phase: 96_oneshot_resilience

All checks complete. Final report:

**Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design)

- `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff
- `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe
- `.env.example` comments updated (chat-turn stream + one-shot summary calls)

**Test / lint / coverage results**
- `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%)
- `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated)
- Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files).

**Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`.

app/rag/folder_summaries.py

@@ -33,6 +33,14 @@ Chat turns never generate folder summaries — the agent's ``ls`` output
 caller's job at sync time (phase 94, task 02), and :func:`generate_
 folder_summaries` only flushes — the sync path owns the transaction
 (the phase-53 ``bump_sources_version`` convention).
+
+Self-heal (phase 96): an exhausted one-shot retry can still leave a
+candidate folder without a row. :func:`missing_folder_summaries`
+names those gaps (the generator's exact candidate computation minus
+the stored keys — one concept, as with :func:`group_by_folder`), and
+:func:`generate_folder_summaries`'s ``only_missing=True`` fills
+EXACTLY those on the next sync — existing rows stay
+byte-identical (text AND ``updated_at``), the prune pass still runs.
 """
 from __future__ import annotations
 
@@ -293,22 +301,85 @@ def _upsert(db: Session, source: str, folder_path: str, summary: str) -> None:
         row.updated_at = now
 
 
-def folder_summary_table_empty(db: Session) -> bool:
-    """Whether ``folder_summaries`` holds no rows (the sync-path gate).
+def _catalog_rows(db: Session) -> list[DocRow]:
+    """The document catalogue in catalogue order (one query).
 
-    Phase 94 (task 02): the ``_overview_row_exists`` pattern
-    (``scripts.import_docs``) extended to a table-emptiness check —
-    after an unchanged re-sync, an EMPTY table (the first full sync
-    after migration 0017, or after a ``--limit`` debug walk that
-    skipped generation) still gets a fresh batch, while a populated
-    table is left untouched until the KB actually changes. One
-    bounded ``LIMIT 1`` probe, never a count scan.
+    ``(source, path, title, summary)`` ordered by ``(source, path)`` —
+    the EXACT query :func:`generate_folder_summaries` runs, so the gap
+detector and the fill can never disagree about what the catalogue
+    contains (phase 96, task 02: one concept, as with
+    :func:`group_by_folder`).
     """
-    return db.execute(select(FolderSummary.source).limit(1)).first() is None
+    result = db.execute(
+        select(Document.source, Document.path, Document.title, Document.summary)
+        .order_by(Document.source, Document.path)
+    ).all()
+    return [
+        (source, path, title, summary)
+        for source, path, title, summary in result
+    ]
+
+
+def _candidates(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow]]:
+    """The generator's candidate map (recursive subtree ≥ 2 docs).
+
+    The :func:`group_by_folder` groups filtered by
+    :data:`MIN_DOCS_PER_FOLDER` — the EXACT set a full regeneration
+    would cover, so a "missing" folder can never disagree with what a
+    regeneration would (re)generate.
+    """
+    groups = group_by_folder(rows)
+    return {
+        key: docs for key, docs in groups.items() if len(docs) >= MIN_DOCS_PER_FOLDER
+    }
+
+
+def _stored_keys(db: Session) -> set[tuple[str, str]]:
+    """The ``(source, folder_path)`` keys that already hold a row.
+
+    One bounded key select — no count scan (the phase-94
+    table-emptiness probe's house pattern).
+    """
+    return {
+        (source, folder_path)
+        for source, folder_path in db.execute(
+            select(FolderSummary.source, FolderSummary.folder_path)
+        ).all()
+    }
+
+
+def missing_folder_summaries(db: Session) -> list[tuple[str, str]]:
+    """The candidate folders whose stored row is ABSENT (phase 96, 02).
+
+    The unchanged-sync self-heal gate: a one-shot reply can still
+    exhaust its retries and leave a candidate folder without a row, and
+    the phase-94 change gate only regenerated on a KB change — so the
+gap persisted until the next KB change. This function names the gap:
+    the generator's candidate folders (the EXACT candidate computation
+    — one catalogue query in catalogue order, :func:`group_by_folder`,
+    recursive subtree ≥ :data:`MIN_DOCS_PER_FOLDER`) minus the stored
+    keys (one bounded select). The gate probe and the
+    :func:`generate_folder_summaries` fill both key off this one
+    concept.
+
+    Returns the missing keys sorted by ``(source, folder_path)``;
+    ``[]`` when there is no gap — including an empty catalogue over an
+    empty table (no candidates, no gap). A single-document folder is
+    never listed (it is not a candidate), and a stored row for a folder
+    that dropped below the minimum is NOT missing (it is stale — the
+    prune pass owns it).
+    """
+    return sorted(
+        key for key in _candidates(_catalog_rows(db)) if key not in _stored_keys(db)
+    )
 
 
 async def generate_folder_summaries(
-    db: Session, llm: FolderSummaryLLM, *, skip: bool = False
+    db: Session,
+    llm: FolderSummaryLLM,
+    *,
+    skip: bool = False,
+    only_missing: bool = False,
 ) -> dict[str, int]:
     """Regenerate the stored folder summaries for the current catalogue.
 
@@ -328,38 +399,45 @@ async def generate_folder_summaries(
        FOLDER: one folder's :class:`LLMError` is logged and counted,
        its previous row (if any) is kept, and the remaining folders
        still land (a ``lite`` outage must never fail the sync — the KB
-       is the product, the summaries are auxiliary).
+       is the product, the summaries are auxiliary). With
+       ``only_missing=True`` the iteration is restricted to the
+       candidates that have NO stored row (the same gap
+       :func:`missing_folder_summaries` reports, derived from the SAME
+       catalogue pass — one bounded stored-key select, no second
+       catalogue query): existing rows stay byte-identical (summary
+       text AND ``updated_at`` — never re-stamped, even a stale-looking
+       one; staleness is the changed-KB regeneration's job) and no
+       ``lite`` call is burned for a folder that already has a summary
+       — the unchanged-sync self-heal fill (phase 96, task 02).
     4. DELETE rows whose folder no longer has ≥ 2 documents — a
        pruned/renamed folder's summary goes stale and is dropped.
        Rows for folders that still qualify persist (regenerated in
-       step 3 — an unchanged folder's summary is still true).
+       step 3 — an unchanged folder's summary is still true). The
+       prune pass runs in BOTH modes: under ``only_missing`` on an
+       unchanged catalogue it is a no-op (the invariant kept), and it
+       still drops rows whose folder fell below the minimum.
 
     Only flushes — the CALLER commits (the phase-53
     ``bump_sources_version`` convention: the sync path owns the
     transaction, so a failed sync rolls the summaries back with it).
 
     Returns the small stats dict ``{"generated", "failed", "pruned"}``
-    for the caller's summary-line logging (PLAN §9 ample logging).
+    for the caller's summary-line logging (PLAN §9 ample logging) —
+    the caller logs the mode, not the generator.
     """
     stats = {"generated": 0, "failed": 0, "pruned": 0}
     if skip:
         return stats
 
-    result = db.execute(
-        select(Document.source, Document.path, Document.title, Document.summary)
-        .order_by(Document.source, Document.path)
-    ).all()
-    rows: list[DocRow] = [
-        (source, path, title, summary)
-        for source, path, title, summary in result
-    ]
-    groups = group_by_folder(rows)
-    candidates = {
-        key: docs for key, docs in groups.items() if len(docs) >= MIN_DOCS_PER_FOLDER
-    }
+    candidates = _candidates(_catalog_rows(db))
+    keys = sorted(candidates)
+    if only_missing:
+        stored = _stored_keys(db)
+        keys = [key for key in keys if key not in stored]
 
-    for source, folder_path in sorted(candidates):
-        docs = candidates[(source, folder_path)]
+    for key in keys:
+        source, folder_path = key
+        docs = candidates[key]
         try:
             summary = await summarize_folder(source, folder_path, docs, llm)
         except LLMError as e:

app/rag/llm.py

@@ -207,6 +207,23 @@ class _TooLarge(RuntimeError):
     """Internal: the endpoint rejected the request's input size."""
 
 
+class _EmptyContentError(RuntimeError):
+    """Internal: the one-shot reply parsed but carries no usable content.
+
+    Raised by :meth:`LLMClient._chat_once` for the retryable failure
+    class (phase 96: ``content`` None or whitespace). Carries the
+    reply's ``finish_reason`` (``None`` when the provider omits it) so
+    :meth:`LLMClient.chat` can log the greppable retry line. Never
+    escapes the client — ``chat()`` converts it to the public
+    :class:`LLMError` (D3). Not an :class:`LLMError` on purpose: callers
+    catching :class:`LLMError` must only ever see final, public errors.
+    """
+
+    def __init__(self, message: str, finish_reason: str | None) -> None:
+        super().__init__(message)
+        self.finish_reason = finish_reason
+
+
 class LLMClient:
     """Thin async wrapper over the aipi OpenAI-compatible API."""
 
@@ -332,24 +349,19 @@ class LLMClient:
         (vec,) = await self.embed([text])
         return vec
 
-    async def chat(
-        self, messages: list[dict[str, Any]], model: str | None = None
-    ) -> str:
-        """One-shot (non-streaming) completion (A5 extended, phase 30).
+    async def _chat_once(self, messages: list[dict[str, Any]], model: str) -> str:
+        """One non-streaming completion attempt (phase 96, private).
 
-        Short, low-temperature request (``temperature=0.2``, 2048-token
-        cap — summaries and outlines are small, so a fixed budget is
-        enough) against ``BOR_LLM_SUMMARY_MODEL`` (default ``lite``)
-        unless *model* names another. Used by the document summarizer
-        (phase 30) and the KB overview generator (phase 31).
-
-        Any transport/HTTP/malformed failure, a choiceless reply, or an
-        empty/missing ``content`` field raises :class:`LLMError` — a
-        silent empty summary must never be stored.
+        The single-attempt body of :meth:`chat` — the transport wrap
+        (→ :class:`LLMError` with the sanitized base URL), the choiceless
+        check, and the empty-content check. Raises
+        :class:`_EmptyContentError` (carrying the reply's
+        ``finish_reason``) for an empty reply — the one failure class
+        :meth:`chat` retries — and :class:`LLMError` for everything else.
         """
         try:
             resp = await self._client.chat.completions.create(
-                model=model or self.settings.llm_summary_model,
+                model=model,
                 messages=cast("list[ChatCompletionMessageParam]", messages),
                 temperature=0.2,
                 max_tokens=2048,
@@ -368,14 +380,81 @@ class LLMClient:
                 f"chat completion from {sanitize_error(self.settings.llm_base_url)} "
                 "returned no choices"
             )
-        content = resp.choices[0].message.content
+        choice = resp.choices[0]
+        content = choice.message.content
         if content is None or not content.strip():
-            raise LLMError(
+            raise _EmptyContentError(
                 f"chat completion from {sanitize_error(self.settings.llm_base_url)} "
-                "returned empty content — refusing to store a silent summary"
+                "returned empty content — refusing to store a silent summary",
+                choice.finish_reason,
             )
         return content.strip()
 
+    async def chat(
+        self, messages: list[dict[str, Any]], model: str | None = None
+    ) -> str:
+        """One-shot (non-streaming) completion (A5 extended, phase 30).
+
+        Short, low-temperature request (``temperature=0.2``, 2048-token
+        cap — summaries and outlines are small, so a fixed budget is
+        enough) against ``BOR_LLM_SUMMARY_MODEL`` (default ``lite``)
+        unless *model* names another. Used by the document summarizer
+        (phase 30), the KB overview generator (phase 31), folder
+        summaries (phase 94), and the pre-sync probe
+        (``check_models``, phase 41).
+
+        Any transport/HTTP/malformed failure, a choiceless reply, or an
+        empty/missing ``content`` field raises :class:`LLMError` — a
+        silent empty summary must never be stored.
+
+        Empty-reply retry (phase 96, house LLM-retry policy D1–D3): the
+        empty-content failure class — the model answered but said
+        nothing (``content`` None or whitespace) — is retried up to
+        ``settings.llm_retries`` times (``BOR_LLM_RETRIES``, default 3;
+        ``0`` = off) with a flat ``settings.llm_retry_delay``
+        (``BOR_LLM_RETRY_DELAY``) between attempts, one WARNING per
+        retry naming the model, the reply's ``finish_reason`` (``None``
+        when the provider omits it), and the attempt count. After
+        ``1 + llm_retries`` empty attempts it raises
+        :class:`LLMError` naming the attempts; with ``llm_retries=0``
+        the single-attempt message stays byte-identical to the
+        pre-phase-96 behavior. NO other failure retries at the app level:
+        a choiceless reply and transport failures raise immediately (the
+        openai SDK's own ``max_retries=2`` already re-POSTs wire-level
+        failures — an app-level transport retry would stack on top of
+        it).
+        """
+        total = 1 + self.settings.llm_retries
+        chosen = model or self.settings.llm_summary_model
+        for attempt in range(1, total + 1):
+            try:
+                return await self._chat_once(messages, chosen)
+            except _EmptyContentError as e:
+                if attempt >= total:
+                    # Retries exhausted — refuse to store the silent
+                    # summary. total == 1 (BOR_LLM_RETRIES=0 kill switch)
+                    # keeps the pre-phase-96 message byte-identical.
+                    if total == 1:
+                        raise LLMError(str(e)) from e
+                    raise LLMError(
+                        f"chat completion from "
+                        f"{sanitize_error(self.settings.llm_base_url)} "
+                        f"returned empty content on all {total} attempts — "
+                        "refusing to store a silent summary"
+                    ) from e
+                logger.warning(
+                    "one-shot LLM reply returned empty content (model=%s, "
+                    "finish_reason=%s, attempt %d/%d) — retrying in %.1fs",
+                    chosen,
+                    e.finish_reason,
+                    attempt,
+                    total,
+                    self.settings.llm_retry_delay,
+                )
+                # Flat delay — the phase-67 convention, no backoff.
+                await asyncio.sleep(self.settings.llm_retry_delay)
+        raise AssertionError("unreachable: every attempt returned or raised")
+
     async def chat_stream(
         self,
         messages: list[dict[str, Any]],