brain-of-reese/.agents/phases/complete/96_oneshot_resilience/00_phase.md

Phase 96 — One-shot LLM resilience: retry empty replies, self-heal missing folder summaries

Source: 2026-09-11 incident (owner chat) — the then-lite model intermittently returned empty completions: its internal monologue (the wire's reasoning_content field) consumed the entire max_tokens=2048 budget, so the reply arrived as content="" + finish_reason="length". Measured 2 empties in 90 folder-summary calls on the deploy source sync (≈2–7% per call in follow-up probes), losing two stored rows: deploy/Deployments (283 docs) and deploy/Deployments/stackexpected/website/app (7 docs). The app handled it exactly as designed (LLMClient.chat refuses to store a silent summary; generate_folder_summaries is per-folder fail-soft) — but nothing retried, and nothing ever filled the hole: the folder-summary gate (changed KB or empty table) skips regeneration on unchanged re-syncs, so missing rows persist until a KB change. The owner replaced the lite model server-side (it no longer emits reasoning_content — verified live: 16/16 clean, no reasoning_content field); this phase removes the remaining app-side exposure so ANY future transient empty reply from ANY one-shot consumer cannot silently lose data. Story: n/a (incident-driven resilience — one Playwright file per phase, run in isolation, A16). Context:

• app/rag/llm.py — LLMClient.chat() (~L335–378) is the one-shot surface: transport failure → LLMError (wrapped), choiceless reply → LLMError, content None/blank → LLMError("…returned empty content — refusing to store a silent summary"). No retry. Phase 67's retry machinery (BOR_LLM_RETRIES=3 / BOR_LLM_RETRY_DELAY=5 s, flat; chat_stream_retried, RetryPiece) covers the streaming chat-turn path only — phase 67's owner-locked A1 explicitly scoped it to the chat turn and left chat() out ("admin/import paths with their own fail-fast behavior").
• chat() consumers (all fail-soft per call, all protected once chat() retries): app/rag/summarizer.py L113 (document summaries — failure leaves documents.summary NULL + a summary_errors count), app/rag/overview.py L182 (KB overview — failure keeps the old row; its _overview_row_exists gate already self-heals a missing row on the next unchanged walk), app/rag/folder_summaries.py L267 via summarize_folder → generate_folder_summaries (per-folder fail-soft; no self-heal — the gap this phase fills), app/rag/llm.py L641 check_models (the sync-time "ping" probe).
• app/config.py — llm_retries: int = 3, llm_retry_delay: float = 5 (phase 67; validators already enforce the house kill-switch pattern: 0 disables, negative fails startup).
• The openai SDK's own max_retries=2 (kept by LLMClient) already re-POSTs wire-level 500s — the gap is the semantic failure: the endpoint is healthy, answers, and says nothing.
• scripts/import_docs.py _run() (~L335–360) and app/api/sync.py _run_sync() (L276–281) — the folder-summary gate, one copy per path: changed = summary.added + summary.updated > 0; script path folders_due = changed or _folder_summaries_table_empty() (--limit skips entirely); API path if changed or folder_summary_table_empty(fs_db). folder_summary_table_empty (app/rag/folder_summaries.py) is the bounded LIMIT 1 probe.
• app/rag/folder_summaries.py — group_by_folder (the one recursive-subtree concept), MIN_DOCS_PER_FOLDER = 2, generate_folder_summaries (sorted candidate iteration, per-folder fail-soft upsert, then a prune pass deleting rows whose folder lost ≥ 2 docs; flushes only — the sync path owns the transaction).
• tests/unit/test_llm_client.py — the fake-client harness (_FakeCompletion, _make_chat_client, SimpleNamespace completions); existing pins test_chat_missing_content_raises_llm_error / test_chat_whitespace_only_content_raises_llm_error (~L989–1000) match "empty content".
• tests/unit/test_folder_summaries.py — the duck-typed fake LLM with a .calls counter (the generator's unit pattern).
• tests/integration/test_sync_folder_summaries.py + tests/integration/test_import_docs_overview.py — the change-gate integration pattern (fake LLMClient keyed on the mode marker; zero-call assertions on unchanged re-runs).
• tests/e2e/mock_llm.py — compose_answer's FOLDER_SUMMARY_MODE branch (L1504) returns the deterministic one-liner Fixture folder summary for <folder>. from the user message's Folder: … header (imported FOLDER_HEADER_PREFIX — the mock can never drift from it); phase 67's failure-injection machinery: module-level _fail_posts counters + _bump_fail, keyed by trigger phrase, reset after the success they guard.
• tests/e2e/test_ls_tree_drilldown.py (phase 94) — the scripted ls drill-down turn pattern (the mock echoes the tool result into its grounded answer — the E2E's only lens on the LLM context) and the temp-local-source seed + POST /api/sync pattern; tests/e2e/test_llm_retry.py (phase 67) — the test-server env-override pattern (BOR_LLM_RETRY_DELAY=0).

Objective

A transient empty one-shot LLM reply can no longer silently lose data: LLMClient.chat() retries an empty-content reply under the existing BOR_LLM_RETRIES/BOR_LLM_RETRY_DELAY policy (logged per attempt) before raising, and an unchanged-KB sync detects folder-summary gaps (candidate folders with no stored row) and fills only the missing rows — so a failed folder summary self-heals on the next sync instead of persisting until a KB change.

Dependencies

• 67_llm_retry (complete) — the retry policy knobs (BOR_LLM_RETRIES/BOR_LLM_RETRY_DELAY), the logging/kill-switch conventions, and the mock's failure-injection pattern.
• 94_ls_tree_drilldown (complete) — folder_summaries, generate_folder_summaries, the change-gate, the E2E drill-down + sync patterns.

Decisions recorded here (no PLAN.md change needed — no new technology, no new env vars, no new SSE events)

• D1 — knob reuse: the one-shot path honors the existing BOR_LLM_RETRIES (default 3) / BOR_LLM_RETRY_DELAY (default 5 s) — no new settings. Phase 67's owner-locked A3 named these the house LLM-retry policy; extending them to chat() is a deliberate extension of that policy, not a deviation (the lock scoped what phase 67 shipped, it does not prohibit chat() from honoring the same knobs). .env.example comments are updated to say so. (If the owner prefers a separate one-shot knob, task 01 is a one-line change.)
• D2 — retry condition: chat() retries empty content only (content None or whitespace — the exact incident failure class). A choiceless reply still raises immediately (no retry), and transport failures keep today's behavior (the SDK's max_retries=2 already covers wire-level 500s; an app-level transport retry would double-stack the SDK policy and multiply the 300 s timeout).
• D3 — exhaustion contract: after 1 + llm_retries empty attempts, chat() raises the same LLMError type with an updated message naming the attempts (…returned empty content on all {N} attempts — refusing to store a silent summary); with llm_retries=0 the message stays byte-identical to today's (the feature is fully off — phase 67 kill-switch convention). All consumers' fail-soft behavior is untouched (document summary → NULL + count, overview → old row, folder → skip + old row, probe → sync fails loudly as today).
• D4 — targeted gap-fill: generate_folder_summaries gains only_missing=False. The changed-KB path is unchanged (full regeneration + prune). The unchanged-KB gate changes from "table empty?" to "any candidate row missing?" (subsumes the table-empty first-run case exactly); when a gap exists it runs only_missing=True — only the missing candidates are generated, existing rows stay byte-identical (text AND updated_at), and the prune pass still runs (a no-op on an unchanged KB, the invariant kept). --limit (script only) still skips generation entirely.
• Non-goals: the streaming path (chat_stream / chat_stream_retried) is untouched (phase 67 territory); no documents.summary-NULL catch-up (the retry eliminates the failure class — expected per-doc loss p⁴; the importer's unchanged-skip would be a separate, larger change); no KB-overview catch-up (its _overview_row_exists gate already self-heals); no UI/SSE/sync-status surface change (folder stats stay log-only, phase 94 contract); no aipi/model-side change (owner already done).

Design (shared by all tasks — the executor reads this, not the chat)

The one-shot retry (task 01)

app/rag/llm.py LLMClient:

• Extract the single-attempt body of chat() into a private _chat_once(messages, model) -> str (the existing try/except transport wrap + choiceless check + empty-content check, verbatim behavior). chat() becomes: attempt 1 via _chat_once; on the empty-content LLMError only, while attempts remain: logger.warning (PLAN §9: model, finish_reason of the empty reply when available, attempt n of N) + await asyncio.sleep(settings.llm_retry_delay) (flat, phase 67) + retry via _chat_once. Any other LLMError propagates immediately.
• N = 1 + settings.llm_retries total attempts. Exhaustion → the D3 message. llm_retries=0 → one attempt, the byte-identical legacy message.
• The empty-content check must capture the reply's finish_reason for the log line (the incident's signature was finish_reason="length" — it makes the warning greppable and self-explaining).
• chat() docstring: the phase-30 "a silent empty summary must never be stored" contract stays, extended with the retry policy (D1/D2/D3). .env.example: the two phase-67 comments gain "(chat-turn stream and one-shot summary calls)".
• check_models benefits automatically (no change).

The gap detector + targeted fill (task 02)

app/rag/folder_summaries.py:

• missing_folder_summaries(db) -> list[tuple[str, str]] — the candidate folders (the generator's exact candidate computation: one catalog query ordered by (source, path), group_by_folder, subtrees with ≥ MIN_DOCS_PER_FOLDER) that have no stored row, sorted by (source, folder_path). The gate probe and the fill both use this one function — one concept, as with group_by_folder.
• generate_folder_summaries(db, llm, *, skip=False, only_missing=False) — only_missing=True restricts the sorted candidate iteration to the keys from missing_folder_summaries(db) (computed from the SAME catalog pass — no second full query inside the generator: derive the missing set from the candidates already grouped). Upsert, fail-soft, and the prune pass are otherwise unchanged; stats dict shape unchanged (the caller logs the mode).
• folder_summary_table_empty is deleted in task 03 (both call sites replaced — the function becomes dead code; its unit tests move to missing_folder_summaries).

The sync gates (task 03)

Both paths, identical shape (the phase-94 "one gate, two call sites" convention):

• Changed KB → generate_folder_summaries(session, llm) — full regeneration (today's behavior, byte-identical).
• Unchanged KB → missing = missing_folder_summaries(session); non-empty → generate_folder_summaries(session, llm, only_missing=True) + a log line naming the gap-fill (script: the summary line token becomes folder_summaries=<g>/<f>/<p> (gap-fill); API: sync: folder_summaries gap-fill stats=…); empty → today's skip log.
• --limit (script) → unchanged full skip. Transaction convention unchanged (generator flushes, the path commits).

The E2E (task 04)

tests/e2e/mock_llm.py — the FOLDER_SUMMARY_MODE branch gains the incident-shape injection, keyed by the folder LABEL (the seeded KB's folder names carry the trigger — deterministic, no env, no request-order state beyond the phase-67 per-key counter, reset after the success it guards):

• label ends with /e2e_empty_once → the FIRST non-stream POST for that label returns the incident envelope: same OpenAI shape, message.content="", finish_reason="length"; later POSTs return the normal Fixture folder summary for <folder>. line.
• label ends with /e2e_empty_always → every non-stream POST for that label returns the empty envelope. tests/e2e/test_oneshot_llm_retry.py (new; app_server + mock_llm conftest fixtures; admin login via tests/e2e/auth_helpers.py; sync via POST /api/sync per test_sync_button.py; the test server boots with BOR_LLM_RETRY_DELAY=0 per the phase-67 pattern; default BOR_LLM_RETRIES=3):

1. Seed a temp local-dir source with three ≥ 2-doc folders: e2e_empty_once/, e2e_empty_always/, normal/.
2. Sync #1 → a scripted ls <source> drill-down turn (the phase-94 echo pattern) asserts: the e2e_empty_once/ line carries : Fixture folder summary for <src>/e2e_empty_once. (the retry recovered the row — without task 01 it would be absent), the normal/ line does too, and the e2e_empty_always/ line has NO : … suffix (exhausted → fail-soft → absent) while the sync still reports success (the folder-stats failure never flips the run, phase 94).
3. Delete the normal stored row directly (simulating a historical failure) → Sync #2 (KB unchanged) → the normal row is back with its deterministic text; the e2e_empty_once row's updated_at is unchanged (targeted fill — no full regeneration); e2e_empty_always remains absent (the gap-fill attempts it, exhausts, stays absent, sync still succeeds).

Tasks

1. 01_chat_retry.md — chat() empty-content retry under BOR_LLM_RETRIES/BOR_LLM_RETRY_DELAY + logging + unit pins
2. 02_gapfill_generator.md — missing_folder_summaries + generate_folder_summaries(only_missing=…) + unit pins
3. 03_sync_gates.md — the gap gate in scripts/import_docs.py + app/api/sync.py (replacing the table-empty probe) + integration pins
4. 04_e2e_oneshot_retry.md — the mock's incident-shape injection + tests/e2e/test_oneshot_llm_retry.py + regressions + commit

Testing & Quality

• Unit: tests/unit/test_llm_client.py (retry matrix: empty-then-success = 2 attempts + 1 sleep; all-empty = 1 + retries attempts + the exhaustion message; first-success = 1 attempt, 0 sleeps, byte-identical return; choiceless reply = no retry, immediate error; transport error = no app-level retry; llm_retries=0 = legacy byte-identical message; sleep value honored — record asyncio.sleep); tests/unit/test_folder_summaries.py (missing_folder_summaries shapes: fresh table → all candidates, full table → empty, one row deleted → that folder; only_missing=True → only missing keys generated, existing rows byte-identical incl. updated_at, zero LLM calls when there is no gap, prune still runs; only_missing=False unchanged).
• Integration: tests/integration/test_sync_folder_summaries.py + the test_import_docs_overview.py pattern (both sync paths: unchanged + gap → targeted fill of exactly the missing rows; unchanged + no gap → zero FOLDER_SUMMARY_MODE calls; changed → full regeneration as today; --limit skips).
• E2E (mandatory, A16): tests/e2e/test_oneshot_llm_retry.py, in isolation, deterministic mock, BOR_LLM_RETRY_DELAY=0.
• Coverage: >90% on app/ (uv run pytest --cov=app --cov-report=term-missing).
• Lint/types: uv run ruff check . && uv run pyright.

Completion Criteria

• a one-shot chat() call whose first reply is empty content (None or whitespace) is retried up to BOR_LLM_RETRIES times with BOR_LLM_RETRY_DELAY between attempts, one WARNING per retry (model + finish_reason + attempt count), and succeeds when a later attempt returns content (unit-pinned)
• an all-empty chat() raises LLMError naming the attempts after exactly 1 + BOR_LLM_RETRIES attempts; BOR_LLM_RETRIES=0 reproduces today's single-attempt behavior and message byte-identically
• the streaming path is untouched: test_llm_retry.py (phase 67) green in isolation
• an unchanged-KB sync with a missing folder-summary row regenerates exactly that row (both scripts/import_docs.py and POST /api/sync), leaves every other row byte-identical (incl. updated_at), and logs the gap-fill (integration-pinned)
• an unchanged-KB sync with no gap burns zero lite folder-summary calls (the phase-94 zero-burn invariant holds)
• uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov green in isolation: the e2e_empty_once folder summary is recovered by the retry (visible in the ls drill-down), e2e_empty_always stays absent with the sync green, and the deleted-row gap self-heals on the next unchanged sync
• regression E2E green in isolation: test_ls_tree_drilldown.py, test_sync_button.py, test_local_directory_sources.py, test_llm_retry.py
• uv run pytest green, coverage >90%, uv run ruff check . && uv run pyright clean
• no behavior change in completed phases; one atomic Conventional Commit, --no-gpg-sign

Commit

git add -A .agents/ app/ scripts/ tests/ && git commit --no-gpg-sign -m "fix(rag): retry empty one-shot LLM replies and self-heal missing folder summaries on unchanged syncs"