phase: 96_oneshot_resilience
All checks complete. Final report: **Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design) - `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff - `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe - `.env.example` comments updated (chat-turn stream + one-shot summary calls) **Test / lint / coverage results** - `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%) - `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated) - Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files). **Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`.
This commit is contained in:
@@ -14,9 +14,19 @@ DB, explicit ``--source``):
|
||||
- a KB-changing import → one row per ≥ 2-doc subtree (the source root
|
||||
+ the 2-doc folder; the 1-doc folder gets none), committed in the
|
||||
run's transaction, the summary line ending
|
||||
``folder_summaries=<generated>/<failed>/<pruned>``;
|
||||
- an unchanged re-import → zero ``lite`` calls,
|
||||
``folder_summaries=skipped``, rows untouched;
|
||||
``folder_summaries=<generated>/<failed>/<pruned>`` (unchanged by
|
||||
phase 96 — no gap-fill suffix on a full regeneration);
|
||||
- an unchanged re-import with a COMPLETE table → zero ``lite`` calls,
|
||||
``folder_summaries=skipped``, rows untouched (the phase-94
|
||||
zero-burn invariant);
|
||||
- an unchanged re-import with a GAP (one stored row deleted) →
|
||||
exactly one ``FOLDER_SUMMARY_MODE`` call (the missing folder only),
|
||||
the row back with the deterministic fake text, every other row
|
||||
byte-identical (summary AND ``updated_at``), the line ending
|
||||
``folder_summaries=1/0/0 (gap-fill)`` (phase 96, task 03 — the
|
||||
failed folder summary self-heals on the next sync);
|
||||
- a KB change on a second run → still a FULL regeneration (call count
|
||||
== candidate count, every row re-stamped, no gap-fill suffix);
|
||||
- a subtree dropping below 2 docs after a changed re-walk → its row
|
||||
pruned;
|
||||
- one folder's ``lite`` failure → its previous row kept, the other
|
||||
@@ -24,7 +34,9 @@ DB, explicit ``--source``):
|
||||
- a ``--limit`` debug run → no generation, no rows,
|
||||
``folder_summaries=skipped``;
|
||||
- a fresh (empty) table after a ``--limit`` first walk → an unchanged
|
||||
full walk generates (the table-empty first-run trigger).
|
||||
full walk generates via the gap-fill path (the subsumed table-empty
|
||||
first-run trigger — every candidate is missing), the line carrying
|
||||
`` (gap-fill)``.
|
||||
|
||||
API path (``POST /api/sync`` end to end, real import over a host temp
|
||||
local dir, deterministic ``FakeEmbedder``):
|
||||
@@ -32,14 +44,20 @@ local dir, deterministic ``FakeEmbedder``):
|
||||
- a KB-changing sync → the rows land (visible via the test's own
|
||||
session) and the status detail keeps its exact pre-phase key set
|
||||
(no folder-summary surface — the stats are log-only);
|
||||
- an unchanged re-sync → zero ``FOLDER_SUMMARY_MODE`` calls;
|
||||
- an unchanged re-sync (complete table) → zero ``FOLDER_SUMMARY_MODE``
|
||||
calls (the phase-94 zero-burn invariant);
|
||||
- an unchanged re-sync with a GAP (one stored row deleted) → targeted
|
||||
fill of exactly that row (one ``FOLDER_SUMMARY_MODE`` call, the
|
||||
``gap-fill`` log line), every other row byte-identical, the status
|
||||
detail shape untouched (phase 96, task 03);
|
||||
- a ``lite`` outage (one folder failing) → the failed folder's row is
|
||||
kept, the run reports ``success`` (never ``failed``), and the
|
||||
sources-version bump still lands (the bump is change-gated on the
|
||||
KB, not on the summaries);
|
||||
- an empty table after a populated sync (the migration-0017 scenario)
|
||||
→ an unchanged walk regenerates (the overview's API gate, purely
|
||||
change-gated, does not).
|
||||
→ an unchanged walk regenerates via the gap probe (the subsumed
|
||||
table-empty trigger; the overview's API gate, purely change-gated,
|
||||
does not fire).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -290,6 +308,101 @@ def test_unchanged_reimport_burns_zero_folder_calls(
|
||||
assert _rows(db) == rows # rows byte-identical
|
||||
|
||||
|
||||
def test_unchanged_reimport_with_gap_fills_only_the_missing_row(
|
||||
db: Session,
|
||||
src: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""Phase 96 (task 03): an unchanged walk with ONE deleted stored
|
||||
row → exactly one ``FOLDER_SUMMARY_MODE`` call (the deleted
|
||||
folder only), the row back with the deterministic fake text, every
|
||||
OTHER row byte-identical (summary AND ``updated_at``), the summary
|
||||
line ending ``folder_summaries=1/0/0 (gap-fill)`` — the failed
|
||||
folder summary self-heals on the next sync instead of persisting
|
||||
until a KB change."""
|
||||
llm1 = FakeEmbedder()
|
||||
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
|
||||
assert rc == 0
|
||||
rows_before = _rows(db)
|
||||
assert set(rows_before) == {("MyDocs", ""), ("MyDocs", "a")}
|
||||
root_stamp_before = _updated_at(db, "MyDocs", "")
|
||||
assert root_stamp_before is not None
|
||||
|
||||
# Simulate the phase-96 incident: a lost row (an exhausted
|
||||
# one-shot retry leaves a candidate without its row).
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries "
|
||||
"WHERE source = 'MyDocs' AND folder_path = 'a'"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
llm2 = FakeEmbedder()
|
||||
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
|
||||
assert rc == 0
|
||||
assert "unchanged=3" in out
|
||||
assert out.rstrip().endswith(
|
||||
"overview=skipped sources_version=skipped "
|
||||
"folder_summaries=1/0/0 (gap-fill)"
|
||||
)
|
||||
# Exactly ONE new folder call — the deleted row's folder only.
|
||||
calls = _folder_calls(llm2)
|
||||
assert len(calls) == 1
|
||||
assert calls[0][1]["content"].splitlines()[0] == "Folder: MyDocs/a"
|
||||
assert len(llm2.chat_calls) == 1 # no other lite traffic at all
|
||||
# The row is back with the deterministic fake text ...
|
||||
rows_after = _rows(db)
|
||||
assert rows_after == rows_before
|
||||
# ... and every OTHER row byte-identical (the root was never
|
||||
# re-stamped by the targeted fill).
|
||||
assert _updated_at(db, "MyDocs", "") == root_stamp_before
|
||||
|
||||
|
||||
def test_changed_reimport_is_a_full_regeneration(
|
||||
db: Session,
|
||||
src: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""Phase 96 (task 03): a KB change is STILL a full regeneration —
|
||||
call count == candidate count, every row re-stamped, and NO
|
||||
`` (gap-fill)`` suffix (byte-identical to today's behavior)."""
|
||||
llm1 = FakeEmbedder()
|
||||
rc, _ = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
|
||||
assert rc == 0
|
||||
rows_before = _rows(db)
|
||||
root_stamp_before = _updated_at(db, "MyDocs", "")
|
||||
a_stamp_before = _updated_at(db, "MyDocs", "a")
|
||||
assert root_stamp_before is not None and a_stamp_before is not None
|
||||
|
||||
# A KB change (one doc edited) — the gate is the change, not the
|
||||
# gap.
|
||||
(src / "a" / "one.md").write_text("# A One\nChanged content.\n", encoding="utf-8")
|
||||
llm2 = FakeEmbedder()
|
||||
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
|
||||
assert rc == 0
|
||||
assert "updated=1" in out
|
||||
# Full-regeneration token — the stats without the gap-fill suffix.
|
||||
assert out.rstrip().endswith(
|
||||
"overview=updated sources_version=2 folder_summaries=2/0/0"
|
||||
)
|
||||
# Call count == candidate count — BOTH folders, not a targeted fill.
|
||||
calls = _folder_calls(llm2)
|
||||
assert [c[1]["content"].splitlines()[0] for c in calls] == [
|
||||
"Folder: MyDocs",
|
||||
"Folder: MyDocs/a",
|
||||
]
|
||||
# All rows re-stamped (the full regeneration re-writes every
|
||||
# candidate, even the unchanging one).
|
||||
root_stamp_after = _updated_at(db, "MyDocs", "")
|
||||
a_stamp_after = _updated_at(db, "MyDocs", "a")
|
||||
assert root_stamp_after is not None and root_stamp_after > root_stamp_before
|
||||
assert a_stamp_after is not None and a_stamp_after > a_stamp_before
|
||||
assert _rows(db) == rows_before # deterministic fake → same texts
|
||||
|
||||
|
||||
def test_subtree_dropping_below_two_docs_is_pruned(
|
||||
db: Session,
|
||||
src: Path,
|
||||
@@ -382,10 +495,11 @@ def test_empty_table_generates_on_unchanged_walk(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""The table-empty first-run trigger: after a ``--limit`` first
|
||||
walk (populated KB, empty table), an unchanged full walk generates
|
||||
— for the folder summaries AND the missing outline, still never
|
||||
bumping the version."""
|
||||
"""The subsumed table-empty first-run trigger (phase 96, task
|
||||
03): after a ``--limit`` first walk (populated KB, empty table),
|
||||
an unchanged full walk generates — for the folder summaries (now
|
||||
via the gap-fill path — every candidate is missing) AND the
|
||||
missing outline, still never bumping the version."""
|
||||
llm1 = FakeEmbedder()
|
||||
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src), "--limit", "3"], capsys)
|
||||
assert rc == 0
|
||||
@@ -399,8 +513,13 @@ def test_empty_table_generates_on_unchanged_walk(
|
||||
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
|
||||
assert rc == 0
|
||||
assert "unchanged=3" in out
|
||||
# Phase 96 (task 03): the old table-empty trigger is now the
|
||||
# subsumed gap case — every candidate is missing, so the unchanged
|
||||
# walk takes the targeted-fill path and the token carries
|
||||
# `` (gap-fill)`` (the generated set is the full candidate set).
|
||||
assert out.rstrip().endswith(
|
||||
"overview=updated sources_version=skipped folder_summaries=2/0/0"
|
||||
"overview=updated sources_version=skipped "
|
||||
"folder_summaries=2/0/0 (gap-fill)"
|
||||
)
|
||||
assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a")}
|
||||
assert len(_folder_calls(llm2)) == 2
|
||||
@@ -576,6 +695,86 @@ def test_api_unchanged_resync_burns_zero_folder_calls(
|
||||
assert current_sources_version(db) == 1
|
||||
|
||||
|
||||
def test_api_unchanged_resync_with_gap_fills_only_the_missing_row(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
local_dir: Path,
|
||||
) -> None:
|
||||
"""Phase 96 (task 03), API path: an unchanged re-sync with ONE
|
||||
deleted stored row → targeted fill of exactly that row (one
|
||||
``FOLDER_SUMMARY_MODE`` call), the ``gap-fill`` log line, every
|
||||
other row byte-identical (summary AND ``updated_at``), status
|
||||
``success``, and the status detail shape untouched (the stats stay
|
||||
log-only — the phase-94 contract)."""
|
||||
_seed_local(db, local_dir)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(str(local_dir.parent / "bor")),
|
||||
)
|
||||
clients = _capture_llm(monkeypatch)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
_poll(sync_client, "success")
|
||||
rows_before = _rows(db)
|
||||
assert set(rows_before) == {("LocalDocs", ""), ("LocalDocs", "a")}
|
||||
root_stamp_before = _updated_at(db, "LocalDocs", "")
|
||||
assert root_stamp_before is not None
|
||||
|
||||
# The phase-96 incident shape: a lost row, deleted directly.
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries "
|
||||
"WHERE source = 'LocalDocs' AND folder_path = 'a'"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Sink(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
sync_logger = logging.getLogger("app.api.sync")
|
||||
sink = _Sink()
|
||||
sync_logger.addHandler(sink)
|
||||
sync_logger.setLevel(logging.INFO)
|
||||
try:
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
finally:
|
||||
sync_logger.removeHandler(sink)
|
||||
|
||||
assert body["error"] is None
|
||||
assert body["detail"]["overview"] is False # unchanged → no overview
|
||||
assert body["detail"]["sources_version"] == 1 # unchanged → no bump
|
||||
# No new sync-status surface: the detail keeps its exact key set.
|
||||
assert set(body["detail"]) == {
|
||||
"files", "added", "updated", "unchanged", "pruned", "errors",
|
||||
"chunks", "summaries", "summary_errors", "overview",
|
||||
"sources_version",
|
||||
}
|
||||
assert len(clients) == 2
|
||||
# Targeted fill — exactly ONE folder call, the missing folder only
|
||||
# (plus the phase-41 probe's ping on the same client).
|
||||
calls = _folder_calls(clients[1])
|
||||
assert len(calls) == 1
|
||||
assert calls[0][1]["content"].splitlines()[0] == "Folder: LocalDocs/a"
|
||||
assert len(clients[1].chat_calls) == 2 # ping + the one fill
|
||||
# The row is back with the deterministic fake text ...
|
||||
assert _rows(db) == rows_before
|
||||
# ... every other row byte-identical (the root was never re-stamped
|
||||
# by the targeted fill).
|
||||
assert _updated_at(db, "LocalDocs", "") == root_stamp_before
|
||||
# The gap-fill log line (PLAN §9 ample logging).
|
||||
assert any(
|
||||
"sync: folder_summaries gap-fill" in r.getMessage() for r in records
|
||||
)
|
||||
|
||||
|
||||
def test_api_folder_lite_failure_keeps_rows_stays_green_and_bumps(
|
||||
sync_client: TestClient,
|
||||
@@ -634,10 +833,12 @@ def test_api_empty_table_first_sync_regenerates(
|
||||
db: Session,
|
||||
local_dir: Path,
|
||||
) -> None:
|
||||
"""The migration-0017 scenario: the KB predates the table — wipe
|
||||
the rows and re-sync an unchanged KB: the empty-table trigger
|
||||
fires for the folder summaries (the overview's API gate, purely
|
||||
change-gated, does not)."""
|
||||
"""The migration-0017 scenario (the subsumed table-empty trigger —
|
||||
phase 96, task 03): the KB predates the table — wipe the rows and
|
||||
re-sync an unchanged KB: the gap probe fires (every candidate is
|
||||
missing) and the targeted fill regenerates the full candidate set
|
||||
for the folder summaries (the overview's API gate, purely
|
||||
change-gated, does not fire)."""
|
||||
_seed_local(db, local_dir)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
|
||||
Reference in New Issue
Block a user