phase: 96_oneshot_resilience
All checks complete. Final report: **Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design) - `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff - `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe - `.env.example` comments updated (chat-turn stream + one-shot summary calls) **Test / lint / coverage results** - `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%) - `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated) - Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files). **Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`.
This commit is contained in:
+131
-1
@@ -385,7 +385,38 @@ test_llm_retry.py``). The mock is single-conversation per e2e server, so
|
||||
SDK-level retries), so one POST per attempt: the counter is per
|
||||
POST here, unlike the chat counter above.
|
||||
Non-streaming requests (document summaries, KB overview) never 500 —
|
||||
the retry scope is the chat turn only (owner-locked A1).
|
||||
the retry scope is the chat turn only (owner-locked A1). The one
|
||||
non-streaming injection is phase 96's incident shape below (it
|
||||
answers 200 with EMPTY content — the semantic failure class, not a
|
||||
dead endpoint).
|
||||
|
||||
Failure injection (phase 96, one-shot resilience, task 04) — the
|
||||
2026-09-11 incident shape for the folder-summary one-shot path
|
||||
(``tests/e2e/test_oneshot_llm_retry.py``): a NON-stream
|
||||
``chat/completions`` request whose system prompt carries
|
||||
``FOLDER_SUMMARY_MODE`` (the folder-summary marker — ``chat()`` is
|
||||
the mock's only non-streaming consumer of it) and whose user
|
||||
message's ``Folder: …`` header (the branch's existing parse, the
|
||||
``FOLDER_HEADER_PREFIX`` tail) labels this suite's own fixture
|
||||
folders:
|
||||
- the label ends with ``/e2e_empty_once``: the FIRST non-stream POST
|
||||
for that label answers the incident envelope — the mock's normal
|
||||
OpenAI chat-completion shape with ``choices[0].message.content =
|
||||
""`` and ``choices[0].finish_reason = "length"`` (the exact wire
|
||||
shape of the empty ``lite`` reply: the budget spent in
|
||||
``reasoning_content``) — and every later POST returns the normal
|
||||
``Fixture folder summary for <folder>.`` line (the one-shot retry,
|
||||
phase 96 task 01, recovers the row).
|
||||
- the label ends with ``/e2e_empty_always``: EVERY non-stream POST
|
||||
for that label answers the empty envelope (the 1 +
|
||||
``BOR_LLM_RETRIES`` exhaustion → per-folder fail-soft → the row
|
||||
stays absent while the sync stays green, phase 94 contract).
|
||||
The once-sequence is driven by a module-level per-label counter that
|
||||
resets after the success it guards (the phase-67 ``_fail_posts``
|
||||
pattern — the mock is single-conversation per e2e server), so a
|
||||
second sync re-drives the sequence deterministically. The trigger
|
||||
strings are this suite's own folder names, so no other E2E can hit
|
||||
them (they seed different trees).
|
||||
|
||||
``max_tokens`` is honored deterministically (token ≈ whitespace word),
|
||||
like a real endpoint: an answer longer than the cap is truncated. This
|
||||
@@ -1496,6 +1527,70 @@ def _history_echo(body: dict[str, Any]) -> str:
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 96 (task 04, one-shot resilience): the incident-shape injection
|
||||
# for the folder-summary one-shot path — see the module docstring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: The folder-label triggers (phase 96, task 04): the ``FOLDER_SUMMARY_MODE``
|
||||
#: branch's folder label (the user message's ``Folder: …`` tail — the
|
||||
#: ``FOLDER_HEADER_PREFIX`` parse) ending with these suffixes is THIS
|
||||
#: SUITE'S own fixture folder (``tests/e2e/test_oneshot_llm_retry.py``
|
||||
#: seeds the names — no other E2E can hit them; they seed different
|
||||
#: trees, and the triggers carry the E2E prefix).
|
||||
ONESHOT_EMPTY_ONCE_SUFFIX = "/e2e_empty_once"
|
||||
ONESHOT_EMPTY_ALWAYS_SUFFIX = "/e2e_empty_always"
|
||||
|
||||
#: Module-level per-label incident counter — the mock is single-
|
||||
#: conversation per e2e server (the phase-67 ``_fail_posts``
|
||||
#: convention). Counts the non-stream folder-summary POSTs served per
|
||||
#: trigger label; the once-sequence resets after the success it guards
|
||||
#: (the first normal reply), so a second sync re-drives the sequence
|
||||
#: deterministically.
|
||||
_empty_once_posts: dict[str, int] = {}
|
||||
|
||||
|
||||
def _folder_summary_incident(body: dict[str, Any]) -> bool:
|
||||
"""Should this NON-stream folder-summary POST answer with the
|
||||
2026-09-11 incident envelope (``content=""`` +
|
||||
``finish_reason="length"`` — the exact wire shape of the empty
|
||||
``lite`` reply phase 96's one-shot retry targets)?
|
||||
|
||||
* the system prompt lacks ``FOLDER_SUMMARY_MODE`` → never (only the
|
||||
folder-summary one-shot carries the marker — ``chat()`` is the
|
||||
mock's only non-streaming consumer of it, so the counter counts
|
||||
exactly the one-shot POSTs the app's retry policy drives);
|
||||
* the label ends with ``/e2e_empty_always`` → EVERY POST (the
|
||||
exhaustion path — 1 + ``BOR_LLM_RETRIES`` empty attempts, the
|
||||
per-folder fail-soft leaves the row absent);
|
||||
* the label ends with ``/e2e_empty_once`` → the FIRST non-stream
|
||||
POST for that label only — every later POST returns the normal
|
||||
line (the retry recovers the row), and the counter resets on
|
||||
that first normal reply (the phase-67 pattern).
|
||||
|
||||
The folder label is the ``FOLDER_SUMMARY_MODE`` branch's existing
|
||||
parse (the user message's first line, the ``FOLDER_HEADER_PREFIX``
|
||||
tail) — the injection is a pure function of the request plus the
|
||||
per-label counter (the house marker-flow convention).
|
||||
"""
|
||||
if "FOLDER_SUMMARY_MODE" not in _system(body):
|
||||
return False
|
||||
user = _user(body)
|
||||
header = user.splitlines()[0] if user else ""
|
||||
if not header.startswith(FOLDER_HEADER_PREFIX):
|
||||
return False
|
||||
label = header.removeprefix(FOLDER_HEADER_PREFIX).strip()
|
||||
if label.endswith(ONESHOT_EMPTY_ALWAYS_SUFFIX):
|
||||
return True
|
||||
if label.endswith(ONESHOT_EMPTY_ONCE_SUFFIX):
|
||||
n = _empty_once_posts.get(label, 0) + 1
|
||||
_empty_once_posts[label] = n
|
||||
if n == 1:
|
||||
return True # the first POST: the incident envelope
|
||||
_empty_once_posts[label] = 0 # the retry went out — restart
|
||||
return False
|
||||
|
||||
|
||||
def compose_answer(body: dict[str, Any]) -> str:
|
||||
system = _system(body)
|
||||
user = _user(body)
|
||||
@@ -1516,6 +1611,12 @@ def compose_answer(body: dict[str, Any]) -> str:
|
||||
# shadow every folder-summary call. Checked BEFORE the
|
||||
# DEFLECT_MODE branch, like the other lite-mode markers (a
|
||||
# deflection prompt never carries one).
|
||||
# Phase 96 (task 04): the incident-shape injection keys on THIS
|
||||
# branch's label (``_folder_summary_incident`` — the module
|
||||
# docstring) and overrides the NON-STREAM response envelope in
|
||||
# ``chat_completions`` (``content=""`` +
|
||||
# ``finish_reason="length"``); the line below is what the later
|
||||
# / normal POSTs return.
|
||||
header = user.splitlines()[0] if user else ""
|
||||
folder = (
|
||||
header.removeprefix(FOLDER_HEADER_PREFIX).strip()
|
||||
@@ -2167,6 +2268,35 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
)
|
||||
|
||||
if not body.get("stream"):
|
||||
# Phase 96 (task 04): the incident-shape injection (the module
|
||||
# docstring) — the trigger-labelled folder summary answers the
|
||||
# exact 2026-09-11 envelope: the mock's normal OpenAI
|
||||
# chat-completion shape with ``content=""`` and
|
||||
# ``finish_reason="length"`` (the budget spent in
|
||||
# ``reasoning_content``). ``chat()`` is the mock's only
|
||||
# non-streaming folder-summary consumer, so this is the one-shot
|
||||
# retry path (``app.rag.llm.LLMClient.chat``, phase 96 task 01)
|
||||
# and nothing else — every other non-stream response is
|
||||
# byte-identical to pre-phase-96.
|
||||
if _folder_summary_incident(body):
|
||||
return {
|
||||
"id": f"chatcmpl-{uuid.uuid4()}",
|
||||
"object": "chat.completion",
|
||||
"created": int(time.time()),
|
||||
"model": body.get("model", "turbo"),
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {"role": "assistant", "content": ""},
|
||||
"finish_reason": "length",
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 100,
|
||||
"completion_tokens": 2048,
|
||||
"total_tokens": 2148,
|
||||
},
|
||||
}
|
||||
message: dict[str, Any] = {"role": "assistant", "content": answer}
|
||||
if thinking:
|
||||
# Harmless future-proofing: the app only uses streaming, but a
|
||||
|
||||
@@ -0,0 +1,633 @@
|
||||
"""Phase 96 task 04 E2E (Playwright, mock-only): the one-shot LLM
|
||||
resilience — the 2026-09-11 incident shape, retried and healed.
|
||||
|
||||
The dedicated story suite for ``96_oneshot_resilience`` (A16 — one
|
||||
Playwright file per phase, run in isolation): a folder whose FIRST
|
||||
one-shot summary reply arrives in the exact incident shape
|
||||
(``content=""`` + ``finish_reason="length"``) still ends up with its
|
||||
stored summary (the task-01 retry recovered it, visible in the
|
||||
``ls`` drill-down), a folder whose replies are ALWAYS empty stays
|
||||
absent without failing the sync (the task-01 exhaustion + the phase-94
|
||||
per-folder fail-soft contract), and a row deleted behind the app's back
|
||||
is self-healed by the next UNCHANGED sync with the other rows untouched
|
||||
(tasks 02/03 targeted fill).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic incident-shape injection in ``tests/e2e/mock_llm.py``
|
||||
(phase 96, task 04): a NON-stream ``chat/completions`` request whose
|
||||
system prompt carries ``FOLDER_SUMMARY_MODE`` and whose ``Folder: …``
|
||||
label ends with ``/e2e_empty_once`` answers the incident envelope
|
||||
(``content=""`` + ``finish_reason="length"`` — the mock's normal OpenAI
|
||||
shape) on its FIRST non-stream POST only, and a label ending with
|
||||
``/e2e_empty_always`` answers it on EVERY non-stream POST. The
|
||||
per-label counter resets after the success it guards (the phase-67
|
||||
``_fail_posts`` pattern), so a re-run of the suite is green without
|
||||
manual state cleanup. The trigger strings are this suite's own folder
|
||||
names, so no other E2E can hit them.
|
||||
|
||||
The e2e app server boots with ``BOR_LLM_RETRY_DELAY=0`` (the conftest
|
||||
pattern) so the 4-attempt exhaustion paths are instant;
|
||||
``BOR_LLM_RETRIES`` is forced to the code default (3 — the conftest
|
||||
leak-guard pattern), so ``e2e_empty_always`` costs exactly 4 POSTs per
|
||||
sync that attempts it and the suite stays fast.
|
||||
|
||||
KB fixture — a host temp dir tree (``tmp_path_factory``; the app runs
|
||||
on the same host) with ONE registered local source (the
|
||||
``test_local_directory_sources.py`` registration + real-Sync pattern;
|
||||
no git anywhere): ``oneshot/`` with three ≥ 2-doc folders —
|
||||
``e2e_empty_once/`` (2 docs), ``e2e_empty_always/`` (2), ``normal/``
|
||||
(2). Every fixture doc carries the words ``drill down the tree`` in
|
||||
its body, so the scripted ``ls`` drill-down questions (the phase-94
|
||||
``DRILL_TRIGGER`` echo pattern — the mock echoes the received tool
|
||||
result into its grounded answer, the E2E's only lens on the LLM's
|
||||
context) FTS-match at least one chunk and run grounded.
|
||||
|
||||
Test → phase mapping (Playwright Mapping Rule):
|
||||
1. ``test_incident_reply_retried_and_always_empty_stays_absent`` —
|
||||
after the changed sync #1 (the module fixture pins the stored rows:
|
||||
the ``e2e_empty_once`` row EXISTS — the retry recovered it, without
|
||||
task 01 it would be absent — and the ``e2e_empty_always`` row does
|
||||
NOT — all 4 attempts empty → ``LLMError`` → per-folder fail-soft),
|
||||
a scripted ``ls oneshot`` turn asserts the drill-down listing: the
|
||||
``e2e_empty_once/`` line carries
|
||||
``: Fixture folder summary for oneshot/e2e_empty_once.`` (the retry
|
||||
recovered the row), the ``normal/`` line carries its summary, and
|
||||
the ``e2e_empty_always/`` line is ``e2e_empty_always/ — 2
|
||||
documents`` with NO ``: …`` suffix — while the sync reported
|
||||
success (a folder-summary exhaustion never flips the run, phase 94).
|
||||
2. ``test_deleted_row_self_heals_on_unchanged_sync`` — the ``normal``
|
||||
stored row is deleted directly (simulating a historical failure),
|
||||
``e2e_empty_once``'s row ``updated_at`` is captured, and the
|
||||
UNCHANGED sync #2 runs the gap gate (task 03): a second scripted
|
||||
``ls oneshot`` turn shows ``normal/`` healed
|
||||
(``: Fixture folder summary for oneshot/normal.`` again) and
|
||||
``e2e_empty_always/`` still absent (the gap-fill attempted it,
|
||||
exhausted, stayed absent — the sync still succeeded), and the DB
|
||||
pins the targeted fill: the ``normal`` row is back with its
|
||||
deterministic text and the ``e2e_empty_once`` row's ``updated_at``
|
||||
is UNCHANGED (a full regeneration would have re-stamped it).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import FolderSummary
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module app binds its own port instead
|
||||
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
# server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_ONESHOT", "8138"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture documents + the pinned drill-down listing (deterministic)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: The local source — the temp directory's basename (``kind=local`` →
|
||||
#: the directory's basename is the source name, phase 38).
|
||||
SOURCE = "oneshot"
|
||||
|
||||
#: The three ≥ 2-doc folders (the mock's trigger labels are the folder
|
||||
#: NAMES — the seeded KB carries them, the house marker-flow
|
||||
#: convention). Path order (the ``ls`` subfolder order):
|
||||
#: e2e_empty_always < e2e_empty_once < normal.
|
||||
FOLDER_ONCE = "e2e_empty_once"
|
||||
FOLDER_ALWAYS = "e2e_empty_always"
|
||||
FOLDER_NORMAL = "normal"
|
||||
|
||||
TOTAL_DOCS = 6 # three folders x 2 docs each
|
||||
|
||||
#: Every fixture body carries ``drill down the tree`` (the
|
||||
#: ``DRILL_TRIGGER`` phrase's words): every scripted question
|
||||
#: FTS-matches at least one chunk → HIGH gate → the ``<tools>`` section
|
||||
#: the drill-down flow keys on (the phase-94 seed convention).
|
||||
DRILL_LEAD = "The drill down the tree fixture note"
|
||||
|
||||
|
||||
def _md(title: str, body: str) -> str:
|
||||
return f"# {title}\n\n{body}\n"
|
||||
|
||||
|
||||
#: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this
|
||||
#: fixture (the phase-94 template — the label is the
|
||||
#: ``FOLDER_HEADER_PREFIX`` tail: ``<source>`` for the root,
|
||||
#: ``<source>/<folder>`` for a folder).
|
||||
SUM_ROOT = f"Fixture folder summary for {SOURCE}."
|
||||
SUM_ONCE = f"Fixture folder summary for {SOURCE}/{FOLDER_ONCE}."
|
||||
SUM_ALWAYS = f"Fixture folder summary for {SOURCE}/{FOLDER_ALWAYS}."
|
||||
SUM_NORMAL = f"Fixture folder summary for {SOURCE}/{FOLDER_NORMAL}."
|
||||
|
||||
# --- the pinned ``ls oneshot`` level (app.rag.agent's phase-94 template)
|
||||
|
||||
#: The source root: 0 direct files, 3 subfolders (path order).
|
||||
LS_HEADER = f"{SOURCE} — 0 documents, 3 folders:"
|
||||
|
||||
#: The subfolder lines — the ``: {summary}`` suffix appended ONLY when
|
||||
#: the subfolder's summary is stored (``render_folder_listing``; the
|
||||
#: renderer's 2-space indent is whitespace-normalized away by the
|
||||
#: ``to_contain_text`` match — the phase-94 pin convention).
|
||||
LINE_ALWAYS = f"{FOLDER_ALWAYS}/ — 2 documents"
|
||||
LINE_ALWAYS_WITH_COLON = f"{FOLDER_ALWAYS}/ — 2 documents: "
|
||||
LINE_ONCE = f"{FOLDER_ONCE}/ — 2 documents: {SUM_ONCE}"
|
||||
LINE_NORMAL = f"{FOLDER_NORMAL}/ — 2 documents: {SUM_NORMAL}"
|
||||
|
||||
# --- the scripted drill-down turns (the phase-94 ``DRILL_TRIGGER``
|
||||
# questions — the mock echoes the listing verbatim into the answer) ---
|
||||
|
||||
LS_QUESTION_1 = f"Drill down the tree: ls {SOURCE} — what's in source {SOURCE}?"
|
||||
LS_QUESTION_2 = f"Drill down the tree: ls {SOURCE} — list the source folders again"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def oneshot_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The one-source temp tree (see the module docstring): the app
|
||||
server runs on the same host, so the paths are visible to it."""
|
||||
root = tmp_path_factory.mktemp("bor_oneshot")
|
||||
src = root / SOURCE
|
||||
for folder, prefix in (
|
||||
(FOLDER_ONCE, "once"),
|
||||
(FOLDER_ALWAYS, "always"),
|
||||
(FOLDER_NORMAL, "normal"),
|
||||
):
|
||||
(src / folder).mkdir(parents=True)
|
||||
for letter, topic in (("a", "A"), ("b", "B")):
|
||||
(src / folder / f"{prefix}-{letter}.md").write_text(
|
||||
_md(
|
||||
f"Oneshot {prefix.title()} {letter.upper()}",
|
||||
f"{DRILL_LEAD} for {SOURCE} {folder} "
|
||||
f"{letter}: this document covers topic {topic} "
|
||||
f"of the {SOURCE} source tree.",
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert len(list(src.rglob("*.md"))) == TOTAL_DOCS
|
||||
return src
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, oneshot_dir: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module app (the conftest pattern,
|
||||
cf. ``test_local_directory_sources.py``): NO ``BOR_GIT_SOURCES``
|
||||
(the env fallback is git-only — the source here is a DB-registered
|
||||
local directory), the mock LLM, the mock-calibrated threshold, and
|
||||
the leak-guarded code defaults. ``BOR_LLM_RETRY_DELAY=0`` + the
|
||||
code-default ``BOR_LLM_RETRIES`` (the phase-67 conftest pattern):
|
||||
the one-shot retry waits are instant and the exhaustion budget is
|
||||
the REAL one (4 attempts). The session app is never started in this
|
||||
isolated run, so no port clash."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated threshold (conftest pattern): every scripted
|
||||
# question FTS-matches the fixture docs (the ``drill down the
|
||||
# tree`` words), so the gate is HIGH either way.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
# Phase 67 / phase 96: instant retry waits + the code-default budget
|
||||
# (the conftest leak-guard pattern) — the one-shot retry and the
|
||||
# 4-attempt exhaustion run in real time at zero delay.
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the registry must hold EXACTLY the one local
|
||||
# directory this suite registers.
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
||||
share one Postgres, so a leftover source or document would pollute
|
||||
the ``ls`` listing the drill answers assert on byte-exactly."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _run_sync_http(base_url: str, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Login + ``POST /api/sync`` + poll the status endpoint until the
|
||||
run reaches a terminal state (the ``test_sync_button.py`` /
|
||||
``test_local_directory_sources.py`` pattern, over plain httpx)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _folder_rows() -> dict[str, tuple[str, datetime]]:
|
||||
"""The source's stored folder summaries
|
||||
``{folder_path: (summary, updated_at)}`` (``""`` = the source
|
||||
root) — the test process's direct DB access, the E2E's other
|
||||
established lens."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(
|
||||
select(
|
||||
FolderSummary.folder_path,
|
||||
FolderSummary.summary,
|
||||
FolderSummary.updated_at,
|
||||
).where(FolderSummary.source == SOURCE)
|
||||
).all()
|
||||
return {folder: (summary, updated_at) for folder, summary, updated_at in rows}
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, oneshot_dir: Path) -> None:
|
||||
"""The story's precondition: the KB synced under the deterministic
|
||||
mock's incident-shape injection.
|
||||
|
||||
Registers the temp directory through the authenticated API (the
|
||||
``test_local_directory_sources.py`` pattern) and runs the REAL
|
||||
in-process sync #1 (``POST /api/sync`` — walk → chunk → embed →
|
||||
overview → folder summaries → version bump). The sync changed the
|
||||
KB → FULL folder regeneration, and the mock's injection drives the
|
||||
incident shapes:
|
||||
|
||||
* ``oneshot`` (the root) + ``oneshot/normal`` — normal replies,
|
||||
one POST each;
|
||||
* ``oneshot/e2e_empty_once`` — the FIRST non-stream POST answers
|
||||
the incident envelope (``content=""`` + ``finish_reason=
|
||||
"length"``), the retry (task 01) recovers the row on the second
|
||||
POST;
|
||||
* ``oneshot/e2e_empty_always`` — EVERY non-stream POST answers the
|
||||
empty envelope: 1 + ``BOR_LLM_RETRIES`` = 4 attempts exhausted →
|
||||
``LLMError`` → the phase-94 per-folder fail-soft leaves the row
|
||||
ABSENT and never flips the run.
|
||||
|
||||
The fixture pins all of that in the DB: the ``e2e_empty_once`` row
|
||||
EXISTS (without task 01 it would be absent — the incident's data
|
||||
loss), the ``e2e_empty_always`` row does NOT, and the sync still
|
||||
reported ``success``.
|
||||
"""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(oneshot_dir)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
body = _run_sync_http(app_server)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == TOTAL_DOCS, detail
|
||||
assert detail["updated"] == 0, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is True, detail
|
||||
# The full regeneration under the injection: the retried row
|
||||
# EXISTS (the incident, healed by task 01's one-shot retry), the
|
||||
# exhausted row is ABSENT (the per-folder fail-soft — the sync
|
||||
# above still reported success), the normal rows landed.
|
||||
rows = _folder_rows()
|
||||
assert set(rows) == {"", FOLDER_ONCE, FOLDER_NORMAL}, rows
|
||||
assert rows[""][0] == SUM_ROOT, rows
|
||||
assert rows[FOLDER_ONCE][0] == SUM_ONCE, rows
|
||||
assert rows[FOLDER_NORMAL][0] == SUM_NORMAL, rows
|
||||
assert FOLDER_ALWAYS not in rows, rows # all 4 attempts empty → no row
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||||
the drill turns never change it, so the folder summaries and the
|
||||
registry persist across the tests of this module)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the phase-94 drill-down house pattern)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` frames, independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page) -> list[dict]:
|
||||
"""The SSE frames captured since the last submit (``_submit``
|
||||
clears the buffer), once the hook's background read settles."""
|
||||
deadline = time.monotonic() + 30.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered
|
||||
(the phase-48 settle wait)."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=60_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=60_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=60_000)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
def _assert_drill_turn(
|
||||
page: Page,
|
||||
expected_tool: dict[str, Any],
|
||||
expected_lines: list[str],
|
||||
forbidden: list[str] | None = None,
|
||||
) -> None:
|
||||
"""One scripted drill turn, fully asserted: the wire carries exactly
|
||||
the expected ``tool`` frame (ahead of the first ``delta``), the
|
||||
bubble carries the expected listing lines (the mock's echo of the
|
||||
tool result the model received), the forbidden substrings are ABSENT
|
||||
(the no-suffix assertions), and the turn was grounded (the ``done``
|
||||
frame is not deflected)."""
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [expected_tool], _tool_frames(frames)
|
||||
first_delta = next(
|
||||
i for i, f in enumerate(frames) if f.get("type") == "delta"
|
||||
)
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
for line in expected_lines:
|
||||
expect(bubble).to_contain_text(line)
|
||||
text = bubble.text_content() or ""
|
||||
for needle in forbidden or []:
|
||||
assert needle not in text, text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. Sync #1: the incident shape is retried (the row lands) and the
|
||||
# always-empty folder stays absent with the sync green
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_incident_reply_retried_and_always_empty_stays_absent(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The ``ls oneshot`` drill-down shows the stored folder summaries
|
||||
the sync #1 produced under the incident-shape injection: the
|
||||
``e2e_empty_once/`` line carries its summary (the retry recovered
|
||||
the row — without task 01 the line would have NO ``: …`` suffix),
|
||||
the ``normal/`` line does too, and the ``e2e_empty_always/`` line
|
||||
is bare (exhausted → fail-soft → absent) — while the sync above
|
||||
reported success (the folder-stats failure never flips the run,
|
||||
phase 94)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, LS_QUESTION_1)
|
||||
_wait_settled(page)
|
||||
lines = _last_brain(page).locator(".tool-call")
|
||||
expect(lines).to_have_count(1)
|
||||
expect(lines.nth(0)).to_contain_text(f"Listing documents in {SOURCE}")
|
||||
_assert_drill_turn(
|
||||
page,
|
||||
{"type": "tool", "name": "ls", "argument": SOURCE},
|
||||
[
|
||||
LS_HEADER,
|
||||
# The bare line (the row is absent — no stored summary to
|
||||
# append) …
|
||||
LINE_ALWAYS,
|
||||
# …and the colon-suffixed lines (the rows the retry + the
|
||||
# normal path stored):
|
||||
LINE_ONCE,
|
||||
LINE_NORMAL,
|
||||
],
|
||||
# The ``e2e_empty_always/`` line must NOT carry a ``: …``
|
||||
# suffix — the bare line above is a prefix of the suffixed
|
||||
# shape, so the absence is pinned here (the all-4-attempts
|
||||
# exhaustion left no row to quote).
|
||||
forbidden=[LINE_ALWAYS_WITH_COLON],
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. The gap-fill: a deleted row self-heals on the next UNCHANGED sync,
|
||||
# the other rows untouched
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_deleted_row_self_heals_on_unchanged_sync(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""Delete the ``normal`` stored row behind the app's back (a
|
||||
historical failure), run the UNCHANGED sync #2, and assert the
|
||||
task-02/03 gap gate: ``missing_folder_summaries`` names the gap,
|
||||
``only_missing=True`` fills EXACTLY the missing rows, and every
|
||||
other row stays byte-identical (text AND ``updated_at``).
|
||||
|
||||
* ``normal`` — healed: the second ``ls oneshot`` turn carries
|
||||
``: Fixture folder summary for oneshot/normal.`` again, and the
|
||||
DB row is back with its deterministic text;
|
||||
* ``e2e_empty_once`` — the gap-fill NEVER calls it (the row
|
||||
exists): its ``updated_at`` is UNCHANGED (a full regeneration
|
||||
would have re-stamped it);
|
||||
* ``e2e_empty_always`` — the gap-fill attempted it, exhausted
|
||||
(4 empty POSTs by design), stayed ABSENT — and the sync still
|
||||
succeeded (the fail-soft never flips the run)."""
|
||||
page.set_default_timeout(30_000)
|
||||
# The gap: delete the ``normal`` row directly (the test process has
|
||||
# DB access via the conftest engine — the same connection the app
|
||||
# uses), capturing the other row's stamp for the targeted-fill
|
||||
# assertion. No KB change anywhere.
|
||||
before = _folder_rows()
|
||||
assert FOLDER_NORMAL in before # the row existed (sync #1 stored it)
|
||||
updated_at_once = before[FOLDER_ONCE][1]
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries "
|
||||
"WHERE source = :s AND folder_path = :f"
|
||||
),
|
||||
{"s": SOURCE, "f": FOLDER_NORMAL},
|
||||
)
|
||||
db.commit()
|
||||
assert FOLDER_NORMAL not in _folder_rows()
|
||||
|
||||
# Sync #2 — the KB is UNCHANGED (nothing in the temp tree moved),
|
||||
# so the gate takes the gap probe: two candidates missing
|
||||
# (``e2e_empty_always`` + ``normal``) → the targeted fill.
|
||||
body = _run_sync_http(app_url)
|
||||
assert body["state"] == "success", body # exhaustion never flips the run
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 0, detail
|
||||
assert detail["updated"] == 0, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is False, detail # unchanged → no overview burn
|
||||
|
||||
# The second scripted drill turn: the healed + the surviving lines,
|
||||
# the always folder still bare.
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
_submit(page, LS_QUESTION_2)
|
||||
_wait_settled(page)
|
||||
lines = _last_brain(page).locator(".tool-call")
|
||||
expect(lines).to_have_count(1)
|
||||
expect(lines.nth(0)).to_contain_text(f"Listing documents in {SOURCE}")
|
||||
_assert_drill_turn(
|
||||
page,
|
||||
{"type": "tool", "name": "ls", "argument": SOURCE},
|
||||
[
|
||||
LS_HEADER,
|
||||
LINE_ALWAYS,
|
||||
LINE_ONCE,
|
||||
LINE_NORMAL, # healed — the suffix is back
|
||||
],
|
||||
forbidden=[LINE_ALWAYS_WITH_COLON],
|
||||
)
|
||||
|
||||
# The DB pins the targeted fill (the E2E's other established lens):
|
||||
rows = _folder_rows()
|
||||
assert set(rows) == {"", FOLDER_ONCE, FOLDER_NORMAL}, rows
|
||||
assert rows[FOLDER_NORMAL][0] == SUM_NORMAL, rows # deterministic text
|
||||
# The targeted fill never touched the other rows — a FULL
|
||||
# regeneration would have re-stamped this one (the ``_upsert``
|
||||
# fresh-UTC-stamp rule).
|
||||
assert rows[FOLDER_ONCE][1] == updated_at_once, (
|
||||
rows[FOLDER_ONCE][1],
|
||||
updated_at_once,
|
||||
)
|
||||
assert FOLDER_ALWAYS not in rows, rows # exhausted again → still absent
|
||||
@@ -111,8 +111,10 @@ def _stub_folder_summaries(monkeypatch: pytest.MonkeyPatch) -> list[dict]:
|
||||
the call record; the canned stats are the zero dict."""
|
||||
calls: list[dict] = []
|
||||
|
||||
async def fake_generate(db: object, llm: object, *, skip: bool = False) -> dict[str, int]:
|
||||
calls.append({"skip": skip})
|
||||
async def fake_generate(
|
||||
db: object, llm: object, *, skip: bool = False, only_missing: bool = False
|
||||
) -> dict[str, int]:
|
||||
calls.append({"skip": skip, "only_missing": only_missing})
|
||||
return {"generated": 0, "failed": 0, "pruned": 0}
|
||||
|
||||
monkeypatch.setattr(import_docs, "generate_folder_summaries", fake_generate)
|
||||
|
||||
@@ -303,7 +303,12 @@ class FakeFolderSummaries:
|
||||
layer boundary — the real generator would read the global
|
||||
``documents`` table and call the (real) ``LLMClient`` over the
|
||||
network. The generator only flushes, so the fake honours the
|
||||
``skip`` flag the same way (the zero stats, no side effects)."""
|
||||
``skip`` flag the same way (the zero stats, no side effects).
|
||||
Phase 96 (task 03): the unchanged-walk gap path calls the
|
||||
generator with ``only_missing=True`` — the fake records the flag
|
||||
the same way it records ``skip`` (the real gap probe,
|
||||
``missing_folder_summaries``, runs against the real tables).
|
||||
"""
|
||||
|
||||
ZERO = {"generated": 0, "failed": 0, "pruned": 0}
|
||||
|
||||
@@ -312,11 +317,18 @@ class FakeFolderSummaries:
|
||||
self.llms: list[LLMClient] = []
|
||||
self.sessions: list[Session] = []
|
||||
self.skip_flags: list[bool] = []
|
||||
self.only_missing_flags: list[bool] = []
|
||||
|
||||
async def __call__(
|
||||
self, db: Session, llm: LLMClient, *, skip: bool = False
|
||||
self,
|
||||
db: Session,
|
||||
llm: LLMClient,
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
) -> dict[str, int]:
|
||||
self.skip_flags.append(skip)
|
||||
self.only_missing_flags.append(only_missing)
|
||||
if skip:
|
||||
return dict(self.ZERO)
|
||||
self.llms.append(llm)
|
||||
|
||||
@@ -14,9 +14,19 @@ DB, explicit ``--source``):
|
||||
- a KB-changing import → one row per ≥ 2-doc subtree (the source root
|
||||
+ the 2-doc folder; the 1-doc folder gets none), committed in the
|
||||
run's transaction, the summary line ending
|
||||
``folder_summaries=<generated>/<failed>/<pruned>``;
|
||||
- an unchanged re-import → zero ``lite`` calls,
|
||||
``folder_summaries=skipped``, rows untouched;
|
||||
``folder_summaries=<generated>/<failed>/<pruned>`` (unchanged by
|
||||
phase 96 — no gap-fill suffix on a full regeneration);
|
||||
- an unchanged re-import with a COMPLETE table → zero ``lite`` calls,
|
||||
``folder_summaries=skipped``, rows untouched (the phase-94
|
||||
zero-burn invariant);
|
||||
- an unchanged re-import with a GAP (one stored row deleted) →
|
||||
exactly one ``FOLDER_SUMMARY_MODE`` call (the missing folder only),
|
||||
the row back with the deterministic fake text, every other row
|
||||
byte-identical (summary AND ``updated_at``), the line ending
|
||||
``folder_summaries=1/0/0 (gap-fill)`` (phase 96, task 03 — the
|
||||
failed folder summary self-heals on the next sync);
|
||||
- a KB change on a second run → still a FULL regeneration (call count
|
||||
== candidate count, every row re-stamped, no gap-fill suffix);
|
||||
- a subtree dropping below 2 docs after a changed re-walk → its row
|
||||
pruned;
|
||||
- one folder's ``lite`` failure → its previous row kept, the other
|
||||
@@ -24,7 +34,9 @@ DB, explicit ``--source``):
|
||||
- a ``--limit`` debug run → no generation, no rows,
|
||||
``folder_summaries=skipped``;
|
||||
- a fresh (empty) table after a ``--limit`` first walk → an unchanged
|
||||
full walk generates (the table-empty first-run trigger).
|
||||
full walk generates via the gap-fill path (the subsumed table-empty
|
||||
first-run trigger — every candidate is missing), the line carrying
|
||||
`` (gap-fill)``.
|
||||
|
||||
API path (``POST /api/sync`` end to end, real import over a host temp
|
||||
local dir, deterministic ``FakeEmbedder``):
|
||||
@@ -32,14 +44,20 @@ local dir, deterministic ``FakeEmbedder``):
|
||||
- a KB-changing sync → the rows land (visible via the test's own
|
||||
session) and the status detail keeps its exact pre-phase key set
|
||||
(no folder-summary surface — the stats are log-only);
|
||||
- an unchanged re-sync → zero ``FOLDER_SUMMARY_MODE`` calls;
|
||||
- an unchanged re-sync (complete table) → zero ``FOLDER_SUMMARY_MODE``
|
||||
calls (the phase-94 zero-burn invariant);
|
||||
- an unchanged re-sync with a GAP (one stored row deleted) → targeted
|
||||
fill of exactly that row (one ``FOLDER_SUMMARY_MODE`` call, the
|
||||
``gap-fill`` log line), every other row byte-identical, the status
|
||||
detail shape untouched (phase 96, task 03);
|
||||
- a ``lite`` outage (one folder failing) → the failed folder's row is
|
||||
kept, the run reports ``success`` (never ``failed``), and the
|
||||
sources-version bump still lands (the bump is change-gated on the
|
||||
KB, not on the summaries);
|
||||
- an empty table after a populated sync (the migration-0017 scenario)
|
||||
→ an unchanged walk regenerates (the overview's API gate, purely
|
||||
change-gated, does not).
|
||||
→ an unchanged walk regenerates via the gap probe (the subsumed
|
||||
table-empty trigger; the overview's API gate, purely change-gated,
|
||||
does not fire).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -290,6 +308,101 @@ def test_unchanged_reimport_burns_zero_folder_calls(
|
||||
assert _rows(db) == rows # rows byte-identical
|
||||
|
||||
|
||||
def test_unchanged_reimport_with_gap_fills_only_the_missing_row(
|
||||
db: Session,
|
||||
src: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""Phase 96 (task 03): an unchanged walk with ONE deleted stored
|
||||
row → exactly one ``FOLDER_SUMMARY_MODE`` call (the deleted
|
||||
folder only), the row back with the deterministic fake text, every
|
||||
OTHER row byte-identical (summary AND ``updated_at``), the summary
|
||||
line ending ``folder_summaries=1/0/0 (gap-fill)`` — the failed
|
||||
folder summary self-heals on the next sync instead of persisting
|
||||
until a KB change."""
|
||||
llm1 = FakeEmbedder()
|
||||
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
|
||||
assert rc == 0
|
||||
rows_before = _rows(db)
|
||||
assert set(rows_before) == {("MyDocs", ""), ("MyDocs", "a")}
|
||||
root_stamp_before = _updated_at(db, "MyDocs", "")
|
||||
assert root_stamp_before is not None
|
||||
|
||||
# Simulate the phase-96 incident: a lost row (an exhausted
|
||||
# one-shot retry leaves a candidate without its row).
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries "
|
||||
"WHERE source = 'MyDocs' AND folder_path = 'a'"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
llm2 = FakeEmbedder()
|
||||
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
|
||||
assert rc == 0
|
||||
assert "unchanged=3" in out
|
||||
assert out.rstrip().endswith(
|
||||
"overview=skipped sources_version=skipped "
|
||||
"folder_summaries=1/0/0 (gap-fill)"
|
||||
)
|
||||
# Exactly ONE new folder call — the deleted row's folder only.
|
||||
calls = _folder_calls(llm2)
|
||||
assert len(calls) == 1
|
||||
assert calls[0][1]["content"].splitlines()[0] == "Folder: MyDocs/a"
|
||||
assert len(llm2.chat_calls) == 1 # no other lite traffic at all
|
||||
# The row is back with the deterministic fake text ...
|
||||
rows_after = _rows(db)
|
||||
assert rows_after == rows_before
|
||||
# ... and every OTHER row byte-identical (the root was never
|
||||
# re-stamped by the targeted fill).
|
||||
assert _updated_at(db, "MyDocs", "") == root_stamp_before
|
||||
|
||||
|
||||
def test_changed_reimport_is_a_full_regeneration(
|
||||
db: Session,
|
||||
src: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""Phase 96 (task 03): a KB change is STILL a full regeneration —
|
||||
call count == candidate count, every row re-stamped, and NO
|
||||
`` (gap-fill)`` suffix (byte-identical to today's behavior)."""
|
||||
llm1 = FakeEmbedder()
|
||||
rc, _ = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
|
||||
assert rc == 0
|
||||
rows_before = _rows(db)
|
||||
root_stamp_before = _updated_at(db, "MyDocs", "")
|
||||
a_stamp_before = _updated_at(db, "MyDocs", "a")
|
||||
assert root_stamp_before is not None and a_stamp_before is not None
|
||||
|
||||
# A KB change (one doc edited) — the gate is the change, not the
|
||||
# gap.
|
||||
(src / "a" / "one.md").write_text("# A One\nChanged content.\n", encoding="utf-8")
|
||||
llm2 = FakeEmbedder()
|
||||
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
|
||||
assert rc == 0
|
||||
assert "updated=1" in out
|
||||
# Full-regeneration token — the stats without the gap-fill suffix.
|
||||
assert out.rstrip().endswith(
|
||||
"overview=updated sources_version=2 folder_summaries=2/0/0"
|
||||
)
|
||||
# Call count == candidate count — BOTH folders, not a targeted fill.
|
||||
calls = _folder_calls(llm2)
|
||||
assert [c[1]["content"].splitlines()[0] for c in calls] == [
|
||||
"Folder: MyDocs",
|
||||
"Folder: MyDocs/a",
|
||||
]
|
||||
# All rows re-stamped (the full regeneration re-writes every
|
||||
# candidate, even the unchanging one).
|
||||
root_stamp_after = _updated_at(db, "MyDocs", "")
|
||||
a_stamp_after = _updated_at(db, "MyDocs", "a")
|
||||
assert root_stamp_after is not None and root_stamp_after > root_stamp_before
|
||||
assert a_stamp_after is not None and a_stamp_after > a_stamp_before
|
||||
assert _rows(db) == rows_before # deterministic fake → same texts
|
||||
|
||||
|
||||
def test_subtree_dropping_below_two_docs_is_pruned(
|
||||
db: Session,
|
||||
src: Path,
|
||||
@@ -382,10 +495,11 @@ def test_empty_table_generates_on_unchanged_walk(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""The table-empty first-run trigger: after a ``--limit`` first
|
||||
walk (populated KB, empty table), an unchanged full walk generates
|
||||
— for the folder summaries AND the missing outline, still never
|
||||
bumping the version."""
|
||||
"""The subsumed table-empty first-run trigger (phase 96, task
|
||||
03): after a ``--limit`` first walk (populated KB, empty table),
|
||||
an unchanged full walk generates — for the folder summaries (now
|
||||
via the gap-fill path — every candidate is missing) AND the
|
||||
missing outline, still never bumping the version."""
|
||||
llm1 = FakeEmbedder()
|
||||
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src), "--limit", "3"], capsys)
|
||||
assert rc == 0
|
||||
@@ -399,8 +513,13 @@ def test_empty_table_generates_on_unchanged_walk(
|
||||
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
|
||||
assert rc == 0
|
||||
assert "unchanged=3" in out
|
||||
# Phase 96 (task 03): the old table-empty trigger is now the
|
||||
# subsumed gap case — every candidate is missing, so the unchanged
|
||||
# walk takes the targeted-fill path and the token carries
|
||||
# `` (gap-fill)`` (the generated set is the full candidate set).
|
||||
assert out.rstrip().endswith(
|
||||
"overview=updated sources_version=skipped folder_summaries=2/0/0"
|
||||
"overview=updated sources_version=skipped "
|
||||
"folder_summaries=2/0/0 (gap-fill)"
|
||||
)
|
||||
assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a")}
|
||||
assert len(_folder_calls(llm2)) == 2
|
||||
@@ -576,6 +695,86 @@ def test_api_unchanged_resync_burns_zero_folder_calls(
|
||||
assert current_sources_version(db) == 1
|
||||
|
||||
|
||||
def test_api_unchanged_resync_with_gap_fills_only_the_missing_row(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
local_dir: Path,
|
||||
) -> None:
|
||||
"""Phase 96 (task 03), API path: an unchanged re-sync with ONE
|
||||
deleted stored row → targeted fill of exactly that row (one
|
||||
``FOLDER_SUMMARY_MODE`` call), the ``gap-fill`` log line, every
|
||||
other row byte-identical (summary AND ``updated_at``), status
|
||||
``success``, and the status detail shape untouched (the stats stay
|
||||
log-only — the phase-94 contract)."""
|
||||
_seed_local(db, local_dir)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(str(local_dir.parent / "bor")),
|
||||
)
|
||||
clients = _capture_llm(monkeypatch)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
_poll(sync_client, "success")
|
||||
rows_before = _rows(db)
|
||||
assert set(rows_before) == {("LocalDocs", ""), ("LocalDocs", "a")}
|
||||
root_stamp_before = _updated_at(db, "LocalDocs", "")
|
||||
assert root_stamp_before is not None
|
||||
|
||||
# The phase-96 incident shape: a lost row, deleted directly.
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries "
|
||||
"WHERE source = 'LocalDocs' AND folder_path = 'a'"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
records: list[logging.LogRecord] = []
|
||||
|
||||
class _Sink(logging.Handler):
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
records.append(record)
|
||||
|
||||
sync_logger = logging.getLogger("app.api.sync")
|
||||
sink = _Sink()
|
||||
sync_logger.addHandler(sink)
|
||||
sync_logger.setLevel(logging.INFO)
|
||||
try:
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
finally:
|
||||
sync_logger.removeHandler(sink)
|
||||
|
||||
assert body["error"] is None
|
||||
assert body["detail"]["overview"] is False # unchanged → no overview
|
||||
assert body["detail"]["sources_version"] == 1 # unchanged → no bump
|
||||
# No new sync-status surface: the detail keeps its exact key set.
|
||||
assert set(body["detail"]) == {
|
||||
"files", "added", "updated", "unchanged", "pruned", "errors",
|
||||
"chunks", "summaries", "summary_errors", "overview",
|
||||
"sources_version",
|
||||
}
|
||||
assert len(clients) == 2
|
||||
# Targeted fill — exactly ONE folder call, the missing folder only
|
||||
# (plus the phase-41 probe's ping on the same client).
|
||||
calls = _folder_calls(clients[1])
|
||||
assert len(calls) == 1
|
||||
assert calls[0][1]["content"].splitlines()[0] == "Folder: LocalDocs/a"
|
||||
assert len(clients[1].chat_calls) == 2 # ping + the one fill
|
||||
# The row is back with the deterministic fake text ...
|
||||
assert _rows(db) == rows_before
|
||||
# ... every other row byte-identical (the root was never re-stamped
|
||||
# by the targeted fill).
|
||||
assert _updated_at(db, "LocalDocs", "") == root_stamp_before
|
||||
# The gap-fill log line (PLAN §9 ample logging).
|
||||
assert any(
|
||||
"sync: folder_summaries gap-fill" in r.getMessage() for r in records
|
||||
)
|
||||
|
||||
|
||||
def test_api_folder_lite_failure_keeps_rows_stays_green_and_bumps(
|
||||
sync_client: TestClient,
|
||||
@@ -634,10 +833,12 @@ def test_api_empty_table_first_sync_regenerates(
|
||||
db: Session,
|
||||
local_dir: Path,
|
||||
) -> None:
|
||||
"""The migration-0017 scenario: the KB predates the table — wipe
|
||||
the rows and re-sync an unchanged KB: the empty-table trigger
|
||||
fires for the folder summaries (the overview's API gate, purely
|
||||
change-gated, does not)."""
|
||||
"""The migration-0017 scenario (the subsumed table-empty trigger —
|
||||
phase 96, task 03): the KB predates the table — wipe the rows and
|
||||
re-sync an unchanged KB: the gap probe fires (every candidate is
|
||||
missing) and the targeted fill regenerates the full candidate set
|
||||
for the folder summaries (the overview's API gate, purely
|
||||
change-gated, does not fire)."""
|
||||
_seed_local(db, local_dir)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
|
||||
@@ -31,9 +31,9 @@ from app.rag.folder_summaries import (
|
||||
SYSTEM_PROMPT,
|
||||
build_folder_summary_prompt,
|
||||
folder_of,
|
||||
folder_summary_table_empty,
|
||||
generate_folder_summaries,
|
||||
group_by_folder,
|
||||
missing_folder_summaries,
|
||||
summarize_folder,
|
||||
)
|
||||
from app.rag.llm import LLMError
|
||||
@@ -613,16 +613,198 @@ def test_generate_only_flushes_caller_commits(db: Session, clean_tables) -> None
|
||||
assert MIN_DOCS_PER_FOLDER == 2 # the ≥ 2 scope rule, pinned by name
|
||||
|
||||
|
||||
def test_folder_summary_table_empty_gate(db: Session, clean_tables) -> None:
|
||||
"""The sync-path gate probe (phase 94, task 02): empty → True
|
||||
(the first full sync after migration 0017 must still generate),
|
||||
one row → False (a populated table waits for a KB change)."""
|
||||
assert folder_summary_table_empty(db) is True # the truncated table
|
||||
_add_doc(db, "FSU", "a/one.md", "One")
|
||||
_add_doc(db, "FSU", "a/two.md", "Two")
|
||||
# ---------- missing_folder_summaries (phase 96, task 02) ----------
|
||||
|
||||
|
||||
def test_missing_fresh_table_is_exactly_the_candidate_set(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""No stored rows → every candidate folder is a gap, sorted by
|
||||
``(source, folder_path)``; the single-doc FSU-solo root is not a
|
||||
candidate and can never be a gap."""
|
||||
_seed_catalogue(db)
|
||||
assert missing_folder_summaries(db) == [
|
||||
("FSU", ""),
|
||||
("FSU", "a"),
|
||||
("FSU", "a/b"),
|
||||
]
|
||||
assert ("FSU-solo", "") not in missing_folder_summaries(db)
|
||||
|
||||
|
||||
def test_missing_fully_populated_table_is_empty(db: Session, clean_tables) -> None:
|
||||
"""Every candidate row present → no gap (the zero-burn gate case)."""
|
||||
_seed_catalogue(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
assert folder_summary_table_empty(db) is False # rows landed
|
||||
assert missing_folder_summaries(db) == []
|
||||
|
||||
|
||||
def test_missing_one_deleted_row_is_that_folder(db: Session, clean_tables) -> None:
|
||||
_seed_catalogue(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries "
|
||||
"WHERE source = 'FSU' AND folder_path = 'a'"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
assert missing_folder_summaries(db) == [("FSU", "a")]
|
||||
|
||||
|
||||
def test_missing_empty_kb_empty_table_is_no_gap(db: Session, clean_tables) -> None:
|
||||
"""No catalogue → no candidates → ``[]`` — an empty table over an
|
||||
empty KB is not a gap (there is nothing to fill)."""
|
||||
assert missing_folder_summaries(db) == []
|
||||
|
||||
|
||||
def test_missing_single_doc_folder_is_never_listed(db: Session, clean_tables) -> None:
|
||||
"""A below-minimum folder without a row is NOT a gap — it is not a
|
||||
candidate (its one file line IS its summary)."""
|
||||
_add_doc(db, "FSU", "solo/one.md", "One")
|
||||
assert missing_folder_summaries(db) == []
|
||||
|
||||
|
||||
def test_missing_stale_row_is_not_a_gap(db: Session, clean_tables) -> None:
|
||||
"""A stored row for a folder that dropped below 2 docs is stale,
|
||||
not missing — the prune pass owns it, the gap detector ignores it."""
|
||||
_seed_catalogue(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
db.add(FolderSummary(source="FSU", folder_path="gone/old", summary="stale"))
|
||||
db.commit()
|
||||
assert missing_folder_summaries(db) == []
|
||||
|
||||
|
||||
# ---------- generate_folder_summaries(only_missing=…) (phase 96, 02) ----------
|
||||
|
||||
|
||||
def _updated_at(db: Session, source: str, folder_path: str) -> object:
|
||||
"""The stored row's ``updated_at`` (raw SQL — bypasses the ORM
|
||||
identity map, so the before/after byte-identity comparison is
|
||||
honest)."""
|
||||
return db.execute(
|
||||
text(
|
||||
"SELECT updated_at FROM folder_summaries "
|
||||
"WHERE source = :s AND folder_path = :f"
|
||||
),
|
||||
{"s": source, "f": folder_path},
|
||||
).scalar_one()
|
||||
|
||||
|
||||
def test_only_missing_fills_exactly_the_missing_keys(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""Two missing + two present → exactly the missing keys are
|
||||
generated (sorted order, one lite call each); the present rows are
|
||||
byte-identical after (text AND ``updated_at``); stats right."""
|
||||
_seed_catalogue(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
full = _rows(db)
|
||||
a_stamp = _updated_at(db, "FSU", "a")
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries "
|
||||
"WHERE (source, folder_path) IN (('FSU', ''), ('FSU', 'a/b'))"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
assert missing_folder_summaries(db) == [("FSU", ""), ("FSU", "a/b")]
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(generate_folder_summaries(db, llm, only_missing=True))
|
||||
assert stats == {"generated": 2, "failed": 0, "pruned": 0}
|
||||
assert llm.calls == 2, "one call per MISSING key — zero for present rows"
|
||||
assert [user.splitlines()[0] for _s, user in llm.requests] == [
|
||||
"Folder: FSU",
|
||||
"Folder: FSU/a/b",
|
||||
], "the missing keys in sorted (source, folder_path) order"
|
||||
|
||||
assert _rows(db) == full, "the fill restores exactly the full candidate set"
|
||||
assert _updated_at(db, "FSU", "a") == a_stamp, (
|
||||
"the present row is byte-identical — never re-stamped by the fill"
|
||||
)
|
||||
|
||||
|
||||
def test_only_missing_no_gap_burns_zero_calls(db: Session, clean_tables) -> None:
|
||||
"""No gap → zero lite calls, zero rows touched, zero stats (the
|
||||
zero-burn invariant the unchanged-sync gate relies on)."""
|
||||
_seed_catalogue(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
before = _rows(db)
|
||||
stamps = {f: _updated_at(db, "FSU", f) for f in ("", "a", "a/b")}
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(generate_folder_summaries(db, llm, only_missing=True))
|
||||
assert stats == {"generated": 0, "failed": 0, "pruned": 0}
|
||||
assert llm.calls == 0, "zero-burn: no gap, no lite call"
|
||||
assert _rows(db) == before
|
||||
for folder, stamp in stamps.items():
|
||||
assert _updated_at(db, "FSU", folder) == stamp, "no row re-stamped"
|
||||
|
||||
|
||||
def test_only_missing_still_prunes_stale_rows(db: Session, clean_tables) -> None:
|
||||
"""The prune pass runs in BOTH modes: the manually seeded stale row
|
||||
(folder gone from the catalogue) is pruned while the genuine
|
||||
missing folders are filled, and the present row stays untouched."""
|
||||
_seed_catalogue(db)
|
||||
db.add(FolderSummary(source="FSU", folder_path="a", summary="keep me"))
|
||||
db.add(FolderSummary(source="FSU", folder_path="gone/old", summary="stale"))
|
||||
db.commit()
|
||||
a_stamp = _updated_at(db, "FSU", "a")
|
||||
assert missing_folder_summaries(db) == [("FSU", ""), ("FSU", "a/b")]
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(generate_folder_summaries(db, llm, only_missing=True))
|
||||
assert stats == {"generated": 2, "failed": 0, "pruned": 1}
|
||||
assert llm.calls == 2
|
||||
|
||||
stored = _rows(db)
|
||||
assert ("FSU", "gone/old") not in stored, (
|
||||
"the stale row is pruned even under only_missing"
|
||||
)
|
||||
assert stored[("FSU", "a")] == "keep me"
|
||||
assert _updated_at(db, "FSU", "a") == a_stamp
|
||||
assert stored[("FSU", "")] == REPLY and stored[("FSU", "a/b")] == REPLY
|
||||
|
||||
|
||||
def test_only_missing_fail_soft_keeps_prior_and_lands_others(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""Per-folder fail-soft applies under ``only_missing`` too: the
|
||||
failing missing folder is counted and stays absent; the other
|
||||
missing folders still land; the present row is untouched."""
|
||||
_seed_catalogue(db)
|
||||
db.add(FolderSummary(source="FSU", folder_path="a", summary="keep me"))
|
||||
db.commit()
|
||||
llm = _FakeLLM(fail_folders=("FSU/a/b",))
|
||||
stats = asyncio.run(generate_folder_summaries(db, llm, only_missing=True))
|
||||
assert stats == {"generated": 1, "failed": 1, "pruned": 0}
|
||||
assert llm.calls == 2 # both missing folders were attempted
|
||||
|
||||
stored = _rows(db)
|
||||
assert stored[("FSU", "")] == REPLY, "the other missing folder still lands"
|
||||
assert ("FSU", "a/b") not in stored, "the failed folder stays absent"
|
||||
assert stored[("FSU", "a")] == "keep me", "the present row is untouched"
|
||||
|
||||
|
||||
def test_gap_probe_subsumes_the_table_empty_gate(db: Session, clean_tables) -> None:
|
||||
"""The deleted phase-94 table-empty gate probe, re-expressed through
|
||||
``missing_folder_summaries`` (phase 96, task 03 — the probe's unit
|
||||
coverage moved here): an empty table over a populated catalogue
|
||||
means EVERY candidate is missing (the targeted fill over all
|
||||
candidates IS a full generation — the first full sync after
|
||||
migration 0017 must still generate), a populated table means no
|
||||
gap (a populated table waits for a KB change or a gap)."""
|
||||
assert missing_folder_summaries(db) == [] # the truncated table, empty KB
|
||||
_add_doc(db, "FSU", "a/one.md", "One")
|
||||
_add_doc(db, "FSU", "a/two.md", "Two")
|
||||
assert missing_folder_summaries(db) == [("FSU", ""), ("FSU", "a")]
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
assert missing_folder_summaries(db) == [] # rows landed → no gap
|
||||
db.execute(text("DELETE FROM folder_summaries"))
|
||||
db.commit()
|
||||
assert folder_summary_table_empty(db) is True # emptied again
|
||||
assert missing_folder_summaries(db) == [("FSU", ""), ("FSU", "a")] # emptied again
|
||||
|
||||
@@ -10,6 +10,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import AsyncGenerator
|
||||
from types import SimpleNamespace
|
||||
from typing import Any, cast
|
||||
@@ -346,14 +347,27 @@ class _FakeCompletion:
|
||||
"""One fake non-streaming ChatCompletion (``choices[].message`` shape).
|
||||
|
||||
``content=None`` mirrors the real wire where the field can be absent or
|
||||
empty (reasoning-only replies, provider quirks).
|
||||
empty (reasoning-only replies, provider quirks). ``finish_reason``
|
||||
(phase 96) defaults to ``None`` — the provider omitting it — and the
|
||||
incident signature is ``"length"`` (the whole ``max_tokens`` budget
|
||||
spent in ``reasoning_content``).
|
||||
"""
|
||||
|
||||
def __init__(self, content: str | None, empty_choices: bool = False) -> None:
|
||||
def __init__(
|
||||
self,
|
||||
content: str | None,
|
||||
empty_choices: bool = False,
|
||||
finish_reason: str | None = None,
|
||||
) -> None:
|
||||
if empty_choices:
|
||||
self.choices = []
|
||||
else:
|
||||
self.choices = [SimpleNamespace(message=SimpleNamespace(content=content))]
|
||||
self.choices = [
|
||||
SimpleNamespace(
|
||||
message=SimpleNamespace(content=content),
|
||||
finish_reason=finish_reason,
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
class _FakeCompletions:
|
||||
@@ -362,12 +376,25 @@ class _FakeCompletions:
|
||||
chunks: list | None = None,
|
||||
fail: Exception | None = None,
|
||||
completion: _FakeCompletion | None = None,
|
||||
completion_seq: list[_FakeCompletion] | None = None,
|
||||
) -> None:
|
||||
self.chunks = chunks or []
|
||||
self.fail = fail
|
||||
self.completion = completion
|
||||
#: Phase 96: a scripted per-``create()`` reply sequence (the retry
|
||||
#: matrix) — popped one per non-streaming call, in order.
|
||||
self.completion_seq = (
|
||||
list(completion_seq) if completion_seq is not None else None
|
||||
)
|
||||
self.kwargs: dict | None = None
|
||||
self.chat_kwargs: dict | None = None
|
||||
#: Every non-streaming ``create()`` call's kwargs (the attempt
|
||||
#: counter for the phase-96 retry matrix).
|
||||
self.chat_calls: list[dict] = []
|
||||
#: Every ``create()`` call (streaming + non-streaming), incl.
|
||||
#: calls that raised (``fail``) — the attempt counter when the
|
||||
#: failure happens inside the SDK call itself.
|
||||
self.create_calls: int = 0
|
||||
#: Every SDK-shaped stream handed out — teardown tests assert the
|
||||
#: phase-48 ``close()`` on them (phase 71 task 02: with/without
|
||||
#: a filter, the teardown path is the same object).
|
||||
@@ -375,6 +402,7 @@ class _FakeCompletions:
|
||||
|
||||
async def create(self, **kwargs) -> _FakeChatStream | _FakeCompletion:
|
||||
self.kwargs = kwargs
|
||||
self.create_calls += 1
|
||||
if self.fail is not None:
|
||||
raise self.fail
|
||||
if kwargs.get("stream"):
|
||||
@@ -382,6 +410,11 @@ class _FakeCompletions:
|
||||
self.streams.append(stream)
|
||||
return stream
|
||||
self.chat_kwargs = kwargs
|
||||
self.chat_calls.append(dict(kwargs))
|
||||
if self.completion_seq is not None:
|
||||
if not self.completion_seq:
|
||||
raise AssertionError("completion script exhausted")
|
||||
return self.completion_seq.pop(0)
|
||||
assert self.completion is not None
|
||||
return self.completion
|
||||
|
||||
@@ -904,9 +937,12 @@ def test_chat_stream_abandon_with_filter_closes_stream() -> None:
|
||||
def _make_chat_client(
|
||||
completion: _FakeCompletion | None = None,
|
||||
fail: Exception | None = None,
|
||||
completion_seq: list[_FakeCompletion] | None = None,
|
||||
**settings_kwargs: Any,
|
||||
) -> tuple[LLMClient, _FakeCompletions]:
|
||||
completions = _FakeCompletions(fail=fail, completion=completion)
|
||||
completions = _FakeCompletions(
|
||||
fail=fail, completion=completion, completion_seq=completion_seq
|
||||
)
|
||||
fake_openai = SimpleNamespace(chat=SimpleNamespace(completions=completions))
|
||||
llm = LLMClient(_settings(**settings_kwargs))
|
||||
llm._client = fake_openai # pyright: ignore[reportAttributeAccessIssue]
|
||||
@@ -984,18 +1020,196 @@ def test_chat_empty_choices_raises_llm_error() -> None:
|
||||
|
||||
|
||||
def test_chat_missing_content_raises_llm_error() -> None:
|
||||
"""A silent empty summary must never be stored — None content fails."""
|
||||
llm, _ = _make_chat_client(_FakeCompletion(None))
|
||||
"""A silent empty summary must never be stored — None content fails.
|
||||
Phase 96: pinned with the kill switch (``llm_retries=0``) so the
|
||||
pre-phase-96 single-attempt behavior and message are asserted
|
||||
verbatim (the retry matrix below pins the retried contract)."""
|
||||
llm, _ = _make_chat_client(_FakeCompletion(None), llm_retries=0)
|
||||
with pytest.raises(LLMError, match="empty content"):
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
|
||||
|
||||
def test_chat_whitespace_only_content_raises_llm_error() -> None:
|
||||
llm, _ = _make_chat_client(_FakeCompletion(" \n\t "))
|
||||
"""Whitespace-only content is empty (phase 96 kill-switch pin)."""
|
||||
llm, _ = _make_chat_client(_FakeCompletion(" \n\t "), llm_retries=0)
|
||||
with pytest.raises(LLMError, match="empty content"):
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
|
||||
|
||||
# ---------- one-shot empty-reply retry (phase 96, task 01) ----------
|
||||
|
||||
_DEFAULT_BASE = "https://aipi.reeseapps.com/v1"
|
||||
|
||||
|
||||
def _empty(finish_reason: str | None = "length") -> _FakeCompletion:
|
||||
"""An incident-shaped empty reply (``content=None``; ``finish_reason``
|
||||
defaults to ``"length"`` — the 2026-09-11 signature)."""
|
||||
return _FakeCompletion(None, finish_reason=finish_reason)
|
||||
|
||||
|
||||
def test_chat_empty_then_success_retries_and_recovers(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""First reply empty (the incident shape), second reply has content →
|
||||
exactly 2 attempts, ONE flat sleep of ``llm_retry_delay`` (default
|
||||
5.0), the trimmed second reply is returned, and ONE warning fired
|
||||
naming the model, the empty reply's ``finish_reason``, and the
|
||||
attempt count."""
|
||||
sleeps = _record_sleeps(monkeypatch)
|
||||
caplog.set_level(logging.WARNING, logger="app.llm")
|
||||
llm, completions = _make_chat_client(
|
||||
completion_seq=[
|
||||
_FakeCompletion(None, finish_reason="length"),
|
||||
_FakeCompletion(" Recovered.\n"),
|
||||
]
|
||||
)
|
||||
out = asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert out == "Recovered."
|
||||
assert completions.create_calls == 2
|
||||
assert len(completions.chat_calls) == 2
|
||||
# Both attempts are byte-identical (same request).
|
||||
assert completions.chat_calls[0] == completions.chat_calls[1]
|
||||
assert sleeps == [5.0] # one flat BOR_LLM_RETRY_DELAY (default)
|
||||
warnings = [r for r in caplog.records if r.levelno == logging.WARNING]
|
||||
assert len(warnings) == 1
|
||||
line = warnings[0].getMessage()
|
||||
assert "lite" in line # the summary model (default)
|
||||
assert "finish_reason=length" in line # the incident signature
|
||||
assert "attempt 1/4" in line # failed attempt 1 of 1 + 3 retries
|
||||
|
||||
|
||||
def test_chat_explicit_model_named_in_the_retry_warning(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""The warning names the model actually requested (an explicit
|
||||
*model* overrides the default)."""
|
||||
_record_sleeps(monkeypatch)
|
||||
caplog.set_level(logging.WARNING, logger="app.llm")
|
||||
llm, _ = _make_chat_client(
|
||||
completion_seq=[_empty(), _FakeCompletion("ok")],
|
||||
llm_summary_model="tiny",
|
||||
)
|
||||
out = asyncio.run(llm.chat([{"role": "user", "content": "q"}], model="special"))
|
||||
assert out == "ok"
|
||||
line = [r.getMessage() for r in caplog.records if r.levelno == logging.WARNING][0]
|
||||
assert "model=special" in line
|
||||
assert "tiny" not in line
|
||||
|
||||
|
||||
def test_chat_all_empty_exhausts_after_1_plus_retries_attempts(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Default ``llm_retries=3`` → exactly 4 attempts, 3 sleeps, then
|
||||
``LLMError`` naming the attempts. One empty reply omits
|
||||
``finish_reason`` (provider quirk) — the log line still formats
|
||||
(``finish_reason=None``) and never crashes the diagnostic path."""
|
||||
sleeps = _record_sleeps(monkeypatch)
|
||||
caplog.set_level(logging.WARNING, logger="app.llm")
|
||||
llm, completions = _make_chat_client(
|
||||
completion_seq=[_empty(), _empty(), _empty(None), _empty()]
|
||||
)
|
||||
with pytest.raises(LLMError) as exc:
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert str(exc.value) == (
|
||||
f"chat completion from {_DEFAULT_BASE} returned empty content on all "
|
||||
"4 attempts — refusing to store a silent summary"
|
||||
)
|
||||
assert completions.create_calls == 4
|
||||
assert sleeps == [5.0, 5.0, 5.0] # no sleep after the last attempt
|
||||
lines = [r.getMessage() for r in caplog.records if r.levelno == logging.WARNING]
|
||||
assert "attempt 1/4" in lines[0]
|
||||
assert "attempt 2/4" in lines[1]
|
||||
assert "attempt 3/4" in lines[2]
|
||||
assert "finish_reason=None" in lines[2] # the omitted-finish_reason reply
|
||||
|
||||
|
||||
def test_chat_all_empty_custom_retry_count_names_the_attempts(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""``llm_retries=1`` → exactly 2 attempts, 1 sleep, the exhaustion
|
||||
message names 2."""
|
||||
sleeps = _record_sleeps(monkeypatch)
|
||||
llm, completions = _make_chat_client(
|
||||
completion_seq=[_empty(), _empty()], llm_retries=1, llm_retry_delay=0.5
|
||||
)
|
||||
with pytest.raises(LLMError, match="all 2 attempts"):
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert completions.create_calls == 2
|
||||
assert sleeps == [0.5]
|
||||
|
||||
|
||||
def test_chat_first_success_never_retries(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Happy path untouched: exactly 1 ``create()`` call, ZERO sleeps,
|
||||
the trimmed content is returned byte-identically."""
|
||||
sleeps = _record_sleeps(monkeypatch)
|
||||
llm, completions = _make_chat_client(_FakeCompletion(" Summary text.\n"))
|
||||
out = asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert out == "Summary text."
|
||||
assert completions.create_calls == 1
|
||||
assert sleeps == []
|
||||
|
||||
|
||||
def test_chat_no_choices_reply_is_not_retried(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""D2: a choiceless reply raises immediately — 1 attempt, no sleep,
|
||||
no retry (only empty content is the retryable class)."""
|
||||
sleeps = _record_sleeps(monkeypatch)
|
||||
llm, completions = _make_chat_client(
|
||||
_FakeCompletion(None, empty_choices=True), llm_retries=3
|
||||
)
|
||||
with pytest.raises(LLMError, match="no choices"):
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert completions.create_calls == 1
|
||||
assert sleeps == []
|
||||
|
||||
|
||||
def test_chat_transport_failure_is_not_retried(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""D2: a transport failure raises immediately — 1 attempt, no sleep,
|
||||
no app-level retry (the openai SDK's own ``max_retries=2`` covers
|
||||
wire-level failures)."""
|
||||
sleeps = _record_sleeps(monkeypatch)
|
||||
llm, completions = _make_chat_client(fail=RuntimeError("HTTP 502 Bad Gateway"))
|
||||
with pytest.raises(LLMError, match="HTTP 502"):
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert completions.create_calls == 1
|
||||
assert completions.chat_calls == [] # the SDK call itself raised
|
||||
assert sleeps == []
|
||||
|
||||
|
||||
def test_chat_zero_retries_raises_legacy_message_byte_identical(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The kill switch (``llm_retries=0``): one attempt, zero sleeps, the
|
||||
PRE-phase-96 message byte-identically (asserted as the exact
|
||||
string, not a pattern)."""
|
||||
sleeps = _record_sleeps(monkeypatch)
|
||||
llm, completions = _make_chat_client(_FakeCompletion(None), llm_retries=0)
|
||||
with pytest.raises(LLMError) as exc:
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert str(exc.value) == (
|
||||
f"chat completion from {_DEFAULT_BASE} returned empty content — "
|
||||
"refusing to store a silent summary"
|
||||
)
|
||||
assert completions.create_calls == 1
|
||||
assert sleeps == []
|
||||
|
||||
|
||||
def test_chat_retry_delay_is_flat_never_backoff(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The recorded sleeps are the flat ``llm_retry_delay`` each time —
|
||||
never a growing backoff (the phase-67 convention)."""
|
||||
sleeps = _record_sleeps(monkeypatch)
|
||||
llm, _ = _make_chat_client(
|
||||
completion_seq=[_empty(), _empty(), _empty(), _empty()],
|
||||
llm_retries=3,
|
||||
llm_retry_delay=1.25,
|
||||
)
|
||||
with pytest.raises(LLMError, match="all 4 attempts"):
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert sleeps == [1.25, 1.25, 1.25]
|
||||
|
||||
|
||||
# ---------- chat_stream_retried (phase 67, task 01) ----------
|
||||
|
||||
_RETRY_MSGS: list[dict[str, str]] = [{"role": "user", "content": "q"}]
|
||||
|
||||
@@ -785,11 +785,19 @@ def _patch_sync_seams(
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
||||
|
||||
async def fake_folder_summaries(
|
||||
db: object, llm: object, *, skip: bool = False
|
||||
db: object,
|
||||
llm: object,
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
) -> dict[str, int]:
|
||||
return {"generated": 0, "failed": 0, "pruned": 0}
|
||||
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folder_summaries)
|
||||
# Phase 96 (task 03): the unchanged-walk gap probe is DB-free in
|
||||
# these state-machine tests — no gap, so the folder step stays
|
||||
# skipped exactly as before the gate change.
|
||||
monkeypatch.setattr(sync_api, "missing_folder_summaries", lambda session: [])
|
||||
|
||||
class _DummySession:
|
||||
def close(self) -> None:
|
||||
|
||||
Reference in New Issue
Block a user