phase: 96_oneshot_resilience
All checks complete. Final report: **Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design) - `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff - `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe - `.env.example` comments updated (chat-turn stream + one-shot summary calls) **Test / lint / coverage results** - `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%) - `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated) - Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files). **Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`.
This commit is contained in:
+131
-1
@@ -385,7 +385,38 @@ test_llm_retry.py``). The mock is single-conversation per e2e server, so
|
||||
SDK-level retries), so one POST per attempt: the counter is per
|
||||
POST here, unlike the chat counter above.
|
||||
Non-streaming requests (document summaries, KB overview) never 500 —
|
||||
the retry scope is the chat turn only (owner-locked A1).
|
||||
the retry scope is the chat turn only (owner-locked A1). The one
|
||||
non-streaming injection is phase 96's incident shape below (it
|
||||
answers 200 with EMPTY content — the semantic failure class, not a
|
||||
dead endpoint).
|
||||
|
||||
Failure injection (phase 96, one-shot resilience, task 04) — the
|
||||
2026-09-11 incident shape for the folder-summary one-shot path
|
||||
(``tests/e2e/test_oneshot_llm_retry.py``): a NON-stream
|
||||
``chat/completions`` request whose system prompt carries
|
||||
``FOLDER_SUMMARY_MODE`` (the folder-summary marker — ``chat()`` is
|
||||
the mock's only non-streaming consumer of it) and whose user
|
||||
message's ``Folder: …`` header (the branch's existing parse, the
|
||||
``FOLDER_HEADER_PREFIX`` tail) labels this suite's own fixture
|
||||
folders:
|
||||
- the label ends with ``/e2e_empty_once``: the FIRST non-stream POST
|
||||
for that label answers the incident envelope — the mock's normal
|
||||
OpenAI chat-completion shape with ``choices[0].message.content =
|
||||
""`` and ``choices[0].finish_reason = "length"`` (the exact wire
|
||||
shape of the empty ``lite`` reply: the budget spent in
|
||||
``reasoning_content``) — and every later POST returns the normal
|
||||
``Fixture folder summary for <folder>.`` line (the one-shot retry,
|
||||
phase 96 task 01, recovers the row).
|
||||
- the label ends with ``/e2e_empty_always``: EVERY non-stream POST
|
||||
for that label answers the empty envelope (the 1 +
|
||||
``BOR_LLM_RETRIES`` exhaustion → per-folder fail-soft → the row
|
||||
stays absent while the sync stays green, phase 94 contract).
|
||||
The once-sequence is driven by a module-level per-label counter that
|
||||
resets after the success it guards (the phase-67 ``_fail_posts``
|
||||
pattern — the mock is single-conversation per e2e server), so a
|
||||
second sync re-drives the sequence deterministically. The trigger
|
||||
strings are this suite's own folder names, so no other E2E can hit
|
||||
them (they seed different trees).
|
||||
|
||||
``max_tokens`` is honored deterministically (token ≈ whitespace word),
|
||||
like a real endpoint: an answer longer than the cap is truncated. This
|
||||
@@ -1496,6 +1527,70 @@ def _history_echo(body: dict[str, Any]) -> str:
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 96 (task 04, one-shot resilience): the incident-shape injection
|
||||
# for the folder-summary one-shot path — see the module docstring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: The folder-label triggers (phase 96, task 04): the ``FOLDER_SUMMARY_MODE``
|
||||
#: branch's folder label (the user message's ``Folder: …`` tail — the
|
||||
#: ``FOLDER_HEADER_PREFIX`` parse) ending with these suffixes is THIS
|
||||
#: SUITE'S own fixture folder (``tests/e2e/test_oneshot_llm_retry.py``
|
||||
#: seeds the names — no other E2E can hit them; they seed different
|
||||
#: trees, and the triggers carry the E2E prefix).
|
||||
ONESHOT_EMPTY_ONCE_SUFFIX = "/e2e_empty_once"
|
||||
ONESHOT_EMPTY_ALWAYS_SUFFIX = "/e2e_empty_always"
|
||||
|
||||
#: Module-level per-label incident counter — the mock is single-
|
||||
#: conversation per e2e server (the phase-67 ``_fail_posts``
|
||||
#: convention). Counts the non-stream folder-summary POSTs served per
|
||||
#: trigger label; the once-sequence resets after the success it guards
|
||||
#: (the first normal reply), so a second sync re-drives the sequence
|
||||
#: deterministically.
|
||||
_empty_once_posts: dict[str, int] = {}
|
||||
|
||||
|
||||
def _folder_summary_incident(body: dict[str, Any]) -> bool:
|
||||
"""Should this NON-stream folder-summary POST answer with the
|
||||
2026-09-11 incident envelope (``content=""`` +
|
||||
``finish_reason="length"`` — the exact wire shape of the empty
|
||||
``lite`` reply phase 96's one-shot retry targets)?
|
||||
|
||||
* the system prompt lacks ``FOLDER_SUMMARY_MODE`` → never (only the
|
||||
folder-summary one-shot carries the marker — ``chat()`` is the
|
||||
mock's only non-streaming consumer of it, so the counter counts
|
||||
exactly the one-shot POSTs the app's retry policy drives);
|
||||
* the label ends with ``/e2e_empty_always`` → EVERY POST (the
|
||||
exhaustion path — 1 + ``BOR_LLM_RETRIES`` empty attempts, the
|
||||
per-folder fail-soft leaves the row absent);
|
||||
* the label ends with ``/e2e_empty_once`` → the FIRST non-stream
|
||||
POST for that label only — every later POST returns the normal
|
||||
line (the retry recovers the row), and the counter resets on
|
||||
that first normal reply (the phase-67 pattern).
|
||||
|
||||
The folder label is the ``FOLDER_SUMMARY_MODE`` branch's existing
|
||||
parse (the user message's first line, the ``FOLDER_HEADER_PREFIX``
|
||||
tail) — the injection is a pure function of the request plus the
|
||||
per-label counter (the house marker-flow convention).
|
||||
"""
|
||||
if "FOLDER_SUMMARY_MODE" not in _system(body):
|
||||
return False
|
||||
user = _user(body)
|
||||
header = user.splitlines()[0] if user else ""
|
||||
if not header.startswith(FOLDER_HEADER_PREFIX):
|
||||
return False
|
||||
label = header.removeprefix(FOLDER_HEADER_PREFIX).strip()
|
||||
if label.endswith(ONESHOT_EMPTY_ALWAYS_SUFFIX):
|
||||
return True
|
||||
if label.endswith(ONESHOT_EMPTY_ONCE_SUFFIX):
|
||||
n = _empty_once_posts.get(label, 0) + 1
|
||||
_empty_once_posts[label] = n
|
||||
if n == 1:
|
||||
return True # the first POST: the incident envelope
|
||||
_empty_once_posts[label] = 0 # the retry went out — restart
|
||||
return False
|
||||
|
||||
|
||||
def compose_answer(body: dict[str, Any]) -> str:
|
||||
system = _system(body)
|
||||
user = _user(body)
|
||||
@@ -1516,6 +1611,12 @@ def compose_answer(body: dict[str, Any]) -> str:
|
||||
# shadow every folder-summary call. Checked BEFORE the
|
||||
# DEFLECT_MODE branch, like the other lite-mode markers (a
|
||||
# deflection prompt never carries one).
|
||||
# Phase 96 (task 04): the incident-shape injection keys on THIS
|
||||
# branch's label (``_folder_summary_incident`` — the module
|
||||
# docstring) and overrides the NON-STREAM response envelope in
|
||||
# ``chat_completions`` (``content=""`` +
|
||||
# ``finish_reason="length"``); the line below is what the later
|
||||
# / normal POSTs return.
|
||||
header = user.splitlines()[0] if user else ""
|
||||
folder = (
|
||||
header.removeprefix(FOLDER_HEADER_PREFIX).strip()
|
||||
@@ -2167,6 +2268,35 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
)
|
||||
|
||||
if not body.get("stream"):
|
||||
# Phase 96 (task 04): the incident-shape injection (the module
|
||||
# docstring) — the trigger-labelled folder summary answers the
|
||||
# exact 2026-09-11 envelope: the mock's normal OpenAI
|
||||
# chat-completion shape with ``content=""`` and
|
||||
# ``finish_reason="length"`` (the budget spent in
|
||||
# ``reasoning_content``). ``chat()`` is the mock's only
|
||||
# non-streaming folder-summary consumer, so this is the one-shot
|
||||
# retry path (``app.rag.llm.LLMClient.chat``, phase 96 task 01)
|
||||
# and nothing else — every other non-stream response is
|
||||
# byte-identical to pre-phase-96.
|
||||
if _folder_summary_incident(body):
|
||||
return {
|
||||
"id": f"chatcmpl-{uuid.uuid4()}",
|
||||
"object": "chat.completion",
|
||||
"created": int(time.time()),
|
||||
"model": body.get("model", "turbo"),
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {"role": "assistant", "content": ""},
|
||||
"finish_reason": "length",
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 100,
|
||||
"completion_tokens": 2048,
|
||||
"total_tokens": 2148,
|
||||
},
|
||||
}
|
||||
message: dict[str, Any] = {"role": "assistant", "content": answer}
|
||||
if thinking:
|
||||
# Harmless future-proofing: the app only uses streaming, but a
|
||||
|
||||
@@ -0,0 +1,633 @@
|
||||
"""Phase 96 task 04 E2E (Playwright, mock-only): the one-shot LLM
|
||||
resilience — the 2026-09-11 incident shape, retried and healed.
|
||||
|
||||
The dedicated story suite for ``96_oneshot_resilience`` (A16 — one
|
||||
Playwright file per phase, run in isolation): a folder whose FIRST
|
||||
one-shot summary reply arrives in the exact incident shape
|
||||
(``content=""`` + ``finish_reason="length"``) still ends up with its
|
||||
stored summary (the task-01 retry recovered it, visible in the
|
||||
``ls`` drill-down), a folder whose replies are ALWAYS empty stays
|
||||
absent without failing the sync (the task-01 exhaustion + the phase-94
|
||||
per-folder fail-soft contract), and a row deleted behind the app's back
|
||||
is self-healed by the next UNCHANGED sync with the other rows untouched
|
||||
(tasks 02/03 targeted fill).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic incident-shape injection in ``tests/e2e/mock_llm.py``
|
||||
(phase 96, task 04): a NON-stream ``chat/completions`` request whose
|
||||
system prompt carries ``FOLDER_SUMMARY_MODE`` and whose ``Folder: …``
|
||||
label ends with ``/e2e_empty_once`` answers the incident envelope
|
||||
(``content=""`` + ``finish_reason="length"`` — the mock's normal OpenAI
|
||||
shape) on its FIRST non-stream POST only, and a label ending with
|
||||
``/e2e_empty_always`` answers it on EVERY non-stream POST. The
|
||||
per-label counter resets after the success it guards (the phase-67
|
||||
``_fail_posts`` pattern), so a re-run of the suite is green without
|
||||
manual state cleanup. The trigger strings are this suite's own folder
|
||||
names, so no other E2E can hit them.
|
||||
|
||||
The e2e app server boots with ``BOR_LLM_RETRY_DELAY=0`` (the conftest
|
||||
pattern) so the 4-attempt exhaustion paths are instant;
|
||||
``BOR_LLM_RETRIES`` is forced to the code default (3 — the conftest
|
||||
leak-guard pattern), so ``e2e_empty_always`` costs exactly 4 POSTs per
|
||||
sync that attempts it and the suite stays fast.
|
||||
|
||||
KB fixture — a host temp dir tree (``tmp_path_factory``; the app runs
|
||||
on the same host) with ONE registered local source (the
|
||||
``test_local_directory_sources.py`` registration + real-Sync pattern;
|
||||
no git anywhere): ``oneshot/`` with three ≥ 2-doc folders —
|
||||
``e2e_empty_once/`` (2 docs), ``e2e_empty_always/`` (2), ``normal/``
|
||||
(2). Every fixture doc carries the words ``drill down the tree`` in
|
||||
its body, so the scripted ``ls`` drill-down questions (the phase-94
|
||||
``DRILL_TRIGGER`` echo pattern — the mock echoes the received tool
|
||||
result into its grounded answer, the E2E's only lens on the LLM's
|
||||
context) FTS-match at least one chunk and run grounded.
|
||||
|
||||
Test → phase mapping (Playwright Mapping Rule):
|
||||
1. ``test_incident_reply_retried_and_always_empty_stays_absent`` —
|
||||
after the changed sync #1 (the module fixture pins the stored rows:
|
||||
the ``e2e_empty_once`` row EXISTS — the retry recovered it, without
|
||||
task 01 it would be absent — and the ``e2e_empty_always`` row does
|
||||
NOT — all 4 attempts empty → ``LLMError`` → per-folder fail-soft),
|
||||
a scripted ``ls oneshot`` turn asserts the drill-down listing: the
|
||||
``e2e_empty_once/`` line carries
|
||||
``: Fixture folder summary for oneshot/e2e_empty_once.`` (the retry
|
||||
recovered the row), the ``normal/`` line carries its summary, and
|
||||
the ``e2e_empty_always/`` line is ``e2e_empty_always/ — 2
|
||||
documents`` with NO ``: …`` suffix — while the sync reported
|
||||
success (a folder-summary exhaustion never flips the run, phase 94).
|
||||
2. ``test_deleted_row_self_heals_on_unchanged_sync`` — the ``normal``
|
||||
stored row is deleted directly (simulating a historical failure),
|
||||
``e2e_empty_once``'s row ``updated_at`` is captured, and the
|
||||
UNCHANGED sync #2 runs the gap gate (task 03): a second scripted
|
||||
``ls oneshot`` turn shows ``normal/`` healed
|
||||
(``: Fixture folder summary for oneshot/normal.`` again) and
|
||||
``e2e_empty_always/`` still absent (the gap-fill attempted it,
|
||||
exhausted, stayed absent — the sync still succeeded), and the DB
|
||||
pins the targeted fill: the ``normal`` row is back with its
|
||||
deterministic text and the ``e2e_empty_once`` row's ``updated_at``
|
||||
is UNCHANGED (a full regeneration would have re-stamped it).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import FolderSummary
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module app binds its own port instead
|
||||
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
# server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_ONESHOT", "8138"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture documents + the pinned drill-down listing (deterministic)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: The local source — the temp directory's basename (``kind=local`` →
|
||||
#: the directory's basename is the source name, phase 38).
|
||||
SOURCE = "oneshot"
|
||||
|
||||
#: The three ≥ 2-doc folders (the mock's trigger labels are the folder
|
||||
#: NAMES — the seeded KB carries them, the house marker-flow
|
||||
#: convention). Path order (the ``ls`` subfolder order):
|
||||
#: e2e_empty_always < e2e_empty_once < normal.
|
||||
FOLDER_ONCE = "e2e_empty_once"
|
||||
FOLDER_ALWAYS = "e2e_empty_always"
|
||||
FOLDER_NORMAL = "normal"
|
||||
|
||||
TOTAL_DOCS = 6 # three folders x 2 docs each
|
||||
|
||||
#: Every fixture body carries ``drill down the tree`` (the
|
||||
#: ``DRILL_TRIGGER`` phrase's words): every scripted question
|
||||
#: FTS-matches at least one chunk → HIGH gate → the ``<tools>`` section
|
||||
#: the drill-down flow keys on (the phase-94 seed convention).
|
||||
DRILL_LEAD = "The drill down the tree fixture note"
|
||||
|
||||
|
||||
def _md(title: str, body: str) -> str:
|
||||
return f"# {title}\n\n{body}\n"
|
||||
|
||||
|
||||
#: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this
|
||||
#: fixture (the phase-94 template — the label is the
|
||||
#: ``FOLDER_HEADER_PREFIX`` tail: ``<source>`` for the root,
|
||||
#: ``<source>/<folder>`` for a folder).
|
||||
SUM_ROOT = f"Fixture folder summary for {SOURCE}."
|
||||
SUM_ONCE = f"Fixture folder summary for {SOURCE}/{FOLDER_ONCE}."
|
||||
SUM_ALWAYS = f"Fixture folder summary for {SOURCE}/{FOLDER_ALWAYS}."
|
||||
SUM_NORMAL = f"Fixture folder summary for {SOURCE}/{FOLDER_NORMAL}."
|
||||
|
||||
# --- the pinned ``ls oneshot`` level (app.rag.agent's phase-94 template)
|
||||
|
||||
#: The source root: 0 direct files, 3 subfolders (path order).
|
||||
LS_HEADER = f"{SOURCE} — 0 documents, 3 folders:"
|
||||
|
||||
#: The subfolder lines — the ``: {summary}`` suffix appended ONLY when
|
||||
#: the subfolder's summary is stored (``render_folder_listing``; the
|
||||
#: renderer's 2-space indent is whitespace-normalized away by the
|
||||
#: ``to_contain_text`` match — the phase-94 pin convention).
|
||||
LINE_ALWAYS = f"{FOLDER_ALWAYS}/ — 2 documents"
|
||||
LINE_ALWAYS_WITH_COLON = f"{FOLDER_ALWAYS}/ — 2 documents: "
|
||||
LINE_ONCE = f"{FOLDER_ONCE}/ — 2 documents: {SUM_ONCE}"
|
||||
LINE_NORMAL = f"{FOLDER_NORMAL}/ — 2 documents: {SUM_NORMAL}"
|
||||
|
||||
# --- the scripted drill-down turns (the phase-94 ``DRILL_TRIGGER``
|
||||
# questions — the mock echoes the listing verbatim into the answer) ---
|
||||
|
||||
LS_QUESTION_1 = f"Drill down the tree: ls {SOURCE} — what's in source {SOURCE}?"
|
||||
LS_QUESTION_2 = f"Drill down the tree: ls {SOURCE} — list the source folders again"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def oneshot_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The one-source temp tree (see the module docstring): the app
|
||||
server runs on the same host, so the paths are visible to it."""
|
||||
root = tmp_path_factory.mktemp("bor_oneshot")
|
||||
src = root / SOURCE
|
||||
for folder, prefix in (
|
||||
(FOLDER_ONCE, "once"),
|
||||
(FOLDER_ALWAYS, "always"),
|
||||
(FOLDER_NORMAL, "normal"),
|
||||
):
|
||||
(src / folder).mkdir(parents=True)
|
||||
for letter, topic in (("a", "A"), ("b", "B")):
|
||||
(src / folder / f"{prefix}-{letter}.md").write_text(
|
||||
_md(
|
||||
f"Oneshot {prefix.title()} {letter.upper()}",
|
||||
f"{DRILL_LEAD} for {SOURCE} {folder} "
|
||||
f"{letter}: this document covers topic {topic} "
|
||||
f"of the {SOURCE} source tree.",
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert len(list(src.rglob("*.md"))) == TOTAL_DOCS
|
||||
return src
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, oneshot_dir: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module app (the conftest pattern,
|
||||
cf. ``test_local_directory_sources.py``): NO ``BOR_GIT_SOURCES``
|
||||
(the env fallback is git-only — the source here is a DB-registered
|
||||
local directory), the mock LLM, the mock-calibrated threshold, and
|
||||
the leak-guarded code defaults. ``BOR_LLM_RETRY_DELAY=0`` + the
|
||||
code-default ``BOR_LLM_RETRIES`` (the phase-67 conftest pattern):
|
||||
the one-shot retry waits are instant and the exhaustion budget is
|
||||
the REAL one (4 attempts). The session app is never started in this
|
||||
isolated run, so no port clash."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated threshold (conftest pattern): every scripted
|
||||
# question FTS-matches the fixture docs (the ``drill down the
|
||||
# tree`` words), so the gate is HIGH either way.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
# Phase 67 / phase 96: instant retry waits + the code-default budget
|
||||
# (the conftest leak-guard pattern) — the one-shot retry and the
|
||||
# 4-attempt exhaustion run in real time at zero delay.
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the registry must hold EXACTLY the one local
|
||||
# directory this suite registers.
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
||||
share one Postgres, so a leftover source or document would pollute
|
||||
the ``ls`` listing the drill answers assert on byte-exactly."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _run_sync_http(base_url: str, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Login + ``POST /api/sync`` + poll the status endpoint until the
|
||||
run reaches a terminal state (the ``test_sync_button.py`` /
|
||||
``test_local_directory_sources.py`` pattern, over plain httpx)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _folder_rows() -> dict[str, tuple[str, datetime]]:
|
||||
"""The source's stored folder summaries
|
||||
``{folder_path: (summary, updated_at)}`` (``""`` = the source
|
||||
root) — the test process's direct DB access, the E2E's other
|
||||
established lens."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(
|
||||
select(
|
||||
FolderSummary.folder_path,
|
||||
FolderSummary.summary,
|
||||
FolderSummary.updated_at,
|
||||
).where(FolderSummary.source == SOURCE)
|
||||
).all()
|
||||
return {folder: (summary, updated_at) for folder, summary, updated_at in rows}
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, oneshot_dir: Path) -> None:
|
||||
"""The story's precondition: the KB synced under the deterministic
|
||||
mock's incident-shape injection.
|
||||
|
||||
Registers the temp directory through the authenticated API (the
|
||||
``test_local_directory_sources.py`` pattern) and runs the REAL
|
||||
in-process sync #1 (``POST /api/sync`` — walk → chunk → embed →
|
||||
overview → folder summaries → version bump). The sync changed the
|
||||
KB → FULL folder regeneration, and the mock's injection drives the
|
||||
incident shapes:
|
||||
|
||||
* ``oneshot`` (the root) + ``oneshot/normal`` — normal replies,
|
||||
one POST each;
|
||||
* ``oneshot/e2e_empty_once`` — the FIRST non-stream POST answers
|
||||
the incident envelope (``content=""`` + ``finish_reason=
|
||||
"length"``), the retry (task 01) recovers the row on the second
|
||||
POST;
|
||||
* ``oneshot/e2e_empty_always`` — EVERY non-stream POST answers the
|
||||
empty envelope: 1 + ``BOR_LLM_RETRIES`` = 4 attempts exhausted →
|
||||
``LLMError`` → the phase-94 per-folder fail-soft leaves the row
|
||||
ABSENT and never flips the run.
|
||||
|
||||
The fixture pins all of that in the DB: the ``e2e_empty_once`` row
|
||||
EXISTS (without task 01 it would be absent — the incident's data
|
||||
loss), the ``e2e_empty_always`` row does NOT, and the sync still
|
||||
reported ``success``.
|
||||
"""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(oneshot_dir)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
body = _run_sync_http(app_server)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == TOTAL_DOCS, detail
|
||||
assert detail["updated"] == 0, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is True, detail
|
||||
# The full regeneration under the injection: the retried row
|
||||
# EXISTS (the incident, healed by task 01's one-shot retry), the
|
||||
# exhausted row is ABSENT (the per-folder fail-soft — the sync
|
||||
# above still reported success), the normal rows landed.
|
||||
rows = _folder_rows()
|
||||
assert set(rows) == {"", FOLDER_ONCE, FOLDER_NORMAL}, rows
|
||||
assert rows[""][0] == SUM_ROOT, rows
|
||||
assert rows[FOLDER_ONCE][0] == SUM_ONCE, rows
|
||||
assert rows[FOLDER_NORMAL][0] == SUM_NORMAL, rows
|
||||
assert FOLDER_ALWAYS not in rows, rows # all 4 attempts empty → no row
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||||
the drill turns never change it, so the folder summaries and the
|
||||
registry persist across the tests of this module)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the phase-94 drill-down house pattern)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` frames, independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page) -> list[dict]:
|
||||
"""The SSE frames captured since the last submit (``_submit``
|
||||
clears the buffer), once the hook's background read settles."""
|
||||
deadline = time.monotonic() + 30.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered
|
||||
(the phase-48 settle wait)."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=60_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=60_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=60_000)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
def _assert_drill_turn(
|
||||
page: Page,
|
||||
expected_tool: dict[str, Any],
|
||||
expected_lines: list[str],
|
||||
forbidden: list[str] | None = None,
|
||||
) -> None:
|
||||
"""One scripted drill turn, fully asserted: the wire carries exactly
|
||||
the expected ``tool`` frame (ahead of the first ``delta``), the
|
||||
bubble carries the expected listing lines (the mock's echo of the
|
||||
tool result the model received), the forbidden substrings are ABSENT
|
||||
(the no-suffix assertions), and the turn was grounded (the ``done``
|
||||
frame is not deflected)."""
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [expected_tool], _tool_frames(frames)
|
||||
first_delta = next(
|
||||
i for i, f in enumerate(frames) if f.get("type") == "delta"
|
||||
)
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
for line in expected_lines:
|
||||
expect(bubble).to_contain_text(line)
|
||||
text = bubble.text_content() or ""
|
||||
for needle in forbidden or []:
|
||||
assert needle not in text, text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. Sync #1: the incident shape is retried (the row lands) and the
|
||||
# always-empty folder stays absent with the sync green
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_incident_reply_retried_and_always_empty_stays_absent(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The ``ls oneshot`` drill-down shows the stored folder summaries
|
||||
the sync #1 produced under the incident-shape injection: the
|
||||
``e2e_empty_once/`` line carries its summary (the retry recovered
|
||||
the row — without task 01 the line would have NO ``: …`` suffix),
|
||||
the ``normal/`` line does too, and the ``e2e_empty_always/`` line
|
||||
is bare (exhausted → fail-soft → absent) — while the sync above
|
||||
reported success (the folder-stats failure never flips the run,
|
||||
phase 94)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, LS_QUESTION_1)
|
||||
_wait_settled(page)
|
||||
lines = _last_brain(page).locator(".tool-call")
|
||||
expect(lines).to_have_count(1)
|
||||
expect(lines.nth(0)).to_contain_text(f"Listing documents in {SOURCE}")
|
||||
_assert_drill_turn(
|
||||
page,
|
||||
{"type": "tool", "name": "ls", "argument": SOURCE},
|
||||
[
|
||||
LS_HEADER,
|
||||
# The bare line (the row is absent — no stored summary to
|
||||
# append) …
|
||||
LINE_ALWAYS,
|
||||
# …and the colon-suffixed lines (the rows the retry + the
|
||||
# normal path stored):
|
||||
LINE_ONCE,
|
||||
LINE_NORMAL,
|
||||
],
|
||||
# The ``e2e_empty_always/`` line must NOT carry a ``: …``
|
||||
# suffix — the bare line above is a prefix of the suffixed
|
||||
# shape, so the absence is pinned here (the all-4-attempts
|
||||
# exhaustion left no row to quote).
|
||||
forbidden=[LINE_ALWAYS_WITH_COLON],
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. The gap-fill: a deleted row self-heals on the next UNCHANGED sync,
|
||||
# the other rows untouched
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_deleted_row_self_heals_on_unchanged_sync(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""Delete the ``normal`` stored row behind the app's back (a
|
||||
historical failure), run the UNCHANGED sync #2, and assert the
|
||||
task-02/03 gap gate: ``missing_folder_summaries`` names the gap,
|
||||
``only_missing=True`` fills EXACTLY the missing rows, and every
|
||||
other row stays byte-identical (text AND ``updated_at``).
|
||||
|
||||
* ``normal`` — healed: the second ``ls oneshot`` turn carries
|
||||
``: Fixture folder summary for oneshot/normal.`` again, and the
|
||||
DB row is back with its deterministic text;
|
||||
* ``e2e_empty_once`` — the gap-fill NEVER calls it (the row
|
||||
exists): its ``updated_at`` is UNCHANGED (a full regeneration
|
||||
would have re-stamped it);
|
||||
* ``e2e_empty_always`` — the gap-fill attempted it, exhausted
|
||||
(4 empty POSTs by design), stayed ABSENT — and the sync still
|
||||
succeeded (the fail-soft never flips the run)."""
|
||||
page.set_default_timeout(30_000)
|
||||
# The gap: delete the ``normal`` row directly (the test process has
|
||||
# DB access via the conftest engine — the same connection the app
|
||||
# uses), capturing the other row's stamp for the targeted-fill
|
||||
# assertion. No KB change anywhere.
|
||||
before = _folder_rows()
|
||||
assert FOLDER_NORMAL in before # the row existed (sync #1 stored it)
|
||||
updated_at_once = before[FOLDER_ONCE][1]
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries "
|
||||
"WHERE source = :s AND folder_path = :f"
|
||||
),
|
||||
{"s": SOURCE, "f": FOLDER_NORMAL},
|
||||
)
|
||||
db.commit()
|
||||
assert FOLDER_NORMAL not in _folder_rows()
|
||||
|
||||
# Sync #2 — the KB is UNCHANGED (nothing in the temp tree moved),
|
||||
# so the gate takes the gap probe: two candidates missing
|
||||
# (``e2e_empty_always`` + ``normal``) → the targeted fill.
|
||||
body = _run_sync_http(app_url)
|
||||
assert body["state"] == "success", body # exhaustion never flips the run
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 0, detail
|
||||
assert detail["updated"] == 0, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is False, detail # unchanged → no overview burn
|
||||
|
||||
# The second scripted drill turn: the healed + the surviving lines,
|
||||
# the always folder still bare.
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
_submit(page, LS_QUESTION_2)
|
||||
_wait_settled(page)
|
||||
lines = _last_brain(page).locator(".tool-call")
|
||||
expect(lines).to_have_count(1)
|
||||
expect(lines.nth(0)).to_contain_text(f"Listing documents in {SOURCE}")
|
||||
_assert_drill_turn(
|
||||
page,
|
||||
{"type": "tool", "name": "ls", "argument": SOURCE},
|
||||
[
|
||||
LS_HEADER,
|
||||
LINE_ALWAYS,
|
||||
LINE_ONCE,
|
||||
LINE_NORMAL, # healed — the suffix is back
|
||||
],
|
||||
forbidden=[LINE_ALWAYS_WITH_COLON],
|
||||
)
|
||||
|
||||
# The DB pins the targeted fill (the E2E's other established lens):
|
||||
rows = _folder_rows()
|
||||
assert set(rows) == {"", FOLDER_ONCE, FOLDER_NORMAL}, rows
|
||||
assert rows[FOLDER_NORMAL][0] == SUM_NORMAL, rows # deterministic text
|
||||
# The targeted fill never touched the other rows — a FULL
|
||||
# regeneration would have re-stamped this one (the ``_upsert``
|
||||
# fresh-UTC-stamp rule).
|
||||
assert rows[FOLDER_ONCE][1] == updated_at_once, (
|
||||
rows[FOLDER_ONCE][1],
|
||||
updated_at_once,
|
||||
)
|
||||
assert FOLDER_ALWAYS not in rows, rows # exhausted again → still absent
|
||||
Reference in New Issue
Block a user