phase: 96_oneshot_resilience
All checks complete. Final report: **Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design) - `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff - `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe - `.env.example` comments updated (chat-turn stream + one-shot summary calls) **Test / lint / coverage results** - `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%) - `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated) - Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files). **Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`.
This commit is contained in:
@@ -0,0 +1,633 @@
|
||||
"""Phase 96 task 04 E2E (Playwright, mock-only): the one-shot LLM
|
||||
resilience — the 2026-09-11 incident shape, retried and healed.
|
||||
|
||||
The dedicated story suite for ``96_oneshot_resilience`` (A16 — one
|
||||
Playwright file per phase, run in isolation): a folder whose FIRST
|
||||
one-shot summary reply arrives in the exact incident shape
|
||||
(``content=""`` + ``finish_reason="length"``) still ends up with its
|
||||
stored summary (the task-01 retry recovered it, visible in the
|
||||
``ls`` drill-down), a folder whose replies are ALWAYS empty stays
|
||||
absent without failing the sync (the task-01 exhaustion + the phase-94
|
||||
per-folder fail-soft contract), and a row deleted behind the app's back
|
||||
is self-healed by the next UNCHANGED sync with the other rows untouched
|
||||
(tasks 02/03 targeted fill).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic incident-shape injection in ``tests/e2e/mock_llm.py``
|
||||
(phase 96, task 04): a NON-stream ``chat/completions`` request whose
|
||||
system prompt carries ``FOLDER_SUMMARY_MODE`` and whose ``Folder: …``
|
||||
label ends with ``/e2e_empty_once`` answers the incident envelope
|
||||
(``content=""`` + ``finish_reason="length"`` — the mock's normal OpenAI
|
||||
shape) on its FIRST non-stream POST only, and a label ending with
|
||||
``/e2e_empty_always`` answers it on EVERY non-stream POST. The
|
||||
per-label counter resets after the success it guards (the phase-67
|
||||
``_fail_posts`` pattern), so a re-run of the suite is green without
|
||||
manual state cleanup. The trigger strings are this suite's own folder
|
||||
names, so no other E2E can hit them.
|
||||
|
||||
The e2e app server boots with ``BOR_LLM_RETRY_DELAY=0`` (the conftest
|
||||
pattern) so the 4-attempt exhaustion paths are instant;
|
||||
``BOR_LLM_RETRIES`` is forced to the code default (3 — the conftest
|
||||
leak-guard pattern), so ``e2e_empty_always`` costs exactly 4 POSTs per
|
||||
sync that attempts it and the suite stays fast.
|
||||
|
||||
KB fixture — a host temp dir tree (``tmp_path_factory``; the app runs
|
||||
on the same host) with ONE registered local source (the
|
||||
``test_local_directory_sources.py`` registration + real-Sync pattern;
|
||||
no git anywhere): ``oneshot/`` with three ≥ 2-doc folders —
|
||||
``e2e_empty_once/`` (2 docs), ``e2e_empty_always/`` (2), ``normal/``
|
||||
(2). Every fixture doc carries the words ``drill down the tree`` in
|
||||
its body, so the scripted ``ls`` drill-down questions (the phase-94
|
||||
``DRILL_TRIGGER`` echo pattern — the mock echoes the received tool
|
||||
result into its grounded answer, the E2E's only lens on the LLM's
|
||||
context) FTS-match at least one chunk and run grounded.
|
||||
|
||||
Test → phase mapping (Playwright Mapping Rule):
|
||||
1. ``test_incident_reply_retried_and_always_empty_stays_absent`` —
|
||||
after the changed sync #1 (the module fixture pins the stored rows:
|
||||
the ``e2e_empty_once`` row EXISTS — the retry recovered it, without
|
||||
task 01 it would be absent — and the ``e2e_empty_always`` row does
|
||||
NOT — all 4 attempts empty → ``LLMError`` → per-folder fail-soft),
|
||||
a scripted ``ls oneshot`` turn asserts the drill-down listing: the
|
||||
``e2e_empty_once/`` line carries
|
||||
``: Fixture folder summary for oneshot/e2e_empty_once.`` (the retry
|
||||
recovered the row), the ``normal/`` line carries its summary, and
|
||||
the ``e2e_empty_always/`` line is ``e2e_empty_always/ — 2
|
||||
documents`` with NO ``: …`` suffix — while the sync reported
|
||||
success (a folder-summary exhaustion never flips the run, phase 94).
|
||||
2. ``test_deleted_row_self_heals_on_unchanged_sync`` — the ``normal``
|
||||
stored row is deleted directly (simulating a historical failure),
|
||||
``e2e_empty_once``'s row ``updated_at`` is captured, and the
|
||||
UNCHANGED sync #2 runs the gap gate (task 03): a second scripted
|
||||
``ls oneshot`` turn shows ``normal/`` healed
|
||||
(``: Fixture folder summary for oneshot/normal.`` again) and
|
||||
``e2e_empty_always/`` still absent (the gap-fill attempted it,
|
||||
exhausted, stayed absent — the sync still succeeded), and the DB
|
||||
pins the targeted fill: the ``normal`` row is back with its
|
||||
deterministic text and the ``e2e_empty_once`` row's ``updated_at``
|
||||
is UNCHANGED (a full regeneration would have re-stamped it).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import FolderSummary
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module app binds its own port instead
|
||||
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
# server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_ONESHOT", "8138"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture documents + the pinned drill-down listing (deterministic)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: The local source — the temp directory's basename (``kind=local`` →
|
||||
#: the directory's basename is the source name, phase 38).
|
||||
SOURCE = "oneshot"
|
||||
|
||||
#: The three ≥ 2-doc folders (the mock's trigger labels are the folder
|
||||
#: NAMES — the seeded KB carries them, the house marker-flow
|
||||
#: convention). Path order (the ``ls`` subfolder order):
|
||||
#: e2e_empty_always < e2e_empty_once < normal.
|
||||
FOLDER_ONCE = "e2e_empty_once"
|
||||
FOLDER_ALWAYS = "e2e_empty_always"
|
||||
FOLDER_NORMAL = "normal"
|
||||
|
||||
TOTAL_DOCS = 6 # three folders x 2 docs each
|
||||
|
||||
#: Every fixture body carries ``drill down the tree`` (the
|
||||
#: ``DRILL_TRIGGER`` phrase's words): every scripted question
|
||||
#: FTS-matches at least one chunk → HIGH gate → the ``<tools>`` section
|
||||
#: the drill-down flow keys on (the phase-94 seed convention).
|
||||
DRILL_LEAD = "The drill down the tree fixture note"
|
||||
|
||||
|
||||
def _md(title: str, body: str) -> str:
|
||||
return f"# {title}\n\n{body}\n"
|
||||
|
||||
|
||||
#: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this
|
||||
#: fixture (the phase-94 template — the label is the
|
||||
#: ``FOLDER_HEADER_PREFIX`` tail: ``<source>`` for the root,
|
||||
#: ``<source>/<folder>`` for a folder).
|
||||
SUM_ROOT = f"Fixture folder summary for {SOURCE}."
|
||||
SUM_ONCE = f"Fixture folder summary for {SOURCE}/{FOLDER_ONCE}."
|
||||
SUM_ALWAYS = f"Fixture folder summary for {SOURCE}/{FOLDER_ALWAYS}."
|
||||
SUM_NORMAL = f"Fixture folder summary for {SOURCE}/{FOLDER_NORMAL}."
|
||||
|
||||
# --- the pinned ``ls oneshot`` level (app.rag.agent's phase-94 template)
|
||||
|
||||
#: The source root: 0 direct files, 3 subfolders (path order).
|
||||
LS_HEADER = f"{SOURCE} — 0 documents, 3 folders:"
|
||||
|
||||
#: The subfolder lines — the ``: {summary}`` suffix appended ONLY when
|
||||
#: the subfolder's summary is stored (``render_folder_listing``; the
|
||||
#: renderer's 2-space indent is whitespace-normalized away by the
|
||||
#: ``to_contain_text`` match — the phase-94 pin convention).
|
||||
LINE_ALWAYS = f"{FOLDER_ALWAYS}/ — 2 documents"
|
||||
LINE_ALWAYS_WITH_COLON = f"{FOLDER_ALWAYS}/ — 2 documents: "
|
||||
LINE_ONCE = f"{FOLDER_ONCE}/ — 2 documents: {SUM_ONCE}"
|
||||
LINE_NORMAL = f"{FOLDER_NORMAL}/ — 2 documents: {SUM_NORMAL}"
|
||||
|
||||
# --- the scripted drill-down turns (the phase-94 ``DRILL_TRIGGER``
|
||||
# questions — the mock echoes the listing verbatim into the answer) ---
|
||||
|
||||
LS_QUESTION_1 = f"Drill down the tree: ls {SOURCE} — what's in source {SOURCE}?"
|
||||
LS_QUESTION_2 = f"Drill down the tree: ls {SOURCE} — list the source folders again"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def oneshot_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The one-source temp tree (see the module docstring): the app
|
||||
server runs on the same host, so the paths are visible to it."""
|
||||
root = tmp_path_factory.mktemp("bor_oneshot")
|
||||
src = root / SOURCE
|
||||
for folder, prefix in (
|
||||
(FOLDER_ONCE, "once"),
|
||||
(FOLDER_ALWAYS, "always"),
|
||||
(FOLDER_NORMAL, "normal"),
|
||||
):
|
||||
(src / folder).mkdir(parents=True)
|
||||
for letter, topic in (("a", "A"), ("b", "B")):
|
||||
(src / folder / f"{prefix}-{letter}.md").write_text(
|
||||
_md(
|
||||
f"Oneshot {prefix.title()} {letter.upper()}",
|
||||
f"{DRILL_LEAD} for {SOURCE} {folder} "
|
||||
f"{letter}: this document covers topic {topic} "
|
||||
f"of the {SOURCE} source tree.",
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert len(list(src.rglob("*.md"))) == TOTAL_DOCS
|
||||
return src
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, oneshot_dir: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module app (the conftest pattern,
|
||||
cf. ``test_local_directory_sources.py``): NO ``BOR_GIT_SOURCES``
|
||||
(the env fallback is git-only — the source here is a DB-registered
|
||||
local directory), the mock LLM, the mock-calibrated threshold, and
|
||||
the leak-guarded code defaults. ``BOR_LLM_RETRY_DELAY=0`` + the
|
||||
code-default ``BOR_LLM_RETRIES`` (the phase-67 conftest pattern):
|
||||
the one-shot retry waits are instant and the exhaustion budget is
|
||||
the REAL one (4 attempts). The session app is never started in this
|
||||
isolated run, so no port clash."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated threshold (conftest pattern): every scripted
|
||||
# question FTS-matches the fixture docs (the ``drill down the
|
||||
# tree`` words), so the gate is HIGH either way.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
# Phase 67 / phase 96: instant retry waits + the code-default budget
|
||||
# (the conftest leak-guard pattern) — the one-shot retry and the
|
||||
# 4-attempt exhaustion run in real time at zero delay.
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the registry must hold EXACTLY the one local
|
||||
# directory this suite registers.
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
||||
share one Postgres, so a leftover source or document would pollute
|
||||
the ``ls`` listing the drill answers assert on byte-exactly."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _run_sync_http(base_url: str, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Login + ``POST /api/sync`` + poll the status endpoint until the
|
||||
run reaches a terminal state (the ``test_sync_button.py`` /
|
||||
``test_local_directory_sources.py`` pattern, over plain httpx)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _folder_rows() -> dict[str, tuple[str, datetime]]:
|
||||
"""The source's stored folder summaries
|
||||
``{folder_path: (summary, updated_at)}`` (``""`` = the source
|
||||
root) — the test process's direct DB access, the E2E's other
|
||||
established lens."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(
|
||||
select(
|
||||
FolderSummary.folder_path,
|
||||
FolderSummary.summary,
|
||||
FolderSummary.updated_at,
|
||||
).where(FolderSummary.source == SOURCE)
|
||||
).all()
|
||||
return {folder: (summary, updated_at) for folder, summary, updated_at in rows}
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, oneshot_dir: Path) -> None:
|
||||
"""The story's precondition: the KB synced under the deterministic
|
||||
mock's incident-shape injection.
|
||||
|
||||
Registers the temp directory through the authenticated API (the
|
||||
``test_local_directory_sources.py`` pattern) and runs the REAL
|
||||
in-process sync #1 (``POST /api/sync`` — walk → chunk → embed →
|
||||
overview → folder summaries → version bump). The sync changed the
|
||||
KB → FULL folder regeneration, and the mock's injection drives the
|
||||
incident shapes:
|
||||
|
||||
* ``oneshot`` (the root) + ``oneshot/normal`` — normal replies,
|
||||
one POST each;
|
||||
* ``oneshot/e2e_empty_once`` — the FIRST non-stream POST answers
|
||||
the incident envelope (``content=""`` + ``finish_reason=
|
||||
"length"``), the retry (task 01) recovers the row on the second
|
||||
POST;
|
||||
* ``oneshot/e2e_empty_always`` — EVERY non-stream POST answers the
|
||||
empty envelope: 1 + ``BOR_LLM_RETRIES`` = 4 attempts exhausted →
|
||||
``LLMError`` → the phase-94 per-folder fail-soft leaves the row
|
||||
ABSENT and never flips the run.
|
||||
|
||||
The fixture pins all of that in the DB: the ``e2e_empty_once`` row
|
||||
EXISTS (without task 01 it would be absent — the incident's data
|
||||
loss), the ``e2e_empty_always`` row does NOT, and the sync still
|
||||
reported ``success``.
|
||||
"""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(oneshot_dir)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
body = _run_sync_http(app_server)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == TOTAL_DOCS, detail
|
||||
assert detail["updated"] == 0, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is True, detail
|
||||
# The full regeneration under the injection: the retried row
|
||||
# EXISTS (the incident, healed by task 01's one-shot retry), the
|
||||
# exhausted row is ABSENT (the per-folder fail-soft — the sync
|
||||
# above still reported success), the normal rows landed.
|
||||
rows = _folder_rows()
|
||||
assert set(rows) == {"", FOLDER_ONCE, FOLDER_NORMAL}, rows
|
||||
assert rows[""][0] == SUM_ROOT, rows
|
||||
assert rows[FOLDER_ONCE][0] == SUM_ONCE, rows
|
||||
assert rows[FOLDER_NORMAL][0] == SUM_NORMAL, rows
|
||||
assert FOLDER_ALWAYS not in rows, rows # all 4 attempts empty → no row
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||||
the drill turns never change it, so the folder summaries and the
|
||||
registry persist across the tests of this module)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the phase-94 drill-down house pattern)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` frames, independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page) -> list[dict]:
|
||||
"""The SSE frames captured since the last submit (``_submit``
|
||||
clears the buffer), once the hook's background read settles."""
|
||||
deadline = time.monotonic() + 30.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered
|
||||
(the phase-48 settle wait)."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=60_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=60_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=60_000)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
def _assert_drill_turn(
|
||||
page: Page,
|
||||
expected_tool: dict[str, Any],
|
||||
expected_lines: list[str],
|
||||
forbidden: list[str] | None = None,
|
||||
) -> None:
|
||||
"""One scripted drill turn, fully asserted: the wire carries exactly
|
||||
the expected ``tool`` frame (ahead of the first ``delta``), the
|
||||
bubble carries the expected listing lines (the mock's echo of the
|
||||
tool result the model received), the forbidden substrings are ABSENT
|
||||
(the no-suffix assertions), and the turn was grounded (the ``done``
|
||||
frame is not deflected)."""
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [expected_tool], _tool_frames(frames)
|
||||
first_delta = next(
|
||||
i for i, f in enumerate(frames) if f.get("type") == "delta"
|
||||
)
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
for line in expected_lines:
|
||||
expect(bubble).to_contain_text(line)
|
||||
text = bubble.text_content() or ""
|
||||
for needle in forbidden or []:
|
||||
assert needle not in text, text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. Sync #1: the incident shape is retried (the row lands) and the
|
||||
# always-empty folder stays absent with the sync green
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_incident_reply_retried_and_always_empty_stays_absent(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The ``ls oneshot`` drill-down shows the stored folder summaries
|
||||
the sync #1 produced under the incident-shape injection: the
|
||||
``e2e_empty_once/`` line carries its summary (the retry recovered
|
||||
the row — without task 01 the line would have NO ``: …`` suffix),
|
||||
the ``normal/`` line does too, and the ``e2e_empty_always/`` line
|
||||
is bare (exhausted → fail-soft → absent) — while the sync above
|
||||
reported success (the folder-stats failure never flips the run,
|
||||
phase 94)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, LS_QUESTION_1)
|
||||
_wait_settled(page)
|
||||
lines = _last_brain(page).locator(".tool-call")
|
||||
expect(lines).to_have_count(1)
|
||||
expect(lines.nth(0)).to_contain_text(f"Listing documents in {SOURCE}")
|
||||
_assert_drill_turn(
|
||||
page,
|
||||
{"type": "tool", "name": "ls", "argument": SOURCE},
|
||||
[
|
||||
LS_HEADER,
|
||||
# The bare line (the row is absent — no stored summary to
|
||||
# append) …
|
||||
LINE_ALWAYS,
|
||||
# …and the colon-suffixed lines (the rows the retry + the
|
||||
# normal path stored):
|
||||
LINE_ONCE,
|
||||
LINE_NORMAL,
|
||||
],
|
||||
# The ``e2e_empty_always/`` line must NOT carry a ``: …``
|
||||
# suffix — the bare line above is a prefix of the suffixed
|
||||
# shape, so the absence is pinned here (the all-4-attempts
|
||||
# exhaustion left no row to quote).
|
||||
forbidden=[LINE_ALWAYS_WITH_COLON],
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. The gap-fill: a deleted row self-heals on the next UNCHANGED sync,
|
||||
# the other rows untouched
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_deleted_row_self_heals_on_unchanged_sync(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""Delete the ``normal`` stored row behind the app's back (a
|
||||
historical failure), run the UNCHANGED sync #2, and assert the
|
||||
task-02/03 gap gate: ``missing_folder_summaries`` names the gap,
|
||||
``only_missing=True`` fills EXACTLY the missing rows, and every
|
||||
other row stays byte-identical (text AND ``updated_at``).
|
||||
|
||||
* ``normal`` — healed: the second ``ls oneshot`` turn carries
|
||||
``: Fixture folder summary for oneshot/normal.`` again, and the
|
||||
DB row is back with its deterministic text;
|
||||
* ``e2e_empty_once`` — the gap-fill NEVER calls it (the row
|
||||
exists): its ``updated_at`` is UNCHANGED (a full regeneration
|
||||
would have re-stamped it);
|
||||
* ``e2e_empty_always`` — the gap-fill attempted it, exhausted
|
||||
(4 empty POSTs by design), stayed ABSENT — and the sync still
|
||||
succeeded (the fail-soft never flips the run)."""
|
||||
page.set_default_timeout(30_000)
|
||||
# The gap: delete the ``normal`` row directly (the test process has
|
||||
# DB access via the conftest engine — the same connection the app
|
||||
# uses), capturing the other row's stamp for the targeted-fill
|
||||
# assertion. No KB change anywhere.
|
||||
before = _folder_rows()
|
||||
assert FOLDER_NORMAL in before # the row existed (sync #1 stored it)
|
||||
updated_at_once = before[FOLDER_ONCE][1]
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries "
|
||||
"WHERE source = :s AND folder_path = :f"
|
||||
),
|
||||
{"s": SOURCE, "f": FOLDER_NORMAL},
|
||||
)
|
||||
db.commit()
|
||||
assert FOLDER_NORMAL not in _folder_rows()
|
||||
|
||||
# Sync #2 — the KB is UNCHANGED (nothing in the temp tree moved),
|
||||
# so the gate takes the gap probe: two candidates missing
|
||||
# (``e2e_empty_always`` + ``normal``) → the targeted fill.
|
||||
body = _run_sync_http(app_url)
|
||||
assert body["state"] == "success", body # exhaustion never flips the run
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 0, detail
|
||||
assert detail["updated"] == 0, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is False, detail # unchanged → no overview burn
|
||||
|
||||
# The second scripted drill turn: the healed + the surviving lines,
|
||||
# the always folder still bare.
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
_submit(page, LS_QUESTION_2)
|
||||
_wait_settled(page)
|
||||
lines = _last_brain(page).locator(".tool-call")
|
||||
expect(lines).to_have_count(1)
|
||||
expect(lines.nth(0)).to_contain_text(f"Listing documents in {SOURCE}")
|
||||
_assert_drill_turn(
|
||||
page,
|
||||
{"type": "tool", "name": "ls", "argument": SOURCE},
|
||||
[
|
||||
LS_HEADER,
|
||||
LINE_ALWAYS,
|
||||
LINE_ONCE,
|
||||
LINE_NORMAL, # healed — the suffix is back
|
||||
],
|
||||
forbidden=[LINE_ALWAYS_WITH_COLON],
|
||||
)
|
||||
|
||||
# The DB pins the targeted fill (the E2E's other established lens):
|
||||
rows = _folder_rows()
|
||||
assert set(rows) == {"", FOLDER_ONCE, FOLDER_NORMAL}, rows
|
||||
assert rows[FOLDER_NORMAL][0] == SUM_NORMAL, rows # deterministic text
|
||||
# The targeted fill never touched the other rows — a FULL
|
||||
# regeneration would have re-stamped this one (the ``_upsert``
|
||||
# fresh-UTC-stamp rule).
|
||||
assert rows[FOLDER_ONCE][1] == updated_at_once, (
|
||||
rows[FOLDER_ONCE][1],
|
||||
updated_at_once,
|
||||
)
|
||||
assert FOLDER_ALWAYS not in rows, rows # exhausted again → still absent
|
||||
Reference in New Issue
Block a user