**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.
- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
840 lines
34 KiB
Python
840 lines
34 KiB
Python
"""Phase 95 task 03 E2E (Playwright, mock-only): the read cap's
|
|
dedicated story suite — the truncated read, end to end.
|
|
|
|
The whole TODO.md L5 item (``95_read_truncation_cap`` task 03 is the
|
|
story gate): a document over the (lowered) cap is read truncated, the
|
|
``tool_result`` frame lands, the Reading line carries the marker, the
|
|
LLM's context carried ``[…truncated…]`` + the grep pointer, and the
|
|
marker survives save → shared.
|
|
|
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
|
|
|
uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov
|
|
|
|
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real
|
|
``turbo`` does whatever it does with the tools, while this story's gate
|
|
is the deterministic SCRIPTED capped-read flow in
|
|
``tests/e2e/mock_llm.py`` (``READ_CAP_TRIGGER``: user message contains
|
|
``read the capped document`` — the question carries its own tool call
|
|
after the colon, ``read the capped document: read source/path`` —
|
|
**and** the system prompt carries the ``<tools>`` section of the HIGH
|
|
prompt). The mock ECHOES the ENTIRE read tool result into its final
|
|
answer (the house scripted-turn way of asserting on tool results — the
|
|
mock is the only E2E lens on the LLM's context), so both the marker's
|
|
presence (the truncated turn) and its absence (the short-document
|
|
control) land on the rendered answer.
|
|
|
|
The app under test boots with the LOWERED cap (the
|
|
env-override-for-the-app-under-test pattern — this suite's module app,
|
|
as in ``test_local_directory_sources.py`` / ``test_ls_tree_drilldown.py``):
|
|
``BOR_READ_MAX_CHARS=1500`` — small enough that the ~3 100-char
|
|
fixture document truncates deterministically at 1 500, big enough that
|
|
the short-document control (~200 chars) never does.
|
|
|
|
KB fixture — one host temp dir (``tmp_path_factory``; the app runs on
|
|
the same host) registered as a local-directory source (the
|
|
``test_local_directory_sources.py`` registration + real-Sync pattern —
|
|
registration through the authenticated API, the real in-process
|
|
``POST /api/sync`` pipeline; no git anywhere), with THREE documents
|
|
whose bodies are token-controlled so every scripted turn stays
|
|
GROUNDED (the hybrid gate HIGH — the mock's read-cap flow needs the
|
|
``<tools>`` section):
|
|
|
|
* ``anchor-2024.md`` — the retrieval ANCHOR: a digit-bearing name.
|
|
Both questions name "anchor 2024", whose joined normalized token
|
|
(``anchor2024``) name-hits this document — the name hit's
|
|
``fts_hit`` keeps the gate HIGH for every turn (grounding: the
|
|
``<tools>`` section is present, and the anchor is never a read
|
|
target);
|
|
* ``zz-capped.md`` — the SUBJECT: ~3 100 chars of varied rotation
|
|
prose (deterministic, pinned below with its exact length — the
|
|
``chars_total`` the assertions use is ``len(CAP_DOC)`` of this very
|
|
string, and ``synced_kb`` pins the stored content byte-identical to
|
|
it). Its body avoids EVERY token of both questions, so on its own
|
|
question it has zero lexical hits and only the common-word cosine —
|
|
it ranks below the anchor (irrelevant to the cap's contract: phase
|
|
118, A6 — the seeds are SUMMARIES, so a first ``read`` of ANY
|
|
document, suggested or not, succeeds, and the cap applies to
|
|
everything ``read`` returns — the retired top-2 seed-read refusal
|
|
that used to constrain this design is gone);
|
|
* ``aa-short.md`` — the CONTROL: ~200 chars, under the cap.
|
|
|
|
The turn-specific flavor words (``note, long form`` planted in the
|
|
SHORT doc's body, ``quick pass`` in the CAPPED doc's) are inert
|
|
leftovers of the retired not-a-seed design — they no longer select a
|
|
seed slot (phase 118: a read of a suggested document succeeds, so the
|
|
target's seed status never changes the wire contract).
|
|
|
|
Test → story mapping (Playwright Mapping Rule; the story is the owner
|
|
TODO item — one Playwright file per story, A16):
|
|
1. ``test_truncated_read_frame_order_live_marker_and_llm_notice`` —
|
|
the wire carries the ``tool`` frame THEN exactly one
|
|
``tool_result`` frame (the pinned counts) THEN the answer's
|
|
``delta``/``done`` frames; the live Reading line carries the pinned
|
|
``(truncated — showing 1500 of N chars)`` marker; the mock's echo
|
|
proves the LLM context carried ``[…truncated…]`` AND the pinned
|
|
``TRUNCATED — … Use grep …`` notice (and the head sentinel reached
|
|
the model while the past-the-cap tail sentinel did NOT).
|
|
2. ``test_truncated_read_save_and_shared_fidelity`` — the auto-saved
|
|
row's (phase 55) brain message ``tools`` record carries
|
|
``truncated: true`` + the counts (saved-chats API); the shared
|
|
page (``/shared/<token>``) renders the SAME marker on the Reading
|
|
line (the phase-50 restore contract, pixel-identical).
|
|
3. ``test_short_read_control_no_frame_no_marker`` — the under-cap read
|
|
executes (its ``tool`` frame lands) but streams NO ``tool_result``
|
|
frame, the Reading line carries NO marker, the echo shows no
|
|
``[…truncated…]`` / ``TRUNCATED —``, and the saved ``tools``
|
|
record is the plain pre-truncation shape.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import time
|
|
from collections.abc import Iterator
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import httpx
|
|
import pytest
|
|
from playwright.sync_api import Browser, BrowserContext, Locator, Page, expect
|
|
from sqlalchemy import select, text
|
|
|
|
from app.config import Settings as _Settings
|
|
from app.db import SessionLocal
|
|
from app.models import Document
|
|
from app.rag.agent import READ_TRUNCATION_NOTICE
|
|
from app.rag.retriever import TRUNCATION_MARKER
|
|
from e2e.auth_helpers import login
|
|
from e2e.conftest import (
|
|
ADMIN_PASSWORD,
|
|
SESSION_SECRET,
|
|
USE_REAL_LLM,
|
|
_wait_http,
|
|
)
|
|
|
|
REPO = Path(__file__).resolve().parents[2]
|
|
|
|
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
|
# port in a combined run — this module app binds its own port instead
|
|
# (a same-port second uvicorn dies on bind and would drive the wrong
|
|
# server). Env-overridable.
|
|
APP_PORT = int(os.environ.get("E2E_APP_PORT_READCAP", "8137"))
|
|
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
|
|
|
#: The lowered read cap the app under test boots with
|
|
# (``BOR_READ_MAX_CHARS``): the subject of the suite.
|
|
READ_CAP = 1500
|
|
|
|
SOURCE = "capkb" # the local directory's basename = the source name
|
|
ANCHOR_REL = "anchor-2024.md"
|
|
CAPPED_REL = "zz-capped.md"
|
|
SHORT_REL = "aa-short.md"
|
|
CAPPED_SP = f"{SOURCE}/{CAPPED_REL}" # the combined read identity
|
|
SHORT_SP = f"{SOURCE}/{SHORT_REL}"
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Fixture documents (deterministic, token-controlled — see the module
|
|
# docstring for the retrieval-anchor design)
|
|
# --------------------------------------------------------------------------
|
|
|
|
ANCHOR_DOC = (
|
|
"# Anchor 2024\n\n"
|
|
"The anchor 2024 record keeps the backup rotation steady: the quiet "
|
|
"window, the mirror pool, and the retention label all stay pinned to "
|
|
"this year's plan.\n"
|
|
)
|
|
|
|
HEAD_SENTINEL = "CAPDOC-HEAD-9f2a" # inside the first 1 500 chars
|
|
TAIL_SENTINEL = "CAPDOC-TAIL-7c3b" # PAST the cap (the rest is NOT shown)
|
|
SHORT_SENTINEL = "CAPSHORT-5d1e" # the whole short doc is under the cap
|
|
|
|
|
|
def _cap_doc() -> str:
|
|
"""The ~3 100-char subject document: a repeated-but-varied paragraph
|
|
block (deterministic), the head sentinel near the top, the planted
|
|
``quick pass`` sentence (the turn-2 flavor words — see the module
|
|
docstring) and the tail sentinel at the very end. The body avoids
|
|
every token of BOTH questions (``read``, ``capped``, ``document``,
|
|
``capkb``, ``zz``, ``md``, ``anchor``, ``2024``, ``note``,
|
|
``long``, ``form``, ``aa``, ``short``, ``quick``, ``pass``,
|
|
``wire``, ``save``, ``control``, ``check``) — the target has zero
|
|
lexical hits on its own turn, so it ranks below the anchor (phase
|
|
118: its seed status is irrelevant — a first read of any document
|
|
succeeds; the anchor keeps the gate HIGH)."""
|
|
topics = [
|
|
"vault", "mirror", "raid", "pool", "drive",
|
|
"chain", "slot", "cycle", "guard", "probe",
|
|
]
|
|
paras: list[str] = []
|
|
i = 0
|
|
while True:
|
|
t = topics[i % len(topics)]
|
|
paras.append(
|
|
f"{t.title()} item {i:02d}: at 02:00 the quiet window opens and "
|
|
f"the archive job copies the {t} image to the mirror pool, then "
|
|
f"the manifest audit verifies the snapshot chain for slot {i:02d}, "
|
|
"keeping the retention label clean and the rotation order exact "
|
|
"for the whole night cycle."
|
|
)
|
|
i += 1
|
|
if len("\n\n".join(paras)) > 2750:
|
|
break
|
|
head = (
|
|
"# Rotation archive\n\n"
|
|
f"{HEAD_SENTINEL}\n\n"
|
|
"A quick pass over the rotation confirms the pass order of the "
|
|
"guard probe before the quiet window closes, and the audit log "
|
|
"records the result.\n\n"
|
|
)
|
|
tail = f"\n\n{TAIL_SENTINEL}\n"
|
|
return head + "\n\n".join(paras) + tail
|
|
|
|
|
|
CAP_DOC = _cap_doc()
|
|
CAP_DOC_TOTAL = len(CAP_DOC)
|
|
|
|
SHORT_DOC = (
|
|
"# Rotation plan\n\n"
|
|
"This note keeps the long form of the rotation plan in one place: the "
|
|
"quiet window, the mirror pool, the retention label, and the guard "
|
|
"probe order for every weekly run.\n\n"
|
|
f"{SHORT_SENTINEL}\n"
|
|
)
|
|
SHORT_DOC_TOTAL = len(SHORT_DOC)
|
|
|
|
# The suite's length contract (the task's pinned-count assumption —
|
|
# ``chars_total`` is asserted against ``len(CAP_DOC)`` of the very
|
|
# string written to the fixture, and ``synced_kb`` pins the stored
|
|
# content byte-identical to it): the subject is OVER the cap with the
|
|
# tail sentinel past the cut; the control is UNDER the cap.
|
|
assert CAP_DOC_TOTAL > READ_CAP, CAP_DOC_TOTAL
|
|
assert CAP_DOC.index(HEAD_SENTINEL) < READ_CAP < CAP_DOC.index(TAIL_SENTINEL)
|
|
assert CAP_DOC.count(HEAD_SENTINEL) == 1 and CAP_DOC.count(TAIL_SENTINEL) == 1
|
|
assert SHORT_DOC_TOTAL < READ_CAP, SHORT_DOC_TOTAL
|
|
assert SHORT_DOC.count(SHORT_SENTINEL) == 1
|
|
|
|
# The pinned marker copy (task 02 — the app.js/shared.js template, plain
|
|
# integers, no thousands separators) and the LLM-side notice (the
|
|
# app.rag.agent constant, formatted for this fixture's counts).
|
|
MARKER = f" (truncated — showing {READ_CAP} of {CAP_DOC_TOTAL} chars)"
|
|
NOTICE = READ_TRUNCATION_NOTICE.format(shown=READ_CAP, total=CAP_DOC_TOTAL)
|
|
|
|
# The scripted turns (the mock's ``READ_CAP_TRIGGER`` questions — each
|
|
# carries its own tool call after the colon; the turn-specific flavor
|
|
# words are inert leftovers of the retired #2-seed design — see the
|
|
# module docstring).
|
|
Q_WIRE = (
|
|
f"Read the capped document: read {CAPPED_SP} — "
|
|
"anchor 2024 note, long form, wire check"
|
|
)
|
|
Q_SAVE = (
|
|
f"Read the capped document: read {CAPPED_SP} — "
|
|
"anchor 2024 note, long form, save check"
|
|
)
|
|
Q_CONTROL = (
|
|
f"Read the capped document: read {SHORT_SP} — "
|
|
"anchor 2024, quick pass, control check"
|
|
)
|
|
|
|
#: The invalid/unknown share token shape (test_share_chat.py's
|
|
#: ``SHARE_URL_RE`` convention).
|
|
SHARE_URL_RE = re.compile(r"^/shared/[0-9a-f-]{36}$")
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Fixtures
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def cap_dirs(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|
"""The story's local-directory source: one host temp dir (the app
|
|
server runs on the same host, so the path is visible to it) holding
|
|
the three token-controlled fixture documents. The directory's
|
|
basename is the source name (``kind=local``, phase 38)."""
|
|
root = tmp_path_factory.mktemp("bor_read_cap") / SOURCE
|
|
root.mkdir()
|
|
for rel, doc in (
|
|
(ANCHOR_REL, ANCHOR_DOC),
|
|
(CAPPED_REL, CAP_DOC),
|
|
(SHORT_REL, SHORT_DOC),
|
|
):
|
|
(root / rel).write_text(doc, encoding="utf-8")
|
|
assert (root / rel).read_text(encoding="utf-8") == doc
|
|
assert not (root / ".git").exists() # the local-kind story: NOT git
|
|
return root
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def app_server(mock_llm: int, cap_dirs: Path) -> Iterator[str]:
|
|
"""The real app under test — per-module env (the conftest pattern,
|
|
cf. ``test_ls_tree_drilldown.py``): NO ``BOR_GIT_SOURCES`` (the env
|
|
fallback is git-only — the source here is a DB-registered local
|
|
directory), the mock LLM, the mock-calibrated threshold, the
|
|
leak-guarded code defaults — and the SUBJECT: the lowered read cap
|
|
``BOR_READ_MAX_CHARS=1500`` (the env-override-for-the-app-under-
|
|
test pattern; the default 128 000 is unit-pinned in
|
|
``tests/unit/test_config.py``). The session app is never started in
|
|
this isolated run, so no port clash."""
|
|
env = dict(os.environ)
|
|
env.pop("DEBUGPY", None)
|
|
env["BOR_ENVIRONMENT"] = "e2e"
|
|
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
|
env["BOR_LLM_BASE_URL"] = (
|
|
"https://aipi.reeseapps.com/v1"
|
|
if USE_REAL_LLM
|
|
else f"http://127.0.0.1:{mock_llm}/v1"
|
|
)
|
|
# Mock-calibrated threshold (conftest pattern) — the retrieval-anchor
|
|
# design keeps every scripted turn grounded regardless.
|
|
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
|
# Phase 67: instant retry waits + the code-default budget (the
|
|
# conftest leak-guard pattern).
|
|
env["BOR_LLM_RETRY_DELAY"] = "0"
|
|
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
|
env.setdefault(
|
|
"BOR_DATABASE_URL",
|
|
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
|
)
|
|
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
|
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
|
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
|
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
|
# reads it from cwd) — override it with an EMPTY value (the env var
|
|
# beats the .env file): the registry must hold EXACTLY the local
|
|
# directory this suite registers (a leftover env git list would
|
|
# pollute the KB the scripted reads run against).
|
|
env["BOR_GIT_SOURCES"] = ""
|
|
# Leak guards (conftest pattern): an operator's local (gitignored)
|
|
# .env cannot leak corpus-specific settings into the app under test.
|
|
env["BOR_DOCS_REPO"] = ""
|
|
env["BOR_SUGGESTIONS"] = json.dumps(
|
|
_Settings.model_fields["suggestions"].default
|
|
)
|
|
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
|
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
|
# THE SUBJECT: the lowered read cap (task 03 — the app under test
|
|
# boots with BOR_READ_MAX_CHARS=1500; the production default of
|
|
# 128 000 stays pinned in tests/unit/test_config.py).
|
|
env["BOR_READ_MAX_CHARS"] = str(READ_CAP)
|
|
proc = subprocess.Popen(
|
|
[sys.executable, "-m", "uvicorn", "app.main:app",
|
|
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
|
cwd=REPO,
|
|
env=env,
|
|
)
|
|
try:
|
|
_wait_http(f"{APP_URL}/api/health")
|
|
yield APP_URL
|
|
finally:
|
|
proc.terminate()
|
|
try:
|
|
proc.wait(timeout=10)
|
|
except subprocess.TimeoutExpired:
|
|
proc.kill()
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def app_url(app_server: str) -> str:
|
|
return app_server
|
|
|
|
|
|
def _truncate_all() -> None:
|
|
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
|
share one Postgres, so a leftover git_sources row or document would
|
|
pollute the retrieval the scripted turns run against (the anchor
|
|
design's margins are pinned against EXACTLY these three
|
|
documents)."""
|
|
with SessionLocal() as db:
|
|
db.execute(
|
|
text(
|
|
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
|
"kb_overview, git_sources, folder_summaries"
|
|
)
|
|
)
|
|
db.commit()
|
|
|
|
|
|
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
|
"""Poll the (cookie-authenticated) status endpoint until the run
|
|
reaches a terminal state (the test_ls_tree_drilldown pattern, over
|
|
plain httpx — this fixture has no browser page yet)."""
|
|
deadline = time.monotonic() + timeout_s
|
|
body: dict[str, Any] = {}
|
|
while time.monotonic() < deadline:
|
|
r = client.get("/api/sync/status")
|
|
assert r.status_code == 200, r.text
|
|
body = r.json()
|
|
if body["state"] in ("success", "failed"):
|
|
return body
|
|
time.sleep(0.5)
|
|
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def synced_kb(app_server: str, cap_dirs: Path) -> None:
|
|
"""The story's precondition: the one-source KB synced under the
|
|
deterministic mock. Registers the temp directory through the
|
|
authenticated API (the ``test_local_directory_sources.py`` pattern),
|
|
runs the REAL in-process sync (``POST /api/sync`` — walk → chunk →
|
|
embed → overview → version bump), pins the stored content
|
|
byte-identical to the fixture strings (the ``chars_total``
|
|
assumption). Phase 118: the retrieval-anchor design only pins the
|
|
gate's grounding (the anchor keeps every turn HIGH) — a read of a
|
|
suggested document succeeds, so the targets' seed status no longer
|
|
constrains the suite."""
|
|
_truncate_all()
|
|
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
|
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
|
assert r.status_code == 204, r.text
|
|
r = client.post(
|
|
"/api/git-sources", json={"kind": "local", "path": str(cap_dirs)}
|
|
)
|
|
assert r.status_code == 201, r.text
|
|
r = client.post("/api/sync")
|
|
assert r.status_code == 202, r.text
|
|
body = _wait_sync_done_http(client)
|
|
assert body["state"] == "success", body
|
|
detail = body["detail"]
|
|
assert detail["added"] == 3, detail
|
|
assert detail["pruned"] == 0, detail
|
|
# The import stored the fixture strings BYTE-IDENTICALLY — the
|
|
# ``chars_total`` every assertion below uses is ``len()`` of the
|
|
# very string written to the directory (the task's pinned-count
|
|
# assumption).
|
|
with SessionLocal() as db:
|
|
for rel, expected in (
|
|
(ANCHOR_REL, ANCHOR_DOC),
|
|
(CAPPED_REL, CAP_DOC),
|
|
(SHORT_REL, SHORT_DOC),
|
|
):
|
|
stored = db.scalar(
|
|
select(Document.content).where(
|
|
Document.source == SOURCE, Document.path == rel
|
|
)
|
|
)
|
|
assert stored == expected, f"stored content drifted for {rel}"
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clean(db_ready: None) -> Iterator[None]:
|
|
"""Per-test query_log isolation (the KB itself is module-scoped —
|
|
the scripted turns never change it, so the registry persists
|
|
across the tests of this module)."""
|
|
with SessionLocal() as db:
|
|
db.execute(text("TRUNCATE query_log"))
|
|
db.commit()
|
|
yield
|
|
with SessionLocal() as db:
|
|
db.execute(text("TRUNCATE query_log"))
|
|
db.commit()
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Page helpers (the test_ls_tree_drilldown / test_share_chat house
|
|
# patterns)
|
|
# --------------------------------------------------------------------------
|
|
|
|
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
|
#: (a response clone read in the background) — wire-level assertions
|
|
#: for the ``tool`` / ``tool_result`` frames, independent of the UI
|
|
#: rendering.
|
|
SSE_HOOK = """
|
|
() => {
|
|
if (window.__sseInstalled) return;
|
|
window.__sseInstalled = true;
|
|
window.__sseFrames = [];
|
|
const origFetch = window.fetch;
|
|
window.fetch = async function (...args) {
|
|
const res = await origFetch.apply(this, args);
|
|
try {
|
|
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
|
if (url.includes('/api/chat')) {
|
|
res.clone().text().then((bodyText) => {
|
|
for (const block of bodyText.split('\\n\\n')) {
|
|
const line = block.trim();
|
|
if (line.startsWith('data: ')) {
|
|
window.__sseFrames.push(line.slice(6));
|
|
}
|
|
}
|
|
});
|
|
}
|
|
} catch (e) { /* non-clonable responses: ignored */ }
|
|
return res;
|
|
};
|
|
}
|
|
"""
|
|
|
|
|
|
def _install_page_hooks(page: Page) -> None:
|
|
page.evaluate(SSE_HOOK)
|
|
|
|
|
|
def _frames(page: Page) -> list[dict]:
|
|
"""The SSE frames captured since the last submit (``_submit``
|
|
clears the buffer), once the hook's background read settles."""
|
|
deadline = time.monotonic() + 30.0
|
|
while True:
|
|
raw = page.evaluate("() => window.__sseFrames || []")
|
|
parsed = [json.loads(line) for line in raw if line]
|
|
if any(f.get("type") == "done" for f in parsed):
|
|
return parsed
|
|
if time.monotonic() > deadline:
|
|
raise AssertionError(
|
|
f"SSE hook captured no `done` frame (frames so far: "
|
|
f"{len(parsed)}) — hook install failed?"
|
|
)
|
|
time.sleep(0.05)
|
|
|
|
|
|
def _tool_frames(frames: list[dict]) -> list[dict]:
|
|
return [f for f in frames if f.get("type") == "tool"]
|
|
|
|
|
|
def _result_frames(frames: list[dict]) -> list[dict]:
|
|
return [f for f in frames if f.get("type") == "tool_result"]
|
|
|
|
|
|
def _submit(page: Page, question: str) -> None:
|
|
page.evaluate("window.__sseFrames = []")
|
|
page.fill("#message-input", question)
|
|
page.click("#send-btn")
|
|
# The user bubble lands synchronously with the submit handler.
|
|
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
|
|
|
|
|
def _wait_settled(page: Page) -> None:
|
|
"""The turn is complete: answer text in the bubble, button recovered
|
|
(the phase-48 settle wait, the test_agent_document_tools helper)."""
|
|
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
|
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
|
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
|
|
|
|
|
def _last_brain(page: Page) -> Locator:
|
|
return page.locator(".msg.brain").last
|
|
|
|
|
|
def _auto_title(question: str) -> str:
|
|
"""The phase-50 auto-title convention: the first question,
|
|
whitespace-collapsed, capped at 120 chars."""
|
|
return " ".join(question.split())[:120]
|
|
|
|
|
|
def _admin_cookies(page: Page) -> dict[str, str]:
|
|
"""The signed session cookies the browser holds after a form login —
|
|
used to call the admin API with plain httpx (the test's API side
|
|
sees exactly what the signed-in browser sees)."""
|
|
return {
|
|
c["name"]: c["value"]
|
|
for c in page.context.cookies()
|
|
if "name" in c and "value" in c
|
|
}
|
|
|
|
|
|
def _chats(app_url: str, cookies: dict[str, str]) -> list[dict[str, Any]]:
|
|
r = httpx.get(f"{app_url}/api/chats", timeout=10, cookies=cookies)
|
|
assert r.status_code == 200
|
|
return r.json()["chats"]
|
|
|
|
|
|
def _find_row(
|
|
rows: list[dict[str, Any]], title: str
|
|
) -> dict[str, Any] | None:
|
|
return next((c for c in rows if c["title"] == title), None)
|
|
|
|
|
|
def _get_chat(app_url: str, cookies: dict[str, str], chat_id: str) -> dict[str, Any]:
|
|
r = httpx.get(f"{app_url}/api/chats/{chat_id}", timeout=10, cookies=cookies)
|
|
assert r.status_code == 200, r.text
|
|
return r.json()
|
|
|
|
|
|
def _delete_chat(app_url: str, cookies: dict[str, str], chat_id: str) -> None:
|
|
"""Best-effort row cleanup (a 404 — already deleted — is fine)."""
|
|
httpx.delete(f"{app_url}/api/chats/{chat_id}", timeout=10, cookies=cookies)
|
|
|
|
|
|
def _wait_saved_row(
|
|
app_url: str,
|
|
cookies: dict[str, str],
|
|
title: str,
|
|
messages: int = 2,
|
|
) -> dict[str, Any]:
|
|
"""Wait for the auto-saved row (phase 55: auto-saves are SILENT —
|
|
A2 — so there is no status line to wait on). The upsert is
|
|
fire-and-forget from the UI's point of view, so poll the admin
|
|
list until the row with the conversation's auto-title appears with
|
|
the expected message count."""
|
|
deadline = time.monotonic() + 15
|
|
last: dict[str, Any] | None = None
|
|
while time.monotonic() < deadline:
|
|
last = _find_row(_chats(app_url, cookies), title)
|
|
if last is not None and last["message_count"] >= messages:
|
|
return last
|
|
time.sleep(0.2)
|
|
raise AssertionError(f"no auto-saved row for {title!r} (last: {last!r})")
|
|
|
|
|
|
def _grant_clipboard(page: Page, app_url: str) -> None:
|
|
"""Grant the async-clipboard permissions on the admin context
|
|
(test_share_chat.py — the assertion branches on the API's
|
|
availability, so a non-secure origin still passes through the
|
|
fallback branch deterministically)."""
|
|
page.context.grant_permissions(
|
|
["clipboard-read", "clipboard-write"], origin=app_url
|
|
)
|
|
|
|
|
|
def _click_share_and_assert_status(page: Page, app_url: str) -> None:
|
|
"""Press the chat page's Share pill and pin the owner-locked
|
|
outcome on the live region (test_share_chat.py's helper, verbatim
|
|
contract): "Share link copied." when ``navigator.clipboard`` is
|
|
available in the context, else the inline fallback link field
|
|
carrying the ``/shared/<uuid>`` URL."""
|
|
page.locator("#share-chat-btn").click()
|
|
if page.evaluate("() => !!navigator.clipboard"):
|
|
expect(page.locator("#send-status")).to_have_text(
|
|
"Share link copied.", timeout=15_000
|
|
)
|
|
expect(page.locator(".share-link-fallback")).to_have_count(0)
|
|
else:
|
|
expect(page.locator("#send-status")).to_have_text(
|
|
"Share link ready — copy it from the field.", timeout=15_000
|
|
)
|
|
field = page.locator(".share-link-fallback")
|
|
expect(field).to_be_visible()
|
|
href = field.get_attribute("href")
|
|
assert href is not None
|
|
assert href.startswith(app_url)
|
|
assert SHARE_URL_RE.fullmatch(href.removeprefix(app_url))
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# 1. The truncated read: the wire's frame order, the live marker, the
|
|
# LLM-visible notice (via the mock's echo)
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
def test_truncated_read_frame_order_live_marker_and_llm_notice(
|
|
page: Page, app_url: str, synced_kb: None, db_ready: None
|
|
) -> None:
|
|
page.set_default_timeout(30_000)
|
|
login(page, app_url, next="/")
|
|
_install_page_hooks(page)
|
|
|
|
_submit(page, Q_WIRE)
|
|
_wait_settled(page)
|
|
|
|
# --- the wire: tool → tool_result → delta… (exactly one, counts) --
|
|
frames = _frames(page)
|
|
assert _tool_frames(frames) == [
|
|
{"type": "tool", "name": "read", "argument": CAPPED_SP}
|
|
], _tool_frames(frames)
|
|
# EXACTLY ONE tool_result frame — the pinned counts (the char-based
|
|
# cap: showing the first READ_CAP of the fixture's true length).
|
|
assert _result_frames(frames) == [
|
|
{
|
|
"type": "tool_result",
|
|
"name": "read",
|
|
"argument": CAPPED_SP,
|
|
"truncated": True,
|
|
"chars_shown": READ_CAP,
|
|
"chars_total": CAP_DOC_TOTAL,
|
|
}
|
|
], _result_frames(frames)
|
|
# Frame order: the marker lands AFTER the call is shown and BEFORE
|
|
# the answer's first delta (the phase-37/48 "calling tool" timing
|
|
# is untouched — the beat between the two is the truncation).
|
|
i_tool = next(i for i, f in enumerate(frames) if f.get("type") == "tool")
|
|
i_result = next(
|
|
i for i, f in enumerate(frames) if f.get("type") == "tool_result"
|
|
)
|
|
i_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
|
assert i_tool < i_result < i_delta, (i_tool, i_result, i_delta)
|
|
done = next(f for f in frames if f.get("type") == "done")
|
|
assert done["deflected"] is False, done
|
|
# The read document is the turn's cited source (retrieval +
|
|
# agent-read, deduped — the grounded-turn record; phase 118: the
|
|
# target may be a suggested (summary-seeded) document — its seed
|
|
# status never changes the cap's wire contract).
|
|
assert any(
|
|
s["path"] == CAPPED_REL and s["source"] == SOURCE for s in done["sources"]
|
|
), done["sources"]
|
|
|
|
# --- the live DOM: the Reading line carries the pinned marker ----
|
|
line = _last_brain(page).locator(".tool-call")
|
|
expect(line).to_have_count(1)
|
|
expect(line).to_contain_text(f"Reading {CAPPED_SP}")
|
|
expect(line.locator("code")).to_have_text(CAPPED_SP)
|
|
expect(line.locator("span.truncated-note")).to_have_text(MARKER)
|
|
|
|
# --- the mock's echo: the LLM's context carried the marker AND ---
|
|
# --- the grep-pointer notice (the house scripted-turn lens) ------
|
|
bubble = _last_brain(page).locator(".bubble")
|
|
text = bubble.text_content() or ""
|
|
assert TRUNCATION_MARKER in text, text
|
|
assert NOTICE in text, text
|
|
# The first cap chars reached the model…
|
|
assert HEAD_SENTINEL in text, text
|
|
# …and the past-the-cap tail did NOT (the cut is real, not cosmetic).
|
|
assert TAIL_SENTINEL not in text, text
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# 2. Save → shared: the stored tools record carries the truncation, and
|
|
# the shared page renders the SAME marker (pixel-identical)
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
def test_truncated_read_save_and_shared_fidelity(
|
|
page: Page,
|
|
browser: Browser,
|
|
app_url: str,
|
|
synced_kb: None,
|
|
db_ready: None,
|
|
) -> None:
|
|
page.set_default_timeout(30_000)
|
|
login(page, app_url, next="/")
|
|
|
|
_submit(page, Q_SAVE)
|
|
_wait_settled(page)
|
|
|
|
# The live marker landed (regression through the save path — the
|
|
# tool_result frame stamped the toolAcc entry the save carries).
|
|
line = _last_brain(page).locator(".tool-call")
|
|
expect(line).to_have_count(1)
|
|
expect(line.locator("span.truncated-note")).to_have_text(MARKER)
|
|
|
|
cookies = _admin_cookies(page)
|
|
title = _auto_title(Q_SAVE)
|
|
row = _wait_saved_row(app_url, cookies, title)
|
|
chat_id: str = row["id"]
|
|
anon_ctx: BrowserContext | None = None
|
|
try:
|
|
# The saved message's ``tools`` record carries the truncation —
|
|
# ``truncated: true`` + the pinned counts (the saved-chats API;
|
|
# the model_dump round-trip fills the non-truncated defaults,
|
|
# so the full five-key shape is asserted).
|
|
detail = _get_chat(app_url, cookies, chat_id)
|
|
brain_msgs = [m for m in detail["messages"] if m["who"] == "brain"]
|
|
assert len(brain_msgs) == 1, detail["messages"]
|
|
assert brain_msgs[0]["tools"] == [
|
|
{
|
|
"name": "read",
|
|
"argument": CAPPED_SP,
|
|
"truncated": True,
|
|
"chars_shown": READ_CAP,
|
|
"chars_total": CAP_DOC_TOTAL,
|
|
}
|
|
], brain_msgs[0]["tools"]
|
|
|
|
# Share the auto-saved row (phase 55: settled = already saved —
|
|
# the Share click shares the linked row) and read back the link.
|
|
_grant_clipboard(page, app_url)
|
|
_click_share_and_assert_status(page, app_url)
|
|
row = _find_row(_chats(app_url, cookies), title)
|
|
assert row is not None and row.get("share_url"), row
|
|
share_url: str = row["share_url"]
|
|
assert SHARE_URL_RE.fullmatch(share_url)
|
|
|
|
# A FRESH context (no cookies — the guest's only credential is
|
|
# the token in the URL): the shared page renders the SAME
|
|
# marker on the Reading line (shared.js renders it from the
|
|
# stored record — the phase-50 restore contract).
|
|
anon_ctx = browser.new_context()
|
|
anon = anon_ctx.new_page()
|
|
anon.set_default_timeout(30_000)
|
|
anon.goto(app_url + share_url)
|
|
expect(anon.locator("#shared-title")).to_have_text(title)
|
|
anon_line = anon.locator(".msg.brain .tool-call")
|
|
expect(anon_line).to_have_count(1)
|
|
expect(anon_line).to_contain_text(f"Reading {CAPPED_SP}")
|
|
expect(anon_line.locator("code")).to_have_text(CAPPED_SP)
|
|
expect(anon_line.locator("span.truncated-note")).to_have_text(MARKER)
|
|
# The shared answer still renders the truncation the LLM was
|
|
# told about (same stored text, same renderer).
|
|
anon_text = anon.locator(".msg.brain .bubble").first.text_content() or ""
|
|
assert TRUNCATION_MARKER in anon_text, anon_text
|
|
assert NOTICE in anon_text, anon_text
|
|
assert HEAD_SENTINEL in anon_text, anon_text
|
|
assert TAIL_SENTINEL not in anon_text, anon_text
|
|
anon_ctx.close()
|
|
anon_ctx = None
|
|
finally:
|
|
if anon_ctx is not None:
|
|
anon_ctx.close()
|
|
_delete_chat(app_url, cookies, chat_id)
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# 3. The control: the under-cap read executes, streams NO tool_result
|
|
# frame, shows NO marker, and saves the plain tools record
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
def test_short_read_control_no_frame_no_marker(
|
|
page: Page, app_url: str, synced_kb: None, db_ready: None
|
|
) -> None:
|
|
page.set_default_timeout(30_000)
|
|
login(page, app_url, next="/")
|
|
_install_page_hooks(page)
|
|
|
|
_submit(page, Q_CONTROL)
|
|
_wait_settled(page)
|
|
|
|
# The read EXECUTED (the tool frame landed)…
|
|
frames = _frames(page)
|
|
assert _tool_frames(frames) == [
|
|
{"type": "tool", "name": "read", "argument": SHORT_SP}
|
|
], _tool_frames(frames)
|
|
# …but a non-truncated read streams NO tool_result frame (one frame
|
|
# = one noteworthy event — the phase-95 additive contract).
|
|
assert _result_frames(frames) == [], _result_frames(frames)
|
|
done = next(f for f in frames if f.get("type") == "done")
|
|
assert done["deflected"] is False, done
|
|
assert any(
|
|
s["path"] == SHORT_REL and s["source"] == SOURCE for s in done["sources"]
|
|
), done["sources"]
|
|
|
|
# No marker on the live Reading line.
|
|
line = _last_brain(page).locator(".tool-call")
|
|
expect(line).to_have_count(1)
|
|
expect(line).to_contain_text(f"Reading {SHORT_SP}")
|
|
expect(line.locator("code")).to_have_text(SHORT_SP)
|
|
expect(line.locator("span.truncated-note")).to_have_count(0)
|
|
|
|
# The mock's echo: the PLAIN read shape — the whole short document
|
|
# (it is under the cap), no marker, no notice.
|
|
bubble = _last_brain(page).locator(".bubble")
|
|
text = bubble.text_content() or ""
|
|
assert SHORT_SENTINEL in text, text
|
|
assert TRUNCATION_MARKER not in text, text
|
|
assert "TRUNCATED —" not in text, text
|
|
|
|
# The saved ``tools`` record is the plain pre-truncation shape
|
|
# (``truncated`` default False, the counts null — the
|
|
# pre-phase-95 shape validates and renders unchanged).
|
|
cookies = _admin_cookies(page)
|
|
title = _auto_title(Q_CONTROL)
|
|
row = _wait_saved_row(app_url, cookies, title)
|
|
try:
|
|
detail = _get_chat(app_url, cookies, row["id"])
|
|
brain_msgs = [m for m in detail["messages"] if m["who"] == "brain"]
|
|
assert len(brain_msgs) == 1, detail["messages"]
|
|
assert brain_msgs[0]["tools"] == [
|
|
{
|
|
"name": "read",
|
|
"argument": SHORT_SP,
|
|
"truncated": False,
|
|
"chars_shown": None,
|
|
"chars_total": None,
|
|
}
|
|
], brain_msgs[0]["tools"]
|
|
finally:
|
|
_delete_chat(app_url, cookies, row["id"])
|