phase: 95_read_truncation_cap
All gates are green. Final verification is complete — no fixes were needed; the phase is fully implemented and passing. **Phase 95 final verification report** - Verified all 3 tasks already implemented (task files in `complete/`): read cap + notice + `ToolResultPiece` (task 01), `tool_result` SSE + live/saved/shared marker (task 02), E2E suite (task 03). - Code checks passed: `BOR_READ_MAX_CHARS` (default 128 000, validator), pinned `READ_TRUNCATION_NOTICE` + `TRUNCATION_MARKER`, byte-identical at/under-cap read, holder `read_truncations` (no `tool_calls` impact), piece order (after `tool`, before next round), `ChatToolResultEvent`, `ToolCall` back-compat fields, `app.js` live/restore handlers, `shared.js`, `.truncated-note` CSS, `.env.example` entry, mock-LLM echo script. - **Tests:** `uv run pytest` → 1966 passed; `uv run pytest --cov=app --cov-report=term-missing` → all green, TOTAL **99%** (>90% gate). - **E2E:** `uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov` → **3 passed** (frame order + live marker + LLM notice via echo; save→shared fidelity; short-read control). - **Regression (isolated):** `test_agent_document_tools` 4 ✓, `test_chat_history` 5 ✓, `test_share_chat` 4 ✓, `test_big_read_progress` 4 ✓, `test_stop_generation` 3 ✓. - **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings. **Completion criteria:** ① over-cap read → first-cap-chars + marker + pinned notice — ✓ (unit-pinned: at-cap/cap+1/notice tests); ② user marker live/saved/shared — ✓ (E2E + frontend tests); ③ at/under cap byte-identical, no frame — ✓ (unit + control E2E); ④ top-2 `<documents>` retrieval untouched — ✓ (`app/rag/retriever.py` unmodified vs HEAD); ⑤ suite green, >90% coverage, ruff+pyright clean — ✓; ⑥ no completed-phase behavior change — ✓ (all gates green; commit left to harness per pass rules). - No defects found; no changes made this pass. Next pending phase: none in `todo/` (96 is the next free number).
This commit is contained in:
@@ -259,6 +259,34 @@ Implements just enough of the aipi surface:
|
||||
the ``<tools>`` section, so deflected turns never hit it); no
|
||||
existing E2E question or fixture file contains the phrase, so
|
||||
every other suite is unaffected.
|
||||
- user message containing ``read the capped document``
|
||||
(``READ_CAP_TRIGGER``, phase 95 task 03 — the read cap's dedicated
|
||||
story suite ``tests/e2e/test_read_truncation_cap.py``) **and** the
|
||||
system prompt carries the ``<tools>`` section -> the deterministic
|
||||
SCRIPTED CAPPED-READ flow: the question carries its own tool call
|
||||
after the colon — ``read the capped document: read source/path`` —
|
||||
parsed by ``_READ_CAP_CALL_RE`` from the RAW user message (the
|
||||
target keeps its case), then discriminated statelessly from the
|
||||
tool results (streaming only):
|
||||
* request 1 (``tools`` offered, no ``tool``-role result in the
|
||||
messages yet): the scripted call — ``read`` with the parsed
|
||||
target (synthetic id ``call_0``);
|
||||
* a ``tool``-role result is in the messages: the deterministic
|
||||
ECHO — the answer carries the LAST tool result VERBATIM
|
||||
(``Here's what the read returned:\n<result>``): a read result
|
||||
(``"Document <source/path>:…``) lands in the answer with its
|
||||
FULL content — the first cap chars + ``[…truncated…]`` + the
|
||||
pinned grep-pointer notice when the cap fired, the plain body
|
||||
byte-identical to the pre-phase-95 shape when it did not — and
|
||||
a refusal (the premise broke) lands just as visibly, so the
|
||||
suite fails loudly on it. The mock is the only E2E lens on the
|
||||
LLM's context, so the echo is the assertion surface for both
|
||||
the marker's presence AND its absence.
|
||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
|
||||
phrases — the phase-71/72/94 ordering convention; the trigger
|
||||
needs the ``<tools>`` section, so deflected turns never hit it);
|
||||
verified 2026-09-10: no existing E2E question or fixture file
|
||||
contains the phrase, so every other suite is unaffected.
|
||||
- user message containing ``what are the correct llama.cpp
|
||||
arguments`` (``GREP_TEACH_TRIGGER``, the 2026-09-05 incident —
|
||||
the harness prior is that grep takes a REGEX; this app's grep is a
|
||||
@@ -1293,6 +1321,79 @@ def _drill_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
return ("echo", last)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 95 (task 03, the read cap's dedicated story suite):
|
||||
# the deterministic SCRIPTED capped read — see the module docstring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: A user message containing this substring (case-insensitive) —
|
||||
#: combined with the ``<tools>`` section in the system prompt — drives
|
||||
#: the scripted CAPPED-READ flow (the read cap's story suite,
|
||||
#: ``tests/e2e/test_read_truncation_cap.py``): the question carries its
|
||||
#: own tool call after the colon — ``read the capped document: read
|
||||
#: source/path`` — the mock emits the scripted ``read``, then ECHOES
|
||||
#: the ENTIRE tool result into its answer (the house scripted-turn lens
|
||||
#: on the LLM's context: the truncated shape — first cap chars +
|
||||
#: ``[…truncated…]`` + the pinned grep-pointer notice — or the plain
|
||||
#: shape, byte-identical to the pre-phase-95 result, lands in the
|
||||
#: rendered answer, and the suite asserts both directions through it).
|
||||
#: Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
|
||||
#: phrases — the phase-71/72/94 ordering convention); verified
|
||||
#: 2026-09-10: no existing E2E question or fixture file contains the
|
||||
#: phrase, so every other suite is unaffected.
|
||||
READ_CAP_TRIGGER = "read the capped document"
|
||||
|
||||
#: The scripted call in the read-cap question (case-insensitive — the
|
||||
#: suite's questions capitalize the trigger's first letter): the verb
|
||||
#: (``read``) plus the target — a combined ``source/path``, parsed from
|
||||
#: the RAW user message so the target keeps its case. The target is a
|
||||
#: ``[a-z0-9_./-]`` run (case-insensitively), so the suite's `` — ``
|
||||
#: flavor separator (em dash) can never bleed into it (the drill-down
|
||||
#: convention, ``_DRILL_CALL_RE``).
|
||||
_READ_CAP_CALL_RE = re.compile(
|
||||
r"read the capped document:\s*read\s+(?P<arg>[a-z0-9_./-]+)",
|
||||
re.I,
|
||||
)
|
||||
|
||||
|
||||
def _read_cap_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a phase-95 scripted read-cap request (see the module
|
||||
docstring). The question carries the scripted call (``read the
|
||||
capped document: read source/path``); the step is then discriminated
|
||||
statelessly from the tool results, like the other marker flows:
|
||||
|
||||
* ``("call", target, "call_0")`` — ``tools`` are offered and no
|
||||
``tool``-role result is in the messages yet: the scripted
|
||||
``read`` on the parsed target (synthetic id ``call_0``).
|
||||
* ``("echo", result)`` — a ``tool``-role result is in the messages:
|
||||
the deterministic ECHO — the answer carries the LAST tool result
|
||||
VERBATIM (``Here's what the read returned:\n<result>``): a read
|
||||
result (``"Document <source/path>:…``) lands in the answer with
|
||||
its full content — the truncation marker + the pinned notice when
|
||||
the cap fired, the plain body when it did not — and a refusal
|
||||
(the premise broke) lands just as visibly, so the suite fails
|
||||
loudly on it.
|
||||
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
|
||||
section is missing (deflected turns never carry it), the scripted
|
||||
call is unparseable, or ``tools`` are not offered and no tool
|
||||
results are in the messages yet (e.g. ``agent_max_rounds=0``).
|
||||
"""
|
||||
user = _user(body)
|
||||
if READ_CAP_TRIGGER not in user.lower():
|
||||
return None
|
||||
if "<tools>" not in _system(body):
|
||||
return None
|
||||
match = _READ_CAP_CALL_RE.search(user)
|
||||
if match is None:
|
||||
return None
|
||||
results = _tool_results(body)
|
||||
if not results:
|
||||
if not body.get("tools"):
|
||||
return None
|
||||
return ("call", match.group("arg"), "call_0")
|
||||
return ("echo", results[-1])
|
||||
|
||||
|
||||
def long_answer() -> str:
|
||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||
a unique final line that must survive the stream untruncated."""
|
||||
@@ -1985,6 +2086,34 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 95 (task 03): the deterministic SCRIPTED capped read
|
||||
# (the question carries its own call — ``read the capped
|
||||
# document: read source/path``): the scripted ``read``, then the
|
||||
# answer that ECHOES the whole tool result (the marker + notice
|
||||
# when the cap fired, the plain shape when it did not — the
|
||||
# suite's lens on the LLM's context). Checked BEFORE the plain
|
||||
# TOOLS_TRIGGER flow (disjoint trigger phrases — the
|
||||
# phase-71/72/94 ordering convention; the trigger needs the
|
||||
# ``<tools>`` section, so deflected turns never hit it).
|
||||
read_cap = _read_cap_flow(body)
|
||||
if read_cap is not None:
|
||||
if read_cap[0] == "call":
|
||||
stream = _tool_call_stream(
|
||||
"read", {"path": read_cap[1]}, read_cap[2]
|
||||
)
|
||||
else: # "echo" — the last tool result verbatim (the lens)
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
f"Here's what the read returned:\n{read_cap[1]}",
|
||||
body.get("max_tokens"),
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
flow = _tool_flow(body)
|
||||
if flow is not None:
|
||||
if flow[0] == "list":
|
||||
|
||||
@@ -0,0 +1,861 @@
|
||||
"""Phase 95 task 03 E2E (Playwright, mock-only): the read cap's
|
||||
dedicated story suite — the truncated read, end to end.
|
||||
|
||||
The whole TODO.md L5 item (``95_read_truncation_cap`` task 03 is the
|
||||
story gate): a document over the (lowered) cap is read truncated, the
|
||||
``tool_result`` frame lands, the Reading line carries the marker, the
|
||||
LLM's context carried ``[…truncated…]`` + the grep pointer, and the
|
||||
marker survives save → shared.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real
|
||||
``turbo`` does whatever it does with the tools, while this story's gate
|
||||
is the deterministic SCRIPTED capped-read flow in
|
||||
``tests/e2e/mock_llm.py`` (``READ_CAP_TRIGGER``: user message contains
|
||||
``read the capped document`` — the question carries its own tool call
|
||||
after the colon, ``read the capped document: read source/path`` —
|
||||
**and** the system prompt carries the ``<tools>`` section of the HIGH
|
||||
prompt). The mock ECHOES the ENTIRE read tool result into its final
|
||||
answer (the house scripted-turn way of asserting on tool results — the
|
||||
mock is the only E2E lens on the LLM's context), so both the marker's
|
||||
presence (the truncated turn) and its absence (the short-document
|
||||
control) land on the rendered answer.
|
||||
|
||||
The app under test boots with the LOWERED cap (the
|
||||
env-override-for-the-app-under-test pattern — this suite's module app,
|
||||
as in ``test_local_directory_sources.py`` / ``test_ls_tree_drilldown.py``):
|
||||
``BOR_READ_MAX_CHARS=1500`` — small enough that the ~3 100-char
|
||||
fixture document truncates deterministically at 1 500, big enough that
|
||||
the short-document control (~200 chars) never does.
|
||||
|
||||
KB fixture — one host temp dir (``tmp_path_factory``; the app runs on
|
||||
the same host) registered as a local-directory source (the
|
||||
``test_local_directory_sources.py`` registration + real-Sync pattern —
|
||||
registration through the authenticated API, the real in-process
|
||||
``POST /api/sync`` pipeline; no git anywhere), with THREE documents
|
||||
whose bodies are token-controlled so the phase-72 ALREADY_IN_CONTEXT
|
||||
dedupe (the read target is refused when it is a top-2 retrieval seed)
|
||||
NEVER fires — each scripted read target must actually execute, not be
|
||||
refused:
|
||||
|
||||
* ``anchor-2024.md`` — the retrieval ANCHOR: a digit-bearing name.
|
||||
Both questions name "anchor 2024", whose joined normalized token
|
||||
(``anchor2024``) name-hits this document — the name-hit list LEADS
|
||||
the lexical side, so the anchor is the #1 seed of BOTH turns
|
||||
(grounding: the name hit's ``fts_hit`` keeps the gate HIGH, the
|
||||
``<tools>`` section is present, and the anchor is never a read
|
||||
target);
|
||||
* ``zz-capped.md`` — the SUBJECT: ~3 100 chars of varied rotation
|
||||
prose (deterministic, pinned below with its exact length — the
|
||||
``chars_total`` the assertions use is ``len(CAP_DOC)`` of this very
|
||||
string, and ``synced_kb`` pins the stored content byte-identical to
|
||||
it). Its body avoids EVERY token of both questions, so on its own
|
||||
(turn 1) question it has zero lexical hits and only the common-word
|
||||
cosine — it is the #3 fused candidate, NOT a seed;
|
||||
* ``aa-short.md`` — the CONTROL: ~200 chars, under the cap.
|
||||
|
||||
The turn-specific flavor words pick the #2 seed: turn 1's question
|
||||
carries ``note, long form`` (planted in the SHORT doc's body) and turn
|
||||
2's carries ``quick pass`` (planted in the CAPPED doc's body), so each
|
||||
turn's NON-target document wins the second seed slot on its own
|
||||
question and the target stays #3/#6. ``synced_kb`` pins this design
|
||||
with the app's real hybrid retrieval (``_assert_target_not_seed`` — a
|
||||
fixture-text regression that makes a target a seed fails at setup with
|
||||
a clear message, not at the wire assertions).
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule; the story is the owner
|
||||
TODO item — one Playwright file per story, A16):
|
||||
1. ``test_truncated_read_frame_order_live_marker_and_llm_notice`` —
|
||||
the wire carries the ``tool`` frame THEN exactly one
|
||||
``tool_result`` frame (the pinned counts) THEN the answer's
|
||||
``delta``/``done`` frames; the live Reading line carries the pinned
|
||||
``(truncated — showing 1500 of N chars)`` marker; the mock's echo
|
||||
proves the LLM context carried ``[…truncated…]`` AND the pinned
|
||||
``TRUNCATED — … Use grep …`` notice (and the head sentinel reached
|
||||
the model while the past-the-cap tail sentinel did NOT).
|
||||
2. ``test_truncated_read_save_and_shared_fidelity`` — the auto-saved
|
||||
row's (phase 55) brain message ``tools`` record carries
|
||||
``truncated: true`` + the counts (saved-chats API); the shared
|
||||
page (``/shared/<token>``) renders the SAME marker on the Reading
|
||||
line (the phase-50 restore contract, pixel-identical).
|
||||
3. ``test_short_read_control_no_frame_no_marker`` — the under-cap read
|
||||
executes (its ``tool`` frame lands) but streams NO ``tool_result``
|
||||
frame, the Reading line carries NO marker, the echo shows no
|
||||
``[…truncated…]`` / ``TRUNCATED —``, and the saved ``tools``
|
||||
record is the plain pre-truncation shape.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Browser, BrowserContext, Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from app.rag.agent import READ_TRUNCATION_NOTICE
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
from e2e.mock_llm import embed_text
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module app binds its own port instead
|
||||
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
# server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_READCAP", "8137"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
#: The lowered read cap the app under test boots with
|
||||
# (``BOR_READ_MAX_CHARS``): the subject of the suite.
|
||||
READ_CAP = 1500
|
||||
|
||||
SOURCE = "capkb" # the local directory's basename = the source name
|
||||
ANCHOR_REL = "anchor-2024.md"
|
||||
CAPPED_REL = "zz-capped.md"
|
||||
SHORT_REL = "aa-short.md"
|
||||
CAPPED_SP = f"{SOURCE}/{CAPPED_REL}" # the combined read identity
|
||||
SHORT_SP = f"{SOURCE}/{SHORT_REL}"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture documents (deterministic, token-controlled — see the module
|
||||
# docstring for the retrieval-anchor design)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
ANCHOR_DOC = (
|
||||
"# Anchor 2024\n\n"
|
||||
"The anchor 2024 record keeps the backup rotation steady: the quiet "
|
||||
"window, the mirror pool, and the retention label all stay pinned to "
|
||||
"this year's plan.\n"
|
||||
)
|
||||
|
||||
HEAD_SENTINEL = "CAPDOC-HEAD-9f2a" # inside the first 1 500 chars
|
||||
TAIL_SENTINEL = "CAPDOC-TAIL-7c3b" # PAST the cap (the rest is NOT shown)
|
||||
SHORT_SENTINEL = "CAPSHORT-5d1e" # the whole short doc is under the cap
|
||||
|
||||
|
||||
def _cap_doc() -> str:
|
||||
"""The ~3 100-char subject document: a repeated-but-varied paragraph
|
||||
block (deterministic), the head sentinel near the top, the planted
|
||||
``quick pass`` sentence (the turn-2 flavor words — see the module
|
||||
docstring) and the tail sentinel at the very end. The body avoids
|
||||
every token of BOTH questions (``read``, ``capped``, ``document``,
|
||||
``capkb``, ``zz``, ``md``, ``anchor``, ``2024``, ``note``,
|
||||
``long``, ``form``, ``aa``, ``short``, ``quick``, ``pass``,
|
||||
``wire``, ``save``, ``control``, ``check``) — the target must have
|
||||
zero lexical hits on its own turn so the dedupe cannot refuse the
|
||||
read."""
|
||||
topics = [
|
||||
"vault", "mirror", "raid", "pool", "drive",
|
||||
"chain", "slot", "cycle", "guard", "probe",
|
||||
]
|
||||
paras: list[str] = []
|
||||
i = 0
|
||||
while True:
|
||||
t = topics[i % len(topics)]
|
||||
paras.append(
|
||||
f"{t.title()} item {i:02d}: at 02:00 the quiet window opens and "
|
||||
f"the archive job copies the {t} image to the mirror pool, then "
|
||||
f"the manifest audit verifies the snapshot chain for slot {i:02d}, "
|
||||
"keeping the retention label clean and the rotation order exact "
|
||||
"for the whole night cycle."
|
||||
)
|
||||
i += 1
|
||||
if len("\n\n".join(paras)) > 2750:
|
||||
break
|
||||
head = (
|
||||
"# Rotation archive\n\n"
|
||||
f"{HEAD_SENTINEL}\n\n"
|
||||
"A quick pass over the rotation confirms the pass order of the "
|
||||
"guard probe before the quiet window closes, and the audit log "
|
||||
"records the result.\n\n"
|
||||
)
|
||||
tail = f"\n\n{TAIL_SENTINEL}\n"
|
||||
return head + "\n\n".join(paras) + tail
|
||||
|
||||
|
||||
CAP_DOC = _cap_doc()
|
||||
CAP_DOC_TOTAL = len(CAP_DOC)
|
||||
|
||||
SHORT_DOC = (
|
||||
"# Rotation plan\n\n"
|
||||
"This note keeps the long form of the rotation plan in one place: the "
|
||||
"quiet window, the mirror pool, the retention label, and the guard "
|
||||
"probe order for every weekly run.\n\n"
|
||||
f"{SHORT_SENTINEL}\n"
|
||||
)
|
||||
SHORT_DOC_TOTAL = len(SHORT_DOC)
|
||||
|
||||
# The suite's length contract (the task's pinned-count assumption —
|
||||
# ``chars_total`` is asserted against ``len(CAP_DOC)`` of the very
|
||||
# string written to the fixture, and ``synced_kb`` pins the stored
|
||||
# content byte-identical to it): the subject is OVER the cap with the
|
||||
# tail sentinel past the cut; the control is UNDER the cap.
|
||||
assert CAP_DOC_TOTAL > READ_CAP, CAP_DOC_TOTAL
|
||||
assert CAP_DOC.index(HEAD_SENTINEL) < READ_CAP < CAP_DOC.index(TAIL_SENTINEL)
|
||||
assert CAP_DOC.count(HEAD_SENTINEL) == 1 and CAP_DOC.count(TAIL_SENTINEL) == 1
|
||||
assert SHORT_DOC_TOTAL < READ_CAP, SHORT_DOC_TOTAL
|
||||
assert SHORT_DOC.count(SHORT_SENTINEL) == 1
|
||||
|
||||
# The pinned marker copy (task 02 — the app.js/shared.js template, plain
|
||||
# integers, no thousands separators) and the LLM-side notice (the
|
||||
# app.rag.agent constant, formatted for this fixture's counts).
|
||||
MARKER = f" (truncated — showing {READ_CAP} of {CAP_DOC_TOTAL} chars)"
|
||||
NOTICE = READ_TRUNCATION_NOTICE.format(shown=READ_CAP, total=CAP_DOC_TOTAL)
|
||||
|
||||
# The scripted turns (the mock's ``READ_CAP_TRIGGER`` questions — each
|
||||
# carries its own tool call after the colon; the turn-specific flavor
|
||||
# words are the #2-seed selectors, see the module docstring).
|
||||
Q_WIRE = (
|
||||
f"Read the capped document: read {CAPPED_SP} — "
|
||||
"anchor 2024 note, long form, wire check"
|
||||
)
|
||||
Q_SAVE = (
|
||||
f"Read the capped document: read {CAPPED_SP} — "
|
||||
"anchor 2024 note, long form, save check"
|
||||
)
|
||||
Q_CONTROL = (
|
||||
f"Read the capped document: read {SHORT_SP} — "
|
||||
"anchor 2024, quick pass, control check"
|
||||
)
|
||||
|
||||
#: The invalid/unknown share token shape (test_share_chat.py's
|
||||
#: ``SHARE_URL_RE`` convention).
|
||||
SHARE_URL_RE = re.compile(r"^/shared/[0-9a-f-]{36}$")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def cap_dirs(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The story's local-directory source: one host temp dir (the app
|
||||
server runs on the same host, so the path is visible to it) holding
|
||||
the three token-controlled fixture documents. The directory's
|
||||
basename is the source name (``kind=local``, phase 38)."""
|
||||
root = tmp_path_factory.mktemp("bor_read_cap") / SOURCE
|
||||
root.mkdir()
|
||||
for rel, doc in (
|
||||
(ANCHOR_REL, ANCHOR_DOC),
|
||||
(CAPPED_REL, CAP_DOC),
|
||||
(SHORT_REL, SHORT_DOC),
|
||||
):
|
||||
(root / rel).write_text(doc, encoding="utf-8")
|
||||
assert (root / rel).read_text(encoding="utf-8") == doc
|
||||
assert not (root / ".git").exists() # the local-kind story: NOT git
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, cap_dirs: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module env (the conftest pattern,
|
||||
cf. ``test_ls_tree_drilldown.py``): NO ``BOR_GIT_SOURCES`` (the env
|
||||
fallback is git-only — the source here is a DB-registered local
|
||||
directory), the mock LLM, the mock-calibrated threshold, the
|
||||
leak-guarded code defaults — and the SUBJECT: the lowered read cap
|
||||
``BOR_READ_MAX_CHARS=1500`` (the env-override-for-the-app-under-
|
||||
test pattern; the default 128 000 is unit-pinned in
|
||||
``tests/unit/test_config.py``). The session app is never started in
|
||||
this isolated run, so no port clash."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated threshold (conftest pattern) — the retrieval-anchor
|
||||
# design keeps every scripted turn grounded regardless.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
# Phase 67: instant retry waits + the code-default budget (the
|
||||
# conftest leak-guard pattern).
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the registry must hold EXACTLY the local
|
||||
# directory this suite registers (a leftover env git list would
|
||||
# pollute the KB the scripted reads run against).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
# THE SUBJECT: the lowered read cap (task 03 — the app under test
|
||||
# boots with BOR_READ_MAX_CHARS=1500; the production default of
|
||||
# 128 000 stays pinned in tests/unit/test_config.py).
|
||||
env["BOR_READ_MAX_CHARS"] = str(READ_CAP)
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
||||
share one Postgres, so a leftover git_sources row or document would
|
||||
pollute the retrieval the scripted turns run against (the anchor
|
||||
design's margins are pinned against EXACTLY these three
|
||||
documents)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _assert_target_not_seed(question: str, target_rel: str) -> None:
|
||||
"""Pin the retrieval-anchor design (see the module docstring) with
|
||||
the app's REAL hybrid retrieval over the mock's embeddings
|
||||
(deterministic): the scripted read target must NOT be a top-2 seed
|
||||
for its own question — the phase-72 ALREADY_IN_CONTEXT dedupe would
|
||||
refuse the read and the cap would never fire (the turn would echo
|
||||
the refusal instead). A fixture-text regression that breaks this
|
||||
fails here, at setup, with a clear message."""
|
||||
from app.rag.retriever import retrieve, select_documents
|
||||
|
||||
with SessionLocal() as db:
|
||||
seeds = select_documents(retrieve(db, question, embed_text(question)))
|
||||
paths = [f"{d.source}/{d.path}" for d in seeds]
|
||||
assert f"{SOURCE}/{target_rel}" not in paths, (
|
||||
f"the read target {SOURCE}/{target_rel} is a top-2 seed for its own "
|
||||
f"question — the ALREADY_IN_CONTEXT dedupe would refuse the read and "
|
||||
f"the cap would never fire (seeds: {paths})"
|
||||
)
|
||||
|
||||
|
||||
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Poll the (cookie-authenticated) status endpoint until the run
|
||||
reaches a terminal state (the test_ls_tree_drilldown pattern, over
|
||||
plain httpx — this fixture has no browser page yet)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, cap_dirs: Path) -> None:
|
||||
"""The story's precondition: the one-source KB synced under the
|
||||
deterministic mock. Registers the temp directory through the
|
||||
authenticated API (the ``test_local_directory_sources.py`` pattern),
|
||||
runs the REAL in-process sync (``POST /api/sync`` — walk → chunk →
|
||||
embed → overview → version bump), pins the stored content
|
||||
byte-identical to the fixture strings (the ``chars_total``
|
||||
assumption), and pins the retrieval-anchor design for all three
|
||||
scripted questions (the dedupe never fires)."""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(cap_dirs)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
body = _wait_sync_done_http(client)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 3, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
# The import stored the fixture strings BYTE-IDENTICALLY — the
|
||||
# ``chars_total`` every assertion below uses is ``len()`` of the
|
||||
# very string written to the directory (the task's pinned-count
|
||||
# assumption).
|
||||
with SessionLocal() as db:
|
||||
for rel, expected in (
|
||||
(ANCHOR_REL, ANCHOR_DOC),
|
||||
(CAPPED_REL, CAP_DOC),
|
||||
(SHORT_REL, SHORT_DOC),
|
||||
):
|
||||
stored = db.scalar(
|
||||
select(Document.content).where(
|
||||
Document.source == SOURCE, Document.path == rel
|
||||
)
|
||||
)
|
||||
assert stored == expected, f"stored content drifted for {rel}"
|
||||
# The retrieval-anchor design (the module docstring): every scripted
|
||||
# read target stays OUT of its own question's top-2 seeds.
|
||||
_assert_target_not_seed(Q_WIRE, CAPPED_REL)
|
||||
_assert_target_not_seed(Q_SAVE, CAPPED_REL)
|
||||
_assert_target_not_seed(Q_CONTROL, SHORT_REL)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||||
the scripted turns never change it, so the registry persists
|
||||
across the tests of this module)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the test_ls_tree_drilldown / test_share_chat house
|
||||
# patterns)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` / ``tool_result`` frames, independent of the UI
|
||||
#: rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page) -> list[dict]:
|
||||
"""The SSE frames captured since the last submit (``_submit``
|
||||
clears the buffer), once the hook's background read settles."""
|
||||
deadline = time.monotonic() + 30.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _result_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool_result"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered
|
||||
(the phase-48 settle wait, the test_agent_document_tools helper)."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
def _auto_title(question: str) -> str:
|
||||
"""The phase-50 auto-title convention: the first question,
|
||||
whitespace-collapsed, capped at 120 chars."""
|
||||
return " ".join(question.split())[:120]
|
||||
|
||||
|
||||
def _admin_cookies(page: Page) -> dict[str, str]:
|
||||
"""The signed session cookies the browser holds after a form login —
|
||||
used to call the admin API with plain httpx (the test's API side
|
||||
sees exactly what the signed-in browser sees)."""
|
||||
return {
|
||||
c["name"]: c["value"]
|
||||
for c in page.context.cookies()
|
||||
if "name" in c and "value" in c
|
||||
}
|
||||
|
||||
|
||||
def _chats(app_url: str, cookies: dict[str, str]) -> list[dict[str, Any]]:
|
||||
r = httpx.get(f"{app_url}/api/chats", timeout=10, cookies=cookies)
|
||||
assert r.status_code == 200
|
||||
return r.json()["chats"]
|
||||
|
||||
|
||||
def _find_row(
|
||||
rows: list[dict[str, Any]], title: str
|
||||
) -> dict[str, Any] | None:
|
||||
return next((c for c in rows if c["title"] == title), None)
|
||||
|
||||
|
||||
def _get_chat(app_url: str, cookies: dict[str, str], chat_id: str) -> dict[str, Any]:
|
||||
r = httpx.get(f"{app_url}/api/chats/{chat_id}", timeout=10, cookies=cookies)
|
||||
assert r.status_code == 200, r.text
|
||||
return r.json()
|
||||
|
||||
|
||||
def _delete_chat(app_url: str, cookies: dict[str, str], chat_id: str) -> None:
|
||||
"""Best-effort row cleanup (a 404 — already deleted — is fine)."""
|
||||
httpx.delete(f"{app_url}/api/chats/{chat_id}", timeout=10, cookies=cookies)
|
||||
|
||||
|
||||
def _wait_saved_row(
|
||||
app_url: str,
|
||||
cookies: dict[str, str],
|
||||
title: str,
|
||||
messages: int = 2,
|
||||
) -> dict[str, Any]:
|
||||
"""Wait for the auto-saved row (phase 55: auto-saves are SILENT —
|
||||
A2 — so there is no status line to wait on). The upsert is
|
||||
fire-and-forget from the UI's point of view, so poll the admin
|
||||
list until the row with the conversation's auto-title appears with
|
||||
the expected message count."""
|
||||
deadline = time.monotonic() + 15
|
||||
last: dict[str, Any] | None = None
|
||||
while time.monotonic() < deadline:
|
||||
last = _find_row(_chats(app_url, cookies), title)
|
||||
if last is not None and last["message_count"] >= messages:
|
||||
return last
|
||||
time.sleep(0.2)
|
||||
raise AssertionError(f"no auto-saved row for {title!r} (last: {last!r})")
|
||||
|
||||
|
||||
def _grant_clipboard(page: Page, app_url: str) -> None:
|
||||
"""Grant the async-clipboard permissions on the admin context
|
||||
(test_share_chat.py — the assertion branches on the API's
|
||||
availability, so a non-secure origin still passes through the
|
||||
fallback branch deterministically)."""
|
||||
page.context.grant_permissions(
|
||||
["clipboard-read", "clipboard-write"], origin=app_url
|
||||
)
|
||||
|
||||
|
||||
def _click_share_and_assert_status(page: Page, app_url: str) -> None:
|
||||
"""Press the chat page's Share pill and pin the owner-locked
|
||||
outcome on the live region (test_share_chat.py's helper, verbatim
|
||||
contract): "Share link copied." when ``navigator.clipboard`` is
|
||||
available in the context, else the inline fallback link field
|
||||
carrying the ``/shared/<uuid>`` URL."""
|
||||
page.locator("#share-chat-btn").click()
|
||||
if page.evaluate("() => !!navigator.clipboard"):
|
||||
expect(page.locator("#send-status")).to_have_text(
|
||||
"Share link copied.", timeout=15_000
|
||||
)
|
||||
expect(page.locator(".share-link-fallback")).to_have_count(0)
|
||||
else:
|
||||
expect(page.locator("#send-status")).to_have_text(
|
||||
"Share link ready — copy it from the field.", timeout=15_000
|
||||
)
|
||||
field = page.locator(".share-link-fallback")
|
||||
expect(field).to_be_visible()
|
||||
href = field.get_attribute("href")
|
||||
assert href is not None
|
||||
assert href.startswith(app_url)
|
||||
assert SHARE_URL_RE.fullmatch(href.removeprefix(app_url))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The truncated read: the wire's frame order, the live marker, the
|
||||
# LLM-visible notice (via the mock's echo)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_truncated_read_frame_order_live_marker_and_llm_notice(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, Q_WIRE)
|
||||
_wait_settled(page)
|
||||
|
||||
# --- the wire: tool → tool_result → delta… (exactly one, counts) --
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "read", "argument": CAPPED_SP}
|
||||
], _tool_frames(frames)
|
||||
# EXACTLY ONE tool_result frame — the pinned counts (the char-based
|
||||
# cap: showing the first READ_CAP of the fixture's true length).
|
||||
assert _result_frames(frames) == [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"name": "read",
|
||||
"argument": CAPPED_SP,
|
||||
"truncated": True,
|
||||
"chars_shown": READ_CAP,
|
||||
"chars_total": CAP_DOC_TOTAL,
|
||||
}
|
||||
], _result_frames(frames)
|
||||
# Frame order: the marker lands AFTER the call is shown and BEFORE
|
||||
# the answer's first delta (the phase-37/48 "calling tool" timing
|
||||
# is untouched — the beat between the two is the truncation).
|
||||
i_tool = next(i for i, f in enumerate(frames) if f.get("type") == "tool")
|
||||
i_result = next(
|
||||
i for i, f in enumerate(frames) if f.get("type") == "tool_result"
|
||||
)
|
||||
i_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert i_tool < i_result < i_delta, (i_tool, i_result, i_delta)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# The read document is the turn's cited source even though it was
|
||||
# NOT a retrieval seed (the anchor design) — retrieval + agent-read,
|
||||
# deduped (the grounded-turn record).
|
||||
assert any(
|
||||
s["path"] == CAPPED_REL and s["source"] == SOURCE for s in done["sources"]
|
||||
), done["sources"]
|
||||
|
||||
# --- the live DOM: the Reading line carries the pinned marker ----
|
||||
line = _last_brain(page).locator(".tool-call")
|
||||
expect(line).to_have_count(1)
|
||||
expect(line).to_contain_text(f"Reading {CAPPED_SP}")
|
||||
expect(line.locator("code")).to_have_text(CAPPED_SP)
|
||||
expect(line.locator("span.truncated-note")).to_have_text(MARKER)
|
||||
|
||||
# --- the mock's echo: the LLM's context carried the marker AND ---
|
||||
# --- the grep-pointer notice (the house scripted-turn lens) ------
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
text = bubble.text_content() or ""
|
||||
assert TRUNCATION_MARKER in text, text
|
||||
assert NOTICE in text, text
|
||||
# The first cap chars reached the model…
|
||||
assert HEAD_SENTINEL in text, text
|
||||
# …and the past-the-cap tail did NOT (the cut is real, not cosmetic).
|
||||
assert TAIL_SENTINEL not in text, text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. Save → shared: the stored tools record carries the truncation, and
|
||||
# the shared page renders the SAME marker (pixel-identical)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_truncated_read_save_and_shared_fidelity(
|
||||
page: Page,
|
||||
browser: Browser,
|
||||
app_url: str,
|
||||
synced_kb: None,
|
||||
db_ready: None,
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
|
||||
_submit(page, Q_SAVE)
|
||||
_wait_settled(page)
|
||||
|
||||
# The live marker landed (regression through the save path — the
|
||||
# tool_result frame stamped the toolAcc entry the save carries).
|
||||
line = _last_brain(page).locator(".tool-call")
|
||||
expect(line).to_have_count(1)
|
||||
expect(line.locator("span.truncated-note")).to_have_text(MARKER)
|
||||
|
||||
cookies = _admin_cookies(page)
|
||||
title = _auto_title(Q_SAVE)
|
||||
row = _wait_saved_row(app_url, cookies, title)
|
||||
chat_id: str = row["id"]
|
||||
anon_ctx: BrowserContext | None = None
|
||||
try:
|
||||
# The saved message's ``tools`` record carries the truncation —
|
||||
# ``truncated: true`` + the pinned counts (the saved-chats API;
|
||||
# the model_dump round-trip fills the non-truncated defaults,
|
||||
# so the full five-key shape is asserted).
|
||||
detail = _get_chat(app_url, cookies, chat_id)
|
||||
brain_msgs = [m for m in detail["messages"] if m["who"] == "brain"]
|
||||
assert len(brain_msgs) == 1, detail["messages"]
|
||||
assert brain_msgs[0]["tools"] == [
|
||||
{
|
||||
"name": "read",
|
||||
"argument": CAPPED_SP,
|
||||
"truncated": True,
|
||||
"chars_shown": READ_CAP,
|
||||
"chars_total": CAP_DOC_TOTAL,
|
||||
}
|
||||
], brain_msgs[0]["tools"]
|
||||
|
||||
# Share the auto-saved row (phase 55: settled = already saved —
|
||||
# the Share click shares the linked row) and read back the link.
|
||||
_grant_clipboard(page, app_url)
|
||||
_click_share_and_assert_status(page, app_url)
|
||||
row = _find_row(_chats(app_url, cookies), title)
|
||||
assert row is not None and row.get("share_url"), row
|
||||
share_url: str = row["share_url"]
|
||||
assert SHARE_URL_RE.fullmatch(share_url)
|
||||
|
||||
# A FRESH context (no cookies — the guest's only credential is
|
||||
# the token in the URL): the shared page renders the SAME
|
||||
# marker on the Reading line (shared.js renders it from the
|
||||
# stored record — the phase-50 restore contract).
|
||||
anon_ctx = browser.new_context()
|
||||
anon = anon_ctx.new_page()
|
||||
anon.set_default_timeout(30_000)
|
||||
anon.goto(app_url + share_url)
|
||||
expect(anon.locator("#shared-title")).to_have_text(title)
|
||||
anon_line = anon.locator(".msg.brain .tool-call")
|
||||
expect(anon_line).to_have_count(1)
|
||||
expect(anon_line).to_contain_text(f"Reading {CAPPED_SP}")
|
||||
expect(anon_line.locator("code")).to_have_text(CAPPED_SP)
|
||||
expect(anon_line.locator("span.truncated-note")).to_have_text(MARKER)
|
||||
# The shared answer still renders the truncation the LLM was
|
||||
# told about (same stored text, same renderer).
|
||||
anon_text = anon.locator(".msg.brain .bubble").first.text_content() or ""
|
||||
assert TRUNCATION_MARKER in anon_text, anon_text
|
||||
assert NOTICE in anon_text, anon_text
|
||||
assert HEAD_SENTINEL in anon_text, anon_text
|
||||
assert TAIL_SENTINEL not in anon_text, anon_text
|
||||
anon_ctx.close()
|
||||
anon_ctx = None
|
||||
finally:
|
||||
if anon_ctx is not None:
|
||||
anon_ctx.close()
|
||||
_delete_chat(app_url, cookies, chat_id)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 3. The control: the under-cap read executes, streams NO tool_result
|
||||
# frame, shows NO marker, and saves the plain tools record
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_short_read_control_no_frame_no_marker(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, Q_CONTROL)
|
||||
_wait_settled(page)
|
||||
|
||||
# The read EXECUTED (the tool frame landed)…
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "read", "argument": SHORT_SP}
|
||||
], _tool_frames(frames)
|
||||
# …but a non-truncated read streams NO tool_result frame (one frame
|
||||
# = one noteworthy event — the phase-95 additive contract).
|
||||
assert _result_frames(frames) == [], _result_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
assert any(
|
||||
s["path"] == SHORT_REL and s["source"] == SOURCE for s in done["sources"]
|
||||
), done["sources"]
|
||||
|
||||
# No marker on the live Reading line.
|
||||
line = _last_brain(page).locator(".tool-call")
|
||||
expect(line).to_have_count(1)
|
||||
expect(line).to_contain_text(f"Reading {SHORT_SP}")
|
||||
expect(line.locator("code")).to_have_text(SHORT_SP)
|
||||
expect(line.locator("span.truncated-note")).to_have_count(0)
|
||||
|
||||
# The mock's echo: the PLAIN read shape — the whole short document
|
||||
# (it is under the cap), no marker, no notice.
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
text = bubble.text_content() or ""
|
||||
assert SHORT_SENTINEL in text, text
|
||||
assert TRUNCATION_MARKER not in text, text
|
||||
assert "TRUNCATED —" not in text, text
|
||||
|
||||
# The saved ``tools`` record is the plain pre-truncation shape
|
||||
# (``truncated`` default False, the counts null — the
|
||||
# pre-phase-95 shape validates and renders unchanged).
|
||||
cookies = _admin_cookies(page)
|
||||
title = _auto_title(Q_CONTROL)
|
||||
row = _wait_saved_row(app_url, cookies, title)
|
||||
try:
|
||||
detail = _get_chat(app_url, cookies, row["id"])
|
||||
brain_msgs = [m for m in detail["messages"] if m["who"] == "brain"]
|
||||
assert len(brain_msgs) == 1, detail["messages"]
|
||||
assert brain_msgs[0]["tools"] == [
|
||||
{
|
||||
"name": "read",
|
||||
"argument": SHORT_SP,
|
||||
"truncated": False,
|
||||
"chars_shown": None,
|
||||
"chars_total": None,
|
||||
}
|
||||
], brain_msgs[0]["tools"]
|
||||
finally:
|
||||
_delete_chat(app_url, cookies, row["id"])
|
||||
@@ -42,7 +42,13 @@ from app.config import Settings
|
||||
from app.models import Document, FolderSummary, GitSource
|
||||
from app.rag import agent
|
||||
from app.rag.agent import AGENT_TOOLS, AgentHolder, run_agent
|
||||
from app.rag.llm import LLMClient, RetryPiece, StreamPiece, ToolCallPiece
|
||||
from app.rag.llm import (
|
||||
LLMClient,
|
||||
RetryPiece,
|
||||
StreamPiece,
|
||||
ToolCallPiece,
|
||||
ToolResultPiece,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.rag.scaffolding import ScaffoldingFilter
|
||||
@@ -254,8 +260,8 @@ def _run_call(
|
||||
|
||||
async def _consume(
|
||||
llm: LLMClient, db: Session, holder: AgentHolder
|
||||
) -> list[StreamPiece | ToolCallPiece | RetryPiece]:
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece] = []
|
||||
) -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
|
||||
async for piece in run_agent(
|
||||
llm,
|
||||
db,
|
||||
|
||||
@@ -16,6 +16,7 @@ import json
|
||||
import logging
|
||||
import math
|
||||
import re
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, cast
|
||||
@@ -24,15 +25,18 @@ import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from pydantic import ValidationError
|
||||
from sqlalchemy import delete, func, select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api import chat as chat_api
|
||||
from app.config import Settings, get_settings
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Chunk, GitSource, QueryLog
|
||||
from app.models import Chunk, Document, GitSource, QueryLog
|
||||
from app.rag import agent
|
||||
from app.rag.agent import AGENT_TOOLS
|
||||
from app.rag.agent import AGENT_TOOLS, READ_TRUNCATION_NOTICE
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.llm import EmbeddingError, LLMError, StreamPiece, ToolCallPiece
|
||||
from app.rag.llm import EmbeddingError, LLMError, StreamPiece, ToolCallPiece, ToolResultPiece
|
||||
from app.rag.prompts import build_high_prompt
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
from app.schemas import ChatDoneEvent, SourceRef
|
||||
from tests.conftest import ADMIN_PASSWORD
|
||||
|
||||
@@ -610,6 +614,352 @@ def test_chat_query_log_failure_still_sends_done(client, db, seeded_kb: FakeRagL
|
||||
# ---------- phase 37: agent document tools on grounded turns ----------
|
||||
|
||||
|
||||
async def _collect_run_agent(
|
||||
llm: FakeRagLLM,
|
||||
db: Session,
|
||||
system_prompt: str,
|
||||
settings: Settings,
|
||||
seed_docs: list[Document],
|
||||
) -> tuple[list[Any], agent.AgentHolder]:
|
||||
"""Consume one ``run_agent`` turn, returning the yielded pieces (in
|
||||
order) and the holder. Phase 95 (task 01): the direct agent-loop
|
||||
drive — the agent-loop yield order on the real prompt path, the
|
||||
complement of the endpoint-level ``tool_result`` SSE tests below
|
||||
(task 02)."""
|
||||
holder = agent.AgentHolder()
|
||||
pieces: list[Any] = []
|
||||
async for piece in agent.run_agent(
|
||||
llm, # pyright: ignore[reportArgumentType] # duck-typed LLMClient
|
||||
db,
|
||||
system_prompt=system_prompt,
|
||||
user_message=QUESTION,
|
||||
seed_docs=seed_docs,
|
||||
settings=settings,
|
||||
holder=holder,
|
||||
):
|
||||
pieces.append(piece)
|
||||
return pieces, holder
|
||||
|
||||
|
||||
def test_read_cap_truncates_and_yields_tool_result_on_real_prompt_path(
|
||||
db,
|
||||
) -> None:
|
||||
"""Phase 95 (task 01): on the REAL prompt path (a real Postgres
|
||||
document + the real ``build_high_prompt``), a ``read`` of a document
|
||||
LONGER than ``settings.read_max_chars`` truncates the result the model
|
||||
sees — first ``cap`` chars + the shared :data:`TRUNCATION_MARKER` + the
|
||||
pinned grep-pointer notice — and ``run_agent`` yields exactly ONE
|
||||
``ToolResultPiece``: AFTER the read's ``tool`` frame (the matching
|
||||
``ToolCallPiece``) and BEFORE the next model round. The endpoint-level
|
||||
``tool_result`` SSE frame is asserted separately below (task 02); this
|
||||
pins the agent-loop yield order on the real prompt path."""
|
||||
cap = 100
|
||||
content = "K" * (cap + 40) # 40 chars over the cap
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source="docs",
|
||||
path="big.md",
|
||||
full_path="/tmp/big.md",
|
||||
title="Big Doc",
|
||||
content=content,
|
||||
content_hash="1" * 64,
|
||||
)
|
||||
db.add(doc)
|
||||
db.commit()
|
||||
try:
|
||||
# The real prompt path: the actual HIGH prompt for the one doc.
|
||||
system_prompt = build_high_prompt([doc])
|
||||
settings = Settings(_env_file=None, read_max_chars=cap) # pyright: ignore[reportCallIssue]
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1", name="read", arguments={"path": "docs/big.md"}
|
||||
)
|
||||
]
|
||||
# the answer request (tools still offered, script
|
||||
# exhausted) falls back to the thinking + answer stream
|
||||
]
|
||||
)
|
||||
pieces, holder = asyncio.run(
|
||||
_collect_run_agent(scripted, db, system_prompt, settings, seed_docs=[])
|
||||
)
|
||||
# The model's context carried the truncated read — the first cap
|
||||
# chars, then the shared marker + the pinned grep-pointer notice
|
||||
# (so a downstream grep is the model's path to the rest). The
|
||||
# fake records the (mutated-in-place) messages list, so the same
|
||||
# tool message is aliased across requests — they all carry the
|
||||
# same content; take the last.
|
||||
tool_msgs = [
|
||||
m for r in scripted.seen_messages for m in r if m.get("role") == "tool"
|
||||
]
|
||||
assert tool_msgs, "the executed read must be appended as a tool message"
|
||||
body = tool_msgs[-1]["content"]
|
||||
assert body.startswith("Document docs/big.md:\n" + content[:cap])
|
||||
assert TRUNCATION_MARKER in body
|
||||
assert (
|
||||
READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) in body
|
||||
)
|
||||
# The yield order: the read's ToolCallPiece, then the ONE
|
||||
# ToolResultPiece, then the next round's answer content.
|
||||
kinds: list[str] = []
|
||||
for p in pieces:
|
||||
if isinstance(p, ToolCallPiece):
|
||||
kinds.append("toolcall")
|
||||
elif isinstance(p, ToolResultPiece):
|
||||
kinds.append("toolresult")
|
||||
elif isinstance(p, StreamPiece):
|
||||
kinds.append(p.kind)
|
||||
assert kinds.count("toolresult") == 1
|
||||
assert kinds.index("toolcall") < kinds.index("toolresult")
|
||||
assert kinds.index("toolresult") < kinds.index("content")
|
||||
# The piece carries (argument, shown, total) — the raw
|
||||
# source/path the model passed (what the tool frame carries), the
|
||||
# cap kept, the true length.
|
||||
(result_piece,) = [p for p in pieces if isinstance(p, ToolResultPiece)]
|
||||
assert result_piece.name == "read"
|
||||
assert result_piece.argument == "docs/big.md"
|
||||
assert result_piece.truncated is True
|
||||
assert result_piece.chars_shown == cap
|
||||
assert result_piece.chars_total == len(content)
|
||||
# Holder accounting: a truncated read is still a SUCCESSFUL call
|
||||
# (counted + added to context); the tuple is the signal only.
|
||||
assert holder.tool_calls == 1
|
||||
assert holder.read_docs == [doc]
|
||||
assert holder.read_truncations == [("docs/big.md", cap, len(content))]
|
||||
finally:
|
||||
db.delete(doc)
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_read_at_or_under_cap_yields_no_tool_result_on_real_prompt_path(
|
||||
db,
|
||||
) -> None:
|
||||
"""Phase 95 (task 01): the complement — a ``read`` of a document at or
|
||||
under the cap on the real prompt path is byte-identical to the
|
||||
pre-phase-95 agent loop: NO ``ToolResultPiece``, no holder entry, no
|
||||
marker in the model's context."""
|
||||
cap = 100
|
||||
content = "K" * cap # exactly at the cap → fits, not truncated
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source="docs",
|
||||
path="fits.md",
|
||||
full_path="/tmp/fits.md",
|
||||
title="Fits Doc",
|
||||
content=content,
|
||||
content_hash="2" * 64,
|
||||
)
|
||||
db.add(doc)
|
||||
db.commit()
|
||||
try:
|
||||
system_prompt = build_high_prompt([doc])
|
||||
settings = Settings(_env_file=None, read_max_chars=cap) # pyright: ignore[reportCallIssue]
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1", name="read", arguments={"path": "docs/fits.md"}
|
||||
)
|
||||
]
|
||||
]
|
||||
)
|
||||
pieces, holder = asyncio.run(
|
||||
_collect_run_agent(scripted, db, system_prompt, settings, seed_docs=[])
|
||||
)
|
||||
# No ToolResultPiece, no holder entry.
|
||||
assert not any(isinstance(p, ToolResultPiece) for p in pieces)
|
||||
assert holder.read_truncations == []
|
||||
# The model's context is the whole document, byte-identical to
|
||||
# the pre-phase-95 read result (no marker, no notice). (The fake
|
||||
# aliases the mutated messages list, so take the last tool msg.)
|
||||
tool_msgs = [
|
||||
m for r in scripted.seen_messages for m in r if m.get("role") == "tool"
|
||||
]
|
||||
assert tool_msgs, "the executed read must be appended as a tool message"
|
||||
assert tool_msgs[-1]["content"] == "Document docs/fits.md:\n" + content
|
||||
assert TRUNCATION_MARKER not in tool_msgs[-1]["content"]
|
||||
# Still a successful read.
|
||||
assert holder.tool_calls == 1
|
||||
assert holder.read_docs == [doc]
|
||||
finally:
|
||||
db.delete(doc)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _insert_big_doc(db, content: str) -> Document:
|
||||
"""One bare ``documents`` row (no chunks — the ``read`` lookup is a
|
||||
(source, path) identity match, not a retrieval) for the SSE-level
|
||||
read-cap tests: a document the model can only reach through the
|
||||
``read`` tool."""
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source="docs",
|
||||
path="big-read.md",
|
||||
full_path="/tmp/big-read.md",
|
||||
title="Big Read Doc",
|
||||
content=content,
|
||||
content_hash="3" * 64,
|
||||
)
|
||||
db.add(doc)
|
||||
db.commit()
|
||||
return doc
|
||||
|
||||
|
||||
def test_truncated_read_streams_tool_result_frame_after_tool_frame(
|
||||
client,
|
||||
db,
|
||||
seeded_kb: FakeRagLLM,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 95 (task 02, the A15 extension): a grounded turn whose
|
||||
scripted ``read`` hits a document LONGER than ``read_max_chars``
|
||||
(the cap lowered via the settings override — the task 01
|
||||
``Settings(_env_file=None, read_max_chars=…)`` pattern) streams the
|
||||
``tool`` → ``tool_result`` → ``delta…`` → ``done`` sequence: EXACTLY
|
||||
ONE ``tool_result`` frame, AFTER the matching ``tool`` frame (the
|
||||
line is already on screen) and BEFORE the next round's first frame,
|
||||
with the right shape and counts (``chars_shown`` = the cap,
|
||||
``chars_total`` = the true length). The model's context carried the
|
||||
truncated read (marker + pinned grep-pointer notice); the read is
|
||||
still cited (a truncated read is a successful call)."""
|
||||
cap = 100
|
||||
content = "K" * (cap + 150)
|
||||
doc = _insert_big_doc(db, content)
|
||||
live = get_settings()
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=live.relevance_threshold,
|
||||
read_max_chars=cap,
|
||||
),
|
||||
)
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1", name="read", arguments={"path": "docs/big-read.md"}
|
||||
)
|
||||
]
|
||||
# the answer request still carries the tools (1 round < the
|
||||
# default cap of 10); the script is exhausted, so the fake
|
||||
# falls back to the thinking + answer stream
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
db.delete(doc)
|
||||
db.commit()
|
||||
|
||||
types = [f["type"] for f in frames]
|
||||
assert "error" not in types
|
||||
tool_i = types.index("tool")
|
||||
tool_result_i = types.index("tool_result")
|
||||
# Exactly one tool_result frame…
|
||||
assert types.count("tool_result") == 1
|
||||
# …AFTER the matching tool frame and BEFORE the next model round's
|
||||
# first frame (the answer's deltas): tool → tool_result → delta…
|
||||
assert tool_i + 1 == tool_result_i
|
||||
assert tool_result_i < min(i for i, t in enumerate(types) if t == "delta")
|
||||
# The frame's exact shape: the additive seventh event type carries
|
||||
# the name/argument of the matching tool frame + the counts.
|
||||
frame = frames[tool_result_i]
|
||||
assert set(frame) == {
|
||||
"type",
|
||||
"name",
|
||||
"argument",
|
||||
"truncated",
|
||||
"chars_shown",
|
||||
"chars_total",
|
||||
}
|
||||
assert frame["name"] == frames[tool_i]["name"] == "read"
|
||||
assert frame["argument"] == frames[tool_i]["argument"] == "docs/big-read.md"
|
||||
assert frame["truncated"] is True
|
||||
assert frame["chars_shown"] == cap # the cap kept
|
||||
assert frame["chars_total"] == len(content) # the true length
|
||||
# The LLM's context carried the honest truncation: first cap chars +
|
||||
# the shared marker + the pinned grep-pointer notice (the fake
|
||||
# aliases the mutated messages list — take the last tool msg).
|
||||
tool_msgs = [
|
||||
m for r in scripted.seen_messages for m in r if m.get("role") == "tool"
|
||||
]
|
||||
assert tool_msgs
|
||||
body = tool_msgs[-1]["content"]
|
||||
assert body.startswith(f"Document docs/big-read.md:\n{content[:cap]}")
|
||||
assert TRUNCATION_MARKER in body
|
||||
assert READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) in body
|
||||
# The truncated read is still a SUCCESSFUL call — cited in done.
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done" and done["deflected"] is False
|
||||
assert ("docs", "big-read.md") in [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert ("docs", "homelab/kubernetes.md") in [
|
||||
(s["source"], s["path"]) for s in done["sources"]
|
||||
]
|
||||
|
||||
|
||||
def test_untruncated_read_streams_no_tool_result_frame(
|
||||
client,
|
||||
db,
|
||||
seeded_kb: FakeRagLLM,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 95 (task 02): the complement at the SSE level — a ``read``
|
||||
of a document AT OR UNDER the cap (the same long document, cap
|
||||
raised past its true length) streams NO ``tool_result`` frame (one
|
||||
frame = one noteworthy event; the six pre-existing event types are
|
||||
byte-identical), the ``tool`` frame is unchanged, and the model's
|
||||
context is the WHOLE document (no marker, no notice)."""
|
||||
content = "K" * 250
|
||||
doc = _insert_big_doc(db, content)
|
||||
live = get_settings()
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=live.relevance_threshold,
|
||||
read_max_chars=10_000, # far over the doc's true length
|
||||
),
|
||||
)
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1", name="read", arguments={"path": "docs/big-read.md"}
|
||||
)
|
||||
]
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
db.delete(doc)
|
||||
db.commit()
|
||||
|
||||
types = [f["type"] for f in frames]
|
||||
assert "error" not in types
|
||||
assert types.count("tool_result") == 0 # one frame = one noteworthy event
|
||||
assert types.count("tool") == 1
|
||||
(tool_frame,) = [f for f in frames if f["type"] == "tool"]
|
||||
assert set(tool_frame) == {"type", "name", "argument"} # byte-identical shape
|
||||
assert tool_frame["argument"] == "docs/big-read.md"
|
||||
assert frames[-1]["type"] == "done" and frames[-1]["deflected"] is False
|
||||
# The model saw the WHOLE document — no marker, no notice.
|
||||
tool_msgs = [
|
||||
m for r in scripted.seen_messages for m in r if m.get("role") == "tool"
|
||||
]
|
||||
assert tool_msgs
|
||||
assert tool_msgs[-1]["content"] == "Document docs/big-read.md:\n" + content
|
||||
assert TRUNCATION_MARKER not in tool_msgs[-1]["content"]
|
||||
|
||||
|
||||
def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
|
||||
@@ -54,6 +54,9 @@ SHARED_OUT_KEYS = {"title", "messages"}
|
||||
|
||||
#: A full ``bor.chat.v1`` brain record (phase 14 shape) — every optional
|
||||
#: key present; the round-trip test asserts it survives byte-identical.
|
||||
#: Phase 95 (task 02): the CURRENT full tool-entry shape — the additive
|
||||
#: truncation fields ride each record (a pre-phase-95 entry WITHOUT them
|
||||
#: still validates — the backward-compat pin in test_schemas.py).
|
||||
FULL_BRAIN: dict[str, Any] = {
|
||||
"who": "brain",
|
||||
"text": "Your k3s cluster runs on three nodes — you've got this.",
|
||||
@@ -64,12 +67,30 @@ FULL_BRAIN: dict[str, Any] = {
|
||||
"suggestions": ["What ports does Traefik expose?"],
|
||||
"thinking": "The kubernetes doc covers the cluster layout…",
|
||||
"tools": [
|
||||
{"name": "read", "argument": "Homelab/kubernetes.md"},
|
||||
{"name": "ls", "argument": None},
|
||||
{
|
||||
"name": "read",
|
||||
"argument": "Homelab/kubernetes.md",
|
||||
"truncated": True,
|
||||
"chars_shown": 128_000,
|
||||
"chars_total": 204_000,
|
||||
},
|
||||
{
|
||||
"name": "ls",
|
||||
"argument": None,
|
||||
"truncated": False,
|
||||
"chars_shown": None,
|
||||
"chars_total": None,
|
||||
},
|
||||
# Saved chats persisting the pre-phase-70 tool names still
|
||||
# validate — ``name`` is opaque to the API (no migration,
|
||||
# locked: old chats render fine).
|
||||
{"name": "read_document", "argument": "Homelab/legacy-notes.md"},
|
||||
{
|
||||
"name": "read_document",
|
||||
"argument": "Homelab/legacy-notes.md",
|
||||
"truncated": False,
|
||||
"chars_shown": None,
|
||||
"chars_total": None,
|
||||
},
|
||||
],
|
||||
"stopped": False,
|
||||
}
|
||||
|
||||
+205
-12
@@ -47,12 +47,21 @@ from app.models import Document, GitSource
|
||||
from app.rag import agent
|
||||
from app.rag.agent import (
|
||||
AGENT_TOOLS,
|
||||
READ_TRUNCATION_NOTICE,
|
||||
AgentHolder,
|
||||
MalformedReplyError,
|
||||
run_agent,
|
||||
)
|
||||
from app.rag.llm import LLMClient, LLMError, RetryPiece, StreamPiece, ToolCallPiece
|
||||
from app.rag.llm import (
|
||||
LLMClient,
|
||||
LLMError,
|
||||
RetryPiece,
|
||||
StreamPiece,
|
||||
ToolCallPiece,
|
||||
ToolResultPiece,
|
||||
)
|
||||
from app.rag.prompts import TOOLS_SECTION, _base, build_deflect_prompt, build_high_prompt
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.rag.scaffolding import ScaffoldingFilter
|
||||
@@ -121,11 +130,12 @@ async def _run(
|
||||
settings: Settings,
|
||||
seed_docs: list[Document] | None = None,
|
||||
history: Sequence[dict[str, Any]] = (),
|
||||
) -> list[StreamPiece | ToolCallPiece | RetryPiece]:
|
||||
) -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
|
||||
"""Consume one ``run_agent`` turn; *history* (phase 74) is the
|
||||
client's prior turns spliced between system and user (default
|
||||
``()`` — the pre-phase-74 two-message request)."""
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece] = []
|
||||
``()`` — the pre-phase-74 two-message request). Phase 95: the loop
|
||||
may also yield a ``ToolResultPiece`` (a truncated ``read``)."""
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
|
||||
async for piece in run_agent(
|
||||
cast("LLMClient", llm),
|
||||
cast("Session", None),
|
||||
@@ -192,15 +202,25 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
# section already carries (every refusal of a 12-call run was
|
||||
# ALREADY_IN_CONTEXT); the rule now leads the description instead
|
||||
# of sitting mid-paragraph, and the tool is framed as "only for
|
||||
# documents NOT already in <documents>".
|
||||
# documents NOT already in <documents>". Phase 95 (task 01): the
|
||||
# read-truncation sentence is inserted before the one-call-at-a-
|
||||
# time discipline clause (the discipline rule stays last, as in the
|
||||
# other two tools) — a capped read carries the TRUNCATED notice and
|
||||
# the `grep` follow-up (the pinned copy).
|
||||
assert read["description"] == (
|
||||
"Do not call this tool for a document already shown in "
|
||||
"the <documents> section, even when the user asks you to "
|
||||
"open or read it — its full text is already in your "
|
||||
"prompt; answer directly from it. Use it only to add a "
|
||||
"document NOT already in <documents> to your context, "
|
||||
"by its combined `source/path` string. Call one tool at "
|
||||
"a time — wait for this result before your next call."
|
||||
"by its combined `source/path` string. Very large "
|
||||
"documents are truncated: you receive the first part "
|
||||
"plus a TRUNCATED notice naming how many more characters "
|
||||
"exist — the notice is authoritative, the document did "
|
||||
"NOT end where it stopped. Follow it and use `grep` "
|
||||
"(pattern) to locate the rest — it searches the whole "
|
||||
"document. Call one tool at a time — wait for this "
|
||||
"result before your next call."
|
||||
)
|
||||
read_params = read["parameters"]
|
||||
assert read_params["type"] == "object"
|
||||
@@ -1519,6 +1539,177 @@ def test_reading_an_already_read_doc_is_deduped(monkeypatch: pytest.MonkeyPatch)
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
# ---------- phase 95: the read cap (bounded reads, honest truncation) ----------
|
||||
|
||||
|
||||
def test_read_exactly_at_cap_is_byte_identical_and_untruncated(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 95 boundary: a document whose content length EQUALS the cap
|
||||
fits — read whole, byte-identical to the pre-phase-95 result (no
|
||||
marker, no notice, no holder entry, no ``ToolResultPiece``)."""
|
||||
cap = 20
|
||||
content = "x" * cap
|
||||
doc = _doc("S", "big.md", "Big", content)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/big.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
out = asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
|
||||
# Byte-identical to today's read result (no marker, no notice).
|
||||
assert llm.requests[1][0][3]["content"] == "Document S/big.md:\n" + content
|
||||
assert TRUNCATION_MARKER not in llm.requests[1][0][3]["content"]
|
||||
# No truncation recorded, none surfaced to the loop.
|
||||
assert holder.read_truncations == []
|
||||
assert not any(isinstance(p, ToolResultPiece) for p in out)
|
||||
# Still a successful read.
|
||||
assert holder.read_docs == [doc]
|
||||
assert holder.tool_calls == 1
|
||||
|
||||
|
||||
def test_read_at_cap_plus_one_truncates_with_marker_and_notice(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 95 boundary: ONE char over the cap truncates — the first
|
||||
``cap`` chars + the shared :data:`TRUNCATION_MARKER` + the pinned
|
||||
grep-pointer notice (``{total}`` = the true length, ``{shown}`` = the
|
||||
cap), and the truncation is recorded on the holder. A truncated read
|
||||
is still a successful call (``tool_calls`` / ``read_docs`` as today)."""
|
||||
cap = 20
|
||||
content = "x" * (cap + 1)
|
||||
doc = _doc("S", "big.md", "Big", content)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/big.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
|
||||
expected = (
|
||||
"Document S/big.md:\n"
|
||||
+ content[:cap]
|
||||
+ "\n"
|
||||
+ TRUNCATION_MARKER
|
||||
+ "\n"
|
||||
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=cap + 1)
|
||||
)
|
||||
assert llm.requests[1][0][3]["content"] == expected
|
||||
# (argument, chars_shown, chars_total) — the raw argument the tool
|
||||
# frame carries, the cap kept, the true length.
|
||||
assert holder.read_truncations == [("S/big.md", cap, cap + 1)]
|
||||
assert holder.read_docs == [doc] # still added to the context
|
||||
assert holder.tool_calls == 1 # still a counted, successful call
|
||||
|
||||
|
||||
def test_read_truncation_notice_is_pinned() -> None:
|
||||
"""Phase 95: the notice copy is pinned — it names the true length
|
||||
(``{total}``), the cap kept (``{shown}``), states the rest is NOT
|
||||
shown (the document did not end where it stopped), and points at
|
||||
``grep`` (which searches the whole document)."""
|
||||
assert READ_TRUNCATION_NOTICE.format(shown=100, total=250) == (
|
||||
"TRUNCATED — this document is 250 characters; only the first "
|
||||
"100 are in your context. The rest is NOT shown. Use grep "
|
||||
"(pattern) to locate what you need — grep searches the whole "
|
||||
"document."
|
||||
)
|
||||
|
||||
|
||||
def test_run_agent_yields_tool_result_after_tool_frame_before_next_round(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 95: ``run_agent`` yields exactly ONE ``ToolResultPiece`` per
|
||||
truncated read — AFTER the round's ``tool`` frame (the matching
|
||||
``ToolCallPiece``) and BEFORE the next model round (the answer
|
||||
pieces). It carries (argument, shown, total); ``argument`` is the
|
||||
same value the matching ``tool`` frame carries (the raw
|
||||
``source/path`` the model passed)."""
|
||||
cap = 20
|
||||
content = "y" * (cap + 5)
|
||||
doc = _doc("S", "big.md", "Big", content)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/big.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
out = asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
|
||||
# The piece order: the read's ToolCallPiece, then the ToolResultPiece,
|
||||
# then the next round's content (the answer).
|
||||
assert isinstance(out[0], ToolCallPiece) and out[0].name == "read"
|
||||
piece = out[1]
|
||||
assert isinstance(piece, ToolResultPiece)
|
||||
assert isinstance(out[2], StreamPiece)
|
||||
assert piece.name == "read"
|
||||
assert piece.argument == "S/big.md"
|
||||
assert piece.truncated is True
|
||||
assert piece.chars_shown == cap
|
||||
assert piece.chars_total == cap + 5
|
||||
# Exactly one ToolResultPiece for the one truncated read.
|
||||
assert [p for p in out if isinstance(p, ToolResultPiece)] == [piece]
|
||||
|
||||
|
||||
def test_run_agent_short_read_yields_no_tool_result_piece(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 95: a read at or under the cap yields NO ``ToolResultPiece``
|
||||
— the non-truncated stream is byte-identical to the pre-phase-95
|
||||
one (just the ``ToolCallPiece`` + the answer)."""
|
||||
cap = 20
|
||||
content = "z" * cap # exactly at the cap → not truncated
|
||||
doc = _doc("S", "small.md", "Small", content)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/small.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
out = asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
|
||||
assert not any(isinstance(p, ToolResultPiece) for p in out)
|
||||
assert holder.read_truncations == []
|
||||
# Order: the ToolCallPiece then the answer content (no piece between).
|
||||
assert isinstance(out[0], ToolCallPiece) and out[0].name == "read"
|
||||
assert isinstance(out[1], StreamPiece)
|
||||
|
||||
|
||||
def test_read_truncation_does_not_touch_refusal_paths(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 95: the read refusal paths are untouched by the cap — a SEED
|
||||
document that is over the cap is still refused with
|
||||
``ALREADY_IN_CONTEXT`` (not truncated, nothing recorded, nothing
|
||||
counted), and an unknown path is still the no-document refusal (no
|
||||
content is read, so no truncation either)."""
|
||||
big = "B" * 5000 # far over the tiny cap below
|
||||
seed = _doc("S", "seed.md", "Seed", big)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [])
|
||||
# (a) Reading the (over-cap) seed doc → ALREADY_IN_CONTEXT (refusal).
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/seed.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(read_max_chars=100), seed_docs=[seed]))
|
||||
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert holder.read_truncations == []
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
# (b) An unknown path → the no-document refusal (argument echoed),
|
||||
# even though a big doc could have truncated — no content is read.
|
||||
holder2 = AgentHolder()
|
||||
llm2 = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/missing.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm2, holder2, _settings(read_max_chars=100), seed_docs=[seed]))
|
||||
assert llm2.requests[1][0][3]["content"] == (
|
||||
"No document at 'S/missing.md' — check the ls output."
|
||||
)
|
||||
assert holder2.read_truncations == []
|
||||
assert holder2.tool_calls == 0 and holder2.read_docs == []
|
||||
|
||||
|
||||
def test_unknown_tool_name_refused(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
@@ -2322,8 +2513,8 @@ def test_round_failure_after_first_piece_is_terminal(monkeypatch: pytest.MonkeyP
|
||||
)
|
||||
sleeps = _record_sleeps(monkeypatch)
|
||||
|
||||
async def drain() -> list[StreamPiece | ToolCallPiece | RetryPiece]:
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece] = []
|
||||
async def drain() -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
|
||||
with pytest.raises(LLMError, match="mid-stream drop"):
|
||||
async for piece in run_agent(
|
||||
cast("LLMClient", llm),
|
||||
@@ -2384,8 +2575,8 @@ def test_zero_retries_is_one_plain_attempt(monkeypatch: pytest.MonkeyPatch) -> N
|
||||
llm = FailingLLM([([], LLMError("connection refused"))])
|
||||
sleeps = _record_sleeps(monkeypatch)
|
||||
|
||||
async def drain() -> list[StreamPiece | ToolCallPiece | RetryPiece]:
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece] = []
|
||||
async def drain() -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
|
||||
with pytest.raises(LLMError, match="connection refused"):
|
||||
async for piece in run_agent(
|
||||
cast("LLMClient", llm),
|
||||
@@ -2434,7 +2625,9 @@ def test_abandon_mid_retry_sleep_leaks_nothing(monkeypatch: pytest.MonkeyPatch)
|
||||
holder=holder,
|
||||
)
|
||||
|
||||
async def consumer() -> list[StreamPiece | ToolCallPiece | RetryPiece]:
|
||||
async def consumer() -> list[
|
||||
StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece
|
||||
]:
|
||||
return [p async for p in gen]
|
||||
|
||||
task = asyncio.ensure_future(consumer())
|
||||
|
||||
@@ -209,6 +209,32 @@ def test_agent_max_rounds_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> N
|
||||
_settings()
|
||||
|
||||
|
||||
def test_read_max_chars_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 95: the agent ``read`` tool's result is capped at
|
||||
``BOR_READ_MAX_CHARS`` (default 128 000 chars ≈ 32k tokens — a
|
||||
quarter of the owner's 128k-token minimum context). Env-tunable in
|
||||
both directions."""
|
||||
monkeypatch.delenv("BOR_READ_MAX_CHARS", raising=False)
|
||||
assert _settings().read_max_chars == 128_000
|
||||
monkeypatch.setenv("BOR_READ_MAX_CHARS", "5000")
|
||||
assert _settings().read_max_chars == 5000
|
||||
# ``0`` is legal (every non-empty read truncates to the marker +
|
||||
# notice) — it is not a kill switch, so no lower-bound error.
|
||||
monkeypatch.setenv("BOR_READ_MAX_CHARS", "0")
|
||||
assert _settings().read_max_chars == 0
|
||||
|
||||
|
||||
def test_read_max_chars_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A negative cap is a typo — it would slice from the END of the
|
||||
content (negative indexing) instead of failing, so the validator
|
||||
fails loudly at startup (the ``agent_max_rounds`` pattern)."""
|
||||
monkeypatch.setenv("BOR_READ_MAX_CHARS", "-1")
|
||||
with pytest.raises(ValidationError, match="read_max_chars"):
|
||||
_settings()
|
||||
|
||||
|
||||
def test_stream_thinking_default_true_and_env_parse(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Phase 17 kill-switch (``BOR_STREAM_THINKING``): on by default,
|
||||
``0``/``false`` turn the ``thinking`` SSE frames off."""
|
||||
|
||||
@@ -229,6 +229,24 @@ def test_tools_section_phase72_contract_clauses() -> None:
|
||||
assert "not a directory or file path" not in TOOLS_SECTION
|
||||
|
||||
|
||||
def test_tools_section_phase95_read_truncation_clause() -> None:
|
||||
"""Phase 95 (task 01): the ``read`` teaching gains exactly one line —
|
||||
very large documents are capped, a cut read returns the first part
|
||||
plus the TRUNCATED notice (the document did not end where it
|
||||
stopped), and ``grep`` is the follow-up (it searches the whole
|
||||
document). The ``ls``/``grep`` teaching is untouched (phase 94 owns
|
||||
``ls``) — the clause is pinned byte-for-byte in the constant."""
|
||||
assert (
|
||||
"Very large documents are capped: a cut read returns the first "
|
||||
"part plus a TRUNCATED notice — the document did not end where "
|
||||
"it stopped; use `grep` (pattern) to find the rest, it searches "
|
||||
"the whole document."
|
||||
) in TOOLS_SECTION
|
||||
# It rides the HIGH prompt and never the LOW (deflected) prompt.
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes")
|
||||
assert "Very large documents are capped" in build_high_prompt([doc])
|
||||
|
||||
|
||||
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
|
||||
"""Phase 72/94: the contract clauses ride the HIGH prompt with the
|
||||
rest of the section and never leak into the LOW/deflection prompt
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
"""Unit: the phase-95 (task 02) truncation-marker frontend contract.
|
||||
|
||||
No new Python app logic exists for the marker itself — the behavior
|
||||
lives in ``frontend/assets/app.js`` (the ``tool_result`` SSE branch +
|
||||
the ``toolAcc`` stamp + the phase-14 restore marker),
|
||||
``frontend/assets/shared.js`` (the shared page's local tool-line
|
||||
render) and ``frontend/assets/styles.css`` (the theme-neutral
|
||||
``.truncated-note`` rule). Like the other frontend-adjacent unit files
|
||||
(``test_frontend_tool_states.py`` is the phase-37 precedent), this
|
||||
module pins the JS/CSS markers the story depends on, so a silent
|
||||
regression in the handler, the pinned marker copy, the persistence
|
||||
stamp, or the styling is caught without a browser. The E2E gate is the
|
||||
phase's story suite (task 03).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
|
||||
APP_JS = FRONTEND / "assets" / "app.js"
|
||||
SHARED_JS = FRONTEND / "assets" / "shared.js"
|
||||
STYLES_CSS = FRONTEND / "assets" / "styles.css"
|
||||
|
||||
#: The pinned marker copy (the unit + E2E assertion target): plain
|
||||
#: integers, no thousands separators, the em-dash per the owner's TODO.
|
||||
MARKER_TEMPLATE = '" (truncated — showing "'
|
||||
|
||||
|
||||
def _js() -> str:
|
||||
return APP_JS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _shared_js() -> str:
|
||||
return SHARED_JS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _css() -> str:
|
||||
return STYLES_CSS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_tool_result_branch_is_a_first_class_turn_branch() -> None:
|
||||
"""The turn handler must branch on `tool_result` frames (the
|
||||
seventh, optional event type — the A15 extension): the branch sits
|
||||
after `done` (a frame for a settled turn is a tolerated late
|
||||
append, never a crash) and before `error`, settles the tool-line
|
||||
clock like every other frame, and must never flip the state
|
||||
machine (the phase-37/48 lifecycle is untouched)."""
|
||||
js = _js()
|
||||
done_idx = js.find('ev.type === "done"')
|
||||
tool_result_idx = js.find('ev.type === "tool_result"')
|
||||
error_idx = js.find('ev.type === "error"')
|
||||
assert -1 < done_idx < tool_result_idx < error_idx, (
|
||||
"the turn handler must branch on tool_result frames (after done,"
|
||||
" before error)"
|
||||
)
|
||||
branch = js[tool_result_idx:error_idx]
|
||||
assert "settleToolLine()" in branch, (
|
||||
"a frame arrived — the latest tool line's elapsed clock settles"
|
||||
)
|
||||
assert "appendTruncatedNote(wrap, argument, shown, total)" in branch
|
||||
# No UI-state transition: the state machine never knows the marker.
|
||||
assert "setUiState" not in branch
|
||||
assert "aborted" not in branch, (
|
||||
"the aborted guard lives at the dispatch top, not per branch"
|
||||
)
|
||||
|
||||
|
||||
def test_truncation_marker_copy_is_pinned() -> None:
|
||||
"""The marker text is pinned EXACTLY: \" (truncated — showing N of
|
||||
M chars)\" — plain integers (no separators), a leading space (it
|
||||
follows the line's <code> child), the em-dash per the TODO copy.
|
||||
Both app.js's helper and shared.js's local render carry the same
|
||||
template (pixel-identical marker — the phase-50 restore
|
||||
contract)."""
|
||||
js = _js()
|
||||
fn = js.find("function appendTruncatedNote")
|
||||
assert fn != -1, "appendTruncatedNote must exist in app.js"
|
||||
body = js[fn : js.find("\n}\n", fn)]
|
||||
assert MARKER_TEMPLATE in body
|
||||
assert '+ charsShown + " of " + charsTotal + " chars)"' in body
|
||||
shared = _shared_js()
|
||||
fn = shared.find("function addToolLines")
|
||||
assert fn != -1
|
||||
sbody = shared[fn : shared.find("\n}\n", fn)]
|
||||
assert MARKER_TEMPLATE in sbody
|
||||
assert 'chars_shown) || 0) + " of " + (Number(t.chars_total) || 0) + " chars)"' in sbody
|
||||
|
||||
|
||||
def test_append_truncated_note_matches_newest_line_and_uses_text_content() -> None:
|
||||
"""appendTruncatedNote: the target is the NEWEST `.tool-call` line
|
||||
whose `<code>` child carries the frame's argument (the raw
|
||||
source/path — the same string the matching `tool` frame put in the
|
||||
line), scanned newest-first; the marker is a SPAN sibling appended
|
||||
to the existing line (createElement + className + textContent only
|
||||
— the house "this file never builds HTML" rule, no innerHTML); a
|
||||
frame whose line is gone (New Chat mid-turn) is a silent no-op."""
|
||||
js = _js()
|
||||
fn = js.find("function appendTruncatedNote")
|
||||
assert fn != -1
|
||||
body = js[fn : js.find("\n}\n", fn)]
|
||||
assert 'wrap?.querySelector?.(".tool-calls")' in body, (
|
||||
"a wrap without tool lines is a silent no-op"
|
||||
)
|
||||
assert 'querySelectorAll(".tool-call")' in body
|
||||
assert "for (let i = lines.length - 1; i >= 0; i -= 1)" in body, (
|
||||
"newest line first — the last call for that argument"
|
||||
)
|
||||
assert 'code.textContent !== argument' in body, (
|
||||
"the match key is the code child's argument (the raw source/path)"
|
||||
)
|
||||
assert 'note.className = "truncated-note"' in body
|
||||
assert 'lines[i].appendChild(note)' in body, (
|
||||
"DOM append to the EXISTING line — no new line, no re-render"
|
||||
)
|
||||
assert "innerHTML" not in body, (
|
||||
"no HTML injection surface — createElement + textContent only"
|
||||
)
|
||||
# The no-op guard: a wrap without tool lines returns silently; a scan
|
||||
# that finds no matching line falls off the loop without appending.
|
||||
assert "if (!calls) return;" in body
|
||||
|
||||
|
||||
def test_tool_result_branch_stamps_the_newest_toolacc_entry() -> None:
|
||||
"""The `tool_result` frame stamps the matching toolAcc entry (same
|
||||
argument, NEWEST — the reverse scan mirrors the line match) with
|
||||
`truncated` + `chars_shown` + `chars_total` — the `done` save point
|
||||
below then carries it with zero other change (the persistence
|
||||
shape rides the existing `tools` key). The stamp is gated on the
|
||||
frame's argument + truncated truth (a malformed frame is a silent
|
||||
no-op for persistence)."""
|
||||
js = _js()
|
||||
tool_result_idx = js.find('ev.type === "tool_result"')
|
||||
error_idx = js.find('ev.type === "error"')
|
||||
assert -1 < tool_result_idx < error_idx
|
||||
branch = js[tool_result_idx:error_idx]
|
||||
assert "t.truncated = true" in branch
|
||||
assert "t.chars_shown = shown" in branch
|
||||
assert "t.chars_total = total" in branch
|
||||
assert "for (let i = toolAcc.length - 1; i >= 0; i -= 1)" in branch, (
|
||||
"the NEWEST matching entry (the reverse scan, same rule as the"
|
||||
" line match) gets stamped — then break"
|
||||
)
|
||||
assert "t.argument === argument" in branch
|
||||
assert "break" in branch
|
||||
# The guard: the stamp only runs for a real truncation with an argument.
|
||||
assert "if (argument && ev.truncated)" in branch
|
||||
# The saved payload rides the existing save point — no second tools
|
||||
# key, no new record field.
|
||||
done_block = js[js.find('ev.type === "done"') : tool_result_idx]
|
||||
assert "tools: toolAcc.length ? toolAcc : undefined" in done_block
|
||||
|
||||
|
||||
def test_restore_path_renders_the_stored_marker() -> None:
|
||||
"""The phase-14 LOCAL restore path (renderStoredMessage): a stored
|
||||
tool record with `truncated` + the counts re-renders the SAME
|
||||
marker next to its Reading line, right after the line is restored
|
||||
(the same order as the live frames). A pre-phase-95 record (no
|
||||
field — `t.truncated` falsy) renders unchanged (no marker, no
|
||||
migration)."""
|
||||
js = _js()
|
||||
fn = js.find("function renderStoredMessage")
|
||||
assert fn != -1
|
||||
end = js.find("function restoreConversation")
|
||||
body = js[fn:end]
|
||||
assert "appendToolLine(wrap, t.name, arg)" in body
|
||||
assert "t.truncated && arg" in body, (
|
||||
"only an argument-bearing (Reading) record with the flag renders"
|
||||
" the marker"
|
||||
)
|
||||
assert (
|
||||
"appendTruncatedNote(wrap, arg, Number(t.chars_shown) || 0,"
|
||||
" Number(t.chars_total) || 0)" in body
|
||||
)
|
||||
# The marker append sits INSIDE the tools loop, after the line append.
|
||||
line_idx = body.find("appendToolLine(wrap, t.name, arg)")
|
||||
note_idx = body.find("appendTruncatedNote(wrap, arg,")
|
||||
assert -1 < line_idx < note_idx
|
||||
|
||||
|
||||
def test_shared_page_renders_the_stored_marker() -> None:
|
||||
"""shared.js's local tool-line render (addToolLines): the same
|
||||
marker from the stored record — a span sibling appended to the
|
||||
line, after the line's existing children (the template text + the
|
||||
<code> argument), textContent only (no HTML from storage, ever). A
|
||||
record saved before phase 95 renders exactly as before."""
|
||||
js = _shared_js()
|
||||
fn = js.find("function addToolLines")
|
||||
assert fn != -1
|
||||
body = js[fn : js.find("\n}\n", fn)]
|
||||
assert "t.truncated && argument" in body
|
||||
assert 'note.className = "truncated-note"' in body
|
||||
assert 'line.appendChild(note)' in body
|
||||
assert MARKER_TEMPLATE in body
|
||||
# Still textContent-only: the marker adds no innerHTML surface, and
|
||||
# the three argument-bearing lines keep their textContent treatment.
|
||||
assert body.count("code.textContent = argument") == 3
|
||||
assert "innerHTML" not in body
|
||||
|
||||
|
||||
def test_truncated_note_style_is_theme_neutral() -> None:
|
||||
"""styles.css: `.tool-call .truncated-note` exists and colors ONLY
|
||||
through a `var(--…)` token (the phase-92 zero-literal invariant —
|
||||
no new hue; under phase 93's monochrome theme it grays
|
||||
automatically, and the marker stays TEXT, never color alone, B5).
|
||||
--ink-soft is the AA-safe soft-ink the status suffixes already
|
||||
borrow."""
|
||||
css = _css()
|
||||
m = re.search(r"\.tool-call \.truncated-note \{([^}]*)\}", css)
|
||||
assert m, "the .tool-call .truncated-note rule must exist"
|
||||
rule = m.group(1)
|
||||
assert "var(--ink-soft)" in rule
|
||||
assert "color: var(--ink-soft)" in rule
|
||||
# Theme-neutral: the whole rule is a single var() color — no hex,
|
||||
# no rgb(), no other property.
|
||||
assert not re.search(r"#[0-9a-fA-F]{3,8}\b|rgb\(", rule)
|
||||
|
||||
|
||||
def test_no_cdn_added() -> None:
|
||||
"""AGENTS.md rule 6: the marker adds no external script/link."""
|
||||
index = (FRONTEND / "index.html").read_text(encoding="utf-8")
|
||||
assert 'src="http' not in index and 'href="http' not in index
|
||||
+165
-3
@@ -41,7 +41,15 @@ def _source_ref() -> dict:
|
||||
|
||||
|
||||
def _tool_call() -> dict:
|
||||
return {"name": "read", "argument": "Homelab/kubernetes.md"}
|
||||
# Phase 95 (task 02): the truncation record rides the same entry —
|
||||
# the CURRENT full shape (additive fields, defaults for a plain read).
|
||||
return {
|
||||
"name": "read",
|
||||
"argument": "Homelab/kubernetes.md",
|
||||
"truncated": False,
|
||||
"chars_shown": None,
|
||||
"chars_total": None,
|
||||
}
|
||||
|
||||
|
||||
def _user_message(text: str = "How did I install k3s?") -> ChatMessage:
|
||||
@@ -341,8 +349,22 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
|
||||
"suggestions": None,
|
||||
"thinking": "The kubernetes doc covers the cluster layout…",
|
||||
"tools": [
|
||||
{"name": "read", "argument": "Homelab/kubernetes.md"},
|
||||
{"name": "ls", "argument": None},
|
||||
# Phase 95 (task 02): the current full shape — one
|
||||
# truncated read (the marker record) + a plain ls.
|
||||
{
|
||||
"name": "read",
|
||||
"argument": "Homelab/kubernetes.md",
|
||||
"truncated": True,
|
||||
"chars_shown": 128_000,
|
||||
"chars_total": 204_000,
|
||||
},
|
||||
{
|
||||
"name": "ls",
|
||||
"argument": None,
|
||||
"truncated": False,
|
||||
"chars_shown": None,
|
||||
"chars_total": None,
|
||||
},
|
||||
],
|
||||
"stopped": None,
|
||||
},
|
||||
@@ -393,3 +415,143 @@ def test_realistic_payload_round_trips_through_update_model() -> None:
|
||||
}
|
||||
payload = SavedChatUpdate.model_validate({"messages": [msg]})
|
||||
assert payload.model_dump()["messages"] == [msg]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 95 (task 02): ToolCall truncation fields + ChatToolResultEvent
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_tool_call_round_trips_with_truncation_fields() -> None:
|
||||
"""The CURRENT full shape (phase 95 task 02): a truncated read's
|
||||
record (``truncated: True`` + the two non-negative counts) validates
|
||||
and round-trips ``model_dump()`` unchanged — the save payload carries
|
||||
it with zero other change (the UI re-renders the marker from it)."""
|
||||
raw = {
|
||||
"name": "read",
|
||||
"argument": "Homelab/big.md",
|
||||
"truncated": True,
|
||||
"chars_shown": 128_000,
|
||||
"chars_total": 204_000,
|
||||
}
|
||||
call = ToolCall.model_validate(raw)
|
||||
assert call.truncated is True
|
||||
assert call.chars_shown == 128_000
|
||||
assert call.chars_total == 204_000
|
||||
assert call.model_dump() == raw
|
||||
|
||||
|
||||
def test_tool_call_old_shape_validates_with_defaults() -> None:
|
||||
"""Backward-compat (the phase-50 rule): a saved chat written BEFORE
|
||||
phase 95 — tool records without the truncation fields — validates
|
||||
UNCHANGED: ``truncated`` defaults to False (the marker is absent),
|
||||
the counts to None. No migration (``ChatMessage.tools`` is JSON);
|
||||
only the dump gains the additive keys with their defaults."""
|
||||
old_shape = {"name": "read", "argument": "Homelab/kubernetes.md"}
|
||||
call = ToolCall.model_validate(old_shape)
|
||||
assert call.truncated is False
|
||||
assert call.chars_shown is None
|
||||
assert call.chars_total is None
|
||||
dumped = call.model_dump()
|
||||
assert dumped["name"] == "read" and dumped["argument"] == "Homelab/kubernetes.md"
|
||||
assert dumped["truncated"] is False
|
||||
assert dumped["chars_shown"] is None and dumped["chars_total"] is None
|
||||
|
||||
|
||||
def test_tool_call_old_shape_message_still_round_trips_as_record() -> None:
|
||||
"""The record-level backward-compat: a pre-phase-95 brain message
|
||||
(old-shape ``tools``) validates inside ``ChatMessage`` and dumps back
|
||||
as a VALID record of the same shape (the frontend renders it without
|
||||
the marker — ``truncated`` falsy)."""
|
||||
old_message = {
|
||||
"who": "brain",
|
||||
"text": "You've got this!",
|
||||
"sources": None,
|
||||
"deflected": False,
|
||||
"suggestions": None,
|
||||
"thinking": None,
|
||||
"tools": [
|
||||
{"name": "read", "argument": "Homelab/kubernetes.md"},
|
||||
{"name": "read_document", "argument": "Homelab/legacy.md"},
|
||||
],
|
||||
"stopped": None,
|
||||
}
|
||||
msg = ChatMessage.model_validate(old_message)
|
||||
assert all(t.truncated is False for t in (msg.tools or []))
|
||||
# Re-validating the dump is a no-op (lossless record round-trip).
|
||||
ChatMessage.model_validate(msg.model_dump())
|
||||
|
||||
|
||||
def test_tool_call_counts_reject_negative() -> None:
|
||||
"""Phase 83 bounds philosophy: the counts are non-negative
|
||||
(``ge=0``) — a negative count is not a real record."""
|
||||
with pytest.raises(ValidationError):
|
||||
ToolCall.model_validate(
|
||||
{
|
||||
"name": "read",
|
||||
"argument": "x",
|
||||
"truncated": True,
|
||||
"chars_shown": -1,
|
||||
"chars_total": 5,
|
||||
}
|
||||
)
|
||||
with pytest.raises(ValidationError):
|
||||
ToolCall.model_validate(
|
||||
{
|
||||
"name": "read",
|
||||
"argument": "x",
|
||||
"truncated": True,
|
||||
"chars_shown": 5,
|
||||
"chars_total": -1,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def test_chat_tool_result_event_shape() -> None:
|
||||
"""The A15 extension's wire shape (phase 95 task 02): the seventh,
|
||||
OPTIONAL SSE event type — ``{type, name, argument, truncated,
|
||||
chars_shown, chars_total}`` — with the pinned field order, the
|
||||
``tool_result`` default, ``truncated`` defaulting True (the emission
|
||||
trigger), and the non-negative counts."""
|
||||
from app.schemas import ChatToolResultEvent
|
||||
|
||||
ev = ChatToolResultEvent(
|
||||
name="read",
|
||||
argument="docs/big.md",
|
||||
truncated=True,
|
||||
chars_shown=128_000,
|
||||
chars_total=204_000,
|
||||
)
|
||||
dumped = ev.model_dump()
|
||||
assert list(dumped) == [
|
||||
"type",
|
||||
"name",
|
||||
"argument",
|
||||
"truncated",
|
||||
"chars_shown",
|
||||
"chars_total",
|
||||
]
|
||||
assert dumped == {
|
||||
"type": "tool_result",
|
||||
"name": "read",
|
||||
"argument": "docs/big.md",
|
||||
"truncated": True,
|
||||
"chars_shown": 128_000,
|
||||
"chars_total": 204_000,
|
||||
}
|
||||
# The emission trigger defaults: the pump always builds the frame from
|
||||
# a piece, so a frame that ever exists carries the truncation truth.
|
||||
minimal = ChatToolResultEvent(
|
||||
name="read", argument=None, chars_shown=0, chars_total=0
|
||||
).model_dump()
|
||||
assert minimal["truncated"] is True
|
||||
|
||||
|
||||
def test_chat_tool_result_event_counts_reject_negative() -> None:
|
||||
"""The frame's counts are non-negative (``ge=0``), like the record's."""
|
||||
from app.schemas import ChatToolResultEvent
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
ChatToolResultEvent(name="read", argument="x", chars_shown=-1, chars_total=5)
|
||||
with pytest.raises(ValidationError):
|
||||
ChatToolResultEvent(name="read", argument="x", chars_shown=5, chars_total=-1)
|
||||
|
||||
Reference in New Issue
Block a user