phase: 95_read_truncation_cap
Build and Push Containers / build-and-push-app (push) Successful in 1m38s
Build and Push Containers / build-and-push-db (push) Successful in 12s

All gates are green. Final verification is complete — no fixes were needed; the phase is fully implemented and passing.

**Phase 95 final verification report**

- Verified all 3 tasks already implemented (task files in `complete/`): read cap + notice + `ToolResultPiece` (task 01), `tool_result` SSE + live/saved/shared marker (task 02), E2E suite (task 03).
- Code checks passed: `BOR_READ_MAX_CHARS` (default 128 000, validator), pinned `READ_TRUNCATION_NOTICE` + `TRUNCATION_MARKER`, byte-identical at/under-cap read, holder `read_truncations` (no `tool_calls` impact), piece order (after `tool`, before next round), `ChatToolResultEvent`, `ToolCall` back-compat fields, `app.js` live/restore handlers, `shared.js`, `.truncated-note` CSS, `.env.example` entry, mock-LLM echo script.
- **Tests:** `uv run pytest` → 1966 passed; `uv run pytest --cov=app --cov-report=term-missing` → all green, TOTAL **99%** (>90% gate).
- **E2E:** `uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov` → **3 passed** (frame order + live marker + LLM notice via echo; save→shared fidelity; short-read control).
- **Regression (isolated):** `test_agent_document_tools` 4 ✓, `test_chat_history` 5 ✓, `test_share_chat` 4 ✓, `test_big_read_progress` 4 ✓, `test_stop_generation` 3 ✓.
- **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings.

**Completion criteria:** ① over-cap read → first-cap-chars + marker + pinned notice — ✓ (unit-pinned: at-cap/cap+1/notice tests); ② user marker live/saved/shared — ✓ (E2E + frontend tests); ③ at/under cap byte-identical, no frame — ✓ (unit + control E2E); ④ top-2 `<documents>` retrieval untouched — ✓ (`app/rag/retriever.py` unmodified vs HEAD); ⑤ suite green, >90% coverage, ruff+pyright clean — ✓; ⑥ no completed-phase behavior change — ✓ (all gates green; commit left to harness per pass rules).

- No defects found; no changes made this pass. Next pending phase: none in `todo/` (96 is the next free number).
This commit is contained in:
2026-09-11 03:42:51 -04:00
parent d4943b4822
commit bcaef800c5
36 changed files with 2836 additions and 43 deletions
+129
View File
@@ -259,6 +259,34 @@ Implements just enough of the aipi surface:
the ``<tools>`` section, so deflected turns never hit it); no
existing E2E question or fixture file contains the phrase, so
every other suite is unaffected.
- user message containing ``read the capped document``
(``READ_CAP_TRIGGER``, phase 95 task 03 — the read cap's dedicated
story suite ``tests/e2e/test_read_truncation_cap.py``) **and** the
system prompt carries the ``<tools>`` section -> the deterministic
SCRIPTED CAPPED-READ flow: the question carries its own tool call
after the colon — ``read the capped document: read source/path`` —
parsed by ``_READ_CAP_CALL_RE`` from the RAW user message (the
target keeps its case), then discriminated statelessly from the
tool results (streaming only):
* request 1 (``tools`` offered, no ``tool``-role result in the
messages yet): the scripted call — ``read`` with the parsed
target (synthetic id ``call_0``);
* a ``tool``-role result is in the messages: the deterministic
ECHO — the answer carries the LAST tool result VERBATIM
(``Here's what the read returned:\n<result>``): a read result
(``"Document <source/path>:…``) lands in the answer with its
FULL content — the first cap chars + ``[…truncated…]`` + the
pinned grep-pointer notice when the cap fired, the plain body
byte-identical to the pre-phase-95 shape when it did not — and
a refusal (the premise broke) lands just as visibly, so the
suite fails loudly on it. The mock is the only E2E lens on the
LLM's context, so the echo is the assertion surface for both
the marker's presence AND its absence.
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
phrases — the phase-71/72/94 ordering convention; the trigger
needs the ``<tools>`` section, so deflected turns never hit it);
verified 2026-09-10: no existing E2E question or fixture file
contains the phrase, so every other suite is unaffected.
- user message containing ``what are the correct llama.cpp
arguments`` (``GREP_TEACH_TRIGGER``, the 2026-09-05 incident —
the harness prior is that grep takes a REGEX; this app's grep is a
@@ -1293,6 +1321,79 @@ def _drill_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
return ("echo", last)
# ---------------------------------------------------------------------------
# Phase 95 (task 03, the read cap's dedicated story suite):
# the deterministic SCRIPTED capped read — see the module docstring
# ---------------------------------------------------------------------------
#: A user message containing this substring (case-insensitive) —
#: combined with the ``<tools>`` section in the system prompt — drives
#: the scripted CAPPED-READ flow (the read cap's story suite,
#: ``tests/e2e/test_read_truncation_cap.py``): the question carries its
#: own tool call after the colon — ``read the capped document: read
#: source/path`` — the mock emits the scripted ``read``, then ECHOES
#: the ENTIRE tool result into its answer (the house scripted-turn lens
#: on the LLM's context: the truncated shape — first cap chars +
#: ``[…truncated…]`` + the pinned grep-pointer notice — or the plain
#: shape, byte-identical to the pre-phase-95 result, lands in the
#: rendered answer, and the suite asserts both directions through it).
#: Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
#: phrases — the phase-71/72/94 ordering convention); verified
#: 2026-09-10: no existing E2E question or fixture file contains the
#: phrase, so every other suite is unaffected.
READ_CAP_TRIGGER = "read the capped document"
#: The scripted call in the read-cap question (case-insensitive — the
#: suite's questions capitalize the trigger's first letter): the verb
#: (``read``) plus the target — a combined ``source/path``, parsed from
#: the RAW user message so the target keeps its case. The target is a
#: ``[a-z0-9_./-]`` run (case-insensitively), so the suite's `` — ``
#: flavor separator (em dash) can never bleed into it (the drill-down
#: convention, ``_DRILL_CALL_RE``).
_READ_CAP_CALL_RE = re.compile(
r"read the capped document:\s*read\s+(?P<arg>[a-z0-9_./-]+)",
re.I,
)
def _read_cap_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
"""Classify a phase-95 scripted read-cap request (see the module
docstring). The question carries the scripted call (``read the
capped document: read source/path``); the step is then discriminated
statelessly from the tool results, like the other marker flows:
* ``("call", target, "call_0")`` — ``tools`` are offered and no
``tool``-role result is in the messages yet: the scripted
``read`` on the parsed target (synthetic id ``call_0``).
* ``("echo", result)`` — a ``tool``-role result is in the messages:
the deterministic ECHO — the answer carries the LAST tool result
VERBATIM (``Here's what the read returned:\n<result>``): a read
result (``"Document <source/path>:…``) lands in the answer with
its full content — the truncation marker + the pinned notice when
the cap fired, the plain body when it did not — and a refusal
(the premise broke) lands just as visibly, so the suite fails
loudly on it.
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
section is missing (deflected turns never carry it), the scripted
call is unparseable, or ``tools`` are not offered and no tool
results are in the messages yet (e.g. ``agent_max_rounds=0``).
"""
user = _user(body)
if READ_CAP_TRIGGER not in user.lower():
return None
if "<tools>" not in _system(body):
return None
match = _READ_CAP_CALL_RE.search(user)
if match is None:
return None
results = _tool_results(body)
if not results:
if not body.get("tools"):
return None
return ("call", match.group("arg"), "call_0")
return ("echo", results[-1])
def long_answer() -> str:
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
a unique final line that must survive the stream untruncated."""
@@ -1985,6 +2086,34 @@ def chat_completions(body: dict[str, Any]) -> Any:
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
# Phase 95 (task 03): the deterministic SCRIPTED capped read
# (the question carries its own call — ``read the capped
# document: read source/path``): the scripted ``read``, then the
# answer that ECHOES the whole tool result (the marker + notice
# when the cap fired, the plain shape when it did not — the
# suite's lens on the LLM's context). Checked BEFORE the plain
# TOOLS_TRIGGER flow (disjoint trigger phrases — the
# phase-71/72/94 ordering convention; the trigger needs the
# ``<tools>`` section, so deflected turns never hit it).
read_cap = _read_cap_flow(body)
if read_cap is not None:
if read_cap[0] == "call":
stream = _tool_call_stream(
"read", {"path": read_cap[1]}, read_cap[2]
)
else: # "echo" — the last tool result verbatim (the lens)
stream = _sse_stream(
_apply_max_tokens(
f"Here's what the read returned:\n{read_cap[1]}",
body.get("max_tokens"),
),
0.0,
)
return StreamingResponse(
stream,
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
flow = _tool_flow(body)
if flow is not None:
if flow[0] == "list":
+861
View File
@@ -0,0 +1,861 @@
"""Phase 95 task 03 E2E (Playwright, mock-only): the read cap's
dedicated story suite — the truncated read, end to end.
The whole TODO.md L5 item (``95_read_truncation_cap`` task 03 is the
story gate): a document over the (lowered) cap is read truncated, the
``tool_result`` frame lands, the Reading line carries the marker, the
LLM's context carried ``[…truncated…]`` + the grep pointer, and the
marker survives save → shared.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real
``turbo`` does whatever it does with the tools, while this story's gate
is the deterministic SCRIPTED capped-read flow in
``tests/e2e/mock_llm.py`` (``READ_CAP_TRIGGER``: user message contains
``read the capped document`` — the question carries its own tool call
after the colon, ``read the capped document: read source/path`` —
**and** the system prompt carries the ``<tools>`` section of the HIGH
prompt). The mock ECHOES the ENTIRE read tool result into its final
answer (the house scripted-turn way of asserting on tool results — the
mock is the only E2E lens on the LLM's context), so both the marker's
presence (the truncated turn) and its absence (the short-document
control) land on the rendered answer.
The app under test boots with the LOWERED cap (the
env-override-for-the-app-under-test pattern — this suite's module app,
as in ``test_local_directory_sources.py`` / ``test_ls_tree_drilldown.py``):
``BOR_READ_MAX_CHARS=1500`` — small enough that the ~3 100-char
fixture document truncates deterministically at 1 500, big enough that
the short-document control (~200 chars) never does.
KB fixture — one host temp dir (``tmp_path_factory``; the app runs on
the same host) registered as a local-directory source (the
``test_local_directory_sources.py`` registration + real-Sync pattern —
registration through the authenticated API, the real in-process
``POST /api/sync`` pipeline; no git anywhere), with THREE documents
whose bodies are token-controlled so the phase-72 ALREADY_IN_CONTEXT
dedupe (the read target is refused when it is a top-2 retrieval seed)
NEVER fires — each scripted read target must actually execute, not be
refused:
* ``anchor-2024.md`` — the retrieval ANCHOR: a digit-bearing name.
Both questions name "anchor 2024", whose joined normalized token
(``anchor2024``) name-hits this document — the name-hit list LEADS
the lexical side, so the anchor is the #1 seed of BOTH turns
(grounding: the name hit's ``fts_hit`` keeps the gate HIGH, the
``<tools>`` section is present, and the anchor is never a read
target);
* ``zz-capped.md`` — the SUBJECT: ~3 100 chars of varied rotation
prose (deterministic, pinned below with its exact length — the
``chars_total`` the assertions use is ``len(CAP_DOC)`` of this very
string, and ``synced_kb`` pins the stored content byte-identical to
it). Its body avoids EVERY token of both questions, so on its own
(turn 1) question it has zero lexical hits and only the common-word
cosine — it is the #3 fused candidate, NOT a seed;
* ``aa-short.md`` — the CONTROL: ~200 chars, under the cap.
The turn-specific flavor words pick the #2 seed: turn 1's question
carries ``note, long form`` (planted in the SHORT doc's body) and turn
2's carries ``quick pass`` (planted in the CAPPED doc's body), so each
turn's NON-target document wins the second seed slot on its own
question and the target stays #3/#6. ``synced_kb`` pins this design
with the app's real hybrid retrieval (``_assert_target_not_seed`` — a
fixture-text regression that makes a target a seed fails at setup with
a clear message, not at the wire assertions).
Test → story mapping (Playwright Mapping Rule; the story is the owner
TODO item — one Playwright file per story, A16):
1. ``test_truncated_read_frame_order_live_marker_and_llm_notice`` —
the wire carries the ``tool`` frame THEN exactly one
``tool_result`` frame (the pinned counts) THEN the answer's
``delta``/``done`` frames; the live Reading line carries the pinned
``(truncated — showing 1500 of N chars)`` marker; the mock's echo
proves the LLM context carried ``[…truncated…]`` AND the pinned
``TRUNCATED — … Use grep …`` notice (and the head sentinel reached
the model while the past-the-cap tail sentinel did NOT).
2. ``test_truncated_read_save_and_shared_fidelity`` — the auto-saved
row's (phase 55) brain message ``tools`` record carries
``truncated: true`` + the counts (saved-chats API); the shared
page (``/shared/<token>``) renders the SAME marker on the Reading
line (the phase-50 restore contract, pixel-identical).
3. ``test_short_read_control_no_frame_no_marker`` — the under-cap read
executes (its ``tool`` frame lands) but streams NO ``tool_result``
frame, the Reading line carries NO marker, the echo shows no
``[…truncated…]`` / ``TRUNCATED —``, and the saved ``tools``
record is the plain pre-truncation shape.
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import httpx
import pytest
from playwright.sync_api import Browser, BrowserContext, Locator, Page, expect
from sqlalchemy import select, text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import Document
from app.rag.agent import READ_TRUNCATION_NOTICE
from app.rag.retriever import TRUNCATION_MARKER
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
from e2e.mock_llm import embed_text
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_READCAP", "8137"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
#: The lowered read cap the app under test boots with
# (``BOR_READ_MAX_CHARS``): the subject of the suite.
READ_CAP = 1500
SOURCE = "capkb" # the local directory's basename = the source name
ANCHOR_REL = "anchor-2024.md"
CAPPED_REL = "zz-capped.md"
SHORT_REL = "aa-short.md"
CAPPED_SP = f"{SOURCE}/{CAPPED_REL}" # the combined read identity
SHORT_SP = f"{SOURCE}/{SHORT_REL}"
# --------------------------------------------------------------------------
# Fixture documents (deterministic, token-controlled — see the module
# docstring for the retrieval-anchor design)
# --------------------------------------------------------------------------
ANCHOR_DOC = (
"# Anchor 2024\n\n"
"The anchor 2024 record keeps the backup rotation steady: the quiet "
"window, the mirror pool, and the retention label all stay pinned to "
"this year's plan.\n"
)
HEAD_SENTINEL = "CAPDOC-HEAD-9f2a" # inside the first 1 500 chars
TAIL_SENTINEL = "CAPDOC-TAIL-7c3b" # PAST the cap (the rest is NOT shown)
SHORT_SENTINEL = "CAPSHORT-5d1e" # the whole short doc is under the cap
def _cap_doc() -> str:
"""The ~3 100-char subject document: a repeated-but-varied paragraph
block (deterministic), the head sentinel near the top, the planted
``quick pass`` sentence (the turn-2 flavor words — see the module
docstring) and the tail sentinel at the very end. The body avoids
every token of BOTH questions (``read``, ``capped``, ``document``,
``capkb``, ``zz``, ``md``, ``anchor``, ``2024``, ``note``,
``long``, ``form``, ``aa``, ``short``, ``quick``, ``pass``,
``wire``, ``save``, ``control``, ``check``) — the target must have
zero lexical hits on its own turn so the dedupe cannot refuse the
read."""
topics = [
"vault", "mirror", "raid", "pool", "drive",
"chain", "slot", "cycle", "guard", "probe",
]
paras: list[str] = []
i = 0
while True:
t = topics[i % len(topics)]
paras.append(
f"{t.title()} item {i:02d}: at 02:00 the quiet window opens and "
f"the archive job copies the {t} image to the mirror pool, then "
f"the manifest audit verifies the snapshot chain for slot {i:02d}, "
"keeping the retention label clean and the rotation order exact "
"for the whole night cycle."
)
i += 1
if len("\n\n".join(paras)) > 2750:
break
head = (
"# Rotation archive\n\n"
f"{HEAD_SENTINEL}\n\n"
"A quick pass over the rotation confirms the pass order of the "
"guard probe before the quiet window closes, and the audit log "
"records the result.\n\n"
)
tail = f"\n\n{TAIL_SENTINEL}\n"
return head + "\n\n".join(paras) + tail
CAP_DOC = _cap_doc()
CAP_DOC_TOTAL = len(CAP_DOC)
SHORT_DOC = (
"# Rotation plan\n\n"
"This note keeps the long form of the rotation plan in one place: the "
"quiet window, the mirror pool, the retention label, and the guard "
"probe order for every weekly run.\n\n"
f"{SHORT_SENTINEL}\n"
)
SHORT_DOC_TOTAL = len(SHORT_DOC)
# The suite's length contract (the task's pinned-count assumption —
# ``chars_total`` is asserted against ``len(CAP_DOC)`` of the very
# string written to the fixture, and ``synced_kb`` pins the stored
# content byte-identical to it): the subject is OVER the cap with the
# tail sentinel past the cut; the control is UNDER the cap.
assert CAP_DOC_TOTAL > READ_CAP, CAP_DOC_TOTAL
assert CAP_DOC.index(HEAD_SENTINEL) < READ_CAP < CAP_DOC.index(TAIL_SENTINEL)
assert CAP_DOC.count(HEAD_SENTINEL) == 1 and CAP_DOC.count(TAIL_SENTINEL) == 1
assert SHORT_DOC_TOTAL < READ_CAP, SHORT_DOC_TOTAL
assert SHORT_DOC.count(SHORT_SENTINEL) == 1
# The pinned marker copy (task 02 — the app.js/shared.js template, plain
# integers, no thousands separators) and the LLM-side notice (the
# app.rag.agent constant, formatted for this fixture's counts).
MARKER = f" (truncated — showing {READ_CAP} of {CAP_DOC_TOTAL} chars)"
NOTICE = READ_TRUNCATION_NOTICE.format(shown=READ_CAP, total=CAP_DOC_TOTAL)
# The scripted turns (the mock's ``READ_CAP_TRIGGER`` questions — each
# carries its own tool call after the colon; the turn-specific flavor
# words are the #2-seed selectors, see the module docstring).
Q_WIRE = (
f"Read the capped document: read {CAPPED_SP} — "
"anchor 2024 note, long form, wire check"
)
Q_SAVE = (
f"Read the capped document: read {CAPPED_SP} — "
"anchor 2024 note, long form, save check"
)
Q_CONTROL = (
f"Read the capped document: read {SHORT_SP} — "
"anchor 2024, quick pass, control check"
)
#: The invalid/unknown share token shape (test_share_chat.py's
#: ``SHARE_URL_RE`` convention).
SHARE_URL_RE = re.compile(r"^/shared/[0-9a-f-]{36}$")
# --------------------------------------------------------------------------
# Fixtures
# --------------------------------------------------------------------------
@pytest.fixture(scope="module")
def cap_dirs(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The story's local-directory source: one host temp dir (the app
server runs on the same host, so the path is visible to it) holding
the three token-controlled fixture documents. The directory's
basename is the source name (``kind=local``, phase 38)."""
root = tmp_path_factory.mktemp("bor_read_cap") / SOURCE
root.mkdir()
for rel, doc in (
(ANCHOR_REL, ANCHOR_DOC),
(CAPPED_REL, CAP_DOC),
(SHORT_REL, SHORT_DOC),
):
(root / rel).write_text(doc, encoding="utf-8")
assert (root / rel).read_text(encoding="utf-8") == doc
assert not (root / ".git").exists() # the local-kind story: NOT git
return root
@pytest.fixture(scope="module")
def app_server(mock_llm: int, cap_dirs: Path) -> Iterator[str]:
"""The real app under test — per-module env (the conftest pattern,
cf. ``test_ls_tree_drilldown.py``): NO ``BOR_GIT_SOURCES`` (the env
fallback is git-only — the source here is a DB-registered local
directory), the mock LLM, the mock-calibrated threshold, the
leak-guarded code defaults — and the SUBJECT: the lowered read cap
``BOR_READ_MAX_CHARS=1500`` (the env-override-for-the-app-under-
test pattern; the default 128 000 is unit-pinned in
``tests/unit/test_config.py``). The session app is never started in
this isolated run, so no port clash."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern) — the retrieval-anchor
# design keeps every scripted turn grounded regardless.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
# Phase 67: instant retry waits + the code-default budget (the
# conftest leak-guard pattern).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the local
# directory this suite registers (a leftover env git list would
# pollute the KB the scripted reads run against).
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under test.
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
# THE SUBJECT: the lowered read cap (task 03 — the app under test
# boots with BOR_READ_MAX_CHARS=1500; the production default of
# 128 000 stays pinned in tests/unit/test_config.py).
env["BOR_READ_MAX_CHARS"] = str(READ_CAP)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
share one Postgres, so a leftover git_sources row or document would
pollute the retrieval the scripted turns run against (the anchor
design's margins are pinned against EXACTLY these three
documents)."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, folder_summaries"
)
)
db.commit()
def _assert_target_not_seed(question: str, target_rel: str) -> None:
"""Pin the retrieval-anchor design (see the module docstring) with
the app's REAL hybrid retrieval over the mock's embeddings
(deterministic): the scripted read target must NOT be a top-2 seed
for its own question — the phase-72 ALREADY_IN_CONTEXT dedupe would
refuse the read and the cap would never fire (the turn would echo
the refusal instead). A fixture-text regression that breaks this
fails here, at setup, with a clear message."""
from app.rag.retriever import retrieve, select_documents
with SessionLocal() as db:
seeds = select_documents(retrieve(db, question, embed_text(question)))
paths = [f"{d.source}/{d.path}" for d in seeds]
assert f"{SOURCE}/{target_rel}" not in paths, (
f"the read target {SOURCE}/{target_rel} is a top-2 seed for its own "
f"question — the ALREADY_IN_CONTEXT dedupe would refuse the read and "
f"the cap would never fire (seeds: {paths})"
)
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
"""Poll the (cookie-authenticated) status endpoint until the run
reaches a terminal state (the test_ls_tree_drilldown pattern, over
plain httpx — this fixture has no browser page yet)."""
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
if body["state"] in ("success", "failed"):
return body
time.sleep(0.5)
raise AssertionError(f"sync did not reach a terminal state: {body}")
@pytest.fixture(scope="module")
def synced_kb(app_server: str, cap_dirs: Path) -> None:
"""The story's precondition: the one-source KB synced under the
deterministic mock. Registers the temp directory through the
authenticated API (the ``test_local_directory_sources.py`` pattern),
runs the REAL in-process sync (``POST /api/sync`` — walk → chunk →
embed → overview → version bump), pins the stored content
byte-identical to the fixture strings (the ``chars_total``
assumption), and pins the retrieval-anchor design for all three
scripted questions (the dedupe never fires)."""
_truncate_all()
with httpx.Client(base_url=app_server, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post(
"/api/git-sources", json={"kind": "local", "path": str(cap_dirs)}
)
assert r.status_code == 201, r.text
r = client.post("/api/sync")
assert r.status_code == 202, r.text
body = _wait_sync_done_http(client)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == 3, detail
assert detail["pruned"] == 0, detail
# The import stored the fixture strings BYTE-IDENTICALLY — the
# ``chars_total`` every assertion below uses is ``len()`` of the
# very string written to the directory (the task's pinned-count
# assumption).
with SessionLocal() as db:
for rel, expected in (
(ANCHOR_REL, ANCHOR_DOC),
(CAPPED_REL, CAP_DOC),
(SHORT_REL, SHORT_DOC),
):
stored = db.scalar(
select(Document.content).where(
Document.source == SOURCE, Document.path == rel
)
)
assert stored == expected, f"stored content drifted for {rel}"
# The retrieval-anchor design (the module docstring): every scripted
# read target stays OUT of its own question's top-2 seeds.
_assert_target_not_seed(Q_WIRE, CAPPED_REL)
_assert_target_not_seed(Q_SAVE, CAPPED_REL)
_assert_target_not_seed(Q_CONTROL, SHORT_REL)
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
"""Per-test query_log isolation (the KB itself is module-scoped —
the scripted turns never change it, so the registry persists
across the tests of this module)."""
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
yield
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
# --------------------------------------------------------------------------
# Page helpers (the test_ls_tree_drilldown / test_share_chat house
# patterns)
# --------------------------------------------------------------------------
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
#: (a response clone read in the background) — wire-level assertions
#: for the ``tool`` / ``tool_result`` frames, independent of the UI
#: rendering.
SSE_HOOK = """
() => {
if (window.__sseInstalled) return;
window.__sseInstalled = true;
window.__sseFrames = [];
const origFetch = window.fetch;
window.fetch = async function (...args) {
const res = await origFetch.apply(this, args);
try {
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
if (url.includes('/api/chat')) {
res.clone().text().then((bodyText) => {
for (const block of bodyText.split('\\n\\n')) {
const line = block.trim();
if (line.startsWith('data: ')) {
window.__sseFrames.push(line.slice(6));
}
}
});
}
} catch (e) { /* non-clonable responses: ignored */ }
return res;
};
}
"""
def _install_page_hooks(page: Page) -> None:
page.evaluate(SSE_HOOK)
def _frames(page: Page) -> list[dict]:
"""The SSE frames captured since the last submit (``_submit``
clears the buffer), once the hook's background read settles."""
deadline = time.monotonic() + 30.0
while True:
raw = page.evaluate("() => window.__sseFrames || []")
parsed = [json.loads(line) for line in raw if line]
if any(f.get("type") == "done" for f in parsed):
return parsed
if time.monotonic() > deadline:
raise AssertionError(
f"SSE hook captured no `done` frame (frames so far: "
f"{len(parsed)}) — hook install failed?"
)
time.sleep(0.05)
def _tool_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool"]
def _result_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool_result"]
def _submit(page: Page, question: str) -> None:
page.evaluate("window.__sseFrames = []")
page.fill("#message-input", question)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
def _wait_settled(page: Page) -> None:
"""The turn is complete: answer text in the bubble, button recovered
(the phase-48 settle wait, the test_agent_document_tools helper)."""
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
def _last_brain(page: Page) -> Locator:
return page.locator(".msg.brain").last
def _auto_title(question: str) -> str:
"""The phase-50 auto-title convention: the first question,
whitespace-collapsed, capped at 120 chars."""
return " ".join(question.split())[:120]
def _admin_cookies(page: Page) -> dict[str, str]:
"""The signed session cookies the browser holds after a form login —
used to call the admin API with plain httpx (the test's API side
sees exactly what the signed-in browser sees)."""
return {
c["name"]: c["value"]
for c in page.context.cookies()
if "name" in c and "value" in c
}
def _chats(app_url: str, cookies: dict[str, str]) -> list[dict[str, Any]]:
r = httpx.get(f"{app_url}/api/chats", timeout=10, cookies=cookies)
assert r.status_code == 200
return r.json()["chats"]
def _find_row(
rows: list[dict[str, Any]], title: str
) -> dict[str, Any] | None:
return next((c for c in rows if c["title"] == title), None)
def _get_chat(app_url: str, cookies: dict[str, str], chat_id: str) -> dict[str, Any]:
r = httpx.get(f"{app_url}/api/chats/{chat_id}", timeout=10, cookies=cookies)
assert r.status_code == 200, r.text
return r.json()
def _delete_chat(app_url: str, cookies: dict[str, str], chat_id: str) -> None:
"""Best-effort row cleanup (a 404 — already deleted — is fine)."""
httpx.delete(f"{app_url}/api/chats/{chat_id}", timeout=10, cookies=cookies)
def _wait_saved_row(
app_url: str,
cookies: dict[str, str],
title: str,
messages: int = 2,
) -> dict[str, Any]:
"""Wait for the auto-saved row (phase 55: auto-saves are SILENT —
A2 — so there is no status line to wait on). The upsert is
fire-and-forget from the UI's point of view, so poll the admin
list until the row with the conversation's auto-title appears with
the expected message count."""
deadline = time.monotonic() + 15
last: dict[str, Any] | None = None
while time.monotonic() < deadline:
last = _find_row(_chats(app_url, cookies), title)
if last is not None and last["message_count"] >= messages:
return last
time.sleep(0.2)
raise AssertionError(f"no auto-saved row for {title!r} (last: {last!r})")
def _grant_clipboard(page: Page, app_url: str) -> None:
"""Grant the async-clipboard permissions on the admin context
(test_share_chat.py — the assertion branches on the API's
availability, so a non-secure origin still passes through the
fallback branch deterministically)."""
page.context.grant_permissions(
["clipboard-read", "clipboard-write"], origin=app_url
)
def _click_share_and_assert_status(page: Page, app_url: str) -> None:
"""Press the chat page's Share pill and pin the owner-locked
outcome on the live region (test_share_chat.py's helper, verbatim
contract): "Share link copied." when ``navigator.clipboard`` is
available in the context, else the inline fallback link field
carrying the ``/shared/<uuid>`` URL."""
page.locator("#share-chat-btn").click()
if page.evaluate("() => !!navigator.clipboard"):
expect(page.locator("#send-status")).to_have_text(
"Share link copied.", timeout=15_000
)
expect(page.locator(".share-link-fallback")).to_have_count(0)
else:
expect(page.locator("#send-status")).to_have_text(
"Share link ready — copy it from the field.", timeout=15_000
)
field = page.locator(".share-link-fallback")
expect(field).to_be_visible()
href = field.get_attribute("href")
assert href is not None
assert href.startswith(app_url)
assert SHARE_URL_RE.fullmatch(href.removeprefix(app_url))
# --------------------------------------------------------------------------
# 1. The truncated read: the wire's frame order, the live marker, the
# LLM-visible notice (via the mock's echo)
# --------------------------------------------------------------------------
def test_truncated_read_frame_order_live_marker_and_llm_notice(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, Q_WIRE)
_wait_settled(page)
# --- the wire: tool → tool_result → delta… (exactly one, counts) --
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "read", "argument": CAPPED_SP}
], _tool_frames(frames)
# EXACTLY ONE tool_result frame — the pinned counts (the char-based
# cap: showing the first READ_CAP of the fixture's true length).
assert _result_frames(frames) == [
{
"type": "tool_result",
"name": "read",
"argument": CAPPED_SP,
"truncated": True,
"chars_shown": READ_CAP,
"chars_total": CAP_DOC_TOTAL,
}
], _result_frames(frames)
# Frame order: the marker lands AFTER the call is shown and BEFORE
# the answer's first delta (the phase-37/48 "calling tool" timing
# is untouched — the beat between the two is the truncation).
i_tool = next(i for i, f in enumerate(frames) if f.get("type") == "tool")
i_result = next(
i for i, f in enumerate(frames) if f.get("type") == "tool_result"
)
i_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert i_tool < i_result < i_delta, (i_tool, i_result, i_delta)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The read document is the turn's cited source even though it was
# NOT a retrieval seed (the anchor design) — retrieval + agent-read,
# deduped (the grounded-turn record).
assert any(
s["path"] == CAPPED_REL and s["source"] == SOURCE for s in done["sources"]
), done["sources"]
# --- the live DOM: the Reading line carries the pinned marker ----
line = _last_brain(page).locator(".tool-call")
expect(line).to_have_count(1)
expect(line).to_contain_text(f"Reading {CAPPED_SP}")
expect(line.locator("code")).to_have_text(CAPPED_SP)
expect(line.locator("span.truncated-note")).to_have_text(MARKER)
# --- the mock's echo: the LLM's context carried the marker AND ---
# --- the grep-pointer notice (the house scripted-turn lens) ------
bubble = _last_brain(page).locator(".bubble")
text = bubble.text_content() or ""
assert TRUNCATION_MARKER in text, text
assert NOTICE in text, text
# The first cap chars reached the model…
assert HEAD_SENTINEL in text, text
# …and the past-the-cap tail did NOT (the cut is real, not cosmetic).
assert TAIL_SENTINEL not in text, text
# --------------------------------------------------------------------------
# 2. Save → shared: the stored tools record carries the truncation, and
# the shared page renders the SAME marker (pixel-identical)
# --------------------------------------------------------------------------
def test_truncated_read_save_and_shared_fidelity(
page: Page,
browser: Browser,
app_url: str,
synced_kb: None,
db_ready: None,
) -> None:
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_submit(page, Q_SAVE)
_wait_settled(page)
# The live marker landed (regression through the save path — the
# tool_result frame stamped the toolAcc entry the save carries).
line = _last_brain(page).locator(".tool-call")
expect(line).to_have_count(1)
expect(line.locator("span.truncated-note")).to_have_text(MARKER)
cookies = _admin_cookies(page)
title = _auto_title(Q_SAVE)
row = _wait_saved_row(app_url, cookies, title)
chat_id: str = row["id"]
anon_ctx: BrowserContext | None = None
try:
# The saved message's ``tools`` record carries the truncation —
# ``truncated: true`` + the pinned counts (the saved-chats API;
# the model_dump round-trip fills the non-truncated defaults,
# so the full five-key shape is asserted).
detail = _get_chat(app_url, cookies, chat_id)
brain_msgs = [m for m in detail["messages"] if m["who"] == "brain"]
assert len(brain_msgs) == 1, detail["messages"]
assert brain_msgs[0]["tools"] == [
{
"name": "read",
"argument": CAPPED_SP,
"truncated": True,
"chars_shown": READ_CAP,
"chars_total": CAP_DOC_TOTAL,
}
], brain_msgs[0]["tools"]
# Share the auto-saved row (phase 55: settled = already saved —
# the Share click shares the linked row) and read back the link.
_grant_clipboard(page, app_url)
_click_share_and_assert_status(page, app_url)
row = _find_row(_chats(app_url, cookies), title)
assert row is not None and row.get("share_url"), row
share_url: str = row["share_url"]
assert SHARE_URL_RE.fullmatch(share_url)
# A FRESH context (no cookies — the guest's only credential is
# the token in the URL): the shared page renders the SAME
# marker on the Reading line (shared.js renders it from the
# stored record — the phase-50 restore contract).
anon_ctx = browser.new_context()
anon = anon_ctx.new_page()
anon.set_default_timeout(30_000)
anon.goto(app_url + share_url)
expect(anon.locator("#shared-title")).to_have_text(title)
anon_line = anon.locator(".msg.brain .tool-call")
expect(anon_line).to_have_count(1)
expect(anon_line).to_contain_text(f"Reading {CAPPED_SP}")
expect(anon_line.locator("code")).to_have_text(CAPPED_SP)
expect(anon_line.locator("span.truncated-note")).to_have_text(MARKER)
# The shared answer still renders the truncation the LLM was
# told about (same stored text, same renderer).
anon_text = anon.locator(".msg.brain .bubble").first.text_content() or ""
assert TRUNCATION_MARKER in anon_text, anon_text
assert NOTICE in anon_text, anon_text
assert HEAD_SENTINEL in anon_text, anon_text
assert TAIL_SENTINEL not in anon_text, anon_text
anon_ctx.close()
anon_ctx = None
finally:
if anon_ctx is not None:
anon_ctx.close()
_delete_chat(app_url, cookies, chat_id)
# --------------------------------------------------------------------------
# 3. The control: the under-cap read executes, streams NO tool_result
# frame, shows NO marker, and saves the plain tools record
# --------------------------------------------------------------------------
def test_short_read_control_no_frame_no_marker(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, Q_CONTROL)
_wait_settled(page)
# The read EXECUTED (the tool frame landed)…
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "read", "argument": SHORT_SP}
], _tool_frames(frames)
# …but a non-truncated read streams NO tool_result frame (one frame
# = one noteworthy event — the phase-95 additive contract).
assert _result_frames(frames) == [], _result_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
assert any(
s["path"] == SHORT_REL and s["source"] == SOURCE for s in done["sources"]
), done["sources"]
# No marker on the live Reading line.
line = _last_brain(page).locator(".tool-call")
expect(line).to_have_count(1)
expect(line).to_contain_text(f"Reading {SHORT_SP}")
expect(line.locator("code")).to_have_text(SHORT_SP)
expect(line.locator("span.truncated-note")).to_have_count(0)
# The mock's echo: the PLAIN read shape — the whole short document
# (it is under the cap), no marker, no notice.
bubble = _last_brain(page).locator(".bubble")
text = bubble.text_content() or ""
assert SHORT_SENTINEL in text, text
assert TRUNCATION_MARKER not in text, text
assert "TRUNCATED —" not in text, text
# The saved ``tools`` record is the plain pre-truncation shape
# (``truncated`` default False, the counts null — the
# pre-phase-95 shape validates and renders unchanged).
cookies = _admin_cookies(page)
title = _auto_title(Q_CONTROL)
row = _wait_saved_row(app_url, cookies, title)
try:
detail = _get_chat(app_url, cookies, row["id"])
brain_msgs = [m for m in detail["messages"] if m["who"] == "brain"]
assert len(brain_msgs) == 1, detail["messages"]
assert brain_msgs[0]["tools"] == [
{
"name": "read",
"argument": SHORT_SP,
"truncated": False,
"chars_shown": None,
"chars_total": None,
}
], brain_msgs[0]["tools"]
finally:
_delete_chat(app_url, cookies, row["id"])
+9 -3
View File
@@ -42,7 +42,13 @@ from app.config import Settings
from app.models import Document, FolderSummary, GitSource
from app.rag import agent
from app.rag.agent import AGENT_TOOLS, AgentHolder, run_agent
from app.rag.llm import LLMClient, RetryPiece, StreamPiece, ToolCallPiece
from app.rag.llm import (
LLMClient,
RetryPiece,
StreamPiece,
ToolCallPiece,
ToolResultPiece,
)
if TYPE_CHECKING:
from app.rag.scaffolding import ScaffoldingFilter
@@ -254,8 +260,8 @@ def _run_call(
async def _consume(
llm: LLMClient, db: Session, holder: AgentHolder
) -> list[StreamPiece | ToolCallPiece | RetryPiece]:
out: list[StreamPiece | ToolCallPiece | RetryPiece] = []
) -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
async for piece in run_agent(
llm,
db,
+353 -3
View File
@@ -16,6 +16,7 @@ import json
import logging
import math
import re
import uuid
from collections.abc import Iterator
from pathlib import Path
from typing import TYPE_CHECKING, Any, cast
@@ -24,15 +25,18 @@ import pytest
from fastapi.testclient import TestClient
from pydantic import ValidationError
from sqlalchemy import delete, func, select, text
from sqlalchemy.orm import Session
from app.api import chat as chat_api
from app.config import Settings, get_settings
from app.main import app as fastapi_app
from app.models import Chunk, GitSource, QueryLog
from app.models import Chunk, Document, GitSource, QueryLog
from app.rag import agent
from app.rag.agent import AGENT_TOOLS
from app.rag.agent import AGENT_TOOLS, READ_TRUNCATION_NOTICE
from app.rag.importer import import_sources
from app.rag.llm import EmbeddingError, LLMError, StreamPiece, ToolCallPiece
from app.rag.llm import EmbeddingError, LLMError, StreamPiece, ToolCallPiece, ToolResultPiece
from app.rag.prompts import build_high_prompt
from app.rag.retriever import TRUNCATION_MARKER
from app.schemas import ChatDoneEvent, SourceRef
from tests.conftest import ADMIN_PASSWORD
@@ -610,6 +614,352 @@ def test_chat_query_log_failure_still_sends_done(client, db, seeded_kb: FakeRagL
# ---------- phase 37: agent document tools on grounded turns ----------
async def _collect_run_agent(
llm: FakeRagLLM,
db: Session,
system_prompt: str,
settings: Settings,
seed_docs: list[Document],
) -> tuple[list[Any], agent.AgentHolder]:
"""Consume one ``run_agent`` turn, returning the yielded pieces (in
order) and the holder. Phase 95 (task 01): the direct agent-loop
drive — the agent-loop yield order on the real prompt path, the
complement of the endpoint-level ``tool_result`` SSE tests below
(task 02)."""
holder = agent.AgentHolder()
pieces: list[Any] = []
async for piece in agent.run_agent(
llm, # pyright: ignore[reportArgumentType] # duck-typed LLMClient
db,
system_prompt=system_prompt,
user_message=QUESTION,
seed_docs=seed_docs,
settings=settings,
holder=holder,
):
pieces.append(piece)
return pieces, holder
def test_read_cap_truncates_and_yields_tool_result_on_real_prompt_path(
db,
) -> None:
"""Phase 95 (task 01): on the REAL prompt path (a real Postgres
document + the real ``build_high_prompt``), a ``read`` of a document
LONGER than ``settings.read_max_chars`` truncates the result the model
sees — first ``cap`` chars + the shared :data:`TRUNCATION_MARKER` + the
pinned grep-pointer notice — and ``run_agent`` yields exactly ONE
``ToolResultPiece``: AFTER the read's ``tool`` frame (the matching
``ToolCallPiece``) and BEFORE the next model round. The endpoint-level
``tool_result`` SSE frame is asserted separately below (task 02); this
pins the agent-loop yield order on the real prompt path."""
cap = 100
content = "K" * (cap + 40) # 40 chars over the cap
doc = Document(
id=uuid.uuid4(),
source="docs",
path="big.md",
full_path="/tmp/big.md",
title="Big Doc",
content=content,
content_hash="1" * 64,
)
db.add(doc)
db.commit()
try:
# The real prompt path: the actual HIGH prompt for the one doc.
system_prompt = build_high_prompt([doc])
settings = Settings(_env_file=None, read_max_chars=cap) # pyright: ignore[reportCallIssue]
scripted = FakeRagLLM(
tool_script=[
[
ToolCallPiece(
id="call_1", name="read", arguments={"path": "docs/big.md"}
)
]
# the answer request (tools still offered, script
# exhausted) falls back to the thinking + answer stream
]
)
pieces, holder = asyncio.run(
_collect_run_agent(scripted, db, system_prompt, settings, seed_docs=[])
)
# The model's context carried the truncated read — the first cap
# chars, then the shared marker + the pinned grep-pointer notice
# (so a downstream grep is the model's path to the rest). The
# fake records the (mutated-in-place) messages list, so the same
# tool message is aliased across requests — they all carry the
# same content; take the last.
tool_msgs = [
m for r in scripted.seen_messages for m in r if m.get("role") == "tool"
]
assert tool_msgs, "the executed read must be appended as a tool message"
body = tool_msgs[-1]["content"]
assert body.startswith("Document docs/big.md:\n" + content[:cap])
assert TRUNCATION_MARKER in body
assert (
READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) in body
)
# The yield order: the read's ToolCallPiece, then the ONE
# ToolResultPiece, then the next round's answer content.
kinds: list[str] = []
for p in pieces:
if isinstance(p, ToolCallPiece):
kinds.append("toolcall")
elif isinstance(p, ToolResultPiece):
kinds.append("toolresult")
elif isinstance(p, StreamPiece):
kinds.append(p.kind)
assert kinds.count("toolresult") == 1
assert kinds.index("toolcall") < kinds.index("toolresult")
assert kinds.index("toolresult") < kinds.index("content")
# The piece carries (argument, shown, total) — the raw
# source/path the model passed (what the tool frame carries), the
# cap kept, the true length.
(result_piece,) = [p for p in pieces if isinstance(p, ToolResultPiece)]
assert result_piece.name == "read"
assert result_piece.argument == "docs/big.md"
assert result_piece.truncated is True
assert result_piece.chars_shown == cap
assert result_piece.chars_total == len(content)
# Holder accounting: a truncated read is still a SUCCESSFUL call
# (counted + added to context); the tuple is the signal only.
assert holder.tool_calls == 1
assert holder.read_docs == [doc]
assert holder.read_truncations == [("docs/big.md", cap, len(content))]
finally:
db.delete(doc)
db.commit()
def test_read_at_or_under_cap_yields_no_tool_result_on_real_prompt_path(
db,
) -> None:
"""Phase 95 (task 01): the complement — a ``read`` of a document at or
under the cap on the real prompt path is byte-identical to the
pre-phase-95 agent loop: NO ``ToolResultPiece``, no holder entry, no
marker in the model's context."""
cap = 100
content = "K" * cap # exactly at the cap → fits, not truncated
doc = Document(
id=uuid.uuid4(),
source="docs",
path="fits.md",
full_path="/tmp/fits.md",
title="Fits Doc",
content=content,
content_hash="2" * 64,
)
db.add(doc)
db.commit()
try:
system_prompt = build_high_prompt([doc])
settings = Settings(_env_file=None, read_max_chars=cap) # pyright: ignore[reportCallIssue]
scripted = FakeRagLLM(
tool_script=[
[
ToolCallPiece(
id="call_1", name="read", arguments={"path": "docs/fits.md"}
)
]
]
)
pieces, holder = asyncio.run(
_collect_run_agent(scripted, db, system_prompt, settings, seed_docs=[])
)
# No ToolResultPiece, no holder entry.
assert not any(isinstance(p, ToolResultPiece) for p in pieces)
assert holder.read_truncations == []
# The model's context is the whole document, byte-identical to
# the pre-phase-95 read result (no marker, no notice). (The fake
# aliases the mutated messages list, so take the last tool msg.)
tool_msgs = [
m for r in scripted.seen_messages for m in r if m.get("role") == "tool"
]
assert tool_msgs, "the executed read must be appended as a tool message"
assert tool_msgs[-1]["content"] == "Document docs/fits.md:\n" + content
assert TRUNCATION_MARKER not in tool_msgs[-1]["content"]
# Still a successful read.
assert holder.tool_calls == 1
assert holder.read_docs == [doc]
finally:
db.delete(doc)
db.commit()
def _insert_big_doc(db, content: str) -> Document:
"""One bare ``documents`` row (no chunks — the ``read`` lookup is a
(source, path) identity match, not a retrieval) for the SSE-level
read-cap tests: a document the model can only reach through the
``read`` tool."""
doc = Document(
id=uuid.uuid4(),
source="docs",
path="big-read.md",
full_path="/tmp/big-read.md",
title="Big Read Doc",
content=content,
content_hash="3" * 64,
)
db.add(doc)
db.commit()
return doc
def test_truncated_read_streams_tool_result_frame_after_tool_frame(
client,
db,
seeded_kb: FakeRagLLM,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 95 (task 02, the A15 extension): a grounded turn whose
scripted ``read`` hits a document LONGER than ``read_max_chars``
(the cap lowered via the settings override — the task 01
``Settings(_env_file=None, read_max_chars=…)`` pattern) streams the
``tool`` → ``tool_result`` → ``delta…`` → ``done`` sequence: EXACTLY
ONE ``tool_result`` frame, AFTER the matching ``tool`` frame (the
line is already on screen) and BEFORE the next round's first frame,
with the right shape and counts (``chars_shown`` = the cap,
``chars_total`` = the true length). The model's context carried the
truncated read (marker + pinned grep-pointer notice); the read is
still cited (a truncated read is a successful call)."""
cap = 100
content = "K" * (cap + 150)
doc = _insert_big_doc(db, content)
live = get_settings()
monkeypatch.setattr(
chat_api,
"get_settings",
lambda: Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
relevance_threshold=live.relevance_threshold,
read_max_chars=cap,
),
)
scripted = FakeRagLLM(
tool_script=[
[
ToolCallPiece(
id="call_1", name="read", arguments={"path": "docs/big-read.md"}
)
]
# the answer request still carries the tools (1 round < the
# default cap of 10); the script is exhausted, so the fake
# falls back to the thinking + answer stream
]
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
db.delete(doc)
db.commit()
types = [f["type"] for f in frames]
assert "error" not in types
tool_i = types.index("tool")
tool_result_i = types.index("tool_result")
# Exactly one tool_result frame…
assert types.count("tool_result") == 1
# …AFTER the matching tool frame and BEFORE the next model round's
# first frame (the answer's deltas): tool → tool_result → delta…
assert tool_i + 1 == tool_result_i
assert tool_result_i < min(i for i, t in enumerate(types) if t == "delta")
# The frame's exact shape: the additive seventh event type carries
# the name/argument of the matching tool frame + the counts.
frame = frames[tool_result_i]
assert set(frame) == {
"type",
"name",
"argument",
"truncated",
"chars_shown",
"chars_total",
}
assert frame["name"] == frames[tool_i]["name"] == "read"
assert frame["argument"] == frames[tool_i]["argument"] == "docs/big-read.md"
assert frame["truncated"] is True
assert frame["chars_shown"] == cap # the cap kept
assert frame["chars_total"] == len(content) # the true length
# The LLM's context carried the honest truncation: first cap chars +
# the shared marker + the pinned grep-pointer notice (the fake
# aliases the mutated messages list — take the last tool msg).
tool_msgs = [
m for r in scripted.seen_messages for m in r if m.get("role") == "tool"
]
assert tool_msgs
body = tool_msgs[-1]["content"]
assert body.startswith(f"Document docs/big-read.md:\n{content[:cap]}")
assert TRUNCATION_MARKER in body
assert READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) in body
# The truncated read is still a SUCCESSFUL call — cited in done.
done = frames[-1]
assert done["type"] == "done" and done["deflected"] is False
assert ("docs", "big-read.md") in [(s["source"], s["path"]) for s in done["sources"]]
assert ("docs", "homelab/kubernetes.md") in [
(s["source"], s["path"]) for s in done["sources"]
]
def test_untruncated_read_streams_no_tool_result_frame(
client,
db,
seeded_kb: FakeRagLLM,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 95 (task 02): the complement at the SSE level — a ``read``
of a document AT OR UNDER the cap (the same long document, cap
raised past its true length) streams NO ``tool_result`` frame (one
frame = one noteworthy event; the six pre-existing event types are
byte-identical), the ``tool`` frame is unchanged, and the model's
context is the WHOLE document (no marker, no notice)."""
content = "K" * 250
doc = _insert_big_doc(db, content)
live = get_settings()
monkeypatch.setattr(
chat_api,
"get_settings",
lambda: Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
relevance_threshold=live.relevance_threshold,
read_max_chars=10_000, # far over the doc's true length
),
)
scripted = FakeRagLLM(
tool_script=[
[
ToolCallPiece(
id="call_1", name="read", arguments={"path": "docs/big-read.md"}
)
]
]
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
db.delete(doc)
db.commit()
types = [f["type"] for f in frames]
assert "error" not in types
assert types.count("tool_result") == 0 # one frame = one noteworthy event
assert types.count("tool") == 1
(tool_frame,) = [f for f in frames if f["type"] == "tool"]
assert set(tool_frame) == {"type", "name", "argument"} # byte-identical shape
assert tool_frame["argument"] == "docs/big-read.md"
assert frames[-1]["type"] == "done" and frames[-1]["deflected"] is False
# The model saw the WHOLE document — no marker, no notice.
tool_msgs = [
m for r in scripted.seen_messages for m in r if m.get("role") == "tool"
]
assert tool_msgs
assert tool_msgs[-1]["content"] == "Document docs/big-read.md:\n" + content
assert TRUNCATION_MARKER not in tool_msgs[-1]["content"]
def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
) -> None:
+24 -3
View File
@@ -54,6 +54,9 @@ SHARED_OUT_KEYS = {"title", "messages"}
#: A full ``bor.chat.v1`` brain record (phase 14 shape) — every optional
#: key present; the round-trip test asserts it survives byte-identical.
#: Phase 95 (task 02): the CURRENT full tool-entry shape — the additive
#: truncation fields ride each record (a pre-phase-95 entry WITHOUT them
#: still validates — the backward-compat pin in test_schemas.py).
FULL_BRAIN: dict[str, Any] = {
"who": "brain",
"text": "Your k3s cluster runs on three nodes — you've got this.",
@@ -64,12 +67,30 @@ FULL_BRAIN: dict[str, Any] = {
"suggestions": ["What ports does Traefik expose?"],
"thinking": "The kubernetes doc covers the cluster layout…",
"tools": [
{"name": "read", "argument": "Homelab/kubernetes.md"},
{"name": "ls", "argument": None},
{
"name": "read",
"argument": "Homelab/kubernetes.md",
"truncated": True,
"chars_shown": 128_000,
"chars_total": 204_000,
},
{
"name": "ls",
"argument": None,
"truncated": False,
"chars_shown": None,
"chars_total": None,
},
# Saved chats persisting the pre-phase-70 tool names still
# validate — ``name`` is opaque to the API (no migration,
# locked: old chats render fine).
{"name": "read_document", "argument": "Homelab/legacy-notes.md"},
{
"name": "read_document",
"argument": "Homelab/legacy-notes.md",
"truncated": False,
"chars_shown": None,
"chars_total": None,
},
],
"stopped": False,
}
+205 -12
View File
@@ -47,12 +47,21 @@ from app.models import Document, GitSource
from app.rag import agent
from app.rag.agent import (
AGENT_TOOLS,
READ_TRUNCATION_NOTICE,
AgentHolder,
MalformedReplyError,
run_agent,
)
from app.rag.llm import LLMClient, LLMError, RetryPiece, StreamPiece, ToolCallPiece
from app.rag.llm import (
LLMClient,
LLMError,
RetryPiece,
StreamPiece,
ToolCallPiece,
ToolResultPiece,
)
from app.rag.prompts import TOOLS_SECTION, _base, build_deflect_prompt, build_high_prompt
from app.rag.retriever import TRUNCATION_MARKER
if TYPE_CHECKING:
from app.rag.scaffolding import ScaffoldingFilter
@@ -121,11 +130,12 @@ async def _run(
settings: Settings,
seed_docs: list[Document] | None = None,
history: Sequence[dict[str, Any]] = (),
) -> list[StreamPiece | ToolCallPiece | RetryPiece]:
) -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
"""Consume one ``run_agent`` turn; *history* (phase 74) is the
client's prior turns spliced between system and user (default
``()`` — the pre-phase-74 two-message request)."""
out: list[StreamPiece | ToolCallPiece | RetryPiece] = []
``()`` — the pre-phase-74 two-message request). Phase 95: the loop
may also yield a ``ToolResultPiece`` (a truncated ``read``)."""
out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
async for piece in run_agent(
cast("LLMClient", llm),
cast("Session", None),
@@ -192,15 +202,25 @@ def test_agent_tools_names_and_parameters() -> None:
# section already carries (every refusal of a 12-call run was
# ALREADY_IN_CONTEXT); the rule now leads the description instead
# of sitting mid-paragraph, and the tool is framed as "only for
# documents NOT already in <documents>".
# documents NOT already in <documents>". Phase 95 (task 01): the
# read-truncation sentence is inserted before the one-call-at-a-
# time discipline clause (the discipline rule stays last, as in the
# other two tools) — a capped read carries the TRUNCATED notice and
# the `grep` follow-up (the pinned copy).
assert read["description"] == (
"Do not call this tool for a document already shown in "
"the <documents> section, even when the user asks you to "
"open or read it — its full text is already in your "
"prompt; answer directly from it. Use it only to add a "
"document NOT already in <documents> to your context, "
"by its combined `source/path` string. Call one tool at "
"a time — wait for this result before your next call."
"by its combined `source/path` string. Very large "
"documents are truncated: you receive the first part "
"plus a TRUNCATED notice naming how many more characters "
"exist — the notice is authoritative, the document did "
"NOT end where it stopped. Follow it and use `grep` "
"(pattern) to locate the rest — it searches the whole "
"document. Call one tool at a time — wait for this "
"result before your next call."
)
read_params = read["parameters"]
assert read_params["type"] == "object"
@@ -1519,6 +1539,177 @@ def test_reading_an_already_read_doc_is_deduped(monkeypatch: pytest.MonkeyPatch)
assert llm.requests[2][1] == AGENT_TOOLS
# ---------- phase 95: the read cap (bounded reads, honest truncation) ----------
def test_read_exactly_at_cap_is_byte_identical_and_untruncated(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 95 boundary: a document whose content length EQUALS the cap
fits — read whole, byte-identical to the pre-phase-95 result (no
marker, no notice, no holder entry, no ``ToolResultPiece``)."""
cap = 20
content = "x" * cap
doc = _doc("S", "big.md", "Big", content)
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/big.md"})],
[StreamPiece("content", "ans")],
)
out = asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
# Byte-identical to today's read result (no marker, no notice).
assert llm.requests[1][0][3]["content"] == "Document S/big.md:\n" + content
assert TRUNCATION_MARKER not in llm.requests[1][0][3]["content"]
# No truncation recorded, none surfaced to the loop.
assert holder.read_truncations == []
assert not any(isinstance(p, ToolResultPiece) for p in out)
# Still a successful read.
assert holder.read_docs == [doc]
assert holder.tool_calls == 1
def test_read_at_cap_plus_one_truncates_with_marker_and_notice(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 95 boundary: ONE char over the cap truncates — the first
``cap`` chars + the shared :data:`TRUNCATION_MARKER` + the pinned
grep-pointer notice (``{total}`` = the true length, ``{shown}`` = the
cap), and the truncation is recorded on the holder. A truncated read
is still a successful call (``tool_calls`` / ``read_docs`` as today)."""
cap = 20
content = "x" * (cap + 1)
doc = _doc("S", "big.md", "Big", content)
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/big.md"})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
expected = (
"Document S/big.md:\n"
+ content[:cap]
+ "\n"
+ TRUNCATION_MARKER
+ "\n"
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=cap + 1)
)
assert llm.requests[1][0][3]["content"] == expected
# (argument, chars_shown, chars_total) — the raw argument the tool
# frame carries, the cap kept, the true length.
assert holder.read_truncations == [("S/big.md", cap, cap + 1)]
assert holder.read_docs == [doc] # still added to the context
assert holder.tool_calls == 1 # still a counted, successful call
def test_read_truncation_notice_is_pinned() -> None:
"""Phase 95: the notice copy is pinned — it names the true length
(``{total}``), the cap kept (``{shown}``), states the rest is NOT
shown (the document did not end where it stopped), and points at
``grep`` (which searches the whole document)."""
assert READ_TRUNCATION_NOTICE.format(shown=100, total=250) == (
"TRUNCATED — this document is 250 characters; only the first "
"100 are in your context. The rest is NOT shown. Use grep "
"(pattern) to locate what you need — grep searches the whole "
"document."
)
def test_run_agent_yields_tool_result_after_tool_frame_before_next_round(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 95: ``run_agent`` yields exactly ONE ``ToolResultPiece`` per
truncated read — AFTER the round's ``tool`` frame (the matching
``ToolCallPiece``) and BEFORE the next model round (the answer
pieces). It carries (argument, shown, total); ``argument`` is the
same value the matching ``tool`` frame carries (the raw
``source/path`` the model passed)."""
cap = 20
content = "y" * (cap + 5)
doc = _doc("S", "big.md", "Big", content)
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/big.md"})],
[StreamPiece("content", "ans")],
)
out = asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
# The piece order: the read's ToolCallPiece, then the ToolResultPiece,
# then the next round's content (the answer).
assert isinstance(out[0], ToolCallPiece) and out[0].name == "read"
piece = out[1]
assert isinstance(piece, ToolResultPiece)
assert isinstance(out[2], StreamPiece)
assert piece.name == "read"
assert piece.argument == "S/big.md"
assert piece.truncated is True
assert piece.chars_shown == cap
assert piece.chars_total == cap + 5
# Exactly one ToolResultPiece for the one truncated read.
assert [p for p in out if isinstance(p, ToolResultPiece)] == [piece]
def test_run_agent_short_read_yields_no_tool_result_piece(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 95: a read at or under the cap yields NO ``ToolResultPiece``
— the non-truncated stream is byte-identical to the pre-phase-95
one (just the ``ToolCallPiece`` + the answer)."""
cap = 20
content = "z" * cap # exactly at the cap → not truncated
doc = _doc("S", "small.md", "Small", content)
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/small.md"})],
[StreamPiece("content", "ans")],
)
out = asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
assert not any(isinstance(p, ToolResultPiece) for p in out)
assert holder.read_truncations == []
# Order: the ToolCallPiece then the answer content (no piece between).
assert isinstance(out[0], ToolCallPiece) and out[0].name == "read"
assert isinstance(out[1], StreamPiece)
def test_read_truncation_does_not_touch_refusal_paths(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 95: the read refusal paths are untouched by the cap — a SEED
document that is over the cap is still refused with
``ALREADY_IN_CONTEXT`` (not truncated, nothing recorded, nothing
counted), and an unknown path is still the no-document refusal (no
content is read, so no truncation either)."""
big = "B" * 5000 # far over the tiny cap below
seed = _doc("S", "seed.md", "Seed", big)
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
monkeypatch.setattr(agent, "all_documents", lambda db: [])
# (a) Reading the (over-cap) seed doc → ALREADY_IN_CONTEXT (refusal).
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/seed.md"})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings(read_max_chars=100), seed_docs=[seed]))
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
assert holder.read_truncations == []
assert holder.tool_calls == 0 and holder.read_docs == []
# (b) An unknown path → the no-document refusal (argument echoed),
# even though a big doc could have truncated — no content is read.
holder2 = AgentHolder()
llm2 = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/missing.md"})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm2, holder2, _settings(read_max_chars=100), seed_docs=[seed]))
assert llm2.requests[1][0][3]["content"] == (
"No document at 'S/missing.md' — check the ls output."
)
assert holder2.read_truncations == []
assert holder2.tool_calls == 0 and holder2.read_docs == []
def test_unknown_tool_name_refused(monkeypatch: pytest.MonkeyPatch) -> None:
holder = AgentHolder()
llm = ScriptedLLM(
@@ -2322,8 +2513,8 @@ def test_round_failure_after_first_piece_is_terminal(monkeypatch: pytest.MonkeyP
)
sleeps = _record_sleeps(monkeypatch)
async def drain() -> list[StreamPiece | ToolCallPiece | RetryPiece]:
out: list[StreamPiece | ToolCallPiece | RetryPiece] = []
async def drain() -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
with pytest.raises(LLMError, match="mid-stream drop"):
async for piece in run_agent(
cast("LLMClient", llm),
@@ -2384,8 +2575,8 @@ def test_zero_retries_is_one_plain_attempt(monkeypatch: pytest.MonkeyPatch) -> N
llm = FailingLLM([([], LLMError("connection refused"))])
sleeps = _record_sleeps(monkeypatch)
async def drain() -> list[StreamPiece | ToolCallPiece | RetryPiece]:
out: list[StreamPiece | ToolCallPiece | RetryPiece] = []
async def drain() -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
with pytest.raises(LLMError, match="connection refused"):
async for piece in run_agent(
cast("LLMClient", llm),
@@ -2434,7 +2625,9 @@ def test_abandon_mid_retry_sleep_leaks_nothing(monkeypatch: pytest.MonkeyPatch)
holder=holder,
)
async def consumer() -> list[StreamPiece | ToolCallPiece | RetryPiece]:
async def consumer() -> list[
StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece
]:
return [p async for p in gen]
task = asyncio.ensure_future(consumer())
+26
View File
@@ -209,6 +209,32 @@ def test_agent_max_rounds_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> N
_settings()
def test_read_max_chars_default_and_env_override(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 95: the agent ``read`` tool's result is capped at
``BOR_READ_MAX_CHARS`` (default 128 000 chars ≈ 32k tokens — a
quarter of the owner's 128k-token minimum context). Env-tunable in
both directions."""
monkeypatch.delenv("BOR_READ_MAX_CHARS", raising=False)
assert _settings().read_max_chars == 128_000
monkeypatch.setenv("BOR_READ_MAX_CHARS", "5000")
assert _settings().read_max_chars == 5000
# ``0`` is legal (every non-empty read truncates to the marker +
# notice) — it is not a kill switch, so no lower-bound error.
monkeypatch.setenv("BOR_READ_MAX_CHARS", "0")
assert _settings().read_max_chars == 0
def test_read_max_chars_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> None:
"""A negative cap is a typo — it would slice from the END of the
content (negative indexing) instead of failing, so the validator
fails loudly at startup (the ``agent_max_rounds`` pattern)."""
monkeypatch.setenv("BOR_READ_MAX_CHARS", "-1")
with pytest.raises(ValidationError, match="read_max_chars"):
_settings()
def test_stream_thinking_default_true_and_env_parse(monkeypatch: pytest.MonkeyPatch) -> None:
"""Phase 17 kill-switch (``BOR_STREAM_THINKING``): on by default,
``0``/``false`` turn the ``thinking`` SSE frames off."""
+18
View File
@@ -229,6 +229,24 @@ def test_tools_section_phase72_contract_clauses() -> None:
assert "not a directory or file path" not in TOOLS_SECTION
def test_tools_section_phase95_read_truncation_clause() -> None:
"""Phase 95 (task 01): the ``read`` teaching gains exactly one line —
very large documents are capped, a cut read returns the first part
plus the TRUNCATED notice (the document did not end where it
stopped), and ``grep`` is the follow-up (it searches the whole
document). The ``ls``/``grep`` teaching is untouched (phase 94 owns
``ls``) — the clause is pinned byte-for-byte in the constant."""
assert (
"Very large documents are capped: a cut read returns the first "
"part plus a TRUNCATED notice — the document did not end where "
"it stopped; use `grep` (pattern) to find the rest, it searches "
"the whole document."
) in TOOLS_SECTION
# It rides the HIGH prompt and never the LOW (deflected) prompt.
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes")
assert "Very large documents are capped" in build_high_prompt([doc])
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
"""Phase 72/94: the contract clauses ride the HIGH prompt with the
rest of the section and never leak into the LOW/deflection prompt
+222
View File
@@ -0,0 +1,222 @@
"""Unit: the phase-95 (task 02) truncation-marker frontend contract.
No new Python app logic exists for the marker itself — the behavior
lives in ``frontend/assets/app.js`` (the ``tool_result`` SSE branch +
the ``toolAcc`` stamp + the phase-14 restore marker),
``frontend/assets/shared.js`` (the shared page's local tool-line
render) and ``frontend/assets/styles.css`` (the theme-neutral
``.truncated-note`` rule). Like the other frontend-adjacent unit files
(``test_frontend_tool_states.py`` is the phase-37 precedent), this
module pins the JS/CSS markers the story depends on, so a silent
regression in the handler, the pinned marker copy, the persistence
stamp, or the styling is caught without a browser. The E2E gate is the
phase's story suite (task 03).
"""
from __future__ import annotations
import re
from pathlib import Path
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
APP_JS = FRONTEND / "assets" / "app.js"
SHARED_JS = FRONTEND / "assets" / "shared.js"
STYLES_CSS = FRONTEND / "assets" / "styles.css"
#: The pinned marker copy (the unit + E2E assertion target): plain
#: integers, no thousands separators, the em-dash per the owner's TODO.
MARKER_TEMPLATE = '" (truncated — showing "'
def _js() -> str:
return APP_JS.read_text(encoding="utf-8")
def _shared_js() -> str:
return SHARED_JS.read_text(encoding="utf-8")
def _css() -> str:
return STYLES_CSS.read_text(encoding="utf-8")
def test_tool_result_branch_is_a_first_class_turn_branch() -> None:
"""The turn handler must branch on `tool_result` frames (the
seventh, optional event type — the A15 extension): the branch sits
after `done` (a frame for a settled turn is a tolerated late
append, never a crash) and before `error`, settles the tool-line
clock like every other frame, and must never flip the state
machine (the phase-37/48 lifecycle is untouched)."""
js = _js()
done_idx = js.find('ev.type === "done"')
tool_result_idx = js.find('ev.type === "tool_result"')
error_idx = js.find('ev.type === "error"')
assert -1 < done_idx < tool_result_idx < error_idx, (
"the turn handler must branch on tool_result frames (after done,"
" before error)"
)
branch = js[tool_result_idx:error_idx]
assert "settleToolLine()" in branch, (
"a frame arrived — the latest tool line's elapsed clock settles"
)
assert "appendTruncatedNote(wrap, argument, shown, total)" in branch
# No UI-state transition: the state machine never knows the marker.
assert "setUiState" not in branch
assert "aborted" not in branch, (
"the aborted guard lives at the dispatch top, not per branch"
)
def test_truncation_marker_copy_is_pinned() -> None:
"""The marker text is pinned EXACTLY: \" (truncated — showing N of
M chars)\" — plain integers (no separators), a leading space (it
follows the line's <code> child), the em-dash per the TODO copy.
Both app.js's helper and shared.js's local render carry the same
template (pixel-identical marker — the phase-50 restore
contract)."""
js = _js()
fn = js.find("function appendTruncatedNote")
assert fn != -1, "appendTruncatedNote must exist in app.js"
body = js[fn : js.find("\n}\n", fn)]
assert MARKER_TEMPLATE in body
assert '+ charsShown + " of " + charsTotal + " chars)"' in body
shared = _shared_js()
fn = shared.find("function addToolLines")
assert fn != -1
sbody = shared[fn : shared.find("\n}\n", fn)]
assert MARKER_TEMPLATE in sbody
assert 'chars_shown) || 0) + " of " + (Number(t.chars_total) || 0) + " chars)"' in sbody
def test_append_truncated_note_matches_newest_line_and_uses_text_content() -> None:
"""appendTruncatedNote: the target is the NEWEST `.tool-call` line
whose `<code>` child carries the frame's argument (the raw
source/path — the same string the matching `tool` frame put in the
line), scanned newest-first; the marker is a SPAN sibling appended
to the existing line (createElement + className + textContent only
— the house "this file never builds HTML" rule, no innerHTML); a
frame whose line is gone (New Chat mid-turn) is a silent no-op."""
js = _js()
fn = js.find("function appendTruncatedNote")
assert fn != -1
body = js[fn : js.find("\n}\n", fn)]
assert 'wrap?.querySelector?.(".tool-calls")' in body, (
"a wrap without tool lines is a silent no-op"
)
assert 'querySelectorAll(".tool-call")' in body
assert "for (let i = lines.length - 1; i >= 0; i -= 1)" in body, (
"newest line first — the last call for that argument"
)
assert 'code.textContent !== argument' in body, (
"the match key is the code child's argument (the raw source/path)"
)
assert 'note.className = "truncated-note"' in body
assert 'lines[i].appendChild(note)' in body, (
"DOM append to the EXISTING line — no new line, no re-render"
)
assert "innerHTML" not in body, (
"no HTML injection surface — createElement + textContent only"
)
# The no-op guard: a wrap without tool lines returns silently; a scan
# that finds no matching line falls off the loop without appending.
assert "if (!calls) return;" in body
def test_tool_result_branch_stamps_the_newest_toolacc_entry() -> None:
"""The `tool_result` frame stamps the matching toolAcc entry (same
argument, NEWEST — the reverse scan mirrors the line match) with
`truncated` + `chars_shown` + `chars_total` — the `done` save point
below then carries it with zero other change (the persistence
shape rides the existing `tools` key). The stamp is gated on the
frame's argument + truncated truth (a malformed frame is a silent
no-op for persistence)."""
js = _js()
tool_result_idx = js.find('ev.type === "tool_result"')
error_idx = js.find('ev.type === "error"')
assert -1 < tool_result_idx < error_idx
branch = js[tool_result_idx:error_idx]
assert "t.truncated = true" in branch
assert "t.chars_shown = shown" in branch
assert "t.chars_total = total" in branch
assert "for (let i = toolAcc.length - 1; i >= 0; i -= 1)" in branch, (
"the NEWEST matching entry (the reverse scan, same rule as the"
" line match) gets stamped — then break"
)
assert "t.argument === argument" in branch
assert "break" in branch
# The guard: the stamp only runs for a real truncation with an argument.
assert "if (argument && ev.truncated)" in branch
# The saved payload rides the existing save point — no second tools
# key, no new record field.
done_block = js[js.find('ev.type === "done"') : tool_result_idx]
assert "tools: toolAcc.length ? toolAcc : undefined" in done_block
def test_restore_path_renders_the_stored_marker() -> None:
"""The phase-14 LOCAL restore path (renderStoredMessage): a stored
tool record with `truncated` + the counts re-renders the SAME
marker next to its Reading line, right after the line is restored
(the same order as the live frames). A pre-phase-95 record (no
field — `t.truncated` falsy) renders unchanged (no marker, no
migration)."""
js = _js()
fn = js.find("function renderStoredMessage")
assert fn != -1
end = js.find("function restoreConversation")
body = js[fn:end]
assert "appendToolLine(wrap, t.name, arg)" in body
assert "t.truncated && arg" in body, (
"only an argument-bearing (Reading) record with the flag renders"
" the marker"
)
assert (
"appendTruncatedNote(wrap, arg, Number(t.chars_shown) || 0,"
" Number(t.chars_total) || 0)" in body
)
# The marker append sits INSIDE the tools loop, after the line append.
line_idx = body.find("appendToolLine(wrap, t.name, arg)")
note_idx = body.find("appendTruncatedNote(wrap, arg,")
assert -1 < line_idx < note_idx
def test_shared_page_renders_the_stored_marker() -> None:
"""shared.js's local tool-line render (addToolLines): the same
marker from the stored record — a span sibling appended to the
line, after the line's existing children (the template text + the
<code> argument), textContent only (no HTML from storage, ever). A
record saved before phase 95 renders exactly as before."""
js = _shared_js()
fn = js.find("function addToolLines")
assert fn != -1
body = js[fn : js.find("\n}\n", fn)]
assert "t.truncated && argument" in body
assert 'note.className = "truncated-note"' in body
assert 'line.appendChild(note)' in body
assert MARKER_TEMPLATE in body
# Still textContent-only: the marker adds no innerHTML surface, and
# the three argument-bearing lines keep their textContent treatment.
assert body.count("code.textContent = argument") == 3
assert "innerHTML" not in body
def test_truncated_note_style_is_theme_neutral() -> None:
"""styles.css: `.tool-call .truncated-note` exists and colors ONLY
through a `var(--…)` token (the phase-92 zero-literal invariant —
no new hue; under phase 93's monochrome theme it grays
automatically, and the marker stays TEXT, never color alone, B5).
--ink-soft is the AA-safe soft-ink the status suffixes already
borrow."""
css = _css()
m = re.search(r"\.tool-call \.truncated-note \{([^}]*)\}", css)
assert m, "the .tool-call .truncated-note rule must exist"
rule = m.group(1)
assert "var(--ink-soft)" in rule
assert "color: var(--ink-soft)" in rule
# Theme-neutral: the whole rule is a single var() color — no hex,
# no rgb(), no other property.
assert not re.search(r"#[0-9a-fA-F]{3,8}\b|rgb\(", rule)
def test_no_cdn_added() -> None:
"""AGENTS.md rule 6: the marker adds no external script/link."""
index = (FRONTEND / "index.html").read_text(encoding="utf-8")
assert 'src="http' not in index and 'href="http' not in index
+165 -3
View File
@@ -41,7 +41,15 @@ def _source_ref() -> dict:
def _tool_call() -> dict:
return {"name": "read", "argument": "Homelab/kubernetes.md"}
# Phase 95 (task 02): the truncation record rides the same entry —
# the CURRENT full shape (additive fields, defaults for a plain read).
return {
"name": "read",
"argument": "Homelab/kubernetes.md",
"truncated": False,
"chars_shown": None,
"chars_total": None,
}
def _user_message(text: str = "How did I install k3s?") -> ChatMessage:
@@ -341,8 +349,22 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
"suggestions": None,
"thinking": "The kubernetes doc covers the cluster layout…",
"tools": [
{"name": "read", "argument": "Homelab/kubernetes.md"},
{"name": "ls", "argument": None},
# Phase 95 (task 02): the current full shape — one
# truncated read (the marker record) + a plain ls.
{
"name": "read",
"argument": "Homelab/kubernetes.md",
"truncated": True,
"chars_shown": 128_000,
"chars_total": 204_000,
},
{
"name": "ls",
"argument": None,
"truncated": False,
"chars_shown": None,
"chars_total": None,
},
],
"stopped": None,
},
@@ -393,3 +415,143 @@ def test_realistic_payload_round_trips_through_update_model() -> None:
}
payload = SavedChatUpdate.model_validate({"messages": [msg]})
assert payload.model_dump()["messages"] == [msg]
# ---------------------------------------------------------------------------
# Phase 95 (task 02): ToolCall truncation fields + ChatToolResultEvent
# ---------------------------------------------------------------------------
def test_tool_call_round_trips_with_truncation_fields() -> None:
"""The CURRENT full shape (phase 95 task 02): a truncated read's
record (``truncated: True`` + the two non-negative counts) validates
and round-trips ``model_dump()`` unchanged — the save payload carries
it with zero other change (the UI re-renders the marker from it)."""
raw = {
"name": "read",
"argument": "Homelab/big.md",
"truncated": True,
"chars_shown": 128_000,
"chars_total": 204_000,
}
call = ToolCall.model_validate(raw)
assert call.truncated is True
assert call.chars_shown == 128_000
assert call.chars_total == 204_000
assert call.model_dump() == raw
def test_tool_call_old_shape_validates_with_defaults() -> None:
"""Backward-compat (the phase-50 rule): a saved chat written BEFORE
phase 95 — tool records without the truncation fields — validates
UNCHANGED: ``truncated`` defaults to False (the marker is absent),
the counts to None. No migration (``ChatMessage.tools`` is JSON);
only the dump gains the additive keys with their defaults."""
old_shape = {"name": "read", "argument": "Homelab/kubernetes.md"}
call = ToolCall.model_validate(old_shape)
assert call.truncated is False
assert call.chars_shown is None
assert call.chars_total is None
dumped = call.model_dump()
assert dumped["name"] == "read" and dumped["argument"] == "Homelab/kubernetes.md"
assert dumped["truncated"] is False
assert dumped["chars_shown"] is None and dumped["chars_total"] is None
def test_tool_call_old_shape_message_still_round_trips_as_record() -> None:
"""The record-level backward-compat: a pre-phase-95 brain message
(old-shape ``tools``) validates inside ``ChatMessage`` and dumps back
as a VALID record of the same shape (the frontend renders it without
the marker — ``truncated`` falsy)."""
old_message = {
"who": "brain",
"text": "You've got this!",
"sources": None,
"deflected": False,
"suggestions": None,
"thinking": None,
"tools": [
{"name": "read", "argument": "Homelab/kubernetes.md"},
{"name": "read_document", "argument": "Homelab/legacy.md"},
],
"stopped": None,
}
msg = ChatMessage.model_validate(old_message)
assert all(t.truncated is False for t in (msg.tools or []))
# Re-validating the dump is a no-op (lossless record round-trip).
ChatMessage.model_validate(msg.model_dump())
def test_tool_call_counts_reject_negative() -> None:
"""Phase 83 bounds philosophy: the counts are non-negative
(``ge=0``) — a negative count is not a real record."""
with pytest.raises(ValidationError):
ToolCall.model_validate(
{
"name": "read",
"argument": "x",
"truncated": True,
"chars_shown": -1,
"chars_total": 5,
}
)
with pytest.raises(ValidationError):
ToolCall.model_validate(
{
"name": "read",
"argument": "x",
"truncated": True,
"chars_shown": 5,
"chars_total": -1,
}
)
def test_chat_tool_result_event_shape() -> None:
"""The A15 extension's wire shape (phase 95 task 02): the seventh,
OPTIONAL SSE event type — ``{type, name, argument, truncated,
chars_shown, chars_total}`` — with the pinned field order, the
``tool_result`` default, ``truncated`` defaulting True (the emission
trigger), and the non-negative counts."""
from app.schemas import ChatToolResultEvent
ev = ChatToolResultEvent(
name="read",
argument="docs/big.md",
truncated=True,
chars_shown=128_000,
chars_total=204_000,
)
dumped = ev.model_dump()
assert list(dumped) == [
"type",
"name",
"argument",
"truncated",
"chars_shown",
"chars_total",
]
assert dumped == {
"type": "tool_result",
"name": "read",
"argument": "docs/big.md",
"truncated": True,
"chars_shown": 128_000,
"chars_total": 204_000,
}
# The emission trigger defaults: the pump always builds the frame from
# a piece, so a frame that ever exists carries the truncation truth.
minimal = ChatToolResultEvent(
name="read", argument=None, chars_shown=0, chars_total=0
).model_dump()
assert minimal["truncated"] is True
def test_chat_tool_result_event_counts_reject_negative() -> None:
"""The frame's counts are non-negative (``ge=0``), like the record's."""
from app.schemas import ChatToolResultEvent
with pytest.raises(ValidationError):
ChatToolResultEvent(name="read", argument="x", chars_shown=-1, chars_total=5)
with pytest.raises(ValidationError):
ChatToolResultEvent(name="read", argument="x", chars_shown=5, chars_total=-1)