phase: 160_tool_budget_telemetry
All green. Final report: **Phase 160 completion pass — verified (final verification; all 5 tasks already in `complete/`)** - **Verified, no code defects found:** budget trailers (`TOOL_BUDGET_REMAINING/LAST/SPENT`), cap-path rejection + `ROUND_CAP_FALLBACK_ANSWER` + `ROUND_CAP_WRAPUP_INSTRUCTION`, `done.budget_exhausted` (ABSENT-when-false), `rounds=/cap_hit=` log fields, `agent_max_rounds=10` default, frontend Continue button + restore + shared-page exclusion, mock markers. - **Tests/lint:** `uv run pytest` 153/153 green (EXIT 0); `uv run pytest --cov=app --cov-report=term-missing` TOTAL **99%** (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors/0 warnings. - **Criteria:** ① decreasing trailer pinned by unit tests (9→…→spent) ✅ ② cap path ends readable, no tool lines after cap (E2E) ✅ ③ `budget_exhausted` true-only-on-cap, absent otherwise (integration) ✅ ④ Continue button → one fresh `continue` turn, fresh budget, record intact (E2E) ✅ ⑤ flag survives refresh; shared chat never renders it (`shared.js` has no `budget_exhausted` path) ✅ ⑥ `max_rounds=0` byte-identical single `tools=None` request; default still 10 ✅ ⑦ full suite/coverage/lint green ✅ ⑧ `test_tool_budget_telemetry.py` **7/7** in isolation; all 6 regression E2E suites green in isolation ✅ ⑨ commit + phase move = harness (I make no git history/index changes). - **Docs:** README.md — fixed: the "Continue researching" button and cap behavior were undocumented (added a Chat-section paragraph + an Agent-tools-section paragraph, incl. the phase-71 recorded-revision note); env-var table, commands, and architecture all verified consistent otherwise. AGENTS.md — verified consistent, no changes needed. - **Decisions:** no code changes required — phase was fully implemented; only README docs added. No live-infrastructure touched. - **Next pending phase:** `161_reasoning_bubble`.
This commit is contained in:
@@ -842,6 +842,57 @@ Implements just enough of the aipi surface:
|
||||
deflected turns never hit it); verified 2026-10-01: no
|
||||
existing E2E question or fixture file contains the phrase, so
|
||||
every other suite is unaffected.
|
||||
- user message containing ``run out of tool calls``
|
||||
(``ROUND_CAP_TRIGGER``, phase 160 task 05 — the
|
||||
tool-call-budget telemetry's dedicated suite
|
||||
``tests/e2e/test_tool_budget_telemetry.py``) — detected on
|
||||
the LAST user message only (the mock's existing invariant:
|
||||
the last ``user`` message is the current question, the
|
||||
earlier non-system messages are prior turns):
|
||||
* while ``tools`` are offered (the request carries
|
||||
``tools`` — the app issues such a request only while the
|
||||
round budget is unspent, i.e. no tool result answers the
|
||||
last emitted call yet) and the flow is active: stream
|
||||
ONLY ``tool_calls`` deltas — ``ls`` with no arguments,
|
||||
incrementing synthetic ids ``call_cap_1``,
|
||||
``call_cap_2``, … (the next id is derived from the
|
||||
``call_cap_N`` ids already in the messages' ``assistant``
|
||||
tool calls), ``finish_reason: "tool_calls"``, no content —
|
||||
the model NEVER stops asking, so the turn is guaranteed
|
||||
to hit ``BOR_AGENT_MAX_ROUNDS``;
|
||||
* on the request WITHOUT ``tools`` (the forced final — the
|
||||
round-cap path; the request still carries the full
|
||||
history, so the trigger stays the last user message):
|
||||
stream a ``tool_calls`` delta ANYWAY — ``read`` on the
|
||||
SEED path (the system prompt's ``<documents>`` section's
|
||||
FIRST ``<document>`` block, the joined ``source/path`` —
|
||||
the ``read`` tool's combined identity, phase 70), id
|
||||
``call_cap_dump``, NO content — the EXACT dumped-call
|
||||
shape the phase kills (and the shape ``SCAFFOLD_TRIGGER``
|
||||
does not cover — that one emits textual markup, this one
|
||||
is structured). The app REJECTS it (never executed, never
|
||||
shown, counted in ``AgentHolder.rejected_calls``);
|
||||
* the ``continue`` question (the last user message is
|
||||
EXACTLY ``continue`` — the UI's ``CONTINUE_QUESTION`` —
|
||||
and an EARLIER user message carries the trigger; the
|
||||
client history rides in the stateless A10 request): the
|
||||
stable ``ROUND_CAP_CONTINUE_ANSWER`` (a fixed sentence
|
||||
naming the flow's seed document — the
|
||||
``SCOPED_SEARCH_PATH`` suite-constant convention: the
|
||||
literal "continue" shares no tokens with the KB, so the
|
||||
continuation turn deflects and its prompt carries no
|
||||
``<documents>`` block; the sentence names the document
|
||||
directly), so a suite asserts a real continuation answer
|
||||
rather than a tool call.
|
||||
Streaming only (the app's chat path is the only streaming
|
||||
consumer; a non-streaming marker request never issues and
|
||||
falls through to the regular answer, like the other marker
|
||||
flows). Checked BEFORE the plain ``TOOLS_TRIGGER`` flow
|
||||
(disjoint trigger phrases — the phase-71/72/94/118/131/
|
||||
139/141/145 ordering convention); verified 2026-10-05: no
|
||||
existing E2E question or fixture file contains the phrase
|
||||
(or sends a ``continue`` question with the trigger in the
|
||||
history), so every other suite is unaffected.
|
||||
|
||||
Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic
|
||||
dead-endpoint behavior for the retry E2E suite (``tests/e2e/
|
||||
@@ -1193,6 +1244,42 @@ HISTORY_TRIGGER = "echo my history"
|
||||
#: phrase, so every other suite is unaffected.
|
||||
FOLDER_MAP_TRIGGER = "repeat your folder map"
|
||||
|
||||
#: Phase 160 (task 05, tool-call budget telemetry, TODO.md L4 —
|
||||
#: the dedicated suite
|
||||
#: ``tests/e2e/test_tool_budget_telemetry.py``): a user message
|
||||
#: containing this substring (case-insensitive), detected on the
|
||||
#: LAST user message only (the mock's existing invariant — the
|
||||
#: last ``user`` message is the current question, the earlier
|
||||
#: non-system messages are prior turns), drives the deterministic
|
||||
#: ROUND-CAP flow: while ``tools`` are offered the mock NEVER
|
||||
#: answers (``ls`` calls, incrementing ``call_cap_N`` ids — the
|
||||
#: turn is guaranteed to hit ``BOR_AGENT_MAX_ROUNDS``), and the
|
||||
#: forced final (no ``tools``) streams a structured ``read`` call
|
||||
#: anyway (id ``call_cap_dump``, the seed path, NO content) — the
|
||||
#: exact dumped-call shape the phase kills. The ``continue``
|
||||
#: question (exactly ``continue``, the trigger in an EARLIER user
|
||||
#: message) gets :data:`ROUND_CAP_CONTINUE_ANSWER`. Disjoint from
|
||||
#: every other trigger phrase; verified 2026-10-05: no existing
|
||||
#: E2E question or fixture file contains the phrase, so every
|
||||
#: other suite is unaffected.
|
||||
ROUND_CAP_TRIGGER = "run out of tool calls"
|
||||
|
||||
#: Phase 160 (task 05): the ROUND-CAP flow's CONTINUATION answer
|
||||
#: — the fixed sentence the ``continue`` question gets. The
|
||||
#: literal "continue" shares no tokens with the KB, so the
|
||||
#: continuation turn DEFLECTS (A8 LOW) and its prompt carries no
|
||||
#: ``<documents>`` block — the sentence therefore names the
|
||||
#: flow's seed document DIRECTLY (the
|
||||
#: :data:`SCOPED_SEARCH_PATH` suite-constant convention; the
|
||||
#: suite seeds exactly that document, so the name IS the
|
||||
#: seed's). Byte-stable on purpose: the suite asserts on it, so
|
||||
#: it must not drift.
|
||||
ROUND_CAP_CONTINUE_ANSWER = (
|
||||
"Picking up where the tool budget ran out — still digging into "
|
||||
"docs/homelab/kubernetes.md: it says the cluster runs Talos "
|
||||
"Linux on three nodes, and Cilium handles networking."
|
||||
)
|
||||
|
||||
#: Phase 122 (image documents, LOCKED A3): the fixed description the
|
||||
#: mock's vision (CHAT) model returns for an ``IMAGE_DESCRIPTION_MODE``
|
||||
#: request — the multimodal user message
|
||||
@@ -3079,6 +3166,106 @@ def _origin_link_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
return ("call", target, "call_0")
|
||||
|
||||
|
||||
#: Phase 160 (task 05): the round-cap flow's incrementing call ids
|
||||
#: — ``call_cap_1``, ``call_cap_2``, … The NEXT id is derived from
|
||||
#: the ids already in the messages' ``assistant`` tool calls (the
|
||||
#: flow is stateless over the message history — a retry that
|
||||
#: re-issues the same request re-derives the same next id).
|
||||
_CAP_CALL_ID_RE = re.compile(r"^call_cap_(\d+)$")
|
||||
|
||||
|
||||
def _round_cap_next_call_id(body: dict[str, Any]) -> str:
|
||||
"""The next incrementing ``call_cap_N`` id (stateless).
|
||||
|
||||
The mock's ``ls`` calls in the round-cap flow carry
|
||||
incrementing synthetic ids — the NEXT id is one past the
|
||||
highest ``call_cap_N`` id already in the messages'
|
||||
``assistant`` tool calls (``call_cap_1`` on the first
|
||||
request — no ids yet).
|
||||
"""
|
||||
n = 0
|
||||
for m in _messages(body):
|
||||
if m.get("role") != "assistant":
|
||||
continue
|
||||
for call in m.get("tool_calls") or []:
|
||||
if not isinstance(call, dict):
|
||||
continue
|
||||
match = _CAP_CALL_ID_RE.match(str(call.get("id") or ""))
|
||||
if match:
|
||||
n = max(n, int(match.group(1)))
|
||||
return f"call_cap_{n + 1}"
|
||||
|
||||
|
||||
def _round_cap_seed_path(body: dict[str, Any]) -> str:
|
||||
"""The seed document's combined ``source/path`` — or the
|
||||
sentinel (the premise broke: no ``<documents>`` block; the
|
||||
suite fails loudly on the sentinel in the dump call).
|
||||
|
||||
Phase 160 (task 05): the FORCED FINAL's request (the
|
||||
round-cap path) still carries the grounded system prompt —
|
||||
the ``<documents>`` section's FIRST ``<document>`` block is
|
||||
the seed document (the suite seeds exactly one). The mock
|
||||
joins the labeled ``source`` / ``path`` fields (the
|
||||
``read`` tool's combined identity, phase 70).
|
||||
"""
|
||||
block = _DOCUMENT_BLOCK_RE.search(_system(body))
|
||||
if not block:
|
||||
return "seed/unknown.md"
|
||||
return f"{block.group('source')}/{block.group('path')}"
|
||||
|
||||
|
||||
def _round_cap_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Phase 160 (task 05): the ROUND-CAP marker flow — or ``None``.
|
||||
|
||||
Discriminated on the LAST user message (the mock's existing
|
||||
invariant: the last ``user`` message is the current
|
||||
question, the earlier non-system messages are prior turns —
|
||||
the ``_history_echo`` ``last_user`` scan convention):
|
||||
|
||||
* the last user message carries :data:`ROUND_CAP_TRIGGER`:
|
||||
- the request offers ``tools`` (the app issues such a
|
||||
request only while the round budget is unspent — no
|
||||
tool result answers the last emitted call yet):
|
||||
``("call", call_cap_N)`` — the model NEVER stops
|
||||
asking (``ls``, no arguments, the next incrementing
|
||||
id) — the turn is guaranteed to hit
|
||||
``BOR_AGENT_MAX_ROUNDS``;
|
||||
- the request carries NO ``tools`` (the forced final —
|
||||
the round-cap path; the trigger is still the last user
|
||||
message): ``("dump", <seed path>)`` — a ``read`` on
|
||||
the seed path (id ``call_cap_dump``, NO content) — the
|
||||
exact dumped-call shape the phase kills;
|
||||
* the last user message is EXACTLY ``continue`` (the UI's
|
||||
``CONTINUE_QUESTION``) and an EARLIER user message
|
||||
carries the trigger (the client history rides in the
|
||||
stateless A10 request): ``("continue",)`` — the stable
|
||||
:data:`ROUND_CAP_CONTINUE_ANSWER` (a fixed sentence
|
||||
naming the flow's seed document — the
|
||||
:data:`SCOPED_SEARCH_PATH` suite-constant convention: the
|
||||
literal "continue" shares no tokens with the KB, so the
|
||||
continuation turn deflects and its prompt carries no
|
||||
``<documents>`` block).
|
||||
"""
|
||||
users = [
|
||||
_content_text(m.get("content"))
|
||||
for m in _messages(body)
|
||||
if m.get("role") == "user"
|
||||
]
|
||||
if not users:
|
||||
return None
|
||||
last_user = users[-1].strip().lower()
|
||||
if ROUND_CAP_TRIGGER in last_user:
|
||||
if body.get("tools"):
|
||||
return ("call", _round_cap_next_call_id(body))
|
||||
return ("dump", _round_cap_seed_path(body))
|
||||
if (
|
||||
last_user == "continue"
|
||||
and any(ROUND_CAP_TRIGGER in u.lower() for u in users[:-1])
|
||||
):
|
||||
return ("continue",)
|
||||
return None
|
||||
|
||||
|
||||
def long_answer() -> str:
|
||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||
a unique final line that must survive the stream untruncated."""
|
||||
@@ -4563,6 +4750,42 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 160 (task 05, tool-call budget telemetry,
|
||||
# TODO.md L4): the deterministic ROUND-CAP marker flow
|
||||
# (the budget-telemetry suite's dedicated flow — see the
|
||||
# module docstring). Checked BEFORE the plain
|
||||
# TOOLS_TRIGGER flow (disjoint trigger phrases — the
|
||||
# phase-71/72/94/118/131/139/141/145 ordering
|
||||
# convention).
|
||||
round_cap = _round_cap_flow(body)
|
||||
if round_cap is not None:
|
||||
if round_cap[0] == "call":
|
||||
# The model never stops asking: ``ls`` with no
|
||||
# arguments, the next incrementing id, NO content
|
||||
# — the turn is guaranteed to hit
|
||||
# ``BOR_AGENT_MAX_ROUNDS``.
|
||||
stream = _tool_call_stream("ls", {}, round_cap[1])
|
||||
elif round_cap[0] == "dump":
|
||||
# The forced final (no ``tools``) — the model
|
||||
# dumps a structured ``read`` call anyway (the
|
||||
# seed path, id ``call_cap_dump``, NO content) —
|
||||
# the exact shape the phase kills (the app
|
||||
# REJECTS it — never executed, never shown).
|
||||
stream = _tool_call_stream(
|
||||
"read", {"path": round_cap[1]}, "call_cap_dump"
|
||||
)
|
||||
else: # "continue" — the stable continuation answer
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
ROUND_CAP_CONTINUE_ANSWER, body.get("max_tokens")
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 124 (task 07, image chips and answers, TODO L3): the
|
||||
# scripted SHOW_IMAGE flow — the mock shows the seeded image
|
||||
# document (the ``<documents>`` block with the image-
|
||||
|
||||
@@ -628,7 +628,8 @@ def test_api_created_chats_carry_the_bor_chat_v1_shape(
|
||||
# brain records (accepted by the ChatMessage schema — its earlier
|
||||
# absence 422'd the done-time auto-save, phase-118 verification fix).
|
||||
allowed = {"who", "text", "sources", "related", "deflected", "suggestions",
|
||||
"thinking", "tools", "stopped", "failed", "error"}
|
||||
"thinking", "tools", "stopped", "failed", "error",
|
||||
"budget_exhausted"}
|
||||
assert all({"who", "text"} <= set(m) <= allowed for m in body["messages"])
|
||||
# The History row renders it (the Open link's text IS the title).
|
||||
page.click("#nav-history")
|
||||
|
||||
@@ -0,0 +1,816 @@
|
||||
"""Phase 160 E2E (Playwright, mock-only): the agent knows its
|
||||
tool-call budget (and never dumps tool calls into the answer).
|
||||
|
||||
Source: ``TODO.md`` L4 — "The LLM need to know how many tool calls
|
||||
it has left. During long research sessions it will sometimes end the
|
||||
conversation with tool calls that just get dumped into chat because
|
||||
it ran out of turns. …"
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_tool_budget_telemetry.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the contract
|
||||
is the deterministic ROUND-CAP marker flow in ``tests/e2e/mock_llm.py``
|
||||
(``ROUND_CAP_TRIGGER`` = ``run out of tool calls``, detected on the
|
||||
LAST user message only — the mock's existing invariant: the last
|
||||
``user`` message is the current question, the earlier non-system
|
||||
messages are prior turns):
|
||||
|
||||
* while ``tools`` are offered (the app issues such a request only
|
||||
while the round budget is unspent — i.e. no tool result answers
|
||||
the last emitted call yet) the mock NEVER answers — it streams
|
||||
ONLY ``tool_calls`` deltas calling ``ls`` with no arguments,
|
||||
incrementing ids ``call_cap_1``, ``call_cap_2``, … (the next id
|
||||
is derived from the ids already in the messages' ``assistant``
|
||||
tool calls), ``finish_reason: "tool_calls"``, no content — the
|
||||
turn is guaranteed to hit ``BOR_AGENT_MAX_ROUNDS``;
|
||||
* on the request WITHOUT ``tools`` (the forced final) the mock
|
||||
streams a ``tool_calls`` delta ANYWAY — ``read`` on the seed
|
||||
path (id ``call_cap_dump``, NO content) — the exact dumped-call
|
||||
shape the phase kills (the app REJECTS it: never executed, never
|
||||
shown, counted in ``AgentHolder.rejected_calls``);
|
||||
* the ``continue`` question (exactly ``continue`` — the UI's
|
||||
``CONTINUE_QUESTION`` — with the trigger in an EARLIER user
|
||||
message; the client history rides in the stateless A10 request)
|
||||
gets the stable ``ROUND_CAP_CONTINUE_ANSWER`` (a fixed sentence
|
||||
naming the seed document — the ``SCOPED_SEARCH_PATH``
|
||||
suite-constant convention: the literal "continue" shares no tokens
|
||||
with the KB, so the continuation turn deflects and its prompt
|
||||
carries no ``<documents>`` block; the sentence names the document
|
||||
directly).
|
||||
|
||||
The e2e app server boots with ``BOR_AGENT_MAX_ROUNDS=3`` (the
|
||||
module-scoped ``app_server`` override — the conftest env shape with
|
||||
one swap): the cap flow burns exactly three executed rounds, and
|
||||
the tool results carry ``[tool calls remaining: 2]``, ``[1 tool
|
||||
call remaining — make it count, then answer]`` and the spent line
|
||||
(``[No tool calls remain — answer now from what you have, or tell
|
||||
the user to say "continue" to keep researching.]`` — the
|
||||
``app.rag.agent`` constants, imported so the pin cannot drift).
|
||||
|
||||
KB fixture (the ``test_failed_turn_retry.py`` TRUNCATE-then-seed
|
||||
pattern): ONE fixture document (``homelab/kubernetes.md``) — the
|
||||
cap question's tokens clear the mock-calibrated gate (≈0.44 against
|
||||
the mock's token-overlap embeddings, past the 0.30 threshold —
|
||||
HIGH without any FTS help; the E2E app pins the mock-calibrated
|
||||
values in conftest).
|
||||
|
||||
Test → contract mapping (``00_phase.md`` §Completion criteria):
|
||||
|
||||
1. ``test_tool_results_carry_the_remaining_count`` — server-side
|
||||
pin at the wire level: the mock's recorded request log (the
|
||||
``/v1/e2e/captured`` pair — reset per turn) shows each
|
||||
successive ``tool``-role message ending with the decreasing
|
||||
trailer (``[tool calls remaining: 2]``, the last-call line, the
|
||||
spent line), the incrementing ``call_cap_N`` ids in the
|
||||
``assistant`` messages, and the final request carrying the spent
|
||||
tool result + the coached system message (no ``tools`` key);
|
||||
2. ``test_capped_turn_never_shows_tool_lines_after_the_cap`` —
|
||||
with ``BOR_AGENT_MAX_ROUNDS=3``: the chat shows EXACTLY three
|
||||
tool lines (one per executed round), the dump call produces NO
|
||||
tool line (no ``Reading <seed path>`` line anywhere), the answer
|
||||
bubble holds the non-empty ``ROUND_CAP_FALLBACK_ANSWER`` (the
|
||||
deterministic backstop — the mock streams no content on the
|
||||
forced final), the turn settles with ``done`` (never the error
|
||||
banner) and the SSE capture has no ``error`` frame;
|
||||
3. ``test_done_flags_budget_exhausted`` — the captured SSE
|
||||
stream's ``done`` frame carries ``budget_exhausted: true`` on
|
||||
the capped turn; a normal grounded turn's ``done`` carries NO
|
||||
``budget_exhausted`` key at all (absent, never false — the
|
||||
byte-identical control case in the same file); the capped
|
||||
answer's Continue button is gone once the newer turn lands;
|
||||
4. ``test_continue_button_runs_a_fresh_turn`` — the capped answer
|
||||
bubble shows **Continue researching**; clicking it appends
|
||||
exactly ONE user bubble ``continue``, the mock answers with
|
||||
``ROUND_CAP_CONTINUE_ANSWER`` (a real continuation answer — a
|
||||
fixed sentence naming the seed document, not a tool call), the
|
||||
earlier tool lines do not duplicate (still exactly three — the
|
||||
continuation turn streams no tool frames), and ``bor.chat.v1``
|
||||
holds ``[user, brain(flagged), user("continue"), brain]`` (the
|
||||
flagged record carries ``budget_exhausted: true``);
|
||||
5. ``test_continue_is_inert_while_streaming_and_single_shot`` —
|
||||
double-clicking Continue (``page.dblclick`` — the second click
|
||||
lands after the first handler removed every ``.continue-btn``
|
||||
and started the turn; the in-flight no-op guard + the
|
||||
remove-first clear make a double-send impossible) starts ONE
|
||||
turn (counted via ``page.on("request")`` — exactly one
|
||||
``/api/chat`` POST; the single ``continue`` user bubble + the
|
||||
single continuation answer pin it);
|
||||
6. ``test_flag_survives_a_refresh`` — ``page.reload()`` after a
|
||||
capped turn: the flagged answer restores WITH the Continue
|
||||
button (the restore path keys on ``m.budget_exhausted``) and
|
||||
clicking it still works (the mock's continuation answer
|
||||
streams);
|
||||
7. negative: ``test_deflected_turn_gets_no_continue_button`` — a
|
||||
deflected question (A8 LOW mode — token-disjoint from the KB:
|
||||
``Who composed Rhapsody in Blue?`` ≈0.05, zero FTS hits)
|
||||
settles deflected with NO ``budget_exhausted`` key, NO ``tool``
|
||||
frame and NO Continue button.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document
|
||||
from app.rag.agent import (
|
||||
ROUND_CAP_FALLBACK_ANSWER,
|
||||
ROUND_CAP_WRAPUP_INSTRUCTION,
|
||||
TOOL_BUDGET_LAST,
|
||||
TOOL_BUDGET_REMAINING,
|
||||
TOOL_BUDGET_SPENT,
|
||||
)
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
from tests.e2e.mock_llm import ROUND_CAP_CONTINUE_ANSWER, embed_text
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
#: This module app binds its OWN port (the conftest session app
|
||||
#: owns its port in a combined run — a same-port second uvicorn
|
||||
#: dies on bind and would drive the wrong server; the
|
||||
#: test_big_read_progress.py convention, env-overridable).
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_TOOLBUDGET", "8129"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
#: The cap the whole suite runs under (the module app's env
|
||||
#: override): the cap flow burns exactly three executed rounds —
|
||||
#: the trailer math runs over a 3-round budget (2 → 1 → spent).
|
||||
MAX_ROUNDS = 3
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Seed + questions (see the module docstring for the gate notes)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
SEED_SOURCE = "docs"
|
||||
SEED_PATH = "homelab/kubernetes.md"
|
||||
SEED_SP = f"{SEED_SOURCE}/{SEED_PATH}"
|
||||
KUB_CONTENT = (REPO / "tests" / "fixtures" / "docs" / SEED_PATH).read_text()
|
||||
|
||||
#: The cap-flow question: the tokens clear the mock-calibrated
|
||||
#: gate (≈0.44 against the mock's token-overlap embeddings, past
|
||||
#: the 0.30 threshold — HIGH without any FTS help), and it carries
|
||||
#: ``ROUND_CAP_TRIGGER``.
|
||||
CAP_QUESTION = (
|
||||
"Explain my kubernetes cluster: talos linux nodes, cilium "
|
||||
"networking, local-path-provisioner and longhorn storage. "
|
||||
"run out of tool calls"
|
||||
)
|
||||
assert "run out of tool calls" in CAP_QUESTION.lower()
|
||||
|
||||
#: The normal grounded control (test 3 — the byte-identical
|
||||
#: ``done`` case): token overlap with the seed (≈0.21, past the
|
||||
#: 0.15 lexical floor) + FTS hits — grounded, no trigger, no tool
|
||||
#: calls (the mock answers plain).
|
||||
CONTROL_QUESTION = "How is my Kubernetes cluster set up?"
|
||||
|
||||
#: The negative (test 7 — A8 LOW): token-disjoint from the KB
|
||||
#: (≈0.05, zero FTS hits — deflected; the phase-112 calibrated
|
||||
#: question).
|
||||
DEFLECT_QUESTION = "Who composed Rhapsody in Blue?"
|
||||
|
||||
STORAGE_KEY = "bor.chat.v1"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# The module app (conftest env shape + the BOR_AGENT_MAX_ROUNDS=3
|
||||
# swap)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int) -> Iterator[str]:
|
||||
"""The app under test — the conftest's session-app env shape
|
||||
(including the phase-61/62 leak guards) with one swap:
|
||||
``BOR_AGENT_MAX_ROUNDS=3`` (the cap flow's denominator — the
|
||||
trailer math runs over a 3-round budget)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# The E2E mock's token-overlap embeddings have their own score
|
||||
# distribution (phase 09) — the mock-calibrated values (the
|
||||
# conftest pattern).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits; the real retry budget (the
|
||||
# conftest pattern).
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# Phase 61/62 leak guards (the conftest pattern): force the code
|
||||
# defaults so an operator's local (gitignored) ``.env`` cannot
|
||||
# leak into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
# The suite's swap: the cap flow burns exactly three executed
|
||||
# rounds (the trailer math: 2 → 1 → spent).
|
||||
env["BOR_AGENT_MAX_ROUNDS"] = str(MAX_ROUNDS)
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# DB seeding (TRUNCATE-then-seed, cf. test_failed_turn_retry.py)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed(db: Session) -> None:
|
||||
"""The single fixture document (see the module docstring)."""
|
||||
md = Document(
|
||||
source=SEED_SOURCE,
|
||||
path=SEED_PATH,
|
||||
full_path=f"/tmp/{SEED_PATH}",
|
||||
title="Kubernetes",
|
||||
content=KUB_CONTENT,
|
||||
content_hash=hashlib.sha256(KUB_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
db.add(md)
|
||||
db.flush()
|
||||
# One chunk carrying the mock's own embedding → genuine token
|
||||
# overlap for the grounded questions (the FTS path carries them
|
||||
# to HIGH).
|
||||
db.add(
|
||||
Chunk(
|
||||
document_id=md.id,
|
||||
position=0,
|
||||
content=KUB_CONTENT,
|
||||
embedding=embed_text(KUB_CONTENT),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _reset_db() -> None:
|
||||
"""Truncate the KB (plus the prompt-shaping tables), then
|
||||
re-seed.
|
||||
|
||||
``steering_notes`` / ``kb_overview`` are truncated too, so the
|
||||
prompts are byte-stable regardless of leftovers from other
|
||||
suites.
|
||||
"""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
_seed(db)
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clean_chats() -> Iterator[None]:
|
||||
"""The send auto-saves a ``saved_chats`` row per turn (phase 55) —
|
||||
truncate around every test so the suite starts from (and leaves)
|
||||
an empty deployment (the phase-80/103 isolation pattern)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE saved_chats"))
|
||||
db.commit()
|
||||
yield
|
||||
# The auto-save is fire-and-forget (the browser never awaits the
|
||||
# PUT): a short margin lets the last turn's PUT land server-side
|
||||
# before the TRUNCATE, so the teardown never 500s an in-flight
|
||||
# upsert (the "Could not refresh instance" race). This runs AFTER
|
||||
# the page closes (LIFO finalization), so the PUT is either done
|
||||
# or aborted — never in flight.
|
||||
time.sleep(0.75)
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE saved_chats"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the SSE capture — the phase-37/67 pattern) + the
|
||||
# mock's request-log capture
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_sse_hook(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page, terminal: str = "done") -> list[dict]:
|
||||
"""The captured SSE frames of the CURRENT turn, once the
|
||||
*terminal* frame lands."""
|
||||
deadline = time.monotonic() + 30.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == terminal for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `{terminal}` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _stored(page: Page) -> dict:
|
||||
"""The persisted ``bor.chat.v1`` conversation (localStorage — the
|
||||
phase-14 store the restore reads on refresh)."""
|
||||
raw = page.evaluate(f"() => localStorage.getItem('{STORAGE_KEY}')")
|
||||
assert raw is not None, "the conversation key must exist in localStorage"
|
||||
return json.loads(raw)
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button
|
||||
recovered (the phase-48 settle convention)."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text(
|
||||
"", timeout=30_000
|
||||
)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
|
||||
def _mock_capture_reset(mock_port: int) -> None:
|
||||
"""Clear the mock's request capture so a suite asserts on
|
||||
EXACTLY the requests of the turn it is about to drive (the
|
||||
phase-123 capture pair)."""
|
||||
r = httpx.post(
|
||||
f"http://127.0.0.1:{mock_port}/v1/e2e/captured/reset", timeout=5
|
||||
)
|
||||
assert r.status_code == 200
|
||||
|
||||
|
||||
def _mock_captured(mock_port: int) -> list[dict]:
|
||||
"""The mock's recorded chat-completions request bodies (oldest
|
||||
first) — the wire-level pin surface for test 1."""
|
||||
r = httpx.get(
|
||||
f"http://127.0.0.1:{mock_port}/v1/e2e/captured", timeout=5
|
||||
)
|
||||
assert r.status_code == 200
|
||||
return r.json()
|
||||
|
||||
|
||||
def _tool_msgs(messages: list[dict]) -> list[str]:
|
||||
"""The ``tool``-role result contents of one request, in order."""
|
||||
return [
|
||||
str(m.get("content") or "")
|
||||
for m in messages
|
||||
if m.get("role") == "tool"
|
||||
]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The budget trailer on every executed tool result (wire level)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_tool_results_carry_the_remaining_count(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_reset_db()
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_mock_capture_reset(mock_llm)
|
||||
|
||||
_submit(page, CAP_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# Wire level (the mock's recorded request log — the app's
|
||||
# EXACT requests): exactly four chat requests — three executed
|
||||
# tool rounds + the forced final.
|
||||
bodies = _mock_captured(mock_llm)
|
||||
assert len(bodies) == 4, len(bodies)
|
||||
|
||||
# Request 1: no tool results yet; the tools are offered.
|
||||
assert bodies[0].get("tools"), "request 1 must offer the tools"
|
||||
assert _tool_msgs(bodies[0]["messages"]) == []
|
||||
|
||||
# Round 1's result (in request 2): the decreasing trailer —
|
||||
# ``[tool calls remaining: 2]`` (two of three left after the
|
||||
# call it is reading).
|
||||
msgs2 = _tool_msgs(bodies[1]["messages"])
|
||||
assert len(msgs2) == 1, msgs2
|
||||
assert msgs2[0].endswith(TOOL_BUDGET_REMAINING.format(n=2)), msgs2[0]
|
||||
|
||||
# Round 2's result (in request 3): the last-call line (one left
|
||||
# — "make it count, then answer").
|
||||
msgs3 = _tool_msgs(bodies[2]["messages"])
|
||||
assert len(msgs3) == 2, msgs3
|
||||
assert msgs3[0].endswith(TOOL_BUDGET_REMAINING.format(n=2)), msgs3[0]
|
||||
assert msgs3[1].endswith(TOOL_BUDGET_LAST), msgs3[1]
|
||||
|
||||
# The boundary call's result (in request 4 — the forced final):
|
||||
# the SPENT line (the budget is gone; say "continue").
|
||||
msgs4 = _tool_msgs(bodies[3]["messages"])
|
||||
assert len(msgs4) == 3, msgs4
|
||||
assert msgs4[2].endswith(TOOL_BUDGET_SPENT), msgs4[2]
|
||||
|
||||
# The mock's incrementing ids (the flow's stateless
|
||||
# derivation): call_cap_1 → call_cap_2 → call_cap_3 in the
|
||||
# assistant messages.
|
||||
asst_ids = [
|
||||
bodies[i + 1]["messages"][2 * i + 2]["tool_calls"][0]["id"]
|
||||
for i in range(3)
|
||||
]
|
||||
assert asst_ids == ["call_cap_1", "call_cap_2", "call_cap_3"], asst_ids
|
||||
|
||||
# The final request is the no-tools one, coached: no ``tools``
|
||||
# key (or null), the system message carries the wrap-up
|
||||
# instruction, and the user question + the tool history stay in
|
||||
# place.
|
||||
assert bodies[3].get("tools") is None
|
||||
systems = [
|
||||
str(m.get("content") or "")
|
||||
for m in bodies[3]["messages"]
|
||||
if m.get("role") == "system"
|
||||
]
|
||||
assert any(ROUND_CAP_WRAPUP_INSTRUCTION in s for s in systems), systems
|
||||
users = [
|
||||
str(m.get("content") or "")
|
||||
for m in bodies[3]["messages"]
|
||||
if m.get("role") == "user"
|
||||
]
|
||||
assert CAP_QUESTION in users, users
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. Past the cap: zero tool lines after the cap, the fallback
|
||||
# answer, no error
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_capped_turn_never_shows_tool_lines_after_the_cap(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_reset_db()
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, CAP_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
|
||||
# The turn settles with ``done`` — never the terminal ``error``
|
||||
# frame (the pre-phase-160 cap path's MalformedReplyError class
|
||||
# is gone; the budget fallback is the terminal).
|
||||
assert not [f for f in frames if f.get("type") == "error"], frames
|
||||
assert frames[-1]["type"] == "done"
|
||||
|
||||
# Wire: EXACTLY three ``tool`` frames — one per executed round
|
||||
# (the cap). The dump call (the forced final's structured
|
||||
# ``read`` on the seed path) produces NO frame — it was
|
||||
# rejected (never executed, never shown).
|
||||
tool_frames = [f for f in frames if f.get("type") == "tool"]
|
||||
assert len(tool_frames) == MAX_ROUNDS, tool_frames
|
||||
assert all(f["name"] == "ls" for f in tool_frames), tool_frames
|
||||
assert not [f for f in tool_frames if f.get("argument") == SEED_SP]
|
||||
|
||||
# UI: exactly three tool lines (all the no-arg top-level
|
||||
# listings); the dump call's ``Reading <seed path>`` line is
|
||||
# NOWHERE — the rejected call never renders.
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(3)
|
||||
for i in range(3):
|
||||
expect(lines.nth(i)).to_contain_text("Listing documents")
|
||||
msg_text = page.locator(".msg.brain").last.text_content() or ""
|
||||
assert "Reading " not in msg_text, msg_text
|
||||
|
||||
# The answer: the deterministic backstop line (the mock streams
|
||||
# no content on the forced final — only the rejected dump
|
||||
# call) — non-empty and readable.
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(ROUND_CAP_FALLBACK_ANSWER)
|
||||
|
||||
# No error banner (the turn settled, never the failure path).
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 3. done.budget_exhausted: true exactly on the capped turn, ABSENT
|
||||
# (never false) on the normal grounded control
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_done_flags_budget_exhausted(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_reset_db()
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_sse_hook(page)
|
||||
|
||||
# The capped turn: ``done`` carries ``budget_exhausted: true``.
|
||||
_submit(page, CAP_QUESTION)
|
||||
_wait_settled(page)
|
||||
frames = _frames(page)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["budget_exhausted"] is True, done
|
||||
assert done["deflected"] is False
|
||||
# The flagged answer offers the Continue affordance (task 04).
|
||||
expect(page.locator(".continue-btn")).to_have_count(1)
|
||||
|
||||
# The normal grounded control: the ``done`` frame carries NO
|
||||
# ``budget_exhausted`` key at all (absent, never false — the
|
||||
# byte-identical off rule, PLAN §13).
|
||||
page.evaluate("() => { window.__sseFrames = []; }")
|
||||
_submit(page, CONTROL_QUESTION)
|
||||
_wait_settled(page)
|
||||
frames = _frames(page)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert "budget_exhausted" not in done, done
|
||||
assert done["deflected"] is False
|
||||
|
||||
# The newer turn's landing cleared the capped answer's Continue
|
||||
# button (the remove-first pattern — a stale "continue" would
|
||||
# append to a history the conversation moved past).
|
||||
expect(page.locator(".continue-btn")).to_have_count(0)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 4. Continue researching: one fresh turn, the real continuation
|
||||
# answer, the record shape
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_continue_button_runs_a_fresh_turn(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_reset_db()
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, CAP_QUESTION)
|
||||
_wait_settled(page)
|
||||
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
|
||||
ROUND_CAP_FALLBACK_ANSWER
|
||||
)
|
||||
|
||||
# The flagged answer offers **Continue researching**.
|
||||
btn = page.locator(".continue-btn")
|
||||
expect(btn).to_have_count(1)
|
||||
expect(btn).to_contain_text("Continue researching")
|
||||
|
||||
# Fresh capture window: the continuation turn's frames only
|
||||
# (the cap turn's frames are still in the hook's buffer).
|
||||
page.evaluate("() => { window.__sseFrames = []; }")
|
||||
btn.click()
|
||||
|
||||
# Exactly ONE ``continue`` user bubble (the plain send).
|
||||
expect(page.locator(".msg.user .bubble")).to_have_count(2)
|
||||
expect(page.locator(".msg.user .bubble").last).to_have_text("continue")
|
||||
|
||||
# … and the mock's REAL continuation answer (a fixed sentence
|
||||
# naming the seed document — not a tool call).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(ROUND_CAP_CONTINUE_ANSWER, timeout=30_000)
|
||||
_wait_settled(page)
|
||||
|
||||
# The continuation turn streams NO tool frames — the earlier
|
||||
# tool lines do not duplicate (still exactly three).
|
||||
frames = _frames(page)
|
||||
assert [f for f in frames if f.get("type") == "tool"] == []
|
||||
expect(page.locator(".msg.brain .tool-call")).to_have_count(3)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
# A FRESH turn: the budget is whole again — the flag is absent
|
||||
# (the continuation turn deflects — the literal "continue"
|
||||
# shares no tokens with the KB, so the A8 gate is LOW).
|
||||
assert "budget_exhausted" not in done, done
|
||||
assert done["deflected"] is True
|
||||
|
||||
# The record: [user, brain(flagged), user("continue"), brain].
|
||||
stored = _stored(page)
|
||||
assert [m["who"] for m in stored["messages"]] == [
|
||||
"user",
|
||||
"brain",
|
||||
"user",
|
||||
"brain",
|
||||
]
|
||||
assert stored["messages"][1]["budget_exhausted"] is True
|
||||
assert stored["messages"][1]["text"] == ROUND_CAP_FALLBACK_ANSWER
|
||||
assert stored["messages"][2]["text"] == "continue"
|
||||
assert ROUND_CAP_CONTINUE_ANSWER in stored["messages"][3]["text"]
|
||||
assert not stored["messages"][3].get("budget_exhausted")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 5. Single-shot: double-clicking Continue starts ONE turn
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_continue_is_inert_while_streaming_and_single_shot(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_reset_db()
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, CAP_QUESTION)
|
||||
_wait_settled(page)
|
||||
expect(page.locator(".continue-btn")).to_have_count(1)
|
||||
|
||||
# Count the ``/api/chat`` POSTs (the single-shot pin) — the
|
||||
# exact path (the auto-save's ``PUT /api/chats/…`` must not
|
||||
# match: ``/api/chats`` is not ``/api/chat``).
|
||||
posts: list[int] = []
|
||||
page.on(
|
||||
"request",
|
||||
lambda req: (
|
||||
posts.append(1)
|
||||
if req.method == "POST"
|
||||
and req.url.split("?", 1)[0].endswith("/api/chat")
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
# The double-click: the FIRST click runs the handler (removes
|
||||
# every ``.continue-btn`` and starts the turn); the SECOND
|
||||
# click lands after the removal — no second target (and the
|
||||
# in-flight no-op guard would swallow it anyway) — ONE turn.
|
||||
page.locator(".continue-btn").dblclick()
|
||||
_wait_settled(page)
|
||||
assert len(posts) == 1, posts
|
||||
|
||||
# ONE ``continue`` user bubble + ONE continuation answer (no
|
||||
# double-send: a second turn would add a second user bubble +
|
||||
# a second brain answer).
|
||||
expect(page.locator(".msg.user .bubble")).to_have_count(2)
|
||||
expect(page.locator(".msg.brain")).to_have_count(2)
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(ROUND_CAP_CONTINUE_ANSWER, timeout=30_000)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 6. The flag survives a refresh (the restore path)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_flag_survives_a_refresh(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_reset_db()
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, CAP_QUESTION)
|
||||
_wait_settled(page)
|
||||
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
|
||||
ROUND_CAP_FALLBACK_ANSWER
|
||||
)
|
||||
expect(page.locator(".continue-btn")).to_have_count(1)
|
||||
|
||||
# The refresh: the flagged answer restores WITH the Continue
|
||||
# button (the restore path keys on ``m.budget_exhausted``).
|
||||
page.reload()
|
||||
expect(page.locator("#messages > .msg")).to_have_count(2)
|
||||
expect(page.locator(".msg.user .bubble")).to_have_text(CAP_QUESTION)
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(ROUND_CAP_FALLBACK_ANSWER, timeout=30_000)
|
||||
btn = page.locator(".continue-btn")
|
||||
expect(btn).to_have_count(1)
|
||||
expect(btn).to_contain_text("Continue researching")
|
||||
|
||||
# The restored button WORKS: clicking it runs the continuation
|
||||
# turn (the mock's continuation answer streams).
|
||||
btn.click()
|
||||
expect(page.locator(".msg.user .bubble")).to_have_count(2)
|
||||
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
|
||||
ROUND_CAP_CONTINUE_ANSWER, timeout=30_000
|
||||
)
|
||||
_wait_settled(page)
|
||||
|
||||
# The record: [user, brain(flagged), user("continue"), brain].
|
||||
stored = _stored(page)
|
||||
assert [m["who"] for m in stored["messages"]] == [
|
||||
"user",
|
||||
"brain",
|
||||
"user",
|
||||
"brain",
|
||||
]
|
||||
assert stored["messages"][1]["budget_exhausted"] is True
|
||||
assert stored["messages"][2]["text"] == "continue"
|
||||
assert ROUND_CAP_CONTINUE_ANSWER in stored["messages"][3]["text"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 7. Negative: a deflected turn — no flag, no Continue button
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_deflected_turn_gets_no_continue_button(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
_reset_db()
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_sse_hook(page)
|
||||
|
||||
_submit(page, DEFLECT_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
# Settled DEFLECTED (A8 LOW), with NO ``budget_exhausted`` key
|
||||
# (the flag is cap-reached turns only) and NO ``tool`` frame
|
||||
# (a tools=None request never emits one).
|
||||
assert done["deflected"] is True, done
|
||||
assert "budget_exhausted" not in done, done
|
||||
assert [f for f in frames if f.get("type") == "tool"] == []
|
||||
|
||||
# The deflected bubble — the honest copy, and NO Continue
|
||||
# button (the affordance is cap-reached answers only).
|
||||
last = page.locator(".msg.brain").last
|
||||
expect(last).to_have_class(re.compile(r"is-deflected"))
|
||||
expect(page.locator(".msg.brain .tool-call")).to_have_count(0)
|
||||
expect(page.locator(".continue-btn")).to_have_count(0)
|
||||
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
|
||||
"I haven't done anything like that"
|
||||
)
|
||||
@@ -320,13 +320,13 @@ def test_ls_top_level_lists_registered_sources_through_run_agent(
|
||||
holder, llm = _run_call(db, "ls", {})
|
||||
|
||||
assert llm.requests[0][1] == AGENT_TOOLS
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"2 sources:\n"
|
||||
"\n"
|
||||
"Deployments — 1 documents\n"
|
||||
"Homelab — 2 documents\n"
|
||||
" Home lab notes."
|
||||
)
|
||||
)
|
||||
assert holder.tool_calls == 1
|
||||
assert holder.read_docs == []
|
||||
|
||||
@@ -352,7 +352,7 @@ def test_ls_top_level_empty_registry_through_run_agent(
|
||||
db.commit()
|
||||
|
||||
holder, llm = _run_call(db, "ls", {})
|
||||
assert llm.requests[1][0][3]["content"] == "0 sources:"
|
||||
assert llm.requests[1][0][3]["content"].startswith("0 sources:")
|
||||
assert holder.tool_calls == 1
|
||||
|
||||
|
||||
@@ -375,20 +375,20 @@ def test_ls_source_scope_lists_root_folder_through_run_agent(kb, registry, db) -
|
||||
|
||||
holder, llm = _run_call(db, "ls", {"path": "Homelab"})
|
||||
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"Homelab — 1 documents, 2 folders:\n"
|
||||
"\n"
|
||||
" backups/ — 2 documents: Backup notes.\n"
|
||||
" networking/ — 1 documents\n"
|
||||
"\n"
|
||||
"source: Homelab | path: readme.md | title: Readme | date: 2024-06-15"
|
||||
)
|
||||
)
|
||||
assert holder.tool_calls == 1
|
||||
assert holder.read_docs == []
|
||||
|
||||
# A registered source with no documents: the header line alone.
|
||||
holder0, llm0 = _run_call(db, "ls", {"path": "Deployments"})
|
||||
assert llm0.requests[1][0][3]["content"] == "Deployments — 0 documents, 0 folders:"
|
||||
assert llm0.requests[1][0][3]["content"].startswith("Deployments — 0 documents, 0 folders:")
|
||||
assert holder0.tool_calls == 1
|
||||
|
||||
|
||||
@@ -405,23 +405,23 @@ def test_ls_nested_folder_scope_drills_one_level_through_run_agent(
|
||||
|
||||
holder, llm = _run_call(db, "ls", {"path": "Homelab/networking"})
|
||||
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"Homelab/networking — 0 documents, 2 folders:\n"
|
||||
"\n"
|
||||
" networking/lan/ — 2 documents\n"
|
||||
" networking/vpn/ — 1 documents"
|
||||
)
|
||||
)
|
||||
assert holder.tool_calls == 1
|
||||
assert holder.read_docs == []
|
||||
|
||||
# One level deeper.
|
||||
holder2, llm2 = _run_call(db, "ls", {"path": "Homelab/networking/lan"})
|
||||
assert llm2.requests[1][0][3]["content"] == (
|
||||
assert llm2.requests[1][0][3]["content"].startswith(
|
||||
"Homelab/networking/lan — 2 documents, 0 folders:\n"
|
||||
"\n"
|
||||
"source: Homelab | path: networking/lan/a.md | title: A | date: 2024-06-15\n"
|
||||
"source: Homelab | path: networking/lan/b.md | title: B | date: 2024-06-15"
|
||||
)
|
||||
)
|
||||
assert holder2.tool_calls == 1
|
||||
|
||||
|
||||
@@ -434,7 +434,9 @@ def test_ls_folder_file_cap_through_run_agent(kb, registry, db) -> None:
|
||||
|
||||
holder, llm = _run_call(db, "ls", {"path": "Homelab/big"})
|
||||
|
||||
content = llm.requests[1][0][3]["content"]
|
||||
# Phase 160 (task 01): the budget-telemetry trailer rides the
|
||||
# listing — the raw listing is the RAW part (the trailer stripped).
|
||||
content = llm.requests[1][0][3]["content"].removesuffix(agent._budget_note(9))
|
||||
lines = content.splitlines()
|
||||
assert lines[0] == "Homelab/big — 51 documents, 0 folders:"
|
||||
assert lines[2] == (
|
||||
@@ -460,9 +462,8 @@ def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None
|
||||
|
||||
holder, llm = _run_call(db, "ls", {"path": "Ghost"})
|
||||
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"]
|
||||
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="Ghost")
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="Ghost")
|
||||
)
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
|
||||
@@ -477,9 +478,8 @@ def test_ls_path_like_scope_unknown_source_gets_no_source_refusal(kb, src, db) -
|
||||
|
||||
holder, llm = _run_call(db, "ls", {"path": "app/rag/importer.py"})
|
||||
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"]
|
||||
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="app")
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="app")
|
||||
)
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||
@@ -500,10 +500,10 @@ def test_ls_unknown_folder_gets_not_a_folder_with_parents_subfolders(
|
||||
|
||||
holder, llm = _run_call(db, "ls", {"path": "Homelab/netwoking"})
|
||||
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"'Homelab/netwoking' is not a folder — Homelab has: "
|
||||
"backups/ containers/ networking/"
|
||||
)
|
||||
)
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||
|
||||
@@ -523,11 +523,11 @@ def test_read_combined_path_through_run_agent(kb, db) -> None:
|
||||
|
||||
# Phase 106 (D5): the date rides every read — the SECOND line (the
|
||||
# first line stays the byte-identical header).
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"Document Alpha/deep/nested/doc.md:\n"
|
||||
"date: 2024-06-15\n"
|
||||
"FULL-TEXT"
|
||||
)
|
||||
)
|
||||
assert holder.tool_calls == 1
|
||||
# SEC-14-04: short-lived session loads fresh copies
|
||||
assert [d.id for d in holder.read_docs] == [created.id]
|
||||
@@ -542,8 +542,8 @@ def test_read_bare_source_name_refused_through_run_agent(kb, db) -> None:
|
||||
|
||||
holder, llm = _run_call(db, "read", {"path": "Alpha"})
|
||||
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"] == "No document at 'Alpha' — check the ls output."
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"No document at 'Alpha' — check the ls output."
|
||||
)
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
|
||||
@@ -558,9 +558,8 @@ def test_read_unknown_combined_path_refused_through_run_agent(kb, db) -> None:
|
||||
|
||||
holder, llm = _run_call(db, "read", {"path": "Alpha/nope/deep.md"})
|
||||
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"]
|
||||
== "No document at 'Alpha/nope/deep.md' — check the ls output."
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"No document at 'Alpha/nope/deep.md' — check the ls output."
|
||||
)
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
|
||||
@@ -592,19 +591,19 @@ def test_read_bare_path_single_source_suggestion_then_corrected_read(kb, db) ->
|
||||
# Round 1: the bare path resolves to no combined identity, but it IS
|
||||
# the indexed document's path — the refusal names the one combined
|
||||
# identity to use (not counted, the tools stay offered).
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"No document at 'deep/nested/doc.md' — "
|
||||
"did you mean 'Alpha/deep/nested/doc.md'?"
|
||||
)
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
# Round 2: the corrected combined identity succeeds — the full
|
||||
# content (plus the phase-106 D5 date line), the holder records the
|
||||
# row, and it counts.
|
||||
assert llm.requests[2][0][5]["content"] == (
|
||||
assert llm.requests[2][0][5]["content"].startswith(
|
||||
"Document Alpha/deep/nested/doc.md:\n"
|
||||
"date: 2024-06-15\n"
|
||||
"FULL-TEXT"
|
||||
)
|
||||
)
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
# SEC-14-04: short-lived session loads fresh copies
|
||||
assert [d.id for d in holder.read_docs] == [created.id]
|
||||
@@ -633,13 +632,13 @@ def test_read_bare_path_two_sources_one_of_suggestion_then_corrected_read(
|
||||
holder = AgentHolder()
|
||||
asyncio.run(_consume(cast("LLMClient", llm), lambda: SessionLocal(), holder))
|
||||
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"No document at 'shared/x.md' — did you mean one of: "
|
||||
"'Alpha/shared/x.md', 'Beta/shared/x.md'?"
|
||||
)
|
||||
assert llm.requests[2][0][5]["content"] == (
|
||||
)
|
||||
assert llm.requests[2][0][5]["content"].startswith(
|
||||
"Document Alpha/shared/x.md:\ndate: 2024-06-15\nA-TEXT"
|
||||
)
|
||||
)
|
||||
# SEC-14-04: short-lived session loads fresh copies
|
||||
assert [d.id for d in holder.read_docs] == [a.id]
|
||||
assert holder.tool_calls == 1 # only the corrected read executed
|
||||
@@ -673,10 +672,10 @@ def test_grep_whole_kb_through_run_agent(kb, db) -> None:
|
||||
# Offered: the first request carries AGENT_TOOLS (the 3-tool list).
|
||||
assert llm.requests[0][1] == AGENT_TOOLS
|
||||
# Executed against the real DB: catalog order, grep-style lines.
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"Alpha/a/one.md:2: needle in one\n"
|
||||
"Beta/b/two.md:2: NEEDLE in two"
|
||||
)
|
||||
)
|
||||
assert holder.tool_calls == 1
|
||||
assert holder.read_docs == [] # locked A5: grep adds no context
|
||||
|
||||
@@ -689,7 +688,7 @@ def test_grep_scoped_through_run_agent(kb, db) -> None:
|
||||
holder, llm = _run_call(db, "grep", {"pattern": "needle", "path": "Alpha/a/one.md"})
|
||||
|
||||
# Only the named document is searched — the other one's hit is absent.
|
||||
assert llm.requests[1][0][3]["content"] == "Alpha/a/one.md:2: Needle here"
|
||||
assert llm.requests[1][0][3]["content"].startswith("Alpha/a/one.md:2: Needle here")
|
||||
assert holder.tool_calls == 1
|
||||
assert holder.read_docs == []
|
||||
|
||||
@@ -703,9 +702,8 @@ def test_grep_scoped_missing_doc_refused_through_run_agent(kb, db) -> None:
|
||||
|
||||
holder, llm = _run_call(db, "grep", {"pattern": "needle", "path": "Alpha/ghost.md"})
|
||||
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"]
|
||||
== "No document at 'Alpha/ghost.md' — check the ls output."
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"No document at 'Alpha/ghost.md' — check the ls output."
|
||||
)
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
|
||||
@@ -716,8 +714,8 @@ def test_grep_no_matches_through_run_agent(kb, db) -> None:
|
||||
|
||||
holder, llm = _run_call(db, "grep", {"pattern": "zebra"})
|
||||
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"No matches for 'zebra' in the knowledge base."
|
||||
)
|
||||
)
|
||||
assert holder.tool_calls == 1 # an executed grep with zero hits
|
||||
assert holder.read_docs == []
|
||||
|
||||
@@ -28,7 +28,7 @@ from sqlalchemy.orm import Session
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document, GitSource
|
||||
from app.rag.agent import AgentHolder, run_agent
|
||||
from app.rag.agent import AgentHolder, _budget_note, run_agent
|
||||
from app.rag.llm import LLMClient, RetryPiece, StreamPiece, ToolCallPiece, ToolResultPiece
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -160,7 +160,9 @@ def test_read_result_second_line_is_stored_date(kb, db) -> None:
|
||||
|
||||
holder, llm = _run_call(db, "read", {"path": "Alpha/deep/nested/doc.md"})
|
||||
|
||||
content = llm.requests[1][0][3]["content"]
|
||||
# Phase 160 (task 01): the trailer rides the read result — the raw
|
||||
# result is the RAW part (the trailer stripped).
|
||||
content = llm.requests[1][0][3]["content"].removesuffix(_budget_note(9))
|
||||
lines = content.splitlines()
|
||||
assert lines[0] == "Document Alpha/deep/nested/doc.md:" # byte-identical header
|
||||
assert lines[1] == f"date: {D2_STR}" # the STORED date (row's UTC date part)
|
||||
@@ -178,13 +180,13 @@ def test_read_result_date_is_the_row_date_not_a_constant(kb, db) -> None:
|
||||
db.commit()
|
||||
|
||||
holder_a, llm_a = _run_call(db, "read", {"path": "Alpha/a.md"})
|
||||
assert llm_a.requests[1][0][3]["content"] == (
|
||||
assert llm_a.requests[1][0][3]["content"].startswith(
|
||||
f"Document Alpha/a.md:\ndate: {D1_STR}\nA-TEXT"
|
||||
)
|
||||
)
|
||||
holder_b, llm_b = _run_call(db, "read", {"path": "Alpha/b.md"})
|
||||
assert llm_b.requests[1][0][3]["content"] == (
|
||||
assert llm_b.requests[1][0][3]["content"].startswith(
|
||||
f"Document Alpha/b.md:\ndate: {D3_STR}\nB-TEXT"
|
||||
)
|
||||
)
|
||||
# SEC-14-04: short-lived sessions
|
||||
assert [d.id for d in holder_a.read_docs] == [a.id]
|
||||
assert [d.id for d in holder_b.read_docs] == [b.id]
|
||||
@@ -207,7 +209,9 @@ def test_ls_drill_file_lines_carry_their_dates(kb, src, db) -> None:
|
||||
|
||||
holder, llm = _run_call(db, "ls", {"path": "Homelab"})
|
||||
|
||||
content = llm.requests[1][0][3]["content"]
|
||||
# Phase 160 (task 01): the trailer rides the listing — stripped for
|
||||
# the raw-part pins below.
|
||||
content = llm.requests[1][0][3]["content"].removesuffix(_budget_note(9))
|
||||
lines = content.splitlines()
|
||||
# The root level: one direct file (readme.md — D1) + the two
|
||||
# subfolder lines (date-free) + the date-free header.
|
||||
@@ -221,7 +225,7 @@ def test_ls_drill_file_lines_carry_their_dates(kb, src, db) -> None:
|
||||
|
||||
# Drill into backups: BOTH files list, each with its OWN date.
|
||||
holder2, llm2 = _run_call(db, "ls", {"path": "Homelab/backups"})
|
||||
lines2 = llm2.requests[1][0][3]["content"].splitlines()
|
||||
lines2 = llm2.requests[1][0][3]["content"].removesuffix(_budget_note(9)).splitlines()
|
||||
assert lines2[0] == "Homelab/backups — 2 documents, 0 folders:"
|
||||
assert lines2[2] == (
|
||||
f"source: Homelab | path: backups/cron.md | title: Cron | date: {D1_STR}"
|
||||
@@ -247,7 +251,7 @@ def test_ls_top_level_source_lines_carry_no_date(kb, db) -> None:
|
||||
|
||||
holder, llm = _run_call(db, "ls", {})
|
||||
|
||||
content = llm.requests[1][0][3]["content"]
|
||||
content = llm.requests[1][0][3]["content"].removesuffix(_budget_note(9))
|
||||
assert content == "1 sources:\n\nHomelab — 1 documents"
|
||||
assert "date" not in content
|
||||
assert holder.tool_calls == 1
|
||||
|
||||
@@ -82,6 +82,7 @@ class FakeRagLLM:
|
||||
stream_error: Exception | None = None,
|
||||
fail_mid_stream: bool = False,
|
||||
tool_script: list[list[StreamPiece | ToolCallPiece]] | None = None,
|
||||
no_tools_script: list[list[StreamPiece | ToolCallPiece]] | None = None,
|
||||
embed_fail_count: int = 0,
|
||||
stream_fail_count: int = 0,
|
||||
answer_sequence: list[str] | None = None,
|
||||
@@ -123,6 +124,18 @@ class FakeRagLLM:
|
||||
#: thinking + answer stream below, so a deflected turn through this
|
||||
#: fake is byte-identical to the plain fake's output.
|
||||
self.tool_script: list[list[StreamPiece | ToolCallPiece]] = list(tool_script or [])
|
||||
#: Phase 160 (task 02): canned piece lists for requests made
|
||||
#: WITHOUT tools (``tools=None``) — the deflected turn, the
|
||||
#: cap-forced final, the kill-switch single-request path, and the
|
||||
#: phase-71 recovery. Empty (the default) keeps the pre-phase
|
||||
#: behavior: a tools=None request yields the thinking + answer
|
||||
#: stream below, so every existing test is untouched. When set,
|
||||
#: the *i*-th tools=None request consumes ``no_tools_script[i]``
|
||||
#: (so a mock can make a tools=None request return structured
|
||||
#: ``ToolCallPiece``s — the documented ``lite`` behavior).
|
||||
self.no_tools_script: list[list[StreamPiece | ToolCallPiece]] = list(
|
||||
no_tools_script or []
|
||||
)
|
||||
|
||||
async def embed(
|
||||
self,
|
||||
@@ -197,6 +210,8 @@ class FakeRagLLM:
|
||||
raw: list[StreamPiece | ToolCallPiece]
|
||||
if tools is not None and self.tool_script:
|
||||
raw = self.tool_script.pop(0)
|
||||
elif tools is None and self.no_tools_script:
|
||||
raw = self.no_tools_script.pop(0)
|
||||
elif self.fail_mid_stream:
|
||||
raw = [StreamPiece("content", "partial ")]
|
||||
mid_stream_drop = True
|
||||
@@ -980,9 +995,10 @@ marker in the model's context."""
|
||||
m for r in scripted.seen_messages for m in r if m.get("role") == "tool"
|
||||
]
|
||||
assert tool_msgs, "the executed read must be appended as a tool message"
|
||||
assert (
|
||||
tool_msgs[-1]["content"]
|
||||
== "Document docs/fits.md:\ndate: 2024-06-15\n" + content
|
||||
# Phase 160 (task 01): the budget-telemetry trailer rides the
|
||||
# read result — the raw result is the RAW part.
|
||||
assert tool_msgs[-1]["content"].startswith(
|
||||
"Document docs/fits.md:\ndate: 2024-06-15\n" + content
|
||||
)
|
||||
assert TRUNCATION_MARKER not in tool_msgs[-1]["content"]
|
||||
# Still a successful read.
|
||||
@@ -1165,9 +1181,10 @@ def test_untruncated_read_streams_no_tool_result_frame(
|
||||
m for r in scripted.seen_messages for m in r if m.get("role") == "tool"
|
||||
]
|
||||
assert tool_msgs
|
||||
assert (
|
||||
tool_msgs[-1]["content"]
|
||||
== "Document docs/big-read.md:\ndate: 2024-06-15\n" + content
|
||||
# Phase 160 (task 01): the budget-telemetry trailer rides the
|
||||
# read result — the raw result is the RAW part.
|
||||
assert tool_msgs[-1]["content"].startswith(
|
||||
"Document docs/big-read.md:\ndate: 2024-06-15\n" + content
|
||||
)
|
||||
assert TRUNCATION_MARKER not in tool_msgs[-1]["content"]
|
||||
|
||||
@@ -1500,6 +1517,176 @@ def test_zero_max_rounds_reproduce_pre_phase_single_request(
|
||||
assert "scaffold_stripped=0" in lines[-1] # phase 71: uniform clean-turn field
|
||||
|
||||
|
||||
def test_cap_path_rejects_late_calls_and_fallbacks(
|
||||
client, db, seeded_kb: FakeRagLLM, monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""A grounded turn with ``agent_max_rounds=1`` whose forced final
|
||||
(``tools=None``) returns a ``ToolCallPiece`` (the documented
|
||||
``lite`` behavior — a no-tools request that still comes back with
|
||||
structured calls): the executed round streams ``tool`` ×1, then
|
||||
NOTHING of the sort — no second ``tool`` frame, no ``error`` frame —
|
||||
and the turn settles with ``done`` whose delta text is the
|
||||
``ROUND_CAP_FALLBACK_ANSWER`` (the model gave no visible content on
|
||||
the forced final). The late call is never executed and never counted
|
||||
in ``tool_calls`` (the executed-only log field stays 1)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[[ToolCallPiece(id="call_1", name="ls", arguments={})]],
|
||||
no_tools_script=[
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_f1",
|
||||
name="read",
|
||||
arguments={"path": "docs/homelab/kubernetes.md"},
|
||||
)
|
||||
]
|
||||
],
|
||||
)
|
||||
live = get_settings()
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=live.relevance_threshold,
|
||||
agent_max_rounds=1,
|
||||
),
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
caplog.set_level(logging.INFO, logger="app.chat")
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
types = [f["type"] for f in frames]
|
||||
# Only the executed round 1 call streams as a tool frame; the forced
|
||||
# final's call is rejected — no second tool frame, no error frame.
|
||||
assert types.count("tool") == 1
|
||||
assert "error" not in types
|
||||
# The turn settles with done; the answer is the budget fallback.
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False
|
||||
delta_text = "".join(f.get("text", "") for f in frames if f["type"] == "delta")
|
||||
assert delta_text == agent.ROUND_CAP_FALLBACK_ANSWER
|
||||
# The late ``read`` was never executed: ``tool_calls=1`` (the
|
||||
# executed round only) in the per-turn log line.
|
||||
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
assert lines and "tool_calls=1" in lines[-1]
|
||||
|
||||
|
||||
def test_cap_reached_turn_reports_budget_exhausted_and_log_fields(
|
||||
client, db, seeded_kb: FakeRagLLM, monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""A grounded turn with ``agent_max_rounds=1`` that hits the cap:
|
||||
the ``done`` frame carries ``"budget_exhausted": true`` (the
|
||||
additive key — the ``data:`` line the client sees) and the
|
||||
per-turn log line reads ``rounds=1 cap_hit=1`` (the cap hit,
|
||||
named — the owner can grep how often the cap bites)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[[ToolCallPiece(id="call_1", name="ls", arguments={})]],
|
||||
no_tools_script=[[StreamPiece("content", "the wrap-up answer")]],
|
||||
)
|
||||
live = get_settings()
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=live.relevance_threshold,
|
||||
agent_max_rounds=1,
|
||||
),
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
caplog.set_level(logging.INFO, logger="app.chat")
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False
|
||||
assert done["budget_exhausted"] is True # the cap-reached flag
|
||||
assert "error" not in [f["type"] for f in frames]
|
||||
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
assert lines and "rounds=1 cap_hit=1" in lines[-1]
|
||||
assert "tool_calls=1" in lines[-1]
|
||||
|
||||
|
||||
def test_in_cap_and_deflected_turns_report_no_budget_exhausted(
|
||||
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""An ordinary grounded turn (answered in-cap) and a deflected
|
||||
turn: the ``done`` frames carry NO ``budget_exhausted`` key
|
||||
(ABSENT, never ``false`` — the byte-identical-off contract), and
|
||||
the per-turn log lines read ``cap_hit=0`` — the grounded one
|
||||
with ``rounds=`` the executed rounds (2), the deflected one
|
||||
``rounds=0 cap_hit=0`` (the agent never runs)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
||||
[ToolCallPiece(id="call_2", name="ls", arguments={})],
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
caplog.set_level(logging.INFO, logger="app.chat")
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
_, _, deflected_frames = _stream_chat(client, OFF_TOPIC)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False
|
||||
assert "budget_exhausted" not in done # ABSENT, never false
|
||||
d_done = deflected_frames[-1]
|
||||
assert d_done["type"] == "done"
|
||||
assert d_done["deflected"] is True
|
||||
assert "budget_exhausted" not in d_done
|
||||
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
assert len(lines) == 2
|
||||
assert "tool_calls=2 rounds=2 cap_hit=0" in lines[0] # the grounded turn
|
||||
assert "rounds=0 cap_hit=0" in lines[1] # the deflected turn
|
||||
|
||||
|
||||
def test_deflected_turn_rejects_structured_calls(
|
||||
client, db, seeded_kb: FakeRagLLM
|
||||
) -> None:
|
||||
"""A deflected turn (``tools=None``, A8) whose model returns a
|
||||
structured ``tool_calls`` (the documented ``lite`` behavior) emits
|
||||
NO ``tool`` frame — the call is rejected at the transport
|
||||
(``_pump`` drops it, ``tools_offered=False``), never rendered, never
|
||||
executed. The turn still settles (``done``) with the model's
|
||||
content (the rejection only drops the call, not the answer)."""
|
||||
scripted = FakeRagLLM(
|
||||
no_tools_script=[
|
||||
[
|
||||
ToolCallPiece(id="call_1", name="ls", arguments={}),
|
||||
StreamPiece("content", "the deflected answer"),
|
||||
]
|
||||
],
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, OFF_TOPIC)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
types = [f["type"] for f in frames]
|
||||
assert not any(t == "tool" for t in types) # no tool frame
|
||||
assert "error" not in types
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is True
|
||||
# The model's content still streamed (the rejection only drops the
|
||||
# call, not the answer).
|
||||
delta_text = "".join(f.get("text", "") for f in frames if f["type"] == "delta")
|
||||
assert delta_text == "the deflected answer"
|
||||
# One request, no tools key (the deflected direct path, A8).
|
||||
assert len(scripted.seen_messages) == 1
|
||||
assert scripted.seen_tools == [None]
|
||||
|
||||
|
||||
def test_tool_execution_db_failure_yields_error_event(
|
||||
client, db, seeded_kb: FakeRagLLM, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
|
||||
@@ -232,6 +232,12 @@ def _expect(records: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"stopped": m.get("stopped"),
|
||||
"failed": m.get("failed"),
|
||||
"error": m.get("error"),
|
||||
# Phase 160 (task 04): the budget-exhausted marker —
|
||||
# explicit null where the record does not carry it (the
|
||||
# bor.chat.v1 shape is explicit-null for every optional
|
||||
# key; a record WITH the key round-trips it byte-
|
||||
# identically — the pins below).
|
||||
"budget_exhausted": m.get("budget_exhausted"),
|
||||
}
|
||||
for m in records
|
||||
]
|
||||
@@ -464,8 +470,10 @@ def test_create_round_trips_full_brain_record(admin_client: TestClient) -> None:
|
||||
assert r.status_code == 201
|
||||
# The bor.chat.v1-shaped payload round-trips losslessly: every
|
||||
# optional key (sources/related/deflected/suggestions/thinking/
|
||||
# tools/stopped) survives identical.
|
||||
assert r.json()["messages"][1] == FULL_BRAIN
|
||||
# tools/stopped/failed/error/budget_exhausted — the phase-160
|
||||
# additive marker, explicit null where absent) survives identical
|
||||
# (the _expect full stored shape).
|
||||
assert r.json()["messages"][1] == _expect([_user(FIRST_QUESTION), FULL_BRAIN])[1]
|
||||
|
||||
|
||||
def test_create_rejects_empty_messages(admin_client: TestClient) -> None:
|
||||
@@ -1036,6 +1044,128 @@ def test_shared_serve_includes_the_brain_answer_image_ref(
|
||||
assert "answer_image" not in messages[0]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 160 (task 04, TODO L4): the budget-exhausted marker — the
|
||||
# persisted answer that ran out of tool calls. The done-time
|
||||
# auto-save (``rememberBrainTurn``) persists the done frame's
|
||||
# ``budget_exhausted`` into the brain record's ``budget_exhausted``
|
||||
# key (the ``answer_image`` precedent — the field must exist so the
|
||||
# ``extra="forbid"`` boundary does NOT 422 those saves; the
|
||||
# phase-113 lesson, applied up front), and the restore path re-
|
||||
# renders the "Continue researching" button from it (task 04 — the
|
||||
# marker is a client restore hint only, never a server behavior; no
|
||||
# migration: ``saved_chats.messages`` is raw JSONB, phase 50, and
|
||||
# ``app/api/chats.py``'s logic is unchanged). These round-trips are
|
||||
# that regression pin.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_create_round_trips_the_budget_exhausted_marker(
|
||||
admin_client: TestClient,
|
||||
) -> None:
|
||||
"""The phase-113 regression pin (the A2 quiet-failure path): a
|
||||
brain record carrying ``budget_exhausted: true`` →
|
||||
``POST /api/chats`` is 201 (NOT a silent 422) and the admin GET
|
||||
round-trips the record BYTE-FOR-BYTE through Postgres (the
|
||||
``_expect`` full bor.chat.v1 shape — the restore path re-renders
|
||||
the Continue button from the marker). The user record carries the
|
||||
key as an explicit null (it never did, and the stored shape is
|
||||
explicit-null for every optional key)."""
|
||||
brain = {**FULL_BRAIN, "budget_exhausted": True}
|
||||
r = admin_client.post(
|
||||
"/api/chats",
|
||||
json={"messages": [_user(FIRST_QUESTION), brain]},
|
||||
)
|
||||
assert r.status_code == 201, f"expected 201, got {r.status_code}: {r.text}"
|
||||
body = r.json()
|
||||
assert body["messages"][1]["budget_exhausted"] is True
|
||||
assert body["messages"][0]["budget_exhausted"] is None
|
||||
|
||||
got = admin_client.get(f"/api/chats/{body['id']}")
|
||||
assert got.status_code == 200
|
||||
assert got.json()["messages"] == _expect(
|
||||
[_user(FIRST_QUESTION), {**FULL_BRAIN, "budget_exhausted": True}]
|
||||
)
|
||||
|
||||
|
||||
def test_create_without_budget_exhausted_marker_round_trips_unchanged(
|
||||
admin_client: TestClient,
|
||||
) -> None:
|
||||
"""A record WITHOUT the marker (every non-cap turn, every pre-
|
||||
phase record) round-trips with the key as an explicit null — the
|
||||
stored shape never gains a value (the byte-identical contract,
|
||||
the phase-50 precedent: the key is absent from the client's
|
||||
``bor.chat.v1`` record via the ``undefined``-drops-the-key
|
||||
pattern, and explicit-null here keeps the restore path null-safe).
|
||||
The ``FULL_BRAIN`` shape (no marker) is the pin: its stored and
|
||||
served form is byte-identical to pre-phase-160 apart from the
|
||||
one new null key."""
|
||||
r = admin_client.post(
|
||||
"/api/chats",
|
||||
json={"messages": [_user(FIRST_QUESTION), FULL_BRAIN]},
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
body = r.json()
|
||||
assert body["messages"][1]["budget_exhausted"] is None
|
||||
got = admin_client.get(f"/api/chats/{body['id']}")
|
||||
assert got.status_code == 200
|
||||
assert got.json()["messages"] == _expect([_user(FIRST_QUESTION), FULL_BRAIN])
|
||||
|
||||
|
||||
def test_shared_serve_includes_the_budget_exhausted_marker(
|
||||
admin_client: TestClient,
|
||||
) -> None:
|
||||
"""The shared view is faithful (the data is in the public
|
||||
snapshot), but the control is NOT: the shared page
|
||||
(``/shared/<token>``, rendered server-side by ``shared.js`` —
|
||||
never by ``app.js``, owner-locked zero interactive controls)
|
||||
IGNORES the key by construction — no Continue control renders for
|
||||
a flagged record (work item 8; pinned in
|
||||
``test_frontend_tool_budget.py::test_shared_page_never_shows_
|
||||
the_continue_control``). The user record stays with the explicit
|
||||
null."""
|
||||
brain = {**FULL_BRAIN, "budget_exhausted": True}
|
||||
r = admin_client.post(
|
||||
"/api/chats",
|
||||
json={"messages": [_user(FIRST_QUESTION), brain]},
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
share_url = _share(admin_client, r.json()["id"])["share_url"]
|
||||
|
||||
anon = TestClient(fastapi_app) # fresh jar: truly anonymous
|
||||
got = anon.get(f"/api{share_url}")
|
||||
assert got.status_code == 200
|
||||
messages = got.json()["messages"]
|
||||
assert messages[1]["budget_exhausted"] is True # faithful snapshot
|
||||
assert messages[0]["budget_exhausted"] is None
|
||||
|
||||
|
||||
def test_put_round_trips_the_budget_exhausted_marker(
|
||||
admin_client: TestClient,
|
||||
) -> None:
|
||||
"""The re-Save upsert (the auto-save path the client's restore
|
||||
relies on) round-trips the marker byte-identically too: a PUT
|
||||
with the flagged record replaces the stored JSONB and the 200
|
||||
body + the GET both carry it (a refreshed chat page restores the
|
||||
Continue affordance from the persisted record)."""
|
||||
created = admin_client.post(
|
||||
"/api/chats",
|
||||
json={"messages": [_user(FIRST_QUESTION), FULL_BRAIN]},
|
||||
).json()
|
||||
brain = {**FULL_BRAIN, "budget_exhausted": True}
|
||||
r = admin_client.put(
|
||||
f"/api/chats/{created['id']}",
|
||||
json={"messages": [_user(FIRST_QUESTION), brain]},
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["messages"][1]["budget_exhausted"] is True
|
||||
got = admin_client.get(f"/api/chats/{created['id']}")
|
||||
assert got.status_code == 200
|
||||
assert got.json()["messages"] == _expect(
|
||||
[_user(FIRST_QUESTION), {**FULL_BRAIN, "budget_exhausted": True}]
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 159 (task 02): edit-and-resend — the SAVE shape. The client
|
||||
# edits the last question: the record is replaced in place, EVERY
|
||||
|
||||
@@ -443,7 +443,10 @@ def test_plain_read_truncates_then_range_read_answers_the_find_traps_question(
|
||||
assert isinstance(plain_result["content"], str)
|
||||
notice = READ_TRUNCATION_NOTICE.format(shown=READ_CAP, total=len(doc.content))
|
||||
assert plain_result["content"].startswith(f"Document {IDENTITY}:\n")
|
||||
assert plain_result["content"].endswith(f"\n{TRUNCATION_MARKER}\n{notice}")
|
||||
# Phase 160 (task 01): the budget-telemetry trailer rides the
|
||||
# truncated result — the marker + notice end the RAW part.
|
||||
raw = plain_result["content"].removesuffix(agent._budget_note(9))
|
||||
assert raw.endswith(f"\n{TRUNCATION_MARKER}\n{notice}")
|
||||
# The truncation surfaced to the API layer as a ToolResultPiece
|
||||
# (the SSE ``tool_result`` frame's wire value).
|
||||
truncations = [p for p in pieces if isinstance(p, ToolResultPiece)]
|
||||
|
||||
@@ -411,7 +411,7 @@ def test_two_ranges_of_one_doc_both_reach_the_model(two_range_doc: Document) ->
|
||||
# the byte-identical line is now true).
|
||||
reread_result = llm.requests[3][0][7]
|
||||
assert reread_result["role"] == "tool"
|
||||
assert reread_result["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert reread_result["content"].startswith(agent.ALREADY_IN_CONTEXT)
|
||||
|
||||
# Round 4: the answer echoes BOTH markers (the bug: pre-phase,
|
||||
# the echo could contain only marker A — range B's slice never
|
||||
@@ -485,7 +485,10 @@ def test_truncated_plain_read_then_range_b_proceeds_and_plain_reread_teaches(
|
||||
assert isinstance(plain_result["content"], str)
|
||||
notice = READ_TRUNCATION_NOTICE.format(shown=cap, total=len(doc.content))
|
||||
assert plain_result["content"].startswith(f"Document {IDENTITY}:\n")
|
||||
assert plain_result["content"].endswith(f"\n{TRUNCATION_MARKER}\n{notice}")
|
||||
# Phase 160 (task 01): the budget-telemetry trailer rides the
|
||||
# truncated result — the marker + notice end the RAW part.
|
||||
raw = plain_result["content"].removesuffix(agent._budget_note(9))
|
||||
assert raw.endswith(f"\n{TRUNCATION_MARKER}\n{notice}")
|
||||
assert MARKER_A in plain_result["content"]
|
||||
truncations = [p for p in pieces if isinstance(p, ToolResultPiece)]
|
||||
assert [(p.argument, p.chars_shown, p.chars_total) for p in truncations] == [
|
||||
@@ -512,12 +515,12 @@ def test_truncated_plain_read_then_range_b_proceeds_and_plain_reread_teaches(
|
||||
# resolved doc.
|
||||
reread_result = llm.requests[3][0][7]
|
||||
assert reread_result["role"] == "tool"
|
||||
assert reread_result["content"] == agent.REREAD_PARTIAL.format(
|
||||
assert reread_result["content"].startswith(agent.REREAD_PARTIAL.format(
|
||||
covered=f"1–{covered}, {marker_b_line - 10}–{marker_b_line - 10 + RANGE_B_LIMIT - 1}",
|
||||
source=SOURCE,
|
||||
path=DOC_PATH,
|
||||
total=total_lines,
|
||||
)
|
||||
))
|
||||
|
||||
# Round 4: the answer echoes both markers.
|
||||
answer = _answer(pieces)
|
||||
|
||||
+572
-235
File diff suppressed because it is too large
Load Diff
@@ -1578,6 +1578,8 @@ def test_endpoint_log_line_records_suggested_after_summary_hits(
|
||||
"sources=",
|
||||
"thinking_chars=",
|
||||
"tool_calls=",
|
||||
"rounds=",
|
||||
"cap_hit=",
|
||||
"total_ms=",
|
||||
"retries=",
|
||||
"scaffold_stripped=",
|
||||
|
||||
@@ -491,7 +491,9 @@ def test_chat_message_image_defaults_to_none_and_is_omitted() -> None:
|
||||
assert m.image is None
|
||||
dumped = m.model_dump()
|
||||
assert "image" not in dumped # ABSENT, never ``null`` (the phase-50 contract)
|
||||
# and every pre-phase key set is untouched
|
||||
# and every pre-phase key set is untouched (the phase-160 additive
|
||||
# ``budget_exhausted`` marker joins as an explicit null — the
|
||||
# phase-120 ``failed`` / phase-48 ``stopped`` style).
|
||||
assert set(dumped) == {
|
||||
"who",
|
||||
"text",
|
||||
@@ -504,6 +506,7 @@ def test_chat_message_image_defaults_to_none_and_is_omitted() -> None:
|
||||
"stopped",
|
||||
"failed",
|
||||
"error",
|
||||
"budget_exhausted",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -173,10 +173,14 @@ def test_pre_phase_record_round_trips_byte_identical_on_existing_keys() -> None:
|
||||
# Every pre-phase key survives byte-identical…
|
||||
for key, value in pre_phase.items():
|
||||
assert dumped[key] == value, f"pre-phase key {key!r} changed: {dumped[key]!r}"
|
||||
# …and the ONLY additions are the two new keys as explicit nulls.
|
||||
assert set(dumped) == set(pre_phase) | {"failed", "error"}
|
||||
# …and the ONLY additions are the new keys as explicit nulls
|
||||
# (the phase-120/48 ``failed`` / ``error`` pair, and the phase-160
|
||||
# additive ``budget_exhausted`` marker — the client's restore
|
||||
# hint, explicit null where absent).
|
||||
assert set(dumped) == set(pre_phase) | {"failed", "error", "budget_exhausted"}
|
||||
assert dumped["failed"] is None
|
||||
assert dumped["error"] is None
|
||||
assert dumped["budget_exhausted"] is None
|
||||
# The stored shape re-validates (round-trip through the DB JSONB).
|
||||
assert ChatMessage.model_validate(dumped).model_dump() == dumped
|
||||
|
||||
@@ -197,4 +201,5 @@ def test_minimal_user_record_unchanged() -> None:
|
||||
"stopped": None,
|
||||
"failed": None,
|
||||
"error": None,
|
||||
"budget_exhausted": None,
|
||||
}
|
||||
|
||||
@@ -796,9 +796,15 @@ def test_retry_last_turn_is_byte_unchanged() -> None:
|
||||
#: task-01 idle-settle ``markLastEditable()`` mark pass (work item 3
|
||||
#: call sites — the in-flight guard's at-rest landing), and the
|
||||
#: task-03 aborted-settle mark pair (the 300s guard's ``aborted``
|
||||
#: catch branch — a timed-out turn must not strand the question).
|
||||
#: The ``reask`` path is consumed as-is. A diff here is a contract
|
||||
#: violation for the phase's remaining tasks.
|
||||
#: catch branch — a timed-out turn must not strand the question);
|
||||
#: plus the phase-160 (task 04, TODO L4) sanctioned additions:
|
||||
#: the done settle's ``budget_exhausted`` meta key (the additive
|
||||
#: done-frame marker — ``undefined`` drops the key, so every non-cap
|
||||
#: record stays byte-identical) and the flag-gated
|
||||
#: ``appendContinueButton`` (the "Continue researching" affordance —
|
||||
#: ABSENT on every other turn). The ``reask`` path is consumed as-is.
|
||||
#: A diff here is a contract violation for the phase's remaining
|
||||
#: tasks.
|
||||
PINNED_RUN_TURN = """function runTurn(text, { reask = false, image = null } = {}) {
|
||||
if (!reask) {
|
||||
// Phase 123 (task 02, TODO L6): the attached image rides the user
|
||||
@@ -1181,6 +1187,12 @@ PINNED_RUN_TURN = """function runTurn(text, { reask = false, image = null } = {}
|
||||
// row exactly as it looked live.
|
||||
related: ev.related?.length ? ev.related : undefined,
|
||||
suggestions: ev.suggestions,
|
||||
// Phase 160 (task 04, TODO L4): the cap-reached marker
|
||||
// persists with the turn — `undefined` drops the key from
|
||||
// the JSON (the house optional-meta pattern, the
|
||||
// answer_image line above is the precedent), so every
|
||||
// non-cap record stays byte-identical to pre-phase.
|
||||
budget_exhausted: ev.budget_exhausted || undefined,
|
||||
}, leavePartialIndex);
|
||||
lastBrainWrap = wrap; // this bubble is now the last brain answer
|
||||
// Phase 130 (task 02): the row order is the DOM order — the
|
||||
@@ -1191,6 +1203,20 @@ PINNED_RUN_TURN = """function runTurn(text, { reask = false, image = null } = {}
|
||||
appendTuneButton(wrap); // every completed brain bubble is tunable
|
||||
markLastRetryable(); // phase 49: the Retry button is last-bubble-only
|
||||
markLastEditable(); // phase 159: the done turn's question is editable (the in-flight guard no-ops here — the runTurn idle settle lands it)
|
||||
// Phase 160 (task 04, TODO L4): a cap-reached answer offers
|
||||
// the one-click continuation — first, every earlier bubble's
|
||||
// continue button is cleared (the markLastRetryable
|
||||
// remove-first pattern: the moment a newer turn lands, an old
|
||||
// bubble's stale affordance is gone — its "continue" would
|
||||
// append to a history the conversation already moved past),
|
||||
// then the button is appended ONLY when the done frame is
|
||||
// flagged (ABSENT on every other turn, so the button never
|
||||
// appears elsewhere); the row order stays Tune → Retry →
|
||||
// Continue → Save-as-doc (the insertion order phase 130
|
||||
// relies on — markLastEditable above acts on the user
|
||||
// bubble, so the brain bubble's row order is unchanged).
|
||||
messagesEl.querySelectorAll(".continue-btn").forEach((b) => b.remove());
|
||||
if (ev.budget_exhausted) appendContinueButton(wrap);
|
||||
// Phase 59: the RAW persisted markdown (never the rendered
|
||||
// HTML) — exactly the string rememberBrainTurn stores above,
|
||||
// so a reload (the restore path) offers the identical draft.
|
||||
@@ -1397,13 +1423,16 @@ PINNED_RUN_TURN = """function runTurn(text, { reask = false, image = null } = {}
|
||||
|
||||
|
||||
def test_run_turn_is_pinned_at_phase_159_state() -> None:
|
||||
"""``runTurn`` is pinned at exactly the state phase 159 left it
|
||||
(task 02 consumes its ``reask`` path AS-IS — the phase's "NOT
|
||||
touched" contract): the full source must match the pin — the
|
||||
``reask`` path, the turn-local resets, the SSE contract, the
|
||||
save points, and the sanctioned additions (the task-01
|
||||
``lastUserWrap`` assignment + the idle-settle mark pass, and the
|
||||
task-03 aborted-settle mark pair) included."""
|
||||
"""``runTurn`` is pinned at exactly the state phase 159 left it,
|
||||
plus phase 160 (task 04)'s sanctioned additions (task 02 consumes
|
||||
its ``reask`` path AS-IS — the phase's "NOT touched" contract;
|
||||
the phase-160 additions are the done settle's additive
|
||||
``budget_exhausted`` meta key + the flag-gated Continue button):
|
||||
the full source must match the pin — the ``reask`` path, the
|
||||
turn-local resets, the SSE contract, the save points, and the
|
||||
sanctioned additions (the task-01 ``lastUserWrap`` assignment +
|
||||
the idle-settle mark pass, the task-03 aborted-settle mark pair,
|
||||
and the phase-160 done-settle pair) included."""
|
||||
assert _fn_source(_js(), "runTurn") == PINNED_RUN_TURN
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,546 @@
|
||||
"""Unit: the phase-160 (task 04, TODO L4) "Continue researching" button.
|
||||
|
||||
When a turn ends because its tool budget was spent (the ``done``
|
||||
frame's additive ``budget_exhausted`` key, task 03 — ABSENT on every
|
||||
non-cap turn), the answer bubble offers **Continue researching**: one
|
||||
click sends the next turn (a plain send — the budget is per-turn, so
|
||||
the fresh turn has a whole new round budget) and the marker persists
|
||||
with the record so a page refresh restores the affordance. The shared
|
||||
page never shows the button (read-only by construction).
|
||||
|
||||
Pins (the house source-pin pattern — no browser):
|
||||
|
||||
* ``CONTINUE_ICON`` / ``CONTINUE_QUESTION`` — the inline aria-hidden
|
||||
SVG in the RETRY_ICON stroke family (no icon font, no CDN — A11)
|
||||
and the literal continuation word (what the model is taught to ask
|
||||
the user for);
|
||||
* ``appendContinueButton`` — the house pattern through
|
||||
``getActionsRow`` (the role=list row joins the button as a
|
||||
listitem), the one-per-bubble guard, click → ``continueTurn()``;
|
||||
* ``continueTurn`` — the ``retryLastTurn`` discipline: no-op while a
|
||||
turn is in flight, every ``.continue-btn`` removed FIRST, then a
|
||||
PLAIN ``runTurn(CONTINUE_QUESTION)`` (no ``reask``);
|
||||
* the ``done`` settle — ``budget_exhausted`` rides
|
||||
``rememberBrainTurn``'s meta (``undefined`` drops the key, so every
|
||||
non-cap record stays byte-identical) and the button is appended
|
||||
ONLY when the frame is flagged, in the row order Tune → Retry →
|
||||
Continue → Save-as-doc (the insertion order phase 130 relies on);
|
||||
* the restore branch — keys on ``m.budget_exhausted``;
|
||||
* ``startNewChat`` needs nothing — its list reset drops the buttons;
|
||||
* ``retryLastTurn`` / ``markLastRetryable`` / ``rememberBrainTurn``
|
||||
are byte-unchanged (the phase-120 shared pin for the first);
|
||||
* the shared page (work item 8) — ``shared.html`` never loads
|
||||
``app.js`` and ``shared.js`` builds no buttons / actions rows, so a
|
||||
flagged record renders without any Continue control there.
|
||||
|
||||
The browser-level assertions (the live click running a fresh turn, the
|
||||
reload restoring the button, the shared page's read-only render) are
|
||||
E2E-gated by tests/e2e/test_tool_budget_telemetry.py (task 05).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from tests.unit.test_frontend_failed_turn import PINNED_RETRY_LAST_TURN
|
||||
|
||||
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
|
||||
APP_JS = FRONTEND / "assets" / "app.js"
|
||||
STYLES_CSS = FRONTEND / "assets" / "styles.css"
|
||||
SHARED_HTML = FRONTEND / "shared.html"
|
||||
SHARED_JS = FRONTEND / "assets" / "shared.js"
|
||||
|
||||
|
||||
def _js() -> str:
|
||||
return APP_JS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _css() -> str:
|
||||
return STYLES_CSS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _fn_body(js: str, header: str) -> str:
|
||||
"""The full source of the function whose header is *header* —
|
||||
brace-matched from the first ``{`` OUTSIDE the parameter list
|
||||
(paren depth 0, the house convention), so destructured parameters
|
||||
are skipped (the pinned functions carry no string literal with a
|
||||
stray brace)."""
|
||||
start = js.index(header)
|
||||
i = start
|
||||
paren = 0
|
||||
while i < len(js):
|
||||
c = js[i]
|
||||
if c == "(":
|
||||
paren += 1
|
||||
elif c == ")":
|
||||
paren -= 1
|
||||
elif c == "{" and paren == 0:
|
||||
break
|
||||
i += 1
|
||||
depth = 0
|
||||
for j in range(i, len(js)):
|
||||
c = js[j]
|
||||
if c == "{":
|
||||
depth += 1
|
||||
elif c == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return js[start : j + 1]
|
||||
raise AssertionError(f"unbalanced braces in {header!r}")
|
||||
|
||||
|
||||
def _done_branch(js: str) -> str:
|
||||
"""The SSE ``done`` branch — from the branch test to the next
|
||||
``else if`` (flat block, the slice is exact)."""
|
||||
start = js.index('ev.type === "done"')
|
||||
return js[start : js.index('ev.type === "tool_result"', start)]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# the glyph + the continuation word
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_continue_icon_is_an_inline_svg_in_the_stroke_family() -> None:
|
||||
"""``CONTINUE_ICON`` is the inline ``aria-hidden`` SVG in the
|
||||
RETRY_ICON stroke family (task 01: 24×24, ``fill="none"``,
|
||||
``stroke="currentColor"``, ``stroke-width="1.7"`` — the CSS themes
|
||||
the stroke) — no icon font, no CDN (A11)."""
|
||||
js = _js()
|
||||
start = js.index("const CONTINUE_ICON =")
|
||||
seg = js[start :]
|
||||
q1 = seg.index("'")
|
||||
q2 = seg.index("'", q1 + 1)
|
||||
literal = seg[q1 + 1 : q2] # the first string literal = the inline SVG
|
||||
for attr in (
|
||||
'aria-hidden="true"',
|
||||
'viewBox="0 0 24 24"',
|
||||
'fill="none"',
|
||||
'stroke="currentColor"',
|
||||
'stroke-width="1.7"',
|
||||
):
|
||||
assert attr in literal, f"the glyph keeps the shared family: {attr!r}"
|
||||
assert literal.startswith("<svg") and literal.endswith("</svg>")
|
||||
|
||||
|
||||
def test_continue_question_is_the_literal_word_continue() -> None:
|
||||
"""``CONTINUE_QUESTION`` is the literal word ``continue`` — the
|
||||
word the model is taught to ask the user for (task 02's
|
||||
ROUND_CAP_WRAPUP_INSTRUCTION / the spent-budget trailer), so the
|
||||
visible bubble, the typed equivalent, and the model's own
|
||||
suggestion all agree."""
|
||||
js = _js()
|
||||
assert 'const CONTINUE_QUESTION = "continue";' in js
|
||||
|
||||
|
||||
def test_continue_icon_is_defined() -> None:
|
||||
"""EXACTLY ONE ``CONTINUE_ICON`` constant (the glyph is shared by
|
||||
the button, never duplicated)."""
|
||||
js = _js()
|
||||
assert js.count("const CONTINUE_ICON =") == 1
|
||||
assert js.count("const CONTINUE_QUESTION =") == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# appendContinueButton — the house pattern
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_append_continue_button_joins_the_actions_row() -> None:
|
||||
"""``appendContinueButton(wrap)`` keeps the house pattern (the
|
||||
appendRetryButton shape, task 02): the getActionsRow lookup (the
|
||||
dedicated actions row, NEVER the chips row), the null no-op, the
|
||||
one-per-bubble guard (``.continue-btn``), the role=list →
|
||||
listitem ARIA join, the icon + the visible "Continue researching"
|
||||
label (the label carries the accessible name), and the click →
|
||||
``continueTurn()`` binding."""
|
||||
js = _js()
|
||||
body = _fn_body(js, "function appendContinueButton(wrap) {")
|
||||
assert "getActionsRow(wrap)" in body, "the shared get-or-create helper"
|
||||
assert "if (!meta) return" in body, "no-op without a .msg-body"
|
||||
assert 'querySelector(".continue-btn")' in body, (
|
||||
"the one-per-bubble guard"
|
||||
)
|
||||
assert 'meta.getAttribute("role") === "list"' in body, (
|
||||
"the role=list check (the row is always role=list)"
|
||||
)
|
||||
assert 'setAttribute("role", "listitem")' in body, "the listitem join"
|
||||
assert 'btn.className = "continue-btn"' in body
|
||||
assert 'btn.innerHTML = CONTINUE_ICON + "<span>Continue researching</span>"' in body, (
|
||||
"icon + the visible label (the accessible name)"
|
||||
)
|
||||
assert "btn.addEventListener(\"click\", () => continueTurn())" in body, (
|
||||
"the click handler — a plain continueTurn() (no wrap, no args)"
|
||||
)
|
||||
assert "meta.appendChild(btn)" in body
|
||||
# NOT admin-gated, like Retry (chat is public).
|
||||
assert "isAdmin" not in body
|
||||
|
||||
|
||||
def test_continue_button_lands_after_the_retry_button() -> None:
|
||||
"""``appendContinueButton`` is defined DIRECTLY AFTER
|
||||
``appendRetryButton`` (task 02's placement — the sibling in the
|
||||
actions-row family, before the Edit button's block)."""
|
||||
js = _js()
|
||||
retry = js.index("function appendRetryButton(wrap) {")
|
||||
cont = js.index("function appendContinueButton(wrap) {")
|
||||
edit = js.index("function appendEditButton(wrap) {")
|
||||
assert retry < cont < edit
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# continueTurn — the retryLastTurn discipline
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_continue_turn_bails_and_clears_and_sends_plain() -> None:
|
||||
"""``continueTurn()`` (task 03): (1) no-op while a turn is in
|
||||
flight (the ``retryLastTurn`` guard — one turn at a time, no
|
||||
double-send); (2) clears EVERY rendered ``.continue-btn`` FIRST
|
||||
(the markLastRetryable remove-first pattern — a stale button on an
|
||||
earlier bubble is gone the moment the new turn's answer lands);
|
||||
(3) a PLAIN send — ``runTurn(CONTINUE_QUESTION)`` with NO
|
||||
``reask`` argument (the default ``false``): the user bubble
|
||||
appears, the record is pushed + saved at the ordinary save
|
||||
points, and the request carries the full history."""
|
||||
js = _js()
|
||||
body = _fn_body(js, "function continueTurn() {")
|
||||
assert (
|
||||
"if (uiState === UI_STATE.thinking || uiState === UI_STATE.streaming) return;"
|
||||
in body
|
||||
), "the in-flight no-op (the retryLastTurn guard)"
|
||||
assert (
|
||||
'messagesEl.querySelectorAll(".continue-btn").forEach((b) => b.remove());'
|
||||
in body
|
||||
), "every continue button is cleared before the send"
|
||||
assert "return runTurn(CONTINUE_QUESTION)" in body, (
|
||||
"a PLAIN send — the returned promise (the retryLastTurn "
|
||||
"convention)"
|
||||
)
|
||||
assert "reask" not in body, (
|
||||
"NO reask argument — a plain send (runTurn's default false)"
|
||||
)
|
||||
order = [
|
||||
body.index("UI_STATE.thinking"),
|
||||
body.index(".remove()"),
|
||||
body.index("runTurn(CONTINUE_QUESTION)"),
|
||||
]
|
||||
assert order == sorted(order), (
|
||||
"guard → clear-every-button → send, in that order"
|
||||
)
|
||||
|
||||
|
||||
def test_continue_turn_sends_one_plain_turn_with_full_history() -> None:
|
||||
"""The plain-send contract on the ``runTurn`` side (task 03's
|
||||
save-point list): with ``reask`` defaulting to false, the turn
|
||||
appends the user bubble, pushes ``{ who: "user", text }`` (no
|
||||
image — a text-only record), runs the ordinary save points
|
||||
(``saveConversation`` + ``persistConversation``), and the request's
|
||||
``history`` is the FULL ``conversation.slice(0, -1)`` — the model
|
||||
resumes with everything it already read."""
|
||||
js = _js()
|
||||
rt = _fn_body(js, "function runTurn(text, { reask = false, image = null } = {}) {")
|
||||
assert "if (!reask) {" in rt, "the default is a plain send"
|
||||
assert 'conversation.push(' in rt
|
||||
assert 'saveConversation();' in rt
|
||||
assert "persistConversation();" in rt
|
||||
assert "const history = conversation.slice(0, -1).map(" in rt, (
|
||||
"the request carries the full history (everything except the "
|
||||
"just-pushed question)"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# the done settle — the flag rides the record + the button is gated
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_done_handler_persists_the_flag_and_appends_the_button() -> None:
|
||||
"""The ``done`` settle (task 04): ``budget_exhausted`` rides
|
||||
``rememberBrainTurn``'s meta as
|
||||
``ev.budget_exhausted || undefined`` (the ``answer_image``
|
||||
precedent — ``undefined`` drops the key from the JSON, so every
|
||||
non-cap record stays byte-identical), every earlier bubble's
|
||||
continue button is cleared FIRST (the completion criterion —
|
||||
"neither does any earlier answer once a newer turn lands" — the
|
||||
markLastRetryable remove-first pattern), and the button is
|
||||
appended ONLY when the frame is flagged."""
|
||||
js = _js()
|
||||
branch = _done_branch(js)
|
||||
assert "budget_exhausted: ev.budget_exhausted || undefined" in branch, (
|
||||
"the flag persists with the turn (undefined drops the key)"
|
||||
)
|
||||
assert (
|
||||
'messagesEl.querySelectorAll(".continue-btn").forEach((b) => b.remove());'
|
||||
in branch
|
||||
), "every earlier bubble's continue button is cleared on the settle"
|
||||
assert "if (ev.budget_exhausted) appendContinueButton(wrap);" in branch, (
|
||||
"the button is appended ONLY on a flagged done"
|
||||
)
|
||||
order = [
|
||||
branch.index('querySelectorAll(".continue-btn")'),
|
||||
branch.index("appendContinueButton(wrap)"),
|
||||
]
|
||||
assert order == sorted(order), "clear-every-button → gated append"
|
||||
|
||||
|
||||
def test_done_branch_row_order_is_tune_retry_continue_save() -> None:
|
||||
"""The row order on a capped answer (task 04): Tune → Retry (the
|
||||
markLastRetryable re-append) → Continue (flag-gated) → Save-as-doc
|
||||
— the INSERTION order phase 130 relies on (the DOM order inside
|
||||
the row is the visual order)."""
|
||||
js = _js()
|
||||
branch = _done_branch(js)
|
||||
order = [
|
||||
branch.index("appendTuneButton(wrap);"),
|
||||
branch.index("markLastRetryable();"),
|
||||
branch.index("appendContinueButton(wrap);"),
|
||||
branch.index('appendSaveAsDocButton(wrap, finalText || acc || "…");'),
|
||||
]
|
||||
assert order == sorted(order), (
|
||||
"done: Tune → Retry → Continue → Save-as-doc (the insertion "
|
||||
"order phase 130 relies on)"
|
||||
)
|
||||
|
||||
|
||||
def test_continue_button_is_not_added_on_the_other_settle_paths() -> None:
|
||||
"""The Continue button is a CAP-REACHED-DONE affordance only: the
|
||||
zero-frame fallback, the stop finalize, and the failed-turn funnel
|
||||
(none of them sees a ``budget_exhausted`` frame — a failed or
|
||||
stopped turn never hit the cap) append NO continue button, and the
|
||||
``done`` branch's flag-gate is the single gate."""
|
||||
js = _js()
|
||||
assert js.count("appendContinueButton(wrap);") == 2, (
|
||||
"exactly TWO call sites (with the terminating semicolon — the "
|
||||
"function header ends in a brace, not a semicolon): the done "
|
||||
"settle + the restore path"
|
||||
)
|
||||
zero = js.index("if (!aborted && !wrap) {")
|
||||
block = js[zero : js.index("} catch (err) {", zero)]
|
||||
assert "appendContinueButton" not in block, (
|
||||
"the zero-frame fallback (a failed turn) adds no continue button"
|
||||
)
|
||||
funnel = _fn_body(
|
||||
js,
|
||||
"function finalizeFailedTurn(\n"
|
||||
" detail,\n"
|
||||
" { acc, thinking, tools, segments, wrap, leavePartialIndex }\n"
|
||||
") {",
|
||||
)
|
||||
assert "appendContinueButton" not in funnel, (
|
||||
"the failed-turn funnel adds no continue button"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# the restore path + startNewChat
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_restore_branch_keys_on_the_persisted_marker() -> None:
|
||||
"""The phase-14 restore (``renderStoredMessage``'s brain branch,
|
||||
task 05): the Continue affordance restores from the persisted
|
||||
``m.budget_exhausted`` marker — a reload keeps the button on a
|
||||
flagged answer (a restored ``history`` is exactly what the
|
||||
continuation turn needs), and a pre-phase / non-cap record (no
|
||||
key) restores byte-identically (no button). The append lands
|
||||
BEFORE the ``lastBrainWrap`` assignment, after the per-message
|
||||
appends (the actions row is the bubble's LAST row, as on the
|
||||
live done path)."""
|
||||
js = _js()
|
||||
body = _fn_body(js, "function renderStoredMessage(m) {")
|
||||
assert "if (m.budget_exhausted) appendContinueButton(wrap);" in body, (
|
||||
"the restore keys on the persisted marker"
|
||||
)
|
||||
order = [
|
||||
body.index("appendFailedNote(wrap, m.error);"),
|
||||
body.index("if (m.budget_exhausted) appendContinueButton(wrap);"),
|
||||
body.index("lastBrainWrap = wrap"),
|
||||
]
|
||||
assert order == sorted(order), (
|
||||
"restore: the per-message appends → the continue button → "
|
||||
"lastBrainWrap (the after-the-loop markLastRetryable lands the "
|
||||
"Retry in the same actions row, as on the live path)"
|
||||
)
|
||||
|
||||
|
||||
def test_start_new_chat_unchanged_drops_the_buttons_with_the_list() -> None:
|
||||
"""``startNewChat`` (task 05: needs nothing) — its list reset
|
||||
removes every ``.msg`` bubble, so the continue buttons die with
|
||||
the list (the phase-49 reasoning). The reset + the in-flight
|
||||
guard stay byte-pinned."""
|
||||
js = _js()
|
||||
body = _fn_body(js, "function startNewChat() {")
|
||||
assert (
|
||||
"if (uiState === UI_STATE.thinking || uiState === UI_STATE.streaming) return;"
|
||||
in body
|
||||
), "the in-flight guard stays"
|
||||
assert 'messagesEl.querySelectorAll(".msg").forEach((el) => el.remove());' in body, (
|
||||
"the list reset drops the buttons along with the list"
|
||||
)
|
||||
assert "appendContinueButton" not in body, (
|
||||
"no continue-button wiring was added to startNewChat"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# NOT touched — the phase's explicit contract (byte-pinned sources)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: The FULL source of ``markLastRetryable`` as of pre-phase-160 task
|
||||
#: 04 — the remove-then-append shape (remove every rendered
|
||||
#: .retry-btn FIRST, then re-append on the last brain bubble) is
|
||||
#: UNCHANGED: the continue button has its OWN clear pass in
|
||||
#: continueTurn, and markLastRetryable's targeting is untouched.
|
||||
PINNED_MARK_LAST_RETRYABLE = """function markLastRetryable() {
|
||||
messagesEl.querySelectorAll(".retry-btn").forEach((b) => b.remove());
|
||||
if (!lastBrainWrap) return;
|
||||
let lastIdx = -1;
|
||||
for (let i = conversation.length - 1; i >= 0; i -= 1) {
|
||||
if (conversation[i].who === "brain") {
|
||||
lastIdx = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
const prev = lastIdx > 0 ? conversation[lastIdx - 1] : null;
|
||||
if (!prev || prev.who !== "user") return;
|
||||
appendRetryButton(lastBrainWrap);
|
||||
}"""
|
||||
|
||||
#: The FULL source of ``rememberBrainTurn`` as of pre-phase-160 task
|
||||
#: 04 — the ``...meta`` spread means the new ``budget_exhausted``
|
||||
#: optional key rides the record with ZERO change to this helper
|
||||
#: (``undefined`` drops the key from the JSON, the house
|
||||
#: optional-meta pattern).
|
||||
PINNED_REMEMBER_BRAIN_TURN = """function rememberBrainTurn(rawText, meta, replaceIndex = -1) {
|
||||
const rec = { who: "brain", text: rawText || "…", ...meta };
|
||||
if (replaceIndex >= 0 && conversation[replaceIndex]?.who === "brain") {
|
||||
conversation[replaceIndex] = rec;
|
||||
} else {
|
||||
conversation.push(rec);
|
||||
}
|
||||
saveConversation();
|
||||
// Phase 55 (A2): the auto-save rides the brain save point — the row
|
||||
// updates with the (possibly replaced) brain turn + metadata. The
|
||||
// pagehide partial reuses this helper, so it rides the same path (no
|
||||
// extra wiring).
|
||||
persistConversation();
|
||||
}"""
|
||||
|
||||
|
||||
def test_mark_last_retryable_is_byte_unchanged() -> None:
|
||||
"""``markLastRetryable`` is byte-unchanged by phase 160 (task 04
|
||||
— the redo-in-place targeting is REUSED, not extended): the full
|
||||
source must match the pin (a diff is a contract violation)."""
|
||||
assert _fn_body(_js(), "function markLastRetryable() {") == PINNED_MARK_LAST_RETRYABLE
|
||||
|
||||
|
||||
def test_remember_brain_turn_is_byte_unchanged() -> None:
|
||||
"""``rememberBrainTurn`` is byte-unchanged by phase 160 (task 04 —
|
||||
the ``budget_exhausted`` meta key rides the existing ``...meta``
|
||||
spread, so the save point is consumed as-is): the full source must
|
||||
match the pin."""
|
||||
assert _fn_body(_js(), "function rememberBrainTurn(rawText, meta, replaceIndex = -1) {") == (
|
||||
PINNED_REMEMBER_BRAIN_TURN
|
||||
)
|
||||
|
||||
|
||||
def test_retry_last_turn_is_byte_unchanged() -> None:
|
||||
"""``retryLastTurn`` is byte-unchanged by phase 160 (task 04 — the
|
||||
continue button REUSES its discipline, the function is consumed
|
||||
as-is): the full source must match the phase-120 pin (the shared
|
||||
constant — one pin, no drift)."""
|
||||
assert _fn_body(_js(), "function retryLastTurn(wrap) {") == PINNED_RETRY_LAST_TURN
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# the shared page (work item 8) — the Continue control never renders there
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_shared_page_never_shows_the_continue_control() -> None:
|
||||
"""Work item 8 (no code change expected): the shared page
|
||||
(``/shared/<token>``, served server-side — ``shared.html``)
|
||||
renders the conversation through ``shared.js``'s OWN read-only
|
||||
renderer, never through ``app.js`` — so the ``budget_exhausted``
|
||||
key is IGNORED there by construction: the page loads no
|
||||
``app.js`` (no ``appendContinueButton`` in its script graph),
|
||||
and ``shared.js`` builds NO buttons, no forms, and no
|
||||
``.msg-actions`` row (owner-locked: zero interactive controls —
|
||||
the "no buttons, no forms, no links, no click handlers"
|
||||
contract), so a flagged record renders without any Continue
|
||||
affordance."""
|
||||
html = SHARED_HTML.read_text(encoding="utf-8")
|
||||
assert "app.js" not in html, (
|
||||
"the shared page never loads app.js — its action buttons "
|
||||
"(incl. Continue) are not in its script graph"
|
||||
)
|
||||
shared = SHARED_JS.read_text(encoding="utf-8")
|
||||
assert "continue-btn" not in shared
|
||||
assert "msg-actions" not in shared, (
|
||||
"shared.js never builds the actions row — zero controls"
|
||||
)
|
||||
assert 'createElement("button")' not in shared, (
|
||||
"owner-locked: NO buttons are ever created on the shared page"
|
||||
)
|
||||
assert 'addEventListener("click"' not in shared, (
|
||||
"owner-locked: NO click handlers on the shared page (the img "
|
||||
"load-failure degradation handler is not a control — the "
|
||||
"zero-interactive-controls contract)"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# styles.css — the .continue-btn family
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_continue_btn_css_is_the_retry_family() -> None:
|
||||
"""``.continue-btn`` is the exact visual family of ``.retry-btn``
|
||||
(task 07): the same pill size/spacing (``min-height: 44px``,
|
||||
``gap: 0.35rem``, ``padding: 0.35rem 0.8rem``,
|
||||
``border-radius: 999px``, ``border: 1px solid var(--line)``),
|
||||
the same neutral color (``--ink-soft`` → ``--ink`` on hover over
|
||||
``--brand-soft``), the 14px currentColor glyph, and the shared
|
||||
GLOBAL ``:focus-visible`` ring (styles.css L162 — the ring rule is
|
||||
not per-class, so no selector-list extension is needed). No
|
||||
animation of its own → the ``prefers-reduced-motion`` blocks need
|
||||
nothing for it (the family's parity — .retry-btn / .edit-btn have
|
||||
none either)."""
|
||||
css = _css()
|
||||
start = css.index(".continue-btn {")
|
||||
end = css.index("}", start)
|
||||
block = css[start : end + 1]
|
||||
for decl in (
|
||||
"display: inline-flex;",
|
||||
"gap: 0.35rem;",
|
||||
"min-height: 44px;",
|
||||
"padding: 0.35rem 0.8rem;",
|
||||
"border-radius: 999px;",
|
||||
"border: 1px solid var(--line);",
|
||||
"background: transparent;",
|
||||
"color: var(--ink-soft);",
|
||||
"font-weight: 600;",
|
||||
"font-size: 0.82rem;",
|
||||
"white-space: nowrap;",
|
||||
"cursor: pointer;",
|
||||
):
|
||||
assert decl in block, f"the pill family keeps {decl!r}"
|
||||
assert ".continue-btn svg { width: 14px; height: 14px; display: block; }" in css
|
||||
assert ".continue-btn:hover { background: var(--brand-soft); color: var(--ink); }" in css
|
||||
|
||||
|
||||
def test_continue_btn_mobile_block_min_height() -> None:
|
||||
"""The mobile block (task 07): ``.continue-btn`` joins the
|
||||
``.tune-btn`` / ``.retry-btn`` / ``.edit-btn`` 44px min-height
|
||||
lockstep next to ``.retry-btn`` — EXACTLY ONE such single-line
|
||||
rule (the base rule is a multi-line block, so the one-line form
|
||||
occurs only in the mobile media block)."""
|
||||
css = _css()
|
||||
assert css.count(".continue-btn { min-height: 44px; }") == 1
|
||||
idx = css.index(".continue-btn { min-height: 44px; }")
|
||||
# It sits beside the family's other lockstep lines (near them in
|
||||
# the same mobile block).
|
||||
retry = css.index(".retry-btn { min-height: 44px; }")
|
||||
assert abs(idx - retry) < 200, (
|
||||
"the continue line is beside .retry-btn in the mobile block"
|
||||
)
|
||||
@@ -565,6 +565,88 @@ def test_done_event_answer_image_ref_drops_its_own_null_image_url() -> None:
|
||||
assert "image_url" not in dump["answer_image"]
|
||||
|
||||
|
||||
# --- the ``budget_exhausted`` done key (phase 160, task 03) ---
|
||||
|
||||
|
||||
def test_done_event_without_budget_exhausted_drops_the_key() -> None:
|
||||
"""``budget_exhausted=False`` (the default — every deflected /
|
||||
in-cap / pre-phase turn) → ``model_dump()`` has NO
|
||||
``budget_exhausted`` key — ABSENT, never ``false`` — and a full
|
||||
done frame (sources + related + suggestions, no flag) serializes
|
||||
byte-identically to the pre-phase expectation (the
|
||||
byte-identical-off contract, PLAN §13)."""
|
||||
event = ChatDoneEvent(
|
||||
deflected=False,
|
||||
sources=[SourceRef(source="docs", path="a.md", title="A")],
|
||||
related=[SourceRef(source="docs", path="b.md", title="B")],
|
||||
suggestions=["Maybe try X?"],
|
||||
)
|
||||
dump = event.model_dump()
|
||||
assert "budget_exhausted" not in dump
|
||||
# The pre-phase-160 key set, exactly (answer_image absent too).
|
||||
assert set(dump) == {"type", "deflected", "sources", "related", "suggestions"}
|
||||
assert _json_dumps(dump, sort_keys=True) == _json_dumps(
|
||||
{
|
||||
"type": "done",
|
||||
"deflected": False,
|
||||
"sources": [{"source": "docs", "path": "a.md", "title": "A"}],
|
||||
"related": [{"source": "docs", "path": "b.md", "title": "B"}],
|
||||
"suggestions": ["Maybe try X?"],
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
raw = _json_dumps(dump)
|
||||
assert "budget_exhausted" not in raw
|
||||
|
||||
|
||||
def test_done_event_with_budget_exhausted_carries_the_key() -> None:
|
||||
"""``budget_exhausted=True`` (a cap-reached turn) → the key is
|
||||
present and ``true``, the rest of the frame untouched; an
|
||||
explicit ``False`` behaves exactly like the default (absent)."""
|
||||
dump = ChatDoneEvent(deflected=False, sources=[], budget_exhausted=True).model_dump()
|
||||
assert dump["budget_exhausted"] is True
|
||||
assert dump["type"] == "done"
|
||||
assert dump["deflected"] is False
|
||||
assert dump["related"] == []
|
||||
assert dump["suggestions"] == []
|
||||
explicit_false = ChatDoneEvent(
|
||||
deflected=False, sources=[], budget_exhausted=False
|
||||
).model_dump()
|
||||
assert "budget_exhausted" not in explicit_false
|
||||
|
||||
|
||||
def test_done_event_budget_exhausted_and_answer_image_are_independent() -> None:
|
||||
"""Both optional keys shown on the same turn: the two omission
|
||||
rules are independent — a cap-reached turn with a shown answer
|
||||
image carries BOTH keys; the answer-image ref's own serializer
|
||||
still applies inside the frame."""
|
||||
dump = ChatDoneEvent(
|
||||
deflected=False,
|
||||
sources=[],
|
||||
answer_image=SourceRef(**_SHOWN_REF),
|
||||
budget_exhausted=True,
|
||||
).model_dump()
|
||||
assert dump["budget_exhausted"] is True
|
||||
assert dump["answer_image"] == _SHOWN_REF
|
||||
|
||||
|
||||
def test_done_event_without_budget_exhausted_parses_a_pre_phase_frame() -> None:
|
||||
"""Back-compat: a pre-phase-160 done frame (no
|
||||
``budget_exhausted`` key) parses with the default ``False`` and
|
||||
re-dumps byte-identically (PLAN §4 — the field is additive in
|
||||
both directions)."""
|
||||
old_payload = {
|
||||
"type": "done",
|
||||
"deflected": True,
|
||||
"sources": [],
|
||||
"related": [],
|
||||
"suggestions": [],
|
||||
}
|
||||
event = ChatDoneEvent(**old_payload)
|
||||
assert event.budget_exhausted is False
|
||||
assert event.model_dump() == old_payload
|
||||
|
||||
|
||||
# --- the ``ChatMessage`` boundary (the saved-chat record key) ---
|
||||
|
||||
|
||||
|
||||
@@ -259,7 +259,9 @@ def test_read_result_plain_carries_date_second_line(
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [doc] and holder.tool_calls == 1
|
||||
content = llm.requests[1][0][3]["content"]
|
||||
# Phase 160 (task 01): the budget-telemetry trailer rides the read
|
||||
# result — the raw result is the RAW part (the trailer stripped).
|
||||
content = llm.requests[1][0][3]["content"].removesuffix(agent._budget_note(9))
|
||||
lines = content.splitlines()
|
||||
assert lines[0] == "Document S/a.md:" # byte-identical first line
|
||||
assert lines[1] == f"date: {DATE}" # the D5 second line
|
||||
@@ -291,7 +293,9 @@ def test_read_result_truncated_carries_date_then_marker_and_notice(
|
||||
+ "\n"
|
||||
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content_body))
|
||||
)
|
||||
assert llm.requests[1][0][3]["content"] == expected
|
||||
# Phase 160 (task 01): the trailer rides the truncated result too —
|
||||
# the truncation contract (phase 95) is the RAW part.
|
||||
assert llm.requests[1][0][3]["content"].startswith(expected)
|
||||
# The truncation still records on the holder (the ToolResultPiece
|
||||
# signal) — the date line adds nothing to the counters.
|
||||
assert holder.read_truncations == [("S/big.md", cap, len(content_body))]
|
||||
|
||||
@@ -386,6 +386,7 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
|
||||
"stopped": None,
|
||||
"failed": None,
|
||||
"error": None,
|
||||
"budget_exhausted": None,
|
||||
},
|
||||
{
|
||||
"who": "brain",
|
||||
@@ -434,6 +435,7 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
|
||||
"stopped": None,
|
||||
"failed": None,
|
||||
"error": None,
|
||||
"budget_exhausted": None,
|
||||
},
|
||||
{
|
||||
"who": "user",
|
||||
@@ -447,6 +449,7 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
|
||||
"stopped": None,
|
||||
"failed": None,
|
||||
"error": None,
|
||||
"budget_exhausted": None,
|
||||
},
|
||||
{
|
||||
"who": "brain",
|
||||
@@ -460,6 +463,7 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
|
||||
"stopped": True, # the owner stopped the generation mid-answer
|
||||
"failed": None,
|
||||
"error": None,
|
||||
"budget_exhausted": None,
|
||||
},
|
||||
]
|
||||
|
||||
@@ -468,7 +472,10 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
|
||||
dumped = payload.model_dump()
|
||||
# the stored shape is byte-identical: every message dict — None-keys
|
||||
# and all — round-trips unchanged (and the top-level record keeps its
|
||||
# ``share`` default).
|
||||
# ``share`` default). The phase-160 additive ``budget_exhausted``
|
||||
# marker joins as an explicit null (the phase-120 ``failed`` /
|
||||
# phase-48 ``stopped`` style — ``None`` flows as null, the
|
||||
# phase-50 byte-identical contract covers the new key).
|
||||
assert dumped["messages"] == messages
|
||||
assert dumped["share"] is False
|
||||
|
||||
@@ -488,8 +495,11 @@ def test_realistic_payload_round_trips_through_update_model() -> None:
|
||||
"stopped": None,
|
||||
"failed": None,
|
||||
"error": None,
|
||||
"budget_exhausted": None,
|
||||
}
|
||||
payload = SavedChatUpdate.model_validate({"messages": [msg]})
|
||||
# Identical stored shape (the phase-160 additive ``budget_exhausted``
|
||||
# marker rides as an explicit null — the phase-120/48 style).
|
||||
assert payload.model_dump()["messages"] == [msg]
|
||||
|
||||
|
||||
@@ -708,8 +718,11 @@ def test_segments_absent_by_default_and_omitted_in_dump() -> None:
|
||||
dumped = msg.model_dump()
|
||||
assert "segments" not in dumped
|
||||
# Byte-identical to the pre-phase stored shape: every bor.chat.v1
|
||||
# optional key present (null), the three serializer-popped keys
|
||||
# (image / answer_image / segments) absent.
|
||||
# optional key present (null) — the phase-160 additive
|
||||
# ``budget_exhausted`` marker included (the phase-120 ``failed`` /
|
||||
# phase-48 ``stopped`` style — explicit null where absent) — and
|
||||
# the three serializer-popped keys (image / answer_image /
|
||||
# segments) absent.
|
||||
assert dumped == {
|
||||
"who": "brain",
|
||||
"text": "ok",
|
||||
@@ -722,6 +735,7 @@ def test_segments_absent_by_default_and_omitted_in_dump() -> None:
|
||||
"stopped": None,
|
||||
"failed": None,
|
||||
"error": None,
|
||||
"budget_exhausted": None,
|
||||
}
|
||||
# An explicit empty list is a different thing: [] is a value, so it
|
||||
# round-trips as [] (present, empty — the pop is None-gated only).
|
||||
|
||||
Reference in New Issue
Block a user