Files
brain-of-reese/tests/e2e/test_tool_call_compact.py
T
ducoterra 31bd903a93
Build and Push Containers / build-and-push-app (push) Successful in 8m38s
Build and Push Containers / build-and-push-db (push) Successful in 32s
phase: 158_thinking_timeline
All verification complete. Every gate is green and the docs check out. Final report:

## Phase 158 final verification pass — COMPLETE

**Verified (all tasks pre-completed; this pass re-ran the full gate):**
- Unit + integration: `uv run pytest --cov=app --cov-report=term-missing` → **3618 passed, 99% coverage on `app/`** (>90% gate)
- Phase E2E in isolation: `uv run pytest tests/e2e/test_thinking_timeline.py -v --no-cov` → **6/6 passed** (order pin, fold-at-rest, live-open, restore+`segments` array, shared page, no-thinking single disclosure)
- Regression E2E, each in isolation: `test_thinking_display` 5✓, `test_tool_call_compact` 4✓, `test_turn_progress_loader` 3✓ (per-run pin updated), `test_big_read_progress` 4✓, `test_agent_document_tools` 4✓, `test_share_chat` 4✓, `test_chat_persistence` 4✓, `test_smoke` 3✓
- Lint/types: `uv run ruff check . && uv run pyright` → **clean (0 errors)**

**Completion criteria:** all 9 met — timeline E2E pins ✓, restore/shared `segments` ✓, no-thinking single-disclosure + the three UNCHANGED regression suites green ✓, pre-phase byte-identical round-trip (integration pin: key absent, never null; >50 / >32 000 / empty text / unknown kind all 422) ✓, full suite + coverage + E2E + lint ✓, `git diff --stat` footprint limited to `app/schemas.py`, `frontend/assets/{app,shared}.js`, `tests/`, README, phase files (a pre-existing owner `TODO.md` edit is out of scope, not staged by the phase commit) ✓, D15→per-run supersession recorded in the updated pin ✓. (Commit + phase move are the harness's job; task dirs already in `complete/`.)

**Documentation:** README.md — Thinking section correctly describes the phase-158 timeline; config reference matches `.env.example` exactly; page table matches registered routes; no stale single-block wording. AGENTS.md — rules and quick-reference commands all accurate. **Verified consistent — no changes needed.**

**Decisions/deviations:** none. `test_tool_call_compact.py` carries a one-line docstring-only update (referenced the removed `closeToolCalls`); its test logic is byte-identical to the phase-117 contract and green.

**Next pending phase:** none — `todo/` contains only 158.
2026-10-05 02:52:55 -04:00

425 lines
18 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Phase 117 E2E (Playwright, mock-only): compact, well-wrapped tool-call lines.
Source: owner visual-glitch report (2026-09-15, mobile viewport,
``https://brain.reeseapps.com``) — "how much space the tool calls take
up, and the tool call text is spit and wrapped poorly": one completed
answer stacked 6+ full-width bordered tool-call cards (one per ``tool``
SSE frame), each a complete card (accent left border + surface
background + radius + a mono path chip inside), with the label flex
item breaking mid-word (``Rea``/``ding``) and the path wrapping to
three lines indented to the right of that broken label.
Phase 117's fix (frontend-only, D1–D6 in ``00_phase.md``): the
per-call lines ride in ONE native ``<details>`` disclosure — a compact
"Tool calls (N)" summary, open while the turn is live and FOLDED when
the answer begins (the first ``delta``), on ``done``, on stop/abort,
and on restore — and the lines themselves are deboxed inline-flow
text (no card, no flex: the label + ``<code>`` run as one continuous
sentence and the path wraps to the left edge).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_tool_call_compact.py -v --no-cov
MOCK-ONLY suite: the marker question drives the SAME deterministic
3-call flow ``test_agent_document_tools.py`` pins — ``ls`` →
``ls(scoped)`` → ``read`` (``tests/e2e/mock_llm.py``: the user message
contains ``use your tools`` and the system prompt carries the
``<tools>`` section of the HIGH prompt) — over the byte-identical
two-document KB fixture (mirrored from that suite so the flow is
deterministic). No slow proxy: the fold is about a COMPLETED turn (it
folds on the first delta), so the fast mock is the deterministic
driver.
Completion marker: the mock's single-read flow answers
``Read <source/path>. <first 80 chars of the read document>`` — the
deterministic quote below (``MOCK_ANSWER_MARKER``), NOT the plain
grounded answer's "Deterministic mock answer for E2E" tail (the tool
flow never reaches that branch). Its presence in the bubble proves the
first delta has landed — the turn is COMPLETE and the disclosure has
already folded (phase 117, D3: the fold rides the first delta).
Test → phase mapping (Playwright Mapping Rule):
1. ``test_tool_calls_fold_to_one_line_after_turn`` — pin 1: a
completed 3-call turn renders ONE visible "Tool calls (3)" summary;
the disclosure is CLOSED (no ``open`` attribute); the three
``.tool-call`` lines are present in the DOM but folded (the first
line is not visible); the answer is present.
2. ``test_summary_click_expands_the_calls`` — pin 2: tapping the
native ``<summary>`` (a real focusable toggle, AA) opens the
disclosure; the three lines become visible, in order, with the
phase-37/94 pinned text.
3. ``test_expanded_line_is_deboxed_inline_flow`` — pin 3: the
expanded "Reading" line's computed ``display`` is NOT ``flex``
(the label + path are one inline run, not two flex items — the
debox), and the label runs directly into the path in the same run.
4. ``test_restored_turn_renders_folded`` — pin 4: a same-context
RELOAD (the phase-14/50 persisted conversation restores) renders
the turn FOLDED — closed disclosure, "Tool calls (3)" summary,
three lines in the DOM, first line hidden — no auto-expand on load.
"""
from __future__ import annotations
import hashlib
from collections.abc import Callable
from datetime import UTC, datetime
from playwright.sync_api import Page, expect
from sqlalchemy import text
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import Chunk, Document, GitSource
from e2e.auth_helpers import login
from tests.e2e.mock_llm import embed_text
# --------------------------------------------------------------------------
# KB fixture — byte-identical to the phase-37 suite (the mirror)
# --------------------------------------------------------------------------
SEED_SOURCE = "Homelab"
SEED_PATH = "aws-route53.md"
SEED_SP = f"{SEED_SOURCE}/{SEED_PATH}"
READ_SOURCE = "Deployments"
READ_PATH = "example-record-file.json"
READ_SP = f"{READ_SOURCE}/{READ_PATH}"
#: The retrievable document: references the JSON file "for the exact
#: JSON shape of reeselink.json" but never includes it. The repeated
#: record-file lines carry the marker question's key tokens (aws,
#: route53, hosted, zone, reeselink, json, exact, shape) — verified
#: ≈0.69 cosine against the mock's embeddings (E2E threshold 0.30)
#: plus FTS hits, so the turn is solidly grounded (HIGH prompt →
#: ``<tools>``).
ROUTE53_CONTENT = (
"# AWS Route 53 Notes\n\n"
"## Record file\n\n"
+ (
"The aws route53 hosted zone for reeselink keeps every record in "
"reseelink.json — the exact JSON shape of reeselink.json is "
"documented in example-record-file.json.\n"
)
* 10
+ "\n## Sync job\n\n"
"A cron job pushes reeselink.json to the aws route53 hosted zone "
"every fifteen minutes; the diff is applied through the route53 api.\n"
)
#: The referenced document: the exact JSON shape, seeded WITHOUT chunks
#: (indexed + catalogued + readable, but never a retrieval candidate).
#: Its ``(source, path)`` sorts FIRST in the catalog — the line the
#: mock's drill + read land on.
RECORD_FILE_CONTENT = (
'{ "version": 3, "comment": "ReeseLink hosted zone records — the exact '
'JSON shape of reeselink.json",\n'
' "hosted_zone_id": "Z0RESEELINK01",\n'
' "record_sets": [\n'
' { "name": "www.reeselink.example", "type": "A", "ttl": 300,\n'
' "resource_records": [ { "value": "10.0.0.20" } ] },\n'
' { "name": "api.reeselink.example", "type": "CNAME", "ttl": 300,\n'
' "resource_records": [ { "value": "www.reeselink.example" } ] }\n'
" ]\n"
"}\n"
)
assert "\n" not in RECORD_FILE_CONTENT[:63] # the quote's content part stays one line
#: The SAME marker that drives the mock's 3-tool flow (mirrored from
#: ``test_agent_document_tools.py`` — deterministic ``ls`` →
#: ``ls(scoped)`` → ``read``).
MARKER_QUESTION = (
"Use your tools: what is the exact JSON shape of reeselink.json "
"for my aws route53 hosted zone?"
)
#: The mock's deterministic single-read answer —
#: "Read <source/path>. <first 80 chars of the read document>"
#: (the phase-37 shape). This suite's completion marker: its presence
#: in the bubble proves the FIRST DELTA has landed — the turn is
#: COMPLETE and the disclosure has already folded (phase 117, D3).
MOCK_ANSWER_MARKER = f"Read {READ_SP}."
#: The phase-117 selectors (scoped to the conversation column).
SUMMARY = "#messages .msg.brain .tool-calls-summary"
DISCLOSURE = "#messages .msg.brain .tool-calls-disclosure"
LINES = "#messages .msg.brain .tool-call"
# --------------------------------------------------------------------------
# DB seeding (TRUNCATE-then-seed, cf. test_agent_document_tools.py)
# --------------------------------------------------------------------------
def _seed(db: Session) -> None:
"""The phase-37 two-document pair, byte-identical (see the module
docstring), plus the phase-94 registry rows: both sources
registered, ``Deployments`` FIRST — the mock's drill (first
source of the top-level listing) lands on the JSON file
deterministically, independent of the operator's
``BOR_GIT_SOURCES`` (a non-empty table ignores the env fallback)."""
# COMMIT between the inserts (not flush): ``added_at`` is
# ``server_default now()`` — the transaction timestamp — and the
# tie-break is the random uuid ``id``, so one-transaction rows order
# nondeterministically.
db.add(GitSource(url=READ_SOURCE, kind="local"))
db.commit()
db.add(GitSource(url=SEED_SOURCE, kind="local"))
md = Document(
source=SEED_SOURCE,
path=SEED_PATH,
full_path=f"/tmp/{SEED_PATH}",
title="AWS Route 53 Notes",
content=ROUTE53_CONTENT,
content_hash=hashlib.sha256(ROUTE53_CONTENT.encode()).hexdigest(),
indexed_at=datetime.now(UTC),
# Phase 106 (D5): explicit dates — byte-stable prompts/quotes
# (the mock's first-80-chars read quote carries the date line).
created_at=datetime(2024, 6, 15, tzinfo=UTC),
)
db.add(md)
db.flush()
# One chunk carrying the mock's own embedding → genuine token
# overlap between the marker question and this document.
db.add(
Chunk(
document_id=md.id,
position=0,
content=ROUTE53_CONTENT,
embedding=embed_text(ROUTE53_CONTENT),
)
)
# The referenced JSON: indexed, catalogued, readable — but NO
# chunks, so retrieval never puts it in context.
db.add(
Document(
source=READ_SOURCE,
path=READ_PATH,
full_path=f"/tmp/{READ_PATH}",
title="Example Record File",
content=RECORD_FILE_CONTENT,
content_hash=hashlib.sha256(RECORD_FILE_CONTENT.encode()).hexdigest(),
indexed_at=datetime.now(UTC),
created_at=datetime(2024, 6, 15, tzinfo=UTC),
)
)
def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
"""Truncate the KB (plus the prompt-shaping tables and the
phase-55 auto-save rows), then re-seed — the E2E isolation
pattern. ``steering_notes`` / ``kb_overview`` are truncated too, so
the HIGH prompt is exactly ``<relevance>`` + ``<documents>`` +
``<tools>`` regardless of leftovers — byte-stable prompts,
byte-stable answers."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, "
"steering_notes, kb_overview, saved_chats, git_sources"
)
)
db.commit()
if seed is not None:
seed(db)
db.commit()
# --------------------------------------------------------------------------
# Page helpers
# --------------------------------------------------------------------------
def _disclosure_open(page: Page) -> bool:
"""The disclosure's open state: a closed ``<details>`` has NO
``open`` attribute (``get_attribute`` → ``None``); an open one
carries it (any value — the native boolean attribute is empty)."""
return page.get_attribute(DISCLOSURE, "open") is not None
def _submit_tools_turn(page: Page, app_url: str) -> None:
"""Log in as admin, submit the marker question, and wait for the
answer bubble — i.e. the turn is COMPLETE (the disclosure has
already folded on the first delta).
Auth: the phase-37 agent-tools E2E's pattern (the mock flow + the
house pattern for this flow win)."""
login(page, app_url, next="/")
page.fill("#message-input", MARKER_QUESTION)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(MARKER_QUESTION)
# The answer's deterministic quote — its presence proves the first
# delta has landed (the fold rides that frame, phase 117 D3)…
expect(
page.locator(".msg.brain .bubble").last
).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
# …and the turn settled: button recovered (phase-48 — the in-flight
# button is the enabled Stop control, so the label assertion carries
# the settle wait).
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
def _expand_disclosure(page: Page) -> None:
"""Tap the summary — the native ``<summary>`` is a real focusable
toggle (AA), so a plain click is the user contract."""
page.locator(SUMMARY).click()
# --------------------------------------------------------------------------
# 1. A completed 3-call turn folds to one "Tool calls (3)" line
# --------------------------------------------------------------------------
def test_tool_calls_fold_to_one_line_after_turn(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""Pin 1 (the space fix): after the deterministic ``ls`` →
``ls(scoped)`` → ``read`` turn COMPLETES, the record is ONE visible
"Tool calls (3)" summary — the disclosure is CLOSED (no ``open``
attribute: the first answer delta folded it, D3), the three
``.tool-call`` lines are still in the DOM (the permanent record)
but hidden (the first line is not visible), and the answer bubble
is present."""
page.set_default_timeout(30_000)
_reset_db(_seed)
_submit_tools_turn(page, app_url)
# One compact summary line, visible, with the live count.
summary = page.locator(SUMMARY)
expect(summary).to_be_visible()
expect(summary).to_contain_text("Tool calls (3)")
# Folded at rest: a closed <details> carries no `open` attribute.
assert page.get_attribute(DISCLOSURE, "open") is None
# The lines are present in the DOM (the permanent record) but
# folded: the FIRST line is not rendered.
expect(page.locator(LINES)).to_have_count(3)
expect(page.locator(LINES).first).not_to_be_visible()
# The answer is present (the mock's deterministic single-read quote).
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
MOCK_ANSWER_MARKER
)
# --------------------------------------------------------------------------
# 2. Tapping the summary expands the calls
# --------------------------------------------------------------------------
def test_summary_click_expands_the_calls(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""Pin 2 (the expand contract): clicking the native ``<summary>``
opens the disclosure (the ``open`` attribute appears) and all
three ``.tool-call`` lines become visible, in order, with the
phase-37/94 pinned text (the drill line is "Listing documents in
<source>" — the top level lists sources only)."""
page.set_default_timeout(30_000)
_reset_db(_seed)
_submit_tools_turn(page, app_url)
assert not _disclosure_open(page) # folded after the turn (pin 1)
_expand_disclosure(page)
# The native boolean attribute is now present (any value — it is
# empty in the DOM; "true" is the attribute-present shorthand).
assert page.get_attribute(DISCLOSURE, "open") is not None
# All three lines, in order, visible with their pinned text.
lines = page.locator(LINES)
expect(lines).to_have_count(3)
expect(lines.first).to_be_visible()
expect(lines.nth(0)).to_be_visible()
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_be_visible()
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1)).to_contain_text(READ_SOURCE)
expect(lines.nth(2)).to_be_visible()
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2)).to_contain_text(READ_SP)
# --------------------------------------------------------------------------
# 3. An expanded line is deboxed inline-flow text (not two flex items)
# --------------------------------------------------------------------------
def test_expanded_line_is_deboxed_inline_flow(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""Pin 3 (the wrapping fix): on the expanded "Reading" line the
computed ``display`` is NOT ``flex`` — the pre-phase-117 card had
``display: flex; align-items: baseline``, which made the label text
node and the ``<code>`` path two separate flex items (the label
broke mid-word, the path wrapped indented). Deboxed (D4), the label
+ inline ``<code>`` flow as one continuous run: the path wraps to
the left edge like a normal sentence."""
page.set_default_timeout(30_000)
_reset_db(_seed)
_submit_tools_turn(page, app_url)
_expand_disclosure(page)
# The visible "Reading" line (the third of the three).
line = page.locator(LINES).nth(2)
expect(line).to_be_visible()
expect(line).to_contain_text("Reading ")
# The debox: NOT a flex item pair — one inline run. (As a flex item
# of the .tool-calls column its used display blockifies to "block";
# the pre-phase-117 card computed "flex" — the discriminator.)
display = page.evaluate(
"""() => {
const lines = document.querySelectorAll('#messages .msg.brain .tool-call');
const el = Array.from(lines).find(
(l) => l.textContent.includes('Reading '));
return el ? getComputedStyle(el).display : null;
}"""
)
assert display is not None and display != "flex", (
"the expanded .tool-call line must be deboxed (label + path one "
f"inline run, not two flex items) — computed display: {display!r}"
)
# --------------------------------------------------------------------------
# 4. A reload restores the turn FOLDED (no auto-expand on load)
# --------------------------------------------------------------------------
def test_restored_turn_renders_folded(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""Pin 4 (the restore path): after the turn completes (the
phase-50/55 auto-save persisted it — the local ``bor.chat.v1``
record is written at the save points, so the reload sees the full
conversation), a same-context RELOAD re-renders the persisted
record through the restore path (``renderStoredMessage`` →
``appendToolLine`` → the after-loop fold): the disclosure is
present, CLOSED (no ``open`` attribute — no auto-expand on load),
the summary reads "Tool calls (3)", the three lines are in the DOM
and the first is hidden (folded on load)."""
page.set_default_timeout(30_000)
_reset_db(_seed)
_submit_tools_turn(page, app_url)
page.reload()
expect(page.locator("#empty-state")).to_be_hidden(timeout=30_000)
# The restored brain message's answer (the restore is synchronous at
# boot — the persisted record is byte-stable for this fixture).
expect(
page.locator(".msg.brain .bubble").last
).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
# Folded on load: one compact summary line, disclosure closed.
expect(page.locator(SUMMARY)).to_be_visible()
expect(page.locator(SUMMARY)).to_contain_text("Tool calls (3)")
assert page.get_attribute(DISCLOSURE, "open") is None
# The three lines are in the DOM (the permanent record) but folded:
# the first line is not rendered — no auto-expand.
expect(page.locator(LINES)).to_have_count(3)
expect(page.locator(LINES).first).not_to_be_visible()