"""Phase 146 (task 02) E2E (Playwright, mock-only): the search tool line shows WHERE the LLM is searching — a scoped grep carries its document. Source: TODO.md L3 — "The "search" tool in the UI doesn't show where the LLM is searching. This results in, what looks like, multiple search tool calls with the same value, like the LLM is repeating itself." A grep that re-searches a specific document now shows THAT document in the chat's tool line and status label — live, persisted on the saved turn's tool record (additively), and re-rendered on restore — so repeated patterns with different scopes read as distinct searches. Every bare-grep rendering (live line, status label, saved record shape, restored line) is byte-identical to pre-phase. Run in isolation (DB must be up: ``podman compose up -d db``): uv run pytest tests/e2e/test_search_tool_scope.py -v --no-cov MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the deterministic SCOPED-SEARCH marker flow in ``tests/e2e/mock_llm.py`` (``SCOPED_SEARCH_TRIGGER`` — user message contains "search within one document" **and** the system prompt carries the ```` section): 1. request 1 (``tools`` offered, no search result yet) → streams ONLY ``tool_calls`` deltas calling ``grep`` with ``{"pattern": SEARCH_PATTERN, "path": SCOPED_SEARCH_PATH}`` (id ``call_0``) — the SCOPED form; the app's ``tool`` frame carries the scope as its additive ``path`` field (phase 146 task 01); 2. request 2 (a ``tool``-role search result — the agent's REAL scoped grep matched the fixture's sentinel line) → the content answer ``Found ``. The EXISTING bare-grep flow (``SEARCH_TRIGGER`` — "search your documents") is untouched and is this suite's byte-identity pin (test 2). KB fixture — ONE document, imported through the real importer against ``tests/fixtures/search_docs/`` with the deterministic mock embeddings (the ``test_search_tool.py`` pattern, reused as-is): * ``search_docs/reese-notes.md`` — homelab kubernetes backup notes; both questions cosine ≈0.51 against it (mock bag-of-words, well past the E2E 0.30 threshold, and they FTS-match too) → grounded, so the HIGH prompt carries the ```` section. Its line 6 carries the sentinel ``reese-sentinel-42`` exactly once — the deterministic match of the agent's (bare or scoped) grep. Test → observable mapping (Playwright Mapping Rule; the story is the owner's TODO item — one Playwright file per story, A16): 1. ``test_scoped_grep_line_and_status_show_the_scope`` — the scoped question → the SSE ``tool`` frame is ``{"type": "tool", "name": "grep", "argument": , "path": }`` (the scope rides the wire); while the turn is live, the status carries "… is searching for in "; the tool line reads ``🔎 Searching for in `` with TWO ```` spans (pattern, then scope) and the `` in `` text node between them (the textContent structure, not a regex over the whole line — the phase-139 range-suffix pin style); the answer quotes the matched line (the scoped grep found it). 2. ``test_bare_grep_is_byte_identical`` — the EXISTING bare-grep question (the phase-68/70 scripted flow — the byte-identity pin) → the SSE ``tool`` frame has NO ``path`` key (ABSENT, never null); the tool line is exactly ``🔎 Searching for `` (ONE ```` span, NO ``.grep-scope`` span); the status label is the pre-phase "… is searching for " (no `` in `` extension); the auto-saved record keeps the pre-phase shape (NO ``path`` key); and the record restored through ``/?chat=`` renders the pre-phase line byte-identically (a no-key record IS the pre-phase saved shape — the backward-compat pin, end to end). 3. ``test_scoped_scope_survives_save_restore`` — the scoped question → the live line carries the scope → the conversation auto-saves (phase 55, silent) with ``path`` on the tool record → ``/?chat=`` re-renders the saved record → the scope comes back through the SAME structure (two ```` spans + the `` in `` text node + the ``.grep-scope`` span), with the phase-117 disclosure FOLDED (a restored turn is one compact "Tool call(s) (N)" line — the shape is unchanged). """ from __future__ import annotations import asyncio import json import re import time from pathlib import Path from threading import Thread from typing import Any import httpx from playwright.sync_api import Page, expect from sqlalchemy import text from app.config import Settings from app.db import SessionLocal from app.rag.importer import ImportSummary, import_sources from app.rag.llm import LLMClient from e2e.auth_helpers import login from tests.e2e.mock_llm import SEARCH_PATTERN, SEARCH_TRIGGER REPO = Path(__file__).resolve().parents[2] FIXTURES = REPO / "tests" / "fixtures" / "search_docs" #: The one imported document (the importer derives ``source`` from the #: fixture root's name and ``path`` from the file's relative position) #: — the SAME fixture the bare-grep scripted flow already greps. SEED_SOURCE = "search_docs" SEED_PATH = "reese-notes.md" SEED_SP = f"{SEED_SOURCE}/{SEED_PATH}" #: The fixture's sentinel line (line 6) — the mock's grep (bare or #: scoped) matches it exactly once, and its ``text`` part is what the #: "Found …" answer quotes. Pinned against the fixture file itself #: below. SENTINEL_LINE = f"The offsite vault passphrase marker is {SEARCH_PATTERN}." FOUND_ANSWER = f"Found {SENTINEL_LINE[:80]}" #: Carries ``SCOPED_SEARCH_TRIGGER`` and is on-topic (the SAME #: vocabulary as the bare question — cosine ≈0.51 against the fixture #: + FTS hits → HIGH gate, the ```` section rides along). SCOPED_QUESTION = ( "Search within one document: the vault passphrase marker in my " "homelab kubernetes backup notes?" ) assert "search within one document" in SCOPED_QUESTION.lower() #: The EXISTING bare-grep question (byte-identical to #: ``test_search_tool.py``'s — the phase-68/70 scripted flow is the #: byte-identity pin, untouched by this phase). BARE_QUESTION = ( "Search your documents for the vault passphrase marker in my homelab " "kubernetes backup notes?" ) assert SEARCH_TRIGGER in BARE_QUESTION.lower() # The two questions must each carry EXACTLY their own trigger — no # other mock marker flow may fire (the phase-68 collision guard). for other in ( "search within one document", "search your documents", "use your tools", "read two documents", "write a long answer", "think in paragraphs", "think out loud", "show the end of your notes", "show me a table", "show me the blockquote", "fail then answer", "always fail", "embed fail once", "fail first turn", "partial then fail", "emit raw tool markup", "list the files in this directory", "what are the correct llama.cpp arguments", "drill down the tree", "read the capped document", "read the suggested document", "show me the picture of", "which diagram is in your notes", "quote the image reference", "read the lines", "read the same file twice", "read two ranges", "original source link", "answer first, then list, then think", ): if other == "search within one document": assert other in SCOPED_QUESTION.lower() assert other not in BARE_QUESTION.lower() elif other == "search your documents": assert other not in SCOPED_QUESTION.lower() assert other in BARE_QUESTION.lower() else: assert other not in SCOPED_QUESTION.lower(), other assert other not in BARE_QUESTION.lower(), other def _pin_fixture() -> None: """The fixture carries the sentinel on line 6, exactly once.""" content = (FIXTURES / SEED_PATH).read_text(encoding="utf-8") lines = content.split("\n") assert lines[5] == SENTINEL_LINE, lines[5] assert sum(SEARCH_PATTERN in line for line in lines) == 1 # -------------------------------------------------------------------------- # DB seeding (TRUNCATE-then-import via the real importer — the # test_search_tool.py pattern, reused as-is: the same admin-import # pipeline the admin Sources page drives, against the session app's # DB) # -------------------------------------------------------------------------- async def _import_fixtures(mock_port: int) -> ImportSummary: kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"} settings = Settings(**kwargs) # pyright: ignore[reportCallIssue] return await import_sources([FIXTURES], LLMClient(settings)) def _run_in_thread(coro: Any) -> Any: """Run a coroutine on a worker thread. Playwright's sync API keeps an asyncio loop running on the test thread, so ``asyncio.run`` cannot be called directly from a test body (the established house helper). """ box: dict[str, Any] = {} def runner() -> None: try: box["value"] = asyncio.run(coro) except BaseException as e: # noqa: BLE001 — re-raised on the test thread box["error"] = e t = Thread(target=runner) t.start() t.join() if "error" in box: raise box["error"] return box["value"] def _reset_db(mock_port: int) -> ImportSummary: """Truncate the KB (plus the prompt-shaping tables), then re-import. ``steering_notes`` / ``kb_overview`` are truncated too, so the HIGH prompt is exactly ```` + ```` + ```` regardless of leftovers from other suites — byte-stable prompts, byte-stable answers. ``saved_chats`` is truncated too: the auto-save rows are global state, and a leftover row with the same auto-title would race ``_wait_saved_row`` (the E2E isolation pattern — this suite runs in isolation, A16).""" with SessionLocal() as db: db.execute( text( "TRUNCATE chunks, documents, query_log, steering_notes, " "kb_overview, saved_chats" ) ) db.commit() summary = _run_in_thread(_import_fixtures(mock_port)) assert summary is not None and summary.added == 1, summary return summary # -------------------------------------------------------------------------- # Page + API helpers (the test_search_tool.py / test_read_range_tool_lines # house patterns) # -------------------------------------------------------------------------- #: Records every value #send-status takes during the turn — the #: transient "… is searching for in " #: calling-tool state is held only from the first `tool` frame until #: the first answer delta, so the pre-submit observer is the #: deterministic source of truth for it (no polling race — the #: phase-44 task-03 flake fix). STATUS_RECORDER = """ () => { if (window.__statusesInstalled) return; window.__statusesInstalled = true; window.__statuses = []; const el = document.querySelector('#send-status'); if (!el) return; const rec = (v) => { const l = window.__statuses; if (!l.length || l[l.length - 1] !== v) l.push(v); }; rec(el.textContent); new MutationObserver(() => rec(el.textContent)).observe(el, { childList: true, subtree: true, }); } """ #: Captures the raw SSE ``data:`` payloads of the /api/chat stream #: (a response clone read in the background) — wire-level assertions #: for the ``tool`` frames, independent of the UI rendering. SSE_HOOK = """ () => { if (window.__sseInstalled) return; window.__sseInstalled = true; window.__sseFrames = []; const origFetch = window.fetch; window.fetch = async function (...args) { const res = await origFetch.apply(this, args); try { const url = typeof args[0] === 'string' ? args[0] : args[0].url; if (url.includes('/api/chat')) { res.clone().text().then((bodyText) => { for (const block of bodyText.split('\\n\\n')) { const line = block.trim(); if (line.startsWith('data: ')) { window.__sseFrames.push(line.slice(6)); } } }); } } catch (e) { /* non-clonable responses: ignored */ } return res; }; } """ def _install_page_hooks(page: Page) -> None: """Install all hooks on the loaded page (post-goto, pre-submit).""" page.evaluate(SSE_HOOK) page.evaluate(STATUS_RECORDER) def _frames(page: Page) -> list[dict]: """The captured SSE frames, once the hook's background read settles. The hook reads ``res.clone().text()`` in a background promise that resolves right after the stream closes — poll briefly until the final ``done`` frame lands (fail loud if the hook captured nothing). """ deadline = time.monotonic() + 10.0 while True: raw = page.evaluate("() => window.__sseFrames || []") parsed = [json.loads(line) for line in raw if line] if any(f.get("type") == "done" for f in parsed): return parsed if time.monotonic() > deadline: raise AssertionError( f"SSE hook captured no `done` frame (frames so far: " f"{len(parsed)}) — hook install failed?" ) time.sleep(0.05) def _tool_frames(frames: list[dict]) -> list[dict]: return [f for f in frames if f.get("type") == "tool"] def _submit(page: Page, question: str) -> None: page.evaluate("window.__sseFrames = []") page.fill("#message-input", question) page.click("#send-btn") # The user bubble lands synchronously with the submit handler. expect(page.locator(".msg.user .bubble").last).to_contain_text(question) def _wait_settled(page: Page) -> None: """The turn is complete: answer text in the bubble, button recovered. Phase 48: the label assertion carries the settle wait with an explicit timeout — the in-flight button is the enabled Stop control (never disabled), so ``to_be_enabled`` no longer blocks until the turn settles, and Playwright expect's default (5s) does not inherit the page default.""" expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000) expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000) expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000) def _assert_no_error_banner(page: Page) -> None: """The turn must settle WITHOUT the terminal error banner (the red role=alert error banner — the KB-offline banner is a separate, non-error state).""" banner = page.locator("#kb-banner") expect(banner).to_be_hidden() expect(banner).not_to_have_attribute("role", "alert") expect(banner).not_to_have_class(re.compile(r"is-error")) def _tool_line(page: Page): """The turn's single tool line (the last .tool-call in the last brain message).""" return page.locator(".msg.brain .tool-call").last def _assert_scoped_line(line, timeout: int = 30_000) -> None: """THE SUBJECT (shared by live + restore): the scoped grep's line reads ``🔎 Searching for in `` — TWO ```` spans (pattern, then scope) with the `` in `` text node between them (the textContent structure, not a regex over the whole line — the phase-139 range-suffix pin style). State assertions only (they retry until the boot load renders the line, and they pass on the HIDDEN line of a folded disclosure — phase 117: a restored turn renders folded).""" expect(line).to_have_count(1, timeout=timeout) expect(line).to_have_text( f"🔎 Searching for {SEARCH_PATTERN} in {SEED_SP}", timeout=timeout ) codes = line.locator("code") expect(codes).to_have_count(2, timeout=timeout) expect(codes.nth(0)).to_have_text(SEARCH_PATTERN, timeout=timeout) expect(codes.nth(1)).to_have_text(SEED_SP, timeout=timeout) scope_span = line.locator(".grep-scope") expect(scope_span).to_have_count(1, timeout=timeout) expect(scope_span).to_have_text(f" in {SEED_SP}", timeout=timeout) def _assert_bare_line(line, timeout: int = 30_000) -> None: """The byte-identity pin: the bare grep's line is EXACTLY the pre-phase line — ONE ```` span (the pattern), NO ``.grep-scope`` span, no extra bytes. (State assertions only — same retry/hidden-line contract as ``_assert_scoped_line``.)""" expect(line).to_have_count(1, timeout=timeout) expect(line).to_have_text(f"🔎 Searching for {SEARCH_PATTERN}", timeout=timeout) expect(line.locator("code")).to_have_count(1, timeout=timeout) expect(line.locator("code")).to_have_text(SEARCH_PATTERN, timeout=timeout) expect(line.locator(".grep-scope")).to_have_count(0, timeout=timeout) def _admin_cookies(page: Page) -> dict[str, str]: """The signed session cookies the browser holds after a form login — used to call the admin API with plain httpx.""" return { c["name"]: c["value"] for c in page.context.cookies() if "name" in c and "value" in c } def _auto_title(question: str) -> str: """The phase-50 auto-title convention: the first question, whitespace-collapsed, capped at 120 chars.""" return " ".join(question.split())[:120] def _find_row(rows: list[dict[str, Any]], title: str) -> dict[str, Any] | None: return next((c for c in rows if c["title"] == title), None) def _wait_saved_row( app_url: str, cookies: dict[str, str], title: str, messages: int = 2 ) -> dict[str, Any]: """Wait for the auto-saved row (phase 55: auto-saves are SILENT — so there is no status line to wait on). Poll the admin list until the row with the conversation's auto-title appears with the expected message count.""" deadline = time.monotonic() + 15 last: dict[str, Any] | None = None while time.monotonic() < deadline: r = httpx.get(f"{app_url}/api/chats", timeout=10, cookies=cookies) assert r.status_code == 200 last = _find_row(r.json()["chats"], title) if last is not None and last["message_count"] >= messages: return last time.sleep(0.2) raise AssertionError(f"no auto-saved row for {title!r} (last: {last!r})") def _saved_tool_record( app_url: str, cookies: dict[str, str], chat_id: str ) -> dict[str, Any]: """The last brain record's single tool record, as the API returns it (the saved ``bor.chat.v1`` JSONB through the boundary).""" r = httpx.get(f"{app_url}/api/chats/{chat_id}", timeout=10, cookies=cookies) assert r.status_code == 200, r.text messages = r.json()["messages"] brain = [m for m in messages if m["who"] == "brain"] assert len(brain) == 1, messages tools = brain[0].get("tools") assert isinstance(tools, list) and len(tools) == 1, messages return tools[0] def _delete_chat(app_url: str, cookies: dict[str, str], chat_id: str) -> None: """Best-effort row cleanup (a 404 — already deleted — is fine).""" httpx.delete(f"{app_url}/api/chats/{chat_id}", timeout=10, cookies=cookies) # -------------------------------------------------------------------------- # 1. The scoped grep: the tool line + status label show the scope # -------------------------------------------------------------------------- def test_scoped_grep_line_and_status_show_the_scope( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: _pin_fixture() page.set_default_timeout(30_000) _reset_db(mock_llm) login(page, app_url, next="/") _install_page_hooks(page) _submit(page, SCOPED_QUESTION) # The "calling tool" STATUS window is transient: the first `tool` # frame sets #send-status and it holds until the FIRST answer delta # (the agent loop completes before the answer stream) — the # pre-submit MutationObserver records are the deterministic source # of truth for the status transitions (the phase-44 flake fix). _wait_settled(page) # The scoped STATUS label (while the turn is live — recorded before # the first delta replaced it): "… is searching for in # ", AFTER the "… is thinking" state. statuses = page.evaluate("() => window.__statuses") i_search = next( ( i for i, s in enumerate(statuses) if s.endswith(f"is searching for {SEARCH_PATTERN} in {SEED_SP}") ), None, ) assert i_search is not None, statuses i_think = next( (i for i, s in enumerate(statuses) if "is thinking" in s), None ) assert i_think is not None and i_think < i_search, statuses # Wire level: exactly ONE `tool` frame — the grep carrying the # PATTERN as its argument AND the scope as the additive `path` # field (phase 146 task 01) — ahead of the first `delta` frame. frames = _frames(page) assert _tool_frames(frames) == [ { "type": "tool", "name": "grep", "argument": SEARCH_PATTERN, "path": SEED_SP, } ] first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta") assert all( i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool" ) done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False # THE SUBJECT: ONE tool line — the scope rendered after the # pattern (two spans + the " in " text node between them). _assert_scoped_line(_tool_line(page)) # The answer quotes the MATCHED LINE — the agent's REAL scoped grep # found the sentinel in the named document (the mock's echo). bubble = page.locator(".msg.brain .bubble").last expect(bubble).to_contain_text(FOUND_ANSWER) # The turn settled to idle WITHOUT the error banner. _assert_no_error_banner(page) # -------------------------------------------------------------------------- # 2. The byte-identity pin: a BARE grep renders exactly as pre-phase — # on the wire, on the line, in the status, and in the saved record # -------------------------------------------------------------------------- def test_bare_grep_is_byte_identical( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: _pin_fixture() page.set_default_timeout(30_000) _reset_db(mock_llm) login(page, app_url, next="/") _install_page_hooks(page) _submit(page, BARE_QUESTION) _wait_settled(page) # Wire level: the bare grep's frame is EXACTLY the pre-phase shape — # the `path` key is ABSENT (never null). frames = _frames(page) tool_frames = _tool_frames(frames) assert len(tool_frames) == 1, tool_frames assert "path" not in tool_frames[0], tool_frames[0] assert tool_frames[0] == { "type": "tool", "name": "grep", "argument": SEARCH_PATTERN, }, tool_frames[0] done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False # The tool line: byte-identical to pre-phase (one , no scope # span — the phase-70/87/117 shape). _assert_bare_line(_tool_line(page)) # The status label: the EXACT pre-phase "… is searching for # " — no " in " extension for a bare grep. statuses = page.evaluate("() => window.__statuses") assert any(s.endswith(f"is searching for {SEARCH_PATTERN}") for s in statuses), ( statuses ) assert not any( f"is searching for {SEARCH_PATTERN} in" in s for s in statuses ), statuses # The saved record: the auto-save (phase 55, silent) persists the # toolAcc record — a bare grep keeps the EXACT pre-phase shape (NO # `path` key — the additive key is stamped only when the frame # carried one). cookies = _admin_cookies(page) row = _wait_saved_row(app_url, cookies, _auto_title(BARE_QUESTION)) try: record = _saved_tool_record(app_url, cookies, row["id"]) assert "path" not in record, record assert record["name"] == "grep", record assert record["argument"] == SEARCH_PATTERN, record # And the no-key record (== the pre-phase saved shape) restores # byte-identically through /?chat= (the phase-50 # backward-compat rule, end to end). The restored line rides the # FOLDED disclosure (phase 117) — the state assertions pass on # the hidden line and retry until the boot load renders it. page.goto(app_url + f"/?chat={row['id']}") _assert_bare_line(_tool_line(page)) finally: _delete_chat(app_url, cookies, row["id"]) # -------------------------------------------------------------------------- # 3. The scope survives save → restore: the saved record's additive # `path` re-renders the same line on /?chat= (the phase-117 # disclosure folded — the shape is unchanged) # -------------------------------------------------------------------------- def test_scoped_scope_survives_save_restore( page: Page, app_url: str, mock_llm: int, db_ready: None ) -> None: _pin_fixture() page.set_default_timeout(30_000) _reset_db(mock_llm) login(page, app_url, next="/") _install_page_hooks(page) _submit(page, SCOPED_QUESTION) _wait_settled(page) # The LIVE line carries the scope (the save point persists the # toolAcc record — {name, argument, path}). _assert_scoped_line(_tool_line(page)) # The conversation auto-saved (phase 55 — silent). Wait for the row # by its auto-title (the first question), then pin the persisted # record: the scope rides `path` ADDITIVELY on the tool record. cookies = _admin_cookies(page) row = _wait_saved_row(app_url, cookies, _auto_title(SCOPED_QUESTION)) try: record = _saved_tool_record(app_url, cookies, row["id"]) assert record["name"] == "grep", record assert record["argument"] == SEARCH_PATTERN, record assert record["path"] == SEED_SP, record # Restore: /?chat= boots into the saved conversation # through the SAME renderStoredMessage loop as the local # restore — the saved record's additive `path` re-renders the # scope through the SAME structure (the phase-139 save → # restore pattern). The restored line rides the FOLDED # disclosure (phase 117) — the state assertions pass on the # hidden line and retry until the boot load renders it. page.goto(app_url + f"/?chat={row['id']}") _assert_scoped_line(_tool_line(page)) # Phase 117 (D3): a restored turn renders FOLDED — one compact # "Tool call(s) (N)" line (the disclosure shape is unchanged by # this phase: the `open` attribute is ABSENT). disclosure = page.locator(".msg.brain .tool-calls-disclosure").last expect(disclosure).not_to_have_attribute("open", timeout=30_000) expect(disclosure.locator("summary")).to_have_text( "Tool call (1)", timeout=30_000 ) # The answer text restored too (the record is intact). expect(page.locator(".msg.brain .bubble").last).to_contain_text( FOUND_ANSWER, timeout=15_000 ) finally: _delete_chat(app_url, cookies, row["id"])