Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 94_ls_tree_drilldown
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Details
Build and Push Containers / build-and-push-db (push) Successful in 25s
Details

Browse Source 
All green. Verification complete.

**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**

- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)

**Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
ducoterra
2026-09-11 00:59:35 -04:00
parent 9188be259b
commit d4943b4822
61 changed files with 6289 additions and 666 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/mock_llm.py
+456 -73
View File
@@ -18,6 +18,14 @@ Implements just enough of the aipi surface:
- ``KB_OVERVIEW_MODE`` -> the deterministic outline: the first 8 tokens
of the user message (the generator puts the document list there) —
byte-stable for a given KB (KB overview, phase 31)
- ``FOLDER_SUMMARY_MODE`` -> the deterministic folder one-liner
``Fixture folder summary for <folder>.`` — <folder> is the user
message's ``Folder: …`` header line (the generator names the
source/folder there, phase 94), so the stored row always names its
folder and an E2E can assert on it. Checked BEFORE the
``SUMMARY_MODE`` branch: the folder marker CONTAINS the summary
marker as a substring, so the summary branch would otherwise
shadow every folder-summary call
- ``DEFLECT_MODE`` -> honest "I haven't done anything like that" answer
- otherwise -> upbeat answer quoting the provided document context
- user message containing ``pretend to think slowly`` -> 3s warm-up delay
@@ -57,22 +65,31 @@ Implements just enough of the aipi surface:
closing tag — same sentinel semantics.)
- user message containing ``use your tools`` (phase 37, agent document
tools; phase 70: the flow emits the harness-aligned names — ``ls``
/ ``read`` with the combined ``source/path`` identity) **and** the
system prompt carries the ``<tools>`` section -> the deterministic
SINGLE-READ tool flow, discriminated statelessly from the messages
(the ``tools`` parameter gates the list/read steps — a no-tools
request with no tool results is not the flow):
/ ``read`` with the combined ``source/path`` identity; phase 94:
the flow DRILLS through the tree ``ls`` — the top-level listing
carries sources only, so the flow takes one drill step, ``ls``
scoped to the first source, before the first file line exists)
**and** the system prompt carries the ``<tools>`` section -> the
deterministic SINGLE-READ tool flow, discriminated statelessly from
the messages (the ``tools`` parameter gates the list/read steps —
a no-tools request with no tool results is not the flow):
* request 1 (``tools`` offered, no tool results yet): stream ONLY
``tool_calls`` deltas — ``ls`` (synthetic id ``call_0``, no
arguments), ``finish_reason: "tool_calls"``, no content;
* request 2 (a ``tool``-role catalog result in the messages):
parse the FIRST catalog line (``source: X | path: Y | title: Z``
— the labeled ``source:`` / ``path:`` fields, phase 63) and
stream a ``tool_calls`` delta calling ``read`` on the JOINED
combined ``source/path`` (the mock joins the two labeled fields
— the catalog format is unchanged, so this join is the only
parse change, phase 70) (id ``call_1``);
* request 3 (a ``tool``-role read result in the messages —
* request 2 (the top-level source listing in the messages — the
agent's ``ls`` tree format, phase 94: no file lines yet):
stream a ``tool_calls`` delta — ``ls`` scoped to the FIRST
source of the listing (id ``call_1``) — the drill step (one
drill per flow — ``_drill_target`` skips already-drilled
sources, so the second listing is the folder level);
* request 3 (a ``tool``-role folder listing with file lines in
the messages): parse the FIRST file line (``source: X | path:
Y | title: Z`` — the labeled ``source:`` / ``path:`` fields,
phase 63) and stream a ``tool_calls`` delta calling ``read`` on
the JOINED combined ``source/path`` (the mock joins the two
labeled fields — the file-line format is unchanged, so this
join is the only parse change, phase 70) (id ``call_2``);
* request 4 (a ``tool``-role read result in the messages —
content starting with the agent's ``"Document <source/path>:"``
header): a content answer, deterministic: ``Read
<source/path>. <first 80 chars of the read document's
@@ -85,19 +102,22 @@ Implements just enough of the aipi surface:
- user message containing BOTH ``use your tools`` AND ``read two
documents`` (``MULTI_READ_TRIGGER``, phase 45 task 02) **and** the
system prompt carries the ``<tools>`` section -> the deterministic
MULTI-READ flow (list → read #1 → read #2 → answer), classified by
the COUNT of ``tool``-role read results (content starting with the
agent's ``"Document <source/path>:"`` prefix); phase 70: the same
flow on the harness-aligned names — ``ls``, then ``read`` on the
JOINED combined ``source/path`` of each catalog line:
* 0 read results, no catalog yet: ``ls`` (id ``call_0``);
* 0 read results, catalog present: ``read`` on the JOINED
combined ``source/path`` of the FIRST catalog line
(id ``call_1``);
MULTI-READ flow (list → drill → read #1 → read #2 → answer; phase
94: the drill step between the top-level listing and the first
read), classified by the COUNT of ``tool``-role read results
(content starting with the agent's ``"Document <source/path>:"``
prefix); phase 70: the same flow on the harness-aligned names —
``ls``, the drill ``ls`` scoped to the first source, then ``read``
on the JOINED combined ``source/path`` of each file line:
* 0 read results, no listing yet: ``ls`` (id ``call_0``);
* 0 read results, top-level listing only (no file lines yet):
the drill — ``ls`` scoped to the first source (id ``call_1``);
* 0 read results, file lines present: ``read`` on the JOINED
combined ``source/path`` of the FIRST file line (id ``call_2``);
* 1 read result: ``read`` on the JOINED combined ``source/path``
of the SECOND catalog line — the first listing line whose
of the SECOND file line — the first listing line whose
``source/path`` differs from the one already read (id
``call_2``); a one-document catalog degenerates to the
``call_3``); a one-file listing degenerates to the
single-read answer (nothing second to read);
* 2 read results: the forced answer, byte-stable: the single-read
shape quoting the FIRST read result, plus the line ``I read
@@ -162,27 +182,83 @@ Implements just enough of the aipi surface:
misuse met the terse refusal and the model re-reasoned the same
paragraphs over and over) **and** the system prompt carries the
``<tools>`` section -> the deterministic LS-TEACHING flow,
discriminated statelessly from the messages (streaming only):
discriminated statelessly from the messages (streaming only;
phase 94: the correction's top-level listing carries sources only,
so the flow drills one level before the first file line exists):
* request 1 (``tools`` offered, no ``tool``-role result in the
messages yet): stream ONLY ``tool_calls`` deltas — ``ls``
with ``{"path": "."}`` (synthetic id ``call_0``),
``finish_reason: "tool_calls"``, no content — the incident's
misuse, deterministic;
* request 2 (a ``tool``-role result present that is NOT a
catalog listing — i.e. the teaching refusal): a ``tool_calls``
delta — ``ls`` with no arguments (id ``call_1``) — the
correction;
* request 3 (a ``tool``-role result whose first line matches the
``^\\d+ documents:`` catalog header): a deterministic content
answer — ``These are the indexed documents: <first catalog
line>`` (the ``source: X | path: Y | title: Z`` line, parsed
with the ``_CATALOG_LINE_RE`` machinery), ``finish_reason:
"stop"`` — the loop ended in ONE correction, not at the round
listing with file lines — i.e. the teaching refusal): a
``tool_calls`` delta — ``ls`` with no arguments (id ``call_1``)
— the correction;
* request 3 (the top-level source listing in the messages —
file lines still absent): the drill — a ``tool_calls`` delta
— ``ls`` scoped to the FIRST source of the listing
(id ``call_2``);
* request 4 (a ``tool``-result with a ``source: X | path: Y |
title: Z`` file line in the messages — the folder listing):
a deterministic content answer — ``These are the indexed
documents: <first file line>`` (the line, parsed with the
``_CATALOG_LINE_RE`` machinery), ``finish_reason: "stop"`` —
the loop ended in ONE correction + ONE drill, not at the round
cap.
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (the trigger
phrases are disjoint substrings — the phase-71 ordering
convention); no existing E2E question or fixture file contains the
phrase, so every other suite is unaffected.
- user message containing ``drill down the tree`` (``DRILL_TRIGGER``,
phase 94 task 04 — the drill-down ``ls``'s dedicated story suite
``tests/e2e/test_ls_tree_drilldown.py``) **and** the system prompt
carries the ``<tools>`` section -> the deterministic SCRIPTED
DRILL-DOWN flow: the question carries its own tool call after the
colon — ``drill down the tree: ls [target]`` (no target = the top
level; ``target`` = a source name or ``source/folder`` path) or
``drill down the tree: read source/path`` — parsed by
``_DRILL_CALL_RE`` from the RAW user message (the target keeps its
case), then discriminated statelessly from the tool results
(streaming only):
* request 1 (``tools`` offered, no ``tool``-role result in the
messages yet): the SCRIPTED call — ``ls`` with no arguments
for the top level, ``ls``/``read`` with the parsed target
otherwise (synthetic id ``call_0``);
* the last tool result is the NOT-A-FOLDER teaching refusal
(it carries ``"is not a folder"`` — the phase-94 task-03
teaching line, the refusal being VISIBLE to the model is what
fires this branch): the scripted one-round recovery — ``ls``
the target's SOURCE segment (id ``call_1``);
* the last tool result is a READ result (``"Document
<source/path>:…``): the deterministic echo answer ``Read
<source/path>. <first 80 chars of the read document's
content>`` (the phase-37 single-read shape — the grounded-
turn citation contract);
* the last tool result is the agent's ALREADY_IN_CONTEXT dedupe
refusal (the read target is already a top-2 retrieval
document — with the drill fixture that is DETERMINISTIC:
the read question names the file's path, so the file
self-matches the hybrid gate and its FULL text is in the
``<documents>`` prompt): the model answers FROM THE PROMPT —
the deterministic answer ``Already in context: Read
<source/path>. <first 80 chars of the target document's text
as it appears in the ``<documents>`` block>`` (same citation
shape as the read-result branch — the document text reached
the model either way, and the answer proves it);
* any other last result (a top-level or folder LISTING landed):
the deterministic ECHO answer ``Here's the level I listed:\n
<the listing, verbatim>`` — the mock echoes what it received
(the house scripted-turn way of asserting on tool results):
the suite asserts on the exact tree level — the ``— N
documents`` lines, the stored folder summaries, the
``source: X | path: Y | title: Z`` file lines, and the 50-line
cap + ``…and N more documents…`` note — through the rendered
answer, the only E2E lens on the LLM's context.
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
phrases — the phase-71/72 ordering convention; the trigger needs
the ``<tools>`` section, so deflected turns never hit it); no
existing E2E question or fixture file contains the phrase, so
every other suite is unaffected.
- user message containing ``what are the correct llama.cpp
arguments`` (``GREP_TEACH_TRIGGER``, the 2026-09-05 incident —
the harness prior is that grep takes a REGEX; this app's grep is a
@@ -302,7 +378,8 @@ from typing import Any
from fastapi import FastAPI
from fastapi.responses import JSONResponse, StreamingResponse
from app.rag.agent import CORRECTION_INSTRUCTION # phase 71: the harness constant
from app.rag.agent import ALREADY_IN_CONTEXT, CORRECTION_INSTRUCTION
from app.rag.folder_summaries import FOLDER_HEADER_PREFIX # phase 94: the mock's key
app = FastAPI()
@@ -588,9 +665,92 @@ GREP_TEACH_PLAIN = "qwen3.8"
#: plain step keys on it (a plain no-match line carries it not).
GREP_TEACH_MARKER = "grep matches a plain substring"
#: The agent's ``ls`` listing header (app.rag.agent ``_execute_tool``):
#: ``"N documents:"`` — the first line of every catalog tool result.
_CATALOG_HEADER_RE = re.compile(r"^\d+ documents:")
# ---------------------------------------------------------------------------
# Phase 94 (task 04, the drill-down ls's dedicated story suite):
# the deterministic SCRIPTED drill-down turns — see the module docstring
# ---------------------------------------------------------------------------
#: A user message containing this substring (case-insensitive) —
#: combined with the ``<tools>`` section in the system prompt — drives
#: the scripted DRILL-DOWN flow: the question carries its own tool call
#: after the colon (``drill down the tree: ls [target]`` / ``drill
#: down the tree: read source/path`` — the suite's scripted turns,
#: ``tests/e2e/test_ls_tree_drilldown.py``). Checked BEFORE the plain
#: ``TOOLS_TRIGGER`` flow (disjoint trigger phrases — the phase-71/72
#: ordering convention); verified: no existing E2E question or fixture
#: file contains the phrase, so every other suite is unaffected.
DRILL_TRIGGER = "drill down the tree"
#: The scripted call in the drill-down question (case-insensitive — the
#: suite's questions capitalize the trigger's first letter): the verb
#: (``ls`` / ``read``) plus the optional target — a source name or
#: ``source/folder`` path (``ls``) or a combined ``source/path``
#: (``read``), parsed from the RAW user message so the target keeps its
#: case. The target is a ``[a-z0-9_./-]`` run (case-insensitively), so
#: the suite's `` — `` flavor separator (em dash) can never bleed into
#: it; no run = the top-level ``ls`` (no ``path`` argument).
_DRILL_CALL_RE = re.compile(
r"drill down the tree:\s*(?P<verb>ls|read)(?:\s+(?P<arg>[a-z0-9_./-]+))?",
re.I,
)
#: The agent's NOT-A-FOLDER teaching refusal marker (app.rag.agent
#: ``NOT_A_FOLDER`` — ``'{arg}' is not a folder — {parent} has:
#: {subfolders}``): the drill-down flow's scripted recovery keys on it
#: in the LAST tool result (the refusal being visible to the model is
#: exactly what triggers the one-round recovery — the phase-72
#: self-correction contract carrying the tree's teaching line).
_NOT_A_FOLDER_MARKER = "is not a folder"
#: The stable substring of the harness-owned dedupe refusal the
#: drill-down flow's ``ctx_answer`` branch keys on (a read target that
#: is already a top-2 retrieval document — the mock then answers from
#: the document's text in the ``<documents>`` prompt block, exactly
#: what the refusal instructs). Keyed on a substring (not the whole
#: constant) so a re-wrap of the constant cannot silently re-route the
#: mock; the module-level assert below fails loudly if the substring
#: ever leaves the constant (the mock must never drift from
#: ``app.rag.agent.ALREADY_IN_CONTEXT``).
_ALREADY_IN_CONTEXT_MARKER = "Already in your context"
assert _ALREADY_IN_CONTEXT_MARKER in ALREADY_IN_CONTEXT, (
"mock drift: the dedupe marker left ALREADY_IN_CONTEXT"
)
#: One ``<document>`` block of the HIGH prompt's ``<documents>``
#: section (``app.rag.prompts.build_high_prompt``): the block is the
#: document identity (``source``/``path``/``title`` attributes) plus
#: the document's FULL text (never truncated on the retrieval path,
#: owner-locked A7) between the tags.
_DOCUMENT_BLOCK_RE = re.compile(
r'<document source="(?P<source>[^"]+)" path="(?P<path>[^"]+)" '
r'title="[^"]*">\n(?P<content>.*?)\n</document>',
re.S,
)
def _document_block(system: str, source: str, path: str) -> str | None:
"""The stored text of one ``<document>`` block (or ``None``).
The drill-down flow's ``ctx_answer`` branch: when the agent's read
of a top-2 retrieval document gets the ALREADY_IN_CONTEXT dedupe,
the document's full text is in the ``<documents>`` prompt — the
mock (the model) extracts it by the block's identity attributes
and quotes it, answering from the prompt as the refusal instructs.
"""
for block in _DOCUMENT_BLOCK_RE.finditer(system):
if block.group("source") == source and block.group("path") == path:
return block.group("content")
return None
#: The agent's drill-down ``ls`` result shapes (app.rag.agent
#: ``render_ls_top`` / ``render_folder_listing``, phase 94): the
#: top-level header ``"N sources:"`` and the per-source block line
#: ``"<source> — N documents"`` (summary lines are indented — they
#: never match); the folder-level header ``"<identity> — N documents,
#: M folders:"`` (identity = the source name or ``source/folder``).
_SOURCE_HEADER_RE = re.compile(r"^\d+ sources:$")
_SOURCE_LINE_RE = re.compile(r"^(?P<source>.+?) — (?P<n>\d+) documents$")
_FOLDER_HEADER_RE = re.compile(r"^(?P<identity>.+?) — \d+ documents, \d+ folders:$")
#: One DEAD app-level chat attempt costs exactly this many HTTP POSTs
#: while the endpoint stays down: the openai SDK's default policy
@@ -713,26 +873,60 @@ def _tool_results(body: dict[str, Any]) -> list[str]:
]
def _first_catalog_line(body: dict[str, Any]) -> str | None:
"""The first catalog line of a catalog listing in the messages.
def _first_file_line(body: dict[str, Any]) -> str | None:
"""The first file line (``source: X | path: Y | title: Z``) across
the ``tool``-role results in the messages, in message order (read
results — full documents, not listings — skipped; the ``_CATALOG_-
LINE_RE`` machinery).
A catalog listing is a ``tool``-role result whose FIRST line is the
agent's ``"N documents:"`` header (``_CATALOG_HEADER_RE``); its
first ``source: X | path: Y | title: Z`` line (the
``_CATALOG_LINE_RE`` machinery) is returned. ``None`` when no
catalog listing is in the messages — e.g. while only the teaching
refusal is there (the phase-72 LS-TEACH flow's request-2 state).
An empty listing (``"0 documents:"`` with no lines) returns
``""`` — the listing is present, it is just empty.
Phase 94: the drill-down ``ls`` carries file lines only at folder
levels — the top-level source listing and the teaching refusals
have none, so ``None`` here means "no folder level reached yet".
"""
for content in _tool_results(body):
lines = content.splitlines()
if not lines or not _CATALOG_HEADER_RE.match(lines[0]):
if content.startswith(_READ_RESULT_PREFIX):
continue
for line in lines[1:]:
for line in content.splitlines():
if _CATALOG_LINE_RE.match(line):
return line
return ""
return None
def _drill_target(body: dict[str, Any]) -> str | None:
"""The next source to drill into (phase 94, the drill step).
The drill-down ``ls`` top level lists SOURCES only (no file lines),
so a deterministic flow that needs a file line must drill one level:
``ls`` scoped to a source. The target is the FIRST source of the
top-level listing (registry order — the listing's own order) that
does not yet have a folder-level listing in the messages (a folder
header whose identity is the source or ``source/…``); ``None`` when
no top-level listing is in the messages yet (nothing to drill from)
or every listed source has been drilled (an empty KB — the flow
degenerates to the old re-list loop, settling at the round cap
exactly like the phase-70 empty-catalog case).
"""
sources: list[str] = []
drilled: set[str] = set()
for content in _tool_results(body):
lines = content.splitlines()
if not lines:
continue
if _SOURCE_HEADER_RE.match(lines[0]):
for line in lines[1:]:
match = _SOURCE_LINE_RE.match(line)
if match:
sources.append(match.group("source"))
else:
folder = _FOLDER_HEADER_RE.match(lines[0])
if folder:
drilled.add(folder.group("identity"))
for source in sources:
if not any(
identity == source or identity.startswith(source + "/")
for identity in drilled
):
return source
return None
@@ -853,11 +1047,14 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
``TOOLS_TRIGGER`` and ``MULTI_READ_TRIGGER``), classified by the
count of ``tool``-role read results:
* 0 read results: ``("list", "", "")`` (no catalog yet) or
``("read", source, path, "call_1")`` on the FIRST catalog doc.
* 1 read result: ``("read", source, path, "call_2")`` on the SECOND
catalog doc — the first listing line whose ``source/path``
differs from the one already read. A one-document catalog
* 0 read results: ``("list", "", "")`` (no listing yet),
``("drill", source, "call_1")`` when the top-level source
listing is in the messages but no file line yet (phase 94: the
drill — the top level carries sources only), or
``("read", source, path, "call_2")`` on the FIRST file-line doc.
* 1 read result: ``("read", source, path, "call_3")`` on the SECOND
file-line doc — the first listing line whose ``source/path``
differs from the one already read. A one-file listing
degenerates to the single-read ``("answer", ...)`` shape (nothing
second to read).
* 2 read results: ``("multi_answer", "", text)`` — the forced
@@ -880,30 +1077,37 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
if not body.get("tools"):
return None
docs = _catalog_docs(body)
if not docs:
return ("list", "", "")
return ("read", docs[0][0], docs[0][1], "call_1")
if docs:
return ("read", docs[0][0], docs[0][1], "call_2")
drill = _drill_target(body)
if drill is not None:
return ("drill", drill, "call_1")
return ("list", "", "")
if len(reads) == 1:
skip = reads[0][0]
second = next(
(d for d in _catalog_docs(body) if f"{d[0]}/{d[1]}" != skip), None
)
if second is None:
# One-document catalog: nothing second to read — the
# One-file listing: nothing second to read — the
# single-read answer shape (deterministic degenerate).
return ("answer", reads[0][0], reads[0][1])
return ("read", second[0], second[1], "call_2")
return ("read", second[0], second[1], "call_3")
(sp1, c1), (sp2, _c2) = reads[0], reads[1]
answer = f"Read {sp1}. {c1[:80]} I read {sp1} and {sp2}."
return ("multi_answer", "", answer)
# Phase-37 single-read flow — byte-identical to the original.
# Phase-37 single-read flow (phase 94: the drill step between the
# top-level listing and the first file line).
if reads:
return ("answer", reads[0][0], reads[0][1])
if not body.get("tools"):
return None
docs = _catalog_docs(body)
if docs:
return ("read", docs[0][0], docs[0][1], "call_1")
return ("read", docs[0][0], docs[0][1], "call_2")
drill = _drill_target(body)
if drill is not None:
return ("drill", drill, "call_1")
return ("list", "", "")
@@ -915,13 +1119,17 @@ def _ls_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
with ``{"path": "."}`` (id ``call_0``), ``finish_reason:
"tool_calls"``, no content.
* ``("correct",)`` — a ``tool``-role result is in the messages and
it is NOT a catalog listing (the teaching refusal): the
correction — ``ls`` with no arguments (id ``call_1``).
* ``("answer", line)`` — a ``tool``-role result whose first line
is the ``"N documents:"`` catalog header: the deterministic
no file line exists yet (the teaching refusal): the correction —
``ls`` with no arguments (id ``call_1``).
* ``("drill", source, "call_2")`` — the top-level source listing
is in the messages (the correction ran) but no file line yet
(phase 94: the top level carries sources only): the drill —
``ls`` scoped to the first source of the listing.
* ``("answer", line)`` — a ``source: X | path: Y | title: Z`` file
line is in the messages (the folder listing): the deterministic
content answer ``These are the indexed documents: <line>`` (the
first catalog line), ``finish_reason: "stop"`` — the loop
settled in ONE correction, not at the round cap.
first file line), ``finish_reason: "stop"`` — the loop settled
in ONE correction + ONE drill, not at the round cap.
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
section is missing (deflected turns never carry it), or
``tools`` are not offered and no tool results are in the
@@ -931,9 +1139,12 @@ def _ls_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
return None
if "<tools>" not in _system(body):
return None
line = _first_catalog_line(body)
line = _first_file_line(body)
if line is not None:
return ("answer", line)
drill = _drill_target(body)
if drill is not None:
return ("drill", drill, "call_2")
if _tool_results(body):
return ("correct",)
if not body.get("tools"):
@@ -996,6 +1207,92 @@ def _grep_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
return ("nomatch",)
def _drill_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
"""Classify a phase-94 scripted drill-down request (see the module
docstring). The question carries the scripted call
(``drill down the tree: ls [target]`` / ``drill down the tree: read
source/path``); the step is then discriminated statelessly from the
tool results, like the other marker flows:
* ``("call", verb, target, "call_0")`` — ``tools`` are offered and
no ``tool``-role result is in the messages yet: the scripted call
(``ls`` with NO path for the top level — empty target —, ``ls``
with the target for source/folder levels, ``read`` with the
combined ``source/path``).
* ``("recover", source)`` — the LAST tool result is the NOT-A-FOLDER
teaching refusal (it carries :data:`_NOT_A_FOLDER_MARKER`): the
scripted one-round recovery — ``ls`` the target's SOURCE segment
(the refusal listing the parent's subfolders is what lets the
model — and this mock — self-correct in the next round, the
phase-72 contract).
* ``("read_answer", combined, quote)`` — the LAST tool result is a
read result (``"Document <source/path>:\n<content>"``): the
deterministic echo answer ``Read <source/path>. <first 80 chars>
`` (the phase-37 single-read shape — the grounded-turn citation
contract the suite asserts).
* ``("ctx_answer", target, quote)`` — the scripted call was a
``read`` and the LAST tool result is the ALREADY_IN_CONTEXT
dedupe refusal (the target is already a top-2 retrieval document
— deterministic for the drill fixture: the read question names
the file's path, so the file self-matches the hybrid gate): the
model answers FROM THE PROMPT — ``Already in context: Read
<target>. <first 80 chars of the target document's text in the
``<documents>`` block>`` (the same citation shape as the
read-result branch — the document text reached the model either
way). Falls through to the echo when the target is not a
``<document>`` block (the premise broke — the suite fails
loudly on the answer).
* ``("echo", listing)`` — a listing landed (top-level or folder —
any other last result): the deterministic ECHO — the answer
carries the listing VERBATIM (``Here's the level I listed:\n
<listing>``), so the suite asserts on the exact tree level the
model saw (source ``— N documents`` lines + stored summaries,
subfolder lines, the ``source: X | path: Y | title: Z`` file
lines, the 50-line cap + note) through the rendered answer — the
house scripted-turn way of asserting on tool results (the mock is
the only E2E lens on the LLM's context).
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
section is missing (deflected turns never carry it), the scripted
call is unparseable, or ``tools`` are not offered and no tool
results are in the messages yet (e.g. ``agent_max_rounds=0``).
"""
user = _user(body)
if DRILL_TRIGGER not in user.lower():
return None
if "<tools>" not in _system(body):
return None
match = _DRILL_CALL_RE.search(user)
if match is None:
return None
verb = match.group("verb")
target = match.group("arg") or ""
results = _tool_results(body)
if not results:
if not body.get("tools"):
return None
return ("call", verb, target, "call_0")
last = results[-1]
if last.startswith(_READ_RESULT_PREFIX):
head, _, content = last.partition("\n")
# The read result is ``"Document <source/path>:\n<content>"`` —
# the head carries the server's appended ``:`` (removed here;
# a document path never legitimately ends with one).
combined = head[len(_READ_RESULT_PREFIX):].strip().removesuffix(":")
return ("read_answer", combined, content[:80])
if _ALREADY_IN_CONTEXT_MARKER in last and verb == "read" and "/" in target:
# The dedupe fired: the read target is already a top-2
# retrieval document, so its FULL text is in the
# ``<documents>`` prompt — answer from the prompt (the
# refusal's instruction), quoting the block's text.
src, _, p = target.partition("/")
content = _document_block(_system(body), src, p)
if content is not None:
return ("ctx_answer", target, content[:80])
if _NOT_A_FOLDER_MARKER in last:
return ("recover", target.split("/")[0])
return ("echo", last)
def long_answer() -> str:
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
a unique final line that must survive the stream untruncated."""
@@ -1103,6 +1400,28 @@ def compose_answer(body: dict[str, Any]) -> str:
user = _user(body)
if LONG_ANSWER_TRIGGER in user.lower():
answer = long_answer()
elif "FOLDER_SUMMARY_MODE" in system:
# Folder summary (phase 94, TODO.md L4): the ``lite`` stand-in
# returns the deterministic one-liner
# ``Fixture folder summary for <folder>.`` — <folder> is the
# user message's ``Folder: …`` header line (the generator puts
# the canonical source/folder identity there, imported as
# ``FOLDER_HEADER_PREFIX`` so the mock can never drift from it).
# The stored row therefore always names its folder — the drill-
# down E2E (``test_ls_tree_drilldown.py``) asserts on it.
# Checked BEFORE the ``SUMMARY_MODE`` branch: the folder marker
# CONTAINS the summary marker as a substring (``FOLDER_`` +
# ``SUMMARY_MODE``), so the summary branch would otherwise
# shadow every folder-summary call. Checked BEFORE the
# DEFLECT_MODE branch, like the other lite-mode markers (a
# deflection prompt never carries one).
header = user.splitlines()[0] if user else ""
folder = (
header.removeprefix(FOLDER_HEADER_PREFIX).strip()
if header.startswith(FOLDER_HEADER_PREFIX)
else "(unnamed folder)"
)
answer = f"Fixture folder summary for {folder}."
elif "SUMMARY_MODE" in system:
# Document summaries (phase 30): the ``lite`` stand-in returns a
# deterministic digest — the first 24 tokens of the user message
@@ -1594,9 +1913,16 @@ def chat_completions(body: dict[str, Any]) -> Any:
# The incident's misuse, deterministic: ls(path='.').
stream = _tool_call_stream("ls", {"path": "."}, "call_0")
elif ls_teach[0] == "correct":
# The one-round correction: the no-arg full listing.
# The one-round correction: the no-arg top-level
# listing (phase 94: sources only — the drill follows).
stream = _tool_call_stream("ls", {}, "call_1")
else: # "answer" — quote the first catalog line
elif ls_teach[0] == "drill":
# Phase 94: the top level lists sources only — drill
# one level into the first source for the file lines.
stream = _tool_call_stream(
"ls", {"path": ls_teach[1]}, ls_teach[2]
)
else: # "answer" — quote the first file line
answer = _apply_max_tokens(
f"These are the indexed documents: {ls_teach[1]}",
body.get("max_tokens"),
@@ -1607,10 +1933,67 @@ def chat_completions(body: dict[str, Any]) -> Any:
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
# Phase 94 (task 04): the deterministic SCRIPTED drill-down
# turns (sources → folders → files → the grounded read; the
# 50-line cap; the NOT-A-FOLDER teaching + scripted recovery) —
# checked BEFORE the plain TOOLS_TRIGGER flow (disjoint trigger
# phrases — the phase-71/72 ordering convention; the trigger
# needs the ``<tools>`` section, so deflected turns never hit
# it).
drill = _drill_flow(body)
if drill is not None:
if drill[0] == "call":
# The scripted call: ls with no path at the top level
# (empty target), ls/read with the parsed target.
stream = _tool_call_stream(
drill[1], {"path": drill[2]} if drill[2] else {}, drill[3]
)
elif drill[0] == "recover":
# The scripted one-round recovery after the NOT-A-FOLDER
# teaching: ls the target's source (the parent level).
stream = _tool_call_stream("ls", {"path": drill[1]}, "call_1")
elif drill[0] == "read_answer":
# Quote the read document (first 80 chars) — the
# phase-37 single-read shape.
stream = _sse_stream(
_apply_max_tokens(
f"Read {drill[1]}. {drill[2]}", body.get("max_tokens")
),
0.0,
)
elif drill[0] == "ctx_answer":
# The dedupe fired: the target's full text is in the
# <documents> prompt — answer from it (same citation
# shape, prefixed so the suite pins the dedupe path).
stream = _sse_stream(
_apply_max_tokens(
f"Already in context: Read {drill[1]}. {drill[2]}",
body.get("max_tokens"),
),
0.0,
)
else: # "echo" — the listing verbatim (the suite's lens)
stream = _sse_stream(
_apply_max_tokens(
f"Here's the level I listed:\n{drill[1]}",
body.get("max_tokens"),
),
0.0,
)
return StreamingResponse(
stream,
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
flow = _tool_flow(body)
if flow is not None:
if flow[0] == "list":
stream = _tool_call_stream("ls", {}, "call_0")
elif flow[0] == "drill":
# Phase 94: the drill step — ls scoped to the first
# source of the top-level listing (flow[1], flow[2] is
# the synthetic call id).
stream = _tool_call_stream("ls", {"path": flow[1]}, flow[2])
elif flow[0] == "read":
# flow[3] is the synthetic call id — "call_1" for the
# single-read flow and the multi-read first read,
tests/e2e/test_agent_document_tools.py
+53 -21
View File
@@ -11,15 +11,22 @@ deterministic marker flow in ``tests/e2e/mock_llm.py`` (user message
contains ``use your tools`` **and** the system prompt carries the
``<tools>`` section of the HIGH prompt; phase 70: the flow emits the
harness-aligned names — ``ls`` / ``read`` with the combined
``source/path`` identity):
``source/path`` identity; phase 94: the drill-down ``ls`` — the top
level lists sources only, so the flow drills one level into the first
source before the first file line exists):
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
``tool_calls`` deltas calling ``ls`` (id ``call_0``, no arguments,
``finish_reason: "tool_calls"``);
2. request 2 (a ``tool``-role catalog result in the messages) → streams a
``tool_calls`` delta calling ``read`` on the JOINED combined
``source/path`` of the FIRST catalog line (id ``call_1``);
3. request 3 (a ``tool``-role read result in the messages) → the content
2. request 2 (the top-level source listing in the messages — no file
lines yet) → a ``tool_calls`` delta — ``ls`` scoped to the FIRST
source of the listing (id ``call_1``) — the drill step (the seed
registers ``Deployments`` first, so the drill — and therefore the
read — lands on the JSON file);
3. request 3 (a ``tool``-role folder listing with file lines) → streams
a ``tool_calls`` delta calling ``read`` on the JOINED combined
``source/path`` of the FIRST file line (id ``call_2``);
4. request 4 (a ``tool``-role read result in the messages) → the content
answer ``Read <source/path>. <first 80 chars of the read document's
content>`` — so the suite can assert the read document reached the
model and landed in the answer.
@@ -72,7 +79,7 @@ from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import Chunk, Document, QueryLog
from app.models import Chunk, Document, GitSource, QueryLog
from e2e.auth_helpers import login
from tests.e2e.mock_llm import embed_text
@@ -148,7 +155,22 @@ READ_CHIP_HREF = f"/document.html?source={READ_SOURCE}&path={READ_PATH}&back=%2F
def _seed(db: Session) -> None:
"""The two-document pair from the TODO (see the module docstring)."""
"""The two-document pair from the TODO (see the module docstring).
Phase 94: the drill-down ``ls`` top level reads the registry — the
seed registers BOTH sources (TRUNCATEd in ``_reset_db``),
``Deployments`` FIRST: registry order is ``(added_at, id)``, so the
mock's drill (first source of the listing) lands on the JSON file
deterministically — independent of the operator's
``BOR_GIT_SOURCES`` (a non-empty table ignores the env fallback).
"""
# COMMIT between the inserts (not flush): ``added_at`` is
# ``server_default now()`` — the transaction timestamp — and the
# tie-break is the random uuid ``id``, so one-transaction rows order
# nondeterministically.
db.add(GitSource(url=READ_SOURCE, kind="local"))
db.commit()
db.add(GitSource(url=SEED_SOURCE, kind="local"))
md = Document(
source=SEED_SOURCE,
path=SEED_PATH,
@@ -195,7 +217,10 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
"""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources"
)
)
db.commit()
if seed is not None:
@@ -389,12 +414,14 @@ def test_marker_question_lists_reads_and_quotes(
assert i_list is not None and i_read is not None, statuses
assert i_list < i_read, statuses
# Wire level: exactly two `tool` frames — ``ls`` then ``read`` (the
# combined source/path as the model passed it) — and both ahead of
# the first `delta` frame.
# Wire level: exactly three `tool` frames — ``ls`` (the top level),
# the drill ``ls`` scoped to the first source (phase 94), then
# ``read`` (the combined source/path as the model passed it) — and
# all three ahead of the first `delta` frame.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": READ_SOURCE},
{"type": "tool", "name": "read", "argument": READ_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
@@ -408,12 +435,15 @@ def test_marker_question_lists_reads_and_quotes(
(READ_SOURCE, READ_PATH),
]
# Both tool lines, in order, above the answer.
# All three tool lines, in order, above the answer (phase 94: the
# drill line is "Listing documents in <source>").
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1)).to_contain_text(READ_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1)).to_contain_text(READ_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2)).to_contain_text(READ_SP)
# The final answer quotes the read document (the mock's deterministic
# quote: "Read <source/path>. <first 80 chars of its content>").
@@ -451,18 +481,20 @@ def test_tool_lines_re_render_after_reload(
_submit(page, MARKER_QUESTION)
_wait_settled(page)
expect(page.locator(".msg.brain .tool-call")).to_have_count(2)
expect(page.locator(".msg.brain .tool-call")).to_have_count(3)
page.reload()
expect(page.locator("#empty-state")).to_be_hidden()
# The persisted record re-renders BOTH tool lines, in saved order,
# through the same append helper as the live frames.
# The persisted record re-renders ALL THREE tool lines, in saved
# order, through the same append helper as the live frames.
restored = page.locator(".msg.brain .tool-call")
expect(restored).to_have_count(2)
expect(restored).to_have_count(3)
expect(restored.nth(0)).to_contain_text("Listing documents")
expect(restored.nth(1)).to_contain_text("Reading ")
expect(restored.nth(1)).to_contain_text(READ_SP)
expect(restored.nth(1)).to_contain_text("Listing documents in")
expect(restored.nth(1)).to_contain_text(READ_SOURCE)
expect(restored.nth(2)).to_contain_text("Reading ")
expect(restored.nth(2)).to_contain_text(READ_SP)
# Answer + the read-document chip are intact (phase-14 restore path).
bubble = page.locator(".msg.brain .bubble").last
tests/e2e/test_agent_unlimited_tools.py
+67 -30
View File
@@ -11,19 +11,26 @@ message contains BOTH ``use your tools`` (``TOOLS_TRIGGER``) and ``read
two documents`` (``MULTI_READ_TRIGGER``) **and** the system prompt
carries the ``<tools>`` section of the HIGH prompt; phase 70: the flow
emits the harness-aligned names — ``ls``, then ``read`` on the JOINED
combined ``source/path`` of each catalog line):
combined ``source/path`` of each file line; phase 94: the drill-down
``ls`` — the top level lists sources only, so the flow drills one
level into the first source before the first file line exists):
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
``tool_calls`` deltas calling ``ls`` (id ``call_0``);
2. request 2 (the ``tool``-role catalog result) → ``read`` on the
JOINED combined ``source/path`` of the FIRST catalog line
(id ``call_1``);
3. request 3 (one ``tool``-role read result) → ``read`` on the JOINED
combined ``source/path`` of the SECOND catalog line (id ``call_2``)
— the pre-phase-45 per-tool budgets would have refused exactly this
2. request 2 (the top-level source listing in the messages — no file
lines yet) → the drill: ``ls`` scoped to the FIRST source of the
listing (id ``call_1``); the seed registers ``Deployments`` first,
and both read documents live in it — so the drill's folder listing
carries BOTH file lines;
3. request 3 (a ``tool``-role folder listing with file lines) →
``read`` on the JOINED combined ``source/path`` of the FIRST file
line (id ``call_2``);
4. request 4 (one ``tool``-role read result) → ``read`` on the JOINED
combined ``source/path`` of the SECOND file line (id ``call_3``) —
the pre-phase-45 per-tool budgets would have refused exactly this
second read (``No reading budget left — answer with what you
have.``);
4. request 4 (two read results) → the forced answer, byte-stable: the
5. request 5 (two read results) → the forced answer, byte-stable: the
single-read shape quoting the FIRST read result, plus the line
``I read <sp1> and <sp2>.`` naming both read paths in read order.
@@ -49,10 +56,11 @@ rejection, not the multi-read
flow this story proves.
Test → story mapping (Playwright Mapping Rule):
1. ``test_multi_read_turn`` — the turn streams THREE ``tool`` frames /
``.tool-call`` lines in order (one list — "is listing documents" —
and two reads — "is reading <source/path>" — the #send-status
transition recorded deterministically via MutationObserver), then a
1. ``test_multi_read_turn`` — the turn streams FOUR ``tool`` frames /
``.tool-call`` lines in order (the top-level list, the drill list —
phase 94 — "is listing documents", and two reads — "is reading
<source/path>" — the #send-status transition recorded
deterministically via MutationObserver), then a
final non-deflected answer containing the mock's byte-stable
``I read <sp1> and <sp2>.`` line; the round cap (default 10) bounds
the turn, no budget refusal anywhere.
@@ -84,7 +92,7 @@ from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import Chunk, Document, QueryLog
from app.models import Chunk, Document, GitSource, QueryLog
from e2e.auth_helpers import login
from tests.e2e.mock_llm import embed_text
@@ -209,7 +217,23 @@ def _doc(source: str, path: str, title: str, content: str) -> Document:
def _seed(db: Session) -> None:
"""The three-document KB from the module docstring."""
"""The three-document KB from the module docstring.
Phase 94: the drill-down ``ls`` top level reads the registry —
register BOTH sources (TRUNCATEd in ``_reset_db``), ``Deployments``
FIRST (registry order is ``(added_at, id)``): the mock's drill
(first source of the listing) lands on the folder that carries
BOTH file lines. A non-empty table also ignores the operator's
``BOR_GIT_SOURCES`` fallback — deterministic.
"""
# COMMIT between the inserts (not flush): ``added_at`` is
# ``server_default now()`` — the transaction timestamp — and the
# tie-break is the RANDOM uuid ``id``, so two rows in one
# transaction order nondeterministically (the integration
# ``registry`` fixture's pattern).
db.add(GitSource(url=READ1_SOURCE, kind="local"))
db.commit()
db.add(GitSource(url=SEED_SOURCE, kind="local"))
md = _doc(SEED_SOURCE, SEED_PATH, "AWS Route 53 Notes", ROUTE53_CONTENT)
db.add(md)
db.flush()
@@ -240,7 +264,10 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
"""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources"
)
)
db.commit()
if seed is not None:
@@ -399,13 +426,16 @@ def test_multi_read_turn(
_submit(page, MULTI_QUESTION)
_wait_settled(page)
# Wire level: exactly THREE `tool` frames — ls, read #1, read #2
# (each read's argument is the JOINED combined source/path), in
# order — and all ahead of the first `delta` frame. This third
# frame is the one the pre-phase-45 read budget refused.
# Wire level: exactly FOUR `tool` frames — the top-level ls, the
# drill ls scoped to the first source (phase 94), then read #1 and
# read #2 (each read's argument is the JOINED combined
# source/path), in order — and all ahead of the first `delta`
# frame. The fourth frame is the one the pre-phase-45 read budget
# refused.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": READ1_SOURCE},
{"type": "tool", "name": "read", "argument": READ1_SP},
{"type": "tool", "name": "read", "argument": READ2_SP},
]
@@ -439,14 +469,17 @@ def test_multi_read_turn(
), statuses
assert i_list < i_read1 < i_read2, statuses
# Three visible tool lines, in order, above the answer.
# Four visible tool lines, in order, above the answer (phase 94:
# the drill line is "Listing documents in <source>").
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(3)
expect(lines).to_have_count(4)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1)).to_contain_text(READ1_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1)).to_contain_text(READ1_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2)).to_contain_text(READ2_SP)
expect(lines.nth(2)).to_contain_text(READ1_SP)
expect(lines.nth(3)).to_contain_text("Reading ")
expect(lines.nth(3)).to_contain_text(READ2_SP)
# The final answer is non-deflected, quotes the FIRST read result,
# and names BOTH read paths (the mock's byte-stable line).
@@ -563,19 +596,23 @@ def test_single_tool_flow_regression(
_submit(page, SINGLE_QUESTION)
_wait_settled(page)
# Exactly TWO tool frames — ls then ONE read of the first catalog
# line (the JOINED combined source/path) — no second read (the
# marker carries no multi-read trigger).
# Exactly THREE tool frames — the top-level ls, the drill ls
# (phase 94), then ONE read of the first file line (the JOINED
# combined source/path) — no second read (the marker carries no
# multi-read trigger).
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": READ1_SOURCE},
{"type": "tool", "name": "read", "argument": READ1_SP},
]
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1)).to_contain_text(READ1_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1)).to_contain_text(READ1_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2)).to_contain_text(READ1_SP)
# The single-read answer shape: quotes the read document; it does
# NOT carry the multi-read both-named line (READ2 was never read).
tests/e2e/test_big_read_progress.py
+58 -31
View File
@@ -19,27 +19,33 @@ phase-64 default 0.15 is far below the 5 s tool-line threshold):
NOTE the prelude: the chat flow's QUESTION EMBEDDING also travels
through the proxy (one 6 s sleep) before the agent loop starts, so
the wall-clock timeline of one turn is:
the wall-clock timeline of one turn is (phase 94: the drill-down
``ls`` adds a drill step — the top level lists sources only, so the
flow drills one level before the first file line exists):
1. t≈12 s — the first SSE ``tool`` frame ("🔎 Listing documents"): the
6 s embedding sleep + request 1's 6 s sleep (``tools`` offered, no
tool results yet — streams ONLY the ``ls`` ``tool_calls`` delta,
id ``call_0``) + the mock's ~0.2 s tool-call stream;
2. t≈19 s — the second ``tool`` frame ("📄 Reading
Deployments/example-record-file.json"): request 2's 6 s sleep (a
``tool``-role catalog result → streams a ``read`` ``tool_calls``
delta on the JOINED combined ``source/path`` of the FIRST catalog
line, id ``call_1``);
3. t≈25 s — the content answer ``Read <source/path>. <first 80 chars
of the read document's content>``: request 3's 6 s sleep (a
2. t≈19 s — the second ``tool`` frame ("🔎 Listing documents in
Deployments"): request 2's 6 s sleep (the top-level source listing
in the messages — no file lines yet → streams the drill ``ls``
scoped to the first source, id ``call_1``);
3. t≈25 s — the third ``tool`` frame ("📄 Reading
Deployments/example-record-file.json"): request 3's 6 s sleep (a
``tool``-role folder listing with file lines → streams a ``read``
``tool_calls`` delta on the JOINED combined ``source/path`` of the
FIRST file line, id ``call_2``);
4. t≈31 s — the content answer ``Read <source/path>. <first 80 chars
of the read document's content>``: request 4's 6 s sleep (a
``tool``-role read result) → the first answer ``delta``.
So each post-tool-frame gap (≈6.3 s — the 6 s sleep + the short
tool-call stream) is PAST the 5 s tool-line threshold
(``TOOL_LINE_ELAPSED_AFTER_MS = 5_000``), and the 10 s pre-token
typing hint (the existing ``startThinkingClock`` gate, ticking from
t≈0) is visible throughout both post-tool gaps — everything before the
answer's first delta (the whole turn runs ≈25 s + overhead, acceptable
t≈0) is visible throughout all post-tool gaps — everything before the
answer's first delta (the whole turn runs ≈31 s + overhead, acceptable
for an isolated story suite). The four tests pin: (1) the latest tool
line's ticking "(Ns)" suffix, (2) the typing indicator's visible "Ns"
hint (+ the kept aria channel), (3) BOTH settling the instant the
@@ -55,15 +61,20 @@ task that writes the visible span) — so the two channels read
polls until they align (the deterministic same-moment pin).
**KB fixture** — byte-identical to the phase-37 suite
(``tests/e2e/test_agent_document_tools.py``): ``Homelab/aws-route53.md``
(``tests/e2e/test_agent_document_tools.py``), plus the phase-94
registry rows: both sources are registered in ``git_sources``
(``Deployments`` FIRST — registry order ``(added_at, id)``), so the
mock's drill (first source of the top-level listing) lands on the
folder that carries the file lines. ``Homelab/aws-route53.md``
carries one chunk embedded with the mock's own bag-of-words vector —
the marker question (phase 37's exact question, which carries the
trigger phrase) cosines ≈0.69 against it, well past the E2E 0.30
threshold, so the turn is grounded and the HIGH prompt carries the
``<tools>`` section; ``Deployments/example-record-file.json`` is
indexed WITHOUT chunks, and its ``(source, path)`` sorts FIRST in the
catalog (``Deployments`` < ``Homelab``) — exactly the line the mock's
second request reads, so the answer is byte-stable.
folder listing's file lines (``Deployments`` < ``Homelab``) — exactly
the line the mock's read request targets, so the answer is
byte-stable.
Test → story mapping (Playwright Mapping Rule):
1. ``test_tool_line_shows_ticking_elapsed``
@@ -92,7 +103,7 @@ from sqlalchemy.orm import Session
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import Chunk, Document
from app.models import Chunk, Document, GitSource
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
@@ -211,8 +222,10 @@ SUFFIX_TIMEOUT_MS = 12_000
#: the indicator is removed at the first delta (≈25 s): 20 s from the
#: check start cannot flake.
TYPING_HINT_TIMEOUT_MS = 20_000
#: The answer settles at ≈19 s from submit; every settle wait here starts
#: ≥12 s in — 30 s keeps ≥2× headroom on the remaining window.
#: The answer settles at ≈31 s from submit (phase 94: the drill step
#: adds a third tool round); every settle wait here starts ≈17 s in
#: (after the first line's suffix) — 30 s keeps ≥2× headroom on the
#: remaining window.
SETTLE_TIMEOUT_MS = 30_000
#: The restore is a synchronous boot re-render — 15 s is ample.
RESTORE_TIMEOUT_MS = 15_000
@@ -360,7 +373,16 @@ def app_url(app_server: str) -> str:
def _seed(db: Session) -> None:
"""The phase-37 two-document pair, byte-identical (see the module
docstring)."""
docstring), plus the phase-94 registry rows: both sources
registered, ``Deployments`` FIRST — the mock's drill (first
source of the top-level listing) lands on the JSON file."""
# COMMIT between the inserts (not flush): ``added_at`` is
# ``server_default now()`` — the transaction timestamp — and the
# tie-break is the random uuid ``id``, so one-transaction rows order
# nondeterministically.
db.add(GitSource(url=READ_SOURCE, kind="local"))
db.commit()
db.add(GitSource(url=SEED_SOURCE, kind="local"))
md = Document(
source=SEED_SOURCE,
path=SEED_PATH,
@@ -408,7 +430,7 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, "
"steering_notes, kb_overview, saved_chats"
"steering_notes, kb_overview, saved_chats, git_sources"
)
)
db.commit()
@@ -514,7 +536,7 @@ def _wait_settled(page: Page) -> None:
#: The FIRST tool line's suffix, pinned to the line ("first .tool-call
#: child of the .tool-calls container" — the "Listing documents" line;
#: the read frame's second line arms its OWN clock 5 s later and is
#: the drill frame's second line arms its OWN clock 5 s later and is
#: never the target here).
FIRST_LINE_SUFFIX = "#messages .tool-calls .tool-call:first-child .tool-elapsed"
@@ -611,8 +633,8 @@ def test_typing_indicator_shows_visible_elapsed(
)
# The clock ticks while the gap holds: ≥1.5 s later the value is
# strictly greater (the read frame is ≥6 s away; the first delta
# — which removes the whole indicator — is ≈25 s from submit).
# strictly greater (the drill frame is ≥6 s away; the first delta
# — which removes the whole indicator — is ≈31 s from submit).
page.wait_for_timeout(int(SAMPLE_GAP_S * 1000))
v2 = _elapsed_value(
page.locator("#typing-indicator .typing-elapsed").first.text_content(),
@@ -644,7 +666,7 @@ def test_indicators_settle_when_the_answer_arrives(
# First the ticking state (the test-1 wait, reused)…
expect(page.locator(FIRST_LINE_SUFFIX)).to_be_visible(timeout=SUFFIX_TIMEOUT_MS)
# …then the answer arrives (≈25 s from submit — see the module
# …then the answer arrives (≈31 s from submit — see the module
# docstring's timeline) and settles.
_wait_settled(page)
@@ -661,13 +683,16 @@ def test_indicators_settle_when_the_answer_arrives(
expect(page.locator("#messages .tool-elapsed")).to_have_count(0)
expect(page.locator("#typing-indicator")).to_have_count(0)
# The tool lines remain — the permanent record, both present with
# their pinned text (phase 37's pattern).
# The tool lines remain — the permanent record, all three present
# with their pinned text (phase 37's pattern + the phase-94 drill
# line).
lines = page.locator("#messages .tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1)).to_contain_text(READ_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1)).to_contain_text(READ_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2)).to_contain_text(READ_SP)
# The answer bubble is complete (the mock's deterministic quote).
bubble = page.locator(".msg.brain .bubble").last
@@ -705,12 +730,14 @@ def test_restored_turn_has_no_timer(
expect(page.locator("#empty-state")).to_be_hidden(timeout=RESTORE_TIMEOUT_MS)
# The persisted record re-renders the tool lines, in saved order
# (the phase-37 reload pin, mirrored)…
# (the phase-37 reload pin, mirrored + the phase-94 drill line)…”
restored = page.locator("#messages .tool-call")
expect(restored).to_have_count(2, timeout=RESTORE_TIMEOUT_MS)
expect(restored).to_have_count(3, timeout=RESTORE_TIMEOUT_MS)
expect(restored.nth(0)).to_contain_text("Listing documents")
expect(restored.nth(1)).to_contain_text("Reading ")
expect(restored.nth(1)).to_contain_text(READ_SP)
expect(restored.nth(1)).to_contain_text("Listing documents in")
expect(restored.nth(1)).to_contain_text(READ_SOURCE)
expect(restored.nth(2)).to_contain_text("Reading ")
expect(restored.nth(2)).to_contain_text(READ_SP)
# …with NO timer (A6 — the restore never arms the clock) and the
# page otherwise settled.
tests/e2e/test_harness_aligned_tools.py
+52 -22
View File
@@ -13,9 +13,13 @@ deterministic marker flows in ``tests/e2e/mock_llm.py`` (phase 70: the
flows emit the NEW names with the NEW argument shapes):
* the READ flow (``use your tools`` (``TOOLS_TRIGGER``) + the HIGH
prompt's ``<tools>`` section): ``ls`` (id ``call_0``, no arguments) →
``read`` on the JOINED combined ``source/path`` of the first catalog
line (id ``call_1``) → the ``Read <source/path>. <quote>`` answer;
prompt's ``<tools>`` section; phase 94: the drill-down ``ls`` — the
top level lists sources only, so the flow drills one level before the
first file line exists): ``ls`` (id ``call_0``, no arguments) → the
drill ``ls`` scoped to the first source of the listing
(id ``call_1``) → ``read`` on the JOINED combined ``source/path`` of
the first file line (id ``call_2``) → the ``Read <source/path>.
<quote>`` answer;
* the SEARCH flow (``search your documents`` (``SEARCH_TRIGGER``) + the
``<tools>`` section): ``grep`` with ``{"pattern": SEARCH_PATTERN}``
(id ``call_0``) → the ``Found <matched line>`` answer.
@@ -41,13 +45,14 @@ KB fixtures:
Test → phase mapping (Playwright Mapping Rule):
1. ``test_read_flow_lines_answer_sources_no_raw_markup`` — the
grounded READ turn: the UI shows the ``ls`` line (unscoped "🔎
Listing documents", no argument) then the "📄 Reading <source/path>"
line with the combined path in a ``<code>`` element, the answer
streams and quotes the read document, the done-state sources
include the read document, and NO raw tool markup (``<|…|>``,
``tool_call``) appears anywhere in the DOM — the live incident this
phase fixes.
grounded READ turn: the UI shows the unscoped ``ls`` line ("🔎
Listing documents", no argument), the drill line ("🔎 Listing
documents in <source>" — phase 94, the source in a ``<code>``
element), then the "📄 Reading <source/path>" line with the
combined path in a ``<code>`` element, the answer streams and quotes
the read document, the done-state sources include the read document,
and NO raw tool markup (``<|…|>``, ``tool_call``) appears anywhere
in the DOM — the live incident this phase fixes.
2. ``test_grep_flow_line_then_answer`` — the grounded SEARCH turn: the
"🔎 Searching for <pattern>" line (sentinel in ``<code>``) then the
matched-line answer.
@@ -74,7 +79,7 @@ from sqlalchemy.orm import Session
from app.config import Settings
from app.db import SessionLocal
from app.models import Chunk, Document
from app.models import Chunk, Document, GitSource
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from e2e.auth_helpers import login
@@ -156,7 +161,22 @@ READ_ANSWER_QUOTE = RECORD_FILE_CONTENT[:80]
def _seed_read_pair(db: Session) -> None:
"""The two-document READ-flow KB (see the module docstring)."""
"""The two-document READ-flow KB (see the module docstring).
Phase 94: the drill-down ``ls`` top level reads the registry —
register BOTH sources (TRUNCATEd in ``_reset_db_read_pair``),
``Deployments`` FIRST (registry order ``(added_at, id)``): the
mock's drill (first source of the listing) lands on the JSON file
— the read the assertions expect. A non-empty table also ignores
the operator's ``BOR_GIT_SOURCES`` fallback — deterministic.
"""
# COMMIT between the inserts (not flush): ``added_at`` is
# ``server_default now()`` — the transaction timestamp — and the
# tie-break is the random uuid ``id``, so one-transaction rows order
# nondeterministically.
db.add(GitSource(url=READ_SOURCE, kind="local"))
db.commit()
db.add(GitSource(url=SEED_SOURCE, kind="local"))
md = Document(
source=SEED_SOURCE,
path=SEED_PATH,
@@ -267,7 +287,10 @@ def _reset_db_read_pair() -> None:
answers."""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources"
)
)
db.commit()
_seed_read_pair(db)
@@ -386,15 +409,18 @@ def test_read_flow_lines_answer_sources_no_raw_markup(
_submit(page, READ_QUESTION)
_wait_settled(page)
# The UI shows the ls line (UNSCOPED — no argument, no <code>) then
# The UI shows the ls line (UNSCOPED — no argument, no <code>),
# the drill line (phase 94 — the source in a <code> element), then
# the "📄 Reading <source/path>" line with the COMBINED path in a
# <code> element (the path is data, never markup).
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(0).locator("code")).to_have_count(0)
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1).locator("code")).to_have_text(READ_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1).locator("code")).to_have_text(READ_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2).locator("code")).to_have_text(READ_SP)
# The answer streamed and quotes the read document (the mock's
# deterministic echo: "Read <source/path>. <first 80 chars>").
@@ -402,12 +428,14 @@ def test_read_flow_lines_answer_sources_no_raw_markup(
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
# Wire level: ls then read — the phase-70 argument rule (ls
# unscoped → null; read → the combined path as passed) — ahead of
# the first delta.
# Wire level: ls, the drill ls scoped to the first source (phase
# 94), then read — the phase-70 argument rule (ls unscoped → null;
# scoped ls → the source name; read → the combined path as passed)
# — ahead of the first delta.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": READ_SOURCE},
{"type": "tool", "name": "read", "argument": READ_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
@@ -508,10 +536,12 @@ def test_wire_argument_rule_across_both_flows(
read_tools = _tool_frames(read_frames)
search_tools = _tool_frames(search_frames)
# The ordered, combined tool-frame sequence across both flows: the
# argument rule end-to-end — read → the combined path as passed,
# grep → the pattern, ls → null when unscoped.
# argument rule end-to-end — the drill ls (phase 94) → the source
# name, read → the combined path as passed, grep → the pattern,
# ls → null when unscoped.
assert read_tools + search_tools == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": READ_SOURCE},
{"type": "tool", "name": "read", "argument": READ_SP},
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN},
]
tests/e2e/test_ls_tree_drilldown.py
+772
View File
@@ -0,0 +1,772 @@
"""Phase 94 task 04 E2E (Playwright, mock-only): the ``ls`` drill-down tree.
The dedicated story suite for ``94_ls_tree_drilldown`` (owner TODO.md L4):
the LLM drills ``ls()`` → ``ls(source)`` → ``ls(source/folder)`` →
``read(source/file)`` through the real UI, the sync-time folder summaries
(must exist after a changed sync under the deterministic mock's
``FOLDER_SUMMARY_MODE`` branch) show up in the ``ls`` output, and the
50-line cap holds on a wide folder.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real ``turbo``
does whatever it does with the tools, while this story's gate is the
deterministic SCRIPTED drill-down flow in ``tests/e2e/mock_llm.py``
(``DRILL_TRIGGER``: user message contains ``drill down the tree`` —
the question carries its own tool call after the colon,
``drill down the tree: ls [target]`` / ``drill down the tree: read
source/path`` — **and** the system prompt carries the ``<tools>``
section of the HIGH prompt). The mock echoes the received tool result
into its final answer (the house scripted-turn way of asserting on tool
results — the mock is the only E2E lens on the LLM's context), so every
tree-level assertion below lands on the rendered answer; the DOM
assertions cover the tool lines + the answer.
KB fixture — a host temp dir tree (``tmp_path_factory``; the app runs on
the same host) with TWO registered local sources (the
``test_local_directory_sources.py`` registration + real-Sync pattern —
registration through the authenticated API, the real in-process
``POST /api/sync`` pipeline; no git anywhere):
* ``alpha/`` — ``root-note.md`` at the source root, ``one/`` (2 docs),
``two/`` (2 docs — the read target lives here) and ``wide/`` (51 tiny
files — the 50-line cap subject);
* ``beta/`` — ``gamma/`` (2 docs).
Every fixture doc carries the words ``drill down the tree`` in its body,
so every scripted question (which contains the trigger phrase) FTS-matches
at least one chunk — the honesty gate is HIGH for all turns regardless
of the mock's cosine distribution, and the ``<tools>`` section is present
(the flow's precondition).
The mock's canned ``FOLDER_SUMMARY_MODE`` branch (phase 94 task 01)
stores, per ≥ 2-doc folder, the deterministic one-liner
``Fixture folder summary for <source>[/<folder>].`` — the ``synced_kb``
module fixture pins those exact rows in ``folder_summaries`` after the
sync, and the drill answers assert on them in the ``ls`` output.
Test → story mapping (Playwright Mapping Rule):
1. ``test_drill_down_sources_folders_files_and_read`` — the four
scripted turns: ``ls()`` (the ``🔎 Listing documents`` line + the
top-level shape — one ``— N documents`` line per source + the
canned source-root summaries), ``ls(alpha)`` (folder lines with
their summaries + the root file line), ``ls(alpha/two)`` (the exact
``source: X | path: Y | title: Z`` file lines), and the grounded
``read(alpha/two/two-a.md)`` (the ``📄 Reading …`` line + the answer
citing the document, the phase-37 assertion pattern). The read
target is DELIBERATELY a top-2 retrieval document for its question
(the question names the file's path, so the file self-matches the
hybrid gate deterministically): the agent's phase-72 dedupe returns
``ALREADY_IN_CONTEXT`` (a refusal — counts in nothing), and the
mock answers FROM THE ``<documents>`` PROMPT with the same citation
shape (``Already in context: Read <sp>. <first 80 chars>``) — the
document text reached the model either way, and the prefix pins
that the dedupe notice itself reached it.
2. ``test_wide_folder_holds_the_fifty_line_cap`` — ``ls(alpha/wide)``
on the 51-file folder: the mock's echo carries exactly 50 file lines
+ the ``…and 1 more documents in this folder — use grep (pattern)…``
note; the 51st file never reaches the model.
3. ``test_not_a_folder_teaching_and_scripted_recovery`` — the mock calls
``ls(alpha/nope)``; the NOT-A-FOLDER teaching line (the argument
echoed, the parent's subfolders listed) is visible to the model —
the mock's scripted recovery branch keys on receiving it — and the
scripted ``ls(alpha)`` recovery lands (the answer is the parent's
listing; the loop ends in one refusal + one correction, not at the
round cap).
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import httpx
import pytest
from playwright.sync_api import Locator, Page, expect
from sqlalchemy import select, text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import FolderSummary, QueryLog
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_LSTREE", "8136"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
# --------------------------------------------------------------------------
# Fixture documents (deterministic, token-controlled)
# --------------------------------------------------------------------------
ALPHA = "alpha"
BETA = "beta"
ROOT_NOTE = "root-note.md"
TWO_A = "two/two-a.md"
TWO_B = "two/two-b.md"
READ_SP = f"{ALPHA}/{TWO_A}"
ALPHA_COUNT = 56 # 1 root note + 2 one/ + 2 two/ + 51 wide/
BETA_COUNT = 2
TOTAL_DOCS = ALPHA_COUNT + BETA_COUNT
WIDE_COUNT = 51
LS_MAX_FILE_LINES = 50 # app.rag.agent.LS_MAX_FILE_LINES — the cap under test
#: Every fixture body carries ``drill down the tree`` (the trigger
#: phrase's words): every scripted question FTS-matches at least one
#: chunk → HIGH gate → the ``<tools>`` section the flow keys on.
DRILL_LEAD = "The drill down the tree fixture note"
def _md(title: str, body: str) -> str:
return f"# {title}\n\n{body}\n"
#: The read target — its FIRST line is ≥ 80 chars, so the mock's
#: first-80-chars quote (the phase-37 single-read shape) is newline-free
#: and the rendered-text assertion matches it verbatim. Pinned by the
#: assert below.
TWO_A_TITLE = (
"Alpha Two A — the drill-down read target for the alpha two folder "
"listing turn in the brain of reese fixture"
)
TWO_A_CONTENT = _md(
TWO_A_TITLE,
f"{DRILL_LEAD} for alpha two: this document covers topic A of the "
"alpha source tree; it is the file the scripted read turn opens "
"from the alpha/two listing.",
)
assert "\n" not in TWO_A_CONTENT[:80] # the quote must stay one line
#: The sync-time folder summaries the mock's canned ``FOLDER_SUMMARY_MODE``
#: branch stores (task 01's byte-stable template), in
#: ``(source, folder_path)`` order: one row per ≥ 2-doc folder (the
#: recursive-subtree rule) — the ``""`` rows are the source roots.
EXPECTED_SUMMARIES: list[tuple[str, str, str]] = [
(ALPHA, "", f"Fixture folder summary for {ALPHA}."),
(ALPHA, "one", f"Fixture folder summary for {ALPHA}/one."),
(ALPHA, "two", f"Fixture folder summary for {ALPHA}/two."),
(ALPHA, "wide", f"Fixture folder summary for {ALPHA}/wide."),
(BETA, "", f"Fixture folder summary for {BETA}."),
(BETA, "gamma", f"Fixture folder summary for {BETA}/gamma."),
]
assert [
(source, folder) for source, folder, _s in EXPECTED_SUMMARIES
] == sorted((source, folder) for source, folder, _s in EXPECTED_SUMMARIES)
# --- the pinned tree levels (app.rag.agent's phase-94 templates) -------
#: ``ls()`` — the top level: sources in registry order (alpha registered
#: first), each with its recursive count + stored source-root summary.
TOP_HEADER = "2 sources:"
TOP_LINES = [
f"{ALPHA} — {ALPHA_COUNT} documents",
EXPECTED_SUMMARIES[0][2],
f"{BETA} — {BETA_COUNT} documents",
EXPECTED_SUMMARIES[4][2],
]
#: ``ls(alpha)`` — the source root: the subfolder lines (path order)
#: with their stored summaries, then the root's own file line (the
#: canonical ``read``/``grep`` identity format, unchanged).
SOURCE_HEADER = f"{ALPHA} — 1 documents, 3 folders:"
SOURCE_LINES = [
f"one/ — 2 documents: {EXPECTED_SUMMARIES[1][2]}",
f"two/ — 2 documents: {EXPECTED_SUMMARIES[2][2]}",
f"wide/ — {WIDE_COUNT} documents: {EXPECTED_SUMMARIES[3][2]}",
f"source: {ALPHA} | path: {ROOT_NOTE} | title: Alpha Root Note",
]
#: ``ls(alpha/two)`` — a leaf folder: the file lines in EXACTLY the
#: existing ``source: X | path: Y | title: Z`` format, path order.
FOLDER_HEADER = f"{ALPHA}/two — 2 documents, 0 folders:"
FOLDER_LINES = [
f"source: {ALPHA} | path: {TWO_A} | title: {TWO_A_TITLE}",
f"source: {ALPHA} | path: {TWO_B} | title: Alpha Two B",
]
#: ``ls(alpha/wide)`` — the 50-line cap: 51 files → 50 lines (path
#: order: wide-01 … wide-50) + one deterministic grep-pointer note;
#: wide-51 never reaches the model.
WIDE_HEADER = f"{ALPHA}/wide — {WIDE_COUNT} documents, 0 folders:"
WIDE_FIRST = f"source: {ALPHA} | path: wide/wide-01.md | title: Wide 01"
WIDE_LAST = f"source: {ALPHA} | path: wide/wide-50.md | title: Wide 50"
WIDE_NOTE = (
"…and 1 more documents in this folder — use grep "
"(pattern) to find a specific one."
)
# --- the scripted turns (the mock's ``DRILL_TRIGGER`` questions) -------
TOP_QUESTION = "Drill down the tree: ls — what sources are indexed?"
SOURCE_QUESTION = f"Drill down the tree: ls {ALPHA} — what's in source {ALPHA}?"
FOLDER_QUESTION = f"Drill down the tree: ls {ALPHA}/two — list that folder"
READ_QUESTION = f"Drill down the tree: read {READ_SP} — read the file"
WIDE_QUESTION = f"Drill down the tree: ls {ALPHA}/wide — how many files does this folder hold?"
NOPE_QUESTION = f"Drill down the tree: ls {ALPHA}/nope — is there such a folder?"
#: The read target is a top-2 retrieval document for its question (the
#: question names the path — the file self-matches the hybrid gate
#: deterministically), so the read gets the phase-72 ALREADY_IN_CONTEXT
#: dedupe and the mock answers from the ``<documents>`` prompt with the
#: same citation shape, prefixed (the suite pins the dedupe path).
READ_ANSWER_PREFIX = f"Already in context: Read {READ_SP}."
READ_ANSWER_QUOTE = TWO_A_CONTENT[:80]
# --------------------------------------------------------------------------
# Fixtures
# --------------------------------------------------------------------------
@pytest.fixture(scope="module")
def drill_dirs(tmp_path_factory: pytest.TempPathFactory) -> tuple[Path, Path]:
"""The two-source temp tree (see the module docstring): the app
server runs on the same host, so the paths are visible to it. The
directory NAMES are the source names (``kind=local`` → the
directory's basename, phase 38)."""
root = tmp_path_factory.mktemp("bor_ls_tree")
alpha = root / ALPHA
beta = root / BETA
(alpha / "one").mkdir(parents=True)
(alpha / "two").mkdir(parents=True)
(alpha / "wide").mkdir(parents=True)
(beta / "gamma").mkdir(parents=True)
(alpha / ROOT_NOTE).write_text(
_md(
"Alpha Root Note",
f"{DRILL_LEAD} at the alpha source root: this file sits "
"directly under the alpha source, not in any folder.",
),
encoding="utf-8",
)
(alpha / "one" / "one-a.md").write_text(
_md(
"Alpha One A",
f"{DRILL_LEAD} for alpha one: this document covers topic A "
"of the alpha source tree.",
),
encoding="utf-8",
)
(alpha / "one" / "one-b.md").write_text(
_md(
"Alpha One B",
f"{DRILL_LEAD} for alpha one: this document covers topic B "
"of the alpha source tree.",
),
encoding="utf-8",
)
(alpha / TWO_A).write_text(TWO_A_CONTENT, encoding="utf-8")
(alpha / TWO_B).write_text(
_md(
"Alpha Two B",
f"{DRILL_LEAD} for alpha two: this document covers topic B "
"of the alpha source tree.",
),
encoding="utf-8",
)
for nn in range(1, WIDE_COUNT + 1):
(alpha / "wide" / f"wide-{nn:02d}.md").write_text(
_md(
f"Wide {nn:02d}",
f"One of {WIDE_COUNT} tiny files in the alpha wide "
f"folder: {DRILL_LEAD.lower()} line {nn:02d}.",
),
encoding="utf-8",
)
(beta / "gamma" / "gamma-a.md").write_text(
_md(
"Beta Gamma A",
f"{DRILL_LEAD} for beta gamma: this document covers topic A "
"of the beta source tree.",
),
encoding="utf-8",
)
(beta / "gamma" / "gamma-b.md").write_text(
_md(
"Beta Gamma B",
f"{DRILL_LEAD} for beta gamma: this document covers topic B "
"of the beta source tree.",
),
encoding="utf-8",
)
assert (alpha / TWO_A).is_file() and (beta / "gamma" / "gamma-b.md").is_file()
return alpha, beta
@pytest.fixture(scope="module")
def app_server(mock_llm: int, drill_dirs: tuple[Path, Path]) -> Iterator[str]:
"""The real app under test — per-module app (the conftest pattern,
cf. ``test_local_directory_sources.py``): NO ``BOR_GIT_SOURCES``
(the env fallback is git-only — the sources here are DB-registered
local directories), the mock LLM, the mock-calibrated threshold,
and the leak-guarded code defaults. The session app is never
started in this isolated run, so no port clash."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern): every scripted
# question FTS-matches the fixture docs (the ``drill down the tree``
# words), so the gate is HIGH either way.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
# Phase 67: instant retry waits + the code-default budget (the
# conftest leak-guard pattern).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the two
# local directories this suite registers (a leftover env git list
# would pollute the top-level ``ls`` the whole story asserts on).
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under test.
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
share one Postgres, so a leftover git_sources row would pollute the
top-level ``ls`` and a leftover document would show up in the
folder listings the drill answers assert on byte-exactly."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, folder_summaries"
)
)
db.commit()
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
"""Poll the (cookie-authenticated) status endpoint until the run
reaches a terminal state (the test_local_directory_sources pattern,
over plain httpx — this fixture has no browser page yet)."""
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
if body["state"] in ("success", "failed"):
return body
time.sleep(0.5)
raise AssertionError(f"sync did not reach a terminal state: {body}")
@pytest.fixture(scope="module")
def synced_kb(app_server: str, drill_dirs: tuple[Path, Path]) -> None:
"""The story's precondition: the folder-structured KB synced under
the deterministic mock.
Registers the two temp directories through the authenticated API
(the ``test_local_directory_sources.py`` pattern — ``alpha`` FIRST,
committed separately, so the registry order — ``(added_at, id)`` —
lists alpha before beta, the top-level ``ls`` order the suite
asserts), runs the REAL in-process sync (``POST /api/sync`` —
walk → chunk → embed → overview → folder summaries → version bump),
and pins the stored folder summaries: the mock's canned
``FOLDER_SUMMARY_MODE`` branch (task 01) makes the sync store one
deterministic row per ≥ 2-doc folder — the drill turns' answers
assert on that exact text.
"""
alpha, beta = drill_dirs
_truncate_all()
with httpx.Client(base_url=app_server, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post(
"/api/git-sources", json={"kind": "local", "path": str(alpha)}
)
assert r.status_code == 201, r.text
time.sleep(0.05) # distinct added_at: alpha before beta (registry order)
r = client.post(
"/api/git-sources", json={"kind": "local", "path": str(beta)}
)
assert r.status_code == 201, r.text
r = client.post("/api/sync")
assert r.status_code == 202, r.text
body = _wait_sync_done_http(client)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == TOTAL_DOCS, detail
assert detail["pruned"] == 0, detail
assert detail["overview"] is True, detail
# The change-gated folder summaries (phase 94 task 02) landed: one
# row per ≥ 2-doc folder, the mock's byte-stable text (the drill
# answers quote exactly these lines).
with SessionLocal() as db:
rows = db.execute(
select(
FolderSummary.source, FolderSummary.folder_path,
FolderSummary.summary,
).order_by(FolderSummary.source, FolderSummary.folder_path)
).all()
assert [(s, f, t) for s, f, t in rows] == EXPECTED_SUMMARIES, rows
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
"""Per-test query_log isolation (the KB itself is module-scoped —
the drill turns never change it, so the folder summaries and the
registry persist across the tests of this module)."""
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
yield
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
# --------------------------------------------------------------------------
# Page helpers (the test_agent_document_tools house pattern)
# --------------------------------------------------------------------------
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
#: (a response clone read in the background) — wire-level assertions
#: for the ``tool`` frames, independent of the UI rendering.
SSE_HOOK = """
() => {
if (window.__sseInstalled) return;
window.__sseInstalled = true;
window.__sseFrames = [];
const origFetch = window.fetch;
window.fetch = async function (...args) {
const res = await origFetch.apply(this, args);
try {
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
if (url.includes('/api/chat')) {
res.clone().text().then((bodyText) => {
for (const block of bodyText.split('\\n\\n')) {
const line = block.trim();
if (line.startsWith('data: ')) {
window.__sseFrames.push(line.slice(6));
}
}
});
}
} catch (e) { /* non-clonable responses: ignored */ }
return res;
};
}
"""
def _install_page_hooks(page: Page) -> None:
page.evaluate(SSE_HOOK)
def _frames(page: Page) -> list[dict]:
"""The SSE frames captured since the last submit (``_submit``
clears the buffer), once the hook's background read settles."""
deadline = time.monotonic() + 30.0
while True:
raw = page.evaluate("() => window.__sseFrames || []")
parsed = [json.loads(line) for line in raw if line]
if any(f.get("type") == "done" for f in parsed):
return parsed
if time.monotonic() > deadline:
raise AssertionError(
f"SSE hook captured no `done` frame (frames so far: "
f"{len(parsed)}) — hook install failed?"
)
time.sleep(0.05)
def _tool_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool"]
def _submit(page: Page, question: str) -> None:
page.evaluate("window.__sseFrames = []")
page.fill("#message-input", question)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
def _wait_settled(page: Page) -> None:
"""The turn is complete: answer text in the bubble, button recovered
(the phase-48 settle wait, the test_agent_document_tools helper)."""
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
def _last_brain(page: Page) -> Locator:
return page.locator(".msg.brain").last
def _assert_turn(
page: Page,
expected_tools: list[dict[str, Any]],
expected_answer_lines: list[str],
) -> None:
"""One scripted drill turn, fully asserted: the wire carries exactly
the expected ``tool`` frames (ahead of the first ``delta``), the
bubble carries the expected answer lines (the mock's echo of the
tool results the model received), and the turn was grounded (the
``done`` frame is not deflected)."""
frames = _frames(page)
assert _tool_frames(frames) == expected_tools, _tool_frames(frames)
if expected_tools:
first_delta = next(
i for i, f in enumerate(frames) if f.get("type") == "delta"
)
assert all(
i < first_delta
for i, f in enumerate(frames)
if f.get("type") == "tool"
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
bubble = _last_brain(page).locator(".bubble")
for line in expected_answer_lines:
expect(bubble).to_contain_text(line)
def _query_log_rows() -> list[QueryLog]:
with SessionLocal() as db:
return list(db.scalars(select(QueryLog)).all())
# --------------------------------------------------------------------------
# 1. The scripted drill: sources → folders (summaries) → files → read
# --------------------------------------------------------------------------
def test_drill_down_sources_folders_files_and_read(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
# --- turn 1: ls() — the top level (sources + source-root summaries) -
_submit(page, TOP_QUESTION)
_wait_settled(page)
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(1)
# The no-arg ls line — NOT a scoped "Listing documents in …" one
# (regex match: string expectations normalize whitespace, so the
# scope check must be a byte-exact pattern).
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(0)).not_to_have_text(re.compile(r"Listing documents in"))
_assert_turn(
page,
[{"type": "tool", "name": "ls", "argument": None}],
[TOP_HEADER, *TOP_LINES],
)
# --- turn 2: ls(alpha) — the source root (subfolders + root files) --
_submit(page, SOURCE_QUESTION)
_wait_settled(page)
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(1)
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}")
_assert_turn(
page,
[{"type": "tool", "name": "ls", "argument": ALPHA}],
[SOURCE_HEADER, *SOURCE_LINES],
)
# --- turn 3: ls(alpha/two) — the folder level (the file lines) ------
_submit(page, FOLDER_QUESTION)
_wait_settled(page)
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(1)
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}/two")
_assert_turn(
page,
[{"type": "tool", "name": "ls", "argument": f"{ALPHA}/two"}],
[FOLDER_HEADER, *FOLDER_LINES],
)
# --- turn 4: read(alpha/two/two-a.md) — the grounded read ------------
_submit(page, READ_QUESTION)
_wait_settled(page)
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(1)
# The phase-37 "Reading <source/path>" line (the combined identity).
expect(lines.nth(0)).to_contain_text(f"Reading {READ_SP}")
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "read", "argument": READ_SP}
], _tool_frames(frames)
bubble = _last_brain(page).locator(".bubble")
# The dedupe notice reached the model (the prefix pins it — the read
# was refused as ALREADY_IN_CONTEXT because the target is a top-2
# retrieval document)…
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
# …and the answer still cites the document: the mock quotes the
# FIRST 80 chars of the target's text from the ``<documents>``
# prompt (the refusal's instruction — answer from that text; the
# quote is newline-free, pinned above).
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The read document is in the turn's sources (retrieval + agent-read,
# deduped — the grounded-turn record).
assert any(
s["path"] == TWO_A and s["source"] == ALPHA for s in done["sources"]
), done["sources"]
# Durable records: all four turns grounded, in order, the read turn
# logging the read document.
rows = _query_log_rows()
assert [r.question for r in rows] == [
TOP_QUESTION, SOURCE_QUESTION, FOLDER_QUESTION, READ_QUESTION
]
assert all(r.deflected is False for r in rows)
assert READ_SP in rows[3].sources, rows[3].sources
# --------------------------------------------------------------------------
# 2. The 50-line cap: a 51-file folder costs the model 50 lines + the note
# --------------------------------------------------------------------------
def test_wide_folder_holds_the_fifty_line_cap(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, WIDE_QUESTION)
_wait_settled(page)
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(1)
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}/wide")
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": f"{ALPHA}/wide"}
], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The mock echoed the listing VERBATIM: the header carries the
# PRE-cap count (51 — the cap hides lines, not the truth), the
# file lines stop at 50, and the one deterministic grep-pointer note
# folds the 51st file away.
bubble = _last_brain(page).locator(".bubble")
text = bubble.text_content() or ""
assert WIDE_HEADER in text, text
assert WIDE_FIRST in text, text
assert WIDE_LAST in text, text
assert WIDE_NOTE in text, text
assert f"wide/wide-{WIDE_COUNT}.md" not in text, text # the 51st file: gone
# Exactly LS_MAX_FILE_LINES file lines reached the model.
assert text.count(f"source: {ALPHA} | path: wide/") == LS_MAX_FILE_LINES, text
row = _query_log_rows()
assert len(row) == 1
assert row[0].deflected is False
# --------------------------------------------------------------------------
# 3. The NOT-A-FOLDER teaching is visible to the model; the scripted
# recovery (ls of the parent level) works
# --------------------------------------------------------------------------
def test_not_a_folder_teaching_and_scripted_recovery(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, NOPE_QUESTION)
_wait_settled(page)
# Two tool lines: the scripted misuse, then the scripted recovery —
# the mock's recovery branch fires ONLY when it RECEIVES the
# NOT-A-FOLDER teaching line (``'alpha/nope' is not a folder —
# alpha has: one/ two/ wide/`` — the argument echoed, the parent's
# subfolders listed): the teaching being visible to the model is
# exactly what the second call proves (the phase-72
# self-correction contract, now carrying the tree's teaching).
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(2)
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}/nope")
expect(lines.nth(1)).to_contain_text(f"Listing documents in {ALPHA}")
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": f"{ALPHA}/nope"},
{"type": "tool", "name": "ls", "argument": ALPHA},
], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The loop ended in ONE refusal + ONE correction: the final answer is
# the recovery's PARENT listing (not the round-cap, not an echo of
# the refusal) — the model self-corrected and got the tree level.
bubble = _last_brain(page).locator(".bubble")
for line in [SOURCE_HEADER, *SOURCE_LINES]:
expect(bubble).to_contain_text(line)
row = _query_log_rows()
assert len(row) == 1
assert row[0].deflected is False
tests/e2e/test_tool_path_teaching.py
+60 -32
View File
@@ -13,11 +13,14 @@ Run in isolation (DB must be up: ``podman compose up -d db``):
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic LS-TEACH flow in ``tests/e2e/mock_llm.py``
(``LS_TEACH_TRIGGER`` — "list the files in this directory" — + the
HIGH prompt's ``<tools>`` section): the incident's misuse (``ls`` with
``{"path": "."}``, id ``call_0``) → the agent's teaching refusal
(``No source named '.' — check the ls output. (…)``) → the corrected
no-arg ``ls()`` (id ``call_1``) → the deterministic
``These are the indexed documents: <first catalog line>`` answer.
HIGH prompt's ``<tools>`` section; phase 94: the drill-down ``ls`` —
the corrected no-arg listing carries sources only, so the flow drills
one level before the first file line exists): the incident's misuse
(``ls`` with ``{"path": "."}``, id ``call_0``) → the agent's teaching
refusal (``No source named '.' — check the ls output. (…)``) → the
corrected no-arg ``ls()`` (id ``call_1``) → the drill ``ls`` scoped to
the first source of the listing (id ``call_2``) → the deterministic
``These are the indexed documents: <first file line>`` answer.
KB fixture (TRUNCATE-then-seed, house pattern): ONE source with TWO
documents of known ``source``/``path``/``title`` (catalog order =
@@ -39,21 +42,24 @@ documents of known ``source``/``path``/``title`` (catalog order =
Test → phase mapping (Playwright Mapping Rule):
1. ``test_ls_misuse_self_corrects_to_noarg_listing`` — the grounded
LS-TEACH turn: the turn settles (composer re-enables, ``done``
observed), the answer bubble carries the first catalog line — the
observed), the answer bubble carries the first file line — the
first document's ``source:`` / ``path:`` / title fields (the
catalog reached the model and landed in the answer), the UI shows
the two tool lines (``🔎 Listing documents in <code>.</code>``
then ``🔎 Listing documents``), and no error banner. Wire level:
the ``tool`` frames arrive in order — first ``ls`` with
``argument: "."``, then ``ls`` with ``argument: null`` — and there
is NO third ``tool`` frame (the loop ended in one correction, not
folder listing reached the model and landed in the answer), the UI
shows the three tool lines (``🔎 Listing documents in
<code>.</code>``, ``🔎 Listing documents``, then the drill
``🔎 Listing documents in <source>`` — phase 94), and no error
banner. Wire level: the ``tool`` frames arrive in order — first
``ls`` with ``argument: "."``, then ``ls`` with ``argument: null``,
then the drill ``ls`` scoped to the source — and there is NO fourth
``tool`` frame (the loop ended in one correction + one drill, not
at the round cap).
2. ``test_plain_tool_flow_not_swallowed_by_new_trigger`` — in the SAME
session, the LS-TEACH turn settles and a follow-up question
carrying ``TOOLS_TRIGGER`` (the single-read flow) still settles
with the read flow's answer (``ls`` → ``read`` on the first
catalog line's combined identity → ``Read <source/path>. <quote>``)
— the new flow did not swallow the existing trigger.
with the read flow's answer (``ls`` → the drill ``ls`` (phase 94)
→ ``read`` on the first file line's combined identity →
``Read <source/path>. <quote>``) — the new flow did not swallow
the existing trigger.
"""
from __future__ import annotations
@@ -69,7 +75,7 @@ from sqlalchemy import text
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import Chunk, Document
from app.models import Chunk, Document, GitSource
from e2e.auth_helpers import login
from tests.e2e.mock_llm import (
LS_TEACH_TRIGGER,
@@ -211,7 +217,14 @@ def _seed_fixture(db: Session) -> None:
cosines well past the E2E 0.30 threshold and FTS-matches too →
grounded). DOC2 is the seed context only — the single-read flow
reads the catalog-FIRST document (DOC1), which is not in context.
Phase 94: the drill-down ``ls`` top level reads the registry —
register the source (TRUNCATEd in ``_reset_db_fixture``): the
corrected no-arg listing names it, and the drill scopes to it. A
non-empty table also ignores the operator's ``BOR_GIT_SOURCES``
fallback — deterministic.
"""
db.add(GitSource(url=SEED_SOURCE, kind="local"))
db.add(
Document(
source=SEED_SOURCE,
@@ -255,7 +268,10 @@ def _reset_db_fixture() -> None:
prompts, byte-stable answers."""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources"
)
)
db.commit()
_seed_fixture(db)
@@ -382,24 +398,30 @@ def test_ls_misuse_self_corrects_to_noarg_listing(
expect(bubble).to_contain_text(FIRST_CATALOG_LINE)
_assert_no_error_banner(page)
# The UI shows the two tool lines in order: the scoped misuse
# (🔎 Listing documents in <code>.</code>) then the corrected
# unscoped listing (🔎 Listing documents — no <code>).
# The UI shows the three tool lines in order: the scoped misuse
# (🔎 Listing documents in <code>.</code>), the corrected
# unscoped listing (🔎 Listing documents — no <code>), then the
# drill (🔎 Listing documents in <source> — phase 94, the top
# level lists sources only, so the file lines need one more level).
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents in")
expect(lines.nth(0).locator("code")).to_have_text(".")
expect(lines.nth(1)).to_contain_text("Listing documents")
expect(lines.nth(1).locator("code")).to_have_count(0)
expect(lines.nth(2)).to_contain_text("Listing documents in")
expect(lines.nth(2).locator("code")).to_have_text(SEED_SOURCE)
# Two rounds on the wire: the tool frames arrive in order — first
# ls with argument "." (the incident's misuse), then ls with
# argument null (the correction) — and there is NO third tool
# frame: the loop ended in one correction, not at the round cap.
# Three rounds on the wire: the tool frames arrive in order —
# first ls with argument "." (the incident's misuse), then ls with
# argument null (the correction), then the drill ls scoped to the
# source (phase 94) — and there is NO fourth tool frame: the loop
# ended in one correction + one drill, not at the round cap.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": "."},
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
@@ -433,6 +455,7 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
assert _tool_frames(teach_frames) == [
{"type": "tool", "name": "ls", "argument": "."},
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
]
expect(
page.locator(".msg.brain .bubble").last
@@ -445,14 +468,17 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
_wait_settled(page)
second_msg = page.locator(".msg.brain").last
# The UI shows the single-read flow's two lines: the unscoped ls
# then the read of the first catalog line's COMBINED identity.
# The UI shows the single-read flow's three lines: the unscoped ls,
# the drill ls (phase 94), then the read of the first file line's
# COMBINED identity.
lines = second_msg.locator(".tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(0).locator("code")).to_have_count(0)
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1).locator("code")).to_have_text(DOC1_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1).locator("code")).to_have_text(SEED_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2).locator("code")).to_have_text(DOC1_SP)
# The answer quotes the read document (the mock's deterministic
# echo: "Read <source/path>. <first 80 chars>").
@@ -461,11 +487,13 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
_assert_no_error_banner(page)
# Wire level for the follow-up: ls (null) → read (the combined
# identity) — the single-read flow, unchanged.
# Wire level for the follow-up: ls (null) → the drill ls (the
# source, phase 94) → read (the combined identity) — the
# single-read flow, grown by the drill step.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
{"type": "tool", "name": "read", "argument": DOC1_SP},
]
done = next(f for f in frames if f.get("type") == "done")