Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 94_ls_tree_drilldown
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Details
Build and Push Containers / build-and-push-db (push) Successful in 25s
Details

Browse Source 
All green. Verification complete.

**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**

- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)

**Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
ducoterra
2026-09-11 00:59:35 -04:00
parent 9188be259b
commit d4943b4822
61 changed files with 6289 additions and 666 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/mock_llm.py
+456 -73
View File
@@ -18,6 +18,14 @@ Implements just enough of the aipi surface:
- ``KB_OVERVIEW_MODE`` -> the deterministic outline: the first 8 tokens
of the user message (the generator puts the document list there) —
byte-stable for a given KB (KB overview, phase 31)
- ``FOLDER_SUMMARY_MODE`` -> the deterministic folder one-liner
``Fixture folder summary for <folder>.`` — <folder> is the user
message's ``Folder: …`` header line (the generator names the
source/folder there, phase 94), so the stored row always names its
folder and an E2E can assert on it. Checked BEFORE the
``SUMMARY_MODE`` branch: the folder marker CONTAINS the summary
marker as a substring, so the summary branch would otherwise
shadow every folder-summary call
- ``DEFLECT_MODE`` -> honest "I haven't done anything like that" answer
- otherwise -> upbeat answer quoting the provided document context
- user message containing ``pretend to think slowly`` -> 3s warm-up delay
@@ -57,22 +65,31 @@ Implements just enough of the aipi surface:
closing tag — same sentinel semantics.)
- user message containing ``use your tools`` (phase 37, agent document
tools; phase 70: the flow emits the harness-aligned names — ``ls``
/ ``read`` with the combined ``source/path`` identity) **and** the
system prompt carries the ``<tools>`` section -> the deterministic
SINGLE-READ tool flow, discriminated statelessly from the messages
(the ``tools`` parameter gates the list/read steps — a no-tools
request with no tool results is not the flow):
/ ``read`` with the combined ``source/path`` identity; phase 94:
the flow DRILLS through the tree ``ls`` — the top-level listing
carries sources only, so the flow takes one drill step, ``ls``
scoped to the first source, before the first file line exists)
**and** the system prompt carries the ``<tools>`` section -> the
deterministic SINGLE-READ tool flow, discriminated statelessly from
the messages (the ``tools`` parameter gates the list/read steps —
a no-tools request with no tool results is not the flow):
* request 1 (``tools`` offered, no tool results yet): stream ONLY
``tool_calls`` deltas — ``ls`` (synthetic id ``call_0``, no
arguments), ``finish_reason: "tool_calls"``, no content;
* request 2 (a ``tool``-role catalog result in the messages):
parse the FIRST catalog line (``source: X | path: Y | title: Z``
— the labeled ``source:`` / ``path:`` fields, phase 63) and
stream a ``tool_calls`` delta calling ``read`` on the JOINED
combined ``source/path`` (the mock joins the two labeled fields
— the catalog format is unchanged, so this join is the only
parse change, phase 70) (id ``call_1``);
* request 3 (a ``tool``-role read result in the messages —
* request 2 (the top-level source listing in the messages — the
agent's ``ls`` tree format, phase 94: no file lines yet):
stream a ``tool_calls`` delta — ``ls`` scoped to the FIRST
source of the listing (id ``call_1``) — the drill step (one
drill per flow — ``_drill_target`` skips already-drilled
sources, so the second listing is the folder level);
* request 3 (a ``tool``-role folder listing with file lines in
the messages): parse the FIRST file line (``source: X | path:
Y | title: Z`` — the labeled ``source:`` / ``path:`` fields,
phase 63) and stream a ``tool_calls`` delta calling ``read`` on
the JOINED combined ``source/path`` (the mock joins the two
labeled fields — the file-line format is unchanged, so this
join is the only parse change, phase 70) (id ``call_2``);
* request 4 (a ``tool``-role read result in the messages —
content starting with the agent's ``"Document <source/path>:"``
header): a content answer, deterministic: ``Read
<source/path>. <first 80 chars of the read document's
@@ -85,19 +102,22 @@ Implements just enough of the aipi surface:
- user message containing BOTH ``use your tools`` AND ``read two
documents`` (``MULTI_READ_TRIGGER``, phase 45 task 02) **and** the
system prompt carries the ``<tools>`` section -> the deterministic
MULTI-READ flow (list → read #1 → read #2 → answer), classified by
the COUNT of ``tool``-role read results (content starting with the
agent's ``"Document <source/path>:"`` prefix); phase 70: the same
flow on the harness-aligned names — ``ls``, then ``read`` on the
JOINED combined ``source/path`` of each catalog line:
* 0 read results, no catalog yet: ``ls`` (id ``call_0``);
* 0 read results, catalog present: ``read`` on the JOINED
combined ``source/path`` of the FIRST catalog line
(id ``call_1``);
MULTI-READ flow (list → drill → read #1 → read #2 → answer; phase
94: the drill step between the top-level listing and the first
read), classified by the COUNT of ``tool``-role read results
(content starting with the agent's ``"Document <source/path>:"``
prefix); phase 70: the same flow on the harness-aligned names —
``ls``, the drill ``ls`` scoped to the first source, then ``read``
on the JOINED combined ``source/path`` of each file line:
* 0 read results, no listing yet: ``ls`` (id ``call_0``);
* 0 read results, top-level listing only (no file lines yet):
the drill — ``ls`` scoped to the first source (id ``call_1``);
* 0 read results, file lines present: ``read`` on the JOINED
combined ``source/path`` of the FIRST file line (id ``call_2``);
* 1 read result: ``read`` on the JOINED combined ``source/path``
of the SECOND catalog line — the first listing line whose
of the SECOND file line — the first listing line whose
``source/path`` differs from the one already read (id
``call_2``); a one-document catalog degenerates to the
``call_3``); a one-file listing degenerates to the
single-read answer (nothing second to read);
* 2 read results: the forced answer, byte-stable: the single-read
shape quoting the FIRST read result, plus the line ``I read
@@ -162,27 +182,83 @@ Implements just enough of the aipi surface:
misuse met the terse refusal and the model re-reasoned the same
paragraphs over and over) **and** the system prompt carries the
``<tools>`` section -> the deterministic LS-TEACHING flow,
discriminated statelessly from the messages (streaming only):
discriminated statelessly from the messages (streaming only;
phase 94: the correction's top-level listing carries sources only,
so the flow drills one level before the first file line exists):
* request 1 (``tools`` offered, no ``tool``-role result in the
messages yet): stream ONLY ``tool_calls`` deltas — ``ls``
with ``{"path": "."}`` (synthetic id ``call_0``),
``finish_reason: "tool_calls"``, no content — the incident's
misuse, deterministic;
* request 2 (a ``tool``-role result present that is NOT a
catalog listing — i.e. the teaching refusal): a ``tool_calls``
delta — ``ls`` with no arguments (id ``call_1``) — the
correction;
* request 3 (a ``tool``-role result whose first line matches the
``^\\d+ documents:`` catalog header): a deterministic content
answer — ``These are the indexed documents: <first catalog
line>`` (the ``source: X | path: Y | title: Z`` line, parsed
with the ``_CATALOG_LINE_RE`` machinery), ``finish_reason:
"stop"`` — the loop ended in ONE correction, not at the round
listing with file lines — i.e. the teaching refusal): a
``tool_calls`` delta — ``ls`` with no arguments (id ``call_1``)
— the correction;
* request 3 (the top-level source listing in the messages —
file lines still absent): the drill — a ``tool_calls`` delta
— ``ls`` scoped to the FIRST source of the listing
(id ``call_2``);
* request 4 (a ``tool``-result with a ``source: X | path: Y |
title: Z`` file line in the messages — the folder listing):
a deterministic content answer — ``These are the indexed
documents: <first file line>`` (the line, parsed with the
``_CATALOG_LINE_RE`` machinery), ``finish_reason: "stop"`` —
the loop ended in ONE correction + ONE drill, not at the round
cap.
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (the trigger
phrases are disjoint substrings — the phase-71 ordering
convention); no existing E2E question or fixture file contains the
phrase, so every other suite is unaffected.
- user message containing ``drill down the tree`` (``DRILL_TRIGGER``,
phase 94 task 04 — the drill-down ``ls``'s dedicated story suite
``tests/e2e/test_ls_tree_drilldown.py``) **and** the system prompt
carries the ``<tools>`` section -> the deterministic SCRIPTED
DRILL-DOWN flow: the question carries its own tool call after the
colon — ``drill down the tree: ls [target]`` (no target = the top
level; ``target`` = a source name or ``source/folder`` path) or
``drill down the tree: read source/path`` — parsed by
``_DRILL_CALL_RE`` from the RAW user message (the target keeps its
case), then discriminated statelessly from the tool results
(streaming only):
* request 1 (``tools`` offered, no ``tool``-role result in the
messages yet): the SCRIPTED call — ``ls`` with no arguments
for the top level, ``ls``/``read`` with the parsed target
otherwise (synthetic id ``call_0``);
* the last tool result is the NOT-A-FOLDER teaching refusal
(it carries ``"is not a folder"`` — the phase-94 task-03
teaching line, the refusal being VISIBLE to the model is what
fires this branch): the scripted one-round recovery — ``ls``
the target's SOURCE segment (id ``call_1``);
* the last tool result is a READ result (``"Document
<source/path>:…``): the deterministic echo answer ``Read
<source/path>. <first 80 chars of the read document's
content>`` (the phase-37 single-read shape — the grounded-
turn citation contract);
* the last tool result is the agent's ALREADY_IN_CONTEXT dedupe
refusal (the read target is already a top-2 retrieval
document — with the drill fixture that is DETERMINISTIC:
the read question names the file's path, so the file
self-matches the hybrid gate and its FULL text is in the
``<documents>`` prompt): the model answers FROM THE PROMPT —
the deterministic answer ``Already in context: Read
<source/path>. <first 80 chars of the target document's text
as it appears in the ``<documents>`` block>`` (same citation
shape as the read-result branch — the document text reached
the model either way, and the answer proves it);
* any other last result (a top-level or folder LISTING landed):
the deterministic ECHO answer ``Here's the level I listed:\n
<the listing, verbatim>`` — the mock echoes what it received
(the house scripted-turn way of asserting on tool results):
the suite asserts on the exact tree level — the ``— N
documents`` lines, the stored folder summaries, the
``source: X | path: Y | title: Z`` file lines, and the 50-line
cap + ``…and N more documents…`` note — through the rendered
answer, the only E2E lens on the LLM's context.
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
phrases — the phase-71/72 ordering convention; the trigger needs
the ``<tools>`` section, so deflected turns never hit it); no
existing E2E question or fixture file contains the phrase, so
every other suite is unaffected.
- user message containing ``what are the correct llama.cpp
arguments`` (``GREP_TEACH_TRIGGER``, the 2026-09-05 incident —
the harness prior is that grep takes a REGEX; this app's grep is a
@@ -302,7 +378,8 @@ from typing import Any
from fastapi import FastAPI
from fastapi.responses import JSONResponse, StreamingResponse
from app.rag.agent import CORRECTION_INSTRUCTION # phase 71: the harness constant
from app.rag.agent import ALREADY_IN_CONTEXT, CORRECTION_INSTRUCTION
from app.rag.folder_summaries import FOLDER_HEADER_PREFIX # phase 94: the mock's key
app = FastAPI()
@@ -588,9 +665,92 @@ GREP_TEACH_PLAIN = "qwen3.8"
#: plain step keys on it (a plain no-match line carries it not).
GREP_TEACH_MARKER = "grep matches a plain substring"
#: The agent's ``ls`` listing header (app.rag.agent ``_execute_tool``):
#: ``"N documents:"`` — the first line of every catalog tool result.
_CATALOG_HEADER_RE = re.compile(r"^\d+ documents:")
# ---------------------------------------------------------------------------
# Phase 94 (task 04, the drill-down ls's dedicated story suite):
# the deterministic SCRIPTED drill-down turns — see the module docstring
# ---------------------------------------------------------------------------
#: A user message containing this substring (case-insensitive) —
#: combined with the ``<tools>`` section in the system prompt — drives
#: the scripted DRILL-DOWN flow: the question carries its own tool call
#: after the colon (``drill down the tree: ls [target]`` / ``drill
#: down the tree: read source/path`` — the suite's scripted turns,
#: ``tests/e2e/test_ls_tree_drilldown.py``). Checked BEFORE the plain
#: ``TOOLS_TRIGGER`` flow (disjoint trigger phrases — the phase-71/72
#: ordering convention); verified: no existing E2E question or fixture
#: file contains the phrase, so every other suite is unaffected.
DRILL_TRIGGER = "drill down the tree"
#: The scripted call in the drill-down question (case-insensitive — the
#: suite's questions capitalize the trigger's first letter): the verb
#: (``ls`` / ``read``) plus the optional target — a source name or
#: ``source/folder`` path (``ls``) or a combined ``source/path``
#: (``read``), parsed from the RAW user message so the target keeps its
#: case. The target is a ``[a-z0-9_./-]`` run (case-insensitively), so
#: the suite's `` — `` flavor separator (em dash) can never bleed into
#: it; no run = the top-level ``ls`` (no ``path`` argument).
_DRILL_CALL_RE = re.compile(
r"drill down the tree:\s*(?P<verb>ls|read)(?:\s+(?P<arg>[a-z0-9_./-]+))?",
re.I,
)
#: The agent's NOT-A-FOLDER teaching refusal marker (app.rag.agent
#: ``NOT_A_FOLDER`` — ``'{arg}' is not a folder — {parent} has:
#: {subfolders}``): the drill-down flow's scripted recovery keys on it
#: in the LAST tool result (the refusal being visible to the model is
#: exactly what triggers the one-round recovery — the phase-72
#: self-correction contract carrying the tree's teaching line).
_NOT_A_FOLDER_MARKER = "is not a folder"
#: The stable substring of the harness-owned dedupe refusal the
#: drill-down flow's ``ctx_answer`` branch keys on (a read target that
#: is already a top-2 retrieval document — the mock then answers from
#: the document's text in the ``<documents>`` prompt block, exactly
#: what the refusal instructs). Keyed on a substring (not the whole
#: constant) so a re-wrap of the constant cannot silently re-route the
#: mock; the module-level assert below fails loudly if the substring
#: ever leaves the constant (the mock must never drift from
#: ``app.rag.agent.ALREADY_IN_CONTEXT``).
_ALREADY_IN_CONTEXT_MARKER = "Already in your context"
assert _ALREADY_IN_CONTEXT_MARKER in ALREADY_IN_CONTEXT, (
"mock drift: the dedupe marker left ALREADY_IN_CONTEXT"
)
#: One ``<document>`` block of the HIGH prompt's ``<documents>``
#: section (``app.rag.prompts.build_high_prompt``): the block is the
#: document identity (``source``/``path``/``title`` attributes) plus
#: the document's FULL text (never truncated on the retrieval path,
#: owner-locked A7) between the tags.
_DOCUMENT_BLOCK_RE = re.compile(
r'<document source="(?P<source>[^"]+)" path="(?P<path>[^"]+)" '
r'title="[^"]*">\n(?P<content>.*?)\n</document>',
re.S,
)
def _document_block(system: str, source: str, path: str) -> str | None:
"""The stored text of one ``<document>`` block (or ``None``).
The drill-down flow's ``ctx_answer`` branch: when the agent's read
of a top-2 retrieval document gets the ALREADY_IN_CONTEXT dedupe,
the document's full text is in the ``<documents>`` prompt — the
mock (the model) extracts it by the block's identity attributes
and quotes it, answering from the prompt as the refusal instructs.
"""
for block in _DOCUMENT_BLOCK_RE.finditer(system):
if block.group("source") == source and block.group("path") == path:
return block.group("content")
return None
#: The agent's drill-down ``ls`` result shapes (app.rag.agent
#: ``render_ls_top`` / ``render_folder_listing``, phase 94): the
#: top-level header ``"N sources:"`` and the per-source block line
#: ``"<source> — N documents"`` (summary lines are indented — they
#: never match); the folder-level header ``"<identity> — N documents,
#: M folders:"`` (identity = the source name or ``source/folder``).
_SOURCE_HEADER_RE = re.compile(r"^\d+ sources:$")
_SOURCE_LINE_RE = re.compile(r"^(?P<source>.+?) — (?P<n>\d+) documents$")
_FOLDER_HEADER_RE = re.compile(r"^(?P<identity>.+?) — \d+ documents, \d+ folders:$")
#: One DEAD app-level chat attempt costs exactly this many HTTP POSTs
#: while the endpoint stays down: the openai SDK's default policy
@@ -713,26 +873,60 @@ def _tool_results(body: dict[str, Any]) -> list[str]:
]
def _first_catalog_line(body: dict[str, Any]) -> str | None:
"""The first catalog line of a catalog listing in the messages.
def _first_file_line(body: dict[str, Any]) -> str | None:
"""The first file line (``source: X | path: Y | title: Z``) across
the ``tool``-role results in the messages, in message order (read
results — full documents, not listings — skipped; the ``_CATALOG_-
LINE_RE`` machinery).
A catalog listing is a ``tool``-role result whose FIRST line is the
agent's ``"N documents:"`` header (``_CATALOG_HEADER_RE``); its
first ``source: X | path: Y | title: Z`` line (the
``_CATALOG_LINE_RE`` machinery) is returned. ``None`` when no
catalog listing is in the messages — e.g. while only the teaching
refusal is there (the phase-72 LS-TEACH flow's request-2 state).
An empty listing (``"0 documents:"`` with no lines) returns
``""`` — the listing is present, it is just empty.
Phase 94: the drill-down ``ls`` carries file lines only at folder
levels — the top-level source listing and the teaching refusals
have none, so ``None`` here means "no folder level reached yet".
"""
for content in _tool_results(body):
lines = content.splitlines()
if not lines or not _CATALOG_HEADER_RE.match(lines[0]):
if content.startswith(_READ_RESULT_PREFIX):
continue
for line in lines[1:]:
for line in content.splitlines():
if _CATALOG_LINE_RE.match(line):
return line
return ""
return None
def _drill_target(body: dict[str, Any]) -> str | None:
"""The next source to drill into (phase 94, the drill step).
The drill-down ``ls`` top level lists SOURCES only (no file lines),
so a deterministic flow that needs a file line must drill one level:
``ls`` scoped to a source. The target is the FIRST source of the
top-level listing (registry order — the listing's own order) that
does not yet have a folder-level listing in the messages (a folder
header whose identity is the source or ``source/…``); ``None`` when
no top-level listing is in the messages yet (nothing to drill from)
or every listed source has been drilled (an empty KB — the flow
degenerates to the old re-list loop, settling at the round cap
exactly like the phase-70 empty-catalog case).
"""
sources: list[str] = []
drilled: set[str] = set()
for content in _tool_results(body):
lines = content.splitlines()
if not lines:
continue
if _SOURCE_HEADER_RE.match(lines[0]):
for line in lines[1:]:
match = _SOURCE_LINE_RE.match(line)
if match:
sources.append(match.group("source"))
else:
folder = _FOLDER_HEADER_RE.match(lines[0])
if folder:
drilled.add(folder.group("identity"))
for source in sources:
if not any(
identity == source or identity.startswith(source + "/")
for identity in drilled
):
return source
return None
@@ -853,11 +1047,14 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
``TOOLS_TRIGGER`` and ``MULTI_READ_TRIGGER``), classified by the
count of ``tool``-role read results:
* 0 read results: ``("list", "", "")`` (no catalog yet) or
``("read", source, path, "call_1")`` on the FIRST catalog doc.
* 1 read result: ``("read", source, path, "call_2")`` on the SECOND
catalog doc — the first listing line whose ``source/path``
differs from the one already read. A one-document catalog
* 0 read results: ``("list", "", "")`` (no listing yet),
``("drill", source, "call_1")`` when the top-level source
listing is in the messages but no file line yet (phase 94: the
drill — the top level carries sources only), or
``("read", source, path, "call_2")`` on the FIRST file-line doc.
* 1 read result: ``("read", source, path, "call_3")`` on the SECOND
file-line doc — the first listing line whose ``source/path``
differs from the one already read. A one-file listing
degenerates to the single-read ``("answer", ...)`` shape (nothing
second to read).
* 2 read results: ``("multi_answer", "", text)`` — the forced
@@ -880,30 +1077,37 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
if not body.get("tools"):
return None
docs = _catalog_docs(body)
if not docs:
return ("list", "", "")
return ("read", docs[0][0], docs[0][1], "call_1")
if docs:
return ("read", docs[0][0], docs[0][1], "call_2")
drill = _drill_target(body)
if drill is not None:
return ("drill", drill, "call_1")
return ("list", "", "")
if len(reads) == 1:
skip = reads[0][0]
second = next(
(d for d in _catalog_docs(body) if f"{d[0]}/{d[1]}" != skip), None
)
if second is None:
# One-document catalog: nothing second to read — the
# One-file listing: nothing second to read — the
# single-read answer shape (deterministic degenerate).
return ("answer", reads[0][0], reads[0][1])
return ("read", second[0], second[1], "call_2")
return ("read", second[0], second[1], "call_3")
(sp1, c1), (sp2, _c2) = reads[0], reads[1]
answer = f"Read {sp1}. {c1[:80]} I read {sp1} and {sp2}."
return ("multi_answer", "", answer)
# Phase-37 single-read flow — byte-identical to the original.
# Phase-37 single-read flow (phase 94: the drill step between the
# top-level listing and the first file line).
if reads:
return ("answer", reads[0][0], reads[0][1])
if not body.get("tools"):
return None
docs = _catalog_docs(body)
if docs:
return ("read", docs[0][0], docs[0][1], "call_1")
return ("read", docs[0][0], docs[0][1], "call_2")
drill = _drill_target(body)
if drill is not None:
return ("drill", drill, "call_1")
return ("list", "", "")
@@ -915,13 +1119,17 @@ def _ls_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
with ``{"path": "."}`` (id ``call_0``), ``finish_reason:
"tool_calls"``, no content.
* ``("correct",)`` — a ``tool``-role result is in the messages and
it is NOT a catalog listing (the teaching refusal): the
correction — ``ls`` with no arguments (id ``call_1``).
* ``("answer", line)`` — a ``tool``-role result whose first line
is the ``"N documents:"`` catalog header: the deterministic
no file line exists yet (the teaching refusal): the correction —
``ls`` with no arguments (id ``call_1``).
* ``("drill", source, "call_2")`` — the top-level source listing
is in the messages (the correction ran) but no file line yet
(phase 94: the top level carries sources only): the drill —
``ls`` scoped to the first source of the listing.
* ``("answer", line)`` — a ``source: X | path: Y | title: Z`` file
line is in the messages (the folder listing): the deterministic
content answer ``These are the indexed documents: <line>`` (the
first catalog line), ``finish_reason: "stop"`` — the loop
settled in ONE correction, not at the round cap.
first file line), ``finish_reason: "stop"`` — the loop settled
in ONE correction + ONE drill, not at the round cap.
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
section is missing (deflected turns never carry it), or
``tools`` are not offered and no tool results are in the
@@ -931,9 +1139,12 @@ def _ls_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
return None
if "<tools>" not in _system(body):
return None
line = _first_catalog_line(body)
line = _first_file_line(body)
if line is not None:
return ("answer", line)
drill = _drill_target(body)
if drill is not None:
return ("drill", drill, "call_2")
if _tool_results(body):
return ("correct",)
if not body.get("tools"):
@@ -996,6 +1207,92 @@ def _grep_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
return ("nomatch",)
def _drill_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
"""Classify a phase-94 scripted drill-down request (see the module
docstring). The question carries the scripted call
(``drill down the tree: ls [target]`` / ``drill down the tree: read
source/path``); the step is then discriminated statelessly from the
tool results, like the other marker flows:
* ``("call", verb, target, "call_0")`` — ``tools`` are offered and
no ``tool``-role result is in the messages yet: the scripted call
(``ls`` with NO path for the top level — empty target —, ``ls``
with the target for source/folder levels, ``read`` with the
combined ``source/path``).
* ``("recover", source)`` — the LAST tool result is the NOT-A-FOLDER
teaching refusal (it carries :data:`_NOT_A_FOLDER_MARKER`): the
scripted one-round recovery — ``ls`` the target's SOURCE segment
(the refusal listing the parent's subfolders is what lets the
model — and this mock — self-correct in the next round, the
phase-72 contract).
* ``("read_answer", combined, quote)`` — the LAST tool result is a
read result (``"Document <source/path>:\n<content>"``): the
deterministic echo answer ``Read <source/path>. <first 80 chars>
`` (the phase-37 single-read shape — the grounded-turn citation
contract the suite asserts).
* ``("ctx_answer", target, quote)`` — the scripted call was a
``read`` and the LAST tool result is the ALREADY_IN_CONTEXT
dedupe refusal (the target is already a top-2 retrieval document
— deterministic for the drill fixture: the read question names
the file's path, so the file self-matches the hybrid gate): the
model answers FROM THE PROMPT — ``Already in context: Read
<target>. <first 80 chars of the target document's text in the
``<documents>`` block>`` (the same citation shape as the
read-result branch — the document text reached the model either
way). Falls through to the echo when the target is not a
``<document>`` block (the premise broke — the suite fails
loudly on the answer).
* ``("echo", listing)`` — a listing landed (top-level or folder —
any other last result): the deterministic ECHO — the answer
carries the listing VERBATIM (``Here's the level I listed:\n
<listing>``), so the suite asserts on the exact tree level the
model saw (source ``— N documents`` lines + stored summaries,
subfolder lines, the ``source: X | path: Y | title: Z`` file
lines, the 50-line cap + note) through the rendered answer — the
house scripted-turn way of asserting on tool results (the mock is
the only E2E lens on the LLM's context).
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
section is missing (deflected turns never carry it), the scripted
call is unparseable, or ``tools`` are not offered and no tool
results are in the messages yet (e.g. ``agent_max_rounds=0``).
"""
user = _user(body)
if DRILL_TRIGGER not in user.lower():
return None
if "<tools>" not in _system(body):
return None
match = _DRILL_CALL_RE.search(user)
if match is None:
return None
verb = match.group("verb")
target = match.group("arg") or ""
results = _tool_results(body)
if not results:
if not body.get("tools"):
return None
return ("call", verb, target, "call_0")
last = results[-1]
if last.startswith(_READ_RESULT_PREFIX):
head, _, content = last.partition("\n")
# The read result is ``"Document <source/path>:\n<content>"`` —
# the head carries the server's appended ``:`` (removed here;
# a document path never legitimately ends with one).
combined = head[len(_READ_RESULT_PREFIX):].strip().removesuffix(":")
return ("read_answer", combined, content[:80])
if _ALREADY_IN_CONTEXT_MARKER in last and verb == "read" and "/" in target:
# The dedupe fired: the read target is already a top-2
# retrieval document, so its FULL text is in the
# ``<documents>`` prompt — answer from the prompt (the
# refusal's instruction), quoting the block's text.
src, _, p = target.partition("/")
content = _document_block(_system(body), src, p)
if content is not None:
return ("ctx_answer", target, content[:80])
if _NOT_A_FOLDER_MARKER in last:
return ("recover", target.split("/")[0])
return ("echo", last)
def long_answer() -> str:
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
a unique final line that must survive the stream untruncated."""
@@ -1103,6 +1400,28 @@ def compose_answer(body: dict[str, Any]) -> str:
user = _user(body)
if LONG_ANSWER_TRIGGER in user.lower():
answer = long_answer()
elif "FOLDER_SUMMARY_MODE" in system:
# Folder summary (phase 94, TODO.md L4): the ``lite`` stand-in
# returns the deterministic one-liner
# ``Fixture folder summary for <folder>.`` — <folder> is the
# user message's ``Folder: …`` header line (the generator puts
# the canonical source/folder identity there, imported as
# ``FOLDER_HEADER_PREFIX`` so the mock can never drift from it).
# The stored row therefore always names its folder — the drill-
# down E2E (``test_ls_tree_drilldown.py``) asserts on it.
# Checked BEFORE the ``SUMMARY_MODE`` branch: the folder marker
# CONTAINS the summary marker as a substring (``FOLDER_`` +
# ``SUMMARY_MODE``), so the summary branch would otherwise
# shadow every folder-summary call. Checked BEFORE the
# DEFLECT_MODE branch, like the other lite-mode markers (a
# deflection prompt never carries one).
header = user.splitlines()[0] if user else ""
folder = (
header.removeprefix(FOLDER_HEADER_PREFIX).strip()
if header.startswith(FOLDER_HEADER_PREFIX)
else "(unnamed folder)"
)
answer = f"Fixture folder summary for {folder}."
elif "SUMMARY_MODE" in system:
# Document summaries (phase 30): the ``lite`` stand-in returns a
# deterministic digest — the first 24 tokens of the user message
@@ -1594,9 +1913,16 @@ def chat_completions(body: dict[str, Any]) -> Any:
# The incident's misuse, deterministic: ls(path='.').
stream = _tool_call_stream("ls", {"path": "."}, "call_0")
elif ls_teach[0] == "correct":
# The one-round correction: the no-arg full listing.
# The one-round correction: the no-arg top-level
# listing (phase 94: sources only — the drill follows).
stream = _tool_call_stream("ls", {}, "call_1")
else: # "answer" — quote the first catalog line
elif ls_teach[0] == "drill":
# Phase 94: the top level lists sources only — drill
# one level into the first source for the file lines.
stream = _tool_call_stream(
"ls", {"path": ls_teach[1]}, ls_teach[2]
)
else: # "answer" — quote the first file line
answer = _apply_max_tokens(
f"These are the indexed documents: {ls_teach[1]}",
body.get("max_tokens"),
@@ -1607,10 +1933,67 @@ def chat_completions(body: dict[str, Any]) -> Any:
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
# Phase 94 (task 04): the deterministic SCRIPTED drill-down
# turns (sources → folders → files → the grounded read; the
# 50-line cap; the NOT-A-FOLDER teaching + scripted recovery) —
# checked BEFORE the plain TOOLS_TRIGGER flow (disjoint trigger
# phrases — the phase-71/72 ordering convention; the trigger
# needs the ``<tools>`` section, so deflected turns never hit
# it).
drill = _drill_flow(body)
if drill is not None:
if drill[0] == "call":
# The scripted call: ls with no path at the top level
# (empty target), ls/read with the parsed target.
stream = _tool_call_stream(
drill[1], {"path": drill[2]} if drill[2] else {}, drill[3]
)
elif drill[0] == "recover":
# The scripted one-round recovery after the NOT-A-FOLDER
# teaching: ls the target's source (the parent level).
stream = _tool_call_stream("ls", {"path": drill[1]}, "call_1")
elif drill[0] == "read_answer":
# Quote the read document (first 80 chars) — the
# phase-37 single-read shape.
stream = _sse_stream(
_apply_max_tokens(
f"Read {drill[1]}. {drill[2]}", body.get("max_tokens")
),
0.0,
)
elif drill[0] == "ctx_answer":
# The dedupe fired: the target's full text is in the
# <documents> prompt — answer from it (same citation
# shape, prefixed so the suite pins the dedupe path).
stream = _sse_stream(
_apply_max_tokens(
f"Already in context: Read {drill[1]}. {drill[2]}",
body.get("max_tokens"),
),
0.0,
)
else: # "echo" — the listing verbatim (the suite's lens)
stream = _sse_stream(
_apply_max_tokens(
f"Here's the level I listed:\n{drill[1]}",
body.get("max_tokens"),
),
0.0,
)
return StreamingResponse(
stream,
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
flow = _tool_flow(body)
if flow is not None:
if flow[0] == "list":
stream = _tool_call_stream("ls", {}, "call_0")
elif flow[0] == "drill":
# Phase 94: the drill step — ls scoped to the first
# source of the top-level listing (flow[1], flow[2] is
# the synthetic call id).
stream = _tool_call_stream("ls", {"path": flow[1]}, flow[2])
elif flow[0] == "read":
# flow[3] is the synthetic call id — "call_1" for the
# single-read flow and the multi-read first read,
tests/e2e/test_agent_document_tools.py
+53 -21
View File
@@ -11,15 +11,22 @@ deterministic marker flow in ``tests/e2e/mock_llm.py`` (user message
contains ``use your tools`` **and** the system prompt carries the
``<tools>`` section of the HIGH prompt; phase 70: the flow emits the
harness-aligned names — ``ls`` / ``read`` with the combined
``source/path`` identity):
``source/path`` identity; phase 94: the drill-down ``ls`` — the top
level lists sources only, so the flow drills one level into the first
source before the first file line exists):
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
``tool_calls`` deltas calling ``ls`` (id ``call_0``, no arguments,
``finish_reason: "tool_calls"``);
2. request 2 (a ``tool``-role catalog result in the messages) → streams a
``tool_calls`` delta calling ``read`` on the JOINED combined
``source/path`` of the FIRST catalog line (id ``call_1``);
3. request 3 (a ``tool``-role read result in the messages) → the content
2. request 2 (the top-level source listing in the messages — no file
lines yet) → a ``tool_calls`` delta — ``ls`` scoped to the FIRST
source of the listing (id ``call_1``) — the drill step (the seed
registers ``Deployments`` first, so the drill — and therefore the
read — lands on the JSON file);
3. request 3 (a ``tool``-role folder listing with file lines) → streams
a ``tool_calls`` delta calling ``read`` on the JOINED combined
``source/path`` of the FIRST file line (id ``call_2``);
4. request 4 (a ``tool``-role read result in the messages) → the content
answer ``Read <source/path>. <first 80 chars of the read document's
content>`` — so the suite can assert the read document reached the
model and landed in the answer.
@@ -72,7 +79,7 @@ from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import Chunk, Document, QueryLog
from app.models import Chunk, Document, GitSource, QueryLog
from e2e.auth_helpers import login
from tests.e2e.mock_llm import embed_text
@@ -148,7 +155,22 @@ READ_CHIP_HREF = f"/document.html?source={READ_SOURCE}&path={READ_PATH}&back=%2F
def _seed(db: Session) -> None:
"""The two-document pair from the TODO (see the module docstring)."""
"""The two-document pair from the TODO (see the module docstring).
Phase 94: the drill-down ``ls`` top level reads the registry — the
seed registers BOTH sources (TRUNCATEd in ``_reset_db``),
``Deployments`` FIRST: registry order is ``(added_at, id)``, so the
mock's drill (first source of the listing) lands on the JSON file
deterministically — independent of the operator's
``BOR_GIT_SOURCES`` (a non-empty table ignores the env fallback).
"""
# COMMIT between the inserts (not flush): ``added_at`` is
# ``server_default now()`` — the transaction timestamp — and the
# tie-break is the random uuid ``id``, so one-transaction rows order
# nondeterministically.
db.add(GitSource(url=READ_SOURCE, kind="local"))
db.commit()
db.add(GitSource(url=SEED_SOURCE, kind="local"))
md = Document(
source=SEED_SOURCE,
path=SEED_PATH,
@@ -195,7 +217,10 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
"""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources"
)
)
db.commit()
if seed is not None:
@@ -389,12 +414,14 @@ def test_marker_question_lists_reads_and_quotes(
assert i_list is not None and i_read is not None, statuses
assert i_list < i_read, statuses
# Wire level: exactly two `tool` frames — ``ls`` then ``read`` (the
# combined source/path as the model passed it) — and both ahead of
# the first `delta` frame.
# Wire level: exactly three `tool` frames — ``ls`` (the top level),
# the drill ``ls`` scoped to the first source (phase 94), then
# ``read`` (the combined source/path as the model passed it) — and
# all three ahead of the first `delta` frame.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": READ_SOURCE},
{"type": "tool", "name": "read", "argument": READ_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
@@ -408,12 +435,15 @@ def test_marker_question_lists_reads_and_quotes(
(READ_SOURCE, READ_PATH),
]
# Both tool lines, in order, above the answer.
# All three tool lines, in order, above the answer (phase 94: the
# drill line is "Listing documents in <source>").
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1)).to_contain_text(READ_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1)).to_contain_text(READ_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2)).to_contain_text(READ_SP)
# The final answer quotes the read document (the mock's deterministic
# quote: "Read <source/path>. <first 80 chars of its content>").
@@ -451,18 +481,20 @@ def test_tool_lines_re_render_after_reload(
_submit(page, MARKER_QUESTION)
_wait_settled(page)
expect(page.locator(".msg.brain .tool-call")).to_have_count(2)
expect(page.locator(".msg.brain .tool-call")).to_have_count(3)
page.reload()
expect(page.locator("#empty-state")).to_be_hidden()
# The persisted record re-renders BOTH tool lines, in saved order,
# through the same append helper as the live frames.
# The persisted record re-renders ALL THREE tool lines, in saved
# order, through the same append helper as the live frames.
restored = page.locator(".msg.brain .tool-call")
expect(restored).to_have_count(2)
expect(restored).to_have_count(3)
expect(restored.nth(0)).to_contain_text("Listing documents")
expect(restored.nth(1)).to_contain_text("Reading ")
expect(restored.nth(1)).to_contain_text(READ_SP)
expect(restored.nth(1)).to_contain_text("Listing documents in")
expect(restored.nth(1)).to_contain_text(READ_SOURCE)
expect(restored.nth(2)).to_contain_text("Reading ")
expect(restored.nth(2)).to_contain_text(READ_SP)
# Answer + the read-document chip are intact (phase-14 restore path).
bubble = page.locator(".msg.brain .bubble").last
tests/e2e/test_agent_unlimited_tools.py
+67 -30
View File
@@ -11,19 +11,26 @@ message contains BOTH ``use your tools`` (``TOOLS_TRIGGER``) and ``read
two documents`` (``MULTI_READ_TRIGGER``) **and** the system prompt
carries the ``<tools>`` section of the HIGH prompt; phase 70: the flow
emits the harness-aligned names — ``ls``, then ``read`` on the JOINED
combined ``source/path`` of each catalog line):
combined ``source/path`` of each file line; phase 94: the drill-down
``ls`` — the top level lists sources only, so the flow drills one
level into the first source before the first file line exists):
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
``tool_calls`` deltas calling ``ls`` (id ``call_0``);
2. request 2 (the ``tool``-role catalog result) → ``read`` on the
JOINED combined ``source/path`` of the FIRST catalog line
(id ``call_1``);
3. request 3 (one ``tool``-role read result) → ``read`` on the JOINED
combined ``source/path`` of the SECOND catalog line (id ``call_2``)
— the pre-phase-45 per-tool budgets would have refused exactly this
2. request 2 (the top-level source listing in the messages — no file
lines yet) → the drill: ``ls`` scoped to the FIRST source of the
listing (id ``call_1``); the seed registers ``Deployments`` first,
and both read documents live in it — so the drill's folder listing
carries BOTH file lines;
3. request 3 (a ``tool``-role folder listing with file lines) →
``read`` on the JOINED combined ``source/path`` of the FIRST file
line (id ``call_2``);
4. request 4 (one ``tool``-role read result) → ``read`` on the JOINED
combined ``source/path`` of the SECOND file line (id ``call_3``) —
the pre-phase-45 per-tool budgets would have refused exactly this
second read (``No reading budget left — answer with what you
have.``);
4. request 4 (two read results) → the forced answer, byte-stable: the
5. request 5 (two read results) → the forced answer, byte-stable: the
single-read shape quoting the FIRST read result, plus the line
``I read <sp1> and <sp2>.`` naming both read paths in read order.
@@ -49,10 +56,11 @@ rejection, not the multi-read
flow this story proves.
Test → story mapping (Playwright Mapping Rule):
1. ``test_multi_read_turn`` — the turn streams THREE ``tool`` frames /
``.tool-call`` lines in order (one list — "is listing documents" —
and two reads — "is reading <source/path>" — the #send-status
transition recorded deterministically via MutationObserver), then a
1. ``test_multi_read_turn`` — the turn streams FOUR ``tool`` frames /
``.tool-call`` lines in order (the top-level list, the drill list —
phase 94 — "is listing documents", and two reads — "is reading
<source/path>" — the #send-status transition recorded
deterministically via MutationObserver), then a
final non-deflected answer containing the mock's byte-stable
``I read <sp1> and <sp2>.`` line; the round cap (default 10) bounds
the turn, no budget refusal anywhere.
@@ -84,7 +92,7 @@ from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import Chunk, Document, QueryLog
from app.models import Chunk, Document, GitSource, QueryLog
from e2e.auth_helpers import login
from tests.e2e.mock_llm import embed_text
@@ -209,7 +217,23 @@ def _doc(source: str, path: str, title: str, content: str) -> Document:
def _seed(db: Session) -> None:
"""The three-document KB from the module docstring."""
"""The three-document KB from the module docstring.
Phase 94: the drill-down ``ls`` top level reads the registry —
register BOTH sources (TRUNCATEd in ``_reset_db``), ``Deployments``
FIRST (registry order is ``(added_at, id)``): the mock's drill
(first source of the listing) lands on the folder that carries
BOTH file lines. A non-empty table also ignores the operator's
``BOR_GIT_SOURCES`` fallback — deterministic.
"""
# COMMIT between the inserts (not flush): ``added_at`` is
# ``server_default now()`` — the transaction timestamp — and the
# tie-break is the RANDOM uuid ``id``, so two rows in one
# transaction order nondeterministically (the integration
# ``registry`` fixture's pattern).
db.add(GitSource(url=READ1_SOURCE, kind="local"))
db.commit()
db.add(GitSource(url=SEED_SOURCE, kind="local"))
md = _doc(SEED_SOURCE, SEED_PATH, "AWS Route 53 Notes", ROUTE53_CONTENT)
db.add(md)
db.flush()
@@ -240,7 +264,10 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
"""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources"
)
)
db.commit()
if seed is not None:
@@ -399,13 +426,16 @@ def test_multi_read_turn(
_submit(page, MULTI_QUESTION)
_wait_settled(page)
# Wire level: exactly THREE `tool` frames — ls, read #1, read #2
# (each read's argument is the JOINED combined source/path), in
# order — and all ahead of the first `delta` frame. This third
# frame is the one the pre-phase-45 read budget refused.
# Wire level: exactly FOUR `tool` frames — the top-level ls, the
# drill ls scoped to the first source (phase 94), then read #1 and
# read #2 (each read's argument is the JOINED combined
# source/path), in order — and all ahead of the first `delta`
# frame. The fourth frame is the one the pre-phase-45 read budget
# refused.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": READ1_SOURCE},
{"type": "tool", "name": "read", "argument": READ1_SP},
{"type": "tool", "name": "read", "argument": READ2_SP},
]
@@ -439,14 +469,17 @@ def test_multi_read_turn(
), statuses
assert i_list < i_read1 < i_read2, statuses
# Three visible tool lines, in order, above the answer.
# Four visible tool lines, in order, above the answer (phase 94:
# the drill line is "Listing documents in <source>").
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(3)
expect(lines).to_have_count(4)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1)).to_contain_text(READ1_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1)).to_contain_text(READ1_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2)).to_contain_text(READ2_SP)
expect(lines.nth(2)).to_contain_text(READ1_SP)
expect(lines.nth(3)).to_contain_text("Reading ")
expect(lines.nth(3)).to_contain_text(READ2_SP)
# The final answer is non-deflected, quotes the FIRST read result,
# and names BOTH read paths (the mock's byte-stable line).
@@ -563,19 +596,23 @@ def test_single_tool_flow_regression(
_submit(page, SINGLE_QUESTION)
_wait_settled(page)
# Exactly TWO tool frames — ls then ONE read of the first catalog
# line (the JOINED combined source/path) — no second read (the
# marker carries no multi-read trigger).
# Exactly THREE tool frames — the top-level ls, the drill ls
# (phase 94), then ONE read of the first file line (the JOINED
# combined source/path) — no second read (the marker carries no
# multi-read trigger).
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": READ1_SOURCE},
{"type": "tool", "name": "read", "argument": READ1_SP},
]
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1)).to_contain_text(READ1_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1)).to_contain_text(READ1_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2)).to_contain_text(READ1_SP)
# The single-read answer shape: quotes the read document; it does
# NOT carry the multi-read both-named line (READ2 was never read).
tests/e2e/test_big_read_progress.py
+58 -31
View File
@@ -19,27 +19,33 @@ phase-64 default 0.15 is far below the 5 s tool-line threshold):
NOTE the prelude: the chat flow's QUESTION EMBEDDING also travels
through the proxy (one 6 s sleep) before the agent loop starts, so
the wall-clock timeline of one turn is:
the wall-clock timeline of one turn is (phase 94: the drill-down
``ls`` adds a drill step — the top level lists sources only, so the
flow drills one level before the first file line exists):
1. t≈12 s — the first SSE ``tool`` frame ("🔎 Listing documents"): the
6 s embedding sleep + request 1's 6 s sleep (``tools`` offered, no
tool results yet — streams ONLY the ``ls`` ``tool_calls`` delta,
id ``call_0``) + the mock's ~0.2 s tool-call stream;
2. t≈19 s — the second ``tool`` frame ("📄 Reading
Deployments/example-record-file.json"): request 2's 6 s sleep (a
``tool``-role catalog result → streams a ``read`` ``tool_calls``
delta on the JOINED combined ``source/path`` of the FIRST catalog
line, id ``call_1``);
3. t≈25 s — the content answer ``Read <source/path>. <first 80 chars
of the read document's content>``: request 3's 6 s sleep (a
2. t≈19 s — the second ``tool`` frame ("🔎 Listing documents in
Deployments"): request 2's 6 s sleep (the top-level source listing
in the messages — no file lines yet → streams the drill ``ls``
scoped to the first source, id ``call_1``);
3. t≈25 s — the third ``tool`` frame ("📄 Reading
Deployments/example-record-file.json"): request 3's 6 s sleep (a
``tool``-role folder listing with file lines → streams a ``read``
``tool_calls`` delta on the JOINED combined ``source/path`` of the
FIRST file line, id ``call_2``);
4. t≈31 s — the content answer ``Read <source/path>. <first 80 chars
of the read document's content>``: request 4's 6 s sleep (a
``tool``-role read result) → the first answer ``delta``.
So each post-tool-frame gap (≈6.3 s — the 6 s sleep + the short
tool-call stream) is PAST the 5 s tool-line threshold
(``TOOL_LINE_ELAPSED_AFTER_MS = 5_000``), and the 10 s pre-token
typing hint (the existing ``startThinkingClock`` gate, ticking from
t≈0) is visible throughout both post-tool gaps — everything before the
answer's first delta (the whole turn runs ≈25 s + overhead, acceptable
t≈0) is visible throughout all post-tool gaps — everything before the
answer's first delta (the whole turn runs ≈31 s + overhead, acceptable
for an isolated story suite). The four tests pin: (1) the latest tool
line's ticking "(Ns)" suffix, (2) the typing indicator's visible "Ns"
hint (+ the kept aria channel), (3) BOTH settling the instant the
@@ -55,15 +61,20 @@ task that writes the visible span) — so the two channels read
polls until they align (the deterministic same-moment pin).
**KB fixture** — byte-identical to the phase-37 suite
(``tests/e2e/test_agent_document_tools.py``): ``Homelab/aws-route53.md``
(``tests/e2e/test_agent_document_tools.py``), plus the phase-94
registry rows: both sources are registered in ``git_sources``
(``Deployments`` FIRST — registry order ``(added_at, id)``), so the
mock's drill (first source of the top-level listing) lands on the
folder that carries the file lines. ``Homelab/aws-route53.md``
carries one chunk embedded with the mock's own bag-of-words vector —
the marker question (phase 37's exact question, which carries the
trigger phrase) cosines ≈0.69 against it, well past the E2E 0.30
threshold, so the turn is grounded and the HIGH prompt carries the
``<tools>`` section; ``Deployments/example-record-file.json`` is
indexed WITHOUT chunks, and its ``(source, path)`` sorts FIRST in the
catalog (``Deployments`` < ``Homelab``) — exactly the line the mock's
second request reads, so the answer is byte-stable.
folder listing's file lines (``Deployments`` < ``Homelab``) — exactly
the line the mock's read request targets, so the answer is
byte-stable.
Test → story mapping (Playwright Mapping Rule):
1. ``test_tool_line_shows_ticking_elapsed``
@@ -92,7 +103,7 @@ from sqlalchemy.orm import Session
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import Chunk, Document
from app.models import Chunk, Document, GitSource
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
@@ -211,8 +222,10 @@ SUFFIX_TIMEOUT_MS = 12_000
#: the indicator is removed at the first delta (≈25 s): 20 s from the
#: check start cannot flake.
TYPING_HINT_TIMEOUT_MS = 20_000
#: The answer settles at ≈19 s from submit; every settle wait here starts
#: ≥12 s in — 30 s keeps ≥2× headroom on the remaining window.
#: The answer settles at ≈31 s from submit (phase 94: the drill step
#: adds a third tool round); every settle wait here starts ≈17 s in
#: (after the first line's suffix) — 30 s keeps ≥2× headroom on the
#: remaining window.
SETTLE_TIMEOUT_MS = 30_000
#: The restore is a synchronous boot re-render — 15 s is ample.
RESTORE_TIMEOUT_MS = 15_000
@@ -360,7 +373,16 @@ def app_url(app_server: str) -> str:
def _seed(db: Session) -> None:
"""The phase-37 two-document pair, byte-identical (see the module
docstring)."""
docstring), plus the phase-94 registry rows: both sources
registered, ``Deployments`` FIRST — the mock's drill (first
source of the top-level listing) lands on the JSON file."""
# COMMIT between the inserts (not flush): ``added_at`` is
# ``server_default now()`` — the transaction timestamp — and the
# tie-break is the random uuid ``id``, so one-transaction rows order
# nondeterministically.
db.add(GitSource(url=READ_SOURCE, kind="local"))
db.commit()
db.add(GitSource(url=SEED_SOURCE, kind="local"))
md = Document(
source=SEED_SOURCE,
path=SEED_PATH,
@@ -408,7 +430,7 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, "
"steering_notes, kb_overview, saved_chats"
"steering_notes, kb_overview, saved_chats, git_sources"
)
)
db.commit()
@@ -514,7 +536,7 @@ def _wait_settled(page: Page) -> None:
#: The FIRST tool line's suffix, pinned to the line ("first .tool-call
#: child of the .tool-calls container" — the "Listing documents" line;
#: the read frame's second line arms its OWN clock 5 s later and is
#: the drill frame's second line arms its OWN clock 5 s later and is
#: never the target here).
FIRST_LINE_SUFFIX = "#messages .tool-calls .tool-call:first-child .tool-elapsed"
@@ -611,8 +633,8 @@ def test_typing_indicator_shows_visible_elapsed(
)
# The clock ticks while the gap holds: ≥1.5 s later the value is
# strictly greater (the read frame is ≥6 s away; the first delta
# — which removes the whole indicator — is ≈25 s from submit).
# strictly greater (the drill frame is ≥6 s away; the first delta
# — which removes the whole indicator — is ≈31 s from submit).
page.wait_for_timeout(int(SAMPLE_GAP_S * 1000))
v2 = _elapsed_value(
page.locator("#typing-indicator .typing-elapsed").first.text_content(),
@@ -644,7 +666,7 @@ def test_indicators_settle_when_the_answer_arrives(
# First the ticking state (the test-1 wait, reused)…
expect(page.locator(FIRST_LINE_SUFFIX)).to_be_visible(timeout=SUFFIX_TIMEOUT_MS)
# …then the answer arrives (≈25 s from submit — see the module
# …then the answer arrives (≈31 s from submit — see the module
# docstring's timeline) and settles.
_wait_settled(page)
@@ -661,13 +683,16 @@ def test_indicators_settle_when_the_answer_arrives(
expect(page.locator("#messages .tool-elapsed")).to_have_count(0)
expect(page.locator("#typing-indicator")).to_have_count(0)
# The tool lines remain — the permanent record, both present with
# their pinned text (phase 37's pattern).
# The tool lines remain — the permanent record, all three present
# with their pinned text (phase 37's pattern + the phase-94 drill
# line).
lines = page.locator("#messages .tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1)).to_contain_text(READ_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1)).to_contain_text(READ_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2)).to_contain_text(READ_SP)
# The answer bubble is complete (the mock's deterministic quote).
bubble = page.locator(".msg.brain .bubble").last
@@ -705,12 +730,14 @@ def test_restored_turn_has_no_timer(
expect(page.locator("#empty-state")).to_be_hidden(timeout=RESTORE_TIMEOUT_MS)
# The persisted record re-renders the tool lines, in saved order
# (the phase-37 reload pin, mirrored)…
# (the phase-37 reload pin, mirrored + the phase-94 drill line)…”
restored = page.locator("#messages .tool-call")
expect(restored).to_have_count(2, timeout=RESTORE_TIMEOUT_MS)
expect(restored).to_have_count(3, timeout=RESTORE_TIMEOUT_MS)
expect(restored.nth(0)).to_contain_text("Listing documents")
expect(restored.nth(1)).to_contain_text("Reading ")
expect(restored.nth(1)).to_contain_text(READ_SP)
expect(restored.nth(1)).to_contain_text("Listing documents in")
expect(restored.nth(1)).to_contain_text(READ_SOURCE)
expect(restored.nth(2)).to_contain_text("Reading ")
expect(restored.nth(2)).to_contain_text(READ_SP)
# …with NO timer (A6 — the restore never arms the clock) and the
# page otherwise settled.
tests/e2e/test_harness_aligned_tools.py
+52 -22
View File
@@ -13,9 +13,13 @@ deterministic marker flows in ``tests/e2e/mock_llm.py`` (phase 70: the
flows emit the NEW names with the NEW argument shapes):
* the READ flow (``use your tools`` (``TOOLS_TRIGGER``) + the HIGH
prompt's ``<tools>`` section): ``ls`` (id ``call_0``, no arguments) →
``read`` on the JOINED combined ``source/path`` of the first catalog
line (id ``call_1``) → the ``Read <source/path>. <quote>`` answer;
prompt's ``<tools>`` section; phase 94: the drill-down ``ls`` — the
top level lists sources only, so the flow drills one level before the
first file line exists): ``ls`` (id ``call_0``, no arguments) → the
drill ``ls`` scoped to the first source of the listing
(id ``call_1``) → ``read`` on the JOINED combined ``source/path`` of
the first file line (id ``call_2``) → the ``Read <source/path>.
<quote>`` answer;
* the SEARCH flow (``search your documents`` (``SEARCH_TRIGGER``) + the
``<tools>`` section): ``grep`` with ``{"pattern": SEARCH_PATTERN}``
(id ``call_0``) → the ``Found <matched line>`` answer.
@@ -41,13 +45,14 @@ KB fixtures:
Test → phase mapping (Playwright Mapping Rule):
1. ``test_read_flow_lines_answer_sources_no_raw_markup`` — the
grounded READ turn: the UI shows the ``ls`` line (unscoped "🔎
Listing documents", no argument) then the "📄 Reading <source/path>"
line with the combined path in a ``<code>`` element, the answer
streams and quotes the read document, the done-state sources
include the read document, and NO raw tool markup (``<|…|>``,
``tool_call``) appears anywhere in the DOM — the live incident this
phase fixes.
grounded READ turn: the UI shows the unscoped ``ls`` line ("🔎
Listing documents", no argument), the drill line ("🔎 Listing
documents in <source>" — phase 94, the source in a ``<code>``
element), then the "📄 Reading <source/path>" line with the
combined path in a ``<code>`` element, the answer streams and quotes
the read document, the done-state sources include the read document,
and NO raw tool markup (``<|…|>``, ``tool_call``) appears anywhere
in the DOM — the live incident this phase fixes.
2. ``test_grep_flow_line_then_answer`` — the grounded SEARCH turn: the
"🔎 Searching for <pattern>" line (sentinel in ``<code>``) then the
matched-line answer.
@@ -74,7 +79,7 @@ from sqlalchemy.orm import Session
from app.config import Settings
from app.db import SessionLocal
from app.models import Chunk, Document
from app.models import Chunk, Document, GitSource
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from e2e.auth_helpers import login
@@ -156,7 +161,22 @@ READ_ANSWER_QUOTE = RECORD_FILE_CONTENT[:80]
def _seed_read_pair(db: Session) -> None:
"""The two-document READ-flow KB (see the module docstring)."""
"""The two-document READ-flow KB (see the module docstring).
Phase 94: the drill-down ``ls`` top level reads the registry —
register BOTH sources (TRUNCATEd in ``_reset_db_read_pair``),
``Deployments`` FIRST (registry order ``(added_at, id)``): the
mock's drill (first source of the listing) lands on the JSON file
— the read the assertions expect. A non-empty table also ignores
the operator's ``BOR_GIT_SOURCES`` fallback — deterministic.
"""
# COMMIT between the inserts (not flush): ``added_at`` is
# ``server_default now()`` — the transaction timestamp — and the
# tie-break is the random uuid ``id``, so one-transaction rows order
# nondeterministically.
db.add(GitSource(url=READ_SOURCE, kind="local"))
db.commit()
db.add(GitSource(url=SEED_SOURCE, kind="local"))
md = Document(
source=SEED_SOURCE,
path=SEED_PATH,
@@ -267,7 +287,10 @@ def _reset_db_read_pair() -> None:
answers."""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources"
)
)
db.commit()
_seed_read_pair(db)
@@ -386,15 +409,18 @@ def test_read_flow_lines_answer_sources_no_raw_markup(
_submit(page, READ_QUESTION)
_wait_settled(page)
# The UI shows the ls line (UNSCOPED — no argument, no <code>) then
# The UI shows the ls line (UNSCOPED — no argument, no <code>),
# the drill line (phase 94 — the source in a <code> element), then
# the "📄 Reading <source/path>" line with the COMBINED path in a
# <code> element (the path is data, never markup).
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(0).locator("code")).to_have_count(0)
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1).locator("code")).to_have_text(READ_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1).locator("code")).to_have_text(READ_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2).locator("code")).to_have_text(READ_SP)
# The answer streamed and quotes the read document (the mock's
# deterministic echo: "Read <source/path>. <first 80 chars>").
@@ -402,12 +428,14 @@ def test_read_flow_lines_answer_sources_no_raw_markup(
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
# Wire level: ls then read — the phase-70 argument rule (ls
# unscoped → null; read → the combined path as passed) — ahead of
# the first delta.
# Wire level: ls, the drill ls scoped to the first source (phase
# 94), then read — the phase-70 argument rule (ls unscoped → null;
# scoped ls → the source name; read → the combined path as passed)
# — ahead of the first delta.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": READ_SOURCE},
{"type": "tool", "name": "read", "argument": READ_SP},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
@@ -508,10 +536,12 @@ def test_wire_argument_rule_across_both_flows(
read_tools = _tool_frames(read_frames)
search_tools = _tool_frames(search_frames)
# The ordered, combined tool-frame sequence across both flows: the
# argument rule end-to-end — read → the combined path as passed,
# grep → the pattern, ls → null when unscoped.
# argument rule end-to-end — the drill ls (phase 94) → the source
# name, read → the combined path as passed, grep → the pattern,
# ls → null when unscoped.
assert read_tools + search_tools == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": READ_SOURCE},
{"type": "tool", "name": "read", "argument": READ_SP},
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN},
]
tests/e2e/test_ls_tree_drilldown.py
+772
View File
@@ -0,0 +1,772 @@
"""Phase 94 task 04 E2E (Playwright, mock-only): the ``ls`` drill-down tree.
The dedicated story suite for ``94_ls_tree_drilldown`` (owner TODO.md L4):
the LLM drills ``ls()`` → ``ls(source)`` → ``ls(source/folder)`` →
``read(source/file)`` through the real UI, the sync-time folder summaries
(must exist after a changed sync under the deterministic mock's
``FOLDER_SUMMARY_MODE`` branch) show up in the ``ls`` output, and the
50-line cap holds on a wide folder.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real ``turbo``
does whatever it does with the tools, while this story's gate is the
deterministic SCRIPTED drill-down flow in ``tests/e2e/mock_llm.py``
(``DRILL_TRIGGER``: user message contains ``drill down the tree`` —
the question carries its own tool call after the colon,
``drill down the tree: ls [target]`` / ``drill down the tree: read
source/path`` — **and** the system prompt carries the ``<tools>``
section of the HIGH prompt). The mock echoes the received tool result
into its final answer (the house scripted-turn way of asserting on tool
results — the mock is the only E2E lens on the LLM's context), so every
tree-level assertion below lands on the rendered answer; the DOM
assertions cover the tool lines + the answer.
KB fixture — a host temp dir tree (``tmp_path_factory``; the app runs on
the same host) with TWO registered local sources (the
``test_local_directory_sources.py`` registration + real-Sync pattern —
registration through the authenticated API, the real in-process
``POST /api/sync`` pipeline; no git anywhere):
* ``alpha/`` — ``root-note.md`` at the source root, ``one/`` (2 docs),
``two/`` (2 docs — the read target lives here) and ``wide/`` (51 tiny
files — the 50-line cap subject);
* ``beta/`` — ``gamma/`` (2 docs).
Every fixture doc carries the words ``drill down the tree`` in its body,
so every scripted question (which contains the trigger phrase) FTS-matches
at least one chunk — the honesty gate is HIGH for all turns regardless
of the mock's cosine distribution, and the ``<tools>`` section is present
(the flow's precondition).
The mock's canned ``FOLDER_SUMMARY_MODE`` branch (phase 94 task 01)
stores, per ≥ 2-doc folder, the deterministic one-liner
``Fixture folder summary for <source>[/<folder>].`` — the ``synced_kb``
module fixture pins those exact rows in ``folder_summaries`` after the
sync, and the drill answers assert on them in the ``ls`` output.
Test → story mapping (Playwright Mapping Rule):
1. ``test_drill_down_sources_folders_files_and_read`` — the four
scripted turns: ``ls()`` (the ``🔎 Listing documents`` line + the
top-level shape — one ``— N documents`` line per source + the
canned source-root summaries), ``ls(alpha)`` (folder lines with
their summaries + the root file line), ``ls(alpha/two)`` (the exact
``source: X | path: Y | title: Z`` file lines), and the grounded
``read(alpha/two/two-a.md)`` (the ``📄 Reading …`` line + the answer
citing the document, the phase-37 assertion pattern). The read
target is DELIBERATELY a top-2 retrieval document for its question
(the question names the file's path, so the file self-matches the
hybrid gate deterministically): the agent's phase-72 dedupe returns
``ALREADY_IN_CONTEXT`` (a refusal — counts in nothing), and the
mock answers FROM THE ``<documents>`` PROMPT with the same citation
shape (``Already in context: Read <sp>. <first 80 chars>``) — the
document text reached the model either way, and the prefix pins
that the dedupe notice itself reached it.
2. ``test_wide_folder_holds_the_fifty_line_cap`` — ``ls(alpha/wide)``
on the 51-file folder: the mock's echo carries exactly 50 file lines
+ the ``…and 1 more documents in this folder — use grep (pattern)…``
note; the 51st file never reaches the model.
3. ``test_not_a_folder_teaching_and_scripted_recovery`` — the mock calls
``ls(alpha/nope)``; the NOT-A-FOLDER teaching line (the argument
echoed, the parent's subfolders listed) is visible to the model —
the mock's scripted recovery branch keys on receiving it — and the
scripted ``ls(alpha)`` recovery lands (the answer is the parent's
listing; the loop ends in one refusal + one correction, not at the
round cap).
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import httpx
import pytest
from playwright.sync_api import Locator, Page, expect
from sqlalchemy import select, text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import FolderSummary, QueryLog
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_LSTREE", "8136"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
# --------------------------------------------------------------------------
# Fixture documents (deterministic, token-controlled)
# --------------------------------------------------------------------------
ALPHA = "alpha"
BETA = "beta"
ROOT_NOTE = "root-note.md"
TWO_A = "two/two-a.md"
TWO_B = "two/two-b.md"
READ_SP = f"{ALPHA}/{TWO_A}"
ALPHA_COUNT = 56 # 1 root note + 2 one/ + 2 two/ + 51 wide/
BETA_COUNT = 2
TOTAL_DOCS = ALPHA_COUNT + BETA_COUNT
WIDE_COUNT = 51
LS_MAX_FILE_LINES = 50 # app.rag.agent.LS_MAX_FILE_LINES — the cap under test
#: Every fixture body carries ``drill down the tree`` (the trigger
#: phrase's words): every scripted question FTS-matches at least one
#: chunk → HIGH gate → the ``<tools>`` section the flow keys on.
DRILL_LEAD = "The drill down the tree fixture note"
def _md(title: str, body: str) -> str:
return f"# {title}\n\n{body}\n"
#: The read target — its FIRST line is ≥ 80 chars, so the mock's
#: first-80-chars quote (the phase-37 single-read shape) is newline-free
#: and the rendered-text assertion matches it verbatim. Pinned by the
#: assert below.
TWO_A_TITLE = (
"Alpha Two A — the drill-down read target for the alpha two folder "
"listing turn in the brain of reese fixture"
)
TWO_A_CONTENT = _md(
TWO_A_TITLE,
f"{DRILL_LEAD} for alpha two: this document covers topic A of the "
"alpha source tree; it is the file the scripted read turn opens "
"from the alpha/two listing.",
)
assert "\n" not in TWO_A_CONTENT[:80] # the quote must stay one line
#: The sync-time folder summaries the mock's canned ``FOLDER_SUMMARY_MODE``
#: branch stores (task 01's byte-stable template), in
#: ``(source, folder_path)`` order: one row per ≥ 2-doc folder (the
#: recursive-subtree rule) — the ``""`` rows are the source roots.
EXPECTED_SUMMARIES: list[tuple[str, str, str]] = [
(ALPHA, "", f"Fixture folder summary for {ALPHA}."),
(ALPHA, "one", f"Fixture folder summary for {ALPHA}/one."),
(ALPHA, "two", f"Fixture folder summary for {ALPHA}/two."),
(ALPHA, "wide", f"Fixture folder summary for {ALPHA}/wide."),
(BETA, "", f"Fixture folder summary for {BETA}."),
(BETA, "gamma", f"Fixture folder summary for {BETA}/gamma."),
]
assert [
(source, folder) for source, folder, _s in EXPECTED_SUMMARIES
] == sorted((source, folder) for source, folder, _s in EXPECTED_SUMMARIES)
# --- the pinned tree levels (app.rag.agent's phase-94 templates) -------
#: ``ls()`` — the top level: sources in registry order (alpha registered
#: first), each with its recursive count + stored source-root summary.
TOP_HEADER = "2 sources:"
TOP_LINES = [
f"{ALPHA} — {ALPHA_COUNT} documents",
EXPECTED_SUMMARIES[0][2],
f"{BETA} — {BETA_COUNT} documents",
EXPECTED_SUMMARIES[4][2],
]
#: ``ls(alpha)`` — the source root: the subfolder lines (path order)
#: with their stored summaries, then the root's own file line (the
#: canonical ``read``/``grep`` identity format, unchanged).
SOURCE_HEADER = f"{ALPHA} — 1 documents, 3 folders:"
SOURCE_LINES = [
f"one/ — 2 documents: {EXPECTED_SUMMARIES[1][2]}",
f"two/ — 2 documents: {EXPECTED_SUMMARIES[2][2]}",
f"wide/ — {WIDE_COUNT} documents: {EXPECTED_SUMMARIES[3][2]}",
f"source: {ALPHA} | path: {ROOT_NOTE} | title: Alpha Root Note",
]
#: ``ls(alpha/two)`` — a leaf folder: the file lines in EXACTLY the
#: existing ``source: X | path: Y | title: Z`` format, path order.
FOLDER_HEADER = f"{ALPHA}/two — 2 documents, 0 folders:"
FOLDER_LINES = [
f"source: {ALPHA} | path: {TWO_A} | title: {TWO_A_TITLE}",
f"source: {ALPHA} | path: {TWO_B} | title: Alpha Two B",
]
#: ``ls(alpha/wide)`` — the 50-line cap: 51 files → 50 lines (path
#: order: wide-01 … wide-50) + one deterministic grep-pointer note;
#: wide-51 never reaches the model.
WIDE_HEADER = f"{ALPHA}/wide — {WIDE_COUNT} documents, 0 folders:"
WIDE_FIRST = f"source: {ALPHA} | path: wide/wide-01.md | title: Wide 01"
WIDE_LAST = f"source: {ALPHA} | path: wide/wide-50.md | title: Wide 50"
WIDE_NOTE = (
"…and 1 more documents in this folder — use grep "
"(pattern) to find a specific one."
)
# --- the scripted turns (the mock's ``DRILL_TRIGGER`` questions) -------
TOP_QUESTION = "Drill down the tree: ls — what sources are indexed?"
SOURCE_QUESTION = f"Drill down the tree: ls {ALPHA} — what's in source {ALPHA}?"
FOLDER_QUESTION = f"Drill down the tree: ls {ALPHA}/two — list that folder"
READ_QUESTION = f"Drill down the tree: read {READ_SP} — read the file"
WIDE_QUESTION = f"Drill down the tree: ls {ALPHA}/wide — how many files does this folder hold?"
NOPE_QUESTION = f"Drill down the tree: ls {ALPHA}/nope — is there such a folder?"
#: The read target is a top-2 retrieval document for its question (the
#: question names the path — the file self-matches the hybrid gate
#: deterministically), so the read gets the phase-72 ALREADY_IN_CONTEXT
#: dedupe and the mock answers from the ``<documents>`` prompt with the
#: same citation shape, prefixed (the suite pins the dedupe path).
READ_ANSWER_PREFIX = f"Already in context: Read {READ_SP}."
READ_ANSWER_QUOTE = TWO_A_CONTENT[:80]
# --------------------------------------------------------------------------
# Fixtures
# --------------------------------------------------------------------------
@pytest.fixture(scope="module")
def drill_dirs(tmp_path_factory: pytest.TempPathFactory) -> tuple[Path, Path]:
"""The two-source temp tree (see the module docstring): the app
server runs on the same host, so the paths are visible to it. The
directory NAMES are the source names (``kind=local`` → the
directory's basename, phase 38)."""
root = tmp_path_factory.mktemp("bor_ls_tree")
alpha = root / ALPHA
beta = root / BETA
(alpha / "one").mkdir(parents=True)
(alpha / "two").mkdir(parents=True)
(alpha / "wide").mkdir(parents=True)
(beta / "gamma").mkdir(parents=True)
(alpha / ROOT_NOTE).write_text(
_md(
"Alpha Root Note",
f"{DRILL_LEAD} at the alpha source root: this file sits "
"directly under the alpha source, not in any folder.",
),
encoding="utf-8",
)
(alpha / "one" / "one-a.md").write_text(
_md(
"Alpha One A",
f"{DRILL_LEAD} for alpha one: this document covers topic A "
"of the alpha source tree.",
),
encoding="utf-8",
)
(alpha / "one" / "one-b.md").write_text(
_md(
"Alpha One B",
f"{DRILL_LEAD} for alpha one: this document covers topic B "
"of the alpha source tree.",
),
encoding="utf-8",
)
(alpha / TWO_A).write_text(TWO_A_CONTENT, encoding="utf-8")
(alpha / TWO_B).write_text(
_md(
"Alpha Two B",
f"{DRILL_LEAD} for alpha two: this document covers topic B "
"of the alpha source tree.",
),
encoding="utf-8",
)
for nn in range(1, WIDE_COUNT + 1):
(alpha / "wide" / f"wide-{nn:02d}.md").write_text(
_md(
f"Wide {nn:02d}",
f"One of {WIDE_COUNT} tiny files in the alpha wide "
f"folder: {DRILL_LEAD.lower()} line {nn:02d}.",
),
encoding="utf-8",
)
(beta / "gamma" / "gamma-a.md").write_text(
_md(
"Beta Gamma A",
f"{DRILL_LEAD} for beta gamma: this document covers topic A "
"of the beta source tree.",
),
encoding="utf-8",
)
(beta / "gamma" / "gamma-b.md").write_text(
_md(
"Beta Gamma B",
f"{DRILL_LEAD} for beta gamma: this document covers topic B "
"of the beta source tree.",
),
encoding="utf-8",
)
assert (alpha / TWO_A).is_file() and (beta / "gamma" / "gamma-b.md").is_file()
return alpha, beta
@pytest.fixture(scope="module")
def app_server(mock_llm: int, drill_dirs: tuple[Path, Path]) -> Iterator[str]:
"""The real app under test — per-module app (the conftest pattern,
cf. ``test_local_directory_sources.py``): NO ``BOR_GIT_SOURCES``
(the env fallback is git-only — the sources here are DB-registered
local directories), the mock LLM, the mock-calibrated threshold,
and the leak-guarded code defaults. The session app is never
started in this isolated run, so no port clash."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern): every scripted
# question FTS-matches the fixture docs (the ``drill down the tree``
# words), so the gate is HIGH either way.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
# Phase 67: instant retry waits + the code-default budget (the
# conftest leak-guard pattern).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the two
# local directories this suite registers (a leftover env git list
# would pollute the top-level ``ls`` the whole story asserts on).
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under test.
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
share one Postgres, so a leftover git_sources row would pollute the
top-level ``ls`` and a leftover document would show up in the
folder listings the drill answers assert on byte-exactly."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, folder_summaries"
)
)
db.commit()
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
"""Poll the (cookie-authenticated) status endpoint until the run
reaches a terminal state (the test_local_directory_sources pattern,
over plain httpx — this fixture has no browser page yet)."""
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
if body["state"] in ("success", "failed"):
return body
time.sleep(0.5)
raise AssertionError(f"sync did not reach a terminal state: {body}")
@pytest.fixture(scope="module")
def synced_kb(app_server: str, drill_dirs: tuple[Path, Path]) -> None:
"""The story's precondition: the folder-structured KB synced under
the deterministic mock.
Registers the two temp directories through the authenticated API
(the ``test_local_directory_sources.py`` pattern — ``alpha`` FIRST,
committed separately, so the registry order — ``(added_at, id)`` —
lists alpha before beta, the top-level ``ls`` order the suite
asserts), runs the REAL in-process sync (``POST /api/sync`` —
walk → chunk → embed → overview → folder summaries → version bump),
and pins the stored folder summaries: the mock's canned
``FOLDER_SUMMARY_MODE`` branch (task 01) makes the sync store one
deterministic row per ≥ 2-doc folder — the drill turns' answers
assert on that exact text.
"""
alpha, beta = drill_dirs
_truncate_all()
with httpx.Client(base_url=app_server, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post(
"/api/git-sources", json={"kind": "local", "path": str(alpha)}
)
assert r.status_code == 201, r.text
time.sleep(0.05) # distinct added_at: alpha before beta (registry order)
r = client.post(
"/api/git-sources", json={"kind": "local", "path": str(beta)}
)
assert r.status_code == 201, r.text
r = client.post("/api/sync")
assert r.status_code == 202, r.text
body = _wait_sync_done_http(client)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == TOTAL_DOCS, detail
assert detail["pruned"] == 0, detail
assert detail["overview"] is True, detail
# The change-gated folder summaries (phase 94 task 02) landed: one
# row per ≥ 2-doc folder, the mock's byte-stable text (the drill
# answers quote exactly these lines).
with SessionLocal() as db:
rows = db.execute(
select(
FolderSummary.source, FolderSummary.folder_path,
FolderSummary.summary,
).order_by(FolderSummary.source, FolderSummary.folder_path)
).all()
assert [(s, f, t) for s, f, t in rows] == EXPECTED_SUMMARIES, rows
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
"""Per-test query_log isolation (the KB itself is module-scoped —
the drill turns never change it, so the folder summaries and the
registry persist across the tests of this module)."""
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
yield
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
# --------------------------------------------------------------------------
# Page helpers (the test_agent_document_tools house pattern)
# --------------------------------------------------------------------------
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
#: (a response clone read in the background) — wire-level assertions
#: for the ``tool`` frames, independent of the UI rendering.
SSE_HOOK = """
() => {
if (window.__sseInstalled) return;
window.__sseInstalled = true;
window.__sseFrames = [];
const origFetch = window.fetch;
window.fetch = async function (...args) {
const res = await origFetch.apply(this, args);
try {
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
if (url.includes('/api/chat')) {
res.clone().text().then((bodyText) => {
for (const block of bodyText.split('\\n\\n')) {
const line = block.trim();
if (line.startsWith('data: ')) {
window.__sseFrames.push(line.slice(6));
}
}
});
}
} catch (e) { /* non-clonable responses: ignored */ }
return res;
};
}
"""
def _install_page_hooks(page: Page) -> None:
page.evaluate(SSE_HOOK)
def _frames(page: Page) -> list[dict]:
"""The SSE frames captured since the last submit (``_submit``
clears the buffer), once the hook's background read settles."""
deadline = time.monotonic() + 30.0
while True:
raw = page.evaluate("() => window.__sseFrames || []")
parsed = [json.loads(line) for line in raw if line]
if any(f.get("type") == "done" for f in parsed):
return parsed
if time.monotonic() > deadline:
raise AssertionError(
f"SSE hook captured no `done` frame (frames so far: "
f"{len(parsed)}) — hook install failed?"
)
time.sleep(0.05)
def _tool_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool"]
def _submit(page: Page, question: str) -> None:
page.evaluate("window.__sseFrames = []")
page.fill("#message-input", question)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
def _wait_settled(page: Page) -> None:
"""The turn is complete: answer text in the bubble, button recovered
(the phase-48 settle wait, the test_agent_document_tools helper)."""
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
def _last_brain(page: Page) -> Locator:
return page.locator(".msg.brain").last
def _assert_turn(
page: Page,
expected_tools: list[dict[str, Any]],
expected_answer_lines: list[str],
) -> None:
"""One scripted drill turn, fully asserted: the wire carries exactly
the expected ``tool`` frames (ahead of the first ``delta``), the
bubble carries the expected answer lines (the mock's echo of the
tool results the model received), and the turn was grounded (the
``done`` frame is not deflected)."""
frames = _frames(page)
assert _tool_frames(frames) == expected_tools, _tool_frames(frames)
if expected_tools:
first_delta = next(
i for i, f in enumerate(frames) if f.get("type") == "delta"
)
assert all(
i < first_delta
for i, f in enumerate(frames)
if f.get("type") == "tool"
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
bubble = _last_brain(page).locator(".bubble")
for line in expected_answer_lines:
expect(bubble).to_contain_text(line)
def _query_log_rows() -> list[QueryLog]:
with SessionLocal() as db:
return list(db.scalars(select(QueryLog)).all())
# --------------------------------------------------------------------------
# 1. The scripted drill: sources → folders (summaries) → files → read
# --------------------------------------------------------------------------
def test_drill_down_sources_folders_files_and_read(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
# --- turn 1: ls() — the top level (sources + source-root summaries) -
_submit(page, TOP_QUESTION)
_wait_settled(page)
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(1)
# The no-arg ls line — NOT a scoped "Listing documents in …" one
# (regex match: string expectations normalize whitespace, so the
# scope check must be a byte-exact pattern).
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(0)).not_to_have_text(re.compile(r"Listing documents in"))
_assert_turn(
page,
[{"type": "tool", "name": "ls", "argument": None}],
[TOP_HEADER, *TOP_LINES],
)
# --- turn 2: ls(alpha) — the source root (subfolders + root files) --
_submit(page, SOURCE_QUESTION)
_wait_settled(page)
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(1)
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}")
_assert_turn(
page,
[{"type": "tool", "name": "ls", "argument": ALPHA}],
[SOURCE_HEADER, *SOURCE_LINES],
)
# --- turn 3: ls(alpha/two) — the folder level (the file lines) ------
_submit(page, FOLDER_QUESTION)
_wait_settled(page)
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(1)
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}/two")
_assert_turn(
page,
[{"type": "tool", "name": "ls", "argument": f"{ALPHA}/two"}],
[FOLDER_HEADER, *FOLDER_LINES],
)
# --- turn 4: read(alpha/two/two-a.md) — the grounded read ------------
_submit(page, READ_QUESTION)
_wait_settled(page)
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(1)
# The phase-37 "Reading <source/path>" line (the combined identity).
expect(lines.nth(0)).to_contain_text(f"Reading {READ_SP}")
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "read", "argument": READ_SP}
], _tool_frames(frames)
bubble = _last_brain(page).locator(".bubble")
# The dedupe notice reached the model (the prefix pins it — the read
# was refused as ALREADY_IN_CONTEXT because the target is a top-2
# retrieval document)…
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
# …and the answer still cites the document: the mock quotes the
# FIRST 80 chars of the target's text from the ``<documents>``
# prompt (the refusal's instruction — answer from that text; the
# quote is newline-free, pinned above).
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The read document is in the turn's sources (retrieval + agent-read,
# deduped — the grounded-turn record).
assert any(
s["path"] == TWO_A and s["source"] == ALPHA for s in done["sources"]
), done["sources"]
# Durable records: all four turns grounded, in order, the read turn
# logging the read document.
rows = _query_log_rows()
assert [r.question for r in rows] == [
TOP_QUESTION, SOURCE_QUESTION, FOLDER_QUESTION, READ_QUESTION
]
assert all(r.deflected is False for r in rows)
assert READ_SP in rows[3].sources, rows[3].sources
# --------------------------------------------------------------------------
# 2. The 50-line cap: a 51-file folder costs the model 50 lines + the note
# --------------------------------------------------------------------------
def test_wide_folder_holds_the_fifty_line_cap(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, WIDE_QUESTION)
_wait_settled(page)
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(1)
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}/wide")
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": f"{ALPHA}/wide"}
], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The mock echoed the listing VERBATIM: the header carries the
# PRE-cap count (51 — the cap hides lines, not the truth), the
# file lines stop at 50, and the one deterministic grep-pointer note
# folds the 51st file away.
bubble = _last_brain(page).locator(".bubble")
text = bubble.text_content() or ""
assert WIDE_HEADER in text, text
assert WIDE_FIRST in text, text
assert WIDE_LAST in text, text
assert WIDE_NOTE in text, text
assert f"wide/wide-{WIDE_COUNT}.md" not in text, text # the 51st file: gone
# Exactly LS_MAX_FILE_LINES file lines reached the model.
assert text.count(f"source: {ALPHA} | path: wide/") == LS_MAX_FILE_LINES, text
row = _query_log_rows()
assert len(row) == 1
assert row[0].deflected is False
# --------------------------------------------------------------------------
# 3. The NOT-A-FOLDER teaching is visible to the model; the scripted
# recovery (ls of the parent level) works
# --------------------------------------------------------------------------
def test_not_a_folder_teaching_and_scripted_recovery(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, NOPE_QUESTION)
_wait_settled(page)
# Two tool lines: the scripted misuse, then the scripted recovery —
# the mock's recovery branch fires ONLY when it RECEIVES the
# NOT-A-FOLDER teaching line (``'alpha/nope' is not a folder —
# alpha has: one/ two/ wide/`` — the argument echoed, the parent's
# subfolders listed): the teaching being visible to the model is
# exactly what the second call proves (the phase-72
# self-correction contract, now carrying the tree's teaching).
lines = _last_brain(page).locator(".tool-call")
expect(lines).to_have_count(2)
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}/nope")
expect(lines.nth(1)).to_contain_text(f"Listing documents in {ALPHA}")
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": f"{ALPHA}/nope"},
{"type": "tool", "name": "ls", "argument": ALPHA},
], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The loop ended in ONE refusal + ONE correction: the final answer is
# the recovery's PARENT listing (not the round-cap, not an echo of
# the refusal) — the model self-corrected and got the tree level.
bubble = _last_brain(page).locator(".bubble")
for line in [SOURCE_HEADER, *SOURCE_LINES]:
expect(bubble).to_contain_text(line)
row = _query_log_rows()
assert len(row) == 1
assert row[0].deflected is False
tests/e2e/test_tool_path_teaching.py
+60 -32
View File
@@ -13,11 +13,14 @@ Run in isolation (DB must be up: ``podman compose up -d db``):
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic LS-TEACH flow in ``tests/e2e/mock_llm.py``
(``LS_TEACH_TRIGGER`` — "list the files in this directory" — + the
HIGH prompt's ``<tools>`` section): the incident's misuse (``ls`` with
``{"path": "."}``, id ``call_0``) → the agent's teaching refusal
(``No source named '.' — check the ls output. (…)``) → the corrected
no-arg ``ls()`` (id ``call_1``) → the deterministic
``These are the indexed documents: <first catalog line>`` answer.
HIGH prompt's ``<tools>`` section; phase 94: the drill-down ``ls`` —
the corrected no-arg listing carries sources only, so the flow drills
one level before the first file line exists): the incident's misuse
(``ls`` with ``{"path": "."}``, id ``call_0``) → the agent's teaching
refusal (``No source named '.' — check the ls output. (…)``) → the
corrected no-arg ``ls()`` (id ``call_1``) → the drill ``ls`` scoped to
the first source of the listing (id ``call_2``) → the deterministic
``These are the indexed documents: <first file line>`` answer.
KB fixture (TRUNCATE-then-seed, house pattern): ONE source with TWO
documents of known ``source``/``path``/``title`` (catalog order =
@@ -39,21 +42,24 @@ documents of known ``source``/``path``/``title`` (catalog order =
Test → phase mapping (Playwright Mapping Rule):
1. ``test_ls_misuse_self_corrects_to_noarg_listing`` — the grounded
LS-TEACH turn: the turn settles (composer re-enables, ``done``
observed), the answer bubble carries the first catalog line — the
observed), the answer bubble carries the first file line — the
first document's ``source:`` / ``path:`` / title fields (the
catalog reached the model and landed in the answer), the UI shows
the two tool lines (``🔎 Listing documents in <code>.</code>``
then ``🔎 Listing documents``), and no error banner. Wire level:
the ``tool`` frames arrive in order — first ``ls`` with
``argument: "."``, then ``ls`` with ``argument: null`` — and there
is NO third ``tool`` frame (the loop ended in one correction, not
folder listing reached the model and landed in the answer), the UI
shows the three tool lines (``🔎 Listing documents in
<code>.</code>``, ``🔎 Listing documents``, then the drill
``🔎 Listing documents in <source>`` — phase 94), and no error
banner. Wire level: the ``tool`` frames arrive in order — first
``ls`` with ``argument: "."``, then ``ls`` with ``argument: null``,
then the drill ``ls`` scoped to the source — and there is NO fourth
``tool`` frame (the loop ended in one correction + one drill, not
at the round cap).
2. ``test_plain_tool_flow_not_swallowed_by_new_trigger`` — in the SAME
session, the LS-TEACH turn settles and a follow-up question
carrying ``TOOLS_TRIGGER`` (the single-read flow) still settles
with the read flow's answer (``ls`` → ``read`` on the first
catalog line's combined identity → ``Read <source/path>. <quote>``)
— the new flow did not swallow the existing trigger.
with the read flow's answer (``ls`` → the drill ``ls`` (phase 94)
→ ``read`` on the first file line's combined identity →
``Read <source/path>. <quote>``) — the new flow did not swallow
the existing trigger.
"""
from __future__ import annotations
@@ -69,7 +75,7 @@ from sqlalchemy import text
from sqlalchemy.orm import Session
from app.db import SessionLocal
from app.models import Chunk, Document
from app.models import Chunk, Document, GitSource
from e2e.auth_helpers import login
from tests.e2e.mock_llm import (
LS_TEACH_TRIGGER,
@@ -211,7 +217,14 @@ def _seed_fixture(db: Session) -> None:
cosines well past the E2E 0.30 threshold and FTS-matches too →
grounded). DOC2 is the seed context only — the single-read flow
reads the catalog-FIRST document (DOC1), which is not in context.
Phase 94: the drill-down ``ls`` top level reads the registry —
register the source (TRUNCATEd in ``_reset_db_fixture``): the
corrected no-arg listing names it, and the drill scopes to it. A
non-empty table also ignores the operator's ``BOR_GIT_SOURCES``
fallback — deterministic.
"""
db.add(GitSource(url=SEED_SOURCE, kind="local"))
db.add(
Document(
source=SEED_SOURCE,
@@ -255,7 +268,10 @@ def _reset_db_fixture() -> None:
prompts, byte-stable answers."""
with SessionLocal() as db:
db.execute(
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources"
)
)
db.commit()
_seed_fixture(db)
@@ -382,24 +398,30 @@ def test_ls_misuse_self_corrects_to_noarg_listing(
expect(bubble).to_contain_text(FIRST_CATALOG_LINE)
_assert_no_error_banner(page)
# The UI shows the two tool lines in order: the scoped misuse
# (🔎 Listing documents in <code>.</code>) then the corrected
# unscoped listing (🔎 Listing documents — no <code>).
# The UI shows the three tool lines in order: the scoped misuse
# (🔎 Listing documents in <code>.</code>), the corrected
# unscoped listing (🔎 Listing documents — no <code>), then the
# drill (🔎 Listing documents in <source> — phase 94, the top
# level lists sources only, so the file lines need one more level).
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents in")
expect(lines.nth(0).locator("code")).to_have_text(".")
expect(lines.nth(1)).to_contain_text("Listing documents")
expect(lines.nth(1).locator("code")).to_have_count(0)
expect(lines.nth(2)).to_contain_text("Listing documents in")
expect(lines.nth(2).locator("code")).to_have_text(SEED_SOURCE)
# Two rounds on the wire: the tool frames arrive in order — first
# ls with argument "." (the incident's misuse), then ls with
# argument null (the correction) — and there is NO third tool
# frame: the loop ended in one correction, not at the round cap.
# Three rounds on the wire: the tool frames arrive in order —
# first ls with argument "." (the incident's misuse), then ls with
# argument null (the correction), then the drill ls scoped to the
# source (phase 94) — and there is NO fourth tool frame: the loop
# ended in one correction + one drill, not at the round cap.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": "."},
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
]
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
@@ -433,6 +455,7 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
assert _tool_frames(teach_frames) == [
{"type": "tool", "name": "ls", "argument": "."},
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
]
expect(
page.locator(".msg.brain .bubble").last
@@ -445,14 +468,17 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
_wait_settled(page)
second_msg = page.locator(".msg.brain").last
# The UI shows the single-read flow's two lines: the unscoped ls
# then the read of the first catalog line's COMBINED identity.
# The UI shows the single-read flow's three lines: the unscoped ls,
# the drill ls (phase 94), then the read of the first file line's
# COMBINED identity.
lines = second_msg.locator(".tool-call")
expect(lines).to_have_count(2)
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
expect(lines.nth(0).locator("code")).to_have_count(0)
expect(lines.nth(1)).to_contain_text("Reading ")
expect(lines.nth(1).locator("code")).to_have_text(DOC1_SP)
expect(lines.nth(1)).to_contain_text("Listing documents in")
expect(lines.nth(1).locator("code")).to_have_text(SEED_SOURCE)
expect(lines.nth(2)).to_contain_text("Reading ")
expect(lines.nth(2).locator("code")).to_have_text(DOC1_SP)
# The answer quotes the read document (the mock's deterministic
# echo: "Read <source/path>. <first 80 chars>").
@@ -461,11 +487,13 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
_assert_no_error_banner(page)
# Wire level for the follow-up: ls (null) → read (the combined
# identity) — the single-read flow, unchanged.
# Wire level for the follow-up: ls (null) → the drill ls (the
# source, phase 94) → read (the combined identity) — the
# single-read flow, grown by the drill step.
frames = _drain_frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
{"type": "tool", "name": "read", "argument": DOC1_SP},
]
done = next(f for f in frames if f.get("type") == "done")
tests/fixtures/test_kb.dump.sql
Vendored
+45 -40
View File
File diff suppressed because one or more lines are too long
tests/integration/test_agent_tools.py
+218 -38
View File
@@ -1,19 +1,28 @@
"""Integration: the agent DB accessors against real Postgres (phase 37;
the harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
the harness-aligned ``ls``/``read``/``grep`` surface, phase 70; the
drill-down tree ``ls``, phase 94).
``list_catalog`` must order rows by ``(source, path)`` — the same order
as ``GET /api/docs`` — ``list_source_names`` must resolve the
registered source names (the scoped ``ls`` join), and ``find_document``
``_source_document_rows`` must order a source's rows by ``path`` (the
file lines' catalog order), ``list_source_names`` must resolve the
registered source names (the registry join), and ``find_document``
must resolve a hit to the full document row (content included, for the
never-truncated read) and return ``None`` for unknown pairs. Phase 70:
the ``ls``/``read``/``grep`` tools are pinned here too — the locked
parameter shape in ``AGENT_TOOLS``, and scripted ``ToolCallPiece``s
executed through ``run_agent`` against the real DB: ``ls`` scoped to a
registered source name (unknown name → refusal), ``read`` on the
canonical combined ``source/path`` form (first-slash split; a bare
source name and an unknown identity get the no-document refusal), and
``grep`` (``all_documents`` for a whole-KB search, ``find_document`` for
a scoped one).
executed through ``run_agent`` against the real DB. Phase 94: the
drill-down ``ls`` against the REAL tables — ``ls()`` lists the
registered sources (registry order, recursive counts, stored
source-root summaries from ``folder_summaries``), ``ls(source)`` /
``ls(source/folder)`` list one folder level (the SQL prefix logic:
subfolders = slash-boundary prefixes, counts = the recursive subtree,
file lines in catalog order, capped at 50 + the grep-pointer note),
and the refusals (unknown source segment → the no-source refusal; an
unknown folder → NOT-A_FOLDER with the parent's subfolders).
``read`` runs on the canonical combined ``source/path`` form
(first-slash split; a bare source name and an unknown identity get the
no-document refusal), and ``grep`` (``all_documents`` for a whole-KB
search, ``find_document`` for a scoped one) — both byte-identical
across the phase-94 change.
Requires: podman compose up -d db
"""
@@ -30,7 +39,7 @@ from sqlalchemy import delete, text
from sqlalchemy.orm import Session
from app.config import Settings
from app.models import Document, GitSource
from app.models import Document, FolderSummary, GitSource
from app.rag import agent
from app.rag.agent import AGENT_TOOLS, AgentHolder, run_agent
from app.rag.llm import LLMClient, RetryPiece, StreamPiece, ToolCallPiece
@@ -55,11 +64,30 @@ def _doc(db: Session, source: str, path: str, title: str, content: str) -> Docum
@pytest.fixture()
def kb(db) -> Iterator[None]:
"""Fresh documents table (chunks first — the FK) for these accessors."""
db.execute(text("TRUNCATE chunks, documents"))
"""Fresh documents + folder_summaries tables (chunks first — the FK)
for these accessors (phase 94: the drill-down ``ls`` reads the
stored summaries too)."""
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
db.commit()
yield
db.execute(text("TRUNCATE chunks, documents"))
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
db.commit()
@pytest.fixture()
def registry(db) -> Iterator[None]:
"""A FRESH two-source registry (phase 94): the drill-down ``ls``
top level IS the registry, so the table is truncated and re-seeded
around the tests in a controlled ``(added_at, id)`` order —
``Deployments`` before ``Homelab`` (the top-level listing order)."""
db.execute(text("TRUNCATE git_sources"))
db.commit()
db.add(GitSource(url="https://github.com/reese/Deployments.git", kind="git"))
db.commit()
db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git"))
db.commit()
yield
db.execute(text("TRUNCATE git_sources"))
db.commit()
@@ -76,21 +104,24 @@ def src(db) -> Iterator[GitSource]:
db.commit()
def test_list_catalog_orders_by_source_then_path(kb, db) -> None:
def test_source_document_rows_order_by_path_within_the_source(kb, db) -> None:
"""Phase 94: the file lines' order — the source's rows in ``path``
order (the old ``list_catalog``'s per-source ordering, now the
``ls`` folder-level accessor's contract; a different source's rows
never leak in)."""
_doc(db, "Zeta", "b/second.md", "Zeta B", "ZB")
_doc(db, "Zeta", "a/first.md", "Zeta A", "ZA")
_doc(db, "Alpha", "c/third.md", "Alpha C", "AC")
db.commit()
assert agent.list_catalog(db) == [
("Alpha", "c/third.md", "Alpha C"),
("Zeta", "a/first.md", "Zeta A"),
("Zeta", "b/second.md", "Zeta B"),
assert agent._source_document_rows(db, "Zeta") == [
("a/first.md", "Zeta A"),
("b/second.md", "Zeta B"),
]
def test_list_catalog_is_empty_without_rows(kb, db) -> None:
assert agent.list_catalog(db) == []
def test_source_document_rows_is_empty_without_rows(kb, db) -> None:
assert agent._source_document_rows(db, "Zeta") == []
def test_list_source_names_resolves_registry_rows(db) -> None:
@@ -238,30 +269,156 @@ async def _consume(
return out
# ---------- ls (scoped through the real registry) ----------
# ---------- ls (the drill-down tree, phase 94 — the real registry + DB) ----------
def test_ls_scoped_to_registered_source_through_run_agent(kb, src, db) -> None:
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
_doc(db, "Other", "b.md", "B", "B-CONTENT")
def test_ls_top_level_lists_registered_sources_through_run_agent(
kb, registry, db
) -> None:
"""No path: the TOP level against the real tables — registry order
(``(added_at, id)`` — Deployments before Homelab), recursive counts
(all of a source's documents), the stored source-root summary
(``folder_path = ''``) shown only when stored."""
_doc(db, "Deployments", "a/one.md", "A1", "A1-CONTENT")
_doc(db, "Homelab", "x.md", "X", "X-CONTENT")
_doc(db, "Homelab", "y/z.md", "Z", "Z-CONTENT")
db.add(FolderSummary(source="Homelab", folder_path="", summary="Home lab notes."))
db.commit()
holder, llm = _run_call(db, "ls", {"path": "Homelab"})
holder, llm = _run_call(db, "ls", {})
# Offered: the first request carries AGENT_TOOLS (the 3-tool list).
assert llm.requests[0][1] == AGENT_TOOLS
# Executed against the real DB: the listing filtered to the source.
assert llm.requests[1][0][3]["content"] == (
"1 documents:\nsource: Homelab | path: a.md | title: A"
"2 sources:\n"
"\n"
"Deployments — 1 documents\n"
"Homelab — 2 documents\n"
" Home lab notes."
)
assert holder.tool_calls == 1
assert holder.read_docs == []
def test_ls_top_level_empty_registry_through_run_agent(
kb, db, monkeypatch: pytest.MonkeyPatch
) -> None:
"""No registered sources: the top level is the header line alone
(``0 sources:`` — the old ``0 documents:`` behavior preserved in
spirit), still counted. The env fallback (``BOR_GIT_SOURCES`` — the
operator's ``.env`` may name sources) is emptied for the test, so
the registry is genuinely empty."""
import app.rag.git_sources as git_sources_mod
db.execute(text("TRUNCATE git_sources"))
db.commit()
monkeypatch.setattr(
git_sources_mod,
"get_settings",
lambda: _settings(git_sources=""),
)
_doc(db, "Orphan", "a.md", "A", "A-CONTENT") # indexed but unregistered
db.commit()
holder, llm = _run_call(db, "ls", {})
assert llm.requests[1][0][3]["content"] == "0 sources:"
assert holder.tool_calls == 1
def test_ls_source_scope_lists_root_folder_through_run_agent(kb, registry, db) -> None:
"""A registered source name: the source's ROOT folder — the direct
subfolders (path order, recursive counts, stored summaries attached)
+ the root's own file lines in catalog order — against the real
tables; a registered source with no documents lists its header
line alone."""
_doc(db, "Homelab", "backups/cron.md", "Cron", "CRON")
_doc(db, "Homelab", "backups/restic.md", "Restic", "RESTIC")
_doc(db, "Homelab", "networking/lan.md", "LAN", "LAN")
_doc(db, "Homelab", "readme.md", "Readme", "README")
db.add(
FolderSummary(
source="Homelab", folder_path="backups", summary="Backup notes."
)
)
db.commit()
holder, llm = _run_call(db, "ls", {"path": "Homelab"})
assert llm.requests[1][0][3]["content"] == (
"Homelab — 1 documents, 2 folders:\n"
"\n"
" backups/ — 2 documents: Backup notes.\n"
" networking/ — 1 documents\n"
"\n"
"source: Homelab | path: readme.md | title: Readme"
)
assert holder.tool_calls == 1
assert holder.read_docs == []
# A registered source with no documents: the header line alone.
holder0, llm0 = _run_call(db, "ls", {"path": "Deployments"})
assert llm0.requests[1][0][3]["content"] == "Deployments — 0 documents, 0 folders:"
assert holder0.tool_calls == 1
def test_ls_nested_folder_scope_drills_one_level_through_run_agent(
kb, registry, db
) -> None:
"""A ``source/folder`` path: that folder's subfolders + own file
lines (identity = ``source/folder``) — the drill-down against the
real tables."""
_doc(db, "Homelab", "networking/lan/a.md", "A", "A")
_doc(db, "Homelab", "networking/lan/b.md", "B", "B")
_doc(db, "Homelab", "networking/vpn/c.md", "C", "C")
db.commit()
holder, llm = _run_call(db, "ls", {"path": "Homelab/networking"})
assert llm.requests[1][0][3]["content"] == (
"Homelab/networking — 0 documents, 2 folders:\n"
"\n"
" networking/lan/ — 2 documents\n"
" networking/vpn/ — 1 documents"
)
assert holder.tool_calls == 1
assert holder.read_docs == []
# One level deeper.
holder2, llm2 = _run_call(db, "ls", {"path": "Homelab/networking/lan"})
assert llm2.requests[1][0][3]["content"] == (
"Homelab/networking/lan — 2 documents, 0 folders:\n"
"\n"
"source: Homelab | path: networking/lan/a.md | title: A\n"
"source: Homelab | path: networking/lan/b.md | title: B"
)
assert holder2.tool_calls == 1
def test_ls_folder_file_cap_through_run_agent(kb, registry, db) -> None:
"""The cap end-to-end: 51 direct files in one folder cost 50 file
lines + the deterministic grep-pointer note, never 51."""
for i in range(51):
_doc(db, "Homelab", f"big/f{i:03d}.md", f"T{i}", "BODY")
db.commit()
holder, llm = _run_call(db, "ls", {"path": "Homelab/big"})
content = llm.requests[1][0][3]["content"]
lines = content.splitlines()
assert lines[0] == "Homelab/big — 51 documents, 0 folders:"
assert lines[2] == "source: Homelab | path: big/f000.md | title: T0"
assert lines[51] == "source: Homelab | path: big/f049.md | title: T49"
assert lines[52] == (
"…and 1 more documents in this folder — use grep (pattern) to "
"find a specific one."
)
assert len(lines) == 53
assert holder.tool_calls == 1
def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None:
"""Phase 72: the no-source refusal now carries the teaching
parenthetical — the prefix byte-identical to the pre-phase-72 line;
still not counted."""
"""A ``path`` without ``/`` matching no source name is a refusal —
the extended line with the teaching parenthetical (phase 72, the
prefix byte-identical to the pre-phase-72 line); still not counted."""
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
db.commit()
@@ -274,11 +431,11 @@ def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None
assert holder.tool_calls == 0 and holder.read_docs == []
def test_ls_path_like_scope_teaching_refusal_through_run_agent(kb, src, db) -> None:
"""Phase 72: a ``/``-containing ``path`` is a document path, not a
source name — the ``LS_PATH_NOT_A_SOURCE`` teaching line (no
registry lookup needed), not counted, the tools stay offered on the
next request."""
def test_ls_path_like_scope_unknown_source_gets_no_source_refusal(kb, src, db) -> None:
"""Phase 94: a ``/`` now names a folder — the phase-72 document-path
teaching is DELETED; a ``source/…`` argument whose FIRST segment
names no registered source gets the no-source refusal (the segment
echoed), not counted, the tools stay offered."""
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
db.commit()
@@ -286,7 +443,30 @@ def test_ls_path_like_scope_teaching_refusal_through_run_agent(kb, src, db) -> N
assert (
llm.requests[1][0][3]["content"]
== agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="app")
)
assert holder.tool_calls == 0 and holder.read_docs == []
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_unknown_folder_gets_not_a_folder_with_parents_subfolders(
kb, src, db,
) -> None:
"""Phase 94: a folder segment matching no indexed prefix gets the
NOT-A_FOLDER teaching — the argument echoed, the source named, its
DIRECT subfolders listed (the self-correction list), not counted,
the tools stay offered."""
_doc(db, "Homelab", "backups/cron.md", "Cron", "CRON")
_doc(db, "Homelab", "containers/caddy.md", "Caddy", "CADDY")
_doc(db, "Homelab", "networking/lan.md", "LAN", "LAN")
_doc(db, "Homelab", "readme.md", "Readme", "README")
db.commit()
holder, llm = _run_call(db, "ls", {"path": "Homelab/netwoking"})
assert llm.requests[1][0][3]["content"] == (
"'Homelab/netwoking' is not a folder — Homelab has: "
"backups/ containers/ networking/"
)
assert holder.tool_calls == 0 and holder.read_docs == []
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
tests/integration/test_chat_api.py
+4 -1
View File
@@ -943,7 +943,10 @@ def test_tool_execution_db_failure_yields_error_event(
def boom(*_a: Any, **_k: Any) -> Any:
raise RuntimeError("db exploded mid tool call")
monkeypatch.setattr(agent, "list_catalog", boom)
# Phase 94: the no-arg ``ls`` executes through ``ls_top`` — the
# failure hook moves with the rewrite (same contract: the tool
# frame goes out first, the structured error ends the turn).
monkeypatch.setattr(agent, "ls_top", boom)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
_, _, frames = _stream_chat(client, QUESTION)
tests/integration/test_import_docs_git.py
+18
View File
@@ -104,6 +104,21 @@ def _stub_bump(monkeypatch: pytest.MonkeyPatch) -> list[None]:
return bumps
def _stub_folder_summaries(monkeypatch: pytest.MonkeyPatch) -> list[dict]:
"""Stub the phase-94 folder-summary regeneration (this file keeps
its no-real-DB / no-network style — the real generator would read
the global ``documents`` table and burn ``lite`` calls). Returns
the call record; the canned stats are the zero dict."""
calls: list[dict] = []
async def fake_generate(db: object, llm: object, *, skip: bool = False) -> dict[str, int]:
calls.append({"skip": skip})
return {"generated": 0, "failed": 0, "pruned": 0}
monkeypatch.setattr(import_docs, "generate_folder_summaries", fake_generate)
return calls
# --- repo_name -------------------------------------------------------------
@@ -293,6 +308,7 @@ def test_main_rows_branch_passes_ignore_map_to_import(
fake_import = FakeImportSources()
monkeypatch.setattr(import_docs, "import_sources", fake_import)
_stub_bump(monkeypatch)
_stub_folder_summaries(monkeypatch)
rc = import_docs.main([])
@@ -329,6 +345,7 @@ def test_main_git_sources_clone_then_import(
fake_import = FakeImportSources()
monkeypatch.setattr(import_docs, "import_sources", fake_import)
bumps = _stub_bump(monkeypatch)
_stub_folder_summaries(monkeypatch)
rc = import_docs.main([])
@@ -371,6 +388,7 @@ def test_main_cli_source_still_imports_manual_dir(
fake_import = FakeImportSources()
monkeypatch.setattr(import_docs, "import_sources", fake_import)
bumps = _stub_bump(monkeypatch)
_stub_folder_summaries(monkeypatch)
rc = import_docs.main(["--source", str(manual)])
tests/integration/test_import_docs_overview.py
+61 -20
View File
@@ -23,6 +23,15 @@ and unchanged re-runs never bump (``sources_version=skipped``), and a
failed ``lite`` never rolls the bump back. The counter is pinned to the
migration-0010 seed (0) around every test by
:func:`_reset_sources_version`.
Phase 94 (task 02, line-extension house rule): the summary line now
ends with the folder-summary stats —
``folder_summaries=<generated>/<failed>/<pruned>`` when the gate fired
(this fixture's 2-doc source holds exactly ONE qualifying subtree: the
source root) or ``folder_summaries=skipped`` otherwise — so the line
pinned here gains that token, and a KB-changing run burns exactly ONE
extra ``lite`` call (the source-root folder summary, markdown files
never get a document summary).
"""
from __future__ import annotations
@@ -98,10 +107,10 @@ def src(tmp_path: Path) -> Path:
@pytest.fixture(autouse=True)
def _clean_kb(db: Session) -> Iterator[None]:
db.execute(text("TRUNCATE chunks, documents, kb_overview"))
db.execute(text("TRUNCATE chunks, documents, kb_overview, folder_summaries"))
db.commit()
yield
db.execute(text("TRUNCATE chunks, documents, kb_overview"))
db.execute(text("TRUNCATE chunks, documents, kb_overview, folder_summaries"))
db.commit()
@@ -149,16 +158,24 @@ def test_changed_import_writes_overview_row(
assert rc == 0
assert "added=2" in out
assert out.rstrip().endswith("overview=updated sources_version=1")
# Phase 94: the line gains the folder-stats token — the 2-doc source
# holds one qualifying subtree (the source root): 1 generated.
assert out.rstrip().endswith(
"overview=updated sources_version=1 folder_summaries=1/0/0"
)
assert _version(db) == 1 # phase 53: a changed import bumps exactly once
# Exactly one lite call — the overview itself (markdown files never
# get a summary, so nothing else may touch ``chat``).
assert len(llm.chat_calls) == 1
# Exactly two lite calls — the overview + the source-root folder
# summary (markdown files never get a document summary, so nothing
# else may touch ``chat``).
assert len(llm.chat_calls) == 2
by_role = {m["role"]: m["content"] for m in llm.chat_calls[0]}
assert "KB_OVERVIEW_MODE" in by_role["system"]
# One line per doc: source — path — title (no summary for markdown).
assert "MyDocs — alpha.md — Alpha" in by_role["user"]
assert "MyDocs — beta.md — Beta" in by_role["user"]
by_role = {m["role"]: m["content"] for m in llm.chat_calls[1]}
assert "FOLDER_SUMMARY_MODE" in by_role["system"]
assert by_role["user"].splitlines()[0] == "Folder: MyDocs"
# The model's outline lands in the single row.
row = _row(db)
assert row is not None
@@ -177,16 +194,20 @@ def test_unchanged_reimport_does_not_call_lite(
llm = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
assert rc == 0
assert out.rstrip().endswith("overview=updated sources_version=1")
assert len(llm.chat_calls) == 1
assert out.rstrip().endswith(
"overview=updated sources_version=1 folder_summaries=1/0/0"
)
assert len(llm.chat_calls) == 2 # overview + source-root folder summary
assert _row(db) is not None
# Same hashes → no KB change → no lite call, previous outline kept.
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
assert rc == 0
assert "unchanged=2" in out
assert out.rstrip().endswith("overview=skipped sources_version=skipped")
assert len(llm.chat_calls) == 1 # no new lite call
assert out.rstrip().endswith(
"overview=skipped sources_version=skipped folder_summaries=skipped"
)
assert len(llm.chat_calls) == 2 # no new lite call
row = _row(db)
assert row is not None and row.content == "Summary of MyDocs"
assert _version(db) == 1 # phase 53: an unchanged re-run never bumps
@@ -201,7 +222,9 @@ def test_lite_failure_is_fail_soft(
good = FakeEmbedder()
rc, out = _run_main(monkeypatch, good, ["--source", str(src)], capsys)
assert rc == 0
assert out.rstrip().endswith("overview=updated sources_version=1")
assert out.rstrip().endswith(
"overview=updated sources_version=1 folder_summaries=1/0/0"
)
previous = _row(db)
assert previous is not None
previous_content = previous.content
@@ -213,8 +236,12 @@ def test_lite_failure_is_fail_soft(
rc, out = _run_main(monkeypatch, bad, ["--source", str(src)], capsys)
assert rc == 0 # a failed outline must not fail the import
assert "updated=1" in out
assert out.rstrip().endswith("overview=failed sources_version=2")
assert len(bad.chat_calls) == 1 # the (failed) attempt was made
# Phase 94: the folder batch fails too (per-folder fail-soft) — the
# failed attempt counts into the stats, the previous row stays.
assert out.rstrip().endswith(
"overview=failed sources_version=2 folder_summaries=0/1/0"
)
assert len(bad.chat_calls) == 2 # the (failed) attempts were made
row = _row(db)
assert row is not None
assert row.content == previous_content # previous row untouched
@@ -232,8 +259,10 @@ def test_limit_run_skips_overview(
llm = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
assert rc == 0
assert out.rstrip().endswith("overview=updated sources_version=1")
assert len(llm.chat_calls) == 1
assert out.rstrip().endswith(
"overview=updated sources_version=1 folder_summaries=1/0/0"
)
assert len(llm.chat_calls) == 2
# An incomplete walk must not rewrite the outline (mirrors the
# --prune-with---limit guard) — and must not advance the version.
@@ -241,8 +270,10 @@ def test_limit_run_skips_overview(
rc, out = _run_main(monkeypatch, llm, ["--source", str(src), "--limit", "1"], capsys)
assert rc == 0
assert "updated=1" in out
assert out.rstrip().endswith("overview=skipped sources_version=skipped")
assert len(llm.chat_calls) == 1 # --limit never burns a lite call
assert out.rstrip().endswith(
"overview=skipped sources_version=skipped folder_summaries=skipped"
)
assert len(llm.chat_calls) == 2 # --limit never burns a lite call
row = _row(db)
assert row is not None and row.content == "Summary of MyDocs"
assert _version(db) == 1 # phase 53: --limit debug runs never bump
@@ -262,7 +293,9 @@ def test_empty_source_without_row_creates_nothing(
assert rc == 0
assert "files=0" in out
assert out.rstrip().endswith("overview=skipped sources_version=skipped")
assert out.rstrip().endswith(
"overview=skipped sources_version=skipped folder_summaries=skipped"
)
assert llm.chat_calls == [] # no KB → no outline, no wasted model call
assert _row(db) is None # nothing created
assert _version(db) == 0 # nothing changed → nothing bumped
@@ -283,7 +316,9 @@ def test_prune_only_run_bumps_sources_version(
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
assert rc == 0
assert "added=2" in out
assert out.rstrip().endswith("overview=updated sources_version=1")
assert out.rstrip().endswith(
"overview=updated sources_version=1 folder_summaries=1/0/0"
)
assert _version(db) == 1
# Delete one file; a --prune run drops exactly it: no add/update,
@@ -292,6 +327,12 @@ def test_prune_only_run_bumps_sources_version(
rc, out = _run_main(monkeypatch, llm, ["--source", str(src), "--prune"], capsys)
assert rc == 0
assert "pruned=1" in out
assert out.rstrip().endswith("overview=skipped sources_version=2")
# Phase 94: the folder gate is the overview's (added + updated > 0
# or empty table) — a prune-only re-walk with a populated table
# skips generation (the remaining 1-doc source stays summarized by
# its existing root row, which still describes it).
assert out.rstrip().endswith(
"overview=skipped sources_version=2 folder_summaries=skipped"
)
assert _version(db) == 2 # the prune-only change bumped exactly once
assert _row(db) is not None # the outline row is untouched
tests/integration/test_migration_0017.py
+248
View File
@@ -0,0 +1,248 @@
"""Integration: migration 0017 (folder_summaries) schema contract
(phase 94, task 01).
Drives the **real Alembic engine** against the live dev database
(``podman compose up -d db``), mirroring the house pattern of
``test_migration_0016.py`` (information_schema assertions on the state
the migration must leave). The tests target the 0016 → 0017 step
explicitly so later migrations cannot break them:
* upgrade 0016 → 0017 → the ``folder_summaries`` table exists with the
full column contract — PK ``(source, folder_path)`` (VARCHAR(120) /
VARCHAR(1000) NOT NULL, mirroring ``documents.source`` /
``documents.path``), ``summary`` TEXT NOT NULL, ``updated_at``
TIMESTAMPTZ NOT NULL with the now() server default (house style) —
while the 0016 ``ui_settings`` schema survives;
* inserted rows round-trip their values (a source-root row with
``folder_path = ''`` and a nested-folder row);
* downgrade to 0016 → the table is GONE (A13 — reversible), the rest of
the schema (``ui_settings`` + ``api_tokens``) survives;
* upgrade back to 0017 → the table is back (round-trip).
The ``alembic`` fixture guarantees the DB ends at head even if a test
fails or the process is interrupted.
"""
from __future__ import annotations
from collections.abc import Iterator
from typing import Any
import pytest
from alembic.config import Config
from sqlalchemy import text
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from alembic import command
from app.db import db_available
@pytest.fixture()
def alembic(db: Session) -> Iterator[Config]:
"""Real Alembic config bound to the dev DB (URL from app settings).
Starts at head (repairs an interrupted earlier run); teardown upgrades
to head no matter what happened, so the dev DB is never left below
head.
"""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
cfg.set_main_option("script_location", "alembic")
command.upgrade(cfg, "head")
try:
yield cfg
finally:
command.upgrade(cfg, "head")
def _version(db: Session) -> str | None:
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
def _table_exists(db: Session, table: str) -> bool:
return (
db.execute(
text("SELECT 1 FROM information_schema.tables WHERE table_name = :t"),
{"t": table},
).scalar()
is not None
)
def _column(db: Session, table: str, column: str) -> tuple[Any, ...] | None:
"""(data_type, is_nullable, column_default, character_maximum_length)
for one table column."""
row = db.execute(
text(
"SELECT data_type, is_nullable, column_default, character_maximum_length"
" FROM information_schema.columns"
" WHERE table_name = :t AND column_name = :c"
),
{"t": table, "c": column},
).fetchone()
return tuple(row) if row is not None else None
def _pk_columns(db: Session, table: str) -> list[str]:
"""The table's PRIMARY KEY columns in ordinal position."""
rows = db.execute(
text(
"SELECT kcu.column_name"
" FROM information_schema.table_constraints tc"
" JOIN information_schema.key_column_usage kcu"
" ON kcu.constraint_name = tc.constraint_name"
" AND kcu.table_name = tc.table_name"
" WHERE tc.table_name = :t"
" AND tc.constraint_type = 'PRIMARY KEY'"
" ORDER BY kcu.ordinal_position"
),
{"t": table},
).fetchall()
return [r[0] for r in rows]
def _clear_rows(db: Session) -> None:
db.execute(text("DELETE FROM folder_summaries"))
db.commit()
def test_upgrade_to_0017_creates_folder_summaries(db: Session, alembic: Config) -> None:
"""Upgrade 0016 → 0017: the table exists with the full column
contract (PK ``(source, folder_path)`` mirroring
``documents.source`` / ``documents.path``; TEXT summary NOT NULL;
TIMESTAMPTZ updated_at NOT NULL with the now() server default), and
the table is ABSENT at 0016 while the 0016 ``ui_settings`` schema
survives the upgrade."""
command.downgrade(alembic, "0016") # start from the pre-0017 state
assert _version(db) == "0016"
assert not _table_exists(db, "folder_summaries"), (
"folder_summaries must be absent at 0016"
)
command.upgrade(alembic, "0017")
assert _version(db) == "0017", "alembic_version must be at 0017"
assert _table_exists(db, "folder_summaries"), "the table must exist at 0017"
source = _column(db, "folder_summaries", "source")
assert source is not None, "folder_summaries.source is missing"
assert source[0] == "character varying", "source must be VARCHAR"
assert source[1] == "NO", "source must be NOT NULL (PK part 1)"
assert source[2] is None, "source must have no server default"
assert source[3] == 120, "source must be String(120) — documents.source"
folder = _column(db, "folder_summaries", "folder_path")
assert folder is not None, "folder_summaries.folder_path is missing"
assert folder[0] == "character varying", "folder_path must be VARCHAR"
assert folder[1] == "NO", "folder_path must be NOT NULL (PK part 2)"
assert folder[2] is None, "folder_path must have no server default"
assert folder[3] == 1000, "folder_path must be String(1000) — documents.path"
summary = _column(db, "folder_summaries", "summary")
assert summary is not None, "folder_summaries.summary is missing"
assert summary[0] == "text", "summary must be TEXT"
assert summary[1] == "NO", "summary must be NOT NULL (never stored empty)"
updated = _column(db, "folder_summaries", "updated_at")
assert updated is not None, "folder_summaries.updated_at is missing"
assert updated[0] == "timestamp with time zone", "updated_at must be TIMESTAMPTZ"
assert updated[1] == "NO", "updated_at must be NOT NULL"
assert updated[2] is not None and "now" in str(updated[2]), (
"updated_at must carry the now() server default (house style)"
)
assert _pk_columns(db, "folder_summaries") == ["source", "folder_path"]
# The 0016 schema survives the additive upgrade.
semantic = _column(db, "ui_settings", "ok_bg")
assert semantic is not None and semantic[3] == 7, (
"ui_settings.ok_bg (0016) must survive the upgrade"
)
def test_inserted_rows_round_trip(db: Session, alembic: Config) -> None:
"""At 0017, a source-root row (``folder_path = ''``) and a nested
folder row round-trip their values, and the composite PK rejects a
duplicate (source, folder_path) pair."""
command.upgrade(alembic, "head")
try:
db.execute(
text(
"INSERT INTO folder_summaries (source, folder_path, summary)"
" VALUES ('Homelab', '', 'Root summary.')"
)
)
db.execute(
text(
"INSERT INTO folder_summaries (source, folder_path, summary)"
" VALUES ('Homelab', 'deployments/ansible', 'Ansible summary.')"
)
)
db.commit()
rows = db.execute(
text(
"SELECT source, folder_path, summary, updated_at"
" FROM folder_summaries ORDER BY folder_path"
)
).fetchall()
assert len(rows) == 2, "both rows must be stored"
assert rows[0][0] == "Homelab" and rows[0][1] == "", (
"the source-root row uses folder_path = ''"
)
assert rows[0][2] == "Root summary.", "summary must round-trip verbatim"
assert rows[0][3] is not None, "updated_at must be stamped (server default)"
assert rows[1][1] == "deployments/ansible", (
"a nested folder path must round-trip verbatim"
)
assert rows[1][2] == "Ansible summary."
with pytest.raises(IntegrityError):
db.execute(
text(
"INSERT INTO folder_summaries (source, folder_path, summary)"
" VALUES ('Homelab', 'deployments/ansible', 'dup')"
)
)
db.rollback() # the IntegrityError aborts the open transaction
finally:
_clear_rows(db)
def test_downgrade_to_0016_drops_the_table(db: Session, alembic: Config) -> None:
"""Downgrade 0017 → 0016: the table is gone (A13 — fully
reversible) while the rest of the schema survives (the 0016
``ui_settings`` semantic columns, ``api_tokens``, ``documents``)."""
command.downgrade(alembic, "0016")
assert _version(db) == "0016"
assert not _table_exists(db, "folder_summaries"), (
"folder_summaries must be dropped"
)
semantic = _column(db, "ui_settings", "accent_line")
assert semantic is not None and semantic[3] == 7, (
"ui_settings.accent_line (0016) must survive the downgrade"
)
token_col = _column(db, "api_tokens", "token_hash")
assert token_col is not None and token_col[0] == "character varying", (
"api_tokens.token_hash must survive the downgrade"
)
doc_path = _column(db, "documents", "path")
assert doc_path is not None and doc_path[3] == 1000, (
"documents.path must survive the downgrade"
)
def test_upgrade_round_trip_restores_the_table(db: Session, alembic: Config) -> None:
"""Downgrade to 0016, then upgrade back to 0017: the table is back
with the column contract and PK intact."""
command.downgrade(alembic, "0016")
command.upgrade(alembic, "0017")
assert _version(db) == "0017", "round-trip upgrade must land at 0017"
assert _table_exists(db, "folder_summaries"), "the table must be back"
folder = _column(db, "folder_summaries", "folder_path")
assert folder is not None, "folder_summaries.folder_path must be back"
assert folder[0] == "character varying", "folder_path must be VARCHAR"
assert folder[1] == "NO", "folder_path must be NOT NULL after the round-trip"
assert folder[3] == 1000, "folder_path must be String(1000) after the round-trip"
assert _pk_columns(db, "folder_summaries") == ["source", "folder_path"]
tests/integration/test_sync_api.py
+62 -3
View File
@@ -48,9 +48,16 @@ and every failure path (git error, model down) never bumps. The
counter is pinned to the migration-0010 seed (0) around every test by
:func:`_reset_sources_version`.
The git / import / overview layers are monkeypatched in ``app.api.sync``
(same fake style as ``test_import_docs_git.py``) — no real git, no LLM:
the runner's state machine and HTTP surface are under test.
The git / import / overview / folder-summary layers are monkeypatched
in ``app.api.sync`` (same fake style as ``test_import_docs_git.py``) —
no real git, no LLM: the runner's state machine and HTTP surface are
under test. Phase 94 (task 02): the folder-summary layer gets the same
treatment — the fake-import tests' canned summaries would otherwise
steer the REAL ``generate_folder_summaries`` at the global
``documents`` table and the real ``LLMClient`` (network); the
real-import tests (host temp dirs, deterministic ``FakeEmbedder``) keep
the real generator, with ``folder_summaries`` truncated around every
test (:func:`_clean_folder_summaries`).
The admin client is used **as a context manager** on purpose: the
background sync task lives on the app's event loop, so the loop must
@@ -182,6 +189,19 @@ def clean_documents(db: Session) -> Iterator[None]:
db.commit()
@pytest.fixture(autouse=True)
def _clean_folder_summaries(db: Session) -> Iterator[None]:
"""Phase 94: the ``folder_summaries`` table is global state the real
generator (the real-import tests) writes — truncated around every
sync test so the change-gate / table-empty-gate assertions start
from a known (empty) table."""
db.execute(text("TRUNCATE folder_summaries"))
db.commit()
yield
db.execute(text("TRUNCATE folder_summaries"))
db.commit()
def _real_llm(monkeypatch: pytest.MonkeyPatch) -> None:
"""The pipeline's ``LLMClient`` becomes the deterministic in-process
``FakeEmbedder`` (real import, no network). ``FakeEmbedder``
@@ -276,6 +296,34 @@ class FakeOverview:
return self.ok
class FakeFolderSummaries:
"""Records every ``generate_folder_summaries`` call; canned stats.
Phase 94 (task 02): keeps the fake-import tests at the deterministic
layer boundary — the real generator would read the global
``documents`` table and call the (real) ``LLMClient`` over the
network. The generator only flushes, so the fake honours the
``skip`` flag the same way (the zero stats, no side effects)."""
ZERO = {"generated": 0, "failed": 0, "pruned": 0}
def __init__(self, stats: dict[str, int] | None = None) -> None:
self.stats = stats if stats is not None else dict(self.ZERO)
self.llms: list[LLMClient] = []
self.sessions: list[Session] = []
self.skip_flags: list[bool] = []
async def __call__(
self, db: Session, llm: LLMClient, *, skip: bool = False
) -> dict[str, int]:
self.skip_flags.append(skip)
if skip:
return dict(self.ZERO)
self.llms.append(llm)
self.sessions.append(db)
return dict(self.stats)
def _fake_clone() -> tuple[list[tuple[str, Path]], object]:
"""A ``clone_or_pull`` that materialises a checkout with one .md file."""
calls: list[tuple[str, Path]] = []
@@ -327,6 +375,7 @@ def test_admin_sync_success_reports_full_detail(
monkeypatch.setattr(sync_api, "import_sources", fake_import)
fake_overview = FakeOverview(ok=True)
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
assert sync_client.get("/api/sync/status").json() == {
@@ -399,6 +448,7 @@ def test_unchanged_kb_skips_overview_refresh(
monkeypatch.setattr(sync_api, "import_sources", fake_import)
fake_overview = FakeOverview(ok=True)
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
@@ -435,6 +485,7 @@ def test_double_trigger_while_running_returns_409(
fake_import = FakeImportSources(ImportSummary(files=1, added=1), delay=0.5)
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
@@ -551,12 +602,17 @@ def test_db_rows_win_over_env(
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
fake_folders = FakeFolderSummaries()
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
_login(sync_client)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# Phase 94: the changed canned summary fired the (stubbed) folder
# step with the run's own session + the import's LLM client.
assert fake_folders.llms == [fake_import.llms[0]]
assert clone_calls == [(db_url, tmp_path / "bor" / "managed")]
assert fake_import.sources == [[tmp_path / "bor" / "managed"]]
assert "env.example.com" not in str(body) # the env URL never reaches the UI
@@ -584,6 +640,7 @@ def test_env_fallback_when_table_empty(
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
@@ -988,6 +1045,7 @@ def test_sync_builds_ignore_map_by_root_string_with_union(
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
@@ -1022,6 +1080,7 @@ def test_sync_without_ignore_lists_passes_empty_map(
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
tests/integration/test_sync_folder_summaries.py
+664
View File
@@ -0,0 +1,664 @@
"""Integration: the sync-time folder-summary wiring (phase 94, task 02).
Extends the KB-overview sync pattern (``test_import_docs_overview.py``
is the template) to the phase-94 folder summaries — both sync paths
regenerate ``folder_summaries`` change-gated (added + updated > 0) or
on an empty table (the first full run after migration 0017 / a
``--limit`` first walk), per-folder fail-soft, best-effort, and in the
run's own short-lived session (the phase-53 flush-then-caller-commits
convention — the generator only flushes, the sync path commits).
Script path (``scripts.import_docs.main`` end to end, fake LLM, real
DB, explicit ``--source``):
- a KB-changing import → one row per ≥ 2-doc subtree (the source root
+ the 2-doc folder; the 1-doc folder gets none), committed in the
run's transaction, the summary line ending
``folder_summaries=<generated>/<failed>/<pruned>``;
- an unchanged re-import → zero ``lite`` calls,
``folder_summaries=skipped``, rows untouched;
- a subtree dropping below 2 docs after a changed re-walk → its row
pruned;
- one folder's ``lite`` failure → its previous row kept, the other
lands, exit code 0, stats ``1/1/0``;
- a ``--limit`` debug run → no generation, no rows,
``folder_summaries=skipped``;
- a fresh (empty) table after a ``--limit`` first walk → an unchanged
full walk generates (the table-empty first-run trigger).
API path (``POST /api/sync`` end to end, real import over a host temp
local dir, deterministic ``FakeEmbedder``):
- a KB-changing sync → the rows land (visible via the test's own
session) and the status detail keeps its exact pre-phase key set
(no folder-summary surface — the stats are log-only);
- an unchanged re-sync → zero ``FOLDER_SUMMARY_MODE`` calls;
- a ``lite`` outage (one folder failing) → the failed folder's row is
kept, the run reports ``success`` (never ``failed``), and the
sources-version bump still lands (the bump is change-gated on the
KB, not on the summaries);
- an empty table after a populated sync (the migration-0017 scenario)
→ an unchanged walk regenerates (the overview's API gate, purely
change-gated, does not).
"""
from __future__ import annotations
import logging
import time
from collections.abc import Iterator
from datetime import datetime
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.api import sync as sync_api
from app.config import Settings
from app.db import SessionLocal, db_available
from app.main import app as fastapi_app
from app.models import FolderSummary, GitSource
from app.rag import git_sources as git_sources_resolver
from app.rag.llm import LLMError
from app.rag.sources_meta import current_sources_version
from scripts import import_docs
from tests.conftest import ADMIN_PASSWORD
from tests.fakes import FakeEmbedder
# --- shared helpers ---------------------------------------------------------
def _folder_calls(llm: FakeEmbedder) -> list[list[dict[str, str]]]:
"""The ``FOLDER_SUMMARY_MODE`` chat calls on the client (the marker
in the system prompt — the KB-overview marker never contains it)."""
return [
msgs
for msgs in llm.chat_calls
if any(
"FOLDER_SUMMARY_MODE" in m.get("content", "")
for m in msgs
if m.get("role") == "system"
)
]
def _rows(db: Session) -> dict[tuple[str, str], str]:
"""The stored folder summaries as ``{(source, folder_path): summary}``."""
db.expire_all()
return {
(source, folder_path): summary
for source, folder_path, summary in db.execute(
select(FolderSummary.source, FolderSummary.folder_path, FolderSummary.summary)
).all()
}
def _updated_at(db: Session, source: str, folder_path: str) -> datetime | None:
db.expire_all()
row = db.get(FolderSummary, (source, folder_path))
return row.updated_at if row is not None else None
class FolderFailingChatEmbedder(FakeEmbedder):
"""``lite`` stand-in that fails exactly one folder-summary call —
the one whose user message contains *fail_label* (a ``Folder: …``
header) — and answers everything else (incl. the KB overview)
normally. Drives the per-folder fail-soft path."""
def __init__(self, fail_label: str) -> None:
super().__init__()
self.fail_label = fail_label
async def chat(
self, messages: list[dict[str, str]], model: str | None = None
) -> str:
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if self.fail_label in user:
self.chat_calls.append(list(messages))
raise LLMError("simulated folder-summary outage (test sentinel)")
return await super().chat(messages, model)
# --- fixtures ----------------------------------------------------------------
@pytest.fixture(autouse=True)
def _clean_kb(db: Session) -> Iterator[None]:
"""Global KB + registry state, truncated around every test (the
real import and the real generator write ``documents``/``chunks``/
``folder_summaries``; the API tests seed ``git_sources``)."""
db.execute(
text("TRUNCATE chunks, documents, kb_overview, folder_summaries, git_sources")
)
db.commit()
yield
db.execute(
text("TRUNCATE chunks, documents, kb_overview, folder_summaries, git_sources")
)
db.commit()
@pytest.fixture(autouse=True)
def _reset_sources_version() -> Iterator[None]:
"""The sources version counter is global mutable state — pin it to
the migration-0010 seed (0) around every test (own session: both
sync paths bump through their own short-lived ``SessionLocal``).
Skips like the ``db`` fixture when Postgres is down."""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
session = SessionLocal()
try:
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
session.commit()
yield
finally:
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
session.commit()
session.close()
@pytest.fixture(autouse=True)
def _fresh_sync_state() -> Iterator[None]:
"""The module-level status object + task are process-global: reset
them around every test (harmless for the script-path tests)."""
sync_api._status = sync_api.SyncStatus()
sync_api._task = None
yield
sync_api._status = sync_api.SyncStatus()
sync_api._task = None
@pytest.fixture()
def src(tmp_path: Path) -> Path:
"""MyDocs: a/ (2 docs) + b/ (1 doc) → subtree counts root 3, a 2, b 1.
md → no document-summary chat calls, so the ``lite`` traffic is
exactly the overview + the folder summaries."""
root = tmp_path / "MyDocs"
(root / "a").mkdir(parents=True)
(root / "b").mkdir()
(root / "a" / "one.md").write_text("# A One\nFirst folder document.\n", encoding="utf-8")
(root / "a" / "two.md").write_text("# A Two\nSecond folder document.\n", encoding="utf-8")
(root / "b" / "three.md").write_text("# B Three\nLone folder document.\n", encoding="utf-8")
return root
def _run_main(
monkeypatch: pytest.MonkeyPatch,
llm: FakeEmbedder,
argv: list[str],
capsys: pytest.CaptureFixture[str],
) -> tuple[int, str]:
"""Run ``import_docs.main`` with fresh settings, a fake LLM, and a
fail-loud git mock (``--source`` always wins, so git must stay
idle) — the phase-31 overview test's runner."""
monkeypatch.setattr(
import_docs,
"get_settings",
lambda: Settings(_env_file=None), # pyright: ignore[reportCallIssue]
)
def _no_git(url: str, dest: Path | str) -> Path:
raise AssertionError("git sync must not run with explicit --source")
monkeypatch.setattr(import_docs, "clone_or_pull", _no_git)
monkeypatch.setattr(import_docs, "LLMClient", lambda: llm)
rc = import_docs.main(argv)
return rc, capsys.readouterr().out
# --- script path: scripts/import_docs.py -------------------------------------
def test_changed_import_generates_folder_rows(
db: Session,
src: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""A KB-changing import upserts one row per ≥ 2-doc subtree in the
run's own transaction — committed and visible afterwards — with the
stats on the summary line (PLAN §9)."""
llm = FakeEmbedder()
records: list[logging.LogRecord] = []
class _Sink(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
records.append(record)
fs_logger = logging.getLogger("app.rag.folder_summaries")
sink = _Sink()
fs_logger.addHandler(sink)
fs_logger.setLevel(logging.INFO)
try:
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
finally:
fs_logger.removeHandler(sink)
assert rc == 0
assert "added=3" in out
assert out.rstrip().endswith(
"overview=updated sources_version=1 folder_summaries=2/0/0"
)
# Two FOLDER_SUMMARY_MODE calls — the source root (3 docs) and the
# 2-doc folder a; the 1-doc folder b gets no row (no lite burn).
calls = _folder_calls(llm)
assert len(calls) == 2
headers = [c[1]["content"].splitlines()[0] for c in calls]
assert headers == ["Folder: MyDocs", "Folder: MyDocs/a"]
for c in calls:
assert "FOLDER_SUMMARY_MODE" in c[0]["content"]
# The rows land committed (the run's own session committed them) —
# one per ≥ 2-doc subtree, never empty, stamped.
rows = _rows(db)
assert set(rows) == {("MyDocs", ""), ("MyDocs", "a")}
assert all(rows.values())
assert _updated_at(db, "MyDocs", "a") is not None
# The stats log line (PLAN §9 ample logging).
assert any(
"folder_summaries: generated=2 failed=0 pruned=0" in r.getMessage()
for r in records
)
def test_unchanged_reimport_burns_zero_folder_calls(
db: Session,
src: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""Same hashes → no KB change → the populated table stays untouched
and zero ``lite`` calls burn (overview AND folder summaries)."""
llm1 = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
assert rc == 0
assert out.rstrip().endswith(
"overview=updated sources_version=1 folder_summaries=2/0/0"
)
rows = _rows(db)
assert set(rows) == {("MyDocs", ""), ("MyDocs", "a")}
llm2 = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
assert rc == 0
assert "unchanged=3" in out
assert out.rstrip().endswith(
"overview=skipped sources_version=skipped folder_summaries=skipped"
)
assert llm2.chat_calls == [] # zero lite calls, any mode
assert _rows(db) == rows # rows byte-identical
def test_subtree_dropping_below_two_docs_is_pruned(
db: Session,
src: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""A subtree that drops below the ≥ 2 rule on a changed re-walk
loses its (stale) row; the remaining qualifiers regenerate."""
llm1 = FakeEmbedder()
rc, _ = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
assert rc == 0
assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a")}
# a/ loses one of its two docs (below the ≥ 2 rule) while b's doc
# changes → the re-walk is a KB change, so generation runs and the
# stale a/ row is pruned; b (1 doc) still gets no row.
(src / "a" / "two.md").unlink()
(src / "b" / "three.md").write_text("# B Three\nChanged content.\n", encoding="utf-8")
llm2 = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src), "--prune"], capsys)
assert rc == 0
assert "pruned=1" in out
assert "updated=1" in out
assert out.rstrip().endswith(
"overview=updated sources_version=2 folder_summaries=1/0/1"
)
# Only the source root (the 2 remaining docs) still qualifies.
assert set(_rows(db)) == {("MyDocs", "")}
assert _updated_at(db, "MyDocs", "a") is None # the row is gone
def test_folder_lite_failure_keeps_previous_row_and_stays_green(
db: Session,
src: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""One folder's ``lite`` failure: its previous row is kept, the
other folder lands, and the run's exit code stays 0 — the stats
carry the failure (``1/1/0``)."""
llm1 = FakeEmbedder()
rc, _ = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
assert rc == 0
rows_before = _rows(db)
a_stamp_before = _updated_at(db, "MyDocs", "a")
root_stamp_before = _updated_at(db, "MyDocs", "")
assert a_stamp_before is not None and root_stamp_before is not None
# a/ changes (a KB change → generation runs); lite fails for a/
# only — its previous row must stay, the root row must land.
(src / "a" / "one.md").write_text("# A One\nChanged content.\n", encoding="utf-8")
llm2 = FolderFailingChatEmbedder(fail_label="Folder: MyDocs/a")
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
assert rc == 0 # a failed folder must not fail the import
assert "updated=1" in out
assert out.rstrip().endswith(
"overview=updated sources_version=2 folder_summaries=1/1/0"
)
rows_after = _rows(db)
assert rows_after["MyDocs", "a"] == rows_before["MyDocs", "a"] # kept
assert _updated_at(db, "MyDocs", "a") == a_stamp_before # untouched
root_stamp_after = _updated_at(db, "MyDocs", "")
assert root_stamp_after is not None and root_stamp_after > root_stamp_before
def test_limit_run_skips_folder_generation(
db: Session,
src: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""A ``--limit`` debug run (an incomplete walk) never generates —
no ``lite`` call at all, no rows, the line says ``skipped``."""
llm = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm, ["--source", str(src), "--limit", "2"], capsys)
assert rc == 0
assert "added=2" in out
assert out.rstrip().endswith(
"overview=skipped sources_version=skipped folder_summaries=skipped"
)
assert llm.chat_calls == [] # no lite call, any mode
assert _rows(db) == {} # an incomplete walk never writes rows
def test_empty_table_generates_on_unchanged_walk(
db: Session,
src: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""The table-empty first-run trigger: after a ``--limit`` first
walk (populated KB, empty table), an unchanged full walk generates
— for the folder summaries AND the missing outline, still never
bumping the version."""
llm1 = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src), "--limit", "3"], capsys)
assert rc == 0
assert "added=3" in out
assert out.rstrip().endswith(
"overview=skipped sources_version=skipped folder_summaries=skipped"
)
assert _rows(db) == {}
llm2 = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
assert rc == 0
assert "unchanged=3" in out
assert out.rstrip().endswith(
"overview=updated sources_version=skipped folder_summaries=2/0/0"
)
assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a")}
assert len(_folder_calls(llm2)) == 2
assert current_sources_version(db) == 0 # the unchanged walk never bumps
# --- API path: POST /api/sync -------------------------------------------------
@pytest.fixture()
def sync_client() -> Iterator[TestClient]:
"""Context-managed TestClient — one app event loop across requests
(the background task must survive between the POST and the polls)."""
with TestClient(fastapi_app) as client:
yield client
def _settings(sources_dir: str) -> Settings:
return Settings(_env_file=None, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
def _stub_env(monkeypatch: pytest.MonkeyPatch, git_sources: str = "") -> None:
"""The resolver's env fallback, driven by a fresh ``Settings`` (the
dev ``.env`` never leaks in)."""
monkeypatch.setattr(
git_sources_resolver,
"get_settings",
lambda: Settings(_env_file=None, git_sources=git_sources), # pyright: ignore[reportCallIssue]
)
def _seed_local(db: Session, path: Path) -> None:
"""A ``kind=local`` row as the phase-38 API stores it: the expanded
absolute path in both ``path`` and the NOT-NULL ``url`` column."""
db.add(GitSource(url=str(path), kind="local", path=str(path)))
db.commit()
def _capture_llm(monkeypatch: pytest.MonkeyPatch) -> list[FakeEmbedder]:
"""The pipeline's ``LLMClient`` becomes a recorded
``FakeEmbedder`` (real import + real generator, no network)."""
clients: list[FakeEmbedder] = []
def _factory() -> FakeEmbedder:
client = FakeEmbedder()
clients.append(client)
return client
monkeypatch.setattr(sync_api, "LLMClient", _factory)
return clients
def _login(client: TestClient) -> None:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
def _poll(client: TestClient, want: str, timeout: float = 10.0) -> dict:
"""Poll ``GET /api/sync/status`` until ``state == want`` (terminal).
Any state other than ``running`` before the deadline fails loudly —
an unexpected ``failed`` must never be masked by the wait."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
body = client.get("/api/sync/status").json()
if body["state"] == want:
return body
assert body["state"] == "running", (
f"unexpected state {body['state']!r} while waiting for {want!r}: {body}"
)
time.sleep(0.05)
raise AssertionError(f"sync did not reach {want!r} within {timeout}s")
@pytest.fixture()
def local_dir(tmp_path: Path) -> Path:
"""LocalDocs: a/ (2 docs) → subtree counts root 2, a 2 (one
qualifying source-root row + one folder row)."""
d = tmp_path / "LocalDocs"
(d / "a").mkdir(parents=True)
(d / "a" / "one.md").write_text("# A One\nfirst local file\n", encoding="utf-8")
(d / "a" / "two.md").write_text("# A Two\nsecond local file\n", encoding="utf-8")
return d
def test_api_changed_sync_generates_folder_rows(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
local_dir: Path,
) -> None:
"""A KB-changing sync upserts the rows (visible via the test's own
session) and keeps the status detail's exact pre-phase shape — the
folder stats are log-only, not a new status surface."""
_seed_local(db, local_dir)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(str(local_dir.parent / "bor")),
)
clients = _capture_llm(monkeypatch)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert body["error"] is None
# No new sync-status surface: the detail keeps its exact key set.
assert set(body["detail"]) == {
"files", "added", "updated", "unchanged", "pruned", "errors",
"chunks", "summaries", "summary_errors", "overview",
"sources_version",
}
assert body["detail"]["files"] == 2
assert body["detail"]["added"] == 2
assert body["detail"]["overview"] is True
assert body["detail"]["sources_version"] == 1
# One LLM client for probe + import + overview + folder summaries;
# exactly two FOLDER_SUMMARY_MODE calls (root + the 2-doc a/ folder).
assert len(clients) == 1
calls = _folder_calls(clients[0])
assert len(calls) == 2
headers = [c[1]["content"].splitlines()[0] for c in calls]
assert headers == ["Folder: LocalDocs", "Folder: LocalDocs/a"]
# The rows land committed — visible from the test's own session.
rows = _rows(db)
assert set(rows) == {("LocalDocs", ""), ("LocalDocs", "a")}
assert all(rows.values())
assert current_sources_version(db) == 1
def test_api_unchanged_resync_burns_zero_folder_calls(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
local_dir: Path,
) -> None:
"""Same files → unchanged re-sync: no ``lite`` call of any kind
(populated table), rows untouched, version unadvanced."""
_seed_local(db, local_dir)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(str(local_dir.parent / "bor")),
)
clients = _capture_llm(monkeypatch)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
rows = _rows(db)
assert set(rows) == {("LocalDocs", ""), ("LocalDocs", "a")}
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert body["detail"]["overview"] is False
assert body["detail"]["sources_version"] == 1 # unchanged → no bump
assert len(clients) == 2
# Zero GENERATION calls — no folder summary, no KB overview. The
# only chat the re-sync's client makes is the phase-41 probe's ping.
assert _folder_calls(clients[1]) == []
assert all(
"KB_OVERVIEW_MODE" not in m.get("content", "")
for msgs in clients[1].chat_calls
for m in msgs
if m.get("role") == "system"
)
assert len(clients[1].chat_calls) == 1
assert clients[1].chat_calls[0][0]["content"] == "ping"
assert _rows(db) == rows
assert current_sources_version(db) == 1
def test_api_folder_lite_failure_keeps_rows_stays_green_and_bumps(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
local_dir: Path,
) -> None:
"""A ``lite`` outage for one folder: the run stays ``success``
(never ``failed``), the failed folder's previous row is kept, the
rest regenerates, and the sources-version bump still lands (the
bump is change-gated on the KB, not on the summaries)."""
_seed_local(db, local_dir)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(str(local_dir.parent / "bor")),
)
clients = _capture_llm(monkeypatch)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
rows_before = _rows(db)
a_stamp_before = _updated_at(db, "LocalDocs", "a")
root_stamp_before = _updated_at(db, "LocalDocs", "")
assert a_stamp_before is not None and root_stamp_before is not None
assert len(clients) == 1 # sync 1 used the recording factory
# a/ changes (the KB changes) while lite fails for a/ only.
(local_dir / "a" / "one.md").write_text(
"# A One\nchanged local file\n", encoding="utf-8"
)
monkeypatch.setattr(
sync_api,
"LLMClient",
lambda: FolderFailingChatEmbedder(fail_label="Folder: LocalDocs/a"),
)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success") # GREEN — a lite outage is fail-soft
assert body["error"] is None
assert body["detail"]["updated"] == 1
assert body["detail"]["sources_version"] == 2 # the bump was not blocked
assert current_sources_version(db) == 2
rows_after = _rows(db)
assert rows_after["LocalDocs", "a"] == rows_before["LocalDocs", "a"]
assert _updated_at(db, "LocalDocs", "a") == a_stamp_before # kept as-is
root_stamp_after = _updated_at(db, "LocalDocs", "")
assert root_stamp_after is not None and root_stamp_after > root_stamp_before
def test_api_empty_table_first_sync_regenerates(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
local_dir: Path,
) -> None:
"""The migration-0017 scenario: the KB predates the table — wipe
the rows and re-sync an unchanged KB: the empty-table trigger
fires for the folder summaries (the overview's API gate, purely
change-gated, does not)."""
_seed_local(db, local_dir)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(str(local_dir.parent / "bor")),
)
clients = _capture_llm(monkeypatch)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
assert set(_rows(db)) == {("LocalDocs", ""), ("LocalDocs", "a")}
db.execute(text("TRUNCATE folder_summaries"))
db.commit()
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert body["detail"]["overview"] is False # unchanged → no overview
assert body["detail"]["sources_version"] == 1 # unchanged → no bump
assert len(clients) == 2
assert len(_folder_calls(clients[1])) == 2 # the folders regenerated
assert set(_rows(db)) == {("LocalDocs", ""), ("LocalDocs", "a")}
tests/unit/test_agent.py
+480 -128
View File
@@ -2,15 +2,20 @@
harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
A scripted fake LLM (canned stream sequences) + monkeypatched
``list_catalog`` / ``list_source_names`` / ``find_document`` /
``ls_top`` / ``ls_folder`` / ``list_source_names`` / ``find_document`` /
``all_documents`` — no database, no network. Covers the loop mechanics:
the ls → read (combined ``source/path``) → answer happy path (event
order, holder state, the tools staying offered on every request —
phase 45 removed the per-tool budgets, the assistant/tool message
history), the ``ls`` scoping (no-arg full catalog in the phase-63
labeled-field format, a one-source scope, a known source with 0
documents → ``0 documents:`` counted, an unknown-source refusal that
counts nothing), ``read`` on the canonical combined form (split at the
history), the phase-94 drill-down ``ls`` (no-arg top level = the
registered sources with counts + stored summaries in the pinned
``{N} sources:`` template, a source scope = its root folder —
subfolders + capped file lines in the pinned folder template —, a
``source/folder`` scope = one level deeper, a registered source with 0
documents → the ``… — 0 documents, 0 folders:`` header counted, an
unknown-source refusal that counts nothing, the NOT-A-FOLDER teaching
with the parent's subfolders, the 50-file cap + grep-pointer note),
``read`` on the canonical combined form (split at the
FIRST slash, full content, the bare-source-name refusal, the
already-in-context dedupe, missing-args refusals), the phase-68 ``grep``
contract under its new name (the locked A5 pins: fixed substring,
@@ -149,28 +154,35 @@ def test_agent_tools_names_and_parameters() -> None:
# Task 05 (live gate iteration 2): the one-call-at-a-time discipline
# clause (the harness prior batches calls; the loop executes one
# per round — the extras count as unexecuted in the gate).
# Phase 94 (task 03): the description is the drill-down tree
# contract (pinned copy — the tool-surface revision, owner
# permission 2026-09-10, ``TODO.md`` L4): one level per call,
# sources at the top, folders + files below, the file-line format
# and the combined-identity handoff to read/grep intact.
assert ls["description"] == (
"List the indexed documents as `source: X | path: Y | "
"title: Z` lines. Call one tool at a time — wait for "
"this result before your next call."
"List the knowledge base as a tree, one level at a time. "
"With no path: the synced sources — each with its document "
"count and a summary of its contents. With a source name (no "
"'/'): that source's top-level folders and files. With a "
"`source/folder` path: that folder's subfolders and files. "
"Folder lines carry a summary of what the folder contains. "
"File lines are `source: X | path: Y | title: Z` — use the "
"combined `source/path` with `read` and `grep`. Call one tool "
"at a time — wait for this result before your next call."
)
ls_params = ls["parameters"]
assert ls_params["type"] == "object"
assert ls_params["required"] == [] # path is optional
assert set(ls_params["properties"]) == {"path"}
assert ls_params["properties"]["path"]["type"] == "string"
# Phase 72: the description states the contract up front — the
# 'path' argument is a source name, not a file or directory path.
# Task 05 (live gate iteration 5): the cross-tool contrast clause
# (ls is the ONLY tool whose path is a source name — the model
# kept transferring that scope to grep's document identity).
# Phase 94 (task 03): the 'path' argument teaches the drill-down
# semantics — a source name lists that source's top level, a
# `source/folder` path drills one level deeper, omitted lists
# every source (pinned copy).
assert ls_params["properties"]["path"]["description"] == (
"Source name to list one source's documents (e.g. 'homelab') — "
"a source name, not a file or directory path; omit to list "
"every document. This is the only tool "
"whose `path` is a source name — for "
"`read` and `grep` it must be a document's "
"combined `source/path`."
"Optional — a source name (e.g. 'homelab') to list its top "
"level, or a `source/folder` path to drill down (e.g. "
"'homelab/active'). Omit it to list every source."
)
read = by_name["read"]["function"]
# Tool-calling fast loop (2026-09-04, controlled fixture gate):
@@ -285,16 +297,11 @@ def test_refusal_constants_are_harness_aligned() -> None:
assert agent.UNKNOWN_TOOL == "Unknown tool."
assert agent.MISSING_READ_ARGS == "read requires a string argument 'path'."
assert agent.MISSING_SEARCH_ARGS == "grep requires a string argument 'pattern'."
# Phase 72: the ls teaching-refusal templates, pinned byte-for-byte
# (task 01 — the read/grep suggestion templates below, task 02).
assert agent.LS_PATH_NOT_A_SOURCE == (
"'{path}' looks like a document path, not a source name. The "
"'path' argument of ls filters by source name (e.g. 'homelab') — "
"omit it to list every document, or read a document by its "
"combined 'source/path' string."
)
# The pre-phase-72 no-source line is the byte-identical prefix of
# the extended line — only the teaching parenthetical was appended.
# Phase 94 (task 03): the phase-72 document-path teaching refusal is
# DELETED (a ``/`` now names a folder — the drill-down contract);
# the no-source refusal stays byte-identical (the task's "existing
# refusal, teaching parenthetical intact" pin).
assert not hasattr(agent, "LS_PATH_NOT_A_SOURCE")
assert agent.NO_SOURCE_NOT_A_DIRECTORY.startswith(
"No source named '{scope}' — check the ls output."
)
@@ -303,6 +310,11 @@ def test_refusal_constants_are_harness_aligned() -> None:
"argument is a source name, not a directory — omit it to list "
"every document.)"
)
# Phase 94 (task 03): the NOT-A-FOLDER drill-down teaching template,
# pinned byte-for-byte (argument echoed, parent's subfolders
# listed), and the pinned file-line cap constant.
assert agent.NOT_A_FOLDER == "'{arg}' is not a folder — {parent} has: {subfolders}"
assert agent.LS_MAX_FILE_LINES == 50
# Phase 72 (task 02): the read/grep "did you mean …?" suggestion
# templates, pinned byte-for-byte, and the suggestion cap.
assert agent.NO_DOCUMENT_DID_YOU_MEAN == (
@@ -348,11 +360,12 @@ def test_list_source_names_empty_registry(monkeypatch: pytest.MonkeyPatch) -> No
def test_ls_then_read_then_answer(monkeypatch: pytest.MonkeyPatch) -> None:
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "aws-route53.md", "AWS Route53 Records"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
# Phase 94 (task 03): the no-arg ``ls`` is the drill-down TOP level
# (the registered sources, registry order) — monkeypatched the way
# the phase-70 full-catalog listing used to be.
monkeypatch.setattr(
agent, "ls_top", lambda db: [("Deployments", 1, None), ("Homelab", 1, None)]
)
target = _doc("Homelab", "aws-route53.md", "AWS Route53 Records", "R53-CONTENT")
calls: list[tuple[str, str]] = []
@@ -423,11 +436,7 @@ def test_ls_then_read_then_answer(monkeypatch: pytest.MonkeyPatch) -> None:
assert msgs[3] == {
"role": "tool",
"tool_call_id": "call_1",
"content": (
"2 documents:\n"
"source: Deployments | path: backups.md | title: Backup Strategy\n"
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
),
"content": "2 sources:\n\nDeployments — 1 documents\nHomelab — 1 documents",
}
# The second follow-up request carries the read call + the FULL text.
msgs = llm.requests[2][0]
@@ -448,7 +457,7 @@ def test_content_and_tool_call_in_one_stream_keeps_both(
) -> None:
"""Rare stream with content AND a tool call: the content stays (it was
already emitted) and the tool still runs."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
monkeypatch.setattr(agent, "ls_top", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -460,7 +469,7 @@ def test_content_and_tool_call_in_one_stream_keeps_both(
pieces = asyncio.run(_run(llm, holder, _settings()))
assert [type(p) for p in pieces] == [StreamPiece, ToolCallPiece, StreamPiece]
assert holder.tool_calls == 1 # the tool ran despite the content
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
assert llm.requests[1][0][3]["content"] == "0 sources:"
# ---------- phase 74: client history between system and user ----------
@@ -519,7 +528,7 @@ def test_run_agent_history_survives_a_tool_round(
"""The tool rounds append assistant/tool messages to the SAME
``messages`` list — the prior history stays in place between the
system prompt and the current question on the SECOND request too."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
monkeypatch.setattr(agent, "ls_top", lambda db: [])
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
[StreamPiece("content", "the answer")],
@@ -534,22 +543,25 @@ def test_run_agent_history_survives_a_tool_round(
]
# ---------- ls: full catalog + scoping ----------
# ---------- ls: the drill-down tree (phase 94, task 03) ----------
def test_ls_full_catalog_format(monkeypatch: pytest.MonkeyPatch) -> None:
"""No argument: the full catalog in the phase-63 labeled-field format
(``source: X | path: Y | title: Z``) — counted; no registry lookup."""
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "aws-route53.md", "AWS Route53 Records"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
def _boom_sources(*_a: Any, **_k: Any) -> None:
raise AssertionError("no registry lookup for an unscoped ls")
monkeypatch.setattr(agent, "list_source_names", _boom_sources)
def test_ls_top_level_lists_sources_with_summaries(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""No argument: the TOP level — the registered sources, registry
order, each ``{source} — {n} documents`` + the indented summary line
only when stored — the pinned template, counted; the registry IS
consulted (unlike the phase-70 full catalog, the top level is the
registry itself)."""
monkeypatch.setattr(
agent,
"ls_top",
lambda db: [
("Deployments", 0, None),
("Homelab", 1, "The homelab notes."),
],
)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -557,22 +569,27 @@ def test_ls_full_catalog_format(monkeypatch: pytest.MonkeyPatch) -> None:
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
"2 documents:\n"
"source: Deployments | path: backups.md | title: Backup Strategy\n"
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
"2 sources:\n"
"\n"
"Deployments — 0 documents\n"
"Homelab — 1 documents\n"
" The homelab notes."
)
assert holder.tool_calls == 1
def test_ls_empty_catalog_says_zero_documents(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
def test_ls_empty_registry_says_zero_sources(monkeypatch: pytest.MonkeyPatch) -> None:
"""No registered sources: the top level is the header line alone —
``0 sources:`` (the old ``0 documents:`` behavior preserved in
spirit), still a counted result."""
monkeypatch.setattr(agent, "ls_top", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
assert llm.requests[1][0][3]["content"] == "0 sources:"
assert holder.tool_calls == 1
@@ -583,13 +600,12 @@ def test_ls_empty_catalog_says_zero_documents(monkeypatch: pytest.MonkeyPatch) -
({"path": 7}, "non-string path"),
],
)
def test_ls_blank_path_lists_full_catalog(
def test_ls_blank_path_lists_top_level(
monkeypatch: pytest.MonkeyPatch, arguments: dict[str, Any], label: str
) -> None:
"""A blank (or non-string) ``path`` is treated as omitted — the full
catalog, counted (no refusal for an empty scope)."""
catalog = [("S", "a.md", "A")]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
"""A blank (or non-string) ``path`` is treated as omitted — the top
level (the sources), counted (no refusal for an empty scope)."""
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -597,21 +613,25 @@ def test_ls_blank_path_lists_full_catalog(
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert (
llm.requests[1][0][3]["content"] == "1 documents:\nsource: S | path: a.md | title: A"
)
assert llm.requests[1][0][3]["content"] == "1 sources:\n\nS — 1 documents"
assert holder.tool_calls == 1
def test_ls_scoped_to_known_source(monkeypatch: pytest.MonkeyPatch) -> None:
"""A known source name: the same listing filtered to that source —
def test_ls_source_scope_lists_root_folder(monkeypatch: pytest.MonkeyPatch) -> None:
"""A registered source name (no ``/``): the source's ROOT folder —
subfolders (2-space-indented, path order, ``: {summary}`` only when
stored) + the root's own file lines in EXACTLY the
``source: X | path: Y | title: Z`` format — the pinned template,
counted."""
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "a.md", "A"),
("Homelab", "b.md", "B"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
monkeypatch.setattr(
agent,
"ls_folder",
lambda db, source, folder: (
[("backups", 2, "Backup notes."), ("networking", 1, None)],
[("Homelab", "readme.md", "Readme")],
1,
),
)
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Deployments", "Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -620,9 +640,50 @@ def test_ls_scoped_to_known_source(monkeypatch: pytest.MonkeyPatch) -> None:
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
"2 documents:\n"
"source: Homelab | path: a.md | title: A\n"
"source: Homelab | path: b.md | title: B"
"Homelab — 1 documents, 2 folders:\n"
"\n"
" backups/ — 2 documents: Backup notes.\n"
" networking/ — 1 documents\n"
"\n"
"source: Homelab | path: readme.md | title: Readme"
)
assert holder.tool_calls == 1
def test_ls_nested_folder_scope_lists_one_level_deeper(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A ``source/folder`` path: that folder's subfolders + own file
lines, identity = ``source/folder`` (the same template as the
root), counted; the fetchers are the source-scoped ones."""
def _rows(db: Any, source: str) -> list[tuple[str, str]]:
assert (source, db) == ("Homelab", None)
return [
("networking/lan.md", "LAN"),
("networking/vpn.md", "VPN"),
]
monkeypatch.setattr(agent, "_source_document_rows", _rows)
monkeypatch.setattr(
agent, "_source_folder_summaries", lambda db, source: {"networking": "Network notes."}
)
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1", name="ls", arguments={"path": "Homelab/networking"}
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
"Homelab/networking — 2 documents, 0 folders:\n"
"\n"
"source: Homelab | path: networking/lan.md | title: LAN\n"
"source: Homelab | path: networking/vpn.md | title: VPN"
)
assert holder.tool_calls == 1
@@ -631,9 +692,13 @@ def test_ls_scoped_known_source_with_zero_docs_counts(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A registered source with no indexed documents is KNOWN (the
registry is the source of truth, not the catalog): it lists as
``0 documents:`` — a valid, counted result, not a refusal."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("Other", "a.md", "A")])
registry is the source of truth, not the catalog): it lists its
header line alone (``… — 0 documents, 0 folders:`` — the old
``0 documents:`` behavior preserved in spirit) — a valid, counted
result, not a refusal."""
monkeypatch.setattr(
agent, "ls_folder", lambda db, source, folder: ([], [], 0)
)
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab", "Other"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -641,7 +706,7 @@ def test_ls_scoped_known_source_with_zero_docs_counts(
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
assert llm.requests[1][0][3]["content"] == "Homelab — 0 documents, 0 folders:"
assert holder.tool_calls == 1 # an executed ls, not a refusal
assert llm.requests[1][1] == AGENT_TOOLS
@@ -650,7 +715,6 @@ def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> No
"""A ``path`` without ``/`` matching no source name is a refusal —
the extended line with the teaching parenthetical (phase 72), not
counted, the round cap bounds its repetition."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -666,20 +730,14 @@ def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> No
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_path_like_scope_gets_document_path_teaching_refusal(
def test_ls_path_like_scope_unknown_source_gets_no_source_refusal(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 72: a stripped scope containing ``/`` looks like a document
path (the incident's ``ls(path='app/rag/importer.py')``) — a source
name is a directory basename and can never contain one, so this gets
the ``LS_PATH_NOT_A_SOURCE`` teaching line with the argument echoed;
no registry lookup, counts in nothing, tools stay offered."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
def _boom_sources(*_a: Any, **_k: Any) -> None:
raise AssertionError("no registry lookup for a path-like scope")
monkeypatch.setattr(agent, "list_source_names", _boom_sources)
"""Phase 94: a ``/`` now names a folder, so the phase-72
document-path teaching is DELETED — a ``source/…`` argument whose
FIRST segment names no registered source gets the no-source refusal
(the segment echoed), counted in nothing, tools stay offered."""
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -695,7 +753,7 @@ def test_ls_path_like_scope_gets_document_path_teaching_refusal(
assert holder.tool_calls == 0 # a refusal counts in nothing
assert (
llm.requests[1][0][3]["content"]
== agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="app")
)
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
@@ -706,8 +764,7 @@ def test_ls_dot_scope_gets_not_a_directory_teaching_refusal(
"""Phase 72: ``ls(path='.')`` (the incident's second round — no
``/``, no matching source) gets the extended no-source refusal with
the teaching parenthetical, ``'.'`` echoed — not counted, tools stay
offered."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
offered (unchanged by phase 94)."""
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -723,6 +780,312 @@ def test_ls_dot_scope_gets_not_a_directory_teaching_refusal(
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_unknown_top_level_folder_gets_not_a_folder_teaching(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 94: a ``source/…`` argument whose TOP-LEVEL folder segment
matches no indexed prefix gets the NOT-A-FOLDER teaching — the
argument echoed, the source named, its direct subfolders listed so
the model self-corrects in the next round; not counted, tools stay
offered."""
monkeypatch.setattr(
agent,
"_source_document_rows",
lambda db, source: [
("backups/cron.md", "Cron"),
("containers/caddy.md", "Caddy"),
("networking/lan.md", "LAN"),
],
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1", name="ls", arguments={"path": "Homelab/netwoking"}
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0 # a refusal counts in nothing
assert llm.requests[1][0][3]["content"] == (
"'Homelab/netwoking' is not a folder — Homelab has: "
"backups/ containers/ networking/"
)
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_unknown_nested_folder_gets_not_a_folder_with_nested_parent(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 94: a nested miss names the DEEPEST existing ancestor —
``source/folder`` — and lists ITS direct subfolders (bounded: the
parent's own listing, no new flood path)."""
monkeypatch.setattr(
agent,
"_source_document_rows",
lambda db, source: [
("networking/lan/a.md", "A"),
("networking/vpn/b.md", "B"),
("readme.md", "Readme"),
],
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1", name="ls", arguments={"path": "Homelab/networking/lan/x"}
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0
assert llm.requests[1][0][3]["content"] == (
"'Homelab/networking/lan/x' is not a folder — "
"Homelab/networking/lan has: none"
)
def test_ls_file_path_scope_gets_not_a_folder(monkeypatch: pytest.MonkeyPatch) -> None:
"""Phase 94: a document's OWN path is never a folder (nothing starts
with ``path + '/'``) — ``ls`` of a file path refuses with the
NOT-A-FOLDER teaching (the parent's subfolders listed)."""
monkeypatch.setattr(
agent,
"_source_document_rows",
lambda db, source: [("notes.md", "Notes"), ("a/b.md", "B")],
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(id="call_1", name="ls", arguments={"path": "S/notes.md"})
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0
assert llm.requests[1][0][3]["content"] == (
"'S/notes.md' is not a folder — S has: a/"
)
# ---------- ls_top / ls_folder: the drill-down accessors (pure + composed) ----------
def test_ls_top_registry_order_zero_docs_and_summaries(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""``ls_top``: registry order (not catalog order), a 0-document
source still lists, the summary is the stored ``(source, "")`` row
or ``None`` when absent; an empty registry → ``[]``."""
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Zeta", "Alpha"])
monkeypatch.setattr(
agent, "_source_document_counts", lambda db: [("Zeta", 3), ("Beta", 1)]
)
monkeypatch.setattr(
agent, "_source_root_summaries", lambda db: [("Zeta", "Zeta stuff.")]
)
assert agent.ls_top(cast("Session", object())) == [
("Zeta", 3, "Zeta stuff."),
("Alpha", 0, None), # 0 docs (no count row) + no stored summary
]
def test_ls_top_empty_registry_is_empty(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setattr(agent, "list_source_names", lambda db: [])
def _boom(*_a: Any, **_k: Any) -> None:
raise AssertionError("no fetches for an empty registry")
monkeypatch.setattr(agent, "_source_document_counts", _boom)
monkeypatch.setattr(agent, "_source_root_summaries", _boom)
assert agent.ls_top(cast("Session", object())) == []
def test_ls_folder_composes_the_fetchers(monkeypatch: pytest.MonkeyPatch) -> None:
"""``ls_folder`` = the source's document rows + stored summaries
through the pure :func:`group_folder_listing` (the fetchers are the
monkeypatch surface)."""
seen: list[tuple[str, str, str]] = []
def _rows(db: Any, source: str) -> list[tuple[str, str]]:
seen.append(("rows", source, ""))
return [("a/b.md", "B"), ("a.md", "A")]
def _summaries(db: Any, source: str) -> dict[str, str]:
seen.append(("summaries", source, ""))
return {"a": "A stuff."}
monkeypatch.setattr(agent, "_source_document_rows", _rows)
monkeypatch.setattr(agent, "_source_folder_summaries", _summaries)
assert agent.ls_folder(cast("Session", object()), "S", "") == (
[("a", 1, "A stuff.")],
[("S", "a.md", "A")],
1,
)
assert seen == [("rows", "S", ""), ("summaries", "S", "")]
def test_group_folder_listing_subfolder_recursion_and_counts() -> None:
"""The recursive count per subfolder — every path equal to the
folder or starting with ``folder + '/'`` (a doc under ``a/b/``
counts for BOTH ``a`` and ``a/b``), path order, the stored summary
attached or ``None``."""
rows = [
("a/b/c.md", "C"),
("a/b/d.md", "D"),
("a/e.md", "E"),
("f.md", "F"),
]
sub, files, total = agent.group_folder_listing(
"S", "", rows, {"a": "A subtree.", "a/b": "B subtree."}
)
# ROOT level: the direct subfolders of "" are the TOP-LEVEL folders
# only (a/b is nested under a, not direct) — a's count is its whole
# recursive subtree (a/e.md + a/b/c.md + a/b/d.md), the stored
# summary attached.
assert sub == [("a", 3, "A subtree.")]
assert files == [("S", "f.md", "F")]
assert total == 1
# One level down: a/b is a's direct subfolder with its own count.
sub2, _files2, _total2 = agent.group_folder_listing("S", "a", rows, {"a/b": "B subtree."})
assert sub2 == [("a/b", 2, "B subtree.")]
def test_group_folder_listing_nested_level_counts_and_membership() -> None:
"""One level down: ``a``'s direct subfolder is ``a/b`` (count 2),
its own direct file is ``a/e.md`` (``a/b/c.md`` is NOT a direct
file of ``a``) — membership is the folder_of rule, order is path
order."""
rows = [
("a/b/c.md", "C"),
("a/b/d.md", "D"),
("a/e.md", "E"),
]
sub, files, total = agent.group_folder_listing("S", "a", rows, {})
assert sub == [("a/b", 2, None)]
assert files == [("S", "a/e.md", "E")]
assert total == 1
def test_group_folder_listing_file_path_is_not_a_folder() -> None:
"""A document whose path is a prefix of NO other path is a file,
never a folder: ``ls`` of it must not list a subfolder (and the
``path == folder`` count arm only fires for TRUE folders — a doc
sharing a real folder's name counts for that folder, the existence
rule intact)."""
rows = [
("a.md", "A"), # a file at the root, and a folder name? NO —
("b/x.md", "X"), # nothing starts with "a.md/"
]
sub, files, total = agent.group_folder_listing("S", "", rows, {})
assert sub == [("b", 1, None)] # "a.md" is NOT a subfolder
assert files == [("S", "a.md", "A")] # b/x.md is NOT a direct root file
assert total == 1
# The path == folder arm: a doc named "a" under a real folder "a/".
rows2 = [("a", "FileA"), ("a/c.md", "C")]
sub2, files2, total2 = agent.group_folder_listing("S", "", rows2, {})
assert sub2 == [("a", 2, None)] # the file "a" counts for folder "a"
assert files2 == [("S", "a", "FileA")] # …and is a direct ROOT file
assert total2 == 1
def test_group_folder_listing_caps_files_at_fifty_keeps_the_total() -> None:
"""The cap: 51 direct files → 50 file lines + the PRE-cap total (51)
for the renderer's note; 50 files → 50 lines, no note material.
A 500-file folder costs 50 lines, never 500."""
rows51 = [(f"big/f{i:03d}.md", f"T{i}") for i in range(51)]
sub, files, total = agent.group_folder_listing("S", "big", rows51, {})
assert sub == []
assert total == 51
assert len(files) == 50
assert files[0] == ("S", "big/f000.md", "T0")
assert files[-1] == ("S", "big/f049.md", "T49")
rows50 = [(f"big/f{i:03d}.md", f"T{i}") for i in range(50)]
_sub, files50, total50 = agent.group_folder_listing("S", "big", rows50, {})
assert total50 == 50 and len(files50) == 50
# ---------- the pinned drill-down templates (byte-for-byte) ----------
def test_render_ls_top_template() -> None:
assert (
agent.render_ls_top(
[("Deployments", 3, None), ("Homelab", 5, "Home lab notes.")]
)
== "2 sources:\n\n"
"Deployments — 3 documents\n"
"Homelab — 5 documents\n"
" Home lab notes."
)
assert agent.render_ls_top([]) == "0 sources:"
def test_render_folder_listing_root_template() -> None:
assert (
agent.render_folder_listing(
"Homelab",
[("backups", 2, "Backup notes."), ("networking", 1, None)],
[("Homelab", "readme.md", "Readme")],
1,
)
== "Homelab — 1 documents, 2 folders:\n"
"\n"
" backups/ — 2 documents: Backup notes.\n"
" networking/ — 1 documents\n"
"\n"
"source: Homelab | path: readme.md | title: Readme"
)
def test_render_folder_listing_empty_level_is_header_alone() -> None:
"""A registered source with no documents: the header line alone —
the old ``0 documents:`` behavior preserved in spirit."""
assert agent.render_folder_listing("Homelab", [], [], 0) == (
"Homelab — 0 documents, 0 folders:"
)
def test_render_folder_listing_subfolders_only_no_blank_trailer() -> None:
"""Subfolders but no own files: header + blank + subfolder lines —
no trailing blank line, no file section."""
assert (
agent.render_folder_listing("S", [("a", 1, None)], [], 0)
== "S — 0 documents, 1 folders:\n\n a/ — 1 documents"
)
def test_render_folder_listing_cap_note_only_past_fifty() -> None:
"""The note appears ONLY when the folder's own files outnumber the
cap: 51 → 50 lines + the deterministic grep-pointer note (the
``…and 1 more…`` shape — unpluralized, the house pin); 50 → no
note."""
files51 = [("S", f"f{i:03d}.md", f"T{i}") for i in range(51)]
capped = files51[:50]
rendered = agent.render_folder_listing("S/big", [], capped, 51)
lines = rendered.splitlines()
assert lines[0] == "S/big — 51 documents, 0 folders:"
assert len(lines) == 1 + 1 + 50 + 1 # header, blank, 50 lines, note
assert lines[-1] == (
"…and 1 more documents in this folder — use grep (pattern) to "
"find a specific one."
)
files50 = [("S", f"f{i:03d}.md", f"T{i}") for i in range(50)]
rendered50 = agent.render_folder_listing("S/big", [], files50, 50)
assert rendered50.splitlines()[-1] == "source: S | path: f049.md | title: T49"
assert "more documents" not in rendered50
# ---------- read: the canonical combined source/path form ----------
@@ -771,7 +1134,6 @@ def test_read_bare_source_name_refused_without_db(monkeypatch: pytest.MonkeyPatc
no-document refusal (the argument echoed as passed), no DB lookup
(NOT even the phase-72 candidate lookup — ``all_documents`` must
not run either), nothing counted."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("Homelab", "a.md", "A")])
def _boom(*_a: Any, **_k: Any) -> None:
raise AssertionError(
@@ -1158,7 +1520,6 @@ def test_reading_an_already_read_doc_is_deduped(monkeypatch: pytest.MonkeyPatch)
def test_unknown_tool_name_refused(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="delete_universe", arguments={"x": 1})],
@@ -1702,13 +2063,11 @@ def test_grep_counts_but_never_adds_context(monkeypatch: pytest.MonkeyPatch) ->
def test_relist_executes_and_counts(monkeypatch: pytest.MonkeyPatch) -> None:
"""Re-lists execute — a second ``ls`` in one turn returns the catalog
again and counts in ``tool_calls`` (no budget to exhaust)."""
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "aws-route53.md", "AWS Route53 Records"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
"""Re-lists execute — a second ``ls`` in one turn returns the top
level again and counts in ``tool_calls`` (no budget to exhaust)."""
monkeypatch.setattr(
agent, "ls_top", lambda db: [("Deployments", 1, None), ("Homelab", 1, None)]
)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -1717,12 +2076,8 @@ def test_relist_executes_and_counts(monkeypatch: pytest.MonkeyPatch) -> None:
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 2 # both re-lists executed and counted
listing = (
"2 documents:\n"
"source: Deployments | path: backups.md | title: Backup Strategy\n"
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
)
# The answer request carries the catalog a second time as a tool result.
listing = "2 sources:\n\nDeployments — 1 documents\nHomelab — 1 documents"
# The answer request carries the listing a second time as a tool result.
assert llm.requests[2][0][3]["content"] == listing # first listing
assert llm.requests[2][0][5]["content"] == listing # the re-list
assert llm.requests[2][1] == AGENT_TOOLS # still offered (no budgets)
@@ -1758,8 +2113,8 @@ def test_always_ls_bounded_by_round_cap(monkeypatch: pytest.MonkeyPatch) -> None
"""A model that keeps calling ``ls`` gets exactly
``agent_max_rounds`` tool rounds, then one forced ``tools=None``
request streams the answer — the cap is the only forced exit."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
listing = "1 documents:\nsource: S | path: a.md | title: A"
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
listing = "1 sources:\n\nS — 1 documents"
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -1923,7 +2278,7 @@ def test_round_retried_before_first_piece(
same messages: the stream carries a RetryPiece BEFORE the tool call,
the tool executes, the final answer streams, and the per-call log line
is still emitted exactly once (retries are invisible to the loop)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = FailingLLM(
[
@@ -1961,7 +2316,6 @@ def test_round_failure_after_first_piece_is_terminal(monkeypatch: pytest.MonkeyP
the LLMError propagates out of ``run_agent``, no RetryPiece, no
sleep, no second request, and the holder is untouched (the tool
never ran)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = FailingLLM(
[([StreamPiece("content", "partial ")], LLMError("mid-stream drop"))]
@@ -1994,7 +2348,7 @@ def test_forced_final_no_tools_call_is_retried(monkeypatch: pytest.MonkeyPatch)
"""The forced final request (round cap reached) goes through the same
retry rule: a failure before its first piece yields a RetryPiece and
restarts with ``tools=None``; the answer from the retry streams."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = FailingLLM(
[
@@ -2026,7 +2380,6 @@ def test_zero_retries_is_one_plain_attempt(monkeypatch: pytest.MonkeyPatch) -> N
"""The kill-switch path (``llm_retries=0``): a dead round raises
immediately — one request, no RetryPiece, no sleep (pre-phase-67
behavior)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = FailingLLM([([], LLMError("connection refused"))])
sleeps = _record_sleeps(monkeypatch)
@@ -2065,7 +2418,6 @@ def test_abandon_mid_retry_sleep_leaks_nothing(monkeypatch: pytest.MonkeyPatch)
await asyncio.Event().wait() # park until the abandon arrives
monkeypatch.setattr(asyncio, "sleep", parking_sleep)
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = FailingLLM(
[([], LLMError("endpoint down")), ([StreamPiece("content", "never")], None)]
@@ -2111,7 +2463,7 @@ def test_retries_are_invisible_to_the_round_cap(
2, the retried first round and the second tool round fill the cap —
the forced final follows the SECOND call, and the log lines read
round=1/2 and round=2/2."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = FailingLLM(
[
@@ -2364,7 +2716,7 @@ def test_scaffolding_round_with_tool_calls_needs_no_recovery(
ran, and the policy keys on the no-calls exit only — no recovery (the
next round is a normal tools-offered round carrying the tool
history)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -2398,7 +2750,7 @@ def test_recovery_after_tool_rounds_keeps_the_history(
keeps the SINGLE (folded) system message at the front and the tool
history intact behind it — no second system message, no duplicated
correction."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -2423,7 +2775,7 @@ def test_recovery_after_tool_rounds_keeps_the_history(
assert recovered[3] == {
"role": "tool",
"tool_call_id": "call_1",
"content": "1 documents:\nsource: S | path: a.md | title: A",
"content": "1 sources:\n\nS — 1 documents",
}
assert sum(1 for m in recovered if m["role"] == "system") == 1
assert holder.scaffold_stripped == len(_INCIDENT_SPAN)
@@ -2436,7 +2788,7 @@ def test_forced_final_scaffolding_only_settles_malformed(
scaffolding-only forced answer never reaches the user raw — the turn
settles with :class:`MalformedReplyError` (the same terminal
semantics; this turn used no recovery, so nothing is doubled up)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
tests/unit/test_folder_summaries.py
+628
View File
@@ -0,0 +1,628 @@
"""Unit: folder summary storage + generator (phase 94, task 01).
The prompt/grouping tests are pure (no DB): ``FOLDER_SUMMARY_MODE``
system prompt, the ``folder_of`` / ``group_by_folder`` recursive-subtree
concept, and the user-message cap with the shared ``[…truncated…]``
marker. The generator tests run against the local compose Postgres
(preferred — real upsert/prune on the ``folder_summaries`` table),
skipping with clear instructions when the stack is not up — same
pattern as ``test_overview.py``.
"""
from __future__ import annotations
import asyncio
import logging
import uuid
from datetime import UTC, datetime
from typing import Any
import pytest
from sqlalchemy import text
from sqlalchemy.orm import Session
from app.config import Settings, get_settings
from app.db import SessionLocal
from app.models import Document, FolderSummary
from app.rag.folder_summaries import (
FOLDER_HEADER_PREFIX,
FOLDER_SUMMARY_INSTRUCTION,
FOLDER_SUMMARY_MODE,
MIN_DOCS_PER_FOLDER,
SYSTEM_PROMPT,
build_folder_summary_prompt,
folder_of,
folder_summary_table_empty,
generate_folder_summaries,
group_by_folder,
summarize_folder,
)
from app.rag.llm import LLMError
from app.rag.retriever import TRUNCATION_MARKER
from tests.e2e.mock_llm import compose_answer
REPLY = "Covers lab automation runbooks: inventories, playbooks, and schedules."
class _FakeLLM:
"""Duck-typed stand-in for ``LLMClient`` (``chat`` + ``settings``).
Records each ``(system, user)`` request and the ``model`` kwarg;
returns the canned reply, or raises — either a fixed exception or a
per-folder failure keyed on the user message's ``Folder: …`` header
(the per-folder fail-soft tests).
"""
def __init__(
self,
reply: str = REPLY,
fail_folders: tuple[str, ...] = (),
fail: Exception | None = None,
) -> None:
self._reply = reply
self._fail_folders = tuple(fail_folders)
self._fail = fail
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
self.calls = 0
self.model: str | None = None
self.requests: list[tuple[str, str]] = []
async def chat(
self, messages: list[dict[str, str]], model: str | None = None
) -> str:
self.calls += 1
self.model = model
system = messages[0]["content"]
user = messages[-1]["content"]
self.requests.append((system, user))
for folder in self._fail_folders:
if FOLDER_HEADER_PREFIX + folder in user:
raise LLMError(f"simulated lite-model failure for {folder}")
if self._fail is not None:
raise self._fail
return self._reply
# ---------- folder_of ----------
def test_folder_of_root_level_file_is_empty() -> None:
assert folder_of("a.md") == ""
def test_folder_of_one_level() -> None:
assert folder_of("a/b.md") == "a"
def test_folder_of_deep_path() -> None:
assert folder_of("a/b/c/d.md") == "a/b/c"
def test_folder_of_iterating_walks_prefixes_to_root() -> None:
"""Iterating ``folder_of`` over its own result walks the folder
prefixes nearest-first, ending at the root (the grouping walk)."""
folder = folder_of("a/b/c.md")
chain: list[str] = []
while folder:
chain.append(folder)
folder = folder_of(folder)
assert chain == ["a/b", "a"] # plus the "" root the grouping adds
# ---------- group_by_folder ----------
def test_group_by_folder_root_file_lands_only_in_source_root() -> None:
rows = [("S", "top.md", "T", None)]
groups = group_by_folder(rows)
assert set(groups) == {("S", "")}
assert groups[("S", "")] == rows
def test_group_by_folder_nested_multi_source_recursive_subtree() -> None:
"""A doc under ``a/b/`` is present in the ``a``, ``a/b``, and ``""``
groups (recursive subtree — the ``ls`` count scope, one concept);
per source the candidates are ``""`` + every distinct folder
prefix; group lists keep the input (catalogue) order."""
rows = [
("S", "a/b/c.md", "C", None),
("S", "a/b/d.md", "D", None),
("S", "a/x.md", "X", None),
("S", "top.md", "T", None),
("T", "a/b/e.md", "E", None),
]
groups = group_by_folder(rows)
assert set(groups) == {
("S", ""),
("S", "a"),
("S", "a/b"),
("T", ""),
("T", "a"),
("T", "a/b"),
}
# The recursive-subtree concept: a/b/ docs in the a/, a/b/, and ""
# groups alike — exactly the set each level's ls count shows.
assert [r[1] for r in groups[("S", "a/b")]] == ["a/b/c.md", "a/b/d.md"]
assert [r[1] for r in groups[("S", "a")]] == ["a/b/c.md", "a/b/d.md", "a/x.md"]
assert [r[1] for r in groups[("S", "")]] == [
"a/b/c.md",
"a/b/d.md",
"a/x.md",
"top.md",
]
# Multi-source: the same folder prefix under another source is a
# separate group (PK is (source, folder_path)).
assert [r[1] for r in groups[("T", "a/b")]] == ["a/b/e.md"]
assert [r[1] for r in groups[("T", "")]] == ["a/b/e.md"]
# Input (catalogue) order is preserved inside each group.
assert [r[2] for r in groups[("S", "")]] == ["C", "D", "X", "T"]
def test_group_by_folder_single_doc_folder_is_a_group_too() -> None:
"""Grouping is pure subtree membership (≥ 1 docs): the ≥ 2 rule is
the GENERATOR's (the recursive count below the minimum yields no
row — pinned by the generator tests, not the grouping)."""
rows = [("S", "a/only.md", "O", None)]
groups = group_by_folder(rows)
assert len(groups[("S", "a")]) == 1 # present, but below the minimum
def test_group_by_folder_doc_path_equal_to_a_folder_prefix_counts_for_it() -> None:
"""The count rule's ``path == folder`` arm: a document whose path
IS one of the source's folder prefixes (a file sharing its name
with a directory) belongs to that folder's group too — the grouping
stays EXACTLY the set the ``ls`` count rule counts (path equal or
starting with ``folder + "/"``), while the returned keys remain the
true folder prefixes only (no file-path keys)."""
rows = [
("S", "a/b", "B", None), # a file named "b" ... (its path is a folder prefix)
("S", "a/b/c.md", "C", None), # ... and a real folder "a/b/" holding a doc
("S", "a/x.md", "X", None),
]
groups = group_by_folder(rows)
assert set(groups) == {("S", ""), ("S", "a"), ("S", "a/b")}, (
"the keys stay the true folder prefixes — the file's own path adds no key"
)
assert [r[1] for r in groups[("S", "a/b")]] == ["a/b", "a/b/c.md"]
assert [r[1] for r in groups[("S", "a")]] == ["a/b", "a/b/c.md", "a/x.md"]
assert [r[1] for r in groups[("S", "")]] == ["a/b", "a/b/c.md", "a/x.md"]
def test_group_by_folder_plain_file_path_is_not_a_group_key() -> None:
"""A file path that is NO folder prefix (no doc under it) adds no
group key of its own — the ``path == folder`` arm only fires when
the path really is a prefix of the catalogue."""
rows = [("S", "top.md", "T", None), ("S", "a/one.md", "O", None)]
groups = group_by_folder(rows)
assert set(groups) == {("S", ""), ("S", "a")}
assert "top.md" not in [folder for _source, folder in groups]
# ---------- build_folder_summary_prompt: system ----------
def test_system_prompt_has_marker_and_locked_instruction() -> None:
assert SYSTEM_PROMPT.startswith(FOLDER_SUMMARY_MODE)
assert FOLDER_SUMMARY_INSTRUCTION in SYSTEM_PROMPT
for fragment in (
"1-3 sentence",
"plain-text summary",
"natural language",
"Do not use markdown",
"not in the list",
):
assert fragment in SYSTEM_PROMPT
system, _ = build_folder_summary_prompt("S", "a/b", [])
assert system == SYSTEM_PROMPT
assert FOLDER_SUMMARY_MODE in system # the marker the E2E mock keys on
# ---------- build_folder_summary_prompt: user ----------
def test_user_prompt_header_names_the_folder() -> None:
"""The first line is the ``FOLDER_HEADER_PREFIX`` header the E2E
mock parses: ``<source>`` for the root, ``<source>/<folder_path>``
for a folder."""
_, user = build_folder_summary_prompt("Homelab", "deployments/ansible", [])
assert user == FOLDER_HEADER_PREFIX + "Homelab/deployments/ansible"
_, user = build_folder_summary_prompt("Homelab", "", [])
assert user == FOLDER_HEADER_PREFIX + "Homelab"
def test_user_lines_carry_path_title_and_first_summary_line() -> None:
docs = [
("S", "a/b/one.md", "One", "First lead.\nSecond line.\nSource: S/a/b/one.md"),
("S", "a/b/two.md", "Two", None),
]
system, user = build_folder_summary_prompt("S", "a/b", docs)
assert system == SYSTEM_PROMPT
assert user == (
"Folder: S/a/b\n"
"a/b/one.md — One — First lead.\n"
"a/b/two.md — Two"
)
def test_user_line_omits_summary_field_when_absent_or_blank() -> None:
docs = [
("S", "a/x.md", "X", None),
("S", "a/y.md", "Y", " \n\t "),
]
_, user = build_folder_summary_prompt("S", "a", docs)
assert user == "Folder: S/a\na/x.md — X\na/y.md — Y"
assert " — " in user # the path — title join only
assert not any(line.endswith(" — ") for line in user.splitlines())
def test_user_line_uses_only_first_summary_line() -> None:
docs = [
("S", "a/x.md", "X", "First line.\nSecond line.\nSource: S/a/x.md"),
]
_, user = build_folder_summary_prompt("S", "a", docs)
assert user == "Folder: S/a\na/x.md — X — First line."
assert "Second line" not in user
assert "Source:" not in user
def test_user_prompt_truncated_with_marker_when_over_custom_cap() -> None:
docs = [("S", f"a/f{i}.md", f"T{i}", None) for i in range(10)]
_, full = build_folder_summary_prompt("S", "a", docs, max_chars=10_000)
cap = 30
_, user = build_folder_summary_prompt("S", "a", docs, max_chars=cap)
assert user == full[:cap] + "\n" + TRUNCATION_MARKER
assert user.endswith(TRUNCATION_MARKER)
assert len(user) > cap # the marker makes the cut visible past the cap
def test_user_prompt_at_exact_cap_not_truncated() -> None:
docs = [("S", "a/x.md", "X", None)] # "Folder: S/a\na/x.md — X" = 22 chars
_, user = build_folder_summary_prompt("S", "a", docs, max_chars=22)
assert user == "Folder: S/a\na/x.md — X"
assert TRUNCATION_MARKER not in user
def test_user_prompt_truncated_at_default_cap() -> None:
"""No explicit cap → ``BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS`` (read
from the live settings, so the test holds for any configured
value)."""
cap = get_settings().folder_summary_input_max_chars
docs = [("S", f"a/f{i}.md", "T", None) for i in range(3_000)]
_, user = build_folder_summary_prompt("S", "a", docs)
assert user.endswith(TRUNCATION_MARKER)
body = user.removesuffix("\n" + TRUNCATION_MARKER)
assert len(body) == cap # cut exactly at the cap, marker on its own line
assert "f2999.md" not in body # the overflow never reaches the model
# ---------- summarize_folder ----------
# ---------- the E2E mock's FOLDER_SUMMARY_MODE branch ----------
def _mock_body(system: str, user: str) -> dict[str, Any]:
"""A minimal chat-completion body for the mock's ``compose_answer``."""
return {"messages": [
{"role": "system", "content": system},
{"role": "user", "content": user},
]}
def test_mock_returns_canned_folder_summary_naming_the_folder() -> None:
"""The deterministic E2E mock keys on the ``FOLDER_SUMMARY_MODE``
marker in the system prompt and returns the canned one-liner naming
the folder from the ``Folder: …`` header — driven through the
GENERATOR's real prompt, so the two can never drift (the drill-down
E2E asserts on this exact template)."""
system, user = build_folder_summary_prompt(
"Homelab", "deployments/ansible",
[("Homelab", "deployments/ansible/lab-inventory.md", "Lab Inventory", None)],
)
assert compose_answer(_mock_body(system, user)) == (
"Fixture folder summary for Homelab/deployments/ansible."
)
# The source-root row names the source itself.
system, user = build_folder_summary_prompt("Homelab", "",
[("Homelab", "top.md", "Top", None)])
assert compose_answer(_mock_body(system, user)) == (
"Fixture folder summary for Homelab."
)
def test_mock_folder_marker_is_not_shadowed_by_the_summary_branch() -> None:
"""``FOLDER_SUMMARY_MODE`` contains ``SUMMARY_MODE`` as a substring —
the mock must check the folder branch FIRST, or every folder call
would land in the document-summary digest (regression pin)."""
system, user = build_folder_summary_prompt(
"S", "a", [("S", "a/x.md", "X", None)]
)
assert "SUMMARY_MODE" in system # the shadowing hazard is real
answer = compose_answer(_mock_body(system, user))
assert answer == "Fixture folder summary for S/a."
assert not answer.startswith("This document covers")
def test_summarize_folder_happy_path_returns_trimmed_text() -> None:
docs = [("S", "a/x.md", "X", None)]
llm = _FakeLLM(reply=f" {REPLY} \n")
out = asyncio.run(summarize_folder("S", "a", docs, llm))
assert out == REPLY # the model's text, trimmed
assert llm.calls == 1
def test_summarize_folder_calls_the_configured_summary_model_with_marker() -> None:
docs = [("S", "a/x.md", "X", "X lead.")]
llm = _FakeLLM()
asyncio.run(summarize_folder("S", "a", docs, llm))
assert llm.model == llm.settings.llm_summary_model # the ``lite`` default
assert llm.model == "lite"
system, user = llm.requests[0]
assert FOLDER_SUMMARY_MODE in system
assert user.startswith(FOLDER_HEADER_PREFIX + "S/a")
assert "a/x.md — X — X lead." in user
def test_summarize_folder_empty_reply_raises_llm_error() -> None:
docs = [("S", "a/x.md", "X", None)]
for reply in ("", " \n\t "):
llm = _FakeLLM(reply=reply)
with pytest.raises(LLMError, match="empty content for S/a"):
asyncio.run(summarize_folder("S", "a", docs, llm))
def test_summarize_folder_error_propagates() -> None:
docs = [("S", "a/x.md", "X", None)]
llm = _FakeLLM(fail=LLMError("simulated transport failure"))
with pytest.raises(LLMError, match="simulated transport failure"):
asyncio.run(summarize_folder("S", "a", docs, llm))
# ---------- generate_folder_summaries (real Postgres) ----------
def _add_doc(
db: Session, source: str, path: str, title: str, summary: str | None = None
) -> Document:
doc = Document(
id=uuid.uuid4(),
source=source,
path=path,
full_path=f"/tmp/{path}",
title=title,
content="body",
content_hash="0" * 64,
summary=summary,
)
db.add(doc)
db.commit()
return doc
def _truncate(db: Session) -> None:
db.execute(text("TRUNCATE chunks, documents"))
db.execute(text("DELETE FROM folder_summaries"))
db.commit()
def _rows(db: Session) -> dict[tuple[str, str], str]:
"""The stored folder summaries: ``{(source, folder_path): summary}``."""
result = db.execute(
text("SELECT source, folder_path, summary FROM folder_summaries")
).all()
return {(source, folder_path): summary for source, folder_path, summary in result}
def _seed_catalogue(db: Session) -> None:
"""The shared catalogue: FSU has four docs in three candidate
folders (root 4, a 3, a/b 2 — all ≥ the minimum); FSU-solo has one
doc (its root folder is below the minimum — no row, no call)."""
_add_doc(db, "FSU", "a/b/one.md", "One", "One lead.\nSource: FSU/a/b/one.md")
_add_doc(db, "FSU", "a/b/two.md", "Two")
_add_doc(db, "FSU", "a/three.md", "Three")
_add_doc(db, "FSU", "root.md", "Root")
_add_doc(db, "FSU-solo", "solo.md", "Solo")
@pytest.fixture()
def clean_tables(db: Session):
_truncate(db)
yield
_truncate(db)
def test_generate_happy_path_upserts_every_candidate_folder(
db: Session, clean_tables, caplog: pytest.LogCaptureFixture
) -> None:
"""Every folder with ≥ 2 recursive docs gets a row (the source root
row included — ``folder_path = ''``); single-doc folders get none;
rows are stamped fresh; the stats dict and the log line are right;
folders are processed in deterministic (source, folder_path) order."""
_seed_catalogue(db)
llm = _FakeLLM()
with caplog.at_level(logging.INFO, logger="app.rag.folder_summaries"):
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 3, "failed": 0, "pruned": 0}
assert llm.calls == 3, "one lite call per candidate folder (the solo folder: none)"
stored = _rows(db)
assert set(stored) == {("FSU", ""), ("FSU", "a"), ("FSU", "a/b")}
assert all(summary == REPLY for summary in stored.values())
assert ("FSU-solo", "") not in stored, (
"a single-doc folder is fully described by its one file line — no row"
)
row = db.get(FolderSummary, ("FSU", "a/b"))
assert row is not None
assert row.summary == REPLY
assert row.updated_at is not None
age = datetime.now(UTC) - row.updated_at
assert age.total_seconds() < 300, "updated_at must be a fresh UTC timestamp"
# Deterministic (source, folder_path) order — root before the
# nested folders, one header per call.
assert [user.splitlines()[0] for _s, user in llm.requests] == [
"Folder: FSU",
"Folder: FSU/a",
"Folder: FSU/a/b",
]
# The recursive-subtree input: the a/ prompt carries a/b's docs too.
a_prompt = llm.requests[1][1]
assert "a/b/one.md — One — One lead." in a_prompt
assert "a/three.md — Three" in a_prompt
assert "root.md — Root" not in a_prompt
assert (
"folder_summaries: generated=3 failed=0 pruned=0" in caplog.text
), "the stats line must be greppable (PLAN §9 ample logging)"
def test_generate_per_folder_fail_soft_keeps_previous_and_lands_others(
db: Session, clean_tables, caplog: pytest.LogCaptureFixture
) -> None:
"""One folder's lite failure is logged and counted, its PREVIOUS
row is kept (an old summary is better than none), and the remaining
folders still land — a lite outage never fails the sync."""
_seed_catalogue(db)
db.add(FolderSummary(source="FSU", folder_path="a/b", summary="old summary"))
db.commit()
llm = _FakeLLM(fail_folders=("FSU/a/b",))
with caplog.at_level(logging.ERROR, logger="app.rag.folder_summaries"):
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 2, "failed": 1, "pruned": 0}
assert llm.calls == 3 # the failing folder was attempted too
stored = _rows(db)
assert stored[("FSU", "a/b")] == "old summary", (
"the previous row survives the per-folder failure"
)
assert stored[("FSU", "")] == REPLY and stored[("FSU", "a")] == REPLY, (
"the other folders still land"
)
assert "folder summary failed for FSU/a/b" in caplog.text
assert "simulated lite-model failure for FSU/a/b" in caplog.text
def test_generate_per_folder_fail_soft_without_previous_row_creates_nothing(
db: Session, clean_tables
) -> None:
_seed_catalogue(db)
llm = _FakeLLM(fail_folders=("FSU/a/b",))
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats["failed"] == 1
stored = _rows(db)
assert ("FSU", "a/b") not in stored, "no row must be invented for a failed folder"
assert ("FSU", "") in stored and ("FSU", "a") in stored
def test_generate_prunes_stale_rows_and_keeps_live_ones(db: Session, clean_tables) -> None:
"""Rows for folders that dropped below 2 recursive docs are deleted
(pruned/renamed — the summary would go stale); rows for folders
that still qualify persist (an unchanged folder's summary is still
true — regenerated in place)."""
_seed_catalogue(db)
# A stale row for a folder no longer in the catalogue (3→1 docs /
# renamed away) + a live row with old content.
db.add(FolderSummary(source="FSU", folder_path="gone/old", summary="stale"))
db.add(FolderSummary(source="FSU", folder_path="a", summary="old a summary"))
db.add(FolderSummary(source="FSU-solo", folder_path="", summary="solo stale"))
db.commit()
stats = asyncio.run(generate_folder_summaries(db, _FakeLLM()))
assert stats["pruned"] == 2 # gone/old + the FSU-solo root (1 doc)
stored = _rows(db)
assert ("FSU", "gone/old") not in stored, "the stale folder row must be pruned"
assert ("FSU-solo", "") not in stored, (
"a folder that dropped below 2 docs loses its row"
)
assert ("FSU", "a") in stored, "the still-qualifying folder keeps its row"
assert stored[("FSU", "a")] == REPLY # regenerated, not stale
assert stored[("FSU", "")] == REPLY and stored[("FSU", "a/b")] == REPLY
def test_generate_skip_is_a_full_noop(db: Session, clean_tables) -> None:
"""``skip=True`` (the ``--limit`` debug run): the LLM is never
called, no rows are touched, zero stats."""
_seed_catalogue(db)
db.add(FolderSummary(source="FSU", folder_path="", summary="existing"))
db.commit()
llm = _FakeLLM()
stats = asyncio.run(generate_folder_summaries(db, llm, skip=True))
assert stats == {"generated": 0, "failed": 0, "pruned": 0}
assert llm.calls == 0
assert _rows(db) == {("FSU", ""): "existing"}
def test_generate_empty_kb_prunes_every_row(db: Session, clean_tables) -> None:
"""No documents → no candidate folders → every stored row is
pruned, with zero wasted lite calls."""
db.add(FolderSummary(source="FSU", folder_path="", summary="old"))
db.add(FolderSummary(source="FSU", folder_path="a/b", summary="old"))
db.commit()
llm = _FakeLLM()
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 0, "failed": 0, "pruned": 2}
assert llm.calls == 0
assert _rows(db) == {}
def test_generate_summarizes_folder_whose_prefix_is_also_a_doc_path(
db: Session, clean_tables
) -> None:
"""The ``path == folder`` arm end to end: a file sharing its name
with a directory counts toward the folder's recursive count (2
docs → the folder is summarized, and BOTH docs are in its prompt).
"""
_add_doc(db, "FSU", "a/b", "B") # a file named "b" (its path is a prefix)
_add_doc(db, "FSU", "a/b/c.md", "C") # and a real folder "a/b/"
llm = _FakeLLM()
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats["generated"] == 3 # root (2), a (2), a/b (2) — all ≥ the minimum
stored = _rows(db)
assert set(stored) == {("FSU", ""), ("FSU", "a"), ("FSU", "a/b")}
a_b_prompt = [
user for _system, user in llm.requests if user.startswith("Folder: FSU/a/b\n")
][0]
assert "a/b — B" in a_b_prompt
assert "a/b/c.md — C" in a_b_prompt
def test_generate_only_flushes_caller_commits(db: Session, clean_tables) -> None:
"""The generator only flushes — the sync path owns the transaction
(the phase-53 ``bump_sources_version`` convention): the catalogue
is committed (the real sync path commits the import before the
summary hooks run), but a second session sees the generator's rows
as NOTHING until the CALLER commits — and sees them after."""
_add_doc(db, "FSU", "x/y/one.md", "One")
_add_doc(db, "FSU", "x/y/two.md", "Two")
stats = asyncio.run(generate_folder_summaries(db, _FakeLLM()))
assert stats["generated"] == 3 # root + x + x/y — all 2 recursive docs
with SessionLocal() as other:
n = other.scalar(
text("SELECT count(*) FROM folder_summaries WHERE source = 'FSU'")
)
assert n == 0, "unflushed-by-caller rows must not be visible yet"
db.commit()
with SessionLocal() as other:
n = other.scalar(
text("SELECT count(*) FROM folder_summaries WHERE source = 'FSU'")
)
assert n == 3, "the caller's commit makes the flushed rows durable"
assert MIN_DOCS_PER_FOLDER == 2 # the ≥ 2 scope rule, pinned by name
def test_folder_summary_table_empty_gate(db: Session, clean_tables) -> None:
"""The sync-path gate probe (phase 94, task 02): empty → True
(the first full sync after migration 0017 must still generate),
one row → False (a populated table waits for a KB change)."""
assert folder_summary_table_empty(db) is True # the truncated table
_add_doc(db, "FSU", "a/one.md", "One")
_add_doc(db, "FSU", "a/two.md", "Two")
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
assert folder_summary_table_empty(db) is False # rows landed
db.execute(text("DELETE FROM folder_summaries"))
db.commit()
assert folder_summary_table_empty(db) is True # emptied again
tests/unit/test_mock_tool_flow.py
+124 -44
View File
@@ -4,12 +4,18 @@ The mock (``tests/e2e/mock_llm.py``) classifies marker requests
statelessly into one step of the agent tool flow. This file pins the
classification at unit speed — no Playwright, no LLM process:
* the phase-37 SINGLE-READ flow (``TOOLS_TRIGGER`` only) stays
byte-identical: list → read (first catalog line, ``call_1``) → answer;
* the phase-45 MULTI-READ flow (``TOOLS_TRIGGER`` + ``MULTI_READ_TRIGGER``)
classifies by the count of ``tool``-role read results: list → read #1
(``call_1``) → read #2 (second catalog line, ``call_2``) → the
byte-stable ``multi_answer`` naming both read paths.
* the phase-37/94 SINGLE-READ flow (``TOOLS_TRIGGER`` only): list (the
top-level source listing) → drill (``ls`` scoped to the first source
— phase 94: the top level carries sources only, so the flow drills
one level for the file lines) → read (first file line, the combined
``source/path``) → answer;
* the phase-45 MULTI-READ flow (``TOOLS_TRIGGER`` +
``MULTI_READ_TRIGGER``) classifies by the count of ``tool``-role read
results: list → drill → read #1 (first file line) → read #2 (second
file line) → the byte-stable ``multi_answer`` naming both read paths;
* the degenerate empty-KB case: every listed source already drilled
with no file lines → the flow falls back to the re-list loop (the
round cap settles it — the phase-70 empty-catalog behavior).
"""
from __future__ import annotations
@@ -36,28 +42,50 @@ SYSTEM_LOW = "<relevance>LOW</relevance>\n"
#: the phase-70 harness-aligned names).
TOOLS = [{"type": "function", "function": {"name": "ls"}}]
#: The agent's ``ls`` output for a two-document KB
#: (``app/rag/agent.py`` ``_execute_tool``): one
#: ``source: X | path: Y | title: Z`` line per document (phase 63: labeled,
#: unambiguous fields), ``(source, path)`` order.
CATALOG_2 = (
"2 documents:\n"
"source: Deployments | path: example-record-file.json | title: Example Record File\n"
"source: Homelab | path: aws-route53.md | title: AWS Route 53 Notes"
#: The agent's drill-down ``ls`` output for a two-source KB
#: (``app/rag/agent.py`` ``render_ls_top`` / ``render_folder_listing``,
#: phase 94): the top level lists the registered sources (registry
#: order, recursive counts — no file lines); the folder level carries
#: the file lines (``source: X | path: Y | title: Z`` — the phase-63
#: labeled fields, unchanged), ``path`` order.
TOP_LEVEL_2 = (
"2 sources:\n"
"\n"
"Deployments — 1 documents\n"
"Homelab — 1 documents"
)
CATALOG_1 = (
"1 documents:\n"
#: The first source's root folder: one file line (the single-read
#: flow's read target).
FOLDER_DEPLOYMENTS = (
"Deployments — 1 documents, 0 folders:\n"
"\n"
"source: Deployments | path: example-record-file.json | title: Example Record File"
)
CATALOG_3 = (
"3 documents:\n"
#: Two file lines in the first source (the multi-read flow's reads).
FOLDER_DEPLOYMENTS_2 = (
"Deployments — 2 documents, 0 folders:\n"
"\n"
"source: Deployments | path: aaa.md | title: AAA\n"
"source: Deployments | path: bbb.md | title: BBB"
)
#: Three file lines in the first source (the listing-order pin: read
#: #2 is the SECOND line, not the last).
FOLDER_DEPLOYMENTS_3 = (
"Deployments — 3 documents, 0 folders:\n"
"\n"
"source: Deployments | path: aaa.md | title: AAA\n"
"source: Deployments | path: bbb.md | title: BBB\n"
"source: Homelab | path: ccc.md | title: CCC"
"source: Deployments | path: ccc.md | title: CCC"
)
#: Empty folder levels (a registered source with no documents — the
#: header line alone; the drill's degenerate arm).
FOLDER_DEPLOYMENTS_EMPTY = "Deployments — 0 documents, 0 folders:"
FOLDER_HOMELAB_EMPTY = "Homelab — 0 documents, 0 folders:"
DOC1_SP = "Deployments/example-record-file.json"
DOC1_CONTENT = (
"The record file keeps every hosted zone record — first line is longer "
@@ -122,10 +150,39 @@ def test_single_flow_list_step() -> None:
assert _tool_flow(_body(SINGLE_USER)) == ("list", "", "")
def test_single_flow_read_step_first_catalog_line() -> None:
flow = _tool_flow(_body(SINGLE_USER, (CATALOG_3,)))
# The FIRST listing line (Deployments/aaa.md), labeled fields.
assert flow == ("read", "Deployments", "aaa.md", "call_1")
def test_single_flow_drill_step_after_top_level() -> None:
# Phase 94: the top level lists SOURCES only — the flow drills one
# level into the FIRST source (listing order = registry order).
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2,)))
assert flow == ("drill", "Deployments", "call_1")
def test_single_flow_drill_skips_already_drilled_source() -> None:
# The first source's folder level is already in the messages (an
# empty listing — header only, no file lines): the drill proceeds
# to the NEXT un-drilled source.
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS_EMPTY)))
assert flow == ("drill", "Homelab", "call_1")
def test_single_flow_all_sources_drilled_empty_falls_back_to_list() -> None:
# Degenerate: every listed source already drilled, no file lines
# anywhere — the flow falls back to the re-list loop (settled at the
# round cap, the phase-70 empty-catalog behavior).
flow = _tool_flow(
_body(
SINGLE_USER,
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS_EMPTY, FOLDER_HOMELAB_EMPTY),
)
)
assert flow == ("list", "", "")
def test_single_flow_read_step_first_file_line() -> None:
# The folder level reached: the FIRST file line (Deployments/aaa.md),
# labeled fields, the combined ``source/path`` join.
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS_2)))
assert flow == ("read", "Deployments", "aaa.md", "call_2")
def test_read_step_nested_path_stays_intact() -> None:
@@ -133,19 +190,20 @@ def test_read_step_nested_path_stays_intact() -> None:
# ``source/path — title`` + ``rpartition("/")`` parse misread the
# split (``source=brain-of-reese-main/homelab``). The labeled fields
# recover the nested path intact, however deep.
catalog = (
"1 documents:\n"
listing = (
"brain-of-reese-main — 1 documents, 0 folders:\n"
"\n"
"source: brain-of-reese-main | path: homelab/aws-route53.md | title: aws-route53"
)
flow = _tool_flow(_body(SINGLE_USER, (catalog,)))
assert flow == ("read", "brain-of-reese-main", "homelab/aws-route53.md", "call_1")
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2, listing)))
assert flow == ("read", "brain-of-reese-main", "homelab/aws-route53.md", "call_2")
def test_single_flow_answer_step_with_tools_offered() -> None:
# Phase 45: the round cap keeps the tools offered until it is hit —
# the answer step fires regardless of the ``tools`` parameter.
flow = _tool_flow(
_body(SINGLE_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT)))
_body(SINGLE_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)))
)
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
@@ -154,7 +212,7 @@ def test_single_flow_answer_step_without_tools() -> None:
flow = _tool_flow(
_body(
SINGLE_USER,
(CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT)),
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)),
tools=None,
)
)
@@ -184,22 +242,39 @@ def test_multi_flow_list_step() -> None:
assert _tool_flow(_body(MULTI_USER)) == ("list", "", "")
def test_multi_flow_drill_step_after_top_level() -> None:
# Phase 94: the top level lists SOURCES only — the multi flow drills
# too, before its first read.
flow = _tool_flow(_body(MULTI_USER, (TOP_LEVEL_2,)))
assert flow == ("drill", "Deployments", "call_1")
def test_multi_flow_read_first_step() -> None:
flow = _tool_flow(_body(MULTI_USER, (CATALOG_2,)))
assert flow == ("read", DOC1_SP.split("/", 1)[0], DOC1_SP.rsplit("/", 1)[1], "call_1")
flow = _tool_flow(_body(MULTI_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS)))
assert flow == ("read", DOC1_SP.split("/", 1)[0], DOC1_SP.rsplit("/", 1)[1], "call_2")
def test_multi_flow_read_second_step_skips_already_read() -> None:
flow = _tool_flow(_body(MULTI_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT))))
# The second catalog line — the first line differing from DOC1.
assert flow == ("read", "Homelab", "aws-route53.md", "call_2")
flow = _tool_flow(
_body(
MULTI_USER,
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS_2, _read_result("Deployments/aaa.md", DOC1_CONTENT)),
)
)
# The second file line — the first line differing from the read doc.
assert flow == ("read", "Deployments", "bbb.md", "call_3")
def test_multi_flow_read_second_is_listing_order_not_last() -> None:
# Three-doc catalog, first doc read: read #2 is the SECOND line
# Three-file listing, first file read: read #2 is the SECOND line
# (Deployments/bbb.md), not the last one.
flow = _tool_flow(_body(MULTI_USER, (CATALOG_3, _read_result("Deployments/aaa.md", "x"))))
assert flow == ("read", "Deployments", "bbb.md", "call_2")
flow = _tool_flow(
_body(
MULTI_USER,
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS_3, _read_result("Deployments/aaa.md", "x")),
)
)
assert flow == ("read", "Deployments", "bbb.md", "call_3")
def test_multi_flow_answer_step_names_both_paths() -> None:
@@ -207,7 +282,8 @@ def test_multi_flow_answer_step_names_both_paths() -> None:
_body(
MULTI_USER,
(
CATALOG_2,
TOP_LEVEL_2,
FOLDER_DEPLOYMENTS_2,
_read_result(DOC1_SP, DOC1_CONTENT),
_read_result(DOC2_SP, DOC2_CONTENT),
),
@@ -227,7 +303,8 @@ def test_multi_flow_answer_step_without_tools_offered() -> None:
_body(
MULTI_USER,
(
CATALOG_2,
TOP_LEVEL_2,
FOLDER_DEPLOYMENTS_2,
_read_result(DOC1_SP, DOC1_CONTENT),
_read_result(DOC2_SP, DOC2_CONTENT),
),
@@ -238,11 +315,11 @@ def test_multi_flow_answer_step_without_tools_offered() -> None:
assert flow[0] == "multi_answer"
def test_multi_flow_one_document_catalog_degenerates_to_single_answer() -> None:
def test_multi_flow_one_file_listing_degenerates_to_single_answer() -> None:
# Nothing second to read — the single-read answer shape, quoting the
# only read result.
flow = _tool_flow(
_body(MULTI_USER, (CATALOG_1, _read_result(DOC1_SP, DOC1_CONTENT)))
_body(MULTI_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)))
)
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
@@ -320,9 +397,12 @@ def test_search_flow_found_step_without_tools_offered() -> None:
def test_search_flow_ignores_catalog_and_read_results() -> None:
# A catalog (labeled lines) and a read result ("Document …" prefix)
# are NOT search results — the flow stays at the search step.
flow = _search_flow(_body(SEARCH_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT))))
# Listings (top-level + folder level, labeled file lines) and a read
# result ("Document …" prefix) are NOT search results — the flow
# stays at the search step.
flow = _search_flow(
_body(SEARCH_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)))
)
assert flow == ("search",)
tests/unit/test_prompts.py
+30 -22
View File
@@ -11,11 +11,15 @@ And the phase-71 deflection plain-text line (owner-permitted
line; the ``DEFLECT_MODE`` marker-keying contract is unchanged and
the line never leaks into the HIGH prompt.
And the phase-72 ``<tools>`` copy: the document-identity contract is
stated up front (the ``ls`` source-name scope, the combined
``source/path`` identity for ``read``/``grep``) — the same contract
the teaching refusals in :mod:`app.rag.agent` re-state; the
``<tools>`` marker keying (HIGH only) is unchanged.
And the ``<tools>`` copy (phase 72: the document-identity contract
stated up front — the combined ``source/path`` identity for
``read``/``grep``; phase 94, task 03: the ``ls`` clause rewritten to
the drill-down tree contract — one level per call, sources at the
top, folders + files below, ``grep`` as the without-listing locator —
while the ``read``/``grep`` clauses and the discipline rules are
byte-identical): the teaching refusals in :mod:`app.rag.agent`
re-state the same contract; the ``<tools>`` marker keying (HIGH
only) is unchanged.
"""
from __future__ import annotations
@@ -194,20 +198,22 @@ pinned byte-for-byte in
def test_tools_section_phase72_contract_clauses() -> None:
"""Phase 72: the two contract clauses the teaching refusals
"""Phase 72 + phase 94: the contract clauses the teaching refusals
re-state after the fact, pinned byte-for-byte in the constant —
the ``ls`` source-name clause (its optional ``path`` is a source
name, not a directory or file path; omit it to list every
document) and the ``read``/``grep`` combined-identity clause
(the combined ``source/path`` string exactly as shown in the
``ls`` output, *including the source name*; a bare document path
will not resolve)."""
# The ls source-name clause.
assert (
"a source name (e.g. 'homelab'), not a directory or file "
"path — omit it to list every document"
) in TOOLS_SECTION
# The read combined-identity clause.
the ``ls`` clause (phase 94: the drill-down tree contract — one
level per call, sources at the top, folders + files below, never
the whole KB in one call, ``grep`` as the without-listing locator)
and the ``read``/``grep`` combined-identity clause (the combined
``source/path`` string exactly as shown in the ``ls`` output,
*including the source name*; a bare document path will not
resolve)."""
# The ls drill-down clauses (phase 94, task 03).
assert "one level at a time" in TOOLS_SECTION
assert "lists every synced source with its document count" in TOOLS_SECTION
assert "that source's top-level folders and files" in TOOLS_SECTION
assert "never the whole knowledge base in one call" in TOOLS_SECTION
assert "to find one specific document without listing, use `grep`" in TOOLS_SECTION
# The read combined-identity clause (byte-identical across phases).
assert (
"combined `source/path` string, exactly as shown in the `ls` "
"output — including the source name"
@@ -217,23 +223,25 @@ def test_tools_section_phase72_contract_clauses() -> None:
"a bare document path (without the source name) will not resolve"
) == 2
# The pre-phase-70 scope wording is gone — replaced by the
# explicit source-name contract.
# explicit source-name contract (and the phase-72 source-name-only
# clause by the phase-94 drill-down contract).
assert "pass a source name as `path`" not in TOOLS_SECTION
assert "not a directory or file path" not in TOOLS_SECTION
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
"""Phase 72: the contract clauses ride the HIGH prompt with the
"""Phase 72/94: the contract clauses ride the HIGH prompt with the
rest of the section and never leak into the LOW/deflection prompt
(whose byte-identity is pinned in
:func:`test_zero_note_prompt_is_byte_identical_to_pre_steering`)."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
high = build_high_prompt([doc])
assert "<tools>" in high
assert "not a directory or file path" in high
assert "one level at a time" in high # the phase-94 ls clause
assert "including the source name" in high
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
assert "<tools>" not in low
assert "not a directory or file path" not in low
assert "one level at a time" not in low
assert "including the source name" not in low
tests/unit/test_sync_button.py
+10 -2
View File
@@ -752,8 +752,9 @@ def _patch_sync_seams(
"""The runner's seams, monkeypatched on ``app.api.sync`` (the house
mock-import pattern): fresh settings (no ``.env`` leak), a no-op
model probe, a sentinel LLM client, one git row, the gated clone +
import, a no-op overview, and the DB-free sources-version step
(dummy session + pinned counters)."""
import, a no-op overview, a no-op folder-summary step (phase 94 —
the sentinel LLM client has no ``chat``), and the DB-free
sources-version step (dummy session + pinned counters)."""
monkeypatch.setattr(
sync_api,
"get_settings",
@@ -783,6 +784,13 @@ def _patch_sync_seams(
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
async def fake_folder_summaries(
db: object, llm: object, *, skip: bool = False
) -> dict[str, int]:
return {"generated": 0, "failed": 0, "pruned": 0}
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folder_summaries)
class _DummySession:
def close(self) -> None:
pass