feat(rag): agent document tools — list/read tools with env-tuned budgets, SSE tool events + "calling tool" UI
Grounded chat turns now run the agent loop (app/rag/agent.py) instead
of a bare chat_stream: while the per-turn budgets last
(BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each) the model
gets list_documents (the indexed catalog, /api/docs order) and
read_document (full text, never truncated — A7-revised contract); once
both budgets are spent the tools key is dropped from the request and
the model must answer. Rejected calls (unknown tool, unknown/missing
path, document already in context, spent budget) consume no budget.
Budgets 0/0 make exactly one tools=None request — byte-identical to
the pre-phase path (budgets-as-kill-switch). Deflected turns keep the
direct chat_stream (A8 unchanged; the LOW prompt never carries the
<tools> section).
SSE contract gains {"type":"tool","name":...,"argument":
"source/path"|null} frames ahead of the answer deltas (PLAN §4
extension, owner permission 2026-08-26); done.sources, query_log.sources
and the per-turn log line (gains tool_calls=N) report the retrieval
docs + read docs, deduped. The UI shows a "calling tool"
button/label state and one visible .tool-call line per call above the
answer; the lines persist with the chat record and re-render on
reload. chat_stream passes tools through and accumulates streaming
tool_calls deltas into ToolCallPiece (tools=None stays byte-identical).
E2E: deterministic mock tool flow ("use your tools" + <tools> marker:
list -> read first catalog line -> quoted answer) plus the story suite
(marker flow, reload re-render, plain/deflected no-tool regressions).
Docs: .env.example + README (the two tools, the budgets, the SSE tool
frame, the "calling tool" UI state).
probe: turbo tool_calls=supported 2026-08-26 (uv run python -m
scripts.llm_probe --tools — non-streaming + streaming
finish_reason=tool_calls, indexed delta.tool_calls partials)
This commit is contained in:
+183
-9
@@ -38,9 +38,33 @@ Implements just enough of the aipi surface:
|
||||
the same echo convention for the overview's prompt injection.
|
||||
- user message containing ``show the end of your notes`` (phase 24,
|
||||
whole-document context) -> the answer quotes the **last 160 chars of
|
||||
the document context** — a tail echo, byte-stable across runs, so a
|
||||
sentinel placed at the *end* of a document appears in the rendered
|
||||
answer iff the whole document was in the prompt.
|
||||
the ``<documents>`` block** — a tail echo, byte-stable across runs, so
|
||||
a sentinel placed at the *end* of a document appears in the rendered
|
||||
answer iff the whole document was in the prompt. (Phase 37: the HIGH
|
||||
prompt now ends with a ``<tools>`` section after ``</documents>``, so
|
||||
the echo targets the block itself; its tail still includes the
|
||||
closing tag — same sentinel semantics.)
|
||||
- user message containing ``use your tools`` (phase 37, agent document
|
||||
tools) **and** the system prompt carries the ``<tools>`` section ->
|
||||
the deterministic tool-calling flow, discriminated statelessly from
|
||||
the messages + the ``tools`` parameter:
|
||||
* request 1 (``tools`` offered, no tool results yet): stream ONLY
|
||||
``tool_calls`` deltas — ``list_documents`` (synthetic id
|
||||
``call_0``, no arguments), ``finish_reason: "tool_calls"``, no
|
||||
content;
|
||||
* request 2 (a ``tool``-role catalog result in the messages):
|
||||
parse the FIRST catalog line (``source/path — title`` → split on
|
||||
``" — "`` → ``rsplit("/", 1)``) and stream a ``tool_calls`` delta
|
||||
calling ``read_document`` on it (id ``call_1``);
|
||||
* request 3 (the read result in the messages, no ``tools``
|
||||
parameter): a content answer, deterministic: ``Read
|
||||
<source/path>. <first 80 chars of the read document's content>``
|
||||
— so a suite can assert the read document reached the model and
|
||||
landed in the answer.
|
||||
All other requests (including the marker without a ``<tools>``
|
||||
section, or with the tool conversation not yet started and no tools
|
||||
offered — e.g. budgets 0/0) behave exactly as today. ``E2E_REAL_LLM=1``
|
||||
ignores the mock entirely (the real model does what it does).
|
||||
|
||||
``max_tokens`` is honored deterministically (token ≈ whitespace word),
|
||||
like a real endpoint: an answer longer than the cap is truncated. This
|
||||
@@ -126,6 +150,67 @@ PRE_CONTENT_PAUSE_S = 4.0
|
||||
#: other suite is unaffected.
|
||||
END_OF_NOTES_TRIGGER = "show the end of your notes"
|
||||
|
||||
#: The ``<documents>`` block of the system prompt (phase 37: the HIGH
|
||||
#: prompt ends with the ``<tools>`` section after ``</documents>``, so the
|
||||
#: phase-24 tail echo targets the block, not the raw message tail).
|
||||
_DOCUMENTS_BLOCK_RE = re.compile(r"<documents>.*?</documents>", re.S)
|
||||
|
||||
#: Phase 37 (agent-document-tools story): a user message containing this
|
||||
#: substring (case-insensitive) — combined with the ``<tools>`` section
|
||||
#: in the system prompt — drives the deterministic tool flow documented
|
||||
#: in the module docstring (list_documents → read_document on the first
|
||||
#: catalog line → the quoted answer). Existing E2E questions do not
|
||||
#: contain the phrase, so every other suite is unaffected.
|
||||
TOOLS_TRIGGER = "use your tools"
|
||||
|
||||
|
||||
#: The agent's ``read_document`` tool-result prefix (app.rag.agent
|
||||
#: ``_execute_tool``): ``"Document <source/path>:\n<content>"``.
|
||||
_READ_RESULT_PREFIX = "Document "
|
||||
|
||||
|
||||
def _tool_flow(body: dict[str, Any]) -> tuple[str, str, str] | None:
|
||||
"""Classify a marker request into one step of the tool flow (phase 37).
|
||||
|
||||
Returns one of:
|
||||
|
||||
* ``("list", "", "")`` — ``tools`` are offered and no tool results
|
||||
are in the messages yet: the model lists the catalog.
|
||||
* ``("read", source, path)`` — a ``tool``-role catalog result is in
|
||||
the messages: the model reads its FIRST ``source/path — title``
|
||||
line (split on ``" — "``, then ``rsplit("/", 1)``).
|
||||
* ``("answer", "source/path", content)`` — a ``tool``-role read
|
||||
result (``"Document <source/path>:\n<content>"``) is in the
|
||||
messages: the model answers, quoting the read document.
|
||||
* ``None`` — not the marker flow: the request behaves exactly as
|
||||
today (marker absent, no ``<tools>`` section, or a no-tools first
|
||||
request — the budgets-0/0 path).
|
||||
"""
|
||||
if TOOLS_TRIGGER not in _user(body).lower():
|
||||
return None
|
||||
if "<tools>" not in _system(body):
|
||||
return None
|
||||
tool_msgs = [m for m in _messages(body) if m.get("role") == "tool"]
|
||||
for m in tool_msgs: # a read result means the forced-answer request
|
||||
content = str(m.get("content") or "")
|
||||
if content.startswith(_READ_RESULT_PREFIX):
|
||||
# The header is "Document <source/path>:" — drop the prefix
|
||||
# AND the trailing colon so the answer quotes a clean path.
|
||||
header, _, doc_content = content.partition("\n")
|
||||
sp = header[len(_READ_RESULT_PREFIX):].strip().removesuffix(":")
|
||||
return ("answer", sp, doc_content)
|
||||
if not body.get("tools"):
|
||||
return None
|
||||
for m in tool_msgs: # a catalog result means the read request
|
||||
content = str(m.get("content") or "")
|
||||
for line in content.splitlines():
|
||||
head = line.split(" — ", 1)[0].strip()
|
||||
if "/" in head:
|
||||
source, _, path = head.rpartition("/")
|
||||
if source and path:
|
||||
return ("read", source, path)
|
||||
return ("list", "", "")
|
||||
|
||||
|
||||
def long_answer() -> str:
|
||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||
@@ -222,12 +307,17 @@ def compose_answer(body: dict[str, Any]) -> str:
|
||||
)
|
||||
elif END_OF_NOTES_TRIGGER in user.lower():
|
||||
# Whole-document-context story (phase 24): echo the tail of the
|
||||
# context. Byte-stable across runs — a sentinel on the document's
|
||||
# last line appears in the answer iff the whole document was in
|
||||
# the prompt. (The tail includes the closing </documents> —
|
||||
# harmless for the E2E sentinel assertions.)
|
||||
# document context. Byte-stable across runs — a sentinel on the
|
||||
# document's last line appears in the answer iff the whole
|
||||
# document was in the prompt. (The tail includes the closing
|
||||
# </documents> — harmless for the E2E sentinel assertions.)
|
||||
# Phase 37: the HIGH prompt now ends with the <tools> section
|
||||
# after </documents>, so the echo targets the <documents> block
|
||||
# itself — the sentinel semantics are unchanged.
|
||||
block = _DOCUMENTS_BLOCK_RE.search(_system(body))
|
||||
tail_source = block.group(0) if block else _context(body)
|
||||
answer = (
|
||||
f"…and the very end of my notes reads: “{_context(body)[-160:]}” "
|
||||
f"…and the very end of my notes reads: “{tail_source[-160:]}” "
|
||||
"(Deterministic mock answer for E2E.)"
|
||||
)
|
||||
else:
|
||||
@@ -436,11 +526,95 @@ def _apply_max_tokens(answer: str, max_tokens: Any) -> str:
|
||||
return " ".join(words[:max_tokens])
|
||||
|
||||
|
||||
def _tool_call_stream(name: str, arguments: dict[str, Any], call_id: str) -> Any:
|
||||
"""SSE frames for one tool-call-only chat completion (phase 37).
|
||||
|
||||
The OpenAI wire convention the app accumulates (``app/rag/llm.py``):
|
||||
the first partial of index 0 carries ``id`` + ``type`` +
|
||||
``function.name`` plus the first ``function.arguments`` fragment;
|
||||
the remaining fragments (deterministic 16-char split — so the
|
||||
multi-fragment accumulation path is exercised) arrive on later
|
||||
chunks; the final chunk carries ``finish_reason: "tool_calls"``.
|
||||
No ``content`` / ``reasoning_content`` frames — the turn asked for a
|
||||
tool instead of answering.
|
||||
|
||||
Pacing: 0.1 s per frame — deliberately SLOWER than the content
|
||||
stream's 0.02 s, so the UI's transient "calling tool" state (held
|
||||
from the first ``tool`` frame until the first answer ``delta``) is a
|
||||
comfortable observation window for the story E2E (~1 s across the
|
||||
two tool requests).
|
||||
"""
|
||||
model = "turbo"
|
||||
chunk_id = f"chatcmpl-{uuid.uuid4()}"
|
||||
raw_args = json_dumps(arguments) if arguments else "{}"
|
||||
frags = [raw_args[i : i + 16] for i in range(0, len(raw_args), 16)] or ["{}"]
|
||||
for i, frag in enumerate(frags):
|
||||
tc: dict[str, Any] = {"index": 0, "function": {"arguments": frag}}
|
||||
delta: dict[str, Any] = {"tool_calls": [tc]}
|
||||
if i == 0:
|
||||
tc = {
|
||||
"index": 0,
|
||||
"id": call_id,
|
||||
"type": "function",
|
||||
"function": {"name": name, "arguments": frag},
|
||||
}
|
||||
delta = {"role": "assistant", "tool_calls": [tc]}
|
||||
payload = {
|
||||
"id": chunk_id,
|
||||
"object": "chat.completion.chunk",
|
||||
"created": int(time.time()),
|
||||
"model": model,
|
||||
"choices": [{"index": 0, "delta": delta, "finish_reason": None}],
|
||||
}
|
||||
yield f"data: {json_dumps(payload)}\n\n"
|
||||
time.sleep(0.1)
|
||||
yield (
|
||||
"data: "
|
||||
+ json_dumps(
|
||||
{
|
||||
"id": chunk_id,
|
||||
"object": "chat.completion.chunk",
|
||||
"created": int(time.time()),
|
||||
"model": model,
|
||||
"choices": [{"index": 0, "delta": {}, "finish_reason": "tool_calls"}],
|
||||
}
|
||||
)
|
||||
+ "\n\n"
|
||||
)
|
||||
yield "data: [DONE]\n\n"
|
||||
|
||||
|
||||
@app.post("/v1/chat/completions")
|
||||
def chat_completions(body: dict[str, Any]) -> Any:
|
||||
user_lower = _user(body).lower()
|
||||
# Phase 37 (agent document tools): the deterministic marker flow.
|
||||
# The app's chat path is the only streaming consumer of this mock, so
|
||||
# the flow handles streaming requests; a non-streaming marker request
|
||||
# (never issued by the app) falls through to the regular answer.
|
||||
if body.get("stream"):
|
||||
flow = _tool_flow(body)
|
||||
if flow is not None:
|
||||
if flow[0] == "list":
|
||||
stream = _tool_call_stream("list_documents", {}, "call_0")
|
||||
elif flow[0] == "read":
|
||||
stream = _tool_call_stream(
|
||||
"read_document",
|
||||
{"source": flow[1], "path": flow[2]},
|
||||
"call_1",
|
||||
)
|
||||
else: # "answer" — quote the read document (first 80 chars)
|
||||
answer = _apply_max_tokens(
|
||||
f"Read {flow[1]}. {flow[2][:80]}", body.get("max_tokens")
|
||||
)
|
||||
stream = _sse_stream(answer, 0.0)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
|
||||
answer = _apply_max_tokens(compose_answer(body), body.get("max_tokens"))
|
||||
delay = 3.0 if "pretend to think slowly" in _user(body) else 0.0
|
||||
user_lower = _user(body).lower()
|
||||
thinking = compose_thinking(body) if THINKING_TRIGGER in user_lower else ""
|
||||
pre_content = (
|
||||
PRE_CONTENT_PAUSE_S if SLOW_PRETOKEN_TRIGGER in user_lower else 0.0
|
||||
|
||||
@@ -0,0 +1,490 @@
|
||||
"""Phase 37 E2E (Playwright, mock-only): agent document tools (list + read).
|
||||
|
||||
Story: ``.agent/user_stories/agent-document-tools.md``
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real ``turbo``
|
||||
does whatever it does with the tools, while this story's gate is the
|
||||
deterministic marker flow in ``tests/e2e/mock_llm.py`` (user message
|
||||
contains ``use your tools`` **and** the system prompt carries the
|
||||
``<tools>`` section of the HIGH prompt):
|
||||
|
||||
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
|
||||
``tool_calls`` deltas calling ``list_documents`` (id ``call_0``, no
|
||||
arguments, ``finish_reason: "tool_calls"``);
|
||||
2. request 2 (a ``tool``-role catalog result in the messages) → streams a
|
||||
``tool_calls`` delta calling ``read_document`` on the FIRST catalog
|
||||
line (id ``call_1``);
|
||||
3. request 3 (no ``tools`` parameter, the read result in the messages) →
|
||||
the content answer ``Read <source/path>. <first 80 chars of the read
|
||||
document's content>`` — so the suite can assert the read document
|
||||
reached the model and landed in the answer.
|
||||
|
||||
KB fixture — reproduces the TODO failure (``aws-route53.md`` references
|
||||
``example-record-file.json`` "for the exact JSON shape of
|
||||
reseelink.json" but does not include it):
|
||||
|
||||
* ``Homelab/aws-route53.md`` — seeded with one chunk whose embedding is
|
||||
the mock's own bag-of-words vector (genuine token overlap: the marker
|
||||
question cosines ≈0.69 against it, well past the E2E 0.30 threshold,
|
||||
and it FTS-matches too) → the only RETRIEVABLE document, i.e. the
|
||||
grounded context;
|
||||
* ``Deployments/example-record-file.json`` — the JSON shape, indexed
|
||||
(a ``documents`` row: it is in the agent's catalog, readable, and a
|
||||
source-chip target) but seeded WITHOUT chunks. In a real
|
||||
hundreds-of-document KB the file would simply fail to rank into the
|
||||
top-2 context; with a two-document corpus every chunk would rank, so
|
||||
"not in context" is expressed as "no retrieval candidates". Its
|
||||
``(source, path)`` also sorts FIRST in the catalog
|
||||
(``Deployments`` < ``Homelab``) — which is exactly the line the mock
|
||||
parses out of the listing and reads.
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
1. ``test_marker_question_lists_reads_and_quotes`` — the SSE carries
|
||||
``tool`` frames (list, then read, ahead of any delta), the UI shows
|
||||
the "calling tool" label while a tool runs, the bubble shows both
|
||||
tool lines, the final answer quotes the read document, and the
|
||||
source chips include the read document (viewer link).
|
||||
2. ``test_tool_lines_re_render_after_reload`` — the persisted record
|
||||
(phase 14) re-renders the tool lines.
|
||||
3. ``test_plain_grounded_question_has_no_tool_frames`` — no marker → no
|
||||
``tool`` frames, the answer renders exactly as today (regression
|
||||
inside the story file).
|
||||
4. ``test_deflected_question_has_no_tool_frames`` — the tools are
|
||||
grounded-only: a deflected turn runs none.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document, QueryLog
|
||||
from tests.e2e.mock_llm import embed_text
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture documents (deterministic, token-controlled)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
SEED_SOURCE = "Homelab"
|
||||
SEED_PATH = "aws-route53.md"
|
||||
SEED_SP = f"{SEED_SOURCE}/{SEED_PATH}"
|
||||
|
||||
READ_SOURCE = "Deployments"
|
||||
READ_PATH = "example-record-file.json"
|
||||
READ_SP = f"{READ_SOURCE}/{READ_PATH}"
|
||||
|
||||
#: The retrievable document: references the JSON file "for the exact JSON
|
||||
#: shape of reeselink.json" but never includes it (the TODO failure).
|
||||
#: The repeated record-file lines carry the marker question's key tokens
|
||||
#: (aws, route53, hosted, zone, reeselink, json, exact, shape) — verified
|
||||
#: ≈0.69 cosine against the mock's embeddings (E2E threshold 0.30) plus
|
||||
#: FTS hits, so the turn is solidly grounded.
|
||||
ROUTE53_CONTENT = (
|
||||
"# AWS Route 53 Notes\n\n"
|
||||
"## Record file\n\n"
|
||||
+ (
|
||||
"The aws route53 hosted zone for reeselink keeps every record in "
|
||||
"reseelink.json — the exact JSON shape of reeselink.json is "
|
||||
"documented in example-record-file.json.\n"
|
||||
)
|
||||
* 10
|
||||
+ "\n## Sync job\n\n"
|
||||
"A cron job pushes reeselink.json to the aws route53 hosted zone "
|
||||
"every fifteen minutes; the diff is applied through the route53 api.\n"
|
||||
)
|
||||
|
||||
#: The referenced document: the exact JSON shape. Its FIRST line is longer
|
||||
#: than 80 chars, so the mock's first-80-chars quote is newline-free (the
|
||||
#: rendered-text assertions below match it verbatim). Pinned by the assert
|
||||
#: below.
|
||||
RECORD_FILE_CONTENT = (
|
||||
'{ "version": 3, "comment": "ReeseLink hosted zone records — the exact '
|
||||
'JSON shape of reeselink.json",\n'
|
||||
' "hosted_zone_id": "Z0RESEELINK01",\n'
|
||||
' "record_sets": [\n'
|
||||
' { "name": "www.reeselink.example", "type": "A", "ttl": 300,\n'
|
||||
' "resource_records": [ { "value": "10.0.0.20" } ] },\n'
|
||||
' { "name": "api.reeselink.example", "type": "CNAME", "ttl": 300,\n'
|
||||
' "resource_records": [ { "value": "www.reeselink.example" } ] }\n'
|
||||
" ]\n"
|
||||
"}\n"
|
||||
)
|
||||
assert "\n" not in RECORD_FILE_CONTENT[:80] # the quote must stay one line
|
||||
|
||||
MARKER_QUESTION = (
|
||||
"Use your tools: what is the exact JSON shape of reeselink.json "
|
||||
"for my aws route53 hosted zone?"
|
||||
)
|
||||
PLAIN_QUESTION = (
|
||||
"How does my aws route53 sync job push reeselink.json to the "
|
||||
"hosted zone?"
|
||||
)
|
||||
DEFLECT_QUESTION = "tell me about quantum wormhole cooling"
|
||||
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
DEFLECT_PHRASE = r"haven't done anything like that"
|
||||
ANSWER_PREFIX = f"Read {READ_SP}."
|
||||
ANSWER_QUOTE = RECORD_FILE_CONTENT[:80]
|
||||
READ_CHIP_HREF = f"/document.html?source={READ_SOURCE}&path={READ_PATH}&back=%2F"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# DB seeding (TRUNCATE-then-seed, cf. test_whole_document_context.py)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed(db: Session) -> None:
|
||||
"""The two-document pair from the TODO (see the module docstring)."""
|
||||
md = Document(
|
||||
source=SEED_SOURCE,
|
||||
path=SEED_PATH,
|
||||
full_path=f"/tmp/{SEED_PATH}",
|
||||
title="AWS Route 53 Notes",
|
||||
content=ROUTE53_CONTENT,
|
||||
content_hash=hashlib.sha256(ROUTE53_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
db.add(md)
|
||||
db.flush()
|
||||
# One chunk carrying the mock's own embedding → genuine token overlap
|
||||
# between the marker question and this document.
|
||||
db.add(
|
||||
Chunk(
|
||||
document_id=md.id,
|
||||
position=0,
|
||||
content=ROUTE53_CONTENT,
|
||||
embedding=embed_text(ROUTE53_CONTENT),
|
||||
)
|
||||
)
|
||||
# The referenced JSON: indexed, catalogued, readable — but NO chunks,
|
||||
# so retrieval never puts it in context (the failure the tools fix).
|
||||
db.add(
|
||||
Document(
|
||||
source=READ_SOURCE,
|
||||
path=READ_PATH,
|
||||
full_path=f"/tmp/{READ_PATH}",
|
||||
title="Example Record File",
|
||||
content=RECORD_FILE_CONTENT,
|
||||
content_hash=hashlib.sha256(RECORD_FILE_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
|
||||
"""Truncate the KB (plus the prompt-shaping tables), then re-seed.
|
||||
|
||||
``steering_notes`` / ``kb_overview`` are truncated too, so the HIGH
|
||||
prompt is exactly ``<relevance>`` + ``<documents>`` + ``<tools>``
|
||||
regardless of leftovers from other suites — byte-stable prompts,
|
||||
byte-stable answers.
|
||||
"""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
||||
)
|
||||
db.commit()
|
||||
if seed is not None:
|
||||
seed(db)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _last_query_log() -> QueryLog:
|
||||
with SessionLocal() as db:
|
||||
rows = db.scalars(select(QueryLog)).all()
|
||||
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
|
||||
return rows[0]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Records every value #send-label takes during the turn (a
|
||||
#: MutationObserver on the element), so the transient "Calling tool…"
|
||||
#: state is captured deterministically — no polling race.
|
||||
LABEL_RECORDER = """
|
||||
() => {
|
||||
if (window.__labelsInstalled) return;
|
||||
window.__labelsInstalled = true;
|
||||
window.__labels = [];
|
||||
const el = document.querySelector('#send-label');
|
||||
if (!el) return;
|
||||
const rec = (v) => {
|
||||
const l = window.__labels;
|
||||
if (!l.length || l[l.length - 1] !== v) l.push(v);
|
||||
};
|
||||
rec(el.textContent);
|
||||
new MutationObserver(() => rec(el.textContent)).observe(el, {
|
||||
childList: true,
|
||||
subtree: true,
|
||||
});
|
||||
}
|
||||
"""
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions for
|
||||
#: the ``tool`` frames, independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
"""Install both hooks on the loaded page (post-goto, pre-submit).
|
||||
|
||||
The fetch wrapper only needs to be in place before the turn's
|
||||
``fetch("/api/chat")`` call; the label observer needs the rendered
|
||||
``#send-label``. (``add_init_script`` would not do — it binds to the
|
||||
NEXT navigation, and the story page is navigated exactly once.)
|
||||
"""
|
||||
page.evaluate(SSE_HOOK)
|
||||
page.evaluate(LABEL_RECORDER)
|
||||
|
||||
|
||||
def _frames(page: Page) -> list[dict]:
|
||||
"""The captured SSE frames, once the hook's background read settles.
|
||||
|
||||
The hook reads ``res.clone().text()`` in a background promise that
|
||||
resolves right after the stream closes — poll briefly until the
|
||||
final ``done`` frame lands (fail loud if the hook captured nothing).
|
||||
"""
|
||||
deadline = time.monotonic() + 10.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The marker question: list → read → quoted answer, "calling tool" UI
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_marker_question_lists_reads_and_quotes(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db(_seed)
|
||||
page.goto(app_url)
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, MARKER_QUESTION)
|
||||
# While a tool runs the button carries the "calling tool" label: the
|
||||
# first `tool` frame sets it and it holds until the FIRST answer
|
||||
# delta (the agent loop completes before the answer stream) — so the
|
||||
# poll issued right after the click must catch it inside that window.
|
||||
expect(page.locator("#send-label")).to_have_text("Calling tool…", timeout=20_000)
|
||||
_wait_settled(page)
|
||||
|
||||
# The label transition is also recorded deterministically (no race):
|
||||
# Thinking… → Calling tool… → … → Send.
|
||||
labels = page.evaluate("() => window.__labels")
|
||||
assert "Calling tool…" in labels, labels
|
||||
assert labels.index("Calling tool…") > labels.index("Thinking…")
|
||||
|
||||
# Wire level: exactly two `tool` frames — list then read — and both
|
||||
# ahead of the first `delta` frame.
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "list_documents", "argument": None},
|
||||
{"type": "tool", "name": "read_document", "argument": READ_SP},
|
||||
]
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
(SEED_SOURCE, SEED_PATH),
|
||||
(READ_SOURCE, READ_PATH),
|
||||
]
|
||||
|
||||
# Both tool lines, in order, above the answer.
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
||||
expect(lines.nth(1)).to_contain_text(READ_SP)
|
||||
|
||||
# The final answer quotes the read document (the mock's deterministic
|
||||
# quote: "Read <source/path>. <first 80 chars of its content>").
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(ANSWER_PREFIX)
|
||||
expect(bubble).to_contain_text(ANSWER_QUOTE)
|
||||
|
||||
# Source chips: the retrieval doc AND the read doc (deduped, in
|
||||
# order) — the read chip links to the viewer.
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(2)
|
||||
expect(chips.nth(0)).to_contain_text(SEED_SP)
|
||||
chip_read = page.locator(".msg.brain .source-chip", has_text=READ_PATH)
|
||||
expect(chip_read).to_have_count(1)
|
||||
expect(chip_read.first).to_have_attribute("href", READ_CHIP_HREF)
|
||||
|
||||
# Durable record: grounded, both sources logged (retrieval + read).
|
||||
row = _last_query_log()
|
||||
assert row.question == MARKER_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == f"{SEED_SP}, {READ_SP}"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. Persistence: the tool lines re-render after a reload (phase 14)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_tool_lines_re_render_after_reload(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db(_seed)
|
||||
page.goto(app_url)
|
||||
|
||||
_submit(page, MARKER_QUESTION)
|
||||
_wait_settled(page)
|
||||
expect(page.locator(".msg.brain .tool-call")).to_have_count(2)
|
||||
|
||||
page.reload()
|
||||
expect(page.locator("#empty-state")).to_be_hidden()
|
||||
|
||||
# The persisted record re-renders BOTH tool lines, in saved order,
|
||||
# through the same append helper as the live frames.
|
||||
restored = page.locator(".msg.brain .tool-call")
|
||||
expect(restored).to_have_count(2)
|
||||
expect(restored.nth(0)).to_contain_text("Listing documents")
|
||||
expect(restored.nth(1)).to_contain_text("Reading ")
|
||||
expect(restored.nth(1)).to_contain_text(READ_SP)
|
||||
|
||||
# Answer + the read-document chip are intact (phase-14 restore path).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(ANSWER_PREFIX)
|
||||
expect(bubble).to_contain_text(ANSWER_QUOTE)
|
||||
chip_read = page.locator(".msg.brain .source-chip", has_text=READ_PATH)
|
||||
expect(chip_read).to_have_count(1)
|
||||
expect(chip_read.first).to_have_attribute("href", READ_CHIP_HREF)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 3. Regression: a plain grounded question (no marker) takes the
|
||||
# no-tool path — the answer renders exactly as today
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_plain_grounded_question_has_no_tool_frames(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db(_seed)
|
||||
page.goto(app_url)
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, PLAIN_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# No tool frames on the wire, no tool lines in the UI.
|
||||
assert _tool_frames(_frames(page)) == []
|
||||
expect(page.locator(".tool-call")).to_have_count(0)
|
||||
|
||||
# The standard grounded answer, citing the retrieval doc only — the
|
||||
# referenced JSON stays OUT of the sources (it was never read).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(PLAIN_QUESTION)
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(1)
|
||||
expect(chips.first).to_contain_text(SEED_SP)
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.question == PLAIN_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == SEED_SP
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 4. Grounded-only scope: a deflected turn runs no tools at all
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_deflected_question_has_no_tool_frames(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db(_seed)
|
||||
page.goto(app_url)
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, DEFLECT_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# The honesty gate fired — and no tool frames / tool lines came with
|
||||
# it (the LOW prompt never carries the tools).
|
||||
last = page.locator(".msg.brain").last
|
||||
expect(last).to_have_class(re.compile(r"is-deflected"))
|
||||
expect(last.locator(".bubble")).to_contain_text(
|
||||
re.compile(DEFLECT_PHRASE, re.IGNORECASE)
|
||||
)
|
||||
assert _tool_frames(_frames(page)) == []
|
||||
expect(page.locator(".tool-call")).to_have_count(0)
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.question == DEFLECT_QUESTION
|
||||
assert row.deflected is True
|
||||
@@ -0,0 +1,83 @@
|
||||
"""Integration: the agent DB accessors against real Postgres (phase 37).
|
||||
|
||||
``list_catalog`` must order rows by ``(source, path)`` — the same order as
|
||||
``GET /api/docs`` — and ``find_document`` must resolve a hit to the full
|
||||
document row (content included, for the never-truncated read) and return
|
||||
``None`` for unknown ``source``/``path`` pairs.
|
||||
|
||||
Requires: podman compose up -d db
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import Document
|
||||
from app.rag import agent
|
||||
|
||||
|
||||
def _doc(db: Session, source: str, path: str, title: str, content: str) -> Document:
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source=source,
|
||||
path=path,
|
||||
full_path=f"/tmp/{source}/{path}",
|
||||
title=title,
|
||||
content=content,
|
||||
content_hash="0" * 64,
|
||||
)
|
||||
db.add(doc)
|
||||
return doc
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def kb(db) -> Iterator[None]:
|
||||
"""Fresh documents table (chunks first — the FK) for these accessors."""
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
yield
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_list_catalog_orders_by_source_then_path(kb, db) -> None:
|
||||
_doc(db, "Zeta", "b/second.md", "Zeta B", "ZB")
|
||||
_doc(db, "Zeta", "a/first.md", "Zeta A", "ZA")
|
||||
_doc(db, "Alpha", "c/third.md", "Alpha C", "AC")
|
||||
db.commit()
|
||||
|
||||
assert agent.list_catalog(db) == [
|
||||
("Alpha", "c/third.md", "Alpha C"),
|
||||
("Zeta", "a/first.md", "Zeta A"),
|
||||
("Zeta", "b/second.md", "Zeta B"),
|
||||
]
|
||||
|
||||
|
||||
def test_list_catalog_is_empty_without_rows(kb, db) -> None:
|
||||
assert agent.list_catalog(db) == []
|
||||
|
||||
|
||||
def test_find_document_hit_returns_full_row(kb, db) -> None:
|
||||
created = _doc(db, "Alpha", "deep/nested/doc.md", "The Doc", "FULL-TEXT")
|
||||
db.commit()
|
||||
|
||||
found = agent.find_document(db, "Alpha", "deep/nested/doc.md")
|
||||
assert found is not None
|
||||
assert found.id == created.id
|
||||
assert found.source == "Alpha"
|
||||
assert found.path == "deep/nested/doc.md"
|
||||
assert found.title == "The Doc"
|
||||
assert found.content == "FULL-TEXT" # the read tool feeds this, untruncated
|
||||
|
||||
|
||||
def test_find_document_none_for_unknown_pairs(kb, db) -> None:
|
||||
_doc(db, "Alpha", "x.md", "X", "X-CONTENT")
|
||||
db.commit()
|
||||
|
||||
assert agent.find_document(db, "Alpha", "nope.md") is None # wrong path
|
||||
assert agent.find_document(db, "Beta", "x.md") is None # wrong source
|
||||
assert agent.find_document(db, "nope", "nope.md") is None # nothing at all
|
||||
@@ -192,10 +192,21 @@ def _find_emoji(text: str) -> list[str]:
|
||||
)
|
||||
def test_ui_chrome_has_no_emoji(client, path: str) -> None:
|
||||
"""Permanent regression guard (phase 08): the UI chrome — all pages,
|
||||
the JS that renders it, and the stylesheet — is emoji-free."""
|
||||
the JS that renders it, and the stylesheet — is emoji-free.
|
||||
|
||||
Phase 37 revision (owner permission 2026-08-26, PLAN §4): the agent's
|
||||
``.tool-call`` line carries two CONTENT marks — 🔎 (list) and 📄
|
||||
(read) — the only emoji in the whole frontend, and only as the exact
|
||||
tool-line template strings in app.js. The guard strips precisely
|
||||
those two literals; any other emoji, or those marks anywhere else,
|
||||
still fails."""
|
||||
r = client.get(path)
|
||||
assert r.status_code == 200
|
||||
assert _find_emoji(r.text) == [], f"emoji found in {path}: {_find_emoji(r.text)!r}"
|
||||
text = r.text
|
||||
if path == "/assets/app.js":
|
||||
text = text.replace('"🔎 Listing documents"', "")
|
||||
text = text.replace('"📄 Reading "', "")
|
||||
assert _find_emoji(text) == [], f"emoji found in {path}: {_find_emoji(text)!r}"
|
||||
|
||||
|
||||
def test_chat_requires_message(client) -> None:
|
||||
|
||||
@@ -13,6 +13,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
@@ -27,8 +28,10 @@ from app.api import chat as chat_api
|
||||
from app.config import Settings, get_settings
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Chunk, QueryLog
|
||||
from app.rag import agent
|
||||
from app.rag.agent import AGENT_TOOLS
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.llm import EmbeddingError, LLMError, StreamPiece
|
||||
from app.rag.llm import EmbeddingError, LLMError, StreamPiece, ToolCallPiece
|
||||
|
||||
FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "docs"
|
||||
QUESTION = "How is my Kubernetes cluster set up?"
|
||||
@@ -57,6 +60,7 @@ class FakeRagLLM:
|
||||
embed_error: Exception | None = None,
|
||||
stream_error: Exception | None = None,
|
||||
fail_mid_stream: bool = False,
|
||||
tool_script: list[list[StreamPiece | ToolCallPiece]] | None = None,
|
||||
) -> None:
|
||||
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
self.embed_batches = 0
|
||||
@@ -67,6 +71,18 @@ class FakeRagLLM:
|
||||
self.fail_mid_stream = fail_mid_stream
|
||||
self.question_embeds: list[str] = []
|
||||
self.seen_messages: list[list[dict[str, str]]] = []
|
||||
#: Every request's ``tools`` value (phase 37) — ``None`` is the
|
||||
#: pre-phase request shape (the key is absent from the payload).
|
||||
self.seen_tools: list[list[dict[str, Any]] | None] = []
|
||||
#: Canned per-agent-round piece lists (phase 37): ``tool_script[i]``
|
||||
#: is yielded for the *i*-th request that carries a non-None
|
||||
#: ``tools`` parameter (a request the agent loop is offering tools
|
||||
#: on). A request without tools — the deflected direct path, the
|
||||
#: post-budget answer request, or the 0/0 single-request path —
|
||||
#: always yields the thinking + answer stream below, so a
|
||||
#: deflected turn through this fake is byte-identical to the
|
||||
#: plain fake's output.
|
||||
self.tool_script: list[list[StreamPiece | ToolCallPiece]] = list(tool_script or [])
|
||||
|
||||
async def embed(self, texts: list[str]) -> list[list[float]]:
|
||||
self.embed_batches += 1
|
||||
@@ -87,14 +103,25 @@ class FakeRagLLM:
|
||||
self.question_embeds.append(text)
|
||||
return _token_vec(text)
|
||||
|
||||
async def chat_stream(self, messages: list[dict[str, str]]):
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
):
|
||||
"""Typed stream (phase 17): ``thinking`` slices (same 12-char
|
||||
cadence as content) **before** the content pieces. With the
|
||||
default ``thinking=""`` this yields content-only pieces — today's
|
||||
behavior, new yield type."""
|
||||
behavior, new yield type. Phase 37: *tools* is the agent loop's
|
||||
``tools=…`` passthrough (recorded in ``seen_tools``); a request
|
||||
with tools consumes the next ``tool_script`` entry, if any."""
|
||||
self.seen_messages.append(messages)
|
||||
self.seen_tools.append(tools)
|
||||
if self.stream_error is not None:
|
||||
raise self.stream_error
|
||||
if tools is not None and self.tool_script:
|
||||
for piece in self.tool_script.pop(0):
|
||||
yield piece
|
||||
return
|
||||
if self.fail_mid_stream:
|
||||
yield StreamPiece("content", "partial ")
|
||||
raise LLMError("mid-stream dropout")
|
||||
@@ -464,3 +491,229 @@ def test_chat_query_log_failure_still_sends_done(client, db, seeded_kb: FakeRagL
|
||||
assert [f["type"] for f in frames if f["type"] == "delta"]
|
||||
assert frames[-1]["type"] == "done"
|
||||
assert frames[-1]["deflected"] is False
|
||||
|
||||
|
||||
# ---------- phase 37: agent document tools on grounded turns ----------
|
||||
|
||||
|
||||
def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""(a) Grounded turn with tool calls: the event sequence is
|
||||
``thinking?/tool/tool/delta…/done``; ``done.sources`` and the
|
||||
``query_log`` row include the read document (deduped, order
|
||||
preserved); the per-turn log line carries ``tool_calls=2``.
|
||||
The agent loop offers tools while budgets last and drops them
|
||||
(``tools=None``) once both are spent."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
StreamPiece("thinking", "Let me list what is indexed…"),
|
||||
ToolCallPiece(id="call_1", name="list_documents", arguments={}),
|
||||
],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={"source": "docs", "path": "homelab/backups.md"},
|
||||
)
|
||||
],
|
||||
# the post-budget answer request (tools=None) falls back to the
|
||||
# fake's thinking + answer stream
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
caplog.set_level(logging.INFO, logger="app.chat")
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
types = [f["type"] for f in frames]
|
||||
assert types[0] == "thinking"
|
||||
assert types[1] == "tool" and types[2] == "tool" # the two executed calls
|
||||
assert "error" not in types
|
||||
assert types[3:-1] == ["delta"] * (len(types) - 4) # deltas, then done last
|
||||
assert frames[-1]["type"] == "done"
|
||||
|
||||
list_frame, read_frame = frames[1], frames[2]
|
||||
assert set(list_frame) == {"type", "name", "argument"}
|
||||
assert list_frame["name"] == "list_documents"
|
||||
assert list_frame["argument"] is None # the tool takes no parameters
|
||||
assert set(read_frame) == {"type", "name", "argument"}
|
||||
assert read_frame["name"] == "read_document"
|
||||
assert read_frame["argument"] == "docs/homelab/backups.md"
|
||||
|
||||
deltas = [f for f in frames if f["type"] == "delta"]
|
||||
assert len(deltas) >= 2 # genuinely streamed
|
||||
assert "".join(d["text"] for d in deltas) == scripted.answer
|
||||
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
# done.sources = the retrieval docs + the read doc, deduped, order kept.
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert sources[-1] == ("docs", "homelab/backups.md") # the read doc is cited
|
||||
assert ("docs", "homelab/kubernetes.md") in sources # …after the retrieval docs
|
||||
assert len(sources) == len(set(sources)) # deduped by (source, path)
|
||||
assert done["sources"][-1]["title"] == "Backup Strategy"
|
||||
|
||||
# The agent loop offered the tools while any budget remained and
|
||||
# dropped them once both were spent (single post-budget request).
|
||||
assert len(scripted.seen_messages) == 3
|
||||
assert scripted.seen_tools[0] == AGENT_TOOLS
|
||||
assert scripted.seen_tools[1] == AGENT_TOOLS # the read budget was still open
|
||||
assert scripted.seen_tools[2] is None
|
||||
|
||||
# The query_log row carries the same combined source list.
|
||||
(row,) = db.scalars(select(QueryLog)).all()
|
||||
assert row.deflected is False
|
||||
assert "docs/homelab/kubernetes.md" in row.sources
|
||||
assert row.sources.endswith(", docs/homelab/backups.md") # the read doc, last
|
||||
|
||||
# The required per-turn log line (PLAN §9 extension) counts both calls
|
||||
# and lists the combined sources (retrieval + read).
|
||||
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
assert lines and "tool_calls=2" in lines[-1]
|
||||
assert "'docs/homelab/kubernetes.md'" in lines[-1]
|
||||
assert "'docs/homelab/backups.md'" in lines[-1]
|
||||
|
||||
|
||||
def test_deflected_turn_stays_byte_identical_without_tools(
|
||||
client, db, seeded_kb: FakeRagLLM
|
||||
) -> None:
|
||||
"""(b) Deflected turn: the agent loop never runs — no ``tool``
|
||||
frames, and the frame sequence is byte-identical to the plain fake's
|
||||
direct-``chat_stream`` output even for a fake scripted to call tools
|
||||
(its script is never consumed). The LLM was called once, without a
|
||||
``tools`` key."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={"source": "docs", "path": "homelab/backups.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "never used — the agent never runs")],
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
|
||||
try:
|
||||
_, _, baseline = _stream_chat(client, OFF_TOPIC)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, OFF_TOPIC)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
assert frames == baseline # byte-identical to the direct path
|
||||
assert not any(f["type"] == "tool" for f in frames)
|
||||
assert frames[-1]["type"] == "done" and frames[-1]["deflected"] is True
|
||||
assert len(scripted.tool_script) == 3 # the script was never consumed
|
||||
assert len(scripted.seen_messages) == 1
|
||||
assert scripted.seen_tools == [None] # one request, no tools key
|
||||
|
||||
# The read document never sneaks into the deflected turn's record.
|
||||
(row,) = [
|
||||
r
|
||||
for r in db.scalars(select(QueryLog)).all()
|
||||
if r.question == OFF_TOPIC
|
||||
][-1:]
|
||||
assert row.deflected is True
|
||||
assert "backups.md" not in row.sources
|
||||
|
||||
|
||||
def test_zero_agent_budgets_reproduce_pre_phase_single_request(
|
||||
client,
|
||||
db,
|
||||
seeded_kb: FakeRagLLM,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""(c) ``BOR_AGENT_LIST_CALLS=0 BOR_AGENT_READ_CALLS=0``: no ``tool``
|
||||
frames, exactly one request **without** a ``tools`` key (the
|
||||
pre-phase request shape), ``done.sources`` unchanged, and
|
||||
``tool_calls=0`` in the log line — budgets-as-kill-switch."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={"source": "docs", "path": "homelab/backups.md"},
|
||||
)
|
||||
],
|
||||
]
|
||||
)
|
||||
live = get_settings()
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=live.relevance_threshold,
|
||||
agent_list_calls=0,
|
||||
agent_read_calls=0,
|
||||
),
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
caplog.set_level(logging.INFO, logger="app.chat")
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
assert not any(f["type"] == "tool" for f in frames)
|
||||
assert "error" not in [f["type"] for f in frames]
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False
|
||||
paths = [s["path"] for s in done["sources"]]
|
||||
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
|
||||
assert "homelab/backups.md" not in paths # nothing was read
|
||||
|
||||
# Exactly one request, and it carried no ``tools`` key at all — the
|
||||
# scripted tool calls were never even offered a chance.
|
||||
assert len(scripted.seen_messages) == 1
|
||||
assert scripted.seen_tools == [None]
|
||||
assert len(scripted.tool_script) == 2 # never consumed
|
||||
|
||||
(row,) = db.scalars(select(QueryLog)).all()
|
||||
assert "docs/homelab/kubernetes.md" in row.sources
|
||||
assert "backups.md" not in row.sources
|
||||
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
assert lines and "tool_calls=0" in lines[-1]
|
||||
|
||||
|
||||
def test_tool_execution_db_failure_yields_error_event(
|
||||
client, db, seeded_kb: FakeRagLLM, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A tool call that hits a dead DB mid-stream gets the same structured
|
||||
``error`` event as the pre-stream retrieval path — never a severed
|
||||
stream (the "never stale" contract, PLAN §7.4)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[[ToolCallPiece(id="call_1", name="list_documents", arguments={})]]
|
||||
)
|
||||
|
||||
def boom(*_a: Any, **_k: Any) -> Any:
|
||||
raise RuntimeError("db exploded mid tool call")
|
||||
|
||||
monkeypatch.setattr(agent, "list_catalog", boom)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
# The ``tool`` frame went out first (the model requested the call);
|
||||
# the failed execution ends the turn with the structured error event.
|
||||
assert [f["type"] for f in frames] == ["tool", "error"]
|
||||
assert frames[0]["name"] == "list_documents"
|
||||
assert "offline mid-question" in frames[1]["detail"]
|
||||
assert db.scalars(select(QueryLog)).all() == [] # no row for a failed turn
|
||||
|
||||
@@ -0,0 +1,519 @@
|
||||
"""Unit: the grounded-turn agent loop (phase 37, ``app.rag.agent``).
|
||||
|
||||
A scripted fake LLM (canned stream sequences) + monkeypatched
|
||||
``list_catalog`` / ``find_document`` — no database, no network. Covers
|
||||
the loop mechanics: the list → read → answer happy path (event order,
|
||||
holder state, the ``tools=None`` request after the budgets are spent,
|
||||
the assistant/tool message history), the 0/0 single-call path, budget
|
||||
exhaustion, dedupe, unknown tool / missing args / unknown path, the
|
||||
round cap, and the ``<tools>`` prompt section (HIGH only).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import uuid
|
||||
from collections.abc import AsyncIterator
|
||||
from copy import deepcopy
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag import agent
|
||||
from app.rag.agent import (
|
||||
AGENT_TOOLS,
|
||||
AgentHolder,
|
||||
run_agent,
|
||||
)
|
||||
from app.rag.llm import LLMClient, StreamPiece, ToolCallPiece
|
||||
from app.rag.prompts import TOOLS_SECTION, _base, build_deflect_prompt, build_high_prompt
|
||||
|
||||
|
||||
def _settings(**kwargs: Any) -> Settings:
|
||||
kwargs.setdefault("_env_file", None)
|
||||
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _doc(source: str, path: str, title: str = "Title", content: str = "CONTENT") -> Document:
|
||||
return Document(
|
||||
id=uuid.uuid4(),
|
||||
source=source,
|
||||
path=path,
|
||||
full_path=f"/tmp/{path}",
|
||||
title=title,
|
||||
content=content,
|
||||
content_hash="0" * 64,
|
||||
)
|
||||
|
||||
|
||||
class ScriptedLLM:
|
||||
"""Canned stream sequences; records every ``chat_stream`` request so
|
||||
the tests can assert on the messages and the ``tools`` passthrough."""
|
||||
|
||||
def __init__(self, *streams: list[StreamPiece | ToolCallPiece]) -> None:
|
||||
self.streams: list[list[StreamPiece | ToolCallPiece]] = list(streams)
|
||||
self.requests: list[tuple[list[dict[str, Any]], list[dict[str, Any]] | None]] = []
|
||||
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
) -> AsyncIterator[StreamPiece | ToolCallPiece]:
|
||||
self.requests.append((deepcopy(messages), tools))
|
||||
if not self.streams:
|
||||
raise AssertionError("ScriptedLLM ran out of canned streams")
|
||||
for piece in self.streams.pop(0):
|
||||
yield piece
|
||||
|
||||
|
||||
async def _run(
|
||||
llm: ScriptedLLM,
|
||||
holder: AgentHolder,
|
||||
settings: Settings,
|
||||
seed_docs: list[Document] | None = None,
|
||||
) -> list[StreamPiece | ToolCallPiece]:
|
||||
out: list[StreamPiece | ToolCallPiece] = []
|
||||
async for piece in run_agent(
|
||||
cast("LLMClient", llm),
|
||||
cast("Session", None),
|
||||
system_prompt="SYSTEM_PROMPT",
|
||||
user_message="QUESTION",
|
||||
seed_docs=seed_docs or [],
|
||||
settings=settings,
|
||||
holder=holder,
|
||||
):
|
||||
out.append(piece)
|
||||
return out
|
||||
|
||||
|
||||
# ---------- AGENT_TOOLS shape ----------
|
||||
|
||||
|
||||
def test_agent_tools_names_and_parameters() -> None:
|
||||
by_name = {t["function"]["name"]: t for t in AGENT_TOOLS}
|
||||
assert set(by_name) == {"list_documents", "read_document"}
|
||||
assert all(t["type"] == "function" for t in AGENT_TOOLS)
|
||||
list_params = by_name["list_documents"]["function"]["parameters"]
|
||||
assert list_params["type"] == "object"
|
||||
assert list_params["properties"] == {} # no parameters
|
||||
read_params = by_name["read_document"]["function"]["parameters"]
|
||||
assert read_params["required"] == ["source", "path"]
|
||||
assert set(read_params["properties"]) == {"source", "path"}
|
||||
|
||||
|
||||
# ---------- happy path: list → read → answer ----------
|
||||
|
||||
|
||||
def test_list_then_read_then_answer(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
catalog = [
|
||||
("Deployments", "backups.md", "Backup Strategy"),
|
||||
("Homelab", "aws-route53.md", "AWS Route53 Records"),
|
||||
]
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
|
||||
target = _doc("Homelab", "aws-route53.md", "AWS Route53 Records", "R53-CONTENT")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: target)
|
||||
seed = [_doc("Homelab", "kubernetes.md", "Kubernetes", "K8S-CONTENT")]
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={"source": "Homelab", "path": "aws-route53.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("thinking", "hmm "), StreamPiece("content", "Done! ")],
|
||||
)
|
||||
|
||||
pieces = asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
|
||||
# Event order: tool pieces before the answer content/thinking.
|
||||
assert [type(p) for p in pieces] == [
|
||||
ToolCallPiece,
|
||||
ToolCallPiece,
|
||||
StreamPiece,
|
||||
StreamPiece,
|
||||
]
|
||||
assert pieces[0] == ToolCallPiece(id="call_1", name="list_documents", arguments={})
|
||||
assert isinstance(pieces[1], ToolCallPiece)
|
||||
assert pieces[1].name == "read_document"
|
||||
assert pieces[3] == StreamPiece("content", "Done! ")
|
||||
# The read document is recorded for done.sources / query_log (task 04).
|
||||
assert holder.read_docs == [target]
|
||||
assert holder.tool_calls == 2
|
||||
|
||||
# Default budgets (1/1): tools offered while any budget remains…
|
||||
assert llm.requests[0][1] == AGENT_TOOLS
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
# …and dropped (tools=None) once both are spent.
|
||||
assert llm.requests[2][1] is None
|
||||
assert len(llm.requests) == 3
|
||||
|
||||
# The follow-up request carries the assistant tool-call + tool result.
|
||||
msgs = llm.requests[1][0]
|
||||
assert msgs[0] == {"role": "system", "content": "SYSTEM_PROMPT"}
|
||||
assert msgs[1] == {"role": "user", "content": "QUESTION"}
|
||||
assert msgs[2] == {
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {"name": "list_documents", "arguments": "{}"},
|
||||
}
|
||||
],
|
||||
}
|
||||
assert msgs[3] == {
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_1",
|
||||
"content": (
|
||||
"2 documents:\n"
|
||||
"Deployments/backups.md — Backup Strategy\n"
|
||||
"Homelab/aws-route53.md — AWS Route53 Records"
|
||||
),
|
||||
}
|
||||
# The second follow-up request carries the read call + the FULL text.
|
||||
msgs = llm.requests[2][0]
|
||||
assert msgs[4]["role"] == "assistant"
|
||||
assert msgs[4]["tool_calls"][0]["id"] == "call_2"
|
||||
assert json.loads(msgs[4]["tool_calls"][0]["function"]["arguments"]) == {
|
||||
"source": "Homelab",
|
||||
"path": "aws-route53.md",
|
||||
}
|
||||
assert msgs[5] == {
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_2",
|
||||
"content": "Document Homelab/aws-route53.md:\nR53-CONTENT", # full text, no cap
|
||||
}
|
||||
|
||||
|
||||
def test_empty_catalog_listing_says_zero_documents(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
|
||||
assert holder.tool_calls == 1
|
||||
|
||||
|
||||
def test_content_and_tool_call_in_one_stream_keeps_both(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Rare stream with content AND a tool call: the content stays (it was
|
||||
already emitted) and the tool still runs."""
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
StreamPiece("content", "Let me check "),
|
||||
ToolCallPiece(id="call_1", name="list_documents", arguments={}),
|
||||
],
|
||||
[StreamPiece("content", "the answer")],
|
||||
)
|
||||
pieces = asyncio.run(_run(llm, holder, _settings()))
|
||||
assert [type(p) for p in pieces] == [StreamPiece, ToolCallPiece, StreamPiece]
|
||||
assert holder.tool_calls == 1 # the tool ran despite the content
|
||||
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
|
||||
|
||||
|
||||
# ---------- budgets ----------
|
||||
|
||||
|
||||
def test_zero_budgets_is_one_request_without_tools() -> None:
|
||||
"""BOR_AGENT_LIST_CALLS=0 BOR_AGENT_READ_CALLS=0 → byte-identical
|
||||
single-call path: exactly one request, tools=None, no history growth."""
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM([StreamPiece("thinking", "t "), StreamPiece("content", "direct answer")])
|
||||
pieces = asyncio.run(
|
||||
_run(llm, holder, _settings(agent_list_calls=0, agent_read_calls=0))
|
||||
)
|
||||
assert [type(p) for p in pieces] == [StreamPiece, StreamPiece]
|
||||
assert len(llm.requests) == 1
|
||||
assert llm.requests[0][1] is None
|
||||
assert llm.requests[0][0] == [
|
||||
{"role": "system", "content": "SYSTEM_PROMPT"},
|
||||
{"role": "user", "content": "QUESTION"},
|
||||
]
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
|
||||
|
||||
def test_read_budget_exhausted_refuses_and_appends_nothing(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
a = _doc("S", "a.md", "A", "A-CONTENT")
|
||||
monkeypatch.setattr(
|
||||
agent, "find_document", lambda db, source, path: a if path == "a.md" else None
|
||||
)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1", name="read_document", arguments={"source": "S", "path": "a.md"}
|
||||
)
|
||||
],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2", name="read_document", arguments={"source": "S", "path": "b.md"}
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1)))
|
||||
|
||||
assert holder.read_docs == [a] # the refused read appended nothing
|
||||
assert holder.tool_calls == 1 # …and consumed no budget
|
||||
refusal = llm.requests[2][0][5]
|
||||
assert refusal == {
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_2",
|
||||
"content": agent.READ_EXHAUSTED,
|
||||
}
|
||||
# The list budget is still open, so tools stay offered after the refusal.
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_list_budget_exhausted_refuses_with_its_own_message(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[ToolCallPiece(id="call_2", name="list_documents", arguments={})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1)))
|
||||
assert holder.tool_calls == 1
|
||||
assert llm.requests[2][0][5]["content"] == agent.LIST_EXHAUSTED
|
||||
# The read budget is still open, so tools stay offered after the refusal.
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
# ---------- rejections (no budget consumed) ----------
|
||||
|
||||
|
||||
def test_reading_a_seed_doc_is_already_in_context(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
seed = [_doc("Homelab", "kubernetes.md", "Kubernetes", "K8S-CONTENT")]
|
||||
|
||||
def _boom(*_a: Any, **_k: Any) -> None:
|
||||
raise AssertionError("find_document must not be called for a seeded doc")
|
||||
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
monkeypatch.setattr(agent, "find_document", _boom)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read_document",
|
||||
arguments={"source": "Homelab", "path": "kubernetes.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
# No budget consumed → tools are still offered on the next request.
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_reading_an_already_read_doc_is_deduped(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
doc = _doc("S", "a.md", "A", "A-CONTENT")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1", name="read_document", arguments={"source": "S", "path": "a.md"}
|
||||
)
|
||||
],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2", name="read_document", arguments={"source": "S", "path": "a.md"}
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1)))
|
||||
assert holder.read_docs == [doc] # appended exactly once
|
||||
assert holder.tool_calls == 1
|
||||
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
# The read budget is intact after the deduped refusal…
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_unknown_path_refused_without_budget(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read_document",
|
||||
arguments={"source": "S", "path": "ghost.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"]
|
||||
== "No document at S/ghost.md — check the list_documents output."
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # budget intact
|
||||
|
||||
|
||||
def test_unknown_tool_name_refused(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="delete_universe", arguments={"x": 1})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == agent.UNKNOWN_TOOL
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # nothing was consumed
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("arguments", "label"),
|
||||
[
|
||||
({}, "no arguments"),
|
||||
({"source": "S"}, "path missing"),
|
||||
({"path": "p.md"}, "source missing"),
|
||||
({"source": "", "path": "p.md"}, "empty source"),
|
||||
({"source": "S", "path": " "}, "blank path"),
|
||||
({"source": 7, "path": "p.md"}, "non-string source"),
|
||||
],
|
||||
)
|
||||
def test_read_document_missing_arguments_refused(
|
||||
monkeypatch: pytest.MonkeyPatch, arguments: dict[str, Any], label: str
|
||||
) -> None:
|
||||
def _boom(*_a: Any, **_k: Any) -> None:
|
||||
raise AssertionError(f"find_document must not be called ({label})")
|
||||
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
monkeypatch.setattr(agent, "find_document", _boom)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read_document", arguments=arguments)],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == agent.MISSING_READ_ARGS
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
# ---------- round cap (pathological stream) ----------
|
||||
|
||||
|
||||
def test_round_cap_forces_a_final_no_tools_answer(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A model that keeps calling a budget-exhausted tool must be forced
|
||||
to answer at ``max_rounds = 2 + list + read`` (= 4 for 1/1)."""
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[ToolCallPiece(id="call_2", name="list_documents", arguments={})],
|
||||
[ToolCallPiece(id="call_3", name="list_documents", arguments={})],
|
||||
[ToolCallPiece(id="call_4", name="list_documents", arguments={})],
|
||||
[StreamPiece("content", "forced answer")],
|
||||
)
|
||||
pieces = asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1)))
|
||||
assert [type(p) for p in pieces] == [
|
||||
ToolCallPiece,
|
||||
ToolCallPiece,
|
||||
ToolCallPiece,
|
||||
ToolCallPiece,
|
||||
StreamPiece,
|
||||
]
|
||||
assert len(llm.requests) == 5
|
||||
# The forced final request carries no tools, whatever is left.
|
||||
assert llm.requests[4][1] is None
|
||||
# Only the first call consumed budget; the three rejections did not.
|
||||
assert holder.tool_calls == 1
|
||||
# The 4th rejection sits at messages[2 + 4*2 - 1] of the final request.
|
||||
assert llm.requests[4][0][9]["content"] == agent.LIST_EXHAUSTED
|
||||
|
||||
|
||||
# ---------- settings ----------
|
||||
|
||||
|
||||
def test_agent_budget_settings_default_to_one_each() -> None:
|
||||
s = _settings()
|
||||
assert s.agent_list_calls == 1
|
||||
assert s.agent_read_calls == 1
|
||||
|
||||
|
||||
def test_agent_budget_settings_env_override(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("BOR_AGENT_LIST_CALLS", "0")
|
||||
monkeypatch.setenv("BOR_AGENT_READ_CALLS", "2")
|
||||
s = _settings()
|
||||
assert s.agent_list_calls == 0
|
||||
assert s.agent_read_calls == 2
|
||||
|
||||
|
||||
# ---------- prompts: <tools> section (HIGH only) ----------
|
||||
|
||||
|
||||
def test_high_prompt_carries_tools_section_after_documents() -> None:
|
||||
prompt = build_high_prompt([_doc("S", "a.md", "A", "A-CONTENT")])
|
||||
assert TOOLS_SECTION in prompt
|
||||
assert "call `list_documents`" in prompt
|
||||
assert "then `read_document` to pull in exactly one more document" in prompt
|
||||
assert "do not read more than one extra document" in prompt
|
||||
# After the mode body: <tools> follows </documents>.
|
||||
assert prompt.index("</documents>") < prompt.index("<tools>")
|
||||
assert prompt.rstrip().endswith("</tools>")
|
||||
|
||||
|
||||
def test_high_prompt_tools_section_with_notes_and_kb() -> None:
|
||||
prompt = build_high_prompt(
|
||||
[_doc("S", "a.md", "A", "A-CONTENT")], notes=["be concise"], kb_overview="- KB"
|
||||
)
|
||||
assert prompt.index("<knowledge_base>") < prompt.index("<tuning>")
|
||||
assert prompt.index("<tuning>") < prompt.index("<documents>")
|
||||
assert prompt.index("<documents>") < prompt.index("<tools>")
|
||||
|
||||
|
||||
def test_low_prompt_is_byte_identical_and_tool_free() -> None:
|
||||
expected = (
|
||||
_base("LOW")
|
||||
+ "\nDEFLECT_MODE: retrieval was weak — the titles below are the closest "
|
||||
"your notes come to the question. They are titles only; do not pretend "
|
||||
"they answer it. Use them to propose 2-3 alternative questions.\n"
|
||||
+ "- T1\n- T2"
|
||||
)
|
||||
assert build_deflect_prompt(["T1", "T2"]) == expected
|
||||
for prompt in (
|
||||
build_deflect_prompt(["T1"]),
|
||||
build_deflect_prompt(["T1"], notes=["be concise"]),
|
||||
build_deflect_prompt(["T1"], kb_overview="- KB"),
|
||||
build_deflect_prompt(["T1"], notes=["be concise"], kb_overview="- KB"),
|
||||
):
|
||||
assert "<tools>" not in prompt
|
||||
assert TOOLS_SECTION not in prompt
|
||||
@@ -20,6 +20,7 @@ from app.api import chat as chat_api
|
||||
from app.config import Settings
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Document, KbOverview, QueryLog
|
||||
from app.rag.agent import AGENT_TOOLS
|
||||
from app.rag.llm import StreamPiece
|
||||
from app.rag.retriever import RetrievedChunk, weak_hit_titles
|
||||
from app.rag.suggestions import MAX_SUGGESTIONS, derive_suggestions
|
||||
@@ -416,19 +417,30 @@ def test_suggestions_empty_input_yields_fallback_only() -> None:
|
||||
|
||||
|
||||
class _CannedLLM:
|
||||
"""Records the messages it is given; streams a canned answer."""
|
||||
"""Records the messages it is given; streams a canned answer.
|
||||
|
||||
Never emits tool calls, so a grounded turn through the phase-37 agent
|
||||
loop ends after the single (tools-offered) request; *seen_tools*
|
||||
records each request's ``tools`` value for the phase-37 wiring pins.
|
||||
"""
|
||||
|
||||
def __init__(self, answer: str = ANSWER) -> None:
|
||||
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
self.embed_batches = 0
|
||||
self.answer = answer
|
||||
self.seen: list[list[dict[str, str]]] = []
|
||||
self.seen_tools: list[list[dict[str, Any]] | None] = []
|
||||
|
||||
async def embed_one(self, _text: str) -> list[float]:
|
||||
return [0.0] * 768
|
||||
|
||||
async def chat_stream(self, messages: list[dict[str, str]]):
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
):
|
||||
self.seen.append(messages)
|
||||
self.seen_tools.append(tools)
|
||||
for i in range(0, len(self.answer), 12):
|
||||
yield StreamPiece("content", self.answer[i : i + 12])
|
||||
|
||||
@@ -545,6 +557,55 @@ def test_endpoint_just_below_threshold_deflects(
|
||||
assert session.commits == 1
|
||||
|
||||
|
||||
def test_endpoint_grounded_turn_runs_agent_loop_with_tools(
|
||||
client: TestClient,
|
||||
gate_env: tuple[_FakeSession, _CannedLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 37: a grounded endpoint turn runs the agent loop — the
|
||||
single no-tool-call request carries ``AGENT_TOOLS`` (default 1/1
|
||||
budgets), no ``tool`` frames stream, and the ``done`` event is the
|
||||
plain retrieval shape (the tool-free answer is byte-identical)."""
|
||||
_session, llm = gate_env
|
||||
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT")
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.90)]))
|
||||
|
||||
frames = _ask(client, "How is my Kubernetes cluster set up?")
|
||||
|
||||
assert frames[-1]["type"] == "done"
|
||||
assert frames[-1]["deflected"] is False
|
||||
assert not any(f["type"] == "tool" for f in frames)
|
||||
assert len(llm.seen) == 1
|
||||
assert llm.seen_tools == [AGENT_TOOLS] # one request, tools offered
|
||||
# The system prompt is the HIGH prompt with the <tools> instructions.
|
||||
(system, _user) = llm.seen[0][0], llm.seen[0][1]
|
||||
assert "<relevance>HIGH</relevance>" in system["content"]
|
||||
assert "<tools>" in system["content"]
|
||||
|
||||
|
||||
def test_endpoint_deflected_turn_never_offers_tools(
|
||||
client: TestClient,
|
||||
gate_env: tuple[_FakeSession, _CannedLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 37: a deflected endpoint turn keeps the direct
|
||||
``chat_stream`` — the single request carries no ``tools`` key
|
||||
(``seen_tools == [None]``), A8 byte-identical."""
|
||||
_session, llm = gate_env
|
||||
doc = _doc("Deploying a New Service", "DOC_CONTENT_NEVER_SENT")
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.2999)]))
|
||||
|
||||
frames = _ask(client, "How do I bake sourdough bread?")
|
||||
|
||||
assert frames[-1]["type"] == "done"
|
||||
assert frames[-1]["deflected"] is True
|
||||
assert not any(f["type"] == "tool" for f in frames)
|
||||
assert len(llm.seen) == 1
|
||||
assert llm.seen_tools == [None]
|
||||
(system, _user) = llm.seen[0][0], llm.seen[0][1]
|
||||
assert "<tools>" not in system["content"] # the LOW prompt never carries it
|
||||
|
||||
|
||||
def test_endpoint_score_at_threshold_answers(
|
||||
client: TestClient,
|
||||
gate_env: tuple[_FakeSession, _CannedLLM],
|
||||
|
||||
@@ -114,7 +114,10 @@ def test_save_points_user_on_send_and_brain_on_done() -> None:
|
||||
# in the same meta object).
|
||||
done_idx = js.find('ev.type === "done"')
|
||||
assert done_idx != -1
|
||||
done_block = js[done_idx : done_idx + 1300]
|
||||
# Window: the whole done branch (up to the error branch) — the meta
|
||||
# object legitimately grows with phases (phase 17: thinking, phase
|
||||
# 37: tools), so a fixed char offset would false-fail.
|
||||
done_block = js[done_idx : js.find('ev.type === "error"')]
|
||||
assert "rememberBrainTurn(finalText || acc" in done_block
|
||||
assert "thinking: thinkingAcc || undefined" in done_block
|
||||
assert "deflected: !!ev.deflected" in done_block
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
"""Unit: the phase-37 "calling tool" frontend contract (task 05).
|
||||
|
||||
No new Python app logic exists for this task — the behavior lives in
|
||||
``frontend/assets/app.js`` + ``styles.css`` and is E2E-gated by the story
|
||||
suite (task 06). Like the other frontend-adjacent unit files, this module
|
||||
pins the JS/CSS markers the story depends on, so a silent regression in
|
||||
the tool branch, the persistence shape, or the tool-line styling is
|
||||
caught without a browser.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
|
||||
APP_JS = FRONTEND / "assets" / "app.js"
|
||||
STYLES_CSS = FRONTEND / "assets" / "styles.css"
|
||||
|
||||
|
||||
def _js() -> str:
|
||||
return APP_JS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _css() -> str:
|
||||
return STYLES_CSS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_tool_branch_is_a_first_class_turn_branch() -> None:
|
||||
"""The turn handler must branch on `tool` frames BETWEEN the
|
||||
thinking and delta branches: the stream stays alive (guard clears),
|
||||
the brain wrap is created on demand, and the label state is
|
||||
applied only while the UI state is still "thinking" (a late frame
|
||||
after the first delta just appends the line — never a crash)."""
|
||||
js = _js()
|
||||
thinking_idx = js.find('ev.type === "thinking"')
|
||||
tool_idx = js.find('ev.type === "tool"')
|
||||
delta_idx = js.find('ev.type === "delta"')
|
||||
assert -1 < thinking_idx < tool_idx < delta_idx, (
|
||||
"the turn handler must branch on tool frames"
|
||||
)
|
||||
branch = js[tool_idx:delta_idx]
|
||||
assert "toolAcc.push" in branch, "every tool frame is recorded for persistence"
|
||||
assert "clearTurnTimeout()" in branch, "a tool frame proves the stream is alive"
|
||||
assert 'addMessage("brain", "")' in branch, "first frame creates the brain wrap"
|
||||
assert "uiState === UI_STATE.thinking" in branch, (
|
||||
"label updates only while the state is still thinking"
|
||||
)
|
||||
assert "appendToolLine(wrap, name, argument)" in branch
|
||||
# No setUiState in the branch: the state stays "thinking" (never stale).
|
||||
assert "setUiState" not in branch, (
|
||||
"the tool branch must keep uiState=thinking (button stays disabled)"
|
||||
)
|
||||
|
||||
|
||||
def test_calling_tool_label_strings() -> None:
|
||||
"""The 'calling tool' label strings the story keys off: the button
|
||||
text and the status/typing-indicator labels (plain literals —
|
||||
phase 39 centralizes brand strings; no helper here)."""
|
||||
js = _js()
|
||||
tool_idx = js.find('ev.type === "tool"')
|
||||
delta_idx = js.find('ev.type === "delta"')
|
||||
branch = js[tool_idx:delta_idx]
|
||||
assert '"Calling tool…"' in branch, "the button carries the calling-tool text"
|
||||
assert '"Brain of Reese is listing documents"' in branch
|
||||
assert "Brain of Reese is reading ${argument}" in branch
|
||||
assert "sendStatus.textContent = toolStatus" in branch, (
|
||||
"the #send-status live region announces what Brain is doing"
|
||||
)
|
||||
assert 'setAttribute("aria-label", toolStatus)' in branch, (
|
||||
"the typing indicator label follows the tool state"
|
||||
)
|
||||
# The elapsed-seconds hint keeps running through tool frames: the
|
||||
# branch must not stop/restart the clock.
|
||||
assert "stopThinkingClock" not in branch
|
||||
assert "startThinkingClock" not in branch
|
||||
|
||||
|
||||
def test_tool_lines_render_into_the_bubble_wrap() -> None:
|
||||
"""appendToolLine: first frame creates the .tool-calls list (role=list
|
||||
+ accessible name) BEFORE the bubble — below an existing Thinking
|
||||
block — and each line is a .tool-call listitem with the exact marks:
|
||||
🔎 Listing documents / 📄 Reading <code>path</code>. The path goes
|
||||
through textContent (storage can never inject HTML)."""
|
||||
js = _js()
|
||||
fn = js.find("function appendToolLine")
|
||||
assert fn != -1, "appendToolLine must exist"
|
||||
body = js[fn : js.find("\n}\n", fn)]
|
||||
assert 'querySelector(".tool-calls")' in body, "idempotent per wrap"
|
||||
assert 'className = "tool-calls"' in body
|
||||
assert 'setAttribute("role", "list")' in body
|
||||
assert 'setAttribute("aria-label", "Tool calls")' in body
|
||||
assert 'insertBefore(container, body.querySelector(".bubble"))' in body, (
|
||||
"the list sits ABOVE the answer"
|
||||
)
|
||||
assert 'className = "tool-call"' in body
|
||||
assert 'setAttribute("role", "listitem")' in body
|
||||
assert 'line.textContent = "📄 Reading "' in body
|
||||
assert 'line.textContent = "🔎 Listing documents"' in body
|
||||
assert "code.textContent = argument" in body, (
|
||||
"the path is data — textContent, never innerHTML"
|
||||
)
|
||||
assert "name === \"read_document\" && argument" in body
|
||||
|
||||
|
||||
def test_tool_branch_is_append_only_and_interleaving_safe() -> None:
|
||||
"""Append-only, the same rule as thinking: multiple calls append
|
||||
multiple lines in order, and a tool frame after the first delta
|
||||
(should not happen in v1) still appends — the branch has no early
|
||||
return gated on acc/delta state."""
|
||||
js = _js()
|
||||
tool_idx = js.find('ev.type === "tool"')
|
||||
delta_idx = js.find('ev.type === "delta"')
|
||||
branch = js[tool_idx:delta_idx]
|
||||
assert "if (aborted) return" not in branch, (
|
||||
"the aborted guard lives at the dispatch top, not per branch"
|
||||
)
|
||||
assert "acc" not in branch.split("appendToolLine")[0].replace("toolAcc", ""), (
|
||||
"the tool branch must not depend on accumulated answer text"
|
||||
)
|
||||
|
||||
|
||||
def test_tool_frames_persist_next_to_thinking() -> None:
|
||||
"""The `done` save point gains an optional `tools` key next to
|
||||
`thinking` (empty turns persist exactly as before — `undefined`
|
||||
drops the key from the JSON), and the accumulator is turn-scoped
|
||||
in handleSend."""
|
||||
js = _js()
|
||||
assert "let toolAcc = []" in js
|
||||
done_idx = js.find('ev.type === "done"')
|
||||
error_idx = js.find('ev.type === "error"')
|
||||
assert -1 < done_idx < error_idx
|
||||
done_block = js[done_idx:error_idx]
|
||||
assert "thinking: thinkingAcc || undefined" in done_block
|
||||
assert "tools: toolAcc.length ? toolAcc : undefined" in done_block, (
|
||||
"tools persisted next to thinking, optional like it"
|
||||
)
|
||||
|
||||
|
||||
def test_tool_lines_re_render_on_restore() -> None:
|
||||
"""Phase 14 convention: a stored brain record with `tools` re-renders
|
||||
the lines on load through the SAME append helper (after the thinking
|
||||
re-render, before the deflected/sources additions)."""
|
||||
js = _js()
|
||||
fn = js.find("function renderStoredMessage")
|
||||
assert fn != -1
|
||||
end = js.find("function restoreConversation")
|
||||
body = js[fn:end]
|
||||
assert "Array.isArray(m.tools)" in body
|
||||
assert "appendToolLine(wrap, t.name, arg)" in body
|
||||
thinking_restore = body.find("if (m.thinking)")
|
||||
tools_restore = body.find("Array.isArray(m.tools)")
|
||||
assert -1 < thinking_restore < tools_restore, (
|
||||
"tools restore sits after the thinking restore (same order as live)"
|
||||
)
|
||||
assert "typeof t.name !== \"string\"" in body, "malformed entries skipped"
|
||||
|
||||
|
||||
def test_thinking_block_stays_on_top_of_tool_lines() -> None:
|
||||
"""If a tool frame precedes the first thinking frame, the Thinking
|
||||
block is still created ABOVE the tool lines (scratchpad on top),
|
||||
not below them."""
|
||||
js = _js()
|
||||
fn = js.find("function ensureThinkingBlock")
|
||||
assert fn != -1
|
||||
body = js[fn : js.find("\n}\n", fn)]
|
||||
assert 'querySelector(".tool-calls")' in body, (
|
||||
"the thinking anchor must account for existing tool lines"
|
||||
)
|
||||
assert 'querySelector(".bubble")' in body
|
||||
assert "insertBefore" in body
|
||||
assert "block.open = true" in body
|
||||
|
||||
|
||||
def test_tool_call_style_is_accent_and_contrast_safe() -> None:
|
||||
"""styles.css: .tool-call is an inline row with the accent palette
|
||||
(distinct from the brand-ink Thinking block) and mono `code` styling
|
||||
for the path; the wrapper stacks lines without shifting the column."""
|
||||
css = _css()
|
||||
assert ".tool-calls" in css
|
||||
assert ".tool-call" in css
|
||||
m = re.search(r"\.tool-call \{([^}]*)\}", css)
|
||||
assert m, "the .tool-call rule must exist"
|
||||
row = m.group(1)
|
||||
assert "display: flex" in row, "inline row: icon + text"
|
||||
assert "var(--accent-ink)" in row, (
|
||||
"accent color distinguishes it from the thinking block (≈10.4:1 on surface)"
|
||||
)
|
||||
assert "var(--accent-line)" in row, "accent left border"
|
||||
code = re.search(r"\.tool-call code \{([^}]*)\}", css)
|
||||
assert code, "the path `code` must be styled"
|
||||
assert "var(--mono)" in code.group(1)
|
||||
assert "var(--ink)" in code.group(1) # ≈11.5:1 on --brand-soft
|
||||
assert "gap" in css.split(".tool-calls {")[1].split("}")[0], (
|
||||
"lines stack with a gap — append-only, no reflow"
|
||||
)
|
||||
|
||||
|
||||
def test_no_cdn_added() -> None:
|
||||
"""AGENTS.md rule 6: the tool state adds no external script/link."""
|
||||
index = (FRONTEND / "index.html").read_text(encoding="utf-8")
|
||||
assert 'src="http' not in index and 'href="http' not in index
|
||||
@@ -11,7 +11,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -22,6 +22,7 @@ from app.rag.llm import (
|
||||
LLMClient,
|
||||
LLMError,
|
||||
StreamPiece,
|
||||
ToolCallPiece,
|
||||
)
|
||||
|
||||
|
||||
@@ -242,21 +243,47 @@ def test_single_oversized_text_fails_actionably() -> None:
|
||||
# ---------- chat streaming (phase 03) ----------
|
||||
|
||||
|
||||
def _tool_call(
|
||||
index: int,
|
||||
id: str | None = None,
|
||||
name: str | None = None,
|
||||
arguments: str | None = None,
|
||||
):
|
||||
"""One fake ``delta.tool_calls[]`` partial (openai SDK shape, phase 37).
|
||||
|
||||
``function`` is None when neither *name* nor *arguments* is given —
|
||||
mirroring the real wire, where id-only fragments carry no function.
|
||||
"""
|
||||
fn = None
|
||||
if name is not None or arguments is not None:
|
||||
fn = SimpleNamespace(name=name, arguments=arguments)
|
||||
return SimpleNamespace(index=index, id=id, function=fn)
|
||||
|
||||
|
||||
def _chunk(
|
||||
content: str | None = "text", empty: bool = False, reasoning: str | None = None
|
||||
content: str | None = "text",
|
||||
empty: bool = False,
|
||||
reasoning: str | None = None,
|
||||
tool_calls: list | None = None,
|
||||
finish_reason: str | None = None,
|
||||
):
|
||||
"""One fake ChatCompletionChunk (``choices[].delta`` shape).
|
||||
|
||||
``reasoning_content`` is present on the delta only when *reasoning*
|
||||
is not None — mirroring the real wire, where the field exists only
|
||||
when the model sends it.
|
||||
``reasoning_content``, ``tool_calls`` and ``finish_reason`` are
|
||||
present only when provided — mirroring the real wire, where the
|
||||
fields exist only when the model sends them.
|
||||
"""
|
||||
if empty:
|
||||
return SimpleNamespace(choices=[])
|
||||
delta: SimpleNamespace = SimpleNamespace(content=content)
|
||||
if reasoning is not None:
|
||||
delta.reasoning_content = reasoning
|
||||
return SimpleNamespace(choices=[SimpleNamespace(delta=delta)])
|
||||
if tool_calls is not None:
|
||||
delta.tool_calls = tool_calls
|
||||
choice = SimpleNamespace(delta=delta)
|
||||
if finish_reason is not None:
|
||||
choice.finish_reason = finish_reason
|
||||
return SimpleNamespace(choices=[choice])
|
||||
|
||||
|
||||
class _FakeChatStream:
|
||||
@@ -326,7 +353,11 @@ def _make_stream_client(
|
||||
|
||||
|
||||
async def _collect(llm: LLMClient, messages: list[dict[str, str]]) -> list[StreamPiece]:
|
||||
return [p async for p in llm.chat_stream(messages)]
|
||||
"""Collect pieces from a tools-less stream (phase 37 task 02, test (a):
|
||||
without tools, no ToolCallPiece can appear)."""
|
||||
pieces = [p async for p in llm.chat_stream(messages)]
|
||||
assert all(isinstance(p, StreamPiece) for p in pieces)
|
||||
return cast("list[StreamPiece]", pieces)
|
||||
|
||||
|
||||
def test_chat_stream_yields_deltas_in_order() -> None:
|
||||
@@ -355,6 +386,9 @@ def test_chat_stream_uses_locked_generation_params() -> None:
|
||||
# BOR_MAX_OUTPUT_TOKENS (default 32 768) so they are not cut off.
|
||||
assert completions.kwargs["max_tokens"] == 32_768
|
||||
assert completions.kwargs["messages"] == messages
|
||||
# Phase 37: no tools passed ⇒ no `tools` key at all (byte-identical
|
||||
# request to pre-phase-37).
|
||||
assert "tools" not in completions.kwargs
|
||||
|
||||
|
||||
def test_chat_stream_max_tokens_comes_from_settings() -> None:
|
||||
@@ -455,6 +489,231 @@ def test_chat_stream_llm_error_passes_through_unwrapped() -> None:
|
||||
asyncio.run(_collect(llm, [{"role": "user", "content": "q"}]))
|
||||
|
||||
|
||||
# ---------- tool-call streaming (phase 37, task 02) ----------
|
||||
|
||||
#: The agent's tool list (phase 37) — the exact wire shape AGENT_TOOLS will
|
||||
#: pass through (the names are whatever the caller's tools list names).
|
||||
_AGENT_TOOLS: list[dict[str, Any]] = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "list_documents",
|
||||
"description": "List the indexed documents.",
|
||||
"parameters": {"type": "object", "properties": {}},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "read_document",
|
||||
"description": "Add one indexed document's full text to the context.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"source": {"type": "string"},
|
||||
"path": {"type": "string"},
|
||||
},
|
||||
"required": ["source", "path"],
|
||||
},
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _collect_with_tools(
|
||||
llm: LLMClient, messages: list[dict[str, str]], tools: list[dict[str, Any]]
|
||||
) -> list[StreamPiece | ToolCallPiece]:
|
||||
async def run() -> list[StreamPiece | ToolCallPiece]:
|
||||
return [p async for p in llm.chat_stream(messages, tools=tools)]
|
||||
|
||||
return asyncio.run(run())
|
||||
|
||||
|
||||
def test_chat_stream_passes_tools_when_given() -> None:
|
||||
"""(e) A non-None tools list is forwarded verbatim to create()."""
|
||||
llm, completions = _make_stream_client([_chunk("ok")], llm_chat_model="turbo")
|
||||
_collect_with_tools(llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS)
|
||||
assert completions.kwargs is not None
|
||||
assert completions.kwargs["tools"] == _AGENT_TOOLS
|
||||
|
||||
|
||||
def test_chat_stream_accumulates_tool_call_across_chunk_partials() -> None:
|
||||
"""(b) name on the first partial, arguments in fragments — merged into
|
||||
one ToolCallPiece with the concatenated JSON, at finish_reason."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(
|
||||
0,
|
||||
id="call_abc",
|
||||
name="read_document",
|
||||
arguments='{"source": "Homelab", "pa',
|
||||
)
|
||||
],
|
||||
),
|
||||
_chunk(None, tool_calls=[_tool_call(0, arguments='th": "kubernetes.md"}')]),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
pieces = _collect_with_tools(
|
||||
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
|
||||
)
|
||||
assert pieces == [
|
||||
ToolCallPiece(
|
||||
id="call_abc",
|
||||
name="read_document",
|
||||
arguments={"source": "Homelab", "path": "kubernetes.md"},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_chat_stream_two_tool_calls_yielded_in_index_order() -> None:
|
||||
"""(c) Indices 0 and 1, interleaved partials (index 1 seen first) —
|
||||
both calls, in index order, each merged from its own fragments."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(1, id="call_b", name="read_document", arguments='{"sou')
|
||||
],
|
||||
),
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(0, id="call_a", name="list_documents"),
|
||||
_tool_call(1, arguments='rce": "Homelab", "path": "a.md"}')
|
||||
],
|
||||
),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
pieces = _collect_with_tools(
|
||||
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
|
||||
)
|
||||
assert pieces == [
|
||||
ToolCallPiece(id="call_a", name="list_documents", arguments={}),
|
||||
ToolCallPiece(
|
||||
id="call_b",
|
||||
name="read_document",
|
||||
arguments={"source": "Homelab", "path": "a.md"},
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def test_chat_stream_tool_calls_yielded_at_stream_end_without_finish_reason() -> None:
|
||||
"""The spec's other emission point: stream ends without a
|
||||
finish_reason="tool_calls" chunk — pieces still materialize."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[_tool_call(0, id="call_z", name="list_documents")],
|
||||
)
|
||||
]
|
||||
)
|
||||
pieces = _collect_with_tools(
|
||||
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
|
||||
)
|
||||
assert pieces == [ToolCallPiece(id="call_z", name="list_documents", arguments={})]
|
||||
|
||||
|
||||
def test_chat_stream_synthesizes_call_id_when_absent() -> None:
|
||||
"""Wire never carried the call id ⇒ synthesized "call_<index>"."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(None, tool_calls=[_tool_call(2, name="read_document", arguments="{}")]),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
pieces = _collect_with_tools(
|
||||
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
|
||||
)
|
||||
assert pieces == [
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_chat_stream_null_arguments_become_empty_dict() -> None:
|
||||
"""JSON "null" (and, by the same branch, absent arguments) ⇒ {}."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(0, id="call_n", name="list_documents", arguments="null")
|
||||
],
|
||||
),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
pieces = _collect_with_tools(
|
||||
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
|
||||
)
|
||||
assert pieces == [ToolCallPiece(id="call_n", name="list_documents", arguments={})]
|
||||
|
||||
|
||||
def test_chat_stream_malformed_tool_arguments_raise_llm_error() -> None:
|
||||
"""(d) A silently dropped tool call would corrupt the loop — malformed
|
||||
arguments JSON must fail loudly."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(
|
||||
0,
|
||||
id="call_x",
|
||||
name="read_document",
|
||||
arguments='{"source": "Homelab",',
|
||||
)
|
||||
],
|
||||
),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
|
||||
async def drain() -> None:
|
||||
async for _ in llm.chat_stream(
|
||||
[{"role": "user", "content": "q"}], tools=_AGENT_TOOLS
|
||||
):
|
||||
pass
|
||||
|
||||
with pytest.raises(LLMError, match="malformed tool-call arguments"):
|
||||
asyncio.run(drain())
|
||||
|
||||
|
||||
def test_chat_stream_non_object_tool_arguments_raise_llm_error() -> None:
|
||||
"""The OpenAI contract says arguments is a JSON *object* — a bare array
|
||||
is malformed too."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(0, id="call_y", name="read_document", arguments='[1, 2]')
|
||||
],
|
||||
),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
|
||||
async def drain() -> None:
|
||||
async for _ in llm.chat_stream(
|
||||
[{"role": "user", "content": "q"}], tools=_AGENT_TOOLS
|
||||
):
|
||||
pass
|
||||
|
||||
with pytest.raises(LLMError, match="non-object tool-call arguments"):
|
||||
asyncio.run(drain())
|
||||
|
||||
|
||||
# ---------- one-shot chat: LLMClient.chat (phase 30, task 01) ----------
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
"""Unit tests: scripts/llm_probe.py --tools response parsing (phase 37, task 01).
|
||||
|
||||
The live probe talks to aipi; the parsing and verdict-classification logic
|
||||
is factored into pure functions and is what this module pins. The fixtures
|
||||
mirror the exact wire shapes observed live against ``turbo`` on 2026-08-26
|
||||
(reasoning_content first, then indexed delta.tool_calls fragments).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from scripts.llm_probe import (
|
||||
classify_tool_calling,
|
||||
parse_tool_response_nonstreaming,
|
||||
parse_tool_response_streaming,
|
||||
)
|
||||
|
||||
|
||||
def test_parse_nonstreaming_tool_calls() -> None:
|
||||
payload = {
|
||||
"choices": [
|
||||
{
|
||||
"finish_reason": "tool_calls",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"reasoning_content": "Let me call it.\n",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "abc123",
|
||||
"type": "function",
|
||||
"function": {"name": "get_time", "arguments": "{}"},
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
]
|
||||
}
|
||||
out = parse_tool_response_nonstreaming(payload)
|
||||
assert out == {"finish_reason": "tool_calls", "calls": [("get_time", "{}")]}
|
||||
|
||||
|
||||
def test_parse_nonstreaming_plain_content_answer() -> None:
|
||||
payload = {
|
||||
"choices": [
|
||||
{"finish_reason": "stop", "message": {"role": "assistant", "content": "It is noon."}}
|
||||
]
|
||||
}
|
||||
assert parse_tool_response_nonstreaming(payload) == {
|
||||
"finish_reason": "stop",
|
||||
"calls": [],
|
||||
}
|
||||
|
||||
|
||||
def test_parse_nonstreaming_empty_or_malformed() -> None:
|
||||
assert parse_tool_response_nonstreaming({"choices": []}) == {
|
||||
"finish_reason": None,
|
||||
"calls": [],
|
||||
}
|
||||
assert parse_tool_response_nonstreaming({}) == {"finish_reason": None, "calls": []}
|
||||
assert parse_tool_response_nonstreaming(None) == {"finish_reason": None, "calls": []}
|
||||
|
||||
|
||||
def _sse(payload: dict) -> str:
|
||||
return "data: " + json.dumps(payload)
|
||||
|
||||
|
||||
def test_parse_streaming_accumulates_fragments() -> None:
|
||||
"""The live wire shape: id+name+partial args in chunk 1, args in chunk 2."""
|
||||
lines = [
|
||||
_sse({"choices": [{"delta": {"reasoning_content": "Let me think."}, "index": 0}]}),
|
||||
_sse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {
|
||||
"tool_calls": [
|
||||
{
|
||||
"index": 0,
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {"name": "get_time", "arguments": "{"},
|
||||
}
|
||||
]
|
||||
},
|
||||
"index": 0,
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
_sse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {"tool_calls": [{"index": 0, "function": {"arguments": "}"}}]},
|
||||
"index": 0,
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
_sse({"choices": [{"delta": {}, "finish_reason": "tool_calls", "index": 0}]}),
|
||||
"data: [DONE]",
|
||||
]
|
||||
out = parse_tool_response_streaming(lines)
|
||||
assert out["finish_reason"] == "tool_calls"
|
||||
assert out["calls"] == [("get_time", "{}")]
|
||||
assert out["delta_chunks"] == 2
|
||||
assert out["indexed"] is True
|
||||
assert out["had_id"] is True
|
||||
assert out["arguments_in_deltas"] is True
|
||||
|
||||
|
||||
def test_parse_streaming_no_tool_calls() -> None:
|
||||
lines = [
|
||||
_sse({"choices": [{"delta": {"content": "It is "}, "index": 0}]}),
|
||||
_sse({"choices": [{"delta": {"content": "noon."}, "index": 0}]}),
|
||||
_sse({"choices": [{"delta": {}, "finish_reason": "stop", "index": 0}]}),
|
||||
"data: [DONE]",
|
||||
]
|
||||
out = parse_tool_response_streaming(lines)
|
||||
assert out["finish_reason"] == "stop"
|
||||
assert out["calls"] == []
|
||||
assert out["delta_chunks"] == 0
|
||||
assert out["had_id"] is False
|
||||
assert out["arguments_in_deltas"] is False
|
||||
|
||||
|
||||
def test_parse_streaming_stops_at_done_and_skips_malformed() -> None:
|
||||
lines = [
|
||||
"data: not-json",
|
||||
"",
|
||||
_sse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {
|
||||
"tool_calls": [
|
||||
{
|
||||
"index": 0,
|
||||
"id": "x",
|
||||
"type": "function",
|
||||
"function": {"name": "get_time", "arguments": "{}"},
|
||||
}
|
||||
]
|
||||
},
|
||||
"index": 0,
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
"data: [DONE]",
|
||||
# After [DONE] nothing must be parsed:
|
||||
_sse({"choices": [{"delta": {"content": "should not appear"}, "index": 0}]}),
|
||||
]
|
||||
out = parse_tool_response_streaming(lines)
|
||||
assert out["calls"] == [("get_time", "{}")]
|
||||
assert out["delta_chunks"] == 1
|
||||
assert out["finish_reason"] is None
|
||||
|
||||
|
||||
def test_parse_streaming_multiple_calls_by_index() -> None:
|
||||
lines = [
|
||||
_sse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {
|
||||
"tool_calls": [
|
||||
{
|
||||
"index": 1,
|
||||
"id": "b",
|
||||
"function": {"name": "second", "arguments": "{\"a\": "},
|
||||
},
|
||||
{
|
||||
"index": 0,
|
||||
"id": "a",
|
||||
"function": {"name": "first", "arguments": "{}"},
|
||||
},
|
||||
]
|
||||
},
|
||||
"index": 0,
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
_sse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {
|
||||
"tool_calls": [{"index": 1, "function": {"arguments": "1}"}}]
|
||||
},
|
||||
"index": 0,
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
]
|
||||
out = parse_tool_response_streaming(lines)
|
||||
assert out["calls"] == [("first", "{}"), ("second", '{"a": 1}')]
|
||||
assert out["delta_chunks"] == 3
|
||||
|
||||
|
||||
def _ns(ok: bool = True) -> dict:
|
||||
return {
|
||||
"finish_reason": "tool_calls" if ok else "stop",
|
||||
"calls": [("get_time", "{}")] if ok else [],
|
||||
}
|
||||
|
||||
|
||||
def _st(**overrides: object) -> dict:
|
||||
result: dict = {
|
||||
"finish_reason": "tool_calls",
|
||||
"calls": [("get_time", "{}")],
|
||||
"delta_chunks": 2,
|
||||
"indexed": True,
|
||||
"had_id": True,
|
||||
"arguments_in_deltas": True,
|
||||
}
|
||||
result.update(overrides)
|
||||
return result
|
||||
|
||||
|
||||
def test_classify_supported() -> None:
|
||||
assert classify_tool_calling(_ns(), _st()) == "supported"
|
||||
|
||||
|
||||
def test_classify_not_supported_variants() -> None:
|
||||
# Non-streaming did not call the tool.
|
||||
assert classify_tool_calling(_ns(False), _st()) == "not-supported"
|
||||
# Streaming did not call the tool.
|
||||
assert classify_tool_calling(_ns(), _st(finish_reason="stop", calls=[])) == "not-supported"
|
||||
# Streamed, but not as delta.tool_calls chunks.
|
||||
assert classify_tool_calling(_ns(), _st(delta_chunks=0)) == "not-supported"
|
||||
# Delta chunks without the OpenAI "index" field.
|
||||
assert classify_tool_calling(_ns(), _st(indexed=False)) == "not-supported"
|
||||
# Delta chunks without a call "id".
|
||||
assert classify_tool_calling(_ns(), _st(had_id=False)) == "not-supported"
|
||||
# Intermittent: non-stream ok, stream answered in content.
|
||||
assert classify_tool_calling(_ns(), _st(finish_reason="stop")) == "not-supported"
|
||||
@@ -16,6 +16,7 @@ from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag.prompts import (
|
||||
PERSONA,
|
||||
TOOLS_SECTION,
|
||||
_base,
|
||||
build_deflect_prompt,
|
||||
build_high_prompt,
|
||||
@@ -116,14 +117,18 @@ def test_low_prompt_with_no_titles() -> None:
|
||||
|
||||
def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
|
||||
"""Phase 15 contract: with no steering notes the prompt is exactly what
|
||||
it was before the <tuning> section existed."""
|
||||
it was before the <tuning> section existed. (Phase 37: the HIGH prompt
|
||||
additionally carries the ``<tools>`` section after the mode body — the
|
||||
fixtures account for it; the LOW prompt is untouched.)"""
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||
block = (
|
||||
'<document source="Homelab" path="kubernetes.md" title="Kubernetes Homelab Cluster">\n'
|
||||
"Talos Linux on three nodes.\n"
|
||||
"</document>"
|
||||
)
|
||||
assert build_high_prompt([doc]) == _base("HIGH") + "\n<documents>\n" + block + "\n</documents>"
|
||||
assert build_high_prompt([doc]) == (
|
||||
_base("HIGH") + "\n<documents>\n" + block + "\n</documents>" + "\n" + TOOLS_SECTION
|
||||
)
|
||||
assert build_deflect_prompt(["T1", "T2"]) == (
|
||||
_base("LOW")
|
||||
+ "\nDEFLECT_MODE: retrieval was weak — the titles below are the closest "
|
||||
@@ -220,14 +225,16 @@ def test_kb_section_tiny_budget_never_exceeds_cap() -> None:
|
||||
def test_no_overview_prompt_is_byte_identical_to_pre_phase() -> None:
|
||||
"""Phase 31 contract: with no KB overview (None, empty, or blank)
|
||||
every prompt is exactly what it was before the ``<knowledge_base>``
|
||||
section existed — with or without steering notes."""
|
||||
section existed — with or without steering notes. (Phase 37: the HIGH
|
||||
prompt additionally carries the ``<tools>`` section after the mode
|
||||
body — the fixtures account for it; the LOW prompt is untouched.)"""
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||
block = (
|
||||
'<document source="Homelab" path="kubernetes.md" title="Kubernetes Homelab Cluster">\n'
|
||||
"Talos Linux on three nodes.\n"
|
||||
"</document>"
|
||||
)
|
||||
docs_block = "\n<documents>\n" + block + "\n</documents>"
|
||||
docs_block = "\n<documents>\n" + block + "\n</documents>" + "\n" + TOOLS_SECTION
|
||||
high_plain = _base("HIGH") + docs_block
|
||||
high_steered = _base("HIGH") + "\n" + build_steering_section(["be concise"]) + docs_block
|
||||
low_plain = (
|
||||
|
||||
@@ -4,7 +4,7 @@ from __future__ import annotations
|
||||
import json
|
||||
|
||||
from app.api.chat import sse_event
|
||||
from app.schemas import ChatErrorEvent, ChatThinkingEvent
|
||||
from app.schemas import ChatErrorEvent, ChatThinkingEvent, ChatToolEvent
|
||||
|
||||
|
||||
def _payload(frame: str) -> dict:
|
||||
@@ -76,3 +76,27 @@ def test_thinking_event_shape_is_type_and_text_only() -> None:
|
||||
dumped = ChatThinkingEvent(text="hmm").model_dump()
|
||||
assert set(dumped.keys()) == {"type", "text"}
|
||||
assert dumped["type"] == "thinking" # default — call sites never spell it out
|
||||
|
||||
|
||||
def test_tool_frame_serializes_exactly() -> None:
|
||||
"""Phase 37 (PLAN §4 extension): the ``tool`` frame is exactly
|
||||
``{type: "tool", name: str, argument: str | null}`` — one per
|
||||
model-requested document tool call, streamed ahead of the ``delta``
|
||||
frames of the answer."""
|
||||
frame = sse_event(ChatToolEvent(name="read_document", argument="S/p.md").model_dump())
|
||||
assert frame == 'data: {"type": "tool", "name": "read_document", "argument": "S/p.md"}\n\n'
|
||||
assert _payload(frame) == {"type": "tool", "name": "read_document", "argument": "S/p.md"}
|
||||
|
||||
|
||||
def test_tool_frame_argument_is_null_for_parameterless_tools() -> None:
|
||||
"""``list_documents`` takes no parameters, so its frame's ``argument``
|
||||
serializes as JSON null (the client renders the name alone)."""
|
||||
dumped = ChatToolEvent(name="list_documents").model_dump()
|
||||
assert dumped == {"type": "tool", "name": "list_documents", "argument": None}
|
||||
assert _payload(sse_event(dumped))["argument"] is None
|
||||
|
||||
|
||||
def test_tool_event_shape_is_type_name_argument_only() -> None:
|
||||
dumped = ChatToolEvent(name="read_document", argument="S/p.md").model_dump()
|
||||
assert set(dumped.keys()) == {"type", "name", "argument"}
|
||||
assert dumped["type"] == "tool" # default — call sites never spell it out
|
||||
|
||||
Reference in New Issue
Block a user