feat(rag): agent document tools — list/read tools with env-tuned budgets, SSE tool events + "calling tool" UI
Grounded chat turns now run the agent loop (app/rag/agent.py) instead
of a bare chat_stream: while the per-turn budgets last
(BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each) the model
gets list_documents (the indexed catalog, /api/docs order) and
read_document (full text, never truncated — A7-revised contract); once
both budgets are spent the tools key is dropped from the request and
the model must answer. Rejected calls (unknown tool, unknown/missing
path, document already in context, spent budget) consume no budget.
Budgets 0/0 make exactly one tools=None request — byte-identical to
the pre-phase path (budgets-as-kill-switch). Deflected turns keep the
direct chat_stream (A8 unchanged; the LOW prompt never carries the
<tools> section).
SSE contract gains {"type":"tool","name":...,"argument":
"source/path"|null} frames ahead of the answer deltas (PLAN §4
extension, owner permission 2026-08-26); done.sources, query_log.sources
and the per-turn log line (gains tool_calls=N) report the retrieval
docs + read docs, deduped. The UI shows a "calling tool"
button/label state and one visible .tool-call line per call above the
answer; the lines persist with the chat record and re-render on
reload. chat_stream passes tools through and accumulates streaming
tool_calls deltas into ToolCallPiece (tools=None stays byte-identical).
E2E: deterministic mock tool flow ("use your tools" + <tools> marker:
list -> read first catalog line -> quoted answer) plus the story suite
(marker flow, reload re-render, plain/deflected no-tool regressions).
Docs: .env.example + README (the two tools, the budgets, the SSE tool
frame, the "calling tool" UI state).
probe: turbo tool_calls=supported 2026-08-26 (uv run python -m
scripts.llm_probe --tools — non-streaming + streaming
finish_reason=tool_calls, indexed delta.tool_calls partials)
This commit is contained in:
+19
-1
@@ -178,6 +178,16 @@ stuck button).
|
||||
> live 2026-08-23; `BOR_STREAM_THINKING=0` suppresses the frames
|
||||
> server-side). `delta` and `done` shapes are unchanged — a recorded
|
||||
> extension of A15, not a silent deviation.
|
||||
>
|
||||
> **SSE revision (phase 37, owner permission 2026-08-26):** the contract
|
||||
> gains a second event type — `{"type":"tool","name":"…","argument":…}` —
|
||||
> carrying the model's document tool calls on grounded turns (phase 37:
|
||||
> `list_documents` / `read_document`, budgeted by `BOR_AGENT_LIST_CALLS`
|
||||
> / `BOR_AGENT_READ_CALLS`; `argument` is `"source/path"` for
|
||||
> `read_document`, null otherwise). Client rule: render each `tool` frame
|
||||
> as a "calling tool" line/state (task 05); `delta` and `done` shapes are
|
||||
> unchanged — the read document is reflected in `done.sources` instead
|
||||
> (deduped) — a recorded extension of A15, not a silent deviation.
|
||||
|
||||
---
|
||||
|
||||
@@ -382,10 +392,18 @@ on the sources and viewer pages (ids shared with chat),
|
||||
- **App logs:** single-line `timestamp LEVEL logger :: message` on stdout;
|
||||
uvicorn access logs on. INFO by default (`BOR_LOG_LEVEL`).
|
||||
- **Per-chat-turn log line (required):**
|
||||
`question=… embed_ms=… top_score=… fts_hits=… tuning=N threshold=… deflected=… sources=… thinking_chars=… total_ms=…`
|
||||
`question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N kb_chars=N threshold=… deflected=… sources=… thinking_chars=… tool_calls=N total_ms=…`
|
||||
(`thinking_chars=` counts the turn's reasoning chars — phase 17, owner
|
||||
permission 2026-08-23 — and is counted even when `BOR_STREAM_THINKING=0`
|
||||
suppresses the frames.)
|
||||
|
||||
> **Log-line revision (phase 37, owner permission 2026-08-26):** the
|
||||
> required per-turn line gains `tool_calls=N` after `thinking_chars=` —
|
||||
> the count of agent tool executions that consumed budget on the turn
|
||||
> (phase 37's `list_documents` / `read_document`; rejected calls do not
|
||||
> count, and deflected turns run no tools). `summary_hits=` (phase 30)
|
||||
> and `kb_chars=` (phase 31) are recorded here as well; `sources=` lists
|
||||
> the retrieval docs plus any agent-read documents, deduped.
|
||||
- **Importer logs:** per-file `added|updated|unchanged|pruned` + summary
|
||||
(counts, embedding batches, total time).
|
||||
- **`query_log` table:** durable record of every question (score, deflection,
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
# Task 01 — Live tool-calling probe
|
||||
|
||||
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "The agent should be able to list the available sources as a tool and the read the ones it thinks are relevant"`
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Verify against the live aipi endpoint whether the `turbo` chat model supports OpenAI-style `tools` + streaming `tool_calls` before building the loop — the phase-17 "verified live" convention — and record which path the phase takes.
|
||||
|
||||
## Work
|
||||
1. `scripts/llm_probe.py` — add a `--tools` flag: when set, send (a) a non-streaming `chat/completions` request with one trivial function (e.g. `get_time`, no parameters) and a user message that makes calling it natural, and (b) the same with `stream=True`; for each, print whether `finish_reason` is `tool_calls`, the parsed `function.name`/`arguments`, and (streaming) whether the calls arrive as `delta.tool_calls` chunks with `index`/`id`/partial `function.arguments` (the OpenAI wire convention). Reuse the existing env reading (`BOR_LLM_BASE_URL` / API key / `BOR_LLM_CHAT_MODEL`) and the script's existing output style.
|
||||
2. Run it against the live endpoint (`uv run python -m scripts.llm_probe --tools`) and classify the verdict:
|
||||
- **supported** → the phase uses OpenAI `tools`/`tool_calls` (tasks 02–03 as written).
|
||||
- **not supported** → the phase uses the prompt-based structured-call fallback (task 03 documents it): the model is instructed to emit a single JSON block (`{"tool": "list_documents"}` / `{"tool": "read_document", "source": …, "path": …}`) before answering; `run_agent` parses it out of the content stream; the SSE `tool` contract and the budgets are identical.
|
||||
- **intermittent** → treat as not supported (fail-loud house style) and note it.
|
||||
3. Record the verdict + date where it will be read later: the task-03 `app/rag/agent.py` module docstring (task 03 writes it) and the phase commit message (task 06) — e.g. `probe: turbo tool_calls=streaming-ok 2026-08-26`.
|
||||
|
||||
## Testing & Quality
|
||||
- The probe is a CLI script (no `app/` coverage impact). If the parsing of the probe response is factored into a function, add a small unit test for it.
|
||||
- `uv run pytest` green (no regressions); ruff + pyright clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run python -m scripts.llm_probe --tools` runs and prints a clear supported / not-supported verdict for both the non-streaming and the streaming request.
|
||||
- [ ] The verdict (with date) is available for task 03's docstring and task 06's commit message.
|
||||
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Task 02 — LLM client: `tools` + tool-call streaming
|
||||
|
||||
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "…read the ones it thinks are relevant… These values should be configured by environment variables"` (the client plumbing the env-tuned loop runs on)
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Teach `LLMClient.chat_stream` to pass an OpenAI `tools` list and to accumulate streaming `tool_calls` deltas into typed pieces — with `tools=None` producing a byte-identical request to today.
|
||||
|
||||
## Work
|
||||
1. `app/rag/llm.py`
|
||||
- New frozen dataclass next to `StreamPiece`:
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class ToolCallPiece:
|
||||
"""One model-requested tool call accumulated from stream deltas (phase 37)."""
|
||||
id: str # the model's tool_call id; synthesized "call_<index>" when absent
|
||||
name: str # "list_documents" | "read_document" (whatever AGENT_TOOLS names)
|
||||
arguments: dict[str, Any]
|
||||
```
|
||||
- `chat_stream(self, messages, tools: list[dict[str, Any]] | None = None)`:
|
||||
- When *tools* is not None, pass `tools=tools` to `chat.completions.create`; when None, do **not** include the key (byte-identical request to today).
|
||||
- In the chunk loop, accumulate `delta.tool_calls` (a list of partials keyed by `index`; `id` and `function.name` arrive on the first partial for an index, `function.arguments` arrives in fragments to concatenate).
|
||||
- At stream end (or when a chunk carries `finish_reason == "tool_calls"`), for each accumulated call **in index order** yield `ToolCallPiece(id, name, json.loads(arguments) or {})`.
|
||||
- Malformed `arguments` JSON → raise `LLMError` (fail-loud house style — a silently dropped tool call would corrupt the loop).
|
||||
- Return annotation becomes `AsyncIterator[StreamPiece | ToolCallPiece]`; update the docstring (wire convention + a pointer to the task-01 probe verdict).
|
||||
- **If the task-01 verdict is "not supported"** (prompt-based fallback): skip the `tools` parameter and the delta accumulation entirely; keep `StreamPiece` unchanged; the JSON-block parse helper lands in `app/rag/agent.py` (task 03) instead.
|
||||
2. `app/api/chat.py` — update the phase-17 typing import line (`StreamPiece # noqa: F401`) to also import `ToolCallPiece` so pyright sees the union; the dispatch wiring itself is task 04.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (`tests/unit/` — extend the existing `chat_stream` tests): synthetic chunk sequences — (a) a content-only stream is unchanged (no `ToolCallPiece`; the captured `create()` kwargs have no `tools` key); (b) a tool-call stream with partials across chunks (name on the first, arguments in 2–3 fragments) → one `ToolCallPiece` with the merged JSON; (c) two calls in one stream (indices 0 and 1) → both, in index order; (d) malformed arguments JSON → `LLMError`; (e) `tools=[…]` present in the request when passed.
|
||||
- Coverage: **>90%** on the modified module.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `chat_stream(messages)` (no tools) — all existing unit tests green unchanged.
|
||||
- [ ] The new tool-call accumulation tests green; full `uv run pytest` green.
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Task 03 — The basic agent loop (`app/rag/agent.py`)
|
||||
|
||||
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "it gets one opportunity to list documents and then one opportunity to add exactly one extra document to its context before being required to answer. These values should be configured by environment variables."`
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
A testable agent loop: the model gets the two tools while budgets last, the app executes them against Postgres, and once both budgets are spent the tools are dropped so the model is required to answer.
|
||||
|
||||
## Work
|
||||
1. `app/config.py` — two settings (documented, `BOR_` prefix per house style, near the RAG-tuning block):
|
||||
```python
|
||||
#: Per-turn opportunities to call the `list_documents` agent tool (phase 37);
|
||||
#: 0 disables the tool entirely (pre-phase behavior with both at 0).
|
||||
agent_list_calls: int = 1
|
||||
#: Per-turn opportunities to call `read_document` (phase 37); 0 disables.
|
||||
agent_read_calls: int = 1
|
||||
```
|
||||
2. `app/rag/agent.py` — new module. The module docstring carries: the phase, the loop contract, and the task-01 probe verdict + date (which path is in use — OpenAI tool_calls vs the prompt-based fallback).
|
||||
- `AGENT_TOOLS`: the two OpenAI function definitions — `list_documents` (no parameters; description: "List every document indexed in the knowledge base, one `source/path — title` line each") and `read_document` (`source` + `path` required; description: "Add the full content of exactly one more indexed document to your context").
|
||||
- DB accessors (module-level functions so unit tests can monkeypatch them):
|
||||
- `list_catalog(db) -> list[tuple[str, str, str]]` — `select(Document.source, Document.path, Document.title).order_by(Document.source, Document.path)` (same order as `GET /api/docs`).
|
||||
- `find_document(db, source, path) -> Document | None`.
|
||||
- `@dataclass AgentHolder: read_docs: list[Document] = field(default_factory=list); tool_calls: int = 0` — the API layer (task 04) reads it after the stream.
|
||||
- `async def run_agent(llm, db, *, system_prompt, user_message, seed_docs, settings, holder) -> AsyncIterator[StreamPiece | ToolCallPiece]`:
|
||||
1. `messages = [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_message}]`; `list_left = settings.agent_list_calls`; `read_left = settings.agent_read_calls`; `tools = AGENT_TOOLS if (list_left or read_left) else None`; `rounds = 0`; cap `max_rounds = 2 + settings.agent_list_calls + settings.agent_read_calls` (belt-and-braces — every tool round must consume a budget, so the cap only catches pathological streams).
|
||||
2. Loop: stream `llm.chat_stream(messages, tools=tools)`; yield every piece as it arrives. Collect any `ToolCallPiece` (handle the first; if a stream yields both content and a tool call — rare — the content stays (it was already emitted) and the tool still runs).
|
||||
- **No tool call** → return (the answer was streamed).
|
||||
- **`list_documents`**: `list_left > 0` → `list_left -= 1`, `holder.tool_calls += 1`, result = `f"{n} documents:\n" + "\n".join(f"{s}/{p} — {t}")` (uncapped in v1 — **ASSUMPTION: the catalog is not truncated; the UI never shows it, only the model does**). Else result = `"No listing budget left — answer with what you have."` (no budget consumed).
|
||||
- **`read_document`**: arguments must carry `source` and `path`. If that document is already in `seed_docs` or `holder.read_docs` → result = `"Already in your context."` (no budget consumed, no append). Else if `read_left > 0` → `doc = find_document(db, source, path)`; found → `read_left -= 1`, `holder.tool_calls += 1`, `holder.read_docs.append(doc)`, result = `f"Document {source}/{path}:\n{doc.content}"` (**full text, never truncated — the A7-revised contract**); not found → result = `f"No document at {source}/{path} — check the list_documents output."` (no budget consumed). `read_left == 0` → result = `"No reading budget left — answer with what you have."`.
|
||||
- **Unknown tool name** → result = `"Unknown tool."` (no budget consumed).
|
||||
- Append to `messages`: the assistant tool-call message (`{"role": "assistant", "content": None, "tool_calls": [{"id": tc.id, "type": "function", "function": {"name": tc.name, "arguments": json.dumps(tc.arguments)}}]}`) + the tool result (`{"role": "tool", "tool_call_id": tc.id, "content": result}`); then `tools = None if (list_left == 0 and read_left == 0) else AGENT_TOOLS` (once both budgets are spent, the next request must be answered).
|
||||
- `rounds += 1`; if `rounds >= max_rounds` → force one final `chat_stream(messages, tools=None)` (yield its pieces) and return.
|
||||
- **Prompt-based fallback (only if the task-01 verdict is "not supported")**: instead of `ToolCallPiece`s, scan each streamed content turn for a leading JSON block matching `{"tool": …}` (strip leading whitespace, parse with `json`; the block must be the first non-whitespace content of the turn). On a match: strip the block from the emitted stream (re-emit only trailing content, if any), then run the identical budget/result machinery. The SSE `tool` events and the budgets are unchanged either way.
|
||||
3. `app/rag/prompts.py` — add the `<tools>` instructions section to the **HIGH prompt only** (a new constant, appended after the mode body; the LOW/deflection prompt stays byte-identical). Wording (tune against the live model if needed, but keep it stable — the E2E mock keys off the `<tools>` marker's *presence*, not the wording):
|
||||
> If the documents in your context reference other files, or you need content that is not included above, call `list_documents` to see what is indexed, then `read_document` to pull in exactly one more document. Answer as soon as you have what you need — do not read more than one extra document.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (`tests/unit/test_agent.py`, new): a scripted fake LLM (canned stream sequences) + monkeypatched `list_catalog`/`find_document` + a `Settings` with budgets set:
|
||||
- list → read → answer: event order (tool pieces before content), `holder.read_docs == [doc]`, `holder.tool_calls == 2`, the request after budgets are spent carries `tools=None`, the follow-up request contains the assistant tool-call + tool-result messages.
|
||||
- Budgets: `read_left` exhausted → a second `read_document` gets "No reading budget left" and appends nothing; `agent_list_calls=0` + `agent_read_calls=0` → exactly one request with `tools=None` (byte-identical single-call path).
|
||||
- Edge: reading a doc already in `seed_docs` → "Already in your context." (no budget consumed); unknown path → "No document at …"; unknown tool name → "Unknown tool."; the round cap forces a final no-tools answer.
|
||||
- The HIGH prompt gains the `<tools>` section; the LOW prompt is byte-identical to pre-phase (assert against the existing prompt-test fixtures).
|
||||
- Integration (`tests/integration/`): `list_catalog` ordering + `find_document` hit/None against real Postgres (follow the existing DB-test patterns).
|
||||
- Coverage: **>90%** on the new module.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `app/rag/agent.py` exists with the loop above; the module docstring records the probe verdict + date (task 01).
|
||||
- [ ] The budget matrix (0/1) unit tests green, including "budgets 0/0 ⇒ one request, no tools".
|
||||
- [ ] HIGH prompt carries `<tools>`; LOW prompt byte-identical to pre-phase.
|
||||
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Task 04 — `POST /api/chat`: `tool` SSE events + agent wiring
|
||||
|
||||
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "This will require some reconfiguring of the UI since it will now need to show 'calling tool' in addition to 'thinking' and it will need a basic agent loop."`
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
Wire the loop into the chat endpoint: the new SSE `tool` event, the agent on grounded turns only, and the read document reflected in `done.sources`, `query_log`, and the per-turn log line.
|
||||
|
||||
## Work
|
||||
1. `app/schemas.py` — add next to `ChatThinkingEvent`:
|
||||
```python
|
||||
class ChatToolEvent(BaseModel):
|
||||
"""SSE frame for one agent tool call (phase 37, PLAN §4 extension)."""
|
||||
type: Literal["tool"] = "tool"
|
||||
name: str # "list_documents" | "read_document"
|
||||
argument: str | None = None # "source/path" for read_document
|
||||
```
|
||||
2. `app/api/chat.py`
|
||||
- In `stream()`, after `plan_turn`: when `not plan.deflected` → run `run_agent(llm, db, system_prompt=plan.system_prompt, user_message=request.message, seed_docs=plan.docs, settings=settings, holder=holder)`; when deflected → keep the current direct `chat_stream` (A8 byte-identical).
|
||||
- Event mapping: `StreamPiece` thinking/delta exactly as today (including the `BOR_STREAM_THINKING` kill-switch and the `thinking_chars` count); `ToolCallPiece` →
|
||||
`yield sse_event(ChatToolEvent(name=tc.name, argument=f"{tc.arguments.get('source')}/{tc.arguments.get('path')}" if tc.name == "read_document" else None).model_dump())`.
|
||||
- `done` event: `sources` = `plan.docs + holder.read_docs` deduped by `(source, path)`, order preserved; the `ChatDoneEvent` shape otherwise unchanged.
|
||||
- `query_log.sources`: the same combined list (replaces the current `source_paths` build).
|
||||
- Per-turn log line: add `tool_calls={holder.tool_calls}` after `thinking_chars=` (PLAN §9 required-line extension — the phase 17/30/31 precedent).
|
||||
- Extend the module docstring with the phase-37 section (the flow, the grounded-only scope, the budgets, the "both budgets 0 ⇒ pre-phase behavior" note).
|
||||
3. `.agent/PLAN.md` — record two revision notes in the project's established format (phase 17/24 precedent, "owner permission 2026-08-26"): the §4 SSE contract gains `{"type":"tool","name":…,"argument":…}` (client rule: render as a "calling tool" line/state; `delta`/`done` unchanged) and the §9 per-turn log line gains `tool_calls=N`.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration (`tests/integration/test_chat*.py` — extend the existing SSE-contract suite, mock LLM that emits tool calls): (a) grounded turn with tool calls → event sequence `thinking?/tool/tool/delta…/done`, `done.sources` includes the read doc, the `query_log` row's sources match, the log line carries `tool_calls=2`; (b) deflected turn → no `tool` events, the sequence byte-identical to today; (c) budgets 0/0 → no `tool` events, the mock received no `tools` parameter, single-request path.
|
||||
- Coverage: **>90%** on the modified modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `POST /api/chat` (mock LLM, grounded) streams `tool` frames ahead of the `delta` frames; the deflected path is unchanged (existing suites green).
|
||||
- [ ] `done.sources`/`query_log` include the read document (deduped); the log line carries `tool_calls=N`.
|
||||
- [ ] The PLAN.md §4 + §9 revision notes exist (dated, owner permission 2026-08-26).
|
||||
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 05 — UI: the "calling tool" state
|
||||
|
||||
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "…it will now need to show 'calling tool' in addition to 'thinking'"`
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md`
|
||||
|
||||
## Objective
|
||||
The chat shell shows a "calling tool" state (button + status label) and a visible tool line in the bubble, distinct from the Thinking scratchpad; tool lines persist with the chat record (phase 14) and re-render after reload.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/app.js` — in the turn handler where the SSE frames are dispatched (the same switch that handles `thinking` and `delta`):
|
||||
- New `tool` branch:
|
||||
- **Status/label:** keep `uiState = thinking` (the button stays disabled — never stale, PLAN §7.4) and set the typing-indicator label to "Brain of Reese is listing documents" (`name: "list_documents"`) / "Brain of Reese is reading <argument>" (`name: "read_document"`); the elapsed-seconds hint (`thinkingClock` aria-label) keeps working through tool frames.
|
||||
- **Bubble line:** append a tool line to the same wrap the thinking block uses (above the answer, beside/below the Thinking summary): `<span class="tool-call">🔎 Listing documents</span>` / `<span class="tool-call">📄 Reading <code>source/path</code></span>` — a real visible line with its own icon + color, distinct from the Thinking block (styles step). Multiple calls append multiple lines, in order.
|
||||
- Tolerate tool frames interleaved with thinking frames (append-only, the same rule as thinking); a tool frame after the first `delta` (should not happen in v1 — the loop completes before the answer stream) still appends rather than crashes.
|
||||
- **Persistence (phase 14 convention):** the saved record gains an optional `tools: [{name, argument}]` array next to `thinking`; re-render the tool lines when a record is loaded (same code path as the thinking re-render).
|
||||
- Keep the new label strings as plain literals — phase 39 centralizes brand strings; do **not** introduce a brand helper here.
|
||||
2. `frontend/assets/styles.css` — `.tool-call` style: inline row, icon + text, an accent color distinguishable from the thinking block's, `code` styling for the path, contrast ≥ 4.5:1 in both themes; no layout shift on append (the centered 46rem chat column is untouched — no new container).
|
||||
3. UI Structure Check (AGENTS.md rule 5) before finalizing: semantic landmarks unchanged; the tool lines live inside the `#messages` `aria-live="polite"` region (announced to screen readers); focus-visible unaffected (the lines are not interactive); no new top-level landmarks.
|
||||
|
||||
## Testing & Quality
|
||||
- No new Python logic — gated by the story E2E (task 06) plus the existing frontend-adjacent suites staying green.
|
||||
- No CDN (AGENTS.md rule 6): no new `<script>`/`<link>` tags.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] During a tool call the button label shows the "calling tool" text (not a stale "Thinking…") and the tool lines render above the answer.
|
||||
- [ ] After a page reload, the persisted record re-renders the tool line(s).
|
||||
- [ ] `uv run pytest` green (no regressions); `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -36,6 +36,10 @@ BOR_HYBRID_VECTOR_CANDIDATES=100 # cosine list width for the fusion
|
||||
BOR_HYBRID_LEXICAL_CANDIDATES=30 # FTS list width for the fusion
|
||||
BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant
|
||||
|
||||
# --- Agent document tools (phase 37: grounded turns may list + read) ---
|
||||
BOR_AGENT_LIST_CALLS=1 # per-turn list_documents opportunities (0 disables the tool)
|
||||
BOR_AGENT_READ_CALLS=1 # per-turn read_document opportunities (0 disables the tool)
|
||||
|
||||
# --- Import scope (A9 formats; may only narrow, never widen) ---
|
||||
# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py
|
||||
# BOR_SUGGESTIONS=["How is my Kubernetes cluster set up?"] # JSON list of onboarding chips
|
||||
|
||||
@@ -129,6 +129,41 @@ exactly as before.
|
||||
To hide it, set `BOR_STREAM_THINKING=0` — the `thinking` events stop
|
||||
(the per-turn log line still counts `thinking_chars`).
|
||||
|
||||
## Agent document tools (list + read)
|
||||
|
||||
Retrieval only puts the top documents in context. When an answer depends
|
||||
on a file a note *references* ("the exact JSON shape is in
|
||||
example-record-file.json"), the model can extend its own context with two
|
||||
server-side tools — on **grounded** (high-relevance) turns only:
|
||||
|
||||
* **`list_documents`** — lists every indexed document, one
|
||||
`source/path — title` line each (the same order as the Sources page);
|
||||
* **`read_document(source, path)`** — appends the **full** text of
|
||||
exactly one more indexed document to the context (never truncated).
|
||||
|
||||
Each call the model requests is executed against Postgres only (no extra
|
||||
LLM round trip) and streamed as an SSE `tool` frame ahead of the answer —
|
||||
`{"type": "tool", "name": …, "argument": "source/path" | null}`. In the
|
||||
chat, each call shows a **"calling tool" state** in addition to
|
||||
"thinking": the send button keeps its busy state ("Calling tool…") and a
|
||||
visible tool line (`🔎 Listing documents` / `📄 Reading source/path`) lands
|
||||
above the answer, one per call, in order. The tool lines persist with the
|
||||
message, so a reloaded conversation re-renders them. The read document is
|
||||
reflected in the answer's **source chips** and in the `query_log` row.
|
||||
|
||||
The opportunities are budgeted per turn:
|
||||
|
||||
| Env | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `BOR_AGENT_LIST_CALLS` | `1` | `list_documents` calls per turn (0 disables the tool) |
|
||||
| `BOR_AGENT_READ_CALLS` | `1` | `read_document` calls per turn (0 disables the tool) |
|
||||
|
||||
Once both budgets are spent the tools are dropped from the LLM request
|
||||
and the model must answer. `BOR_AGENT_LIST_CALLS=0 BOR_AGENT_READ_CALLS=0`
|
||||
reproduces the pre-agent chat behavior exactly (no `tools` in the
|
||||
request, no `tool` frames). Deflected turns run no tools at all — the
|
||||
low-relevance path is unchanged.
|
||||
|
||||
## Admin & sign-in
|
||||
|
||||
Brain of Reese has exactly **one account: the admin (you)**. Signing in
|
||||
@@ -559,6 +594,8 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
|
||||
| `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | cosine list width for the RRF fusion |
|
||||
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
|
||||
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
|
||||
| `BOR_AGENT_LIST_CALLS` | `1` | per-turn `list_documents` tool opportunities on grounded turns (0 disables the tool) |
|
||||
| `BOR_AGENT_READ_CALLS` | `1` | per-turn `read_document` tool opportunities on grounded turns (0 disables the tool) |
|
||||
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
|
||||
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*) |
|
||||
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) |
|
||||
|
||||
+93
-5
@@ -46,6 +46,28 @@ prompts stay byte-identical to the pre-phase text (phase 15
|
||||
convention); ``TurnPlan.kb_chars`` records the length of the stored
|
||||
outline (0 when absent) and the per-turn log line records
|
||||
``kb_chars=N`` after ``tuning=N`` (PLAN §9 line extension).
|
||||
|
||||
Agent document tools (phase 37, PLAN §4 extension, owner permission
|
||||
2026-08-26): a **grounded** turn (``not plan.deflected``) no longer
|
||||
streams a bare ``chat_stream`` — it runs the agent loop
|
||||
(``app.rag.agent.run_agent``), which offers the model the two
|
||||
server-side tools ``list_documents`` / ``read_document`` while the
|
||||
per-turn budgets (``BOR_AGENT_LIST_CALLS`` / ``BOR_AGENT_READ_CALLS``,
|
||||
default 1 each) last; once both budgets are spent the ``tools`` key is
|
||||
dropped from the request and the model must answer. Each
|
||||
model-requested call streams as an SSE ``tool`` event —
|
||||
``{"type": "tool", "name": …, "argument": "source/path" | null}`` —
|
||||
ahead of the answer's ``delta`` frames. ``done.sources``,
|
||||
``query_log.sources`` and the per-turn log line all report the same
|
||||
combined source list (retrieval docs + the agent's read docs, deduped
|
||||
by ``(source, path)``, order preserved), and the log line records
|
||||
``tool_calls=N`` after ``thinking_chars=N`` (PLAN §9 line extension —
|
||||
``N`` counts budget-consuming executions; rejected calls do not
|
||||
count). **Deflected turns keep the direct ``chat_stream`` — byte-
|
||||
identical to the pre-phase path (A8):** the LOW prompt never carries
|
||||
tools, and with **both budgets at 0** ``run_agent`` makes exactly one
|
||||
``tools=None`` request, reproducing the pre-phase behavior (budgets-
|
||||
as-kill-switch).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -64,11 +86,13 @@ from app.api.steering import load_steering_notes
|
||||
from app.config import Settings, get_settings
|
||||
from app.db import db_available, get_db
|
||||
from app.models import Document, QueryLog
|
||||
from app.rag.agent import AgentHolder, run_agent
|
||||
from app.rag.llm import (
|
||||
EmbeddingError,
|
||||
LLMClient,
|
||||
LLMError,
|
||||
StreamPiece, # noqa: F401 (phase 17 typing: chat_stream yields StreamPiece)
|
||||
StreamPiece, # type of the answer pieces streamed by the agent loop
|
||||
ToolCallPiece, # phase 37: one model-requested tool call
|
||||
)
|
||||
from app.rag.overview import load_kb_overview
|
||||
from app.rag.prompts import build_deflect_prompt, build_high_prompt
|
||||
@@ -79,6 +103,7 @@ from app.schemas import (
|
||||
ChatErrorEvent,
|
||||
ChatRequest,
|
||||
ChatThinkingEvent,
|
||||
ChatToolEvent,
|
||||
SourceRef,
|
||||
)
|
||||
|
||||
@@ -260,7 +285,6 @@ async def chat(
|
||||
).model_dump()
|
||||
)
|
||||
return
|
||||
source_paths = [f"{d.source}/{d.path}" for d in plan.docs]
|
||||
messages = [
|
||||
{"role": "system", "content": plan.system_prompt},
|
||||
{"role": "user", "content": request.message},
|
||||
@@ -271,9 +295,44 @@ async def chat(
|
||||
# ahead of the ``delta`` events (PLAN §4 extension); the
|
||||
# kill-switch (``BOR_STREAM_THINKING=0``) suppresses the
|
||||
# frames, not the counting.
|
||||
# Phase 37: a grounded turn runs the agent loop instead of a
|
||||
# bare ``chat_stream`` — its ``ToolCallPiece``s stream as
|
||||
# ``tool`` events ahead of the answer. A deflected turn keeps
|
||||
# the direct ``chat_stream`` (byte-identical, A8): the LOW
|
||||
# prompt never carries tools, and with both budgets at 0
|
||||
# ``run_agent`` is a single ``tools=None`` request anyway.
|
||||
holder = AgentHolder()
|
||||
answer_stream: AsyncIterator[StreamPiece | ToolCallPiece]
|
||||
if plan.deflected:
|
||||
answer_stream = llm.chat_stream(messages)
|
||||
else:
|
||||
answer_stream = run_agent(
|
||||
llm,
|
||||
db,
|
||||
system_prompt=plan.system_prompt,
|
||||
user_message=request.message,
|
||||
seed_docs=plan.docs,
|
||||
settings=settings,
|
||||
holder=holder,
|
||||
)
|
||||
thinking_chars = 0
|
||||
try:
|
||||
async for piece in llm.chat_stream(messages): # StreamPiece (phase 17)
|
||||
async for piece in answer_stream: # StreamPiece | ToolCallPiece
|
||||
if isinstance(piece, ToolCallPiece):
|
||||
# Phase 37 (PLAN §4 extension): one SSE ``tool``
|
||||
# frame per model-requested call; ``argument`` is the
|
||||
# read_document "source/path" (null otherwise).
|
||||
yield sse_event(
|
||||
ChatToolEvent(
|
||||
name=piece.name,
|
||||
argument=(
|
||||
f"{piece.arguments.get('source')}/{piece.arguments.get('path')}"
|
||||
if piece.name == "read_document"
|
||||
else None
|
||||
),
|
||||
).model_dump()
|
||||
)
|
||||
continue
|
||||
if piece.kind == "thinking":
|
||||
thinking_chars += len(piece.text)
|
||||
if settings.stream_thinking:
|
||||
@@ -293,8 +352,35 @@ async def chat(
|
||||
).model_dump()
|
||||
)
|
||||
return
|
||||
except Exception: # noqa: BLE001 — a tool call hit the DB mid-stream
|
||||
# Phase 37: tool execution (list_catalog / find_document) runs
|
||||
# inside the stream now; a mid-turn DB failure gets the same
|
||||
# structured ``error`` event as the pre-stream retrieval path.
|
||||
logger.exception(
|
||||
"chat: tool execution failed question=%r total_ms=%d",
|
||||
request.message,
|
||||
int((time.monotonic() - started) * 1000),
|
||||
)
|
||||
yield sse_event(
|
||||
ChatErrorEvent(
|
||||
detail="The knowledge base went offline mid-question — is Postgres up?"
|
||||
).model_dump()
|
||||
)
|
||||
return
|
||||
|
||||
# 4. Durable record + required per-turn log line (PLAN §9).
|
||||
# Phase 37: the agent's read documents join the retrieval's —
|
||||
# deduped by (source, path), order preserved — and the same
|
||||
# combined list feeds done.sources, query_log.sources and the
|
||||
# log line (empty on deflected turns: the agent never runs).
|
||||
cited_docs: list[Document] = []
|
||||
seen: set[tuple[str, str]] = set()
|
||||
for doc in [*plan.docs, *holder.read_docs]:
|
||||
key = (doc.source, doc.path)
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
cited_docs.append(doc)
|
||||
source_paths = [f"{d.source}/{d.path}" for d in cited_docs]
|
||||
total_ms = int((time.monotonic() - started) * 1000)
|
||||
try:
|
||||
db.add(
|
||||
@@ -314,7 +400,8 @@ async def chat(
|
||||
|
||||
logger.info(
|
||||
"question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d tuning=%d "
|
||||
"kb_chars=%d threshold=%.2f deflected=%s sources=%r thinking_chars=%d total_ms=%d",
|
||||
"kb_chars=%d threshold=%.2f deflected=%s sources=%r thinking_chars=%d "
|
||||
"tool_calls=%d total_ms=%d",
|
||||
request.message,
|
||||
embed_ms,
|
||||
plan.top_score,
|
||||
@@ -326,13 +413,14 @@ async def chat(
|
||||
plan.deflected,
|
||||
source_paths,
|
||||
thinking_chars,
|
||||
holder.tool_calls,
|
||||
total_ms,
|
||||
)
|
||||
yield sse_event(
|
||||
ChatDoneEvent(
|
||||
deflected=plan.deflected,
|
||||
sources=[
|
||||
SourceRef(source=d.source, path=d.path, title=d.title) for d in plan.docs
|
||||
SourceRef(source=d.source, path=d.path, title=d.title) for d in cited_docs
|
||||
],
|
||||
suggestions=plan.suggestions,
|
||||
).model_dump()
|
||||
|
||||
@@ -87,6 +87,13 @@ class Settings(BaseSettings):
|
||||
#: ``app.rag.overview``). Overflow is cut at the cap and the shared
|
||||
#: ``[…truncated…]`` marker is appended (summarizer convention).
|
||||
overview_input_max_chars: int = 40_000
|
||||
#: Per-turn opportunities to call the ``list_documents`` agent tool
|
||||
#: (phase 37, ``app.rag.agent``); 0 disables the tool entirely
|
||||
#: (pre-phase behavior with both budgets at 0).
|
||||
agent_list_calls: int = 1
|
||||
#: Per-turn opportunities to call the ``read_document`` agent tool
|
||||
#: (phase 37, ``app.rag.agent``); 0 disables the tool entirely.
|
||||
agent_read_calls: int = 1
|
||||
|
||||
# --- Hybrid retrieval (A7, revised 2026-08-21) ---
|
||||
# cosine top-N ∪ Postgres FTS top-N, fused with Reciprocal Rank Fusion
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
"""Agent loop: the grounded-turn document tools (phase 37, task 03).
|
||||
|
||||
Probe verdict (task 01 — ``uv run python -m scripts.llm_probe --tools``
|
||||
run live against aipi): **``probe: turbo tool_calls=supported 2026-08-26``**
|
||||
— ``turbo`` answers OpenAI ``tools`` requests with
|
||||
``finish_reason="tool_calls"`` and streams the calls as indexed
|
||||
``delta.tool_calls`` partials (id + name on the first partial, arguments
|
||||
in fragments). This module therefore uses the **native tool-calling
|
||||
path**: tool calls arrive as :class:`app.rag.llm.ToolCallPiece` values
|
||||
from ``chat_stream(messages, tools=AGENT_TOOLS)``. The prompt-based
|
||||
JSON-block fallback (documented in the task file) is *not* implemented —
|
||||
it exists only for a "not supported"/"intermittent" verdict, and the
|
||||
probe came back "supported".
|
||||
|
||||
Loop contract (one grounded chat turn; the API layer wires this in,
|
||||
task 04):
|
||||
|
||||
1. While budget remains the model is offered the two OpenAI functions in
|
||||
:data:`AGENT_TOOLS`: up to ``settings.agent_list_calls``
|
||||
(``BOR_AGENT_LIST_CALLS``, default 1) ``list_documents`` calls and up
|
||||
to ``settings.agent_read_calls`` (``BOR_AGENT_READ_CALLS``, default 1)
|
||||
``read_document`` calls. With both budgets at 0 the loop makes exactly
|
||||
one request with ``tools=None`` — byte-identical to the pre-phase chat
|
||||
path (budgets-as-kill-switch, phase 37 locked decision).
|
||||
2. Each tool call the model emits is executed server-side against
|
||||
Postgres only (no LLM, no network): ``list_documents`` returns the
|
||||
indexed catalog — one ``source/path — title`` line per document,
|
||||
``GET /api/docs`` order (uncapped in v1; the UI never shows it, only
|
||||
the model does) — and ``read_document`` returns the document's **full**
|
||||
content (A7-revised contract: never truncated).
|
||||
3. Rejected calls consume **no** budget and get a one-line refusal:
|
||||
unknown tool name → ``"Unknown tool."``; missing ``source``/``path``
|
||||
arguments; a document already in context (seed or previously read) →
|
||||
``"Already in your context."``; an unknown ``source/path`` →
|
||||
``"No document at …"``; an exhausted list/read budget → the matching
|
||||
``"No … budget left"`` refusal.
|
||||
4. Every executed call is appended back to the message history as the
|
||||
assistant tool-call message + the tool result, and the model is called
|
||||
again. Once **both** budgets are spent, ``tools`` is dropped from the
|
||||
request and the model must answer. Belt-and-braces round cap:
|
||||
``max_rounds = 2 + agent_list_calls + agent_read_calls`` (every tool
|
||||
round consumes a budget, so the cap only catches pathological streams
|
||||
that keep calling rejected tools) — at the cap the loop forces one
|
||||
final ``chat_stream(messages, tools=None)`` and returns.
|
||||
5. A rare stream that carries both content and a tool call keeps the
|
||||
content (it was already emitted) **and** still runs the tool.
|
||||
6. *holder* (an :class:`AgentHolder`) records the read documents and the
|
||||
number of budget-consuming tool executions; the API layer (task 04)
|
||||
reads it after the stream to extend ``done.sources`` /
|
||||
``query_log.sources`` and the per-turn log line (``tool_calls=N``).
|
||||
|
||||
The DB accessors (:func:`list_catalog`, :func:`find_document`) are
|
||||
module-level functions so unit tests can monkeypatch them without a
|
||||
database.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import AsyncIterator, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, cast
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag.llm import LLMClient, StreamPiece, ToolCallPiece
|
||||
|
||||
logger = logging.getLogger("app.agent")
|
||||
|
||||
#: The two agent tools (phase 37): OpenAI function definitions passed as
|
||||
#: ``tools=AGENT_TOOLS`` to ``chat_stream`` while the per-turn budgets
|
||||
#: (``BOR_AGENT_LIST_CALLS`` / ``BOR_AGENT_READ_CALLS``) remain.
|
||||
AGENT_TOOLS: list[dict[str, Any]] = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "list_documents",
|
||||
"description": (
|
||||
"List every document indexed in the knowledge base, one "
|
||||
"`source/path — title` line each"
|
||||
),
|
||||
"parameters": {"type": "object", "properties": {}, "required": []},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "read_document",
|
||||
"description": (
|
||||
"Add the full content of exactly one more indexed document "
|
||||
"to your context"
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"The document's source (a directory basename, "
|
||||
"e.g. 'Homelab')."
|
||||
),
|
||||
},
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"The document's path relative to its source "
|
||||
"directory."
|
||||
),
|
||||
},
|
||||
},
|
||||
"required": ["source", "path"],
|
||||
},
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
#: Tool refusal texts (phase 37): rejected calls consume no budget.
|
||||
LIST_EXHAUSTED = "No listing budget left — answer with what you have."
|
||||
READ_EXHAUSTED = "No reading budget left — answer with what you have."
|
||||
ALREADY_IN_CONTEXT = "Already in your context."
|
||||
UNKNOWN_TOOL = "Unknown tool."
|
||||
MISSING_READ_ARGS = "read_document requires string arguments 'source' and 'path'."
|
||||
|
||||
|
||||
def list_catalog(db: Session) -> list[tuple[str, str, str]]:
|
||||
"""Every indexed document as ``(source, path, title)``.
|
||||
|
||||
Ordered by ``(source, path)`` — the same order as ``GET /api/docs``.
|
||||
Module-level (not a method) so unit tests can monkeypatch it.
|
||||
"""
|
||||
rows = db.execute(
|
||||
select(Document.source, Document.path, Document.title).order_by(
|
||||
Document.source, Document.path
|
||||
)
|
||||
).all()
|
||||
return [(source, path, title) for source, path, title in rows]
|
||||
|
||||
|
||||
def find_document(db: Session, source: str, path: str) -> Document | None:
|
||||
"""The indexed document at ``(source, path)``, or ``None``.
|
||||
|
||||
Module-level (not a method) so unit tests can monkeypatch it.
|
||||
"""
|
||||
return db.scalar(
|
||||
select(Document).where(Document.source == source, Document.path == path)
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AgentHolder:
|
||||
"""Per-turn agent state the API layer reads after the stream (task 04).
|
||||
|
||||
``read_docs``: the documents ``read_document`` added to the context,
|
||||
in read order (deduped — re-reading a document appends nothing).
|
||||
``tool_calls``: how many tool executions consumed budget; rejected
|
||||
calls (unknown tool, unknown/missing document, exhausted budget,
|
||||
already-in-context) do not count. Drives the per-turn log line's
|
||||
``tool_calls=N`` field (task 04).
|
||||
"""
|
||||
|
||||
read_docs: list[Document] = field(default_factory=list)
|
||||
tool_calls: int = 0
|
||||
|
||||
|
||||
def _execute_tool(
|
||||
db: Session,
|
||||
call: ToolCallPiece,
|
||||
seed_docs: Sequence[Document],
|
||||
holder: AgentHolder,
|
||||
list_left: int,
|
||||
read_left: int,
|
||||
) -> tuple[str, int, int]:
|
||||
"""Execute one tool call server-side (DB only).
|
||||
|
||||
Returns ``(result, list_left, read_left)``. Rejected calls consume no
|
||||
budget; a successful read appends the :class:`Document` to
|
||||
``holder.read_docs`` and bumps ``holder.tool_calls``.
|
||||
"""
|
||||
if call.name == "list_documents":
|
||||
if list_left <= 0:
|
||||
return LIST_EXHAUSTED, list_left, read_left
|
||||
rows = list_catalog(db)
|
||||
listing = f"{len(rows)} documents:\n" + "\n".join(
|
||||
f"{source}/{path} — {title}" for source, path, title in rows
|
||||
)
|
||||
holder.tool_calls += 1
|
||||
return listing, list_left - 1, read_left
|
||||
if call.name == "read_document":
|
||||
raw_source = call.arguments.get("source")
|
||||
raw_path = call.arguments.get("path")
|
||||
source = raw_source.strip() if isinstance(raw_source, str) else ""
|
||||
path = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||
if not source or not path:
|
||||
return MISSING_READ_ARGS, list_left, read_left
|
||||
known = {(doc.source, doc.path) for doc in (*seed_docs, *holder.read_docs)}
|
||||
if (source, path) in known:
|
||||
return ALREADY_IN_CONTEXT, list_left, read_left
|
||||
if read_left <= 0:
|
||||
return READ_EXHAUSTED, list_left, read_left
|
||||
doc = find_document(db, source, path)
|
||||
if doc is None:
|
||||
return (
|
||||
f"No document at {source}/{path} — check the list_documents output.",
|
||||
list_left,
|
||||
read_left,
|
||||
)
|
||||
holder.read_docs.append(doc)
|
||||
holder.tool_calls += 1
|
||||
return f"Document {source}/{path}:\n{doc.content}", list_left, read_left - 1
|
||||
return UNKNOWN_TOOL, list_left, read_left
|
||||
|
||||
|
||||
async def run_agent(
|
||||
llm: LLMClient,
|
||||
db: Session,
|
||||
*,
|
||||
system_prompt: str,
|
||||
user_message: str,
|
||||
seed_docs: Sequence[Document],
|
||||
settings: Settings,
|
||||
holder: AgentHolder,
|
||||
) -> AsyncIterator[StreamPiece | ToolCallPiece]:
|
||||
"""Run the grounded-turn tool loop, yielding every stream piece.
|
||||
|
||||
Every piece (``thinking`` / ``content`` / tool calls) is yielded as it
|
||||
arrives; the API layer (task 04) turns tool-call pieces into SSE
|
||||
``tool`` events. After the loop finishes, *holder* carries the read
|
||||
documents and the budget-consuming tool count.
|
||||
|
||||
``seed_docs`` are the documents the retrieval already put in context
|
||||
(they shape the *system_prompt* the caller built); re-reading one of
|
||||
them is rejected as "Already in your context." without spending budget.
|
||||
"""
|
||||
messages: list[dict[str, Any]] = [
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": user_message},
|
||||
]
|
||||
list_left = settings.agent_list_calls
|
||||
read_left = settings.agent_read_calls
|
||||
tools: list[dict[str, Any]] | None = AGENT_TOOLS if (list_left or read_left) else None
|
||||
# Every tool round consumes a budget, so this cap only catches
|
||||
# pathological streams that keep calling rejected tools (belt and
|
||||
# braces — the budgets already force the answer after
|
||||
# list + read rounds).
|
||||
max_rounds = 2 + settings.agent_list_calls + settings.agent_read_calls
|
||||
rounds = 0
|
||||
while True:
|
||||
calls: list[ToolCallPiece] = []
|
||||
async for piece in llm.chat_stream(
|
||||
cast("list[dict[str, str]]", messages), tools=tools
|
||||
):
|
||||
if isinstance(piece, ToolCallPiece):
|
||||
calls.append(piece)
|
||||
yield piece
|
||||
if not calls:
|
||||
return # the answer was streamed
|
||||
call = calls[0] # a stream can carry several calls; run the first
|
||||
result, list_left, read_left = _execute_tool(
|
||||
db, call, seed_docs, holder, list_left, read_left
|
||||
)
|
||||
logger.info(
|
||||
"agent tool=%s args=%s budget list_left=%d read_left=%d",
|
||||
call.name,
|
||||
json.dumps(call.arguments, ensure_ascii=False)[:200],
|
||||
list_left,
|
||||
read_left,
|
||||
)
|
||||
messages.append(
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": call.id,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": call.name,
|
||||
"arguments": json.dumps(call.arguments),
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
|
||||
tools = None if (list_left == 0 and read_left == 0) else AGENT_TOOLS
|
||||
rounds += 1
|
||||
if rounds >= max_rounds:
|
||||
logger.warning(
|
||||
"agent round cap reached (rounds=%d) — forcing a final "
|
||||
"no-tools answer",
|
||||
rounds,
|
||||
)
|
||||
async for piece in llm.chat_stream(
|
||||
cast("list[dict[str, str]]", messages), tools=None
|
||||
):
|
||||
yield piece
|
||||
return
|
||||
+139
-15
@@ -4,9 +4,10 @@ Provides the embeddings surface (importer, retrieval), one-shot chat
|
||||
completions (phase 30: the ``lite`` model summarizes non-markdown
|
||||
documents at import time), and chat streaming (PLAN A15) for the RAG
|
||||
pipeline. Chat streaming yields typed :class:`StreamPiece` values
|
||||
(phase 17): aipi's ``turbo`` model streams its reasoning as
|
||||
``delta.reasoning_content`` chunks (deepseek/litellm wire convention,
|
||||
verified live 2026-08-23) **before** the answer's
|
||||
(phase 17) and — when the caller passes a ``tools`` list —
|
||||
:class:`ToolCallPiece` values (phase 37): aipi's ``turbo`` model streams
|
||||
its reasoning as ``delta.reasoning_content`` chunks (deepseek/litellm
|
||||
wire convention, verified live 2026-08-23) **before** the answer's
|
||||
``delta.content`` chunks, and reasoning counts against ``max_tokens``
|
||||
(an answer can in principle be empty).
|
||||
|
||||
@@ -17,10 +18,11 @@ vectors that pgvector rejects.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import AsyncIterator
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal, cast
|
||||
from typing import Any, Literal, cast
|
||||
|
||||
from openai import AsyncOpenAI
|
||||
from openai.types.chat import ChatCompletionMessageParam
|
||||
@@ -55,6 +57,78 @@ class StreamPiece:
|
||||
text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ToolCallPiece:
|
||||
"""One model-requested tool call accumulated from stream deltas (phase 37).
|
||||
|
||||
``id`` is the model's tool_call id (synthesized as ``call_<index>``
|
||||
when the wire never carried one), ``name`` is the function name
|
||||
(whatever the caller's ``tools`` list names — for the agent loop,
|
||||
``list_documents`` / ``read_document``), and ``arguments`` is the
|
||||
parsed JSON object (``{}`` when the model sent none).
|
||||
"""
|
||||
|
||||
id: str # the model's tool_call id; synthesized "call_<index>" when absent
|
||||
name: str # "list_documents" | "read_document" (whatever AGENT_TOOLS names)
|
||||
arguments: dict[str, Any]
|
||||
|
||||
|
||||
@dataclass
|
||||
class _ToolCallSlot:
|
||||
"""Mutable accumulator for one streamed tool call (phase 37, private).
|
||||
|
||||
``id`` and ``function.name`` arrive on the first partial for an index;
|
||||
``function.arguments`` arrives in fragments to concatenate (OpenAI wire
|
||||
convention, verified live against aipi 2026-08-26).
|
||||
"""
|
||||
|
||||
id: str | None = None
|
||||
name: str = ""
|
||||
arguments: str = ""
|
||||
|
||||
|
||||
def _materialize_tool_calls(
|
||||
slots: dict[int, _ToolCallSlot],
|
||||
) -> list[ToolCallPiece]:
|
||||
"""Turn accumulated slots into ordered :class:`ToolCallPiece` values.
|
||||
|
||||
Malformed ``arguments`` JSON raises :class:`LLMError` — a silently
|
||||
dropped tool call would corrupt the agent loop (fail-loud house
|
||||
style). Empty/``null`` arguments become ``{}`` (a no-parameter call
|
||||
such as ``list_documents``).
|
||||
"""
|
||||
pieces: list[ToolCallPiece] = []
|
||||
for index in sorted(slots):
|
||||
slot = slots[index]
|
||||
raw = slot.arguments.strip()
|
||||
label = slot.name or f"index {index}"
|
||||
if raw:
|
||||
try:
|
||||
parsed: Any = json.loads(raw)
|
||||
except json.JSONDecodeError as e:
|
||||
raise LLMError(
|
||||
f"model sent malformed tool-call arguments for '{label}': "
|
||||
f"{raw[:200]!r} ({e})"
|
||||
) from e
|
||||
else:
|
||||
parsed = None
|
||||
if parsed is None:
|
||||
arguments: dict[str, Any] = {}
|
||||
elif isinstance(parsed, dict):
|
||||
arguments = cast("dict[str, Any]", parsed)
|
||||
else:
|
||||
raise LLMError(
|
||||
f"model sent non-object tool-call arguments for '{label}': "
|
||||
f"{raw[:200]!r}"
|
||||
)
|
||||
pieces.append(
|
||||
ToolCallPiece(
|
||||
id=slot.id or f"call_{index}", name=slot.name, arguments=arguments
|
||||
)
|
||||
)
|
||||
return pieces
|
||||
|
||||
|
||||
# aipi's local embedding model rejects requests over ~1024 input tokens
|
||||
# ("input is too large to process"). Batch by estimated tokens, with a
|
||||
# safety margin under that cap — code-dense text can tokenize at ~3
|
||||
@@ -232,8 +306,10 @@ class LLMClient:
|
||||
return content.strip()
|
||||
|
||||
async def chat_stream(
|
||||
self, messages: list[dict[str, str]]
|
||||
) -> AsyncIterator[StreamPiece]:
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
) -> AsyncIterator[StreamPiece | ToolCallPiece]:
|
||||
"""Stream assistant pieces from the chat model (PLAN A5/A15, phase 17).
|
||||
|
||||
``stream=True`` against the OpenAI-compatible endpoint, yielding
|
||||
@@ -252,24 +328,61 @@ class LLMClient:
|
||||
32 768) output tokens — the old hard 700-token cap cut long
|
||||
answers off mid-sentence (owner report 2026-08-22).
|
||||
|
||||
Tool calls (phase 37): when *tools* (an OpenAI ``tools`` list) is
|
||||
not None it is passed through as ``tools=…``; when None the key is
|
||||
**not** included, so the request is byte-identical to pre-phase-37
|
||||
and no tool pieces can be produced. A tool-calling model replies
|
||||
with ``delta.tool_calls`` partials — keyed by ``index``, with
|
||||
``id`` and ``function.name`` on the first partial and
|
||||
``function.arguments`` in fragments — which are accumulated into
|
||||
one :class:`ToolCallPiece` per call, yielded in index order at
|
||||
stream end (or immediately once a chunk carries
|
||||
``finish_reason="tool_calls"``). Malformed ``arguments`` JSON
|
||||
raises :class:`LLMError`. Wire convention verified live against
|
||||
aipi's ``turbo`` on 2026-08-26 via
|
||||
``uv run python -m scripts.llm_probe --tools`` (phase 37, task 01:
|
||||
``probe: turbo tool_calls=supported 2026-08-26``).
|
||||
|
||||
Any failure (network, HTTP, malformed stream) surfaces as
|
||||
:class:`LLMError` so the API layer can turn it into an SSE
|
||||
``error`` event instead of a hung request.
|
||||
"""
|
||||
try:
|
||||
kwargs: dict[str, Any] = {
|
||||
# ``{role, content}`` dicts are exactly what the message params
|
||||
# accept; the cast keeps pyright honest about the SDK's union.
|
||||
stream = await self._client.chat.completions.create(
|
||||
model=self.settings.llm_chat_model,
|
||||
messages=cast("list[ChatCompletionMessageParam]", messages),
|
||||
temperature=0.4,
|
||||
max_tokens=self.settings.max_output_tokens,
|
||||
stream=True,
|
||||
)
|
||||
"model": self.settings.llm_chat_model,
|
||||
"messages": cast("list[ChatCompletionMessageParam]", messages),
|
||||
"temperature": 0.4,
|
||||
"max_tokens": self.settings.max_output_tokens,
|
||||
"stream": True,
|
||||
}
|
||||
if tools is not None:
|
||||
kwargs["tools"] = tools
|
||||
try:
|
||||
stream = await self._client.chat.completions.create(**kwargs)
|
||||
calls: dict[int, _ToolCallSlot] = {}
|
||||
emitted = False
|
||||
async for chunk in stream:
|
||||
if not chunk.choices:
|
||||
continue
|
||||
delta = chunk.choices[0].delta
|
||||
choice = chunk.choices[0]
|
||||
delta = choice.delta
|
||||
# Tool-call partials (phase 37) accumulate across chunks,
|
||||
# keyed by index; a missing index (not seen on aipi) falls
|
||||
# back to the next synthetic slot.
|
||||
for tc in getattr(delta, "tool_calls", None) or []:
|
||||
idx = getattr(tc, "index", None)
|
||||
key = idx if isinstance(idx, int) else (max(calls) + 1 if calls else 0)
|
||||
slot = calls.setdefault(key, _ToolCallSlot())
|
||||
tc_id = getattr(tc, "id", None)
|
||||
if tc_id and slot.id is None:
|
||||
slot.id = tc_id
|
||||
fn = getattr(tc, "function", None)
|
||||
if fn is not None:
|
||||
if fn.name:
|
||||
slot.name += fn.name
|
||||
if fn.arguments:
|
||||
slot.arguments += fn.arguments
|
||||
reasoning = getattr(delta, "reasoning_content", None)
|
||||
if not reasoning:
|
||||
# Future-proofing: the same wire convention under a
|
||||
@@ -280,6 +393,17 @@ class LLMClient:
|
||||
content = delta.content
|
||||
if content:
|
||||
yield StreamPiece("content", content)
|
||||
if (
|
||||
calls
|
||||
and not emitted
|
||||
and getattr(choice, "finish_reason", None) == "tool_calls"
|
||||
):
|
||||
for piece in _materialize_tool_calls(calls):
|
||||
yield piece
|
||||
emitted = True
|
||||
if calls and not emitted:
|
||||
for piece in _materialize_tool_calls(calls):
|
||||
yield piece
|
||||
except LLMError:
|
||||
raise
|
||||
except Exception as e: # noqa: BLE001 — wrap transport-level failures
|
||||
|
||||
+32
-5
@@ -23,6 +23,12 @@ the ``<tuning>`` section (order: ``<relevance>`` →
|
||||
``<knowledge_base>`` → ``<tuning>`` → mode body) — the agent knows
|
||||
roughly what the KB contains before retrieval. With an empty row the
|
||||
prompt is byte-identical to the pre-phase text.
|
||||
|
||||
Agent tools (phase 37): the **HIGH** prompt only carries a ``<tools>``
|
||||
section after the ``<documents>`` body — the grounded turn may call the
|
||||
server-side ``list_documents`` / ``read_document`` tools (budgeted, see
|
||||
:mod:`app.rag.agent`). The LOW/deflection prompt never carries it and
|
||||
stays byte-identical to the pre-phase text.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -66,6 +72,25 @@ _KB_INTRO = (
|
||||
"(generated at import time):\n"
|
||||
)
|
||||
|
||||
#: The ``<tools>`` instructions section — **HIGH prompt only** (phase 37,
|
||||
#: task 03): a grounded turn may extend its context through the two
|
||||
#: server-side tools (budgets: ``BOR_AGENT_LIST_CALLS`` /
|
||||
#: ``BOR_AGENT_READ_CALLS``, see :mod:`app.rag.agent`). Appended after
|
||||
#: the mode body (``<documents>``), so the instructions are the last
|
||||
#: thing the model reads. The LOW/deflection prompt never carries it —
|
||||
#: a deflection has no grounded context to extend — and stays
|
||||
#: byte-identical to the pre-phase text. The E2E mock keys off the
|
||||
#: ``<tools>`` marker's *presence*, not this wording.
|
||||
TOOLS_SECTION: str = (
|
||||
"<tools>\n"
|
||||
"If the documents in your context reference other files, or you need "
|
||||
"content that is not included above, call `list_documents` to see what "
|
||||
"is indexed, then `read_document` to pull in exactly one more document. "
|
||||
"Answer as soon as you have what you need — do not read more than one "
|
||||
"extra document.\n"
|
||||
"</tools>"
|
||||
)
|
||||
|
||||
|
||||
def _base(relevance: str) -> str:
|
||||
if relevance not in ("HIGH", "LOW"):
|
||||
@@ -155,11 +180,13 @@ def build_high_prompt(
|
||||
kb_overview: str | None = None,
|
||||
) -> str:
|
||||
"""Grounded turn: locked persona (+ steering, + KB overview) + full
|
||||
texts of the top documents.
|
||||
texts of the top documents + the ``<tools>`` instructions (phase 37).
|
||||
|
||||
Section order (phase 31): ``<relevance>`` → ``<knowledge_base>`` →
|
||||
``<tuning>`` → ``<documents>``; empty steering/overview omit their
|
||||
section, keeping the prompt byte-identical to the pre-phase text.
|
||||
Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>``
|
||||
→ ``<documents>`` → ``<tools>``; empty steering/overview omit their
|
||||
section. ``<tools>`` is always present in the HIGH prompt (the
|
||||
budgets — not the prompt — decide whether the tools are actually
|
||||
offered to the model, see :mod:`app.rag.agent`).
|
||||
"""
|
||||
blocks = [
|
||||
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}">\n'
|
||||
@@ -174,7 +201,7 @@ texts of the top documents.
|
||||
for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or [])):
|
||||
if part:
|
||||
prompt += "\n" + part
|
||||
return prompt + "\n<documents>\n" + body + "\n</documents>"
|
||||
return prompt + "\n<documents>\n" + body + "\n</documents>\n" + TOOLS_SECTION
|
||||
|
||||
|
||||
def build_deflect_prompt(
|
||||
|
||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
@@ -60,6 +61,25 @@ class ChatThinkingEvent(BaseModel):
|
||||
text: str
|
||||
|
||||
|
||||
class ChatToolEvent(BaseModel):
|
||||
"""SSE frame for one agent tool call (phase 37, PLAN §4 extension).
|
||||
|
||||
A15 extension (owner permission 2026-08-26): a grounded turn may call
|
||||
the server-side document tools (``list_documents`` / ``read_document``,
|
||||
see :mod:`app.rag.agent`); each model-requested call streams as
|
||||
``{type: "tool", name: str, argument: str | null}`` ahead of the
|
||||
answer's ``delta`` frames. ``argument`` is the read document's
|
||||
``"source/path"`` for ``read_document`` and null otherwise. The client
|
||||
renders each frame as a "calling tool" line/state (phase 37 task 05);
|
||||
the ``delta`` / ``done`` shapes are unchanged — the read document is
|
||||
reflected in ``done.sources`` instead.
|
||||
"""
|
||||
|
||||
type: Literal["tool"] = "tool"
|
||||
name: str # "list_documents" | "read_document"
|
||||
argument: str | None = None # "source/path" for read_document
|
||||
|
||||
|
||||
class ChatDoneEvent(BaseModel):
|
||||
"""Final SSE event of a chat turn: metadata for the finished answer."""
|
||||
|
||||
|
||||
+109
-7
@@ -40,6 +40,23 @@
|
||||
* "New chat" (#new-chat-btn — bound by the shared header module,
|
||||
* phase 34 task 02) clears the key + the list back to the empty state.
|
||||
*
|
||||
* Agent tool calls (phase 37, PLAN §4 extension): a grounded turn may
|
||||
* call the two server-side document tools (list_documents /
|
||||
* read_document, budgeted server-side). Each call streams a `tool` SSE
|
||||
* frame, and the UI shows the "calling tool" state IN ADDITION to
|
||||
* "thinking": the UI state itself stays "thinking" (button stays
|
||||
* disabled — never stale, PLAN §7.4) while the LABELS change — the
|
||||
* button says "Calling tool…", the #send-status + typing-indicator
|
||||
* labels say what Brain is doing ("Brain of Reese is listing documents"
|
||||
* / "Brain of Reese is reading source/path"), and a visible `.tool-call`
|
||||
* line (own icon + accent color, distinct from the brand-ink Thinking
|
||||
* block) is appended above the answer, one per call, in order.
|
||||
* Append-only like thinking: frames are tolerated in any interleaving
|
||||
* (a frame after the first delta just appends — the agent loop never
|
||||
* emits one, but it must not crash). The turn record persists an
|
||||
* optional `tools: [{name, argument}]` array next to `thinking` and
|
||||
* restore re-renders the lines (phase 14 convention).
|
||||
*
|
||||
* Steering notes (phase 15) let the owner tune how Brain answers: a
|
||||
* "Tune" button under every completed brain bubble (deflected included)
|
||||
* opens an inline form → POST /api/steering → the note is stored in
|
||||
@@ -362,7 +379,12 @@ function ensureThinkingBlock(wrap) {
|
||||
block.innerHTML =
|
||||
`<summary>Thinking</summary><div class="thinking-text"></div>`;
|
||||
const body = wrap.querySelector(".msg-body");
|
||||
body.insertBefore(block, body.querySelector(".bubble"));
|
||||
// Phase 37: the scratchpad stays the TOP row of the wrap — if tool
|
||||
// lines are already there (a `tool` frame preceded the first
|
||||
// `thinking` frame), the block lands above them, not below.
|
||||
const anchor =
|
||||
body.querySelector(".tool-calls") ?? body.querySelector(".bubble");
|
||||
body.insertBefore(block, anchor);
|
||||
}
|
||||
return block;
|
||||
}
|
||||
@@ -372,6 +394,44 @@ function closeThinkingBlock(wrap) {
|
||||
if (block) block.open = false; // idempotent; no-op without a block
|
||||
}
|
||||
|
||||
/* ---------- tool-call lines (phase 37, PLAN §4 extension) ----------
|
||||
* One visible "calling tool" row per `tool` SSE frame, in the same wrap
|
||||
* the Thinking block uses — above the answer, below the Thinking
|
||||
* summary (ensureThinkingBlock keeps the scratchpad on top). The first
|
||||
* frame creates the .tool-calls list; later frames — any tool, any
|
||||
* interleaving with thinking frames, even after the first delta (the
|
||||
* agent loop never emits one, but a late frame must not crash) — just
|
||||
* append another line, in order. The SAME helper re-renders the
|
||||
* persisted lines on restore (phase 14 convention): the path argument
|
||||
* goes through textContent, so nothing HTML-shaped can come from
|
||||
* storage. Lines are not interactive (no focus targets). */
|
||||
function appendToolLine(wrap, name, argument) {
|
||||
const body = wrap?.querySelector?.(".msg-body");
|
||||
if (!body) return;
|
||||
let container = body.querySelector(".tool-calls");
|
||||
if (!container) {
|
||||
container = document.createElement("div");
|
||||
container.className = "tool-calls";
|
||||
container.setAttribute("role", "list");
|
||||
container.setAttribute("aria-label", "Tool calls");
|
||||
// Before the bubble; below an existing Thinking block (both insert
|
||||
// before the bubble, so document order is preserved).
|
||||
body.insertBefore(container, body.querySelector(".bubble"));
|
||||
}
|
||||
const line = document.createElement("span");
|
||||
line.className = "tool-call";
|
||||
line.setAttribute("role", "listitem");
|
||||
if (name === "read_document" && argument) {
|
||||
line.textContent = "📄 Reading ";
|
||||
const code = document.createElement("code");
|
||||
code.textContent = argument; // the path is data, never markup
|
||||
line.appendChild(code);
|
||||
} else {
|
||||
line.textContent = "🔎 Listing documents";
|
||||
}
|
||||
container.appendChild(line);
|
||||
}
|
||||
|
||||
/* ---------- suggestions (shared chip component, phase 05) ----------
|
||||
*
|
||||
* One component, two homes: the onboarding row in the empty state and the
|
||||
@@ -616,7 +676,7 @@ function appendMaybeTry(wrap, suggestions) {
|
||||
*
|
||||
* bor.chat.v1 → { v: 1, messages: [{ who: "user"|"brain", text,
|
||||
* sources?, deflected?, suggestions?,
|
||||
* thinking? }] }
|
||||
* thinking?, tools? }] }
|
||||
*
|
||||
* Only RAW TEXT is stored — restore re-renders it through the escape-first
|
||||
* markdown renderer, so no HTML is ever persisted. Save points: the user
|
||||
@@ -704,6 +764,16 @@ function renderStoredMessage(m) {
|
||||
block.open = false;
|
||||
block.querySelector(".thinking-text").innerHTML = renderMarkdown(m.thinking);
|
||||
}
|
||||
if (Array.isArray(m.tools)) {
|
||||
// Phase 37: restore the tool lines in saved order through the SAME
|
||||
// append helper as the live frames (no HTML from storage, ever).
|
||||
for (const t of m.tools) {
|
||||
if (!t || typeof t.name !== "string") continue;
|
||||
const arg =
|
||||
typeof t.argument === "string" && t.argument ? t.argument : null;
|
||||
appendToolLine(wrap, t.name, arg);
|
||||
}
|
||||
}
|
||||
if (m.deflected) {
|
||||
wrap.classList.add("is-deflected");
|
||||
appendMaybeTry(wrap, m.suggestions);
|
||||
@@ -721,10 +791,10 @@ function restoreConversation() {
|
||||
}
|
||||
|
||||
/* Brain message save point (on `done`): raw accumulated text + metadata.
|
||||
Phase 17: meta.thinking is optional — `undefined` drops the key from
|
||||
the JSON, so turns without thinking persist exactly as before. An empty
|
||||
answer keeps the fallback/"…" text that was actually rendered — what
|
||||
the user saw is what is stored. */
|
||||
Phase 17: meta.thinking and phase 37: meta.tools are optional —
|
||||
`undefined` drops the key from the JSON, so turns without them persist
|
||||
exactly as before. An empty answer keeps the fallback/"…" text that
|
||||
was actually rendered — what the user saw is what is stored. */
|
||||
function rememberBrainTurn(rawText, meta) {
|
||||
conversation.push({ who: "brain", text: rawText || "…", ...meta });
|
||||
saveConversation();
|
||||
@@ -828,6 +898,8 @@ async function handleSend(e) {
|
||||
persistedOnLeave = false;
|
||||
let sawThinking = false; // did any `thinking` frame arrive this turn?
|
||||
let sawDone = false; // did the stream end with a `done` event?
|
||||
let toolAcc = []; // phase 37: {name, argument} per `tool` frame —
|
||||
// persisted with the turn (optional `tools` key)
|
||||
|
||||
try {
|
||||
// thinking = pre-token: dots + busy button. The guard is armed so a
|
||||
@@ -874,6 +946,34 @@ async function handleSend(e) {
|
||||
textEl.scrollTop = textEl.scrollHeight; // pin the stream to the bottom
|
||||
scrollReveal(wrap); // page follows only while pinned (phase 18)
|
||||
}
|
||||
} else if (ev.type === "tool") {
|
||||
// Phase 37 (PLAN §4 extension): an agent tool call. The UI
|
||||
// state stays "thinking" — the button remains disabled (never
|
||||
// stale, PLAN §7.4); what changes are the LABELS: the button
|
||||
// carries the "calling tool" text, #send-status + the typing
|
||||
// indicator (if still visible) say what Brain is doing, and a
|
||||
// .tool-call line lands above the answer (append-only, in
|
||||
// order). The elapsed-seconds hint (thinkingClock) keeps
|
||||
// running through tool frames — no clock changes here.
|
||||
const name = typeof ev.name === "string" ? ev.name : "";
|
||||
const argument =
|
||||
typeof ev.argument === "string" && ev.argument ? ev.argument : null;
|
||||
toolAcc.push({ name, argument });
|
||||
clearTurnTimeout(); // the stream is alive — a frame arrived
|
||||
if (!wrap) wrap = addMessage("brain", "");
|
||||
const toolStatus =
|
||||
name === "read_document" && argument
|
||||
? `Brain of Reese is reading ${argument}`
|
||||
: "Brain of Reese is listing documents";
|
||||
if (uiState === UI_STATE.thinking) {
|
||||
sendLabel.textContent = "Calling tool…";
|
||||
sendStatus.textContent = toolStatus;
|
||||
document
|
||||
.querySelector("#typing-indicator .bubble")
|
||||
?.setAttribute("aria-label", toolStatus);
|
||||
}
|
||||
appendToolLine(wrap, name, argument);
|
||||
scrollReveal(wrap); // page follows only while pinned (phase 18)
|
||||
} else if (ev.type === "delta") {
|
||||
acc += ev.text || "";
|
||||
if (uiState === UI_STATE.thinking) setUiState(UI_STATE.streaming);
|
||||
@@ -903,9 +1003,11 @@ async function handleSend(e) {
|
||||
}
|
||||
// Persistence save point 2: the answer lands only when the turn is
|
||||
// complete (raw text + the done metadata; phase 17: + optional
|
||||
// thinking — `undefined` drops the key from the JSON).
|
||||
// thinking, phase 37: + optional tools — `undefined` drops the
|
||||
// key from the JSON).
|
||||
rememberBrainTurn(finalText || acc, {
|
||||
thinking: thinkingAcc || undefined,
|
||||
tools: toolAcc.length ? toolAcc : undefined,
|
||||
deflected: !!ev.deflected,
|
||||
sources: ev.sources,
|
||||
suggestions: ev.suggestions,
|
||||
|
||||
@@ -569,6 +569,45 @@ details.thinking .thinking-text {
|
||||
details.thinking .thinking-text p,
|
||||
details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
|
||||
|
||||
/* Agent tool-call lines (phase 37): one visible "calling tool" row per
|
||||
`tool` SSE frame — in the same wrap as the Thinking block, above the
|
||||
answer, below the Thinking summary. Deliberately distinct from the
|
||||
scratchpad: accent palette (--accent-ink) vs the brand-ink summary,
|
||||
own icon, own accent left border. Contrast: --accent-ink on the row's
|
||||
--surface ≈10.4:1 (11.6:1 on the page bg), and --ink on --brand-soft
|
||||
in the path `code` ≈11.5:1 — all comfortably AA in the (single dark)
|
||||
theme. Inline rows only: appending lines never shifts the 46rem chat
|
||||
column (no new container), and the rows are not interactive — no
|
||||
focus targets. */
|
||||
.tool-calls {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 0.25rem;
|
||||
}
|
||||
.tool-call {
|
||||
display: flex;
|
||||
align-items: baseline;
|
||||
gap: 0.45rem;
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--line);
|
||||
border-left: 3px solid var(--accent-line);
|
||||
border-radius: var(--radius-sm);
|
||||
padding: 0.3rem 0.75rem;
|
||||
color: var(--accent-ink);
|
||||
font-size: 0.8rem;
|
||||
line-height: 1.4;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
.tool-call code {
|
||||
font-family: var(--mono);
|
||||
font-size: 0.95em;
|
||||
background: var(--brand-soft);
|
||||
color: var(--ink);
|
||||
padding: 0.05em 0.35em;
|
||||
border-radius: 5px;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.msg-meta {
|
||||
font-size: 0.75rem;
|
||||
color: var(--ink-soft);
|
||||
|
||||
+268
-5
@@ -2,22 +2,281 @@
|
||||
|
||||
Lists available models and verifies the embedding dimension of the
|
||||
configured ``embed`` model against ``BOR_EMBEDDING_DIM`` (default 768).
|
||||
Run this before the first import if the LLM backend ever changes:
|
||||
With ``--tools`` it additionally probes the configured chat model's
|
||||
OpenAI-style tool-calling support (phase 37, task 01): a trivial
|
||||
no-parameter ``get_time`` function is offered in a non-streaming and a
|
||||
streaming ``chat/completions`` request, and a supported / not-supported
|
||||
verdict is printed for each — the agent loop (``app/rag/agent.py``) is
|
||||
built against that verdict. Run this before the first import if the LLM
|
||||
backend ever changes:
|
||||
|
||||
uv run python -m scripts.llm_probe
|
||||
uv run python -m scripts.llm_probe --tools
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from collections.abc import Iterable
|
||||
from datetime import date
|
||||
|
||||
import httpx
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
#: Trivial probe function (phase 37, task 01): no parameters, so a
|
||||
#: compliant tool call carries arguments of exactly ``{}``.
|
||||
_PROBE_TOOL: dict = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_time",
|
||||
"description": "Get the current time.",
|
||||
"parameters": {"type": "object", "properties": {}},
|
||||
},
|
||||
}
|
||||
|
||||
#: A question for which calling ``get_time`` is the natural move.
|
||||
_PROBE_MESSAGES: list[dict] = [{"role": "user", "content": "What time is it right now?"}]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
def parse_tool_response_nonstreaming(payload: dict | None) -> dict:
|
||||
"""Extract tool-call facts from one non-streaming chat.completions body.
|
||||
|
||||
Returns ``{"finish_reason": str | None, "calls": [(name, arguments)]}``
|
||||
— empty/None when the reply carries no tool calls (a plain content
|
||||
answer) or is malformed (including a non-dict body).
|
||||
"""
|
||||
choices = (payload or {}).get("choices") or []
|
||||
if not isinstance(choices, list) or not choices:
|
||||
return {"finish_reason": None, "calls": []}
|
||||
first = choices[0]
|
||||
message = first.get("message") or {}
|
||||
calls: list[tuple[str, str]] = []
|
||||
for tc in message.get("tool_calls") or []:
|
||||
fn = tc.get("function") or {}
|
||||
calls.append((str(fn.get("name") or ""), str(fn.get("arguments") or "")))
|
||||
return {"finish_reason": first.get("finish_reason"), "calls": calls}
|
||||
|
||||
|
||||
def parse_tool_response_streaming(lines: Iterable[str]) -> dict:
|
||||
"""Accumulate ``delta.tool_calls`` fragments across SSE ``data:`` lines.
|
||||
|
||||
Wire convention (OpenAI): the first fragment of a call carries
|
||||
``index`` + ``id`` + ``function.name`` and (possibly partial)
|
||||
``function.arguments``; later fragments carry ``index`` + further
|
||||
``arguments`` pieces; the final chunk carries ``finish_reason``.
|
||||
Parsing stops at ``data: [DONE]``; malformed ``data:`` lines are
|
||||
skipped (aipi sometimes interleaves keep-alive noise). Returns::
|
||||
|
||||
{
|
||||
"finish_reason": str | None,
|
||||
"calls": [(name, arguments)], # accumulated per index
|
||||
"delta_chunks": int, # chunks carrying tool_calls
|
||||
"indexed": bool, # every such chunk had int "index"
|
||||
"had_id": bool, # some chunk carried the call "id"
|
||||
"arguments_in_deltas": bool, # some fragment carried arguments
|
||||
}
|
||||
"""
|
||||
finish_reason: str | None = None
|
||||
by_index: dict[int, dict[str, str]] = {}
|
||||
delta_chunks = 0
|
||||
indexed = True
|
||||
had_id = False
|
||||
arguments_in_deltas = False
|
||||
for raw in lines:
|
||||
line = raw.strip()
|
||||
if not line.startswith("data:"):
|
||||
continue
|
||||
data = line[5:].strip()
|
||||
if data == "[DONE]":
|
||||
break
|
||||
try:
|
||||
payload = json.loads(data)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
choices = payload.get("choices") or []
|
||||
if not isinstance(choices, list):
|
||||
continue
|
||||
for choice in choices:
|
||||
fr = choice.get("finish_reason")
|
||||
if fr:
|
||||
finish_reason = fr
|
||||
delta = choice.get("delta") or {}
|
||||
for tc in delta.get("tool_calls") or []:
|
||||
idx = tc.get("index")
|
||||
if isinstance(idx, int):
|
||||
key = idx
|
||||
else:
|
||||
indexed = False
|
||||
key = 0 if not by_index else max(by_index) + 1
|
||||
slot = by_index.setdefault(key, {"name": "", "arguments": ""})
|
||||
delta_chunks += 1
|
||||
if tc.get("id"):
|
||||
had_id = True
|
||||
fn = tc.get("function") or {}
|
||||
if fn.get("name"):
|
||||
slot["name"] += str(fn["name"])
|
||||
if fn.get("arguments"):
|
||||
arguments_in_deltas = True
|
||||
slot["arguments"] += str(fn["arguments"])
|
||||
calls = [(slot["name"], slot["arguments"]) for _, slot in sorted(by_index.items())]
|
||||
return {
|
||||
"finish_reason": finish_reason,
|
||||
"calls": calls,
|
||||
"delta_chunks": delta_chunks,
|
||||
"indexed": indexed,
|
||||
"had_id": had_id,
|
||||
"arguments_in_deltas": arguments_in_deltas,
|
||||
}
|
||||
|
||||
|
||||
def _called(result: dict, expected: str) -> bool:
|
||||
"""True when the request finished with tool_calls invoking *expected*."""
|
||||
return (
|
||||
result["finish_reason"] == "tool_calls"
|
||||
and any(name == expected for name, _ in result["calls"])
|
||||
)
|
||||
|
||||
|
||||
def classify_tool_calling(nonstream: dict, stream: dict, expected: str = "get_time") -> str:
|
||||
"""Phase-37 verdict: ``"supported"`` or ``"not-supported"``.
|
||||
|
||||
Supported requires *both* requests to finish with
|
||||
``finish_reason="tool_calls"`` calling *expected*, and the streaming
|
||||
request to deliver the calls as indexed ``delta.tool_calls`` chunks
|
||||
with a call ``id`` (the OpenAI wire convention). Anything less —
|
||||
including an intermittent split outcome — is ``"not-supported"``
|
||||
(fail-loud house style); the phase then uses the documented
|
||||
prompt-based structured-call fallback.
|
||||
"""
|
||||
if (
|
||||
_called(nonstream, expected)
|
||||
and _called(stream, expected)
|
||||
and stream["delta_chunks"] > 0
|
||||
and stream["indexed"]
|
||||
and stream["had_id"]
|
||||
):
|
||||
return "supported"
|
||||
return "not-supported"
|
||||
|
||||
|
||||
def _no_stream_result() -> dict:
|
||||
return {
|
||||
"finish_reason": None,
|
||||
"calls": [],
|
||||
"delta_chunks": 0,
|
||||
"indexed": False,
|
||||
"had_id": False,
|
||||
"arguments_in_deltas": False,
|
||||
}
|
||||
|
||||
|
||||
def probe_tools(client: httpx.Client, chat_model: str) -> int:
|
||||
"""Run the ``--tools`` probe (phase 37, task 01) and print the verdicts.
|
||||
|
||||
Returns 0 when both requests were made and classified (either verdict
|
||||
is a valid, recorded outcome) and 1 when the endpoint is unreachable
|
||||
(the probe is inconclusive, not a "not supported" signal).
|
||||
"""
|
||||
base_payload: dict = {
|
||||
"model": chat_model,
|
||||
"messages": _PROBE_MESSAGES,
|
||||
"tools": [_PROBE_TOOL],
|
||||
}
|
||||
|
||||
# (a) non-streaming request
|
||||
ns_raw: dict | None = None
|
||||
try:
|
||||
resp = client.post("/chat/completions", json={**base_payload, "stream": False})
|
||||
resp.raise_for_status()
|
||||
ns_raw = resp.json()
|
||||
nonstream = parse_tool_response_nonstreaming(ns_raw)
|
||||
except httpx.HTTPStatusError as e:
|
||||
print(f"[probe] tools(non-stream) HTTP {e.response.status_code}: {e.response.text[:200]}")
|
||||
nonstream = {"finish_reason": None, "calls": []}
|
||||
except httpx.TransportError as e:
|
||||
print(f"[probe] tools(non-stream) transport error: {e}")
|
||||
return 1
|
||||
ns_ok = _called(nonstream, "get_time")
|
||||
print(f"[probe] tools(non-stream) verdict : {'supported' if ns_ok else 'not supported'}")
|
||||
print(
|
||||
f"[probe] tools(non-stream) finish_reason={nonstream['finish_reason']!r} "
|
||||
f"calls={nonstream['calls']!r}"
|
||||
)
|
||||
if not ns_ok and ns_raw:
|
||||
content = ((ns_raw.get("choices") or [{}])[0].get("message") or {}).get("content") or ""
|
||||
if content:
|
||||
print(f"[probe] tools(non-stream) answered in content instead: {content[:160]!r}")
|
||||
|
||||
# (b) streaming request
|
||||
lines: list[str] = []
|
||||
try:
|
||||
with client.stream(
|
||||
"POST", "/chat/completions", json={**base_payload, "stream": True}
|
||||
) as resp:
|
||||
resp.raise_for_status()
|
||||
for line in resp.iter_lines():
|
||||
lines.append(line)
|
||||
if line.startswith("data:") and line[5:].strip() == "[DONE]":
|
||||
break
|
||||
stream = parse_tool_response_streaming(lines)
|
||||
except httpx.HTTPStatusError as e:
|
||||
print(f"[probe] tools(stream) HTTP {e.response.status_code}: {e.response.text[:200]}")
|
||||
stream = _no_stream_result()
|
||||
except httpx.TransportError as e:
|
||||
print(f"[probe] tools(stream) transport error: {e}")
|
||||
return 1
|
||||
st_ok = (
|
||||
_called(stream, "get_time")
|
||||
and stream["delta_chunks"] > 0
|
||||
and stream["indexed"]
|
||||
and stream["had_id"]
|
||||
)
|
||||
print(f"[probe] tools(stream) verdict : {'supported' if st_ok else 'not supported'}")
|
||||
print(
|
||||
f"[probe] tools(stream) finish_reason={stream['finish_reason']!r} "
|
||||
f"calls={stream['calls']!r}"
|
||||
)
|
||||
print(
|
||||
f"[probe] tools(stream) delta.tool_calls: chunks={stream['delta_chunks']} "
|
||||
f"indexed={stream['indexed']} had_id={stream['had_id']} "
|
||||
f"arguments_in_deltas={stream['arguments_in_deltas']}"
|
||||
)
|
||||
|
||||
verdict = classify_tool_calling(nonstream, stream)
|
||||
today = date.today().isoformat()
|
||||
detail = (
|
||||
"non-streaming + streaming tool_calls follow the OpenAI wire convention"
|
||||
if verdict == "supported"
|
||||
else "phase falls back to the documented prompt-based structured-call path"
|
||||
)
|
||||
print(f"[probe] TOOLS VERDICT ({today}) : {verdict} — {detail}")
|
||||
print(f"[probe] summary : probe: {chat_model} tool_calls={verdict} {today}")
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
# CLI-only: pick up .env without side effects on import (the parse/
|
||||
# classify functions above are imported from unit tests, where the
|
||||
# ambient environment must stay pristine).
|
||||
load_dotenv()
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Probe the aipi endpoint: models, embedding dimension, and "
|
||||
"(with --tools) the chat model's tool-calling support."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--tools",
|
||||
action="store_true",
|
||||
help=(
|
||||
"also probe the chat model's OpenAI-style tools/tool_calls support "
|
||||
"(non-streaming + streaming) and print a supported / not-supported verdict"
|
||||
),
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
base_url = os.environ.get("BOR_LLM_BASE_URL", "https://aipi.reeseapps.com/v1").rstrip("/")
|
||||
api_key = (
|
||||
os.environ.get("BOR_LLM_API_KEY")
|
||||
@@ -29,7 +288,8 @@ def main() -> int:
|
||||
expected_dim = int(os.environ.get("BOR_EMBEDDING_DIM", "768"))
|
||||
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
with httpx.Client(base_url=base_url, headers=headers, timeout=30.0) as client:
|
||||
tools_rc = 0
|
||||
with httpx.Client(base_url=base_url, headers=headers, timeout=60.0) as client:
|
||||
r = client.get("/models")
|
||||
r.raise_for_status()
|
||||
models = [m["id"] for m in r.json()["data"]]
|
||||
@@ -49,6 +309,9 @@ def main() -> int:
|
||||
dims = sorted({len(d["embedding"]) for d in r.json()["data"]})
|
||||
print(f"[probe] dims({embed_model}): {dims}")
|
||||
|
||||
if args.tools:
|
||||
tools_rc = probe_tools(client, chat_model)
|
||||
|
||||
if dims != [expected_dim]:
|
||||
print(
|
||||
f"[probe] MISMATCH: expected {expected_dim}, got {dims}. "
|
||||
@@ -56,7 +319,7 @@ def main() -> int:
|
||||
)
|
||||
return 1
|
||||
print("[probe] OK — models present, embedding dimension matches configuration.")
|
||||
return 0
|
||||
return tools_rc
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+183
-9
@@ -38,9 +38,33 @@ Implements just enough of the aipi surface:
|
||||
the same echo convention for the overview's prompt injection.
|
||||
- user message containing ``show the end of your notes`` (phase 24,
|
||||
whole-document context) -> the answer quotes the **last 160 chars of
|
||||
the document context** — a tail echo, byte-stable across runs, so a
|
||||
sentinel placed at the *end* of a document appears in the rendered
|
||||
answer iff the whole document was in the prompt.
|
||||
the ``<documents>`` block** — a tail echo, byte-stable across runs, so
|
||||
a sentinel placed at the *end* of a document appears in the rendered
|
||||
answer iff the whole document was in the prompt. (Phase 37: the HIGH
|
||||
prompt now ends with a ``<tools>`` section after ``</documents>``, so
|
||||
the echo targets the block itself; its tail still includes the
|
||||
closing tag — same sentinel semantics.)
|
||||
- user message containing ``use your tools`` (phase 37, agent document
|
||||
tools) **and** the system prompt carries the ``<tools>`` section ->
|
||||
the deterministic tool-calling flow, discriminated statelessly from
|
||||
the messages + the ``tools`` parameter:
|
||||
* request 1 (``tools`` offered, no tool results yet): stream ONLY
|
||||
``tool_calls`` deltas — ``list_documents`` (synthetic id
|
||||
``call_0``, no arguments), ``finish_reason: "tool_calls"``, no
|
||||
content;
|
||||
* request 2 (a ``tool``-role catalog result in the messages):
|
||||
parse the FIRST catalog line (``source/path — title`` → split on
|
||||
``" — "`` → ``rsplit("/", 1)``) and stream a ``tool_calls`` delta
|
||||
calling ``read_document`` on it (id ``call_1``);
|
||||
* request 3 (the read result in the messages, no ``tools``
|
||||
parameter): a content answer, deterministic: ``Read
|
||||
<source/path>. <first 80 chars of the read document's content>``
|
||||
— so a suite can assert the read document reached the model and
|
||||
landed in the answer.
|
||||
All other requests (including the marker without a ``<tools>``
|
||||
section, or with the tool conversation not yet started and no tools
|
||||
offered — e.g. budgets 0/0) behave exactly as today. ``E2E_REAL_LLM=1``
|
||||
ignores the mock entirely (the real model does what it does).
|
||||
|
||||
``max_tokens`` is honored deterministically (token ≈ whitespace word),
|
||||
like a real endpoint: an answer longer than the cap is truncated. This
|
||||
@@ -126,6 +150,67 @@ PRE_CONTENT_PAUSE_S = 4.0
|
||||
#: other suite is unaffected.
|
||||
END_OF_NOTES_TRIGGER = "show the end of your notes"
|
||||
|
||||
#: The ``<documents>`` block of the system prompt (phase 37: the HIGH
|
||||
#: prompt ends with the ``<tools>`` section after ``</documents>``, so the
|
||||
#: phase-24 tail echo targets the block, not the raw message tail).
|
||||
_DOCUMENTS_BLOCK_RE = re.compile(r"<documents>.*?</documents>", re.S)
|
||||
|
||||
#: Phase 37 (agent-document-tools story): a user message containing this
|
||||
#: substring (case-insensitive) — combined with the ``<tools>`` section
|
||||
#: in the system prompt — drives the deterministic tool flow documented
|
||||
#: in the module docstring (list_documents → read_document on the first
|
||||
#: catalog line → the quoted answer). Existing E2E questions do not
|
||||
#: contain the phrase, so every other suite is unaffected.
|
||||
TOOLS_TRIGGER = "use your tools"
|
||||
|
||||
|
||||
#: The agent's ``read_document`` tool-result prefix (app.rag.agent
|
||||
#: ``_execute_tool``): ``"Document <source/path>:\n<content>"``.
|
||||
_READ_RESULT_PREFIX = "Document "
|
||||
|
||||
|
||||
def _tool_flow(body: dict[str, Any]) -> tuple[str, str, str] | None:
|
||||
"""Classify a marker request into one step of the tool flow (phase 37).
|
||||
|
||||
Returns one of:
|
||||
|
||||
* ``("list", "", "")`` — ``tools`` are offered and no tool results
|
||||
are in the messages yet: the model lists the catalog.
|
||||
* ``("read", source, path)`` — a ``tool``-role catalog result is in
|
||||
the messages: the model reads its FIRST ``source/path — title``
|
||||
line (split on ``" — "``, then ``rsplit("/", 1)``).
|
||||
* ``("answer", "source/path", content)`` — a ``tool``-role read
|
||||
result (``"Document <source/path>:\n<content>"``) is in the
|
||||
messages: the model answers, quoting the read document.
|
||||
* ``None`` — not the marker flow: the request behaves exactly as
|
||||
today (marker absent, no ``<tools>`` section, or a no-tools first
|
||||
request — the budgets-0/0 path).
|
||||
"""
|
||||
if TOOLS_TRIGGER not in _user(body).lower():
|
||||
return None
|
||||
if "<tools>" not in _system(body):
|
||||
return None
|
||||
tool_msgs = [m for m in _messages(body) if m.get("role") == "tool"]
|
||||
for m in tool_msgs: # a read result means the forced-answer request
|
||||
content = str(m.get("content") or "")
|
||||
if content.startswith(_READ_RESULT_PREFIX):
|
||||
# The header is "Document <source/path>:" — drop the prefix
|
||||
# AND the trailing colon so the answer quotes a clean path.
|
||||
header, _, doc_content = content.partition("\n")
|
||||
sp = header[len(_READ_RESULT_PREFIX):].strip().removesuffix(":")
|
||||
return ("answer", sp, doc_content)
|
||||
if not body.get("tools"):
|
||||
return None
|
||||
for m in tool_msgs: # a catalog result means the read request
|
||||
content = str(m.get("content") or "")
|
||||
for line in content.splitlines():
|
||||
head = line.split(" — ", 1)[0].strip()
|
||||
if "/" in head:
|
||||
source, _, path = head.rpartition("/")
|
||||
if source and path:
|
||||
return ("read", source, path)
|
||||
return ("list", "", "")
|
||||
|
||||
|
||||
def long_answer() -> str:
|
||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||
@@ -222,12 +307,17 @@ def compose_answer(body: dict[str, Any]) -> str:
|
||||
)
|
||||
elif END_OF_NOTES_TRIGGER in user.lower():
|
||||
# Whole-document-context story (phase 24): echo the tail of the
|
||||
# context. Byte-stable across runs — a sentinel on the document's
|
||||
# last line appears in the answer iff the whole document was in
|
||||
# the prompt. (The tail includes the closing </documents> —
|
||||
# harmless for the E2E sentinel assertions.)
|
||||
# document context. Byte-stable across runs — a sentinel on the
|
||||
# document's last line appears in the answer iff the whole
|
||||
# document was in the prompt. (The tail includes the closing
|
||||
# </documents> — harmless for the E2E sentinel assertions.)
|
||||
# Phase 37: the HIGH prompt now ends with the <tools> section
|
||||
# after </documents>, so the echo targets the <documents> block
|
||||
# itself — the sentinel semantics are unchanged.
|
||||
block = _DOCUMENTS_BLOCK_RE.search(_system(body))
|
||||
tail_source = block.group(0) if block else _context(body)
|
||||
answer = (
|
||||
f"…and the very end of my notes reads: “{_context(body)[-160:]}” "
|
||||
f"…and the very end of my notes reads: “{tail_source[-160:]}” "
|
||||
"(Deterministic mock answer for E2E.)"
|
||||
)
|
||||
else:
|
||||
@@ -436,11 +526,95 @@ def _apply_max_tokens(answer: str, max_tokens: Any) -> str:
|
||||
return " ".join(words[:max_tokens])
|
||||
|
||||
|
||||
def _tool_call_stream(name: str, arguments: dict[str, Any], call_id: str) -> Any:
|
||||
"""SSE frames for one tool-call-only chat completion (phase 37).
|
||||
|
||||
The OpenAI wire convention the app accumulates (``app/rag/llm.py``):
|
||||
the first partial of index 0 carries ``id`` + ``type`` +
|
||||
``function.name`` plus the first ``function.arguments`` fragment;
|
||||
the remaining fragments (deterministic 16-char split — so the
|
||||
multi-fragment accumulation path is exercised) arrive on later
|
||||
chunks; the final chunk carries ``finish_reason: "tool_calls"``.
|
||||
No ``content`` / ``reasoning_content`` frames — the turn asked for a
|
||||
tool instead of answering.
|
||||
|
||||
Pacing: 0.1 s per frame — deliberately SLOWER than the content
|
||||
stream's 0.02 s, so the UI's transient "calling tool" state (held
|
||||
from the first ``tool`` frame until the first answer ``delta``) is a
|
||||
comfortable observation window for the story E2E (~1 s across the
|
||||
two tool requests).
|
||||
"""
|
||||
model = "turbo"
|
||||
chunk_id = f"chatcmpl-{uuid.uuid4()}"
|
||||
raw_args = json_dumps(arguments) if arguments else "{}"
|
||||
frags = [raw_args[i : i + 16] for i in range(0, len(raw_args), 16)] or ["{}"]
|
||||
for i, frag in enumerate(frags):
|
||||
tc: dict[str, Any] = {"index": 0, "function": {"arguments": frag}}
|
||||
delta: dict[str, Any] = {"tool_calls": [tc]}
|
||||
if i == 0:
|
||||
tc = {
|
||||
"index": 0,
|
||||
"id": call_id,
|
||||
"type": "function",
|
||||
"function": {"name": name, "arguments": frag},
|
||||
}
|
||||
delta = {"role": "assistant", "tool_calls": [tc]}
|
||||
payload = {
|
||||
"id": chunk_id,
|
||||
"object": "chat.completion.chunk",
|
||||
"created": int(time.time()),
|
||||
"model": model,
|
||||
"choices": [{"index": 0, "delta": delta, "finish_reason": None}],
|
||||
}
|
||||
yield f"data: {json_dumps(payload)}\n\n"
|
||||
time.sleep(0.1)
|
||||
yield (
|
||||
"data: "
|
||||
+ json_dumps(
|
||||
{
|
||||
"id": chunk_id,
|
||||
"object": "chat.completion.chunk",
|
||||
"created": int(time.time()),
|
||||
"model": model,
|
||||
"choices": [{"index": 0, "delta": {}, "finish_reason": "tool_calls"}],
|
||||
}
|
||||
)
|
||||
+ "\n\n"
|
||||
)
|
||||
yield "data: [DONE]\n\n"
|
||||
|
||||
|
||||
@app.post("/v1/chat/completions")
|
||||
def chat_completions(body: dict[str, Any]) -> Any:
|
||||
user_lower = _user(body).lower()
|
||||
# Phase 37 (agent document tools): the deterministic marker flow.
|
||||
# The app's chat path is the only streaming consumer of this mock, so
|
||||
# the flow handles streaming requests; a non-streaming marker request
|
||||
# (never issued by the app) falls through to the regular answer.
|
||||
if body.get("stream"):
|
||||
flow = _tool_flow(body)
|
||||
if flow is not None:
|
||||
if flow[0] == "list":
|
||||
stream = _tool_call_stream("list_documents", {}, "call_0")
|
||||
elif flow[0] == "read":
|
||||
stream = _tool_call_stream(
|
||||
"read_document",
|
||||
{"source": flow[1], "path": flow[2]},
|
||||
"call_1",
|
||||
)
|
||||
else: # "answer" — quote the read document (first 80 chars)
|
||||
answer = _apply_max_tokens(
|
||||
f"Read {flow[1]}. {flow[2][:80]}", body.get("max_tokens")
|
||||
)
|
||||
stream = _sse_stream(answer, 0.0)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
|
||||
answer = _apply_max_tokens(compose_answer(body), body.get("max_tokens"))
|
||||
delay = 3.0 if "pretend to think slowly" in _user(body) else 0.0
|
||||
user_lower = _user(body).lower()
|
||||
thinking = compose_thinking(body) if THINKING_TRIGGER in user_lower else ""
|
||||
pre_content = (
|
||||
PRE_CONTENT_PAUSE_S if SLOW_PRETOKEN_TRIGGER in user_lower else 0.0
|
||||
|
||||
@@ -0,0 +1,490 @@
|
||||
"""Phase 37 E2E (Playwright, mock-only): agent document tools (list + read).
|
||||
|
||||
Story: ``.agent/user_stories/agent-document-tools.md``
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real ``turbo``
|
||||
does whatever it does with the tools, while this story's gate is the
|
||||
deterministic marker flow in ``tests/e2e/mock_llm.py`` (user message
|
||||
contains ``use your tools`` **and** the system prompt carries the
|
||||
``<tools>`` section of the HIGH prompt):
|
||||
|
||||
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
|
||||
``tool_calls`` deltas calling ``list_documents`` (id ``call_0``, no
|
||||
arguments, ``finish_reason: "tool_calls"``);
|
||||
2. request 2 (a ``tool``-role catalog result in the messages) → streams a
|
||||
``tool_calls`` delta calling ``read_document`` on the FIRST catalog
|
||||
line (id ``call_1``);
|
||||
3. request 3 (no ``tools`` parameter, the read result in the messages) →
|
||||
the content answer ``Read <source/path>. <first 80 chars of the read
|
||||
document's content>`` — so the suite can assert the read document
|
||||
reached the model and landed in the answer.
|
||||
|
||||
KB fixture — reproduces the TODO failure (``aws-route53.md`` references
|
||||
``example-record-file.json`` "for the exact JSON shape of
|
||||
reseelink.json" but does not include it):
|
||||
|
||||
* ``Homelab/aws-route53.md`` — seeded with one chunk whose embedding is
|
||||
the mock's own bag-of-words vector (genuine token overlap: the marker
|
||||
question cosines ≈0.69 against it, well past the E2E 0.30 threshold,
|
||||
and it FTS-matches too) → the only RETRIEVABLE document, i.e. the
|
||||
grounded context;
|
||||
* ``Deployments/example-record-file.json`` — the JSON shape, indexed
|
||||
(a ``documents`` row: it is in the agent's catalog, readable, and a
|
||||
source-chip target) but seeded WITHOUT chunks. In a real
|
||||
hundreds-of-document KB the file would simply fail to rank into the
|
||||
top-2 context; with a two-document corpus every chunk would rank, so
|
||||
"not in context" is expressed as "no retrieval candidates". Its
|
||||
``(source, path)`` also sorts FIRST in the catalog
|
||||
(``Deployments`` < ``Homelab``) — which is exactly the line the mock
|
||||
parses out of the listing and reads.
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
1. ``test_marker_question_lists_reads_and_quotes`` — the SSE carries
|
||||
``tool`` frames (list, then read, ahead of any delta), the UI shows
|
||||
the "calling tool" label while a tool runs, the bubble shows both
|
||||
tool lines, the final answer quotes the read document, and the
|
||||
source chips include the read document (viewer link).
|
||||
2. ``test_tool_lines_re_render_after_reload`` — the persisted record
|
||||
(phase 14) re-renders the tool lines.
|
||||
3. ``test_plain_grounded_question_has_no_tool_frames`` — no marker → no
|
||||
``tool`` frames, the answer renders exactly as today (regression
|
||||
inside the story file).
|
||||
4. ``test_deflected_question_has_no_tool_frames`` — the tools are
|
||||
grounded-only: a deflected turn runs none.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document, QueryLog
|
||||
from tests.e2e.mock_llm import embed_text
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture documents (deterministic, token-controlled)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
SEED_SOURCE = "Homelab"
|
||||
SEED_PATH = "aws-route53.md"
|
||||
SEED_SP = f"{SEED_SOURCE}/{SEED_PATH}"
|
||||
|
||||
READ_SOURCE = "Deployments"
|
||||
READ_PATH = "example-record-file.json"
|
||||
READ_SP = f"{READ_SOURCE}/{READ_PATH}"
|
||||
|
||||
#: The retrievable document: references the JSON file "for the exact JSON
|
||||
#: shape of reeselink.json" but never includes it (the TODO failure).
|
||||
#: The repeated record-file lines carry the marker question's key tokens
|
||||
#: (aws, route53, hosted, zone, reeselink, json, exact, shape) — verified
|
||||
#: ≈0.69 cosine against the mock's embeddings (E2E threshold 0.30) plus
|
||||
#: FTS hits, so the turn is solidly grounded.
|
||||
ROUTE53_CONTENT = (
|
||||
"# AWS Route 53 Notes\n\n"
|
||||
"## Record file\n\n"
|
||||
+ (
|
||||
"The aws route53 hosted zone for reeselink keeps every record in "
|
||||
"reseelink.json — the exact JSON shape of reeselink.json is "
|
||||
"documented in example-record-file.json.\n"
|
||||
)
|
||||
* 10
|
||||
+ "\n## Sync job\n\n"
|
||||
"A cron job pushes reeselink.json to the aws route53 hosted zone "
|
||||
"every fifteen minutes; the diff is applied through the route53 api.\n"
|
||||
)
|
||||
|
||||
#: The referenced document: the exact JSON shape. Its FIRST line is longer
|
||||
#: than 80 chars, so the mock's first-80-chars quote is newline-free (the
|
||||
#: rendered-text assertions below match it verbatim). Pinned by the assert
|
||||
#: below.
|
||||
RECORD_FILE_CONTENT = (
|
||||
'{ "version": 3, "comment": "ReeseLink hosted zone records — the exact '
|
||||
'JSON shape of reeselink.json",\n'
|
||||
' "hosted_zone_id": "Z0RESEELINK01",\n'
|
||||
' "record_sets": [\n'
|
||||
' { "name": "www.reeselink.example", "type": "A", "ttl": 300,\n'
|
||||
' "resource_records": [ { "value": "10.0.0.20" } ] },\n'
|
||||
' { "name": "api.reeselink.example", "type": "CNAME", "ttl": 300,\n'
|
||||
' "resource_records": [ { "value": "www.reeselink.example" } ] }\n'
|
||||
" ]\n"
|
||||
"}\n"
|
||||
)
|
||||
assert "\n" not in RECORD_FILE_CONTENT[:80] # the quote must stay one line
|
||||
|
||||
MARKER_QUESTION = (
|
||||
"Use your tools: what is the exact JSON shape of reeselink.json "
|
||||
"for my aws route53 hosted zone?"
|
||||
)
|
||||
PLAIN_QUESTION = (
|
||||
"How does my aws route53 sync job push reeselink.json to the "
|
||||
"hosted zone?"
|
||||
)
|
||||
DEFLECT_QUESTION = "tell me about quantum wormhole cooling"
|
||||
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
DEFLECT_PHRASE = r"haven't done anything like that"
|
||||
ANSWER_PREFIX = f"Read {READ_SP}."
|
||||
ANSWER_QUOTE = RECORD_FILE_CONTENT[:80]
|
||||
READ_CHIP_HREF = f"/document.html?source={READ_SOURCE}&path={READ_PATH}&back=%2F"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# DB seeding (TRUNCATE-then-seed, cf. test_whole_document_context.py)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed(db: Session) -> None:
|
||||
"""The two-document pair from the TODO (see the module docstring)."""
|
||||
md = Document(
|
||||
source=SEED_SOURCE,
|
||||
path=SEED_PATH,
|
||||
full_path=f"/tmp/{SEED_PATH}",
|
||||
title="AWS Route 53 Notes",
|
||||
content=ROUTE53_CONTENT,
|
||||
content_hash=hashlib.sha256(ROUTE53_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
db.add(md)
|
||||
db.flush()
|
||||
# One chunk carrying the mock's own embedding → genuine token overlap
|
||||
# between the marker question and this document.
|
||||
db.add(
|
||||
Chunk(
|
||||
document_id=md.id,
|
||||
position=0,
|
||||
content=ROUTE53_CONTENT,
|
||||
embedding=embed_text(ROUTE53_CONTENT),
|
||||
)
|
||||
)
|
||||
# The referenced JSON: indexed, catalogued, readable — but NO chunks,
|
||||
# so retrieval never puts it in context (the failure the tools fix).
|
||||
db.add(
|
||||
Document(
|
||||
source=READ_SOURCE,
|
||||
path=READ_PATH,
|
||||
full_path=f"/tmp/{READ_PATH}",
|
||||
title="Example Record File",
|
||||
content=RECORD_FILE_CONTENT,
|
||||
content_hash=hashlib.sha256(RECORD_FILE_CONTENT.encode()).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
|
||||
"""Truncate the KB (plus the prompt-shaping tables), then re-seed.
|
||||
|
||||
``steering_notes`` / ``kb_overview`` are truncated too, so the HIGH
|
||||
prompt is exactly ``<relevance>`` + ``<documents>`` + ``<tools>``
|
||||
regardless of leftovers from other suites — byte-stable prompts,
|
||||
byte-stable answers.
|
||||
"""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
||||
)
|
||||
db.commit()
|
||||
if seed is not None:
|
||||
seed(db)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _last_query_log() -> QueryLog:
|
||||
with SessionLocal() as db:
|
||||
rows = db.scalars(select(QueryLog)).all()
|
||||
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
|
||||
return rows[0]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Records every value #send-label takes during the turn (a
|
||||
#: MutationObserver on the element), so the transient "Calling tool…"
|
||||
#: state is captured deterministically — no polling race.
|
||||
LABEL_RECORDER = """
|
||||
() => {
|
||||
if (window.__labelsInstalled) return;
|
||||
window.__labelsInstalled = true;
|
||||
window.__labels = [];
|
||||
const el = document.querySelector('#send-label');
|
||||
if (!el) return;
|
||||
const rec = (v) => {
|
||||
const l = window.__labels;
|
||||
if (!l.length || l[l.length - 1] !== v) l.push(v);
|
||||
};
|
||||
rec(el.textContent);
|
||||
new MutationObserver(() => rec(el.textContent)).observe(el, {
|
||||
childList: true,
|
||||
subtree: true,
|
||||
});
|
||||
}
|
||||
"""
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions for
|
||||
#: the ``tool`` frames, independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
"""Install both hooks on the loaded page (post-goto, pre-submit).
|
||||
|
||||
The fetch wrapper only needs to be in place before the turn's
|
||||
``fetch("/api/chat")`` call; the label observer needs the rendered
|
||||
``#send-label``. (``add_init_script`` would not do — it binds to the
|
||||
NEXT navigation, and the story page is navigated exactly once.)
|
||||
"""
|
||||
page.evaluate(SSE_HOOK)
|
||||
page.evaluate(LABEL_RECORDER)
|
||||
|
||||
|
||||
def _frames(page: Page) -> list[dict]:
|
||||
"""The captured SSE frames, once the hook's background read settles.
|
||||
|
||||
The hook reads ``res.clone().text()`` in a background promise that
|
||||
resolves right after the stream closes — poll briefly until the
|
||||
final ``done`` frame lands (fail loud if the hook captured nothing).
|
||||
"""
|
||||
deadline = time.monotonic() + 10.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The marker question: list → read → quoted answer, "calling tool" UI
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_marker_question_lists_reads_and_quotes(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db(_seed)
|
||||
page.goto(app_url)
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, MARKER_QUESTION)
|
||||
# While a tool runs the button carries the "calling tool" label: the
|
||||
# first `tool` frame sets it and it holds until the FIRST answer
|
||||
# delta (the agent loop completes before the answer stream) — so the
|
||||
# poll issued right after the click must catch it inside that window.
|
||||
expect(page.locator("#send-label")).to_have_text("Calling tool…", timeout=20_000)
|
||||
_wait_settled(page)
|
||||
|
||||
# The label transition is also recorded deterministically (no race):
|
||||
# Thinking… → Calling tool… → … → Send.
|
||||
labels = page.evaluate("() => window.__labels")
|
||||
assert "Calling tool…" in labels, labels
|
||||
assert labels.index("Calling tool…") > labels.index("Thinking…")
|
||||
|
||||
# Wire level: exactly two `tool` frames — list then read — and both
|
||||
# ahead of the first `delta` frame.
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "list_documents", "argument": None},
|
||||
{"type": "tool", "name": "read_document", "argument": READ_SP},
|
||||
]
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
(SEED_SOURCE, SEED_PATH),
|
||||
(READ_SOURCE, READ_PATH),
|
||||
]
|
||||
|
||||
# Both tool lines, in order, above the answer.
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
||||
expect(lines.nth(1)).to_contain_text(READ_SP)
|
||||
|
||||
# The final answer quotes the read document (the mock's deterministic
|
||||
# quote: "Read <source/path>. <first 80 chars of its content>").
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(ANSWER_PREFIX)
|
||||
expect(bubble).to_contain_text(ANSWER_QUOTE)
|
||||
|
||||
# Source chips: the retrieval doc AND the read doc (deduped, in
|
||||
# order) — the read chip links to the viewer.
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(2)
|
||||
expect(chips.nth(0)).to_contain_text(SEED_SP)
|
||||
chip_read = page.locator(".msg.brain .source-chip", has_text=READ_PATH)
|
||||
expect(chip_read).to_have_count(1)
|
||||
expect(chip_read.first).to_have_attribute("href", READ_CHIP_HREF)
|
||||
|
||||
# Durable record: grounded, both sources logged (retrieval + read).
|
||||
row = _last_query_log()
|
||||
assert row.question == MARKER_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == f"{SEED_SP}, {READ_SP}"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. Persistence: the tool lines re-render after a reload (phase 14)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_tool_lines_re_render_after_reload(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db(_seed)
|
||||
page.goto(app_url)
|
||||
|
||||
_submit(page, MARKER_QUESTION)
|
||||
_wait_settled(page)
|
||||
expect(page.locator(".msg.brain .tool-call")).to_have_count(2)
|
||||
|
||||
page.reload()
|
||||
expect(page.locator("#empty-state")).to_be_hidden()
|
||||
|
||||
# The persisted record re-renders BOTH tool lines, in saved order,
|
||||
# through the same append helper as the live frames.
|
||||
restored = page.locator(".msg.brain .tool-call")
|
||||
expect(restored).to_have_count(2)
|
||||
expect(restored.nth(0)).to_contain_text("Listing documents")
|
||||
expect(restored.nth(1)).to_contain_text("Reading ")
|
||||
expect(restored.nth(1)).to_contain_text(READ_SP)
|
||||
|
||||
# Answer + the read-document chip are intact (phase-14 restore path).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(ANSWER_PREFIX)
|
||||
expect(bubble).to_contain_text(ANSWER_QUOTE)
|
||||
chip_read = page.locator(".msg.brain .source-chip", has_text=READ_PATH)
|
||||
expect(chip_read).to_have_count(1)
|
||||
expect(chip_read.first).to_have_attribute("href", READ_CHIP_HREF)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 3. Regression: a plain grounded question (no marker) takes the
|
||||
# no-tool path — the answer renders exactly as today
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_plain_grounded_question_has_no_tool_frames(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db(_seed)
|
||||
page.goto(app_url)
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, PLAIN_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# No tool frames on the wire, no tool lines in the UI.
|
||||
assert _tool_frames(_frames(page)) == []
|
||||
expect(page.locator(".tool-call")).to_have_count(0)
|
||||
|
||||
# The standard grounded answer, citing the retrieval doc only — the
|
||||
# referenced JSON stays OUT of the sources (it was never read).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(PLAIN_QUESTION)
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(1)
|
||||
expect(chips.first).to_contain_text(SEED_SP)
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.question == PLAIN_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == SEED_SP
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 4. Grounded-only scope: a deflected turn runs no tools at all
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_deflected_question_has_no_tool_frames(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
_reset_db(_seed)
|
||||
page.goto(app_url)
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, DEFLECT_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# The honesty gate fired — and no tool frames / tool lines came with
|
||||
# it (the LOW prompt never carries the tools).
|
||||
last = page.locator(".msg.brain").last
|
||||
expect(last).to_have_class(re.compile(r"is-deflected"))
|
||||
expect(last.locator(".bubble")).to_contain_text(
|
||||
re.compile(DEFLECT_PHRASE, re.IGNORECASE)
|
||||
)
|
||||
assert _tool_frames(_frames(page)) == []
|
||||
expect(page.locator(".tool-call")).to_have_count(0)
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.question == DEFLECT_QUESTION
|
||||
assert row.deflected is True
|
||||
@@ -0,0 +1,83 @@
|
||||
"""Integration: the agent DB accessors against real Postgres (phase 37).
|
||||
|
||||
``list_catalog`` must order rows by ``(source, path)`` — the same order as
|
||||
``GET /api/docs`` — and ``find_document`` must resolve a hit to the full
|
||||
document row (content included, for the never-truncated read) and return
|
||||
``None`` for unknown ``source``/``path`` pairs.
|
||||
|
||||
Requires: podman compose up -d db
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import Document
|
||||
from app.rag import agent
|
||||
|
||||
|
||||
def _doc(db: Session, source: str, path: str, title: str, content: str) -> Document:
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source=source,
|
||||
path=path,
|
||||
full_path=f"/tmp/{source}/{path}",
|
||||
title=title,
|
||||
content=content,
|
||||
content_hash="0" * 64,
|
||||
)
|
||||
db.add(doc)
|
||||
return doc
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def kb(db) -> Iterator[None]:
|
||||
"""Fresh documents table (chunks first — the FK) for these accessors."""
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
yield
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_list_catalog_orders_by_source_then_path(kb, db) -> None:
|
||||
_doc(db, "Zeta", "b/second.md", "Zeta B", "ZB")
|
||||
_doc(db, "Zeta", "a/first.md", "Zeta A", "ZA")
|
||||
_doc(db, "Alpha", "c/third.md", "Alpha C", "AC")
|
||||
db.commit()
|
||||
|
||||
assert agent.list_catalog(db) == [
|
||||
("Alpha", "c/third.md", "Alpha C"),
|
||||
("Zeta", "a/first.md", "Zeta A"),
|
||||
("Zeta", "b/second.md", "Zeta B"),
|
||||
]
|
||||
|
||||
|
||||
def test_list_catalog_is_empty_without_rows(kb, db) -> None:
|
||||
assert agent.list_catalog(db) == []
|
||||
|
||||
|
||||
def test_find_document_hit_returns_full_row(kb, db) -> None:
|
||||
created = _doc(db, "Alpha", "deep/nested/doc.md", "The Doc", "FULL-TEXT")
|
||||
db.commit()
|
||||
|
||||
found = agent.find_document(db, "Alpha", "deep/nested/doc.md")
|
||||
assert found is not None
|
||||
assert found.id == created.id
|
||||
assert found.source == "Alpha"
|
||||
assert found.path == "deep/nested/doc.md"
|
||||
assert found.title == "The Doc"
|
||||
assert found.content == "FULL-TEXT" # the read tool feeds this, untruncated
|
||||
|
||||
|
||||
def test_find_document_none_for_unknown_pairs(kb, db) -> None:
|
||||
_doc(db, "Alpha", "x.md", "X", "X-CONTENT")
|
||||
db.commit()
|
||||
|
||||
assert agent.find_document(db, "Alpha", "nope.md") is None # wrong path
|
||||
assert agent.find_document(db, "Beta", "x.md") is None # wrong source
|
||||
assert agent.find_document(db, "nope", "nope.md") is None # nothing at all
|
||||
@@ -192,10 +192,21 @@ def _find_emoji(text: str) -> list[str]:
|
||||
)
|
||||
def test_ui_chrome_has_no_emoji(client, path: str) -> None:
|
||||
"""Permanent regression guard (phase 08): the UI chrome — all pages,
|
||||
the JS that renders it, and the stylesheet — is emoji-free."""
|
||||
the JS that renders it, and the stylesheet — is emoji-free.
|
||||
|
||||
Phase 37 revision (owner permission 2026-08-26, PLAN §4): the agent's
|
||||
``.tool-call`` line carries two CONTENT marks — 🔎 (list) and 📄
|
||||
(read) — the only emoji in the whole frontend, and only as the exact
|
||||
tool-line template strings in app.js. The guard strips precisely
|
||||
those two literals; any other emoji, or those marks anywhere else,
|
||||
still fails."""
|
||||
r = client.get(path)
|
||||
assert r.status_code == 200
|
||||
assert _find_emoji(r.text) == [], f"emoji found in {path}: {_find_emoji(r.text)!r}"
|
||||
text = r.text
|
||||
if path == "/assets/app.js":
|
||||
text = text.replace('"🔎 Listing documents"', "")
|
||||
text = text.replace('"📄 Reading "', "")
|
||||
assert _find_emoji(text) == [], f"emoji found in {path}: {_find_emoji(text)!r}"
|
||||
|
||||
|
||||
def test_chat_requires_message(client) -> None:
|
||||
|
||||
@@ -13,6 +13,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
@@ -27,8 +28,10 @@ from app.api import chat as chat_api
|
||||
from app.config import Settings, get_settings
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Chunk, QueryLog
|
||||
from app.rag import agent
|
||||
from app.rag.agent import AGENT_TOOLS
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.llm import EmbeddingError, LLMError, StreamPiece
|
||||
from app.rag.llm import EmbeddingError, LLMError, StreamPiece, ToolCallPiece
|
||||
|
||||
FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "docs"
|
||||
QUESTION = "How is my Kubernetes cluster set up?"
|
||||
@@ -57,6 +60,7 @@ class FakeRagLLM:
|
||||
embed_error: Exception | None = None,
|
||||
stream_error: Exception | None = None,
|
||||
fail_mid_stream: bool = False,
|
||||
tool_script: list[list[StreamPiece | ToolCallPiece]] | None = None,
|
||||
) -> None:
|
||||
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
self.embed_batches = 0
|
||||
@@ -67,6 +71,18 @@ class FakeRagLLM:
|
||||
self.fail_mid_stream = fail_mid_stream
|
||||
self.question_embeds: list[str] = []
|
||||
self.seen_messages: list[list[dict[str, str]]] = []
|
||||
#: Every request's ``tools`` value (phase 37) — ``None`` is the
|
||||
#: pre-phase request shape (the key is absent from the payload).
|
||||
self.seen_tools: list[list[dict[str, Any]] | None] = []
|
||||
#: Canned per-agent-round piece lists (phase 37): ``tool_script[i]``
|
||||
#: is yielded for the *i*-th request that carries a non-None
|
||||
#: ``tools`` parameter (a request the agent loop is offering tools
|
||||
#: on). A request without tools — the deflected direct path, the
|
||||
#: post-budget answer request, or the 0/0 single-request path —
|
||||
#: always yields the thinking + answer stream below, so a
|
||||
#: deflected turn through this fake is byte-identical to the
|
||||
#: plain fake's output.
|
||||
self.tool_script: list[list[StreamPiece | ToolCallPiece]] = list(tool_script or [])
|
||||
|
||||
async def embed(self, texts: list[str]) -> list[list[float]]:
|
||||
self.embed_batches += 1
|
||||
@@ -87,14 +103,25 @@ class FakeRagLLM:
|
||||
self.question_embeds.append(text)
|
||||
return _token_vec(text)
|
||||
|
||||
async def chat_stream(self, messages: list[dict[str, str]]):
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
):
|
||||
"""Typed stream (phase 17): ``thinking`` slices (same 12-char
|
||||
cadence as content) **before** the content pieces. With the
|
||||
default ``thinking=""`` this yields content-only pieces — today's
|
||||
behavior, new yield type."""
|
||||
behavior, new yield type. Phase 37: *tools* is the agent loop's
|
||||
``tools=…`` passthrough (recorded in ``seen_tools``); a request
|
||||
with tools consumes the next ``tool_script`` entry, if any."""
|
||||
self.seen_messages.append(messages)
|
||||
self.seen_tools.append(tools)
|
||||
if self.stream_error is not None:
|
||||
raise self.stream_error
|
||||
if tools is not None and self.tool_script:
|
||||
for piece in self.tool_script.pop(0):
|
||||
yield piece
|
||||
return
|
||||
if self.fail_mid_stream:
|
||||
yield StreamPiece("content", "partial ")
|
||||
raise LLMError("mid-stream dropout")
|
||||
@@ -464,3 +491,229 @@ def test_chat_query_log_failure_still_sends_done(client, db, seeded_kb: FakeRagL
|
||||
assert [f["type"] for f in frames if f["type"] == "delta"]
|
||||
assert frames[-1]["type"] == "done"
|
||||
assert frames[-1]["deflected"] is False
|
||||
|
||||
|
||||
# ---------- phase 37: agent document tools on grounded turns ----------
|
||||
|
||||
|
||||
def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""(a) Grounded turn with tool calls: the event sequence is
|
||||
``thinking?/tool/tool/delta…/done``; ``done.sources`` and the
|
||||
``query_log`` row include the read document (deduped, order
|
||||
preserved); the per-turn log line carries ``tool_calls=2``.
|
||||
The agent loop offers tools while budgets last and drops them
|
||||
(``tools=None``) once both are spent."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
StreamPiece("thinking", "Let me list what is indexed…"),
|
||||
ToolCallPiece(id="call_1", name="list_documents", arguments={}),
|
||||
],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={"source": "docs", "path": "homelab/backups.md"},
|
||||
)
|
||||
],
|
||||
# the post-budget answer request (tools=None) falls back to the
|
||||
# fake's thinking + answer stream
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
caplog.set_level(logging.INFO, logger="app.chat")
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
types = [f["type"] for f in frames]
|
||||
assert types[0] == "thinking"
|
||||
assert types[1] == "tool" and types[2] == "tool" # the two executed calls
|
||||
assert "error" not in types
|
||||
assert types[3:-1] == ["delta"] * (len(types) - 4) # deltas, then done last
|
||||
assert frames[-1]["type"] == "done"
|
||||
|
||||
list_frame, read_frame = frames[1], frames[2]
|
||||
assert set(list_frame) == {"type", "name", "argument"}
|
||||
assert list_frame["name"] == "list_documents"
|
||||
assert list_frame["argument"] is None # the tool takes no parameters
|
||||
assert set(read_frame) == {"type", "name", "argument"}
|
||||
assert read_frame["name"] == "read_document"
|
||||
assert read_frame["argument"] == "docs/homelab/backups.md"
|
||||
|
||||
deltas = [f for f in frames if f["type"] == "delta"]
|
||||
assert len(deltas) >= 2 # genuinely streamed
|
||||
assert "".join(d["text"] for d in deltas) == scripted.answer
|
||||
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
# done.sources = the retrieval docs + the read doc, deduped, order kept.
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert sources[-1] == ("docs", "homelab/backups.md") # the read doc is cited
|
||||
assert ("docs", "homelab/kubernetes.md") in sources # …after the retrieval docs
|
||||
assert len(sources) == len(set(sources)) # deduped by (source, path)
|
||||
assert done["sources"][-1]["title"] == "Backup Strategy"
|
||||
|
||||
# The agent loop offered the tools while any budget remained and
|
||||
# dropped them once both were spent (single post-budget request).
|
||||
assert len(scripted.seen_messages) == 3
|
||||
assert scripted.seen_tools[0] == AGENT_TOOLS
|
||||
assert scripted.seen_tools[1] == AGENT_TOOLS # the read budget was still open
|
||||
assert scripted.seen_tools[2] is None
|
||||
|
||||
# The query_log row carries the same combined source list.
|
||||
(row,) = db.scalars(select(QueryLog)).all()
|
||||
assert row.deflected is False
|
||||
assert "docs/homelab/kubernetes.md" in row.sources
|
||||
assert row.sources.endswith(", docs/homelab/backups.md") # the read doc, last
|
||||
|
||||
# The required per-turn log line (PLAN §9 extension) counts both calls
|
||||
# and lists the combined sources (retrieval + read).
|
||||
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
assert lines and "tool_calls=2" in lines[-1]
|
||||
assert "'docs/homelab/kubernetes.md'" in lines[-1]
|
||||
assert "'docs/homelab/backups.md'" in lines[-1]
|
||||
|
||||
|
||||
def test_deflected_turn_stays_byte_identical_without_tools(
|
||||
client, db, seeded_kb: FakeRagLLM
|
||||
) -> None:
|
||||
"""(b) Deflected turn: the agent loop never runs — no ``tool``
|
||||
frames, and the frame sequence is byte-identical to the plain fake's
|
||||
direct-``chat_stream`` output even for a fake scripted to call tools
|
||||
(its script is never consumed). The LLM was called once, without a
|
||||
``tools`` key."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={"source": "docs", "path": "homelab/backups.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "never used — the agent never runs")],
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
|
||||
try:
|
||||
_, _, baseline = _stream_chat(client, OFF_TOPIC)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, OFF_TOPIC)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
assert frames == baseline # byte-identical to the direct path
|
||||
assert not any(f["type"] == "tool" for f in frames)
|
||||
assert frames[-1]["type"] == "done" and frames[-1]["deflected"] is True
|
||||
assert len(scripted.tool_script) == 3 # the script was never consumed
|
||||
assert len(scripted.seen_messages) == 1
|
||||
assert scripted.seen_tools == [None] # one request, no tools key
|
||||
|
||||
# The read document never sneaks into the deflected turn's record.
|
||||
(row,) = [
|
||||
r
|
||||
for r in db.scalars(select(QueryLog)).all()
|
||||
if r.question == OFF_TOPIC
|
||||
][-1:]
|
||||
assert row.deflected is True
|
||||
assert "backups.md" not in row.sources
|
||||
|
||||
|
||||
def test_zero_agent_budgets_reproduce_pre_phase_single_request(
|
||||
client,
|
||||
db,
|
||||
seeded_kb: FakeRagLLM,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""(c) ``BOR_AGENT_LIST_CALLS=0 BOR_AGENT_READ_CALLS=0``: no ``tool``
|
||||
frames, exactly one request **without** a ``tools`` key (the
|
||||
pre-phase request shape), ``done.sources`` unchanged, and
|
||||
``tool_calls=0`` in the log line — budgets-as-kill-switch."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={"source": "docs", "path": "homelab/backups.md"},
|
||||
)
|
||||
],
|
||||
]
|
||||
)
|
||||
live = get_settings()
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=live.relevance_threshold,
|
||||
agent_list_calls=0,
|
||||
agent_read_calls=0,
|
||||
),
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
caplog.set_level(logging.INFO, logger="app.chat")
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
assert not any(f["type"] == "tool" for f in frames)
|
||||
assert "error" not in [f["type"] for f in frames]
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False
|
||||
paths = [s["path"] for s in done["sources"]]
|
||||
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
|
||||
assert "homelab/backups.md" not in paths # nothing was read
|
||||
|
||||
# Exactly one request, and it carried no ``tools`` key at all — the
|
||||
# scripted tool calls were never even offered a chance.
|
||||
assert len(scripted.seen_messages) == 1
|
||||
assert scripted.seen_tools == [None]
|
||||
assert len(scripted.tool_script) == 2 # never consumed
|
||||
|
||||
(row,) = db.scalars(select(QueryLog)).all()
|
||||
assert "docs/homelab/kubernetes.md" in row.sources
|
||||
assert "backups.md" not in row.sources
|
||||
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
assert lines and "tool_calls=0" in lines[-1]
|
||||
|
||||
|
||||
def test_tool_execution_db_failure_yields_error_event(
|
||||
client, db, seeded_kb: FakeRagLLM, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A tool call that hits a dead DB mid-stream gets the same structured
|
||||
``error`` event as the pre-stream retrieval path — never a severed
|
||||
stream (the "never stale" contract, PLAN §7.4)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[[ToolCallPiece(id="call_1", name="list_documents", arguments={})]]
|
||||
)
|
||||
|
||||
def boom(*_a: Any, **_k: Any) -> Any:
|
||||
raise RuntimeError("db exploded mid tool call")
|
||||
|
||||
monkeypatch.setattr(agent, "list_catalog", boom)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
# The ``tool`` frame went out first (the model requested the call);
|
||||
# the failed execution ends the turn with the structured error event.
|
||||
assert [f["type"] for f in frames] == ["tool", "error"]
|
||||
assert frames[0]["name"] == "list_documents"
|
||||
assert "offline mid-question" in frames[1]["detail"]
|
||||
assert db.scalars(select(QueryLog)).all() == [] # no row for a failed turn
|
||||
|
||||
@@ -0,0 +1,519 @@
|
||||
"""Unit: the grounded-turn agent loop (phase 37, ``app.rag.agent``).
|
||||
|
||||
A scripted fake LLM (canned stream sequences) + monkeypatched
|
||||
``list_catalog`` / ``find_document`` — no database, no network. Covers
|
||||
the loop mechanics: the list → read → answer happy path (event order,
|
||||
holder state, the ``tools=None`` request after the budgets are spent,
|
||||
the assistant/tool message history), the 0/0 single-call path, budget
|
||||
exhaustion, dedupe, unknown tool / missing args / unknown path, the
|
||||
round cap, and the ``<tools>`` prompt section (HIGH only).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import uuid
|
||||
from collections.abc import AsyncIterator
|
||||
from copy import deepcopy
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag import agent
|
||||
from app.rag.agent import (
|
||||
AGENT_TOOLS,
|
||||
AgentHolder,
|
||||
run_agent,
|
||||
)
|
||||
from app.rag.llm import LLMClient, StreamPiece, ToolCallPiece
|
||||
from app.rag.prompts import TOOLS_SECTION, _base, build_deflect_prompt, build_high_prompt
|
||||
|
||||
|
||||
def _settings(**kwargs: Any) -> Settings:
|
||||
kwargs.setdefault("_env_file", None)
|
||||
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _doc(source: str, path: str, title: str = "Title", content: str = "CONTENT") -> Document:
|
||||
return Document(
|
||||
id=uuid.uuid4(),
|
||||
source=source,
|
||||
path=path,
|
||||
full_path=f"/tmp/{path}",
|
||||
title=title,
|
||||
content=content,
|
||||
content_hash="0" * 64,
|
||||
)
|
||||
|
||||
|
||||
class ScriptedLLM:
|
||||
"""Canned stream sequences; records every ``chat_stream`` request so
|
||||
the tests can assert on the messages and the ``tools`` passthrough."""
|
||||
|
||||
def __init__(self, *streams: list[StreamPiece | ToolCallPiece]) -> None:
|
||||
self.streams: list[list[StreamPiece | ToolCallPiece]] = list(streams)
|
||||
self.requests: list[tuple[list[dict[str, Any]], list[dict[str, Any]] | None]] = []
|
||||
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
) -> AsyncIterator[StreamPiece | ToolCallPiece]:
|
||||
self.requests.append((deepcopy(messages), tools))
|
||||
if not self.streams:
|
||||
raise AssertionError("ScriptedLLM ran out of canned streams")
|
||||
for piece in self.streams.pop(0):
|
||||
yield piece
|
||||
|
||||
|
||||
async def _run(
|
||||
llm: ScriptedLLM,
|
||||
holder: AgentHolder,
|
||||
settings: Settings,
|
||||
seed_docs: list[Document] | None = None,
|
||||
) -> list[StreamPiece | ToolCallPiece]:
|
||||
out: list[StreamPiece | ToolCallPiece] = []
|
||||
async for piece in run_agent(
|
||||
cast("LLMClient", llm),
|
||||
cast("Session", None),
|
||||
system_prompt="SYSTEM_PROMPT",
|
||||
user_message="QUESTION",
|
||||
seed_docs=seed_docs or [],
|
||||
settings=settings,
|
||||
holder=holder,
|
||||
):
|
||||
out.append(piece)
|
||||
return out
|
||||
|
||||
|
||||
# ---------- AGENT_TOOLS shape ----------
|
||||
|
||||
|
||||
def test_agent_tools_names_and_parameters() -> None:
|
||||
by_name = {t["function"]["name"]: t for t in AGENT_TOOLS}
|
||||
assert set(by_name) == {"list_documents", "read_document"}
|
||||
assert all(t["type"] == "function" for t in AGENT_TOOLS)
|
||||
list_params = by_name["list_documents"]["function"]["parameters"]
|
||||
assert list_params["type"] == "object"
|
||||
assert list_params["properties"] == {} # no parameters
|
||||
read_params = by_name["read_document"]["function"]["parameters"]
|
||||
assert read_params["required"] == ["source", "path"]
|
||||
assert set(read_params["properties"]) == {"source", "path"}
|
||||
|
||||
|
||||
# ---------- happy path: list → read → answer ----------
|
||||
|
||||
|
||||
def test_list_then_read_then_answer(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
catalog = [
|
||||
("Deployments", "backups.md", "Backup Strategy"),
|
||||
("Homelab", "aws-route53.md", "AWS Route53 Records"),
|
||||
]
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
|
||||
target = _doc("Homelab", "aws-route53.md", "AWS Route53 Records", "R53-CONTENT")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: target)
|
||||
seed = [_doc("Homelab", "kubernetes.md", "Kubernetes", "K8S-CONTENT")]
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={"source": "Homelab", "path": "aws-route53.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("thinking", "hmm "), StreamPiece("content", "Done! ")],
|
||||
)
|
||||
|
||||
pieces = asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
|
||||
# Event order: tool pieces before the answer content/thinking.
|
||||
assert [type(p) for p in pieces] == [
|
||||
ToolCallPiece,
|
||||
ToolCallPiece,
|
||||
StreamPiece,
|
||||
StreamPiece,
|
||||
]
|
||||
assert pieces[0] == ToolCallPiece(id="call_1", name="list_documents", arguments={})
|
||||
assert isinstance(pieces[1], ToolCallPiece)
|
||||
assert pieces[1].name == "read_document"
|
||||
assert pieces[3] == StreamPiece("content", "Done! ")
|
||||
# The read document is recorded for done.sources / query_log (task 04).
|
||||
assert holder.read_docs == [target]
|
||||
assert holder.tool_calls == 2
|
||||
|
||||
# Default budgets (1/1): tools offered while any budget remains…
|
||||
assert llm.requests[0][1] == AGENT_TOOLS
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
# …and dropped (tools=None) once both are spent.
|
||||
assert llm.requests[2][1] is None
|
||||
assert len(llm.requests) == 3
|
||||
|
||||
# The follow-up request carries the assistant tool-call + tool result.
|
||||
msgs = llm.requests[1][0]
|
||||
assert msgs[0] == {"role": "system", "content": "SYSTEM_PROMPT"}
|
||||
assert msgs[1] == {"role": "user", "content": "QUESTION"}
|
||||
assert msgs[2] == {
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {"name": "list_documents", "arguments": "{}"},
|
||||
}
|
||||
],
|
||||
}
|
||||
assert msgs[3] == {
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_1",
|
||||
"content": (
|
||||
"2 documents:\n"
|
||||
"Deployments/backups.md — Backup Strategy\n"
|
||||
"Homelab/aws-route53.md — AWS Route53 Records"
|
||||
),
|
||||
}
|
||||
# The second follow-up request carries the read call + the FULL text.
|
||||
msgs = llm.requests[2][0]
|
||||
assert msgs[4]["role"] == "assistant"
|
||||
assert msgs[4]["tool_calls"][0]["id"] == "call_2"
|
||||
assert json.loads(msgs[4]["tool_calls"][0]["function"]["arguments"]) == {
|
||||
"source": "Homelab",
|
||||
"path": "aws-route53.md",
|
||||
}
|
||||
assert msgs[5] == {
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_2",
|
||||
"content": "Document Homelab/aws-route53.md:\nR53-CONTENT", # full text, no cap
|
||||
}
|
||||
|
||||
|
||||
def test_empty_catalog_listing_says_zero_documents(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
|
||||
assert holder.tool_calls == 1
|
||||
|
||||
|
||||
def test_content_and_tool_call_in_one_stream_keeps_both(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Rare stream with content AND a tool call: the content stays (it was
|
||||
already emitted) and the tool still runs."""
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
StreamPiece("content", "Let me check "),
|
||||
ToolCallPiece(id="call_1", name="list_documents", arguments={}),
|
||||
],
|
||||
[StreamPiece("content", "the answer")],
|
||||
)
|
||||
pieces = asyncio.run(_run(llm, holder, _settings()))
|
||||
assert [type(p) for p in pieces] == [StreamPiece, ToolCallPiece, StreamPiece]
|
||||
assert holder.tool_calls == 1 # the tool ran despite the content
|
||||
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
|
||||
|
||||
|
||||
# ---------- budgets ----------
|
||||
|
||||
|
||||
def test_zero_budgets_is_one_request_without_tools() -> None:
|
||||
"""BOR_AGENT_LIST_CALLS=0 BOR_AGENT_READ_CALLS=0 → byte-identical
|
||||
single-call path: exactly one request, tools=None, no history growth."""
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM([StreamPiece("thinking", "t "), StreamPiece("content", "direct answer")])
|
||||
pieces = asyncio.run(
|
||||
_run(llm, holder, _settings(agent_list_calls=0, agent_read_calls=0))
|
||||
)
|
||||
assert [type(p) for p in pieces] == [StreamPiece, StreamPiece]
|
||||
assert len(llm.requests) == 1
|
||||
assert llm.requests[0][1] is None
|
||||
assert llm.requests[0][0] == [
|
||||
{"role": "system", "content": "SYSTEM_PROMPT"},
|
||||
{"role": "user", "content": "QUESTION"},
|
||||
]
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
|
||||
|
||||
def test_read_budget_exhausted_refuses_and_appends_nothing(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
a = _doc("S", "a.md", "A", "A-CONTENT")
|
||||
monkeypatch.setattr(
|
||||
agent, "find_document", lambda db, source, path: a if path == "a.md" else None
|
||||
)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1", name="read_document", arguments={"source": "S", "path": "a.md"}
|
||||
)
|
||||
],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2", name="read_document", arguments={"source": "S", "path": "b.md"}
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1)))
|
||||
|
||||
assert holder.read_docs == [a] # the refused read appended nothing
|
||||
assert holder.tool_calls == 1 # …and consumed no budget
|
||||
refusal = llm.requests[2][0][5]
|
||||
assert refusal == {
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_2",
|
||||
"content": agent.READ_EXHAUSTED,
|
||||
}
|
||||
# The list budget is still open, so tools stay offered after the refusal.
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_list_budget_exhausted_refuses_with_its_own_message(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[ToolCallPiece(id="call_2", name="list_documents", arguments={})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1)))
|
||||
assert holder.tool_calls == 1
|
||||
assert llm.requests[2][0][5]["content"] == agent.LIST_EXHAUSTED
|
||||
# The read budget is still open, so tools stay offered after the refusal.
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
# ---------- rejections (no budget consumed) ----------
|
||||
|
||||
|
||||
def test_reading_a_seed_doc_is_already_in_context(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
seed = [_doc("Homelab", "kubernetes.md", "Kubernetes", "K8S-CONTENT")]
|
||||
|
||||
def _boom(*_a: Any, **_k: Any) -> None:
|
||||
raise AssertionError("find_document must not be called for a seeded doc")
|
||||
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
monkeypatch.setattr(agent, "find_document", _boom)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read_document",
|
||||
arguments={"source": "Homelab", "path": "kubernetes.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
# No budget consumed → tools are still offered on the next request.
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_reading_an_already_read_doc_is_deduped(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
doc = _doc("S", "a.md", "A", "A-CONTENT")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1", name="read_document", arguments={"source": "S", "path": "a.md"}
|
||||
)
|
||||
],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2", name="read_document", arguments={"source": "S", "path": "a.md"}
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1)))
|
||||
assert holder.read_docs == [doc] # appended exactly once
|
||||
assert holder.tool_calls == 1
|
||||
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
# The read budget is intact after the deduped refusal…
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_unknown_path_refused_without_budget(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read_document",
|
||||
arguments={"source": "S", "path": "ghost.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert (
|
||||
llm.requests[1][0][3]["content"]
|
||||
== "No document at S/ghost.md — check the list_documents output."
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # budget intact
|
||||
|
||||
|
||||
def test_unknown_tool_name_refused(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="delete_universe", arguments={"x": 1})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == agent.UNKNOWN_TOOL
|
||||
assert llm.requests[1][1] == AGENT_TOOLS # nothing was consumed
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("arguments", "label"),
|
||||
[
|
||||
({}, "no arguments"),
|
||||
({"source": "S"}, "path missing"),
|
||||
({"path": "p.md"}, "source missing"),
|
||||
({"source": "", "path": "p.md"}, "empty source"),
|
||||
({"source": "S", "path": " "}, "blank path"),
|
||||
({"source": 7, "path": "p.md"}, "non-string source"),
|
||||
],
|
||||
)
|
||||
def test_read_document_missing_arguments_refused(
|
||||
monkeypatch: pytest.MonkeyPatch, arguments: dict[str, Any], label: str
|
||||
) -> None:
|
||||
def _boom(*_a: Any, **_k: Any) -> None:
|
||||
raise AssertionError(f"find_document must not be called ({label})")
|
||||
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
monkeypatch.setattr(agent, "find_document", _boom)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read_document", arguments=arguments)],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == agent.MISSING_READ_ARGS
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
# ---------- round cap (pathological stream) ----------
|
||||
|
||||
|
||||
def test_round_cap_forces_a_final_no_tools_answer(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A model that keeps calling a budget-exhausted tool must be forced
|
||||
to answer at ``max_rounds = 2 + list + read`` (= 4 for 1/1)."""
|
||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="list_documents", arguments={})],
|
||||
[ToolCallPiece(id="call_2", name="list_documents", arguments={})],
|
||||
[ToolCallPiece(id="call_3", name="list_documents", arguments={})],
|
||||
[ToolCallPiece(id="call_4", name="list_documents", arguments={})],
|
||||
[StreamPiece("content", "forced answer")],
|
||||
)
|
||||
pieces = asyncio.run(_run(llm, holder, _settings(agent_list_calls=1, agent_read_calls=1)))
|
||||
assert [type(p) for p in pieces] == [
|
||||
ToolCallPiece,
|
||||
ToolCallPiece,
|
||||
ToolCallPiece,
|
||||
ToolCallPiece,
|
||||
StreamPiece,
|
||||
]
|
||||
assert len(llm.requests) == 5
|
||||
# The forced final request carries no tools, whatever is left.
|
||||
assert llm.requests[4][1] is None
|
||||
# Only the first call consumed budget; the three rejections did not.
|
||||
assert holder.tool_calls == 1
|
||||
# The 4th rejection sits at messages[2 + 4*2 - 1] of the final request.
|
||||
assert llm.requests[4][0][9]["content"] == agent.LIST_EXHAUSTED
|
||||
|
||||
|
||||
# ---------- settings ----------
|
||||
|
||||
|
||||
def test_agent_budget_settings_default_to_one_each() -> None:
|
||||
s = _settings()
|
||||
assert s.agent_list_calls == 1
|
||||
assert s.agent_read_calls == 1
|
||||
|
||||
|
||||
def test_agent_budget_settings_env_override(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("BOR_AGENT_LIST_CALLS", "0")
|
||||
monkeypatch.setenv("BOR_AGENT_READ_CALLS", "2")
|
||||
s = _settings()
|
||||
assert s.agent_list_calls == 0
|
||||
assert s.agent_read_calls == 2
|
||||
|
||||
|
||||
# ---------- prompts: <tools> section (HIGH only) ----------
|
||||
|
||||
|
||||
def test_high_prompt_carries_tools_section_after_documents() -> None:
|
||||
prompt = build_high_prompt([_doc("S", "a.md", "A", "A-CONTENT")])
|
||||
assert TOOLS_SECTION in prompt
|
||||
assert "call `list_documents`" in prompt
|
||||
assert "then `read_document` to pull in exactly one more document" in prompt
|
||||
assert "do not read more than one extra document" in prompt
|
||||
# After the mode body: <tools> follows </documents>.
|
||||
assert prompt.index("</documents>") < prompt.index("<tools>")
|
||||
assert prompt.rstrip().endswith("</tools>")
|
||||
|
||||
|
||||
def test_high_prompt_tools_section_with_notes_and_kb() -> None:
|
||||
prompt = build_high_prompt(
|
||||
[_doc("S", "a.md", "A", "A-CONTENT")], notes=["be concise"], kb_overview="- KB"
|
||||
)
|
||||
assert prompt.index("<knowledge_base>") < prompt.index("<tuning>")
|
||||
assert prompt.index("<tuning>") < prompt.index("<documents>")
|
||||
assert prompt.index("<documents>") < prompt.index("<tools>")
|
||||
|
||||
|
||||
def test_low_prompt_is_byte_identical_and_tool_free() -> None:
|
||||
expected = (
|
||||
_base("LOW")
|
||||
+ "\nDEFLECT_MODE: retrieval was weak — the titles below are the closest "
|
||||
"your notes come to the question. They are titles only; do not pretend "
|
||||
"they answer it. Use them to propose 2-3 alternative questions.\n"
|
||||
+ "- T1\n- T2"
|
||||
)
|
||||
assert build_deflect_prompt(["T1", "T2"]) == expected
|
||||
for prompt in (
|
||||
build_deflect_prompt(["T1"]),
|
||||
build_deflect_prompt(["T1"], notes=["be concise"]),
|
||||
build_deflect_prompt(["T1"], kb_overview="- KB"),
|
||||
build_deflect_prompt(["T1"], notes=["be concise"], kb_overview="- KB"),
|
||||
):
|
||||
assert "<tools>" not in prompt
|
||||
assert TOOLS_SECTION not in prompt
|
||||
@@ -20,6 +20,7 @@ from app.api import chat as chat_api
|
||||
from app.config import Settings
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Document, KbOverview, QueryLog
|
||||
from app.rag.agent import AGENT_TOOLS
|
||||
from app.rag.llm import StreamPiece
|
||||
from app.rag.retriever import RetrievedChunk, weak_hit_titles
|
||||
from app.rag.suggestions import MAX_SUGGESTIONS, derive_suggestions
|
||||
@@ -416,19 +417,30 @@ def test_suggestions_empty_input_yields_fallback_only() -> None:
|
||||
|
||||
|
||||
class _CannedLLM:
|
||||
"""Records the messages it is given; streams a canned answer."""
|
||||
"""Records the messages it is given; streams a canned answer.
|
||||
|
||||
Never emits tool calls, so a grounded turn through the phase-37 agent
|
||||
loop ends after the single (tools-offered) request; *seen_tools*
|
||||
records each request's ``tools`` value for the phase-37 wiring pins.
|
||||
"""
|
||||
|
||||
def __init__(self, answer: str = ANSWER) -> None:
|
||||
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
self.embed_batches = 0
|
||||
self.answer = answer
|
||||
self.seen: list[list[dict[str, str]]] = []
|
||||
self.seen_tools: list[list[dict[str, Any]] | None] = []
|
||||
|
||||
async def embed_one(self, _text: str) -> list[float]:
|
||||
return [0.0] * 768
|
||||
|
||||
async def chat_stream(self, messages: list[dict[str, str]]):
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
):
|
||||
self.seen.append(messages)
|
||||
self.seen_tools.append(tools)
|
||||
for i in range(0, len(self.answer), 12):
|
||||
yield StreamPiece("content", self.answer[i : i + 12])
|
||||
|
||||
@@ -545,6 +557,55 @@ def test_endpoint_just_below_threshold_deflects(
|
||||
assert session.commits == 1
|
||||
|
||||
|
||||
def test_endpoint_grounded_turn_runs_agent_loop_with_tools(
|
||||
client: TestClient,
|
||||
gate_env: tuple[_FakeSession, _CannedLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 37: a grounded endpoint turn runs the agent loop — the
|
||||
single no-tool-call request carries ``AGENT_TOOLS`` (default 1/1
|
||||
budgets), no ``tool`` frames stream, and the ``done`` event is the
|
||||
plain retrieval shape (the tool-free answer is byte-identical)."""
|
||||
_session, llm = gate_env
|
||||
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT")
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.90)]))
|
||||
|
||||
frames = _ask(client, "How is my Kubernetes cluster set up?")
|
||||
|
||||
assert frames[-1]["type"] == "done"
|
||||
assert frames[-1]["deflected"] is False
|
||||
assert not any(f["type"] == "tool" for f in frames)
|
||||
assert len(llm.seen) == 1
|
||||
assert llm.seen_tools == [AGENT_TOOLS] # one request, tools offered
|
||||
# The system prompt is the HIGH prompt with the <tools> instructions.
|
||||
(system, _user) = llm.seen[0][0], llm.seen[0][1]
|
||||
assert "<relevance>HIGH</relevance>" in system["content"]
|
||||
assert "<tools>" in system["content"]
|
||||
|
||||
|
||||
def test_endpoint_deflected_turn_never_offers_tools(
|
||||
client: TestClient,
|
||||
gate_env: tuple[_FakeSession, _CannedLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 37: a deflected endpoint turn keeps the direct
|
||||
``chat_stream`` — the single request carries no ``tools`` key
|
||||
(``seen_tools == [None]``), A8 byte-identical."""
|
||||
_session, llm = gate_env
|
||||
doc = _doc("Deploying a New Service", "DOC_CONTENT_NEVER_SENT")
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.2999)]))
|
||||
|
||||
frames = _ask(client, "How do I bake sourdough bread?")
|
||||
|
||||
assert frames[-1]["type"] == "done"
|
||||
assert frames[-1]["deflected"] is True
|
||||
assert not any(f["type"] == "tool" for f in frames)
|
||||
assert len(llm.seen) == 1
|
||||
assert llm.seen_tools == [None]
|
||||
(system, _user) = llm.seen[0][0], llm.seen[0][1]
|
||||
assert "<tools>" not in system["content"] # the LOW prompt never carries it
|
||||
|
||||
|
||||
def test_endpoint_score_at_threshold_answers(
|
||||
client: TestClient,
|
||||
gate_env: tuple[_FakeSession, _CannedLLM],
|
||||
|
||||
@@ -114,7 +114,10 @@ def test_save_points_user_on_send_and_brain_on_done() -> None:
|
||||
# in the same meta object).
|
||||
done_idx = js.find('ev.type === "done"')
|
||||
assert done_idx != -1
|
||||
done_block = js[done_idx : done_idx + 1300]
|
||||
# Window: the whole done branch (up to the error branch) — the meta
|
||||
# object legitimately grows with phases (phase 17: thinking, phase
|
||||
# 37: tools), so a fixed char offset would false-fail.
|
||||
done_block = js[done_idx : js.find('ev.type === "error"')]
|
||||
assert "rememberBrainTurn(finalText || acc" in done_block
|
||||
assert "thinking: thinkingAcc || undefined" in done_block
|
||||
assert "deflected: !!ev.deflected" in done_block
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
"""Unit: the phase-37 "calling tool" frontend contract (task 05).
|
||||
|
||||
No new Python app logic exists for this task — the behavior lives in
|
||||
``frontend/assets/app.js`` + ``styles.css`` and is E2E-gated by the story
|
||||
suite (task 06). Like the other frontend-adjacent unit files, this module
|
||||
pins the JS/CSS markers the story depends on, so a silent regression in
|
||||
the tool branch, the persistence shape, or the tool-line styling is
|
||||
caught without a browser.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
|
||||
APP_JS = FRONTEND / "assets" / "app.js"
|
||||
STYLES_CSS = FRONTEND / "assets" / "styles.css"
|
||||
|
||||
|
||||
def _js() -> str:
|
||||
return APP_JS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _css() -> str:
|
||||
return STYLES_CSS.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_tool_branch_is_a_first_class_turn_branch() -> None:
|
||||
"""The turn handler must branch on `tool` frames BETWEEN the
|
||||
thinking and delta branches: the stream stays alive (guard clears),
|
||||
the brain wrap is created on demand, and the label state is
|
||||
applied only while the UI state is still "thinking" (a late frame
|
||||
after the first delta just appends the line — never a crash)."""
|
||||
js = _js()
|
||||
thinking_idx = js.find('ev.type === "thinking"')
|
||||
tool_idx = js.find('ev.type === "tool"')
|
||||
delta_idx = js.find('ev.type === "delta"')
|
||||
assert -1 < thinking_idx < tool_idx < delta_idx, (
|
||||
"the turn handler must branch on tool frames"
|
||||
)
|
||||
branch = js[tool_idx:delta_idx]
|
||||
assert "toolAcc.push" in branch, "every tool frame is recorded for persistence"
|
||||
assert "clearTurnTimeout()" in branch, "a tool frame proves the stream is alive"
|
||||
assert 'addMessage("brain", "")' in branch, "first frame creates the brain wrap"
|
||||
assert "uiState === UI_STATE.thinking" in branch, (
|
||||
"label updates only while the state is still thinking"
|
||||
)
|
||||
assert "appendToolLine(wrap, name, argument)" in branch
|
||||
# No setUiState in the branch: the state stays "thinking" (never stale).
|
||||
assert "setUiState" not in branch, (
|
||||
"the tool branch must keep uiState=thinking (button stays disabled)"
|
||||
)
|
||||
|
||||
|
||||
def test_calling_tool_label_strings() -> None:
|
||||
"""The 'calling tool' label strings the story keys off: the button
|
||||
text and the status/typing-indicator labels (plain literals —
|
||||
phase 39 centralizes brand strings; no helper here)."""
|
||||
js = _js()
|
||||
tool_idx = js.find('ev.type === "tool"')
|
||||
delta_idx = js.find('ev.type === "delta"')
|
||||
branch = js[tool_idx:delta_idx]
|
||||
assert '"Calling tool…"' in branch, "the button carries the calling-tool text"
|
||||
assert '"Brain of Reese is listing documents"' in branch
|
||||
assert "Brain of Reese is reading ${argument}" in branch
|
||||
assert "sendStatus.textContent = toolStatus" in branch, (
|
||||
"the #send-status live region announces what Brain is doing"
|
||||
)
|
||||
assert 'setAttribute("aria-label", toolStatus)' in branch, (
|
||||
"the typing indicator label follows the tool state"
|
||||
)
|
||||
# The elapsed-seconds hint keeps running through tool frames: the
|
||||
# branch must not stop/restart the clock.
|
||||
assert "stopThinkingClock" not in branch
|
||||
assert "startThinkingClock" not in branch
|
||||
|
||||
|
||||
def test_tool_lines_render_into_the_bubble_wrap() -> None:
|
||||
"""appendToolLine: first frame creates the .tool-calls list (role=list
|
||||
+ accessible name) BEFORE the bubble — below an existing Thinking
|
||||
block — and each line is a .tool-call listitem with the exact marks:
|
||||
🔎 Listing documents / 📄 Reading <code>path</code>. The path goes
|
||||
through textContent (storage can never inject HTML)."""
|
||||
js = _js()
|
||||
fn = js.find("function appendToolLine")
|
||||
assert fn != -1, "appendToolLine must exist"
|
||||
body = js[fn : js.find("\n}\n", fn)]
|
||||
assert 'querySelector(".tool-calls")' in body, "idempotent per wrap"
|
||||
assert 'className = "tool-calls"' in body
|
||||
assert 'setAttribute("role", "list")' in body
|
||||
assert 'setAttribute("aria-label", "Tool calls")' in body
|
||||
assert 'insertBefore(container, body.querySelector(".bubble"))' in body, (
|
||||
"the list sits ABOVE the answer"
|
||||
)
|
||||
assert 'className = "tool-call"' in body
|
||||
assert 'setAttribute("role", "listitem")' in body
|
||||
assert 'line.textContent = "📄 Reading "' in body
|
||||
assert 'line.textContent = "🔎 Listing documents"' in body
|
||||
assert "code.textContent = argument" in body, (
|
||||
"the path is data — textContent, never innerHTML"
|
||||
)
|
||||
assert "name === \"read_document\" && argument" in body
|
||||
|
||||
|
||||
def test_tool_branch_is_append_only_and_interleaving_safe() -> None:
|
||||
"""Append-only, the same rule as thinking: multiple calls append
|
||||
multiple lines in order, and a tool frame after the first delta
|
||||
(should not happen in v1) still appends — the branch has no early
|
||||
return gated on acc/delta state."""
|
||||
js = _js()
|
||||
tool_idx = js.find('ev.type === "tool"')
|
||||
delta_idx = js.find('ev.type === "delta"')
|
||||
branch = js[tool_idx:delta_idx]
|
||||
assert "if (aborted) return" not in branch, (
|
||||
"the aborted guard lives at the dispatch top, not per branch"
|
||||
)
|
||||
assert "acc" not in branch.split("appendToolLine")[0].replace("toolAcc", ""), (
|
||||
"the tool branch must not depend on accumulated answer text"
|
||||
)
|
||||
|
||||
|
||||
def test_tool_frames_persist_next_to_thinking() -> None:
|
||||
"""The `done` save point gains an optional `tools` key next to
|
||||
`thinking` (empty turns persist exactly as before — `undefined`
|
||||
drops the key from the JSON), and the accumulator is turn-scoped
|
||||
in handleSend."""
|
||||
js = _js()
|
||||
assert "let toolAcc = []" in js
|
||||
done_idx = js.find('ev.type === "done"')
|
||||
error_idx = js.find('ev.type === "error"')
|
||||
assert -1 < done_idx < error_idx
|
||||
done_block = js[done_idx:error_idx]
|
||||
assert "thinking: thinkingAcc || undefined" in done_block
|
||||
assert "tools: toolAcc.length ? toolAcc : undefined" in done_block, (
|
||||
"tools persisted next to thinking, optional like it"
|
||||
)
|
||||
|
||||
|
||||
def test_tool_lines_re_render_on_restore() -> None:
|
||||
"""Phase 14 convention: a stored brain record with `tools` re-renders
|
||||
the lines on load through the SAME append helper (after the thinking
|
||||
re-render, before the deflected/sources additions)."""
|
||||
js = _js()
|
||||
fn = js.find("function renderStoredMessage")
|
||||
assert fn != -1
|
||||
end = js.find("function restoreConversation")
|
||||
body = js[fn:end]
|
||||
assert "Array.isArray(m.tools)" in body
|
||||
assert "appendToolLine(wrap, t.name, arg)" in body
|
||||
thinking_restore = body.find("if (m.thinking)")
|
||||
tools_restore = body.find("Array.isArray(m.tools)")
|
||||
assert -1 < thinking_restore < tools_restore, (
|
||||
"tools restore sits after the thinking restore (same order as live)"
|
||||
)
|
||||
assert "typeof t.name !== \"string\"" in body, "malformed entries skipped"
|
||||
|
||||
|
||||
def test_thinking_block_stays_on_top_of_tool_lines() -> None:
|
||||
"""If a tool frame precedes the first thinking frame, the Thinking
|
||||
block is still created ABOVE the tool lines (scratchpad on top),
|
||||
not below them."""
|
||||
js = _js()
|
||||
fn = js.find("function ensureThinkingBlock")
|
||||
assert fn != -1
|
||||
body = js[fn : js.find("\n}\n", fn)]
|
||||
assert 'querySelector(".tool-calls")' in body, (
|
||||
"the thinking anchor must account for existing tool lines"
|
||||
)
|
||||
assert 'querySelector(".bubble")' in body
|
||||
assert "insertBefore" in body
|
||||
assert "block.open = true" in body
|
||||
|
||||
|
||||
def test_tool_call_style_is_accent_and_contrast_safe() -> None:
|
||||
"""styles.css: .tool-call is an inline row with the accent palette
|
||||
(distinct from the brand-ink Thinking block) and mono `code` styling
|
||||
for the path; the wrapper stacks lines without shifting the column."""
|
||||
css = _css()
|
||||
assert ".tool-calls" in css
|
||||
assert ".tool-call" in css
|
||||
m = re.search(r"\.tool-call \{([^}]*)\}", css)
|
||||
assert m, "the .tool-call rule must exist"
|
||||
row = m.group(1)
|
||||
assert "display: flex" in row, "inline row: icon + text"
|
||||
assert "var(--accent-ink)" in row, (
|
||||
"accent color distinguishes it from the thinking block (≈10.4:1 on surface)"
|
||||
)
|
||||
assert "var(--accent-line)" in row, "accent left border"
|
||||
code = re.search(r"\.tool-call code \{([^}]*)\}", css)
|
||||
assert code, "the path `code` must be styled"
|
||||
assert "var(--mono)" in code.group(1)
|
||||
assert "var(--ink)" in code.group(1) # ≈11.5:1 on --brand-soft
|
||||
assert "gap" in css.split(".tool-calls {")[1].split("}")[0], (
|
||||
"lines stack with a gap — append-only, no reflow"
|
||||
)
|
||||
|
||||
|
||||
def test_no_cdn_added() -> None:
|
||||
"""AGENTS.md rule 6: the tool state adds no external script/link."""
|
||||
index = (FRONTEND / "index.html").read_text(encoding="utf-8")
|
||||
assert 'src="http' not in index and 'href="http' not in index
|
||||
@@ -11,7 +11,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -22,6 +22,7 @@ from app.rag.llm import (
|
||||
LLMClient,
|
||||
LLMError,
|
||||
StreamPiece,
|
||||
ToolCallPiece,
|
||||
)
|
||||
|
||||
|
||||
@@ -242,21 +243,47 @@ def test_single_oversized_text_fails_actionably() -> None:
|
||||
# ---------- chat streaming (phase 03) ----------
|
||||
|
||||
|
||||
def _tool_call(
|
||||
index: int,
|
||||
id: str | None = None,
|
||||
name: str | None = None,
|
||||
arguments: str | None = None,
|
||||
):
|
||||
"""One fake ``delta.tool_calls[]`` partial (openai SDK shape, phase 37).
|
||||
|
||||
``function`` is None when neither *name* nor *arguments* is given —
|
||||
mirroring the real wire, where id-only fragments carry no function.
|
||||
"""
|
||||
fn = None
|
||||
if name is not None or arguments is not None:
|
||||
fn = SimpleNamespace(name=name, arguments=arguments)
|
||||
return SimpleNamespace(index=index, id=id, function=fn)
|
||||
|
||||
|
||||
def _chunk(
|
||||
content: str | None = "text", empty: bool = False, reasoning: str | None = None
|
||||
content: str | None = "text",
|
||||
empty: bool = False,
|
||||
reasoning: str | None = None,
|
||||
tool_calls: list | None = None,
|
||||
finish_reason: str | None = None,
|
||||
):
|
||||
"""One fake ChatCompletionChunk (``choices[].delta`` shape).
|
||||
|
||||
``reasoning_content`` is present on the delta only when *reasoning*
|
||||
is not None — mirroring the real wire, where the field exists only
|
||||
when the model sends it.
|
||||
``reasoning_content``, ``tool_calls`` and ``finish_reason`` are
|
||||
present only when provided — mirroring the real wire, where the
|
||||
fields exist only when the model sends them.
|
||||
"""
|
||||
if empty:
|
||||
return SimpleNamespace(choices=[])
|
||||
delta: SimpleNamespace = SimpleNamespace(content=content)
|
||||
if reasoning is not None:
|
||||
delta.reasoning_content = reasoning
|
||||
return SimpleNamespace(choices=[SimpleNamespace(delta=delta)])
|
||||
if tool_calls is not None:
|
||||
delta.tool_calls = tool_calls
|
||||
choice = SimpleNamespace(delta=delta)
|
||||
if finish_reason is not None:
|
||||
choice.finish_reason = finish_reason
|
||||
return SimpleNamespace(choices=[choice])
|
||||
|
||||
|
||||
class _FakeChatStream:
|
||||
@@ -326,7 +353,11 @@ def _make_stream_client(
|
||||
|
||||
|
||||
async def _collect(llm: LLMClient, messages: list[dict[str, str]]) -> list[StreamPiece]:
|
||||
return [p async for p in llm.chat_stream(messages)]
|
||||
"""Collect pieces from a tools-less stream (phase 37 task 02, test (a):
|
||||
without tools, no ToolCallPiece can appear)."""
|
||||
pieces = [p async for p in llm.chat_stream(messages)]
|
||||
assert all(isinstance(p, StreamPiece) for p in pieces)
|
||||
return cast("list[StreamPiece]", pieces)
|
||||
|
||||
|
||||
def test_chat_stream_yields_deltas_in_order() -> None:
|
||||
@@ -355,6 +386,9 @@ def test_chat_stream_uses_locked_generation_params() -> None:
|
||||
# BOR_MAX_OUTPUT_TOKENS (default 32 768) so they are not cut off.
|
||||
assert completions.kwargs["max_tokens"] == 32_768
|
||||
assert completions.kwargs["messages"] == messages
|
||||
# Phase 37: no tools passed ⇒ no `tools` key at all (byte-identical
|
||||
# request to pre-phase-37).
|
||||
assert "tools" not in completions.kwargs
|
||||
|
||||
|
||||
def test_chat_stream_max_tokens_comes_from_settings() -> None:
|
||||
@@ -455,6 +489,231 @@ def test_chat_stream_llm_error_passes_through_unwrapped() -> None:
|
||||
asyncio.run(_collect(llm, [{"role": "user", "content": "q"}]))
|
||||
|
||||
|
||||
# ---------- tool-call streaming (phase 37, task 02) ----------
|
||||
|
||||
#: The agent's tool list (phase 37) — the exact wire shape AGENT_TOOLS will
|
||||
#: pass through (the names are whatever the caller's tools list names).
|
||||
_AGENT_TOOLS: list[dict[str, Any]] = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "list_documents",
|
||||
"description": "List the indexed documents.",
|
||||
"parameters": {"type": "object", "properties": {}},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "read_document",
|
||||
"description": "Add one indexed document's full text to the context.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"source": {"type": "string"},
|
||||
"path": {"type": "string"},
|
||||
},
|
||||
"required": ["source", "path"],
|
||||
},
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _collect_with_tools(
|
||||
llm: LLMClient, messages: list[dict[str, str]], tools: list[dict[str, Any]]
|
||||
) -> list[StreamPiece | ToolCallPiece]:
|
||||
async def run() -> list[StreamPiece | ToolCallPiece]:
|
||||
return [p async for p in llm.chat_stream(messages, tools=tools)]
|
||||
|
||||
return asyncio.run(run())
|
||||
|
||||
|
||||
def test_chat_stream_passes_tools_when_given() -> None:
|
||||
"""(e) A non-None tools list is forwarded verbatim to create()."""
|
||||
llm, completions = _make_stream_client([_chunk("ok")], llm_chat_model="turbo")
|
||||
_collect_with_tools(llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS)
|
||||
assert completions.kwargs is not None
|
||||
assert completions.kwargs["tools"] == _AGENT_TOOLS
|
||||
|
||||
|
||||
def test_chat_stream_accumulates_tool_call_across_chunk_partials() -> None:
|
||||
"""(b) name on the first partial, arguments in fragments — merged into
|
||||
one ToolCallPiece with the concatenated JSON, at finish_reason."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(
|
||||
0,
|
||||
id="call_abc",
|
||||
name="read_document",
|
||||
arguments='{"source": "Homelab", "pa',
|
||||
)
|
||||
],
|
||||
),
|
||||
_chunk(None, tool_calls=[_tool_call(0, arguments='th": "kubernetes.md"}')]),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
pieces = _collect_with_tools(
|
||||
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
|
||||
)
|
||||
assert pieces == [
|
||||
ToolCallPiece(
|
||||
id="call_abc",
|
||||
name="read_document",
|
||||
arguments={"source": "Homelab", "path": "kubernetes.md"},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_chat_stream_two_tool_calls_yielded_in_index_order() -> None:
|
||||
"""(c) Indices 0 and 1, interleaved partials (index 1 seen first) —
|
||||
both calls, in index order, each merged from its own fragments."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(1, id="call_b", name="read_document", arguments='{"sou')
|
||||
],
|
||||
),
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(0, id="call_a", name="list_documents"),
|
||||
_tool_call(1, arguments='rce": "Homelab", "path": "a.md"}')
|
||||
],
|
||||
),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
pieces = _collect_with_tools(
|
||||
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
|
||||
)
|
||||
assert pieces == [
|
||||
ToolCallPiece(id="call_a", name="list_documents", arguments={}),
|
||||
ToolCallPiece(
|
||||
id="call_b",
|
||||
name="read_document",
|
||||
arguments={"source": "Homelab", "path": "a.md"},
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def test_chat_stream_tool_calls_yielded_at_stream_end_without_finish_reason() -> None:
|
||||
"""The spec's other emission point: stream ends without a
|
||||
finish_reason="tool_calls" chunk — pieces still materialize."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[_tool_call(0, id="call_z", name="list_documents")],
|
||||
)
|
||||
]
|
||||
)
|
||||
pieces = _collect_with_tools(
|
||||
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
|
||||
)
|
||||
assert pieces == [ToolCallPiece(id="call_z", name="list_documents", arguments={})]
|
||||
|
||||
|
||||
def test_chat_stream_synthesizes_call_id_when_absent() -> None:
|
||||
"""Wire never carried the call id ⇒ synthesized "call_<index>"."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(None, tool_calls=[_tool_call(2, name="read_document", arguments="{}")]),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
pieces = _collect_with_tools(
|
||||
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
|
||||
)
|
||||
assert pieces == [
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read_document",
|
||||
arguments={},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_chat_stream_null_arguments_become_empty_dict() -> None:
|
||||
"""JSON "null" (and, by the same branch, absent arguments) ⇒ {}."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(0, id="call_n", name="list_documents", arguments="null")
|
||||
],
|
||||
),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
pieces = _collect_with_tools(
|
||||
llm, [{"role": "user", "content": "q"}], _AGENT_TOOLS
|
||||
)
|
||||
assert pieces == [ToolCallPiece(id="call_n", name="list_documents", arguments={})]
|
||||
|
||||
|
||||
def test_chat_stream_malformed_tool_arguments_raise_llm_error() -> None:
|
||||
"""(d) A silently dropped tool call would corrupt the loop — malformed
|
||||
arguments JSON must fail loudly."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(
|
||||
0,
|
||||
id="call_x",
|
||||
name="read_document",
|
||||
arguments='{"source": "Homelab",',
|
||||
)
|
||||
],
|
||||
),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
|
||||
async def drain() -> None:
|
||||
async for _ in llm.chat_stream(
|
||||
[{"role": "user", "content": "q"}], tools=_AGENT_TOOLS
|
||||
):
|
||||
pass
|
||||
|
||||
with pytest.raises(LLMError, match="malformed tool-call arguments"):
|
||||
asyncio.run(drain())
|
||||
|
||||
|
||||
def test_chat_stream_non_object_tool_arguments_raise_llm_error() -> None:
|
||||
"""The OpenAI contract says arguments is a JSON *object* — a bare array
|
||||
is malformed too."""
|
||||
llm, _ = _make_stream_client(
|
||||
[
|
||||
_chunk(
|
||||
None,
|
||||
tool_calls=[
|
||||
_tool_call(0, id="call_y", name="read_document", arguments='[1, 2]')
|
||||
],
|
||||
),
|
||||
_chunk(None, finish_reason="tool_calls"),
|
||||
]
|
||||
)
|
||||
|
||||
async def drain() -> None:
|
||||
async for _ in llm.chat_stream(
|
||||
[{"role": "user", "content": "q"}], tools=_AGENT_TOOLS
|
||||
):
|
||||
pass
|
||||
|
||||
with pytest.raises(LLMError, match="non-object tool-call arguments"):
|
||||
asyncio.run(drain())
|
||||
|
||||
|
||||
# ---------- one-shot chat: LLMClient.chat (phase 30, task 01) ----------
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
"""Unit tests: scripts/llm_probe.py --tools response parsing (phase 37, task 01).
|
||||
|
||||
The live probe talks to aipi; the parsing and verdict-classification logic
|
||||
is factored into pure functions and is what this module pins. The fixtures
|
||||
mirror the exact wire shapes observed live against ``turbo`` on 2026-08-26
|
||||
(reasoning_content first, then indexed delta.tool_calls fragments).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from scripts.llm_probe import (
|
||||
classify_tool_calling,
|
||||
parse_tool_response_nonstreaming,
|
||||
parse_tool_response_streaming,
|
||||
)
|
||||
|
||||
|
||||
def test_parse_nonstreaming_tool_calls() -> None:
|
||||
payload = {
|
||||
"choices": [
|
||||
{
|
||||
"finish_reason": "tool_calls",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"reasoning_content": "Let me call it.\n",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "abc123",
|
||||
"type": "function",
|
||||
"function": {"name": "get_time", "arguments": "{}"},
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
]
|
||||
}
|
||||
out = parse_tool_response_nonstreaming(payload)
|
||||
assert out == {"finish_reason": "tool_calls", "calls": [("get_time", "{}")]}
|
||||
|
||||
|
||||
def test_parse_nonstreaming_plain_content_answer() -> None:
|
||||
payload = {
|
||||
"choices": [
|
||||
{"finish_reason": "stop", "message": {"role": "assistant", "content": "It is noon."}}
|
||||
]
|
||||
}
|
||||
assert parse_tool_response_nonstreaming(payload) == {
|
||||
"finish_reason": "stop",
|
||||
"calls": [],
|
||||
}
|
||||
|
||||
|
||||
def test_parse_nonstreaming_empty_or_malformed() -> None:
|
||||
assert parse_tool_response_nonstreaming({"choices": []}) == {
|
||||
"finish_reason": None,
|
||||
"calls": [],
|
||||
}
|
||||
assert parse_tool_response_nonstreaming({}) == {"finish_reason": None, "calls": []}
|
||||
assert parse_tool_response_nonstreaming(None) == {"finish_reason": None, "calls": []}
|
||||
|
||||
|
||||
def _sse(payload: dict) -> str:
|
||||
return "data: " + json.dumps(payload)
|
||||
|
||||
|
||||
def test_parse_streaming_accumulates_fragments() -> None:
|
||||
"""The live wire shape: id+name+partial args in chunk 1, args in chunk 2."""
|
||||
lines = [
|
||||
_sse({"choices": [{"delta": {"reasoning_content": "Let me think."}, "index": 0}]}),
|
||||
_sse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {
|
||||
"tool_calls": [
|
||||
{
|
||||
"index": 0,
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {"name": "get_time", "arguments": "{"},
|
||||
}
|
||||
]
|
||||
},
|
||||
"index": 0,
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
_sse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {"tool_calls": [{"index": 0, "function": {"arguments": "}"}}]},
|
||||
"index": 0,
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
_sse({"choices": [{"delta": {}, "finish_reason": "tool_calls", "index": 0}]}),
|
||||
"data: [DONE]",
|
||||
]
|
||||
out = parse_tool_response_streaming(lines)
|
||||
assert out["finish_reason"] == "tool_calls"
|
||||
assert out["calls"] == [("get_time", "{}")]
|
||||
assert out["delta_chunks"] == 2
|
||||
assert out["indexed"] is True
|
||||
assert out["had_id"] is True
|
||||
assert out["arguments_in_deltas"] is True
|
||||
|
||||
|
||||
def test_parse_streaming_no_tool_calls() -> None:
|
||||
lines = [
|
||||
_sse({"choices": [{"delta": {"content": "It is "}, "index": 0}]}),
|
||||
_sse({"choices": [{"delta": {"content": "noon."}, "index": 0}]}),
|
||||
_sse({"choices": [{"delta": {}, "finish_reason": "stop", "index": 0}]}),
|
||||
"data: [DONE]",
|
||||
]
|
||||
out = parse_tool_response_streaming(lines)
|
||||
assert out["finish_reason"] == "stop"
|
||||
assert out["calls"] == []
|
||||
assert out["delta_chunks"] == 0
|
||||
assert out["had_id"] is False
|
||||
assert out["arguments_in_deltas"] is False
|
||||
|
||||
|
||||
def test_parse_streaming_stops_at_done_and_skips_malformed() -> None:
|
||||
lines = [
|
||||
"data: not-json",
|
||||
"",
|
||||
_sse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {
|
||||
"tool_calls": [
|
||||
{
|
||||
"index": 0,
|
||||
"id": "x",
|
||||
"type": "function",
|
||||
"function": {"name": "get_time", "arguments": "{}"},
|
||||
}
|
||||
]
|
||||
},
|
||||
"index": 0,
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
"data: [DONE]",
|
||||
# After [DONE] nothing must be parsed:
|
||||
_sse({"choices": [{"delta": {"content": "should not appear"}, "index": 0}]}),
|
||||
]
|
||||
out = parse_tool_response_streaming(lines)
|
||||
assert out["calls"] == [("get_time", "{}")]
|
||||
assert out["delta_chunks"] == 1
|
||||
assert out["finish_reason"] is None
|
||||
|
||||
|
||||
def test_parse_streaming_multiple_calls_by_index() -> None:
|
||||
lines = [
|
||||
_sse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {
|
||||
"tool_calls": [
|
||||
{
|
||||
"index": 1,
|
||||
"id": "b",
|
||||
"function": {"name": "second", "arguments": "{\"a\": "},
|
||||
},
|
||||
{
|
||||
"index": 0,
|
||||
"id": "a",
|
||||
"function": {"name": "first", "arguments": "{}"},
|
||||
},
|
||||
]
|
||||
},
|
||||
"index": 0,
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
_sse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"delta": {
|
||||
"tool_calls": [{"index": 1, "function": {"arguments": "1}"}}]
|
||||
},
|
||||
"index": 0,
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
]
|
||||
out = parse_tool_response_streaming(lines)
|
||||
assert out["calls"] == [("first", "{}"), ("second", '{"a": 1}')]
|
||||
assert out["delta_chunks"] == 3
|
||||
|
||||
|
||||
def _ns(ok: bool = True) -> dict:
|
||||
return {
|
||||
"finish_reason": "tool_calls" if ok else "stop",
|
||||
"calls": [("get_time", "{}")] if ok else [],
|
||||
}
|
||||
|
||||
|
||||
def _st(**overrides: object) -> dict:
|
||||
result: dict = {
|
||||
"finish_reason": "tool_calls",
|
||||
"calls": [("get_time", "{}")],
|
||||
"delta_chunks": 2,
|
||||
"indexed": True,
|
||||
"had_id": True,
|
||||
"arguments_in_deltas": True,
|
||||
}
|
||||
result.update(overrides)
|
||||
return result
|
||||
|
||||
|
||||
def test_classify_supported() -> None:
|
||||
assert classify_tool_calling(_ns(), _st()) == "supported"
|
||||
|
||||
|
||||
def test_classify_not_supported_variants() -> None:
|
||||
# Non-streaming did not call the tool.
|
||||
assert classify_tool_calling(_ns(False), _st()) == "not-supported"
|
||||
# Streaming did not call the tool.
|
||||
assert classify_tool_calling(_ns(), _st(finish_reason="stop", calls=[])) == "not-supported"
|
||||
# Streamed, but not as delta.tool_calls chunks.
|
||||
assert classify_tool_calling(_ns(), _st(delta_chunks=0)) == "not-supported"
|
||||
# Delta chunks without the OpenAI "index" field.
|
||||
assert classify_tool_calling(_ns(), _st(indexed=False)) == "not-supported"
|
||||
# Delta chunks without a call "id".
|
||||
assert classify_tool_calling(_ns(), _st(had_id=False)) == "not-supported"
|
||||
# Intermittent: non-stream ok, stream answered in content.
|
||||
assert classify_tool_calling(_ns(), _st(finish_reason="stop")) == "not-supported"
|
||||
@@ -16,6 +16,7 @@ from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag.prompts import (
|
||||
PERSONA,
|
||||
TOOLS_SECTION,
|
||||
_base,
|
||||
build_deflect_prompt,
|
||||
build_high_prompt,
|
||||
@@ -116,14 +117,18 @@ def test_low_prompt_with_no_titles() -> None:
|
||||
|
||||
def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
|
||||
"""Phase 15 contract: with no steering notes the prompt is exactly what
|
||||
it was before the <tuning> section existed."""
|
||||
it was before the <tuning> section existed. (Phase 37: the HIGH prompt
|
||||
additionally carries the ``<tools>`` section after the mode body — the
|
||||
fixtures account for it; the LOW prompt is untouched.)"""
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||
block = (
|
||||
'<document source="Homelab" path="kubernetes.md" title="Kubernetes Homelab Cluster">\n'
|
||||
"Talos Linux on three nodes.\n"
|
||||
"</document>"
|
||||
)
|
||||
assert build_high_prompt([doc]) == _base("HIGH") + "\n<documents>\n" + block + "\n</documents>"
|
||||
assert build_high_prompt([doc]) == (
|
||||
_base("HIGH") + "\n<documents>\n" + block + "\n</documents>" + "\n" + TOOLS_SECTION
|
||||
)
|
||||
assert build_deflect_prompt(["T1", "T2"]) == (
|
||||
_base("LOW")
|
||||
+ "\nDEFLECT_MODE: retrieval was weak — the titles below are the closest "
|
||||
@@ -220,14 +225,16 @@ def test_kb_section_tiny_budget_never_exceeds_cap() -> None:
|
||||
def test_no_overview_prompt_is_byte_identical_to_pre_phase() -> None:
|
||||
"""Phase 31 contract: with no KB overview (None, empty, or blank)
|
||||
every prompt is exactly what it was before the ``<knowledge_base>``
|
||||
section existed — with or without steering notes."""
|
||||
section existed — with or without steering notes. (Phase 37: the HIGH
|
||||
prompt additionally carries the ``<tools>`` section after the mode
|
||||
body — the fixtures account for it; the LOW prompt is untouched.)"""
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||
block = (
|
||||
'<document source="Homelab" path="kubernetes.md" title="Kubernetes Homelab Cluster">\n'
|
||||
"Talos Linux on three nodes.\n"
|
||||
"</document>"
|
||||
)
|
||||
docs_block = "\n<documents>\n" + block + "\n</documents>"
|
||||
docs_block = "\n<documents>\n" + block + "\n</documents>" + "\n" + TOOLS_SECTION
|
||||
high_plain = _base("HIGH") + docs_block
|
||||
high_steered = _base("HIGH") + "\n" + build_steering_section(["be concise"]) + docs_block
|
||||
low_plain = (
|
||||
|
||||
@@ -4,7 +4,7 @@ from __future__ import annotations
|
||||
import json
|
||||
|
||||
from app.api.chat import sse_event
|
||||
from app.schemas import ChatErrorEvent, ChatThinkingEvent
|
||||
from app.schemas import ChatErrorEvent, ChatThinkingEvent, ChatToolEvent
|
||||
|
||||
|
||||
def _payload(frame: str) -> dict:
|
||||
@@ -76,3 +76,27 @@ def test_thinking_event_shape_is_type_and_text_only() -> None:
|
||||
dumped = ChatThinkingEvent(text="hmm").model_dump()
|
||||
assert set(dumped.keys()) == {"type", "text"}
|
||||
assert dumped["type"] == "thinking" # default — call sites never spell it out
|
||||
|
||||
|
||||
def test_tool_frame_serializes_exactly() -> None:
|
||||
"""Phase 37 (PLAN §4 extension): the ``tool`` frame is exactly
|
||||
``{type: "tool", name: str, argument: str | null}`` — one per
|
||||
model-requested document tool call, streamed ahead of the ``delta``
|
||||
frames of the answer."""
|
||||
frame = sse_event(ChatToolEvent(name="read_document", argument="S/p.md").model_dump())
|
||||
assert frame == 'data: {"type": "tool", "name": "read_document", "argument": "S/p.md"}\n\n'
|
||||
assert _payload(frame) == {"type": "tool", "name": "read_document", "argument": "S/p.md"}
|
||||
|
||||
|
||||
def test_tool_frame_argument_is_null_for_parameterless_tools() -> None:
|
||||
"""``list_documents`` takes no parameters, so its frame's ``argument``
|
||||
serializes as JSON null (the client renders the name alone)."""
|
||||
dumped = ChatToolEvent(name="list_documents").model_dump()
|
||||
assert dumped == {"type": "tool", "name": "list_documents", "argument": None}
|
||||
assert _payload(sse_event(dumped))["argument"] is None
|
||||
|
||||
|
||||
def test_tool_event_shape_is_type_name_argument_only() -> None:
|
||||
dumped = ChatToolEvent(name="read_document", argument="S/p.md").model_dump()
|
||||
assert set(dumped.keys()) == {"type", "name", "argument"}
|
||||
assert dumped["type"] == "tool" # default — call sites never spell it out
|
||||
|
||||
Reference in New Issue
Block a user