feat(rag): agent document tools — list/read tools with env-tuned budgets, SSE tool events + "calling tool" UI

Grounded chat turns now run the agent loop (app/rag/agent.py) instead
of a bare chat_stream: while the per-turn budgets last
(BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each) the model
gets list_documents (the indexed catalog, /api/docs order) and
read_document (full text, never truncated — A7-revised contract); once
both budgets are spent the tools key is dropped from the request and
the model must answer. Rejected calls (unknown tool, unknown/missing
path, document already in context, spent budget) consume no budget.
Budgets 0/0 make exactly one tools=None request — byte-identical to
the pre-phase path (budgets-as-kill-switch). Deflected turns keep the
direct chat_stream (A8 unchanged; the LOW prompt never carries the
<tools> section).

SSE contract gains {"type":"tool","name":...,"argument":
"source/path"|null} frames ahead of the answer deltas (PLAN §4
extension, owner permission 2026-08-26); done.sources, query_log.sources
and the per-turn log line (gains tool_calls=N) report the retrieval
docs + read docs, deduped. The UI shows a "calling tool"
button/label state and one visible .tool-call line per call above the
answer; the lines persist with the chat record and re-render on
reload. chat_stream passes tools through and accumulates streaming
tool_calls deltas into ToolCallPiece (tools=None stays byte-identical).

E2E: deterministic mock tool flow ("use your tools" + <tools> marker:
list -> read first catalog line -> quoted answer) plus the story suite
(marker flow, reload re-render, plain/deflected no-tool regressions).
Docs: .env.example + README (the two tools, the budgets, the SSE tool
frame, the "calling tool" UI state).

probe: turbo tool_calls=supported 2026-08-26 (uv run python -m
scripts.llm_probe --tools — non-streaming + streaming
finish_reason=tool_calls, indexed delta.tool_calls partials)

.agent/phases/complete/37_agent_document_tools/01_probe_tool_calling.md

@@ -0,0 +1,24 @@
+# Task 01 — Live tool-calling probe
+
+**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "The agent should be able to list the available sources as a tool and the read the ones it thinks are relevant"`
+**Story:** `.agent/user_stories/agent-document-tools.md`
+
+## Objective
+Verify against the live aipi endpoint whether the `turbo` chat model supports OpenAI-style `tools` + streaming `tool_calls` before building the loop — the phase-17 "verified live" convention — and record which path the phase takes.
+
+## Work
+1. `scripts/llm_probe.py` — add a `--tools` flag: when set, send (a) a non-streaming `chat/completions` request with one trivial function (e.g. `get_time`, no parameters) and a user message that makes calling it natural, and (b) the same with `stream=True`; for each, print whether `finish_reason` is `tool_calls`, the parsed `function.name`/`arguments`, and (streaming) whether the calls arrive as `delta.tool_calls` chunks with `index`/`id`/partial `function.arguments` (the OpenAI wire convention). Reuse the existing env reading (`BOR_LLM_BASE_URL` / API key / `BOR_LLM_CHAT_MODEL`) and the script's existing output style.
+2. Run it against the live endpoint (`uv run python -m scripts.llm_probe --tools`) and classify the verdict:
+   - **supported** → the phase uses OpenAI `tools`/`tool_calls` (tasks 02–03 as written).
+   - **not supported** → the phase uses the prompt-based structured-call fallback (task 03 documents it): the model is instructed to emit a single JSON block (`{"tool": "list_documents"}` / `{"tool": "read_document", "source": …, "path": …}`) before answering; `run_agent` parses it out of the content stream; the SSE `tool` contract and the budgets are identical.
+   - **intermittent** → treat as not supported (fail-loud house style) and note it.
+3. Record the verdict + date where it will be read later: the task-03 `app/rag/agent.py` module docstring (task 03 writes it) and the phase commit message (task 06) — e.g. `probe: turbo tool_calls=streaming-ok 2026-08-26`.
+
+## Testing & Quality
+- The probe is a CLI script (no `app/` coverage impact). If the parsing of the probe response is factored into a function, add a small unit test for it.
+- `uv run pytest` green (no regressions); ruff + pyright clean.
+
+## Completion Criteria
+- [ ] `uv run python -m scripts.llm_probe --tools` runs and prints a clear supported / not-supported verdict for both the non-streaming and the streaming request.
+- [ ] The verdict (with date) is available for task 03's docstring and task 06's commit message.
+- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.

.agent/phases/complete/37_agent_document_tools/02_llm_client_tools.md

@@ -0,0 +1,36 @@
+# Task 02 — LLM client: `tools` + tool-call streaming
+
+**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "…read the ones it thinks are relevant… These values should be configured by environment variables"` (the client plumbing the env-tuned loop runs on)
+**Story:** `.agent/user_stories/agent-document-tools.md`
+
+## Objective
+Teach `LLMClient.chat_stream` to pass an OpenAI `tools` list and to accumulate streaming `tool_calls` deltas into typed pieces — with `tools=None` producing a byte-identical request to today.
+
+## Work
+1. `app/rag/llm.py`
+   - New frozen dataclass next to `StreamPiece`:
+     ```python
+     @dataclass(frozen=True)
+     class ToolCallPiece:
+         """One model-requested tool call accumulated from stream deltas (phase 37)."""
+         id: str  # the model's tool_call id; synthesized "call_<index>" when absent
+         name: str  # "list_documents" | "read_document" (whatever AGENT_TOOLS names)
+         arguments: dict[str, Any]
+     ```
+   - `chat_stream(self, messages, tools: list[dict[str, Any]] | None = None)`:
+     - When *tools* is not None, pass `tools=tools` to `chat.completions.create`; when None, do **not** include the key (byte-identical request to today).
+     - In the chunk loop, accumulate `delta.tool_calls` (a list of partials keyed by `index`; `id` and `function.name` arrive on the first partial for an index, `function.arguments` arrives in fragments to concatenate).
+     - At stream end (or when a chunk carries `finish_reason == "tool_calls"`), for each accumulated call **in index order** yield `ToolCallPiece(id, name, json.loads(arguments) or {})`.
+     - Malformed `arguments` JSON → raise `LLMError` (fail-loud house style — a silently dropped tool call would corrupt the loop).
+   - Return annotation becomes `AsyncIterator[StreamPiece | ToolCallPiece]`; update the docstring (wire convention + a pointer to the task-01 probe verdict).
+   - **If the task-01 verdict is "not supported"** (prompt-based fallback): skip the `tools` parameter and the delta accumulation entirely; keep `StreamPiece` unchanged; the JSON-block parse helper lands in `app/rag/agent.py` (task 03) instead.
+2. `app/api/chat.py` — update the phase-17 typing import line (`StreamPiece  # noqa: F401`) to also import `ToolCallPiece` so pyright sees the union; the dispatch wiring itself is task 04.
+
+## Testing & Quality
+- Unit (`tests/unit/` — extend the existing `chat_stream` tests): synthetic chunk sequences — (a) a content-only stream is unchanged (no `ToolCallPiece`; the captured `create()` kwargs have no `tools` key); (b) a tool-call stream with partials across chunks (name on the first, arguments in 2–3 fragments) → one `ToolCallPiece` with the merged JSON; (c) two calls in one stream (indices 0 and 1) → both, in index order; (d) malformed arguments JSON → `LLMError`; (e) `tools=[…]` present in the request when passed.
+- Coverage: **>90%** on the modified module.
+
+## Completion Criteria
+- [ ] `chat_stream(messages)` (no tools) — all existing unit tests green unchanged.
+- [ ] The new tool-call accumulation tests green; full `uv run pytest` green.
+- [ ] `uv run ruff check . && uv run pyright` clean.

.agent/phases/complete/37_agent_document_tools/03_agent_loop.md

@@ -0,0 +1,50 @@
+# Task 03 — The basic agent loop (`app/rag/agent.py`)
+
+**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "it gets one opportunity to list documents and then one opportunity to add exactly one extra document to its context before being required to answer. These values should be configured by environment variables."`
+**Story:** `.agent/user_stories/agent-document-tools.md`
+
+## Objective
+A testable agent loop: the model gets the two tools while budgets last, the app executes them against Postgres, and once both budgets are spent the tools are dropped so the model is required to answer.
+
+## Work
+1. `app/config.py` — two settings (documented, `BOR_` prefix per house style, near the RAG-tuning block):
+   ```python
+   #: Per-turn opportunities to call the `list_documents` agent tool (phase 37);
+   #: 0 disables the tool entirely (pre-phase behavior with both at 0).
+   agent_list_calls: int = 1
+   #: Per-turn opportunities to call `read_document` (phase 37); 0 disables.
+   agent_read_calls: int = 1
+   ```
+2. `app/rag/agent.py` — new module. The module docstring carries: the phase, the loop contract, and the task-01 probe verdict + date (which path is in use — OpenAI tool_calls vs the prompt-based fallback).
+   - `AGENT_TOOLS`: the two OpenAI function definitions — `list_documents` (no parameters; description: "List every document indexed in the knowledge base, one `source/path — title` line each") and `read_document` (`source` + `path` required; description: "Add the full content of exactly one more indexed document to your context").
+   - DB accessors (module-level functions so unit tests can monkeypatch them):
+     - `list_catalog(db) -> list[tuple[str, str, str]]` — `select(Document.source, Document.path, Document.title).order_by(Document.source, Document.path)` (same order as `GET /api/docs`).
+     - `find_document(db, source, path) -> Document | None`.
+   - `@dataclass AgentHolder: read_docs: list[Document] = field(default_factory=list); tool_calls: int = 0` — the API layer (task 04) reads it after the stream.
+   - `async def run_agent(llm, db, *, system_prompt, user_message, seed_docs, settings, holder) -> AsyncIterator[StreamPiece | ToolCallPiece]`:
+     1. `messages = [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_message}]`; `list_left = settings.agent_list_calls`; `read_left = settings.agent_read_calls`; `tools = AGENT_TOOLS if (list_left or read_left) else None`; `rounds = 0`; cap `max_rounds = 2 + settings.agent_list_calls + settings.agent_read_calls` (belt-and-braces — every tool round must consume a budget, so the cap only catches pathological streams).
+     2. Loop: stream `llm.chat_stream(messages, tools=tools)`; yield every piece as it arrives. Collect any `ToolCallPiece` (handle the first; if a stream yields both content and a tool call — rare — the content stays (it was already emitted) and the tool still runs).
+        - **No tool call** → return (the answer was streamed).
+        - **`list_documents`**: `list_left > 0` → `list_left -= 1`, `holder.tool_calls += 1`, result = `f"{n} documents:\n" + "\n".join(f"{s}/{p} — {t}")` (uncapped in v1 — **ASSUMPTION: the catalog is not truncated; the UI never shows it, only the model does**). Else result = `"No listing budget left — answer with what you have."` (no budget consumed).
+        - **`read_document`**: arguments must carry `source` and `path`. If that document is already in `seed_docs` or `holder.read_docs` → result = `"Already in your context."` (no budget consumed, no append). Else if `read_left > 0` → `doc = find_document(db, source, path)`; found → `read_left -= 1`, `holder.tool_calls += 1`, `holder.read_docs.append(doc)`, result = `f"Document {source}/{path}:\n{doc.content}"` (**full text, never truncated — the A7-revised contract**); not found → result = `f"No document at {source}/{path} — check the list_documents output."` (no budget consumed). `read_left == 0` → result = `"No reading budget left — answer with what you have."`.
+        - **Unknown tool name** → result = `"Unknown tool."` (no budget consumed).
+        - Append to `messages`: the assistant tool-call message (`{"role": "assistant", "content": None, "tool_calls": [{"id": tc.id, "type": "function", "function": {"name": tc.name, "arguments": json.dumps(tc.arguments)}}]}`) + the tool result (`{"role": "tool", "tool_call_id": tc.id, "content": result}`); then `tools = None if (list_left == 0 and read_left == 0) else AGENT_TOOLS` (once both budgets are spent, the next request must be answered).
+        - `rounds += 1`; if `rounds >= max_rounds` → force one final `chat_stream(messages, tools=None)` (yield its pieces) and return.
+   - **Prompt-based fallback (only if the task-01 verdict is "not supported")**: instead of `ToolCallPiece`s, scan each streamed content turn for a leading JSON block matching `{"tool": …}` (strip leading whitespace, parse with `json`; the block must be the first non-whitespace content of the turn). On a match: strip the block from the emitted stream (re-emit only trailing content, if any), then run the identical budget/result machinery. The SSE `tool` events and the budgets are unchanged either way.
+3. `app/rag/prompts.py` — add the `<tools>` instructions section to the **HIGH prompt only** (a new constant, appended after the mode body; the LOW/deflection prompt stays byte-identical). Wording (tune against the live model if needed, but keep it stable — the E2E mock keys off the `<tools>` marker's *presence*, not the wording):
+   > If the documents in your context reference other files, or you need content that is not included above, call `list_documents` to see what is indexed, then `read_document` to pull in exactly one more document. Answer as soon as you have what you need — do not read more than one extra document.
+
+## Testing & Quality
+- Unit (`tests/unit/test_agent.py`, new): a scripted fake LLM (canned stream sequences) + monkeypatched `list_catalog`/`find_document` + a `Settings` with budgets set:
+  - list → read → answer: event order (tool pieces before content), `holder.read_docs == [doc]`, `holder.tool_calls == 2`, the request after budgets are spent carries `tools=None`, the follow-up request contains the assistant tool-call + tool-result messages.
+  - Budgets: `read_left` exhausted → a second `read_document` gets "No reading budget left" and appends nothing; `agent_list_calls=0` + `agent_read_calls=0` → exactly one request with `tools=None` (byte-identical single-call path).
+  - Edge: reading a doc already in `seed_docs` → "Already in your context." (no budget consumed); unknown path → "No document at …"; unknown tool name → "Unknown tool."; the round cap forces a final no-tools answer.
+  - The HIGH prompt gains the `<tools>` section; the LOW prompt is byte-identical to pre-phase (assert against the existing prompt-test fixtures).
+- Integration (`tests/integration/`): `list_catalog` ordering + `find_document` hit/None against real Postgres (follow the existing DB-test patterns).
+- Coverage: **>90%** on the new module.
+
+## Completion Criteria
+- [ ] `app/rag/agent.py` exists with the loop above; the module docstring records the probe verdict + date (task 01).
+- [ ] The budget matrix (0/1) unit tests green, including "budgets 0/0 ⇒ one request, no tools".
+- [ ] HIGH prompt carries `<tools>`; LOW prompt byte-identical to pre-phase.
+- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.

.agent/phases/complete/37_agent_document_tools/04_api_sse_tool_event.md

@@ -0,0 +1,36 @@
+# Task 04 — `POST /api/chat`: `tool` SSE events + agent wiring
+
+**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "This will require some reconfiguring of the UI since it will now need to show 'calling tool' in addition to 'thinking' and it will need a basic agent loop."`
+**Story:** `.agent/user_stories/agent-document-tools.md`
+
+## Objective
+Wire the loop into the chat endpoint: the new SSE `tool` event, the agent on grounded turns only, and the read document reflected in `done.sources`, `query_log`, and the per-turn log line.
+
+## Work
+1. `app/schemas.py` — add next to `ChatThinkingEvent`:
+   ```python
+   class ChatToolEvent(BaseModel):
+       """SSE frame for one agent tool call (phase 37, PLAN §4 extension)."""
+       type: Literal["tool"] = "tool"
+       name: str  # "list_documents" | "read_document"
+       argument: str | None = None  # "source/path" for read_document
+   ```
+2. `app/api/chat.py`
+   - In `stream()`, after `plan_turn`: when `not plan.deflected` → run `run_agent(llm, db, system_prompt=plan.system_prompt, user_message=request.message, seed_docs=plan.docs, settings=settings, holder=holder)`; when deflected → keep the current direct `chat_stream` (A8 byte-identical).
+   - Event mapping: `StreamPiece` thinking/delta exactly as today (including the `BOR_STREAM_THINKING` kill-switch and the `thinking_chars` count); `ToolCallPiece` →
+     `yield sse_event(ChatToolEvent(name=tc.name, argument=f"{tc.arguments.get('source')}/{tc.arguments.get('path')}" if tc.name == "read_document" else None).model_dump())`.
+   - `done` event: `sources` = `plan.docs + holder.read_docs` deduped by `(source, path)`, order preserved; the `ChatDoneEvent` shape otherwise unchanged.
+   - `query_log.sources`: the same combined list (replaces the current `source_paths` build).
+   - Per-turn log line: add `tool_calls={holder.tool_calls}` after `thinking_chars=` (PLAN §9 required-line extension — the phase 17/30/31 precedent).
+   - Extend the module docstring with the phase-37 section (the flow, the grounded-only scope, the budgets, the "both budgets 0 ⇒ pre-phase behavior" note).
+3. `.agent/PLAN.md` — record two revision notes in the project's established format (phase 17/24 precedent, "owner permission 2026-08-26"): the §4 SSE contract gains `{"type":"tool","name":…,"argument":…}` (client rule: render as a "calling tool" line/state; `delta`/`done` unchanged) and the §9 per-turn log line gains `tool_calls=N`.
+
+## Testing & Quality
+- Integration (`tests/integration/test_chat*.py` — extend the existing SSE-contract suite, mock LLM that emits tool calls): (a) grounded turn with tool calls → event sequence `thinking?/tool/tool/delta…/done`, `done.sources` includes the read doc, the `query_log` row's sources match, the log line carries `tool_calls=2`; (b) deflected turn → no `tool` events, the sequence byte-identical to today; (c) budgets 0/0 → no `tool` events, the mock received no `tools` parameter, single-request path.
+- Coverage: **>90%** on the modified modules.
+
+## Completion Criteria
+- [ ] `POST /api/chat` (mock LLM, grounded) streams `tool` frames ahead of the `delta` frames; the deflected path is unchanged (existing suites green).
+- [ ] `done.sources`/`query_log` include the read document (deduped); the log line carries `tool_calls=N`.
+- [ ] The PLAN.md §4 + §9 revision notes exist (dated, owner permission 2026-08-26).
+- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.

.agent/phases/complete/37_agent_document_tools/05_frontend_tool_states.md

@@ -0,0 +1,27 @@
+# Task 05 — UI: the "calling tool" state
+
+**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "…it will now need to show 'calling tool' in addition to 'thinking'"`
+**Story:** `.agent/user_stories/agent-document-tools.md`
+
+## Objective
+The chat shell shows a "calling tool" state (button + status label) and a visible tool line in the bubble, distinct from the Thinking scratchpad; tool lines persist with the chat record (phase 14) and re-render after reload.
+
+## Work
+1. `frontend/assets/app.js` — in the turn handler where the SSE frames are dispatched (the same switch that handles `thinking` and `delta`):
+   - New `tool` branch:
+     - **Status/label:** keep `uiState = thinking` (the button stays disabled — never stale, PLAN §7.4) and set the typing-indicator label to "Brain of Reese is listing documents" (`name: "list_documents"`) / "Brain of Reese is reading <argument>" (`name: "read_document"`); the elapsed-seconds hint (`thinkingClock` aria-label) keeps working through tool frames.
+     - **Bubble line:** append a tool line to the same wrap the thinking block uses (above the answer, beside/below the Thinking summary): `<span class="tool-call">🔎 Listing documents</span>` / `<span class="tool-call">📄 Reading <code>source/path</code></span>` — a real visible line with its own icon + color, distinct from the Thinking block (styles step). Multiple calls append multiple lines, in order.
+     - Tolerate tool frames interleaved with thinking frames (append-only, the same rule as thinking); a tool frame after the first `delta` (should not happen in v1 — the loop completes before the answer stream) still appends rather than crashes.
+   - **Persistence (phase 14 convention):** the saved record gains an optional `tools: [{name, argument}]` array next to `thinking`; re-render the tool lines when a record is loaded (same code path as the thinking re-render).
+   - Keep the new label strings as plain literals — phase 39 centralizes brand strings; do **not** introduce a brand helper here.
+2. `frontend/assets/styles.css` — `.tool-call` style: inline row, icon + text, an accent color distinguishable from the thinking block's, `code` styling for the path, contrast ≥ 4.5:1 in both themes; no layout shift on append (the centered 46rem chat column is untouched — no new container).
+3. UI Structure Check (AGENTS.md rule 5) before finalizing: semantic landmarks unchanged; the tool lines live inside the `#messages` `aria-live="polite"` region (announced to screen readers); focus-visible unaffected (the lines are not interactive); no new top-level landmarks.
+
+## Testing & Quality
+- No new Python logic — gated by the story E2E (task 06) plus the existing frontend-adjacent suites staying green.
+- No CDN (AGENTS.md rule 6): no new `<script>`/`<link>` tags.
+
+## Completion Criteria
+- [ ] During a tool call the button label shows the "calling tool" text (not a stale "Thinking…") and the tool lines render above the answer.
+- [ ] After a page reload, the persisted record re-renders the tool line(s).
+- [ ] `uv run pytest` green (no regressions); `uv run ruff check . && uv run pyright` clean.