Phase 68 — Agent Search Tool: grep the Indexed Documents

Source: TODO.md L4 — "Add a search tool that allows the LLM to grep through the uploaded documents for a given string" Story: n/a (TODO-derived — owner roadmap confirmation 2026-09-01) Context:

app/rag/agent.py — AGENT_TOOLS currently defines two OpenAI functions (list_documents, read_document); _execute_tool runs them server-side against the DB with fixed refusal strings (ALREADY_IN_CONTEXT, UNKNOWN_TOOL, MISSING_READ_ARGS); AgentHolder counts executed calls (tool_calls) and context additions (read_docs).
documents table (app/models.py) — content is the FULL document text (Text column), so a grep is a plain in-process scan: no new index, no migration.
app/schemas.py — ChatToolEvent ({type: "tool", name, argument}): argument is "source/path" for read_document, null otherwise.
frontend/assets/app.js — runTurn's tool branch builds the status label (${brand()} is reading … / … is listing documents) and appendToolLine (L796) renders the per-call line (📄 Reading <code>path</code> / 🔎 Listing documents); both special-case by tool name.
tests/e2e/mock_llm.py — the marker-driven deterministic tool flow (use your tools → list → read → answer) is the template for a search flow; tests/e2e/test_agent_document_tools.py is the pattern for the E2E assertions (.tool-call lines, #send-status recording).
Phases 37/45/63 (complete) established the tool infrastructure: native tool-calling, unlimited calls bounded by the round cap, labeled source:/path: catalog lines.

Objective

A third agent tool, search_documents, lets the model grep every indexed document (or one named document) for an exact string and get back path:line: text matches — so it can LOCATE content cheaply and then read_document the winner, instead of reading whole documents hoping the string is in them.

Dependencies

67_llm_retry (todo, preceding — no functional dependency; ordering by number)

Tasks

01_search_tool_backend.md — the tool definition, the grep_document helper, and the _execute_tool branch.
02_search_tool_api_ui.md — the SSE tool argument mapping and the frontend status/tool-line for the search.
03_e2e_and_commit.md — the mock search flow, the dedicated E2E suite, regressions, commit.

Testing & Quality

Unit: tests/unit/test_agent.py — match semantics (case-insensitivity, 1-based line numbers, the 20-match cap, 200-char line truncation, scoped single-doc search, no-match/missing-arg/unknown-doc refusals, tool_calls counting, read_docs untouched).
Integration: tests/integration/test_agent_tools.py — the tool appears in AGENT_TOOLS with the locked parameter shape; tests/integration/test_chat_api.py (or the SSE pin file) — a search_documents call streams argument = pattern.
E2E (mandatory, house rule): tests/e2e/test_search_tool.py, run in isolation (deterministic mock flow: the model searches, sees the match line, answers from it).
Coverage: >90% on app/ (validate.sh gate).

Completion Criteria

search_documents is the third entry in AGENT_TOOLS; a model call with pattern (optionally source+path) returns grep-style matches or a no-match line.
The UI shows Brain is searching for '…' in the status line and a 🔎 Searching for '<pattern>' tool line, persisted/restored like the other tool lines.
uv run pytest green; coverage TOTAL >90%; uv run ruff check . && uv run pyright clean.
uv run pytest tests/e2e/test_search_tool.py -v --no-cov green in isolation (DB up).
Regression E2E suites green in isolation: test_agent_document_tools.py, test_agent_unlimited_tools.py.
One --no-gpg-sign commit; phase dir moved to .agent/phases/complete/.

Locked decisions

Owner-locked (2026-09-01, roadmap confirmation, A5): the match is a case-insensitive fixed substring (no regex — no ReDoS surface, a simple contract for the model); output is grep-style source/path:LINE: text lines; 20 matches per call maximum (global cap across documents, in catalog order), each line truncated to 200 chars; a search does not add the document to the answer context (read_document remains the only context-adder — holder.read_docs is untouched by a search).
Owner-locked (2026-09-01, roadmap confirmation, A6): the tool name is search_documents (alongside list_documents / read_document).
Scope: search is offered on grounded (HIGH) turns only, exactly like the existing tools — deflected turns keep tools=None (A8 byte-identical deflection path), and BOR_AGENT_MAX_ROUNDS=0 stays the no-tools kill switch (phase 45).

Commit

git add -A .agent/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): search_documents tool — the model can grep the indexed documents for an exact string"

4.9 KiB Raw Blame History