feat(agent): search_documents tool — the model can grep the indexed documents for an exact string

app/api/chat.py

@@ -49,25 +49,30 @@ outline (0 when absent) and the per-turn log line records
 
 Agent document tools (phase 37, PLAN §4 extension, owner permission
 2026-08-26; phase 45 removed the per-tool budgets — owner permission
-2026-08-27): a **grounded** turn (``not plan.deflected``) no longer
-streams a bare ``chat_stream`` — it runs the agent loop
-(``app.rag.agent.run_agent``), which offers the model the two
-server-side tools ``list_documents`` / ``read_document`` for the whole
-turn (as many calls as the model wants, re-lists included) until it
-answers or the round cap (``BOR_AGENT_MAX_ROUNDS``, default 10) forces
-one final no-tools answer. Each model-requested call streams as an SSE
-``tool`` event — ``{"type": "tool", "name": …, "argument":
-"source/path" | null}`` — ahead of the answer's ``delta`` frames.
-``done.sources``, ``query_log.sources`` and the per-turn log line all
-report the same combined source list (retrieval docs + the agent's
-read docs, deduped by ``(source, path)``, order preserved), and the log
-line records ``tool_calls=N`` after ``thinking_chars=N`` (PLAN §9 line
-extension — ``N`` counts executed tool calls; rejected calls do not
-count). **Deflected turns keep the direct ``chat_stream`` —
-byte-identical to the pre-phase path (A8):** the LOW prompt never
-carries tools, and with ``agent_max_rounds`` at **0** ``run_agent``
-makes exactly one ``tools=None`` request, reproducing the pre-phase
-behavior (the kill switch).
+2026-08-27; phase 68 added the ``search_documents`` grep): a
+**grounded** turn (``not plan.deflected``) no longer streams a bare
+``chat_stream`` — it runs the agent loop (``app.rag.agent.run_agent``),
+which offers the model the three server-side tools
+``list_documents`` / ``read_document`` / ``search_documents`` for the
+whole turn (as many calls as the model wants, re-lists and re-searches
+included) until it answers or the round cap (``BOR_AGENT_MAX_ROUNDS``,
+default 10) forces one final no-tools answer. Each model-requested call
+streams as an SSE ``tool`` event — ``{"type": "tool", "name": …,
+"argument": "source/path" | pattern | null}`` — ahead of the answer's
+``delta`` frames: ``argument`` is the read document's path for
+``read_document``, the raw search pattern for ``search_documents``
+(a non-string pattern — a model error the backend refuses — yields
+null), and null for ``list_documents``. ``done.sources``,
+``query_log.sources`` and the per-turn log line all report the same
+combined source list (retrieval docs + the agent's read docs, deduped
+by ``(source, path)``, order preserved — a search adds no source; it is
+a locator, locked A5), and the log line records ``tool_calls=N`` after
+``thinking_chars=N`` (PLAN §9 line extension — ``N`` counts executed
+tool calls; rejected calls do not count). **Deflected turns keep the
+direct ``chat_stream`` — byte-identical to the pre-phase path (A8):**
+the LOW prompt never carries tools, and with ``agent_max_rounds`` at
+**0** ``run_agent`` makes exactly one ``tools=None`` request,
+reproducing the pre-phase behavior (the kill switch).
 
 LLM retries (phase 67, ``BOR_LLM_RETRIES`` / ``BOR_LLM_RETRY_DELAY``,
 owner-locked 2026-09-01): when the aipi endpoint dies before a request
@@ -396,18 +401,22 @@ async def chat(
                 async for piece in answer_stream:  # StreamPiece | ToolCallPiece | RetryPiece
                     if isinstance(piece, ToolCallPiece):
                         # Phase 37 (PLAN §4 extension): one SSE ``tool``
-                        # frame per model-requested call; ``argument`` is
-                        # the read_document "source/path" (null
-                        # otherwise).
+                        # frame per model-requested call. ``argument`` is
+                        # the read_document "source/path"; phase 68
+                        # extends it with the search_documents pattern
+                        # (a non-string pattern — a model error the
+                        # backend refuses — is null); null otherwise.
+                        if piece.name == "read_document":
+                            argument = (
+                                f"{piece.arguments.get('source')}/{piece.arguments.get('path')}"
+                            )
+                        elif piece.name == "search_documents":
+                            pattern = piece.arguments.get("pattern")
+                            argument = pattern if isinstance(pattern, str) else None
+                        else:
+                            argument = None
                         yield sse_event(
-                            ChatToolEvent(
-                                name=piece.name,
-                                argument=(
-                                    f"{piece.arguments.get('source')}/{piece.arguments.get('path')}"
-                                    if piece.name == "read_document"
-                                    else None
-                                ),
-                            ).model_dump()
+                            ChatToolEvent(name=piece.name, argument=argument).model_dump()
                         )
                         continue
                     if isinstance(piece, RetryPiece):

app/rag/agent.py

@@ -15,27 +15,38 @@ probe came back "supported".
 Loop contract (one grounded chat turn; the API layer wires this in,
 task 04):
 
-1. The model is offered the two OpenAI functions in :data:`AGENT_TOOLS`
+1. The model is offered the three OpenAI functions in :data:`AGENT_TOOLS`
    for the whole turn — phase 45 removed the phase-37 per-tool budgets
    (owner permission 2026-08-27, ``TODO.md`` L8: "allow the LLM to make
-   as many tool calls as it wants"): ``list_documents`` and
-   ``read_document`` can each be called as many times as the model needs,
-   re-lists included. With ``settings.agent_max_rounds``
-   (``BOR_AGENT_MAX_ROUNDS``, default 10) at 0 the loop makes exactly one
-   request with ``tools=None`` — byte-identical to the pre-phase-37 chat
-   path (the kill switch).
+   as many tool calls as it wants"): ``list_documents``,
+   ``read_document`` and ``search_documents`` can each be called as many
+   times as the model needs, re-lists and re-searches included. With
+   ``settings.agent_max_rounds`` (``BOR_AGENT_MAX_ROUNDS``, default 10)
+   at 0 the loop makes exactly one request with ``tools=None`` —
+   byte-identical to the pre-phase-37 chat path (the kill switch).
 2. Each tool call the model emits is executed server-side against
    Postgres only (no LLM, no network): ``list_documents`` returns the
    indexed catalog — one ``source: X | path: Y | title: Z`` line per
    document (phase 63: labeled fields — unambiguous for LLM parsing),
    ``GET /api/docs`` order (uncapped in v1; the UI never shows it, only
-   the model does) — and ``read_document`` returns the document's **full**
-   content (A7-revised contract: never truncated).
+   the model does) — ``read_document`` returns the document's **full**
+   content (A7-revised contract: never truncated) — and
+   ``search_documents`` greps the indexed documents (or one named
+   document) for a case-insensitive fixed substring and returns up to 20
+   ``source/path:line: text`` match lines (owner-locked A5, phase 68),
+   each line truncated to 200 chars. A search is a **locator**, not a
+   context-adder: it never appends to the answer context (only
+   ``read_document`` does — ``holder.read_docs`` is untouched by a
+   search).
 3. Rejected calls get a one-line refusal and count in nothing
    (``holder.tool_calls`` tracks executed calls only): unknown tool name
-   → ``"Unknown tool."``; missing ``source``/``path`` arguments; a
-   document already in context (seed or previously read) → ``"Already in
-   your context."``; an unknown ``source/path`` → ``"No document at …"``.
+   → ``"Unknown tool."``; missing ``source``/``path`` arguments; a search
+   without a usable ``pattern`` (missing, blank or non-string) or with a
+   half-specified ``source``/``path`` target; a document already in
+   context (seed or previously read) → ``"Already in your
+   context."``; an unknown ``source/path`` (read or scoped search) →
+   ``"No document at …"``. A search that ran but found nothing is NOT a
+   rejection — its ``"No matches for …"`` line is a (counted) result.
    A rejected call still consumes a *round* in the loop, so a
    pathological stream that keeps emitting rejected calls is bounded by
    the cap (point 4).
@@ -65,7 +76,8 @@ task 04):
    exactly one round. With ``settings.llm_retries=0`` every request is a
    single plain attempt (the pre-phase-67 path).
 
-The DB accessors (:func:`list_catalog`, :func:`find_document`) are
+The DB accessors (:func:`list_catalog`, :func:`find_document`,
+:func:`all_documents`) and the :func:`grep_document` line matcher are
 module-level functions so unit tests can monkeypatch them without a
 database.
 """
@@ -92,10 +104,11 @@ from app.rag.llm import (
 
 logger = logging.getLogger("app.agent")
 
-#: The two agent tools (phase 37): OpenAI function definitions passed as
-#: ``tools=AGENT_TOOLS`` to ``chat_stream`` for the whole grounded turn —
-#: phase 45 removed the per-tool budgets; the round cap
-#: (``BOR_AGENT_MAX_ROUNDS``) is the only bound.
+#: The three agent tools (phase 37; ``search_documents`` added in phase
+#: 68): OpenAI function definitions passed as ``tools=AGENT_TOOLS`` to
+#: ``chat_stream`` for the whole grounded turn — phase 45 removed the
+#: per-tool budgets; the round cap (``BOR_AGENT_MAX_ROUNDS``) is the only
+#: bound.
 AGENT_TOOLS: list[dict[str, Any]] = [
     {
         "type": "function",
@@ -140,6 +153,49 @@ AGENT_TOOLS: list[dict[str, Any]] = [
             },
         },
     },
+    {
+        "type": "function",
+        "function": {
+            "name": "search_documents",
+            "description": (
+                "Search every indexed document for an exact string "
+                "(case-insensitive) and return up to 20 matching lines as "
+                "'source/path:line: text' — use this to locate content, "
+                "then read_document the winner. Optionally pass 'source' "
+                "and 'path' (as shown in list_documents) to search one "
+                "document only."
+            ),
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "pattern": {
+                        "type": "string",
+                        "description": (
+                            "The exact text to search for (a plain "
+                            "substring, not a regex)"
+                        ),
+                    },
+                    "source": {
+                        "type": "string",
+                        "description": (
+                            "The document's source, as shown after 'source: ' in the "
+                            "list_documents output (e.g. 'Homelab' from "
+                            "'source: Homelab | path: homelab/aws-route53.md')."
+                        ),
+                    },
+                    "path": {
+                        "type": "string",
+                        "description": (
+                            "The document's path, as shown after 'path: ' in the "
+                            "list_documents output (e.g. 'homelab/aws-route53.md' from "
+                            "'source: Homelab | path: homelab/aws-route53.md')."
+                        ),
+                    },
+                },
+                "required": ["pattern"],
+            },
+        },
+    },
 ]
 
 #: Tool refusal texts (phase 37): rejected calls count in nothing
@@ -148,6 +204,19 @@ AGENT_TOOLS: list[dict[str, Any]] = [
 ALREADY_IN_CONTEXT = "Already in your context."
 UNKNOWN_TOOL = "Unknown tool."
 MISSING_READ_ARGS = "read_document requires string arguments 'source' and 'path'."
+MISSING_SEARCH_ARGS = "search_documents requires a string argument 'pattern'."
+
+#: Search caps (owner-locked A5, phase 68): a global per-call match cap
+#: (across documents, in catalog order) and a per-match-line char limit.
+SEARCH_MAX_MATCHES = 20
+SEARCH_LINE_LIMIT = 200
+
+#: No-match result lines (templates — the pattern is truncated to 100
+#: chars before formatting, to keep a long pattern from bloating the
+#: tool result). A no-match line is a *result* of an executed search,
+#: not a refusal (see the module docstring, point 3).
+NO_MATCHES = "No matches for '{pattern}' in the knowledge base."
+NO_MATCHES_SCOPED = "No matches for '{pattern}' in {source}/{path}."
 
 
 def list_catalog(db: Session) -> list[tuple[str, str, str]]:
@@ -174,6 +243,37 @@ def find_document(db: Session, source: str, path: str) -> Document | None:
     )
 
 
+def all_documents(db: Session) -> list[Document]:
+    """Every indexed document (full rows), ordered by ``(source, path)``
+    — catalog order.
+
+    The whole-KB ``search_documents`` path loads all contents in this one
+    bulk query (catalog order is the locked match order, owner-locked A5).
+    Module-level (not a method) so unit tests can monkeypatch it.
+    """
+    return list(
+        db.execute(
+            select(Document).order_by(Document.source, Document.path)
+        ).scalars()
+    )
+
+
+def grep_document(content: str, pattern: str) -> list[tuple[int, str]]:
+    """Every line of *content* that contains *pattern*, in file order.
+
+    Case-insensitive **fixed substring** (owner-locked A5: no regex — no
+    ReDoS surface, a simple contract for the model). Returns
+    ``(1-based line number, line.rstrip())`` pairs; an empty *content*
+    never matches a non-empty pattern.
+    """
+    needle = pattern.lower()
+    return [
+        (number, line.rstrip())
+        for number, line in enumerate(content.split("\n"), start=1)
+        if needle in line.lower()
+    ]
+
+
 @dataclass
 class AgentHolder:
     """Per-turn agent state the API layer reads after the stream (task 04).
@@ -200,8 +300,11 @@ def _execute_tool(
 
     Returns the tool result text. A successful call bumps
     ``holder.tool_calls`` (a successful read also appends the
-    :class:`Document` to ``holder.read_docs``); rejected calls return
-    their refusal line and count in nothing.
+    :class:`Document` to ``holder.read_docs``; a search never does — it
+    is a locator, locked A5); rejected calls return their refusal line
+    and count in nothing. A search that ran but found nothing is still a
+    successful (counted) call — its no-match line is a result, not a
+    refusal.
     """
     if call.name == "list_documents":
         rows = list_catalog(db)
@@ -229,6 +332,49 @@ def _execute_tool(
         holder.read_docs.append(doc)
         holder.tool_calls += 1
         return f"Document {source}/{path}:\n{doc.content}"
+    if call.name == "search_documents":
+        raw_pattern = call.arguments.get("pattern")
+        pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
+        if not pattern:
+            return MISSING_SEARCH_ARGS
+        raw_source = call.arguments.get("source")
+        raw_path = call.arguments.get("path")
+        source = raw_source.strip() if isinstance(raw_source, str) else ""
+        path = raw_path.strip() if isinstance(raw_path, str) else ""
+        if (source == "") != (path == ""):
+            # A half-specified target is a model error — fail loud with
+            # the missing-args refusal instead of silently widening to a
+            # whole-KB search (house style).
+            return MISSING_SEARCH_ARGS
+        if source:
+            target = find_document(db, source, path)
+            if target is None:
+                return (
+                    f"No document at {source}/{path} — check the list_documents output."
+                )
+            docs: list[Document] = [target]
+        else:
+            docs = all_documents(db)
+        matches: list[str] = []
+        for doc in docs:
+            for lineno, line in grep_document(doc.content, pattern):
+                matches.append(
+                    f"{doc.source}/{doc.path}:{lineno}: {line[:SEARCH_LINE_LIMIT]}"
+                )
+                if len(matches) >= SEARCH_MAX_MATCHES:
+                    break
+            if len(matches) >= SEARCH_MAX_MATCHES:
+                break  # the global cap is hit — stop scanning
+        holder.tool_calls += 1  # the search executed (no-match counts too)
+        # Locked A5: a search never adds context — read_docs untouched.
+        if not matches:
+            shown = pattern[:100]  # keep a long pattern short in the line
+            if source:
+                return NO_MATCHES_SCOPED.format(
+                    pattern=shown, source=source, path=path
+                )
+            return NO_MATCHES.format(pattern=shown)
+        return "\n".join(matches)
     return UNKNOWN_TOOL
 
 

app/schemas.py

@@ -71,20 +71,24 @@ class ChatThinkingEvent(BaseModel):
 class ChatToolEvent(BaseModel):
     """SSE frame for one agent tool call (phase 37, PLAN §4 extension).
 
-    A15 extension (owner permission 2026-08-26): a grounded turn may call
-    the server-side document tools (``list_documents`` / ``read_document``,
-    see :mod:`app.rag.agent`); each model-requested call streams as
-    ``{type: "tool", name: str, argument: str | null}`` ahead of the
-    answer's ``delta`` frames. ``argument`` is the read document's
-    ``"source/path"`` for ``read_document`` and null otherwise. The client
+    A15 extension (owner permission 2026-08-26; ``search_documents``
+    added in phase 68): a grounded turn may call the server-side
+    document tools (``list_documents`` / ``read_document`` /
+    ``search_documents``, see :mod:`app.rag.agent`); each model-requested
+    call streams as ``{type: "tool", name: str, argument: str | null}``
+    ahead of the answer's ``delta`` frames. ``argument`` is the read
+    document's ``"source/path"`` for ``read_document``, the search
+    pattern for ``search_documents``, and null otherwise (a non-string
+    pattern — a model error the backend refuses — is null). The client
     renders each frame as a "calling tool" line/state (phase 37 task 05);
     the ``delta`` / ``done`` shapes are unchanged — the read document is
-    reflected in ``done.sources`` instead.
+    reflected in ``done.sources`` instead (a search adds no source: it is
+    a locator, locked A5).
     """
 
     type: Literal["tool"] = "tool"
-    name: str  # "list_documents" | "read_document"
-    argument: str | None = None  # "source/path" for read_document
+    name: str  # "list_documents" | "read_document" | "search_documents"
+    argument: str | None = None  # "source/path" for read_document, pattern for search_documents
 
 
 class ChatDoneEvent(BaseModel):