feat(agent): search_documents tool — the model can grep the indexed documents for an exact string

app/rag/agent.py

@@ -15,27 +15,38 @@ probe came back "supported".
 Loop contract (one grounded chat turn; the API layer wires this in,
 task 04):
 
-1. The model is offered the two OpenAI functions in :data:`AGENT_TOOLS`
+1. The model is offered the three OpenAI functions in :data:`AGENT_TOOLS`
    for the whole turn — phase 45 removed the phase-37 per-tool budgets
    (owner permission 2026-08-27, ``TODO.md`` L8: "allow the LLM to make
-   as many tool calls as it wants"): ``list_documents`` and
-   ``read_document`` can each be called as many times as the model needs,
-   re-lists included. With ``settings.agent_max_rounds``
-   (``BOR_AGENT_MAX_ROUNDS``, default 10) at 0 the loop makes exactly one
-   request with ``tools=None`` — byte-identical to the pre-phase-37 chat
-   path (the kill switch).
+   as many tool calls as it wants"): ``list_documents``,
+   ``read_document`` and ``search_documents`` can each be called as many
+   times as the model needs, re-lists and re-searches included. With
+   ``settings.agent_max_rounds`` (``BOR_AGENT_MAX_ROUNDS``, default 10)
+   at 0 the loop makes exactly one request with ``tools=None`` —
+   byte-identical to the pre-phase-37 chat path (the kill switch).
 2. Each tool call the model emits is executed server-side against
    Postgres only (no LLM, no network): ``list_documents`` returns the
    indexed catalog — one ``source: X | path: Y | title: Z`` line per
    document (phase 63: labeled fields — unambiguous for LLM parsing),
    ``GET /api/docs`` order (uncapped in v1; the UI never shows it, only
-   the model does) — and ``read_document`` returns the document's **full**
-   content (A7-revised contract: never truncated).
+   the model does) — ``read_document`` returns the document's **full**
+   content (A7-revised contract: never truncated) — and
+   ``search_documents`` greps the indexed documents (or one named
+   document) for a case-insensitive fixed substring and returns up to 20
+   ``source/path:line: text`` match lines (owner-locked A5, phase 68),
+   each line truncated to 200 chars. A search is a **locator**, not a
+   context-adder: it never appends to the answer context (only
+   ``read_document`` does — ``holder.read_docs`` is untouched by a
+   search).
 3. Rejected calls get a one-line refusal and count in nothing
    (``holder.tool_calls`` tracks executed calls only): unknown tool name
-   → ``"Unknown tool."``; missing ``source``/``path`` arguments; a
-   document already in context (seed or previously read) → ``"Already in
-   your context."``; an unknown ``source/path`` → ``"No document at …"``.
+   → ``"Unknown tool."``; missing ``source``/``path`` arguments; a search
+   without a usable ``pattern`` (missing, blank or non-string) or with a
+   half-specified ``source``/``path`` target; a document already in
+   context (seed or previously read) → ``"Already in your
+   context."``; an unknown ``source/path`` (read or scoped search) →
+   ``"No document at …"``. A search that ran but found nothing is NOT a
+   rejection — its ``"No matches for …"`` line is a (counted) result.
    A rejected call still consumes a *round* in the loop, so a
    pathological stream that keeps emitting rejected calls is bounded by
    the cap (point 4).
@@ -65,7 +76,8 @@ task 04):
    exactly one round. With ``settings.llm_retries=0`` every request is a
    single plain attempt (the pre-phase-67 path).
 
-The DB accessors (:func:`list_catalog`, :func:`find_document`) are
+The DB accessors (:func:`list_catalog`, :func:`find_document`,
+:func:`all_documents`) and the :func:`grep_document` line matcher are
 module-level functions so unit tests can monkeypatch them without a
 database.
 """
@@ -92,10 +104,11 @@ from app.rag.llm import (
 
 logger = logging.getLogger("app.agent")
 
-#: The two agent tools (phase 37): OpenAI function definitions passed as
-#: ``tools=AGENT_TOOLS`` to ``chat_stream`` for the whole grounded turn —
-#: phase 45 removed the per-tool budgets; the round cap
-#: (``BOR_AGENT_MAX_ROUNDS``) is the only bound.
+#: The three agent tools (phase 37; ``search_documents`` added in phase
+#: 68): OpenAI function definitions passed as ``tools=AGENT_TOOLS`` to
+#: ``chat_stream`` for the whole grounded turn — phase 45 removed the
+#: per-tool budgets; the round cap (``BOR_AGENT_MAX_ROUNDS``) is the only
+#: bound.
 AGENT_TOOLS: list[dict[str, Any]] = [
     {
         "type": "function",
@@ -140,6 +153,49 @@ AGENT_TOOLS: list[dict[str, Any]] = [
             },
         },
     },
+    {
+        "type": "function",
+        "function": {
+            "name": "search_documents",
+            "description": (
+                "Search every indexed document for an exact string "
+                "(case-insensitive) and return up to 20 matching lines as "
+                "'source/path:line: text' — use this to locate content, "
+                "then read_document the winner. Optionally pass 'source' "
+                "and 'path' (as shown in list_documents) to search one "
+                "document only."
+            ),
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "pattern": {
+                        "type": "string",
+                        "description": (
+                            "The exact text to search for (a plain "
+                            "substring, not a regex)"
+                        ),
+                    },
+                    "source": {
+                        "type": "string",
+                        "description": (
+                            "The document's source, as shown after 'source: ' in the "
+                            "list_documents output (e.g. 'Homelab' from "
+                            "'source: Homelab | path: homelab/aws-route53.md')."
+                        ),
+                    },
+                    "path": {
+                        "type": "string",
+                        "description": (
+                            "The document's path, as shown after 'path: ' in the "
+                            "list_documents output (e.g. 'homelab/aws-route53.md' from "
+                            "'source: Homelab | path: homelab/aws-route53.md')."
+                        ),
+                    },
+                },
+                "required": ["pattern"],
+            },
+        },
+    },
 ]
 
 #: Tool refusal texts (phase 37): rejected calls count in nothing
@@ -148,6 +204,19 @@ AGENT_TOOLS: list[dict[str, Any]] = [
 ALREADY_IN_CONTEXT = "Already in your context."
 UNKNOWN_TOOL = "Unknown tool."
 MISSING_READ_ARGS = "read_document requires string arguments 'source' and 'path'."
+MISSING_SEARCH_ARGS = "search_documents requires a string argument 'pattern'."
+
+#: Search caps (owner-locked A5, phase 68): a global per-call match cap
+#: (across documents, in catalog order) and a per-match-line char limit.
+SEARCH_MAX_MATCHES = 20
+SEARCH_LINE_LIMIT = 200
+
+#: No-match result lines (templates — the pattern is truncated to 100
+#: chars before formatting, to keep a long pattern from bloating the
+#: tool result). A no-match line is a *result* of an executed search,
+#: not a refusal (see the module docstring, point 3).
+NO_MATCHES = "No matches for '{pattern}' in the knowledge base."
+NO_MATCHES_SCOPED = "No matches for '{pattern}' in {source}/{path}."
 
 
 def list_catalog(db: Session) -> list[tuple[str, str, str]]:
@@ -174,6 +243,37 @@ def find_document(db: Session, source: str, path: str) -> Document | None:
     )
 
 
+def all_documents(db: Session) -> list[Document]:
+    """Every indexed document (full rows), ordered by ``(source, path)``
+    — catalog order.
+
+    The whole-KB ``search_documents`` path loads all contents in this one
+    bulk query (catalog order is the locked match order, owner-locked A5).
+    Module-level (not a method) so unit tests can monkeypatch it.
+    """
+    return list(
+        db.execute(
+            select(Document).order_by(Document.source, Document.path)
+        ).scalars()
+    )
+
+
+def grep_document(content: str, pattern: str) -> list[tuple[int, str]]:
+    """Every line of *content* that contains *pattern*, in file order.
+
+    Case-insensitive **fixed substring** (owner-locked A5: no regex — no
+    ReDoS surface, a simple contract for the model). Returns
+    ``(1-based line number, line.rstrip())`` pairs; an empty *content*
+    never matches a non-empty pattern.
+    """
+    needle = pattern.lower()
+    return [
+        (number, line.rstrip())
+        for number, line in enumerate(content.split("\n"), start=1)
+        if needle in line.lower()
+    ]
+
+
 @dataclass
 class AgentHolder:
     """Per-turn agent state the API layer reads after the stream (task 04).
@@ -200,8 +300,11 @@ def _execute_tool(
 
     Returns the tool result text. A successful call bumps
     ``holder.tool_calls`` (a successful read also appends the
-    :class:`Document` to ``holder.read_docs``); rejected calls return
-    their refusal line and count in nothing.
+    :class:`Document` to ``holder.read_docs``; a search never does — it
+    is a locator, locked A5); rejected calls return their refusal line
+    and count in nothing. A search that ran but found nothing is still a
+    successful (counted) call — its no-match line is a result, not a
+    refusal.
     """
     if call.name == "list_documents":
         rows = list_catalog(db)
@@ -229,6 +332,49 @@ def _execute_tool(
         holder.read_docs.append(doc)
         holder.tool_calls += 1
         return f"Document {source}/{path}:\n{doc.content}"
+    if call.name == "search_documents":
+        raw_pattern = call.arguments.get("pattern")
+        pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
+        if not pattern:
+            return MISSING_SEARCH_ARGS
+        raw_source = call.arguments.get("source")
+        raw_path = call.arguments.get("path")
+        source = raw_source.strip() if isinstance(raw_source, str) else ""
+        path = raw_path.strip() if isinstance(raw_path, str) else ""
+        if (source == "") != (path == ""):
+            # A half-specified target is a model error — fail loud with
+            # the missing-args refusal instead of silently widening to a
+            # whole-KB search (house style).
+            return MISSING_SEARCH_ARGS
+        if source:
+            target = find_document(db, source, path)
+            if target is None:
+                return (
+                    f"No document at {source}/{path} — check the list_documents output."
+                )
+            docs: list[Document] = [target]
+        else:
+            docs = all_documents(db)
+        matches: list[str] = []
+        for doc in docs:
+            for lineno, line in grep_document(doc.content, pattern):
+                matches.append(
+                    f"{doc.source}/{doc.path}:{lineno}: {line[:SEARCH_LINE_LIMIT]}"
+                )
+                if len(matches) >= SEARCH_MAX_MATCHES:
+                    break
+            if len(matches) >= SEARCH_MAX_MATCHES:
+                break  # the global cap is hit — stop scanning
+        holder.tool_calls += 1  # the search executed (no-match counts too)
+        # Locked A5: a search never adds context — read_docs untouched.
+        if not matches:
+            shown = pattern[:100]  # keep a long pattern short in the line
+            if source:
+                return NO_MATCHES_SCOPED.format(
+                    pattern=shown, source=source, path=path
+                )
+            return NO_MATCHES.format(pattern=shown)
+        return "\n".join(matches)
     return UNKNOWN_TOOL