Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Details
Build and Push Containers / build-and-push-db (push) Successful in 14s
Details

Browse Source 
Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled
methodology (owner directive: stop clearing/re-importing the homelab KB per
iteration; measure tool-calling accuracy on a controlled fixture KB, target
>90%).

Real-model gate verdicts (live, configured chat model 'lite', fixture KB):
- Controlled fixture battery (the new methodology's pass condition —
  contract accuracy >= 90%): PASS, 4 consecutive runs:
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s)
- Locked derived battery (phase-72 task 05, executed >= 90% bar, run
  unchanged on the same fixture KB):
  gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s)
  The teaching works — every bare-path trap self-corrects in exactly one
  round, zero cap hits, zero repeat loops, 10/10 answered. The locked
  executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the
  corrected re-reads (the trap question seeds its target, so the correct
  combined-form read is refused for redundancy) — a copy-invariant model
  behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04)
  and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md
  sections 5 and 7), not a copy lever.

Copy changes this phase owns (unit pins updated to follow):
- app/rag/agent.py: ls teaching refusals (path-like scope -> document-path
  line; unknown source -> no-source line with the source-name
  parenthetical), read/grep 'did you mean source/path?' teaching
  (find_path_candidates: exact or suffix path match, catalog order, cap 3),
  ALREADY_IN_CONTEXT naming the correct action (answer from the text
  already in the prompt), read tool description front-loaded with the
  do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is
  the only remaining refusal class; contract accuracy 92-100% across runs)
- app/rag/prompts.py: TOOLS_SECTION states the document-identity contract
  up front (ls path = source name; read/grep = combined source/path
  including the source name; do-not-read for <documents> documents placed
  next to the read teaching; one-call-per-reply and never-repeat rules)
- tests: refusal pins (unit + integration), new dedicated E2E suite
  tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in
  isolation), regression suites green in isolation (harness_aligned_tools,
  agent_document_tools, agent_unlimited_tools, search_tool, chat_rag).

Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff +
pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/
move and both phases' .agent/reports/ (AGENTS.md 8).
This commit is contained in:
ducoterra
2026-09-04 13:11:07 -04:00
parent 7909bdb8da
commit 988ff78526
42 changed files with 2987 additions and 88 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/rag/agent.py
+222 -24
View File
@@ -13,6 +13,20 @@ JSON-block fallback (documented in the task file) is *not* implemented —
it exists only for a "not supported"/"intermittent" verdict, and the
probe came back "supported".
Real-model gate (phase 72, task 05 — live vs the configured chat
model; re-run 2026-09-04 on the controlled fixture KB — see
``TOOL_CALLING_TESTING.md``): the bare-path teaching (did-you-mean
refusals) makes every trap self-correct in exactly one round — zero
cap hits, zero repeat loops. Locked derived battery (phase 72,
executed ≥ 90 % bar): ``gate: lite FAIL turns=10 answered=10 caps=0
tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04
(wall 47.7s)`` — the bar is blocked by :data:`ALREADY_IN_CONTEXT` dedupe
refusals on the corrected re-reads, a copy-invariant model behavior
(five copy variants, 2026-09-03 → 04) and an app-semantics decision
(TOOL_CALLING_TESTING.md §7). Controlled fixture battery (the 2026-09-04
methodology — contract accuracy ≥ 90 %): PASS on three consecutive runs,
``contract 11/11 (100%)`` / ``12/13 (92%)`` / ``11/11 (100%)``.
Loop contract (one grounded chat turn; the API layer wires this in,
task 04):
@@ -58,14 +72,38 @@ task 04):
blank or non-string) → ``"read requires a string argument
'path'."``; a ``grep`` without a usable ``pattern`` (missing, blank
or non-string) → ``"grep requires a string argument
'pattern'."``; a scoped ``ls`` whose ``path`` matches no source name
→ ``"No source named '…' — check the ls output."``; a document
already in context (seed or previously read) → ``"Already in your
context."``; an unknown document (a ``read`` or scoped ``grep`` whose
'pattern'."``; a scoped ``ls`` whose (stripped) ``path`` contains a
``/`` — a document path where a source name belongs (source names
are directory basenames and can never contain one; the 2026-09-03
incident's ``ls(path='app/rag/importer.py')``) →
:data:`LS_PATH_NOT_A_SOURCE`, the document-path teaching line with
the argument echoed; a scoped ``ls`` whose ``path`` names no
registered source (no ``/`` — the incident's ``ls(path='.')``)
→ :data:`NO_SOURCE_NOT_A_DIRECTORY`, the no-source refusal with the
teaching parenthetical appended; a document
already in context (seed or previously read) →
:data:`ALREADY_IN_CONTEXT` (phase 72, task 05 gate iteration:
the line names the correct action — answer from the text already
in the prompt, do not call read again — so a fired refusal ends
the loop instead of inviting a repeat); an unknown document (a ``read`` or scoped ``grep`` whose
combined ``source/path`` matches nothing — a bare source name, which
can never be a document, included) → ``"No document at '…' — check
the ls output."`` with the argument echoed as passed (the model sees
its own form). A grep that ran but found nothing is NOT a rejection
its own form) — EXCEPT the phase-72 "did you mean …?" teaching
(task 02): when the argument is a path (contains ``/``) that matches
an indexed document's ``path`` (exact or as a ``/arg`` suffix,
case-sensitive, catalog order — :func:`find_path_candidates`, a pure
catalog lookup, one bulk query, called only from this refusal path),
the refusal names the combined identity instead — exactly one match
→ :data:`NO_DOCUMENT_DID_YOU_MEAN` (``did you mean
'source/path'?``), two or more → :data:`NO_DOCUMENT_DID_YOU_MEAN_MANY`
(up to :data:`SUGGESTION_LIMIT` identities), so the harness-prior
misuse (the bare document path missing the source prefix,
``read('app/rag/importer.py')``) self-corrects in one round; it is
still a refusal (counts in nothing, consumes a round — no silent
argument normalization), and a bare argument (no ``/``) or a
zero-candidate path keeps the line above byte-identical (the bare
form never hits the DB). A grep that ran but found nothing is NOT a rejection
— its ``"No matches for …"`` line is a (counted) result. A rejected
call still consumes a *round* in the loop, so a pathological stream
that keeps emitting rejected calls is bounded by the cap (point 4).
@@ -164,7 +202,8 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"name": "ls",
"description": (
"List the indexed documents as `source: X | path: Y | "
"title: Z` lines."
"title: Z` lines. Call one tool at a time — wait for "
"this result before your next call."
),
"parameters": {
"type": "object",
@@ -173,8 +212,12 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"type": "string",
"description": (
"Source name to list one source's documents "
"(e.g. 'homelab'); omit to list every "
"document."
"(e.g. 'homelab') — a source name, not a "
"file or directory path; omit to list "
"every document. This is the only tool "
"whose `path` is a source name — for "
"`read` and `grep` it must be a document's "
"combined `source/path`."
),
}
},
@@ -187,8 +230,13 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"function": {
"name": "read",
"description": (
"Add the full content of one indexed document to your "
"context."
"Do not call this tool for a document already shown in "
"the <documents> section, even when the user asks you to "
"open or read it — its full text is already in your "
"prompt; answer directly from it. Use it only to add a "
"document NOT already in <documents> to your context, "
"by its combined `source/path` string. Call one tool at "
"a time — wait for this result before your next call."
),
"parameters": {
"type": "object",
@@ -199,7 +247,12 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"The document to add to your context, as the "
"combined `source/path` string exactly as "
"shown in the `ls` output (e.g. "
"'homelab/active/container_caddy/caddy.md')."
"'homelab/active/container_caddy/caddy.md'). "
"A bare document path (without the source "
"name) will not resolve. Only pass a document "
"NOT already shown in the <documents> "
"section — it is already in your context; do "
"not re-read it."
),
}
},
@@ -215,7 +268,12 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"Search the indexed documents for an exact string "
"(case-insensitive) and return up to 20 matching lines "
"as `source/path:line: text` — a locator, not a "
"context-adder: read the winner with `read`."
"context-adder: read the winner with `read`. For a "
"normal search pass ONLY `pattern` — it searches every "
"document and that is how you search the knowledge "
"base; never pass a source name as `path` (a source "
"name is not a document). Call one tool at a time — "
"wait for this result before your next call."
),
"parameters": {
"type": "object",
@@ -230,10 +288,13 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"path": {
"type": "string",
"description": (
"Limit the search to one document, as a "
"combined `source/path` string from the "
"`ls` output (omit to search every "
"document)."
"Rarely needed — only for re-searching one "
"document you already know: that document's "
"combined `source/path` identity (e.g. "
"'homelab/ansible/inventory.yaml'). Never a "
"source name. A bare document path (without "
"the source name) will not resolve. Omit it "
"for a normal search (pass only `pattern`)."
),
},
},
@@ -245,12 +306,78 @@ AGENT_TOOLS: list[dict[str, Any]] = [
#: Tool refusal texts (phase 37): rejected calls count in nothing
#: (``holder.tool_calls`` tracks executed calls); the round cap bounds
#: their pathological repetition (phase 45).
ALREADY_IN_CONTEXT = "Already in your context."
#: their pathological repetition (phase 45). The in-context line is a
#: phase-72, task 05 gate-iteration teaching (live telemetry: the
#: ``lite`` model obeyed the user's "open it / read it" and re-read
#: seed-context documents, then repeated the call against the terse
#: phase-37 line — the refusal itself carried no correct action): same
#: behavior (a refusal: counts in nothing, consumes a round, changes
#: no context), the copy now names the action, so even a fired
#: refusal ends the loop instead of inviting a repeat.
ALREADY_IN_CONTEXT = (
"Already in your context — the full text is already in your "
"prompt. Do not call read on it again; answer from that text."
)
UNKNOWN_TOOL = "Unknown tool."
MISSING_READ_ARGS = "read requires a string argument 'path'."
MISSING_SEARCH_ARGS = "grep requires a string argument 'pattern'."
#: Teaching refusal for a scoped ``ls`` whose stripped ``path``
#: contains a ``/`` (phase 72): a source name is a directory basename
#: and can never contain one, so the argument is a document path passed
#: where a source name belongs (the 2026-09-03 incident's
#: ``ls(path='app/rag/importer.py')``). One ``{path}`` field — the
#: argument echoed; a fixed template states the correct contract
#: instead of the terse pre-phase-72 line, so the harness-prior misuse
#: self-corrects in one round.
LS_PATH_NOT_A_SOURCE = (
"'{path}' looks like a document path, not a source name. The "
"'path' argument of ls filters by source name (e.g. 'homelab') — "
"omit it to list every document, or read a document by its "
"combined 'source/path' string."
)
#: The no-source ``ls`` refusal with the teaching parenthetical
#: appended (phase 72): used when a stripped scope has no ``/`` and
#: matches no registered source (the incident's ``ls(path='.')``). The
#: prefix — the pre-phase-72 line — stays byte-identical; one ``{scope}``
#: field, the argument echoed.
NO_SOURCE_NOT_A_DIRECTORY = (
"No source named '{scope}' — check the ls output. (The 'path' "
"argument is a source name, not a directory — omit it to list "
"every document.)"
)
#: Teaching refusal for a ``read`` / scoped ``grep`` argument that
#: resolves to no combined identity but matches ONE indexed document's
#: ``path`` (phase 72, task 02): names the exact combined
#: ``source/path`` identity to use, so the harness-prior misuse — the
#: bare document path missing the source prefix
#: (``read('app/rag/importer.py')``) — self-corrects in one round.
#: One each of the fields ``{arg}`` (the argument echoed as passed),
#: ``{source}`` and ``{path}`` (the one candidate). Still a refusal:
#: it counts in nothing and consumes a round (no silent argument
#: normalization).
NO_DOCUMENT_DID_YOU_MEAN = (
"No document at '{arg}' — did you mean '{source}/{path}'?"
)
#: The ambiguous form of the same teaching (phase 72, task 02): the
#: argument matches SEVERAL indexed documents' ``path`` (the same path
#: under several sources). ``{candidates}`` holds up to
#: :data:`SUGGESTION_LIMIT` combined ``source/path`` identities, each
#: single-quoted, joined with ``", "`` in catalog order; ``{arg}`` is
#: the argument echoed as passed.
NO_DOCUMENT_DID_YOU_MEAN_MANY = (
"No document at '{arg}' — did you mean one of: {candidates}?"
)
#: Cap on the suggested combined identities per "did you mean …?"
#: refusal (phase 72, task 02): the same document ``path`` under
#: several sources suggests up to this many (catalog order, the rest
#: dropped).
SUGGESTION_LIMIT = 3
#: The harness-owned recovery line (phase 71, task 03) — folded into the
#: ORIGINAL single system message of the one bounded recovery request
#: (``system_prompt + "\n" + CORRECTION_INSTRUCTION``; provider-safe,
@@ -376,6 +503,61 @@ def all_documents(db: Session) -> list[Document]:
)
def find_path_candidates(db: Session, arg: str) -> list[tuple[str, str, str]]:
"""The indexed documents a bare document *arg* names by ``path``.
Phase 72, task 02: a ``read`` / scoped-``grep`` argument that
resolves to no combined identity but *is* a document path (contains
``/``) is matched against the indexed ``Document.path`` values so
the refusal can name the combined ``source/path`` identity to use
(the "did you mean …?" teaching). The documents whose ``path``
equals *arg* (the exact bare path) or ends with ``f"/{arg}"`` (the
file is nested deeper — the suffix match) — in catalog order (the
:func:`all_documents` order), case-sensitive (these are file
paths) — as ``(source, path, title)`` triples. One bulk query via
:func:`all_documents` (at most one); called ONLY from the refusal
path of :func:`_execute_tool` (never on the happy path) and only
when *arg* contains ``/`` (a bare name keeps today's no-DB-lookup
refusal). Module-level (not a method) so unit tests can
monkeypatch it.
"""
return [
(doc.source, doc.path, doc.title)
for doc in all_documents(db)
if doc.path == arg or doc.path.endswith(f"/{arg}")
]
def _no_document_refusal(db: Session, arg: str) -> str:
"""The no-document refusal for an unresolved ``read`` / scoped-
``grep`` argument (phase 72, task 02).
The pre-phase-72 line — the argument echoed as passed — whenever
there is nothing to suggest: a bare argument (no ``/`` — a bare
source name or any other bare name gets the no-DB-lookup refusal,
byte-identical to today) or a path-like argument that matches no
indexed document's ``path`` (zero candidates). A path-like argument
(contains ``/``) that matches exactly one indexed document's
``path`` gets :data:`NO_DOCUMENT_DID_YOU_MEAN` (the combined
identity named); two or more get :data:`NO_DOCUMENT_DID_YOU_MEAN_MANY`
(up to :data:`SUGGESTION_LIMIT`, catalog order). Deterministic
only: the suggestion is a pure catalog lookup, no model. A refusal
still counts in nothing and consumes a round.
"""
if "/" in arg:
candidates = find_path_candidates(db, arg)
if len(candidates) == 1:
source, path, _title = candidates[0]
return NO_DOCUMENT_DID_YOU_MEAN.format(arg=arg, source=source, path=path)
if len(candidates) > 1:
identities = ", ".join(
f"'{source}/{path}'"
for source, path, _title in candidates[:SUGGESTION_LIMIT]
)
return NO_DOCUMENT_DID_YOU_MEAN_MANY.format(arg=arg, candidates=identities)
return f"No document at '{arg}' — check the ls output."
def grep_document(content: str, pattern: str) -> list[tuple[int, str]]:
"""Every line of *content* that contains *pattern*, in file order.
@@ -437,8 +619,19 @@ def _execute_tool(
scope = raw_path.strip() if isinstance(raw_path, str) else ""
rows = list_catalog(db)
if scope:
if "/" in scope:
# A source name (a directory basename) can never
# contain '/' — this is a document path where a source
# name belongs (phase 72): teach the contract; no
# registry lookup needed, counts in nothing, consumes
# a round like every refusal.
return LS_PATH_NOT_A_SOURCE.format(path=scope)
if scope not in list_source_names(db):
return f"No source named '{scope}' — check the ls output."
# The no-source refusal with the teaching parenthetical
# (phase 72) — the prefix byte-identical to the
# pre-phase-72 line; counts in nothing, consumes a
# round like every refusal.
return NO_SOURCE_NOT_A_DIRECTORY.format(scope=scope)
rows = [row for row in rows if row[0] == scope]
listing = f"{len(rows)} documents:\n" + "\n".join(
f"source: {source} | path: {path} | title: {title}"
@@ -462,8 +655,10 @@ def _execute_tool(
doc, _source, _path = _resolve_path(db, arg)
if doc is None:
# Echo the argument as passed — the model sees its own form
# (a bare source name can never be a document, no DB lookup).
return f"No document at '{arg}' — check the ls output."
# (a bare argument can never be a document, no DB lookup);
# a path-like argument that matches an indexed document's
# path gets the "did you mean …?" teaching (phase 72).
return _no_document_refusal(db, arg)
holder.read_docs.append(doc)
holder.tool_calls += 1
return f"Document {doc.source}/{doc.path}:\n{doc.content}"
@@ -478,7 +673,9 @@ def _execute_tool(
if scope:
target, src, p = _resolve_path(db, scope)
if target is None:
return f"No document at '{scope}' — check the ls output."
# The same phase-72 "did you mean …?" teaching as the
# read branch (a refusal — not counted, no context).
return _no_document_refusal(db, scope)
docs: list[Document] = [target]
scoped_to = (src, p) # the resolved (canonical) identity
else:
@@ -552,8 +749,9 @@ async def run_agent(
``seed_docs`` are the documents the retrieval already put in context
(they shape the *system_prompt* the caller built); re-reading one of
them is rejected as "Already in your context." — the rejection counts
in nothing, but it still consumes a round.
them is rejected with :data:`ALREADY_IN_CONTEXT` (the phase-72
teaching line — answer from the text already in the prompt) — the
rejection counts in nothing, but it still consumes a round.
"""
messages: list[dict[str, Any]] = [
{"role": "system", "content": system_prompt},
app/rag/prompts.py
+94 -27
View File
@@ -25,13 +25,20 @@ roughly what the KB contains before retrieval. With an empty row the
prompt is byte-identical to the pre-phase text.
Agent tools (phase 37; phase 70: the copy teaches the harness-aligned
``ls`` / ``read`` / ``grep`` shapes): the **HIGH** prompt only carries a
``<tools>`` section after the ``<documents>`` body — the grounded turn
may extend its context through the three server-side tools (round-
capped, see :mod:`app.rag.agent`; the cap is the bound and this section
does not re-state it, phase 45). The LOW/deflection prompt never
carries it (phase 71: the LOW prompt's only addition is the
plain-text line below — it still has no ``<tools>`` section).
``ls`` / ``read`` / ``grep`` shapes; phase 72: the copy states the
document-identity contract up front — ``ls``'s ``path`` is a *source
name*, not a directory or file path, and ``read``/``grep`` take the
combined ``source/path`` string *including the source name* (a bare
document path will not resolve) — the same two things the phase-72
teaching refusals in :mod:`app.rag.agent` re-state after the fact, so
the model carries the contract before it calls a tool): the **HIGH**
prompt only carries a ``<tools>`` section after the ``<documents>``
body — the grounded turn may extend its context through the three
server-side tools (round-capped, see :mod:`app.rag.agent`; the cap is
the bound and this section does not re-state it, phase 45). The
LOW/deflection prompt never carries it (phase 71: the LOW prompt's
only addition is the plain-text line below — it still has no
``<tools>`` section).
Deflection plain-text line (phase 71, owner-permitted 2026-09-03):
the otherwise-locked ``LOW`` prompt gains exactly one instruction
@@ -89,28 +96,69 @@ _KB_INTRO = (
#: The ``<tools>`` instructions section — **HIGH prompt only** (phase 37,
#: task 03; phase 70: the copy is rewritten for the harness-aligned
#: ``ls`` / ``read`` / ``grep`` shapes, names/args exactly as the
#: ``AGENT_TOOLS`` schemas in :mod:`app.rag.agent`): a grounded turn may
#: extend its context through the three server-side tools (round cap:
#: ``BOR_AGENT_MAX_ROUNDS`` — the cap is the bound and this section does
#: not re-state it, phase 45). Appended after the mode body
#: (``<documents>``), so the instructions are the last thing the model
#: reads. The LOW/deflection prompt never carries it — a deflection has
#: no grounded context to extend (phase 71: the LOW prompt's only
#: addition is the plain-text line in :func:`build_deflect_prompt`).
#: The E2E mock keys off the ``<tools>`` marker's *presence*, not this
#: wording.
#: ``AGENT_TOOLS`` schemas in :mod:`app.rag.agent`; phase 72: the copy
#: states the document-identity contract UP FRONT — ``ls``'s optional
#: ``path`` is a *source name* (not a directory or file path) and
#: ``read``/``grep`` take the combined ``source/path`` string *including
#: the source name* (a bare document path will not resolve) — the same
#: two things the phase-72 teaching refusals re-state after the fact):
#: a grounded turn may extend its context through the three server-side
#: tools (round cap: ``BOR_AGENT_MAX_ROUNDS`` — the cap is the bound and
#: this section does not re-state it, phase 45). Appended after the mode
#: body (``<documents>``), so the instructions are the last thing the
#: model reads. The LOW/deflection prompt never carries it — a
#: deflection has no grounded context to extend (phase 71: the LOW
#: prompt's only addition is the plain-text line in
#: :func:`build_deflect_prompt`). The E2E mock keys off the ``<tools>``
#: marker's *presence*, not this wording. Task 05 (the live gate's
#: iteration loop) keeps the baseline and carries the discipline rules
#: the live telemetry motivated, refined across the task-05 re-runs of
#: 2026-09-03/04 (run 1: 7/16 executed, 44% — the ``lite`` model obeyed
#: the user's "open it / read it" and ``read`` seed-context documents,
#: then repeated the refused call; run 2: 8/18, 44% — the repeat is
#: gone, but a grep turn batched two calls per reply and the harness
#: runs only the first of a batch): the do-not-read rule names the
#: user-command scenario (a document already in the ``<documents>``
#: section: do not call ``read``, answer from the text already in the
#: prompt — an anchor on the concrete ``<document path="...">`` markup
#: was tried and REVERTED: it primed the model to latch the seed
#: documents' paths as ``ls`` scopes, regressing the incident turn);
#: the one-call rule names the consequence (a batched second call is
#: discarded — runs only the first); the never-repeat rule says why
#: (the refusal already told you the correct form); the ``grep`` clause
#: leads with "for a normal search pass only ``pattern``" (the gate's
#: live runs showed the model scoping ``grep`` with an ``ls``-style
#: source name — the incident shape, but on grep). The behavioral
#: contract lives in the ``AGENT_TOOLS`` descriptions as well (the most
#: local text at call time): ``read`` must not be called for a
#: ``<documents>`` document at all; ``grep`` with only ``pattern``
#: searches the whole knowledge base, and a source name is not a
#: document.
TOOLS_SECTION: str = (
"<tools>\n"
"You may extend your context with three tools. `ls` lists the "
"indexed documents as `source: X | path: Y | title: Z` lines "
"(pass a source name as `path` to list one source's documents; "
"omit it to list every document). `grep` locates an exact string "
"(case-insensitive) in the indexed documents and returns up to 20 "
"matching `source/path:line: text` lines — a locator, not a "
"context-adder: read the winner with `read`. `read` pulls in one "
"document by its combined `source/path` string, exactly as shown in "
"the `ls` output, adding its full content to your context. Answer "
"as soon as you have what you need.\n"
"indexed documents as `source: X | path: Y | title: Z` lines; its "
"optional `path` argument is a source name (e.g. 'homelab'), not a "
"directory or file path — omit it to list every document. `read` "
"pulls in one document by its combined `source/path` string, "
"exactly as shown in the `ls` output — including the source name — "
"adding its full content to your context. Do not call `read` for a "
"document already shown in the <documents> section, even when the "
"user asks you to open or read it — its full text is already in "
"your prompt; answer directly from it. For `read`, a bare document "
"path (without the source name) will not resolve. `grep` locates an "
"exact string (case-insensitive) in the indexed documents and "
"returns up to 20 matching `source/path:line: text` lines — a "
"locator, not a context-adder: read the winner with `read`; for a "
"normal search pass only `pattern` — its optional `path` argument "
"limits the search to one document you already know, by the same "
"combined `source/path` string; never a source name — a bare "
"document path (without the source name) will not resolve there "
"either. Make exactly one tool call per reply — a reply carrying "
"two tool calls runs only the first, the second is discarded — and "
"wait for the result before the next call. Never repeat a call that "
"was refused or already succeeded — the refusal already told you "
"the correct form. Answer as soon as you have what you need.\n"
"</tools>"
)
@@ -204,14 +252,33 @@ def build_high_prompt(
) -> str:
"""Grounded turn: locked persona (+ steering, + KB overview) + full
texts of the top documents + the ``<tools>`` instructions (phase 37;
the phase-70 copy teaches the ``ls`` / ``read`` / ``grep`` shapes).
phase 70: the harness-aligned ``ls`` / ``read`` / ``grep`` shapes;
phase 72: the copy states the document-identity contract — the
source-name ``ls`` scope, the combined ``source/path`` identity for
``read``/``grep`` — up front).
Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>``
→ ``<documents>`` → ``<tools>``; empty steering/overview omit their
section. ``<tools>`` is always present in the HIGH prompt (the round
cap — not the prompt — decides whether the tools are actually
offered to the model, see :mod:`app.rag.agent`).
Gate-iteration note (task 05, 2026-09-03/04): an in-context reminder
LEADING this section (the document texts are already context — do
not ``read`` one the user asked to open) was tried and REVERTED:
it never flipped the seed-doc reads (15/15 across gate runs 1-5)
and correlated with the incident-turn regression (the model latched
the seed documents' paths as ``ls`` scopes — cap reached on the
"list the files in this directory" turn) whenever the copy named
the ``<document>`` blocks explicitly.
"""
# 2026-09-04 (controlled tool-calling fast loop): the do-not-read
# rule for seed documents lives in TOOLS_SECTION and the ``read``
# tool descriptions (the copy levers that stuck — see the gate's
# telemetry in TOOL_CALLING_TESTING.md). A per-block instruction
# attribute at the ``source``/``path`` copy site was TRIED and
# REVERTED the same day (no improvement across runs; the block stays
# exactly the document identity + full text).
blocks = [
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}">\n'
f"{doc.content}\n"