Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Details
Build and Push Containers / build-and-push-db (push) Successful in 14s
Details

Browse Source 
Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled
methodology (owner directive: stop clearing/re-importing the homelab KB per
iteration; measure tool-calling accuracy on a controlled fixture KB, target
>90%).

Real-model gate verdicts (live, configured chat model 'lite', fixture KB):
- Controlled fixture battery (the new methodology's pass condition —
  contract accuracy >= 90%): PASS, 4 consecutive runs:
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s)
- Locked derived battery (phase-72 task 05, executed >= 90% bar, run
  unchanged on the same fixture KB):
  gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s)
  The teaching works — every bare-path trap self-corrects in exactly one
  round, zero cap hits, zero repeat loops, 10/10 answered. The locked
  executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the
  corrected re-reads (the trap question seeds its target, so the correct
  combined-form read is refused for redundancy) — a copy-invariant model
  behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04)
  and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md
  sections 5 and 7), not a copy lever.

Copy changes this phase owns (unit pins updated to follow):
- app/rag/agent.py: ls teaching refusals (path-like scope -> document-path
  line; unknown source -> no-source line with the source-name
  parenthetical), read/grep 'did you mean source/path?' teaching
  (find_path_candidates: exact or suffix path match, catalog order, cap 3),
  ALREADY_IN_CONTEXT naming the correct action (answer from the text
  already in the prompt), read tool description front-loaded with the
  do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is
  the only remaining refusal class; contract accuracy 92-100% across runs)
- app/rag/prompts.py: TOOLS_SECTION states the document-identity contract
  up front (ls path = source name; read/grep = combined source/path
  including the source name; do-not-read for <documents> documents placed
  next to the read teaching; one-call-per-reply and never-repeat rules)
- tests: refusal pins (unit + integration), new dedicated E2E suite
  tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in
  isolation), regression suites green in isolation (harness_aligned_tools,
  agent_document_tools, agent_unlimited_tools, search_tool, chat_rag).

Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff +
pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/
move and both phases' .agent/reports/ (AGENTS.md 8).
This commit is contained in:
ducoterra
2026-09-04 13:11:07 -04:00
parent 7909bdb8da
commit 988ff78526
42 changed files with 2987 additions and 88 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/rag/agent.py
+222 -24
View File
@@ -13,6 +13,20 @@ JSON-block fallback (documented in the task file) is *not* implemented —
it exists only for a "not supported"/"intermittent" verdict, and the
probe came back "supported".
Real-model gate (phase 72, task 05 — live vs the configured chat
model; re-run 2026-09-04 on the controlled fixture KB — see
``TOOL_CALLING_TESTING.md``): the bare-path teaching (did-you-mean
refusals) makes every trap self-correct in exactly one round — zero
cap hits, zero repeat loops. Locked derived battery (phase 72,
executed ≥ 90 % bar): ``gate: lite FAIL turns=10 answered=10 caps=0
tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04
(wall 47.7s)`` — the bar is blocked by :data:`ALREADY_IN_CONTEXT` dedupe
refusals on the corrected re-reads, a copy-invariant model behavior
(five copy variants, 2026-09-03 → 04) and an app-semantics decision
(TOOL_CALLING_TESTING.md §7). Controlled fixture battery (the 2026-09-04
methodology — contract accuracy ≥ 90 %): PASS on three consecutive runs,
``contract 11/11 (100%)`` / ``12/13 (92%)`` / ``11/11 (100%)``.
Loop contract (one grounded chat turn; the API layer wires this in,
task 04):
@@ -58,14 +72,38 @@ task 04):
blank or non-string) → ``"read requires a string argument
'path'."``; a ``grep`` without a usable ``pattern`` (missing, blank
or non-string) → ``"grep requires a string argument
'pattern'."``; a scoped ``ls`` whose ``path`` matches no source name
→ ``"No source named '…' — check the ls output."``; a document
already in context (seed or previously read) → ``"Already in your
context."``; an unknown document (a ``read`` or scoped ``grep`` whose
'pattern'."``; a scoped ``ls`` whose (stripped) ``path`` contains a
``/`` — a document path where a source name belongs (source names
are directory basenames and can never contain one; the 2026-09-03
incident's ``ls(path='app/rag/importer.py')``) →
:data:`LS_PATH_NOT_A_SOURCE`, the document-path teaching line with
the argument echoed; a scoped ``ls`` whose ``path`` names no
registered source (no ``/`` — the incident's ``ls(path='.')``)
→ :data:`NO_SOURCE_NOT_A_DIRECTORY`, the no-source refusal with the
teaching parenthetical appended; a document
already in context (seed or previously read) →
:data:`ALREADY_IN_CONTEXT` (phase 72, task 05 gate iteration:
the line names the correct action — answer from the text already
in the prompt, do not call read again — so a fired refusal ends
the loop instead of inviting a repeat); an unknown document (a ``read`` or scoped ``grep`` whose
combined ``source/path`` matches nothing — a bare source name, which
can never be a document, included) → ``"No document at '…' — check
the ls output."`` with the argument echoed as passed (the model sees
its own form). A grep that ran but found nothing is NOT a rejection
its own form) — EXCEPT the phase-72 "did you mean …?" teaching
(task 02): when the argument is a path (contains ``/``) that matches
an indexed document's ``path`` (exact or as a ``/arg`` suffix,
case-sensitive, catalog order — :func:`find_path_candidates`, a pure
catalog lookup, one bulk query, called only from this refusal path),
the refusal names the combined identity instead — exactly one match
→ :data:`NO_DOCUMENT_DID_YOU_MEAN` (``did you mean
'source/path'?``), two or more → :data:`NO_DOCUMENT_DID_YOU_MEAN_MANY`
(up to :data:`SUGGESTION_LIMIT` identities), so the harness-prior
misuse (the bare document path missing the source prefix,
``read('app/rag/importer.py')``) self-corrects in one round; it is
still a refusal (counts in nothing, consumes a round — no silent
argument normalization), and a bare argument (no ``/``) or a
zero-candidate path keeps the line above byte-identical (the bare
form never hits the DB). A grep that ran but found nothing is NOT a rejection
— its ``"No matches for …"`` line is a (counted) result. A rejected
call still consumes a *round* in the loop, so a pathological stream
that keeps emitting rejected calls is bounded by the cap (point 4).
@@ -164,7 +202,8 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"name": "ls",
"description": (
"List the indexed documents as `source: X | path: Y | "
"title: Z` lines."
"title: Z` lines. Call one tool at a time — wait for "
"this result before your next call."
),
"parameters": {
"type": "object",
@@ -173,8 +212,12 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"type": "string",
"description": (
"Source name to list one source's documents "
"(e.g. 'homelab'); omit to list every "
"document."
"(e.g. 'homelab') — a source name, not a "
"file or directory path; omit to list "
"every document. This is the only tool "
"whose `path` is a source name — for "
"`read` and `grep` it must be a document's "
"combined `source/path`."
),
}
},
@@ -187,8 +230,13 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"function": {
"name": "read",
"description": (
"Add the full content of one indexed document to your "
"context."
"Do not call this tool for a document already shown in "
"the <documents> section, even when the user asks you to "
"open or read it — its full text is already in your "
"prompt; answer directly from it. Use it only to add a "
"document NOT already in <documents> to your context, "
"by its combined `source/path` string. Call one tool at "
"a time — wait for this result before your next call."
),
"parameters": {
"type": "object",
@@ -199,7 +247,12 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"The document to add to your context, as the "
"combined `source/path` string exactly as "
"shown in the `ls` output (e.g. "
"'homelab/active/container_caddy/caddy.md')."
"'homelab/active/container_caddy/caddy.md'). "
"A bare document path (without the source "
"name) will not resolve. Only pass a document "
"NOT already shown in the <documents> "
"section — it is already in your context; do "
"not re-read it."
),
}
},
@@ -215,7 +268,12 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"Search the indexed documents for an exact string "
"(case-insensitive) and return up to 20 matching lines "
"as `source/path:line: text` — a locator, not a "
"context-adder: read the winner with `read`."
"context-adder: read the winner with `read`. For a "
"normal search pass ONLY `pattern` — it searches every "
"document and that is how you search the knowledge "
"base; never pass a source name as `path` (a source "
"name is not a document). Call one tool at a time — "
"wait for this result before your next call."
),
"parameters": {
"type": "object",
@@ -230,10 +288,13 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"path": {
"type": "string",
"description": (
"Limit the search to one document, as a "
"combined `source/path` string from the "
"`ls` output (omit to search every "
"document)."
"Rarely needed — only for re-searching one "
"document you already know: that document's "
"combined `source/path` identity (e.g. "
"'homelab/ansible/inventory.yaml'). Never a "
"source name. A bare document path (without "
"the source name) will not resolve. Omit it "
"for a normal search (pass only `pattern`)."
),
},
},
@@ -245,12 +306,78 @@ AGENT_TOOLS: list[dict[str, Any]] = [
#: Tool refusal texts (phase 37): rejected calls count in nothing
#: (``holder.tool_calls`` tracks executed calls); the round cap bounds
#: their pathological repetition (phase 45).
ALREADY_IN_CONTEXT = "Already in your context."
#: their pathological repetition (phase 45). The in-context line is a
#: phase-72, task 05 gate-iteration teaching (live telemetry: the
#: ``lite`` model obeyed the user's "open it / read it" and re-read
#: seed-context documents, then repeated the call against the terse
#: phase-37 line — the refusal itself carried no correct action): same
#: behavior (a refusal: counts in nothing, consumes a round, changes
#: no context), the copy now names the action, so even a fired
#: refusal ends the loop instead of inviting a repeat.
ALREADY_IN_CONTEXT = (
"Already in your context — the full text is already in your "
"prompt. Do not call read on it again; answer from that text."
)
UNKNOWN_TOOL = "Unknown tool."
MISSING_READ_ARGS = "read requires a string argument 'path'."
MISSING_SEARCH_ARGS = "grep requires a string argument 'pattern'."
#: Teaching refusal for a scoped ``ls`` whose stripped ``path``
#: contains a ``/`` (phase 72): a source name is a directory basename
#: and can never contain one, so the argument is a document path passed
#: where a source name belongs (the 2026-09-03 incident's
#: ``ls(path='app/rag/importer.py')``). One ``{path}`` field — the
#: argument echoed; a fixed template states the correct contract
#: instead of the terse pre-phase-72 line, so the harness-prior misuse
#: self-corrects in one round.
LS_PATH_NOT_A_SOURCE = (
"'{path}' looks like a document path, not a source name. The "
"'path' argument of ls filters by source name (e.g. 'homelab') — "
"omit it to list every document, or read a document by its "
"combined 'source/path' string."
)
#: The no-source ``ls`` refusal with the teaching parenthetical
#: appended (phase 72): used when a stripped scope has no ``/`` and
#: matches no registered source (the incident's ``ls(path='.')``). The
#: prefix — the pre-phase-72 line — stays byte-identical; one ``{scope}``
#: field, the argument echoed.
NO_SOURCE_NOT_A_DIRECTORY = (
"No source named '{scope}' — check the ls output. (The 'path' "
"argument is a source name, not a directory — omit it to list "
"every document.)"
)
#: Teaching refusal for a ``read`` / scoped ``grep`` argument that
#: resolves to no combined identity but matches ONE indexed document's
#: ``path`` (phase 72, task 02): names the exact combined
#: ``source/path`` identity to use, so the harness-prior misuse — the
#: bare document path missing the source prefix
#: (``read('app/rag/importer.py')``) — self-corrects in one round.
#: One each of the fields ``{arg}`` (the argument echoed as passed),
#: ``{source}`` and ``{path}`` (the one candidate). Still a refusal:
#: it counts in nothing and consumes a round (no silent argument
#: normalization).
NO_DOCUMENT_DID_YOU_MEAN = (
"No document at '{arg}' — did you mean '{source}/{path}'?"
)
#: The ambiguous form of the same teaching (phase 72, task 02): the
#: argument matches SEVERAL indexed documents' ``path`` (the same path
#: under several sources). ``{candidates}`` holds up to
#: :data:`SUGGESTION_LIMIT` combined ``source/path`` identities, each
#: single-quoted, joined with ``", "`` in catalog order; ``{arg}`` is
#: the argument echoed as passed.
NO_DOCUMENT_DID_YOU_MEAN_MANY = (
"No document at '{arg}' — did you mean one of: {candidates}?"
)
#: Cap on the suggested combined identities per "did you mean …?"
#: refusal (phase 72, task 02): the same document ``path`` under
#: several sources suggests up to this many (catalog order, the rest
#: dropped).
SUGGESTION_LIMIT = 3
#: The harness-owned recovery line (phase 71, task 03) — folded into the
#: ORIGINAL single system message of the one bounded recovery request
#: (``system_prompt + "\n" + CORRECTION_INSTRUCTION``; provider-safe,
@@ -376,6 +503,61 @@ def all_documents(db: Session) -> list[Document]:
)
def find_path_candidates(db: Session, arg: str) -> list[tuple[str, str, str]]:
"""The indexed documents a bare document *arg* names by ``path``.
Phase 72, task 02: a ``read`` / scoped-``grep`` argument that
resolves to no combined identity but *is* a document path (contains
``/``) is matched against the indexed ``Document.path`` values so
the refusal can name the combined ``source/path`` identity to use
(the "did you mean …?" teaching). The documents whose ``path``
equals *arg* (the exact bare path) or ends with ``f"/{arg}"`` (the
file is nested deeper — the suffix match) — in catalog order (the
:func:`all_documents` order), case-sensitive (these are file
paths) — as ``(source, path, title)`` triples. One bulk query via
:func:`all_documents` (at most one); called ONLY from the refusal
path of :func:`_execute_tool` (never on the happy path) and only
when *arg* contains ``/`` (a bare name keeps today's no-DB-lookup
refusal). Module-level (not a method) so unit tests can
monkeypatch it.
"""
return [
(doc.source, doc.path, doc.title)
for doc in all_documents(db)
if doc.path == arg or doc.path.endswith(f"/{arg}")
]
def _no_document_refusal(db: Session, arg: str) -> str:
"""The no-document refusal for an unresolved ``read`` / scoped-
``grep`` argument (phase 72, task 02).
The pre-phase-72 line — the argument echoed as passed — whenever
there is nothing to suggest: a bare argument (no ``/`` — a bare
source name or any other bare name gets the no-DB-lookup refusal,
byte-identical to today) or a path-like argument that matches no
indexed document's ``path`` (zero candidates). A path-like argument
(contains ``/``) that matches exactly one indexed document's
``path`` gets :data:`NO_DOCUMENT_DID_YOU_MEAN` (the combined
identity named); two or more get :data:`NO_DOCUMENT_DID_YOU_MEAN_MANY`
(up to :data:`SUGGESTION_LIMIT`, catalog order). Deterministic
only: the suggestion is a pure catalog lookup, no model. A refusal
still counts in nothing and consumes a round.
"""
if "/" in arg:
candidates = find_path_candidates(db, arg)
if len(candidates) == 1:
source, path, _title = candidates[0]
return NO_DOCUMENT_DID_YOU_MEAN.format(arg=arg, source=source, path=path)
if len(candidates) > 1:
identities = ", ".join(
f"'{source}/{path}'"
for source, path, _title in candidates[:SUGGESTION_LIMIT]
)
return NO_DOCUMENT_DID_YOU_MEAN_MANY.format(arg=arg, candidates=identities)
return f"No document at '{arg}' — check the ls output."
def grep_document(content: str, pattern: str) -> list[tuple[int, str]]:
"""Every line of *content* that contains *pattern*, in file order.
@@ -437,8 +619,19 @@ def _execute_tool(
scope = raw_path.strip() if isinstance(raw_path, str) else ""
rows = list_catalog(db)
if scope:
if "/" in scope:
# A source name (a directory basename) can never
# contain '/' — this is a document path where a source
# name belongs (phase 72): teach the contract; no
# registry lookup needed, counts in nothing, consumes
# a round like every refusal.
return LS_PATH_NOT_A_SOURCE.format(path=scope)
if scope not in list_source_names(db):
return f"No source named '{scope}' — check the ls output."
# The no-source refusal with the teaching parenthetical
# (phase 72) — the prefix byte-identical to the
# pre-phase-72 line; counts in nothing, consumes a
# round like every refusal.
return NO_SOURCE_NOT_A_DIRECTORY.format(scope=scope)
rows = [row for row in rows if row[0] == scope]
listing = f"{len(rows)} documents:\n" + "\n".join(
f"source: {source} | path: {path} | title: {title}"
@@ -462,8 +655,10 @@ def _execute_tool(
doc, _source, _path = _resolve_path(db, arg)
if doc is None:
# Echo the argument as passed — the model sees its own form
# (a bare source name can never be a document, no DB lookup).
return f"No document at '{arg}' — check the ls output."
# (a bare argument can never be a document, no DB lookup);
# a path-like argument that matches an indexed document's
# path gets the "did you mean …?" teaching (phase 72).
return _no_document_refusal(db, arg)
holder.read_docs.append(doc)
holder.tool_calls += 1
return f"Document {doc.source}/{doc.path}:\n{doc.content}"
@@ -478,7 +673,9 @@ def _execute_tool(
if scope:
target, src, p = _resolve_path(db, scope)
if target is None:
return f"No document at '{scope}' — check the ls output."
# The same phase-72 "did you mean …?" teaching as the
# read branch (a refusal — not counted, no context).
return _no_document_refusal(db, scope)
docs: list[Document] = [target]
scoped_to = (src, p) # the resolved (canonical) identity
else:
@@ -552,8 +749,9 @@ async def run_agent(
``seed_docs`` are the documents the retrieval already put in context
(they shape the *system_prompt* the caller built); re-reading one of
them is rejected as "Already in your context." — the rejection counts
in nothing, but it still consumes a round.
them is rejected with :data:`ALREADY_IN_CONTEXT` (the phase-72
teaching line — answer from the text already in the prompt) — the
rejection counts in nothing, but it still consumes a round.
"""
messages: list[dict[str, Any]] = [
{"role": "system", "content": system_prompt},