Experimental/brain-of-reese

Fork 0

Files

T

ducoterra 988ff78526

Build and Push Containers / build-and-push-app (push) Successful in 1m51s

Details

Build and Push Containers / build-and-push-db (push) Successful in 14s

Details

fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling

Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled
methodology (owner directive: stop clearing/re-importing the homelab KB per
iteration; measure tool-calling accuracy on a controlled fixture KB, target
>90%).

Real-model gate verdicts (live, configured chat model 'lite', fixture KB):
- Controlled fixture battery (the new methodology's pass condition —
  contract accuracy >= 90%): PASS, 4 consecutive runs:
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s)
- Locked derived battery (phase-72 task 05, executed >= 90% bar, run
  unchanged on the same fixture KB):
  gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s)
  The teaching works — every bare-path trap self-corrects in exactly one
  round, zero cap hits, zero repeat loops, 10/10 answered. The locked
  executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the
  corrected re-reads (the trap question seeds its target, so the correct
  combined-form read is refused for redundancy) — a copy-invariant model
  behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04)
  and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md
  sections 5 and 7), not a copy lever.

Copy changes this phase owns (unit pins updated to follow):
- app/rag/agent.py: ls teaching refusals (path-like scope -> document-path
  line; unknown source -> no-source line with the source-name
  parenthetical), read/grep 'did you mean source/path?' teaching
  (find_path_candidates: exact or suffix path match, catalog order, cap 3),
  ALREADY_IN_CONTEXT naming the correct action (answer from the text
  already in the prompt), read tool description front-loaded with the
  do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is
  the only remaining refusal class; contract accuracy 92-100% across runs)
- app/rag/prompts.py: TOOLS_SECTION states the document-identity contract
  up front (ls path = source name; read/grep = combined source/path
  including the source name; do-not-read for <documents> documents placed
  next to the read teaching; one-call-per-reply and never-repeat rules)
- tests: refusal pins (unit + integration), new dedicated E2E suite
  tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in
  isolation), regression suites green in isolation (harness_aligned_tools,
  agent_document_tools, agent_unlimited_tools, search_tool, chat_rag).

Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff +
pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/
move and both phases' .agent/reports/ (AGENTS.md 8).

2026-09-04 13:11:07 -04:00

4.6 KiB

Raw Blame History

Task 05 — Mock Triggers, Dedicated E2E Suite, Gates, Commit

Phase: 71_scaffolding_guardrails · Story: n/a (owner request from chat, 2026-09-03)

Objective

Prove the guardrail end-to-end through the real UI: a mock-LLM scaffolding flow (recovery case + terminal case), a dedicated Playwright suite pinning that raw tokens never reach the DOM, then the full quality gates and the phase commit.

Work

tests/e2e/mock_llm.py — two new deterministic flows (checked in the flow table before the plain TOOLS_TRIGGER flow, after SEARCH_TRIGGER ordering rules as they fit — the triggers are independent of the <tools> marker, so both grounded and deflected turns hit them):
- SCAFFOLD_TRIGGER = "emit raw tool markup" — request 1 (no correction in the system prompt): stream ONLY delta.content chunks carrying the incident text <|tool_call_start|>[read(path='search_docs/reese-notes.md')]<|tool_call_end|> (split across ≥2 chunks to exercise the boundary path), finish_reason: "stop", no structured tool_calls, no reasoning. Request 2 (system prompt contains the stable substring of CORRECTION_INSTRUCTION — import it from app.rag.agent so the mock can never drift from the constant): stream a clean plain answer ("Here is the plain-text answer the recovery produced.")
  - finish_reason: "stop".
- SCAFFOLD_ALWAYS_TRIGGER = "always emit raw tool markup" — every request (recovery included): the same scaffolding-only stream, forever.
- Update the module docstring's flow table + the phase-71 note.
tests/e2e/test_tool_scaffolding_guardrails.py (NEW — the phase's dedicated suite, house pattern, run in isolation; DB up, mock LLM):
- Recovery case — ask a question containing SCAFFOLD_TRIGGER: the turn settles (the composer re-enables, done observed); the final answer bubble contains the recovery's clean text; document.body.innerText contains neither tool_call_start nor tool_call_end (nor the raw [read(path=…] fragment); no error banner.
- Terminal case — ask a question containing SCAFFOLD_ALWAYS_TRIGGER: the existing error status renders with the dedicated copy ("The model returned a malformed reply — please try again."); no raw tokens in the DOM; no answer bubble with the scaffolding; the app stays usable — a follow-up plain question (no trigger) gets a normal streamed answer in the same session.
- No false positive — a plain question (existing CHAT_TRIGGER-style flow, no tools needed): the answer streams byte-clean, no error state, no recovery request visible (the turn settles on the first request).
- The SSE wire itself: in the recovery case, no delta frame ever carries a tool_call_start/tool_call_end fragment (the existing SSE-capture house pattern) — the strip happens server-side, not in the UI.
Gates + commit:
- uv run pytest green; uv run pytest --cov=app --cov-report=term-missing TOTAL >90%; uv run ruff check . && uv run pyright clean.
- E2E in isolation (DB up): test_tool_scaffolding_guardrails.py, then the regression suites test_harness_aligned_tools.py, test_chat_rag.py, test_agent_document_tools.py.
- One atomic commit (message below); move .agent/phases/todo/71_scaffolding_guardrails/ → .agent/phases/complete/71_scaffolding_guardrails/.

Testing & Quality

E2E: the new dedicated suite (recovery / terminal / no-false-positive / wire pins) + the three regression suites (isolation runs).
Coverage: >90% on app/ (phase-level gate).

Completion Criteria

rg "tool_call_start|tool_call_end" frontend/ → no matches.
Raw scaffolding is never visible in the DOM in any case (recovery, terminal, mixed) — pinned by the dedicated suite.
Exactly one recovery per malformed turn (mock request counts implied by the flows); clean turns never carry the correction line (pinned by the unit suites from task 03).
uv run pytest green; uv run pytest --cov=app TOTAL >90%; uv run ruff check . && uv run pyright clean.
uv run pytest tests/e2e/test_tool_scaffolding_guardrails.py -v --no-cov green in isolation; regression suites green in isolation.
One --no-gpg-sign commit; phase dir moved to .agent/phases/complete/.

Commit

git add -A .agent/ app/ tests/ frontend/ && git commit --no-gpg-sign -m "feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery"

4.6 KiB Raw Blame History