Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 118_summary_seed_context
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Details
Build and Push Containers / build-and-push-db (push) Successful in 14s
Details

Browse Source 
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.

- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
ducoterra
2026-09-16 06:57:49 -04:00
parent 21aad84a6d
commit 9820c361b0
80 changed files with 4690 additions and 1302 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/mock_llm.py
+211 -46
View File
@@ -55,14 +55,20 @@ Implements just enough of the aipi surface:
- system prompt containing ``<knowledge_base>`` (phase 31, KB overview)
-> the composed answer ends with `` (kb: <first bullet line>)`` —
the same echo convention for the overview's prompt injection.
- user message containing ``show the end of your notes`` (phase 24,
whole-document context) -> the answer quotes the **last 160 chars of
the ``<documents>`` block** — a tail echo, byte-stable across runs, so
a sentinel placed at the *end* of a document appears in the rendered
answer iff the whole document was in the prompt. (Phase 37: the HIGH
prompt now ends with a ``<tools>`` section after ``</documents>``, so
the echo targets the block itself; its tail still includes the
closing tag — same sentinel semantics.)
- user message containing ``show the end of your notes`` (phase 24;
phase 118 re-targeted — the story's dedicated suite
``tests/e2e/test_summary_seed_context.py``) -> the answer quotes
the **last 160 chars of the ``<documents>`` block** — a tail echo,
byte-stable across runs. (Phase 37: the HIGH prompt now ends with
a ``<tools>`` section after ``</documents>``, so the echo targets
the block itself; its tail still includes the closing tag.)
Phase 118 (A6): the block carries the suggested documents'
SUMMARIES (never full texts), so the echoed tail is the LAST
suggested document's SUMMARY tail (its digest + the
``Source: <source>/<path>`` pointer line) — a sentinel on a
document's *last line* appears in the rendered answer iff the
FULL content (not the summary) was in the prompt, which under the
summary-seed contract is only through a ``read`` tool result.
- user message containing ``use your tools`` (phase 37, agent document
tools; phase 70: the flow emits the harness-aligned names — ``ls``
/ ``read`` with the combined ``source/path`` identity; phase 94:
@@ -235,16 +241,19 @@ Implements just enough of the aipi surface:
content>`` (the phase-37 single-read shape — the grounded-
turn citation contract);
* the last tool result is the agent's ALREADY_IN_CONTEXT dedupe
refusal (the read target is already a top-2 retrieval
document — with the drill fixture that is DETERMINISTIC:
the read question names the file's path, so the file
self-matches the hybrid gate and its FULL text is in the
``<documents>`` prompt): the model answers FROM THE PROMPT —
the deterministic answer ``Already in context: Read
<source/path>. <first 80 chars of the target document's text
as it appears in the ``<documents>`` block>`` (same citation
shape as the read-result branch — the document text reached
the model either way, and the answer proves it);
refusal (phase 118: the read target is a document ALREADY
READ into full-text context earlier in the same turn — the
seeds are summaries, so a first read of any document
succeeds and only a re-read is refused): the model answers
FROM THE PROMPT — the deterministic answer ``Already in
context: Read <source/path>. <first 80 chars of the target
document's text as it appears in the ``<documents>`` block>``
(same citation shape as the read-result branch — the document
text reached the model either way, and the answer proves it;
the block it quotes now carries the document's SUMMARY).
No suite exercises this branch today (the drill questions
read once per turn) — it is kept for the still-real
already-read refusal;
* any other last result (a top-level or folder LISTING landed):
the deterministic ECHO answer ``Here's the level I listed:\n
<the listing, verbatim>`` — the mock echoes what it received
@@ -287,6 +296,38 @@ Implements just enough of the aipi surface:
needs the ``<tools>`` section, so deflected turns never hit it);
verified 2026-09-10: no existing E2E question or fixture file
contains the phrase, so every other suite is unaffected.
- user message containing ``read the suggested document``
(``SUMMARY_SEED_READ_TRIGGER``, phase 118 task 06 — the
summary-seed context's dedicated story suite
``tests/e2e/test_summary_seed_context.py``) **and** the system
prompt carries the ``<tools>`` section -> the deterministic
SCRIPTED SUMMARY-READ flow: the question carries its own tool call
after the colon — ``read the suggested document: read source/path``
— parsed by ``_SUMMARY_SEED_READ_CALL_RE`` from the RAW user
message (the target keeps its case), then discriminated
statelessly from the tool results (streaming only):
* request 1 (``tools`` offered, no ``tool``-role result in the
messages yet): the scripted call — ``read`` with the parsed
target (synthetic id ``call_0``);
* a ``tool``-role result is in the messages: the deterministic
ECHO — the answer carries the LAST tool result VERBATIM
(``Here's what the read returned:\n<result>``): under the
phase-118 summary-seed contract a first ``read`` of ANY
document succeeds (the seeds are summaries, not full text), so
a read result (``"Document <source/path>:…`` — header + the
phase-106 D5 ``date:`` line + the FULL content) lands in the
answer with its tail intact — the story suite's lens on the
full text the ``read`` tool delivered (a tail sentinel on the
document's last line appears in the answer iff the full content
reached the model through the read, not the seed); a refusal
(the premise broke) lands just as visibly, so the suite fails
loudly on it. The mock is the only E2E lens on the LLM's
context, so the echo is the assertion surface.
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
phrases — the phase-71/72/94 ordering convention; the trigger
needs the ``<tools>`` section, so deflected turns never hit it);
no existing E2E question or fixture file contains the phrase, so
every other suite is unaffected.
- user message containing ``what are the correct llama.cpp
arguments`` (``GREP_TEACH_TRIGGER``, the 2026-09-05 incident —
the harness prior is that grep takes a REGEX; this app's grep is a
@@ -802,10 +843,12 @@ _DRILL_CALL_RE = re.compile(
_NOT_A_FOLDER_MARKER = "is not a folder"
#: The stable substring of the harness-owned dedupe refusal the
#: drill-down flow's ``ctx_answer`` branch keys on (a read target that
#: is already a top-2 retrieval document — the mock then answers from
#: the document's text in the ``<documents>`` prompt block, exactly
#: what the refusal instructs). Keyed on a substring (not the whole
#: drill-down flow's ``ctx_answer`` branch keys on (phase 118: a read
#: target ALREADY READ into full-text context in the same turn — the
#: seeds are summaries, so a first read of any document succeeds and
#: only a re-read is refused; the mock then answers from the document's
#: summary in the ``<documents>`` prompt block, exactly what the
#: refusal instructs). Keyed on a substring (not the whole
#: constant) so a re-wrap of the constant cannot silently re-route the
#: mock; the module-level assert below fails loudly if the substring
#: ever leaves the constant (the mock must never drift from
@@ -819,11 +862,13 @@ assert _ALREADY_IN_CONTEXT_MARKER in ALREADY_IN_CONTEXT, (
#: section (``app.rag.prompts.build_high_prompt``): the block is the
#: document identity (``source``/``path``/``title`` attributes — plus,
#: since phase 106 D5, the ``date`` attribute, the row's ``created_at``
#: UTC date part, APPENDED after ``title``) plus the document's FULL
#: text (never truncated on the retrieval path, owner-locked A7)
#: between the tags. The ``date`` group is OPTIONAL so the mock
#: tolerates the pre- and post-phase block shapes (house rule: the
#: marker/regex lands with the prompt change).
#: UTC date part, APPENDED after ``title``) plus the document's
#: SUMMARY (phase 118, A6 — the summary-seed contract: the seeded
#: blocks are summaries, never full text; full text enters the context
#: only through the capped ``read`` tool) between the tags. The
#: ``date`` group is OPTIONAL so the mock tolerates the pre- and
#: post-phase block shapes (house rule: the marker/regex lands with
#: the prompt change).
_DOCUMENT_BLOCK_RE = re.compile(
r'<document source="(?P<source>[^"]+)" path="(?P<path>[^"]+)" '
r'title="[^"]*"(\sdate="[^"]*")?>\n(?P<content>.*?)\n</document>',
@@ -835,10 +880,14 @@ def _document_block(system: str, source: str, path: str) -> str | None:
"""The stored text of one ``<document>`` block (or ``None``).
The drill-down flow's ``ctx_answer`` branch: when the agent's read
of a top-2 retrieval document gets the ALREADY_IN_CONTEXT dedupe,
the document's full text is in the ``<documents>`` prompt — the
mock (the model) extracts it by the block's identity attributes
and quotes it, answering from the prompt as the refusal instructs.
of a document ALREADY READ in the same turn gets the
ALREADY_IN_CONTEXT dedupe (phase 118 — the seeds are summaries, so
a first read of any document succeeds and only a re-read is
refused), the mock (the model) extracts the block's text by the
identity attributes and quotes it, answering from the prompt as
the refusal instructs. The block carries the document's SUMMARY
(phase 118 A6) — no suite exercises the branch today; it is kept
for the still-real already-read refusal.
"""
for block in _DOCUMENT_BLOCK_RE.finditer(system):
if block.group("source") == source and block.group("path") == path:
@@ -1377,16 +1426,22 @@ def _drill_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
last = results[-1]
if last.startswith(_READ_RESULT_PREFIX):
head, _, content = last.partition("\n")
# The read result is ``"Document <source/path>:\n<content>"`` —
# the head carries the server's appended ``:`` (removed here;
# a document path never legitimately ends with one).
# The read result is ``"Document <source/path>:\ndate: …\n<content>"``
# — the head carries the server's appended ``:`` (removed here;
# a document path never legitimately ends with one); the
# phase-106 D5 ``date:`` line sits between the header and the
# document text — skipped so the quote stays pure document
# content (the suite's byte-identical pin).
combined = head[len(_READ_RESULT_PREFIX):].strip().removesuffix(":")
if content.startswith("date: "):
content = content.partition("\n")[2]
return ("read_answer", combined, content[:80])
if _ALREADY_IN_CONTEXT_MARKER in last and verb == "read" and "/" in target:
# The dedupe fired: the read target is already a top-2
# retrieval document, so its FULL text is in the
# ``<documents>`` prompt — answer from the prompt (the
# refusal's instruction), quoting the block's text.
# Phase 118: the dedupe fires only for a document ALREADY READ
# in the same turn (the seeds are summaries — a first read of
# a suggested document succeeds). Answer from the prompt (the
# refusal's instruction), quoting the block's text (now the
# document's summary).
src, _, p = target.partition("/")
content = _document_block(_system(body), src, p)
if content is not None:
@@ -1469,6 +1524,81 @@ def _read_cap_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
return ("echo", results[-1])
# ---------------------------------------------------------------------------
# Phase 118 (task 06, the summary-seed context's dedicated story suite):
# the deterministic SCRIPTED summary read — see the module docstring
# ---------------------------------------------------------------------------
#: A user message containing this substring (case-insensitive) —
#: combined with the ``<tools>`` section in the system prompt — drives
#: the scripted SUMMARY-READ flow (the summary-seed context's story
#: suite, ``tests/e2e/test_summary_seed_context.py``): the question
#: carries its own tool call after the colon — ``read the suggested
#: document: read source/path`` — the mock emits the scripted ``read``
#: (phase 118: the seeds are summaries, so a first read of a suggested
#: document SUCCEEDS — the full text arrives through the read), then
#: ECHOES the ENTIRE tool result into its answer (the house
#: scripted-turn lens on the LLM's context — the full content's tail
#: reaches the rendered answer iff the read delivered it). Checked
#: BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger phrases —
#: the phase-71/72/94 ordering convention); no existing E2E question
#: or fixture file contains the phrase, so every other suite is
#: unaffected.
SUMMARY_SEED_READ_TRIGGER = "read the suggested document"
#: The scripted call in the summary-read question (case-insensitive —
#: the suite's questions capitalize the trigger's first letter): the
#: verb (``read``) plus the target — a combined ``source/path``, parsed
#: from the RAW user message so the target keeps its case. The target
#: is a ``[a-z0-9_./-]`` run (case-insensitively), so the suite's
#: `` — `` flavor separator (em dash) can never bleed into it (the
#: read-cap/drill convention, ``_READ_CAP_CALL_RE`` / ``_DRILL_CALL_RE``).
_SUMMARY_SEED_READ_CALL_RE = re.compile(
r"read the suggested document:\s*read\s+(?P<arg>[a-z0-9_./-]+)",
re.I,
)
def _summary_seed_read_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
"""Classify a phase-118 scripted summary-read request (see the
module docstring). The question carries the scripted call (``read
the suggested document: read source/path``); the step is then
discriminated statelessly from the tool results, like the other
marker flows:
* ``("call", target, "call_0")`` — ``tools`` are offered and no
``tool``-role result is in the messages yet: the scripted
``read`` on the parsed target (synthetic id ``call_0``).
* ``("echo", result)`` — a ``tool``-role result is in the messages:
the deterministic ECHO — the answer carries the LAST tool result
VERBATIM (``Here's what the read returned:\n<result>``): a read
result (``"Document <source/path>:…`` — header + date line +
FULL content) lands in the answer with its tail intact (the
full text the ``read`` tool delivered — the phase-118 contract:
a first read of a suggested document succeeds, the seed was a
summary); a refusal (the premise broke) lands just as visibly,
so the suite fails loudly on it.
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
section is missing (deflected turns never carry it), the scripted
call is unparseable, or ``tools`` are not offered and no tool
results are in the messages yet (e.g. ``agent_max_rounds=0``).
"""
user = _user(body)
if SUMMARY_SEED_READ_TRIGGER not in user.lower():
return None
if "<tools>" not in _system(body):
return None
match = _SUMMARY_SEED_READ_CALL_RE.search(user)
if match is None:
return None
results = _tool_results(body)
if not results:
if not body.get("tools"):
return None
return ("call", match.group("arg"), "call_0")
return ("echo", results[-1])
def long_answer() -> str:
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
a unique final line that must survive the stream untruncated."""
@@ -1924,14 +2054,19 @@ def compose_answer(body: dict[str, Any]) -> str:
"You've got this!"
)
elif END_OF_NOTES_TRIGGER in user.lower():
# Whole-document-context story (phase 24): echo the tail of the
# document context. Byte-stable across runs — a sentinel on the
# document's last line appears in the answer iff the whole
# document was in the prompt. (The tail includes the closing
# </documents> — harmless for the E2E sentinel assertions.)
# Phase 37: the HIGH prompt now ends with the <tools> section
# after </documents>, so the echo targets the <documents> block
# itself — the sentinel semantics are unchanged.
# Phase 24 (whole-document context) — phase 118 re-targeted
# (the summary-seed context's story suite): echo the tail of
# the <documents> block. Byte-stable across runs — under the
# phase-118 summary-seed contract the block carries the
# suggested docs' SUMMARIES, so the echoed tail is the LAST
# suggested doc's summary tail (digest + Source: pointer line),
# and a sentinel on a document's last line appears in the
# answer only if the FULL content reached the model (through a
# read tool result — never the seed). (The tail includes the
# closing </documents> — harmless for the E2E sentinel
# assertions.) Phase 37: the HIGH prompt ends with the <tools>
# section after </documents>, so the echo targets the
# <documents> block itself.
block = _DOCUMENTS_BLOCK_RE.search(_system(body))
tail_source = block.group(0) if block else _context(body)
answer = (
@@ -2461,6 +2596,36 @@ def chat_completions(body: dict[str, Any]) -> Any:
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
# Phase 118 (task 06): the deterministic SCRIPTED summary read
# (the question carries its own call — ``read the suggested
# document: read source/path``): the scripted ``read`` (phase
# 118: a first read of a suggested document succeeds — the
# seeds are summaries), then the answer that ECHOES the whole
# tool result (the full text's tail reaches the rendered answer
# iff the read delivered it — the suite's lens on the LLM's
# context). Checked BEFORE the plain TOOLS_TRIGGER flow
# (disjoint trigger phrases — the phase-71/72/94 ordering
# convention; the trigger needs the ``<tools>`` section, so
# deflected turns never hit it).
seed_read = _summary_seed_read_flow(body)
if seed_read is not None:
if seed_read[0] == "call":
stream = _tool_call_stream(
"read", {"path": seed_read[1]}, seed_read[2]
)
else: # "echo" — the last tool result verbatim (the lens)
stream = _sse_stream(
_apply_max_tokens(
f"Here's what the read returned:\n{seed_read[1]}",
body.get("max_tokens"),
),
0.0,
)
return StreamingResponse(
stream,
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
# Phase 109 (task 03, never-frozen-turn story suite): the
# deterministic REPORTED-REPRO turn (delta → tool →
# thinking-after-delta — TODO.md L3): the scripted first answer,