phase: 118_summary_seed_context
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.
- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
+99
-41
@@ -7,7 +7,14 @@ no mandated deflection opening; the honesty gate itself is unchanged.)
|
||||
|
||||
Two modes:
|
||||
|
||||
* ``HIGH`` — grounded turn: full top-document texts under ``<documents>``.
|
||||
* ``HIGH`` — grounded turn: the top-ranked documents' SUMMARIES under
|
||||
``<documents>`` (phase 118, LOCKED A6 re-revising A7 — the "start here"
|
||||
suggestion seeding: each block is the document's stored summary, never
|
||||
the full content; the LLM extends its context by ``read``-ing a
|
||||
document's full text through the capped ``read`` tool, the ONLY full-
|
||||
text path. A NULL/blank summary — a fail-soft import miss — falls back
|
||||
to a ``suggestion_preview_chars`` content preview + the shared
|
||||
``[…truncated…]`` marker; no LLM call at chat time, LOCKED A5).
|
||||
* ``LOW`` — deflection turn: weak-hit *titles only* plus the
|
||||
``DEFLECT_MODE`` marker (the E2E mock LLM keys on that marker).
|
||||
|
||||
@@ -124,6 +131,24 @@ _KB_INTRO = (
|
||||
"(generated at import time):\n"
|
||||
)
|
||||
|
||||
#: The ``<documents>`` section's start-here intro (phase 118, task 03 —
|
||||
#: the owner directive, TODO L3): the seeded blocks are the SUMMARIES of
|
||||
#: the top-ranked documents for the question, opt-in starting points —
|
||||
#: start here if one seems right; ``read`` the document's combined
|
||||
#: ``source/path`` to add its FULL content before answering specifics
|
||||
#: (the full text is NOT in the prompt until you read it); cite the
|
||||
#: document(s) you used, by path. Rendered BEFORE the first block, only
|
||||
#: when at least one block is present. The ``<documents>`` tag and the
|
||||
#: per-block ``<document>`` markup stay byte-stable around it (the E2E
|
||||
#: mock's block parser and the ``read``-tool copy key off both).
|
||||
SUGGEST_INTRO = (
|
||||
"The blocks below are the summaries of the top-ranked documents for "
|
||||
"your question — start here if one seems right to you: call `read` "
|
||||
"with that document's combined `source/path` to add its full content "
|
||||
"before answering specifics (its full text is not in the prompt until "
|
||||
"you read it). Cite the document(s) you used, by path."
|
||||
)
|
||||
|
||||
#: The ``<tools>`` instructions section — **HIGH prompt only** (phase 37,
|
||||
#: task 03; phase 70: the copy is rewritten for the harness-aligned
|
||||
#: ``ls`` / ``read`` / ``grep`` shapes, names/args exactly as the
|
||||
@@ -158,12 +183,15 @@ _KB_INTRO = (
|
||||
#: the user's "open it / read it" and ``read`` seed-context documents,
|
||||
#: then repeated the refused call; run 2: 8/18, 44% — the repeat is
|
||||
#: gone, but a grep turn batched two calls per reply and the harness
|
||||
#: runs only the first of a batch): the do-not-read rule names the
|
||||
#: user-command scenario (a document already in the ``<documents>``
|
||||
#: section: do not call ``read``, answer from the text already in the
|
||||
#: prompt — an anchor on the concrete ``<document path="...">`` markup
|
||||
#: was tried and REVERTED: it primed the model to latch the seed
|
||||
#: documents' paths as ``ls`` scopes, regressing the incident turn);
|
||||
#: runs only the first of a batch): the ``read`` clause carries the
|
||||
#: phase-118 summary-seed contract (the ``<documents>`` section holds
|
||||
#: SUMMARIES — a suggested document's full text is not in the prompt
|
||||
#: until ``read`` adds it; do not re-read a document you have already
|
||||
#: read — answer from the text already in the prompt; if the user
|
||||
#: asks to open or read a suggested document, ``read`` it — that is
|
||||
#: the point of the section — the phase-72 do-not-read rule and the
|
||||
#: reverted ``<document path="...">`` anchor are retired with the
|
||||
#: full-text seeds);
|
||||
#: the one-call rule names the consequence (a batched second call is
|
||||
#: discarded — runs only the first); the never-repeat rule says why
|
||||
#: (the refusal already told you the correct form); the ``grep`` clause
|
||||
@@ -171,10 +199,11 @@ _KB_INTRO = (
|
||||
#: live runs showed the model scoping ``grep`` with an ``ls``-style
|
||||
#: source name — the incident shape, but on grep). The behavioral
|
||||
#: contract lives in the ``AGENT_TOOLS`` descriptions as well (the most
|
||||
#: local text at call time): ``read`` must not be called for a
|
||||
#: ``<documents>`` document at all; ``grep`` with only ``pattern``
|
||||
#: searches the whole knowledge base, and a source name is not a
|
||||
#: document.
|
||||
#: local text at call time): ``read`` adds a document's FULL text by
|
||||
#: its combined ``source/path`` identity — the ``<documents>``
|
||||
#: summaries are the starting points, not the content; ``grep`` with
|
||||
#: only ``pattern`` searches the whole knowledge base, and a source
|
||||
#: name is not a document.
|
||||
TOOLS_SECTION: str = (
|
||||
"<tools>\n"
|
||||
"You may extend your context with three tools. `ls` lists the "
|
||||
@@ -189,12 +218,16 @@ TOOLS_SECTION: str = (
|
||||
"`source: X | path: Y | title: Z`; to find one specific document "
|
||||
"without listing, use `grep`. `read` pulls in one document by its "
|
||||
"combined `source/path` string, exactly as shown in the `ls` "
|
||||
"output — including the source name — adding its full content to "
|
||||
"your context. Do not call `read` for a document already shown in "
|
||||
"the <documents> section, even when the user asks you to open or "
|
||||
"read it — its full text is already in your prompt; answer "
|
||||
"directly from it. For `read`, a bare document path (without the "
|
||||
"source name) will not resolve. Very large documents are capped: a "
|
||||
"output — including the source name — or in the <documents> "
|
||||
"summary blocks — adding its full content to your context. The "
|
||||
"<documents> section holds SUMMARIES — the full text of a "
|
||||
"suggested document is not in your prompt until you `read` it. Do "
|
||||
"not re-read a document you have already read — its full text is "
|
||||
"already in your prompt; answer directly from it. If the user "
|
||||
"asks you to open or read a suggested document, `read` it — that "
|
||||
"is the point of the section. For `read`, a bare document path "
|
||||
"(without the source name) will not resolve. Very large documents "
|
||||
"are capped: a "
|
||||
"cut read returns the first part plus a TRUNCATED notice — the "
|
||||
"document did not end where it stopped; use `grep` (pattern) to "
|
||||
"find the rest, it searches the whole document. `grep` locates an exact string "
|
||||
@@ -355,17 +388,42 @@ def build_kb_section(overview: str, max_chars: int | None = None) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _document_body(doc: Document) -> str:
|
||||
"""The body of one ``<document>`` suggestion block (phase 118, LOCKED
|
||||
A6): the document's stored summary (stripped) — NEVER the full
|
||||
content.
|
||||
|
||||
Defensive fallback ONLY when the summary is missing (``None`` or
|
||||
whitespace — a fail-soft import miss, LOCKED A5): the first
|
||||
``suggestion_preview_chars`` characters of the content plus the
|
||||
shared :data:`TRUNCATION_MARKER` on its own line — the settings read
|
||||
happens on this fallback path ONLY (a prompt built from
|
||||
summary-bearing docs makes no settings call for the cap). Content at
|
||||
or under the cap rides whole, unmarked (nothing was cut). No LLM
|
||||
call at chat time — the preview is deterministic.
|
||||
"""
|
||||
summary = (doc.summary or "").strip()
|
||||
if summary:
|
||||
return summary
|
||||
limit = get_settings().suggestion_preview_chars
|
||||
content = doc.content
|
||||
if len(content) > limit:
|
||||
return content[:limit] + "\n" + TRUNCATION_MARKER
|
||||
return content
|
||||
|
||||
|
||||
def build_high_prompt(
|
||||
documents: Sequence[Document],
|
||||
notes: Sequence[str] | None = None,
|
||||
kb_overview: str | None = None,
|
||||
) -> str:
|
||||
"""Grounded turn: locked persona (+ steering, + KB overview) + full
|
||||
texts of the top documents + the ``<tools>`` instructions (phase 37;
|
||||
phase 70: the harness-aligned ``ls`` / ``read`` / ``grep`` shapes;
|
||||
phase 72: the copy states the document-identity contract — the
|
||||
source-name ``ls`` scope, the combined ``source/path`` identity for
|
||||
``read``/``grep`` — up front).
|
||||
"""Grounded turn: locked persona (+ steering, + KB overview) + the
|
||||
top-ranked documents' SUMMARY blocks + the ``<tools>`` instructions
|
||||
(phase 37; phase 70: the harness-aligned ``ls`` / ``read`` /
|
||||
``grep`` shapes; phase 72: the document-identity contract up front;
|
||||
phase 118, LOCKED A6 re-revising A7: the ``<documents>`` section
|
||||
seeds SUMMARIES, never full texts — full text enters the context
|
||||
only through the capped ``read`` tool).
|
||||
|
||||
Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>``
|
||||
→ ``<documents>`` → ``<tools>``; empty steering/overview omit their
|
||||
@@ -373,42 +431,42 @@ source-name ``ls`` scope, the combined ``source/path`` identity for
|
||||
cap — not the prompt — decides whether the tools are actually
|
||||
offered to the model, see :mod:`app.rag.agent`).
|
||||
|
||||
When at least one block is present, the section leads with the
|
||||
start-here :data:`SUGGEST_INTRO` line (before the first block — the
|
||||
phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent): the
|
||||
blocks are the summaries of the top-ranked documents, ``read`` adds
|
||||
the full text, and the answer cites the document(s) used by path.
|
||||
This is NOT the reverted phase-72 in-context reminder (that copy
|
||||
taught the seed documents as already-read context; this one teaches
|
||||
the summary-as-starting-point contract the A6 re-revision requires).
|
||||
|
||||
Each ``<document>`` block carries the identity attributes
|
||||
``source`` / ``path`` / ``title`` — and, since phase 106 (D5),
|
||||
``date`` (the row's ``created_at`` UTC date part, ``YYYY-MM-DD``,
|
||||
APPENDED after ``title`` — the only position; always present,
|
||||
``created_at`` is NOT NULL) — plus the document's full text.
|
||||
|
||||
Gate-iteration note (task 05, 2026-09-03/04): an in-context reminder
|
||||
LEADING this section (the document texts are already context — do
|
||||
not ``read`` one the user asked to open) was tried and REVERTED:
|
||||
it never flipped the seed-doc reads (15/15 across gate runs 1-5)
|
||||
and correlated with the incident-turn regression (the model latched
|
||||
the seed documents' paths as ``ls`` scopes — cap reached on the
|
||||
"list the files in this directory" turn) whenever the copy named
|
||||
the ``<document>`` blocks explicitly.
|
||||
``created_at`` is NOT NULL) — plus the document's summary
|
||||
(:func:`_document_body`; the NULL/whitespace-summary preview
|
||||
fallback is LOCKED A5).
|
||||
"""
|
||||
# 2026-09-04 (controlled tool-calling fast loop): the do-not-read
|
||||
# rule for seed documents lives in TOOLS_SECTION and the ``read``
|
||||
# tool descriptions (the copy levers that stuck — see the gate's
|
||||
# telemetry in TOOL_CALLING_TESTING.md). A per-block instruction
|
||||
# attribute at the ``source``/``path`` copy site was TRIED and
|
||||
# REVERTED the same day (no improvement across runs; the block stays
|
||||
# exactly the document identity + full text).
|
||||
# Phase 106 (D5): every document the model sees carries its
|
||||
# creation date — the block's ``date`` attribute (the row's
|
||||
# ``created_at`` UTC date part, appended after ``title`` — the
|
||||
# only position; always present, ``created_at`` is NOT NULL).
|
||||
# Phase 118 (A6): the block body is the summary — the full content
|
||||
# stays out of the prompt (the capped ``read`` tool is the only
|
||||
# full-text path).
|
||||
blocks = [
|
||||
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
|
||||
f'date="{doc.created_at:%Y-%m-%d}">\n'
|
||||
f"{doc.content}\n"
|
||||
f"{_document_body(doc)}\n"
|
||||
"</document>"
|
||||
for doc in documents
|
||||
]
|
||||
body = "\n\n".join(blocks) if blocks else (
|
||||
"(no documents matched — do not invent specifics)"
|
||||
)
|
||||
if blocks:
|
||||
body = SUGGEST_INTRO + "\n\n" + body
|
||||
prompt = _base("HIGH")
|
||||
for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or [])):
|
||||
if part:
|
||||
|
||||
Reference in New Issue
Block a user