phase: 118_summary_seed_context
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 14s

**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.

- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
2026-09-16 06:57:49 -04:00
parent 21aad84a6d
commit 9820c361b0
80 changed files with 4690 additions and 1302 deletions
+303 -24
View File
@@ -17,9 +17,13 @@ stated up front — the combined ``source/path`` identity for
the drill-down tree contract — one level per call, sources at the
top, folders + files below, ``grep`` as the without-listing locator —
while the ``read``/``grep`` clauses and the discipline rules are
byte-identical): the teaching refusals in :mod:`app.rag.agent`
re-state the same contract; the ``<tools>`` marker keying (HIGH
only) is unchanged.
byte-identical; phase 118, task 04: the ``read`` clause rewritten
for the summary-seed mode — the ``<documents>`` section holds
SUMMARIES, ``read`` adds the full text, and only an already-read
document is refused — while the ``ls``/``grep`` clauses and the
discipline rules stay byte-identical): the teaching refusals in
:mod:`app.rag.agent` re-state the same contract; the ``<tools>``
marker keying (HIGH only) is unchanged.
"""
from __future__ import annotations
@@ -32,6 +36,7 @@ from app.config import Settings
from app.models import Document
from app.rag.prompts import (
PERSONA,
SUGGEST_INTRO,
TOOLS_SECTION,
_base,
build_deflect_prompt,
@@ -55,7 +60,12 @@ KB_INTRO = "The basic categories of everything in this knowledge base (generated
_FIXTURE_CREATED_AT = datetime(2024, 6, 15, 12, 0, 0, tzinfo=UTC)
def _doc(path: str, content: str, title: str) -> Document:
def _doc(
path: str,
content: str,
title: str,
summary: str | None = None,
) -> Document:
return Document(
id=uuid.uuid4(),
source="Homelab",
@@ -65,6 +75,7 @@ def _doc(path: str, content: str, title: str) -> Document:
content=content,
content_hash="0" * 64,
created_at=_FIXTURE_CREATED_AT,
summary=summary,
)
@@ -93,23 +104,64 @@ def test_persona_owner_edits_are_preserved() -> None:
assert "HONESTY GATE" in PERSONA # the gate itself is intact
def test_high_prompt_carries_relevance_marker_and_full_documents() -> None:
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
def test_high_prompt_carries_relevance_marker_and_summary_block() -> None:
"""Phase 118 (LOCKED A6): the grounded turn seeds the document's
SUMMARY — the full content never reaches the prompt (the ``read``
tool is the only full-text path)."""
doc = _doc(
"kubernetes.md",
"Talos Linux on three nodes. FULL_CONTENT_SENTINEL_987654",
"Kubernetes Homelab Cluster",
summary="A Talos Linux cluster on three nodes.",
)
prompt = build_high_prompt([doc])
assert "<relevance>HIGH</relevance>" in prompt
assert "DEFLECT_MODE" not in prompt
assert "<documents>" in prompt and "</documents>" in prompt
assert 'path="kubernetes.md"' in prompt
assert "Talos Linux on three nodes." in prompt
assert "A Talos Linux cluster on three nodes." in prompt # the summary
assert "FULL_CONTENT_SENTINEL_987654" not in prompt # never the content
assert "Talos Linux on three nodes." not in prompt # nor the full sentence
assert "HONESTY GATE" in prompt # persona intact
def test_high_prompt_lists_multiple_documents_in_order() -> None:
a = _doc("a.md", "CONTENT_A", "Title A")
b = _doc("b.md", "CONTENT_B", "Title B")
a = _doc("a.md", "CONTENT_A", "Title A", summary="Summary A.")
b = _doc("b.md", "CONTENT_B", "Title B", summary="Summary B.")
prompt = build_high_prompt([a, b])
assert prompt.index("CONTENT_A") < prompt.index("CONTENT_B")
assert prompt.index("Summary A.") < prompt.index("Summary B.")
assert 'title="Title B"' in prompt
assert "CONTENT_A" not in prompt and "CONTENT_B" not in prompt
def test_high_prompt_seeds_summaries_of_all_suggested_docs() -> None:
"""The phase-118 completion pin: a grounded prompt built from five
summary-bearing documents contains ALL FIVE summaries + the intro,
and ZERO full-content characters (the full texts are not in the
prompt at all)."""
docs = [
_doc(
f"doc{i}.md",
f"FULL_CONTENT_SENTINEL_{i} " + "x" * 100,
f"Title {i}",
summary=f"SUMMARY_{i} of the document.",
)
for i in range(5)
]
prompt = build_high_prompt(docs)
assert SUGGEST_INTRO in prompt
for i in range(5):
assert f"SUMMARY_{i} of the document.\n" in prompt
assert f"FULL_CONTENT_SENTINEL_{i}" not in prompt
assert f'path="doc{i}.md"' in prompt
# One block per document, in the given order.
assert prompt.count("<document ") == 5
positions = [prompt.index(f"SUMMARY_{i}") for i in range(5)]
assert positions == sorted(positions)
# The intro leads the section: <documents> → intro → first block.
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
def test_high_prompt_without_documents_stays_honest() -> None:
@@ -143,7 +195,9 @@ def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
"""Phase 15 contract: with no steering notes the prompt is exactly what
it was before the <tuning> section existed. (Phase 37: the HIGH prompt
additionally carries the ``<tools>`` section after the mode body — the
fixtures account for it; the LOW prompt is untouched.)"""
fixtures account for it; phase 118: the ``<documents>`` section leads
with the ``SUGGEST_INTRO`` line — the fixture accounts for it; the
LOW prompt is untouched.)"""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
block = (
'<document source="Homelab" path="kubernetes.md" '
@@ -152,7 +206,14 @@ def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
"</document>"
)
assert build_high_prompt([doc]) == (
_base("HIGH") + "\n<documents>\n" + block + "\n</documents>" + "\n" + TOOLS_SECTION
_base("HIGH")
+ "\n<documents>\n"
+ SUGGEST_INTRO
+ "\n\n"
+ block
+ "\n</documents>"
+ "\n"
+ TOOLS_SECTION
)
# Phase 71: the LOW prompt carries the owner-permitted plain-text
# line after the DEFLECT_MODE sentence (the marker-keying contract
@@ -257,6 +318,86 @@ def test_tools_section_phase95_read_truncation_clause() -> None:
assert "Very large documents are capped" in build_high_prompt([doc])
def test_tools_section_phase118_summary_seed_read_clause() -> None:
"""Phase 118 (task 04, A6): the ``read`` clause is rewritten for
the summary-seed mode — the ``<documents>`` section holds
SUMMARIES (a suggested document's full text is not in the prompt
until ``read`` adds it); do not re-read an already-read document
(answer from the text already in the prompt); if the user asks to
open or read a suggested document, ``read`` it. The phase-72 "do
not call ``read`` for a ``<documents>`` document" copy is retired.
The ``ls`` clause (phase 94 drill-down contract), the ``grep``
clause, and the discipline rules stay byte-identical — the
prompt lock's prefix/suffix anchors survive (see
``test_prompt_lock``)."""
# The new summary-seed copy (pinned byte-for-byte).
assert (
"The <documents> section holds SUMMARIES — the full text of a "
"suggested document is not in your prompt until you `read` it. "
"Do not re-read a document you have already read — its full "
"text is already in your prompt; answer directly from it. If "
"the user asks you to open or read a suggested document, "
"`read` it — that is the point of the section."
) in TOOLS_SECTION
# The read identity handoff now also names the <documents> summary
# blocks (the combined identity is shown there), keeping the
# "including the source name" contract.
assert (
"exactly as shown in the `ls` output — including the source "
"name — or in the <documents> summary blocks — adding its "
"full content to your context"
) in TOOLS_SECTION
# The retired pre-phase-118 copy is gone.
assert "Do not call `read` for a document already shown in" not in TOOLS_SECTION
assert "even when the user asks you to open or read it" not in TOOLS_SECTION
# The ls clause (phase 94 drill-down contract) — byte-identical.
assert (
"`ls` lists the knowledge base as a tree, one level at a "
"time: with no `path` it lists every synced source with its "
"document count and a summary of its contents; with a source "
"name (e.g. 'homelab') it lists that source's top-level "
"folders and files; with a `source/folder` path it drills one "
"level deeper. A listing shows only that level's subfolders "
"and its own files — never the whole knowledge base in one "
"call — and each folder line's summary says what the folder "
"contains before you drill into it. File lines are `source: X "
"| path: Y | title: Z`; to find one specific document without "
"listing, use `grep`."
) in TOOLS_SECTION
# The grep clause — byte-identical.
assert (
"`grep` locates an exact string (case-insensitive) in the "
"indexed documents and returns up to 20 matching "
"`source/path:line: text` lines — a locator, not a "
"context-adder: read the winner with `read`. A grep pattern "
"is a plain substring, NEVER a regex — '.*' and '\\.' are "
"literal text there; if such a pattern returns no matches, "
"retry with the plain text you expect to see. For a normal "
"search pass only `pattern` — its optional `path` argument "
"limits the search to one document you already know, by the "
"same combined `source/path` string; never a source name — a "
"bare document path (without the source name) will not "
"resolve there either."
) in TOOLS_SECTION
# The discipline rules — byte-identical.
assert (
"Make exactly one tool call per reply — a reply carrying two "
"tool calls runs only the first, the second is discarded — "
"and wait for the result before the next call. Never repeat a "
"call that was refused or already succeeded — the refusal "
"already told you the correct form. Answer as soon as you "
"have what you need."
) in TOOLS_SECTION
# The new read clause rides the built HIGH prompt and never the
# LOW (deflected) prompt.
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes", summary="K8S")
high = build_high_prompt([doc])
assert "The <documents> section holds SUMMARIES" in high
assert "`read` it — that is the point of the section." in high
low = build_deflect_prompt(["T1"])
assert "The <documents> section holds SUMMARIES" not in low
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
"""Phase 72/94: the contract clauses ride the HIGH prompt with the
rest of the section and never leak into the LOW/deflection prompt
@@ -273,20 +414,37 @@ def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
assert "including the source name" not in low
def test_documents_section_has_no_leading_intro() -> None:
"""Phase 72, task 05 (gate iterations 2-3, reverted): the
``<documents>`` section must NOT lead with an in-context reminder
or name the ``<document>`` blocks — the live telemetry showed that
copy primed the model to latch the seed documents' paths as
``ls`` scopes (the incident turn regressed to a cap-reached loop
on run 2 and re-trapped on run 5), and the reminder never flipped
the seed-doc ``read``s (15/15 across gate runs 1-5). The section
is exactly the document blocks again."""
def test_documents_section_leads_with_the_suggest_intro() -> None:
"""Phase 118, task 03: the ``<documents>`` section leads with the
start-here :data:`SUGGEST_INTRO` line BEFORE the first block (the
phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent) —
only when at least one block is present. This is the summary-as-
starting-point framing, not the reverted phase-72 do-not-read
reminder (that copy taught the seed texts as already-read context;
with A6's summary seeding the blocks are starting points the model
may ``read`` through to full text)."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
high = build_high_prompt([doc])
i_open = high.index("<documents>")
i_block = high.index('<document source="Homelab"')
assert high[i_open : i_block] == "<documents>\n" # no intro line
assert high[i_open : i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
# The intro occurs exactly once and sits inside the section.
assert high.count(SUGGEST_INTRO) == 1
assert high.index(SUGGEST_INTRO) > high.index("<documents>")
assert high.index(SUGGEST_INTRO) < high.index("</documents>")
def test_documents_section_without_blocks_has_no_intro() -> None:
"""The intro rides only on present blocks: an empty ``<documents>``
section is exactly the fallback line again (no intro, no blocks)."""
high = build_high_prompt([])
assert SUGGEST_INTRO not in high
i_open = high.index("<documents>")
i_close = high.index("</documents>")
assert (
high[i_open : i_close]
== "<documents>\n(no documents matched — do not invent specifics)\n"
)
def test_tools_section_old_names_and_budget_copy_gone() -> None:
@@ -314,6 +472,117 @@ def test_high_prompt_still_ends_with_tools_section() -> None:
assert old not in prompt
# ---------- phase 118 (task 03): the NULL-summary preview fallback (A5) ----------
def _content_only_doc(content: str) -> Document:
return _doc("kubernetes.md", content, "Kubernetes Homelab Cluster")
def test_null_summary_falls_back_to_preview_plus_marker() -> None:
"""A doc whose ``summary`` is None (a fail-soft import miss) seeds the
first ``suggestion_preview_chars`` (default 400) content characters +
the shared ``[…truncated…]`` marker on its own line — never the full
content, never an LLM call."""
content = "A" * 400 + "B" * 600 # 1000 chars
prompt = build_high_prompt([_content_only_doc(content)])
body_start = prompt.index('date="2024-06-15">\n') + len('date="2024-06-15">\n')
body_end = prompt.index("\n</document>", body_start)
assert prompt[body_start:body_end] == content[:400] + "\n" + TRUNCATION_MARKER
assert "B" * 40 not in prompt # beyond the 400-char cut
assert prompt.count(TRUNCATION_MARKER) == 1 # only the block's marker
def test_whitespace_summary_uses_the_same_preview_fallback() -> None:
for blank in ("", " ", "\n \t "):
doc = _doc("kubernetes.md", "A" * 500 + "B" * 500, "T", summary=blank)
prompt = build_high_prompt([doc])
assert "A" * 400 in prompt
assert "B" * 40 not in prompt
assert TRUNCATION_MARKER in prompt
def test_short_content_preview_is_the_whole_content_unmarked() -> None:
"""Content at or under the cap rides whole — nothing was cut, so no
marker (the marker signals truncation, not the fallback)."""
for content in ("short body", "C" * 399, "D" * 400):
prompt = build_high_prompt([_content_only_doc(content)])
assert content + "\n</document>" in prompt
assert TRUNCATION_MARKER not in prompt
def test_summary_stripped_and_never_truncated_by_the_preview_cap() -> None:
"""The summary body is the stripped ``doc.summary`` — even when it is
longer than the preview cap (the cap bounds only the FALLBACK; a
stored summary is trusted context, LOCKED A5)."""
summary = "S" * 900
content = "CONTENT_SENTINEL " + "x" * 50
prompt = build_high_prompt([_doc("kubernetes.md", content, "T", summary=f" {summary}\n")])
assert summary in prompt
assert content not in prompt
assert TRUNCATION_MARKER not in prompt
def test_preview_cap_setting_is_honored_on_the_fallback_path(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A non-default ``suggestion_preview_chars`` (``BOR_SUGGESTION_PREVIEW_CHARS``
— the env mapping is pinned in :mod:`tests.unit.test_config`) cuts the
preview at the setting's cap on the fallback path."""
from app.rag import prompts as prompts_mod
monkeypatch.setattr(
prompts_mod,
"get_settings",
lambda: Settings(_env_file=None, suggestion_preview_chars=10), # pyright: ignore[reportCallIssue]
)
content = "A" * 100 + "B" * 100
prompt = build_high_prompt([_content_only_doc(content)])
assert "A" * 10 in prompt
assert "A" * 11 not in prompt
assert TRUNCATION_MARKER in prompt
def test_preview_fallback_never_reads_settings_for_summarized_docs(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""House pattern (``build_steering_section`` / ``build_kb_section``):
the fallback cap is read from settings on the fallback path ONLY — a
prompt built from summary-bearing docs makes no settings call for
it (a dead ``get_settings`` must not break such a build)."""
from app.rag import prompts as prompts_mod
doc = _doc("kubernetes.md", "CONTENT_SENTINEL", "T", summary="the summary")
def _dead() -> None: # pragma: no cover - must never be called
raise AssertionError("get_settings() called for a summarized doc")
monkeypatch.setattr(prompts_mod, "get_settings", _dead)
prompt = build_high_prompt([doc])
assert "the summary\n</document>" in prompt
assert "CONTENT_SENTINEL" not in prompt
# ---------- phase 118: the LOW prompt stays byte-identical ----------
def test_low_prompt_byte_identical_to_pre_task() -> None:
"""LOCKED A8 surface: the deflection prompt is untouched by the
summary seeding — byte-identical build on the same inputs (the full
sha pin lives in :mod:`tests.unit.test_prompt_lock`)."""
for titles, notes, kb in (
(["T1", "T2"], None, None),
(["T1"], ["be concise"], None),
([], None, OVERVIEW),
(["T1", "T2"], ["be concise"], OVERVIEW),
):
prompt = build_deflect_prompt(titles, notes=notes, kb_overview=kb)
assert "DEFLECT_MODE" in prompt
assert "<documents>" not in prompt
assert SUGGEST_INTRO not in prompt
assert "<tools>" not in prompt
def test_relevance_placeholder_rejected_for_garbage() -> None:
with pytest.raises(ValueError, match="HIGH or LOW"):
_base("MEDIUM")
@@ -401,7 +670,9 @@ def test_no_overview_prompt_is_byte_identical_to_pre_phase() -> None:
every prompt is exactly what it was before the ``<knowledge_base>``
section existed — with or without steering notes. (Phase 37: the HIGH
prompt additionally carries the ``<tools>`` section after the mode
body — the fixtures account for it; the LOW prompt is untouched.)"""
body; phase 118: the ``<documents>`` section leads with the
``SUGGEST_INTRO`` line — the fixtures account for both; the LOW
prompt is untouched.)"""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
block = (
'<document source="Homelab" path="kubernetes.md" '
@@ -409,7 +680,15 @@ def test_no_overview_prompt_is_byte_identical_to_pre_phase() -> None:
"Talos Linux on three nodes.\n"
"</document>"
)
docs_block = "\n<documents>\n" + block + "\n</documents>" + "\n" + TOOLS_SECTION
docs_block = (
"\n<documents>\n"
+ SUGGEST_INTRO
+ "\n\n"
+ block
+ "\n</documents>"
+ "\n"
+ TOOLS_SECTION
)
high_plain = _base("HIGH") + docs_block
high_steered = _base("HIGH") + "\n" + build_steering_section(["be concise"]) + docs_block
# Phase 71: the owner-permitted plain-text line is part of the