phase: 118_summary_seed_context
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.
- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
@@ -1,4 +1,5 @@
|
||||
"""Phase 30 E2E (Playwright): a summary hit delivers the full source doc.
|
||||
"""Phase 30 E2E (Playwright) — phase 118 re-targeted (A2/A6): a summary
|
||||
hit seeds the SUMMARY into the prompt — never the full source doc.
|
||||
|
||||
Story: ``.agents/user_stories/document-summaries.md``
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
@@ -7,27 +8,33 @@ Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
The fixture KB is a story-dedicated directory
|
||||
(``tests/fixtures/summary_kb/`` — the shared ``tests/fixtures/docs/``
|
||||
stays at its 9 pinned files) with two documents:
|
||||
stays at its pinned files) with two documents:
|
||||
|
||||
* ``quadlet/qwen-llamacpp.yaml`` — a non-markdown A9 doc. At import the
|
||||
mock ``lite`` model (``SUMMARY_MODE`` marker, ``tests/e2e/mock_llm.py``)
|
||||
reduces it to a deterministic 24-token digest, stored on
|
||||
``documents.summary`` and indexed as one ``is_summary`` chunk. The raw
|
||||
yaml body is deliberately token-diluted, so the document's best fused
|
||||
chunk is its summary chunk. The sentinel ``RESE-SUMMARY-SENTINEL-7f3a``
|
||||
sits on the document's LAST line — outside the 24-token digest,
|
||||
unreachable from the summary.
|
||||
* ``notes/qwen-llamacpp-notes.md`` — a markdown control doc (never
|
||||
summarized) that ranks first, which puts the yaml document LAST inside
|
||||
``<documents>``.
|
||||
* ``quadlet/qwen-llamacpp.yaml`` — at import the mock ``lite`` model
|
||||
(``SUMMARY_MODE`` marker, ``tests/e2e/mock_llm.py``) reduces it to a
|
||||
deterministic 24-token digest, stored on ``documents.summary`` and
|
||||
indexed as one ``is_summary`` chunk. The raw yaml body is deliberately
|
||||
token-diluted, so the document's top fused chunks are the summary and
|
||||
the one lexical-hitting raw chunk. The sentinel
|
||||
``RESE-SUMMARY-SENTINEL-7f3a`` sits on the document's LAST line —
|
||||
outside the 24-token digest, unreachable from the summary.
|
||||
* ``notes/qwen-llamacpp-notes.md`` — a markdown doc (phase 118 A2: it is
|
||||
summarized TOO — the phase-30 non-markdown-only scope is retired) that
|
||||
ranks first, which puts the yaml document LAST inside
|
||||
``<documents>`` (a two-doc KB → both docs are suggested, the related
|
||||
tier is empty).
|
||||
|
||||
The mock LLM's tail-echo trigger (``END_OF_NOTES_TRIGGER``) makes the
|
||||
answer quote the last 160 chars of the document context — the tail of
|
||||
the LAST selected document. The sentinel therefore appears in the
|
||||
rendered answer **iff the entire yaml source document (not the summary
|
||||
digest) reached the LLM prompt** — the summary→parent-document resolution
|
||||
through the unchanged chunk→document mapping (A7 revised: never
|
||||
truncated), which is what this story is about.
|
||||
answer quote the last 160 chars of the seeded ``<documents>`` block —
|
||||
under the phase-118 summary-seed contract (A6) that block carries the
|
||||
suggested docs' SUMMARIES, never their full texts, so the echoed tail is
|
||||
the LAST suggested doc's summary (the yaml doc's byte-stable digest tail
|
||||
+ ``Source:`` pointer line). The sentinel therefore appears in the
|
||||
rendered answer **only if the entire yaml source document (not the
|
||||
summary) reached the LLM prompt** — under the locked contract it must be
|
||||
ABSENT (full text enters the context only through the capped ``read``
|
||||
tool), which is the inverse of the retired phase-30/24 full-text pin
|
||||
and what this story is about now.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -47,7 +54,7 @@ from app.models import Document, QueryLog
|
||||
from app.rag.chunker import chunk_document
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from app.rag.retriever import RetrievedChunk, retrieve
|
||||
from app.rag.retriever import RetrievedChunk, retrieve, select_suggested
|
||||
from e2e.auth_helpers import login
|
||||
from tests.e2e.mock_llm import TOKEN_RE, embed_text
|
||||
|
||||
@@ -156,17 +163,20 @@ def _chunks_by_path(chunks: Sequence[RetrievedChunk], path: str) -> list[Retriev
|
||||
# --- Story tests -------------------------------------------------------------
|
||||
|
||||
|
||||
def test_summary_hit_retrieves_full_source_document(
|
||||
def test_summary_hit_seeds_the_summary_not_the_full_text(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""A question whose best yaml match is the summary chunk yields an
|
||||
answer grounded in the FULL yaml source document: its tail sentinel —
|
||||
which the summary digest cannot contain — is echoed back, and the
|
||||
source chip cites the yaml path (deflected: false)."""
|
||||
"""A question whose best yaml match is its summary chunk yields a
|
||||
grounded answer seeded from the SUMMARY — the mock's tail echo
|
||||
quotes the last suggested doc's summary tail (digest + pointer), and
|
||||
the sentinel (the yaml doc's last line, outside the digest) is
|
||||
ABSENT: the full source doc never reached the prompt (A6). Both
|
||||
fixture docs are suggested (two-doc KB, no floor) and both chips
|
||||
render (deflected: false)."""
|
||||
_reset_db()
|
||||
summary = _run_in_thread(_import_fixtures(mock_llm))
|
||||
assert summary.added == 2 # yaml + md control
|
||||
assert summary.summaries == 1 and summary.summary_errors == 0
|
||||
assert summary.summaries == 2 and summary.summary_errors == 0 # A2: md too
|
||||
assert summary.errors == 0
|
||||
|
||||
# Import state: exactly one embedded ``is_summary`` chunk (position
|
||||
@@ -182,30 +192,48 @@ def test_summary_hit_retrieves_full_source_document(
|
||||
assert schunks[0].embedding is not None
|
||||
assert yaml_doc.summary == _expected_summary(yaml_content, SOURCE, YAML_PATH)
|
||||
|
||||
# Retrieval state: the summary chunk is the yaml document's best fused
|
||||
# chunk — the document enters the context through its summary, not
|
||||
# through the diluted raw yaml chunks.
|
||||
# Retrieval state (phase 118, A2/A6): the EMBEDDED summary chunk is
|
||||
# a retrieval candidate (the seeded text ranks on its own), and the
|
||||
# suggested tier is the two docs in rank order — md first, yaml
|
||||
# LAST (so the yaml's summary is the tail of the <documents> block,
|
||||
# the mock echo's target). The document enters the prompt through
|
||||
# its SUMMARY block, not through the diluted raw yaml chunks.
|
||||
with SessionLocal() as db:
|
||||
chunks = retrieve(db, QUESTION, embed_text(QUESTION))
|
||||
yaml_chunks = _chunks_by_path(chunks, YAML_PATH)
|
||||
best_yaml = max(yaml_chunks, key=lambda c: c.score)
|
||||
assert best_yaml.is_summary
|
||||
assert any(c.is_summary for c in yaml_chunks) # the embedded summary ranks
|
||||
assert len(yaml_chunks) >= 2 # summary + at least one raw candidate
|
||||
assert [d.path for d in select_suggested(chunks)] == [MD_PATH, YAML_PATH]
|
||||
|
||||
bubble = _ask(page, app_url, QUESTION)
|
||||
|
||||
# The tail sentinel exists only on the document's last line and
|
||||
# cannot be in the summary digest — its presence proves the entire
|
||||
# source document was in the LLM prompt (summary→parent resolution).
|
||||
expect(bubble).to_contain_text(SENTINEL, timeout=30_000)
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
# Phase 118 (A6): the seed is the SUMMARY, not the full text — the
|
||||
# mock's tail echo quotes the last 160 chars of the <documents>
|
||||
# block, which end in the LAST suggested doc's summary: the yaml
|
||||
# doc's byte-stable digest tail + pointer line. The sentinel
|
||||
# (document's last line, outside the digest) is therefore ABSENT —
|
||||
# the full source doc never reached the prompt (full text enters
|
||||
# only through the capped read tool; the inverse of the retired
|
||||
# phase-30/24 full-text pin). The bubble renders the answer as
|
||||
# markdown, which collapses the summary's newline — so pin each
|
||||
# LINE separately (the digest line's tail sits inside the echoed
|
||||
# 160 chars; the pointer line is single-line too).
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
yaml_summary = _expected_summary(yaml_content, SOURCE, YAML_PATH)
|
||||
yaml_digest_line = yaml_summary.split("\n", 1)[0]
|
||||
expect(bubble).to_contain_text(f"Source: {SOURCE}/{YAML_PATH}")
|
||||
expect(bubble).to_contain_text(yaml_digest_line[-80:])
|
||||
expect(bubble).not_to_contain_text(SENTINEL)
|
||||
|
||||
# Grounded: the yaml source chip renders (the md doc ranks first, so
|
||||
# both fixtures are cited).
|
||||
# Grounded: both fixture docs are suggested (two-doc KB — no floor)
|
||||
# and both chips render, in rank order (md first, yaml last).
|
||||
chip = page.locator(".msg.brain .source-chip", has_text=YAML_PATH)
|
||||
expect(chip).to_have_count(1)
|
||||
expect(chip.first).to_contain_text(f"{SOURCE}/{YAML_PATH}")
|
||||
expect(page.locator(".msg.brain .source-chip", has_text=MD_PATH)).to_have_count(1)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(2)
|
||||
# No rank-6+ doc in a two-doc KB → the related row is absent.
|
||||
expect(page.locator(".msg.brain .related-docs")).to_have_count(0)
|
||||
|
||||
# Button recovers (never stale) and the turn was grounded, not
|
||||
# deflected.
|
||||
@@ -214,20 +242,26 @@ def test_summary_hit_retrieves_full_source_document(
|
||||
row = _last_query_log()
|
||||
assert row.question == QUESTION
|
||||
assert row.deflected is False
|
||||
assert f"{SOURCE}/{YAML_PATH}" in row.sources
|
||||
assert f"{SOURCE}/{MD_PATH}" in row.sources
|
||||
# The durable record: suggested + related + read (deduped) — here
|
||||
# exactly the two suggested docs, in rank order (LOCKED A3).
|
||||
assert row.sources == (
|
||||
f"{SOURCE}/{MD_PATH}, {SOURCE}/{YAML_PATH}"
|
||||
), row.sources
|
||||
|
||||
|
||||
def test_markdown_control_doc_gets_no_summary_chunk(
|
||||
def test_markdown_control_doc_gets_a_summary_chunk(
|
||||
mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""Control: in the same KB the markdown doc gets no summary at all —
|
||||
its chunk count is exactly the raw chunks; the yaml doc has exactly
|
||||
one ``is_summary`` row and its raw chunk count is untouched by the
|
||||
summary."""
|
||||
"""Phase 118 (A2): in the same KB the markdown doc gets a summary TOO
|
||||
— the phase-30 non-markdown-only scope is retired (markdown docs
|
||||
backfill + summarize like every other doc): the same byte-stable
|
||||
digest + pointer line, exactly one ``is_summary`` chunk, and the raw
|
||||
chunk count untouched. The yaml doc keeps its exactly-one summary
|
||||
row with its raw chunks untouched."""
|
||||
_reset_db()
|
||||
summary = _run_in_thread(_import_fixtures(mock_llm))
|
||||
assert summary.added == 2
|
||||
assert summary.summaries == 2 # A2: the markdown doc is summarized too
|
||||
|
||||
md_content = (FIXTURES / MD_PATH).read_text(encoding="utf-8")
|
||||
yaml_content = (FIXTURES / YAML_PATH).read_text(encoding="utf-8")
|
||||
@@ -235,11 +269,20 @@ def test_markdown_control_doc_gets_no_summary_chunk(
|
||||
md_doc = _doc(db, MD_PATH)
|
||||
yaml_doc = _doc(db, YAML_PATH)
|
||||
md_chunks = [c for c in md_doc.chunks if not c.is_summary]
|
||||
md_summary = [c for c in md_doc.chunks if c.is_summary]
|
||||
yaml_raw = [c for c in yaml_doc.chunks if not c.is_summary]
|
||||
yaml_summary = [c for c in yaml_doc.chunks if c.is_summary]
|
||||
|
||||
# Markdown: never summarized (phase 30 scope — A9 non-markdown only).
|
||||
assert md_doc.summary is None
|
||||
# Markdown: NOW summarized (phase 118 A2) — the same byte-stable
|
||||
# digest + deterministic pointer line, exactly one is_summary chunk
|
||||
# (position −1, embedded), raw chunks untouched.
|
||||
expected_md_summary = _expected_summary(md_content, SOURCE, MD_PATH)
|
||||
assert md_doc.summary == expected_md_summary
|
||||
assert len(md_summary) == 1
|
||||
assert md_summary[0].position == -1
|
||||
assert md_summary[0].embedding is not None
|
||||
assert md_summary[0].content == expected_md_summary
|
||||
assert expected_md_summary.endswith(f"\nSource: {SOURCE}/{MD_PATH}")
|
||||
assert len(md_chunks) == len(
|
||||
chunk_document(md_content, MD_PATH, CHUNK_TARGET, CHUNK_OVERLAP)
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user