phase: 118_summary_seed_context

**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed. - **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0). - **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`. - **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors. - **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed. - **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits). - **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step). - **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
2026-09-16 06:57:49 -04:00
parent 21aad84a6d
commit 9820c361b0
80 changed files with 4690 additions and 1302 deletions
@@ -37,13 +37,30 @@ Steering (phase 15): the owner's stored tuning notes are loaded per turn
 section — both the HIGH and the LOW prompt carry it. The per-turn log
 line records ``tuning=N`` (the number of injected notes).

-Summaries (phase 30): a lite-model summary chunk's parent *is* the
-source document, so a summary hit resolves to the full source document
-through the unchanged chunk→document mapping (A7 revised) — context
-assembly is untouched. ``TurnPlan.summary_hits`` counts the hit chunks
-with ``is_summary`` whose parent document landed in the selected
-top-N context, and the per-turn log line records ``summary_hits=N``
-after ``fts_hits`` (PLAN §9 line extension).
+Summaries (phase 30; phase 118 redefinition): a lite-model summary
+chunk's parent *is* the source document, so a summary hit resolves to
+the full source document through the unchanged chunk→document mapping
+(A7 revised). ``TurnPlan.summary_hits`` counts the hit chunks with
+``is_summary`` whose parent document is in the SUGGESTED set (phase 118:
+redefined from the phase-113 cited set — the suggestion tier is the
+seeded context now), and the per-turn log line records ``summary_hits=N``
+after ``fts_hits`` (PLAN §9 line extension; phase 118 adds
+``suggested=N`` after ``summary_hits=N``).
+
+Summary seed context (phase 118, TODO L3 — the owner directive that
+re-revises A7, LOCKED A6): a grounded turn's ``<documents>`` section
+seeds the top-5 suggested documents' SUMMARIES (never their full
+texts) — the "start here if these summaries seem right to you" starting
+point — and the LLM extends its context by ``read``-ing what it needs
+(the capped ``read`` tool is the ONLY full-text path). Both tiers are
+computed once per turn, for BOTH branches: ``suggested_docs`` (no
+floor, A3) and ``related_docs`` (rank 6+, the done frame's row).
+``done.sources`` is the citation surface — suggested + agent-read,
+deduped (A4: a grounded turn always shows chips); ``query_log.sources``
+and the per-turn log line keep recording the full retrieval
+(suggested + related + read, LOCKED A3). The deflected branch's prompt
+is byte-identical (weak-hit titles only — A8 untouched); its TurnPlan
+still carries both tiers for the durable record.

 KB overview (phase 31): the lite-generated outline of the knowledge
 base (single ``kb_overview`` row) is read per turn (one indexed PK
@@ -195,7 +212,13 @@ from app.rag.llm import (
 )
 from app.rag.overview import load_kb_overview
 from app.rag.prompts import build_deflect_prompt, build_high_prompt, history_to_messages
-from app.rag.retriever import RetrievedChunk, retrieve, select_documents_tiered, weak_hit_titles
+from app.rag.retriever import (
+    RetrievedChunk,
+    retrieve,
+    select_related,
+    select_suggested,
+    weak_hit_titles,
+)
 from app.rag.scaffolding import ScaffoldingFilter  # phase 71: the streaming filter
 from app.rag.suggestions import derive_suggestions
 from app.schemas import (
@@ -258,12 +281,19 @@ class TurnPlan:
    fts_hits: int  # lexical (OR-tsquery) candidates matched
    deflected: bool
    system_prompt: str
-    docs: list[Document]  # cited sources (phase 113: the bar-clearing tier)
-    related_docs: list[Document] = field(default_factory=list)  # phase 113
+    # The summary-seeded suggestion tier (phase 118, A6: top-N distinct
+    # documents, NO floor, A3 — the HIGH prompt carries their SUMMARIES;
+    # the done frame's citation surface + run_agent's seed_docs, A4).
+    suggested_docs: list[Document]
+    # Rank 6+ after the suggested set (phase 118: the next ranked docs
+    # that are not already suggested, at most related_max_docs — the
+    # done frame's de-emphasized "nearby docs" row, never a citation).
+    related_docs: list[Document] = field(default_factory=list)
    suggestions: list[str] = field(default_factory=list)  # "Maybe try" chips (deflected turns only)
    tuning_count: int = 0  # steering notes injected into the system prompt
-    #: Hit chunks with ``is_summary`` whose parent document made it into
-    #: *docs* (phase 30; per-turn log line ``summary_hits=N``).
+    #: Hit chunks with ``is_summary`` whose parent document is in the
+    #: SUGGESTED set (phase 30; redefined from the phase-113 cited set
+    #: in phase 118; per-turn log line ``summary_hits=N``).
    summary_hits: int = 0
    #: Length of the stored KB overview injected as the
    #: ``<knowledge_base>`` section (phase 31; per-turn log line
@@ -281,7 +311,8 @@ def plan_turn(

    * **HIGH (grounded)** when ``best_cosine >= threshold`` **or**
      (``fts_hits > 0`` **and** ``best_cosine >= lexical_support_floor``):
-      HIGH prompt with the full top-N documents, no suggestions. A cosine
+      HIGH prompt seeded with the suggested documents' SUMMARIES (phase
+      118, A6 — never their full texts), no suggestions. A cosine
      exactly at the threshold is an answer — the gate is strict
      (``< threshold``). An FTS hit alone, without vector corroboration
      (cosine < lexical_support_floor), stays LOW (A8 revised 2026-09-14).
@@ -305,42 +336,52 @@ def plan_turn(
    prompts are byte-identical to the pre-phase text and ``kb_chars``
    is 0.

-    ``summary_hits`` (phase 30) counts the hit chunks with
-    ``is_summary`` whose parent document is among the selected
-    top-N documents — both the HIGH and the LOW branch record it.
+    ``summary_hits`` (phase 30; phase 118 redefinition) counts the hit
+    chunks with ``is_summary`` whose parent document is in the
+    SUGGESTED set — both the HIGH and the LOW branch record it.

-    Phase 113 (the usefulness bar, LOCKED A2): retrieval documents are
-    tiered before either branch — ``docs`` (cited) are the distinct
-    parent documents whose best hit-chunk cosine clears
-    ``settings.source_usefulness_floor`` (at most ``top_n_docs`` — the
-    ceiling, never a quota); ``related_docs`` are the next ranked
-    documents (at most ``related_max_docs``) that did not earn a cited
-    slot. On a deflected turn the weak hits fall to ``related_docs``
-    (the cited tier is usually empty — nothing below the bar earned a
-    citation slot); the LOW prompt itself is unchanged (weak-hit titles
-    only).
+    Phase 118 (the summary seeding, LOCKED A3/A6): retrieval documents
+    are tiered once, before either branch — ``suggested_docs`` are the
+    top-N distinct parent documents in fused rank order with NO cosine
+    floor (``settings.suggested_docs``, default 5 — the "start here"
+    seeding: the HIGH prompt carries their SUMMARIES, never their full
+    texts; the floor never filters, so a lexical-only hit is suggested
+    when it ranks) and ``related_docs`` are the next ranked documents
+    that are not already suggested (rank 6+ for the contiguous top-5
+    suggestion set, at most ``related_max_docs`` — the done frame's
+    de-emphasized "nearby docs" row). On a deflected turn the weak hits
+    are suggested too (no floor); the LOW prompt itself is unchanged
+    (weak-hit titles only), and both tiers still ride the TurnPlan for
+    the durable record (LOCKED A3: query_log records retrieval, not
+    citations).
+
+    ``settings.top_n_docs`` and ``settings.source_usefulness_floor`` are
+    NOT consulted here (phase 118 retired their seeding role, A6) —
+    they remain settings for env back-compat only.
    """
    steering = list(notes or [])
    kb_text = (kb_overview or "").strip()
    kb_chars = len(kb_text)
    best_cosine = max((c.cosine for c in chunks), default=0.0)
    fts_hits = sum(1 for c in chunks if c.fts_hit)
-    docs, related_docs = select_documents_tiered(
-        chunks,
-        n=settings.top_n_docs,
-        floor=settings.source_usefulness_floor,
-        related_cap=settings.related_max_docs,
+    # Phase 118 (A3/A6): BOTH tiers, computed once, for BOTH branches —
+    # the suggested tier (top-N, NO floor) seeds the HIGH prompt and the
+    # agent; the related tier (rank 6+ after the suggested set) feeds
+    # the done frame's row and the durable record.
+    suggested = select_suggested(chunks, n=settings.suggested_docs)
+    related_docs = select_related(
+        chunks, {d.id for d in suggested}, settings.related_max_docs
    )
-    selected_ids = {d.id for d in docs}
-    summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in selected_ids)
+    suggested_ids = {d.id for d in suggested}
+    summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in suggested_ids)
    lexical_supported = fts_hits > 0 and best_cosine >= settings.lexical_support_floor
    if best_cosine >= settings.relevance_threshold or lexical_supported:
        return TurnPlan(
            best_cosine,
            fts_hits,
            False,
-            build_high_prompt(docs, notes=steering, kb_overview=kb_text),
-            docs,
+            build_high_prompt(suggested, notes=steering, kb_overview=kb_text),
+            suggested,
            related_docs,
            [],
            len(steering),
@@ -353,7 +394,7 @@ def plan_turn(
        fts_hits,
        True,
        build_deflect_prompt(titles, notes=steering, kb_overview=kb_text),
-        docs,
+        suggested,
        related_docs,
        derive_suggestions(titles, settings.suggestions),
        len(steering),
@@ -606,7 +647,7 @@ async def chat(
                        db_factory,  # SEC-14-04: session factory, not a long-lived session
                        system_prompt=plan.system_prompt,
                        user_message=request.message,
-                        seed_docs=plan.docs,
+                        seed_docs=plan.suggested_docs,  # phase 118 (A4): the suggestion tier
                        settings=settings,
                        holder=holder,
                        history=hist,  # phase 74: the same trimmed prior turns
@@ -804,30 +845,33 @@ async def chat(
                # 4. Durable record + required per-turn log line (PLAN §9).
                #    Phase 37: the agent's read documents join the
                #    retrieval's — deduped by (source, path), order
-                #    preserved (the read doc stays last). Phase 113:
-                #    the retrieval now arrives in two tiers — the cited
-                #    docs (``plan.docs``) and the related docs
-                #    (``plan.related_docs``, the scored-but-below-the-bar
-                #    documents). The DURABLE record keeps the full
+                #    preserved (a read doc not already suggested stays
+                #    last). Phase 118: the retrieval arrives in two
+                #    tiers — the suggested docs
+                #    (``plan.suggested_docs``, the summary-seeded
+                #    "start here" tier, no floor, A3) and the related
+                #    docs (``plan.related_docs``, rank 6+ after the
+                #    suggested set). The DURABLE record keeps the full
                #    retrieval (LOCKED A3: query_log records retrieval,
                #    not citations — even on deflected turns, where the
-                #    weak hits live in the related tier). Phase 112:
+                #    weak hits are suggested). Phase 112/118:
                #    done.sources is the CITATION surface — it carries the
-                #    cited docs + the agent-read docs on grounded turns
-                #    and [] on deflected ones (a deflected answer cites
-                #    nothing; the weak hits stay in the durable record).
+                #    suggested docs + the agent-read docs (deduped,
+                #    LOCKED A4) on grounded turns and [] on deflected
+                #    ones (a deflected answer cites nothing; the weak
+                #    hits stay in the durable record).
                #    A cancelled turn (the generator closed by the
                #    consumer) never reaches this step — no query_log row.
                cited_docs: list[Document] = []
                cited_seen: set[tuple[str, str]] = set()
-                for doc in [*plan.docs, *holder.read_docs]:
+                for doc in [*plan.suggested_docs, *holder.read_docs]:
                    key = (doc.source, doc.path)
                    if key not in cited_seen:
                        cited_seen.add(key)
                        cited_docs.append(doc)
                record_docs: list[Document] = []
                seen: set[tuple[str, str]] = set()
-                for doc in [*plan.docs, *plan.related_docs, *holder.read_docs]:
+                for doc in [*plan.suggested_docs, *plan.related_docs, *holder.read_docs]:
                    key = (doc.source, doc.path)
                    if key not in seen:
                        seen.add(key)
@@ -852,7 +896,8 @@ async def chat(
                    logger.exception("chat: failed to write query_log question=%r", request.message)

                logger.info(
-                    "question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d tuning=%d "
+                    "question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d "
+                    "suggested=%d tuning=%d "
                    "kb_chars=%d history_msgs=%d threshold=%.2f deflected=%s sources=%r "
                    "thinking_chars=%d tool_calls=%d total_ms=%d retries=%d scaffold_stripped=%d",
                    request.message,
@@ -860,6 +905,7 @@ async def chat(
                    plan.top_score,
                    plan.fts_hits,
                    plan.summary_hits,
+                    len(plan.suggested_docs),  # phase 118: the seeded suggestion tier size
                    plan.tuning_count,
                    plan.kb_chars,
                    len(hist),
@@ -879,14 +925,15 @@ async def chat(
                # citations). The retrieval stays durably recorded above
                # (query_log.sources + the log line — observability
                # unchanged).
-                # Phase 113 (A2/A4): done.related carries the related
-                # tier — the scored documents that did not clear the
-                # usefulness bar (deduped against the cited list, the
-                # same (source, path) pattern as cited_docs: an agent-
-                # read related doc is a citation, never a "nearby doc"
-                # — and capped by related_max_docs in the tiering). The
-                # UI renders it as the de-emphasized related-docs row,
-                # never a citation chip; old clients ignore the field.
+                # Phase 118 (A3/A4): done.related carries the related
+                # tier — the ranked documents beyond the suggested set
+                # (rank 6+ after the contiguous top-N suggestion, capped
+                # by related_max_docs in the tiering), deduped against
+                # the cited list, the same (source, path) pattern as
+                # cited_docs: an agent-read related doc is a citation,
+                # never a "nearby doc". The UI renders it as the
+                # de-emphasized related-docs row, never a citation chip;
+                # old clients ignore the field.
                cited_refs: list[SourceRef] = []
                if not plan.deflected:
                    cited_refs = [