Phase 118 — Summary seed context: top-5 "start here" suggestions (TODO L3)

Source: TODO.md L3 — "I want to change the way the initial chat context works. Right now we feed two entire documents into context before the LLM responds. That was a good idea before we had complex tools and document summaries, but now we have faster ways for the LLM to identify what information it needs without reading 2 files. Moving forward, after the question is embedded, I want the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content — basically a 'start here if these summaries seem right to you' approach. That way the LLM responds faster and doesn't begin by reading massive amounts of data that might be irrelevant because the similarity search had a bad day." Story: n/a (owner-directed RAG behavior change; extends the phase-30 summaries, the phase-37/70/94 agent tools, and the phase-113 tiering assets). Context: A grounded (HIGH) turn currently seeds build_high_prompt with the full text of the top-2 cited documents (select_documents_tiered's cited tier — floor BOR_SOURCE_USEFULNESS_FLOOR 0.35, ceiling BOR_TOP_N_DOCS 2, app/api/chat.py:plan_turn), and run_agent receives them as seed_docs whose read is refused (ALREADY_IN_CONTEXT — the full text is already in the prompt). Summaries exist on Document.summary (phase-30 lite digest — currently non-markdown only; NULL for markdown docs, pre-30 rows and fail-soft misses) and are embedded as one is_summary chunk (position −1) per document. LOCKED A7 (full text of the top-2, never truncated, on the retrieval path) is re-revised by the owner directive in this TODO: full text enters the context only through the capped read tool (locked A6 below).

Objective

Grounded turns seed the top 5 related documents as summary blocks — a "start here if these summaries seem right to you" starting point — and the LLM extends its context by read-ing only what it needs; every document (markdown included) has a stored summary (generated for all new/changed docs at import, backfilled on the next sync for existing NULL rows). Answers get faster and never start by ingesting massive possibly-irrelevant full texts.

Dependencies

117_tool_call_compact (complete) — the last completed phase; this phase rewrites the read-tool copy the battery gate pins, so the gate (task 07) runs against its final state.

Design (shared by all tasks — the executor reads this, not the chat)

Suggested tier (task 02): new pure select_suggested(chunks, n=None) in app/rag/retriever.py — the top-N DISTINCT parent documents in fused rank order (the same stable score-descending walk as select_documents_tiered — a document's rank is fixed by its FIRST seen chunk), no cosine floor (suggestions are opt-in starting points, not citations — the floor stays for the done-event related tier only). n defaults to the new setting suggested_docs (BOR_SUGGESTED_DOCS, default 5, validator >= 1).
Suggestion blocks (task 03): the HIGH prompt keeps the <documents> section tag and the per-block <document …> markup (the E2E mock keys off both), but each block's BODY becomes the document's summary: the identity attributes (source/path/title/date — the phase-106 D5 date stays, always present) + doc.summary (stripped). Defensive fallback ONLY when doc.summary is None/whitespace (a fail-soft miss — locked A5): the first suggestion_preview_chars (BOR_SUGGESTION_PREVIEW_CHARS, default 400) content characters + the shared TRUNCATION_MARKER — no LLM call at chat time. A new pinned intro line (a module constant in app/rag/prompts.py) precedes the blocks inside the section: the start-here framing — the blocks are summaries of the top-ranked documents; if one seems right, read that document (combined source/path) to add its full content; answer from what you have and what you read; cite the document(s) you used by path.
Agent read (task 04): with summary seeds, read on a suggested document adds its full text — the existing mechanics are all unchanged (the BOR_READ_MAX_CHARS cap + TRUNCATION_MARKER + READ_TRUNCATION_NOTICE + ToolResultPiece/SSE tool_result frame, the phase-106 D5 date: line, holder.read_docs append, tool_calls increment). The dedupe set in _execute_tool's read branch becomes holder.read_docs only (the seeds drop out — a suggested doc is a summary, not full text), so ALREADY_IN_CONTEXT fires only for a document ALREADY READ into full-text context (its copy stays accurate: "the full text is already in your prompt"). The read tool description in AGENT_TOOLS (app/rag/agent.py) and the read clause of TOOLS_SECTION (app/rag/prompts.py) are rewritten for the summary-seed mode — SINGLE mode (no kill switch, locked A1): the <documents> section shows summaries, read adds a document's full text, and re-reading an already-read document is refused. The ls/grep clauses and the discipline rules (one call per reply, never repeat) stay byte-identical. tests/unit/test_prompt_lock.py byte-pins TOOLS_SECTION (sha256 + length + prefix/suffix): re-cut its anchors in the same change; PERSONA and the LOW DEFLECT_MODE body MUST stay byte-identical (the lock pins those too — the E2E mock keys on the markers' presence).
Turn wiring (task 05): plan_turn computes BOTH tiers once, for BOTH branches: suggested = select_suggested(chunks) and related = select_related(chunks, excluded={suggested ids}, cap=settings.related_max_docs) (new pure helper in app/rag/retriever.py — the same rank walk, skipping the excluded ids, no floor; with a 5-doc suggestion the related tier is rank order 6..n, the phase-113 UI "nearby docs" row). The HIGH branch: build_high_prompt(suggested, …) and run_agent(..., seed_docs=suggested); the LOW branch: prompt BYTE-IDENTICAL (weak-hit titles only, DEFLECT_MODE, no tools, chips — A8 gate untouched) but the TurnPlan still carries suggested + related for the durable record. TurnPlan.docs is RENAMED suggested_docs (same dataclass position); summary_hits is redefined as the hit chunks with is_summary whose parent document is in the SUGGESTED set. Citations: done.sources = suggested + agent-read, deduped by (source, path) order-preserving (locked A4 — a grounded turn always shows chips; the deflected turn keeps []). Durable record (LOCKED A3 — the log records retrieval, not citations): query_log.sources + the per-turn log line = suggested + related + read, deduped. The per-turn log line gains suggested=N after summary_hits=N (PLAN §9 extension). source_usefulness_floor stops being a seeding input (it may stay for other consumers — the executor checks; the SETTING itself is never removed).
Summaries for every document (task 01): the importer's summary step covers every new/changed A9 document — the markdown exclusion (_import_one_file's if suffix in (".md", ".markdown"): return) is removed. Backfill: on the UNCHANGED branch (same content_hash), a document whose summary is None gets the same best-effort _store_summary pass — summary text stored on doc.summary + one new embedded is_summary chunk — with a backfill=True flag that counts success in a NEW ImportSummary.summary_backfilled counter (instead of summaries). The backfill runs BEFORE the created_at_manual early-return and the date refresh (the manual flag protects the DATE only, phase 106 D1 — a manually-dated doc still gets its summary), is still fail-soft (LLMError/EmbeddingError → session rollback + summary_errors += 1, the doc row untouched, the date refresh still runs), and NEVER counts added/updated/pruned (so no sources_meta bump, no KB-overview/folder-summary regeneration — the content is unchanged, locked A2). Non-NULL summaries (owner-edited via phase 57 included) are never touched. The import log line (PLAN §9) gains summary_backfilled=N after summary_errors=N.
Old-contract retirement (task 06): tests/e2e/test_whole_document_context.py (the phase-24 full-text sentinel E2E) is DELETED — its contract no longer exists (locked A1/A6). Every test that pins the retired behavior is updated in place to pin the new one: markers to audit across tests/unit/, tests/integration/, tests/e2e/ — the ALREADY_IN_CONTEXT seed-read flows (test_agent.py, test_ls_tree_drilldown.py, test_read_truncation_cap.py), the <documents> full-text pins (test_prompts.py, test_prompts_dates.py, test_chat_gate.py), the markdown-summary pin (test_importer.py::test_markdown_file_never_gets_summary — inverted), the per-turn log-line pins (test_chat_gate.py — summary_hits=). A pin of UNCHANGED behavior stays untouched.
Copy gate (task 07): the read-tool copy is the lever the gate telemetry proved matters (the phase-70/72/94 precedent) — the real-model fixture battery re-runs before the phase completes and its verdict is recorded in TOOL_CALLING_TESTING.md (gate semantics: locked A7 below).

Tasks

01_all_doc_summaries.md — importer: every document gets a summary (markdown included) + the NULL-summary backfill on the unchanged path.
02_suggested_tier_selection.md — retriever: select_suggested (top-N, no floor) + BOR_SUGGESTED_DOCS.
03_summary_document_blocks.md — prompts: the "start here" suggestion blocks in the HIGH prompt (+ the preview fallback).
04_read_suggested_docs.md — agent: read on a suggested doc adds its full text; the read copy rewritten (both surfaces); the prompt-lock re-cut.
05_chat_turn_wiring.md — plan_turn: suggested seeding, select_related, citations, the durable record, suggested=N.
06_e2e_summary_context.md — new Playwright E2E; the phase-24 E2E retired; the old pins updated.
07_fixture_battery_gate.md — the real-model fixture battery re-run + the verdict recorded.

Testing & Quality

Unit: tests/unit/test_importer.py (markdown summarized; backfill success/failure; the manual-date case; the log-line counter), tests/unit/test_retriever.py (select_suggested / select_related rank order, cap, no floor, dedupe), tests/unit/test_prompts.py + test_prompts_dates.py (summary blocks, the date attribute, the preview fallback, the intro line; the LOW prompt byte-identical), tests/unit/test_prompt_lock.py (the re-cut TOOLS_SECTION anchors; PERSONA + LOW body untouched), tests/unit/test_agent.py (read-of-suggested success + re-read refusal + cap path), tests/unit/test_chat_gate.py (HIGH seeds 5 summaries — no full text in the prompt; related = rank 6+; both branches' durable record; suggested=N), tests/unit/test_config.py (the two new settings).
E2E: tests/e2e/test_summary_seed_context.py (new — run in isolation: uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov).
Regression: the remaining E2E suites green (the tool-flow suites re-targeted in task 06), and the battery gate (task 07).
Coverage: >90% on app/ (the validate.sh gate).

Completion Criteria

A grounded turn seeds the top-5 suggested documents as SUMMARY blocks (no full text in the prompt); the LLM reads what it needs and answers faster.
Every imported document (markdown included) has a summary; an existing NULL-summary doc backfills on the next sync (no content re-embed, no sources_meta bump, fail-soft).
read on a suggested doc adds its full text (cap/truncation/date line unchanged); a re-read of an already-read doc is refused.
done.sources = suggested + read (deduped); query_log + the log line record suggested + related + read; the log line carries suggested=N.
The deflection path and every SSE frame shape are byte-identical; PERSONA + the LOW body stay byte-locked.
uv run pytest green; coverage >90%; the new E2E green in isolation; the fixture battery gate PASS (task 07); uv run ruff check . && uv run pyright clean.
One --no-gpg-sign commit; the phase dir moves to complete/ by the pipeline gate.

Locked decisions

A1 — no kill switch (owner-confirmed 2026-09-15, roadmap confirmation): the summary-seed context IS the behavior; the legacy full-text top-2 seeding is gone and the phase-24 E2E is retired (not re-scoped).
A2 — every document gets a summary, no exceptions, including markdown (owner-confirmed 2026-09-15): generated for all new/changed docs at import; existing NULL rows backfill on the next sync as a summary-only pass (no content re-chunk/re-embed, no sources_meta bump); owner-edited (non-NULL) summaries are never touched.
A3 — the suggestion set = the top 5 distinct documents in fused rank order, NO cosine floor (BOR_SUGGESTED_DOCS default 5); the floor stays for the done-event related tier only (owner-confirmed 2026-09-15).
A4 — done.sources (the citation surface) = suggested + agent-read, deduped — a grounded turn always shows chips (owner-confirmed 2026-09-15).
A5 — a NULL summary at prompt time (a fail-soft miss) falls back to a 400-char content preview (BOR_SUGGESTION_PREVIEW_CHARS) — deterministic, no LLM call at chat time (owner-confirmed 2026-09-15).
A6 — A7 re-revised (owner directive, TODO L3, 2026-09-15): the retrieval path never seeds full texts; full text enters the context only through the capped read tool. PLAN.md is untouched (Protocol B) — the owner folds it in at the next redo.
A7 — the battery-gate semantics under summary seeding (owner-confirmed 2026-09-15): conditions 1 (every turn answers) and 2 (zero round-cap hits) still gate; condition 4 (fixture-mode contract accuracy ≥ 0.90) still gates; condition 3 (≥6/10 turns emit ≥1 tool call) is REPORTED, not gated — a summary-only answer is now the intended fast path, not tool abandonment. The full telemetry is recorded in TOOL_CALLING_TESTING.md regardless.

Commit

git add app/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): seed grounded turns with top-5 document summaries instead of full texts — every document summarized, read extends the context"

14 KiB Raw Blame History Unescape Escape