chore(agent): phase roadmap from TODO.md — phase 118 summary seed context (top-5 summary seeding, all-doc summaries)

.agents/phases/todo/118_summary_seed_context/00_phase.md

@@ -0,0 +1,58 @@
+# Phase 118 — Summary seed context: top-5 "start here" suggestions (TODO L3)
+
+**Source:** `TODO.md` L3 — "I want to change the way the initial chat context works. Right now we feed two entire documents into context before the LLM responds. That was a good idea before we had complex tools and document summaries, but now we have faster ways for the LLM to identify what information it needs without reading 2 files. Moving forward, after the question is embedded, I want the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content — basically a 'start here if these summaries seem right to you' approach. That way the LLM responds faster and doesn't begin by reading massive amounts of data that might be irrelevant because the similarity search had a bad day."
+**Story:** n/a (owner-directed RAG behavior change; extends the phase-30 summaries, the phase-37/70/94 agent tools, and the phase-113 tiering assets).
+**Context:** A grounded (HIGH) turn currently seeds `build_high_prompt` with the **full text** of the top-2 cited documents (`select_documents_tiered`'s cited tier — floor `BOR_SOURCE_USEFULNESS_FLOOR` 0.35, ceiling `BOR_TOP_N_DOCS` 2, `app/api/chat.py:plan_turn`), and `run_agent` receives them as `seed_docs` whose `read` is refused (`ALREADY_IN_CONTEXT` — the full text is already in the prompt). Summaries exist on `Document.summary` (phase-30 lite digest — currently **non-markdown only**; NULL for markdown docs, pre-30 rows and fail-soft misses) and are embedded as one `is_summary` chunk (position −1) per document. LOCKED A7 (full text of the top-2, never truncated, on the retrieval path) is **re-revised by the owner directive in this TODO**: full text enters the context only through the capped `read` tool (locked A6 below).
+
+## Objective
+Grounded turns seed the top 5 related documents as **summary** blocks — a "start here if these summaries seem right to you" starting point — and the LLM extends its context by `read`-ing only what it needs; every document (markdown included) has a stored summary (generated for all new/changed docs at import, backfilled on the next sync for existing NULL rows). Answers get faster and never start by ingesting massive possibly-irrelevant full texts.
+
+## Dependencies
+- `117_tool_call_compact` (complete) — the last completed phase; this phase rewrites the `read`-tool copy the battery gate pins, so the gate (task 07) runs against its final state.
+
+## Design (shared by all tasks — the executor reads this, not the chat)
+- **Suggested tier (task 02):** new pure `select_suggested(chunks, n=None)` in `app/rag/retriever.py` — the top-N DISTINCT parent documents in fused rank order (the same stable score-descending walk as `select_documents_tiered` — a document's rank is fixed by its FIRST seen chunk), **no cosine floor** (suggestions are opt-in starting points, not citations — the floor stays for the `done`-event related tier only). `n` defaults to the new setting `suggested_docs` (`BOR_SUGGESTED_DOCS`, default **5**, validator `>= 1`).
+- **Suggestion blocks (task 03):** the HIGH prompt keeps the `<documents>` section tag and the per-block `<document …>` markup (the E2E mock keys off both), but each block's BODY becomes the document's **summary**: the identity attributes (`source`/`path`/`title`/`date` — the phase-106 D5 `date` stays, always present) + `doc.summary` (stripped). Defensive fallback ONLY when `doc.summary` is None/whitespace (a fail-soft miss — locked A5): the first `suggestion_preview_chars` (`BOR_SUGGESTION_PREVIEW_CHARS`, default **400**) content characters + the shared `TRUNCATION_MARKER` — no LLM call at chat time. A new pinned intro line (a module constant in `app/rag/prompts.py`) precedes the blocks inside the section: the start-here framing — the blocks are summaries of the top-ranked documents; if one seems right, `read` that document (combined `source/path`) to add its full content; answer from what you have and what you read; cite the document(s) you used by path.
+- **Agent `read` (task 04):** with summary seeds, `read` on a suggested document **adds its full text** — the existing mechanics are all unchanged (the `BOR_READ_MAX_CHARS` cap + `TRUNCATION_MARKER` + `READ_TRUNCATION_NOTICE` + `ToolResultPiece`/SSE `tool_result` frame, the phase-106 D5 `date:` line, `holder.read_docs` append, `tool_calls` increment). The dedupe set in `_execute_tool`'s read branch becomes **`holder.read_docs` only** (the seeds drop out — a suggested doc is a summary, not full text), so `ALREADY_IN_CONTEXT` fires only for a document ALREADY READ into full-text context (its copy stays accurate: "the full text is already in your prompt"). The `read` tool description in `AGENT_TOOLS` (`app/rag/agent.py`) and the `read` clause of `TOOLS_SECTION` (`app/rag/prompts.py`) are rewritten for the summary-seed mode — SINGLE mode (no kill switch, locked A1): the `<documents>` section shows summaries, `read` adds a document's full text, and re-reading an already-read document is refused. The `ls`/`grep` clauses and the discipline rules (one call per reply, never repeat) stay byte-identical. `tests/unit/test_prompt_lock.py` byte-pins `TOOLS_SECTION` (sha256 + length + prefix/suffix): re-cut its anchors in the same change; `PERSONA` and the LOW `DEFLECT_MODE` body MUST stay byte-identical (the lock pins those too — the E2E mock keys on the markers' presence).
+- **Turn wiring (task 05):** `plan_turn` computes BOTH tiers once, for BOTH branches: `suggested = select_suggested(chunks)` and `related = select_related(chunks, excluded={suggested ids}, cap=settings.related_max_docs)` (new pure helper in `app/rag/retriever.py` — the same rank walk, skipping the excluded ids, no floor; with a 5-doc suggestion the related tier is rank order 6..n, the phase-113 UI "nearby docs" row). The HIGH branch: `build_high_prompt(suggested, …)` and `run_agent(..., seed_docs=suggested)`; the LOW branch: prompt BYTE-IDENTICAL (weak-hit titles only, `DEFLECT_MODE`, no tools, chips — A8 gate untouched) but the TurnPlan still carries suggested + related for the durable record. `TurnPlan.docs` is RENAMED `suggested_docs` (same dataclass position); `summary_hits` is redefined as the hit chunks with `is_summary` whose parent document is in the SUGGESTED set. Citations: `done.sources` = suggested + agent-read, deduped by `(source, path)` order-preserving (locked A4 — a grounded turn always shows chips; the deflected turn keeps `[]`). Durable record (LOCKED A3 — the log records retrieval, not citations): `query_log.sources` + the per-turn log line = suggested + related + read, deduped. The per-turn log line gains `suggested=N` after `summary_hits=N` (PLAN §9 extension). `source_usefulness_floor` stops being a seeding input (it may stay for other consumers — the executor checks; the SETTING itself is never removed).
+- **Summaries for every document (task 01):** the importer's summary step covers **every** new/changed A9 document — the markdown exclusion (`_import_one_file`'s `if suffix in (".md", ".markdown"): return`) is removed. **Backfill:** on the UNCHANGED branch (same `content_hash`), a document whose `summary is None` gets the same best-effort `_store_summary` pass — summary text stored on `doc.summary` + one new embedded `is_summary` chunk — with a `backfill=True` flag that counts success in a NEW `ImportSummary.summary_backfilled` counter (instead of `summaries`). The backfill runs BEFORE the `created_at_manual` early-return and the date refresh (the manual flag protects the DATE only, phase 106 D1 — a manually-dated doc still gets its summary), is still fail-soft (`LLMError`/`EmbeddingError` → session rollback + `summary_errors += 1`, the doc row untouched, the date refresh still runs), and NEVER counts `added`/`updated`/`pruned` (so no `sources_meta` bump, no KB-overview/folder-summary regeneration — the content is unchanged, locked A2). Non-NULL summaries (owner-edited via phase 57 included) are never touched. The import log line (PLAN §9) gains `summary_backfilled=N` after `summary_errors=N`.
+- **Old-contract retirement (task 06):** `tests/e2e/test_whole_document_context.py` (the phase-24 full-text sentinel E2E) is DELETED — its contract no longer exists (locked A1/A6). Every test that pins the retired behavior is updated in place to pin the new one: markers to audit across `tests/unit/`, `tests/integration/`, `tests/e2e/` — the `ALREADY_IN_CONTEXT` seed-read flows (`test_agent.py`, `test_ls_tree_drilldown.py`, `test_read_truncation_cap.py`), the `<documents>` full-text pins (`test_prompts.py`, `test_prompts_dates.py`, `test_chat_gate.py`), the markdown-summary pin (`test_importer.py::test_markdown_file_never_gets_summary` — inverted), the per-turn log-line pins (`test_chat_gate.py` — `summary_hits=`). A pin of UNCHANGED behavior stays untouched.
+- **Copy gate (task 07):** the `read`-tool copy is the lever the gate telemetry proved matters (the phase-70/72/94 precedent) — the real-model fixture battery re-runs before the phase completes and its verdict is recorded in `TOOL_CALLING_TESTING.md` (gate semantics: locked A7 below).
+
+## Tasks
+1. `01_all_doc_summaries.md` — importer: every document gets a summary (markdown included) + the NULL-summary backfill on the unchanged path.
+2. `02_suggested_tier_selection.md` — retriever: `select_suggested` (top-N, no floor) + `BOR_SUGGESTED_DOCS`.
+3. `03_summary_document_blocks.md` — prompts: the "start here" suggestion blocks in the HIGH prompt (+ the preview fallback).
+4. `04_read_suggested_docs.md` — agent: `read` on a suggested doc adds its full text; the `read` copy rewritten (both surfaces); the prompt-lock re-cut.
+5. `05_chat_turn_wiring.md` — `plan_turn`: suggested seeding, `select_related`, citations, the durable record, `suggested=N`.
+6. `06_e2e_summary_context.md` — new Playwright E2E; the phase-24 E2E retired; the old pins updated.
+7. `07_fixture_battery_gate.md` — the real-model fixture battery re-run + the verdict recorded.
+
+## Testing & Quality
+- Unit: `tests/unit/test_importer.py` (markdown summarized; backfill success/failure; the manual-date case; the log-line counter), `tests/unit/test_retriever.py` (`select_suggested` / `select_related` rank order, cap, no floor, dedupe), `tests/unit/test_prompts.py` + `test_prompts_dates.py` (summary blocks, the `date` attribute, the preview fallback, the intro line; the LOW prompt byte-identical), `tests/unit/test_prompt_lock.py` (the re-cut `TOOLS_SECTION` anchors; `PERSONA` + LOW body untouched), `tests/unit/test_agent.py` (read-of-suggested success + re-read refusal + cap path), `tests/unit/test_chat_gate.py` (HIGH seeds 5 summaries — no full text in the prompt; related = rank 6+; both branches' durable record; `suggested=N`), `tests/unit/test_config.py` (the two new settings).
+- E2E: `tests/e2e/test_summary_seed_context.py` (new — run in isolation: `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov`).
+- Regression: the remaining E2E suites green (the tool-flow suites re-targeted in task 06), and the battery gate (task 07).
+- Coverage: **>90%** on `app/` (the validate.sh gate).
+
+## Completion Criteria
+- [ ] A grounded turn seeds the top-5 suggested documents as SUMMARY blocks (no full text in the prompt); the LLM `read`s what it needs and answers faster.
+- [ ] Every imported document (markdown included) has a `summary`; an existing NULL-summary doc backfills on the next sync (no content re-embed, no `sources_meta` bump, fail-soft).
+- [ ] `read` on a suggested doc adds its full text (cap/truncation/date line unchanged); a re-read of an already-read doc is refused.
+- [ ] `done.sources` = suggested + read (deduped); `query_log` + the log line record suggested + related + read; the log line carries `suggested=N`.
+- [ ] The deflection path and every SSE frame shape are byte-identical; `PERSONA` + the LOW body stay byte-locked.
+- [ ] `uv run pytest` green; coverage >90%; the new E2E green in isolation; the fixture battery gate PASS (task 07); `uv run ruff check . && uv run pyright` clean.
+- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
+
+## Locked decisions
+- **A1 — no kill switch (owner-confirmed 2026-09-15, roadmap confirmation):** the summary-seed context IS the behavior; the legacy full-text top-2 seeding is gone and the phase-24 E2E is retired (not re-scoped).
+- **A2 — every document gets a summary, no exceptions, including markdown (owner-confirmed 2026-09-15):** generated for all new/changed docs at import; existing NULL rows backfill on the next sync as a summary-only pass (no content re-chunk/re-embed, no `sources_meta` bump); owner-edited (non-NULL) summaries are never touched.
+- **A3 — the suggestion set = the top 5 distinct documents in fused rank order, NO cosine floor** (`BOR_SUGGESTED_DOCS` default 5); the floor stays for the `done`-event related tier only (owner-confirmed 2026-09-15).
+- **A4 — `done.sources` (the citation surface) = suggested + agent-read, deduped** — a grounded turn always shows chips (owner-confirmed 2026-09-15).
+- **A5 — a NULL summary at prompt time (a fail-soft miss) falls back to a 400-char content preview** (`BOR_SUGGESTION_PREVIEW_CHARS`) — deterministic, no LLM call at chat time (owner-confirmed 2026-09-15).
+- **A6 — A7 re-revised (owner directive, TODO L3, 2026-09-15):** the retrieval path never seeds full texts; full text enters the context only through the capped `read` tool. `PLAN.md` is untouched (Protocol B) — the owner folds it in at the next redo.
+- **A7 — the battery-gate semantics under summary seeding (owner-confirmed 2026-09-15):** conditions 1 (every turn answers) and 2 (zero round-cap hits) still gate; condition 4 (fixture-mode contract accuracy ≥ 0.90) still gates; condition 3 (≥6/10 turns emit ≥1 tool call) is REPORTED, not gated — a summary-only answer is now the intended fast path, not tool abandonment. The full telemetry is recorded in `TOOL_CALLING_TESTING.md` regardless.
+
+## Commit
+```bash
+git add app/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): seed grounded turns with top-5 document summaries instead of full texts — every document summarized, read extends the context"
+```