feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc

.agent/user_stories/document-summaries.md

@@ -0,0 +1,70 @@
+# Story: Document Summaries (lite-model summaries for non-markdown docs)
+
+**Phase:** `30_document_summaries.md` · **E2E:** `tests/e2e/test_document_summaries.py`
+
+## Narrative
+
+As **any user of Reese**, I ask questions about notes that live in
+machine-formatted files (yaml, json, py, txt). Raw flags and keys embed
+badly, so retrieval misses those documents. I want a small model to
+**analyze every non-markdown document at import time** and store a
+natural-language summary with a pointer back to the source. When
+retrieval hits a summary, Reese follows the pointer and answers from the
+**full source document** — the summary is a retrieval target, never the
+answer.
+
+- **Given** a knowledge base containing non-markdown A9 documents (yaml,
+  yml, json, py, txt)
+- **When** I run `import_docs` and then ask a question those documents
+  answer
+- **Then** each non-markdown document carries a `lite`-model summary
+  (`documents.summary` plus one embedded `is_summary` chunk at position
+  −1), markdown documents carry neither, and a question whose best match
+  is a summary chunk gets an answer grounded in the **full source
+  document**, with the hit counted in the per-turn log line
+  (`summary_hits=N`).
+
+## Acceptance criteria
+1. `BOR_LLM_SUMMARY_MODEL` (default `lite`) + non-streaming
+   `LLMClient.chat()` — A5 extended (same OpenAI-compatible endpoint),
+   empty/missing content raises `LLMError` (no silent summaries).
+2. Migration 0004: `documents.summary TEXT NULL` +
+   `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE` (reversible,
+   integration-tested up/down).
+3. `app/rag/summarizer.py`: the `SUMMARY_MODE` prompt (document content
+   capped at `BOR_SUMMARY_MAX_CHARS`, overflow marked with the shared
+   truncation marker), the lite call, output validation, and the
+   **code-deterministic** `Source: <source>/<path>` pointer line (never
+   model-generated).
+4. Importer: every non-markdown file gets its summary stored + indexed
+   (best-effort — a lite failure logs, counts in `summary_errors`, and
+   still leaves the document fully indexed); markdown files get neither;
+   re-import replaces the old summary chunk (exactly one at a time).
+5. `is_summary` flows through the retriever's vector and lexical
+   candidate lists and `fuse`; `TurnPlan.summary_hits` counts hit chunks
+   with `is_summary` whose parent document made the selected top-N
+   context; the per-turn log line records `summary_hits=N` after
+   `fts_hits=N` (PLAN §9 extension).
+6. A summary hit resolves to its parent through the unchanged
+   chunk→document mapping — the LLM receives the full source document
+   (A7 revised: never truncated).
+7. `.env.example` + README document `BOR_LLM_SUMMARY_MODEL` /
+   `BOR_SUMMARY_MAX_CHARS` and the fail-soft behavior; the deterministic
+   mock LLM answers `SUMMARY_MODE` with a byte-stable 24-token digest.
+8. Unit + integration green, `app/` coverage >90%, story E2E green in
+   isolation, `ruff` + `pyright` clean, one `--no-gpg-sign` commit.
+
+## Playwright Mapping Rule
+`tests/e2e/test_document_summaries.py` — one story, one file, run in
+isolation. It imports the story-dedicated fixture KB
+(`tests/fixtures/summary_kb/`: one yaml document with a tail sentinel +
+one markdown control document) through the real importer against the
+deterministic mock LLM, asserts the yaml document's import state (one
+embedded `is_summary` chunk at position −1 whose text is the byte-stable
+mock digest + the deterministic `Source:` pointer), asserts the summary
+chunk is the yaml document's best fused chunk, then asks the question
+whose best yaml match is that summary chunk and asserts the rendered
+answer quotes the document's tail sentinel `RESE-SUMMARY-SENTINEL-7f3a`
+(only possible if the full source document — not the summary digest —
+reached the LLM), the source chip cites the yaml path, the turn is not
+deflected, and the markdown control document carries no summary chunk.