brain-of-reese/.agent/user_stories/document-summaries.md

Story: Document Summaries (lite-model summaries for non-markdown docs)

Phase: 30_document_summaries.md · E2E: tests/e2e/test_document_summaries.py

Narrative

As any user of Reese, I ask questions about notes that live in machine-formatted files (yaml, json, py, txt). Raw flags and keys embed badly, so retrieval misses those documents. I want a small model to analyze every non-markdown document at import time and store a natural-language summary with a pointer back to the source. When retrieval hits a summary, Reese follows the pointer and answers from the full source document — the summary is a retrieval target, never the answer.

• Given a knowledge base containing non-markdown A9 documents (yaml, yml, json, py, txt)
• When I run import_docs and then ask a question those documents answer
• Then each non-markdown document carries a lite-model summary (documents.summary plus one embedded is_summary chunk at position −1), markdown documents carry neither, and a question whose best match is a summary chunk gets an answer grounded in the full source document, with the hit counted in the per-turn log line (summary_hits=N).

Acceptance criteria

1. BOR_LLM_SUMMARY_MODEL (default lite) + non-streaming LLMClient.chat() — A5 extended (same OpenAI-compatible endpoint), empty/missing content raises LLMError (no silent summaries).
2. Migration 0004: documents.summary TEXT NULL + chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE (reversible, integration-tested up/down).
3. app/rag/summarizer.py: the SUMMARY_MODE prompt (document content capped at BOR_SUMMARY_MAX_CHARS, overflow marked with the shared truncation marker), the lite call, output validation, and the code-deterministic Source: <source>/<path> pointer line (never model-generated).
4. Importer: every non-markdown file gets its summary stored + indexed (best-effort — a lite failure logs, counts in summary_errors, and still leaves the document fully indexed); markdown files get neither; re-import replaces the old summary chunk (exactly one at a time).
5. is_summary flows through the retriever's vector and lexical candidate lists and fuse; TurnPlan.summary_hits counts hit chunks with is_summary whose parent document made the selected top-N context; the per-turn log line records summary_hits=N after fts_hits=N (PLAN §9 extension).
6. A summary hit resolves to its parent through the unchanged chunk→document mapping — the LLM receives the full source document (A7 revised: never truncated).
7. .env.example + README document BOR_LLM_SUMMARY_MODEL / BOR_SUMMARY_MAX_CHARS and the fail-soft behavior; the deterministic mock LLM answers SUMMARY_MODE with a byte-stable 24-token digest.
8. Unit + integration green, app/ coverage >90%, story E2E green in isolation, ruff + pyright clean, one --no-gpg-sign commit.

Playwright Mapping Rule

tests/e2e/test_document_summaries.py — one story, one file, run in isolation. It imports the story-dedicated fixture KB (tests/fixtures/summary_kb/: one yaml document with a tail sentinel + one markdown control document) through the real importer against the deterministic mock LLM, asserts the yaml document's import state (one embedded is_summary chunk at position −1 whose text is the byte-stable mock digest + the deterministic Source: pointer), asserts the summary chunk is the yaml document's best fused chunk, then asks the question whose best yaml match is that summary chunk and asserts the rendered answer quotes the document's tail sentinel RESE-SUMMARY-SENTINEL-7f3a (only possible if the full source document — not the summary digest — reached the LLM), the source chip cites the yaml path, the turn is not deflected, and the markdown control document carries no summary chunk.