brain-of-reese/.agent/phases/complete/30_document_summaries/00_phase.md

Phase 30 — Document Summaries (lite-model summaries for non-markdown documents)

Source: TODO.md L3 — "One issue I'm having is bad context for the embedder which causes poor retrieval results… we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source… if retrieval == summary, fetch documents referenced by summary… The small model available on aipi.reeseapps.com is 'lite'." Story: .agent/user_stories/document-summaries.md Context: The importer (app/rag/importer.py::_index_file — chunk → embed → upsert per file, A9 scope), hybrid retrieval (app/rag/retriever.py — A7: cosine ∪ FTS, RRF, chunk→parent-document mapping, full-document context never truncated, phase 24), the locked persona prompts (app/rag/prompts.py), the aipi client (app/rag/llm.py — A5: turbo chat streaming + embed embeddings), and the E2E mock LLM (tests/e2e/mock_llm.py — deterministic, keys on system-prompt markers like DEFLECT_MODE / <tuning>).

Objective

Give every non-markdown A9 document (txt, yaml, yml, json, py) a natural-language summary generated at import time by the aipi lite model. The summary is stored on the document (documents.summary) and indexed as one extra embedded chunk (chunks.is_summary), so hybrid search has a well-embedding natural-language target to hit instead of the badly-formatted raw text. A summary hit resolves to its parent (the source document) — the existing chunk→document mapping then feeds the full source document to the LLM, implementing the TODO's "if retrieval == summary, fetch the documents referenced by the summary" step. The per-turn log line records how many summary hits landed in the selected context.

Dependencies

• 29_tuning_nav_link (complete) — the latest finished phase (sequencing only).
• Substantively builds on: 02_story_import_documents / 24_whole_document_context (import pipeline + full-document context contract), 09_story_retrieval_quality (A7 hybrid retrieval the summary chunk flows through unchanged), 01_infrastructure (models/alembic, LLM client, E2E mock).

Tasks

1. 01_lite_model_client.md — BOR_LLM_SUMMARY_MODEL (default lite) + non-streaming LLMClient.chat() for the lite model.
2. 02_migration_summary_columns.md — Alembic 0004: documents.summary TEXT NULL + chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE.
3. 03_summarizer_module.md — app/rag/summarizer.py: SUMMARY_MODE prompt (capped input), lite call, output validation + deterministic Source: <source>/<path> pointer line.
4. 04_importer_summary_integration.md — importer generates/stores/indexes summaries for non-md files (best-effort fail-soft) + summary counters.
5. 05_pipeline_summary_resolution.md — is_summary through the retriever, summary_hits in TurnPlan + the per-turn log line; full source document on summary hit (existing mapping, asserted).
6. 06_mock_and_e2e.md — deterministic lite in mock_llm.py, sentinel fixture, tests/e2e/test_document_summaries.py, story file, README, commit.

Testing & Quality

• Unit: summarizer (prompt/cap/pointer/errors), importer (summary happy path, md exclusion, fail-soft, replacement on re-import), retriever (is_summary through both candidate lists + fuse), chat gate (TurnPlan.summary_hits), LLM client (chat()).
• Integration: migration 0004 up/down.
• Coverage: >90% on app/ (uv run pytest --cov=app --cov-report=term-missing, TOTAL ≥ pre-change number).
• E2E (mandatory, A16): tests/e2e/test_document_summaries.py — one story, run in isolation (uv run pytest tests/e2e/test_document_summaries.py -v --no-cov); proves summary hit → full source document reaches the answer (sentinel in the raw doc, absent from the mock summary).
• All existing E2E suites stay green (new columns are defaulted; all existing chunks have is_summary=false).

Completion Criteria

• After uv run python -m scripts.import_docs, every non-markdown fixture/doc has documents.summary set and exactly one is_summary chunk (position −1, embedded); markdown docs have neither.
• A question whose best match is a summary chunk yields an answer grounded in the full source document (E2E sentinel) and the per-turn log line shows summary_hits>=1.
• A lite-model failure during import does not drop the document — it is indexed without a summary, logged, and counted (summary_errors).
• uv run pytest green; uv run pytest --cov=app --cov-report=term-missing TOTAL ≥ pre-change number (app/ >90%).
• uv run pytest tests/e2e/test_document_summaries.py -v --no-cov green in isolation; existing suites (test_chat_rag.py, test_retrieval_quality.py, test_import_documents.py, test_whole_document_context.py) stay green.
• uv run ruff check . && uv run pyright clean.
• .agent/user_stories/document-summaries.md exists.
• .env.example + README document BOR_LLM_SUMMARY_MODEL / BOR_SUMMARY_MAX_CHARS and the summary behavior.
• One --no-gpg-sign commit staging only this phase's files (e.g. feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc); .agent/phases/todo/30_document_summaries/ moved to .agent/phases/complete/.

Locked decisions

• A5 extended, not revised — the lite model is served by the same OpenAI-compatible endpoint (https://aipi.reeseapps.com/v1) via a new BOR_LLM_SUMMARY_MODEL setting (default lite); no new model management, no new package.
• A7 untouched — hybrid retrieval logic is unchanged; the summary is an ordinary chunk, so it flows through the existing cosine ∪ FTS ∪ RRF path and the chunk→document mapping. The "fetch the referenced document" step is the existing full-document context contract (phase 24) — never truncated.
• A9 untouched — "non-markdown" means every already-imported A9 document except md/markdown. The TODO's quadlet-file example is out of scope: .quadlet is not an A9 format and BOR_IMPORT_EXTENSIONS may only narrow the locked set (flagged at roadmap confirmation; importing quadlet files would require an owner-permission A9 revision).
• A13 — migration 0004 adds two columns (documents.summary, chunks.is_summary); no table rework, both reversible.
• Summary generation is best-effort — a lite failure logs + counts (summary_errors) and the file is still indexed without a summary (same fail-soft spirit as the per-file EmbeddingError handling, but weaker: the doc is already committed).
• Pointer is code-deterministic — the Source: <source>/<path> line is appended by summarizer.py, never trusted to the model.
• A16 honoured — one dedicated story E2E suite; E2E stays deterministic via the mock LLM's SUMMARY_MODE marker.
• A17 honoured — one atomic --no-gpg-sign commit.