feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc

.agent/phases/todo/30_document_summaries/01_lite_model_client.md

@@ -1,26 +0,0 @@
-# Task 01 — lite model setting + non-streaming chat()
-
-**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "The small model available on aipi.reeseapps.com is 'lite'." (enabler for "we need a small model to analyze non markdown documents")`
-**Story:** `.agent/user_stories/document-summaries.md`
-
-## Objective
-Make the aipi **`lite`** model callable from the app: a new model setting plus a non-streaming `LLMClient.chat()` one-shot completion method that the summarizer (task 03) and the phase-31 overview generator will use.
-
-## Work
-1. `app/config.py` — add `llm_summary_model: str = "lite"` (env `BOR_LLM_SUMMARY_MODEL`), documented like the other LLM settings.
-2. `app/rag/llm.py` — add to `LLMClient`:
-   - `async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str` — `chat.completions.create(model=model or self.settings.llm_summary_model, messages=…, temperature=0.2, max_tokens=2048, stream=False)`; returns the first choice's `message.content` stripped.
-   - Raise `LLMError` (wrapped, with the base URL in the message — same style as `chat_stream`) on any transport/HTTP/malformed failure, and on an empty/missing content field (a silent empty summary must never be stored).
-3. `.env.example` — document `BOR_LLM_SUMMARY_MODEL` (default `lite`).
-4. `README.md` — models section: add `lite` (document summaries — this phase; KB overview in phase 31) next to `turbo`/`embed`.
-
-- ASSUMPTION: the model name is `lite` per the TODO item; single (non-streaming) completion with `temperature=0.2`, `max_tokens=2048` — summaries/outlines are short, so a fixed budget is enough (no new setting).
-
-## Testing & Quality
-- Unit: `tests/unit/test_llm_client.py` — extend the existing mock-transport pattern: `chat()` returns content (trimmed); HTTP ≥400 → `LLMError`; empty content → `LLMError`; explicit `model=` overrides the default (`llm_summary_model`).
-- Coverage: **>90%** on this task's new/modified code (`app/` TOTAL ≥ pre-change).
-
-## Completion Criteria
-- [ ] `Settings().llm_summary_model == "lite"` by default; `BOR_LLM_SUMMARY_MODEL` env override works (config test).
-- [ ] `chat()` unit tests green; `uv run ruff check . && uv run pyright` clean.
-- [ ] No change to `embed`/`chat_stream` behavior (existing suite green).

.agent/phases/todo/30_document_summaries/02_migration_summary_columns.md

@@ -1,23 +0,0 @@
-# Task 02 — Migration 0004: summary columns
-
-**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "provide a textual summary of those documents with a pointer back to the source" (storage)`
-**Story:** `.agent/user_stories/document-summaries.md`
-
-## Objective
-Add the schema for storing a document's summary and for marking the extra summary chunk: `documents.summary TEXT NULL` and `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE`.
-
-## Work
-1. `alembic/versions/0004_summary_columns.py` — new revision (down_revision = the 0003 steering-notes revision, whatever `alembic/versions/` currently heads to):
-   - upgrade: `op.add_column("documents", sa.Column("summary", sa.Text(), nullable=True))`; `op.add_column("chunks", sa.Column("is_summary", sa.Boolean(), nullable=False, server_default=sa.text("false")))`.
-   - downgrade: drop both columns.
-2. `app/models.py` — `Document.summary: Mapped[str | None] = mapped_column(Text, default=None)` (comment: lite-model summary, phase 30); `Chunk.is_summary: Mapped[bool] = mapped_column(Boolean, default=False)` (comment: summary chunk, position −1, phase 30).
-3. `tests/integration/test_migration_0004.py` — mirror `tests/integration/test_migration_0002.py`'s style: upgrade to head → both columns exist, `is_summary` default `false`; downgrade to 0003 → both gone; upgrade again → back (round-trip).
-
-## Testing & Quality
-- Integration: the migration test above (real Postgres, as `test_migration_0002.py` does).
-- Coverage: models are exercised by the existing model tests; `app/` TOTAL ≥ pre-change.
-
-## Completion Criteria
-- [ ] `uv run alembic upgrade head` applies cleanly on the dev DB (and `alembic downgrade -1` + `upgrade head` round-trips).
-- [ ] `uv run pytest` green (including all pre-existing migration/importer tests — `is_summary` default keeps old rows valid).
-- [ ] `uv run ruff check . && uv run pyright` clean.

.agent/phases/todo/30_document_summaries/03_summarizer_module.md

@@ -1,30 +0,0 @@
-# Task 03 — app/rag/summarizer.py (prompt + lite call + pointer)
-
-**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source"`
-**Story:** `.agent/user_stories/document-summaries.md`
-
-## Objective
-Create the summarizer module: build the `lite` prompt for one document, call the model (task 01), validate the output, and return the summary text with a **code-deterministic** pointer line back to the source.
-
-## Work
-1. `app/config.py` — add `summary_max_chars: int = 12_000` (env `BOR_SUMMARY_MAX_CHARS`): the cap on document content sent to the lite model in one call.
-2. `app/rag/summarizer.py` (new) —
-   - `SUMMARY_MODE = "SUMMARY_MODE"` — marker constant the E2E mock keys on in the system prompt (same convention as `DEFLECT_MODE`).
-   - `build_summary_prompt(source: str, path: str, content: str, max_chars: int | None = None) -> tuple[str, str]` → `(system, user)`:
-     - system: `SUMMARY_MODE` + instruction — "Write a 3–6 sentence plain-text summary of this document in natural language. Cover what it configures/defines and its most important values. Do not use markdown. Do not invent anything that is not in the document."
-     - user: the document content, capped at *max_chars* (default `get_settings().summary_max_chars`); on overflow cut at the cap and append the shared `TRUNCATION_MARKER` (imported from `app.rag.retriever`).
-   - `async def generate_summary(llm, *, source: str, path: str, content: str) -> str` — calls `llm.chat([{"role":"system",…},{"role":"user",…}], model=llm.settings.llm_summary_model)`; validates non-empty after trim (else raise `LLMError` — the client already does this, but re-assert defensively); appends the deterministic pointer line: `f"\nSource: {source}/{path}"` (the pointer is **never** model-generated).
-3. `tests/unit/test_summarizer.py` (new) — fake LLM object (duck-typed `chat` + `settings`):
-   - prompt: system contains `SUMMARY_MODE`; user == full content when under cap; user truncated + `TRUNCATION_MARKER` when over cap (custom and default cap).
-   - generation: returned text = model text + pointer line `Source: <source>/<path>`; whitespace model text → `LLMError`; `LLMError` from the client propagates.
-
-- ASSUMPTION: the pointer is the literal line `Source: <source>/<path>` appended by code (the TODO's "pointer back to the source"); the model is told what to summarize but not to write the pointer.
-
-## Testing & Quality
-- Unit: the tests in Work step 3.
-- Coverage: **>90%** on `app/rag/summarizer.py`.
-
-## Completion Criteria
-- [ ] `generate_summary` returns a non-empty summary ending in the deterministic pointer line; all unit tests green.
-- [ ] `uv run ruff check . && uv run pyright` clean.
-- [ ] No app endpoint change yet (pipeline integration is task 05).

.agent/phases/todo/30_document_summaries/04_importer_summary_integration.md

@@ -1,39 +0,0 @@
-# Task 04 — Importer: generate, store, index summaries (best-effort)
-
-**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "a small model to analyze non markdown documents and provide a textual summary… The similarity search will have a higher chance of hitting those summaries than the original document"`
-**Story:** `.agent/user_stories/document-summaries.md`
-
-## Objective
-Hook summarization into the import pipeline: every **non-markdown** file gets a lite summary stored on `documents.summary` and indexed as one extra embedded chunk (`is_summary`, position −1) — best-effort, so a lite failure never loses the document.
-
-## Work
-1. `app/rag/importer.py`:
-   - `Embedder` protocol — add `async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str: ...` (task 01's `LLMClient.chat` already satisfies it; the protocol is what tests duck-type).
-   - `ImportSummary` — new counters `summaries: int = 0` and `summary_errors: int = 0`; include both in the `log()` summary line (`… summaries=%d summary_errors=%d`).
-   - `_index_file` — **after** the existing doc+chunks commit (so the document is safe):
-     - If `full_path.suffix.lower()` is **not** in `(".md", ".markdown")`: try
-       `summary = await generate_summary(llm, source=source, path=rel, content=content)`;
-       delete any existing `is_summary` chunk of this document (re-import replacement);
-       add `Chunk(document_id=doc.id, position=-1, content=summary, is_summary=True)`;
-       `vector = (await llm.embed([summary]))[0]`; set `chunk.embedding = vector`, `doc.summary = summary`; `session.commit()`; `summary.summaries += 1`; log `import: summary source=%s path=%s chars=%d`.
-     - On `LLMError | EmbeddingError`: `session.rollback()`, `summary.summary_errors += 1`, log `import: summary failed source=%s path=%s — %s`, and **continue** (the document row + content chunks stay committed; `doc.summary` remains NULL).
-     - Markdown files: no summary, `doc.summary` stays NULL, no `is_summary` chunk.
-2. `scripts/import_docs.py` — the final `print` gains `summaries=%d summary_errors=%d` from the `ImportSummary`.
-3. `tests/unit/test_importer.py` — extend the existing fake embeder with a `chat` method (deterministic: returns `"Summary of " + first token of content`; raises `LLMError` when the content contains the sentinel word `SUMMARY-BLOWUP`):
-   - non-md file (e.g. `.yaml`) → after import: `doc.summary` set, exactly one `is_summary` chunk at position −1 with a non-NULL embedding; `summary.summaries == 1`.
-   - `.md` file → `doc.summary is None`, no `is_summary` chunk, `summaries == 0`.
-   - fail-soft: content with `SUMMARY-BLOWUP` → document fully indexed (chunks present, embedding set), `doc.summary is None`, `summary_errors == 1`, no exception.
-   - replacement: re-import the same file with changed content → still exactly **one** `is_summary` chunk (old one deleted), new text.
-   - `log()` line includes the new counters (existing log-format test updated accordingly).
-
-- ASSUMPTION: the summary chunk sits at `position = -1` (content chunks stay 0-based in order) so chunk ordering and the viewer are undisturbed; only one summary chunk per document at a time.
-- ASSUMPTION: "non-markdown" = suffix not in `(.md, .markdown)` — every other A9 format (txt, yaml, yml, json, py) gets a summary.
-
-## Testing & Quality
-- Unit: the tests in Work step 3 (reuse the existing test file's session/fake-embedder fixtures).
-- Coverage: **>90%** on the modified `app/rag/importer.py`; `app/` TOTAL ≥ pre-change.
-
-## Completion Criteria
-- [ ] All new unit tests green; existing importer tests green (protocol change is additive).
-- [ ] `uv run ruff check . && uv run pyright` clean.
-- [ ] A manual `uv run python -m scripts.import_docs` run against the dev KB logs `summaries=N` for the non-md docs (observable in the importer log line, PLAN §9).

.agent/phases/todo/30_document_summaries/05_pipeline_summary_resolution.md

@@ -1,31 +0,0 @@
-# Task 05 — Pipeline: summary hits resolve to the full source document
-
-**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "the chat LLM can retrieve the source pointed to by the summary document (so as part of the pipeline: if retrieval == summary, fetch documents referenced by summary)"`
-**Story:** `.agent/user_stories/document-summaries.md`
-
-## Objective
-Make the retrieval + chat pipeline summary-aware: carry `is_summary` through both candidate lists into the fused result, and record in the per-turn log how many of the selected documents were hit via their summary chunk. The context assembly itself is **unchanged** — a summary chunk's parent *is* the source document, and `select_documents` already feeds the full document (A7 revised / phase 24); this task makes that resolution explicit, asserted, and observable.
-
-## Work
-1. `app/rag/retriever.py`:
-   - `RetrievedChunk` — add field `is_summary: bool = False` (documented: True for the lite-model summary chunk, position −1).
-   - `_vector_candidates` — select `Chunk.is_summary` and pass it into the constructed `RetrievedChunk`s.
-   - `_LEXICAL_SQL` — add `c.is_summary AS is_summary`; `_lexical_candidates` passes `row.is_summary`.
-   - `fuse` needs no change (dataclass passthrough) — but assert in tests that the flag survives fusion.
-2. `app/api/chat.py`:
-   - `TurnPlan` — add `summary_hits: int = 0` (count of selected-document hit chunks with `is_summary`).
-   - `plan_turn` — after the selected-docs decision is known for each branch, compute `summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in selected_ids)` and store it on the `TurnPlan` (both HIGH and LOW branches).
-   - Per-turn log line — add `summary_hits=%d` after `fts_hits=%d` (PLAN §9 line extension; record it in the phase's locked decisions). Update any existing test that asserts the log line format verbatim.
-   - **No change** to `build_high_prompt`/`build_deflect_prompt` inputs or to `select_documents` — the full source document of a summary hit already lands in `<documents>`; task 06's E2E proves it end-to-end.
-3. `tests/unit/test_retriever.py` — `is_summary` survives: vector candidates (flag set), lexical candidates (flag set), `fuse` (both a double-hit and a summary-only lexical hit keep the flag; default stays `False` for legacy chunks).
-4. `tests/unit/test_chat_gate.py` — `plan_turn`: a summary chunk on a selected top document → `summary_hits == 1`; a summary chunk on a document **outside** the top-N selection → not counted; no summaries → `0` (existing cases unchanged).
-
-## Testing & Quality
-- Unit: the tests in Work steps 3–4; existing chat-gate and retriever tests stay green (new field is defaulted).
-- Coverage: **>90%** on modified `app/rag/retriever.py` + `app/api/chat.py`; `app/` TOTAL ≥ pre-change.
-
-## Completion Criteria
-- [ ] A summary chunk retrieved via vector **or** lexical carries `is_summary=True` through `retrieve()` (unit).
-- [ ] The per-turn log line (PLAN §9) now reads `… fts_hits=… summary_hits=… …` and existing log-format tests are updated + green.
-- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
-- [ ] No prompt change: HIGH/LOW prompts byte-identical for summary-less KBs (covered by existing prompt tests).