feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc

README.md

@@ -316,6 +316,32 @@ absent. The top `BOR_TOP_N_DOCS` full documents are still what the LLM sees.
 `chunk_hits`, `deflected`, `sources`, `latency_ms`) — the raw material for
 tuning: `psql … -c 'SELECT question, top_score, fts_hits, deflected FROM query_log ORDER BY created_at DESC LIMIT 20'`.
 
+## Document summaries (non-markdown)
+
+Raw yaml/json/py/txt embeds badly — flags and keys are not language, so
+retrieval can miss exactly the documents that are all configuration. At
+import time, every **non-markdown** A9 document is summarized by the aipi
+`lite` model (one-shot `LLMClient.chat`, `BOR_LLM_SUMMARY_MODEL`, default
+`lite`):
+
+* the summary is stored on `documents.summary` **and** indexed as one
+  extra embedded chunk (`chunks.is_summary`, position −1), so hybrid
+  search has a natural-language target to hit instead of the raw text;
+* the last line is a **code-deterministic** pointer — `Source:
+  <source>/<path>` — appended by the app, never model-generated;
+* the model only sees the first `BOR_SUMMARY_MAX_CHARS` (default 12000)
+  characters of the document; overflow is cut and marked with the shared
+  `[…truncated…]` marker.
+
+A summary hit resolves through the normal chunk→document mapping: the
+LLM receives the **full source document** (never the summary alone,
+never truncated). Markdown files are natural language already and get no
+summary. Summary generation is best-effort: if `lite` fails for a file,
+the document is still indexed (without a summary), the failure is logged
+and counted in the import summary line (`summaries=N summary_errors=N`).
+On a chat turn, the per-turn log line records `summary_hits=N` — how many
+summary chunks of the selected context the question landed on.
+
 ## Debugging
 
 `debugpy` is **off by default** and *never imported* unless you opt in —
@@ -416,6 +442,7 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
 | `BOR_LLM_API_KEY` | — (falls back to `$AIPI_KEY`) | aipi API key |
 | `BOR_LLM_CHAT_MODEL` | `turbo` | chat model |
 | `BOR_LLM_EMBED_MODEL` | `embed` | embedding model |
+| `BOR_LLM_SUMMARY_MODEL` | `lite` | one-shot (non-streaming) completions: document summaries at import (phase 30) and the KB overview (phase 31) |
 | `BOR_EMBEDDING_DIM` | `768` | vector dimension (fixed at table creation) |
 | `BOR_TOP_N_DOCS` | `2` | full documents fed to the LLM |
 | `BOR_RELEVANCE_THRESHOLD` | `0.62` | answer when best cosine ≥ this **or** an FTS hit; below + no FTS ⇒ honest deflection |
@@ -426,6 +453,7 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
 | `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs; `import_docs` clones/pulls them into `BOR_SOURCES_DIR` and indexes the checkouts (see *Git-based sources*) |
 | `BOR_SOURCES_DIR` | `~/bor-sources` | where the `BOR_GIT_SOURCES` repos are cloned/pulled (one subdirectory per repo) |
 | `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
+| `BOR_SUMMARY_MAX_CHARS` | `12000` | cap on document content sent to the `lite` summary model at import (see *Document summaries*) |
 | `BOR_SUGGESTIONS` | built-in list | JSON list of onboarding chips |
 | `BOR_ADMIN_PASSWORD` | *(required)* | the single admin's password (plaintext, `.env`); app refuses to start when empty |
 | `BOR_SESSION_SECRET` | *(required)* | signing key for the `bor_session` cookie; `python -c 'import secrets;print(secrets.token_hex(32))'` |