feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts

2026-08-25 20:22:51 -04:00
parent 572a4190a6
commit 0654b304e1
23 changed files with 2276 additions and 34 deletions
@@ -0,0 +1,80 @@
+# Story: KB Overview in the System Prompt (lite-generated knowledge-base outline)
+
+**Phase:** `31_kb_overview_prompt.md` · **E2E:** `tests/e2e/test_kb_overview.py`
+
+## Narrative
+
+As **any user of Reese**, I ask questions without knowing what the
+knowledge base actually covers. Retrieval only reveals the answer
+documents after it has run, and when it finds nothing the deflection has
+no picture of what Reese DOES know. I want the system prompt to carry,
+on every turn, a plain-text outline of the **basic categories** of
+everything that has been read — so the agent knows roughly what its
+knowledge base contains before retrieval returns documents. The outline
+is **generated by the `lite` model** from the document catalogue
+(source, path, title, first summary line per document) and **stored in
+a single row**, so it is regenerated whenever an import changes the
+knowledge base — never per chat turn.
+
+- **Given** a knowledge base that was imported (with at least one
+  document added or updated, or no outline stored yet)
+- **When** I ask any question
+- **Then** the system prompt of that turn — grounded (HIGH) or
+  deflected (LOW) — contains a `<knowledge_base>` section with the
+  stored outline (budgeted by `BOR_KB_OVERVIEW_MAX_CHARS`, overflow
+  marked with the shared `[…truncated…]` marker), and the per-turn log
+  line records `kb_chars=N`; when no outline is stored, both prompts
+  are byte-identical to the pre-phase text.
+
+## Acceptance criteria
+1. Migration 0005: single-row `kb_overview` table (`id INTEGER PK
+   DEFAULT 1`, `content TEXT NOT NULL DEFAULT ''`,
+   `updated_at TIMESTAMPTZ`) + `KbOverview` model — no other schema
+   change (reversible, integration-tested up/down).
+2. `app/rag/overview.py`: the `KB_OVERVIEW_MODE` prompt builder
+   (document list capped at `BOR_OVERVIEW_INPUT_MAX_CHARS`, overflow
+   cut at the cap with the shared truncation marker), `load_kb_overview`
+   (one indexed PK lookup, `""` when the row is missing/empty), and
+   `regenerate_overview` (best-effort upsert into the single row with a
+   fresh UTC `updated_at`, `overview: regenerated docs=… chars=…` log
+   line; a `lite` failure logs and leaves the previous outline intact;
+   zero documents → no model call, any existing row untouched).
+3. Prompt injection (phase 15 convention): the `<knowledge_base>`
+   section is budgeted by `BOR_KB_OVERVIEW_MAX_CHARS` (default 4000)
+   with the shared `[…truncated…]` overflow, ordered
+   `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body in
+   **both** the HIGH and LOW prompts; a missing/empty row keeps both
+   prompts **byte-identical** to the pre-phase text (unit-asserted).
+4. Chat turn: `plan_turn` reads the stored row per turn (no per-turn
+   LLM call) and records `TurnPlan.kb_chars`; the per-turn log line
+   records `kb_chars=N` after `tuning=N` (PLAN §9 extension).
+5. Import trigger: after a run that added/updated at least one document
+   (or no row exists yet), `import_docs` regenerates the outline with
+   the same `lite` model (`BOR_LLM_SUMMARY_MODEL` — A5 extended);
+   unchanged re-imports and `--limit` debug runs never burn a `lite`
+   call; the summary line ends `overview=updated|skipped|failed` and a
+   `lite` failure never changes the import's exit code.
+6. `.env.example` + README document `BOR_KB_OVERVIEW_MAX_CHARS` /
+   `BOR_OVERVIEW_INPUT_MAX_CHARS` and the regeneration behavior; the
+   deterministic mock LLM answers `KB_OVERVIEW_MODE` with a byte-stable
+   outline (first 8 tokens of the document list) and echoes the
+   `<knowledge_base>` section's first bullet as `(kb: …)` — the
+   `(tuning: …)` steering-echo precedent.
+7. Unit + integration green, `app/` coverage >90% (TOTAL ≥ pre-change),
+   story E2E green in isolation, `ruff` + `pyright` clean, one
+   `--no-gpg-sign` commit.
+
+## Playwright Mapping Rule
+`tests/e2e/test_kb_overview.py` — one story, one file, run in
+isolation. It seeds the `kb_overview` row directly in the DB (the
+import-trigger path is integration-tested) with a recognizable
+multi-bullet outline, imports the shared fixture KB, asks an on-topic
+question and asserts the rendered brain answer ends with
+`(kb: Kubernetes cluster and node maintenance notes)` (the mock's echo
+of the injected section's first bullet — only possible if the
+`<knowledge_base>` section reached the LLM prompt), asks an off-topic
+question and asserts the deflected answer carries the same echo (the
+section is in the LOW prompt too), deletes the row and asserts a fresh
+answer has no `(kb: …)` suffix (absence end-to-end), and runs the real
+`regenerate_overview` against the mock to assert the `KB_OVERVIEW_MODE`
+branch's byte-stable outline is stored and echoed by a subsequent turn.