81 lines
4.6 KiB
Markdown
81 lines
4.6 KiB
Markdown
# Story: KB Overview in the System Prompt (lite-generated knowledge-base outline)
|
|
|
|
**Phase:** `31_kb_overview_prompt.md` · **E2E:** `tests/e2e/test_kb_overview.py`
|
|
|
|
## Narrative
|
|
|
|
As **any user of Reese**, I ask questions without knowing what the
|
|
knowledge base actually covers. Retrieval only reveals the answer
|
|
documents after it has run, and when it finds nothing the deflection has
|
|
no picture of what Reese DOES know. I want the system prompt to carry,
|
|
on every turn, a plain-text outline of the **basic categories** of
|
|
everything that has been read — so the agent knows roughly what its
|
|
knowledge base contains before retrieval returns documents. The outline
|
|
is **generated by the `lite` model** from the document catalogue
|
|
(source, path, title, first summary line per document) and **stored in
|
|
a single row**, so it is regenerated whenever an import changes the
|
|
knowledge base — never per chat turn.
|
|
|
|
- **Given** a knowledge base that was imported (with at least one
|
|
document added or updated, or no outline stored yet)
|
|
- **When** I ask any question
|
|
- **Then** the system prompt of that turn — grounded (HIGH) or
|
|
deflected (LOW) — contains a `<knowledge_base>` section with the
|
|
stored outline (budgeted by `BOR_KB_OVERVIEW_MAX_CHARS`, overflow
|
|
marked with the shared `[…truncated…]` marker), and the per-turn log
|
|
line records `kb_chars=N`; when no outline is stored, both prompts
|
|
are byte-identical to the pre-phase text.
|
|
|
|
## Acceptance criteria
|
|
1. Migration 0005: single-row `kb_overview` table (`id INTEGER PK
|
|
DEFAULT 1`, `content TEXT NOT NULL DEFAULT ''`,
|
|
`updated_at TIMESTAMPTZ`) + `KbOverview` model — no other schema
|
|
change (reversible, integration-tested up/down).
|
|
2. `app/rag/overview.py`: the `KB_OVERVIEW_MODE` prompt builder
|
|
(document list capped at `BOR_OVERVIEW_INPUT_MAX_CHARS`, overflow
|
|
cut at the cap with the shared truncation marker), `load_kb_overview`
|
|
(one indexed PK lookup, `""` when the row is missing/empty), and
|
|
`regenerate_overview` (best-effort upsert into the single row with a
|
|
fresh UTC `updated_at`, `overview: regenerated docs=… chars=…` log
|
|
line; a `lite` failure logs and leaves the previous outline intact;
|
|
zero documents → no model call, any existing row untouched).
|
|
3. Prompt injection (phase 15 convention): the `<knowledge_base>`
|
|
section is budgeted by `BOR_KB_OVERVIEW_MAX_CHARS` (default 4000)
|
|
with the shared `[…truncated…]` overflow, ordered
|
|
`<relevance>` → `<knowledge_base>` → `<tuning>` → mode body in
|
|
**both** the HIGH and LOW prompts; a missing/empty row keeps both
|
|
prompts **byte-identical** to the pre-phase text (unit-asserted).
|
|
4. Chat turn: `plan_turn` reads the stored row per turn (no per-turn
|
|
LLM call) and records `TurnPlan.kb_chars`; the per-turn log line
|
|
records `kb_chars=N` after `tuning=N` (PLAN §9 extension).
|
|
5. Import trigger: after a run that added/updated at least one document
|
|
(or no row exists yet), `import_docs` regenerates the outline with
|
|
the same `lite` model (`BOR_LLM_SUMMARY_MODEL` — A5 extended);
|
|
unchanged re-imports and `--limit` debug runs never burn a `lite`
|
|
call; the summary line ends `overview=updated|skipped|failed` and a
|
|
`lite` failure never changes the import's exit code.
|
|
6. `.env.example` + README document `BOR_KB_OVERVIEW_MAX_CHARS` /
|
|
`BOR_OVERVIEW_INPUT_MAX_CHARS` and the regeneration behavior; the
|
|
deterministic mock LLM answers `KB_OVERVIEW_MODE` with a byte-stable
|
|
outline (first 8 tokens of the document list) and echoes the
|
|
`<knowledge_base>` section's first bullet as `(kb: …)` — the
|
|
`(tuning: …)` steering-echo precedent.
|
|
7. Unit + integration green, `app/` coverage >90% (TOTAL ≥ pre-change),
|
|
story E2E green in isolation, `ruff` + `pyright` clean, one
|
|
`--no-gpg-sign` commit.
|
|
|
|
## Playwright Mapping Rule
|
|
`tests/e2e/test_kb_overview.py` — one story, one file, run in
|
|
isolation. It seeds the `kb_overview` row directly in the DB (the
|
|
import-trigger path is integration-tested) with a recognizable
|
|
multi-bullet outline, imports the shared fixture KB, asks an on-topic
|
|
question and asserts the rendered brain answer ends with
|
|
`(kb: Kubernetes cluster and node maintenance notes)` (the mock's echo
|
|
of the injected section's first bullet — only possible if the
|
|
`<knowledge_base>` section reached the LLM prompt), asks an off-topic
|
|
question and asserts the deflected answer carries the same echo (the
|
|
section is in the LOW prompt too), deletes the row and asserts a fresh
|
|
answer has no `(kb: …)` suffix (absence end-to-end), and runs the real
|
|
`regenerate_overview` against the mock to assert the `KB_OVERVIEW_MODE`
|
|
branch's byte-stable outline is stored and echoed by a subsequent turn.
|