feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts

.agent/phases/todo/31_kb_overview_prompt/01_migration_kb_overview.md

@@ -1,22 +0,0 @@
-# Task 01 — Migration 0005: kb_overview table
-
-**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "should be stored somewhere so it can be updated whenever we import new documents"`
-**Story:** `.agent/user_stories/kb-overview-prompt.md`
-
-## Objective
-Create the storage for the knowledge-base outline: a single-row `kb_overview` table and its SQLAlchemy model.
-
-## Work
-1. `alembic/versions/0005_kb_overview.py` — new revision (down_revision = phase 30's 0004):
-   - upgrade: create table `kb_overview` (`id INTEGER` PK `server_default sa.text("1")`, `content TEXT NOT NULL server_default sa.text("''")`, `updated_at TIMESTAMPTZ NOT NULL server_default=sa.func.now()`).
-   - downgrade: drop the table.
-2. `app/models.py` — `class KbOverview(Base)`: `id: Mapped[int] = mapped_column(Integer, primary_key=True, server_default="1")`, `content: Mapped[str] = mapped_column(Text, server_default="")`, `updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())`. Docstring: single row, lite-generated KB outline, phase 31.
-3. `tests/integration/test_migration_0005.py` — same style as `test_migration_0002.py` / `test_migration_0004.py`: upgrade → table exists with the three columns and defaults; downgrade → gone; upgrade again → back.
-
-## Testing & Quality
-- Integration: the migration test above (real Postgres).
-- Coverage: model exercised by existing model-test patterns; `app/` TOTAL ≥ pre-change.
-
-## Completion Criteria
-- [ ] `uv run alembic upgrade head` clean on the dev DB; round-trip with `alembic downgrade -1` + `upgrade head`.
-- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.

.agent/phases/todo/31_kb_overview_prompt/02_overview_generator.md

@@ -1,29 +0,0 @@
-# Task 02 — app/rag/overview.py (generator + loader)
-
-**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "basic categories of everything that's been read… generated by the lite model… updated whenever we import new documents"`
-**Story:** `.agent/user_stories/kb-overview-prompt.md`
-
-## Objective
-Create the overview generator module: build the `lite` prompt from the document catalogue, generate the outline, store it in the single row, and expose a cheap loader for the chat path.
-
-## Work
-1. `app/config.py` — add:
-   - `kb_overview_max_chars: int = 4_000` (`BOR_KB_OVERVIEW_MAX_CHARS`) — prompt-section budget (task 03).
-   - `overview_input_max_chars: int = 40_000` (`BOR_OVERVIEW_INPUT_MAX_CHARS`) — cap on the document list sent to the model.
-2. `app/rag/overview.py` (new):
-   - `KB_OVERVIEW_MODE = "KB_OVERVIEW_MODE"` — marker the E2E mock keys on (same convention as `SUMMARY_MODE` / `DEFLECT_MODE`).
-   - `build_overview_prompt(rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None) -> tuple[str, str]` → `(system, user)`. Each row is `(source, path, title, summary)`; system = `KB_OVERVIEW_MODE` + instruction ("From the document list below, write a compact plain-text outline of the basic categories and topics this knowledge base covers. Group by source where useful, use `-` bullet lines, at most ~1500 characters, no markdown headings, and no topics not present in the list."); user = one line per doc `source — path — title — {first line of summary or ''}` joined by newlines, capped at *max_chars* (default `overview_input_max_chars`, overflow → shared `TRUNCATION_MARKER`).
-   - `load_kb_overview(db: Session) -> str` — the single row's `content` (trimmed) or `""` when the row is missing/empty.
-   - `async def regenerate_overview(llm, session: Session | None = None) -> bool` — load all documents (`source, path, title, summary` ordered by source, path); **zero documents → leave the existing row untouched, return False**; build the prompt; `text = await llm.chat([system, user], model=llm.settings.llm_summary_model)`; upsert the single row (`id=1`, `content=text`, `updated_at=now(UTC)`); commit; log `overview: regenerated docs=%d chars=%d`; return True. On `LLMError`: log `overview: regeneration failed — %s` and return False (previous row stays — see phase locked decisions).
-3. `tests/unit/test_overview.py` (new) — fake LLM (duck-typed `chat` + `settings`), in-memory/SQLite session where the existing test infra allows (else a real-DB integration test in the same style as `test_steering.py`):
-   - prompt: system contains `KB_OVERVIEW_MODE`; user lines carry source/path/title/first summary line; cap truncates + marker.
-   - `load_kb_overview`: no row → `""`; row present → content.
-   - `regenerate_overview`: happy path upserts (content + fresh `updated_at`, returns True); zero docs → no DB write, returns False; `LLMError` → previous row unchanged, returns False.
-
-## Testing & Quality
-- Unit/integration: the tests in Work step 3.
-- Coverage: **>90%** on `app/rag/overview.py`.
-
-## Completion Criteria
-- [ ] `regenerate_overview` is idempotent (single row, always id=1) and fail-soft; all tests green.
-- [ ] `uv run ruff check . && uv run pyright` clean.

.agent/phases/todo/31_kb_overview_prompt/03_prompt_injection.md

@@ -1,30 +0,0 @@
-# Task 03 — `<knowledge_base>` section in both prompts + chat wiring
-
-**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "The system prompt should inject basic categories of everything that's been read so the agent knows roughly what its knowledge base contains before the rag retrieval returns documents"`
-**Story:** `.agent/user_stories/kb-overview-prompt.md`
-
-## Objective
-Inject the stored overview into **every** chat turn's system prompt (HIGH and LOW modes) as a budgeted `<knowledge_base>` section — absent row → byte-identical prompts — and record `kb_chars` in the per-turn log line.
-
-## Work
-1. `app/rag/prompts.py`:
-   - `build_kb_section(overview: str, max_chars: int | None = None) -> str` — empty/whitespace → `""`; otherwise `<knowledge_base>\n` + intro line ("The basic categories of everything in this knowledge base (generated at import time):") + the overview content, budgeted at *max_chars* (default `get_settings().kb_overview_max_chars`) with the shared `TRUNCATION_MARKER` for overflow (exact pattern of `build_steering_section`, including its pathological-budget handling).
-   - `build_high_prompt(documents, notes=None, kb_overview: str | None = None)` and `build_deflect_prompt(titles, notes=None, kb_overview: str | None = None)` — insert the section **between `<relevance>` and the `<tuning>` section** (i.e. order: `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body); with an empty overview the output is byte-identical to today's text in both modes.
-2. `app/api/chat.py`:
-   - Load per turn: `kb_overview = load_kb_overview(db)` next to the steering-notes load (one PK lookup — no LLM call).
-   - `plan_turn(chunks, settings, notes=None, kb_overview: str | None = None)` — pass it to both prompt builders; `TurnPlan` gains `kb_chars: int = 0` (length of the stored overview text when a non-empty row exists, else 0).
-   - Per-turn log line (PLAN §9): add `kb_chars=%d` after `tuning=%d`. Update any existing test asserting the log line format verbatim.
-3. `tests/unit/test_prompts.py` —
-   - HIGH: no overview → byte-identical to the pre-phase builder output (build the expected string with `notes=None, kb_overview=None`); with overview → section present, ordered before `<tuning>` when both exist.
-   - LOW: same pair of assertions (deflection prompt).
-   - budget: overview longer than `kb_overview_max_chars` → capped + `TRUNCATION_MARKER`.
-4. `tests/unit/test_chat_gate.py` — `plan_turn` with an overview: both branches' `system_prompt` contains the section; `TurnPlan.kb_chars` == len(overview); empty overview → `kb_chars == 0` and prompt unchanged.
-
-## Testing & Quality
-- Unit: Work steps 3–4; existing steering/prompt/gate tests stay green (new param is defaulted).
-- Coverage: **>90%** on modified `app/rag/prompts.py` + `app/api/chat.py`; `app/` TOTAL ≥ pre-change.
-
-## Completion Criteria
-- [ ] HIGH and LOW prompts are byte-identical to pre-phase text when no overview row exists (unit-asserted against the exact strings).
-- [ ] With a row, both prompts carry the budgeted `<knowledge_base>` section in the locked order; the per-turn log line shows `kb_chars=…`.
-- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.

.agent/phases/todo/31_kb_overview_prompt/04_import_trigger.md

@@ -1,33 +0,0 @@
-# Task 04 — import_docs regenerates the overview after a KB-changing import
-
-**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "should be updated whenever we import new documents"`
-**Story:** `.agent/user_stories/kb-overview-prompt.md`
-
-## Objective
-Wire the "update whenever we import new documents" trigger into the import script: after an import that changed the KB, regenerate the stored overview (best-effort). Phase 32's admin sync button reuses the exact same `regenerate_overview` call.
-
-## Work
-1. `scripts/import_docs.py`:
-   - Restructure `main()`'s single `asyncio.run(import_sources(…))` into one `async def _run()` that (a) runs `import_sources(sources, llm, prune=args.prune, limit=args.limit)` and (b) — when the summary has `added + updated > 0` **or** no overview row exists yet — awaits `regenerate_overview(llm)` (import it from `app.rag.overview`). One event loop, same `LLMClient` instance.
-   - `--limit` debug runs skip the regeneration (an incomplete walk must not rewrite the outline — mirrors the existing `--prune`-with-`--limit` guard).
-   - The final `print` gains `overview=updated|skipped|failed` (failed = `regenerate_overview` returned False via LLMError; the import's own exit code is **unchanged** — a failed outline must not fail the import).
-   - `Limit` guard: when `limit` is set, `added + updated > 0` does *not* trigger regeneration (log `overview: skipped (--limit)`).
-2. `tests/integration/test_import_docs_overview.py` (new) — with the git sync mocked out (reuse `test_import_docs_git.py`'s mocking style) and a fake LLM whose `chat` records calls:
-   - import with changed files → `kb_overview` row written; `chat` called once; print shows `overview=updated`.
-   - unchanged re-import (same hashes) → `chat` **not** called; print shows `overview=skipped`.
-   - `chat` raising `LLMError` → exit code still `0` (no import errors), print shows `overview=failed`, previous row untouched.
-   - `--limit` run with changes → `overview=skipped`.
-   - first-ever import (no row) with zero *changed* docs is not possible (new docs are "added") — but an empty-source run with no row → no row created, `overview=skipped`.
-3. `.env.example` — `BOR_KB_OVERVIEW_MAX_CHARS` (default 4000), `BOR_OVERVIEW_INPUT_MAX_CHARS` (default 40000).
-4. `README.md` — import workflow section: the import now refreshes the KB overview after a KB-changing run (fail-soft, `overview=` token in the summary line).
-
-- ASSUMPTION: "whenever we import new documents" = whenever an import **added or updated** at least one document (or the row doesn't exist yet); unchanged re-imports and `--limit` debug runs do not burn a lite call.
-
-## Testing & Quality
-- Integration: Work step 2 (real Postgres, mocked git + fake LLM — no live aipi).
-- Coverage: `app/` TOTAL ≥ pre-change (the script change is covered by the integration tests; the script itself is outside the `app/` gate).
-
-## Completion Criteria
-- [ ] All new integration tests green; `tests/integration/test_import_docs_git.py` stays green.
-- [ ] A manual run (`uv run python -m scripts.import_docs`) against the dev KB logs `overview: regenerated docs=… chars=…` after a KB-changing import and `overview=skipped` otherwise.
-- [ ] `uv run ruff check . && uv run pyright` clean.