diff --git a/.agent/phases/todo/30_document_summaries/00_phase.md b/.agent/phases/todo/30_document_summaries/00_phase.md new file mode 100644 index 0000000..2262a06 --- /dev/null +++ b/.agent/phases/todo/30_document_summaries/00_phase.md @@ -0,0 +1,48 @@ +# Phase 30 — Document Summaries (lite-model summaries for non-markdown documents) + +**Source:** `TODO.md L3 — "One issue I'm having is bad context for the embedder which causes poor retrieval results… we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source… if retrieval == summary, fetch documents referenced by summary… The small model available on aipi.reeseapps.com is 'lite'."` +**Story:** `.agent/user_stories/document-summaries.md` +**Context:** The importer (`app/rag/importer.py::_index_file` — chunk → embed → upsert per file, A9 scope), hybrid retrieval (`app/rag/retriever.py` — A7: cosine ∪ FTS, RRF, chunk→parent-document mapping, full-document context never truncated, phase 24), the locked persona prompts (`app/rag/prompts.py`), the aipi client (`app/rag/llm.py` — A5: `turbo` chat streaming + `embed` embeddings), and the E2E mock LLM (`tests/e2e/mock_llm.py` — deterministic, keys on system-prompt markers like `DEFLECT_MODE` / ``). + +## Objective +Give every **non-markdown** A9 document (txt, yaml, yml, json, py) a natural-language summary generated at import time by the aipi **`lite`** model. The summary is stored on the document (`documents.summary`) **and indexed as one extra embedded chunk** (`chunks.is_summary`), so hybrid search has a well-embedding natural-language target to hit instead of the badly-formatted raw text. A summary hit resolves to its parent (the source document) — the existing chunk→document mapping then feeds the **full source document** to the LLM, implementing the TODO's "if retrieval == summary, fetch the documents referenced by the summary" step. The per-turn log line records how many summary hits landed in the selected context. + +## Dependencies +- `29_tuning_nav_link` (complete) — the latest finished phase (sequencing only). +- Substantively builds on: `02_story_import_documents` / `24_whole_document_context` (import pipeline + full-document context contract), `09_story_retrieval_quality` (A7 hybrid retrieval the summary chunk flows through unchanged), `01_infrastructure` (models/alembic, LLM client, E2E mock). + +## Tasks +1. `01_lite_model_client.md` — `BOR_LLM_SUMMARY_MODEL` (default `lite`) + non-streaming `LLMClient.chat()` for the lite model. +2. `02_migration_summary_columns.md` — Alembic 0004: `documents.summary TEXT NULL` + `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE`. +3. `03_summarizer_module.md` — `app/rag/summarizer.py`: `SUMMARY_MODE` prompt (capped input), lite call, output validation + deterministic `Source: /` pointer line. +4. `04_importer_summary_integration.md` — importer generates/stores/indexes summaries for non-md files (best-effort fail-soft) + summary counters. +5. `05_pipeline_summary_resolution.md` — `is_summary` through the retriever, `summary_hits` in `TurnPlan` + the per-turn log line; full source document on summary hit (existing mapping, asserted). +6. `06_mock_and_e2e.md` — deterministic `lite` in `mock_llm.py`, sentinel fixture, `tests/e2e/test_document_summaries.py`, story file, README, commit. + +## Testing & Quality +- Unit: summarizer (prompt/cap/pointer/errors), importer (summary happy path, md exclusion, fail-soft, replacement on re-import), retriever (`is_summary` through both candidate lists + `fuse`), chat gate (`TurnPlan.summary_hits`), LLM client (`chat()`). +- Integration: migration 0004 up/down. +- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`, TOTAL ≥ pre-change number). +- E2E (mandatory, A16): `tests/e2e/test_document_summaries.py` — one story, run **in isolation** (`uv run pytest tests/e2e/test_document_summaries.py -v --no-cov`); proves summary hit → full source document reaches the answer (sentinel in the raw doc, absent from the mock summary). +- All existing E2E suites stay green (new columns are defaulted; all existing chunks have `is_summary=false`). + +## Completion Criteria +- [ ] After `uv run python -m scripts.import_docs`, every non-markdown fixture/doc has `documents.summary` set and exactly one `is_summary` chunk (position −1, embedded); markdown docs have neither. +- [ ] A question whose best match is a summary chunk yields an answer grounded in the **full source document** (E2E sentinel) and the per-turn log line shows `summary_hits>=1`. +- [ ] A lite-model failure during import does **not** drop the document — it is indexed without a summary, logged, and counted (`summary_errors`). +- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%). +- [ ] `uv run pytest tests/e2e/test_document_summaries.py -v --no-cov` green in isolation; existing suites (`test_chat_rag.py`, `test_retrieval_quality.py`, `test_import_documents.py`, `test_whole_document_context.py`) stay green. +- [ ] `uv run ruff check . && uv run pyright` clean. +- [ ] `.agent/user_stories/document-summaries.md` exists. +- [ ] `.env.example` + README document `BOR_LLM_SUMMARY_MODEL` / `BOR_SUMMARY_MAX_CHARS` and the summary behavior. +- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc`); `.agent/phases/todo/30_document_summaries/` moved to `.agent/phases/complete/`. + +## Locked decisions +- **A5 extended, not revised** — the `lite` model is served by the same OpenAI-compatible endpoint (`https://aipi.reeseapps.com/v1`) via a new `BOR_LLM_SUMMARY_MODEL` setting (default `lite`); no new model management, no new package. +- **A7 untouched** — hybrid retrieval logic is unchanged; the summary is an ordinary chunk, so it flows through the existing cosine ∪ FTS ∪ RRF path and the chunk→document mapping. The "fetch the referenced document" step is the existing full-document context contract (phase 24) — never truncated. +- **A9 untouched** — "non-markdown" means every *already-imported* A9 document except `md`/`markdown`. The TODO's quadlet-file example is **out of scope**: `.quadlet` is not an A9 format and `BOR_IMPORT_EXTENSIONS` may only narrow the locked set (flagged at roadmap confirmation; importing quadlet files would require an owner-permission A9 revision). +- **A13** — migration 0004 adds two columns (`documents.summary`, `chunks.is_summary`); no table rework, both reversible. +- **Summary generation is best-effort** — a lite failure logs + counts (`summary_errors`) and the file is still indexed without a summary (same fail-soft spirit as the per-file `EmbeddingError` handling, but weaker: the doc is already committed). +- **Pointer is code-deterministic** — the `Source: /` line is appended by `summarizer.py`, never trusted to the model. +- **A16 honoured** — one dedicated story E2E suite; E2E stays deterministic via the mock LLM's `SUMMARY_MODE` marker. +- **A17 honoured** — one atomic `--no-gpg-sign` commit. diff --git a/.agent/phases/todo/30_document_summaries/01_lite_model_client.md b/.agent/phases/todo/30_document_summaries/01_lite_model_client.md new file mode 100644 index 0000000..375a34e --- /dev/null +++ b/.agent/phases/todo/30_document_summaries/01_lite_model_client.md @@ -0,0 +1,26 @@ +# Task 01 — lite model setting + non-streaming chat() + +**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "The small model available on aipi.reeseapps.com is 'lite'." (enabler for "we need a small model to analyze non markdown documents")` +**Story:** `.agent/user_stories/document-summaries.md` + +## Objective +Make the aipi **`lite`** model callable from the app: a new model setting plus a non-streaming `LLMClient.chat()` one-shot completion method that the summarizer (task 03) and the phase-31 overview generator will use. + +## Work +1. `app/config.py` — add `llm_summary_model: str = "lite"` (env `BOR_LLM_SUMMARY_MODEL`), documented like the other LLM settings. +2. `app/rag/llm.py` — add to `LLMClient`: + - `async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str` — `chat.completions.create(model=model or self.settings.llm_summary_model, messages=…, temperature=0.2, max_tokens=2048, stream=False)`; returns the first choice's `message.content` stripped. + - Raise `LLMError` (wrapped, with the base URL in the message — same style as `chat_stream`) on any transport/HTTP/malformed failure, and on an empty/missing content field (a silent empty summary must never be stored). +3. `.env.example` — document `BOR_LLM_SUMMARY_MODEL` (default `lite`). +4. `README.md` — models section: add `lite` (document summaries — this phase; KB overview in phase 31) next to `turbo`/`embed`. + +- ASSUMPTION: the model name is `lite` per the TODO item; single (non-streaming) completion with `temperature=0.2`, `max_tokens=2048` — summaries/outlines are short, so a fixed budget is enough (no new setting). + +## Testing & Quality +- Unit: `tests/unit/test_llm_client.py` — extend the existing mock-transport pattern: `chat()` returns content (trimmed); HTTP ≥400 → `LLMError`; empty content → `LLMError`; explicit `model=` overrides the default (`llm_summary_model`). +- Coverage: **>90%** on this task's new/modified code (`app/` TOTAL ≥ pre-change). + +## Completion Criteria +- [ ] `Settings().llm_summary_model == "lite"` by default; `BOR_LLM_SUMMARY_MODEL` env override works (config test). +- [ ] `chat()` unit tests green; `uv run ruff check . && uv run pyright` clean. +- [ ] No change to `embed`/`chat_stream` behavior (existing suite green). diff --git a/.agent/phases/todo/30_document_summaries/02_migration_summary_columns.md b/.agent/phases/todo/30_document_summaries/02_migration_summary_columns.md new file mode 100644 index 0000000..1fbf587 --- /dev/null +++ b/.agent/phases/todo/30_document_summaries/02_migration_summary_columns.md @@ -0,0 +1,23 @@ +# Task 02 — Migration 0004: summary columns + +**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "provide a textual summary of those documents with a pointer back to the source" (storage)` +**Story:** `.agent/user_stories/document-summaries.md` + +## Objective +Add the schema for storing a document's summary and for marking the extra summary chunk: `documents.summary TEXT NULL` and `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE`. + +## Work +1. `alembic/versions/0004_summary_columns.py` — new revision (down_revision = the 0003 steering-notes revision, whatever `alembic/versions/` currently heads to): + - upgrade: `op.add_column("documents", sa.Column("summary", sa.Text(), nullable=True))`; `op.add_column("chunks", sa.Column("is_summary", sa.Boolean(), nullable=False, server_default=sa.text("false")))`. + - downgrade: drop both columns. +2. `app/models.py` — `Document.summary: Mapped[str | None] = mapped_column(Text, default=None)` (comment: lite-model summary, phase 30); `Chunk.is_summary: Mapped[bool] = mapped_column(Boolean, default=False)` (comment: summary chunk, position −1, phase 30). +3. `tests/integration/test_migration_0004.py` — mirror `tests/integration/test_migration_0002.py`'s style: upgrade to head → both columns exist, `is_summary` default `false`; downgrade to 0003 → both gone; upgrade again → back (round-trip). + +## Testing & Quality +- Integration: the migration test above (real Postgres, as `test_migration_0002.py` does). +- Coverage: models are exercised by the existing model tests; `app/` TOTAL ≥ pre-change. + +## Completion Criteria +- [ ] `uv run alembic upgrade head` applies cleanly on the dev DB (and `alembic downgrade -1` + `upgrade head` round-trips). +- [ ] `uv run pytest` green (including all pre-existing migration/importer tests — `is_summary` default keeps old rows valid). +- [ ] `uv run ruff check . && uv run pyright` clean. diff --git a/.agent/phases/todo/30_document_summaries/03_summarizer_module.md b/.agent/phases/todo/30_document_summaries/03_summarizer_module.md new file mode 100644 index 0000000..8ceb90e --- /dev/null +++ b/.agent/phases/todo/30_document_summaries/03_summarizer_module.md @@ -0,0 +1,30 @@ +# Task 03 — app/rag/summarizer.py (prompt + lite call + pointer) + +**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source"` +**Story:** `.agent/user_stories/document-summaries.md` + +## Objective +Create the summarizer module: build the `lite` prompt for one document, call the model (task 01), validate the output, and return the summary text with a **code-deterministic** pointer line back to the source. + +## Work +1. `app/config.py` — add `summary_max_chars: int = 12_000` (env `BOR_SUMMARY_MAX_CHARS`): the cap on document content sent to the lite model in one call. +2. `app/rag/summarizer.py` (new) — + - `SUMMARY_MODE = "SUMMARY_MODE"` — marker constant the E2E mock keys on in the system prompt (same convention as `DEFLECT_MODE`). + - `build_summary_prompt(source: str, path: str, content: str, max_chars: int | None = None) -> tuple[str, str]` → `(system, user)`: + - system: `SUMMARY_MODE` + instruction — "Write a 3–6 sentence plain-text summary of this document in natural language. Cover what it configures/defines and its most important values. Do not use markdown. Do not invent anything that is not in the document." + - user: the document content, capped at *max_chars* (default `get_settings().summary_max_chars`); on overflow cut at the cap and append the shared `TRUNCATION_MARKER` (imported from `app.rag.retriever`). + - `async def generate_summary(llm, *, source: str, path: str, content: str) -> str` — calls `llm.chat([{"role":"system",…},{"role":"user",…}], model=llm.settings.llm_summary_model)`; validates non-empty after trim (else raise `LLMError` — the client already does this, but re-assert defensively); appends the deterministic pointer line: `f"\nSource: {source}/{path}"` (the pointer is **never** model-generated). +3. `tests/unit/test_summarizer.py` (new) — fake LLM object (duck-typed `chat` + `settings`): + - prompt: system contains `SUMMARY_MODE`; user == full content when under cap; user truncated + `TRUNCATION_MARKER` when over cap (custom and default cap). + - generation: returned text = model text + pointer line `Source: /`; whitespace model text → `LLMError`; `LLMError` from the client propagates. + +- ASSUMPTION: the pointer is the literal line `Source: /` appended by code (the TODO's "pointer back to the source"); the model is told what to summarize but not to write the pointer. + +## Testing & Quality +- Unit: the tests in Work step 3. +- Coverage: **>90%** on `app/rag/summarizer.py`. + +## Completion Criteria +- [ ] `generate_summary` returns a non-empty summary ending in the deterministic pointer line; all unit tests green. +- [ ] `uv run ruff check . && uv run pyright` clean. +- [ ] No app endpoint change yet (pipeline integration is task 05). diff --git a/.agent/phases/todo/30_document_summaries/04_importer_summary_integration.md b/.agent/phases/todo/30_document_summaries/04_importer_summary_integration.md new file mode 100644 index 0000000..691a925 --- /dev/null +++ b/.agent/phases/todo/30_document_summaries/04_importer_summary_integration.md @@ -0,0 +1,39 @@ +# Task 04 — Importer: generate, store, index summaries (best-effort) + +**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "a small model to analyze non markdown documents and provide a textual summary… The similarity search will have a higher chance of hitting those summaries than the original document"` +**Story:** `.agent/user_stories/document-summaries.md` + +## Objective +Hook summarization into the import pipeline: every **non-markdown** file gets a lite summary stored on `documents.summary` and indexed as one extra embedded chunk (`is_summary`, position −1) — best-effort, so a lite failure never loses the document. + +## Work +1. `app/rag/importer.py`: + - `Embedder` protocol — add `async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str: ...` (task 01's `LLMClient.chat` already satisfies it; the protocol is what tests duck-type). + - `ImportSummary` — new counters `summaries: int = 0` and `summary_errors: int = 0`; include both in the `log()` summary line (`… summaries=%d summary_errors=%d`). + - `_index_file` — **after** the existing doc+chunks commit (so the document is safe): + - If `full_path.suffix.lower()` is **not** in `(".md", ".markdown")`: try + `summary = await generate_summary(llm, source=source, path=rel, content=content)`; + delete any existing `is_summary` chunk of this document (re-import replacement); + add `Chunk(document_id=doc.id, position=-1, content=summary, is_summary=True)`; + `vector = (await llm.embed([summary]))[0]`; set `chunk.embedding = vector`, `doc.summary = summary`; `session.commit()`; `summary.summaries += 1`; log `import: summary source=%s path=%s chars=%d`. + - On `LLMError | EmbeddingError`: `session.rollback()`, `summary.summary_errors += 1`, log `import: summary failed source=%s path=%s — %s`, and **continue** (the document row + content chunks stay committed; `doc.summary` remains NULL). + - Markdown files: no summary, `doc.summary` stays NULL, no `is_summary` chunk. +2. `scripts/import_docs.py` — the final `print` gains `summaries=%d summary_errors=%d` from the `ImportSummary`. +3. `tests/unit/test_importer.py` — extend the existing fake embeder with a `chat` method (deterministic: returns `"Summary of " + first token of content`; raises `LLMError` when the content contains the sentinel word `SUMMARY-BLOWUP`): + - non-md file (e.g. `.yaml`) → after import: `doc.summary` set, exactly one `is_summary` chunk at position −1 with a non-NULL embedding; `summary.summaries == 1`. + - `.md` file → `doc.summary is None`, no `is_summary` chunk, `summaries == 0`. + - fail-soft: content with `SUMMARY-BLOWUP` → document fully indexed (chunks present, embedding set), `doc.summary is None`, `summary_errors == 1`, no exception. + - replacement: re-import the same file with changed content → still exactly **one** `is_summary` chunk (old one deleted), new text. + - `log()` line includes the new counters (existing log-format test updated accordingly). + +- ASSUMPTION: the summary chunk sits at `position = -1` (content chunks stay 0-based in order) so chunk ordering and the viewer are undisturbed; only one summary chunk per document at a time. +- ASSUMPTION: "non-markdown" = suffix not in `(.md, .markdown)` — every other A9 format (txt, yaml, yml, json, py) gets a summary. + +## Testing & Quality +- Unit: the tests in Work step 3 (reuse the existing test file's session/fake-embedder fixtures). +- Coverage: **>90%** on the modified `app/rag/importer.py`; `app/` TOTAL ≥ pre-change. + +## Completion Criteria +- [ ] All new unit tests green; existing importer tests green (protocol change is additive). +- [ ] `uv run ruff check . && uv run pyright` clean. +- [ ] A manual `uv run python -m scripts.import_docs` run against the dev KB logs `summaries=N` for the non-md docs (observable in the importer log line, PLAN §9). diff --git a/.agent/phases/todo/30_document_summaries/05_pipeline_summary_resolution.md b/.agent/phases/todo/30_document_summaries/05_pipeline_summary_resolution.md new file mode 100644 index 0000000..9fbc79a --- /dev/null +++ b/.agent/phases/todo/30_document_summaries/05_pipeline_summary_resolution.md @@ -0,0 +1,31 @@ +# Task 05 — Pipeline: summary hits resolve to the full source document + +**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "the chat LLM can retrieve the source pointed to by the summary document (so as part of the pipeline: if retrieval == summary, fetch documents referenced by summary)"` +**Story:** `.agent/user_stories/document-summaries.md` + +## Objective +Make the retrieval + chat pipeline summary-aware: carry `is_summary` through both candidate lists into the fused result, and record in the per-turn log how many of the selected documents were hit via their summary chunk. The context assembly itself is **unchanged** — a summary chunk's parent *is* the source document, and `select_documents` already feeds the full document (A7 revised / phase 24); this task makes that resolution explicit, asserted, and observable. + +## Work +1. `app/rag/retriever.py`: + - `RetrievedChunk` — add field `is_summary: bool = False` (documented: True for the lite-model summary chunk, position −1). + - `_vector_candidates` — select `Chunk.is_summary` and pass it into the constructed `RetrievedChunk`s. + - `_LEXICAL_SQL` — add `c.is_summary AS is_summary`; `_lexical_candidates` passes `row.is_summary`. + - `fuse` needs no change (dataclass passthrough) — but assert in tests that the flag survives fusion. +2. `app/api/chat.py`: + - `TurnPlan` — add `summary_hits: int = 0` (count of selected-document hit chunks with `is_summary`). + - `plan_turn` — after the selected-docs decision is known for each branch, compute `summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in selected_ids)` and store it on the `TurnPlan` (both HIGH and LOW branches). + - Per-turn log line — add `summary_hits=%d` after `fts_hits=%d` (PLAN §9 line extension; record it in the phase's locked decisions). Update any existing test that asserts the log line format verbatim. + - **No change** to `build_high_prompt`/`build_deflect_prompt` inputs or to `select_documents` — the full source document of a summary hit already lands in ``; task 06's E2E proves it end-to-end. +3. `tests/unit/test_retriever.py` — `is_summary` survives: vector candidates (flag set), lexical candidates (flag set), `fuse` (both a double-hit and a summary-only lexical hit keep the flag; default stays `False` for legacy chunks). +4. `tests/unit/test_chat_gate.py` — `plan_turn`: a summary chunk on a selected top document → `summary_hits == 1`; a summary chunk on a document **outside** the top-N selection → not counted; no summaries → `0` (existing cases unchanged). + +## Testing & Quality +- Unit: the tests in Work steps 3–4; existing chat-gate and retriever tests stay green (new field is defaulted). +- Coverage: **>90%** on modified `app/rag/retriever.py` + `app/api/chat.py`; `app/` TOTAL ≥ pre-change. + +## Completion Criteria +- [ ] A summary chunk retrieved via vector **or** lexical carries `is_summary=True` through `retrieve()` (unit). +- [ ] The per-turn log line (PLAN §9) now reads `… fts_hits=… summary_hits=… …` and existing log-format tests are updated + green. +- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean. +- [ ] No prompt change: HIGH/LOW prompts byte-identical for summary-less KBs (covered by existing prompt tests). diff --git a/.agent/phases/todo/30_document_summaries/06_mock_and_e2e.md b/.agent/phases/todo/30_document_summaries/06_mock_and_e2e.md new file mode 100644 index 0000000..a44b071 --- /dev/null +++ b/.agent/phases/todo/30_document_summaries/06_mock_and_e2e.md @@ -0,0 +1,35 @@ +# Task 06 — Deterministic lite mock + story E2E + docs + commit + +**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — (whole item: bad embedder context for non-markdown docs → lite summaries → summary hits fetch the referenced source; the aipi 'lite' model)` +**Story:** `.agent/user_stories/document-summaries.md` + +## Objective +Close the loop: a deterministic `lite` behavior in the E2E mock, a sentinel fixture proving that **a summary hit still delivers the full source document to the LLM**, the story E2E suite, the story file, README docs, and the phase commit. + +## Work +1. `tests/e2e/mock_llm.py` — in the chat-completions handler (non-stream and stream paths), **before** the `DEFLECT_MODE` check: if the system prompt contains `SUMMARY_MODE`, return the deterministic digest + `f"This document covers {' '.join(TOKEN_RE.findall(_user(body).lower())[:24])}."` + — the first 24 tokens of the document content (the summarizer puts the content in the *user* message). Byte-stable for a given fixture. +2. E2E fixture — a new non-markdown fixture doc, e.g. `quadlet/qwen-llamacpp.yaml` under the existing E2E fixture KB (follow the import-dependent fixtures' pattern in `tests/e2e/conftest.py` / the fixture dir used by `test_whole_document_context.py`): + - The document **opens** with a header comment line dense in the question tokens (e.g. `# qwen 3.8 llama.cpp optimal parameters deployment notes`) so the mock's 24-token summary digest contains the question's words, followed by ~4–5 k of other yaml content (so the raw chunks dilute their overlap and the summary chunk ranks first — the mock's embeddings are a pure function of tokens, so the ranking is fully deterministic for a fixed fixture; iterate the fixture text until the E2E assertions hold). + - A unique sentinel `RESE-SUMMARY-SENTINEL-7f3a` on the **last line** of the document (outside the 24-token digest, unreachable from the summary). +3. `tests/e2e/test_document_summaries.py` (new, the story gate) — reuse the E2E conftest app/DB fixtures: + - Import the fixture KB (re-import pattern used by import-dependent stories). + - Ask the question (e.g. "What are the optimal parameters for qwen 3.8 on llama.cpp? show the end of your notes" — the phase-24 tail-echo trigger makes the answer quote the **last 160 chars of the document context**). + - Assert the rendered brain answer contains `RESE-SUMMARY-SENTINEL-7f3a` → the full **source** document was in the LLM context (only possible via the summary→parent-document resolution, since the summary digest cannot contain the sentinel). + - Assert the source chip shows the fixture doc's path and `deflected` is false. + - Control: a markdown fixture doc in the same KB gets **no** summary chunk — assert via the Sources table (admin, `#docs-table`) or a direct DB check in the test: markdown doc's chunk count == raw chunks only; the yaml doc has exactly one `is_summary` row. +4. `.agent/user_stories/document-summaries.md` (new) — narrative + acceptance criteria + Playwright mapping rule (story → `tests/e2e/test_document_summaries.py`), matching the style of `.agent/user_stories/git-sources.md`. +5. `README.md` — new "Document summaries" section: what gets summarized (non-markdown A9 docs), the `Source:` pointer, `BOR_LLM_SUMMARY_MODEL` / `BOR_SUMMARY_MAX_CHARS`, fail-soft behavior, and how summary hits appear in the per-turn log. +6. Commit: `git add` the phase's app/script/test/README files; `git commit --no-gpg-sign -m "feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc"`; move `.agent/phases/todo/30_document_summaries/` → `.agent/phases/complete/` (`.agent/` is gitignored by design — force-add only if the commit must record the plan change, otherwise leave the move out of git). + +## Testing & Quality +- E2E: `uv run pytest tests/e2e/test_document_summaries.py -v --no-cov` green **in isolation** (Chromium installed; `podman compose up -d db` up; mock LLM — no live aipi needed). +- Regression: run `test_chat_rag.py`, `test_retrieval_quality.py`, `test_import_documents.py`, `test_whole_document_context.py` in isolation — all stay green. +- Full gate: `uv run pytest` + `uv run pytest --cov=app --cov-report=term-missing` (TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`. + +## Completion Criteria +- [ ] `tests/e2e/test_document_summaries.py` green in isolation; the sentinel assertion proves summary hit → full source document. +- [ ] All regression suites listed above green in isolation. +- [ ] Full test gate + lint/type gate green (per this phase's 00_phase.md). +- [ ] Story file + README + `.env.example` complete; one `--no-gpg-sign` commit made. diff --git a/.agent/phases/todo/31_kb_overview_prompt/00_phase.md b/.agent/phases/todo/31_kb_overview_prompt/00_phase.md new file mode 100644 index 0000000..7d14b6e --- /dev/null +++ b/.agent/phases/todo/31_kb_overview_prompt/00_phase.md @@ -0,0 +1,45 @@ +# Phase 31 — KB Overview in the System Prompt (lite-generated knowledge-base outline) + +**Source:** `TODO.md L4 — "The system prompt should inject basic categories of everything that's been read so the agent knows roughly what its knowledge base contains before the rag retrieval returns documents. This part of the system prompt should be generated by the lite model and should be stored somewhere so it can be updated whenever we import new documents."` +**Story:** `.agent/user_stories/kb-overview-prompt.md` +**Context:** Phase 15 steering notes (the `` prompt section, its char budget, and the **byte-identical-when-absent** convention — `app/rag/prompts.py::build_steering_section`), phase 30 (the `lite` client method `LLMClient.chat`, and per-document summaries that make a much better overview input than raw titles), `scripts/import_docs.py` (the place "whenever we import new documents" happens), and the E2E mock's answer-echo convention (the `(tuning: …)` suffix — `tests/e2e/mock_llm.py`). + +## Objective +Store a lite-generated, plain-text outline of the knowledge base's basic **categories** in a single-row `kb_overview` table, inject it into **both** chat prompts (HIGH and LOW) as a `` section so the agent knows roughly what the KB contains before retrieval, and regenerate it automatically whenever an import changes the KB. + +## Dependencies +- `30_document_summaries` (todo) — `LLMClient.chat` + `BOR_LLM_SUMMARY_MODEL` (task 01) and the stored per-document summaries (task 04) that feed the overview input. +- `15_steering_notes` (complete) — the prompt-section pattern this phase mirrors (budget, marker, byte-identical-when-absent, per-turn load in `app/api/chat.py`). +- `11_long_answers` / README import workflow (complete) — `scripts/import_docs.py`'s `main()` structure, which this phase extends with the post-import regeneration. + +## Tasks +1. `01_migration_kb_overview.md` — Alembic 0005: single-row `kb_overview` table + `KbOverview` model. +2. `02_overview_generator.md` — `app/rag/overview.py`: `KB_OVERVIEW_MODE` prompt builder, `load_kb_overview`, `regenerate_overview` (best-effort upsert). +3. `03_prompt_injection.md` — `` section in HIGH + LOW prompts (budgeted, byte-identical when absent); `plan_turn`/chat wire it in; `kb_chars` in the per-turn log. +4. `04_import_trigger.md` — `import_docs` regenerates the overview after a KB-changing import (shared with phase 32's sync). +5. `05_mock_and_e2e.md` — deterministic `KB_OVERVIEW_MODE` mock + `(kb: …)` echo, `tests/e2e/test_kb_overview.py`, story file, commit. + +## Testing & Quality +- Unit: overview generator (prompt build/cap, load, regenerate upsert/fail-soft/zero-docs), prompts (section present/budgeted/absent → byte-identical, ordering vs ``), chat gate (`kb_chars`, prompt carries the section). +- Integration: migration 0005 up/down; `import_docs` regeneration trigger (changed vs unchanged imports, failure isolation). +- Coverage: **>90%** on `app/` (`app/` TOTAL ≥ pre-change). +- E2E (mandatory, A16): `tests/e2e/test_kb_overview.py` — one story, run **in isolation**; the injected section is observable in the mock answer via the `(kb: …)` echo (steering precedent). + +## Completion Criteria +- [ ] After a KB-changing import, `kb_overview` holds a fresh outline (log line `overview: regenerated docs=… chars=…`); an unchanged re-import does **not** call the lite model. +- [ ] Every chat turn's system prompt (HIGH and LOW) contains the `` section when a row exists; with no row, both prompts are **byte-identical** to the pre-phase text (unit-asserted). +- [ ] The per-turn log line records `kb_chars=`; section overflow beyond `BOR_KB_OVERVIEW_MAX_CHARS` is capped with the shared `[…truncated…]` marker. +- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%). +- [ ] `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green in isolation; existing prompt/steering/chat suites stay green. +- [ ] `uv run ruff check . && uv run pyright` clean. +- [ ] `.agent/user_stories/kb-overview-prompt.md` exists; `.env.example` + README document `BOR_KB_OVERVIEW_MAX_CHARS` / `BOR_OVERVIEW_INPUT_MAX_CHARS` and the regeneration behavior. +- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, section in HIGH+LOW prompts`); `.agent/phases/todo/31_kb_overview_prompt/` moved to `.agent/phases/complete/`. + +## Locked decisions +- **A13** — migration 0005 adds one single-row table `kb_overview` (`id INTEGER PK DEFAULT 1`, `content TEXT NOT NULL DEFAULT ''`, `updated_at TIMESTAMPTZ`); no other schema change. +- **A5 extended** — the overview is generated by the same `lite` model via the same `BOR_LLM_SUMMARY_MODEL` setting and `LLMClient.chat` (phase 30); no new model or package. +- **Prompt-section convention (phase 15 precedent)** — the section is budgeted by `BOR_KB_OVERVIEW_MAX_CHARS` (default **4000**) with the shared `TRUNCATION_MARKER` overflow; **zero/empty row → prompts byte-identical** to pre-phase text. Section order: `` → `` → `` → mode body. +- **Regeneration is best-effort and change-gated** — runs only when an import added/updated at least one document (or no row exists yet); a lite failure logs and leaves the previous overview intact (an old outline is better than none). +- **Overview input is capped** — `BOR_OVERVIEW_INPUT_MAX_CHARS` (default **40 000**) on the document list (source/path/title/first summary line) sent to the model. +- **No per-turn LLM call** — chat turns only *read* the stored row (one indexed PK lookup); generation happens at import/sync time (phase 32's button triggers the same `regenerate_overview`). +- **A16 / A17 honoured** — one dedicated story E2E suite; one atomic `--no-gpg-sign` commit. diff --git a/.agent/phases/todo/31_kb_overview_prompt/01_migration_kb_overview.md b/.agent/phases/todo/31_kb_overview_prompt/01_migration_kb_overview.md new file mode 100644 index 0000000..56569fb --- /dev/null +++ b/.agent/phases/todo/31_kb_overview_prompt/01_migration_kb_overview.md @@ -0,0 +1,22 @@ +# Task 01 — Migration 0005: kb_overview table + +**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "should be stored somewhere so it can be updated whenever we import new documents"` +**Story:** `.agent/user_stories/kb-overview-prompt.md` + +## Objective +Create the storage for the knowledge-base outline: a single-row `kb_overview` table and its SQLAlchemy model. + +## Work +1. `alembic/versions/0005_kb_overview.py` — new revision (down_revision = phase 30's 0004): + - upgrade: create table `kb_overview` (`id INTEGER` PK `server_default sa.text("1")`, `content TEXT NOT NULL server_default sa.text("''")`, `updated_at TIMESTAMPTZ NOT NULL server_default=sa.func.now()`). + - downgrade: drop the table. +2. `app/models.py` — `class KbOverview(Base)`: `id: Mapped[int] = mapped_column(Integer, primary_key=True, server_default="1")`, `content: Mapped[str] = mapped_column(Text, server_default="")`, `updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())`. Docstring: single row, lite-generated KB outline, phase 31. +3. `tests/integration/test_migration_0005.py` — same style as `test_migration_0002.py` / `test_migration_0004.py`: upgrade → table exists with the three columns and defaults; downgrade → gone; upgrade again → back. + +## Testing & Quality +- Integration: the migration test above (real Postgres). +- Coverage: model exercised by existing model-test patterns; `app/` TOTAL ≥ pre-change. + +## Completion Criteria +- [ ] `uv run alembic upgrade head` clean on the dev DB; round-trip with `alembic downgrade -1` + `upgrade head`. +- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean. diff --git a/.agent/phases/todo/31_kb_overview_prompt/02_overview_generator.md b/.agent/phases/todo/31_kb_overview_prompt/02_overview_generator.md new file mode 100644 index 0000000..4ea9aad --- /dev/null +++ b/.agent/phases/todo/31_kb_overview_prompt/02_overview_generator.md @@ -0,0 +1,29 @@ +# Task 02 — app/rag/overview.py (generator + loader) + +**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "basic categories of everything that's been read… generated by the lite model… updated whenever we import new documents"` +**Story:** `.agent/user_stories/kb-overview-prompt.md` + +## Objective +Create the overview generator module: build the `lite` prompt from the document catalogue, generate the outline, store it in the single row, and expose a cheap loader for the chat path. + +## Work +1. `app/config.py` — add: + - `kb_overview_max_chars: int = 4_000` (`BOR_KB_OVERVIEW_MAX_CHARS`) — prompt-section budget (task 03). + - `overview_input_max_chars: int = 40_000` (`BOR_OVERVIEW_INPUT_MAX_CHARS`) — cap on the document list sent to the model. +2. `app/rag/overview.py` (new): + - `KB_OVERVIEW_MODE = "KB_OVERVIEW_MODE"` — marker the E2E mock keys on (same convention as `SUMMARY_MODE` / `DEFLECT_MODE`). + - `build_overview_prompt(rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None) -> tuple[str, str]` → `(system, user)`. Each row is `(source, path, title, summary)`; system = `KB_OVERVIEW_MODE` + instruction ("From the document list below, write a compact plain-text outline of the basic categories and topics this knowledge base covers. Group by source where useful, use `-` bullet lines, at most ~1500 characters, no markdown headings, and no topics not present in the list."); user = one line per doc `source — path — title — {first line of summary or ''}` joined by newlines, capped at *max_chars* (default `overview_input_max_chars`, overflow → shared `TRUNCATION_MARKER`). + - `load_kb_overview(db: Session) -> str` — the single row's `content` (trimmed) or `""` when the row is missing/empty. + - `async def regenerate_overview(llm, session: Session | None = None) -> bool` — load all documents (`source, path, title, summary` ordered by source, path); **zero documents → leave the existing row untouched, return False**; build the prompt; `text = await llm.chat([system, user], model=llm.settings.llm_summary_model)`; upsert the single row (`id=1`, `content=text`, `updated_at=now(UTC)`); commit; log `overview: regenerated docs=%d chars=%d`; return True. On `LLMError`: log `overview: regeneration failed — %s` and return False (previous row stays — see phase locked decisions). +3. `tests/unit/test_overview.py` (new) — fake LLM (duck-typed `chat` + `settings`), in-memory/SQLite session where the existing test infra allows (else a real-DB integration test in the same style as `test_steering.py`): + - prompt: system contains `KB_OVERVIEW_MODE`; user lines carry source/path/title/first summary line; cap truncates + marker. + - `load_kb_overview`: no row → `""`; row present → content. + - `regenerate_overview`: happy path upserts (content + fresh `updated_at`, returns True); zero docs → no DB write, returns False; `LLMError` → previous row unchanged, returns False. + +## Testing & Quality +- Unit/integration: the tests in Work step 3. +- Coverage: **>90%** on `app/rag/overview.py`. + +## Completion Criteria +- [ ] `regenerate_overview` is idempotent (single row, always id=1) and fail-soft; all tests green. +- [ ] `uv run ruff check . && uv run pyright` clean. diff --git a/.agent/phases/todo/31_kb_overview_prompt/03_prompt_injection.md b/.agent/phases/todo/31_kb_overview_prompt/03_prompt_injection.md new file mode 100644 index 0000000..4c08980 --- /dev/null +++ b/.agent/phases/todo/31_kb_overview_prompt/03_prompt_injection.md @@ -0,0 +1,30 @@ +# Task 03 — `` section in both prompts + chat wiring + +**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "The system prompt should inject basic categories of everything that's been read so the agent knows roughly what its knowledge base contains before the rag retrieval returns documents"` +**Story:** `.agent/user_stories/kb-overview-prompt.md` + +## Objective +Inject the stored overview into **every** chat turn's system prompt (HIGH and LOW modes) as a budgeted `` section — absent row → byte-identical prompts — and record `kb_chars` in the per-turn log line. + +## Work +1. `app/rag/prompts.py`: + - `build_kb_section(overview: str, max_chars: int | None = None) -> str` — empty/whitespace → `""`; otherwise `\n` + intro line ("The basic categories of everything in this knowledge base (generated at import time):") + the overview content, budgeted at *max_chars* (default `get_settings().kb_overview_max_chars`) with the shared `TRUNCATION_MARKER` for overflow (exact pattern of `build_steering_section`, including its pathological-budget handling). + - `build_high_prompt(documents, notes=None, kb_overview: str | None = None)` and `build_deflect_prompt(titles, notes=None, kb_overview: str | None = None)` — insert the section **between `` and the `` section** (i.e. order: `` → `` → `` → mode body); with an empty overview the output is byte-identical to today's text in both modes. +2. `app/api/chat.py`: + - Load per turn: `kb_overview = load_kb_overview(db)` next to the steering-notes load (one PK lookup — no LLM call). + - `plan_turn(chunks, settings, notes=None, kb_overview: str | None = None)` — pass it to both prompt builders; `TurnPlan` gains `kb_chars: int = 0` (length of the stored overview text when a non-empty row exists, else 0). + - Per-turn log line (PLAN §9): add `kb_chars=%d` after `tuning=%d`. Update any existing test asserting the log line format verbatim. +3. `tests/unit/test_prompts.py` — + - HIGH: no overview → byte-identical to the pre-phase builder output (build the expected string with `notes=None, kb_overview=None`); with overview → section present, ordered before `` when both exist. + - LOW: same pair of assertions (deflection prompt). + - budget: overview longer than `kb_overview_max_chars` → capped + `TRUNCATION_MARKER`. +4. `tests/unit/test_chat_gate.py` — `plan_turn` with an overview: both branches' `system_prompt` contains the section; `TurnPlan.kb_chars` == len(overview); empty overview → `kb_chars == 0` and prompt unchanged. + +## Testing & Quality +- Unit: Work steps 3–4; existing steering/prompt/gate tests stay green (new param is defaulted). +- Coverage: **>90%** on modified `app/rag/prompts.py` + `app/api/chat.py`; `app/` TOTAL ≥ pre-change. + +## Completion Criteria +- [ ] HIGH and LOW prompts are byte-identical to pre-phase text when no overview row exists (unit-asserted against the exact strings). +- [ ] With a row, both prompts carry the budgeted `` section in the locked order; the per-turn log line shows `kb_chars=…`. +- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean. diff --git a/.agent/phases/todo/31_kb_overview_prompt/04_import_trigger.md b/.agent/phases/todo/31_kb_overview_prompt/04_import_trigger.md new file mode 100644 index 0000000..d63bedf --- /dev/null +++ b/.agent/phases/todo/31_kb_overview_prompt/04_import_trigger.md @@ -0,0 +1,33 @@ +# Task 04 — import_docs regenerates the overview after a KB-changing import + +**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "should be updated whenever we import new documents"` +**Story:** `.agent/user_stories/kb-overview-prompt.md` + +## Objective +Wire the "update whenever we import new documents" trigger into the import script: after an import that changed the KB, regenerate the stored overview (best-effort). Phase 32's admin sync button reuses the exact same `regenerate_overview` call. + +## Work +1. `scripts/import_docs.py`: + - Restructure `main()`'s single `asyncio.run(import_sources(…))` into one `async def _run()` that (a) runs `import_sources(sources, llm, prune=args.prune, limit=args.limit)` and (b) — when the summary has `added + updated > 0` **or** no overview row exists yet — awaits `regenerate_overview(llm)` (import it from `app.rag.overview`). One event loop, same `LLMClient` instance. + - `--limit` debug runs skip the regeneration (an incomplete walk must not rewrite the outline — mirrors the existing `--prune`-with-`--limit` guard). + - The final `print` gains `overview=updated|skipped|failed` (failed = `regenerate_overview` returned False via LLMError; the import's own exit code is **unchanged** — a failed outline must not fail the import). + - `Limit` guard: when `limit` is set, `added + updated > 0` does *not* trigger regeneration (log `overview: skipped (--limit)`). +2. `tests/integration/test_import_docs_overview.py` (new) — with the git sync mocked out (reuse `test_import_docs_git.py`'s mocking style) and a fake LLM whose `chat` records calls: + - import with changed files → `kb_overview` row written; `chat` called once; print shows `overview=updated`. + - unchanged re-import (same hashes) → `chat` **not** called; print shows `overview=skipped`. + - `chat` raising `LLMError` → exit code still `0` (no import errors), print shows `overview=failed`, previous row untouched. + - `--limit` run with changes → `overview=skipped`. + - first-ever import (no row) with zero *changed* docs is not possible (new docs are "added") — but an empty-source run with no row → no row created, `overview=skipped`. +3. `.env.example` — `BOR_KB_OVERVIEW_MAX_CHARS` (default 4000), `BOR_OVERVIEW_INPUT_MAX_CHARS` (default 40000). +4. `README.md` — import workflow section: the import now refreshes the KB overview after a KB-changing run (fail-soft, `overview=` token in the summary line). + +- ASSUMPTION: "whenever we import new documents" = whenever an import **added or updated** at least one document (or the row doesn't exist yet); unchanged re-imports and `--limit` debug runs do not burn a lite call. + +## Testing & Quality +- Integration: Work step 2 (real Postgres, mocked git + fake LLM — no live aipi). +- Coverage: `app/` TOTAL ≥ pre-change (the script change is covered by the integration tests; the script itself is outside the `app/` gate). + +## Completion Criteria +- [ ] All new integration tests green; `tests/integration/test_import_docs_git.py` stays green. +- [ ] A manual run (`uv run python -m scripts.import_docs`) against the dev KB logs `overview: regenerated docs=… chars=…` after a KB-changing import and `overview=skipped` otherwise. +- [ ] `uv run ruff check . && uv run pyright` clean. diff --git a/.agent/phases/todo/31_kb_overview_prompt/05_mock_and_e2e.md b/.agent/phases/todo/31_kb_overview_prompt/05_mock_and_e2e.md new file mode 100644 index 0000000..4a4efda --- /dev/null +++ b/.agent/phases/todo/31_kb_overview_prompt/05_mock_and_e2e.md @@ -0,0 +1,31 @@ +# Task 05 — Deterministic KB_OVERVIEW_MODE mock + story E2E + commit + +**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — (whole item: system prompt injects basic categories of everything read, lite-generated, stored, updated on import)` +**Story:** `.agent/user_stories/kb-overview-prompt.md` + +## Objective +Make the overview observable end-to-end in a deterministic E2E: the mock generates a `KB_OVERVIEW_MODE` outline and echoes the injected section into its answer (the `(tuning: …)` precedent), plus the story suite, story file, and the phase commit. + +## Work +1. `tests/e2e/mock_llm.py`: + - Generation: in the chat-completions handler, if the system prompt contains `KB_OVERVIEW_MODE` → return the deterministic outline `f"Knowledge base outline:\n- {first 8 tokens of _user(body), space-joined}"` (the user message carries the document list). + - Echo: in `compose_answer` (all answer paths), if the system prompt contains a `` section, append `(kb: )` — parse with a regex in the `first_tuning_note` style (skip the intro line, take the first `-` line, strip the dash). This mirrors the steering echo exactly. +2. `tests/e2e/test_kb_overview.py` (new, the story gate): + - Seed the `kb_overview` row directly in the DB (the E2E test has DB access via the conftest fixtures — content with a recognizable first bullet, e.g. `- Kubernetes cluster and node maintenance notes`), so the test exercises the **injection** path deterministically (the CLI trigger path is covered by task 04's integration tests). + - Ask a normal on-topic question (the fixture KB is already imported by the conftest pattern) → assert the rendered brain answer ends with `(kb: Kubernetes cluster and node maintenance notes)`. + - Deflection control: ask an off-topic question (deflection path) → the answer still carries the `(kb: …)` echo (the section is in the LOW prompt too). + - Absence control: delete the row → a fresh question's answer has **no** `(kb: …)` suffix (byte-identical prompt behavior is unit-asserted in task 03; this proves it end-to-end). +3. Integration test (same task): extend `tests/integration/test_chat_api.py` (or a new `test_kb_overview_api.py`) with the capturing-fake-LLM pattern — no row: the system prompt sent to the model equals the pre-phase construction (assert the exact string via the existing `build_high_prompt`/`build_deflect_prompt` with `kb_overview=None`); row present: it contains the section in both HIGH and LOW turns. +4. `.agent/user_stories/kb-overview-prompt.md` (new) — narrative + acceptance criteria + Playwright mapping rule, styled like the other story files. +5. Commit: `git commit --no-gpg-sign -m "feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, section in HIGH+LOW prompts"`; move `.agent/phases/todo/31_kb_overview_prompt/` → `.agent/phases/complete/`. + +## Testing & Quality +- E2E: `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green **in isolation**. +- Regression: `test_steering.py`, `test_chat_rag.py`, `test_honest_deflection.py` stay green in isolation (prompt change is additive and defaulted). +- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`. + +## Completion Criteria +- [ ] `tests/e2e/test_kb_overview.py` green in isolation (injection, deflection, and absence all asserted). +- [ ] Integration prompt-capture tests green; existing steering/chat suites green. +- [ ] Full test + lint/type gates green (per this phase's 00_phase.md). +- [ ] Story file + `.env.example` + README complete; one `--no-gpg-sign` commit made. diff --git a/.agent/phases/todo/32_admin_sync_button/00_phase.md b/.agent/phases/todo/32_admin_sync_button/00_phase.md new file mode 100644 index 0000000..e15dfc3 --- /dev/null +++ b/.agent/phases/todo/32_admin_sync_button/00_phase.md @@ -0,0 +1,45 @@ +# Phase 32 — Admin Sync Button (one-click doc import sync) + +**Source:** `TODO.md L5 — "Need a button that only the admin can see that triggers a doc import sync by cloning the relevant repos and then running import doc script"` +**Story:** `.agent/user_stories/admin-sync-button.md` +**Context:** Phase 28 (`scripts/git_sync.py::clone_or_pull` — shallow clone / `--ff-only` pull; `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`; `repo_name` in `scripts/import_docs.py`), phase 31 (`regenerate_overview` — the sync refreshes the KB outline), phase 16 (`require_admin` dependency + the `header.js` `fetchIsAdmin()` reveal gate for admin-only UI like `#nav-sources` / `#nav-tuning`), PLAN §7.4 "never stale" feedback contract (the UI can never sit on a stale button state). + +## Objective +Give the admin a **"Sync sources"** button (Sources page, visible to the admin only) that triggers the full document sync in-process — clone/pull every `BOR_GIT_SOURCES` repo, re-import (with prune) so the KB mirrors the repos, and refresh the KB overview — with live, non-stale UI feedback driven by a polled sync-status endpoint. + +## Dependencies +- `31_kb_overview_prompt` (todo) — `regenerate_overview(llm)` is the sync's final step; `LLMClient.chat` for it. +- `28_git_based_sources` (complete) — `clone_or_pull` / `GitSyncError` / `BOR_GIT_SOURCES` / `repo_name` (the sync reuses them, does not re-implement git). +- `16_admin_auth` (complete) — `require_admin` for the new endpoints; the `header.js` whoami gate for the button. +- `19_shared_header` / `29_tuning_nav_link` (complete) — the Sources page header actions area where the button lives. + +## Tasks +1. `01_sync_api.md` — in-process sync runner: `POST /api/sync` (admin, 409 when running) + `GET /api/sync/status` (admin). +2. `02_ui_button.md` — the admin-only button on Sources with §7.4 feedback states (polling, last-result, error banner) + frontend unit assertions. +3. `03_e2e_and_docs.md` — `tests/e2e/test_sync_button.py` (real `file://` git fixture), README, story file, commit. + +## Testing & Quality +- Integration: sync API — anonymous 403s, admin idle/running/success/failed transitions, 409 double-trigger, GitSyncError → `failed` with the repo named (git + import + overview mocked, as `test_import_docs_git.py` does). +- Unit (frontend-assertion style, cf. `tests/unit/test_shared_header.py`): button markup hidden-by-default + labeled; `header.js` reveal; `sources.js` polling/terminal-state logic. +- Coverage: **>90%** on `app/` (`app/api/sync.py` fully covered); `app/` TOTAL ≥ pre-change. +- E2E (mandatory, A16): `tests/e2e/test_sync_button.py` — one story, run **in isolation**; uses a **real** local `file://` git repo fixture (deterministic, no network) with the mock LLM for embeddings. +- UI Structure Check (AGENTS.md rule 5): labeled button, focus-visible, contrast ≥4.5:1, `aria-live` result region, no CDN. + +## Completion Criteria +- [ ] Anonymous: the button is not revealed (stays `hidden`) and both endpoints return 403. +- [ ] Admin: clicking "Sync sources" starts the sync (202), the button goes disabled with "Syncing…" while polling `GET /api/sync/status` every 2 s, and on completion shows the last result (`Synced HH:MM` + `N added · M updated`); a failed sync re-enables the button with an error banner (`role="alert"`) naming the failure. +- [ ] A double trigger while running returns 409 and the UI never starts a second poll loop. +- [ ] After a successful sync against the `file://` fixture repo, the newly committed fixture doc appears in the Sources table and the `kb_overview` row is fresh (phase-31 trigger). +- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%). +- [ ] `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green in isolation; `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py` stay green. +- [ ] `uv run ruff check . && uv run pyright` clean. +- [ ] `.agent/user_stories/admin-sync-button.md` exists; README documents the button (behavior, states, prerequisites). +- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status`); `.agent/phases/todo/32_admin_sync_button/` moved to `.agent/phases/complete/`. + +## Locked decisions +- **A10 extended (recorded, not a revision)** — two new **admin-only** endpoints (`POST /api/sync`, `GET /api/sync/status`) behind the existing `require_admin`; the public API surface stays stateless, the signed cookie remains the only session state (same pattern as `/api/steering`). +- **A12 untouched** — the sync runs **in-process** (one `asyncio` background task + a module-level status object in `app/api/sync.py`). The app is a single instance on the homelab; no Valkey/queue. Status is in memory — a restart mid-sync loses the running state (accepted: the next click re-syncs idempotently). +- **Sync semantics** — the button targets `BOR_GIT_SOURCES` only (manual `--source` dirs have no repo to clone; an unset/empty `BOR_GIT_SOURCES` → the sync fails loudly with "no git sources configured"); the import runs with **`prune=True`** so files deleted upstream leave the index (the button is the canonical "mirror the repos" action — the CLI default of no-prune is unchanged); phase-31's `regenerate_overview` runs after the import when docs changed. +- **Concurrency** — one sync at a time: `POST /api/sync` while running → `409 {"detail": "a sync is already running"}`; the UI reflects the in-flight run (re-attaches on page load while a sync is running). +- **§7.4 adaptation (recorded)** — the 120 s client guard applies to LLM turns; a sync can legitimately run for minutes (clone + embed), so the button has **no client-side hard timeout** — the 2 s status poll is the feedback loop and the server state is authoritative. The button is disabled until the run reaches a terminal state, so it can never be stale *or* stuck: a failed run re-enables it, a running run always shows "Syncing…". +- **A16 / A17 honoured** — one dedicated story E2E suite (real `file://` git fixture — git is a documented environment prerequisite, as in phase 28); one atomic `--no-gpg-sign` commit. diff --git a/.agent/phases/todo/32_admin_sync_button/01_sync_api.md b/.agent/phases/todo/32_admin_sync_button/01_sync_api.md new file mode 100644 index 0000000..5fa9359 --- /dev/null +++ b/.agent/phases/todo/32_admin_sync_button/01_sync_api.md @@ -0,0 +1,37 @@ +# Task 01 — Sync API: in-process runner + status + +**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "triggers a doc import sync by cloning the relevant repos and then running import doc script"` +**Story:** `.agent/user_stories/admin-sync-button.md` + +## Objective +The backend of the sync button: an admin-only `POST /api/sync` that starts the clone → import → overview pipeline as one in-process background task, and `GET /api/sync/status` for the UI's polling loop. + +## Work +1. `app/api/sync.py` (new): + - `@dataclass SyncStatus` — `state: Literal["idle", "running", "success", "failed"] = "idle"`, `started_at: datetime | None`, `finished_at: datetime | None`, `detail: dict[str, Any] = field(default_factory=dict)`, `error: str | None`; module-level `_status` + `_task: asyncio.Task | None`. + - `GET /api/sync/status` (`Depends(require_admin)`) → JSON `{state, started_at, finished_at, detail, error}` (datetimes ISO-8601 or null). + - `POST /api/sync` (`Depends(require_admin)`) — if `_task` is not done → `409 {"detail": "a sync is already running"}`; else `_task = asyncio.create_task(_run_sync())` → `202 {"detail": "sync started"}`. + - `async def _run_sync()`: + 1. `_status.state = "running"`, `started_at = now(UTC)`. + 2. Resolve repos from `settings.git_source_list` — empty → fail with `"no git sources configured (BOR_GIT_SOURCES)"`. + 3. For each URL: `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))` (imported from `scripts.git_sync` / `scripts.import_docs` — no git re-implementation; `GitSyncError` carries git's stderr). + 4. `summary = await import_sources(sources, LLMClient(), prune=True)` (prune per phase locked decision). + 5. If `summary.added + summary.updated > 0`: `await regenerate_overview(llm)`. + 6. `_status.state = "success"`, `finished_at`, `detail = {files, added, updated, unchanged, pruned, errors, chunks, summaries, summary_errors, overview: bool}`; log `sync: done detail=…`. + 7. Any `GitSyncError | EmbeddingError | Exception` → `_status.state = "failed"`, `finished_at`, `error = str(e)` (sanitized: no secrets; git's stderr is fine), `logger.exception("sync: failed")`. +2. `app/main.py` — `from app.api.sync import router as sync_router` + `app.include_router(sync_router, prefix="/api")` (next to the other routers). +3. `tests/integration/test_sync_api.py` (new) — sign in via the existing auth test helper (`tests/integration/test_auth_api.py` pattern): + - anonymous: `GET /api/sync/status` → 403; `POST /api/sync` → 403. + - admin: idle state initially; `BOR_GIT_SOURCES` set to one `file://` URL with `clone_or_pull`, `import_sources`, `regenerate_overview` **monkeypatched** in `app.api.sync` (the mock import returns a canned `ImportSummary`; the mock overview returns True) → `POST` → 202; poll status → `success` with the canned detail (all ImportSummary fields + `overview: true`). + - 409: mock runner sleeps briefly (asyncio.sleep) → second `POST` while running → 409. + - failure: mock `clone_or_pull` raises `GitSyncError("git clone failed …")` → status `failed`, `error` names the failure; import is **not** called. + - empty `BOR_GIT_SOURCES` → `POST` 202 → status `failed` with the "no git sources configured" message. + - prune: assert the monkeypatched `import_sources` received `prune=True`. + +## Testing & Quality +- Integration: Work step 3 (real Postgres not required for the runner logic beyond none — keep DB-free; if the session needs Postgres for nothing, use the app fixture without DB). +- Coverage: **>90%** on `app/api/sync.py` (all states/branches hit). + +## Completion Criteria +- [ ] All integration tests green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean. +- [ ] SSE/API routes untouched — `test_chat_api.py` green (no middleware or router precedence change). diff --git a/.agent/phases/todo/32_admin_sync_button/02_ui_button.md b/.agent/phases/todo/32_admin_sync_button/02_ui_button.md new file mode 100644 index 0000000..c3abe3d --- /dev/null +++ b/.agent/phases/todo/32_admin_sync_button/02_ui_button.md @@ -0,0 +1,40 @@ +# Task 02 — The admin-only Sync button on Sources (§7.4 feedback) + +**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "a button that only the admin can see that triggers a doc import sync"` +**Story:** `.agent/user_stories/admin-sync-button.md` + +## Objective +The UI: a **"Sync sources"** button in the Sources page header — hidden by default, revealed only for the signed-in admin (the existing `header.js` whoami gate) — with the full "never stale" feedback lifecycle: idle → "Syncing…" (disabled, spinner, 2 s status polling) → last-result label or error banner. + +## Work +1. `frontend/sources.html` — in the header actions area (next to `.new-chat-btn`, inside the same `.header-inner` container the phase-19 shared header uses on this page): + - `