feat(rag): honest deflection gate with amber UI state and alternative-question chips

.agent/phases/complete/02_story_import_documents.md

@@ -0,0 +1,76 @@
+# Phase 02 — Story: Import Documents
+
+**Story:** `.agent/user_stories/import-documents.md`
+**Context:** `.agent/PLAN.md` §5 (data model), §9 (logging), §11 (import workflow)
+
+## Goal
+The importer (`scripts/import_docs.py`) + `GET /api/docs` + the Sources page
+rendering the indexed documents — the knowledge base becomes refreshable.
+
+## Implementation steps
+1. `app/rag/__init__.py`, `app/rag/chunker.py` — markdown-aware chunker
+   (PLAN §5 policy: heading splits, 2000-char target, 200 overlap, keep
+   nearest heading). Pure functions, fully unit-testable.
+2. `app/rag/llm.py` — `LLMClient` (openai async) with `embed(texts) ->
+   list[list[float]]` (batched, `BOR_EMBED_BATCH_SIZE`) and a
+   `embed_one`; dimension check vs `settings.embedding_dim` with a loud,
+   actionable error. (Chat streaming is added in Phase 03 on this client.)
+3. `app/rag/importer.py` — the core: directory walk (exclusion list, PLAN
+   A9; `*.md` only), sha256 delta vs `documents.content_hash`,
+   upsert-or-skip, two-phase chunk replace (insert doc → replace chunks →
+   embed → commit), `--prune` support, per-file + summary logging.
+4. `scripts/import_docs.py` — CLI wrapper (argparse): repeatable
+   `--source` (default `~/Homelab` `~/Deployments`, `expanduser`),
+   `--prune`, `--limit`.
+5. `app/api/docs.py` — `GET /api/docs` → `{"documents": [DocSummary]}`
+   (include `chunks` count via `func.count`); mount in `app/main.py`
+   **before** the static mount.
+6. `frontend/assets/sources.js` + `sources.html` polish — wire the real
+   endpoint (already scaffolded to expect this shape); keep the empty state.
+7. Update `README.md` §Knowledge Base Import with the final commands +
+   exclusion list + "update your docs → re-run the script" workflow.
+
+## UI Verification
+Compare `/sources.html` against the story's "UI Visualization & Structure":
+stat cards `auto-fit minmax(170px,1fr)`; full-width table (≥85% container);
+mono path column with `title` ellipsis; empty state with the exact command;
+`<caption class="visually-hidden">`, `scope="col"`, scroll wrapper
+`role="region" tabindex="0"`. No CDN refs. Take a 1280px and 375px
+screenshot pass before finishing.
+
+## Testing & Quality
+- Unit: chunker (heading splits, overlap, short-doc single chunk, code
+  fences kept intact), exclusion walk (temp tree with `.venv` junk),
+  delta logic (unchanged/changed/pruned via tmp Postgres or in-memory fakes
+  — real DB preferred since compose runs locally).
+- Integration: `GET /api/docs` empty shape + populated shape; importer
+  end-to-end against `tests/fixtures/docs/` into a test schema.
+- Coverage: `uv run pytest --cov=app --cov-report=term-missing` — **>90%**
+  on `app/` (importer + chunker + client are the bulk; test them hard).
+
+## Playwright Execution Phase
+Run ONLY this story's suite (DB must be up: `podman compose up -d db`):
+
+```bash
+uv run pytest tests/e2e/test_import_documents.py -v --no-cov
+```
+
+The test file implements the story's Playwright Mapping Rule (seed via the
+import function against `tests/fixtures/docs/` with the mock LLM; assert
+Sources page rows, layout width, and the empty state).
+
+## Success criteria
+- [ ] `uv run python -m scripts.import_docs` (fixtures) imports all 3 docs,
+      re-run reports `unchanged`
+- [ ] `GET /api/docs` + Sources page show the docs (real run: `~/Homelab`
+      + `~/Deployments` counts logged)
+- [ ] unit + integration green, coverage >90%
+- [ ] UI verification passed (screenshots attached to the phase record)
+- [ ] story E2E green in isolation
+- [ ] README import section updated
+- [ ] committed
+
+## Commit
+```bash
+git add -A && git commit --no-gpg-sign -m "feat(kb): markdown importer with sha256 deltas, chunking, batched embeddings, and Sources page"
+```

.agent/phases/complete/03_story_chat_rag.md

@@ -0,0 +1,74 @@
+# Phase 03 — Story: Chat RAG Answer (happy path)
+
+**Story:** `.agent/user_stories/chat-rag-answer.md`
+**Context:** `.agent/PLAN.md` §3 (data flow), §4 (SSE contract), §6 (persona), §9 (logging)
+
+## Goal
+The core product loop: question → embed → cosine top-4 → full top-2
+documents → `turbo` (streamed) → chippy grounded answer with source chips.
+
+## Implementation steps
+1. `app/rag/retriever.py` — `retrieve(db, question_embedding) ->
+   list[RetrievedChunk]` (score = 1 − distance, `ORDER BY embedding <=> $1
+   LIMIT BOR_TOP_K_CHUNKS`) + `select_documents(chunks, n) -> list[Document]`
+   (distinct by `document_id`, ranked by best chunk score, cap content at
+   `BOR_MAX_CONTEXT_CHARS` with `[…truncated…]`).
+2. `app/rag/prompts.py` — locked persona + HONESTY GATE prompt builder
+   (PLAN §6 verbatim, `<relevance>HIGH|LOW</relevance>`, `<documents>`
+   block; LOW mode includes the `DEFLECT_MODE` marker + weak-hit titles).
+3. `app/rag/llm.py` — add `chat_stream(messages) -> AsyncIterator[str]`
+   (openai async, `stream=True`, `model=turbo`, temperature 0.4,
+   max_tokens ~700).
+4. `app/api/chat.py` — `POST /api/chat` (ChatRequest) → `StreamingResponse`
+   (SSE): emit `delta` events from the stream, then the `done` event
+   (deflected, sources, suggestions); insert `query_log` row (deflected=
+   false this phase); per-turn log line (PLAN §9); structured error events
+   (`{"type":"error","detail":…}`) on LLM/DB failure.
+5. `frontend/assets/app.js` — replace the placeholder handler: `fetch` +
+   `ReadableStream` SSE parser; render deltas live into a brain bubble
+   (reuse the typing-indicator → streaming handoff); on `done`, append
+   `.source-chip`s under the bubble; on error, show the banner (full
+   state machine is Phase 06 — keep it simple-correct here).
+6. Tune `settings.suggestions` if the real Homelab import revealed better
+   defaults (optional here; Phase 05 owns the chips).
+
+## UI Verification
+Against the story's "UI Visualization & Structure": bubbles right/left
+(brand vs surface, ≥4.5:1 text), avatar 🧠, source chips mono/brand-soft
+with `source/path` and ellipsis, safe markdown (paste an answer containing
+`<script>alert(1)</script>` from the mock to prove it's escaped). Chat
+column 46rem centered. 1280px + 375px screenshot pass.
+
+## Testing & Quality
+- Unit: retriever ordering/dedup/cap (fake rows), prompt builder (HIGH
+  contains documents + `HIGH`, LOW contains `DEFLECT_MODE` + titles only,
+  persona rules present verbatim), SSE event serialization.
+- Integration: `/api/chat` against the mock LLM with a seeded temp schema —
+  assert SSE delta sequence, `done` payload (sources non-empty,
+  deflected false), `query_log` row, error event when LLM unreachable.
+- Coverage: `uv run pytest --cov=app --cov-report=term-missing` — **>90%**.
+
+## Playwright Execution Phase
+Run ONLY this story's suite:
+
+```bash
+uv run pytest tests/e2e/test_chat_rag.py -v --no-cov
+```
+
+Implements the story mapping: streamed grounded answer + `kubernetes.md`
+source chip + button recovery; DB `query_log` assertion; raw SSE shape
+check via `httpx`.
+
+## Success criteria
+- [ ] end-to-end: question → streamed chippy answer citing `kubernetes.md`
+- [ ] `query_log` row per turn; per-turn log line in stdout
+- [ ] LLM-down path shows error banner, no stuck button
+- [ ] unit + integration green, coverage >90%
+- [ ] UI verification passed
+- [ ] story E2E green in isolation
+- [ ] committed
+
+## Commit
+```bash
+git add -A && git commit --no-gpg-sign -m "feat(rag): stream grounded chat answers via pgvector cosine retrieval with source citations"
+```