feat: scaffold Brain of Reese — FastAPI RAG chat over Postgres 17 + pgvector
Foundation (phase 01, verified): - FastAPI app: /api/health, /api/suggestions, /api/chat (placeholder), static frontend served locally (no CDN) - Postgres 17 + pgvector via db/Containerfile + compose.yaml (podman compose up -d db), Alembic initial migration (documents, chunks with vector(768), query_log) - LLM client targeting https://aipi.reeseapps.com/v1 (turbo/embed); scripts/llm_probe.py verified models + 768-dim embeddings live - Conditional debugpy: imported only when DEBUGPY=1 (attach on demand, :5678); logging config for clean single-line logs - Frontend shell: mobile-first chat + Sources pages, tokens, a11y baselines - Tests: 24 unit+integration (99% coverage on app/), ruff + pyright clean, Playwright smoke E2E (3 tests) against a deterministic mock LLM - Planning: .agent/PLAN.md (architecture + LOCKED decisions), AGENTS.md, 6 user stories, 7 phase files (one story / one phase / one Playwright suite each)
This commit is contained in:
@@ -0,0 +1,57 @@
|
||||
# Story: Honest Deflection
|
||||
|
||||
**Phase:** `04_story_honest_deflection.md` · **E2E:** `tests/e2e/test_honest_deflection.py`
|
||||
|
||||
## Narrative
|
||||
|
||||
As **a user**, when I ask something Brain genuinely has no notes about, I
|
||||
want it to **admit it plainly** and still be helpful — so I never walk away
|
||||
with a confident-sounding hallucination.
|
||||
|
||||
- **Given** the knowledge base is about homelab/infra topics
|
||||
- **When** I ask "How do I bake sourdough bread?"
|
||||
- **Then** retrieval's best similarity is below `BOR_RELEVANCE_THRESHOLD`,
|
||||
Brain switches to deflection mode, opens with a variant of
|
||||
**"I haven't done anything like that"**, stays chippy, and offers 2–3
|
||||
alternative questions about things it *does* know (from the weak hits).
|
||||
|
||||
## Acceptance criteria
|
||||
1. Gate: `max(1 − cosine_distance) < BOR_RELEVANCE_THRESHOLD` ⇒
|
||||
`<relevance>LOW</relevance>` + `DEFLECT_MODE` system prompt (weak-hit
|
||||
**titles only**, no full docs).
|
||||
2. The LLM is still called (voice stays chippy); the prompt forces the
|
||||
honesty phrasing + alternative suggestions (PLAN §6).
|
||||
3. `done` event carries `deflected: true` and `suggestions[]` (2–3 strings).
|
||||
4. `query_log` row has `deflected=true` + the weak `top_score`.
|
||||
5. UI: the deflected bubble is visually distinct (amber border/background),
|
||||
and "Maybe try:" chips render below it; clicking a chip asks that
|
||||
question (delegated to the suggestion-chips story for chip behavior;
|
||||
here only rendering).
|
||||
6. Threshold is env-tunable; lowering it to ~0 makes every question an
|
||||
"answer" (documented in README troubleshooting).
|
||||
7. Unit tests cover the gate boundary (score == threshold → answer mode;
|
||||
just below → deflect) using a fake retriever — no LLM needed.
|
||||
|
||||
## UI Visualization & Structure
|
||||
- Deflected brain bubble: `background: var(--accent-bg) #fff7e8`,
|
||||
`border: 1px solid var(--accent-line) #f59e0b`, text stays `var(--ink)`
|
||||
(or accent-ink for emphasis ≥4.5:1) — clearly "different" from a normal
|
||||
answer without being alarm-red (it's honesty, not an error).
|
||||
- Below the bubble: `Maybe try:` label (visually hidden for SR, `aria-label`
|
||||
on the chip group) + 2–3 `.suggestion-chip` pills (same chip component as
|
||||
onboarding: ≥44px height, brand-soft bg, brand-ink text).
|
||||
- Bubble may include the model's alternative list in text too; chips are the
|
||||
one-click affordance.
|
||||
- Contrast audit: `#92400e` on `#fff7e8` ≈ 8.7:1 ✓; chip text on chip bg ≥6:1 ✓.
|
||||
|
||||
## Playwright Mapping Rule
|
||||
**Test Scenario → `tests/e2e/test_honest_deflection.py`** (mock LLM, seeded KB):
|
||||
1. `test_off_topic_question_deflects_honestly` — ask "How do I bake
|
||||
sourdough bread?"; assert the answer bubble is `.is-deflected`, its text
|
||||
matches /haven't done anything like that/i, and ≥2 "Maybe try:" chips
|
||||
render below it.
|
||||
2. `test_deflection_suggestions_are_clickable` — click the first deflection
|
||||
chip; assert the input is populated/focus behavior per chip contract and a
|
||||
new user bubble is created.
|
||||
3. `test_threshold_gate_unit_boundary` is a **unit** test (not Playwright):
|
||||
retriever returns score 0.30 → HIGH; 0.2999 → LOW (mocked components).
|
||||
Reference in New Issue
Block a user