ducoterra
022da8e2bc
Foundation (phase 01, verified): - FastAPI app: /api/health, /api/suggestions, /api/chat (placeholder), static frontend served locally (no CDN) - Postgres 17 + pgvector via db/Containerfile + compose.yaml (podman compose up -d db), Alembic initial migration (documents, chunks with vector(768), query_log) - LLM client targeting https://aipi.reeseapps.com/v1 (turbo/embed); scripts/llm_probe.py verified models + 768-dim embeddings live - Conditional debugpy: imported only when DEBUGPY=1 (attach on demand, :5678); logging config for clean single-line logs - Frontend shell: mobile-first chat + Sources pages, tokens, a11y baselines - Tests: 24 unit+integration (99% coverage on app/), ruff + pyright clean, Playwright smoke E2E (3 tests) against a deterministic mock LLM - Planning: .agent/PLAN.md (architecture + LOCKED decisions), AGENTS.md, 6 user stories, 7 phase files (one story / one phase / one Playwright suite each)
58 lines
3.0 KiB
Markdown
58 lines
3.0 KiB
Markdown
# Story: Honest Deflection
|
||
|
||
**Phase:** `04_story_honest_deflection.md` · **E2E:** `tests/e2e/test_honest_deflection.py`
|
||
|
||
## Narrative
|
||
|
||
As **a user**, when I ask something Brain genuinely has no notes about, I
|
||
want it to **admit it plainly** and still be helpful — so I never walk away
|
||
with a confident-sounding hallucination.
|
||
|
||
- **Given** the knowledge base is about homelab/infra topics
|
||
- **When** I ask "How do I bake sourdough bread?"
|
||
- **Then** retrieval's best similarity is below `BOR_RELEVANCE_THRESHOLD`,
|
||
Brain switches to deflection mode, opens with a variant of
|
||
**"I haven't done anything like that"**, stays chippy, and offers 2–3
|
||
alternative questions about things it *does* know (from the weak hits).
|
||
|
||
## Acceptance criteria
|
||
1. Gate: `max(1 − cosine_distance) < BOR_RELEVANCE_THRESHOLD` ⇒
|
||
`<relevance>LOW</relevance>` + `DEFLECT_MODE` system prompt (weak-hit
|
||
**titles only**, no full docs).
|
||
2. The LLM is still called (voice stays chippy); the prompt forces the
|
||
honesty phrasing + alternative suggestions (PLAN §6).
|
||
3. `done` event carries `deflected: true` and `suggestions[]` (2–3 strings).
|
||
4. `query_log` row has `deflected=true` + the weak `top_score`.
|
||
5. UI: the deflected bubble is visually distinct (amber border/background),
|
||
and "Maybe try:" chips render below it; clicking a chip asks that
|
||
question (delegated to the suggestion-chips story for chip behavior;
|
||
here only rendering).
|
||
6. Threshold is env-tunable; lowering it to ~0 makes every question an
|
||
"answer" (documented in README troubleshooting).
|
||
7. Unit tests cover the gate boundary (score == threshold → answer mode;
|
||
just below → deflect) using a fake retriever — no LLM needed.
|
||
|
||
## UI Visualization & Structure
|
||
- Deflected brain bubble: `background: var(--accent-bg) #fff7e8`,
|
||
`border: 1px solid var(--accent-line) #f59e0b`, text stays `var(--ink)`
|
||
(or accent-ink for emphasis ≥4.5:1) — clearly "different" from a normal
|
||
answer without being alarm-red (it's honesty, not an error).
|
||
- Below the bubble: `Maybe try:` label (visually hidden for SR, `aria-label`
|
||
on the chip group) + 2–3 `.suggestion-chip` pills (same chip component as
|
||
onboarding: ≥44px height, brand-soft bg, brand-ink text).
|
||
- Bubble may include the model's alternative list in text too; chips are the
|
||
one-click affordance.
|
||
- Contrast audit: `#92400e` on `#fff7e8` ≈ 8.7:1 ✓; chip text on chip bg ≥6:1 ✓.
|
||
|
||
## Playwright Mapping Rule
|
||
**Test Scenario → `tests/e2e/test_honest_deflection.py`** (mock LLM, seeded KB):
|
||
1. `test_off_topic_question_deflects_honestly` — ask "How do I bake
|
||
sourdough bread?"; assert the answer bubble is `.is-deflected`, its text
|
||
matches /haven't done anything like that/i, and ≥2 "Maybe try:" chips
|
||
render below it.
|
||
2. `test_deflection_suggestions_are_clickable` — click the first deflection
|
||
chip; assert the input is populated/focus behavior per chip contract and a
|
||
new user bubble is created.
|
||
3. `test_threshold_gate_unit_boundary` is a **unit** test (not Playwright):
|
||
retriever returns score 0.30 → HIGH; 0.2999 → LOW (mocked components).
|