Files

T

ducoterra 022da8e2bc feat: scaffold Brain of Reese — FastAPI RAG chat over Postgres 17 + pgvector

Foundation (phase 01, verified):
- FastAPI app: /api/health, /api/suggestions, /api/chat (placeholder),
  static frontend served locally (no CDN)
- Postgres 17 + pgvector via db/Containerfile + compose.yaml
  (podman compose up -d db), Alembic initial migration (documents,
  chunks with vector(768), query_log)
- LLM client targeting https://aipi.reeseapps.com/v1 (turbo/embed);
  scripts/llm_probe.py verified models + 768-dim embeddings live
- Conditional debugpy: imported only when DEBUGPY=1 (attach on demand,
  :5678); logging config for clean single-line logs
- Frontend shell: mobile-first chat + Sources pages, tokens, a11y baselines
- Tests: 24 unit+integration (99% coverage on app/), ruff + pyright clean,
  Playwright smoke E2E (3 tests) against a deterministic mock LLM
- Planning: .agent/PLAN.md (architecture + LOCKED decisions), AGENTS.md,
  6 user stories, 7 phase files (one story / one phase / one Playwright
  suite each)

2026-08-21 13:42:21 -04:00

3.0 KiB

Raw Blame History

Story: Honest Deflection

Phase: 04_story_honest_deflection.md · E2E: tests/e2e/test_honest_deflection.py

Narrative

As a user, when I ask something Brain genuinely has no notes about, I want it to admit it plainly and still be helpful — so I never walk away with a confident-sounding hallucination.

Given the knowledge base is about homelab/infra topics
When I ask "How do I bake sourdough bread?"
Then retrieval's best similarity is below BOR_RELEVANCE_THRESHOLD, Brain switches to deflection mode, opens with a variant of "I haven't done anything like that", stays chippy, and offers 2–3 alternative questions about things it does know (from the weak hits).

Acceptance criteria

Gate: max(1 − cosine_distance) < BOR_RELEVANCE_THRESHOLD ⇒ <relevance>LOW</relevance> + DEFLECT_MODE system prompt (weak-hit titles only, no full docs).
The LLM is still called (voice stays chippy); the prompt forces the honesty phrasing + alternative suggestions (PLAN §6).
done event carries deflected: true and suggestions[] (2–3 strings).
query_log row has deflected=true + the weak top_score.
UI: the deflected bubble is visually distinct (amber border/background), and "Maybe try:" chips render below it; clicking a chip asks that question (delegated to the suggestion-chips story for chip behavior; here only rendering).
Threshold is env-tunable; lowering it to ~0 makes every question an "answer" (documented in README troubleshooting).
Unit tests cover the gate boundary (score == threshold → answer mode; just below → deflect) using a fake retriever — no LLM needed.

UI Visualization & Structure

Deflected brain bubble: background: var(--accent-bg) #fff7e8, border: 1px solid var(--accent-line) #f59e0b, text stays var(--ink) (or accent-ink for emphasis ≥4.5:1) — clearly "different" from a normal answer without being alarm-red (it's honesty, not an error).
Below the bubble: Maybe try: label (visually hidden for SR, aria-label on the chip group) + 2–3 .suggestion-chip pills (same chip component as onboarding: ≥44px height, brand-soft bg, brand-ink text).
Bubble may include the model's alternative list in text too; chips are the one-click affordance.
Contrast audit: #92400e on #fff7e8 ≈ 8.7:1 ✓; chip text on chip bg ≥6:1 ✓.

Playwright Mapping Rule

Test Scenario → tests/e2e/test_honest_deflection.py (mock LLM, seeded KB):

test_off_topic_question_deflects_honestly — ask "How do I bake sourdough bread?"; assert the answer bubble is .is-deflected, its text matches /haven't done anything like that/i, and ≥2 "Maybe try:" chips render below it.
test_deflection_suggestions_are_clickable — click the first deflection chip; assert the input is populated/focus behavior per chip contract and a new user bubble is created.
test_threshold_gate_unit_boundary is a unit test (not Playwright): retriever returns score 0.30 → HIGH; 0.2999 → LOW (mocked components).

3.0 KiB Raw Blame History Unescape Escape