brain-of-reese/.agents/user_stories/retrieval-quality.md

Story: Retrieval Quality — Multi-Format Ingestion + Hybrid Search

Phase: 09_story_retrieval_quality.md · E2E: tests/e2e/test_retrieval_quality.py

Narrative

As a user, I want to ask questions that name my tools ("How did I install gitlab?") and get a grounded answer citing the right document — even when the answer lives in a .yaml, .py, .json or .txt file — so Brain actually covers my notes instead of deflecting around them.

• Given my notes are a mix of markdown write-ups and config/code files, and today's index is polluted with vendored-cache docs (.esphome/.espressif/**, merge-request templates, …) that outrank real content, while retrieval is pure cosine top-4
• When the importer covers the full text-format set, skips hidden directories, and retrieval fuses vector similarity with Postgres full-text search
• Then "How did I install gitlab?" surfaces active/container_gitlab/ gitlab.md as the top document, the junk docs are pruned from the index, and genuinely off-topic questions still get the honest deflection.

Acceptance criteria

1. Ingestion scope (A9, revised with owner permission 2026-08-21): default extensions md, markdown, txt, yaml, yml, json, py, overridable via BOR_IMPORT_EXTENSIONS; hidden (dot) directories skipped by default; the A9 exclusion list stays.
2. Format-aware chunking: md unchanged (heading-aware); yaml/yml split on top-level keys and --- document separators (key kept as chunk anchor); json pretty-printed and split on top-level keys; py split on top-level defs/classes (stdlib ast); txt on paragraphs. Every format honors the 1200-char hard cap (aipi ~1024-token request cap).
3. Hybrid retrieval (A7, revised with owner permission 2026-08-21): cosine top-30 + FTS top-30 (OR tsquery over the question's tokens, ts_rank-ordered) fused with Reciprocal Rank Fusion (k=60); document-level ranking by best fused chunk score; the LLM still receives the full top-2 documents capped at 24k chars (contract preserved).
4. Honesty gate (A8, revised with owner permission 2026-08-21): deflect when best cosine < BOR_RELEVANCE_THRESHOLD and no candidate chunk FTS-matches the question; threshold re-tuned for the embed model's measured 0.41–0.84 score compression (default 0.62, documented tuning procedure); fts_hits logged per turn and stored in query_log.
5. Live verification: scripts/eval_retrieval.py run against the imported KB shows gitlab.md as the top document for "How did I install gitlab?"; a known off-topic question gates LOW.
6. Index hygiene: after a --prune re-import, zero documents with a dot-prefixed path component exist in documents; prune also drops docs that no longer match the format filter.
7. No regressions: off-topic deflection (story 04) and grounded answers (story 03) still pass their E2E suites in isolation; the E2E mock keeps its own threshold calibration via the app fixture.
8. Unit + integration green, app/ coverage >90%, story E2E green in isolation, one --no-gpg-sign commit.

UI Visualization & Structure

No UI change. The Sources page document count drops after the junk prune (intended cleanup — noted in the README).

Playwright Mapping Rule

Test Scenario → tests/e2e/test_retrieval_quality.py (mock LLM, seeded KB):

1. test_multiformat_docs_indexed — fixture tree with .md, .yaml, .py, .json, .txt plus .hidden/junk.md: import in-process → /api/docs lists every non-hidden doc and none of the hidden ones.
2. test_named_tool_question_retrieves_right_doc — "How did I install gitlab?" → not deflected, streamed answer, a .source-chip containing the gitlab doc, query_log row deflected=false with the gitlab doc in sources.
3. test_keyword_only_question_beats_vector_ranking — a question whose rare token exists in exactly one fixture doc (weak mock-embedding overlap, e.g. "kafkabridge") → grounded answer citing that doc (FTS-OR gate + lexical RRF lift, end to end).
4. test_off_topic_still_deflects — "sourdough starter" → .is-deflected bubble matching /haven't done anything like that/i + ≥2 "Maybe try:" chips (story 04 behavior under the new gate).