# Story: Retrieval Quality — Multi-Format Ingestion + Hybrid Search

**Phase:** `09_story_retrieval_quality.md` · **E2E:** `tests/e2e/test_retrieval_quality.py`

## Narrative

As **a user**, I want to ask questions that name my tools ("How did I
install gitlab?") and get a grounded answer citing the right document —
even when the answer lives in a `.yaml`, `.py`, `.json` or `.txt` file —
so Brain actually covers my notes instead of deflecting around them.

- **Given** my notes are a mix of markdown write-ups and config/code
  files, and today's index is polluted with vendored-cache docs
  (`.esphome/.espressif/**`, merge-request templates, …) that outrank
  real content, while retrieval is pure cosine top-4
- **When** the importer covers the full text-format set, skips hidden
  directories, and retrieval fuses vector similarity with Postgres
  full-text search
- **Then** "How did I install gitlab?" surfaces `active/container_gitlab/
  gitlab.md` as the top document, the junk docs are pruned from the
  index, and genuinely off-topic questions still get the honest
  deflection.

## Acceptance criteria
1. **Ingestion scope (A9, revised with owner permission 2026-08-21):**
   default extensions `md, markdown, txt, yaml, yml, json, py`,
   overridable via `BOR_IMPORT_EXTENSIONS`; **hidden (dot) directories
   skipped by default**; the A9 exclusion list stays.
2. **Format-aware chunking:** md unchanged (heading-aware); yaml/yml
   split on top-level keys and `---` document separators (key kept as
   chunk anchor); json pretty-printed and split on top-level keys; py
   split on top-level defs/classes (stdlib `ast`); txt on paragraphs.
   Every format honors the 1200-char hard cap (aipi ~1024-token request
   cap).
3. **Hybrid retrieval (A7, revised with owner permission 2026-08-21):**
   cosine top-30 + FTS top-30 (OR tsquery over the question's tokens,
   `ts_rank`-ordered) fused with Reciprocal Rank Fusion (k=60);
   document-level ranking by best fused chunk score; the LLM still
   receives the **full top-2 documents** capped at 24k chars (contract
   preserved).
4. **Honesty gate (A8, revised with owner permission 2026-08-21):**
   deflect when `best cosine < BOR_RELEVANCE_THRESHOLD` **and** no
   candidate chunk FTS-matches the question; threshold re-tuned for the
   `embed` model's measured 0.41–0.84 score compression (default 0.62,
   documented tuning procedure); `fts_hits` logged per turn and stored in
   `query_log`.
5. **Live verification:** `scripts/eval_retrieval.py` run against the
   imported KB shows `gitlab.md` as the top document for "How did I
   install gitlab?"; a known off-topic question gates LOW.
6. **Index hygiene:** after a `--prune` re-import, **zero** documents
   with a dot-prefixed path component exist in `documents`; prune also
   drops docs that no longer match the format filter.
7. **No regressions:** off-topic deflection (story 04) and grounded
   answers (story 03) still pass their E2E suites in isolation; the E2E
   mock keeps its own threshold calibration via the app fixture.
8. Unit + integration green, `app/` coverage >90%, story E2E green in
   isolation, one `--no-gpg-sign` commit.

## UI Visualization & Structure
No UI change. The Sources page document count drops after the junk prune
(intended cleanup — noted in the README).

## Playwright Mapping Rule
**Test Scenario → `tests/e2e/test_retrieval_quality.py`** (mock LLM, seeded KB):
1. `test_multiformat_docs_indexed` — fixture tree with `.md`, `.yaml`,
   `.py`, `.json`, `.txt` plus `.hidden/junk.md`: import in-process →
   `/api/docs` lists every non-hidden doc and none of the hidden ones.
2. `test_named_tool_question_retrieves_right_doc` — "How did I install
   gitlab?" → not deflected, streamed answer, a `.source-chip` containing
   the gitlab doc, `query_log` row `deflected=false` with the gitlab doc
   in sources.
3. `test_keyword_only_question_beats_vector_ranking` — a question whose
   rare token exists in exactly one fixture doc (weak mock-embedding
   overlap, e.g. "kafkabridge") → grounded answer citing that doc
   (FTS-OR gate + lexical RRF lift, end to end).
4. `test_off_topic_still_deflects` — "sourdough starter" →
   `.is-deflected` bubble matching /haven't done anything like that/i +
   ≥2 "Maybe try:" chips (story 04 behavior under the new gate).