chore(agent): track .agent/ planning tree in git

Remove the blanket .agent/ gitignore so the phase roadmap, user
stories, reports, and PLAN.md are versioned with the code. Only
runtime artifacts (.agent/phase-sessions/, .agent/pipeline.log)
remain ignored. Update AGENTS.md git protocol rule to match.

.agent/user_stories/retrieval-quality.md

@@ -0,0 +1,78 @@
+# Story: Retrieval Quality — Multi-Format Ingestion + Hybrid Search
+
+**Phase:** `09_story_retrieval_quality.md` · **E2E:** `tests/e2e/test_retrieval_quality.py`
+
+## Narrative
+
+As **a user**, I want to ask questions that name my tools ("How did I
+install gitlab?") and get a grounded answer citing the right document —
+even when the answer lives in a `.yaml`, `.py`, `.json` or `.txt` file —
+so Brain actually covers my notes instead of deflecting around them.
+
+- **Given** my notes are a mix of markdown write-ups and config/code
+  files, and today's index is polluted with vendored-cache docs
+  (`.esphome/.espressif/**`, merge-request templates, …) that outrank
+  real content, while retrieval is pure cosine top-4
+- **When** the importer covers the full text-format set, skips hidden
+  directories, and retrieval fuses vector similarity with Postgres
+  full-text search
+- **Then** "How did I install gitlab?" surfaces `active/container_gitlab/
+  gitlab.md` as the top document, the junk docs are pruned from the
+  index, and genuinely off-topic questions still get the honest
+  deflection.
+
+## Acceptance criteria
+1. **Ingestion scope (A9, revised with owner permission 2026-08-21):**
+   default extensions `md, markdown, txt, yaml, yml, json, py`,
+   overridable via `BOR_IMPORT_EXTENSIONS`; **hidden (dot) directories
+   skipped by default**; the A9 exclusion list stays.
+2. **Format-aware chunking:** md unchanged (heading-aware); yaml/yml
+   split on top-level keys and `---` document separators (key kept as
+   chunk anchor); json pretty-printed and split on top-level keys; py
+   split on top-level defs/classes (stdlib `ast`); txt on paragraphs.
+   Every format honors the 1200-char hard cap (aipi ~1024-token request
+   cap).
+3. **Hybrid retrieval (A7, revised with owner permission 2026-08-21):**
+   cosine top-30 + FTS top-30 (OR tsquery over the question's tokens,
+   `ts_rank`-ordered) fused with Reciprocal Rank Fusion (k=60);
+   document-level ranking by best fused chunk score; the LLM still
+   receives the **full top-2 documents** capped at 24k chars (contract
+   preserved).
+4. **Honesty gate (A8, revised with owner permission 2026-08-21):**
+   deflect when `best cosine < BOR_RELEVANCE_THRESHOLD` **and** no
+   candidate chunk FTS-matches the question; threshold re-tuned for the
+   `embed` model's measured 0.41–0.84 score compression (default 0.62,
+   documented tuning procedure); `fts_hits` logged per turn and stored in
+   `query_log`.
+5. **Live verification:** `scripts/eval_retrieval.py` run against the
+   imported KB shows `gitlab.md` as the top document for "How did I
+   install gitlab?"; a known off-topic question gates LOW.
+6. **Index hygiene:** after a `--prune` re-import, **zero** documents
+   with a dot-prefixed path component exist in `documents`; prune also
+   drops docs that no longer match the format filter.
+7. **No regressions:** off-topic deflection (story 04) and grounded
+   answers (story 03) still pass their E2E suites in isolation; the E2E
+   mock keeps its own threshold calibration via the app fixture.
+8. Unit + integration green, `app/` coverage >90%, story E2E green in
+   isolation, one `--no-gpg-sign` commit.
+
+## UI Visualization & Structure
+No UI change. The Sources page document count drops after the junk prune
+(intended cleanup — noted in the README).
+
+## Playwright Mapping Rule
+**Test Scenario → `tests/e2e/test_retrieval_quality.py`** (mock LLM, seeded KB):
+1. `test_multiformat_docs_indexed` — fixture tree with `.md`, `.yaml`,
+   `.py`, `.json`, `.txt` plus `.hidden/junk.md`: import in-process →
+   `/api/docs` lists every non-hidden doc and none of the hidden ones.
+2. `test_named_tool_question_retrieves_right_doc` — "How did I install
+   gitlab?" → not deflected, streamed answer, a `.source-chip` containing
+   the gitlab doc, `query_log` row `deflected=false` with the gitlab doc
+   in sources.
+3. `test_keyword_only_question_beats_vector_ranking` — a question whose
+   rare token exists in exactly one fixture doc (weak mock-embedding
+   overlap, e.g. "kafkabridge") → grounded answer citing that doc
+   (FTS-OR gate + lexical RRF lift, end to end).
+4. `test_off_topic_still_deflects` — "sourdough starter" →
+   `.is-deflected` bubble matching /haven't done anything like that/i +
+   ≥2 "Maybe try:" chips (story 04 behavior under the new gate).