# Phase 09 — Story: Retrieval Quality — Multi-Format Ingestion + Hybrid Search

**Story:** `.agent/user_stories/retrieval-quality.md`
**Context:** `.agent/PLAN.md` §3 (data flow), §5 (data model), §6 (retrieval), §11 (import)

## Goal
Fix "RAG retrieval is terrible": ingest the full text-format set (not
just `.md`), purge vendored-cache junk from the index, and replace
pure-cosine top-4 with hybrid (vector + Postgres FTS, RRF-fused)
retrieval so name-your-tool questions find the right document.

## Owner permission (recorded per phase protocol)
> "I'm giving you explicit permission to update the locked decisions and
> proceed with writing all 3 of these phases" — Reese, 2026-08-21.

This phase revises anchors **A9** (content scope: `*.md` only →
`md, markdown, txt, yaml, yml, json, py` + hidden-dir skip), **A7**
(pure-cosine top-4 → hybrid RRF retrieval; the whole-document context
contract is preserved), **A8** (gate: LOW only when best cosine <
threshold **and** zero FTS hits; threshold re-tuned 0.30 → 0.62 default).
`PLAN.md` anchors were updated 2026-08-21 under this permission.

## Evidence (measured 2026-08-21 against the live KB + `embed` model)
- "How did I install gitlab?": the best `gitlab.md` chunk ranks **7th**
  (cosine 0.804) — outside the top-4 window. Ranks 1–6: a vendored-cache
  README (`.esphome/.espressif/…/esp-tflite-micro/README.md`, 0.838) and
  generic templates (`project_readme_template.md`, `templates/…/foobar.
  md`, 0.81–0.82). The LLM therefore answered from junk docs and honestly
  reported "no notes on gitlab".
- Corpus cosine range: **0.41–0.84** — the old 0.30 gate never
  discriminated.
- FTS: `plainto_tsquery('english','gitlab')` matches **exactly**
  gitlab.md's 4 chunks and nothing else.
- ~470 of 672 indexed docs live under dot-prefixed path components
  (vendored caches) that A9's exclusion list doesn't cover.

## Dependencies
Phases 01–07 (02 importer, 03 retriever, 04 gate especially).
Independent of 08 (backend + E2E only). Phase 10 builds on the new
multi-format corpus.

## Implementation steps
1. **Config** (`app/config.py`): `import_extensions` (csv, default
   `md,markdown,txt,yaml,yml,json,py`; env `BOR_IMPORT_EXTENSIONS`),
   `hybrid_vector_candidates` (30, `BOR_HYBRID_VECTOR_CANDIDATES`),
   `hybrid_lexical_candidates` (30, `BOR_HYBRID_LEXICAL_CANDIDATES`),
   `rrf_k` (60, `BOR_RRF_K`), `relevance_threshold` default **0.62**
   (re-tuned; env override stays). In `tests/e2e/conftest.py`'s
   `app_server` fixture set `BOR_RELEVANCE_THRESHOLD=0.30` — the mock's
   token-overlap embeddings need their own calibration; this keeps
   stories 02–07's E2E suites green.
2. **Chunker** (`app/rag/chunker.py`): add a `chunk_document(content,
   path)` dispatcher by lowercased suffix + per-format functions —
   **stdlib only, no new dependencies**:
   - `yaml`/`yml`: split on `---` document separators and top-level keys
     (indent-0 `key:` lines); every chunk keeps its key line as anchor.
   - `json`: `json.dumps(obj, indent=2)` then split at top-level keys
     (track brace depth); unparseable JSON → paragraph packing.
   - `py`: stdlib `ast` top-level node line ranges → split at
     defs/classes; oversized functions fall back to line packing.
   - `txt`: paragraph packing (reuse `_paragraph_blocks`/`_pack_blocks`).
   - All formats honor `HARD_MAX_CHARS` (1200 — the aipi ~1024-token
     request cap) and the target/overlap settings; the `md` path stays
     byte-for-byte unchanged (existing chunker tests must stay green).
3. **Importer** (`app/rag/importer.py`, `scripts/import_docs.py`):
   extension filter (case-insensitive, config-driven); **skip any path
   containing a dot-prefixed component** (hidden dirs); chunker dispatch
   by suffix; `--prune` now also drops docs whose files **no longer
   match the filter** (this is how the ~470 junk docs leave the index);
   summary log gains per-format counts
   (`formats=md:203,yaml:267,…`).
4. **Migration `0002_hybrid_retrieval.py`** (alembic):
   - `chunks.tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english',
     content)) STORED` + `CREATE INDEX … USING gin (chunks.tsv)`.
   - `query_log.fts_hits INT` (nullable; pre-existing rows stay NULL).
5. **Retriever** (`app/rag/retriever.py`) — hybrid path:
   - `retrieve(db, question, question_embedding)`: vector top-N (cosine,
     as today) ∪ lexical top-N — `to_tsquery('english', <OR-joined
     stemmed tokens of the question>)` (skip pure-stopword/no-token
     questions → empty lexical list), ordered by `ts_rank` — fused with
     RRF: `score = Σ 1/(k + rank)` over the lists a chunk appears in
     (single-list chunks get one term; k from config).
   - `RetrievedChunk` gains `cosine` (for the gate) and `fts_hit: bool`
     alongside `score` (now the fused score, used for ranking);
     `select_documents` / `weak_hit_titles` keep working off `score`.
   - Deterministic tie-break: `(−fused, −cosine, document.path,
     chunk.position)`.
6. **Chat flow + gate** (`app/api/chat.py`): pass the raw question into
   `retrieve`; **LOW only when `best_cosine < threshold and fts_hits ==
   0`** (`fts_hits` = count of lexical candidates matched); per-turn log
   line gains `fts_hits=…` (PLAN §9); `query_log` row stores `fts_hits`.
7. **Eval script** `scripts/eval_retrieval.py`:
   `uv run python -m scripts.eval_retrieval "q1" "q2" …` (or
   `--from-file questions.txt`) — embeds via aipi, runs the hybrid
   search, prints top-5 docs per question with cosine/fts/fused scores +
   the gate verdict. Requires `AIPI_KEY` in the environment (same
   convention as `llm_probe.py`).
8. **Re-import the live KB** (one-time; expect ~15–40 min of embedding
   batches — the importer logs per file):
   `uv run python -m scripts.import_docs --prune`. Expect ~470
   hidden-dir docs pruned and ~500 docs indexed (md + new formats). Then
   verify with the eval script:
   - "How did I install gitlab?" → top doc `active/container_gitlab/
     gitlab.md` (the compose yaml should land in the top-2).
   - "How is my Kubernetes cluster set up?" → kubernetes docs.
   - "sourdough starter" → LOW (deflect).
   If the gitlab case isn't #1, iterate the **fusion** (k, candidate
   counts, token handling) — not the threshold — until it is, and record
   the final numbers in the phase report.
9. **E2E fixtures** (`tests/fixtures/docs/`): add
   `homelab/container_gitlab/gitlab.md` (H1 "Gitlab", docker install
   steps, "gitlab" repeated), `homelab/container_gitlab/
   gitlab-compose.yaml` (`services: gitlab: …`), a `.py` note, a `.json`
   note, a `.txt` note, and `.hidden/junk.md` (must never be imported).
   Follow the existing in-process seeding pattern from
   `tests/e2e/test_import_documents.py`.
10. **README**: import workflow section — supported formats, hidden-dir
    skip, `scripts/eval_retrieval.py`, threshold tuning; note that the
    Sources count drops after the prune (intended cleanup).

## Testing & Quality
- **Unit:** chunker per format (yaml top-level + `---` split, json
  top-level keys + pretty-print + unparseable fallback, py ast split +
  oversized-func fallback, txt paragraphs, dispatch, 1200-cap) with md
  output unchanged; importer (hidden-dir skip, extension filter,
  prune-when-filtered-out, per-format summary); retriever (RRF math:
  both-lists / one-list / tie-break; OR tsquery construction incl.
  no-token and stopword-only questions; gate: `cosine ≥ T` → HIGH;
  `cosine < T` + `fts>0` → HIGH; `cosine < T` + `fts=0` → LOW; boundary
  exactly `T` → HIGH).
- **Integration:** `/api/chat` hybrid against a seeded temp schema —
  keyword question grounded + `fts_hits` in `query_log`; off-topic
  deflected with `fts_hits=0`; migration up clean.
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` —
  **>90%** on `app/`.
- **No regressions:** existing story E2E suites (02–07) green in
  isolation after the change (the conftest threshold override is what
  keeps them green — verify each one).

## Playwright Execution Phase
Run ONLY this story's suite:

```bash
uv run pytest tests/e2e/test_retrieval_quality.py -v --no-cov
```

Implements the story mapping: multi-format fixture import (hidden doc
excluded, `/api/docs` counts); "How did I install gitlab?" → grounded,
not deflected, gitlab chip, `query_log` row; keyword-only question beats
vector ranking (FTS-OR gate end to end); "sourdough" → deflected bubble +
≥2 chips.

## Success criteria
- [ ] live eval: "How did I install gitlab?" → `gitlab.md` is the top doc
- [ ] zero dot-prefixed path components in `documents` after re-import
- [ ] off-topic still deflects; on-topic still grounds (new + existing E2E)
- [ ] unit + integration green, coverage >90%, ruff + pyright green
- [ ] README documents formats / hidden-dir skip / eval / tuning
- [ ] committed

## Commit
```bash
git add -A && git commit --no-gpg-sign -m "feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document"
```