brain-of-reese/.agent/phases/complete/09_story_retrieval_quality.md

Phase 09 — Story: Retrieval Quality — Multi-Format Ingestion + Hybrid Search

Story: .agent/user_stories/retrieval-quality.md Context: .agent/PLAN.md §3 (data flow), §5 (data model), §6 (retrieval), §11 (import)

Goal

Fix "RAG retrieval is terrible": ingest the full text-format set (not just .md), purge vendored-cache junk from the index, and replace pure-cosine top-4 with hybrid (vector + Postgres FTS, RRF-fused) retrieval so name-your-tool questions find the right document.

Owner permission (recorded per phase protocol)

"I'm giving you explicit permission to update the locked decisions and proceed with writing all 3 of these phases" — Reese, 2026-08-21.

This phase revises anchors A9 (content scope: *.md only → md, markdown, txt, yaml, yml, json, py + hidden-dir skip), A7 (pure-cosine top-4 → hybrid RRF retrieval; the whole-document context contract is preserved), A8 (gate: LOW only when best cosine < threshold and zero FTS hits; threshold re-tuned 0.30 → 0.62 default). PLAN.md anchors were updated 2026-08-21 under this permission.

Evidence (measured 2026-08-21 against the live KB + embed model)

• "How did I install gitlab?": the best gitlab.md chunk ranks 7th (cosine 0.804) — outside the top-4 window. Ranks 1–6: a vendored-cache README (.esphome/.espressif/…/esp-tflite-micro/README.md, 0.838) and generic templates (project_readme_template.md, templates/…/foobar. md, 0.81–0.82). The LLM therefore answered from junk docs and honestly reported "no notes on gitlab".
• Corpus cosine range: 0.41–0.84 — the old 0.30 gate never discriminated.
• FTS: plainto_tsquery('english','gitlab') matches exactly gitlab.md's 4 chunks and nothing else.
• ~470 of 672 indexed docs live under dot-prefixed path components (vendored caches) that A9's exclusion list doesn't cover.

Dependencies

Phases 01–07 (02 importer, 03 retriever, 04 gate especially). Independent of 08 (backend + E2E only). Phase 10 builds on the new multi-format corpus.

Implementation steps

1. Config (app/config.py): import_extensions (csv, default md,markdown,txt,yaml,yml,json,py; env BOR_IMPORT_EXTENSIONS), hybrid_vector_candidates (30, BOR_HYBRID_VECTOR_CANDIDATES), hybrid_lexical_candidates (30, BOR_HYBRID_LEXICAL_CANDIDATES), rrf_k (60, BOR_RRF_K), relevance_threshold default 0.62 (re-tuned; env override stays). In tests/e2e/conftest.py's app_server fixture set BOR_RELEVANCE_THRESHOLD=0.30 — the mock's token-overlap embeddings need their own calibration; this keeps stories 02–07's E2E suites green.
2. Chunker (app/rag/chunker.py): add a chunk_document(content, path) dispatcher by lowercased suffix + per-format functions — stdlib only, no new dependencies:
   • yaml/yml: split on --- document separators and top-level keys (indent-0 key: lines); every chunk keeps its key line as anchor.
   • json: json.dumps(obj, indent=2) then split at top-level keys (track brace depth); unparseable JSON → paragraph packing.
   • py: stdlib ast top-level node line ranges → split at defs/classes; oversized functions fall back to line packing.
   • txt: paragraph packing (reuse _paragraph_blocks/_pack_blocks).
   • All formats honor HARD_MAX_CHARS (1200 — the aipi ~1024-token request cap) and the target/overlap settings; the md path stays byte-for-byte unchanged (existing chunker tests must stay green).
3. Importer (app/rag/importer.py, scripts/import_docs.py): extension filter (case-insensitive, config-driven); skip any path containing a dot-prefixed component (hidden dirs); chunker dispatch by suffix; --prune now also drops docs whose files no longer match the filter (this is how the ~470 junk docs leave the index); summary log gains per-format counts (formats=md:203,yaml:267,…).
4. Migration 0002_hybrid_retrieval.py (alembic):
   • chunks.tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english', content)) STORED + CREATE INDEX … USING gin (chunks.tsv).
   • query_log.fts_hits INT (nullable; pre-existing rows stay NULL).
5. Retriever (app/rag/retriever.py) — hybrid path:
   • retrieve(db, question, question_embedding): vector top-N (cosine, as today) ∪ lexical top-N — to_tsquery('english', <OR-joined stemmed tokens of the question>) (skip pure-stopword/no-token questions → empty lexical list), ordered by ts_rank — fused with RRF: score = Σ 1/(k + rank) over the lists a chunk appears in (single-list chunks get one term; k from config).
   • RetrievedChunk gains cosine (for the gate) and fts_hit: bool alongside score (now the fused score, used for ranking); select_documents / weak_hit_titles keep working off score.
   • Deterministic tie-break: (−fused, −cosine, document.path, chunk.position).
6. Chat flow + gate (app/api/chat.py): pass the raw question into retrieve; LOW only when best_cosine < threshold and fts_hits == 0 (fts_hits = count of lexical candidates matched); per-turn log line gains fts_hits=… (PLAN §9); query_log row stores fts_hits.
7. Eval script scripts/eval_retrieval.py: uv run python -m scripts.eval_retrieval "q1" "q2" … (or --from-file questions.txt) — embeds via aipi, runs the hybrid search, prints top-5 docs per question with cosine/fts/fused scores + the gate verdict. Requires AIPI_KEY in the environment (same convention as llm_probe.py).
8. Re-import the live KB (one-time; expect ~15–40 min of embedding batches — the importer logs per file): uv run python -m scripts.import_docs --prune. Expect ~470 hidden-dir docs pruned and ~500 docs indexed (md + new formats). Then verify with the eval script:
   • "How did I install gitlab?" → top doc active/container_gitlab/ gitlab.md (the compose yaml should land in the top-2).
   • "How is my Kubernetes cluster set up?" → kubernetes docs.
   • "sourdough starter" → LOW (deflect). If the gitlab case isn't #1, iterate the fusion (k, candidate counts, token handling) — not the threshold — until it is, and record the final numbers in the phase report.
9. E2E fixtures (tests/fixtures/docs/): add homelab/container_gitlab/gitlab.md (H1 "Gitlab", docker install steps, "gitlab" repeated), homelab/container_gitlab/ gitlab-compose.yaml (services: gitlab: …), a .py note, a .json note, a .txt note, and .hidden/junk.md (must never be imported). Follow the existing in-process seeding pattern from tests/e2e/test_import_documents.py.
10. README: import workflow section — supported formats, hidden-dir skip, scripts/eval_retrieval.py, threshold tuning; note that the Sources count drops after the prune (intended cleanup).

Testing & Quality

• Unit: chunker per format (yaml top-level + --- split, json top-level keys + pretty-print + unparseable fallback, py ast split + oversized-func fallback, txt paragraphs, dispatch, 1200-cap) with md output unchanged; importer (hidden-dir skip, extension filter, prune-when-filtered-out, per-format summary); retriever (RRF math: both-lists / one-list / tie-break; OR tsquery construction incl. no-token and stopword-only questions; gate: cosine ≥ T → HIGH; cosine < T + fts>0 → HIGH; cosine < T + fts=0 → LOW; boundary exactly T → HIGH).
• Integration: /api/chat hybrid against a seeded temp schema — keyword question grounded + fts_hits in query_log; off-topic deflected with fts_hits=0; migration up clean.
• Coverage: uv run pytest --cov=app --cov-report=term-missing — >90% on app/.
• No regressions: existing story E2E suites (02–07) green in isolation after the change (the conftest threshold override is what keeps them green — verify each one).

Playwright Execution Phase

Run ONLY this story's suite:

uv run pytest tests/e2e/test_retrieval_quality.py -v --no-cov

Implements the story mapping: multi-format fixture import (hidden doc excluded, /api/docs counts); "How did I install gitlab?" → grounded, not deflected, gitlab chip, query_log row; keyword-only question beats vector ranking (FTS-OR gate end to end); "sourdough" → deflected bubble + ≥2 chips.

Success criteria

• live eval: "How did I install gitlab?" → gitlab.md is the top doc
• zero dot-prefixed path components in documents after re-import
• off-topic still deflects; on-topic still grounds (new + existing E2E)
• unit + integration green, coverage >90%, ruff + pyright green
• README documents formats / hidden-dir skip / eval / tuning
• committed

Commit

git add -A && git commit --no-gpg-sign -m "feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document"