feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document
This commit is contained in:
@@ -0,0 +1,164 @@
|
||||
# Phase 09 — Story: Retrieval Quality — Multi-Format Ingestion + Hybrid Search
|
||||
|
||||
**Story:** `.agent/user_stories/retrieval-quality.md`
|
||||
**Context:** `.agent/PLAN.md` §3 (data flow), §5 (data model), §6 (retrieval), §11 (import)
|
||||
|
||||
## Goal
|
||||
Fix "RAG retrieval is terrible": ingest the full text-format set (not
|
||||
just `.md`), purge vendored-cache junk from the index, and replace
|
||||
pure-cosine top-4 with hybrid (vector + Postgres FTS, RRF-fused)
|
||||
retrieval so name-your-tool questions find the right document.
|
||||
|
||||
## Owner permission (recorded per phase protocol)
|
||||
> "I'm giving you explicit permission to update the locked decisions and
|
||||
> proceed with writing all 3 of these phases" — Reese, 2026-08-21.
|
||||
|
||||
This phase revises anchors **A9** (content scope: `*.md` only →
|
||||
`md, markdown, txt, yaml, yml, json, py` + hidden-dir skip), **A7**
|
||||
(pure-cosine top-4 → hybrid RRF retrieval; the whole-document context
|
||||
contract is preserved), **A8** (gate: LOW only when best cosine <
|
||||
threshold **and** zero FTS hits; threshold re-tuned 0.30 → 0.62 default).
|
||||
`PLAN.md` anchors were updated 2026-08-21 under this permission.
|
||||
|
||||
## Evidence (measured 2026-08-21 against the live KB + `embed` model)
|
||||
- "How did I install gitlab?": the best `gitlab.md` chunk ranks **7th**
|
||||
(cosine 0.804) — outside the top-4 window. Ranks 1–6: a vendored-cache
|
||||
README (`.esphome/.espressif/…/esp-tflite-micro/README.md`, 0.838) and
|
||||
generic templates (`project_readme_template.md`, `templates/…/foobar.
|
||||
md`, 0.81–0.82). The LLM therefore answered from junk docs and honestly
|
||||
reported "no notes on gitlab".
|
||||
- Corpus cosine range: **0.41–0.84** — the old 0.30 gate never
|
||||
discriminated.
|
||||
- FTS: `plainto_tsquery('english','gitlab')` matches **exactly**
|
||||
gitlab.md's 4 chunks and nothing else.
|
||||
- ~470 of 672 indexed docs live under dot-prefixed path components
|
||||
(vendored caches) that A9's exclusion list doesn't cover.
|
||||
|
||||
## Dependencies
|
||||
Phases 01–07 (02 importer, 03 retriever, 04 gate especially).
|
||||
Independent of 08 (backend + E2E only). Phase 10 builds on the new
|
||||
multi-format corpus.
|
||||
|
||||
## Implementation steps
|
||||
1. **Config** (`app/config.py`): `import_extensions` (csv, default
|
||||
`md,markdown,txt,yaml,yml,json,py`; env `BOR_IMPORT_EXTENSIONS`),
|
||||
`hybrid_vector_candidates` (30, `BOR_HYBRID_VECTOR_CANDIDATES`),
|
||||
`hybrid_lexical_candidates` (30, `BOR_HYBRID_LEXICAL_CANDIDATES`),
|
||||
`rrf_k` (60, `BOR_RRF_K`), `relevance_threshold` default **0.62**
|
||||
(re-tuned; env override stays). In `tests/e2e/conftest.py`'s
|
||||
`app_server` fixture set `BOR_RELEVANCE_THRESHOLD=0.30` — the mock's
|
||||
token-overlap embeddings need their own calibration; this keeps
|
||||
stories 02–07's E2E suites green.
|
||||
2. **Chunker** (`app/rag/chunker.py`): add a `chunk_document(content,
|
||||
path)` dispatcher by lowercased suffix + per-format functions —
|
||||
**stdlib only, no new dependencies**:
|
||||
- `yaml`/`yml`: split on `---` document separators and top-level keys
|
||||
(indent-0 `key:` lines); every chunk keeps its key line as anchor.
|
||||
- `json`: `json.dumps(obj, indent=2)` then split at top-level keys
|
||||
(track brace depth); unparseable JSON → paragraph packing.
|
||||
- `py`: stdlib `ast` top-level node line ranges → split at
|
||||
defs/classes; oversized functions fall back to line packing.
|
||||
- `txt`: paragraph packing (reuse `_paragraph_blocks`/`_pack_blocks`).
|
||||
- All formats honor `HARD_MAX_CHARS` (1200 — the aipi ~1024-token
|
||||
request cap) and the target/overlap settings; the `md` path stays
|
||||
byte-for-byte unchanged (existing chunker tests must stay green).
|
||||
3. **Importer** (`app/rag/importer.py`, `scripts/import_docs.py`):
|
||||
extension filter (case-insensitive, config-driven); **skip any path
|
||||
containing a dot-prefixed component** (hidden dirs); chunker dispatch
|
||||
by suffix; `--prune` now also drops docs whose files **no longer
|
||||
match the filter** (this is how the ~470 junk docs leave the index);
|
||||
summary log gains per-format counts
|
||||
(`formats=md:203,yaml:267,…`).
|
||||
4. **Migration `0002_hybrid_retrieval.py`** (alembic):
|
||||
- `chunks.tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english',
|
||||
content)) STORED` + `CREATE INDEX … USING gin (chunks.tsv)`.
|
||||
- `query_log.fts_hits INT` (nullable; pre-existing rows stay NULL).
|
||||
5. **Retriever** (`app/rag/retriever.py`) — hybrid path:
|
||||
- `retrieve(db, question, question_embedding)`: vector top-N (cosine,
|
||||
as today) ∪ lexical top-N — `to_tsquery('english', <OR-joined
|
||||
stemmed tokens of the question>)` (skip pure-stopword/no-token
|
||||
questions → empty lexical list), ordered by `ts_rank` — fused with
|
||||
RRF: `score = Σ 1/(k + rank)` over the lists a chunk appears in
|
||||
(single-list chunks get one term; k from config).
|
||||
- `RetrievedChunk` gains `cosine` (for the gate) and `fts_hit: bool`
|
||||
alongside `score` (now the fused score, used for ranking);
|
||||
`select_documents` / `weak_hit_titles` keep working off `score`.
|
||||
- Deterministic tie-break: `(−fused, −cosine, document.path,
|
||||
chunk.position)`.
|
||||
6. **Chat flow + gate** (`app/api/chat.py`): pass the raw question into
|
||||
`retrieve`; **LOW only when `best_cosine < threshold and fts_hits ==
|
||||
0`** (`fts_hits` = count of lexical candidates matched); per-turn log
|
||||
line gains `fts_hits=…` (PLAN §9); `query_log` row stores `fts_hits`.
|
||||
7. **Eval script** `scripts/eval_retrieval.py`:
|
||||
`uv run python -m scripts.eval_retrieval "q1" "q2" …` (or
|
||||
`--from-file questions.txt`) — embeds via aipi, runs the hybrid
|
||||
search, prints top-5 docs per question with cosine/fts/fused scores +
|
||||
the gate verdict. Requires `AIPI_KEY` in the environment (same
|
||||
convention as `llm_probe.py`).
|
||||
8. **Re-import the live KB** (one-time; expect ~15–40 min of embedding
|
||||
batches — the importer logs per file):
|
||||
`uv run python -m scripts.import_docs --prune`. Expect ~470
|
||||
hidden-dir docs pruned and ~500 docs indexed (md + new formats). Then
|
||||
verify with the eval script:
|
||||
- "How did I install gitlab?" → top doc `active/container_gitlab/
|
||||
gitlab.md` (the compose yaml should land in the top-2).
|
||||
- "How is my Kubernetes cluster set up?" → kubernetes docs.
|
||||
- "sourdough starter" → LOW (deflect).
|
||||
If the gitlab case isn't #1, iterate the **fusion** (k, candidate
|
||||
counts, token handling) — not the threshold — until it is, and record
|
||||
the final numbers in the phase report.
|
||||
9. **E2E fixtures** (`tests/fixtures/docs/`): add
|
||||
`homelab/container_gitlab/gitlab.md` (H1 "Gitlab", docker install
|
||||
steps, "gitlab" repeated), `homelab/container_gitlab/
|
||||
gitlab-compose.yaml` (`services: gitlab: …`), a `.py` note, a `.json`
|
||||
note, a `.txt` note, and `.hidden/junk.md` (must never be imported).
|
||||
Follow the existing in-process seeding pattern from
|
||||
`tests/e2e/test_import_documents.py`.
|
||||
10. **README**: import workflow section — supported formats, hidden-dir
|
||||
skip, `scripts/eval_retrieval.py`, threshold tuning; note that the
|
||||
Sources count drops after the prune (intended cleanup).
|
||||
|
||||
## Testing & Quality
|
||||
- **Unit:** chunker per format (yaml top-level + `---` split, json
|
||||
top-level keys + pretty-print + unparseable fallback, py ast split +
|
||||
oversized-func fallback, txt paragraphs, dispatch, 1200-cap) with md
|
||||
output unchanged; importer (hidden-dir skip, extension filter,
|
||||
prune-when-filtered-out, per-format summary); retriever (RRF math:
|
||||
both-lists / one-list / tie-break; OR tsquery construction incl.
|
||||
no-token and stopword-only questions; gate: `cosine ≥ T` → HIGH;
|
||||
`cosine < T` + `fts>0` → HIGH; `cosine < T` + `fts=0` → LOW; boundary
|
||||
exactly `T` → HIGH).
|
||||
- **Integration:** `/api/chat` hybrid against a seeded temp schema —
|
||||
keyword question grounded + `fts_hits` in `query_log`; off-topic
|
||||
deflected with `fts_hits=0`; migration up clean.
|
||||
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` —
|
||||
**>90%** on `app/`.
|
||||
- **No regressions:** existing story E2E suites (02–07) green in
|
||||
isolation after the change (the conftest threshold override is what
|
||||
keeps them green — verify each one).
|
||||
|
||||
## Playwright Execution Phase
|
||||
Run ONLY this story's suite:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/e2e/test_retrieval_quality.py -v --no-cov
|
||||
```
|
||||
|
||||
Implements the story mapping: multi-format fixture import (hidden doc
|
||||
excluded, `/api/docs` counts); "How did I install gitlab?" → grounded,
|
||||
not deflected, gitlab chip, `query_log` row; keyword-only question beats
|
||||
vector ranking (FTS-OR gate end to end); "sourdough" → deflected bubble +
|
||||
≥2 chips.
|
||||
|
||||
## Success criteria
|
||||
- [ ] live eval: "How did I install gitlab?" → `gitlab.md` is the top doc
|
||||
- [ ] zero dot-prefixed path components in `documents` after re-import
|
||||
- [ ] off-topic still deflects; on-topic still grounds (new + existing E2E)
|
||||
- [ ] unit + integration green, coverage >90%, ruff + pyright green
|
||||
- [ ] README documents formats / hidden-dir skip / eval / tuning
|
||||
- [ ] committed
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A && git commit --no-gpg-sign -m "feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document"
|
||||
```
|
||||
Reference in New Issue
Block a user