feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document
This commit is contained in:
@@ -1,10 +1,12 @@
|
||||
# 🧠 Brain of Reese
|
||||
|
||||
A chippy, honest **RAG chatbot** over the `~/Homelab` and `~/Deployments`
|
||||
projects. Point it at your markdown docs, ask it anything — it retrieves
|
||||
the relevant notes with **Postgres 17 + pgvector** cosine search, feeds the
|
||||
**whole relevant document** to a **self-hosted LLM** (`turbo` via
|
||||
`https://aipi.reeseapps.com/v1`), and streams a grounded answer back.
|
||||
projects. Point it at your notes — markdown, YAML, JSON, Python, plain
|
||||
text — ask it anything, and it retrieves the relevant chunks with
|
||||
**hybrid search** (pgvector cosine ∪ Postgres full-text search, fused with
|
||||
Reciprocal Rank Fusion), feeds the **whole relevant document** to a
|
||||
**self-hosted LLM** (`turbo` via `https://aipi.reeseapps.com/v1`), and
|
||||
streams a grounded answer back.
|
||||
|
||||
If it doesn't have notes for your question, it admits it:
|
||||
*"I haven't done anything like that"* — plus suggestions for what it **does** know.
|
||||
@@ -77,9 +79,9 @@ uv run uvicorn app.main:app --reload
|
||||
file), so a refresh after a normal editing session takes seconds:
|
||||
|
||||
```bash
|
||||
# After editing/adding/removing markdown in your projects:
|
||||
# After editing/adding/removing notes in your projects:
|
||||
uv run python -m scripts.import_docs # re-index what changed
|
||||
uv run python -m scripts.import_docs --prune # also drop deleted files
|
||||
uv run python -m scripts.import_docs --prune # also drop deleted/out-of-scope files
|
||||
|
||||
# Point it at extra directories (repeatable):
|
||||
uv run python -m scripts.import_docs --source ~/SomeOtherDocs
|
||||
@@ -92,16 +94,61 @@ embedded.
|
||||
|
||||
- The import prints one line per file (`import: added|updated|unchanged|
|
||||
pruned …`) and ends with a greppable summary (`import: summary files=…
|
||||
added=… updated=… unchanged=… pruned=… chunks=… embed_batches=…`), so it
|
||||
is safe to run from a cron job or after every commit.
|
||||
- Only **`*.md`** files are indexed. Directories like `.venv`,
|
||||
`node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`
|
||||
are skipped (see `.agent/PLAN.md` anchor A9).
|
||||
added=… updated=… unchanged=… pruned=… chunks=… embed_batches=…
|
||||
formats=md:203,yaml:267,…`), so it is safe to run from a cron job or
|
||||
after every commit.
|
||||
- Indexed formats (A9): **`md, markdown, txt, yaml, yml, json, py`**
|
||||
(case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`). Any path with a
|
||||
**dot-prefixed component** — hidden files or vendored caches like
|
||||
`.esphome/.espressif/**` — is skipped, along with `.venv`,
|
||||
`node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`.
|
||||
`--prune` also drops documents whose files no longer match the filter —
|
||||
that's how previously imported junk leaves the index.
|
||||
- Non-markdown files get format-aware chunking (YAML top-level keys /
|
||||
`---` docs, JSON top-level keys, Python top-level defs/classes via
|
||||
stdlib `ast`) and their title comes from the file stem.
|
||||
- Unchanged files are **not re-embedded** — only new/changed ones, so
|
||||
refreshes are cheap.
|
||||
- To sanity-check the LLM backend (models + embedding dimension) after any
|
||||
aipi change: `uv run python -m scripts.llm_probe`.
|
||||
|
||||
## Checking retrieval quality
|
||||
|
||||
Ask the *real* pipeline (live aipi embeddings + the current KB) whether a
|
||||
question lands on the right document, with the gate verdict and per-document
|
||||
cosine / FTS / fused scores:
|
||||
|
||||
```bash
|
||||
uv run python -m scripts.eval_retrieval "How did I install gitlab?"
|
||||
uv run python -m scripts.eval_retrieval --from-file questions.txt --top 8
|
||||
```
|
||||
|
||||
Requires `AIPI_KEY` in the environment (same convention as
|
||||
`scripts/llm_probe.py`) and an imported knowledge base.
|
||||
|
||||
## How retrieval works (hybrid)
|
||||
|
||||
Every question is embedded and also lexically tokenized (OR-joined, English
|
||||
stemming) and searched **twice** against Postgres:
|
||||
|
||||
1. **Vector** — pgvector cosine top-N (default `BOR_HYBRID_VECTOR_CANDIDATES=100`)
|
||||
2. **Lexical** — a stored `tsvector` (GIN-indexed) matched with `to_tsquery`,
|
||||
top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`)
|
||||
|
||||
The two ranked lists are fused with **Reciprocal Rank Fusion**
|
||||
(`score = Σ 1/(k + rank)`, `BOR_RRF_K=60`) — a chunk in both lists scores
|
||||
nearly double, which is what lets a name-your-tool question ("gitlab") find
|
||||
its own document even when the question embeds close to generic templates.
|
||||
|
||||
The **honesty gate** (A8) then answers (HIGH) when the best cosine is ≥
|
||||
`BOR_RELEVANCE_THRESHOLD` (default `0.62`) **or** at least one chunk matched
|
||||
lexically (`fts_hits > 0`) — it deflects (LOW) only when *both* signals are
|
||||
absent. The top `BOR_TOP_N_DOCS` full documents are still what the LLM sees.
|
||||
|
||||
`query_log` records every turn (`top_score` = best cosine, `fts_hits`,
|
||||
`chunk_hits`, `deflected`, `sources`, `latency_ms`) — the raw material for
|
||||
tuning: `psql … -c 'SELECT question, top_score, fts_hits, deflected FROM query_log ORDER BY created_at DESC LIMIT 20'`.
|
||||
|
||||
## Debugging
|
||||
|
||||
`debugpy` is **off by default** and *never imported* unless you opt in —
|
||||
@@ -203,9 +250,12 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
|
||||
| `BOR_LLM_CHAT_MODEL` | `turbo` | chat model |
|
||||
| `BOR_LLM_EMBED_MODEL` | `embed` | embedding model |
|
||||
| `BOR_EMBEDDING_DIM` | `768` | vector dimension (fixed at table creation) |
|
||||
| `BOR_TOP_K_CHUNKS` | `4` | chunks retrieved per question |
|
||||
| `BOR_TOP_N_DOCS` | `2` | full documents fed to the LLM |
|
||||
| `BOR_RELEVANCE_THRESHOLD` | `0.30` | best cosine similarity required to answer; below ⇒ honest deflection |
|
||||
| `BOR_RELEVANCE_THRESHOLD` | `0.62` | answer when best cosine ≥ this **or** an FTS hit; below + no FTS ⇒ honest deflection |
|
||||
| `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | cosine list width for the RRF fusion |
|
||||
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
|
||||
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
|
||||
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
|
||||
| `BOR_MAX_CONTEXT_CHARS` | `24000` | cap on total document text sent to the LLM |
|
||||
| `BOR_SUGGESTIONS` | built-in list | JSON list of onboarding chips |
|
||||
| `DEBUGPY` | `0` | `1` ⇒ attach-on-demand debugpy on `DEBUGPY_PORT` (default 5678) |
|
||||
@@ -227,23 +277,26 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
|
||||
drop + recreate the chunks table (new migration or manual `TRUNCATE
|
||||
chunks, documents`).
|
||||
- **Honest deflection (the amber “I haven't done anything like that”
|
||||
bubble)** — every question passes the honesty gate: when the best
|
||||
cosine similarity is below `BOR_RELEVANCE_THRESHOLD` (default `0.30`),
|
||||
Brain switches to deflection mode instead of guessing. The LLM prompt
|
||||
then carries weak-hit *titles only* (no document content), the reply
|
||||
opens with “I haven't done anything like that”, the bubble renders
|
||||
amber with “Maybe try” chips derived from the closest indexed titles,
|
||||
the SSE `done` event carries `deflected: true` + `suggestions[]`, and
|
||||
the `query_log` row records `deflected=true` + the weak `top_score`.
|
||||
This is a feature, not a bug — the KB simply has no notes that close;
|
||||
the chips always point at topics Brain really covers.
|
||||
bubble)** — every question passes the honesty gate: deflection happens
|
||||
only when the best cosine similarity is below
|
||||
`BOR_RELEVANCE_THRESHOLD` (default `0.62`) **and** no chunk matched the
|
||||
question lexically (`fts_hits = 0`). A weak cosine with a lexical hit
|
||||
(name-your-tool questions) still gets a grounded answer. When it does
|
||||
deflect, the LLM prompt carries weak-hit *titles only* (no document
|
||||
content), the reply opens with “I haven't done anything like that”, the
|
||||
bubble renders amber with “Maybe try” chips derived from the closest
|
||||
indexed titles, the SSE `done` event carries `deflected: true` +
|
||||
`suggestions[]`, and the `query_log` row records `deflected=true` + the
|
||||
weak `top_score` + `fts_hits`. This is a feature, not a bug — the KB
|
||||
simply has no notes that close; the chips always point at topics Brain
|
||||
really covers.
|
||||
- **Answers deflect too often / too rarely** — tune
|
||||
`BOR_RELEVANCE_THRESHOLD` (lower = answers more, higher = more honest
|
||||
deflection): `0.0` ⇒ every question gets answered, even unknown topics
|
||||
(expect confident-sounding guesses); `1.0` ⇒ everything deflects
|
||||
(nothing but a perfect 1.0 score counts as relevant). After changing
|
||||
it, check the real scores:
|
||||
`psql … -c 'SELECT question, top_score, deflected FROM query_log ORDER BY created_at DESC LIMIT 20'`
|
||||
deflection): `0.0` ⇒ the gate leans entirely on FTS hits; `1.0` ⇒
|
||||
everything deflects unless a chunk matches lexically. The `embed` model's
|
||||
cosines cluster in a ~0.6–0.85 band on the live KB, so the default is
|
||||
`0.62`; after changing it, check the real scores:
|
||||
`psql … -c 'SELECT question, top_score, fts_hits, deflected FROM query_log ORDER BY created_at DESC LIMIT 20'`
|
||||
- **KB offline banner in the chat** — Postgres isn't running:
|
||||
`podman compose up -d db`.
|
||||
- **Stuck "Thinking…"** — the LLM is slow or down; a 120s client timeout
|
||||
|
||||
Reference in New Issue
Block a user