feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document

.agent/phases/complete/09_story_retrieval_quality.md

@@ -0,0 +1,164 @@
+# Phase 09 — Story: Retrieval Quality — Multi-Format Ingestion + Hybrid Search
+
+**Story:** `.agent/user_stories/retrieval-quality.md`
+**Context:** `.agent/PLAN.md` §3 (data flow), §5 (data model), §6 (retrieval), §11 (import)
+
+## Goal
+Fix "RAG retrieval is terrible": ingest the full text-format set (not
+just `.md`), purge vendored-cache junk from the index, and replace
+pure-cosine top-4 with hybrid (vector + Postgres FTS, RRF-fused)
+retrieval so name-your-tool questions find the right document.
+
+## Owner permission (recorded per phase protocol)
+> "I'm giving you explicit permission to update the locked decisions and
+> proceed with writing all 3 of these phases" — Reese, 2026-08-21.
+
+This phase revises anchors **A9** (content scope: `*.md` only →
+`md, markdown, txt, yaml, yml, json, py` + hidden-dir skip), **A7**
+(pure-cosine top-4 → hybrid RRF retrieval; the whole-document context
+contract is preserved), **A8** (gate: LOW only when best cosine <
+threshold **and** zero FTS hits; threshold re-tuned 0.30 → 0.62 default).
+`PLAN.md` anchors were updated 2026-08-21 under this permission.
+
+## Evidence (measured 2026-08-21 against the live KB + `embed` model)
+- "How did I install gitlab?": the best `gitlab.md` chunk ranks **7th**
+  (cosine 0.804) — outside the top-4 window. Ranks 1–6: a vendored-cache
+  README (`.esphome/.espressif/…/esp-tflite-micro/README.md`, 0.838) and
+  generic templates (`project_readme_template.md`, `templates/…/foobar.
+  md`, 0.81–0.82). The LLM therefore answered from junk docs and honestly
+  reported "no notes on gitlab".
+- Corpus cosine range: **0.41–0.84** — the old 0.30 gate never
+  discriminated.
+- FTS: `plainto_tsquery('english','gitlab')` matches **exactly**
+  gitlab.md's 4 chunks and nothing else.
+- ~470 of 672 indexed docs live under dot-prefixed path components
+  (vendored caches) that A9's exclusion list doesn't cover.
+
+## Dependencies
+Phases 01–07 (02 importer, 03 retriever, 04 gate especially).
+Independent of 08 (backend + E2E only). Phase 10 builds on the new
+multi-format corpus.
+
+## Implementation steps
+1. **Config** (`app/config.py`): `import_extensions` (csv, default
+   `md,markdown,txt,yaml,yml,json,py`; env `BOR_IMPORT_EXTENSIONS`),
+   `hybrid_vector_candidates` (30, `BOR_HYBRID_VECTOR_CANDIDATES`),
+   `hybrid_lexical_candidates` (30, `BOR_HYBRID_LEXICAL_CANDIDATES`),
+   `rrf_k` (60, `BOR_RRF_K`), `relevance_threshold` default **0.62**
+   (re-tuned; env override stays). In `tests/e2e/conftest.py`'s
+   `app_server` fixture set `BOR_RELEVANCE_THRESHOLD=0.30` — the mock's
+   token-overlap embeddings need their own calibration; this keeps
+   stories 02–07's E2E suites green.
+2. **Chunker** (`app/rag/chunker.py`): add a `chunk_document(content,
+   path)` dispatcher by lowercased suffix + per-format functions —
+   **stdlib only, no new dependencies**:
+   - `yaml`/`yml`: split on `---` document separators and top-level keys
+     (indent-0 `key:` lines); every chunk keeps its key line as anchor.
+   - `json`: `json.dumps(obj, indent=2)` then split at top-level keys
+     (track brace depth); unparseable JSON → paragraph packing.
+   - `py`: stdlib `ast` top-level node line ranges → split at
+     defs/classes; oversized functions fall back to line packing.
+   - `txt`: paragraph packing (reuse `_paragraph_blocks`/`_pack_blocks`).
+   - All formats honor `HARD_MAX_CHARS` (1200 — the aipi ~1024-token
+     request cap) and the target/overlap settings; the `md` path stays
+     byte-for-byte unchanged (existing chunker tests must stay green).
+3. **Importer** (`app/rag/importer.py`, `scripts/import_docs.py`):
+   extension filter (case-insensitive, config-driven); **skip any path
+   containing a dot-prefixed component** (hidden dirs); chunker dispatch
+   by suffix; `--prune` now also drops docs whose files **no longer
+   match the filter** (this is how the ~470 junk docs leave the index);
+   summary log gains per-format counts
+   (`formats=md:203,yaml:267,…`).
+4. **Migration `0002_hybrid_retrieval.py`** (alembic):
+   - `chunks.tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english',
+     content)) STORED` + `CREATE INDEX … USING gin (chunks.tsv)`.
+   - `query_log.fts_hits INT` (nullable; pre-existing rows stay NULL).
+5. **Retriever** (`app/rag/retriever.py`) — hybrid path:
+   - `retrieve(db, question, question_embedding)`: vector top-N (cosine,
+     as today) ∪ lexical top-N — `to_tsquery('english', <OR-joined
+     stemmed tokens of the question>)` (skip pure-stopword/no-token
+     questions → empty lexical list), ordered by `ts_rank` — fused with
+     RRF: `score = Σ 1/(k + rank)` over the lists a chunk appears in
+     (single-list chunks get one term; k from config).
+   - `RetrievedChunk` gains `cosine` (for the gate) and `fts_hit: bool`
+     alongside `score` (now the fused score, used for ranking);
+     `select_documents` / `weak_hit_titles` keep working off `score`.
+   - Deterministic tie-break: `(−fused, −cosine, document.path,
+     chunk.position)`.
+6. **Chat flow + gate** (`app/api/chat.py`): pass the raw question into
+   `retrieve`; **LOW only when `best_cosine < threshold and fts_hits ==
+   0`** (`fts_hits` = count of lexical candidates matched); per-turn log
+   line gains `fts_hits=…` (PLAN §9); `query_log` row stores `fts_hits`.
+7. **Eval script** `scripts/eval_retrieval.py`:
+   `uv run python -m scripts.eval_retrieval "q1" "q2" …` (or
+   `--from-file questions.txt`) — embeds via aipi, runs the hybrid
+   search, prints top-5 docs per question with cosine/fts/fused scores +
+   the gate verdict. Requires `AIPI_KEY` in the environment (same
+   convention as `llm_probe.py`).
+8. **Re-import the live KB** (one-time; expect ~15–40 min of embedding
+   batches — the importer logs per file):
+   `uv run python -m scripts.import_docs --prune`. Expect ~470
+   hidden-dir docs pruned and ~500 docs indexed (md + new formats). Then
+   verify with the eval script:
+   - "How did I install gitlab?" → top doc `active/container_gitlab/
+     gitlab.md` (the compose yaml should land in the top-2).
+   - "How is my Kubernetes cluster set up?" → kubernetes docs.
+   - "sourdough starter" → LOW (deflect).
+   If the gitlab case isn't #1, iterate the **fusion** (k, candidate
+   counts, token handling) — not the threshold — until it is, and record
+   the final numbers in the phase report.
+9. **E2E fixtures** (`tests/fixtures/docs/`): add
+   `homelab/container_gitlab/gitlab.md` (H1 "Gitlab", docker install
+   steps, "gitlab" repeated), `homelab/container_gitlab/
+   gitlab-compose.yaml` (`services: gitlab: …`), a `.py` note, a `.json`
+   note, a `.txt` note, and `.hidden/junk.md` (must never be imported).
+   Follow the existing in-process seeding pattern from
+   `tests/e2e/test_import_documents.py`.
+10. **README**: import workflow section — supported formats, hidden-dir
+    skip, `scripts/eval_retrieval.py`, threshold tuning; note that the
+    Sources count drops after the prune (intended cleanup).
+
+## Testing & Quality
+- **Unit:** chunker per format (yaml top-level + `---` split, json
+  top-level keys + pretty-print + unparseable fallback, py ast split +
+  oversized-func fallback, txt paragraphs, dispatch, 1200-cap) with md
+  output unchanged; importer (hidden-dir skip, extension filter,
+  prune-when-filtered-out, per-format summary); retriever (RRF math:
+  both-lists / one-list / tie-break; OR tsquery construction incl.
+  no-token and stopword-only questions; gate: `cosine ≥ T` → HIGH;
+  `cosine < T` + `fts>0` → HIGH; `cosine < T` + `fts=0` → LOW; boundary
+  exactly `T` → HIGH).
+- **Integration:** `/api/chat` hybrid against a seeded temp schema —
+  keyword question grounded + `fts_hits` in `query_log`; off-topic
+  deflected with `fts_hits=0`; migration up clean.
+- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` —
+  **>90%** on `app/`.
+- **No regressions:** existing story E2E suites (02–07) green in
+  isolation after the change (the conftest threshold override is what
+  keeps them green — verify each one).
+
+## Playwright Execution Phase
+Run ONLY this story's suite:
+
+```bash
+uv run pytest tests/e2e/test_retrieval_quality.py -v --no-cov
+```
+
+Implements the story mapping: multi-format fixture import (hidden doc
+excluded, `/api/docs` counts); "How did I install gitlab?" → grounded,
+not deflected, gitlab chip, `query_log` row; keyword-only question beats
+vector ranking (FTS-OR gate end to end); "sourdough" → deflected bubble +
+≥2 chips.
+
+## Success criteria
+- [ ] live eval: "How did I install gitlab?" → `gitlab.md` is the top doc
+- [ ] zero dot-prefixed path components in `documents` after re-import
+- [ ] off-topic still deflects; on-topic still grounds (new + existing E2E)
+- [ ] unit + integration green, coverage >90%, ruff + pyright green
+- [ ] README documents formats / hidden-dir skip / eval / tuning
+- [ ] committed
+
+## Commit
+```bash
+git add -A && git commit --no-gpg-sign -m "feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document"
+```

.agent/phases/todo/08_story_dark_tech_theme.md

@@ -1,136 +0,0 @@
-# Phase 08 — Story: Dark Tech Theme
-
-**Story:** `.agent/user_stories/dark-tech-theme.md`
-**Context:** `.agent/PLAN.md` §7 (UI/UX strategy), §10 (testing)
-
-## Goal
-Re-skin the whole UI to a dark, techy, emoji-free look with a subtly
-animated pure-CSS background — zero behavior or layout changes, WCAG 2.1
-AA re-proven on the new palette.
-
-## Dependencies
-Phases 01–07 (all complete). Independent of 09/10; phase 10's viewer page
-inherits this theme, so 08 must land first.
-
-## Locked decisions
-No anchors changed. Replaces the §7.2 light contrast pairs with the dark
-palette below (PLAN §7.2 already updated 2026-08-21 with owner
-permission). No new technology — pure CSS/HTML/inline SVG (A11).
-
-## Implementation steps
-1. **Palette swap** — `frontend/assets/styles.css` `:root` tokens (all
-   pairs computed, ≥4.5:1):
-
-   | token | dark value | computed pair |
-   |---|---|---|
-   | `--bg` | `#0a0e17` | ink on bg 16.2:1 |
-   | `--surface` | `#121a2e` | ink on surface 14.5:1 |
-   | `--ink` | `#e8ebf4` | — |
-   | `--ink-soft` | `#9aa4bd` | ink-soft on surface 6.9:1 |
-   | `--line` | `#26304a` | decorative |
-   | `--brand` | `#6d78f2` | **dark ink `--bg` on brand 5.2:1** |
-   | `--brand-soft` | `#232b52` | brand-ink on brand-soft 6.9:1 |
-   | `--brand-ink` | `#a5b4fc` | brand-ink on surface 8.7:1 |
-   | `--accent-bg` | `#2b2110` | accent-ink on accent-bg 9.5:1 |
-   | `--accent-ink` | `#fbbf24` | — |
-   | `--accent-line` | `#f59e0b` | unchanged (8.9:1 on bg) |
-   | `--err-bg` / `--err-ink` | `#2d1318` / `#fca5a5` | 9.1:1 |
-   | `--err-line` | `#ef4444` | 4.6:1 on err-bg (UI boundary) |
-   | `--ok-bg` / `--ok-ink` | `#10241b` / `#6ee7a8` | 10.6:1 |
-
-   Button text is `--bg` (dark) on `--brand` — **never white on brand**
-   (3.7:1, fails). Busy button: keep the `#a5b4fc` background (the
-   `tests/unit/test_frontend_feedback.py` assertion greps this token)
-   with a **dark** arc (`--bg`, 9.7:1). Update derived light-mode values:
-   shadows (black-based, lower alpha), selection, typing dots, chip
-   hover.
-2. **Emoji purge** — replace every emoji in chrome with inline SVG
-   (`aria-hidden` kept, ~16–20px, `currentColor` where sensible):
-   - `frontend/assets/app.js` (~L116, ~L132): avatars 🧠/🧑 → SVG
-     circuit-node glyph (brain) / minimal silhouette (user) as JS string
-     constants.
-   - `frontend/index.html`: favicon 🧠 data-URI → SVG tech mark (hex +
-     node, brand color on dark, <1 KB), still a `data:` URI;
-     `.brand-mark` 🧠 → same mark; ⚠️ banner icon → SVG triangle; 👋
-     empty state → SVG glyph.
-   - `frontend/sources.html`: favicon, `.brand-mark`, 📂 empty state →
-     SVG marks.
-3. **Tech details** — mono wordmark with letter-spacing; stat values
-   mono; radii `10px`/`6px`; 1px `--line` borders on cards/bubbles/table;
-   2px gradient hairline (brand→cyan, low alpha) under the sticky header.
-4. **Animated background (pure CSS, zero JS)** — working recipe:
-   `html { background: var(--bg) }`, `body { background: transparent;
-   position: relative }` (body must not create a stacking context):
-   - `body::before` — fine grid: two `linear-gradient`s (1px lines,
-     `--line` at ~35% alpha), `background-size: 44px 44px`, masked with a
-     radial fade (visible center-top, fading to the edges), animated
-     `background-position` `0 0 → 44px 44px`, 60s linear infinite
-     (seamless loop — the delta equals one cell).
-   - `body::after` — two large soft radial glows: indigo
-     `rgba(109,120,242,0.14)` top-left, cyan `rgba(34,211,238,0.10)`
-     bottom-right; 14s ease-in-out infinite alternate breathing
-     (opacity/scale). No `filter: blur` (perf).
-   - Both: `position: fixed; inset: 0; pointer-events: none; z-index:
-     -1`. Keep glow alpha low — subtle, never competing with text.
-5. **Reduced motion** — `@media (prefers-reduced-motion: reduce)`:
-   `body::before, body::after { animation: none }` (static grid + glows
-   remain). Existing typing/spinner reduced-motion handling stays.
-6. **Test updates (behavior unchanged):**
-   - `tests/e2e/test_honest_deflection.py` (~L104): deflection bubble
-     `backgroundColor` assertion `rgb(255, 247, 232)` → `rgb(43, 33,
-     16)`; the border assertion `rgb(245, 158, 11)` is unchanged.
-   - New integration test
-     `tests/integration/test_api.py::test_ui_chrome_has_no_emoji`: GET
-     `/`, `/sources.html`, `/assets/app.js`, `/assets/styles.css` — assert
-     no characters in the emoji code-point set (U+1F300–U+1FAFF,
-     U+2600–U+27BF, U+2B00–U+2BFF, U+FE0F, U+200D, plus the specific
-     glyphs previously used: 🧠 🧑 👋 📂 ⚠️).
-7. **PLAN.md §7.2** — dark contrast table already applied (2026-08-21,
-   owner permission); no further plan edits in this phase.
-
-## UI Verification
-Manual screenshot pass (1280px + 375px, both pages): grid is faint
-(barely-there), glows soft, no banding; brand button legible (dark
-text); deflection bubble distinct from normal answers; avatars crisp at
-16px; reduced-motion preview (DevTools emulation) shows the static
-background.
-
-## Testing & Quality
-- Integration: no-CDN check still green on both pages (it covers
-  `/sources.html` since Phase 07); new emoji-guard test above.
-- **Existing E2E regression check:** after the reskin, run the existing
-  story suites in isolation and confirm they stay green — at minimum
-  `test_chat_rag.py`, `test_honest_deflection.py`, `test_responsive_
-  polish.py` (the contrast helper computes from live styles and must pass
-  on the new palette).
-- Coverage: `uv run pytest --cov=app --cov-report=term-missing` — **>90%**
-  on `app/`.
-- `uv run ruff check . && uv run pyright` green.
-
-## Playwright Execution Phase
-Run ONLY this story's suite:
-
-```bash
-uv run pytest tests/e2e/test_dark_tech_theme.py -v --no-cov
-```
-
-Implements the story mapping: dark bg + computed contrast pairs ≥4.5:1;
-no emoji in innerText/outerHTML on both pages; `body::before`/`::after`
-animate; reduced-motion context → `animation-name: none`; on-topic smoke
-(stream + chip + button recovery) unchanged; all assets local.
-
-## Success criteria
-- [ ] both pages dark; every text pair ≥4.5:1 (computed in E2E)
-- [ ] zero emoji in chrome (E2E + new integration guard)
-- [ ] animated background subtle, pure CSS, reduced-motion honored
-- [ ] layout metrics + chat behavior unchanged (smoke E2E)
-- [ ] existing story E2E suites still green in isolation
-- [ ] unit + integration green, coverage >90%, ruff + pyright green
-- [ ] committed (force-add `.agent/PLAN.md` + this phase record — rule 8)
-
-## Commit
-```bash
-git add -A
-git add -f .agent/PLAN.md .agent/phases/todo/08_story_dark_tech_theme.md
-git commit --no-gpg-sign -m "feat(ui): dark tech theme — emoji-free chrome, subtle animated CSS background, WCAG AA dark palette"
-```