ducoterra
dbf2af26c6
Standardize on the .agents/ directory (shared with project skills): phases/, user_stories/, reports/, screenshots/, validate.sh, and phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves history; runtime artifacts move alongside). Updates every reference in AGENTS.md, README.md, .gitignore, app docstrings, and test story headers. Historical KB content in data/ and the runtime pipeline.log transcript are left untouched.
79 lines
4.3 KiB
Markdown
79 lines
4.3 KiB
Markdown
# Story: Retrieval Quality — Multi-Format Ingestion + Hybrid Search
|
||
|
||
**Phase:** `09_story_retrieval_quality.md` · **E2E:** `tests/e2e/test_retrieval_quality.py`
|
||
|
||
## Narrative
|
||
|
||
As **a user**, I want to ask questions that name my tools ("How did I
|
||
install gitlab?") and get a grounded answer citing the right document —
|
||
even when the answer lives in a `.yaml`, `.py`, `.json` or `.txt` file —
|
||
so Brain actually covers my notes instead of deflecting around them.
|
||
|
||
- **Given** my notes are a mix of markdown write-ups and config/code
|
||
files, and today's index is polluted with vendored-cache docs
|
||
(`.esphome/.espressif/**`, merge-request templates, …) that outrank
|
||
real content, while retrieval is pure cosine top-4
|
||
- **When** the importer covers the full text-format set, skips hidden
|
||
directories, and retrieval fuses vector similarity with Postgres
|
||
full-text search
|
||
- **Then** "How did I install gitlab?" surfaces `active/container_gitlab/
|
||
gitlab.md` as the top document, the junk docs are pruned from the
|
||
index, and genuinely off-topic questions still get the honest
|
||
deflection.
|
||
|
||
## Acceptance criteria
|
||
1. **Ingestion scope (A9, revised with owner permission 2026-08-21):**
|
||
default extensions `md, markdown, txt, yaml, yml, json, py`,
|
||
overridable via `BOR_IMPORT_EXTENSIONS`; **hidden (dot) directories
|
||
skipped by default**; the A9 exclusion list stays.
|
||
2. **Format-aware chunking:** md unchanged (heading-aware); yaml/yml
|
||
split on top-level keys and `---` document separators (key kept as
|
||
chunk anchor); json pretty-printed and split on top-level keys; py
|
||
split on top-level defs/classes (stdlib `ast`); txt on paragraphs.
|
||
Every format honors the 1200-char hard cap (aipi ~1024-token request
|
||
cap).
|
||
3. **Hybrid retrieval (A7, revised with owner permission 2026-08-21):**
|
||
cosine top-30 + FTS top-30 (OR tsquery over the question's tokens,
|
||
`ts_rank`-ordered) fused with Reciprocal Rank Fusion (k=60);
|
||
document-level ranking by best fused chunk score; the LLM still
|
||
receives the **full top-2 documents** capped at 24k chars (contract
|
||
preserved).
|
||
4. **Honesty gate (A8, revised with owner permission 2026-08-21):**
|
||
deflect when `best cosine < BOR_RELEVANCE_THRESHOLD` **and** no
|
||
candidate chunk FTS-matches the question; threshold re-tuned for the
|
||
`embed` model's measured 0.41–0.84 score compression (default 0.62,
|
||
documented tuning procedure); `fts_hits` logged per turn and stored in
|
||
`query_log`.
|
||
5. **Live verification:** `scripts/eval_retrieval.py` run against the
|
||
imported KB shows `gitlab.md` as the top document for "How did I
|
||
install gitlab?"; a known off-topic question gates LOW.
|
||
6. **Index hygiene:** after a `--prune` re-import, **zero** documents
|
||
with a dot-prefixed path component exist in `documents`; prune also
|
||
drops docs that no longer match the format filter.
|
||
7. **No regressions:** off-topic deflection (story 04) and grounded
|
||
answers (story 03) still pass their E2E suites in isolation; the E2E
|
||
mock keeps its own threshold calibration via the app fixture.
|
||
8. Unit + integration green, `app/` coverage >90%, story E2E green in
|
||
isolation, one `--no-gpg-sign` commit.
|
||
|
||
## UI Visualization & Structure
|
||
No UI change. The Sources page document count drops after the junk prune
|
||
(intended cleanup — noted in the README).
|
||
|
||
## Playwright Mapping Rule
|
||
**Test Scenario → `tests/e2e/test_retrieval_quality.py`** (mock LLM, seeded KB):
|
||
1. `test_multiformat_docs_indexed` — fixture tree with `.md`, `.yaml`,
|
||
`.py`, `.json`, `.txt` plus `.hidden/junk.md`: import in-process →
|
||
`/api/docs` lists every non-hidden doc and none of the hidden ones.
|
||
2. `test_named_tool_question_retrieves_right_doc` — "How did I install
|
||
gitlab?" → not deflected, streamed answer, a `.source-chip` containing
|
||
the gitlab doc, `query_log` row `deflected=false` with the gitlab doc
|
||
in sources.
|
||
3. `test_keyword_only_question_beats_vector_ranking` — a question whose
|
||
rare token exists in exactly one fixture doc (weak mock-embedding
|
||
overlap, e.g. "kafkabridge") → grounded answer citing that doc
|
||
(FTS-OR gate + lexical RRF lift, end to end).
|
||
4. `test_off_topic_still_deflects` — "sourdough starter" →
|
||
`.is-deflected` bubble matching /haven't done anything like that/i +
|
||
≥2 "Maybe try:" chips (story 04 behavior under the new gate).
|