brain-of-reese/.agents/phases/todo/113_source_chip_quality/00_phase.md

Phase 113 — Source chip quality: usefulness bar + related-docs tier (TODO L5 + L2c)

Source: TODO.md L101–147 — "L5 — Recurring weak-hit source chips: the 2nd chip is often noise the answer never used (2026-09-15, brain-of-reese interactive test)" — root-cause chain (L125–129), suggested directions (L131–142), acceptance (L144–146) — plus TODO.md L57–62 (L2 part c): "c) Misleading chips on non-grounded answers (chat.py:252 — done.sources = weak hits when deflected; by design, but visually a citation). … At minimum: never render them as answer citations." Story: n/a (interactive-test follow-up fix; extends the phase-09 retrieval-quality and phase-05 chip assets). Context: top_n_docs = 2 (app/config.py:114) forces two docs into plan.docs; done.sources (app/api/chat.py ~L796, ChatDoneEvent.sources) carries every entry and appendSources (frontend/assets/app.js:1369) chips them all with identical visual weight — "the answer used this" vs "this also scored" is indistinguishable. Deflected turns carry weak hits in sources "by design" but they render as citations. Owner-confirmed direction: the deterministic usefulness bar (server-side) + the visual split (UI) — cite-gated chips (parsing the model's cited paths) are rejected for now (stochastic with a small model).

Objective

A document earns a citation slot only when its retrieval signal is vector-corroborated (or the agent explicitly read it via a tool call); everything else that scored is demoted to a clearly secondary "nearby docs" row that never reads as a citation. For a single-document question the turn shows exactly one citation chip; a deflected turn shows no citation chips at all.

Dependencies

• 112_honesty_gate_weak_hits (todo) — the gate fix stops weak hits being injected into the HIGH prompt; this phase stops weak docs earning a done.sources slot. Same workstream, ordered after the gate.

Design (shared by all tasks — the executor reads this, not the chat)

• Usefulness bar (task 01): new settings source_usefulness_floor: float = 0.35 (env BOR_SOURCE_USEFULNESS_FLOOR, validated 0 <= floor <= relevance_threshold, mirroring phase 112's floor) and related_max_docs: int = 2 (env BOR_RELATED_MAX_DOCS, validated >= 0). In plan_turn, retrieval docs are tiered: cited = distinct parent docs (best fused-score order, at most top_n_docs) whose best hit-chunk cosine clears the floor; related = the next scored distinct docs (at most related_max_docs) that did not clear it. Agent-read docs (holder.read_docs, the phase-37 agent tool reads) always stay cited — the model read them via tool calls, so they were used by definition. TurnPlan gains related_docs: list[Document] = []; the tiering is a new select_documents_tiered(chunks, n, floor, related_cap) -> tuple[list[Document], list[Document]] in app/rag/retriever.py, with select_documents becoming a thin wrapper (legacy behavior byte-identical for existing callers/tests). query_log.sources is unchanged (it records retrieval, not citations — locked A3).
• Done frame (task 01): ChatDoneEvent (app/schemas.py) gains related: list[SourceRef] = [] — additive; old clients ignore unknown fields (house contract, PLAN §4). Built from plan.related_docs with the same (source, path) dedupe against the cited list as cited_docs already does.
• UI split (task 02): frontend/assets/app.js — new appendRelated(wrap, related) renders a .msg-meta.related-docs row under the bubble (only when related is non-empty): a small de-emphasized label "Nearby docs, in case:" + one link per doc with the class related-doc (NOT source-chip) — same documentUrl(...) href and left-click → openDocumentModal behavior as citation chips, visually secondary (reduced opacity/size/dashed border via theme variables; link contrast ≥4.5:1, WCAG 2.1 AA). The done-frame handler (~L2421) also calls appendRelated(wrap, ev.related); the restored-chat path (~L1561) likewise when the stored payload carries related (pre-phase chats don't — graceful). Deflected turns: ev.sources is empty (the server change) → no chips; the weak hits arrive in ev.related → the row only.
• NOT touched: the citation-chip component (.source-chip / appendSources) for the cited tier; the suggestion chips; top_n_docs (ceiling, not quota); the phase-37 agent-read dedupe; the Sources/RAG pages.

Tasks

1. 01_usefulness_bar_sources.md — retriever tiering + TurnPlan.related_docs + ChatDoneEvent.related + the two settings.
2. 02_secondary_related_docs_ui.md — the related-docs row in the chat UI (app.js + styles.css); deflected turns show no chips.
3. 03_chip_filter_tests.md — unit pins on the four observed live shapes + E2E chip-count assertions.

Testing & Quality

• Unit: tests/unit/test_retriever.py (extend — the tiering table), tests/unit/test_source_chip_quality.py (new, task 03 — the four observed shapes), the done-frame schema tests (related defaults []; old payloads without the field still parse).
• E2E: tests/e2e/test_source_chip_quality.py (new, task 03; run in isolation: uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov) — a known single-source question → exactly one citation chip; a deflected question → zero .source-chip elements (the row, if any, is .related-doc, never .source-chip).
• Regression: tests/e2e/test_retrieval_quality.py, test_honest_deflection.py, test_chat_rag.py, test_sources_midstream_bug.py stay green.
• Coverage: >90% on app/ (validate.sh gate).

Completion Criteria

• For a single-document question, the turn shows one citation chip (E2E).
• A weak 2nd doc renders only in the de-emphasized related row, never as a .source-chip (unit + E2E).
• A deflected turn renders zero citation chips (the weak hits, if any, live in the related row).
• uv run pytest green; coverage >90%; the e2e file green in isolation; uv run ruff check . && uv run pyright clean.
• One --no-gpg-sign commit; phase dir moved to complete/ by the pipeline gate.

Locked decisions

• A1 — direction: usefulness bar (server, deterministic) + visual split (UI); cite-gated chips rejected for now (owner-confirmed 2026-09-14, roadmap confirmation).
• A2 — citation slot = vector-corroborated retrieval doc (best hit-chunk cosine >= source_usefulness_floor, default 0.35, env-tunable) OR agent-read doc; top_n_docs stays a ceiling, not a quota (owner-confirmed 2026-09-14).
• A3 — query_log.sources keeps recording the full retrieval (observability); done.sources records only the cited tier (owner-confirmed 2026-09-14).
• A4 — the related tier is capped at 2 docs (related_max_docs, env BOR_RELATED_MAX_DOCS — owner-confirmed 2026-09-14).

Commit

git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): tier sources by a usefulness bar — weak hits become a de-emphasized related-docs row, never citation chips"