phase: 113_source_chip_quality
All gates green — no defects found; this pass was verification only. **Phase 113 final verification pass — report** - Verified (no code changes needed): `select_documents_tiered` cited/related tiering + `select_documents` wrapper, `TurnPlan.related_docs`, `ChatDoneEvent.related` (additive, old payloads parse), `appendRelated` UI row (`.related-doc`, never `.source-chip`), done-frame + restore-path wiring, two settings with validators, `.env.example` entries - `uv run pytest --cov=app --cov-report=term-missing` → 2422 passed, app/ coverage **99%** (>90% gate) - `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (isolated) → 2 passed - Regression E2E `test_retrieval_quality.py` + `test_honest_deflection.py` + `test_chat_rag.py` + `test_sources_midstream_bug.py` → 17 passed - `uv run ruff check . && uv run pyright` → clean (0 errors); `bash .agents/validate.sh` → "validation OK" Completion criteria: 1. Single-doc question → exactly one `.source-chip` (E2E): ✅ passed 2. Weak 2nd doc only in de-emphasized related row, never `.source-chip` (unit + E2E): ✅ passed 3. Deflected turn → zero citation chips, weak hits in related row: ✅ passed 4. Full suite green, coverage >90%, isolated E2E green, lint/types clean: ✅ passed 5. `--no-gpg-sign` commit + phase dir move: left to harness per pass rules (task files already in `complete/`) No deviations. Next pending phase: `114_embed_question_length`.
This commit is contained in:
@@ -0,0 +1,46 @@
|
||||
# Phase 113 — Source chip quality: usefulness bar + related-docs tier (TODO L5 + L2c)
|
||||
|
||||
**Source:** `TODO.md` L101–147 — "L5 — Recurring weak-hit source chips: the 2nd chip is often noise the answer never used (2026-09-15, brain-of-reese interactive test)" — root-cause chain (L125–129), suggested directions (L131–142), acceptance (L144–146) — plus `TODO.md` L57–62 (L2 part c): "c) **Misleading chips on non-grounded answers** (`chat.py:252` — `done.sources` = weak hits when deflected; by design, but visually a citation). … At minimum: never render them as answer citations."
|
||||
**Story:** n/a (interactive-test follow-up fix; extends the phase-09 retrieval-quality and phase-05 chip assets).
|
||||
**Context:** `top_n_docs = 2` (`app/config.py:114`) forces two docs into `plan.docs`; `done.sources` (`app/api/chat.py` ~L796, `ChatDoneEvent.sources`) carries every entry and `appendSources` (`frontend/assets/app.js:1369`) chips them all with identical visual weight — "the answer used this" vs "this also scored" is indistinguishable. Deflected turns carry weak hits in `sources` "by design" but they render as citations. Owner-confirmed direction: the deterministic **usefulness bar** (server-side) + the **visual split** (UI) — cite-gated chips (parsing the model's cited paths) are rejected for now (stochastic with a small model).
|
||||
|
||||
## Objective
|
||||
A document earns a citation slot only when its retrieval signal is vector-corroborated (or the agent explicitly read it via a tool call); everything else that scored is demoted to a clearly secondary "nearby docs" row that never reads as a citation. For a single-document question the turn shows exactly one citation chip; a deflected turn shows no citation chips at all.
|
||||
|
||||
## Dependencies
|
||||
- `112_honesty_gate_weak_hits` (todo) — the gate fix stops weak hits being injected into the HIGH prompt; this phase stops weak docs earning a `done.sources` slot. Same workstream, ordered after the gate.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Usefulness bar (task 01):** new settings `source_usefulness_floor: float = 0.35` (env `BOR_SOURCE_USEFULNESS_FLOOR`, validated `0 <= floor <= relevance_threshold`, mirroring phase 112's floor) and `related_max_docs: int = 2` (env `BOR_RELATED_MAX_DOCS`, validated `>= 0`). In `plan_turn`, retrieval docs are tiered: **cited** = distinct parent docs (best fused-score order, at most `top_n_docs`) whose best hit-chunk **cosine** clears the floor; **related** = the next scored distinct docs (at most `related_max_docs`) that did not clear it. Agent-read docs (`holder.read_docs`, the phase-37 agent tool reads) always stay cited — the model read them via tool calls, so they were used by definition. `TurnPlan` gains `related_docs: list[Document] = []`; the tiering is a new `select_documents_tiered(chunks, n, floor, related_cap) -> tuple[list[Document], list[Document]]` in `app/rag/retriever.py`, with `select_documents` becoming a thin wrapper (legacy behavior byte-identical for existing callers/tests). `query_log.sources` is unchanged (it records retrieval, not citations — locked A3).
|
||||
- **Done frame (task 01):** `ChatDoneEvent` (`app/schemas.py`) gains `related: list[SourceRef] = []` — additive; old clients ignore unknown fields (house contract, PLAN §4). Built from `plan.related_docs` with the same (source, path) dedupe against the cited list as `cited_docs` already does.
|
||||
- **UI split (task 02):** `frontend/assets/app.js` — new `appendRelated(wrap, related)` renders a `.msg-meta.related-docs` row under the bubble (only when `related` is non-empty): a small de-emphasized label "Nearby docs, in case:" + one link per doc with the class `related-doc` (NOT `source-chip`) — same `documentUrl(...)` href and left-click → `openDocumentModal` behavior as citation chips, visually secondary (reduced opacity/size/dashed border via theme variables; link contrast ≥4.5:1, WCAG 2.1 AA). The done-frame handler (~L2421) also calls `appendRelated(wrap, ev.related)`; the restored-chat path (~L1561) likewise when the stored payload carries `related` (pre-phase chats don't — graceful). Deflected turns: `ev.sources` is empty (the server change) → no chips; the weak hits arrive in `ev.related` → the row only.
|
||||
- **NOT touched:** the citation-chip component (`.source-chip` / `appendSources`) for the cited tier; the suggestion chips; `top_n_docs` (ceiling, not quota); the phase-37 agent-read dedupe; the Sources/RAG pages.
|
||||
|
||||
## Tasks
|
||||
1. `01_usefulness_bar_sources.md` — retriever tiering + `TurnPlan.related_docs` + `ChatDoneEvent.related` + the two settings.
|
||||
2. `02_secondary_related_docs_ui.md` — the related-docs row in the chat UI (app.js + styles.css); deflected turns show no chips.
|
||||
3. `03_chip_filter_tests.md` — unit pins on the four observed live shapes + E2E chip-count assertions.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_retriever.py` (extend — the tiering table), `tests/unit/test_source_chip_quality.py` (new, task 03 — the four observed shapes), the done-frame schema tests (`related` defaults `[]`; old payloads without the field still parse).
|
||||
- E2E: `tests/e2e/test_source_chip_quality.py` (new, task 03; run in isolation: `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov`) — a known single-source question → exactly one citation chip; a deflected question → zero `.source-chip` elements (the row, if any, is `.related-doc`, never `.source-chip`).
|
||||
- Regression: `tests/e2e/test_retrieval_quality.py`, `test_honest_deflection.py`, `test_chat_rag.py`, `test_sources_midstream_bug.py` stay green.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] For a single-document question, the turn shows one citation chip (E2E).
|
||||
- [ ] A weak 2nd doc renders only in the de-emphasized related row, never as a `.source-chip` (unit + E2E).
|
||||
- [ ] A deflected turn renders zero citation chips (the weak hits, if any, live in the related row).
|
||||
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
|
||||
|
||||
## Locked decisions
|
||||
- **A1 — direction: usefulness bar (server, deterministic) + visual split (UI); cite-gated chips rejected for now (owner-confirmed 2026-09-14, roadmap confirmation).**
|
||||
- **A2 — citation slot = vector-corroborated retrieval doc (best hit-chunk cosine >= `source_usefulness_floor`, default 0.35, env-tunable) OR agent-read doc; `top_n_docs` stays a ceiling, not a quota (owner-confirmed 2026-09-14).**
|
||||
- **A3 — `query_log.sources` keeps recording the full retrieval (observability); `done.sources` records only the cited tier (owner-confirmed 2026-09-14).**
|
||||
- **A4 — the related tier is capped at 2 docs (`related_max_docs`, env `BOR_RELATED_MAX_DOCS` — owner-confirmed 2026-09-14).**
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): tier sources by a usefulness bar — weak hits become a de-emphasized related-docs row, never citation chips"
|
||||
```
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 01 — Usefulness bar: tier done.sources into cited + related
|
||||
|
||||
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:125–129, 137–140` — "Root cause chain: `top_n_docs = 2` (`app/config.py` L114) forces retrieval to return two documents, `done.sources` carries both (`app/api/chat.py` L252), and the UI chips every entry without distinguishing 'the answer used this' from 'this also scored'." + "**Usefulness bar on the 2nd doc** — only include a document in `done.sources` when its fused/cosine score clears a threshold (a single weak FTS token hit should not earn a citation slot); `top_n_docs` stays a ceiling, not a quota." + `TODO.md:57–62` (L2 part c: "`done.sources` = weak hits when deflected; by design, but visually a citation … At minimum: never render them as answer citations.")
|
||||
|
||||
## Objective
|
||||
Retrieval docs are tiered at the honesty gate: cited (vector-corroborated, ≤ `top_n_docs`) vs related (scored but under the floor, ≤ `related_max_docs`); the SSE done frame carries both; agent-read docs always stay cited.
|
||||
|
||||
## Work
|
||||
1. `app/config.py` — add `source_usefulness_floor: float = Field(default=0.35)` (env `BOR_SOURCE_USEFULNESS_FLOOR`) and `related_max_docs: int = Field(default=2)` (env `BOR_RELATED_MAX_DOCS`); validators: `0 <= source_usefulness_floor <= relevance_threshold`, `related_max_docs >= 0`; `.env.example` entries.
|
||||
2. `app/rag/retriever.py` — add `select_documents_tiered(chunks, n, floor, related_cap) -> tuple[list[Document], list[Document]]`:
|
||||
- rank distinct parent docs by best fused score (the existing `select_documents` ordering), tracking each doc's best hit-chunk cosine;
|
||||
- **cited** = the docs whose best-chunk cosine >= `floor`, up to `n` (ceiling — a single strong doc yields one cited doc);
|
||||
- **related** = the next docs in rank order (any cosine, including 0.0 lexical-only), up to `related_cap`, never overlapping the cited list.
|
||||
- `select_documents` becomes a wrapper: `cited, _ = select_documents_tiered(chunks, n, 0.0, 0)` — floor 0.0 + cap 0 keeps the legacy "any score, top-N" behavior byte-identical for existing callers/tests.
|
||||
3. `app/api/chat.py` — `TurnPlan` gains `related_docs: list[Document] = []` (after `docs`); `plan_turn` calls `select_documents_tiered(chunks, settings.top_n_docs, settings.source_usefulness_floor, settings.related_max_docs)` → `docs`, `related_docs` (both the HIGH and LOW branches — deflected turns: the weak hits fall to related, cited is usually empty). The done-frame build (~L796): add `related=[SourceRef(source=d.source, path=d.path, title=d.title) for d in <plan.related_docs deduped against cited_docs by (source, path)>]` — the same dedupe pattern `cited_docs` already uses.
|
||||
4. `app/schemas.py` — `ChatDoneEvent` gains `related: list[SourceRef] = []` (additive; docstring note: old clients ignore unknown fields, PLAN §4).
|
||||
5. `query_log.sources` — unchanged (locked A3).
|
||||
6. ASSUMPTION: the bar is on the **cosine** of the doc's best hit chunk, not the RRF fused score — the fused `score` is a rank key, not a similarity; a lexical-only hit has cosine 0.0 and is vector-unsupported by definition (consistent with the phase-112 gate; locked A2).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_retriever.py` (extend) — the tiering table: both clear → both cited; strong + weak → 1 cited + 1 related; both weak → 0 cited + 2 related; related cap respected; `select_documents` wrapper legacy behavior unchanged (existing tests stay green without edits).
|
||||
- Unit: the existing done-frame/schema tests — `related` defaults to `[]`; a payload without the field still parses (back-compat).
|
||||
- Coverage: **>90%** on `app/` including the new function.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `plan_turn` returns tiered docs; HIGH and LOW branches both populated correctly.
|
||||
- [ ] The done frame carries `related` (≤ `related_max_docs`, deduped against cited); old frames (no field) parse.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 02 — UI: the related-docs row (never a citation chip)
|
||||
|
||||
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:141–142` — "**Visual split** — keep both, but render uncited/weak docs as a clearly secondary 'related docs' row, not citation chips." + `TODO.md:59–62` — "When the answer is disclosed general knowledge (or a deflected turn), the chips should be visually de-emphasized / labeled (e.g. 'nearby docs I have, in case'), or omitted when the brain says it didn't use them. At minimum: never render them as answer citations."
|
||||
|
||||
## Objective
|
||||
The chat bubble renders the cited tier exactly as today (`.source-chip` via `appendSources`) and the related tier as a clearly secondary labeled row; a deflected turn renders zero citation chips.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/app.js`:
|
||||
- new `appendRelated(wrap, related)` next to `appendSources` (L1369): early-return when empty; a `.msg-meta.related-docs` row (`role="list"`, `aria-label="Nearby docs, in case"`) + a small `<span class="related-docs-label">Nearby docs, in case:</span>` + one link per doc — class `related-doc` (NOT `source-chip`), the same `documentUrl(s.source, s.path, "/")` href and left-click → `openDocumentModal(s.source, s.path, link)` behavior, `title`/`aria-label` carrying the full path.
|
||||
- the done-frame handler (~L2421, next to `appendSources(wrap, ev.sources)`): also `appendRelated(wrap, ev.related)`.
|
||||
- the restored-chat path (~L1561): same, when the stored payload carries `related` (pre-phase saved chats don't — the row is simply absent, graceful).
|
||||
- deflected turns: `ev.sources` is empty (the task-01 server change) → `appendSources` no-ops; the weak hits arrive in `ev.related` → row only.
|
||||
2. `frontend/assets/styles.css` — `.related-docs` (muted row: smaller font, theme-variable color — link text contrast ≥4.5:1, WCAG 2.1 AA), `.related-doc` (dashed border, no hover elevation of the citation chips; `focus-visible` ring), `.related-docs-label` (small caps or muted small text); the row stacks below the citation `.msg-meta` row with the existing gap.
|
||||
3. Frontend unit test (house source-assertion style — lives in `tests/unit/test_source_chip_quality.py`, extended by task 03): `appendRelated` exists and uses `related-doc` (assert `source-chip` is NOT in the `appendRelated` body); the row renders only when related is non-empty; the label text is present; the done handler calls `appendRelated`.
|
||||
4. ASSUMPTION: label copy "Nearby docs, in case:" (the TODO's suggested wording, trimmed).
|
||||
5. ASSUMPTION: pre-phase saved chats (stored payload without `related`) restore exactly as today — no related row.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the frontend source-assertion tests (task 03's file).
|
||||
- E2E: pinned by task 03.
|
||||
- Coverage: n/a (frontend) — the `app/` gate stays green.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A related doc renders only as `.related-doc` in the labeled row — never as `.source-chip`.
|
||||
- [ ] A deflected turn (mock) renders zero `.source-chip` elements under the bubble.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 03 — Pin the chip contract: the four observed shapes + E2E chip counts
|
||||
|
||||
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:110–123, 144–146` — the four observed live cases (1. "What is the capital of Mongolia?" → `Trooper_Nagraz.pl` + `Trooper_Begzei.pl`, both unrelated; 2. phase-gate question answered from `brain-of-reese/.agents/validate.sh` → second chip `ServMon/README.md` unused; 3. Trooper_Nagraz question answered from `Trooper_Nagraz.pl` → second chip `Trooper_Byzin.pl` uncited; 4. meta question about the conversation's own history → chips `app/api/suggestions.py` + `108_history_wire_check/00_phase.md`, neither used) + "Acceptance: for a single-document question, the turn shows one citation chip; a unit test pins `done.sources` filtering (or the chip renderer's cite-gate) on the four observed shapes; E2E asserts chip count for a known single-source question."
|
||||
|
||||
## Objective
|
||||
A unit test pins the cited/related tiering on the four live shapes; the E2E suite asserts the visible chip counts.
|
||||
|
||||
## Work
|
||||
1. `tests/unit/test_source_chip_quality.py` (new) — model each of the four observed shapes as a `plan_turn`/done-frame fixture (retrieval chunks with controlled cosine/`fts_hit`/fused `score`) and assert the tiering:
|
||||
1. **both docs weak** (cosine < floor, the Mongolia case) → `sources` empty, `related` ≤ 2.
|
||||
2. **one strong + one weak** (the validate.sh case: `validate.sh` cosine ≥ floor, `ServMon/README.md` below) → exactly 1 in `sources`, the weak one in `related`.
|
||||
3. **the Nagraz case** — same shape, different fixtures (`Trooper_Nagraz.pl` strong, `Trooper_Byzin.pl` weak) → 1 cited, 1 related.
|
||||
4. **the meta/history question** (no doc clears the floor, the agent reads nothing) → `sources` empty, `related` ≤ 2; assert the frame shape that the UI renders as row-only (the rendering is pinned by task 02's source tests + the E2E).
|
||||
5. **agent-read exemption**: a doc under the floor that is in `holder.read_docs` (agent tool read) still lands in `sources` (cited).
|
||||
- plus the frontend source-assertion tests from task 02 work item 3 (same file).
|
||||
2. `tests/e2e/test_source_chip_quality.py` (new; conftest/mock-LLM/fixture-KB pattern per `tests/e2e/test_retrieval_quality.py`):
|
||||
- a known single-source question (a fixture-KB question whose answer comes from one doc) → the done bubble has **exactly one** `.source-chip`.
|
||||
- a deflected question (known-out-of-KB) → **zero** `.source-chip`; if a `.related-docs` row exists, its links are `.related-doc`, never `.source-chip`.
|
||||
- if the fixture KB cannot produce a strong+weak two-tier shape, say so in the test docstring and rely on the unit table for that shape.
|
||||
3. Run in isolation: `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (DB up).
|
||||
4. Regression: `tests/e2e/test_retrieval_quality.py`, `test_honest_deflection.py`, `test_chat_rag.py` green.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the four-shape table (the acceptance pin) + the agent-read exemption.
|
||||
- E2E: the chip-count assertions (the acceptance pin).
|
||||
- Coverage: **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] The four observed shapes are unit-pinned (plus the agent-read exemption).
|
||||
- [ ] E2E: single-source question → exactly one citation chip; deflected turn → zero.
|
||||
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||
Reference in New Issue
Block a user