phase: 113_source_chip_quality
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 15s

All gates green — no defects found; this pass was verification only.

**Phase 113 final verification pass — report**

- Verified (no code changes needed): `select_documents_tiered` cited/related tiering + `select_documents` wrapper, `TurnPlan.related_docs`, `ChatDoneEvent.related` (additive, old payloads parse), `appendRelated` UI row (`.related-doc`, never `.source-chip`), done-frame + restore-path wiring, two settings with validators, `.env.example` entries
- `uv run pytest --cov=app --cov-report=term-missing` → 2422 passed, app/ coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (isolated) → 2 passed
- Regression E2E `test_retrieval_quality.py` + `test_honest_deflection.py` + `test_chat_rag.py` + `test_sources_midstream_bug.py` → 17 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors); `bash .agents/validate.sh` → "validation OK"

Completion criteria:
1. Single-doc question → exactly one `.source-chip` (E2E): ✅ passed
2. Weak 2nd doc only in de-emphasized related row, never `.source-chip` (unit + E2E): ✅ passed
3. Deflected turn → zero citation chips, weak hits in related row: ✅ passed
4. Full suite green, coverage >90%, isolated E2E green, lint/types clean: ✅ passed
5. `--no-gpg-sign` commit + phase dir move: left to harness per pass rules (task files already in `complete/`)

No deviations. Next pending phase: `114_embed_question_length`.
This commit is contained in:
2026-09-15 03:11:05 -04:00
parent 1374faf136
commit 97d663d16d
31 changed files with 2370 additions and 52 deletions
+85 -12
View File
@@ -27,8 +27,13 @@
unchanged (same lists, same ``1/(k+rank)`` terms).
* **Fusion** — Reciprocal Rank Fusion (``score = Σ 1/(k + rank)`` over the
lists a chunk appears in; chunks hit by both lists get both terms). The
fused score ranks; :meth:`select_documents` and :func:`weak_hit_titles`
keep working off ``score``.
fused score ranks; :func:`select_documents` and :func:`weak_hit_titles`
keep working off ``score``. Phase 113: the document tiering
(:func:`select_documents_tiered`) keeps the same rank order and adds the
usefulness bar — a document earns the cited tier only when its best
hit-chunk cosine clears ``BOR_SOURCE_USEFULNESS_FLOOR``; the rest of the
ranked documents (up to ``BOR_RELATED_MAX_DOCS``) become the related
tier.
The product requirement (LOCKED A7, revised 2026-08-24): the LLM receives
the **entire relevant document**, not just the chunk — chunk hits map back
@@ -560,6 +565,77 @@ def weak_hit_titles(chunks: Sequence[RetrievedChunk]) -> list[str]:
return titles
def select_documents_tiered(
chunks: Sequence[RetrievedChunk],
n: int | None = None,
floor: float = 0.0,
related_cap: int = 0,
) -> tuple[list[Document], list[Document]]:
"""Tier chunk hits into the cited and the related parent documents
(phase 113, LOCKED A2/A4 — the usefulness bar).
Distinct parent documents are ranked exactly like :func:`select_documents`
(best fused score first — the same stable score-descending walk, so a
document's rank position is fixed by its FIRST seen chunk) and each
document's **best hit-chunk cosine** is tracked across all of its
chunks. The tiers are then cut in that rank order:
* **cited** — the documents whose best-chunk cosine clears *floor*,
up to *n* (default ``BOR_TOP_N_DOCS``). The ceiling, never a quota:
a single strong document yields one cited document, and documents
whose cosine stays below the bar are skipped (the next-ranked
clearing document takes their slot — the bar filters, it does not
backfill). The bar is on the **cosine**, not the RRF fused score:
the fused ``score`` is a rank key, not a similarity, and a
lexical-only hit has cosine 0.0 — vector-unsupported by definition
(LOCKED A2, consistent with the A8 honesty gate).
* **related** — the next distinct documents in the same rank order
that are not already cited (any cosine, including 0.0 lexical-only
hits), up to *related_cap* (default 0). Never overlaps the cited
list. The done frame carries them in the secondary ``related``
tier — the UI's de-emphasized "nearby docs" row, never a citation
chip (LOCKED A4).
With ``floor=0.0`` (no bar — a zero floor admits every scored
document, so the legacy "any score, top-N" selection holds exactly)
and ``related_cap=0`` the tiering degenerates to the legacy behavior:
:func:`select_documents` is a thin wrapper on that.
The returned rows carry the full document content, byte-identical —
a matched parent document is **never truncated** (A7 revised, owner
permission 2026-08-24).
"""
top_n = n if n is not None else get_settings().top_n_docs
no_bar = floor <= 0.0
order: list[Document] = []
best_cosine: dict[uuid.UUID, float] = {}
for rc in sorted(chunks, key=lambda c: c.score, reverse=True):
doc = rc.document
if doc.id in best_cosine:
if rc.cosine > best_cosine[doc.id]:
best_cosine[doc.id] = rc.cosine
continue
best_cosine[doc.id] = rc.cosine
order.append(doc)
cited: list[Document] = []
for doc in order:
if len(cited) >= top_n:
break
if no_bar or best_cosine[doc.id] >= floor:
cited.append(doc)
cited_ids = {doc.id for doc in cited}
related: list[Document] = []
for doc in order:
if len(related) >= related_cap:
break
if doc.id not in cited_ids:
related.append(doc)
return cited, related
def select_documents(
chunks: Sequence[RetrievedChunk],
n: int | None = None,
@@ -572,14 +648,11 @@ def select_documents(
permission 2026-08-24). There is deliberately no context budget: an
oversized prompt must fail loudly through the ``LLMError`` → SSE
``error`` path, never arrive as silent partial context.
"""
top_n = n if n is not None else get_settings().top_n_docs
docs: list[Document] = []
seen: set[uuid.UUID] = set()
for rc in sorted(chunks, key=lambda c: c.score, reverse=True):
if rc.document.id in seen:
continue
seen.add(rc.document.id)
docs.append(rc.document)
return docs[:top_n]
Phase 113: a thin wrapper on :func:`select_documents_tiered` — the
legacy "any score, top-N" behavior is the cited tier with a zero
floor (no bar) and an empty related tier, byte-identical for all
existing callers.
"""
cited, _ = select_documents_tiered(chunks, n, 0.0, 0)
return cited