phase: 113_source_chip_quality
All gates green — no defects found; this pass was verification only. **Phase 113 final verification pass — report** - Verified (no code changes needed): `select_documents_tiered` cited/related tiering + `select_documents` wrapper, `TurnPlan.related_docs`, `ChatDoneEvent.related` (additive, old payloads parse), `appendRelated` UI row (`.related-doc`, never `.source-chip`), done-frame + restore-path wiring, two settings with validators, `.env.example` entries - `uv run pytest --cov=app --cov-report=term-missing` → 2422 passed, app/ coverage **99%** (>90% gate) - `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (isolated) → 2 passed - Regression E2E `test_retrieval_quality.py` + `test_honest_deflection.py` + `test_chat_rag.py` + `test_sources_midstream_bug.py` → 17 passed - `uv run ruff check . && uv run pyright` → clean (0 errors); `bash .agents/validate.sh` → "validation OK" Completion criteria: 1. Single-doc question → exactly one `.source-chip` (E2E): ✅ passed 2. Weak 2nd doc only in de-emphasized related row, never `.source-chip` (unit + E2E): ✅ passed 3. Deflected turn → zero citation chips, weak hits in related row: ✅ passed 4. Full suite green, coverage >90%, isolated E2E green, lint/types clean: ✅ passed 5. `--no-gpg-sign` commit + phase dir move: left to harness per pass rules (task files already in `complete/`) No deviations. Next pending phase: `114_embed_question_length`.
This commit is contained in:
+85
-12
@@ -27,8 +27,13 @@
|
||||
unchanged (same lists, same ``1/(k+rank)`` terms).
|
||||
* **Fusion** — Reciprocal Rank Fusion (``score = Σ 1/(k + rank)`` over the
|
||||
lists a chunk appears in; chunks hit by both lists get both terms). The
|
||||
fused score ranks; :meth:`select_documents` and :func:`weak_hit_titles`
|
||||
keep working off ``score``.
|
||||
fused score ranks; :func:`select_documents` and :func:`weak_hit_titles`
|
||||
keep working off ``score``. Phase 113: the document tiering
|
||||
(:func:`select_documents_tiered`) keeps the same rank order and adds the
|
||||
usefulness bar — a document earns the cited tier only when its best
|
||||
hit-chunk cosine clears ``BOR_SOURCE_USEFULNESS_FLOOR``; the rest of the
|
||||
ranked documents (up to ``BOR_RELATED_MAX_DOCS``) become the related
|
||||
tier.
|
||||
|
||||
The product requirement (LOCKED A7, revised 2026-08-24): the LLM receives
|
||||
the **entire relevant document**, not just the chunk — chunk hits map back
|
||||
@@ -560,6 +565,77 @@ def weak_hit_titles(chunks: Sequence[RetrievedChunk]) -> list[str]:
|
||||
return titles
|
||||
|
||||
|
||||
def select_documents_tiered(
|
||||
chunks: Sequence[RetrievedChunk],
|
||||
n: int | None = None,
|
||||
floor: float = 0.0,
|
||||
related_cap: int = 0,
|
||||
) -> tuple[list[Document], list[Document]]:
|
||||
"""Tier chunk hits into the cited and the related parent documents
|
||||
(phase 113, LOCKED A2/A4 — the usefulness bar).
|
||||
|
||||
Distinct parent documents are ranked exactly like :func:`select_documents`
|
||||
(best fused score first — the same stable score-descending walk, so a
|
||||
document's rank position is fixed by its FIRST seen chunk) and each
|
||||
document's **best hit-chunk cosine** is tracked across all of its
|
||||
chunks. The tiers are then cut in that rank order:
|
||||
|
||||
* **cited** — the documents whose best-chunk cosine clears *floor*,
|
||||
up to *n* (default ``BOR_TOP_N_DOCS``). The ceiling, never a quota:
|
||||
a single strong document yields one cited document, and documents
|
||||
whose cosine stays below the bar are skipped (the next-ranked
|
||||
clearing document takes their slot — the bar filters, it does not
|
||||
backfill). The bar is on the **cosine**, not the RRF fused score:
|
||||
the fused ``score`` is a rank key, not a similarity, and a
|
||||
lexical-only hit has cosine 0.0 — vector-unsupported by definition
|
||||
(LOCKED A2, consistent with the A8 honesty gate).
|
||||
* **related** — the next distinct documents in the same rank order
|
||||
that are not already cited (any cosine, including 0.0 lexical-only
|
||||
hits), up to *related_cap* (default 0). Never overlaps the cited
|
||||
list. The done frame carries them in the secondary ``related``
|
||||
tier — the UI's de-emphasized "nearby docs" row, never a citation
|
||||
chip (LOCKED A4).
|
||||
|
||||
With ``floor=0.0`` (no bar — a zero floor admits every scored
|
||||
document, so the legacy "any score, top-N" selection holds exactly)
|
||||
and ``related_cap=0`` the tiering degenerates to the legacy behavior:
|
||||
:func:`select_documents` is a thin wrapper on that.
|
||||
|
||||
The returned rows carry the full document content, byte-identical —
|
||||
a matched parent document is **never truncated** (A7 revised, owner
|
||||
permission 2026-08-24).
|
||||
"""
|
||||
top_n = n if n is not None else get_settings().top_n_docs
|
||||
no_bar = floor <= 0.0
|
||||
|
||||
order: list[Document] = []
|
||||
best_cosine: dict[uuid.UUID, float] = {}
|
||||
for rc in sorted(chunks, key=lambda c: c.score, reverse=True):
|
||||
doc = rc.document
|
||||
if doc.id in best_cosine:
|
||||
if rc.cosine > best_cosine[doc.id]:
|
||||
best_cosine[doc.id] = rc.cosine
|
||||
continue
|
||||
best_cosine[doc.id] = rc.cosine
|
||||
order.append(doc)
|
||||
|
||||
cited: list[Document] = []
|
||||
for doc in order:
|
||||
if len(cited) >= top_n:
|
||||
break
|
||||
if no_bar or best_cosine[doc.id] >= floor:
|
||||
cited.append(doc)
|
||||
|
||||
cited_ids = {doc.id for doc in cited}
|
||||
related: list[Document] = []
|
||||
for doc in order:
|
||||
if len(related) >= related_cap:
|
||||
break
|
||||
if doc.id not in cited_ids:
|
||||
related.append(doc)
|
||||
return cited, related
|
||||
|
||||
|
||||
def select_documents(
|
||||
chunks: Sequence[RetrievedChunk],
|
||||
n: int | None = None,
|
||||
@@ -572,14 +648,11 @@ def select_documents(
|
||||
permission 2026-08-24). There is deliberately no context budget: an
|
||||
oversized prompt must fail loudly through the ``LLMError`` → SSE
|
||||
``error`` path, never arrive as silent partial context.
|
||||
"""
|
||||
top_n = n if n is not None else get_settings().top_n_docs
|
||||
|
||||
docs: list[Document] = []
|
||||
seen: set[uuid.UUID] = set()
|
||||
for rc in sorted(chunks, key=lambda c: c.score, reverse=True):
|
||||
if rc.document.id in seen:
|
||||
continue
|
||||
seen.add(rc.document.id)
|
||||
docs.append(rc.document)
|
||||
return docs[:top_n]
|
||||
Phase 113: a thin wrapper on :func:`select_documents_tiered` — the
|
||||
legacy "any score, top-N" behavior is the cited tier with a zero
|
||||
floor (no bar) and an empty related tier, byte-identical for all
|
||||
existing callers.
|
||||
"""
|
||||
cited, _ = select_documents_tiered(chunks, n, 0.0, 0)
|
||||
return cited
|
||||
|
||||
Reference in New Issue
Block a user