phase: 112_honesty_gate_weak_hits
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):** - Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`. - Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length). - Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented. - Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW). - E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**. - Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors. **Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree). **Deviations:** none. Next pending phase: `113_source_chip_quality`.
This commit is contained in:
+49
-26
@@ -15,14 +15,22 @@ turn's thinking is counted in the per-turn log line
|
||||
(``thinking_chars=N``); ``BOR_STREAM_THINKING=0`` suppresses the
|
||||
``thinking`` frames (the pieces are still counted).
|
||||
|
||||
Honesty gate (A8, revised 2026-08-21): LOW — deflection — only when the
|
||||
best cosine is strictly below ``BOR_RELEVANCE_THRESHOLD`` **and** no
|
||||
candidate chunk FTS-matches the question (``fts_hits == 0``). A
|
||||
name-your-tool question with weak vector overlap but a lexical hit still
|
||||
gets a grounded answer. Deflection mode carries weak-hit *titles only*
|
||||
(never document content) plus deterministic "Maybe try" chips, and the
|
||||
``done`` event / ``query_log`` row record ``deflected=true``, the weak
|
||||
score and the ``fts_hits`` count.
|
||||
Honesty gate (A8, revised 2026-09-14): LOW — deflection — only when
|
||||
``best_cosine < BOR_RELEVANCE_THRESHOLD`` **and** (``fts_hits == 0``
|
||||
or ``best_cosine < BOR_LEXICAL_SUPPORT_FLOOR``). A lexical hit alone
|
||||
(cosine < lexical_support_floor) no longer promotes to HIGH — the vector
|
||||
signal must corroborate the lexical match (A8 revised 2026-09-14, the
|
||||
"Mongolia" fix). HIGH fires when ``best_cosine >= threshold`` OR
|
||||
(fts>0 AND cosine >= lexical_support_floor). Deflection mode carries
|
||||
weak-hit *titles only* (never document content) plus deterministic
|
||||
"Maybe try" chips, and the ``done`` event / ``query_log`` row record
|
||||
``deflected=true``, the weak score and the ``fts_hits`` count. A
|
||||
deflected turn cites nothing: the ``done`` frame's ``sources`` is
|
||||
``[]`` (``done.sources`` is the citation surface — the UI chips every
|
||||
entry as "the answer used this" — and weak hits are scored docs, not
|
||||
citations, TODO L2), while ``query_log.sources`` and the per-turn log
|
||||
line keep recording the retrieval for tuning (phase 112, A8 revised).
|
||||
|
||||
|
||||
Steering (phase 15): the owner's stored tuning notes are loaded per turn
|
||||
(oldest first) and injected into the system prompt as a ``<tuning>``
|
||||
@@ -270,13 +278,15 @@ def plan_turn(
|
||||
"""Apply the honesty gate (A8, revised) and assemble prompt + context.
|
||||
|
||||
* **HIGH (grounded)** when ``best_cosine >= threshold`` **or**
|
||||
``fts_hits > 0``: HIGH prompt with the full top-N documents, no
|
||||
suggestions. A cosine exactly at the threshold is an answer — the
|
||||
gate is strict (``< threshold``).
|
||||
* **LOW (deflected)** only when ``best_cosine < threshold`` **and**
|
||||
``fts_hits == 0`` (or no hits at all): LOW prompt (``DEFLECT_MODE``)
|
||||
with weak-hit titles only — never document content — plus
|
||||
deterministic alternative-question chips derived from those titles.
|
||||
(``fts_hits > 0`` **and** ``best_cosine >= lexical_support_floor``):
|
||||
HIGH prompt with the full top-N documents, no suggestions. A cosine
|
||||
exactly at the threshold is an answer — the gate is strict
|
||||
(``< threshold``). An FTS hit alone, without vector corroboration
|
||||
(cosine < lexical_support_floor), stays LOW (A8 revised 2026-09-14).
|
||||
* **LOW (deflected)** otherwise — including the fts>0 / cosine < floor
|
||||
case (the "Mongolia" case): LOW prompt (``DEFLECT_MODE``) with
|
||||
weak-hit titles only — never document content — plus deterministic
|
||||
alternative-question chips derived from those titles.
|
||||
|
||||
``top_score`` (stored in ``query_log``) is the best cosine, so the
|
||||
gate input is always a pure vector-similarity number; the lexical
|
||||
@@ -305,7 +315,8 @@ def plan_turn(
|
||||
docs = select_documents(chunks, n=settings.top_n_docs)
|
||||
selected_ids = {d.id for d in docs}
|
||||
summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in selected_ids)
|
||||
if best_cosine >= settings.relevance_threshold or fts_hits > 0:
|
||||
lexical_supported = fts_hits > 0 and best_cosine >= settings.lexical_support_floor
|
||||
if best_cosine >= settings.relevance_threshold or lexical_supported:
|
||||
return TurnPlan(
|
||||
best_cosine,
|
||||
fts_hits,
|
||||
@@ -736,10 +747,14 @@ async def chat(
|
||||
|
||||
# 4. Durable record + required per-turn log line (PLAN §9).
|
||||
# Phase 37: the agent's read documents join the
|
||||
# retrieval's — deduped by (source, path), order preserved
|
||||
# — and the same combined list feeds done.sources,
|
||||
# query_log.sources and the log line (empty on deflected
|
||||
# turns: the agent never runs). A cancelled turn (the
|
||||
# retrieval's — deduped by (source, path), order preserved.
|
||||
# The combined list feeds query_log.sources and the log
|
||||
# line (retrieval docs even on deflected turns —
|
||||
# observability, the phase-113 A3 precedent). Phase 112:
|
||||
# done.sources is the CITATION surface — it carries the
|
||||
# combined list on grounded turns and [] on deflected
|
||||
# ones (a deflected answer cites nothing; the weak hits
|
||||
# stay in the durable record). A cancelled turn (the
|
||||
# generator closed by the consumer) never reaches this
|
||||
# step — no query_log row.
|
||||
cited_docs: list[Document] = []
|
||||
@@ -790,15 +805,23 @@ async def chat(
|
||||
scaffold_stripped,
|
||||
)
|
||||
settled = True # terminal: the done frame settles the turn
|
||||
# Phase 112 (A8 revised, TODO L2): a deflected turn cites
|
||||
# nothing — done.sources is [] (the UI chips every entry
|
||||
# as a citation; the weak hits are scored docs, not
|
||||
# citations). The retrieval stays durably recorded above
|
||||
# (query_log.sources + the log line — observability
|
||||
# unchanged); the phase-113 related-doc tier is the home
|
||||
# for the weak hits' visibility.
|
||||
cited_refs: list[SourceRef] = []
|
||||
if not plan.deflected:
|
||||
cited_refs = [
|
||||
SourceRef(source=d.source, path=d.path, title=d.title)
|
||||
for d in cited_docs
|
||||
]
|
||||
yield sse_event(
|
||||
ChatDoneEvent(
|
||||
deflected=plan.deflected,
|
||||
sources=[
|
||||
SourceRef(
|
||||
source=d.source, path=d.path, title=d.title
|
||||
)
|
||||
for d in cited_docs
|
||||
],
|
||||
sources=cited_refs,
|
||||
suggestions=plan.suggestions,
|
||||
).model_dump()
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user