phase: 112_honesty_gate_weak_hits
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):** - Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`. - Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length). - Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented. - Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW). - E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**. - Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors. **Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree). **Deviations:** none. Next pending phase: `113_source_chip_quality`.
This commit is contained in:
@@ -58,6 +58,27 @@ recovery in :mod:`app.rag.scaffolding` / :mod:`app.rag.agent` is the
|
||||
backstop). The ``DEFLECT_MODE`` marker and everything else in the
|
||||
prompt stay put — the E2E mock LLM keys on the marker's *presence*,
|
||||
not the wording, so that contract is unchanged.
|
||||
|
||||
Disclosed-answer contract (phase 112, task 03; owner decision
|
||||
2026-09-14, roadmap confirmation — option (iii) of the three the
|
||||
2026-09-15 interactive deflection test raised): that test found the
|
||||
HONESTY GATE's compliance is **stochastic** across runs when
|
||||
misleading context is injected — the "What is the capital of
|
||||
Mongolia?" question had two *irrelevant* docs promoted into the HIGH
|
||||
prompt by a weak single-token FTS hit (the pre-phase A8 rule: any
|
||||
``fts_hits > 0`` flipped HIGH), and run 1 answered parametrically —
|
||||
"Ulaanbaatar" — *with an explicit disclosure*, while the identical
|
||||
one-tap re-run produced a clean, textbook deflection. The owner's
|
||||
decision: the disclosed general-knowledge answer is treated as
|
||||
**acceptable and documented** — a small local model cannot be relied
|
||||
on to obey Rules 1/3 100% when handed misleading context, so this
|
||||
prompt text stays byte-identical (LOCKED verbatim; options (i) tighten
|
||||
the copy / (ii) amend this locked prompt via the plan to explicitly
|
||||
permit disclosed general-knowledge answers remain open to a future
|
||||
owner decision). The deterministic lever is the honesty gate itself
|
||||
(A8 revised 2026-09-14: an FTS hit flips HIGH only when
|
||||
``best_cosine >= lexical_support_floor``, so the misleading docs are
|
||||
no longer injected — see :mod:`app.api.chat`).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -21,8 +21,9 @@
|
||||
match a document's ``qwen3``/``8``/``27b`` tokens, while every
|
||||
unrelated llama.cpp quadlet out-ranks the target on the shared
|
||||
``llama``/``cpp`` tokens). The name-hit rows carry ``fts_hit=True``
|
||||
(they ARE the lexical signal — the A8 honesty gate then answers
|
||||
instead of deflecting) and ``cosine=0.0``; the RRF fusion is
|
||||
(they ARE the lexical signal — the A8 honesty gate answers on them
|
||||
only when the best cosine clears ``BOR_LEXICAL_SUPPORT_FLOOR``,
|
||||
A8 revised 2026-09-14) and ``cosine=0.0``; the RRF fusion is
|
||||
unchanged (same lists, same ``1/(k+rank)`` terms).
|
||||
* **Fusion** — Reciprocal Rank Fusion (``score = Σ 1/(k + rank)`` over the
|
||||
lists a chunk appears in; chunks hit by both lists get both terms). The
|
||||
@@ -445,7 +446,7 @@ def _name_hit_chunks(db: Session, question: str) -> list[RetrievedChunk]:
|
||||
score=0.0, # filled in by :func:`fuse`
|
||||
document=doc,
|
||||
cosine=0.0, # no vector rank — name-only hit
|
||||
fts_hit=True, # the lexical signal — the A8 gate answers
|
||||
fts_hit=True, # lexical signal — A8 answers if cosine corroborates
|
||||
is_summary=bool(row.is_summary),
|
||||
)
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user