phase: 112_honesty_gate_weak_hits
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):** - Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`. - Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length). - Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented. - Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW). - E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**. - Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors. **Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree). **Deviations:** none. Next pending phase: `113_source_chip_quality`.
This commit is contained in:
@@ -13,7 +13,8 @@ The four tests map the story's acceptance criteria:
|
||||
2. "How did I install gitlab?" — grounded (not deflected), gitlab chip,
|
||||
``query_log`` row with the gitlab doc in ``sources``
|
||||
3. keyword-only question ("kafkabridge") beats the vector ranking — the
|
||||
FTS-OR gate grounds it end to end despite weak cosine
|
||||
corroborated-lexical gate (A8 revised 2026-09-14) grounds it end to
|
||||
end: weak cosine, but an FTS hit AND cosine >= lexical_support_floor
|
||||
4. "sourdough" — deflected bubble + ≥2 "Maybe try" chips
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -37,7 +38,15 @@ from e2e.auth_helpers import login
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "docs"
|
||||
GITLAB_QUESTION = "How did I install gitlab?"
|
||||
KEYWORD_QUESTION = "How does kafkabridge work?"
|
||||
# Phase 112 (A8 revised): the pre-phase question ("How does kafkabridge
|
||||
# work?" — mock cosine 0.134) now sits BELOW lexical_support_floor
|
||||
# (0.15, mock-calibrated) with fts>0 — the new gate's deflection
|
||||
# quadrant, so it can no longer demonstrate the grounded lexical path.
|
||||
# "handle DNS" adds static-dns.json's own tokens: cosine ≈0.24 — still
|
||||
# weak (below the 0.30 threshold) yet corroborated (>= floor) with the
|
||||
# same single-doc FTS hit, and the FTS-matched doc still tops the fused
|
||||
# ranking (the test's actual assertion).
|
||||
KEYWORD_QUESTION = "How does kafkabridge handle DNS?"
|
||||
OFF_TOPIC = "sourdough starter"
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
|
||||
@@ -162,9 +171,11 @@ def test_gitlab_question_is_grounded_with_gitlab_chip(
|
||||
def test_keyword_only_question_beats_vector_ranking(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""The FTS-OR gate end to end: "kafkabridge" appears in exactly one
|
||||
fixture doc (static-dns.json) and the question's cosine overlap is
|
||||
weak — the lexical branch is what grounds the answer."""
|
||||
"""The corroborated-lexical gate end to end (A8 revised 2026-09-14):
|
||||
"kafkabridge" appears in exactly one fixture doc (static-dns.json)
|
||||
and the question's cosine overlap is weak (below the threshold) —
|
||||
the lexical hit plus cosine >= lexical_support_floor is what grounds
|
||||
the answer (a lexical-only hit below the floor would deflect)."""
|
||||
_reset_db(mock_llm, seed=True)
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/") # phase 79: chat is require_user-gated
|
||||
@@ -182,9 +193,11 @@ def test_keyword_only_question_beats_vector_ranking(
|
||||
|
||||
with SessionLocal() as db:
|
||||
row = db.scalars(select(QueryLog)).one()
|
||||
# Weak vector score…
|
||||
# Weak vector score — below the answer threshold…
|
||||
assert row.top_score < get_settings().relevance_threshold
|
||||
# …but a lexical hit grounded it (the FTS-OR branch).
|
||||
# …but cleared the lexical support floor, and a lexical hit fired —
|
||||
# the corroborated-lexical path (A8 revised 2026-09-14) grounded it.
|
||||
assert row.top_score >= get_settings().lexical_support_floor
|
||||
assert (row.fts_hits or 0) >= 1
|
||||
assert row.deflected is False
|
||||
assert "homelab/networking/static-dns.json" in row.sources
|
||||
|
||||
Reference in New Issue
Block a user