phase: 112_honesty_gate_weak_hits
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):** - Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`. - Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length). - Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented. - Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW). - E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**. - Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors. **Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree). **Deviations:** none. Next pending phase: `113_source_chip_quality`.
This commit is contained in:
@@ -413,7 +413,11 @@ def test_off_topic_question_deflects_honestly(client, db, seeded_kb: FakeRagLLM)
|
||||
assert any(
|
||||
"Deploying a New Service" in s for s in done["suggestions"]
|
||||
), "the best weak-hit title must be offered as a chip"
|
||||
assert done["sources"], "weak hits are still reported as the closest sources"
|
||||
# Phase 112 (A8 revised, TODO L2): a deflected turn cites nothing —
|
||||
# done.sources is the citation surface (the UI chips every entry as
|
||||
# "the answer used this"), and the weak hits are scored docs, not
|
||||
# citations. (Pre-phase: they rode the wire as sources.)
|
||||
assert done["sources"] == []
|
||||
|
||||
# The LLM saw the LOW prompt: DEFLECT_MODE + titles, never doc content.
|
||||
(system, user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
|
||||
@@ -433,19 +437,34 @@ def test_off_topic_question_deflects_honestly(client, db, seeded_kb: FakeRagLLM)
|
||||
assert 0.0 < row.top_score < get_settings().relevance_threshold
|
||||
assert row.fts_hits == 0
|
||||
assert row.chunk_hits >= 1
|
||||
# The retrieval stays durably recorded for threshold tuning
|
||||
# (observability unchanged — query_log records retrieval, not
|
||||
# citations; the done frame's [] above is the citation surface).
|
||||
assert row.sources
|
||||
|
||||
|
||||
def test_keyword_question_grounded_by_lexical_hit_despite_weak_cosine(
|
||||
client, db, seeded_kb: FakeRagLLM
|
||||
client, db, seeded_kb: FakeRagLLM,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 09: a name-your-tool question the vector model barely ranks
|
||||
("kafkabridge" only appears in static-dns.json) must still be grounded
|
||||
via the FTS branch — LOW only fires at weak cosine AND zero hits."""
|
||||
via the FTS branch — HIGH when cosine >= lexical_support_floor AND
|
||||
fts_hits > 0 (A8 revised 2026-09-14).
|
||||
|
||||
The conftest floor (0.15) is above the mock's cosine (~0.134), so we
|
||||
lower the floor here so the corroborated-lexical path fires."""
|
||||
from app.config import get_settings # noqa: E402
|
||||
|
||||
monkeypatch.setenv("BOR_LEXICAL_SUPPORT_FLOOR", "0.10")
|
||||
# get_settings is lru_cached — clear the cache so the new env var takes effect.
|
||||
get_settings.cache_clear()
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, "How does kafkabridge work?")
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
get_settings.cache_clear()
|
||||
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
|
||||
Reference in New Issue
Block a user