phase: 112_honesty_gate_weak_hits
Build and Push Containers / build-and-push-app (push) Successful in 2m15s
Build and Push Containers / build-and-push-db (push) Successful in 14s

**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):**

- Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`.
- Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length).
- Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented.
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW).
- E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**.
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors.

**Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree).

**Deviations:** none. Next pending phase: `113_source_chip_quality`.
This commit is contained in:
2026-09-15 00:37:38 -04:00
parent 2683128876
commit 1374faf136
36 changed files with 1240 additions and 67 deletions
+20 -7
View File
@@ -13,7 +13,8 @@ The four tests map the story's acceptance criteria:
2. "How did I install gitlab?" — grounded (not deflected), gitlab chip,
``query_log`` row with the gitlab doc in ``sources``
3. keyword-only question ("kafkabridge") beats the vector ranking — the
FTS-OR gate grounds it end to end despite weak cosine
corroborated-lexical gate (A8 revised 2026-09-14) grounds it end to
end: weak cosine, but an FTS hit AND cosine >= lexical_support_floor
4. "sourdough" — deflected bubble + ≥2 "Maybe try" chips
"""
from __future__ import annotations
@@ -37,7 +38,15 @@ from e2e.auth_helpers import login
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "docs"
GITLAB_QUESTION = "How did I install gitlab?"
KEYWORD_QUESTION = "How does kafkabridge work?"
# Phase 112 (A8 revised): the pre-phase question ("How does kafkabridge
# work?" — mock cosine 0.134) now sits BELOW lexical_support_floor
# (0.15, mock-calibrated) with fts>0 — the new gate's deflection
# quadrant, so it can no longer demonstrate the grounded lexical path.
# "handle DNS" adds static-dns.json's own tokens: cosine ≈0.24 — still
# weak (below the 0.30 threshold) yet corroborated (>= floor) with the
# same single-doc FTS hit, and the FTS-matched doc still tops the fused
# ranking (the test's actual assertion).
KEYWORD_QUESTION = "How does kafkabridge handle DNS?"
OFF_TOPIC = "sourdough starter"
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
@@ -162,9 +171,11 @@ def test_gitlab_question_is_grounded_with_gitlab_chip(
def test_keyword_only_question_beats_vector_ranking(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""The FTS-OR gate end to end: "kafkabridge" appears in exactly one
fixture doc (static-dns.json) and the question's cosine overlap is
weak — the lexical branch is what grounds the answer."""
"""The corroborated-lexical gate end to end (A8 revised 2026-09-14):
"kafkabridge" appears in exactly one fixture doc (static-dns.json)
and the question's cosine overlap is weak (below the threshold) —
the lexical hit plus cosine >= lexical_support_floor is what grounds
the answer (a lexical-only hit below the floor would deflect)."""
_reset_db(mock_llm, seed=True)
page.set_default_timeout(30_000)
login(page, app_url, next="/") # phase 79: chat is require_user-gated
@@ -182,9 +193,11 @@ def test_keyword_only_question_beats_vector_ranking(
with SessionLocal() as db:
row = db.scalars(select(QueryLog)).one()
# Weak vector score…
# Weak vector score — below the answer threshold…
assert row.top_score < get_settings().relevance_threshold
# …but a lexical hit grounded it (the FTS-OR branch).
# …but cleared the lexical support floor, and a lexical hit fired —
# the corroborated-lexical path (A8 revised 2026-09-14) grounded it.
assert row.top_score >= get_settings().lexical_support_floor
assert (row.fts_hits or 0) >= 1
assert row.deflected is False
assert "homelab/networking/static-dns.json" in row.sources