phase: 112_honesty_gate_weak_hits
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):** - Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`. - Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length). - Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented. - Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW). - E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**. - Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors. **Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree). **Deviations:** none. Next pending phase: `113_source_chip_quality`.
This commit is contained in:
@@ -117,6 +117,15 @@ class Settings(BaseSettings):
|
||||
# default never discriminated. LOW only fires when best cosine < this
|
||||
# AND no candidate chunk matches the question lexically (see A8).
|
||||
relevance_threshold: float = 0.62
|
||||
#: Lexical support floor (A8 revised 2026-09-14): an FTS hit promotes
|
||||
# a turn to HIGH only when the best cosine is >= this value — it
|
||||
# requires the vector signal to corroborate the lexical match. A
|
||||
# single weak token hit with vector-unsupported docs (cosine < floor)
|
||||
# stays LOW (deflected). Default 0.35 ≈ half the relevance threshold;
|
||||
# tunable via ``BOR_LEXICAL_SUPPORT_FLOOR``. Must be <=
|
||||
# ``relevance_threshold`` (a floor above the threshold is a typo that
|
||||
# would make every FTS hit require a HIGH cosine anyway).
|
||||
lexical_support_floor: float = 0.35
|
||||
#: Maximum output tokens a chat answer may use (owner instruction
|
||||
#: 2026-08-22: answers must run to their natural end — the old hard
|
||||
#: 700-token cap cut long answers off mid-sentence).
|
||||
@@ -302,6 +311,22 @@ class Settings(BaseSettings):
|
||||
#: separate from ``sources_dir`` (the source checkouts).
|
||||
docs_work_dir: str = "~/bor-docs"
|
||||
|
||||
@field_validator("lexical_support_floor")
|
||||
@classmethod
|
||||
def _lexical_support_floor_bounds(cls, v: float, info: ValidationInfo) -> float:
|
||||
"""The lexical support floor must be in [0, relevance_threshold].
|
||||
A value above the relevance threshold would be a typo — it would
|
||||
make every FTS hit require a HIGH cosine anyway, defeating the
|
||||
purpose of the floor (A8 revised 2026-09-14)."""
|
||||
if v < 0:
|
||||
raise ValueError("lexical_support_floor must be >= 0")
|
||||
threshold = info.data.get("relevance_threshold")
|
||||
if isinstance(threshold, float) and v > threshold:
|
||||
raise ValueError(
|
||||
f"lexical_support_floor ({v}) must be <= relevance_threshold ({threshold})"
|
||||
)
|
||||
return v
|
||||
|
||||
@field_validator("import_extensions")
|
||||
@classmethod
|
||||
def _import_extensions_known(cls, v: str) -> str:
|
||||
|
||||
Reference in New Issue
Block a user