phase: 112_honesty_gate_weak_hits
Build and Push Containers / build-and-push-app (push) Successful in 2m15s
Build and Push Containers / build-and-push-db (push) Successful in 14s

**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):**

- Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`.
- Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length).
- Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented.
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW).
- E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**.
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors.

**Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree).

**Deviations:** none. Next pending phase: `113_source_chip_quality`.
This commit is contained in:
2026-09-15 00:37:38 -04:00
parent 2683128876
commit 1374faf136
36 changed files with 1240 additions and 67 deletions
+122
View File
@@ -0,0 +1,122 @@
"""Prompt-lock pin (phase 112, task 03 — owner decision iii, 2026-09-14).
The persona + HONESTY GATE text is **locked verbatim** (PLAN §6): it
changes through the plan, never in code. This module byte-pins the
locked prompt constants against their pre-phase-112 anchor values —
sha256 + exact prefix/suffix + total length, so *any* byte change
(option (i)'s copy tightening, option (ii)'s plan amendment, or an
accidental edit) fails loudly until the anchors are re-cut as part of
the same plan revision. ``tests.unit.test_prompts`` pins the
assembled-prompt structure and the behavioral contracts on top of
these constants; this file pins the constants themselves.
"""
from __future__ import annotations
import hashlib
from app.rag.prompts import (
PERSONA,
TOOLS_SECTION,
_base,
build_deflect_prompt,
)
def _sha256(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()
# ---------- PERSONA (the locked base of BOTH the HIGH and LOW prompts) ----------
#: Pre-phase-112 anchors for ``PERSONA`` (the ``{relevance}`` placeholder
#: and the line wrapping included).
PERSONA_SHA256 = "e31792a73e64c53853097e0f7b6df8b96c5f2d05c286944c3edba16dde7777fe"
PERSONA_LEN = 706
PERSONA_PREFIX = (
'You are "Brain of Reese" — the digital brain of Reese, a self-hoster and\n'
"homelab tinkerer. Personality: chippy, upbeat, warm, and genuinely\n"
"optimistic about the user's ability to do things.\n"
"\n"
"Rules:\n"
)
PERSONA_SUFFIX = (
"4. Never invent facts, hosts, or steps that are not in the context.\n"
"5. Keep answers tight: short paragraphs, bullets where helpful.\n"
"\n"
"<relevance>{relevance}</relevance>"
)
def test_persona_byte_locked() -> None:
"""Any byte change to the locked persona (opening, any rule line, the
``<relevance>`` placeholder) fails on the sha256; the prefix/suffix
anchors name the damaged region for the diff."""
assert len(PERSONA) == PERSONA_LEN
assert _sha256(PERSONA) == PERSONA_SHA256
assert PERSONA.startswith(PERSONA_PREFIX)
assert PERSONA.endswith(PERSONA_SUFFIX)
def test_high_and_low_bases_byte_locked() -> None:
"""``_base`` only substitutes ``{relevance}`` — the per-mode base
lengths pin the substitution against a moved or re-spelled
placeholder in the locked text."""
assert len(_base("HIGH")) == 699 # PERSONA_LEN - 11 + 4
assert len(_base("LOW")) == 698 # PERSONA_LEN - 11 + 3
# ---------- TOOLS_SECTION (the HIGH prompt's locked ``<tools>`` copy) ----------
#: Pre-phase-112 anchors for ``TOOLS_SECTION``.
TOOLS_SECTION_SHA256 = "b834cbe368055e65da82ae3e37a91e6c658c713954703fc79b849a6ebdf4aa53"
TOOLS_SECTION_LEN = 2273
TOOLS_SECTION_PREFIX = (
"<tools>\n"
"You may extend your context with three tools. `ls` lists the "
"knowledge base as a tree, one level at a time: "
)
TOOLS_SECTION_SUFFIX = (
"Never repeat a call that was refused or already succeeded — the refusal "
"already told you the correct form. Answer as soon as you have what "
"you need.\n</tools>"
)
def test_tools_section_byte_locked() -> None:
"""The ``<tools>`` teaching is LOCKED verbatim too (the E2E mock keys
on the ``<tools>`` marker's presence; the wording is the owner's):
sha256 + exact prefix/suffix + total length."""
assert len(TOOLS_SECTION) == TOOLS_SECTION_LEN
assert _sha256(TOOLS_SECTION) == TOOLS_SECTION_SHA256
assert TOOLS_SECTION.startswith(TOOLS_SECTION_PREFIX)
assert TOOLS_SECTION.endswith(TOOLS_SECTION_SUFFIX)
# ---------- the LOW prompt's locked DEFLECT_MODE body ----------
#: The ``DEFLECT_MODE`` body exactly as it was pre-phase-112 (the
#: phase-71 plain-text line included) — inline in
#: :func:`app.rag.prompts.build_deflect_prompt`, so it is pinned through
#: the built prompt rather than a module constant.
LOW_BODY_SHA256 = "c9868cfccdd0ff79d7c1de5df5f6dea0182d3726ca912c9ef609ab54b563a0a4"
LOW_BODY_LEN = 259
LOW_BODY = (
"DEFLECT_MODE: retrieval was weak — the titles below are the closest "
"your notes come to the question. They are titles only; do not pretend "
"they answer it. Use them to propose 2-3 alternative questions.\n"
"Reply in plain text only — you have no tools in this mode."
)
def test_deflect_body_byte_locked() -> None:
"""The LOW build = locked base + exactly the locked DEFLECT_MODE body
+ the weak-hit title list — byte for byte (the mock keys on the
``DEFLECT_MODE`` marker's presence; the body wording is locked)."""
assert len(LOW_BODY) == LOW_BODY_LEN
assert _sha256(LOW_BODY) == LOW_BODY_SHA256
assert build_deflect_prompt(["T1", "T2"]) == _base("LOW") + "\n" + LOW_BODY + "\n- T1\n- T2"
prompt = build_deflect_prompt(["T1", "T2"])
assert prompt.count(LOW_BODY) == 1
assert prompt.index("DEFLECT_MODE") < prompt.index(
"Reply in plain text only"
) # the marker precedes the plain-text line