phase: 112_honesty_gate_weak_hits
Build and Push Containers / build-and-push-app (push) Successful in 2m15s
Build and Push Containers / build-and-push-db (push) Successful in 14s

**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):**

- Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`.
- Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length).
- Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented.
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW).
- E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**.
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors.

**Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree).

**Deviations:** none. Next pending phase: `113_source_chip_quality`.
This commit is contained in:
2026-09-15 00:37:38 -04:00
parent 2683128876
commit 1374faf136
36 changed files with 1240 additions and 67 deletions
+33 -19
View File
@@ -8,8 +8,8 @@ Reciprocal Rank Fusion), feeds the **whole relevant document** to a
**self-hosted LLM** (`turbo` via `https://aipi.reeseapps.com/v1`), and
streams a grounded answer back.
If it doesn't have notes for your question, it admits it: *"I haven't done
anything like that"* — plus suggestions for what it **does** know.
If it doesn't have notes for your question, it admits it — and offers
2-3 concrete alternative questions about things it **does** have notes on.
> **Updated your notes?** Re-run the import — it's idempotent and only
> re-embeds what changed:
@@ -345,9 +345,12 @@ nearly double, which lets a name-your-tool question ("gitlab") find its own
document even when the question embeds close to generic templates.
The **honesty gate** (A8) then answers (HIGH) when the best cosine is ≥
`BOR_RELEVANCE_THRESHOLD` (default `0.62`) **or** at least one chunk matched
lexically (`fts_hits > 0`) — it deflects (LOW) only when *both* signals are
absent. The top `BOR_TOP_N_DOCS` full documents are what the LLM sees.
`BOR_RELEVANCE_THRESHOLD` (default `0.62`) **or** when at least one chunk
matched lexically (`fts_hits > 0`) **and** the best cosine clears
`BOR_LEXICAL_SUPPORT_FLOOR` (default `0.35`) — the vector signal must
corroborate a lexical hit (A8 revised 2026-09-14). It deflects (LOW)
otherwise, including a weak single-token hit with vector-unsupported
docs. The top `BOR_TOP_N_DOCS` full documents are what the LLM sees.
---
@@ -412,7 +415,8 @@ asset references of the known pages in flight.
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview |
| `BOR_EMBEDDING_DIM` | `768` | Vector dimension (fixed at table creation) |
| `BOR_TOP_N_DOCS` | `2` | Full documents fed to the LLM |
| `BOR_RELEVANCE_THRESHOLD` | `0.62` | Answer when best cosine ≥ this **or** an FTS hit; below + no FTS ⇒ honest deflection |
| `BOR_RELEVANCE_THRESHOLD` | `0.62` | Answer when best cosine ≥ this **or** a lexical hit corroborated by cosine ≥ `BOR_LEXICAL_SUPPORT_FLOOR`; otherwise honest deflection |
| `BOR_LEXICAL_SUPPORT_FLOOR` | `0.35` | Best-cosine floor an FTS hit must clear to flip the gate HIGH (A8 revised 2026-09-14) |
| `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | Cosine list width for the RRF fusion |
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
@@ -572,22 +576,32 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
- **Embedding dimension mismatch** — aipi changed models; run
`uv run python -m scripts.llm_probe`, update `BOR_EMBEDDING_DIM`, then
drop + recreate the chunks table.
- **Honest deflection** (the amber *"I haven't done anything like that"*
bubble) — every question passes the honesty gate: deflection happens only
- **Honest deflection** (the amber deflection bubble) — every question
passes the honesty gate: deflection happens
when the best cosine similarity is below `BOR_RELEVANCE_THRESHOLD`
(default `0.62`) **and** no chunk matched lexically (`fts_hits = 0`).
A weak cosine with a lexical hit (name-your-tool questions) still gets a
grounded answer. When it does deflect, the LLM prompt carries weak-hit
*titles only* (no document content), the reply opens with *"I haven't
done anything like that"*, the bubble renders amber with *"Maybe try"*
chips derived from the closest indexed titles, and the `query_log` row
records `deflected=true`. This is a feature, not a bug.
(default `0.62`) **and** the lexical signal is absent or not
vector-corroborated (`fts_hits = 0`, or best cosine below
`BOR_LEXICAL_SUPPORT_FLOOR` — default `0.35` — A8 revised 2026-09-14).
A weak cosine with a lexical hit (name-your-tool questions) still gets
a grounded answer, provided the best cosine clears the floor. When it
does deflect, the LLM prompt carries weak-hit *titles only* (no document
content), the reply admits it has no notes on that and offers 2-3
concrete alternative questions about things it **does** have notes on,
the bubble renders amber with *"Maybe try"* chips derived from the
closest indexed titles, and the `query_log` row records
`deflected=true`. This is a feature, not a bug. With a small local
model, a rare turn may still answer from general knowledge with an
explicit disclosure when retrieval was borderline — the gate (A8
revised) minimizes this by keeping vector-unsupported docs out of the
grounded prompt, and the disclosure is surfaced, never silent.
- **Answers deflect too often / too rarely** — tune
`BOR_RELEVANCE_THRESHOLD` (lower = answers more, higher = more honest
deflection): `0.0` ⇒ the gate leans entirely on FTS hits; `1.0` ⇒
everything deflects unless a chunk matches lexically. The `embed` model's
cosines cluster in a ~0.6–0.85 band on the live KB, so the default is
`0.62`. Check real scores:
deflection) and `BOR_LEXICAL_SUPPORT_FLOOR` (the best-cosine floor a
lexical hit must clear to flip HIGH — lower = lexical hits ground more
easily): `0.0` ⇒ the gate leans entirely on FTS hits; `1.0` ⇒
everything deflects unless a lexical hit is vector-corroborated. The
`embed` model's cosines cluster in a ~0.6–0.85 band on the live KB, so
the default is `0.62`. Check real scores:
```sql
SELECT question, top_score, fts_hits, deflected
FROM query_log ORDER BY created_at DESC LIMIT 20;