phase: 112_honesty_gate_weak_hits
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):** - Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`. - Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length). - Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented. - Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW). - E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**. - Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors. **Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree). **Deviations:** none. Next pending phase: `113_source_chip_quality`.
This commit is contained in:
@@ -0,0 +1,46 @@
|
||||
# Phase 112 — Honesty gate: FTS hits need cosine corroboration (TODO L2a/b + README nit)
|
||||
|
||||
**Source:** `TODO.md` L23–86 — "L2 — Deflection test: non-KB question got answered parametrically + unrelated source chips (2026-09-15, brain-of-reese interactive test)" — part a (L36–44), part b (L46–55) + the stochastic follow-up (L64–75), the README nit (L77–81), acceptance (L83–86). Part c (chips on non-grounded answers, L57–62) is delivered by phase 113.
|
||||
**Story:** the completed story 04 (`04_story_honest_deflection`) — this phase fixes the A8 gate eagerness the live deflection test exposed.
|
||||
**Context:** `app/api/chat.py::plan_turn` (L272–277): HIGH when `best_cosine >= settings.relevance_threshold (0.62) or fts_hits > 0` — a single weak FTS token hit (suspected: the token "capital" inside a quest file) promoted a non-KB question into grounded mode and injected two irrelevant docs into the HIGH prompt, which then tempted the model into a parametric answer. The follow-up one-tap re-run of the identical question produced a **clean, textbook deflection** with the same docs injected → HONESTY GATE compliance is stochastic across runs; the deterministic lever is the gate (don't inject irrelevant docs), not prompt copy alone.
|
||||
|
||||
## Objective
|
||||
Make the A8 honesty gate deterministic against weak lexical hits: an FTS hit flips the turn to HIGH (grounded) only when the vector signal corroborates it (best cosine clears a new `lexical_support_floor`); below the floor the turn stays LOW (deflected) even with FTS matches. The owner-confirmed locked-prompt contract decision (treat the stochastic disclosed-general-knowledge behavior as acceptable — documented, prompt text byte-identical) and the stale README deflection copy are fixed in the same phase.
|
||||
|
||||
## Dependencies
|
||||
- `111_chat_banner_retry` (todo) — pipeline predecessor (execution order) only; no code dependency.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Gate rule (task 01):** HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`). LOW otherwise — including the fts>0 / cosine<floor case (the Mongolia case). `lexical_support_floor` is a new setting (default 0.35, env `BOR_LEXICAL_SUPPORT_FLOOR`), validated `0 <= floor <= relevance_threshold`. Rationale: a real lexical match on a genuinely similar doc (cosine ≥ floor) still grounds; a single weak token match with vector-unsupported docs no longer promotes. `plan_turn`'s docstring (the A8 bullets) and the A8 entry in `.agents/PLAN.md` are updated with the revision note (house precedent: "A8 revised 2026-08-21" — an owner-confirmed change to a LOCKED decision is recorded in the plan, not silently deviated from).
|
||||
- **No schema/API change:** `query_log.top_score` / `fts_hits` are recorded exactly as today (observability unchanged); `TurnPlan` shape unchanged; the LOW branch (deflect prompt, weak-hit titles, derived suggestions) unchanged.
|
||||
- **Prompt contract (task 03):** owner decision (iii) — the model's stochastic disclosed-general-knowledge answer (when misleading docs are injected) is documented as acceptable; the `app/rag/prompts.py` module docstring (the house location for locked-prompt revision history, e.g. the 2026-08-22 note) gains a dated entry; the prompt strings stay byte-identical (LOCKED verbatim); the README deflection section notes the behavior.
|
||||
- **README nit (task 04):** the quoted deflection opening *"I haven't done anything like that"* was removed in the 2026-08-22 locked-prompt revision — the two README spots (L11, L575) are updated to describe the current behavior (admit no notes + 2–3 concrete alternative questions).
|
||||
|
||||
## Tasks
|
||||
1. `01_gate_meaningful_fts.md` — the `plan_turn` gate fix + `lexical_support_floor` setting + the A8 plan revision note.
|
||||
2. `02_gate_tests.md` — unit pins for the HIGH/LOW quadrants on weak single-token FTS hits + the E2E deflection check.
|
||||
3. `03_prompt_contract_documentation.md` — document the owner decision (iii) (prompts.py docstring + README); prompt text unchanged.
|
||||
4. `04_readme_deflection_copy.md` — README stale deflection copy (L11, L575) updated.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_chat_gate.py` (existing — extend) + a new quadrant file (task 02): cosine ≥ threshold → HIGH regardless of FTS; fts>0 + cosine ≥ floor → HIGH; **fts>0 + cosine < floor → LOW** (the new behavior, the Mongolia regression pin); no hits → LOW.
|
||||
- E2E: `tests/e2e/test_honest_deflection.py` (existing — extend, task 02): a known-out-of-KB question (mock LLM, so the test pins the gate not the model) → deflected, no citation chips, 2–3 alternative questions. Run in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov`.
|
||||
- Regression: `tests/unit/test_chat_gate.py`, `tests/e2e/test_chat_rag.py`, `tests/e2e/test_retrieval_quality.py` stay green.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A weak single-token FTS hit with vector-unsupported docs (cosine < floor) → LOW/deflected (unit-pinned).
|
||||
- [ ] A known-out-of-KB question produces no false citations and 2–3 concrete alternative questions (E2E).
|
||||
- [ ] `app/rag/prompts.py` prompt strings byte-identical to pre-phase (test-pinned); README deflection copy matches current behavior.
|
||||
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
|
||||
|
||||
## Locked decisions
|
||||
- **A1 — lever: cosine corroboration; `lexical_support_floor` default 0.35, env-tunable via `BOR_LEXICAL_SUPPORT_FLOOR` (owner-confirmed 2026-09-14, roadmap confirmation).** Of the TODO's three options (rank threshold / stopword-short-token exclusion / cosine corroboration), cosine corroboration is the deterministic one; the tests pin the decision logic, not the default value.
|
||||
- **A2 — locked-prompt contract: option (iii) — treat the stochastic disclosed-general-knowledge behavior as acceptable and document it (owner-confirmed 2026-09-14, roadmap confirmation).** The prompt text is unchanged (LOCKED verbatim); the deterministic protection is the gate (A1). Options (i) tighten copy / (ii) amend the prompt via the plan remain open to a future owner decision.
|
||||
- **A3 — the A8 entry in `.agents/PLAN.md` is amended in this phase with a dated revision note** (owner-confirmed 2026-09-14) — house precedent for recording owner-confirmed LOCKED-decision changes.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ tests/ .agents/ README.md && git commit --no-gpg-sign -m "fix(rag): require cosine-corroborated FTS hits before the honesty gate flips HIGH — document the disclosed-answer behavior, refresh README deflection copy"
|
||||
```
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 01 — Gate: FTS hits must be cosine-corroborated
|
||||
|
||||
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:36–44` — "a) **Honesty gate too eager** (`app/api/chat.py` L272–277, A8 revised 2026-08-21): HIGH/grounded when `best_cosine >= threshold` OR `fts_hits > 0`. A single weak FTS token hit (suspected: the token 'capital' inside a quest file) promotes a non-KB question into grounded mode and injects two irrelevant top-docs into the HIGH prompt — which then tempts the model into a parametric answer instead of deflection. Consider: require FTS hits to be *meaningful* (e.g. rank threshold, stopword/short-token exclusion, or cosine corroboration) before flipping to HIGH; or run the LOW prompt when top-docs score below a usefulness bar." (+ the follow-up, L64–75: "the deterministic lever is the gate (don't inject irrelevant docs — part a), not prompt copy alone")
|
||||
|
||||
## Objective
|
||||
An FTS hit flips `plan_turn` to HIGH only when the best cosine clears the new `lexical_support_floor`; a weak single-token hit with vector-unsupported docs stays LOW (deflected).
|
||||
|
||||
## Work
|
||||
1. `app/config.py` — add `lexical_support_floor: float = Field(default=0.35)` (env `BOR_LEXICAL_SUPPORT_FLOOR`) next to `relevance_threshold` (L119), with validation `0 <= lexical_support_floor <= relevance_threshold` (mirror the file's existing validator style); document it in `.env.example`.
|
||||
2. `app/api/chat.py::plan_turn` — replace the gate line (`if best_cosine >= settings.relevance_threshold or fts_hits > 0:`):
|
||||
```python
|
||||
lexical_supported = fts_hits > 0 and best_cosine >= settings.lexical_support_floor
|
||||
if best_cosine >= settings.relevance_threshold or lexical_supported:
|
||||
...
|
||||
```
|
||||
Update the `plan_turn` docstring's gate section (the A8 bullets): HIGH when `best_cosine >= threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); LOW otherwise — a lexical-only hit without vector support deflects (A8 revised 2026-09-14).
|
||||
3. `.agents/PLAN.md` — amend the A8 locked-decision entry with a dated revision note (house precedent: "A8 revised 2026-08-21"): "A8 revised 2026-09-14 (owner-confirmed, TODO L2a): an FTS hit flips HIGH only when `best_cosine >= lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`); lexical-only hits without vector support deflect."
|
||||
4. `app/rag/prompts.py` — prompt strings NOT touched (LOCKED verbatim).
|
||||
5. ASSUMPTION: the chosen lever is cosine corroboration (one of the TODO's three listed options); floor default 0.35 (≈ half the 0.62 threshold) is tunable via env against the live KB — the unit/E2E tests pin the decision logic, not the default value (locked A1).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: covered by task 02 (`tests/unit/test_chat_gate.py` extension) — this task's code must keep it green.
|
||||
- Coverage: **>90%** on `app/` including the new setting and the modified gate line.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `plan_turn`: fts_hits>0 + best_cosine < floor → LOW (`TurnPlan.deflected` True, LOW prompt, suggestions derived from weak-hit titles).
|
||||
- [ ] fts_hits>0 + floor <= best_cosine < threshold → HIGH (the new "corroborated lexical" path).
|
||||
- [ ] best_cosine >= threshold → HIGH regardless of FTS (unchanged); no hits → LOW (unchanged).
|
||||
- [ ] `.env.example` documents `BOR_LEXICAL_SUPPORT_FLOOR`; the config validation rejects floor > threshold.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 02 — Pin the gate's HIGH/LOW decision on weak FTS hits
|
||||
|
||||
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:83–86` — "Acceptance: a known-out-of-KB question (LLM-known, e.g. capitals, sports results) produces no false citations, follows the HONESTY GATE (or the amended contract), and unit/E2E tests pin the gate's HIGH/LOW decision on a weak single-token FTS hit."
|
||||
|
||||
## Objective
|
||||
Unit tests pin all four gate quadrants — especially the new fts>0 / cosine<floor → LOW quadrant (the Mongolia/"capital" regression); the E2E deflection story asserts a known-out-of-KB question deflects with no false citations and 2–3 alternatives.
|
||||
|
||||
## Work
|
||||
1. `tests/unit/test_chat_gate.py` (existing — extend; reuse its chunk-building helpers) — the quadrants:
|
||||
- cosine ≥ threshold, fts=0 → HIGH (unchanged).
|
||||
- fts>0, cosine ≥ floor (e.g. 0.50 with default settings) → HIGH (corroborated lexical — the new path).
|
||||
- **fts>0, cosine < floor** (e.g. one `fts_hit=True` chunk with cosine 0.10 — the "capital" case) → LOW: `deflected=True`, LOW prompt, `suggestions` non-empty, the weak docs do not enter a HIGH prompt.
|
||||
- no chunks → LOW (unchanged).
|
||||
- boundary: cosine exactly at the floor → HIGH (`>=`, mirroring the threshold's convention); config with floor > threshold → validation error.
|
||||
2. `tests/e2e/test_honest_deflection.py` (existing — extend): a test asking a known-out-of-KB question (e.g. "What is the capital of Mongolia?" — LLM-known, absent from the fixture KB; the mock LLM's deflection path keeps the test deterministic on the gate, not on model compliance) → the done frame is `deflected: true`, `sources` is empty (no false citations), `suggestions` has 2–3 items.
|
||||
3. Run the E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` (DB up).
|
||||
4. Regression: `tests/unit/test_chat_gate.py` (all), `tests/e2e/test_chat_rag.py`, `tests/e2e/test_retrieval_quality.py` green.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the quadrant table (the new quadrant is the regression pin for TODO L2a).
|
||||
- E2E: the deflection story extension.
|
||||
- Coverage: **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] All quadrants green, including fts>0 + cosine<floor → LOW.
|
||||
- [ ] E2E: known-out-of-KB question → deflected, zero source chips, 2–3 alternatives.
|
||||
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
# Task 03 — Document the disclosed-answer contract (owner decision iii)
|
||||
|
||||
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:46–55, 64–75` — "b) **Model violates the locked prompt** (Rule 1 'Answer ONLY from the provided document context', Rule 3 HONESTY GATE): with irrelevant docs injected it answered from general knowledge. The disclosure is better UX than silence, but the contract says no pretending to know + 2-3 concrete alternative questions — decide whether to (i) tighten the prompt copy … (ii) amend the locked prompt via the plan to explicitly permit disclosed general-knowledge answers, or (iii) treat the observed behavior as acceptable and document it. Owner decision required — the prompt text is locked verbatim (change through the plan, not here)." + the follow-up: "HONESTY GATE compliance is **stochastic** across runs (run 1: parametric answer; run 2: perfect deflection). Implication for the fix direction: the deterministic lever is the gate …, not prompt copy alone; a small local model cannot be relied on to obey Rules 1/3 100% when handed misleading context."
|
||||
|
||||
## Objective
|
||||
Record the owner-confirmed decision (iii): the stochastic disclosed-general-knowledge answer (when misleading docs are injected) is acceptable and documented — the prompt text stays byte-identical (LOCKED verbatim); the deterministic protection is the task-01 gate fix.
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — module docstring (the house location for the locked-prompt revision history, e.g. the 2026-08-22 note): add a dated entry recording the 2026-09-15 interactive-test finding (parametric "Ulaanbaatar" answer with a disclosure, on the Mongolia question with two irrelevant docs injected; the clean textbook deflection on the identical one-tap re-run — stochastic compliance) and the owner decision (2026-09-14, roadmap confirmation): treat the disclosed general-knowledge answer as acceptable; the deterministic lever is the gate (A8 revised, task 01); options (i)/(ii) remain open to a future plan amendment. Prompt strings: byte-identical.
|
||||
2. `README.md` — in the deflection paragraph (the one task 04 rewrites), one sentence: with a small local model, a rare turn may answer from general knowledge with an explicit disclosure when retrieval was borderline — the gate (phase 112) minimizes this; the disclosure is surfaced, never silent.
|
||||
3. Prompt-lock pin: if no existing test byte-pins the prompt text, add a small `tests/unit/test_prompt_lock.py` asserting the HIGH/LOW prompt constants against pre-phase anchor strings (the executor extracts the pre-phase values when writing the test — e.g. exact prefix/suffix + total length, so any byte change fails).
|
||||
4. ASSUMPTION: the documentation lives in the prompts.py docstring (the existing revision-history location) + the README — no new docs file.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the prompt-lock pin (work item 3).
|
||||
- Coverage: n/a (docs-only change) — the suite stays green.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `app/rag/prompts.py` docstring carries the dated decision entry; the prompt strings are byte-identical (test-pinned).
|
||||
- [ ] The README deflection section notes the rare disclosed-answer behavior.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Task 04 — README: fix the stale deflection copy
|
||||
|
||||
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:77–81` — "Plus a docs nit: the README still promises the exact deflection copy *'I haven't done anything like that'* — the mandated deflection opening was removed in the 2026-08-22 locked-prompt revision (`app/rag/prompts.py` module docstring). Update the README's 'If it doesn't have notes…' paragraph to match current behavior."
|
||||
|
||||
## Objective
|
||||
The README describes the current deflection behavior (admit no notes + 2–3 concrete alternative questions) instead of the removed mandated opening.
|
||||
|
||||
## Work
|
||||
1. `README.md` L11 ("If it doesn't have notes for your question, it admits it: *'I haven't done…*") — rewrite the quoted copy to match the current locked-prompt behavior: it admits it has no notes on that and offers 2–3 concrete alternative questions about things it DOES have notes on. No exact-copy promise (the opening is no longer mandated).
|
||||
2. `README.md` L575 ("**Honest deflection** (the amber *'I haven't done anything like that'*…") — same update; keep the amber-banner description accurate.
|
||||
3. Grep the README for any other occurrence of the old quoted opening and update it.
|
||||
|
||||
## Testing & Quality
|
||||
- Docs-only: no code tests. Grep `tests/` for `haven't done anything` — if any test asserts the old copy, update it to the new behavior.
|
||||
- Coverage: n/a.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `grep -rn "haven't done anything" README.md` → no hits (the quoted opening is gone).
|
||||
- [ ] The deflection paragraph matches current behavior (admit + 2–3 concrete alternatives).
|
||||
- [ ] `uv run pytest` green.
|
||||
Reference in New Issue
Block a user