phase: 112_honesty_gate_weak_hits
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):** - Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`. - Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length). - Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented. - Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW). - E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**. - Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors. **Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree). **Deviations:** none. Next pending phase: `113_source_chip_quality`.
This commit is contained in:
+1
-1
@@ -46,7 +46,7 @@ accounts (one admin + hand-out tokens is the model).
|
||||
| A5 | LLM backend | OpenAI-compatible self-hosted endpoint `https://aipi.reeseapps.com/v1` via the `openai` async client: **`turbo`** (chat, streams `reasoning_content` thinking), **`embed`** (embeddings), **`lite`** (one-shot: document summaries, KB overview, folder summaries) |
|
||||
| A6 | Embedding dim | **768** (verified against the live endpoint); `chunks.embedding` is fixed at table creation — a dim mismatch must **fail loudly**, never silently re-embed |
|
||||
| A7 | Retrieval→context | **Hybrid:** cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, `ts_rank`), fused with **RRF (k=60)** → parent docs ranked by best fused chunk → **full text of top-N=2 documents, never truncated** on the retrieval path (owner 2026-08-24: "this should never happen"; the agent `read`-tool cap is a separate owner-permitted path) |
|
||||
| A8 | Honesty gate | Deflect (LOW mode) **only when** best cosine < `BOR_RELEVANCE_THRESHOLD` (0.62) **and** zero FTS hits; LOW prompt carries weak-hit *titles only* + the `DEFLECT_MODE` marker (the E2E mock keys on its presence) + the plain-text no-tools line |
|
||||
| A8 | Honesty gate | Deflect (LOW mode) **only when** best cosine < `BOR_RELEVANCE_THRESHOLD` (0.62) **and** (zero FTS hits **or** best cosine < `BOR_LEXICAL_SUPPORT_FLOOR` (0.35)); an FTS hit flips HIGH only when `best_cosine >= lexical_support_floor` — the vector signal must corroborate the lexical match (A8 revised 2026-09-14, owner-confirmed, TODO L2a: lexical-only hits without vector support deflect). LOW prompt carries weak-hit *titles only* + the `DEFLECT_MODE` marker (the E2E mock keys on its presence) + the plain-text no-tools line |
|
||||
| A9 | Content scope | Default `md, markdown, txt, yaml, yml, json, py` + Podman quadlet family + `j2`; `BOR_IMPORT_EXTENSIONS` may name **any** well-formed extension or narrow the set; hidden (dot) paths + exclusion list + per-source `ignore_paths` (raw prefixes, no globs) always apply |
|
||||
| A10 | State & auth | `POST /api/chat` is **stateless** (client-provided `history` only, budget-trimmed — nothing stored per conversation). Auth = single admin, signed `bor_session` cookie (Starlette `SessionMiddleware` + itsdangerous; no server-side session store); admin-issued SHA-256-hashed access tokens are the only other identity; **only** shared chats stay anonymous. Fail-loud at boot while `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` are empty |
|
||||
| A11 | Frontend | Vanilla HTML/CSS/JS in git, **no CDN** — everything served by FastAPI `StaticFiles`; system font stack; the navbar views are views of ONE shell document (`frontend/index.html` + `router.js` deep-links) |
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):**
|
||||
|
||||
- Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`.
|
||||
- Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length).
|
||||
- Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented.
|
||||
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW).
|
||||
- E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**.
|
||||
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
|
||||
|
||||
**Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree).
|
||||
|
||||
**Deviations:** none. Next pending phase: `113_source_chip_quality`.
|
||||
+102
@@ -0,0 +1,102 @@
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 99%]
|
||||
.. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 208 1 99%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 94 0 100%
|
||||
app/api/docs.py 156 1 99%
|
||||
app/api/git_sources.py 232 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 186 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 66 0 100%
|
||||
app/models.py 128 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 317 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 123 0 100%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 215 3 99%
|
||||
app/rag/llm.py 243 1 99%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 88 0 100%
|
||||
app/rag/retriever.py 172 3 98%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 327 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 4079 16 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+1
@@ -0,0 +1 @@
|
||||
All 2373 tests pass with 99% coverage. Now let me run the linter:
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 97%]
|
||||
..................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 205 1 99%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 94 0 100%
|
||||
app/api/docs.py 156 1 99%
|
||||
app/api/git_sources.py 232 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 186 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 66 0 100%
|
||||
app/models.py 128 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 317 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 123 0 100%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 215 3 99%
|
||||
app/rag/llm.py 243 1 99%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 88 0 100%
|
||||
app/rag/retriever.py 172 3 98%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 327 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 4076 16 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+1
@@ -0,0 +1 @@
|
||||
All 2373 tests pass with 99% coverage. Now let me run the linter:
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 97%]
|
||||
..................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 205 1 99%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 94 0 100%
|
||||
app/api/docs.py 156 1 99%
|
||||
app/api/git_sources.py 232 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 186 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 66 0 100%
|
||||
app/models.py 128 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 317 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 123 0 100%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 215 3 99%
|
||||
app/rag/llm.py 243 1 99%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 88 0 100%
|
||||
app/rag/retriever.py 172 3 98%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 327 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 4076 16 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
All green. Final verification complete.
|
||||
|
||||
**Task 02 (gate tests) — complete**
|
||||
|
||||
Implemented:
|
||||
- Unit quadrants in `tests/unit/test_chat_gate.py`: added the code-defaults quadrant (fts>0 + cosine 0.50 ≥ floor 0.35 → HIGH) and pinned the new done-frame contract on the endpoint (deflected → `done["sources"] == []`, `query_log.sources` still records retrieval)
|
||||
- E2E `test_honest_deflection.py`: new `test_out_of_kb_question_deflects_without_citations` — "Who composed Rhapsody in Blue?" (LLM-known, absent from fixture KB; fts_hits≥1, cosine 0.124 < floor 0.15 — the fts>0/cosine<floor quadrant pinned end-to-end) → `deflected: true`, `sources: []`, 2–3 suggestions
|
||||
- Server: deflected turns' `done` frame now sends `sources=[]` (no false citations — the L2 acceptance; UI chips nothing); `query_log.sources` + per-turn log line keep recording the retrieval (observability unchanged, phase-113 A3 precedent)
|
||||
- Updated the phase-04 integration test that pinned the old contract; reworded `test_retrieval_quality`'s kafkabridge question ("How does kafkabridge handle DNS?") — the old one (cos 0.134) now correctly falls in the new deflection quadrant
|
||||
|
||||
Results:
|
||||
- `uv run pytest` → 2374 passed; `uv run pytest --cov=app --cov-report=term-missing` → 99% on `app/` (>90%)
|
||||
- `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → 4 passed (in isolation); `test_chat_rag.py` → 3 passed; `test_retrieval_quality.py` → 4 passed
|
||||
- At-risk audit: scanned every E2E question literal for gate flips (only the intended one + branch-independent/empty-KB suites); ran 13 gate-sensitive E2E suites (38 tests) → all passed
|
||||
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
|
||||
|
||||
Decisions: the task's example question scores 0.303 (> E2E threshold 0.30) under the mock, so it would ground — replaced with Rhapsody in Blue, which also exercises the new quadrant; deflected-done-`sources=[]` is a deliberate contract change required by the task's acceptance (matches phase 113's assumed end state).
|
||||
|
||||
Next pending task: `112_honesty_gate_weak_hits/03_prompt_contract_documentation.md`
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 97%]
|
||||
...................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 208 1 99%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 94 0 100%
|
||||
app/api/docs.py 156 1 99%
|
||||
app/api/git_sources.py 232 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 186 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 66 0 100%
|
||||
app/models.py 128 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 317 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 123 0 100%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 215 3 99%
|
||||
app/rag/llm.py 243 1 99%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 88 0 100%
|
||||
app/rag/retriever.py 172 3 98%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 327 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 4079 16 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
All green. Task 03 complete.
|
||||
|
||||
**Report**
|
||||
|
||||
- **prompts.py docstring**: dated entry recording the 2026-09-15 interactive-test finding (stochastic "Ulaanbaatar" disclosed answer vs. clean deflection on re-run, Mongolia case) and owner decision (iii, 2026-09-14): disclosed answers acceptable + documented; gate (A8 revised) is the deterministic lever; options (i)/(ii) left open. Prompt strings untouched (diff is docstring-only).
|
||||
- **Prompt-lock pin**: new `tests/unit/test_prompt_lock.py` (4 tests) — sha256 + exact prefix/suffix + total length pinning `PERSONA` (706 B), `TOOLS_SECTION` (2273 B), the inline LOW `DEFLECT_MODE` body (259 B), and both `_base` lengths; any byte change fails.
|
||||
- **README**: added the rare disclosed-answer sentence to the Honest-deflection bullet (work item 2). Defect fix (task 01 leftovers): refreshed the now-stale A8 gate copy in the retrieval paragraph, env table (added `BOR_LEXICAL_SUPPORT_FLOOR` row), tuning bullet, and two spots in `app/rag/retriever.py`.
|
||||
- **Results**: `uv run pytest --cov=app --cov-report=term-missing` → 2378 passed, **99%** on `app/` (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors; `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → 4 passed in isolation.
|
||||
- **Decisions**: kept the quoted *"I haven't done anything like that"* openings intact — task 04 rewrites those; my gate-copy fixes there will fold into it.
|
||||
- **Next pending task**: `04_readme_deflection_copy.md`.
|
||||
+102
@@ -0,0 +1,102 @@
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 99%]
|
||||
.. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 208 1 99%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 94 0 100%
|
||||
app/api/docs.py 156 1 99%
|
||||
app/api/git_sources.py 232 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 186 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 66 0 100%
|
||||
app/models.py 128 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 317 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 123 0 100%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 215 3 99%
|
||||
app/rag/llm.py 243 1 99%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 88 0 100%
|
||||
app/rag/retriever.py 172 3 98%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 327 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 4079 16 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
**Task 04 complete — README deflection copy refreshed (docs-only, no code changes).**
|
||||
|
||||
- Rewrote README intro (L11): now "it admits it — and offers 2-3 concrete alternative questions about things it **does** have notes on" (no exact-copy promise; the mandated opening was removed in the 2026-08-22 prompt revision).
|
||||
- Rewrote Troubleshooting "Honest deflection" entry (L579): amber-bubble reference no longer quotes the removed opening; kept the accurate amber-banner + "Maybe try" chips description.
|
||||
- Third stale occurrence (L588, the "reply opens with…" clause) found via the required README grep — updated to admit + 2-3 concrete alternatives. `grep -rn "haven't done anything" README.md` → no hits (exit 1).
|
||||
- Tests grep: all `haven't done anything` hits in `tests/` are mock-LLM/stub-LLM *fixture* copies (the mock's own reply, which already matches the new contract; `DEFLECT_PHRASE` is documented as "the mock's deflection answer must match this") — none assert the app/prompt copy, so no test changes were needed.
|
||||
- `uv run pytest` → 2378 passed.
|
||||
- `uv run pytest --cov=app` → TOTAL 99% (>90% gate).
|
||||
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors.
|
||||
- `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → 4 passed (phase E2E green in isolation; last task of phase 112).
|
||||
- No git add/commit (left in working tree for the harness phase commit).
|
||||
- Next pending task: `.agents/phases/todo/113_source_chip_quality/01_usefulness_bar_sources.md`
|
||||
+102
@@ -0,0 +1,102 @@
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 99%]
|
||||
.. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 208 1 99%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 94 0 100%
|
||||
app/api/docs.py 156 1 99%
|
||||
app/api/git_sources.py 232 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 186 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 66 0 100%
|
||||
app/models.py 128 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 317 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 206 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 123 0 100%
|
||||
app/rag/git_sources.py 14 0 100%
|
||||
app/rag/importer.py 215 3 99%
|
||||
app/rag/llm.py 243 1 99%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/prompts.py 88 0 100%
|
||||
app/rag/retriever.py 172 3 98%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 24 0 100%
|
||||
app/schemas.py 327 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 4079 16 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
Reference in New Issue
Block a user