phase: 178_split_thinking_levels
## Phase 178 — final verification pass: 100% green
**Verified (all 5 tasks complete in `complete/178_split_thinking_levels/`):**
- `config.py` rename (`llm_chat_thinking_level` default `low`) + new `llm_summary_thinking_level` (default `""`); two `CATALOG` rows; `.env.example` replaced; no stale old-key references (only intentional negative pins in tests)
- Wire split: `chat_stream()` → chat level, one-shot `chat()` → summary level (absent `extra_body` when empty); per-turn line reads chat level only
**Gates:**
- `uv run pytest -k "config and thinking" -v` → 7 passed (defaults + env overrides both knobs)
- `uv run pytest` (unit + integration) → 4170 tests, exit 0; `--cov=app` → **99%** (>90% ✓)
- `uv run ruff check .` clean; `uv run pyright` clean
- `uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov` → 3 passed in isolation
- `test_admin_settings_panel.py` + `test_admin_settings_layout.py` → 17 passed in isolation
**Completion criteria:** all 6 test/doc criteria ✅ (admin GET lists both keys, old key gone — integration + E2E pins); commit + phase-dir move = harness's job.
**Documentation:** README.md — both new env vars correctly documented in the LLM (aipi) table with defaults/wire-mapping/off-rule, defaults match `config.py`, all `.env.example` vars covered: verified consistent, no changes needed. AGENTS.md — all referenced files/commands verified to exist, no phase-178 impact (no UI code change, A4): verified consistent, no changes needed.
**Decisions:** none required — no live changes made (E2E booted its own app subprocess against the existing DB).
**Next pending phase:** none — 178 is the last phase in `todo/`.
This commit is contained in:
@@ -0,0 +1,532 @@
|
||||
"""Phase 178 E2E (Playwright): the SPLIT thinking levels — a Chat
|
||||
thinking level row and a Summary thinking level row in the admin
|
||||
Settings panel — the owner's acceptance for TWO independently
|
||||
tunable, pi-style LLM reasoning knobs, proven in a real browser.
|
||||
|
||||
Source: ``TODO.md`` L3 — "Need a separate summary thinking level and
|
||||
chat thinking level. Default thinking level for summary should be
|
||||
none. Chat should stay at low"
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov
|
||||
|
||||
Test → contract mapping (one story, one phase, one isolated file):
|
||||
|
||||
1. ``test_both_rows_render_free_form_with_defaults`` — the rows:
|
||||
the ``LLM (aipi)`` group band is visible (the phase-171 band
|
||||
contract), a row labelled **Chat thinking level** with the
|
||||
``BOR_LLM_CHAT_THINKING_LEVEL`` env-var display shows ``low``
|
||||
with the ``env`` badge (chat stays at low, TODO L3), a row
|
||||
labelled **Summary thinking level** with the
|
||||
``BOR_LLM_SUMMARY_THINKING_LEVEL`` display shows an **empty**
|
||||
input with the ``env`` badge (the TODO's "none" = the phase-172
|
||||
off switch, LOCKED A2 — the parameter is ABSENT, never a
|
||||
literal "none"), BOTH inputs are ``type="text"`` (free-form —
|
||||
NOT ``number``/``select``, the A4 contract: no allowlist, the
|
||||
endpoint is the source of truth), the ``llm_chat_model`` row's
|
||||
input is the same control (the "like chat model" reference, one
|
||||
assertion pair), and ``tr[data-key="llm_thinking_level"]`` is
|
||||
**absent** (the old phase-172 key is gone, LOCKED A1).
|
||||
2. ``test_saved_levels_persist_without_restart`` — the save (the
|
||||
phase-168 §7.4 Save lifecycle per row: dirty → save → ``DB``
|
||||
badge): ``think-soft`` (a free-form custom — NOT a known pi
|
||||
level, A4) is typed into the summary row and ``high`` into the
|
||||
chat row (the dirty hints appear), then BOTH are saved; the
|
||||
API catalogue (``_catalog_item``) AND the raw ``app_settings``
|
||||
rows carry the saved values for EACH key independently (the
|
||||
independence pin — the two rows' controls are distinct and the
|
||||
saved values do not cross), and a reload of ``/settings.html``
|
||||
still shows ``think-soft`` / ``high`` with the ``DB`` badges
|
||||
(no restart, the phase-168 contract).
|
||||
3. ``test_reset_returns_to_env_defaults`` — the per-row Reset (the
|
||||
per-row ``DELETE``): after the saves, both Resets are live;
|
||||
clicking the summary Reset lands the value at **empty** with
|
||||
the ``env`` badge and clicking the chat Reset lands ``low``
|
||||
with the ``env`` badge (the overlay reset, task 02's contract,
|
||||
in the browser); the status lines show the house "… is back to
|
||||
the environment value." toast for each.
|
||||
|
||||
DB isolation: the shared e2e Postgres keeps ``app_settings``
|
||||
(the overlay rows — a leftover row would retune every later
|
||||
suite's app through the conftest pin's failure mode),
|
||||
``ui_settings`` (the single row the caching middleware reads
|
||||
for EVERY served page) and ``api_tokens`` rows across suites.
|
||||
An autouse fixture truncates ``app_settings`` and
|
||||
``ui_settings`` and deletes the ``e2e-``-labeled tokens before
|
||||
AND after every test (never a TRUNCATE on ``api_tokens`` — the
|
||||
shared DB may hold the owner's real tokens), and — because the
|
||||
direct SQL truncate does NOT invalidate the app's settings cache
|
||||
(the admin API is the only mutation surface that does) — follows
|
||||
it with the app's own clear-all (``DELETE /api/admin/settings``;
|
||||
without it, a test that types the SAME value its predecessor
|
||||
saved sees the stale value as the served one, registers no dirty
|
||||
state, and the Save button stays disabled). The module teardown
|
||||
pins ``app_settings`` empty. The module app opts BACK IN over
|
||||
the conftest's global ``BOR_SETTINGS_DB=0`` pin (``env["BOR_
|
||||
SETTINGS_DB"] = "1"`` — the ``test_pictorial_pdf.py`` opt-in
|
||||
pattern) and pins ``BOR_LLM_CHAT_THINKING_LEVEL=low`` AND
|
||||
``BOR_LLM_SUMMARY_THINKING_LEVEL=`` (the empty string SET — so
|
||||
both badges read ``env``, not ``default``, and an operator's
|
||||
local ``.env`` cannot leak a different level into the suite;
|
||||
the house leak-guard pattern) together with the "like chat
|
||||
model" reference (``BOR_LLM_CHAT_MODEL``). A module-scope ``.env``
|
||||
mtime guard pins LOCKED A1: no code path the suite drives opens
|
||||
the env file for writing.
|
||||
|
||||
ASSUMPTION A5 (phase-locked): the per-turn-line assertion
|
||||
(``thinking=<chat level>``) and the wire split (``chat_stream()``
|
||||
→ chat level, ``chat()`` → summary level) live in the
|
||||
unit/integration layer (tasks 03/04) — this suite proves the
|
||||
settings surface only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
APP_PORT,
|
||||
REPO,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
#: The chat row's saved value (test 2) — a known pi level, the
|
||||
#: operator's "bump the chat turn" case.
|
||||
SAVED_CHAT_LEVEL = "high"
|
||||
#: The summary row's saved value (test 2) — NOT a known pi level
|
||||
#: (proves the A4 contract: no allowlist, the endpoint is the
|
||||
#: source of truth).
|
||||
SAVED_SUMMARY_LEVEL = "think-soft"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-module app env (the phase-168 tuning pattern)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _boot_app() -> subprocess.Popen[bytes]:
|
||||
"""The one module app: the shared env block + the phase-178
|
||||
pins, with the overlay opted BACK IN (``BOR_SETTINGS_DB=1``)."""
|
||||
global _MOCK_PORT
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{_MOCK_PORT}/v1"
|
||||
)
|
||||
# The E2E-calibrated gate pins (the conftest pattern — the mock's
|
||||
# token-overlap distribution, explicit so an operator's local
|
||||
# .env cannot leak a different gate into the suite's app).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
Settings.model_fields["suggestions"].default
|
||||
)
|
||||
# Leak guards: the branding base and the "like chat model"
|
||||
# reference row stay the CODE defaults (derived from the class
|
||||
# fields, never drift from app/config.py).
|
||||
env["BOR_APP_NAME"] = Settings.model_fields["app_name"].default
|
||||
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_LLM_CHAT_MODEL"] = Settings.model_fields["llm_chat_model"].default
|
||||
# Phase 168 (the panel suite's pattern): the module opts BACK IN
|
||||
# over the conftest's global BOR_SETTINGS_DB=0 pin (the
|
||||
# test_pictorial_pdf.py pattern) — the rows' save/reset
|
||||
# contract needs the overlay on.
|
||||
env["BOR_SETTINGS_DB"] = "1"
|
||||
# Phase 178 (task 05): pin BOTH thinking levels to their CODE
|
||||
# DEFAULTS — each env var is SET (so the badge reads ``env``,
|
||||
# not ``default``) and an operator's local .env cannot leak a
|
||||
# different level into the suite's app. The summary level is
|
||||
# set to EMPTY (the TODO's "none" = the off switch, A2).
|
||||
env["BOR_LLM_CHAT_THINKING_LEVEL"] = Settings.model_fields[
|
||||
"llm_chat_thinking_level"
|
||||
].default
|
||||
env["BOR_LLM_SUMMARY_THINKING_LEVEL"] = Settings.model_fields[
|
||||
"llm_summary_thinking_level"
|
||||
].default
|
||||
return subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
|
||||
|
||||
def _stop_app(proc: subprocess.Popen[bytes]) -> None:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int) -> Iterator[str]:
|
||||
"""The real app under test — per-module env: BOTH thinking
|
||||
levels pinned to their code defaults (chat ``low``, summary
|
||||
empty = the A2 off switch) and ``BOR_SETTINGS_DB=1`` (the
|
||||
module opts back in over the conftest pin)."""
|
||||
global _MOCK_PORT
|
||||
_MOCK_PORT = mock_llm
|
||||
proc = _boot_app()
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
_stop_app(proc)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB isolation + helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _clean_settings_state() -> None:
|
||||
"""Fresh settings + theme + token state per test: truncate
|
||||
``app_settings`` (a leftover row is the cross-suite poison this
|
||||
phase's overlay contract exists to prevent — the rows are wiped
|
||||
between tests) and the single-row ``ui_settings`` (the
|
||||
middleware reads it for EVERY page), and delete this suite's
|
||||
issued tokens (label-scoped on ``e2e-`` — never a TRUNCATE: the
|
||||
shared DB may hold the owner's real tokens)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE app_settings"))
|
||||
db.execute(text("TRUNCATE ui_settings"))
|
||||
db.execute(text("DELETE FROM api_tokens WHERE label LIKE 'e2e-%'"))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _invalidate_app_cache(app_url: str) -> None:
|
||||
"""The direct SQL truncate does NOT invalidate the app's
|
||||
``get_settings()`` lru-cache (the admin API is the only
|
||||
mutation surface that invalidates — the phase-168 contract)
|
||||
— so after the truncate, hit the app's own clear-all
|
||||
(``DELETE /api/admin/settings``): zero rows to delete, but the
|
||||
invalidate call rebuilds the cache env-only. Without it, a test
|
||||
that types the SAME value its predecessor saved sees the STALE
|
||||
value as the served one (``markDirty`` finds no diff, the Save
|
||||
button stays disabled)."""
|
||||
client = _admin_client(app_url)
|
||||
r = client.delete(f"{app_url}/api/admin/settings", timeout=30.0)
|
||||
assert r.status_code == 200, f"clear-all → {r.status_code}: {r.text[:200]}"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(app_server: str, db_ready: None) -> Iterator[None]:
|
||||
_clean_settings_state()
|
||||
_invalidate_app_cache(app_server)
|
||||
yield
|
||||
_clean_settings_state()
|
||||
_invalidate_app_cache(app_server)
|
||||
# The module's final state: app_settings EMPTY (the cross-suite
|
||||
# poison guard — the next suite's app must boot with no
|
||||
# overrides).
|
||||
with SessionLocal() as db:
|
||||
assert db.execute(text("SELECT count(*) FROM app_settings")).scalar() == 0
|
||||
|
||||
|
||||
@pytest.fixture(scope="module", autouse=True)
|
||||
def _env_never_written() -> Iterator[None]:
|
||||
"""LOCKED A1, mtime-pinned: no code path the suite drives opens
|
||||
``.env`` for writing (the file is gitignored — a ``git status``
|
||||
check cannot see it, so ``stat``'s mtime is the assertion)."""
|
||||
env_file = REPO / ".env"
|
||||
before = env_file.stat().st_mtime_ns if env_file.exists() else None
|
||||
yield
|
||||
after = env_file.stat().st_mtime_ns if env_file.exists() else None
|
||||
assert before == after, "the app under test must never write .env"
|
||||
|
||||
|
||||
def _cookies(page: Page) -> dict[str, str]:
|
||||
"""The session cookies the browser context holds (the test's API
|
||||
side sees exactly what that browser sees)."""
|
||||
return {
|
||||
c["name"]: c["value"]
|
||||
for c in page.context.cookies()
|
||||
if "name" in c and "value" in c
|
||||
}
|
||||
|
||||
|
||||
def _admin_client(app_url: str) -> httpx.Client:
|
||||
"""The API side of the form login (the 204 + the signed session
|
||||
cookie in the jar)."""
|
||||
client = httpx.Client(timeout=60.0)
|
||||
r = client.post(f"{app_url}/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, f"admin login → {r.status_code}: {r.text[:120]}"
|
||||
return client
|
||||
|
||||
|
||||
def _catalog_item(client: httpx.Client, app_url: str, key: str) -> dict[str, Any]:
|
||||
r = client.get(f"{app_url}/api/admin/settings", timeout=30.0)
|
||||
assert r.status_code == 200, r.text[:200]
|
||||
for item in r.json()["items"]:
|
||||
if item["key"] == key:
|
||||
return item
|
||||
raise AssertionError(f"{key} missing from the catalogue")
|
||||
|
||||
|
||||
def _raw_value(key: str) -> str | None:
|
||||
"""The raw ``app_settings`` row for ``key`` (the overlay's D1
|
||||
env-native string form), or None when no override is stored."""
|
||||
with SessionLocal() as db:
|
||||
return db.execute(
|
||||
text("SELECT value FROM app_settings WHERE key = :k"), {"k": key}
|
||||
).scalar_one_or_none()
|
||||
|
||||
|
||||
def _raw_count(key: str) -> int:
|
||||
with SessionLocal() as db:
|
||||
return db.execute(
|
||||
text("SELECT count(*) FROM app_settings WHERE key = :k"), {"k": key}
|
||||
).scalar_one()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Both rows render free-form with defaults (chat ``low``,
|
||||
# summary empty = the A2 off switch; the old key is gone)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_both_rows_render_free_form_with_defaults(
|
||||
page: Page, app_url: str, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/settings.html")
|
||||
expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
|
||||
expect(page.locator("#settings-disabled")).to_be_hidden() # the overlay is on
|
||||
client = _admin_client(app_url)
|
||||
|
||||
# The LLM (aipi) group band is visible (the phase-171 band
|
||||
# contract — both rows sit in it)…
|
||||
expect(page.locator("tr.settings-group-row", has_text="LLM (aipi)")).to_be_visible()
|
||||
|
||||
# The CHAT row: the label, the env-var display (the catalogue
|
||||
# pattern), the free-form text control (A4: type="text" — NOT
|
||||
# number/select; no allowlist, the endpoint is the source of
|
||||
# truth)…
|
||||
chat_row = page.locator('tr[data-key="llm_chat_thinking_level"]')
|
||||
expect(chat_row).to_be_visible()
|
||||
expect(chat_row.locator(".cell-clip")).to_have_text("Chat thinking level")
|
||||
expect(chat_row.locator(".settings-env")).to_have_text("BOR_LLM_CHAT_THINKING_LEVEL")
|
||||
assert chat_row.locator(".settings-control").get_attribute("type") == "text"
|
||||
# …chat stays at low (TODO L3): the env-pinned ``low`` with the
|
||||
# env badge (the env var is set — no app_settings row, no
|
||||
# code-default badge).
|
||||
expect(chat_row.locator(".settings-control")).to_have_value("low")
|
||||
expect(chat_row.locator(".settings-src")).to_have_text("env")
|
||||
expect(chat_row.locator(".settings-reset")).to_be_disabled()
|
||||
|
||||
# The SUMMARY row: the label, the env-var display, the free-form
|
||||
# text control…
|
||||
summary_row = page.locator('tr[data-key="llm_summary_thinking_level"]')
|
||||
expect(summary_row).to_be_visible()
|
||||
expect(summary_row.locator(".cell-clip")).to_have_text("Summary thinking level")
|
||||
expect(summary_row.locator(".settings-env")).to_have_text(
|
||||
"BOR_LLM_SUMMARY_THINKING_LEVEL"
|
||||
)
|
||||
assert summary_row.locator(".settings-control").get_attribute("type") == "text"
|
||||
# …and the TODO's "none" = the off switch (A2): an EMPTY input
|
||||
# with the env badge (the env var is SET to the empty string —
|
||||
# the badge reads env, not default).
|
||||
expect(summary_row.locator(".settings-control")).to_have_value("")
|
||||
expect(summary_row.locator(".settings-src")).to_have_text("env")
|
||||
expect(summary_row.locator(".settings-reset")).to_be_disabled()
|
||||
|
||||
# The "like chat model" reference: the llm_chat_model row's
|
||||
# input is the SAME type="text" control (one assertion pair
|
||||
# covering both thinking rows).
|
||||
chat_model_type = page.locator(
|
||||
'tr[data-key="llm_chat_model"] .settings-control'
|
||||
).get_attribute("type")
|
||||
assert chat_model_type == "text"
|
||||
assert (
|
||||
chat_row.locator(".settings-control").get_attribute("type")
|
||||
== summary_row.locator(".settings-control").get_attribute("type")
|
||||
== chat_model_type
|
||||
)
|
||||
|
||||
# The old phase-172 key is GONE (LOCKED A1 — the rename, not
|
||||
# add-on): no row for llm_thinking_level anywhere in the panel.
|
||||
assert page.locator('tr[data-key="llm_thinking_level"]').count() == 0
|
||||
|
||||
# The server agrees with the panel: both defaults, both sourced
|
||||
# from the env (the module pins), stored raw on no row.
|
||||
chat_item = _catalog_item(client, app_url, "llm_chat_thinking_level")
|
||||
assert chat_item["source"] == "env" and chat_item["value"] == "low"
|
||||
summary_item = _catalog_item(client, app_url, "llm_summary_thinking_level")
|
||||
assert summary_item["source"] == "env" and summary_item["value"] == ""
|
||||
assert _raw_count("llm_chat_thinking_level") == 0
|
||||
assert _raw_count("llm_summary_thinking_level") == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. The saves: dirty → save → DB badge, EACH key's value persists
|
||||
# independently and survives a reload WITHOUT a restart
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_saved_levels_persist_without_restart(
|
||||
page: Page, app_url: str, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/settings.html")
|
||||
expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
|
||||
client = _admin_client(app_url)
|
||||
|
||||
# The §7.4 Save lifecycle: type the summary's free-form custom
|
||||
# value (A4 — NOT a known pi level: no allowlist, the endpoint
|
||||
# is the source of truth) and the chat's bump — two dirty
|
||||
# rows, the hints appear…
|
||||
summary_row = page.locator('tr[data-key="llm_summary_thinking_level"]')
|
||||
chat_row = page.locator('tr[data-key="llm_chat_thinking_level"]')
|
||||
summary_row.locator(".settings-control").fill(SAVED_SUMMARY_LEVEL)
|
||||
chat_row.locator(".settings-control").fill(SAVED_CHAT_LEVEL)
|
||||
expect(page.locator("#settings-dirty")).to_have_text("2 unsaved changes")
|
||||
expect(page.locator("#settings-save")).to_be_enabled()
|
||||
|
||||
# Save: the status line, then BOTH badges flip to DB (the
|
||||
# re-GET settled the rows).
|
||||
page.click("#settings-save")
|
||||
expect(page.locator("#settings-status")).to_contain_text(
|
||||
"Saved 2 settings.", timeout=30_000
|
||||
)
|
||||
expect(summary_row.locator(".settings-src")).to_have_text("DB")
|
||||
expect(chat_row.locator(".settings-src")).to_have_text("DB")
|
||||
|
||||
# The effective values are the saved ones for EACH key
|
||||
# independently (the independence pin — the two rows' controls
|
||||
# are distinct and the saved values do not cross): the API
|
||||
# catalogue (what the LLM layer reads per request via
|
||||
# get_settings)…
|
||||
summary_item = _catalog_item(client, app_url, "llm_summary_thinking_level")
|
||||
assert summary_item["source"] == "db" and summary_item["value"] == SAVED_SUMMARY_LEVEL
|
||||
chat_item = _catalog_item(client, app_url, "llm_chat_thinking_level")
|
||||
assert chat_item["source"] == "db" and chat_item["value"] == SAVED_CHAT_LEVEL
|
||||
# …and the raw app_settings rows are stored RAW (the overlay's
|
||||
# D1 env-native string form), one per key.
|
||||
assert _raw_value("llm_summary_thinking_level") == SAVED_SUMMARY_LEVEL
|
||||
assert _raw_value("llm_chat_thinking_level") == SAVED_CHAT_LEVEL
|
||||
|
||||
# No restart: the DB values survive a reload of /settings.html
|
||||
# (the panel re-reads the settled server state), the summary
|
||||
# row's custom value and the chat row's bump each still on
|
||||
# their OWN row with the DB badge still on.
|
||||
page.goto(app_url + "/settings.html")
|
||||
expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
|
||||
expect(
|
||||
page.locator('tr[data-key="llm_summary_thinking_level"] .settings-control')
|
||||
).to_have_value(SAVED_SUMMARY_LEVEL, timeout=15_000)
|
||||
expect(
|
||||
page.locator('tr[data-key="llm_summary_thinking_level"] .settings-src')
|
||||
).to_have_text("DB")
|
||||
expect(
|
||||
page.locator('tr[data-key="llm_chat_thinking_level"] .settings-control')
|
||||
).to_have_value(SAVED_CHAT_LEVEL)
|
||||
expect(
|
||||
page.locator('tr[data-key="llm_chat_thinking_level"] .settings-src')
|
||||
).to_have_text("DB")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. The Resets: each row goes back to ITS environment default (the
|
||||
# per-row DELETE — task 02's contract, in the browser)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_reset_returns_to_env_defaults(page: Page, app_url: str, db_ready: None) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/settings.html")
|
||||
expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
|
||||
client = _admin_client(app_url)
|
||||
|
||||
# Establish saved overrides first (the Resets are disabled with
|
||||
# nothing to reset — the phase-168 render contract): save a
|
||||
# value on each row so both Resets go live…
|
||||
summary_row = page.locator('tr[data-key="llm_summary_thinking_level"]')
|
||||
chat_row = page.locator('tr[data-key="llm_chat_thinking_level"]')
|
||||
summary_row.locator(".settings-control").fill(SAVED_SUMMARY_LEVEL)
|
||||
chat_row.locator(".settings-control").fill(SAVED_CHAT_LEVEL)
|
||||
page.click("#settings-save")
|
||||
expect(page.locator("#settings-status")).to_contain_text(
|
||||
"Saved 2 settings.", timeout=30_000
|
||||
)
|
||||
# The re-GET settles the rows (DB badges) before the Resets go
|
||||
# live (the phase-168 render contract — the button tracks the
|
||||
# SETTLED source, not the click).
|
||||
expect(summary_row.locator(".settings-src")).to_have_text("DB")
|
||||
expect(chat_row.locator(".settings-src")).to_have_text("DB")
|
||||
expect(summary_row.locator(".settings-reset")).to_be_enabled()
|
||||
expect(chat_row.locator(".settings-reset")).to_be_enabled()
|
||||
|
||||
# The summary Reset (the per-row DELETE): the status line…
|
||||
summary_row.locator(".settings-reset").click()
|
||||
expect(page.locator("#settings-status")).to_contain_text(
|
||||
"BOR_LLM_SUMMARY_THINKING_LEVEL is back to the environment value.",
|
||||
timeout=30_000,
|
||||
)
|
||||
# …the value lands at EMPTY (the env var is set to the empty
|
||||
# string — the env badge, not the code-default badge) and the
|
||||
# row's Reset goes live-free again.
|
||||
expect(summary_row.locator(".settings-control")).to_have_value("")
|
||||
expect(summary_row.locator(".settings-src")).to_have_text("env")
|
||||
expect(summary_row.locator(".settings-reset")).to_be_disabled()
|
||||
|
||||
# The chat Reset (the per-row DELETE): the status line…
|
||||
chat_row.locator(".settings-reset").click()
|
||||
expect(page.locator("#settings-status")).to_contain_text(
|
||||
"BOR_LLM_CHAT_THINKING_LEVEL is back to the environment value.",
|
||||
timeout=30_000,
|
||||
)
|
||||
# …the value reads ``low`` again (chat stays at low, TODO L3)
|
||||
# with the env badge, and the row's Reset is disabled.
|
||||
expect(chat_row.locator(".settings-control")).to_have_value("low")
|
||||
expect(chat_row.locator(".settings-src")).to_have_text("env")
|
||||
expect(chat_row.locator(".settings-reset")).to_be_disabled()
|
||||
|
||||
# The server agrees: both overrides are gone, the effective
|
||||
# values are the env defaults, and no raw row survives.
|
||||
summary_item = _catalog_item(client, app_url, "llm_summary_thinking_level")
|
||||
assert summary_item["source"] == "env" and summary_item["value"] == ""
|
||||
chat_item = _catalog_item(client, app_url, "llm_chat_thinking_level")
|
||||
assert chat_item["source"] == "env" and chat_item["value"] == "low"
|
||||
assert _raw_count("llm_summary_thinking_level") == 0
|
||||
assert _raw_count("llm_chat_thinking_level") == 0
|
||||
@@ -1,405 +0,0 @@
|
||||
"""Phase 172 E2E (Playwright): the Thinking level row in the admin Settings
|
||||
panel — the owner's acceptance for a pi-style, operator-tunable LLM
|
||||
reasoning knob, proven in a real browser.
|
||||
|
||||
Source: ``TODO.md`` L3 — "Brain of Reese needs to support thinking levels
|
||||
like pi does. should be a free-form text entry like chat model. Default
|
||||
should be 'low'. Update .env and admin settings."
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_thinking_level.py -v --no-cov
|
||||
|
||||
Test → contract mapping (one story, one phase, one isolated file):
|
||||
|
||||
1. ``test_row_renders_free_form_with_default_low`` — the row: the
|
||||
``LLM (aipi)`` group band is visible (the phase-171 band
|
||||
contract), a row labelled **Thinking level** with the
|
||||
``BOR_LLM_THINKING_LEVEL`` env-var display (the catalogue
|
||||
pattern) is present, its input is a **free-form text** input
|
||||
(``type="text"`` — NOT ``number``/``select``, the A4
|
||||
contract), it shows ``low`` with the ``env`` badge, and the
|
||||
``llm_chat_model`` row's input is the same ``type="text"``
|
||||
control (the "like chat model" reference, in one assertion
|
||||
pair).
|
||||
2. ``test_saved_level_persists_without_restart`` — the save
|
||||
(the phase-168 §7.4 Save lifecycle: dirty → save → ``DB``
|
||||
badge): ``high`` is typed (the dirty hint appears), then the
|
||||
free-form custom value ``think-soft`` is typed once (no
|
||||
allowlist — A4) and saved; the effective value reads
|
||||
``think-soft`` (the API catalogue AND the raw
|
||||
``app_settings`` row), and a reload of ``/settings.html``
|
||||
still shows ``think-soft`` with the ``DB`` badge (no
|
||||
restart, the phase-168 contract).
|
||||
3. ``test_reset_returns_to_env_default`` — the per-row Reset
|
||||
(the per-row ``DELETE``): after a save, the Reset is live;
|
||||
clicking it lands the status line and the value reads
|
||||
``low`` again with the ``env`` badge (the overlay reset,
|
||||
task 02's contract, in the browser).
|
||||
|
||||
DB isolation: the shared e2e Postgres keeps ``app_settings``
|
||||
(the overlay rows — a leftover row would retune every later
|
||||
suite's app through the conftest pin's failure mode),
|
||||
``ui_settings`` (the single row the caching middleware reads
|
||||
for EVERY served page) and ``api_tokens`` rows across suites.
|
||||
An autouse fixture truncates ``app_settings`` and
|
||||
``ui_settings`` and deletes the ``e2e-``-labeled tokens before
|
||||
AND after every test (never a TRUNCATE on ``api_tokens`` — the
|
||||
shared DB may hold the owner's real tokens); the module teardown
|
||||
pins ``app_settings`` empty. The module app opts BACK IN over
|
||||
the conftest's global ``BOR_SETTINGS_DB=0`` pin (``env["BOR_
|
||||
SETTINGS_DB"] = "1"`` — the ``test_pictorial_pdf.py`` opt-in
|
||||
pattern) and pins ``BOR_LLM_THINKING_LEVEL`` (and the
|
||||
"like chat model" reference, ``BOR_LLM_CHAT_MODEL``) to their
|
||||
CODE DEFAULTS — the env var is set (so the badge reads ``env``,
|
||||
not ``default``) and an operator's local ``.env`` cannot leak a
|
||||
different level into the suite's app (the house leak-guard
|
||||
pattern). A module-scope ``.env`` mtime guard pins LOCKED A1:
|
||||
no code path the suite drives opens the env file for writing.
|
||||
|
||||
ASSUMPTION A6 (phase-locked): the per-turn-line assertion
|
||||
(``thinking=<value>``) lives in the unit/integration layer
|
||||
(task 04) — the mock LLM turn does not exercise the wire, and
|
||||
this suite proves the settings surface only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
APP_PORT,
|
||||
REPO,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
#: The save's custom free-form value (test 2) — NOT a known pi
|
||||
#: level (proves the A4 contract: no allowlist, the endpoint is
|
||||
#: the source of truth).
|
||||
SAVED_LEVEL = "think-soft"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-module app env (the phase-168 tuning pattern)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _boot_app() -> subprocess.Popen[bytes]:
|
||||
"""The one module app: the shared env block + the phase-172 pins,
|
||||
with the overlay opted BACK IN (``BOR_SETTINGS_DB=1``)."""
|
||||
global _MOCK_PORT
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{_MOCK_PORT}/v1"
|
||||
)
|
||||
# The E2E-calibrated gate pins (the conftest pattern — the mock's
|
||||
# token-overlap distribution, explicit so an operator's local
|
||||
# .env cannot leak a different gate into the suite's app).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
Settings.model_fields["suggestions"].default
|
||||
)
|
||||
# Leak guards: the branding base and the "like chat model"
|
||||
# reference row stay the CODE defaults (derived from the class
|
||||
# fields, never drift from app/config.py).
|
||||
env["BOR_APP_NAME"] = Settings.model_fields["app_name"].default
|
||||
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_LLM_CHAT_MODEL"] = Settings.model_fields["llm_chat_model"].default
|
||||
# Phase 168 (the panel suite's pattern): the module opts BACK IN
|
||||
# over the conftest's global BOR_SETTINGS_DB=0 pin (the
|
||||
# test_pictorial_pdf.py pattern) — the row's save/reset
|
||||
# contract needs the overlay on.
|
||||
env["BOR_SETTINGS_DB"] = "1"
|
||||
# Phase 172 (task 05): pin the thinking level to the CODE
|
||||
# DEFAULT — the env var is SET (so the badge reads ``env``, not
|
||||
# ``default``) and an operator's local .env cannot leak a
|
||||
# different level into the suite's app.
|
||||
env["BOR_LLM_THINKING_LEVEL"] = Settings.model_fields[
|
||||
"llm_thinking_level"
|
||||
].default
|
||||
return subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
|
||||
|
||||
def _stop_app(proc: subprocess.Popen[bytes]) -> None:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int) -> Iterator[str]:
|
||||
"""The real app under test — per-module env: the thinking level
|
||||
pinned to the code default ``low`` and ``BOR_SETTINGS_DB=1``
|
||||
(the module opts back in over the conftest pin)."""
|
||||
global _MOCK_PORT
|
||||
_MOCK_PORT = mock_llm
|
||||
proc = _boot_app()
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
_stop_app(proc)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB isolation + helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _clean_settings_state() -> None:
|
||||
"""Fresh settings + theme + token state per test: truncate
|
||||
``app_settings`` (a leftover row is the cross-suite poison this
|
||||
phase's overlay contract exists to prevent — the new key's rows
|
||||
are wiped between tests) and the single-row ``ui_settings``
|
||||
(the middleware reads it for EVERY page), and delete this
|
||||
suite's issued tokens (label-scoped on ``e2e-`` — never a
|
||||
TRUNCATE: the shared DB may hold the owner's real tokens)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE app_settings"))
|
||||
db.execute(text("TRUNCATE ui_settings"))
|
||||
db.execute(text("DELETE FROM api_tokens WHERE label LIKE 'e2e-%'"))
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
_clean_settings_state()
|
||||
yield
|
||||
_clean_settings_state()
|
||||
# The module's final state: app_settings EMPTY (the cross-suite
|
||||
# poison guard — the next suite's app must boot with no
|
||||
# overrides).
|
||||
with SessionLocal() as db:
|
||||
assert db.execute(text("SELECT count(*) FROM app_settings")).scalar() == 0
|
||||
|
||||
|
||||
@pytest.fixture(scope="module", autouse=True)
|
||||
def _env_never_written() -> Iterator[None]:
|
||||
"""LOCKED A1, mtime-pinned: no code path the suite drives opens
|
||||
``.env`` for writing (the file is gitignored — a ``git status``
|
||||
check cannot see it, so ``stat``'s mtime is the assertion)."""
|
||||
env_file = REPO / ".env"
|
||||
before = env_file.stat().st_mtime_ns if env_file.exists() else None
|
||||
yield
|
||||
after = env_file.stat().st_mtime_ns if env_file.exists() else None
|
||||
assert before == after, "the app under test must never write .env"
|
||||
|
||||
|
||||
def _cookies(page: Page) -> dict[str, str]:
|
||||
"""The session cookies the browser context holds (the test's API
|
||||
side sees exactly what that browser sees)."""
|
||||
return {
|
||||
c["name"]: c["value"]
|
||||
for c in page.context.cookies()
|
||||
if "name" in c and "value" in c
|
||||
}
|
||||
|
||||
|
||||
def _admin_client(app_url: str) -> httpx.Client:
|
||||
"""The API side of the form login (the 204 + the signed session
|
||||
cookie in the jar)."""
|
||||
client = httpx.Client(timeout=60.0)
|
||||
r = client.post(f"{app_url}/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, f"admin login → {r.status_code}: {r.text[:120]}"
|
||||
return client
|
||||
|
||||
|
||||
def _catalog_item(client: httpx.Client, app_url: str, key: str) -> dict[str, Any]:
|
||||
r = client.get(f"{app_url}/api/admin/settings", timeout=30.0)
|
||||
assert r.status_code == 200, r.text[:200]
|
||||
for item in r.json()["items"]:
|
||||
if item["key"] == key:
|
||||
return item
|
||||
raise AssertionError(f"{key} missing from the catalogue")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The row renders free-form with default low (the "like chat
|
||||
# model" reference, in one assertion pair)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_row_renders_free_form_with_default_low(page: Page, app_url: str, db_ready: None) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/settings.html")
|
||||
expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
|
||||
expect(page.locator("#settings-disabled")).to_be_hidden() # the overlay is on
|
||||
|
||||
# The LLM (aipi) group band is visible (the phase-171 band
|
||||
# contract — the row sits in it)…
|
||||
expect(page.locator("tr.settings-group-row", has_text="LLM (aipi)")).to_be_visible()
|
||||
# …and the row is present with its label…
|
||||
row = page.locator('tr[data-key="llm_thinking_level"]')
|
||||
expect(row).to_be_visible()
|
||||
expect(row.locator(".cell-clip")).to_have_text("Thinking level")
|
||||
# …and the env-var display (the catalogue pattern).
|
||||
expect(row.locator(".settings-env")).to_have_text("BOR_LLM_THINKING_LEVEL")
|
||||
|
||||
# The input is a FREE-FORM TEXT control (A4: type="text" — NOT
|
||||
# number/select; no allowlist, the endpoint is the source of
|
||||
# truth).
|
||||
assert row.locator(".settings-control").get_attribute("type") == "text"
|
||||
# The "like chat model" reference: the llm_chat_model row's
|
||||
# input is the SAME type="text" control (one assertion pair).
|
||||
thinking_type = row.locator(".settings-control").get_attribute("type")
|
||||
chat_type = page.locator(
|
||||
'tr[data-key="llm_chat_model"] .settings-control'
|
||||
).get_attribute("type")
|
||||
assert thinking_type == chat_type == "text"
|
||||
|
||||
# The default: the env-pinned ``low`` with the env badge (the
|
||||
# env var is set — no app_settings row, no code-default badge).
|
||||
expect(row.locator(".settings-control")).to_have_value("low")
|
||||
expect(row.locator(".settings-src")).to_have_text("env")
|
||||
# Nothing is saved yet: the row's Reset is disabled (no saved
|
||||
# override — the phase-168 render contract).
|
||||
expect(row.locator(".settings-reset")).to_be_disabled()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. The save: dirty → save → DB badge, the effective value is the
|
||||
# free-form custom, and it survives a reload WITHOUT a restart
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_saved_level_persists_without_restart(page: Page, app_url: str, db_ready: None) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/settings.html")
|
||||
expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
|
||||
client = _admin_client(app_url)
|
||||
|
||||
# The §7.4 Save lifecycle: type ``high`` — the dirty hint
|
||||
# appears (one unsaved change on the row)…
|
||||
row = page.locator('tr[data-key="llm_thinking_level"]')
|
||||
row.locator(".settings-control").fill("high")
|
||||
expect(page.locator("#settings-dirty")).to_have_text("1 unsaved changes")
|
||||
# …then type the free-form custom value once (A4 — NOT a known
|
||||
# pi level: no allowlist, the endpoint is the source of truth).
|
||||
row.locator(".settings-control").fill(SAVED_LEVEL)
|
||||
expect(page.locator("#settings-save")).to_be_enabled()
|
||||
|
||||
# Save: the status line, then the badge flips to DB (the
|
||||
# re-GET settled the row).
|
||||
page.click("#settings-save")
|
||||
expect(page.locator("#settings-status")).to_contain_text(
|
||||
"Saved 1 settings.", timeout=30_000
|
||||
)
|
||||
expect(row.locator(".settings-src")).to_have_text("DB")
|
||||
|
||||
# The effective value reads ``think-soft`` — the API catalogue
|
||||
# (what the LLM layer reads per request via get_settings)…
|
||||
item = _catalog_item(client, app_url, "llm_thinking_level")
|
||||
assert item["source"] == "db" and item["value"] == SAVED_LEVEL
|
||||
# …and the raw app_settings row is stored RAW (the overlay's
|
||||
# D1 env-native string form).
|
||||
with SessionLocal() as db:
|
||||
raw = db.execute(
|
||||
text("SELECT value FROM app_settings WHERE key = 'llm_thinking_level'")
|
||||
).scalar_one()
|
||||
assert raw == SAVED_LEVEL, f"the row must be stored raw (got {raw!r})"
|
||||
|
||||
# No restart: the DB value survives a reload of /settings.html
|
||||
# (the panel re-reads the settled server state) with the DB
|
||||
# badge still on.
|
||||
page.goto(app_url + "/settings.html")
|
||||
expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
|
||||
expect(
|
||||
page.locator('tr[data-key="llm_thinking_level"] .settings-control')
|
||||
).to_have_value(SAVED_LEVEL, timeout=15_000)
|
||||
expect(
|
||||
page.locator('tr[data-key="llm_thinking_level"] .settings-src')
|
||||
).to_have_text("DB")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. The Reset: control goes back to the environment (the per-row
|
||||
# DELETE — task 02's contract, in the browser)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_reset_returns_to_env_default(page: Page, app_url: str, db_ready: None) -> None:
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/settings.html")
|
||||
expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
|
||||
client = _admin_client(app_url)
|
||||
|
||||
# Establish a saved override first (the Reset is disabled with
|
||||
# nothing to reset — the phase-168 render contract): save
|
||||
# ``high`` so the row's Reset goes live…
|
||||
row = page.locator('tr[data-key="llm_thinking_level"]')
|
||||
row.locator(".settings-control").fill("high")
|
||||
page.click("#settings-save")
|
||||
expect(page.locator("#settings-status")).to_contain_text(
|
||||
"Saved 1 settings.", timeout=30_000
|
||||
)
|
||||
expect(row.locator(".settings-src")).to_have_text("DB")
|
||||
expect(row.locator(".settings-reset")).to_be_enabled()
|
||||
|
||||
# The per-row Reset (the per-row DELETE): the status line…
|
||||
row.locator(".settings-reset").click()
|
||||
expect(page.locator("#settings-status")).to_contain_text(
|
||||
"BOR_LLM_THINKING_LEVEL is back to the environment value.", timeout=30_000
|
||||
)
|
||||
# …the value reads ``low`` again with the env badge (the env
|
||||
# var is set — not the code-default badge)…
|
||||
expect(row.locator(".settings-control")).to_have_value("low")
|
||||
expect(row.locator(".settings-src")).to_have_text("env")
|
||||
# …and the server agrees: the override is gone and the
|
||||
# effective value is the env value again.
|
||||
item = _catalog_item(client, app_url, "llm_thinking_level")
|
||||
assert item["source"] == "env" and item["value"] == "low"
|
||||
with SessionLocal() as db:
|
||||
assert (
|
||||
db.execute(
|
||||
text("SELECT count(*) FROM app_settings WHERE key = 'llm_thinking_level'")
|
||||
).scalar()
|
||||
== 0
|
||||
), "the DELETE must remove the row"
|
||||
@@ -518,11 +518,13 @@ def test_catalog_covers_every_settings_field(admin_client) -> None:
|
||||
assert set(CATALOG) == set(Settings.model_fields)
|
||||
|
||||
|
||||
def test_llm_thinking_level_row_contract(
|
||||
def test_llm_chat_thinking_level_row_contract(
|
||||
admin_client: TestClient, db: Session, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Phase 172 (task 02): the new ``llm_thinking_level`` row behaves
|
||||
exactly like ``llm_chat_model`` — the machinery is generic over
|
||||
"""Phase 178 (task 01; task 02 extends for the summary row): the
|
||||
renamed ``llm_chat_thinking_level`` row (the phase-172 knob, the
|
||||
rename per locked A1) behaves exactly like ``llm_chat_model`` —
|
||||
the machinery is generic over
|
||||
the catalogue (D4 / A5), so the per-key contract is: (a) present in
|
||||
the GET body (group ``LLM (aipi)``, ``str``, effective ``low``
|
||||
with no row in ``app_settings``, ``env_var`` named); (b) PUT stores
|
||||
@@ -538,20 +540,20 @@ def test_llm_thinking_level_row_contract(
|
||||
"""
|
||||
# A deterministic base (machine-independent, the kill-switch test's
|
||||
# idiom): the code default, no row in the table.
|
||||
monkeypatch.delenv("BOR_LLM_THINKING_LEVEL", raising=False)
|
||||
monkeypatch.delenv("BOR_LLM_CHAT_THINKING_LEVEL", raising=False)
|
||||
invalidate_settings()
|
||||
assert get_settings().llm_thinking_level == "low"
|
||||
assert get_settings().llm_chat_thinking_level == "low"
|
||||
|
||||
# (a) the row: group, type, env var, effective ``low`` with NO row
|
||||
# in app_settings, not secret, not restart-required (the LLM layer
|
||||
# reads the setting per request — live).
|
||||
r = admin_client.get(API)
|
||||
assert r.status_code == 200, r.text
|
||||
item = _item(r.json(), "llm_thinking_level")
|
||||
item = _item(r.json(), "llm_chat_thinking_level")
|
||||
assert item["group"] == "LLM (aipi)"
|
||||
assert item["label"] == "Thinking level"
|
||||
assert item["label"] == "Chat thinking level"
|
||||
assert item["type"] == "str"
|
||||
assert item["env_var"] == "BOR_LLM_THINKING_LEVEL"
|
||||
assert item["env_var"] == "BOR_LLM_CHAT_THINKING_LEVEL"
|
||||
assert item["secret"] is False
|
||||
assert item["restart_required"] is False
|
||||
assert item["value"] == "low"
|
||||
@@ -562,48 +564,150 @@ def test_llm_thinking_level_row_contract(
|
||||
|
||||
# (b) PUT stores the RAW string (the overlay's D1 — env-native
|
||||
# string form) and the response reports the new effective value.
|
||||
r = admin_client.put(f"{API}/llm_thinking_level", json={"value": "high"})
|
||||
r = admin_client.put(f"{API}/llm_chat_thinking_level", json={"value": "high"})
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["value"] == "high"
|
||||
assert r.json()["source"] == "db"
|
||||
assert r.json()["has_override"] is True
|
||||
assert r.json()["db_value"] == "high"
|
||||
assert db.execute(
|
||||
text("SELECT value FROM app_settings WHERE key = 'llm_thinking_level'")
|
||||
text("SELECT value FROM app_settings WHERE key = 'llm_chat_thinking_level'")
|
||||
).scalar() == "high" # stored raw, exactly as sent
|
||||
assert get_settings().llm_thinking_level == "high" # live, no restart
|
||||
assert get_settings().llm_chat_thinking_level == "high" # live, no restart
|
||||
|
||||
# (c) the EMPTY value is storable (the off switch, A2/A4 — the
|
||||
# API stores as-is; no allowlist, no value validation): the row
|
||||
# holds the empty string and the effective value is ``""``.
|
||||
r = admin_client.put(f"{API}/llm_thinking_level", json={"value": ""})
|
||||
r = admin_client.put(f"{API}/llm_chat_thinking_level", json={"value": ""})
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["value"] == ""
|
||||
assert r.json()["has_value"] is False # the effective value is empty
|
||||
assert db.execute(
|
||||
text("SELECT value FROM app_settings WHERE key = 'llm_thinking_level'")
|
||||
text("SELECT value FROM app_settings WHERE key = 'llm_chat_thinking_level'")
|
||||
).scalar() == "" # stored as-is (the off switch)
|
||||
assert get_settings().llm_thinking_level == ""
|
||||
assert get_settings().llm_chat_thinking_level == ""
|
||||
|
||||
# (d) DELETE: the row is gone and the effective value is back to
|
||||
# the env value (``low`` — the base pinned above).
|
||||
r = admin_client.delete(f"{API}/llm_thinking_level")
|
||||
r = admin_client.delete(f"{API}/llm_chat_thinking_level")
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["value"] == "low"
|
||||
assert r.json()["has_override"] is False
|
||||
assert db.execute(
|
||||
text("SELECT count(*) FROM app_settings WHERE key = 'llm_thinking_level'")
|
||||
text("SELECT count(*) FROM app_settings WHERE key = 'llm_chat_thinking_level'")
|
||||
).scalar() == 0 # the row is gone
|
||||
assert get_settings().llm_thinking_level == "low"
|
||||
assert get_settings().llm_chat_thinking_level == "low"
|
||||
|
||||
# (e) the non-admin 403 gate still applies (asserted once for this
|
||||
# key — the router-wide dependency is pinned on ALL four routes in
|
||||
# ``test_auth_matrix_all_four_routes``).
|
||||
anon = TestClient(fastapi_app)
|
||||
r = anon.put(f"{API}/llm_thinking_level", json={"value": "high"})
|
||||
r = anon.put(f"{API}/llm_chat_thinking_level", json={"value": "high"})
|
||||
assert r.status_code == 403 and r.json() == {"detail": "admin only"}
|
||||
|
||||
|
||||
def test_llm_summary_thinking_level_row_contract(
|
||||
admin_client: TestClient, db: Session, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Phase 178 (task 02): the ``llm_summary_thinking_level`` row (the
|
||||
new one-shot knob) behaves like the chat row — the machinery is
|
||||
generic over the catalogue (D4 / A4) — with the summary specifics:
|
||||
(a) present in the GET body (group ``LLM (aipi)``, ``str``, label
|
||||
``"Summary thinking level"``, ``env_var``
|
||||
``BOR_LLM_SUMMARY_THINKING_LEVEL``, effective ``""`` with no row in
|
||||
``app_settings``); (b) PUT stores the RAW string in
|
||||
``app_settings`` and the value resolves live WITHOUT a restart;
|
||||
(c) the EMPTY value is storable (the summary off switch, A2 — the
|
||||
API stores as-is, no allowlist, an empty stored value is never
|
||||
rejected as "no value"); (d) DELETE resets to the env value
|
||||
(``""`` — the code default); (e) the non-admin 403 gate applies;
|
||||
(f) INDEPENDENCE — the two overlays resolve independently (the
|
||||
whole point of the split): a saved summary value never changes the
|
||||
chat row's effective value.
|
||||
"""
|
||||
# A deterministic base (machine-independent, the kill-switch test's
|
||||
# idiom): BOTH knobs at their code defaults, no row in the table.
|
||||
monkeypatch.delenv("BOR_LLM_SUMMARY_THINKING_LEVEL", raising=False)
|
||||
monkeypatch.delenv("BOR_LLM_CHAT_THINKING_LEVEL", raising=False)
|
||||
invalidate_settings()
|
||||
assert get_settings().llm_summary_thinking_level == ""
|
||||
assert get_settings().llm_chat_thinking_level == "low"
|
||||
|
||||
# (a) the row: group, label, type, env var, effective ``""`` with NO
|
||||
# row in app_settings, not secret, not restart-required (the LLM
|
||||
# layer reads the setting per request — live).
|
||||
r = admin_client.get(API)
|
||||
assert r.status_code == 200, r.text
|
||||
item = _item(r.json(), "llm_summary_thinking_level")
|
||||
assert item["group"] == "LLM (aipi)"
|
||||
assert item["label"] == "Summary thinking level"
|
||||
assert item["type"] == "str"
|
||||
assert item["env_var"] == "BOR_LLM_SUMMARY_THINKING_LEVEL"
|
||||
assert item["secret"] is False
|
||||
assert item["restart_required"] is False
|
||||
assert item["value"] == ""
|
||||
assert item["source"] == "default" # the env var is unset, no row
|
||||
assert item["has_override"] is False
|
||||
assert item["db_value"] is None
|
||||
assert item["default"] == ""
|
||||
|
||||
# (b) PUT stores the RAW string (the overlay's D1 — env-native
|
||||
# string form) and the response reports the new effective value.
|
||||
r = admin_client.put(f"{API}/llm_summary_thinking_level", json={"value": "high"})
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["value"] == "high"
|
||||
assert r.json()["source"] == "db"
|
||||
assert r.json()["has_override"] is True
|
||||
assert r.json()["db_value"] == "high"
|
||||
assert db.execute(
|
||||
text("SELECT value FROM app_settings WHERE key = 'llm_summary_thinking_level'")
|
||||
).scalar() == "high" # stored raw, exactly as sent
|
||||
assert get_settings().llm_summary_thinking_level == "high" # live, no restart
|
||||
|
||||
# (c) the EMPTY value is storable — the summary off switch (A2; the
|
||||
# API stores as-is; no allowlist, no value validation): an empty
|
||||
# stored value must not be rejected as "no value"; the row holds
|
||||
# the empty string and the effective value is ``""``.
|
||||
r = admin_client.put(f"{API}/llm_summary_thinking_level", json={"value": ""})
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["value"] == ""
|
||||
assert r.json()["has_value"] is False # the effective value is empty
|
||||
assert db.execute(
|
||||
text("SELECT value FROM app_settings WHERE key = 'llm_summary_thinking_level'")
|
||||
).scalar() == "" # stored as-is (the off switch)
|
||||
assert get_settings().llm_summary_thinking_level == ""
|
||||
|
||||
# (d) DELETE: the row is gone and the effective value is back to
|
||||
# the env value (``""`` — the base pinned above).
|
||||
r = admin_client.delete(f"{API}/llm_summary_thinking_level")
|
||||
assert r.status_code == 200, r.text
|
||||
assert r.json()["value"] == ""
|
||||
assert r.json()["has_override"] is False
|
||||
assert db.execute(
|
||||
text("SELECT count(*) FROM app_settings WHERE key = 'llm_summary_thinking_level'")
|
||||
).scalar() == 0 # the row is gone
|
||||
assert get_settings().llm_summary_thinking_level == ""
|
||||
|
||||
# (e) the non-admin 403 gate still applies (asserted once for this
|
||||
# key — the router-wide dependency is pinned on ALL four routes in
|
||||
# ``test_auth_matrix_all_four_routes``).
|
||||
anon = TestClient(fastapi_app)
|
||||
r = anon.put(f"{API}/llm_summary_thinking_level", json={"value": "high"})
|
||||
assert r.status_code == 403 and r.json() == {"detail": "admin only"}
|
||||
|
||||
# (f) INDEPENDENCE — the whole point of the split: save the summary
|
||||
# row "high" again; the chat row's effective value is STILL "low"
|
||||
# (the two overlays resolve independently — a saved summary value
|
||||
# never touches the chat level).
|
||||
r = admin_client.put(f"{API}/llm_summary_thinking_level", json={"value": "high"})
|
||||
assert r.status_code == 200, r.text
|
||||
assert get_settings().llm_summary_thinking_level == "high"
|
||||
assert get_settings().llm_chat_thinking_level == "low" # untouched
|
||||
chat_item = _item(admin_client.get(API).json(), "llm_chat_thinking_level")
|
||||
assert chat_item["value"] == "low" # the panel agrees, too
|
||||
admin_client.delete(f"{API}/llm_summary_thinking_level") # no row survives
|
||||
|
||||
|
||||
def test_docs_token_is_write_only_masked_end_to_end(
|
||||
admin_client: TestClient, db: Session, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
|
||||
@@ -10,9 +10,10 @@ snapshot, ``app/api/chat.py::get_llm``), and the chat concurrency
|
||||
semaphore (``chat_max_concurrent``). Plus the kill-switch counterpart
|
||||
(``BOR_SETTINGS_DB=0`` with rows present → env values are back) and
|
||||
the ``scripts/``-consumer surface (``import_extension_set``), and the
|
||||
phase-172 thinking level (``llm_thinking_level`` — the value the LLM
|
||||
layer reads per request via ``get_settings()``; the off switch is
|
||||
live-settable, A2).
|
||||
phase-172 thinking level (``llm_chat_thinking_level`` — the value the
|
||||
LLM layer reads per request via ``get_settings()``; the off switch is
|
||||
live-settable, A2) and its phase-178 summary twin (task 02:
|
||||
``llm_summary_thinking_level``, default ``""`` = off, A2).
|
||||
|
||||
The rows are inserted directly through the ``db`` session (the admin
|
||||
API is task 04); the save itself is ``invalidate_settings()`` —
|
||||
@@ -164,44 +165,88 @@ def test_llm_client_repoints_on_the_next_save(db: Session) -> None:
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# the thinking level: save/reset resolve the live value (phase 172, task 02)
|
||||
# the chat thinking level: save/reset resolve the live value
|
||||
# (phase 178, task 01 — the phase-172 test, renamed)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_llm_thinking_level_live_apply_no_restart(
|
||||
def test_llm_chat_thinking_level_live_apply_no_restart(
|
||||
db: Session, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Phase 172 (task 02): the thinking level follows the overlay
|
||||
"""Phase 178 (task 01): the chat thinking level follows the overlay
|
||||
live — a save (the PUT's write, minus the API: ``_save`` +
|
||||
``invalidate_settings()`` is exactly what task 04's PUT/DELETE
|
||||
call after the write) resolves
|
||||
``get_settings().llm_thinking_level`` WITHOUT a restart (the proof
|
||||
the LLM layer — task 03 reads the setting per request via
|
||||
``get_settings().llm_chat_thinking_level`` WITHOUT a restart (the
|
||||
proof the LLM layer — task 03 reads the setting per request via
|
||||
``get_settings()`` — picks the override up live), the EMPTY value
|
||||
resolves live too (the off switch is live-settable, A2/A4), and a
|
||||
reset (the DELETE) returns the env value.
|
||||
"""
|
||||
# A deterministic base (machine-independent, the kill-switch test's
|
||||
# idiom): the code default, no row in the table.
|
||||
monkeypatch.delenv("BOR_LLM_THINKING_LEVEL", raising=False)
|
||||
monkeypatch.delenv("BOR_LLM_CHAT_THINKING_LEVEL", raising=False)
|
||||
invalidate_settings()
|
||||
assert get_settings().llm_thinking_level == "low"
|
||||
_save(db, "llm_thinking_level", "high")
|
||||
assert get_settings().llm_chat_thinking_level == "low"
|
||||
_save(db, "llm_chat_thinking_level", "high")
|
||||
invalidate_settings()
|
||||
assert get_settings().llm_thinking_level == "high" # NO restart
|
||||
assert get_settings().llm_chat_thinking_level == "high" # NO restart
|
||||
# The off switch is live-settable too (A2/A4): the empty value
|
||||
# resolves through the overlay, not stripped or coerced.
|
||||
_save(db, "llm_thinking_level", "")
|
||||
_save(db, "llm_chat_thinking_level", "")
|
||||
invalidate_settings()
|
||||
assert get_settings().llm_thinking_level == ""
|
||||
assert get_settings().llm_chat_thinking_level == ""
|
||||
# The reset returns the env value (no process restart in between —
|
||||
# the LLM layer re-reads per request, task 03).
|
||||
db.execute(
|
||||
text("DELETE FROM app_settings WHERE key = 'llm_thinking_level'")
|
||||
text("DELETE FROM app_settings WHERE key = 'llm_chat_thinking_level'")
|
||||
)
|
||||
db.commit()
|
||||
invalidate_settings()
|
||||
assert get_settings().llm_thinking_level == "low"
|
||||
assert get_settings().llm_chat_thinking_level == "low"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# the summary thinking level: save/reset resolve the live value
|
||||
# (phase 178, task 02 — the chat twin's summary counterpart)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_llm_summary_thinking_level_live_apply_no_restart(
|
||||
db: Session, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Phase 178 (task 02): the summary thinking level follows the
|
||||
overlay live — a save (the PUT's write, minus the API: ``_save`` +
|
||||
``invalidate_settings()`` is exactly what task 04's PUT/DELETE call
|
||||
after the write) resolves ``get_settings().llm_summary_thinking_level``
|
||||
WITHOUT a restart (the proof the LLM layer — task 03 reads the
|
||||
setting per request via ``get_settings()`` — picks the override up
|
||||
live), the EMPTY value resolves live too (the off switch is
|
||||
live-settable, A2/A4), and a reset (the DELETE) returns the env
|
||||
value (``""`` — the code default; the chat twin's reset returns
|
||||
``"low"``, the two levels are independent).
|
||||
"""
|
||||
# A deterministic base (machine-independent, the kill-switch test's
|
||||
# idiom): the code default, no row in the table.
|
||||
monkeypatch.delenv("BOR_LLM_SUMMARY_THINKING_LEVEL", raising=False)
|
||||
invalidate_settings()
|
||||
assert get_settings().llm_summary_thinking_level == ""
|
||||
_save(db, "llm_summary_thinking_level", "high")
|
||||
invalidate_settings()
|
||||
assert get_settings().llm_summary_thinking_level == "high" # NO restart
|
||||
# The off switch is live-settable too (A2/A4): the empty value
|
||||
# resolves through the overlay, not stripped or coerced.
|
||||
_save(db, "llm_summary_thinking_level", "")
|
||||
invalidate_settings()
|
||||
assert get_settings().llm_summary_thinking_level == ""
|
||||
# The reset returns the env value (no process restart in between —
|
||||
# the LLM layer re-reads per request, task 03).
|
||||
db.execute(
|
||||
text("DELETE FROM app_settings WHERE key = 'llm_summary_thinking_level'")
|
||||
)
|
||||
db.commit()
|
||||
invalidate_settings()
|
||||
assert get_settings().llm_summary_thinking_level == ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1550,8 +1550,13 @@ def test_endpoint_log_line_records_suggested_after_summary_hits(
|
||||
seeded suggestion tier's size (``len(plan.suggested_docs)``).
|
||||
Phase 172 (A6): the line gains ``thinking=<level>`` immediately
|
||||
BEFORE ``thinking_chars=N`` — the effective (overlay-resolved)
|
||||
``BOR_LLM_THINKING_LEVEL``, trimmed, ``-`` when empty (the off
|
||||
switch)."""
|
||||
``BOR_LLM_CHAT_THINKING_LEVEL`` (the chat level), trimmed,
|
||||
``-`` when empty (the off switch).
|
||||
Phase 178 (A5): the field stays pinned to the CHAT level — the
|
||||
level the turn actually ran at; the one-shot summary level
|
||||
(``llm_summary_thinking_level``, a different knob) is never read
|
||||
by the line and must not leak into the line's ``thinking=``
|
||||
value (cross-pin + inverse off-pin below)."""
|
||||
_session, _llm = gate_env
|
||||
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT")
|
||||
chunks = [_chunk(doc, 0.30), _chunk(doc, 0.20, is_summary=True)]
|
||||
@@ -1606,7 +1611,7 @@ def test_endpoint_log_line_records_suggested_after_summary_hits(
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(_env_file=None, relevance_threshold=0.30, llm_thinking_level="high"), # pyright: ignore[reportCallIssue]
|
||||
lambda: Settings(_env_file=None, relevance_threshold=0.30, llm_chat_thinking_level="high"), # pyright: ignore[reportCallIssue]
|
||||
)
|
||||
line = _last_log_line()
|
||||
assert "thinking=high thinking_chars=" in line
|
||||
@@ -1615,7 +1620,7 @@ def test_endpoint_log_line_records_suggested_after_summary_hits(
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(_env_file=None, relevance_threshold=0.30, llm_thinking_level=""), # pyright: ignore[reportCallIssue]
|
||||
lambda: Settings(_env_file=None, relevance_threshold=0.30, llm_chat_thinking_level=""), # pyright: ignore[reportCallIssue]
|
||||
)
|
||||
line = _last_log_line()
|
||||
assert "thinking=- thinking_chars=" in line
|
||||
@@ -1625,7 +1630,42 @@ def test_endpoint_log_line_records_suggested_after_summary_hits(
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(_env_file=None, relevance_threshold=0.30, llm_thinking_level=" "), # pyright: ignore[reportCallIssue]
|
||||
lambda: Settings(_env_file=None, relevance_threshold=0.30, llm_chat_thinking_level=" "), # pyright: ignore[reportCallIssue]
|
||||
)
|
||||
line = _last_log_line()
|
||||
assert "thinking=- thinking_chars=" in line
|
||||
|
||||
# Phase 178 (A5) cross-pin: the line is pinned to the CHAT level
|
||||
# — with the one-shot summary level set to a DIFFERENT value,
|
||||
# the line still shows the chat level; the summary's "high"
|
||||
# does not appear as the line's ``thinking=`` value.
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=0.30,
|
||||
llm_chat_thinking_level="low",
|
||||
llm_summary_thinking_level="high",
|
||||
),
|
||||
)
|
||||
line = _last_log_line()
|
||||
assert "thinking=low thinking_chars=" in line
|
||||
assert "thinking=high" not in line
|
||||
|
||||
# The inverse off-pin: the chat off-switch logs ``-`` even
|
||||
# though the one-shot summary knob is set — only the chat level
|
||||
# (the level this turn ran at) rides the line.
|
||||
monkeypatch.setattr(
|
||||
chat_api,
|
||||
"get_settings",
|
||||
lambda: Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=0.30,
|
||||
llm_chat_thinking_level="",
|
||||
llm_summary_thinking_level="high",
|
||||
),
|
||||
)
|
||||
line = _last_log_line()
|
||||
assert "thinking=- thinking_chars=" in line
|
||||
assert "thinking=high" not in line
|
||||
|
||||
+56
-23
@@ -105,32 +105,65 @@ def test_llm_summary_model_env_override(monkeypatch) -> None:
|
||||
assert s.llm_summary_model == "mini"
|
||||
|
||||
|
||||
def test_llm_thinking_level_default_is_low(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Phase 172: the model's thinking level defaults to ``low``
|
||||
(``BOR_LLM_THINKING_LEVEL`` unset)."""
|
||||
monkeypatch.delenv("BOR_LLM_THINKING_LEVEL", raising=False)
|
||||
s = _settings()
|
||||
assert s.llm_thinking_level == "low"
|
||||
|
||||
|
||||
def test_llm_thinking_level_env_override(monkeypatch) -> None:
|
||||
"""Phase 172 (A4): free-form like ``llm_chat_model`` —
|
||||
``BOR_LLM_THINKING_LEVEL=high`` resolves to ``"high"`` (no
|
||||
allowlist, no validator)."""
|
||||
monkeypatch.setenv("BOR_LLM_THINKING_LEVEL", "high")
|
||||
s = _settings()
|
||||
assert s.llm_thinking_level == "high"
|
||||
|
||||
|
||||
def test_llm_thinking_level_empty_env_value_stays_empty(
|
||||
def test_llm_chat_thinking_level_default_is_low(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 172 (A2): the EMPTY value is the off switch — it must not
|
||||
be coerced back to the default (trimming happens only at the wire,
|
||||
task 03; no normalization at load)."""
|
||||
monkeypatch.setenv("BOR_LLM_THINKING_LEVEL", "")
|
||||
"""Phase 178 (the phase-172 knob, renamed): the chat turn's
|
||||
thinking level defaults to ``low`` (``BOR_LLM_CHAT_THINKING_LEVEL``
|
||||
unset)."""
|
||||
monkeypatch.delenv("BOR_LLM_CHAT_THINKING_LEVEL", raising=False)
|
||||
s = _settings()
|
||||
assert s.llm_thinking_level == ""
|
||||
assert s.llm_chat_thinking_level == "low"
|
||||
|
||||
|
||||
def test_llm_chat_thinking_level_env_override(monkeypatch) -> None:
|
||||
"""Phase 178 (A4, kept): free-form like ``llm_chat_model`` —
|
||||
``BOR_LLM_CHAT_THINKING_LEVEL=high`` resolves to ``"high"`` (no
|
||||
allowlist, no validator)."""
|
||||
monkeypatch.setenv("BOR_LLM_CHAT_THINKING_LEVEL", "high")
|
||||
s = _settings()
|
||||
assert s.llm_chat_thinking_level == "high"
|
||||
|
||||
|
||||
def test_llm_chat_thinking_level_empty_env_value_stays_empty(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 178 (A2, kept): the EMPTY value is the off switch — it
|
||||
must not be coerced back to the default (trimming happens only at
|
||||
the wire; no normalization at load)."""
|
||||
monkeypatch.setenv("BOR_LLM_CHAT_THINKING_LEVEL", "")
|
||||
s = _settings()
|
||||
assert s.llm_chat_thinking_level == ""
|
||||
|
||||
|
||||
def test_llm_summary_thinking_level_default_is_empty(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 178 (A2): the one-shot summary thinking level defaults to
|
||||
EMPTY (``BOR_LLM_SUMMARY_THINKING_LEVEL`` unset) — the TODO's
|
||||
"none" means absence, never a literal "none" string."""
|
||||
monkeypatch.delenv("BOR_LLM_SUMMARY_THINKING_LEVEL", raising=False)
|
||||
s = _settings()
|
||||
assert s.llm_summary_thinking_level == ""
|
||||
|
||||
|
||||
def test_llm_summary_thinking_level_env_override(monkeypatch) -> None:
|
||||
"""Phase 178: free-form like the chat level —
|
||||
``BOR_LLM_SUMMARY_THINKING_LEVEL=high`` resolves to ``"high"``
|
||||
(no allowlist, no validator)."""
|
||||
monkeypatch.setenv("BOR_LLM_SUMMARY_THINKING_LEVEL", "high")
|
||||
s = _settings()
|
||||
assert s.llm_summary_thinking_level == "high"
|
||||
|
||||
|
||||
def test_llm_summary_thinking_level_empty_env_value_stays_empty(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 178 (A2, kept): the EMPTY value is the off switch — it
|
||||
must not be coerced to anything at load (no normalization)."""
|
||||
monkeypatch.setenv("BOR_LLM_SUMMARY_THINKING_LEVEL", "")
|
||||
s = _settings()
|
||||
assert s.llm_summary_thinking_level == ""
|
||||
|
||||
|
||||
def test_summary_max_chars_default_and_env_override(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
|
||||
+116
-33
@@ -1324,48 +1324,62 @@ def test_chat_retry_delay_is_flat_never_backoff(monkeypatch: pytest.MonkeyPatch)
|
||||
assert sleeps == [1.25, 1.25, 1.25]
|
||||
|
||||
|
||||
# ---------- thinking level passthrough (phase 172, task 03) ----------
|
||||
# ---------- thinking level passthrough (phase 172; the split, 178 A3) ----------
|
||||
# chat() (one-shot) rides the SUMMARY level; chat_stream() rides the CHAT
|
||||
# level; an empty (trimmed) level is the off switch (A2 — the parameter
|
||||
# is absent, never a literal "none").
|
||||
|
||||
|
||||
def test_chat_sends_reasoning_effort_extra_body_when_level_set() -> None:
|
||||
"""Phase 172 (A1): a non-empty level rides the one-shot request as
|
||||
the top-level ``reasoning_effort`` parameter via the SDK's
|
||||
# --- the one-shot path: the summary level (phase 178, A3) ---
|
||||
|
||||
|
||||
def test_chat_sends_thinking_level_extra_body_when_summary_level_set() -> None:
|
||||
"""Phase 178 (A3): a non-empty summary level rides the one-shot
|
||||
request as the top-level ``reasoning_effort`` parameter via the SDK's
|
||||
``extra_body``."""
|
||||
llm, completions = _make_chat_client(
|
||||
_FakeCompletion("ok"), llm_thinking_level="low"
|
||||
_FakeCompletion("ok"), llm_summary_thinking_level="high"
|
||||
)
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert completions.chat_kwargs is not None
|
||||
assert completions.chat_kwargs["extra_body"] == {"reasoning_effort": "low"}
|
||||
assert completions.chat_kwargs["extra_body"] == {"reasoning_effort": "high"}
|
||||
|
||||
|
||||
def test_chat_default_level_low_is_always_sent() -> None:
|
||||
"""The default (``low``) is always sent (A2): a client built without
|
||||
naming the setting still carries the parameter."""
|
||||
def test_chat_default_summary_thinking_off_is_pre_phase172_byte_identical() -> None:
|
||||
"""The default summary level is EMPTY (the TODO's "none", A2): a
|
||||
client built without naming either setting sends NO ``extra_body`` —
|
||||
one-shot requests are byte-identical to pre-phase-172."""
|
||||
llm, completions = _make_chat_client(_FakeCompletion("ok"))
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert completions.chat_kwargs is not None
|
||||
assert completions.chat_kwargs["extra_body"] == {"reasoning_effort": "low"}
|
||||
assert "extra_body" not in completions.chat_kwargs
|
||||
assert set(completions.chat_kwargs) == {
|
||||
"model",
|
||||
"messages",
|
||||
"temperature",
|
||||
"max_tokens",
|
||||
"stream",
|
||||
}
|
||||
|
||||
|
||||
def test_chat_trims_the_level_before_sending() -> None:
|
||||
"""TRIM: a hand-edited ``.env`` value with trailing whitespace
|
||||
(``"low "``) is sent trimmed — the raw stored string never reaches
|
||||
the wire with surrounding whitespace."""
|
||||
def test_chat_trims_the_summary_thinking_level_before_sending() -> None:
|
||||
"""TRIM: a hand-edited ``.env`` value with surrounding whitespace
|
||||
(``" high "``) is sent trimmed — the raw stored string never reaches
|
||||
the wire with whitespace."""
|
||||
llm, completions = _make_chat_client(
|
||||
_FakeCompletion("ok"), llm_thinking_level="low "
|
||||
_FakeCompletion("ok"), llm_summary_thinking_level=" high "
|
||||
)
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert completions.chat_kwargs is not None
|
||||
assert completions.chat_kwargs["extra_body"] == {"reasoning_effort": "low"}
|
||||
assert completions.chat_kwargs["extra_body"] == {"reasoning_effort": "high"}
|
||||
|
||||
|
||||
def test_chat_empty_level_absent_from_kwargs_byte_identical() -> None:
|
||||
"""Phase 172 (A2): a whitespace value is the off switch —
|
||||
``extra_body`` is ABSENT (never None, never ``"off"``) and the
|
||||
request kwargs keep the pre-phase-172 shape."""
|
||||
def test_chat_empty_summary_thinking_level_absent_from_kwargs_byte_identical() -> None:
|
||||
"""Phase 172 (A2): an empty value is the off switch —
|
||||
``extra_body`` is ABSENT (never None, never a literal ``"none"``) and
|
||||
the request kwargs keep the pre-phase-172 shape."""
|
||||
llm, completions = _make_chat_client(
|
||||
_FakeCompletion("ok"), llm_thinking_level=" "
|
||||
_FakeCompletion("ok"), llm_summary_thinking_level=""
|
||||
)
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert completions.chat_kwargs is not None
|
||||
@@ -1379,30 +1393,42 @@ def test_chat_empty_level_absent_from_kwargs_byte_identical() -> None:
|
||||
}
|
||||
|
||||
|
||||
def test_chat_stream_sends_reasoning_effort_extra_body_when_level_set() -> None:
|
||||
"""Phase 172 (A1): the chat turn rides the parameter too."""
|
||||
def test_chat_whitespace_summary_thinking_level_is_the_off_switch() -> None:
|
||||
"""Whitespace-only (``" "``) is empty for wire purposes (A2)."""
|
||||
llm, completions = _make_chat_client(
|
||||
_FakeCompletion("ok"), llm_summary_thinking_level=" "
|
||||
)
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert completions.chat_kwargs is not None
|
||||
assert "extra_body" not in completions.chat_kwargs
|
||||
|
||||
|
||||
def test_chat_stream_sends_thinking_level_extra_body_when_level_set() -> None:
|
||||
"""Phase 178 (A3): the chat turn rides the parameter from the
|
||||
**chat** level."""
|
||||
llm, completions = _make_stream_client(
|
||||
[_chunk("x")], llm_thinking_level="high"
|
||||
[_chunk("x")], llm_chat_thinking_level="high"
|
||||
)
|
||||
asyncio.run(_collect(llm, [{"role": "user", "content": "q"}]))
|
||||
assert completions.kwargs is not None
|
||||
assert completions.kwargs["extra_body"] == {"reasoning_effort": "high"}
|
||||
|
||||
|
||||
def test_chat_stream_trims_the_level_before_sending() -> None:
|
||||
def test_chat_stream_trims_the_thinking_level_before_sending() -> None:
|
||||
llm, completions = _make_stream_client(
|
||||
[_chunk("x")], llm_thinking_level=" high "
|
||||
[_chunk("x")], llm_chat_thinking_level=" high "
|
||||
)
|
||||
asyncio.run(_collect(llm, [{"role": "user", "content": "q"}]))
|
||||
assert completions.kwargs is not None
|
||||
assert completions.kwargs["extra_body"] == {"reasoning_effort": "high"}
|
||||
|
||||
|
||||
def test_chat_stream_empty_level_absent_from_kwargs_byte_identical() -> None:
|
||||
"""Phase 172 (A2): the stream's off switch — ``extra_body`` absent
|
||||
(never None) and the kwargs dict keeps the pre-phase-172 shape."""
|
||||
def test_chat_stream_empty_thinking_level_absent_from_kwargs_byte_identical() -> None:
|
||||
"""Phase 178 (A2): the stream's off switch — ``extra_body`` absent
|
||||
(never None, never a literal ``"none"``) and the kwargs dict keeps
|
||||
the pre-phase-172 shape."""
|
||||
llm, completions = _make_stream_client(
|
||||
[_chunk("x")], llm_thinking_level=""
|
||||
[_chunk("x")], llm_chat_thinking_level=""
|
||||
)
|
||||
asyncio.run(_collect(llm, [{"role": "user", "content": "q"}]))
|
||||
assert completions.kwargs is not None
|
||||
@@ -1416,22 +1442,79 @@ def test_chat_stream_empty_level_absent_from_kwargs_byte_identical() -> None:
|
||||
}
|
||||
|
||||
|
||||
def test_chat_stream_whitespace_level_is_the_off_switch() -> None:
|
||||
def test_chat_stream_whitespace_thinking_level_is_the_off_switch() -> None:
|
||||
"""Whitespace-only (``" "``) is empty for wire purposes (A2)."""
|
||||
llm, completions = _make_stream_client(
|
||||
[_chunk("x")], llm_thinking_level=" "
|
||||
[_chunk("x")], llm_chat_thinking_level=" "
|
||||
)
|
||||
asyncio.run(_collect(llm, [{"role": "user", "content": "q"}]))
|
||||
assert completions.kwargs is not None
|
||||
assert "extra_body" not in completions.kwargs
|
||||
|
||||
|
||||
# --- the cross-pins (phase 178, A3 — the whole point of the split) ---
|
||||
|
||||
|
||||
def _make_both_client(
|
||||
chunks: list,
|
||||
completion: _FakeCompletion,
|
||||
**settings_kwargs: Any,
|
||||
) -> tuple[LLMClient, _FakeCompletions]:
|
||||
"""One client for BOTH paths (a shared fake ``completions`` serves the
|
||||
streaming and the non-streaming ``create()`` alike) so the two levels
|
||||
can be observed on one client."""
|
||||
completions = _FakeCompletions(chunks, None, completion)
|
||||
fake_openai = SimpleNamespace(chat=SimpleNamespace(completions=completions))
|
||||
llm = LLMClient(_settings(**settings_kwargs))
|
||||
llm._client = fake_openai # pyright: ignore[reportAttributeAccessIssue]
|
||||
return llm, completions
|
||||
|
||||
|
||||
def test_split_thinking_levels_inverted_defaults_chat_high_summary_off() -> None:
|
||||
"""Phase 178 (A3): the defaults INVERTED — the chat level ``high``
|
||||
rides ``chat_stream`` while the (empty) summary level keeps ``chat``
|
||||
free of ``extra_body``. The two knobs are independent on the wire."""
|
||||
llm, completions = _make_both_client(
|
||||
[_chunk("x")],
|
||||
_FakeCompletion("ok"),
|
||||
llm_chat_thinking_level="high",
|
||||
llm_summary_thinking_level="",
|
||||
)
|
||||
asyncio.run(_collect(llm, [{"role": "user", "content": "q"}]))
|
||||
assert completions.kwargs is not None
|
||||
assert completions.kwargs["extra_body"] == {"reasoning_effort": "high"}
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert completions.chat_kwargs is not None
|
||||
assert "extra_body" not in completions.chat_kwargs
|
||||
|
||||
|
||||
def test_split_thinking_levels_default_client_stream_low_summary_off() -> None:
|
||||
"""The DEFAULTS (chat ``low`` + summary ``""``) — the stream carries
|
||||
``reasoning_effort="low"`` while the one-shot request is
|
||||
byte-identical to pre-phase-172 (no ``extra_body``; the TODO's
|
||||
"summary should be none")."""
|
||||
llm, completions = _make_both_client([_chunk("x")], _FakeCompletion("ok"))
|
||||
asyncio.run(_collect(llm, [{"role": "user", "content": "q"}]))
|
||||
assert completions.kwargs is not None
|
||||
assert completions.kwargs["extra_body"] == {"reasoning_effort": "low"}
|
||||
asyncio.run(llm.chat([{"role": "user", "content": "q"}]))
|
||||
assert completions.chat_kwargs is not None
|
||||
assert "extra_body" not in completions.chat_kwargs
|
||||
assert set(completions.chat_kwargs) == {
|
||||
"model",
|
||||
"messages",
|
||||
"temperature",
|
||||
"max_tokens",
|
||||
"stream",
|
||||
}
|
||||
|
||||
|
||||
def test_embeddings_body_never_carries_reasoning_effort() -> None:
|
||||
"""Phase 172 (A3): the embeddings path is untouched — with the level
|
||||
set, every POST body stays the minimal ``{model, input}`` payload;
|
||||
``reasoning_effort`` never appears in it."""
|
||||
service = _FakeEmbeddingsService()
|
||||
llm, http = _make_client(service, llm_thinking_level="high")
|
||||
llm, http = _make_client(service, llm_chat_thinking_level="high")
|
||||
asyncio.run(llm.embed(["a", "b"]))
|
||||
vec = asyncio.run(llm.embed_one("c"))
|
||||
assert len(vec) == 768
|
||||
|
||||
@@ -96,19 +96,25 @@ def test_env_var_is_bare_key_upper_prefixed() -> None:
|
||||
assert env_var(key).startswith("BOR_")
|
||||
|
||||
|
||||
def test_llm_thinking_level_is_a_plain_live_overridable_row() -> None:
|
||||
"""Phase 172 (task 01): the new ``llm_thinking_level`` pair sits in
|
||||
``LLM (aipi)`` with the ``Thinking level`` label and is plain —
|
||||
not ``secret`` (a free-form text value, like ``llm_chat_model``),
|
||||
not ``restart_required`` (the LLM client reads the setting per
|
||||
request via ``get_settings()`` — live), overridable (the overlay is
|
||||
generic over ``str`` fields)."""
|
||||
meta = CATALOG["llm_thinking_level"]
|
||||
assert meta.group == "LLM (aipi)"
|
||||
assert meta.label == "Thinking level"
|
||||
assert "llm_thinking_level" not in SECRETS
|
||||
assert "llm_thinking_level" not in RESTART_REQUIRED
|
||||
assert "llm_thinking_level" not in NON_OVERRIDABLE
|
||||
def test_llm_thinking_levels_are_plain_live_overridable_rows() -> None:
|
||||
"""Phase 178 (task 01): the two thinking-level rows (the phase-172
|
||||
knob, renamed + the new summary level) sit in ``LLM (aipi)`` with
|
||||
the ``Chat thinking level`` / ``Summary thinking level`` labels and
|
||||
are plain — not ``secret`` (free-form text values, like
|
||||
``llm_chat_model``), not ``restart_required`` (the LLM client
|
||||
reads the settings per request via ``get_settings()`` — live),
|
||||
overridable (the overlay is generic over ``str`` fields)."""
|
||||
for key, label in (
|
||||
("llm_chat_thinking_level", "Chat thinking level"),
|
||||
("llm_summary_thinking_level", "Summary thinking level"),
|
||||
):
|
||||
meta = CATALOG[key]
|
||||
assert meta.group == "LLM (aipi)"
|
||||
assert meta.label == label
|
||||
assert meta.secret is False
|
||||
assert key not in SECRETS
|
||||
assert key not in RESTART_REQUIRED
|
||||
assert key not in NON_OVERRIDABLE
|
||||
|
||||
|
||||
def test_docs_user_is_a_plain_live_overridable_row() -> None:
|
||||
|
||||
Reference in New Issue
Block a user