"""Phase 178 E2E (Playwright): the SPLIT thinking levels — a Chat thinking level row and a Summary thinking level row in the admin Settings panel — the owner's acceptance for TWO independently tunable, pi-style LLM reasoning knobs, proven in a real browser. Source: ``TODO.md`` L3 — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low" Run in isolation (DB must be up: ``podman compose up -d db``): uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov Test → contract mapping (one story, one phase, one isolated file): 1. ``test_both_rows_render_free_form_with_defaults`` — the rows: the ``LLM (aipi)`` group band is visible (the phase-171 band contract), a row labelled **Chat thinking level** with the ``BOR_LLM_CHAT_THINKING_LEVEL`` env-var display shows ``low`` with the ``env`` badge (chat stays at low, TODO L3), a row labelled **Summary thinking level** with the ``BOR_LLM_SUMMARY_THINKING_LEVEL`` display shows an **empty** input with the ``env`` badge (the TODO's "none" = the phase-172 off switch, LOCKED A2 — the parameter is ABSENT, never a literal "none"), BOTH inputs are ``type="text"`` (free-form — NOT ``number``/``select``, the A4 contract: no allowlist, the endpoint is the source of truth), the ``llm_chat_model`` row's input is the same control (the "like chat model" reference, one assertion pair), and ``tr[data-key="llm_thinking_level"]`` is **absent** (the old phase-172 key is gone, LOCKED A1). 2. ``test_saved_levels_persist_without_restart`` — the save (the phase-168 §7.4 Save lifecycle per row: dirty → save → ``DB`` badge): ``think-soft`` (a free-form custom — NOT a known pi level, A4) is typed into the summary row and ``high`` into the chat row (the dirty hints appear), then BOTH are saved; the API catalogue (``_catalog_item``) AND the raw ``app_settings`` rows carry the saved values for EACH key independently (the independence pin — the two rows' controls are distinct and the saved values do not cross), and a reload of ``/settings.html`` still shows ``think-soft`` / ``high`` with the ``DB`` badges (no restart, the phase-168 contract). 3. ``test_reset_returns_to_env_defaults`` — the per-row Reset (the per-row ``DELETE``): after the saves, both Resets are live; clicking the summary Reset lands the value at **empty** with the ``env`` badge and clicking the chat Reset lands ``low`` with the ``env`` badge (the overlay reset, task 02's contract, in the browser); the status lines show the house "… is back to the environment value." toast for each. DB isolation: the shared e2e Postgres keeps ``app_settings`` (the overlay rows — a leftover row would retune every later suite's app through the conftest pin's failure mode), ``ui_settings`` (the single row the caching middleware reads for EVERY served page) and ``api_tokens`` rows across suites. An autouse fixture truncates ``app_settings`` and ``ui_settings`` and deletes the ``e2e-``-labeled tokens before AND after every test (never a TRUNCATE on ``api_tokens`` — the shared DB may hold the owner's real tokens), and — because the direct SQL truncate does NOT invalidate the app's settings cache (the admin API is the only mutation surface that does) — follows it with the app's own clear-all (``DELETE /api/admin/settings``; without it, a test that types the SAME value its predecessor saved sees the stale value as the served one, registers no dirty state, and the Save button stays disabled). The module teardown pins ``app_settings`` empty. The module app opts BACK IN over the conftest's global ``BOR_SETTINGS_DB=0`` pin (``env["BOR_ SETTINGS_DB"] = "1"`` — the ``test_pictorial_pdf.py`` opt-in pattern) and pins ``BOR_LLM_CHAT_THINKING_LEVEL=low`` AND ``BOR_LLM_SUMMARY_THINKING_LEVEL=`` (the empty string SET — so both badges read ``env``, not ``default``, and an operator's local ``.env`` cannot leak a different level into the suite; the house leak-guard pattern) together with the "like chat model" reference (``BOR_LLM_CHAT_MODEL``). A module-scope ``.env`` mtime guard pins LOCKED A1: no code path the suite drives opens the env file for writing. ASSUMPTION A5 (phase-locked): the per-turn-line assertion (``thinking=``) and the wire split (``chat_stream()`` → chat level, ``chat()`` → summary level) live in the unit/integration layer (tasks 03/04) — this suite proves the settings surface only. """ from __future__ import annotations import json import os import subprocess import sys from collections.abc import Iterator from typing import Any import httpx import pytest from playwright.sync_api import Page, expect from sqlalchemy import text from app.config import Settings from app.db import SessionLocal from e2e.auth_helpers import login from e2e.conftest import ( ADMIN_PASSWORD, APP_PORT, REPO, SESSION_SECRET, USE_REAL_LLM, _wait_http, ) APP_URL = f"http://127.0.0.1:{APP_PORT}" #: The chat row's saved value (test 2) — a known pi level, the #: operator's "bump the chat turn" case. SAVED_CHAT_LEVEL = "high" #: The summary row's saved value (test 2) — NOT a known pi level #: (proves the A4 contract: no allowlist, the endpoint is the #: source of truth). SAVED_SUMMARY_LEVEL = "think-soft" # --------------------------------------------------------------------------- # Per-module app env (the phase-168 tuning pattern) # --------------------------------------------------------------------------- def _boot_app() -> subprocess.Popen[bytes]: """The one module app: the shared env block + the phase-178 pins, with the overlay opted BACK IN (``BOR_SETTINGS_DB=1``).""" global _MOCK_PORT env = dict(os.environ) env.pop("DEBUGPY", None) env["BOR_ENVIRONMENT"] = "e2e" env["BOR_STATIC_DIR"] = str(REPO / "frontend") env["BOR_LLM_BASE_URL"] = ( "https://aipi.reeseapps.com/v1" if USE_REAL_LLM else f"http://127.0.0.1:{_MOCK_PORT}/v1" ) # The E2E-calibrated gate pins (the conftest pattern — the mock's # token-overlap distribution, explicit so an operator's local # .env cannot leak a different gate into the suite's app). env["BOR_RELEVANCE_THRESHOLD"] = "0.30" env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15" env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15" env["BOR_LLM_RETRY_DELAY"] = "0" env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default) env.setdefault( "BOR_DATABASE_URL", "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese", ) # Phase 16: admin auth must be set or create_app() refuses to boot. env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD env["BOR_SESSION_SECRET"] = SESSION_SECRET env["BOR_DOCS_REPO"] = "" env["BOR_SUGGESTIONS"] = json.dumps( Settings.model_fields["suggestions"].default ) # Leak guards: the branding base and the "like chat model" # reference row stay the CODE defaults (derived from the class # fields, never drift from app/config.py). env["BOR_APP_NAME"] = Settings.model_fields["app_name"].default env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default env["BOR_GIT_SOURCES"] = "" env["BOR_LLM_CHAT_MODEL"] = Settings.model_fields["llm_chat_model"].default # Phase 168 (the panel suite's pattern): the module opts BACK IN # over the conftest's global BOR_SETTINGS_DB=0 pin (the # test_pictorial_pdf.py pattern) — the rows' save/reset # contract needs the overlay on. env["BOR_SETTINGS_DB"] = "1" # Phase 178 (task 05): pin BOTH thinking levels to their CODE # DEFAULTS — each env var is SET (so the badge reads ``env``, # not ``default``) and an operator's local .env cannot leak a # different level into the suite's app. The summary level is # set to EMPTY (the TODO's "none" = the off switch, A2). env["BOR_LLM_CHAT_THINKING_LEVEL"] = Settings.model_fields[ "llm_chat_thinking_level" ].default env["BOR_LLM_SUMMARY_THINKING_LEVEL"] = Settings.model_fields[ "llm_summary_thinking_level" ].default return subprocess.Popen( [sys.executable, "-m", "uvicorn", "app.main:app", "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"], cwd=REPO, env=env, ) def _stop_app(proc: subprocess.Popen[bytes]) -> None: proc.terminate() try: proc.wait(timeout=10) except subprocess.TimeoutExpired: proc.kill() @pytest.fixture(scope="module") def app_server(mock_llm: int) -> Iterator[str]: """The real app under test — per-module env: BOTH thinking levels pinned to their code defaults (chat ``low``, summary empty = the A2 off switch) and ``BOR_SETTINGS_DB=1`` (the module opts back in over the conftest pin).""" global _MOCK_PORT _MOCK_PORT = mock_llm proc = _boot_app() try: _wait_http(f"{APP_URL}/api/health") yield APP_URL finally: _stop_app(proc) @pytest.fixture(scope="module") def app_url(app_server: str) -> str: return app_server # --------------------------------------------------------------------------- # DB isolation + helpers # --------------------------------------------------------------------------- def _clean_settings_state() -> None: """Fresh settings + theme + token state per test: truncate ``app_settings`` (a leftover row is the cross-suite poison this phase's overlay contract exists to prevent — the rows are wiped between tests) and the single-row ``ui_settings`` (the middleware reads it for EVERY page), and delete this suite's issued tokens (label-scoped on ``e2e-`` — never a TRUNCATE: the shared DB may hold the owner's real tokens).""" with SessionLocal() as db: db.execute(text("TRUNCATE app_settings")) db.execute(text("TRUNCATE ui_settings")) db.execute(text("DELETE FROM api_tokens WHERE label LIKE 'e2e-%'")) db.commit() def _invalidate_app_cache(app_url: str) -> None: """The direct SQL truncate does NOT invalidate the app's ``get_settings()`` lru-cache (the admin API is the only mutation surface that invalidates — the phase-168 contract) — so after the truncate, hit the app's own clear-all (``DELETE /api/admin/settings``): zero rows to delete, but the invalidate call rebuilds the cache env-only. Without it, a test that types the SAME value its predecessor saved sees the STALE value as the served one (``markDirty`` finds no diff, the Save button stays disabled).""" client = _admin_client(app_url) r = client.delete(f"{app_url}/api/admin/settings", timeout=30.0) assert r.status_code == 200, f"clear-all → {r.status_code}: {r.text[:200]}" @pytest.fixture(autouse=True) def _clean(app_server: str, db_ready: None) -> Iterator[None]: _clean_settings_state() _invalidate_app_cache(app_server) yield _clean_settings_state() _invalidate_app_cache(app_server) # The module's final state: app_settings EMPTY (the cross-suite # poison guard — the next suite's app must boot with no # overrides). with SessionLocal() as db: assert db.execute(text("SELECT count(*) FROM app_settings")).scalar() == 0 @pytest.fixture(scope="module", autouse=True) def _env_never_written() -> Iterator[None]: """LOCKED A1, mtime-pinned: no code path the suite drives opens ``.env`` for writing (the file is gitignored — a ``git status`` check cannot see it, so ``stat``'s mtime is the assertion).""" env_file = REPO / ".env" before = env_file.stat().st_mtime_ns if env_file.exists() else None yield after = env_file.stat().st_mtime_ns if env_file.exists() else None assert before == after, "the app under test must never write .env" def _cookies(page: Page) -> dict[str, str]: """The session cookies the browser context holds (the test's API side sees exactly what that browser sees).""" return { c["name"]: c["value"] for c in page.context.cookies() if "name" in c and "value" in c } def _admin_client(app_url: str) -> httpx.Client: """The API side of the form login (the 204 + the signed session cookie in the jar).""" client = httpx.Client(timeout=60.0) r = client.post(f"{app_url}/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, f"admin login → {r.status_code}: {r.text[:120]}" return client def _catalog_item(client: httpx.Client, app_url: str, key: str) -> dict[str, Any]: r = client.get(f"{app_url}/api/admin/settings", timeout=30.0) assert r.status_code == 200, r.text[:200] for item in r.json()["items"]: if item["key"] == key: return item raise AssertionError(f"{key} missing from the catalogue") def _raw_value(key: str) -> str | None: """The raw ``app_settings`` row for ``key`` (the overlay's D1 env-native string form), or None when no override is stored.""" with SessionLocal() as db: return db.execute( text("SELECT value FROM app_settings WHERE key = :k"), {"k": key} ).scalar_one_or_none() def _raw_count(key: str) -> int: with SessionLocal() as db: return db.execute( text("SELECT count(*) FROM app_settings WHERE key = :k"), {"k": key} ).scalar_one() # --------------------------------------------------------------------------- # 1. Both rows render free-form with defaults (chat ``low``, # summary empty = the A2 off switch; the old key is gone) # --------------------------------------------------------------------------- def test_both_rows_render_free_form_with_defaults( page: Page, app_url: str, db_ready: None ) -> None: page.set_default_timeout(30_000) login(page, app_url, next="/settings.html") expect(page.locator("#settings-content")).to_be_visible(timeout=15_000) expect(page.locator("#settings-disabled")).to_be_hidden() # the overlay is on client = _admin_client(app_url) # The LLM (aipi) group band is visible (the phase-171 band # contract — both rows sit in it)… expect(page.locator("tr.settings-group-row", has_text="LLM (aipi)")).to_be_visible() # The CHAT row: the label, the env-var display (the catalogue # pattern), the free-form text control (A4: type="text" — NOT # number/select; no allowlist, the endpoint is the source of # truth)… chat_row = page.locator('tr[data-key="llm_chat_thinking_level"]') expect(chat_row).to_be_visible() expect(chat_row.locator(".cell-clip")).to_have_text("Chat thinking level") expect(chat_row.locator(".settings-env")).to_have_text("BOR_LLM_CHAT_THINKING_LEVEL") assert chat_row.locator(".settings-control").get_attribute("type") == "text" # …chat stays at low (TODO L3): the env-pinned ``low`` with the # env badge (the env var is set — no app_settings row, no # code-default badge). expect(chat_row.locator(".settings-control")).to_have_value("low") expect(chat_row.locator(".settings-src")).to_have_text("env") expect(chat_row.locator(".settings-reset")).to_be_disabled() # The SUMMARY row: the label, the env-var display, the free-form # text control… summary_row = page.locator('tr[data-key="llm_summary_thinking_level"]') expect(summary_row).to_be_visible() expect(summary_row.locator(".cell-clip")).to_have_text("Summary thinking level") expect(summary_row.locator(".settings-env")).to_have_text( "BOR_LLM_SUMMARY_THINKING_LEVEL" ) assert summary_row.locator(".settings-control").get_attribute("type") == "text" # …and the TODO's "none" = the off switch (A2): an EMPTY input # with the env badge (the env var is SET to the empty string — # the badge reads env, not default). expect(summary_row.locator(".settings-control")).to_have_value("") expect(summary_row.locator(".settings-src")).to_have_text("env") expect(summary_row.locator(".settings-reset")).to_be_disabled() # The "like chat model" reference: the llm_chat_model row's # input is the SAME type="text" control (one assertion pair # covering both thinking rows). chat_model_type = page.locator( 'tr[data-key="llm_chat_model"] .settings-control' ).get_attribute("type") assert chat_model_type == "text" assert ( chat_row.locator(".settings-control").get_attribute("type") == summary_row.locator(".settings-control").get_attribute("type") == chat_model_type ) # The old phase-172 key is GONE (LOCKED A1 — the rename, not # add-on): no row for llm_thinking_level anywhere in the panel. assert page.locator('tr[data-key="llm_thinking_level"]').count() == 0 # The server agrees with the panel: both defaults, both sourced # from the env (the module pins), stored raw on no row. chat_item = _catalog_item(client, app_url, "llm_chat_thinking_level") assert chat_item["source"] == "env" and chat_item["value"] == "low" summary_item = _catalog_item(client, app_url, "llm_summary_thinking_level") assert summary_item["source"] == "env" and summary_item["value"] == "" assert _raw_count("llm_chat_thinking_level") == 0 assert _raw_count("llm_summary_thinking_level") == 0 # --------------------------------------------------------------------------- # 2. The saves: dirty → save → DB badge, EACH key's value persists # independently and survives a reload WITHOUT a restart # --------------------------------------------------------------------------- def test_saved_levels_persist_without_restart( page: Page, app_url: str, db_ready: None ) -> None: page.set_default_timeout(30_000) login(page, app_url, next="/settings.html") expect(page.locator("#settings-content")).to_be_visible(timeout=15_000) client = _admin_client(app_url) # The §7.4 Save lifecycle: type the summary's free-form custom # value (A4 — NOT a known pi level: no allowlist, the endpoint # is the source of truth) and the chat's bump — two dirty # rows, the hints appear… summary_row = page.locator('tr[data-key="llm_summary_thinking_level"]') chat_row = page.locator('tr[data-key="llm_chat_thinking_level"]') summary_row.locator(".settings-control").fill(SAVED_SUMMARY_LEVEL) chat_row.locator(".settings-control").fill(SAVED_CHAT_LEVEL) expect(page.locator("#settings-dirty")).to_have_text("2 unsaved changes") expect(page.locator("#settings-save")).to_be_enabled() # Save: the status line, then BOTH badges flip to DB (the # re-GET settled the rows). page.click("#settings-save") expect(page.locator("#settings-status")).to_contain_text( "Saved 2 settings.", timeout=30_000 ) expect(summary_row.locator(".settings-src")).to_have_text("DB") expect(chat_row.locator(".settings-src")).to_have_text("DB") # The effective values are the saved ones for EACH key # independently (the independence pin — the two rows' controls # are distinct and the saved values do not cross): the API # catalogue (what the LLM layer reads per request via # get_settings)… summary_item = _catalog_item(client, app_url, "llm_summary_thinking_level") assert summary_item["source"] == "db" and summary_item["value"] == SAVED_SUMMARY_LEVEL chat_item = _catalog_item(client, app_url, "llm_chat_thinking_level") assert chat_item["source"] == "db" and chat_item["value"] == SAVED_CHAT_LEVEL # …and the raw app_settings rows are stored RAW (the overlay's # D1 env-native string form), one per key. assert _raw_value("llm_summary_thinking_level") == SAVED_SUMMARY_LEVEL assert _raw_value("llm_chat_thinking_level") == SAVED_CHAT_LEVEL # No restart: the DB values survive a reload of /settings.html # (the panel re-reads the settled server state), the summary # row's custom value and the chat row's bump each still on # their OWN row with the DB badge still on. page.goto(app_url + "/settings.html") expect(page.locator("#settings-content")).to_be_visible(timeout=15_000) expect( page.locator('tr[data-key="llm_summary_thinking_level"] .settings-control') ).to_have_value(SAVED_SUMMARY_LEVEL, timeout=15_000) expect( page.locator('tr[data-key="llm_summary_thinking_level"] .settings-src') ).to_have_text("DB") expect( page.locator('tr[data-key="llm_chat_thinking_level"] .settings-control') ).to_have_value(SAVED_CHAT_LEVEL) expect( page.locator('tr[data-key="llm_chat_thinking_level"] .settings-src') ).to_have_text("DB") # --------------------------------------------------------------------------- # 3. The Resets: each row goes back to ITS environment default (the # per-row DELETE — task 02's contract, in the browser) # --------------------------------------------------------------------------- def test_reset_returns_to_env_defaults(page: Page, app_url: str, db_ready: None) -> None: page.set_default_timeout(30_000) login(page, app_url, next="/settings.html") expect(page.locator("#settings-content")).to_be_visible(timeout=15_000) client = _admin_client(app_url) # Establish saved overrides first (the Resets are disabled with # nothing to reset — the phase-168 render contract): save a # value on each row so both Resets go live… summary_row = page.locator('tr[data-key="llm_summary_thinking_level"]') chat_row = page.locator('tr[data-key="llm_chat_thinking_level"]') summary_row.locator(".settings-control").fill(SAVED_SUMMARY_LEVEL) chat_row.locator(".settings-control").fill(SAVED_CHAT_LEVEL) page.click("#settings-save") expect(page.locator("#settings-status")).to_contain_text( "Saved 2 settings.", timeout=30_000 ) # The re-GET settles the rows (DB badges) before the Resets go # live (the phase-168 render contract — the button tracks the # SETTLED source, not the click). expect(summary_row.locator(".settings-src")).to_have_text("DB") expect(chat_row.locator(".settings-src")).to_have_text("DB") expect(summary_row.locator(".settings-reset")).to_be_enabled() expect(chat_row.locator(".settings-reset")).to_be_enabled() # The summary Reset (the per-row DELETE): the status line… summary_row.locator(".settings-reset").click() expect(page.locator("#settings-status")).to_contain_text( "BOR_LLM_SUMMARY_THINKING_LEVEL is back to the environment value.", timeout=30_000, ) # …the value lands at EMPTY (the env var is set to the empty # string — the env badge, not the code-default badge) and the # row's Reset goes live-free again. expect(summary_row.locator(".settings-control")).to_have_value("") expect(summary_row.locator(".settings-src")).to_have_text("env") expect(summary_row.locator(".settings-reset")).to_be_disabled() # The chat Reset (the per-row DELETE): the status line… chat_row.locator(".settings-reset").click() expect(page.locator("#settings-status")).to_contain_text( "BOR_LLM_CHAT_THINKING_LEVEL is back to the environment value.", timeout=30_000, ) # …the value reads ``low`` again (chat stays at low, TODO L3) # with the env badge, and the row's Reset is disabled. expect(chat_row.locator(".settings-control")).to_have_value("low") expect(chat_row.locator(".settings-src")).to_have_text("env") expect(chat_row.locator(".settings-reset")).to_be_disabled() # The server agrees: both overrides are gone, the effective # values are the env defaults, and no raw row survives. summary_item = _catalog_item(client, app_url, "llm_summary_thinking_level") assert summary_item["source"] == "env" and summary_item["value"] == "" chat_item = _catalog_item(client, app_url, "llm_chat_thinking_level") assert chat_item["source"] == "env" and chat_item["value"] == "low" assert _raw_count("llm_summary_thinking_level") == 0 assert _raw_count("llm_chat_thinking_level") == 0