brain-of-reese/tests/e2e/test_split_thinking_levels.py

"""Phase 178 E2E (Playwright): the SPLIT thinking levels — a Chat
thinking level row and a Summary thinking level row in the admin
Settings panel — the owner's acceptance for TWO independently
tunable, pi-style LLM reasoning knobs, proven in a real browser.

Source: ``TODO.md`` L3 — "Need a separate summary thinking level and
chat thinking level. Default thinking level for summary should be
none. Chat should stay at low"

Run in isolation (DB must be up: ``podman compose up -d db``):

    uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov

Test → contract mapping (one story, one phase, one isolated file):

1. ``test_both_rows_render_free_form_with_defaults`` — the rows:
   the ``LLM (aipi)`` group band is visible (the phase-171 band
   contract), a row labelled **Chat thinking level** with the
   ``BOR_LLM_CHAT_THINKING_LEVEL`` env-var display shows ``low``
   with the ``env`` badge (chat stays at low, TODO L3), a row
   labelled **Summary thinking level** with the
   ``BOR_LLM_SUMMARY_THINKING_LEVEL`` display shows an **empty**
   input with the ``env`` badge (the TODO's "none" = the phase-172
   off switch, LOCKED A2 — the parameter is ABSENT, never a
   literal "none"), BOTH inputs are ``type="text"`` (free-form —
   NOT ``number``/``select``, the A4 contract: no allowlist, the
   endpoint is the source of truth), the ``llm_chat_model`` row's
   input is the same control (the "like chat model" reference, one
   assertion pair), and ``tr[data-key="llm_thinking_level"]`` is
   **absent** (the old phase-172 key is gone, LOCKED A1).
2. ``test_saved_levels_persist_without_restart`` — the save (the
   phase-168 §7.4 Save lifecycle per row: dirty → save → ``DB``
   badge): ``think-soft`` (a free-form custom — NOT a known pi
   level, A4) is typed into the summary row and ``high`` into the
   chat row (the dirty hints appear), then BOTH are saved; the
   API catalogue (``_catalog_item``) AND the raw ``app_settings``
   rows carry the saved values for EACH key independently (the
   independence pin — the two rows' controls are distinct and the
   saved values do not cross), and a reload of ``/settings.html``
   still shows ``think-soft`` / ``high`` with the ``DB`` badges
   (no restart, the phase-168 contract).
3. ``test_reset_returns_to_env_defaults`` — the per-row Reset (the
   per-row ``DELETE``): after the saves, both Resets are live;
   clicking the summary Reset lands the value at **empty** with
   the ``env`` badge and clicking the chat Reset lands ``low``
   with the ``env`` badge (the overlay reset, task 02's contract,
   in the browser); the status lines show the house "… is back to
   the environment value." toast for each.

DB isolation: the shared e2e Postgres keeps ``app_settings``
(the overlay rows — a leftover row would retune every later
suite's app through the conftest pin's failure mode),
``ui_settings`` (the single row the caching middleware reads
for EVERY served page) and ``api_tokens`` rows across suites.
An autouse fixture truncates ``app_settings`` and
``ui_settings`` and deletes the ``e2e-``-labeled tokens before
AND after every test (never a TRUNCATE on ``api_tokens`` — the
shared DB may hold the owner's real tokens), and — because the
direct SQL truncate does NOT invalidate the app's settings cache
(the admin API is the only mutation surface that does) — follows
it with the app's own clear-all (``DELETE /api/admin/settings``;
without it, a test that types the SAME value its predecessor
saved sees the stale value as the served one, registers no dirty
state, and the Save button stays disabled). The module teardown
pins ``app_settings`` empty. The module app opts BACK IN over
the conftest's global ``BOR_SETTINGS_DB=0`` pin (``env["BOR_
SETTINGS_DB"] = "1"`` — the ``test_pictorial_pdf.py`` opt-in
pattern) and pins ``BOR_LLM_CHAT_THINKING_LEVEL=low`` AND
``BOR_LLM_SUMMARY_THINKING_LEVEL=`` (the empty string SET — so
both badges read ``env``, not ``default``, and an operator's
local ``.env`` cannot leak a different level into the suite;
the house leak-guard pattern) together with the "like chat
model" reference (``BOR_LLM_CHAT_MODEL``). A module-scope ``.env``
mtime guard pins LOCKED A1: no code path the suite drives opens
the env file for writing.

ASSUMPTION A5 (phase-locked): the per-turn-line assertion
(``thinking=<chat level>``) and the wire split (``chat_stream()``
→ chat level, ``chat()`` → summary level) live in the
unit/integration layer (tasks 03/04) — this suite proves the
settings surface only.
"""
from __future__ import annotations

import json
import os
import subprocess
import sys
from collections.abc import Iterator
from typing import Any

import httpx
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text

from app.config import Settings
from app.db import SessionLocal
from e2e.auth_helpers import login
from e2e.conftest import (
    ADMIN_PASSWORD,
    APP_PORT,
    REPO,
    SESSION_SECRET,
    USE_REAL_LLM,
    _wait_http,
)

APP_URL = f"http://127.0.0.1:{APP_PORT}"

#: The chat row's saved value (test 2) — a known pi level, the
#: operator's "bump the chat turn" case.
SAVED_CHAT_LEVEL = "high"
#: The summary row's saved value (test 2) — NOT a known pi level
#: (proves the A4 contract: no allowlist, the endpoint is the
#: source of truth).
SAVED_SUMMARY_LEVEL = "think-soft"


# ---------------------------------------------------------------------------
# Per-module app env (the phase-168 tuning pattern)
# ---------------------------------------------------------------------------


def _boot_app() -> subprocess.Popen[bytes]:
    """The one module app: the shared env block + the phase-178
    pins, with the overlay opted BACK IN (``BOR_SETTINGS_DB=1``)."""
    global _MOCK_PORT
    env = dict(os.environ)
    env.pop("DEBUGPY", None)
    env["BOR_ENVIRONMENT"] = "e2e"
    env["BOR_STATIC_DIR"] = str(REPO / "frontend")
    env["BOR_LLM_BASE_URL"] = (
        "https://aipi.reeseapps.com/v1"
        if USE_REAL_LLM
        else f"http://127.0.0.1:{_MOCK_PORT}/v1"
    )
    # The E2E-calibrated gate pins (the conftest pattern — the mock's
    # token-overlap distribution, explicit so an operator's local
    # .env cannot leak a different gate into the suite's app).
    env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
    env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
    env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
    env["BOR_LLM_RETRY_DELAY"] = "0"
    env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
    env.setdefault(
        "BOR_DATABASE_URL",
        "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
    )
    # Phase 16: admin auth must be set or create_app() refuses to boot.
    env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
    env["BOR_SESSION_SECRET"] = SESSION_SECRET
    env["BOR_DOCS_REPO"] = ""
    env["BOR_SUGGESTIONS"] = json.dumps(
        Settings.model_fields["suggestions"].default
    )
    # Leak guards: the branding base and the "like chat model"
    # reference row stay the CODE defaults (derived from the class
    # fields, never drift from app/config.py).
    env["BOR_APP_NAME"] = Settings.model_fields["app_name"].default
    env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
    env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
    env["BOR_GIT_SOURCES"] = ""
    env["BOR_LLM_CHAT_MODEL"] = Settings.model_fields["llm_chat_model"].default
    # Phase 168 (the panel suite's pattern): the module opts BACK IN
    # over the conftest's global BOR_SETTINGS_DB=0 pin (the
    # test_pictorial_pdf.py pattern) — the rows' save/reset
    # contract needs the overlay on.
    env["BOR_SETTINGS_DB"] = "1"
    # Phase 178 (task 05): pin BOTH thinking levels to their CODE
    # DEFAULTS — each env var is SET (so the badge reads ``env``,
    # not ``default``) and an operator's local .env cannot leak a
    # different level into the suite's app. The summary level is
    # set to EMPTY (the TODO's "none" = the off switch, A2).
    env["BOR_LLM_CHAT_THINKING_LEVEL"] = Settings.model_fields[
        "llm_chat_thinking_level"
    ].default
    env["BOR_LLM_SUMMARY_THINKING_LEVEL"] = Settings.model_fields[
        "llm_summary_thinking_level"
    ].default
    return subprocess.Popen(
        [sys.executable, "-m", "uvicorn", "app.main:app",
         "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
        cwd=REPO,
        env=env,
    )


def _stop_app(proc: subprocess.Popen[bytes]) -> None:
    proc.terminate()
    try:
        proc.wait(timeout=10)
    except subprocess.TimeoutExpired:
        proc.kill()


@pytest.fixture(scope="module")
def app_server(mock_llm: int) -> Iterator[str]:
    """The real app under test — per-module env: BOTH thinking
    levels pinned to their code defaults (chat ``low``, summary
    empty = the A2 off switch) and ``BOR_SETTINGS_DB=1`` (the
    module opts back in over the conftest pin)."""
    global _MOCK_PORT
    _MOCK_PORT = mock_llm
    proc = _boot_app()
    try:
        _wait_http(f"{APP_URL}/api/health")
        yield APP_URL
    finally:
        _stop_app(proc)


@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
    return app_server


# ---------------------------------------------------------------------------
# DB isolation + helpers
# ---------------------------------------------------------------------------


def _clean_settings_state() -> None:
    """Fresh settings + theme + token state per test: truncate
    ``app_settings`` (a leftover row is the cross-suite poison this
    phase's overlay contract exists to prevent — the rows are wiped
    between tests) and the single-row ``ui_settings`` (the
    middleware reads it for EVERY page), and delete this suite's
    issued tokens (label-scoped on ``e2e-`` — never a TRUNCATE: the
    shared DB may hold the owner's real tokens)."""
    with SessionLocal() as db:
        db.execute(text("TRUNCATE app_settings"))
        db.execute(text("TRUNCATE ui_settings"))
        db.execute(text("DELETE FROM api_tokens WHERE label LIKE 'e2e-%'"))
        db.commit()


def _invalidate_app_cache(app_url: str) -> None:
    """The direct SQL truncate does NOT invalidate the app's
    ``get_settings()`` lru-cache (the admin API is the only
    mutation surface that invalidates — the phase-168 contract)
    — so after the truncate, hit the app's own clear-all
    (``DELETE /api/admin/settings``): zero rows to delete, but the
    invalidate call rebuilds the cache env-only. Without it, a test
    that types the SAME value its predecessor saved sees the STALE
    value as the served one (``markDirty`` finds no diff, the Save
    button stays disabled)."""
    client = _admin_client(app_url)
    r = client.delete(f"{app_url}/api/admin/settings", timeout=30.0)
    assert r.status_code == 200, f"clear-all → {r.status_code}: {r.text[:200]}"


@pytest.fixture(autouse=True)
def _clean(app_server: str, db_ready: None) -> Iterator[None]:
    _clean_settings_state()
    _invalidate_app_cache(app_server)
    yield
    _clean_settings_state()
    _invalidate_app_cache(app_server)
    # The module's final state: app_settings EMPTY (the cross-suite
    # poison guard — the next suite's app must boot with no
    # overrides).
    with SessionLocal() as db:
        assert db.execute(text("SELECT count(*) FROM app_settings")).scalar() == 0


@pytest.fixture(scope="module", autouse=True)
def _env_never_written() -> Iterator[None]:
    """LOCKED A1, mtime-pinned: no code path the suite drives opens
    ``.env`` for writing (the file is gitignored — a ``git status``
    check cannot see it, so ``stat``'s mtime is the assertion)."""
    env_file = REPO / ".env"
    before = env_file.stat().st_mtime_ns if env_file.exists() else None
    yield
    after = env_file.stat().st_mtime_ns if env_file.exists() else None
    assert before == after, "the app under test must never write .env"


def _cookies(page: Page) -> dict[str, str]:
    """The session cookies the browser context holds (the test's API
    side sees exactly what that browser sees)."""
    return {
        c["name"]: c["value"]
        for c in page.context.cookies()
        if "name" in c and "value" in c
    }


def _admin_client(app_url: str) -> httpx.Client:
    """The API side of the form login (the 204 + the signed session
    cookie in the jar)."""
    client = httpx.Client(timeout=60.0)
    r = client.post(f"{app_url}/api/login", json={"password": ADMIN_PASSWORD})
    assert r.status_code == 204, f"admin login → {r.status_code}: {r.text[:120]}"
    return client


def _catalog_item(client: httpx.Client, app_url: str, key: str) -> dict[str, Any]:
    r = client.get(f"{app_url}/api/admin/settings", timeout=30.0)
    assert r.status_code == 200, r.text[:200]
    for item in r.json()["items"]:
        if item["key"] == key:
            return item
    raise AssertionError(f"{key} missing from the catalogue")


def _raw_value(key: str) -> str | None:
    """The raw ``app_settings`` row for ``key`` (the overlay's D1
    env-native string form), or None when no override is stored."""
    with SessionLocal() as db:
        return db.execute(
            text("SELECT value FROM app_settings WHERE key = :k"), {"k": key}
        ).scalar_one_or_none()


def _raw_count(key: str) -> int:
    with SessionLocal() as db:
        return db.execute(
            text("SELECT count(*) FROM app_settings WHERE key = :k"), {"k": key}
        ).scalar_one()


# ---------------------------------------------------------------------------
# 1. Both rows render free-form with defaults (chat ``low``,
#    summary empty = the A2 off switch; the old key is gone)
# ---------------------------------------------------------------------------


def test_both_rows_render_free_form_with_defaults(
    page: Page, app_url: str, db_ready: None
) -> None:
    page.set_default_timeout(30_000)
    login(page, app_url, next="/settings.html")
    expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
    expect(page.locator("#settings-disabled")).to_be_hidden()  # the overlay is on
    client = _admin_client(app_url)

    # The LLM (aipi) group band is visible (the phase-171 band
    # contract — both rows sit in it)…
    expect(page.locator("tr.settings-group-row", has_text="LLM (aipi)")).to_be_visible()

    # The CHAT row: the label, the env-var display (the catalogue
    # pattern), the free-form text control (A4: type="text" — NOT
    # number/select; no allowlist, the endpoint is the source of
    # truth)…
    chat_row = page.locator('tr[data-key="llm_chat_thinking_level"]')
    expect(chat_row).to_be_visible()
    expect(chat_row.locator(".cell-clip")).to_have_text("Chat thinking level")
    expect(chat_row.locator(".settings-env")).to_have_text("BOR_LLM_CHAT_THINKING_LEVEL")
    assert chat_row.locator(".settings-control").get_attribute("type") == "text"
    # …chat stays at low (TODO L3): the env-pinned ``low`` with the
    # env badge (the env var is set — no app_settings row, no
    # code-default badge).
    expect(chat_row.locator(".settings-control")).to_have_value("low")
    expect(chat_row.locator(".settings-src")).to_have_text("env")
    expect(chat_row.locator(".settings-reset")).to_be_disabled()

    # The SUMMARY row: the label, the env-var display, the free-form
    # text control…
    summary_row = page.locator('tr[data-key="llm_summary_thinking_level"]')
    expect(summary_row).to_be_visible()
    expect(summary_row.locator(".cell-clip")).to_have_text("Summary thinking level")
    expect(summary_row.locator(".settings-env")).to_have_text(
        "BOR_LLM_SUMMARY_THINKING_LEVEL"
    )
    assert summary_row.locator(".settings-control").get_attribute("type") == "text"
    # …and the TODO's "none" = the off switch (A2): an EMPTY input
    # with the env badge (the env var is SET to the empty string —
    # the badge reads env, not default).
    expect(summary_row.locator(".settings-control")).to_have_value("")
    expect(summary_row.locator(".settings-src")).to_have_text("env")
    expect(summary_row.locator(".settings-reset")).to_be_disabled()

    # The "like chat model" reference: the llm_chat_model row's
    # input is the SAME type="text" control (one assertion pair
    # covering both thinking rows).
    chat_model_type = page.locator(
        'tr[data-key="llm_chat_model"] .settings-control'
    ).get_attribute("type")
    assert chat_model_type == "text"
    assert (
        chat_row.locator(".settings-control").get_attribute("type")
        == summary_row.locator(".settings-control").get_attribute("type")
        == chat_model_type
    )

    # The old phase-172 key is GONE (LOCKED A1 — the rename, not
    # add-on): no row for llm_thinking_level anywhere in the panel.
    assert page.locator('tr[data-key="llm_thinking_level"]').count() == 0

    # The server agrees with the panel: both defaults, both sourced
    # from the env (the module pins), stored raw on no row.
    chat_item = _catalog_item(client, app_url, "llm_chat_thinking_level")
    assert chat_item["source"] == "env" and chat_item["value"] == "low"
    summary_item = _catalog_item(client, app_url, "llm_summary_thinking_level")
    assert summary_item["source"] == "env" and summary_item["value"] == ""
    assert _raw_count("llm_chat_thinking_level") == 0
    assert _raw_count("llm_summary_thinking_level") == 0


# ---------------------------------------------------------------------------
# 2. The saves: dirty → save → DB badge, EACH key's value persists
#    independently and survives a reload WITHOUT a restart
# ---------------------------------------------------------------------------


def test_saved_levels_persist_without_restart(
    page: Page, app_url: str, db_ready: None
) -> None:
    page.set_default_timeout(30_000)
    login(page, app_url, next="/settings.html")
    expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
    client = _admin_client(app_url)

    # The §7.4 Save lifecycle: type the summary's free-form custom
    # value (A4 — NOT a known pi level: no allowlist, the endpoint
    # is the source of truth) and the chat's bump — two dirty
    # rows, the hints appear…
    summary_row = page.locator('tr[data-key="llm_summary_thinking_level"]')
    chat_row = page.locator('tr[data-key="llm_chat_thinking_level"]')
    summary_row.locator(".settings-control").fill(SAVED_SUMMARY_LEVEL)
    chat_row.locator(".settings-control").fill(SAVED_CHAT_LEVEL)
    expect(page.locator("#settings-dirty")).to_have_text("2 unsaved changes")
    expect(page.locator("#settings-save")).to_be_enabled()

    # Save: the status line, then BOTH badges flip to DB (the
    # re-GET settled the rows).
    page.click("#settings-save")
    expect(page.locator("#settings-status")).to_contain_text(
        "Saved 2 settings.", timeout=30_000
    )
    expect(summary_row.locator(".settings-src")).to_have_text("DB")
    expect(chat_row.locator(".settings-src")).to_have_text("DB")

    # The effective values are the saved ones for EACH key
    # independently (the independence pin — the two rows' controls
    # are distinct and the saved values do not cross): the API
    # catalogue (what the LLM layer reads per request via
    # get_settings)…
    summary_item = _catalog_item(client, app_url, "llm_summary_thinking_level")
    assert summary_item["source"] == "db" and summary_item["value"] == SAVED_SUMMARY_LEVEL
    chat_item = _catalog_item(client, app_url, "llm_chat_thinking_level")
    assert chat_item["source"] == "db" and chat_item["value"] == SAVED_CHAT_LEVEL
    # …and the raw app_settings rows are stored RAW (the overlay's
    # D1 env-native string form), one per key.
    assert _raw_value("llm_summary_thinking_level") == SAVED_SUMMARY_LEVEL
    assert _raw_value("llm_chat_thinking_level") == SAVED_CHAT_LEVEL

    # No restart: the DB values survive a reload of /settings.html
    # (the panel re-reads the settled server state), the summary
    # row's custom value and the chat row's bump each still on
    # their OWN row with the DB badge still on.
    page.goto(app_url + "/settings.html")
    expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
    expect(
        page.locator('tr[data-key="llm_summary_thinking_level"] .settings-control')
    ).to_have_value(SAVED_SUMMARY_LEVEL, timeout=15_000)
    expect(
        page.locator('tr[data-key="llm_summary_thinking_level"] .settings-src')
    ).to_have_text("DB")
    expect(
        page.locator('tr[data-key="llm_chat_thinking_level"] .settings-control')
    ).to_have_value(SAVED_CHAT_LEVEL)
    expect(
        page.locator('tr[data-key="llm_chat_thinking_level"] .settings-src')
    ).to_have_text("DB")


# ---------------------------------------------------------------------------
# 3. The Resets: each row goes back to ITS environment default (the
#    per-row DELETE — task 02's contract, in the browser)
# ---------------------------------------------------------------------------


def test_reset_returns_to_env_defaults(page: Page, app_url: str, db_ready: None) -> None:
    page.set_default_timeout(30_000)
    login(page, app_url, next="/settings.html")
    expect(page.locator("#settings-content")).to_be_visible(timeout=15_000)
    client = _admin_client(app_url)

    # Establish saved overrides first (the Resets are disabled with
    # nothing to reset — the phase-168 render contract): save a
    # value on each row so both Resets go live…
    summary_row = page.locator('tr[data-key="llm_summary_thinking_level"]')
    chat_row = page.locator('tr[data-key="llm_chat_thinking_level"]')
    summary_row.locator(".settings-control").fill(SAVED_SUMMARY_LEVEL)
    chat_row.locator(".settings-control").fill(SAVED_CHAT_LEVEL)
    page.click("#settings-save")
    expect(page.locator("#settings-status")).to_contain_text(
        "Saved 2 settings.", timeout=30_000
    )
    # The re-GET settles the rows (DB badges) before the Resets go
    # live (the phase-168 render contract — the button tracks the
    # SETTLED source, not the click).
    expect(summary_row.locator(".settings-src")).to_have_text("DB")
    expect(chat_row.locator(".settings-src")).to_have_text("DB")
    expect(summary_row.locator(".settings-reset")).to_be_enabled()
    expect(chat_row.locator(".settings-reset")).to_be_enabled()

    # The summary Reset (the per-row DELETE): the status line…
    summary_row.locator(".settings-reset").click()
    expect(page.locator("#settings-status")).to_contain_text(
        "BOR_LLM_SUMMARY_THINKING_LEVEL is back to the environment value.",
        timeout=30_000,
    )
    # …the value lands at EMPTY (the env var is set to the empty
    # string — the env badge, not the code-default badge) and the
    # row's Reset goes live-free again.
    expect(summary_row.locator(".settings-control")).to_have_value("")
    expect(summary_row.locator(".settings-src")).to_have_text("env")
    expect(summary_row.locator(".settings-reset")).to_be_disabled()

    # The chat Reset (the per-row DELETE): the status line…
    chat_row.locator(".settings-reset").click()
    expect(page.locator("#settings-status")).to_contain_text(
        "BOR_LLM_CHAT_THINKING_LEVEL is back to the environment value.",
        timeout=30_000,
    )
    # …the value reads ``low`` again (chat stays at low, TODO L3)
    # with the env badge, and the row's Reset is disabled.
    expect(chat_row.locator(".settings-control")).to_have_value("low")
    expect(chat_row.locator(".settings-src")).to_have_text("env")
    expect(chat_row.locator(".settings-reset")).to_be_disabled()

    # The server agrees: both overrides are gone, the effective
    # values are the env defaults, and no raw row survives.
    summary_item = _catalog_item(client, app_url, "llm_summary_thinking_level")
    assert summary_item["source"] == "env" and summary_item["value"] == ""
    chat_item = _catalog_item(client, app_url, "llm_chat_thinking_level")
    assert chat_item["source"] == "env" and chat_item["value"] == "low"
    assert _raw_count("llm_summary_thinking_level") == 0
    assert _raw_count("llm_chat_thinking_level") == 0