brain-of-reese/.agents/phases/complete/168_admin_settings_panel/00_phase.md

Phase 168 — Admin Settings panel: edit any BOR_* setting in the browser, DB over env, live on Save

Source: TODO.md L3 — "I want an admin panel that can edit anything in .env and save it to the database. The database value should override the env var. The change should take effect on clicking 'save' on that page." Story: n/a (owner-directed feature) Context: app/config.py (75 Settings fields, env_prefix="BOR_", env_file=".env", extra="ignore", get_settings() @lru_cached, ~40 call sites across app/), app/db.py (module-level settings = get_settings() + create_engine), app/main.py (settings = get_settings() at import, middleware wired at boot, _shell_routes), app/core/theming.py::effective_settings (the DB-over-env resolver precedent, B1), app/models.py::UiSettings (the single-row admin-persistence precedent), app/api/ui_settings.py + app/api/tokens.py (router-wide require_admin), app/core/auth.py::require_admin, frontend/index.html (#view-* shell views; #view-tokens L1077, #view-theme L1269), frontend/assets/router.js (VIEW / VIEW_PATH / VIEW_MODULES), frontend/assets/header.js (~L197-230, the ship-hidden / reveal-for-admin nav contract), frontend/assets/styles.css (.table-wrap L2825 + .table-wrap table thead th sticky L2841 — the phase-156 bounded-scroll precedent), app/core/caching.py::HTML_PAGES (L169), alembic/versions/ (next free: 0028), tests/e2e/conftest.py (the os.environ["BOR_PDF_PICTORIAL"] = "0" global-pin precedent, ~L52).

Objective

An admin opens Settings in the navbar, edits any setting that today lives only in .env (75 fields, grouped, type-correct inputs), clicks Save, and the new value is (a) stored in Postgres, (b) preferred over the env var from that instant on with no restart, and (c) visible in the running UI. Per-row Reset deletes the DB row so the env value wins again. The .env file is never written by the app.

Design (shared by all tasks — the executor reads this, not the chat)

D1 — Storage: one app_settings table (task 01). key = the bare Settings field name (relevance_threshold), value = the env-native string form ("0.55", "true", "md,txt", a JSON array for the one list[str] field), updated_at. Storing bare strings (not typed columns, not a wide row) is what makes "anything in .env" work for all 75 fields with zero per-field DDL, and keeps the coercion in exactly one place (the pydantic validators). The BOR_ prefix stays a presentation concern (env_var = "BOR_" + key.upper()), never stored — a stored prefix is a double-prefix typo waiting to happen.

D2 — The overlay lives INSIDE get_settings() (task 02). The ~40 existing call sites (app/rag/retriever.py, app/api/chat.py, middlewares, scripts/) must not change — that is the only way "anything" becomes live rather than "the three fields someone remembered to thread through". get_settings() keeps its @lru_cache and its zero-arg signature (tests/{integration,e2e}/*.py call get_settings.cache_clear() in 6 files — that surface must survive), and resolves: Settings() (env/.env, exactly today) → overlay the DB rows → validate → return. invalidate_settings() clears the cache and runs the reset hooks (D3). Hot-path cost after the first call: one lru_cache dict hit.

D3 — "Takes effect on Save" = cache + the two in-process singletons (task 03). Everything except two objects is read per use, so clearing the cache is enough for ~70 fields. The exceptions are app/api/chat.py::_llm (an LLMClient built once from get_settings() — holds llm_base_url / effective_api_key / llm_chat_model / llm_embed_model / timeouts / retries, L318-326) and _chat_semaphore (built once from chat_max_concurrent, L302-315). The overlay exposes a hook registry; app/main.py::create_app() registers the chat reset (_llm = None, _chat_semaphore = None) so app/core/ never imports app/api/ (dependency direction, PLAN §3). Settings wired at boot cannot be applied in-process (engine pools, SessionMiddleware, the static mount, the gzip middleware, logging) — they stay editable but carry a restart_required badge (LOCKED A4).

D4 — The catalog is code, derived-checked, and complete (task 02). app/core/settings_catalog.py declares, per field: group (mirrors app/config.py's own section comments: App & branding, HTTP & logging, Database, LLM, Chat history, RAG & retrieval, Agent tools, Import & sources, Images, PDF, Docs push, Admin & security), label, secret, restart_required, overridable. A unit test asserts the catalog covers exactly Settings.model_fields (a new setting added without catalog metadata fails the test → it can never silently disappear from the panel).

D5 — Fail loud on write, fail soft on read (tasks 02/04, LOCKED A5). A write validates the FULL candidate set — every stored override plus the new one, layered over the env base — through the real Settings validators, so a cross-field typo (lexical_support_floor above a DB-overridden relevance_threshold) is a 422 that names the field and writes nothing. A read that finds an unusable row (hand-edited, or a validator changed in a later release) skips THAT key, logs settings override ignored key=… reason=… and applies the rest: a DB value must never brick boot, because a bricked app cannot be un-bricked from the panel.

D6 — The kill switch, and the E2E poison guard (task 02). BOR_SETTINGS_DB=0 (env-only, never itself DB-overridable, read straight from os.environ) disables the overlay entirely — the pre-phase behavior, the house agent_max_rounds/compress_min_bytes convention. tests/e2e/conftest.py pins it to 0 for every E2E app the same way it already pins BOR_PDF_PICTORIAL: a row left behind by one suite would otherwise silently retune every later suite's app subprocess. The phase-168 suite opts back in on its own app fixture (the test_pictorial_pdf.py pattern) and TRUNCATE app_settings before AND after (the test_admin_theme_tab.py:286 precedent).

D7 — The API (task 04). app/api/admin_settings.py, prefix /admin/settings, router-wide dependencies=[Depends(require_admin)] (the ui_settings/tokens pattern: anonymous AND token users 403): GET the catalog + effective values with a source badge (db | env | default), PUT /{key} set one value, DELETE /{key} reset to env, DELETE reset all (the panel's "Clear all overrides"). Secrets are write-only (LOCKED A4): GET reports is_set, never the plaintext, and the write log line carries the key and the action, never the value.

D8 — The view (task 05). The shell's eighth view: #view-settings in frontend/index.html + frontend/assets/settings.js, /settings.html added to main.py::_shell_routes and caching.HTML_PAGES, VIEW/VIEW_PATH/VIEW_MODULES in router.js, and #nav-settings (ship-hidden) in index.html / document.html / login.html / shared.html revealed by header.js — the exact phase-91 Theme-view wiring. Layout per PLAN §7 + AGENTS rule 5: the 72rem .container, one .table-wrap card (bounded max-height: min(70vh, 64rem), sticky thead) with a <tbody> per group, type-correct controls (<input type=number> for int/float, a true/false <select> for bool, <textarea> for the CSV/JSON and long strings, a password input for secrets), a text source badge (B5: never color alone), a client-side type=search filter (the phase-101 tokens pattern), a page-level Save settings (dirty rows only), per-row Reset, and a role="status" / role="alert" live region (§7.4).

Dependencies

• 91_admin_theme_tab + 92_theme_save_and_coverage + 93_theme_semantic_completion (complete) — the DB-over-default resolver (B1), the admin-only shell view + gate pattern this view copies, and the nav reveal contract.
• 79_api_tokens (complete) — router-wide require_admin + the bounded, searchable admin catalogue table with per-row actions.
• 156_table_ux_polish (complete) — .table-wrap bounded scroll + sticky header, the cell-clip ellipsis contract for long action labels.
• 167_api_payload_diet (complete) — the response byte budget: the new GET /api/admin/settings (~75 rows ≈ 20 KB) must stay under BOR_LOG_LARGE_RESPONSE_BYTES and the audit script's 256 KB bar; never a 413.
• 82_security_headers + 81_login_rate_limit (complete) — the CSP/default-src 'self' posture the new JS must live inside (no CDN, rule 6).

Tasks

1. 01_app_settings_table.md — AppSetting model + reversible migration 0028_app_settings.py + its migration test.
2. 02_settings_overlay_resolver.md — app/core/settings_catalog.py + app/core/settings_overlay.py, get_settings() becomes the cached env+DB resolver, invalidate_settings(), the BOR_SETTINGS_DB kill switch + the E2E conftest pin, the catalog-completeness unit pins.
3. 03_live_apply_on_save.md — the reset-hook registry, the chat _llm / _chat_semaphore reset registered from create_app(), and the integration proof that a saved change bites without a restart.
4. 04_admin_settings_api.md — /api/admin/settings GET / PUT /{key} / DELETE /{key} / DELETE all, the schemas, the full-candidate validation, the masked secrets, the value-free log line, the README API note.
5. 05_settings_view_ui.md — the Settings shell view (#view-settings, settings.js, router + shell route + HTML_PAGES + nav reveal, .table-wrap catalogue, Save/Reset + live region).
6. 06_e2e_settings_panel.md — tests/e2e/test_admin_settings_panel.py, the regression set, the full gates, the atomic commit.

Testing & Quality

• Unit: tests/unit/test_settings_catalog.py (catalog ⇄ Settings.model_fields equality; the masked set is exactly llm_api_key + admin_password + session_secret + database_url (the last one is read-only but its URL carries credentials); the non-overridable set is exactly database_url + embedding_dim), tests/unit/test_settings_overlay.py (merge/coercion/validation-skip/BOR_SETTINGS_DB=0), tests/unit/test_settings_frontend_view.py (static source pins in the house style: the view section exists, router.js maps /settings.html, no CDN, settings.js builds DOM with createElement/textContent).
• Integration: tests/integration/test_migration_0028.py (up/down, A13), tests/integration/test_admin_settings_api.py (auth matrix anonymous/token-user/admin, 404 unknown key, 422 validator failures naming the field, masked secrets in the response body, the DB-over-env flip visible through GET /api/config, reset-to-env, get_settings() reflecting a write with no restart), plus the existing get_settings.cache_clear() suites staying green untouched.
• Coverage: >90% on app/ — uv run pytest --cov=app --cov-report=term-missing.
• E2E: uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov green in isolation; regression set green in isolation: test_admin_theme_tab.py, test_api_tokens.py, test_configurable_brand.py, test_global_tuning.py, test_admin_auth.py, test_cache_busting.py.
• Lint/types: uv run ruff check . && uv run pyright.
• Log contract (AGENTS rule 10): every settings write logs one line (settings action=set|clear|clear_all key=… source=db|env) with no value; GET /api/admin/settings appears in the grep ' :: api ' byte audit and under the 256 KB bar (uv run python -m scripts.api_payload_audit).

Completion Criteria

• /settings.html is an admin-only shell view: anonymous sees the sign-in gate, a token user never sees the nav link, and all four API routes 403 for both.
• Editing a value + Save settings stores it in app_settings, flips the row's badge from env/default to DB, and the effect is visible in the running app with no restart (pinned E2E: BOR_INPUT_PLACEHOLDER shows up in the composer; BOR_RELEVANCE_THRESHOLD / BOR_LLM_CHAT_MODEL are read from the overlay by a live request).
• Reset deletes the row and the env value wins again; Clear all overrides empties the table; .env is never opened for writing by any code path (unit-pinned: no open( on the env file in app/).
• Secrets never leave the server (GET body has no plaintext, page.content() has no secret), no write log line contains a value, and a bogus value is a 422 that names the field and stores nothing.
• A stored-but-invalid row (inserted directly, bypassing the API) is ignored at read with a logged reason — the app still boots and /api/health is 200.
• uv run pytest green, app/ coverage >90%, ruff + pyright clean, the phase E2E + the regression set green in isolation.
• One atomic --no-gpg-sign Conventional Commit; the phase dir moves to .agents/phases/complete/.

Locked decisions (owner-confirmed 2026-10-09 during phase authoring)

• A1 — DB-only persistence: the panel writes app_settings and never touches .env; the env var stays the base and the DB row wins.
• A2 — the overlay sits inside get_settings() (cached, cache_clear()-compatible) so all ~40 consumers go DB-over-env at once, behind the BOR_SETTINGS_DB=0 kill switch.
• A3 — database_url and embedding_dim are NOT DB-overridable (read-only rows, reason shown: the engine is built from database_url, so an override is a chicken-and-egg brick; A6 fixes chunks.embedding at 768, so a dim override would corrupt retrieval).
• A4 — boot-wired settings stay editable with a "restart required" badge (pools, session middleware, static mount, gzip, log level); secrets are masked and write-only and never appear in a response body or a log line.
• A5 — validate the full candidate set on write (422, nothing stored), fail soft on read (skip the bad key, log it, boot anyway).
• A6 — one story, one phase, one Playwright file (AGENTS rule 4) — the backend tasks are covered by unit + integration, the live-effect contract by tests/e2e/test_admin_settings_panel.py.

Assumptions (the TODO left these open; the executor does not guess)

• K1: the key is the bare field name; BOR_UPPER is derived for display/API presentation only.
• K2: the one list[str] field (suggestions) is edited as a JSON array in a <textarea> and stored in that same JSON form (pydantic parses it exactly like the env var); every other field is a scalar or CSV string.
• K3: a missing app_settings table (fresh deployment, migrations not applied) means "no overrides" and logs at DEBUG once — only an unusable value logs at ERROR.
• K4: "Save settings" sends only the dirty rows; the first failing row aborts the batch, shows its 422 detail, and keeps the other rows' edits in the form for correction.
• K5: history/audit beyond updated_at (who changed what, when, old value) is out of scope — single-admin homelab (PLAN §1 "out of scope").
• K6: the overlay is single-process (PLAN A12: in-process state, restart-clearing accepted, no Valkey); with more than one uvicorn worker a save is visible in the worker that served it and in every worker after its next resolution — the deployment runs one app process, and the panel notes nothing about it.
• K7: .env keys that are not Settings fields (extra="ignore", e.g. AIPI_KEY, DEBUGPY*) are not editable — they have no app/ consumer to retune. BOR_LLM_API_KEY remains the way to set the key (it wins over $AIPI_KEY via effective_api_key).

Commit

git add app/ alembic/ frontend/ tests/ .env.example README.md .agents/phases/ && git commit --no-gpg-sign -m "feat(settings): admin settings panel with DB-over-env overrides applied live on save"