phase: 168_admin_settings_panel

**Phase 168 — Admin Settings panel — final verification: PASS**

1. **Implemented/verified:** `/settings.html` (admin-only) with the 75-field catalogue (`app/core/settings_catalog.py`), DB-over-env overrides in Postgres (`app_settings`, migration 0028 at head, `settings_overlay.py` resolver, `app/api/admin_settings.py` — 4 admin-only routes), live-apply on save with no restart + restart-required badges, Reset/Clear all, write-only secrets, `.env`-never-written, `BOR_SETTINGS_DB=0` kill switch.
2. **Phase E2E: 12/12** — run with `E2E_APP_PORT=8200` (ports 8123/8124 are held by an unrelated "undercooked" project; no process killed, no code change — the documented env override).
3. **Regression set: all 7 suites green in isolation** (theme tab, API tokens, configurable brand, global tuning, admin auth, cache busting, wide desktop column — 38 tests).
4. **Full suite: unit+integration exit 0 (4066 passed), app/ coverage 99%** (gate >90%); **ruff clean, pyright 0 errors / 0 warnings.**
5. **Log contract (rule 10) verified live:** per-turn + per-response byte-budget lines present; settings write lines carry key + action, **never the value**.
6. **Byte-budget audit: all endpoints under the 262,144 B budget**; settings catalogue 23,513 B (gzip 3,326).
7. **Defect fixed (one):** criterion 5 had no integration pin — added `test_bad_row_inserted_directly_never_bricks_boot` to `tests/integration/test_admin_settings_api.py` (malformed row inserted directly → fresh `create_app()` boots, row skipped with logged reason, rest of table applies, `/api/health` 200) — 12/12 green.
8. **Docs pass:** `README.md` and `AGENTS.md` reviewed in full — both consistent with the as-built feature (Settings page table, Settings section, Configuration-reference note, `.env.example` block); **no changes required**.
9. **Cleanup:** audit app killed (by PID); `app_settings` left empty (0 rows); all changes left in the working tree (no commit, per protocol).
10. **Completion criteria:** all 7 checkboxes verified green (criterion 7's commit is the harness's).
11. **Next pending phase:** `170_edit_bubble_sizing`.

.agents/phases/complete/168_admin_settings_panel/00_phase.md

@@ -0,0 +1,81 @@
+# Phase 168 — Admin Settings panel: edit any `BOR_*` setting in the browser, DB over env, live on Save
+
+**Source:** `TODO.md` L3 — "I want an admin panel that can edit anything in .env and save it to the database. The database value should override the env var. The change should take effect on clicking 'save' on that page."
+**Story:** n/a (owner-directed feature)
+**Context:** `app/config.py` (75 `Settings` fields, `env_prefix="BOR_"`, `env_file=".env"`, `extra="ignore"`, `get_settings()` `@lru_cache`d, ~40 call sites across `app/`), `app/db.py` (module-level `settings = get_settings()` + `create_engine`), `app/main.py` (`settings = get_settings()` at import, middleware wired at boot, `_shell_routes`), `app/core/theming.py::effective_settings` (the DB-over-env resolver precedent, B1), `app/models.py::UiSettings` (the single-row admin-persistence precedent), `app/api/ui_settings.py` + `app/api/tokens.py` (router-wide `require_admin`), `app/core/auth.py::require_admin`, `frontend/index.html` (`#view-*` shell views; `#view-tokens` L1077, `#view-theme` L1269), `frontend/assets/router.js` (`VIEW` / `VIEW_PATH` / `VIEW_MODULES`), `frontend/assets/header.js` (~L197-230, the ship-hidden / reveal-for-admin nav contract), `frontend/assets/styles.css` (`.table-wrap` L2825 + `.table-wrap table thead th` sticky L2841 — the phase-156 bounded-scroll precedent), `app/core/caching.py::HTML_PAGES` (L169), `alembic/versions/` (next free: **0028**), `tests/e2e/conftest.py` (the `os.environ["BOR_PDF_PICTORIAL"] = "0"` global-pin precedent, ~L52).
+
+## Objective
+
+An admin opens **Settings** in the navbar, edits any setting that today lives only in `.env` (75 fields, grouped, type-correct inputs), clicks **Save**, and the new value is (a) stored in Postgres, (b) preferred over the env var from that instant on with **no restart**, and (c) visible in the running UI. Per-row **Reset** deletes the DB row so the env value wins again. The `.env` file is never written by the app.
+
+## Design (shared by all tasks — the executor reads this, not the chat)
+
+**D1 — Storage: one `app_settings` table (task 01).** `key` = the **bare `Settings` field name** (`relevance_threshold`), `value` = the **env-native string form** (`"0.55"`, `"true"`, `"md,txt"`, a JSON array for the one `list[str]` field), `updated_at`. Storing bare strings (not typed columns, not a wide row) is what makes "anything in .env" work for all 75 fields with zero per-field DDL, and keeps the coercion in exactly one place (the pydantic validators). The `BOR_` prefix stays a *presentation* concern (`env_var = "BOR_" + key.upper()`), never stored — a stored prefix is a double-prefix typo waiting to happen.
+
+**D2 — The overlay lives INSIDE `get_settings()` (task 02).** The ~40 existing call sites (`app/rag/retriever.py`, `app/api/chat.py`, middlewares, `scripts/`) must not change — that is the only way "anything" becomes live rather than "the three fields someone remembered to thread through". `get_settings()` keeps its `@lru_cache` and its zero-arg signature (`tests/{integration,e2e}/*.py` call `get_settings.cache_clear()` in 6 files — that surface must survive), and resolves: `Settings()` (env/`.env`, exactly today) → overlay the DB rows → validate → return. `invalidate_settings()` clears the cache and runs the reset hooks (D3). Hot-path cost after the first call: one `lru_cache` dict hit.
+
+**D3 — "Takes effect on Save" = cache + the two in-process singletons (task 03).** Everything except two objects is read per use, so clearing the cache is enough for ~70 fields. The exceptions are `app/api/chat.py::_llm` (an `LLMClient` built once from `get_settings()` — holds `llm_base_url` / `effective_api_key` / `llm_chat_model` / `llm_embed_model` / timeouts / retries, L318-326) and `_chat_semaphore` (built once from `chat_max_concurrent`, L302-315). The overlay exposes a hook registry; `app/main.py::create_app()` registers the chat reset (`_llm = None`, `_chat_semaphore = None`) so `app/core/` never imports `app/api/` (dependency direction, PLAN §3). Settings wired at boot cannot be applied in-process (engine pools, `SessionMiddleware`, the static mount, the gzip middleware, logging) — they stay **editable** but carry a `restart_required` badge (LOCKED A4).
+
+**D4 — The catalog is code, derived-checked, and complete (task 02).** `app/core/settings_catalog.py` declares, per field: `group` (mirrors `app/config.py`'s own section comments: App & branding, HTTP & logging, Database, LLM, Chat history, RAG & retrieval, Agent tools, Import & sources, Images, PDF, Docs push, Admin & security), `label`, `secret`, `restart_required`, `overridable`. A unit test asserts the catalog covers **exactly** `Settings.model_fields` (a new setting added without catalog metadata fails the test → it can never silently disappear from the panel).
+
+**D5 — Fail loud on write, fail soft on read (tasks 02/04, LOCKED A5).** A write validates the FULL candidate set — every stored override plus the new one, layered over the env base — through the real `Settings` validators, so a cross-field typo (`lexical_support_floor` above a DB-overridden `relevance_threshold`) is a 422 that names the field and writes nothing. A read that finds an unusable row (hand-edited, or a validator changed in a later release) skips THAT key, logs `settings override ignored key=… reason=…` and applies the rest: a DB value must never brick boot, because a bricked app cannot be un-bricked from the panel.
+
+**D6 — The kill switch, and the E2E poison guard (task 02).** `BOR_SETTINGS_DB=0` (env-only, never itself DB-overridable, read straight from `os.environ`) disables the overlay entirely — the pre-phase behavior, the house `agent_max_rounds`/`compress_min_bytes` convention. `tests/e2e/conftest.py` pins it to `0` for **every** E2E app the same way it already pins `BOR_PDF_PICTORIAL`: a row left behind by one suite would otherwise silently retune every later suite's app subprocess. The phase-168 suite opts back in on its own app fixture (the `test_pictorial_pdf.py` pattern) and `TRUNCATE app_settings` before AND after (the `test_admin_theme_tab.py:286` precedent).
+
+**D7 — The API (task 04).** `app/api/admin_settings.py`, prefix `/admin/settings`, router-wide `dependencies=[Depends(require_admin)]` (the `ui_settings`/`tokens` pattern: anonymous AND token users 403): `GET` the catalog + effective values with a `source` badge (`db` | `env` | `default`), `PUT /{key}` set one value, `DELETE /{key}` reset to env, `DELETE` reset all (the panel's "Clear all overrides"). Secrets are **write-only** (LOCKED A4): `GET` reports `is_set`, never the plaintext, and the write log line carries the key and the action, never the value.
+
+**D8 — The view (task 05).** The shell's eighth view: `#view-settings` in `frontend/index.html` + `frontend/assets/settings.js`, `/settings.html` added to `main.py::_shell_routes` and `caching.HTML_PAGES`, `VIEW`/`VIEW_PATH`/`VIEW_MODULES` in `router.js`, and `#nav-settings` (ship-hidden) in `index.html` / `document.html` / `login.html` / `shared.html` revealed by `header.js` — the exact phase-91 Theme-view wiring. Layout per PLAN §7 + AGENTS rule 5: the 72rem `.container`, one `.table-wrap` card (bounded `max-height: min(70vh, 64rem)`, sticky `thead`) with a `<tbody>` per group, type-correct controls (`<input type=number>` for int/float, a `true/false` `<select>` for bool, `<textarea>` for the CSV/JSON and long strings, a password input for secrets), a text source badge (B5: never color alone), a client-side `type=search` filter (the phase-101 tokens pattern), a page-level **Save settings** (dirty rows only), per-row **Reset**, and a `role="status"` / `role="alert"` live region (§7.4).
+
+## Dependencies
+- `91_admin_theme_tab` + `92_theme_save_and_coverage` + `93_theme_semantic_completion` (complete) — the DB-over-default resolver (B1), the admin-only shell view + gate pattern this view copies, and the nav reveal contract.
+- `79_api_tokens` (complete) — router-wide `require_admin` + the bounded, searchable admin catalogue table with per-row actions.
+- `156_table_ux_polish` (complete) — `.table-wrap` bounded scroll + sticky header, the `cell-clip` ellipsis contract for long action labels.
+- `167_api_payload_diet` (complete) — the response byte budget: the new `GET /api/admin/settings` (~75 rows ≈ 20 KB) must stay under `BOR_LOG_LARGE_RESPONSE_BYTES` and the audit script's 256 KB bar; never a 413.
+- `82_security_headers` + `81_login_rate_limit` (complete) — the CSP/`default-src 'self'` posture the new JS must live inside (no CDN, rule 6).
+
+## Tasks
+1. `01_app_settings_table.md` — `AppSetting` model + reversible migration `0028_app_settings.py` + its migration test.
+2. `02_settings_overlay_resolver.md` — `app/core/settings_catalog.py` + `app/core/settings_overlay.py`, `get_settings()` becomes the cached env+DB resolver, `invalidate_settings()`, the `BOR_SETTINGS_DB` kill switch + the E2E conftest pin, the catalog-completeness unit pins.
+3. `03_live_apply_on_save.md` — the reset-hook registry, the chat `_llm` / `_chat_semaphore` reset registered from `create_app()`, and the integration proof that a saved change bites **without a restart**.
+4. `04_admin_settings_api.md` — `/api/admin/settings` GET / PUT `/{key}` / DELETE `/{key}` / DELETE all, the schemas, the full-candidate validation, the masked secrets, the value-free log line, the README API note.
+5. `05_settings_view_ui.md` — the Settings shell view (`#view-settings`, `settings.js`, router + shell route + `HTML_PAGES` + nav reveal, `.table-wrap` catalogue, Save/Reset + live region).
+6. `06_e2e_settings_panel.md` — `tests/e2e/test_admin_settings_panel.py`, the regression set, the full gates, the atomic commit.
+
+## Testing & Quality
+- **Unit:** `tests/unit/test_settings_catalog.py` (catalog ⇄ `Settings.model_fields` equality; the masked set is exactly `llm_api_key` + `admin_password` + `session_secret` + `database_url` (the last one is read-only but its URL carries credentials); the non-overridable set is exactly `database_url` + `embedding_dim`), `tests/unit/test_settings_overlay.py` (merge/coercion/validation-skip/`BOR_SETTINGS_DB=0`), `tests/unit/test_settings_frontend_view.py` (static source pins in the house style: the view section exists, `router.js` maps `/settings.html`, no CDN, `settings.js` builds DOM with `createElement`/`textContent`).
+- **Integration:** `tests/integration/test_migration_0028.py` (up/down, A13), `tests/integration/test_admin_settings_api.py` (auth matrix anonymous/token-user/admin, 404 unknown key, 422 validator failures naming the field, masked secrets in the response body, the DB-over-env flip visible through `GET /api/config`, reset-to-env, `get_settings()` reflecting a write with no restart), plus the existing `get_settings.cache_clear()` suites staying green untouched.
+- **Coverage:** **>90%** on `app/` — `uv run pytest --cov=app --cov-report=term-missing`.
+- **E2E:** `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` green **in isolation**; regression set green in isolation: `test_admin_theme_tab.py`, `test_api_tokens.py`, `test_configurable_brand.py`, `test_global_tuning.py`, `test_admin_auth.py`, `test_cache_busting.py`.
+- **Lint/types:** `uv run ruff check . && uv run pyright`.
+- **Log contract (AGENTS rule 10):** every settings write logs one line (`settings action=set|clear|clear_all key=… source=db|env`) with no value; `GET /api/admin/settings` appears in the `grep ' :: api '` byte audit and under the 256 KB bar (`uv run python -m scripts.api_payload_audit`).
+
+## Completion Criteria
+- [ ] `/settings.html` is an admin-only shell view: anonymous sees the sign-in gate, a token user never sees the nav link, and all four API routes 403 for both.
+- [ ] Editing a value + **Save settings** stores it in `app_settings`, flips the row's badge from `env`/`default` to `DB`, and the effect is visible in the running app with **no restart** (pinned E2E: `BOR_INPUT_PLACEHOLDER` shows up in the composer; `BOR_RELEVANCE_THRESHOLD` / `BOR_LLM_CHAT_MODEL` are read from the overlay by a live request).
+- [ ] **Reset** deletes the row and the env value wins again; **Clear all overrides** empties the table; `.env` is never opened for writing by any code path (unit-pinned: no `open(` on the env file in `app/`).
+- [ ] Secrets never leave the server (`GET` body has no plaintext, `page.content()` has no secret), no write log line contains a value, and a bogus value is a 422 that names the field and stores nothing.
+- [ ] A stored-but-invalid row (inserted directly, bypassing the API) is ignored at read with a logged reason — the app still boots and `/api/health` is 200.
+- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean, the phase E2E + the regression set green in isolation.
+- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
+
+## Locked decisions (owner-confirmed 2026-10-09 during phase authoring)
+- **A1 — DB-only persistence:** the panel writes `app_settings` and **never** touches `.env`; the env var stays the base and the DB row wins.
+- **A2 — the overlay sits inside `get_settings()`** (cached, `cache_clear()`-compatible) so all ~40 consumers go DB-over-env at once, behind the `BOR_SETTINGS_DB=0` kill switch.
+- **A3 — `database_url` and `embedding_dim` are NOT DB-overridable** (read-only rows, reason shown: the engine is built from `database_url`, so an override is a chicken-and-egg brick; A6 fixes `chunks.embedding` at 768, so a dim override would corrupt retrieval).
+- **A4 — boot-wired settings stay editable with a "restart required" badge** (pools, session middleware, static mount, gzip, log level); **secrets are masked and write-only** and never appear in a response body or a log line.
+- **A5 — validate the full candidate set on write (422, nothing stored), fail soft on read** (skip the bad key, log it, boot anyway).
+- **A6 — one story, one phase, one Playwright file** (AGENTS rule 4) — the backend tasks are covered by unit + integration, the live-effect contract by `tests/e2e/test_admin_settings_panel.py`.
+
+## Assumptions (the TODO left these open; the executor does not guess)
+- **K1:** the key is the bare field name; `BOR_UPPER` is derived for display/API presentation only.
+- **K2:** the one `list[str]` field (`suggestions`) is edited as a JSON array in a `<textarea>` and stored in that same JSON form (pydantic parses it exactly like the env var); every other field is a scalar or CSV string.
+- **K3:** a missing `app_settings` table (fresh deployment, migrations not applied) means "no overrides" and logs at DEBUG once — only an unusable *value* logs at ERROR.
+- **K4:** "Save settings" sends only the dirty rows; the first failing row aborts the batch, shows its 422 detail, and keeps the other rows' edits in the form for correction.
+- **K5:** history/audit beyond `updated_at` (who changed what, when, old value) is out of scope — single-admin homelab (PLAN §1 "out of scope").
+- **K6:** the overlay is single-process (PLAN A12: in-process state, restart-clearing accepted, no Valkey); with more than one uvicorn worker a save is visible in the worker that served it and in every worker after its next resolution — the deployment runs one app process, and the panel notes nothing about it.
+- **K7:** `.env` keys that are not `Settings` fields (`extra="ignore"`, e.g. `AIPI_KEY`, `DEBUGPY*`) are not editable — they have no `app/` consumer to retune. `BOR_LLM_API_KEY` remains the way to set the key (it wins over `$AIPI_KEY` via `effective_api_key`).
+
+## Commit
+```bash
+git add app/ alembic/ frontend/ tests/ .env.example README.md .agents/phases/ && git commit --no-gpg-sign -m "feat(settings): admin settings panel with DB-over-env overrides applied live on save"
+```