phase: 02_overview_fleet_and_alerts

**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.** - Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live). - Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`. - Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`. - Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`. - `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**. - Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`. - Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅. - Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent. - Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward. - Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
2026-10-17 00:59:09 -04:00
parent ad993c7b03
commit 8b048599f3
34 changed files with 1955 additions and 225 deletions
@@ -0,0 +1,12 @@
 **Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**
 - Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
 - Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
 - Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
 - Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
 - `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
 - Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
 - Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
 - Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
 - Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
 - Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,46 @@
 == uv run pytest --cov (hermetic: no network, live tests skip) ==
 ........................................................................ [ 16%]
 ........................................................................ [ 32%]
 ........................................................................ [ 48%]
 .......ssssss........................................................... [ 64%]
 ........................................................................ [ 80%]
 ........................................................................ [ 97%]
 .............                                                            [100%]
 =============================== warnings summary ===============================
 .venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.14.4-final-0 ________________
 Name                                 Stmts   Miss  Cover   Missing
 ------------------------------------------------------------------
 src/dashboards/__init__.py               3      0   100%
 src/dashboards/__main__.py              14      1    93%   28
 src/dashboards/alerts.py               175      0   100%
 src/dashboards/api.py                  193      0   100%
 src/dashboards/app.py                  109      6    94%   50-51, 56, 58, 60, 131
 src/dashboards/collectors.py           140      5    96%   92-93, 97-98, 115
 src/dashboards/config.py               110      0   100%
 src/dashboards/debug.py                 14      0   100%
 src/dashboards/poller.py               139      1    99%   226
 src/dashboards/prom.py                 105      5    95%   107, 144, 158, 161, 164
 src/dashboards/rates.py                 37      1    97%   51
 src/dashboards/severity.py              49      0   100%
 src/dashboards/sources/__init__.py       0      0   100%
 src/dashboards/sources/host.py         199      2    99%   122, 147
 src/dashboards/sources/llm.py          150      1    99%   226
 src/dashboards/sources/nut.py          367     22    94%   118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
 src/dashboards/sources/nut_sim.py      122     21    83%   141, 167-185, 226
 src/dashboards/store.py                178      2    99%   243-244
 ------------------------------------------------------------------
 TOTAL                                 2104     67    97%
 Required test coverage of 94% reached. Total coverage: 96.82%
 439 passed, 6 skipped, 1 warning in 43.39s
 == uv run ruff check src tests agent ==
 All checks passed!
 == uv run pyright ==
 0 errors, 0 warnings, 0 informations
 == node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1,17 @@
 **Implemented**
 - `src/dashboards/alerts.py` (new, 100% covered): pure `build(settings, hub_states, host_payloads, engine_payloads, ups_payload, recent_events, *, now)` → `[{id, severity, source, message, href, at}]`, fail→warn then by source. Rules: unconfigured never alerts; `ok=false` → `fail` (collectors' name prefix de-duplicated, `at` from the poller's `unreachable` event); `ok` but older than 3× its own poll interval → `warn`; host/engine/UPS threshold breaches read only phase-01 `severity` dicts + `health.checks[]` (no new thresholds; bound quoted back from `severity.THRESHOLDS`/`INVERTED`); degraded-agent sections → `warn`; `HOST_TARGETS` problems → `warn`; dedupe by (source, metric) keeping the worst verdict; capped at `MAX_ALERTS=12` with a marker stating how many were hidden; a dead source yields exactly one line (stale payload suppressed).
 - `src/dashboards/api.py`: `/api/summary` calls it and adds `"alerts"`; `system`/`llm`/`ups`/`recent_events` byte-identical otherwise (additive-only).
 - Tests: `tests/test_alerts.py` (40 tests incl. all-clear `[]`, unconfigured, stale-vs-dead, per-source intervals, KV pressure, UPS detail strings, simulated-only, dedupe/cap/sort/shape), 4 new `/api/summary` tests + `healthy_client` fixture in `test_api.py`, shared `host_scrape` fixture in `conftest.py`.
 - Defects fixed in passing: `nut_sim` hard-coded `battery.mfr.date` (2019/2022) aged into permanent amber → computed relative to today (+ test); `build_client()` leaked its `SampleStore` (28 ResourceWarnings under `--cov`) → now a context manager that closes it.
 **Gate** (`bash .agents/validate.sh`, exit 0): `uv run pytest --cov --cov-fail-under=94` → **400 passed, 6 skipped, 1 warning; total coverage 95.18%**, `alerts.py` 100% (changed part of `api.py` fully covered); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; repo hygiene clean. `uv run pytest -m live --live` → 6 passed (unchanged). Dev server restarted: all 6 sources `ok`; `curl :8090/api/summary` keys `['alerts','llm','recent_events','system','ts','ups']` with live alerts (e.g. `fail deskwork /system deskwork: GPU utilisation 100% … (fail ≥95)`, `warn apc /ups Battery age: 5.3 years — past the 4-year replacement window`).
 **Docs**: AGENTS.md — `alerts.py` responsibility bullet, `/api/summary` additive note, nut_sim date behaviour, refreshed test/coverage counts (400 / 95.2%); README.md — `alerts.py` bullet in Architecture.
 **Decisions / deviations**
 - LLM alerts come from the cells `llm.SEVERITY_CELLS` already rates (KV pressure only) so no thresholds are invented here; `engine_payloads` therefore has a real purpose beyond reachability.
 - The cap keeps the list at exactly `MAX_ALERTS` by replacing the last slot with an `alerts:suppressed` marker whose severity is the worst item it hid (tested both directions).
 - Live fleet today is *not* all-clear: the real UPSes emit `warn` checks (battery age, "No test initiated"), so the `[]`-when-healthy and "NUT_SIMULATE=1 → exactly the simulated warning" criteria are asserted hermetically instead; I did not change `nut.assess()` semantics.
 - No live infra change was made (no tunnel was killed); "one dead host → exactly one alert with `/system`" is covered by `test_summary_raises_one_alert_for_the_host_that_stopped_answering`, and the browser proof belongs to task 02.
 **Next pending task**: `.agents/phases/todo/02_overview_fleet_and_alerts/02_rewrite_overview_page.md` (reduce `index.html` + `page-index.js` to fleet table + alerts panel).
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,45 @@
 == uv run pytest --cov (hermetic: no network, live tests skip) ==
 ........................................................................ [ 17%]
 ........................................................................ [ 35%]
 ........................................ssssss.......................... [ 53%]
 ........................................................................ [ 70%]
 ........................................................................ [ 88%]
 ..............................................                           [100%]
 =============================== warnings summary ===============================
 .venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.14.4-final-0 ________________
 Name                                 Stmts   Miss  Cover   Missing
 ------------------------------------------------------------------
 src/dashboards/__init__.py               3      0   100%
 src/dashboards/__main__.py              14      1    93%   28
 src/dashboards/alerts.py               166      0   100%
 src/dashboards/api.py                  193     28    85%   60-61, 63, 294-300, 317-325, 332-336, 420, 425-426, 432-443
 src/dashboards/app.py                  109      7    94%   50-51, 56, 58, 60, 97, 131
 src/dashboards/collectors.py           140      9    94%   78-79, 92-93, 97-98, 115, 232, 236
 src/dashboards/config.py               110      0   100%
 src/dashboards/debug.py                 14      0   100%
 src/dashboards/poller.py               139      1    99%   226
 src/dashboards/prom.py                 105      5    95%   107, 144, 158, 161, 164
 src/dashboards/rates.py                 37      1    97%   51
 src/dashboards/severity.py              49      0   100%
 src/dashboards/sources/__init__.py       0      0   100%
 src/dashboards/sources/host.py         199      2    99%   122, 147
 src/dashboards/sources/llm.py          150      1    99%   226
 src/dashboards/sources/nut.py          367     22    94%   118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
 src/dashboards/sources/nut_sim.py      122     21    83%   141, 167-185, 226
 src/dashboards/store.py                178      3    98%   204, 243-244
 ------------------------------------------------------------------
 TOTAL                                 2095    101    95%
 Required test coverage of 94% reached. Total coverage: 95.18%
 400 passed, 6 skipped, 1 warning in 40.23s
 == uv run ruff check src tests agent ==
 All checks passed!
 == uv run pyright ==
 0 errors, 0 warnings, 0 informations
 == node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1,17 @@
 **Task 02 complete — all green.**
 **Implemented**
 - `web/pages/index.html`: deleted the three summary cards, card dots, and events panel; page is now exactly two `<section>`s — **Warnings & errors** (`#alerts`, `aria-live="polite"`, "All clear" empty state) and **Fleet** (`#fleet-table` columns unchanged); masthead/`#nav-status`/`#clock`/`#theme-toggle`/`#config-chip`/`#banners`/`#foot-versions` kept.
 - `web/static/page-index.js`: `renderEvents()` + all `#sys-*`/`#llm-*`/`#ups-list` code deleted; new `renderAlerts()` (row = verdict class via `Dash.severityFrom`, source linked via the API's `href`, message, age via `Dash.age()`) and `renderFleet()` (cells coloured from `/api/summary` row `severity` verdicts only — no JS thresholds); `#summary-line` is "3/3 hosts · 2/2 engines · 2 UPS · N warnings / all clear"; one `/api/summary` call + poll loop kept.
 - `dash.css`: `ul.alerts` rows + verdict colours, `table.data td.warn/.fail`, panel swap via `#overview-panels[data-state="alerting"]` (alerts above fleet when problems exist, all-clear below when empty).
 - Defect fixed in passing: `Dash.age()` printed "2$ ago" (`replace(/…$/, '$')` — lone `$` is a literal); now "2m ago".
 - `tests/test_api.py`: 3 new served-HTML tests (two-panel shape, removed ids, alert href/CSS contract); fallback test updated — `/` is converted, 03–05 remain.
 - Screenshots in `data/logs/` (live data): `overview-alert.png` (driveripper tunnel stopped → one red row naming it, link to `/system`, alerts panel floated above fleet) and `overview-clear.png` (live gateway-only instance, `alerts: []` → "All clear" below fleet). Tunnel restored; all 6 sources `ok`.
 **Gate**: `bash .agents/validate.sh` exit 0 — `uv run pytest --cov --cov-fail-under=94` → **403 passed, 6 skipped, 95.18%**; ruff clean; pyright 0 errors; `node --check` clean. `uv run pytest -m live --live` → 6 passed (unchanged).
 **Docs**: AGENTS.md (severityFor fallback now phases 03–05; UI-conventions paragraph pinning the two-panel overview), README.md (`web/` bullet describes the two panels), `app.js` fallback docstring.
 **Decisions**: all-clear screenshot uses a throwaway 8091 instance (gateway only) because the real UPSes permanently warn — a full live config can never show `[]`; fleet table colours only warn/fail (ok stays plain text) so a healthy table is quiet.
 **Next pending task**: `.agents/phases/todo/02_overview_fleet_and_alerts/03_retire_unused_overview_code.md`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,45 @@
 == uv run pytest --cov (hermetic: no network, live tests skip) ==
 ........................................................................ [ 17%]
 ........................................................................ [ 35%]
 ...........................................ssssss....................... [ 52%]
 ........................................................................ [ 70%]
 ........................................................................ [ 88%]
 .................................................                        [100%]
 =============================== warnings summary ===============================
 .venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.14.4-final-0 ________________
 Name                                 Stmts   Miss  Cover   Missing
 ------------------------------------------------------------------
 src/dashboards/__init__.py               3      0   100%
 src/dashboards/__main__.py              14      1    93%   28
 src/dashboards/alerts.py               166      0   100%
 src/dashboards/api.py                  193     28    85%   60-61, 63, 294-300, 317-325, 332-336, 420, 425-426, 432-443
 src/dashboards/app.py                  109      7    94%   50-51, 56, 58, 60, 97, 131
 src/dashboards/collectors.py           140      9    94%   78-79, 92-93, 97-98, 115, 232, 236
 src/dashboards/config.py               110      0   100%
 src/dashboards/debug.py                 14      0   100%
 src/dashboards/poller.py               139      1    99%   226
 src/dashboards/prom.py                 105      5    95%   107, 144, 158, 161, 164
 src/dashboards/rates.py                 37      1    97%   51
 src/dashboards/severity.py              49      0   100%
 src/dashboards/sources/__init__.py       0      0   100%
 src/dashboards/sources/host.py         199      2    99%   122, 147
 src/dashboards/sources/llm.py          150      1    99%   226
 src/dashboards/sources/nut.py          367     22    94%   118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
 src/dashboards/sources/nut_sim.py      122     21    83%   141, 167-185, 226
 src/dashboards/store.py                178      3    98%   204, 243-244
 ------------------------------------------------------------------
 TOTAL                                 2095    101    95%
 Required test coverage of 94% reached. Total coverage: 95.18%
 403 passed, 6 skipped, 1 warning in 45.80s
 == uv run ruff check src tests agent ==
 All checks passed!
 == uv run pyright ==
 0 errors, 0 warnings, 0 informations
 == node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1,16 @@
 All green. Report:
 **Implemented**
 - `index.html`: removed the inert `#config-chip`, `#fleet-note` and the unread `#alerts-panel`/`#fleet-panel` ids; `system.html`: dropped the unread `#f-power` id (element kept, still renders `n/a`); `app.js`: deleted the dead `getElementById('sim-flag')` + `void target` in the clock tick.
 - `dash.css`: removed `.grid.wide` + its 1150px media query (the only rule the deleted fleet+events row made dead) plus `.cardset`/`.stale-flag` (duplicates of the live `.stack`/`.stale`); `.metric*`, `.events`, `.hero-bar` stay — still used/pinned.
 - `api.py`: `/api/summary` docstring now states the page renders only `alerts` + `system.fleet`/`hosts`, the rest is programmatic (additive-only).
 - Tests (`tests/test_api.py`): `test_summary_serves_every_path_another_consumer_reads` pins `system.enabled|fleet|hosts`, `llm.engines`, `ups.ups`, `recent_events`, `alerts` with **and** without targets (mutation-checked: renaming `llm.engines` fails it); `test_no_page_carries_an_id_nothing_reads` (orphan markup tripwire, all four pages); `parse_range` accept/refuse matrix; `/api/system/history`, `/api/llm/snapshot`, `/api/llm/history` contracts; UPS gates — driver-does-not-advertise 400, `load.off`+`DANGER` → **409 while on battery** (via the labelled simulator), and a scripted `nut.NutClient` proving login→INSTCMD→audit event and 502+error-event on refusal.
 - Docs: README `/` row → “fleet + alerts in one request (`/api/summary`)”; AGENTS.md — `/api/summary` render-vs-programmatic bullet, orphan-id/CSS convention, counts (435 tests, 96.8%), `ScriptedNutClient` note.
 **Gate** — `bash .agents/validate.sh` **exit 0**: `pytest --cov --cov-fail-under=94` → **435 passed, 6 skipped, coverage 96.80%** (`api.py` 100%, `alerts.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean. `uv run pytest -m live --live` → **6 passed** (read-only). `uv pip list --outdated` empty.
 **Browser** — `data/logs/retire-index.png` (live fleet, 6 alerts, alerts above Fleet, 2 `<section>`s, zero console/request errors) and `data/logs/retire-system.png` (Power still `n/a`). `curl :8090/api/summary` returns every pinned key.
 **Grep evidence** — `grep -rn "sys-cpu\|llm-kv\|ups-list\|renderEvents" src/dashboards/web/` → *no output*; `grep -rn "cardset\|stale-flag\|grid wide" src/dashboards/web/` → *no output*.
 **Decisions** — Removed 3 pre-existing dead CSS rules that duplicated a live mechanism; left the remaining unused-but-intentional vocabulary (`.hero-bar` is pinned by phase 01's test, `.cols-3`/`.vh`/`.bar.tall`/`table.data.scroll` are components phases 03–05 adopt), and wrote that line into AGENTS.md rather than deleting rework. No live infrastructure was touched (no tunnel/host changed); the on-battery refusal is proven through the labelled simulator only.
 **Next pending task**: `.agents/phases/todo/03_system_per_host_panels/01_host_card_payload_contract.md` (this was phase 02's last task).
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,46 @@
 == uv run pytest --cov (hermetic: no network, live tests skip) ==
 ........................................................................ [ 16%]
 ........................................................................ [ 32%]
 ........................................................................ [ 48%]
 ...ssssss............................................................... [ 65%]
 ........................................................................ [ 81%]
 ........................................................................ [ 97%]
 .........                                                                [100%]
 =============================== warnings summary ===============================
 .venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.14.4-final-0 ________________
 Name                                 Stmts   Miss  Cover   Missing
 ------------------------------------------------------------------
 src/dashboards/__init__.py               3      0   100%
 src/dashboards/__main__.py              14      1    93%   28
 src/dashboards/alerts.py               166      0   100%
 src/dashboards/api.py                  193      0   100%
 src/dashboards/app.py                  109      6    94%   50-51, 56, 58, 60, 131
 src/dashboards/collectors.py           140      5    96%   92-93, 97-98, 115
 src/dashboards/config.py               110      0   100%
 src/dashboards/debug.py                 14      0   100%
 src/dashboards/poller.py               139      1    99%   226
 src/dashboards/prom.py                 105      5    95%   107, 144, 158, 161, 164
 src/dashboards/rates.py                 37      1    97%   51
 src/dashboards/severity.py              49      0   100%
 src/dashboards/sources/__init__.py       0      0   100%
 src/dashboards/sources/host.py         199      2    99%   122, 147
 src/dashboards/sources/llm.py          150      1    99%   226
 src/dashboards/sources/nut.py          367     22    94%   118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
 src/dashboards/sources/nut_sim.py      122     21    83%   141, 167-185, 226
 src/dashboards/store.py                178      2    99%   243-244
 ------------------------------------------------------------------
 TOTAL                                 2095     67    97%
 Required test coverage of 94% reached. Total coverage: 96.80%
 435 passed, 6 skipped, 1 warning in 44.68s
 == uv run ruff check src tests agent ==
 All checks passed!
 == uv run pyright ==
 0 errors, 0 warnings, 0 informations
 == node --check src/dashboards/web/static/*.js ==
@@ -54,8 +54,10 @@ page still renders.
   panels that matter.
 4. **One dead source must not blink another.** Each source is its own asyncio task with its
   own timeout and health record (`poller.py`). A failed poll keeps the last good payload and
-   marks it stale in the UI instead of zeroing it. Transitions write events; repeated failure
+   marks it stale in the UI instead of zeroing it — on `/` too, where a host that has never
-   writes one line, not 500.
+   answered still gets a fleet row (`unreachable: true`, dimmed, every number `n/a`) because
   dropping the row would turn a dead agent into "never configured". Transitions write
   events; repeated failure writes one line, not 500.
 5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
   charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
   no outbound access — a CDN reference is a request that never comes back.
@@ -113,6 +115,8 @@ Responsibilities, one line each:
  kelvin-vs-celsius normalisation.
 - `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a
  battery. It must never be the default, and every payload it produces says `simulated`.
  Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that
  ages), so a fresh simulator reads healthy and amber keeps meaning something.
 - `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
 - `rates.py` — Δ/window estimator, keyed by resolved series name.
 - `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
@@ -123,8 +127,29 @@ Responsibilities, one line each:
  (charge/load/runtime), `severity` inside rated LLM cells (KV pressure, classified on a
  percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
  row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
 - `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload,
  recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts`
  (`{id, severity, source, message, href, at}`, fail first then warn, then by source). A
  pure projection: no I/O, no store, no thresholds of its own — it reads job states for
  reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions.
  Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but
  older than 3× its own poll interval is `warn` while a source that stopped answering is
  `fail`; a dead source gets exactly one line (its stale payload adds nothing) that names it
  once — the poller's `"UpstreamError: "` wrapper and the duplicated host name are stripped,
  because a message that says `gateway` twice reads as two problems; `unknown` and missing
  numbers are silence; one alert per (source, metric) and at most 12, the last being a marker
  that says how many it hid. This is why the overview can be two panels.
 - `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
 - `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
  `/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
  the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `llm.engines`,
  `ups.ups`, the `states` maps and `recent_events` stay for programmatic consumers (live
  tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events` are unchanged
  and `alerts` is additive — `tests/test_api.py` pins every one of those key paths, with and
  without targets configured, so retiring a panel can never shrink the payload. A configured
  job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
  `system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
  counts the machines there are and `hosts_live` the ones answering.
 - `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
  with an in-memory DB and no targets.
 - `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the
@@ -133,10 +158,12 @@ Responsibilities, one line each:
  `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
  `.hero` big number (plus an optional `.bar`) from that class — it never compares a number
  to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
-  values the API does not rate; converting its remaining call sites is phases 02–05's job.
+  values the API does not rate; converting its remaining call sites is phases 03–05's job
  (`/` was converted in phase 02: the overview now colours only from `/api/summary` verdicts).
 Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
-`gpu.busy_pct@0`, `llm.decode_tps`, `ups.charge_pct`), `@stream` for per-device series.
+`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
 series — `/api/metrics` answers with exactly these names.
 That scheme is the history API's key — treat it as a public interface.
 ## The fleet (real environment, verified 2026-10)
@@ -178,8 +205,8 @@ cp .env.dev.example .env.dev              # dev-only: ssh tunnels + live targets
 ./scripts/dev-server.sh status            # per-source ok/DOWN
 ./scripts/dev-server.sh logs
-uv run pytest                             # hermetic: 356 tests, no network
+uv run pytest                             # hermetic: 439 passed + 6 live-skipped, no network
-uv run pytest --cov                       # coverage gate (94.75%; floor 94%)
+uv run pytest --cov                       # coverage gate (96.8%; floor 94%)
 uv run pytest -m live --live              # opt-in: real fleet / engines / upsd
 uv run ruff check src tests agent && uv run pyright
 ./scripts/capture-fixtures.sh             # re-record ground truth
@@ -215,18 +242,39 @@ service here.
 - The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
  so it stays in the default run and is *not* marked `live`.
 - Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
-  94% (`--cov-fail-under`), and ~95% is where we are; keep it there. New/modified Python
+  94% (`--cov-fail-under`), and ~97% is where we are; keep it there. New/modified Python
  still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
-  never a real target.
+  never a real target, and never a real upstream: `/api/ups/command`'s send is proven against
  a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live
  suite is read-only — that fake is the only place the write reaches a "driver".
 ## UI conventions
 Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
 measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
-dots in the header, window chips (1h…90d) on history pages. A panel that cannot get data says
+dots in the header, window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API
 accepts 5m…90d). A panel that cannot get data says
 why in the panel — the operator should never have to open devtools to learn that a source is
 unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
 The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per
 `/api/summary` alert: verdict class, source linked to its dashboard via the alert's own
 `href`, message, age) and **Fleet**. A non-empty list floats the alerts panel above the
 fleet table (`#overview-panels[data-state]` swaps them); an empty one is an explicit
 "all clear" state below it, because an empty list is a message, not a blank box. A host that
 is not answering keeps its row, dimmed with the shared `.stale` class and reading `n/a`, so
 the table never implies last good numbers are live and never silently loses a machine.
 No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the
 two-panel shape against the served HTML.
 Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an
 `id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for`
 attribute references — the orphan a deleted renderer leaves behind, caught on served bytes
 because there is no JS test runner (a locked anchor). Same rule for `dash.css`: delete the
 rules a removed panel made dead (`.grid.wide` went with the overview's fleet+events row);
 component vocabulary the pages are still being converted to (`.hero-bar`, the grid and bar
 variants) stays even while only one page uses it.
 A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
 muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's
 `severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*`
@@ -4,7 +4,7 @@ Three dashboards, one domain, one process, one SQLite file:
 | Page        | What it answers                                                                 | Source                                             |
 | ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
-| `/`         | Is anything wrong, and which dashboard do I need?                               | all of the below, one request                      |
+| `/`         | Is anything wrong, and which dashboard do I need?                               | fleet + alerts in one request (`/api/summary`)     |
 | `/system`   | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
 | `/llm`      | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
 | `/ups`      | Battery health, runtime left, power events, and battery-test buttons             | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
@@ -136,8 +136,23 @@ store.py  ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
  colours the same way and a value nobody reported renders `n/a` instead of green. A metric
  published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays
  what the upstream published.
 * `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states,
  host_payloads, engine_payloads, ups_payload, recent_events)` is a pure projection of the
  state the poller already keeps plus the verdicts `severity.py` and the UPS health checks
  already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as
  `alerts`, worst first: `{id, severity, source, message, href, at}`. A configured source
  that stopped answering is `fail`; one that is stale but still serving its last good value
  is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than
  one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a
  marker that says how many it hid.
 * `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No
-  framework, no CDN, no bundler. A big colour-coded number is `Dash.hero(el, {...})` on the
+  framework, no CDN, no bundler. The overview (`/`) is exactly two panels — a
  **Warnings & errors** list built from `/api/summary`'s `alerts` (each row links to the
  dashboard that explains it) and the **Fleet** table — and nothing else: an empty alert
  list reads as an explicit "all clear", and a live problem moves the panel above the
  table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the
  fleet never quietly gets smaller and last good values are never mistaken for live ones.
  A big colour-coded number is `Dash.hero(el, {...})` on the
  `.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
  `Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
  with no verdict renders `n/a` with no colour at all.
@@ -0,0 +1,414 @@
 """What is wrong right now, derived once on the server instead of on every page.
 The landing page has one question — *is anything wrong, and where do I look?* — and the
 answer must be identical in the browser, in a future mobile view, and in whatever
 notifies someone later. So the answer is a function of the state the poller already
 keeps and the verdicts `severity.py` / `sources/nut.assess()` already produced, and this
 module is the only place that assembles it.
 It is a pure projection by design: no I/O, no store, no HTTP, no clock it cannot be given.
 That is what makes every rule below unit-testable without a hub, and what keeps a page
 from growing its own opinion about what red means.
 The rules, and the reason each exists:
 * **Never alert for an unconfigured source.** `AGENTS.md` invariant 3: a server the
  operator never installed must not raise a red flag, because that destroys the signal
  from the panels that matter. `SourceState.configured` is the gate.
 * **Stale is not dead.** A source that stopped answering *and* has no last good value is
  `fail`. A source that is still serving its last good value but has not refreshed in
  three of its own poll intervals is `warn` — saying "dead" while the page shows numbers
  would be a lie. Per-source intervals because system polls at 10s and UPS at 5s; one
  global number would be either noise or too late.
 * **Verdicts come from elsewhere.** Host/LLM breaches read the `severity` dicts phase 01
  added; UPS breaches read `health.checks[].status`. Nothing here compares a number to a
  threshold of its own — the message *quotes* the bound from `severity.THRESHOLDS` so the
  operator can see which line was crossed, and the numbers still live in one file.
 * **A dead source is one alert.** Its last good payload is still in the hub, so its stale
  threshold breaches would pile three more amber lines on top of the red one that already
  names it. Payload-derived alerts are skipped for a source that is not answering.
 * **Bounded and deduped.** One alert per (source, metric) — the `id` is that pair — and at
  most :data:`MAX_ALERTS` of them, the last being a marker that says how many it hid. A
  40-disk fleet must not turn the landing page into a log tail.
 * **Sorted fail → warn, then by source**, so the thing that needs action is at the top
  without the operator scanning.
 """
 from __future__ import annotations
 import re
 import time
 from collections.abc import Mapping, Sequence
 from typing import Any
 from .config import Settings
 from .severity import INVERTED, THRESHOLDS
 from .sources.llm import SEVERITY_CELLS
 #: The most recent verdicts the landing page shows. Beyond this it is a wall of text and
 #: the question it answers becomes "read this list", which is a different page.
 MAX_ALERTS = 12
 FAIL = "fail"
 WARN = "warn"
 #: Only these two are alerts. `ok` is not news, and `None`/`unknown` means nothing honest
 #: could be said — `AGENTS.md` rule 2, no measurement, no verdict, no colour.
 _RANK = {FAIL: 0, WARN: 1}
 #: Where each dashboard lives, so every alert can say where to look.
 HREF = {"system": "/system", "llm": "/llm", "ups": "/ups"}
 DEFAULT_HREF = "/"
 #: A source whose last good value is older than this many of its own poll intervals is
 #: serving data the operator should not act on. It is still on screen, hence amber.
 STALE_FACTOR = 3.0
 #: How to say a metric out loud: label, unit, decimals. Keyed by the dotted metric name
 #: from `severity.THRESHOLDS`; an unlisted metric prints its own name rather than nothing.
 _DISPLAY: dict[str, tuple[str, str, int]] = {
    "cpu.usage_pct": ("CPU", "%", 0),
    "cpu.temp_c": ("CPU temperature", "°C", 0),
    "mem.used_pct": ("memory", "%", 0),
    "disk.used_pct": ("disk", "%", 0),
    "gpu.busy_pct": ("GPU utilisation", "%", 0),
    "gpu.mem_used_pct": ("GPU memory", "%", 0),
    "llm.kv_cache_usage_pct": ("KV cache", "%", 0),
 }
 #: host payload section -> (field, severity metric). Mirrors the `severity` sub-dicts that
 #: `sources/host.py` attaches, so a new rated field is one row here.
 _HOST_CELLS: tuple[tuple[str, str, str], ...] = (
    ("cpu", "usage_pct", "cpu.usage_pct"),
    ("cpu", "temp_c", "cpu.temp_c"),
    ("memory", "used_pct", "mem.used_pct"),
    ("disks", "worst_used_pct", "disk.used_pct"),
 )
 _GPU_CELLS: tuple[tuple[str, str], ...] = (
    ("busy_pct", "gpu.busy_pct"),
    ("mem_used_pct", "gpu.mem_used_pct"),
 )
 # --------------------------------------------------------------------------- #
 # formatting helpers (words, not thresholds)
 # --------------------------------------------------------------------------- #
 def _value(metric: str, number: float, scale: float = 1.0) -> str:
    """`CPU 92%` — the metric said out loud, with the value the upstream published.
    Callers only get here with a real number (no measurement, no alert), so there is no
    `n/a` case. ``scale`` is a unit for the *display* of a ratio the threshold table keeps
    in percent (`llamacpp:kv_cache_usage_ratio` is 0-1); it is the same conversion
    `with_severity()` applies to the verdict, so the number and the colour cannot disagree.
    """
    label, unit, digits = _DISPLAY.get(metric, (metric, "", 1))
    return f"{label} {number * scale:.{digits}f}{unit}"
 def _trip(metric: str, verdict: Any) -> str:
    """` (fail ≥90)` — which bound was crossed, quoted from `severity.THRESHOLDS`.
    The threshold still lives in one place; this only reads it back so the alert is
    actionable without opening the source. Direction comes from `severity.INVERTED`, so a
    low battery charge prints `≤` instead of an arithmetic-looking `≥20`.
    """
    if verdict not in _RANK:
        return ""
    base = metric.split("@", 1)[0]
    bounds = THRESHOLDS.get(base)
    if bounds is None:
        return ""
    warn, danger = bounds
    low_is_worse = base in INVERTED
    chosen = danger if verdict == FAIL else warn
    return f" ({verdict} {'≤' if low_is_worse else '≥'}{chosen:g})"
 #: `poller.Job` records a failure as `"<ExceptionClass>: <message>"`, and collectors build
 #: their message as `"<name>: <what happened>"`. Left alone, a dead host prints
 #: `gateway: UpstreamError: gateway: http://… unreachable` — the name twice and the one
 #: exception class this codebase guarantees everything else raises (`AGENTS.md`: collectors
 #: surface a single `UpstreamError` type, so its name is never information). Other class
 #: names stay: an unexpected `ValueError` is a bug worth reading about. The full string
 #: survives in `/api/health` and the event feed.
 _POLLER_CLASS = re.compile(r"^UpstreamError: ")
 def _clean_error(label: str, error: str) -> str:
    """`http://h:9121/snapshot unreachable (ConnectError)` from the poller's wrapped string."""
    text = error.strip()
    while True:
        stripped = _POLLER_CLASS.sub("", text, count=1).strip()
        if stripped.startswith(f"{label}: "):
            stripped = stripped[len(label) + 2:].strip()
        if stripped == text:
            return text
        text = stripped
 def _sections(degraded: Any) -> str:
    """The agent's `errors` map (or list) as words: `temperatures (no hwmon sensors)`."""
    if isinstance(degraded, Mapping):
        return ", ".join(f"{name} ({detail})" if detail else str(name)
                         for name, detail in degraded.items()) or "nothing"
    if isinstance(degraded, (list, tuple)):
        return ", ".join(str(item) for item in degraded) or "nothing"
    return str(degraded)
 def _interval(settings: Settings, dashboard: str) -> float:
    """The source's own poll interval — the unit staleness is measured in."""
    if dashboard == "llm":
        return float(settings.llm_interval)
    if dashboard == "ups":
        return float(settings.nut_interval)
    if dashboard == "system":
        return float(settings.host_interval)
    return 60.0
 # --------------------------------------------------------------------------- #
 # the sink: dedupe, bound, sort
 # --------------------------------------------------------------------------- #
 class _Sink:
    """Collects alerts keyed by (source, metric), keeping the worst verdict per key."""
    def __init__(self, now: float) -> None:
        self.now = now
        self.items: dict[str, dict[str, Any]] = {}
    def add(self, verdict: Any, source: str, key: str, message: str, href: str,
            at: float | None = None) -> None:
        """Record one verdict — unless it is `ok`, `unknown`, or `None`, which are not alerts."""
        if verdict not in _RANK:
            return
        alert_id = f"{source}:{key}"
        existing = self.items.get(alert_id)
        if existing is not None:
            # same source + same metric twice (two UPS cards answering as one name, a
            # payload re-projected after a state change): one line, and the worse verdict
            if _RANK[str(verdict)] < _RANK[existing["severity"]]:
                existing["severity"] = str(verdict)
                existing["message"] = message
                existing["at"] = self.now if at is None else at
            return
        self.items[alert_id] = {
            "id": alert_id,
            "severity": str(verdict),
            "source": source,
            "message": message,
            "href": href,
            # when this became true, best effort: the last good poll, the outage event, or
            # now. The page prints it as "since", so an alert that resets its clock every
            # refresh is an alert whose duration nobody can read.
            "at": self.now if at is None else at,
        }
    def result(self) -> list[dict[str, Any]]:
        ordered = sorted(self.items.values(),
                         key=lambda item: (_RANK.get(item["severity"], 9), item["source"],
                                           item["id"]))
        if len(ordered) <= MAX_ALERTS:
            return ordered
        visible, hidden = ordered[:MAX_ALERTS - 1], ordered[MAX_ALERTS - 1:]
        failing = sum(1 for item in hidden if item["severity"] == FAIL)
        visible.append({
            "id": "alerts:suppressed",
            # the marker inherits the worst thing it hid: a list that ends in amber while
            # three reds are behind it would be a lie of the same shape as hiding them
            "severity": hidden[0]["severity"],
            "source": "alerts",
            "message": (f"{len(hidden)} more not shown "
                        f"({failing} failing, {len(hidden) - failing} warning)"),
            "href": DEFAULT_HREF,
            "at": self.now,
        })
        return visible
 # --------------------------------------------------------------------------- #
 # the rules
 # --------------------------------------------------------------------------- #
 def _state_for(states: Sequence[Mapping[str, Any]], name: str) -> Mapping[str, Any] | None:
    """The health record for the job that produced `name` (a host or engine label)."""
    for state in states:
        if state.get("label") == name or str(state.get("key") or "").endswith(f":{name}"):
            return state
    return None
 def _answering(states: Sequence[Mapping[str, Any]], name: str) -> bool:
    """Is that source currently answering? Unknown (no state at all) counts as yes.
    A payload that exists is data; inventing a reachability verdict for a source with no
    health record would be this module's opinion, not the poller's.
    """
    state = _state_for(states, name)
    return True if state is None else bool(state.get("ok"))
 def _dashboard_answering(states: Sequence[Mapping[str, Any]], dashboard: str) -> bool:
    mine = [state for state in states if state.get("dashboard") == dashboard]
    return any(bool(state.get("ok")) for state in mine) if mine else True
 def _outage_started(events: Sequence[Mapping[str, Any]], dashboard: str,
                    label: str) -> float | None:
    """When the poller first said this source was down, from its own transition event.
    Best effort by construction: `recent_events` is the short list the summary already
    carries, so an outage older than that reads as `at: now`. The alert itself never
    depends on the event being there.
    """
    matches = [float(event["ts"]) for event in events
               if event.get("kind") == "unreachable" and event.get("host") == label
               and (event.get("source") == dashboard or event.get("source") is None)
               and isinstance(event.get("ts"), (int, float))]
    return max(matches) if matches else None
 def _source_alerts(sink: _Sink, settings: Settings, states: Sequence[Mapping[str, Any]],
                   events: Sequence[Mapping[str, Any]]) -> None:
    for state in states:
        if not state.get("configured"):
            continue  # invariant 3: never red for a source nobody installed
        label = str(state.get("label") or state.get("key") or "source")
        dashboard = str(state.get("dashboard") or "")
        href = HREF.get(dashboard, DEFAULT_HREF)
        key = str(state.get("key") or label)
        if not state.get("ok"):
            # the alert starts with the name, so the name inside the message is removed —
            # "driveripper: driveripper: ..." reads like two problems instead of one
            error = _clean_error(label, str(state.get("error") or "")) or "is not answering"
            sink.add(FAIL, label, f"{key}:down", f"{label}: {error}", href,
                     _outage_started(events, dashboard, label) or sink.now)
            continue
        age = state.get("age_seconds")
        interval = _interval(settings, dashboard)
        if isinstance(age, (int, float)) and age > interval * STALE_FACTOR:
            sink.add(WARN, label, f"{key}:stale",
                     f"{label} last answered {age:.0f}s ago (every {interval:g}s): the page "
                     f"is showing the last good value, not a live one",
                     href, sink.now - age)
 def _host_alerts(sink: _Sink, host: Mapping[str, Any],
                 states: Sequence[Mapping[str, Any]]) -> None:
    name = str(host.get("name") or "host")
    if host.get("unreachable") or not _answering(states, name):
        return  # the job-level alert already names this host; one line, not four
    at = host.get("ts") if isinstance(host.get("ts"), (int, float)) else None
    for section, field, metric in _HOST_CELLS:
        block = host.get(section) or {}
        number = block.get(field)
        verdict = (block.get("severity") or {}).get(field)
        # no number, no alert: `n/a` is not a breach (and the verdict is None anyway)
        if number is None or verdict not in _RANK:
            continue
        mount = block.get("worst_mount")
        where = f" on {mount}" if mount else ""
        sink.add(verdict, name, metric,
                 f"{name}: {_value(metric, number)}{where}{_trip(metric, verdict)}",
                 HREF["system"], at)
    for gpu in host.get("gpus") or []:
        card = str(gpu.get("model") or f"GPU {gpu.get('index', 0)}")
        for field, metric in _GPU_CELLS:
            number = gpu.get(field)
            verdict = (gpu.get("severity") or {}).get(field)
            if number is None or verdict not in _RANK:
                continue
            sink.add(verdict, name, f"{metric}@{gpu.get('index', 0)}",
                     f"{name}: {_value(metric, number)} on {card}{_trip(metric, verdict)}",
                     HREF["system"], at)
    degraded = host.get("degraded")
    if degraded:
        # a /proc read failed on the box: the numbers that did come back are real, but the
        # snapshot is not the whole picture and the page must not imply that it is
        sink.add(WARN, name, "agent.sections",
                 f"{name}: the agent could not read {_sections(degraded)}", HREF["system"], at)
 def _engine_alerts(sink: _Sink, engine: Mapping[str, Any],
                   states: Sequence[Mapping[str, Any]]) -> None:
    name = str(engine.get("engine") or "engine")
    if engine.get("unreachable") or not _answering(states, name):
        return
    at = engine.get("scraped_at") if isinstance(engine.get("scraped_at"), (int, float)) else None
    metrics = engine.get("metrics") or {}
    # the rated cells are declared next to the field maps they come from (`SEVERITY_CELLS`),
    # so a new one is rated by severity.py and alerted here without touching this loop
    for key, (metric, scale) in SEVERITY_CELLS.items():
        cell = metrics.get(key)
        if not isinstance(cell, Mapping):
            continue
        value, verdict = cell.get("value"), cell.get("severity")
        # no number in the cell, no claim (rule 2); no verdict, no colour to justify one
        if not isinstance(value, (int, float)) or verdict not in _RANK:
            continue
        sink.add(verdict, name, f"{metric}@{key}",
                 f"{name}: {_value(metric, float(value), scale)}{_trip(metric, verdict)}",
                 HREF["llm"], at)
 def _ups_alerts(sink: _Sink, payload: Mapping[str, Any],
                states: Sequence[Mapping[str, Any]]) -> None:
    if payload.get("simulated"):
        # `AGENTS.md`: the simulator must never be silent about what it is. The UPS page
        # badges it; the landing page has to say it too, or a modelled battery reads as a
        # healthy real one from two clicks away.
        sink.add(WARN, "ups", "simulated",
                 "UPS numbers are simulated — NUT_SIMULATE=1 is set, so these are modelled "
                 "figures and not your batteries", HREF["ups"])
    if not _dashboard_answering(states, "ups"):
        return
    for card in payload.get("ups") or []:
        name = str(card.get("name") or "ups")
        at = card.get("ts") if isinstance(card.get("ts"), (int, float)) else None
        for check in (card.get("health") or {}).get("checks") or []:
            label = str(check.get("name") or "check")
            detail = str(check.get("detail") or "").strip()
            # `unknown` stays out: a driver that publishes no battery temperature is not a
            # fault, and the UPS page already says "not reported by this driver"
            sink.add(check.get("status"), name, f"health:{label}",
                     f"{label}: {detail}" if detail else label, HREF["ups"], at)
 def _config_alerts(sink: _Sink, settings: Settings) -> None:
    for problem in settings.host_target_problems:
        # a typo'd entry is not a dead host: the operator needs to know it is not being
        # polled at all, which is otherwise invisible (the host simply is not in the list)
        sink.add(WARN, "config", f"HOST_TARGETS:{problem}",
                 f"HOST_TARGETS: {problem} — that entry is not being polled", HREF["system"])
 # --------------------------------------------------------------------------- #
 # entry point
 # --------------------------------------------------------------------------- #
 def build(settings: Settings, hub_states: Sequence[Mapping[str, Any]] = (),
          host_payloads: Sequence[Mapping[str, Any]] = (),
          engine_payloads: Sequence[Mapping[str, Any]] = (),
          ups_payload: Mapping[str, Any] | None = None,
          recent_events: Sequence[Mapping[str, Any]] = (), *,
          now: float | None = None) -> list[dict[str, Any]]:
    """Everything that is wrong right now, worst first.
    Each alert is `{"id", "severity", "source", "message", "href", "at"}`; `id` is
    `"<source>:<metric>"` and is stable across refreshes so a page can key DOM off it, and
    `at` is a unix timestamp for when the condition was first observed (best effort).
    Empty means all clear — there is no `ok` alert, because an empty list is the clearest
    possible answer to "what is wrong".
    Inputs are the same objects `/api/summary` already assembles (job states, last good
    payloads, the UPS payload, the short event list), so answering costs nothing extra and
    the answer cannot disagree with the panels underneath it.
    """
    sink = _Sink(time.time() if now is None else now)
    _source_alerts(sink, settings, hub_states, recent_events)
    for host in host_payloads:
        _host_alerts(sink, host, hub_states)
    for engine in engine_payloads:
        _engine_alerts(sink, engine, hub_states)
    if ups_payload is not None:
        _ups_alerts(sink, ups_payload, hub_states)
    _config_alerts(sink, settings)
    return sink.result()
@@ -18,6 +18,7 @@ from typing import Any
 from fastapi import APIRouter, Body, Depends, HTTPException, Query, Request
 from . import alerts as alert_rules
 from .collectors import NutMonitor
 from .config import Settings
 from .poller import Hub
@@ -190,11 +191,30 @@ def common_router() -> APIRouter:
    @router.get("/summary")
    async def summary(ctx: AppContext = Depends(get_context)) -> dict[str, Any]:
-        """One call for the landing page: is each dashboard alive, and what stands out."""
+        """One call for the landing page: is anything wrong, and what is the fleet doing.
-        hosts = [job.state.payload for job in ctx.hub.jobs.values()
+
-                 if job.dashboard == "system" and job.state.payload]
+        What `/` renders is `alerts` plus `system.fleet`/`system.hosts` and nothing else.
-        engines = [job.state.payload for job in ctx.hub.jobs.values()
+        Everything else here is for **programmatic use**: this is the cheapest whole-fleet
-                   if job.dashboard == "llm" and job.state.payload]
+        probe that exists, so the live tests, an external check and a notifier later read
        `llm.engines`, `ups.ups` and the `states`/`recent_events` no panel paints any more.
        Keys are additive-only — `alerts` was added, nothing was renamed, dropped, or given
        a new meaning. `tests/test_api.py` pins each path below against a future cleanup.
        """
        # A configured job that has never answered gets the same stub `/api/*/snapshot`
        # hands out: a host in `HOST_TARGETS` with no payload yet is still in the fleet, and
        # dropping its row would turn "dead agent" into "never configured" (invariants 3
        # and 4) while quietly shrinking `fleet.hosts_total` to the answering subset.
        hosts: list[dict[str, Any]] = [
            job.state.payload or {"name": job.label, "ts": job.state.last_attempt,
                                  "unreachable": True}
            for job in ctx.hub.jobs.values() if job.dashboard == "system"
        ]
        engines: list[dict[str, Any]] = [
            job.state.payload or {"engine": job.label, "metrics": {},
                                  "sessions": {"active": None, "list": []},
                                  "unreachable": True}
            for job in ctx.hub.jobs.values() if job.dashboard == "llm"
        ]
        ups_payload = None
        nut_job = ctx.hub.jobs.get("nut")
        if nut_job and nut_job.state.payload:
@@ -202,28 +222,40 @@ def common_router() -> APIRouter:
        fleet = fleet_summary(hosts) if hosts else None
        if fleet is not None:
            fleet["severity"] = _fleet_severity(fleet)
        # one list of events, reused twice: the feed itself, and the timestamp for a source
        # that is currently failing (alerts.py reads `kind: unreachable` out of it)
        recent_events = [event.as_json() for event in ctx.store.recent_events(12)]
        # "what is wrong right now" is a server-side answer (see alerts.py) so the overview,
        # a mobile view, and a notifier later all agree on the same verdicts
        summary_alerts = alert_rules.build(ctx.settings, ctx.hub.states(), hosts, engines,
                                           ups_payload, recent_events)
        return {
            "ts": time.time(),
            "system": {"enabled": bool(ctx.settings.host_targets),
                       "fleet": fleet,
-                       "hosts": [{"name": h["name"], "os": (h.get("info") or {}).get("os_pretty"),
+                       # every read is `.get()` because a stub has none of these sections —
-                                  "cpu_usage_pct": h["cpu"]["usage_pct"],
+                       # an unreachable agent renders as `n/a` (rule 2), never as a 500
-                                  "load1": h["cpu"].get("load1"),
+                       "hosts": [{"name": h["name"],
                                  "os": (h.get("info") or {}).get("os_pretty"),
                                  "cpu_usage_pct": (h.get("cpu") or {}).get("usage_pct"),
                                  "load1": (h.get("cpu") or {}).get("load1"),
                                  "cores": (h.get("info") or {}).get("cpu_cores_logical"),
-                                  "mem_used_pct": h["memory"]["used_pct"],
+                                  "mem_used_pct": (h.get("memory") or {}).get("used_pct"),
-                                  "disk_worst_used_pct": h["disks"]["worst_used_pct"],
+                                  "disk_worst_used_pct": (h.get("disks") or {}).get("worst_used_pct"),
                                  "gpu_busy_pct": [g.get("busy_pct") for g in h.get("gpus") or []],
-                                  "temp_c": h["cpu"].get("temp_c"),
+                                  "temp_c": (h.get("cpu") or {}).get("temp_c"),
                                  "uptime_seconds": (h.get("info") or {}).get("uptime_seconds"),
                                  "unreachable": bool(h.get("unreachable")),
                                  "severity": _host_severity(h)}
                                 for h in hosts],
                       "states": ctx.hub.states("system")},
            "llm": {"enabled": bool(ctx.settings.gufo_url or ctx.settings.vllm_url),
                    "fleet": fleet_totals(engines) if engines else None,
                    "engines": [{"engine": e["engine"],
-                                 "decode_tokens_per_second": e["metrics"].get(
+                                 "decode_tokens_per_second": (e.get("metrics") or {}).get(
                                     "decode_tokens_per_second", {}).get("value"),
-                                 "active": e["sessions"].get("active")}
+                                 "active": (e.get("sessions") or {}).get("active"),
                                 "unreachable": bool(e.get("unreachable"))}
                                for e in engines],
                    "states": ctx.hub.states("llm")},
            "ups": {"enabled": ctx.settings.nut_enabled,
@@ -236,7 +268,8 @@ def common_router() -> APIRouter:
                             "severity": _ups_severity(card)}
                            for card in (ups_payload or {}).get("ups", [])],
                    "states": ctx.hub.states("ups")},
-            "recent_events": [event.as_json() for event in ctx.store.recent_events(12)],
+            "recent_events": recent_events,
            "alerts": summary_alerts,
        }
    return router
@@ -23,6 +23,18 @@ from typing import Any
 from .nut import UpsReading
 def _mfr_date(years_ago: float) -> str:
    """A battery manufacture date `years_ago` years back, computed instead of hard-coded.
    A fixed date is fine for a year or two and then quietly turns every development page
    amber forever (`Battery age: 7.0 years — past the 4-year replacement window`), which is
    the same failure `assess()` refuses for a UPS that floats at 90 %: a warning nobody can
    clear teaches people to ignore the warnings that matter. The simulator stands in for a
    working UPS, so its pack stays a fixed age behind today.
    """
    return time.strftime("%Y-%m-%d", time.gmtime(time.time() - years_ago * 365.25 * 86_400))
@dataclass
 class SimUps:
    """One fake UPS: nameplate values plus the state a driver would report."""
@@ -180,10 +192,10 @@ class NutSimulator:
        self.ups = [
            SimUps("cyberpower", "PR1500LCDRT2U", "Cyber Power Systems", "CPS-04213",
                   nominal_va=1500.0, nominal_w=900.0, load_w=210.0, capacity_ah=17.0,
-                   mfr_date="2022-04-15"),
+                   mfr_date=_mfr_date(2.0)),
            SimUps("apc", "Smart-UPS 1500", "APC", "AS-11902",
                   nominal_va=1500.0, nominal_w=865.0, load_w=140.0, capacity_ah=8.0,
-                   mfr_date="2019-11-02", driver="usbhid-ups",
+                   mfr_date=_mfr_date(3.0), driver="usbhid-ups",
                   test_result="PASSED", test_date="2026-06-14"),
        ]
        self._last = time.time()
@@ -30,70 +30,29 @@
      <div class="sub" id="summary-line">reading fleet…</div>
    </div>
    <div class="spacer"></div>
    <span class="chip" id="config-chip" hidden>loading config</span>
  </div>
  <div id="banners" class="stack" style="margin-bottom:.85rem"></div>
-  <section class="grid auto" aria-label="Dashboards">
+  <!-- Exactly two panels: what is wrong, and what the machines are doing. Their order is
-    <article class="card">
+       the page's only trick: a problem moves alerts above the fleet table, an empty list
-      <header><h2>System resources</h2><span class="spacer"></span><span class="dot idle" id="dot-system"></span></header>
+       leaves the morning calm. `data-state` is set by page-index.js from /api/summary. -->
  <div class="stack" id="overview-panels" data-state="clear">
    <section class="card panel-alerts" aria-labelledby="alerts-title">
      <header><h2 id="alerts-title">Warnings &amp; errors</h2><span class="spacer"></span>
        <span class="faint mono" id="alerts-count"></span></header>
      <div class="body">
-        <div class="metrics">
+        <ul class="alerts" id="alerts" aria-live="polite"></ul>
-          <div class="metric"><div class="label">Hosts answering</div>
+        <div class="empty-state" id="alerts-clear" hidden>
-            <div class="value"><span id="sys-hosts-live">n/a</span><span class="unit" id="sys-hosts-total"></span></div></div>
+          <h3>All clear</h3>
-          <div class="metric"><div class="label">CPU average</div>
+          <p>Nothing is failing and no threshold is breached right now. Sources the operator
-            <div class="value"><span id="sys-cpu">n/a</span><span class="unit">%</span></div>
+             never configured are not problems, so they are not here either.</p>
            <div class="bar" style="margin-top:.3rem"><i></i></div></div>
          <div class="metric"><div class="label">Memory</div>
            <div class="value"><span id="sys-mem">n/a</span><span class="unit" id="sys-mem-unit"></span></div>
            <div class="bar" style="margin-top:.3rem"><i></i></div></div>
          <div class="metric"><div class="label">Worst disk</div>
            <div class="value"><span id="sys-disk">n/a</span><span class="unit">%</span></div>
            <div class="bar" style="margin-top:.3rem"><i></i></div></div>
          <div class="metric"><div class="label">GPUs</div>
            <div class="value sm" id="sys-gpus">n/a</div></div>
        </div>
        <div class="gap" style="margin-top:.7rem"><a class="action" href="/system" style="text-decoration:none">Open system dashboard →</a></div>
      </div>
-    </article>
+    </section>
-    <article class="card">
+    <section class="card panel-fleet" aria-labelledby="fleet-title">
-      <header><h2>LLM engines</h2><span class="spacer"></span><span class="dot idle" id="dot-llm"></span></header>
+      <header><h2 id="fleet-title">Fleet</h2></header>
      <div class="body">
        <div class="metrics">
          <div class="metric"><div class="label">Engines answering</div>
            <div class="value"><span id="llm-live">n/a</span><span class="unit" id="llm-total"></span></div></div>
          <div class="metric"><div class="label">Decode</div>
            <div class="value"><span id="llm-decode">n/a</span><span class="unit">tok/s</span></div></div>
          <div class="metric"><div class="label">Prompt</div>
            <div class="value"><span id="llm-prompt">n/a</span><span class="unit">tok/s</span></div></div>
          <div class="metric"><div class="label">Active sessions</div>
            <div class="value" id="llm-active">n/a</div></div>
          <div class="metric"><div class="label">Queued</div>
            <div class="value" id="llm-waiting">n/a</div></div>
          <div class="metric"><div class="label" title="highest KV-cache use across engines right now">KV cache now</div>
            <div class="value"><span id="llm-kv">n/a</span><span class="unit">%</span></div>
            <div class="bar" style="margin-top:.3rem"><i></i></div></div>
        </div>
        <div class="gap" style="margin-top:.7rem"><a class="action" href="/llm" style="text-decoration:none">Open LLM dashboard →</a></div>
      </div>
    </article>
    <article class="card">
      <header><h2>UPS / battery</h2><span class="spacer"></span><span class="dot idle" id="dot-ups"></span></header>
      <div class="body">
        <div id="ups-list" class="stack">
          <div class="muted">loading…</div>
        </div>
        <div class="gap" style="margin-top:.7rem"><a class="action" href="/ups" style="text-decoration:none">Open UPS dashboard →</a></div>
      </div>
    </article>
  </section>
  <section class="grid wide" style="margin-top:.85rem">
    <article class="card">
      <header><h2>Fleet</h2><span class="spacer"></span><span class="faint mono" id="fleet-note"></span></header>
      <div class="body tight">
        <table class="data" id="fleet-table">
          <thead><tr><th>Host</th><th>CPU</th><th>Load 1</th><th>Mem used</th><th>Worst disk</th><th>GPU busy</th><th>Temp</th><th>Up</th></tr></thead>
@@ -105,14 +64,8 @@
             <code>HOST_TARGETS</code>.</p>
        </div>
      </div>
-    </article>
+    </section>
-
+  </div>
    <article class="card">
      <header><h2>Recent events</h2><span class="spacer"></span>
        <span class="faint">transitions only</span></header>
      <div class="body"><ul class="events" id="events"><li class="muted">loading…</li></ul></div>
    </article>
  </section>
  <footer class="pagefoot">
    <span id="foot-versions"></span>
@@ -57,7 +57,7 @@
        <div class="metric"><div class="label">GPU busy (avg)</div>
          <div class="value"><span id="f-gpu">n/a</span><span class="unit">%</span></div></div>
        <div class="metric"><div class="label">Power</div>
-          <div class="value sm" id="f-power" title="No whole-host power source is wired up yet; this stays n/a rather than guessing.">n/a</div></div>
+          <div class="value sm" title="No whole-host power source is wired up yet; no script writes this, so it stays n/a rather than guessing.">n/a</div></div>
      </div>
    </div>
  </section>
@@ -62,7 +62,10 @@ const Dash = (() => {
    if (ts == null) return 'never';
    const seconds = Math.max(0, Date.now() / 1000 - ts);
    if (seconds < 2) return 'now';
-    return duration(seconds).replace(/(h|m|s|d|y)$/, '$') + ' ago';
+    // `duration()` already carries the unit: "2m ago", "5h ago". (It used to be stripped
    // with `replace(/(h|m|s|d|y)$/, '$')` — a lone `$` is a literal in a JS replacement,
    // so every age printed "2$ ago". The unit is what makes the number mean anything.)
    return `${duration(seconds)} ago`;
  }
  const naText = 'n/a';
@@ -107,10 +110,10 @@ const Dash = (() => {
   *
   * Thresholds live in `src/dashboards/severity.py` and travel in the payload as
   * `severity`; `severityFrom()` below is how a page reads them. This function survives
-   * only because `page-index.js`, `page-system.js`, `page-llm.js` and `page-ups.js` still
+   * only because `page-system.js`, `page-llm.js` and `page-ups.js` still call it with
-   * call it with their own numbers (phases 02–05 convert them page by page, so each page's
+   * their own numbers (phases 03–05 convert them page by page, so each page's change is
-   * change is reviewable on its own). Do not add a new call site: a threshold written in
+   * reviewable on its own — `/` was already converted in phase 02). Do not add a new call
-   * JS is a threshold that can disagree with the other three pages.
+   * site: a threshold written in JS is a threshold that can disagree with the other pages.
   */
  function severityFor(value, warnAt, failAt) {
    const warn = warnAt == null ? 75 : warnAt;
@@ -312,10 +315,7 @@ const Dash = (() => {
    const clock = $('#clock');
    if (clock) {
      const tickClock = () => {
-        const now = new Date();
+        clock.textContent = new Date().toLocaleTimeString(undefined, { hour12: false });
        const target = document.getElementById('sim-flag');
        clock.textContent = now.toLocaleTimeString(undefined, { hour12: false });
        void target;
      };
      tickClock();
      setInterval(tickClock, 1000);
@@ -179,8 +179,6 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
 .grid.auto { grid-template-columns: repeat(auto-fit, minmax(min(360px, 100%), 1fr)); }
 .grid.cols-2 { grid-template-columns: repeat(auto-fit, minmax(min(480px, 100%), 1fr)); }
 .grid.cols-3 { grid-template-columns: repeat(auto-fit, minmax(min(300px, 100%), 1fr)); }
 .grid.wide { grid-template-columns: minmax(0, 2fr) minmax(0, 1fr); }
@media (max-width: 1150px) { .grid.wide { grid-template-columns: 1fr; } }
 .stack { display: grid; gap: .85rem; align-content: start; }
 /* ---- cards -------------------------------------------------------------- */
@@ -208,8 +206,6 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
 .card > .body.tight { padding: .45rem .8rem .6rem; }
 .card.flat { box-shadow: none; }
 .cardset { display: grid; gap: .85rem; }
 /* ---- metrics ------------------------------------------------------------ */
 .metrics { display: grid; gap: .7rem .9rem; grid-template-columns: repeat(auto-fit, minmax(104px, 1fr)); }
 .metric { min-width: 0; }
@@ -353,6 +349,11 @@ table.data tbody tr:hover td { background: var(--bg-sunken); }
 table.data td.name { font-weight: 600; }
 table.data .num { font-family: var(--mono); }
 table.data .mute { color: var(--text-muted); }
 /* verdict colour on a table cell (overview fleet rows): only a breach paints, `ok` stays
   plain text so a healthy table is quiet — the words are the same `ok|warn|fail` the API
   sends, never a JS-side threshold */
 table.data td.warn { color: var(--warn); }
 table.data td.fail { color: var(--fail); }
 table.data.scroll { display: block; overflow: auto; max-height: 320px; }
 /* ---- charts ------------------------------------------------------------- */
@@ -404,6 +405,29 @@ ul.events .sev.info { color: var(--text-muted); }
 ul.events .sev.warn { color: var(--warn); }
 ul.events .sev.error { color: var(--fail); }
 /* overview alerts: source (link to its dashboard) · message · age. The class on the row
   is the server's verdict — `alerts.py` decided, the page only paints it. */
 ul.alerts { list-style: none; margin: 0; padding: 0; display: grid; gap: .1rem; }
 ul.alerts li {
  display: grid; grid-template-columns: max-content 1fr max-content; gap: .7rem; align-items: baseline;
  padding: .4rem .1rem; border-bottom: 1px solid color-mix(in srgb, var(--border) 50%, transparent);
  font-size: 13px;
 }
 ul.alerts li:last-child { border-bottom: 0; }
 ul.alerts .src { font-weight: 650; white-space: nowrap; }
 ul.alerts .msg { color: var(--text-muted); overflow-wrap: anywhere; }
 ul.alerts time { font-family: var(--mono); font-size: 11px; color: var(--text-faint); white-space: nowrap; }
 ul.alerts li.ok .src { color: var(--ok); }
 ul.alerts li.warn .src { color: var(--warn); }
 ul.alerts li.fail .src { color: var(--fail); }
 /* the overview's two panels swap by verdict, not by re-rendering: an active problem reads
   first, an empty list leaves the fleet table on top and the all-clear state below it */
 #overview-panels .panel-fleet { order: 1; }
 #overview-panels .panel-alerts { order: 2; }
 #overview-panels[data-state="alerting"] .panel-alerts { order: 1; }
 #overview-panels[data-state="alerting"] .panel-fleet { order: 2; }
 ul.checks { list-style: none; margin: 0; padding: 0; display: grid; gap: .4rem; }
 ul.checks li { display: grid; grid-template-columns: 18px 1fr; gap: .5rem; align-items: start; }
 ul.checks .what { font-weight: 600; font-size: 12.5px; }
@@ -472,7 +496,6 @@ dialog footer { padding: .7rem .9rem; border-top: 1px solid var(--border); displ
 .empty-state ul { text-align: left; display: inline-block; margin: .6rem 0 0; }
 .stale { opacity: .62; }
 .stale-flag { color: var(--warn); font-size: 11px; }
 select, input[type="text"] {
  background: var(--bg-sunken); color: var(--text); border: 1px solid var(--border);
@@ -1,7 +1,13 @@
-/* Overview page: one /api/summary call, plus the config-driven setup hints. */
+/* Overview page: exactly two answers — what needs attention, and what the fleet is doing.
 *
 * One `/api/summary` call feeds both panels; "what needs attention" is the server's list
 * (`alerts.py`), never a browser-side opinion. The page owns no thresholds: cells are
 * coloured from the `severity` verdicts in the payload via `Dash.severityFrom`, and an
 * unrated value gets no colour (which is also what keeps `n/a` from looking green).
 */
 'use strict';
-const { $, fetchJSON, setTxt, setValue, setBar, setDot, bytes, pct, num, duration, poll } = Dash;
+const { $, fetchJSON, num, duration, age, poll } = Dash;
 async function loadConfig() {
  try {
@@ -48,101 +54,84 @@ async function loadConfig() {
  }
 }
-function renderEvents(events) {
+/**
-  const list = $('#events');
+ * The warnings/errors panel: one row per alert from `/api/summary`'s `alerts`.
-  if (!events || !events.length) {
+ *
-    list.innerHTML = '<li class="muted" style="grid-template-columns:1fr">Nothing has changed ' +
+ * The row is the source (a link to the dashboard that explains it), the server's message,
-      'since this server started — the log only records transitions.</li>';
+ * and the age. Colour comes from `alert.severity` through `Dash.severityFrom` — the page
 * never re-derives a verdict. An empty list is a message ("all clear"), not a blank box,
 * and it also moves the panel below the fleet table so a calm morning reads calm.
 */
 function renderAlerts(alerts) {
  const list = $('#alerts');
  const clear = $('#alerts-clear');
  const panels = $('#overview-panels');
  const items = Array.isArray(alerts) ? alerts : [];
  panels.dataset.state = items.length ? 'alerting' : 'clear';
  $('#alerts-count').textContent = items.length
    ? `${items.length} active`
    : '';
  clear.hidden = items.length > 0;
  list.hidden = items.length === 0;
  if (!items.length) {
    list.innerHTML = '';
    return;
  }
-  list.innerHTML = events.map((event) => {
+  list.innerHTML = items.map((alert) => {
-    const when = new Date(event.ts * 1000).toLocaleTimeString(undefined,
+    // alerts.py names the source at the head of most messages; the link already says it,
-      { hour: '2-digit', minute: '2-digit', hour12: false });
+    // and "deskwork: deskwork: CPU 92%" reads like two problems instead of one
-    return `<li><time>${when}</time><span><span class="sev ${event.severity}">${event.host}</span> ` +
+    const source = String(alert.source || 'fleet');
-      `${DashCharts.escapeHtml(event.message.replace(event.host + ':', '').trim() || event.kind)}</span></li>`;
+    const raw = String(alert.message || '');
    const prefix = `${source}:`;
    const message = raw.startsWith(prefix) ? raw.slice(prefix.length).trim() : raw;
    const sev = Dash.severityFrom(alert.severity);
    return `<li${sev ? ` class="${sev}"` : ''}>
      <a class="src" href="${DashCharts.escapeHtml(alert.href || '/')}">${DashCharts.escapeHtml(source)}</a>
      <span class="msg">${DashCharts.escapeHtml(message)}</span>
      <time>${age(alert.at)}</time>
    </li>`;
  }).join('');
 }
-function renderSummary(summary) {
+/**
-  const system = summary.system || {};
+ * The fleet table: one row per host, columns unchanged.
-  const fleet = system.fleet || {};
+ *
-  setDot($('#dot-system'), system.enabled ? (fleet.hosts_live === fleet.hosts_total ? 'ok'
+ * Cell colour comes from the row's `severity` verdicts (`/api/summary`, phase 01). `ok`
-    : fleet.hosts_live ? 'warn' : 'fail') : 'idle');
+ * prints as plain text on purpose — an all-green table is noise; only a breach earns a
-  setTxt($('#sys-hosts-live'), fleet.hosts_live == null ? 'n/a' : String(fleet.hosts_live));
+ * colour here, and the alerts panel is where anything rated gets its line.
-  setTxt($('#sys-hosts-total'), fleet.hosts_total == null ? '' : `of ${fleet.hosts_total}`);
+ *
-  setValue($('#sys-cpu'), fleet.cpu_usage_pct_avg, 0);
+ * A host whose agent stopped answering keeps its last good numbers rather than being
-  setBar($('#sys-cpu').closest('.metric').querySelector('.bar'), fleet.cpu_usage_pct_avg);
+ * zeroed, so its row is dimmed with the shared `.stale` class (and says so in its
-  if (fleet.memory_used_bytes != null) {
+ * tooltip): the table must never imply dead numbers are live while the panel above
-    const used = Dash.fmtBytes(fleet.memory_used_bytes, 0);
+ * says the box is down.
-    const total = Dash.fmtBytes(fleet.memory_total_bytes, 0);
+ */
-    // "200 / 253 GiB", not "200 GiB of 253 GiB": the card column is narrow and the long form
+function renderFleet(rows, states) {
-    // ran into the worst-disk number next to it
+  const answering = {};
-    setTxt($('#sys-mem'), total ? `${used.text}/${total.text}` : used.text);
+  for (const state of states || []) {
-    setTxt($('#sys-mem-unit'), used.unit);
+    const name = String(state.label || state.key || '');
-  } else {
+    if (name) answering[name] = Boolean(state.ok);
    setTxt($('#sys-mem'), 'n/a');
    setTxt($('#sys-mem-unit'), '');
  }
  setBar($('#sys-mem').closest('.metric').querySelector('.bar'), fleet.memory_used_pct);
  setValue($('#sys-disk'), fleet.disk_worst_used_pct, 0);
  setBar($('#sys-disk').closest('.metric').querySelector('.bar'), fleet.disk_worst_used_pct,
    Dash.severityFor(fleet.disk_worst_used_pct, 80, 92));
  setTxt($('#sys-gpus'), fleet.gpu_count == null ? 'n/a'
    : `${fleet.gpu_count}${fleet.gpu_busy_pct_avg != null ? ' · ' + num(fleet.gpu_busy_pct_avg, 0) + '%' : ''}`);
  const llm = summary.llm || {};
  const engines = llm.fleet || {};
  setDot($('#dot-llm'), !llm.enabled ? 'idle'
    : (engines.engines_live === engines.engines_total ? 'ok' : engines.engines_live ? 'warn' : 'fail'));
  setTxt($('#llm-live'), engines.engines_live == null ? 'n/a' : String(engines.engines_live));
  setTxt($('#llm-total'), engines.engines_total == null ? '' : `of ${engines.engines_total}`);
  setValue($('#llm-decode'), engines.decode_tokens_per_second, 1);
  setValue($('#llm-prompt'), engines.prompt_tokens_per_second, 0);
  setTxt($('#llm-active'), engines.requests_active == null ? 'n/a' : num(engines.requests_active, 0));
  setTxt($('#llm-waiting'), engines.requests_waiting == null ? 'n/a' : num(engines.requests_waiting, 0));
  setValue($('#llm-kv'), engines.kv_cache_usage_max == null ? null : engines.kv_cache_usage_max * 100, 0);
  setBar($('#llm-kv').closest('.metric').querySelector('.bar'),
    engines.kv_cache_usage_max == null ? null : engines.kv_cache_usage_max * 100,
    Dash.severityFor(engines.kv_cache_usage_max == null ? null : engines.kv_cache_usage_max * 100, 75, 90));
  const ups = summary.ups || {};
  setDot($('#dot-ups'), !ups.enabled ? 'idle'
    : (ups.ups || []).some((u) => u.overall === 'at risk') ? 'fail'
    : (ups.ups || []).some((u) => u.overall !== 'healthy') ? 'warn' : 'ok');
  const list = $('#ups-list');
  if (!ups.enabled) {
    list.innerHTML = '<div class="muted">No NUT server configured.</div>';
  } else if (!(ups.ups || []).length) {
    list.innerHTML = '<div class="muted">No UPS answering from this upsd yet.</div>';
  } else {
    list.innerHTML = ups.ups.map((item) => {
      const kind = item.overall === 'healthy' ? 'ok' : item.overall === 'at risk' ? 'fail'
        : item.overall === 'degraded' ? 'warn' : '';
      return `<div class="gap" style="justify-content:space-between">
        <span class="gap"><span class="chip ${kind}">${DashCharts.escapeHtml(item.name)}</span>
          <span class="mono faint">${DashCharts.escapeHtml(item.status || 'no state')}</span>
          ${ups.simulated ? '<span class="sim-badge">sim</span>' : ''}</span>
        <span class="mono">${item.charge_pct == null ? 'n/a' : num(item.charge_pct, 0) + '%'} ·
          ${item.load_pct == null ? 'n/a' : num(item.load_pct, 0) + '% load'} ·
          ${item.runtime_seconds == null ? 'n/a' : duration(item.runtime_seconds) + ' runtime'}</span>
      </div>`;
    }).join('');
  }
  const rows = system.hosts || [];
  const tbody = $('#fleet-table tbody');
  $('#fleet-empty').hidden = rows.length > 0;
  if (!rows.length) { tbody.innerHTML = ''; return; }
  const verdict = (severity, key) => {
    const kind = Dash.severityFrom((severity || {})[key]);
    return kind === 'ok' || kind === '' ? '' : ` ${kind}`;
  };
  tbody.innerHTML = rows.map((host) => {
    const sev = host.severity || {};
    const gpus = (host.gpu_busy_pct || []).filter((v) => v != null);
-    return `<tr>
+    // the row's own flag (a host that has never answered has no payload to be stale about)
    // or the job state: either way the numbers below are the last good ones, not live ones
    const stale = host.unreachable === true || answering[host.name] === false;
    return `<tr${stale ? ' class="stale" title="Last good values — this host is not answering"' : ''}>
      <td class="name">${DashCharts.escapeHtml(host.name)}</td>
-      <td class="num">${host.cpu_usage_pct == null ? 'n/a' : num(host.cpu_usage_pct, 0) + '%'}</td>
+      <td class="num${verdict(sev, 'cpu_usage_pct')}">${host.cpu_usage_pct == null ? 'n/a' : num(host.cpu_usage_pct, 0) + '%'}</td>
      <td class="num mute">${host.load1 == null ? 'n/a' : num(host.load1, 2)}</td>
-      <td class="num">${host.mem_used_pct == null ? 'n/a' : num(host.mem_used_pct, 0) + '%'}</td>
+      <td class="num${verdict(sev, 'mem_used_pct')}">${host.mem_used_pct == null ? 'n/a' : num(host.mem_used_pct, 0) + '%'}</td>
-      <td class="num">${host.disk_worst_used_pct == null ? 'n/a' : num(host.disk_worst_used_pct, 0) + '%'}</td>
+      <td class="num${verdict(sev, 'disk_worst_used_pct')}">${host.disk_worst_used_pct == null ? 'n/a' : num(host.disk_worst_used_pct, 0) + '%'}</td>
      <td class="num">${gpus.length ? num(gpus[0], 0) + '%' : '<span class="mute">—</span>'}</td>
-      <td class="num">${host.temp_c == null ? '<span class="mute">n/a</span>' : num(host.temp_c, 0) + '°C'}</td>
+      <td class="num${verdict(sev, 'temp_c')}">${host.temp_c == null ? '<span class="mute">n/a</span>' : num(host.temp_c, 0) + '°C'}</td>
      <td class="num mute">${host.uptime_seconds == null ? 'n/a' : duration(host.uptime_seconds)}</td>
    </tr>`;
  }).join('');
@@ -151,14 +140,24 @@ function renderSummary(summary) {
 async function refresh() {
  try {
    const summary = await fetchJSON('/api/summary');
-    const total = (summary.system.hosts || []).length;
+    const system = summary.system || {};
-    const live = summary.system.fleet ? summary.system.fleet.hosts_live : 0;
+    const fleet = system.fleet || {};
-    const engines = summary.llm.fleet ? `${summary.llm.fleet.engines_live}/${summary.llm.fleet.engines_total} engines` : 'no engines';
+    const hosts = system.hosts || [];
-    const ups = summary.ups.ups ? `${summary.ups.ups.length} UPS` : 'no UPS';
+    const live = fleet.hosts_live == null ? hosts.length : fleet.hosts_live;
-    $('#summary-line').textContent = `${live}/${total} hosts · ${engines} · ${ups} · ` +
+    const total = fleet.hosts_total == null ? hosts.length : fleet.hosts_total;
-      `${(summary.recent_events || []).length} recent events`;
+    const engines = summary.llm && summary.llm.fleet
-    renderSummary(summary);
+      ? `${summary.llm.fleet.engines_live}/${summary.llm.fleet.engines_total} engines`
-    renderEvents(summary.recent_events);
+      : 'no engines';
    const ups = summary.ups && (summary.ups.ups || []).length
      ? `${summary.ups.ups.length} UPS`
      : 'no UPS';
    const alerts = summary.alerts || [];
    const problems = alerts.length ? `${alerts.length} warning${alerts.length === 1 ? '' : 's'}`
                                   : 'all clear';
    $('#summary-line').textContent =
      `${live}/${total} hosts · ${engines} · ${ups} · ${problems}`;
    renderAlerts(alerts);
    renderFleet(hosts, system.states);
  } catch (err) {
    $('#summary-line').textContent = `summary failed: ${err.message}`;
  }
@@ -11,6 +11,7 @@ letting `uv run pytest -m live --live` catch an upstream that renamed something.
 """
 from __future__ import annotations
 import copy
 import json
 import sys
 from pathlib import Path
@@ -42,6 +43,7 @@ if str(SRC) not in sys.path:
    sys.path.insert(0, str(SRC))
 from dashboards.rates import RateEstimator  # noqa: E402
 from dashboards.sources.host import HostCollector, parse_host_snapshot  # noqa: E402
 from dashboards.store import SampleStore  # noqa: E402
 FIXTURES = Path(__file__).resolve().parent / "fixtures"
@@ -77,6 +79,34 @@ def agent_snapshots() -> dict[str, dict[str, Any]]:
    return out
@pytest.fixture
 def host_scrape() -> Any:
    """`host_scrape(snapshot, cpu_busy=10.0, mutate=None)` -> one parsed host payload.
    Two scrapes are needed before any rate exists (the agent reports counters; the
    dashboard owns the window), and the payload the API serves is the second one. `mutate`
    edits the snapshot before both scrapes so a test can put a value on a chosen side of
    `severity.THRESHOLDS`; `cpu_busy` sets the CPU percentage over the window directly.
    Driven from the real capture so a schema drift breaks here rather than in a browser.
    """
    def scrape(snapshot: dict[str, Any], *, name: str = "driveripper", cpu_busy: float = 10.0,
               mutate: Any = None) -> dict[str, Any]:
        snap = copy.deepcopy(snapshot)
        if mutate is not None:
            mutate(snap)
        collector = HostCollector(RateEstimator(span=60.0))
        parse_host_snapshot(name, snap, collector, now=1_000.0)
        second = copy.deepcopy(snap)
        second["ts"] = 1_010.0
        modes = second["cpu"]["modes"]
        # busy = (Δtotal - Δidle)/Δtotal, and iowait is left untouched so it rates at 0
        modes["total"] += 1_000.0
        modes["idle"] += 1_000.0 - (cpu_busy * 10.0)
        return parse_host_snapshot(name, second, collector, now=1_010.0).payload
    return scrape
@pytest.fixture
 def metrics_text() -> Any:
    """metrics_text('gufo') / metrics_text('vllm') -> the captured exposition verbatim."""
@@ -0,0 +1,476 @@
 """`alerts.build`: the whole "what is wrong right now" answer, with no I/O anywhere.
 The builder is a pure projection of state the poller already keeps and verdicts
 `severity.py` / `sources/nut.assess()` already reached, so every rule can be tested by
 handing it a dict — no hub, no DB, no sockets, no clock it was not given. The cases below
 are the ones that make the landing page trustworthy rather than merely busy:
 * a source nobody configured is **never** an alert (`AGENTS.md` invariant 3);
 * a source that is stale but still serving is amber, not red;
 * a dead source is **one** line, not one per stale metric in its last good payload;
 * only `fail` and `warn` come through — `ok`, `unknown`, and a missing number are silence;
 * the list dedupes per (source, metric), is bounded, and says how much it hid.
 """
 from __future__ import annotations
 from typing import Any
 from dashboards import alerts
 from dashboards.config import Settings
 from dashboards.sources import nut
 from dashboards.sources.nut_sim import NutSimulator
 SETTINGS = Settings()  # defaults: no targets, intervals 10s system / 2s llm / 5s ups
 # --------------------------------------------------------------------------- #
 # state shapers
 # --------------------------------------------------------------------------- #
 def state(key: str = "host:driveripper", dashboard: str = "system",
          label: str = "driveripper", *, ok: bool = True, configured: bool = True,
          error: str | None = None, age: float | None = 0.0) -> dict[str, Any]:
    """A `SourceState.as_json()`-shaped health record — the only shape the builder reads."""
    return {"key": key, "dashboard": dashboard, "label": label, "configured": configured,
            "ok": ok, "error": error, "age_seconds": age, "attempts": 4, "successes": 3,
            "consecutive_failures": 0 if ok else 1, "last_ok": None, "latency_ms": None}
 def check(name: str, status: str, detail: str = "") -> dict[str, Any]:
    return {"name": name, "status": status, "detail": detail, "metric": None, "value": None}
 def ups_payload(*cards: dict[str, Any], simulated: bool = False) -> dict[str, Any]:
    return {"ts": 1_000.0, "simulated": simulated, "ups": list(cards)}
 def ups_card(name: str = "apc", *checks: dict[str, Any]) -> dict[str, Any]:
    return {"name": name, "ts": 1_000.0, "health": {"overall": "healthy", "checks": list(checks)}}
 def engine(name: str = "gufo", **cells: Any) -> dict[str, Any]:
    return {"engine": name, "scraped_at": 1_000.0, "metrics": dict(cells)}
 def ratio(value: float | None, verdict: str | None) -> dict[str, Any]:
    """A rated LLM cell, exactly as `sources/llm.py` publishes it (0-1 ratio + verdict)."""
    return {"value": value, "source": "llamacpp:kv_cache_usage_ratio", "derived": False,
            "severity": verdict}
 # --------------------------------------------------------------------------- #
 # the all-clear, and the things that must not look like a problem
 # --------------------------------------------------------------------------- #
 def test_a_healthy_fleet_alerts_nothing(host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
    payload = host_scrape(agent_snapshot, cpu_busy=10.0)
    assert alerts.build(SETTINGS, [state()], [payload], [], ups_payload(), [], now=2_000.0) == []
 def test_no_inputs_at_all_is_all_clear_rather_than_an_error() -> None:
    """A fresh install with nothing wired up has nothing wrong with it."""
    assert alerts.build(SETTINGS) == []
 def test_an_unconfigured_source_never_alerts() -> None:
    """`AGENTS.md` invariant 3: not installed is not broken.
    The health record for a disabled source says `ok: false` exactly like a dead one, so
    `configured` is the only thing that tells those two apart — and the state map having
    nothing for a source must not be read as a failure either.
    """
    out = alerts.build(SETTINGS, [
        state(ok=False, configured=False, error="Connection refused"),
        state(key="nut", dashboard="ups", label="NUT server", ok=False, configured=False),
    ], [], [], ups_payload(), now=2_000.0)
    assert out == []
 def test_a_card_with_no_checks_and_a_source_with_no_state_say_nothing() -> None:
    assert alerts.build(SETTINGS, [], [], [], ups_payload(ups_card("apc"))) == []
 # --------------------------------------------------------------------------- #
 # reachability
 # --------------------------------------------------------------------------- #
 def test_a_source_that_stopped_answering_is_a_fail_naming_it_and_linking_its_page() -> None:
    out = alerts.build(SETTINGS, [state(ok=False, error="Connection refused")], now=2_000.0)
    assert len(out) == 1
    alert = out[0]
    assert alert["severity"] == "fail"
    assert alert["source"] == "driveripper"
    assert alert["href"] == "/system"
    assert alert["message"] == "driveripper: Connection refused"
    assert alert["at"] == 2_000.0
 def test_the_collectors_host_prefix_is_not_doubled() -> None:
    """Collectors already say `driveripper: …`; the alert starts with the name too."""
    out = alerts.build(SETTINGS, [state(ok=False,
                                        error="driveripper: http://h:9121/snapshot unreachable")])
    assert out[0]["message"] == "driveripper: http://h:9121/snapshot unreachable"
 def test_the_pollers_exception_prefix_is_not_printed_either() -> None:
    """The live shape is `poller`'s `UpstreamError: name: …`; one name, no class noise.
    Verified against a real stopped agent tunnel: the health record reads
    `UpstreamError: gateway: http://127.0.0.1:9123/snapshot unreachable (ConnectError)`,
    and printing that verbatim put the host name on the page twice.
    """
    out = alerts.build(SETTINGS, [state(key="host:gateway", label="gateway", ok=False,
                                        error="UpstreamError: gateway: http://h:9123/snapshot "
                                              "unreachable (ConnectError)")])
    assert out[0]["message"] == ("gateway: http://h:9123/snapshot unreachable (ConnectError)")
 def test_an_unexpected_exception_class_is_still_named() -> None:
    """Only `UpstreamError` is noise; a bug in our own code is worth reading about."""
    out = alerts.build(SETTINGS, [state(ok=False, error="ValueError: bad snapshot")])
    assert out[0]["message"] == "driveripper: ValueError: bad snapshot"
 def test_a_failing_source_with_no_recorded_error_still_says_so() -> None:
    out = alerts.build(SETTINGS, [state(key="llm:gufo", dashboard="llm", label="gufo",
                                        ok=False, error=None)])
    assert out[0]["message"] == "gufo: is not answering"
    assert out[0]["href"] == "/llm"
 def test_a_stale_but_serving_source_is_a_warn_because_the_page_still_shows_numbers() -> None:
    """Last good value on screen + no refresh for 3 intervals = amber, not `dead`."""
    out = alerts.build(SETTINGS, [state(age=45.0)], now=2_000.0)
    assert [item["severity"] for item in out] == ["warn"]
    assert "45s ago" in out[0]["message"]
    assert "last good value" in out[0]["message"]
    assert out[0]["at"] == 2_000.0 - 45.0
    # inside the grace period (3 x 10s) the same source is not news at all
    assert alerts.build(SETTINGS, [state(age=25.0)]) == []
 def test_staleness_is_measured_in_each_sources_own_poll_interval() -> None:
    """system polls every 10s and the UPS every 5s — one global number is wrong for one."""
    llm = state(key="llm:gufo", dashboard="llm", label="gufo", age=5.0)      # 3 x 2s = 6s
    ups = state(key="nut", dashboard="ups", label="NUT server", age=20.0)    # 3 x 5s = 15s
    assert alerts.build(SETTINGS, [llm]) == []
    out = alerts.build(SETTINGS, [ups])
    assert out[0]["severity"] == "warn" and out[0]["href"] == "/ups"
 def test_a_dashboard_the_builder_does_not_know_about_still_gets_a_verdict_and_a_home() -> None:
    """A new dashboard must not be silently un-alertable while its interval is unknown."""
    unknown_down = {"key": "probe:0", "dashboard": "telemetry", "label": "probe",
                    "configured": True, "ok": False, "error": "no route"}
    assert alerts.build(SETTINGS, [unknown_down])[0]["href"] == "/"
    unknown_stale = {**unknown_down, "ok": True, "error": None, "age_seconds": 200.0}
    out = alerts.build(SETTINGS, [unknown_stale])
    assert "every 60s" in out[0]["message"]
 def test_a_failing_source_says_when_the_poller_first_noticed() -> None:
    events = [{"ts": 1_500.0, "source": "system", "host": "driveripper", "kind": "unreachable",
               "severity": "error", "message": "ConnectError", "data": None},
              {"ts": 1_600.0, "source": "host", "host": "deskwork", "kind": "reboot",
               "severity": "warn", "message": "deskwork rebooted", "data": None}]
    out = alerts.build(SETTINGS, [state(ok=False, error="nope")], [], [], None, events,
                       now=2_000.0)
    assert out[0]["at"] == 1_500.0, "the outage clock must not reset on every refresh"
 # --------------------------------------------------------------------------- #
 # host thresholds, degraded agents
 # --------------------------------------------------------------------------- #
 def test_a_threshold_breach_names_the_host_the_metric_and_the_value(
        host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
    payload = host_scrape(agent_snapshot, cpu_busy=95.0)
    out = alerts.build(SETTINGS, [state()], [payload], now=2_000.0)
    assert out[0]["id"] == "driveripper:cpu.usage_pct"
    assert out[0]["message"] == "driveripper: CPU 95% (fail ≥90)"
    assert out[0]["href"] == "/system"
    assert out[0]["at"] == 1_010.0, "`at` is the poll that measured it"
 def test_every_rated_host_field_alerts_at_its_own_verdict(
        host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
    def nudge(snap: dict[str, Any]) -> None:
        snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.85   # warn ≥80
        snap["cpu_temperature_celsius"] = 80.0                                # warn ≥75
        for fs in snap["filesystems"]:
            if fs["mountpoint"] == "/btrfs/pool1":
                fs["used_pct"] = 91.0                                         # fail ≥90
    payload = host_scrape(agent_snapshot, cpu_busy=10.0, mutate=nudge)
    out = {item["id"]: item for item in alerts.build(SETTINGS, [state()], [payload])}
    assert out["driveripper:mem.used_pct"]["message"] == "driveripper: memory 85% (warn ≥80)"
    assert out["driveripper:cpu.temp_c"]["message"] == (
        "driveripper: CPU temperature 80°C (warn ≥75)")
    assert out["driveripper:disk.used_pct"]["message"] == (
        "driveripper: disk 91% on /btrfs/pool1 (fail ≥90)")
    assert "driveripper:cpu.usage_pct" not in out, "10% busy is not a problem"
 def test_a_number_the_upstream_does_not_publish_is_never_a_breach(
        host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
    """No measurement, no verdict, no alert — the same rule `severity.classify` upholds."""
    payload = host_scrape(agent_snapshot, cpu_busy=10.0,
                          mutate=lambda snap: snap.pop("cpu_temperature_celsius"))
    assert alerts.build(SETTINGS, [state()], [payload]) == []
 def test_a_gpu_breach_names_the_card_and_dedupes_per_metric(
        host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
    def fill_the_card(snap: dict[str, Any]) -> None:
        gpu = snap["gpus"][0]
        gpu["busy_pct"] = 97.0
        for used, total in (("vram_used_bytes", "vram_total_bytes"),
                            ("gtt_used_bytes", "gtt_total_bytes")):
            if gpu.get(total) is not None:
                gpu[used] = gpu[total]
    payload = host_scrape(agent_snapshot, cpu_busy=10.0, mutate=fill_the_card)
    messages = {item["id"]: item["message"] for item in alerts.build(SETTINGS, [state()], [payload])}
    assert any("GPU utilisation 97% on Radeon AI PRO R9700" in text
               for text in messages.values()), messages
    assert any("GPU memory" in text and "R9700" in text for text in messages.values())
 def test_a_degraded_agent_warns_and_names_the_sections_that_failed(
        host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
    payload = host_scrape(agent_snapshot, cpu_busy=10.0)
    payload["degraded"] = {"temperatures": "no hwmon sensors", "disk_io": ""}
    out = alerts.build(SETTINGS, [state()], [payload])
    assert len(out) == 1, "a partial snapshot is one warning, not one per number that is missing"
    assert out[0]["id"] == "driveripper:agent.sections"
    assert out[0]["severity"] == "warn"
    assert "temperatures (no hwmon sensors)" in out[0]["message"]
    assert "disk_io" in out[0]["message"]
 def test_a_degraded_list_from_an_older_agent_is_readable_too(
        host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
    payload = host_scrape(agent_snapshot, cpu_busy=10.0)
    payload["degraded"] = ["network"]
    out = alerts.build(SETTINGS, [state()], [payload])
    assert out[0]["message"] == "driveripper: the agent could not read network"
 def test_a_degraded_note_that_is_a_plain_string_is_still_reported(
        host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
    """An agent (or a future collector) that reports one string instead of a map."""
    payload = host_scrape(agent_snapshot, cpu_busy=10.0)
    payload["degraded"] = "hwmon"
    out = alerts.build(SETTINGS, [state()], [payload])
    assert out[0]["message"] == "driveripper: the agent could not read hwmon"
 def test_an_unreachable_host_is_one_alert_even_with_its_last_good_payload(
        host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
    """The exact completion criterion for the phase: one line naming the dead host."""
    def nudge(snap: dict[str, Any]) -> None:
        snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.99
        snap["cpu_temperature_celsius"] = 95.0
    payload = host_scrape(agent_snapshot, cpu_busy=96.0, mutate=nudge)
    out = alerts.build(SETTINGS, [state(ok=False, error="timed out")], [payload])
    assert len(out) == 1
    assert out[0]["severity"] == "fail" and "timed out" in out[0]["message"]
 def test_a_stub_payload_for_a_host_that_never_answered_alerts_once() -> None:
    """/api/system/snapshot represents that host as {name, unreachable}; still one line."""
    stub = {"name": "gateway", "ts": 1_000.0, "unreachable": True}
    out = alerts.build(SETTINGS, [state(key="host:gateway", label="gateway", ok=False,
                                        error="unreachable")], [stub])
    assert [item["id"] for item in out] == ["gateway:host:gateway:down"]
 # --------------------------------------------------------------------------- #
 # LLM engines
 # --------------------------------------------------------------------------- #
 def test_kv_pressure_comes_from_the_rated_cell_and_reads_as_a_percent() -> None:
    """The engine publishes 0-1; the operator reads percent, matching the verdict's scale."""
    out = alerts.build(SETTINGS, [state(key="llm:gufo", dashboard="llm", label="gufo")], [],
                       [engine(kv_cache_usage_ratio=ratio(0.95, "fail"))])
    assert out[0]["message"] == "gufo: KV cache 95% (fail ≥90)"
    assert out[0]["href"] == "/llm"
    assert out[0]["id"] == "gufo:llm.kv_cache_usage_pct@kv_cache_usage_ratio"
 def test_vllm_publishes_the_same_verdict_under_its_own_series_name() -> None:
    out = alerts.build(SETTINGS, [], [], [engine("vllm", gpu_cache_usage=ratio(0.8, "warn"))])
    assert out[0]["message"] == "vllm: KV cache 80% (warn ≥75)"
 def test_a_cool_or_absent_kv_cell_is_not_an_alert() -> None:
    for cell in (ratio(0.10, "ok"), ratio(None, None), ratio(0.95, None), {}, "not a cell"):
        out = alerts.build(SETTINGS, [], [], [engine(kv_cache_usage_ratio=cell)])
        assert out == [], cell
 def test_an_engine_that_is_not_answering_alerts_only_its_own_job() -> None:
    dead = {"engine": "vllm", "metrics": {}, "sessions": {}, "unreachable": True}
    out = alerts.build(SETTINGS, [state(key="llm:vllm", dashboard="llm", label="vLLM",
                                        ok=False, error="502 Bad Gateway")], [], [dead])
    assert [item["id"] for item in out] == ["vLLM:llm:vllm:down"]
 # --------------------------------------------------------------------------- #
 # UPS
 # --------------------------------------------------------------------------- #
 def test_a_failing_ups_check_is_a_fail_carrying_the_check_and_its_detail() -> None:
    payload = ups_payload(ups_card("apc",
                                   check("Power path", "fail", "The UPS is on battery"),
                                   check("Charge", "warn", "42% and discharging")))
    out = alerts.build(SETTINGS, [state(key="nut", dashboard="ups", label="NUT server")],
                       [], [], payload)
    by_id = {item["id"]: item for item in out}
    assert by_id["apc:health:Power path"]["severity"] == "fail"
    assert by_id["apc:health:Power path"]["message"] == "Power path: The UPS is on battery"
    assert by_id["apc:health:Charge"]["severity"] == "warn"
    assert all(item["href"] == "/ups" for item in out)
 def test_the_landing_page_is_where_battery_age_belongs() -> None:
    """The task's own example: a sentence an operator can act on, not a score."""
    detail = "5.3 years — past the 4-year replacement window"
    out = alerts.build(SETTINGS, [], [], [], ups_payload(ups_card("apc",
               check("Battery age", "warn", detail))))
    assert out[0]["message"] == f"Battery age: {detail}"
    assert out[0]["source"] == "apc"
 def test_an_unknown_check_is_silence_and_a_check_with_no_detail_is_still_readable() -> None:
    payload = ups_payload(ups_card("apc", check("Battery temperature", "unknown",
                                                "not reported by this driver"),
                                   check("Alarms", "fail")))
    out = alerts.build(SETTINGS, [], [], [], payload)
    assert [item["message"] for item in out] == ["Alarms"]
 def test_simulated_ups_data_is_announced_once_and_loudly() -> None:
    """With NUT_SIMULATE=1 and otherwise-healthy cards, that warning is the whole list."""
    payload = ups_payload(ups_card("apc", check("Charge", "ok", "100%")),
                          ups_card("cyberpower", check("Charge", "ok", "100%")),
                          simulated=True)
    out = alerts.build(SETTINGS, [state(key="nut", dashboard="ups", label="NUT server")],
                       [], [], payload)
    assert len(out) == 1
    assert out[0]["severity"] == "warn"
    assert "NUT_SIMULATE=1" in out[0]["message"]
    assert out[0]["href"] == "/ups"
 async def test_the_simulator_on_its_own_produces_exactly_that_warning() -> None:
    """The task's acceptance line, through the real simulator and the real payload builder."""
    readings = await NutSimulator().read()
    payload = {"ts": 1_000.0, "simulated": True,
               "ups": [nut.payload(reading) for reading in readings]}
    out = alerts.build(SETTINGS,
                       [state(key="nut", dashboard="ups", label="NUT server")], [], [], payload)
    assert [item["id"] for item in out] == ["ups:simulated"], out
 def test_a_real_upsd_does_not_claim_to_be_simulated() -> None:
    out = alerts.build(SETTINGS, [], [], [], ups_payload(ups_card("apc")))
    assert out == []
 def test_a_dead_upsd_is_one_alert_and_not_its_stale_cards_too() -> None:
    payload = ups_payload(ups_card("apc", check("Power path", "fail", "on battery")))
    out = alerts.build(SETTINGS, [state(key="nut", dashboard="ups", label="NUT server",
                                        ok=False, error="upsd closed the connection")],
                       [], [], payload)
    assert [item["source"] for item in out] == ["NUT server"]
 # --------------------------------------------------------------------------- #
 # configuration
 # --------------------------------------------------------------------------- #
 def test_a_bad_target_entry_is_a_warn_that_says_it_is_not_being_polled() -> None:
    settings = Settings(host_target_problems=("nope is not name=url",
                                              "deskwork: url must start with http://"))
    out = alerts.build(settings)
    assert len(out) == 2
    assert all(item["severity"] == "warn" and item["source"] == "config" for item in out)
    assert all(item["href"] == "/system" for item in out)
    assert "HOST_TARGETS" in out[0]["message"] and "not being polled" in out[0]["message"]
 # --------------------------------------------------------------------------- #
 # dedupe, bound, order, shape
 # --------------------------------------------------------------------------- #
 def test_one_alert_per_source_and_metric_with_the_worst_verdict_winning() -> None:
    """Two cards answering as the same name (a re-exported UPS) must not double the alarm."""
    payload = ups_payload(ups_card("apc", check("Charge", "warn", "60%")),
                          ups_card("apc", check("Charge", "fail", "12%")))
    out = alerts.build(SETTINGS, [], [], [], payload)
    assert len(out) == 1
    assert out[0]["severity"] == "fail"
    assert out[0]["message"] == "Charge: 12%"
 def test_the_list_is_bounded_and_says_how_many_it_hid() -> None:
    cards = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery"))
             for i in range(alerts.MAX_ALERTS + 5)]
    out = alerts.build(SETTINGS, [], [], [], ups_payload(*cards))
    assert len(out) == alerts.MAX_ALERTS
    marker = out[-1]
    assert marker["id"] == "alerts:suppressed"
    assert marker["severity"] == "fail", "hiding reds behind an amber marker is a lie"
    hidden = len(cards) - (alerts.MAX_ALERTS - 1)  # one slot is the marker itself
    assert marker["message"] == f"{hidden} more not shown ({hidden} failing, 0 warning)"
 def test_the_marker_is_the_worst_verdict_it_did_not_show() -> None:
    cards = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery"))
             for i in range(alerts.MAX_ALERTS)]
    cards += [ups_card(f"warn{i:02d}", check("Charge", "warn", "40%")) for i in range(3)]
    out = alerts.build(SETTINGS, [], [], [], ups_payload(*cards))
    assert len(out) == alerts.MAX_ALERTS
    assert out[-1]["id"] == "alerts:suppressed"
    assert out[-1]["severity"] == "fail", "a red is behind the marker, so the marker is red"
    # and when only warnings are hidden, the marker says amber — it describes what it hid
    mild = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery"))
            for i in range(3)]
    mild += [ups_card(f"warn{i:02d}", check("Charge", "warn", "40%"))
             for i in range(alerts.MAX_ALERTS)]
    out = alerts.build(SETTINGS, [], [], [], ups_payload(*mild))
    assert out[-1]["severity"] == "warn"
 def test_exactly_the_cap_adds_no_marker() -> None:
    cards = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery"))
             for i in range(alerts.MAX_ALERTS)]
    out = alerts.build(SETTINGS, [], [], [], ups_payload(*cards))
    assert len(out) == alerts.MAX_ALERTS
    assert all(item["id"] != "alerts:suppressed" for item in out)
 def test_fails_sort_before_warns_and_then_by_source() -> None:
    payload = ups_payload(ups_card("zeta", check("Power path", "fail", "on battery")),
                          ups_card("alpha", check("Charge", "warn", "40%")),
                          ups_card("beta", check("Load", "warn", "82%")))
    out = alerts.build(SETTINGS, [], [], [], payload)
    assert [(item["severity"], item["source"]) for item in out] == [
        ("fail", "zeta"), ("warn", "alpha"), ("warn", "beta")]
 def test_the_trip_note_quotes_the_threshold_table_in_the_right_direction() -> None:
    """`_trip` reads `severity.THRESHOLDS` back instead of carrying its own numbers.
    Reached through `build()` only by metrics that are not inverted today, so this checks
    the helper directly: a low-is-worse metric must print `≤` (an `alert` for a battery is
    exactly where a wrong sign would be published), and a metric with no threshold gets no
    note rather than a made-up one.
    """
    assert alerts._trip("cpu.usage_pct", "fail") == " (fail ≥90)"
    assert alerts._trip("ups.battery_runtime", "warn") == " (warn ≤600)"
    assert alerts._trip("ups.charge_pct", "fail") == " (fail ≤20)"
    assert alerts._trip("cpu.usage_pct", "ok") == ""
    assert alerts._trip("mystery.metric", "fail") == ""
 def test_the_alert_shape_is_the_contract_the_pages_render() -> None:
    out = alerts.build(SETTINGS, [state(ok=False, error="nope")], now=2_000.0)
    assert out[0] == {"id": "driveripper:host:driveripper:down", "severity": "fail",
                      "source": "driveripper", "message": "driveripper: nope",
                      "href": "/system", "at": 2_000.0}
@@ -7,30 +7,38 @@ asset the network can fail to deliver.
 """
 from __future__ import annotations
 import copy
 import re
 import time
 from collections.abc import Iterator
 from contextlib import contextmanager
 from pathlib import Path
 from typing import Any
 import pytest
 from fastapi import HTTPException
 from fastapi.testclient import TestClient
 from dashboards import alerts
 from dashboards.api import RANGE_SECONDS, parse_range
 from dashboards.app import create_app
 from dashboards.config import Settings, Target
 from dashboards.rates import RateEstimator
 from dashboards.severity import classify
-from dashboards.sources.host import HostCollector, parse_host_snapshot
+from dashboards.sources import nut
 from dashboards.store import Event, Sample, SampleStore
 # refused immediately, and never dials out of the test environment
 DEAD_URL = "http://127.0.0.1:1"
@contextmanager
 def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: bool = False,
-                 **overrides: Any) -> TestClient:
+                 **overrides: Any) -> Iterator[TestClient]:
-    """A TestClient whose lifespan runs, against a private DB and no real network."""
+    """A TestClient whose lifespan runs, against a private DB and no real network.
    The store is injected, so `create_app()` deliberately does not close it (production
    owns its own); the caller here does, or every test leaks a SQLite handle and the
    `--cov` log fills with ResourceWarnings that bury the warnings worth reading.
    """
    if simulate:
        monkeypatch.setenv("NUT_SIMULATE", "1")
    else:
@@ -38,10 +46,13 @@ def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: b
    db = str(tmp_path / "api.sqlite3")
    settings = Settings(db_path=db, **overrides)
    store = SampleStore(db)
-    app = create_app(settings, store)
+    client = TestClient(create_app(settings, store))
    client = TestClient(app)
    client.__enter__()
-    return client
+    try:
        yield client
    finally:
        client.__exit__(None, None, None)
        store.close()
@pytest.fixture
@@ -68,29 +79,23 @@ def wait_for_payload(client: TestClient, path: str, *, timeout: float = 5.0) ->
    pytest.fail(f"{path} never became available")
-def host_payload(agent_snapshot: dict[str, Any]) -> dict[str, Any]:
+def failing_host_payload(scrape: Any, agent_snapshot: dict[str, Any]) -> dict[str, Any]:
    """Two scrapes of the real driveripper capture, nudged onto known sides of the table.
    Driven from the fixture rather than hand-written so a schema drift breaks here first.
    """
-    snap = copy.deepcopy(agent_snapshot)
+    def nudge(snap: dict[str, Any]) -> None:
-    snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95   # mem.used_pct -> fail
+        snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95   # mem.used_pct -> fail
-    snap["cpu_temperature_celsius"] = 91.0                                 # cpu.temp_c -> fail
+        snap["cpu_temperature_celsius"] = 91.0                                # cpu.temp_c -> fail
-    for fs in snap["filesystems"]:
+        for fs in snap["filesystems"]:
-        fs["used_pct"] = 93.0                                              # disk -> fail
+            fs["used_pct"] = 93.0                                             # disk -> fail
-    collector = HostCollector(RateEstimator(span=60.0))
+
-    parse_host_snapshot("driveripper", snap, collector, now=1_000.0)
+    return scrape(agent_snapshot, cpu_busy=10.0, mutate=nudge)
    second = copy.deepcopy(snap)
    second["ts"] = 1_010.0
    modes = second["cpu"]["modes"]
    modes["total"] += 1_000.0
    modes["idle"] += 900.0            # 10 % busy over the window -> ok
    return parse_host_snapshot("driveripper", second, collector, now=1_010.0).payload
@pytest.fixture
 def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
-                agent_snapshot: dict[str, Any]) -> Iterator[TestClient]:
+                agent_snapshot: dict[str, Any], host_scrape: Any) -> Iterator[TestClient]:
    """Overview-shaped app: one host payload in the hub plus the simulated UPSes.
    The host job points at a dead port, so a poll can never overwrite the injected payload
@@ -104,7 +109,8 @@ def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
    test_client = TestClient(app)
    test_client.__enter__()
    try:
-        app.state.ctx.hub.jobs["host:driveripper"].state.payload = host_payload(agent_snapshot)
+        app.state.ctx.hub.jobs["host:driveripper"].state.payload = failing_host_payload(
            host_scrape, agent_snapshot)
        wait_for_payload(test_client, "/api/ups/snapshot")
        yield test_client
    finally:
@@ -112,6 +118,41 @@ def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
        store.close()
@pytest.fixture
 def healthy_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
                   agent_snapshot: dict[str, Any],
                   host_scrape: Any) -> Iterator[TestClient]:
    """The all-clear app: one host answering, nothing past a threshold, no UPS configured.
    The job's collector is swapped for one that hands back the injected payload, so the
    health record says `ok` truthfully instead of being overridden behind the poller.
    """
    with build_client(tmp_path, monkeypatch,
                      host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
        from dashboards.poller import Result  # noqa: PLC0415 - only this fixture needs it
        payload = host_scrape(agent_snapshot, cpu_busy=10.0)
        async def collect() -> Result:
            return Result(payload=payload, samples=[])
        job = test_client.app.state.ctx.hub.jobs["host:driveripper"]  # type: ignore[attr-defined]
        job.fn = collect
        # let the poll already in flight against the dead port land first (it fails in
        # microseconds), then take the health record over: the alerts must describe a host
        # that is answering, not the half-second of startup where it was not
        deadline = time.time() + 2.0
        while time.time() < deadline and job.state.attempts == 0:
            time.sleep(0.02)
        time.sleep(0.05)
        job.state.payload = payload
        job.state.ok = True
        job.state.error = None
        job.state.consecutive_failures = 0
        job.state.last_ok = time.time()
        yield test_client
 # ---- severity travels with the value --------------------------------------- #
 def test_summary_carries_the_verdict_for_every_column_the_page_colours(data_client) -> None:
@@ -147,13 +188,121 @@ def test_summary_shape_did_not_shrink(data_client) -> None:
    assert fleet_keys <= set(body["system"]["fleet"]), fleet_keys - set(body["system"]["fleet"])
    assert host_keys <= set(body["system"]["hosts"][0]), host_keys - set(body["system"]["hosts"][0])
    assert ups_keys <= set(body["ups"]["ups"][0]), ups_keys - set(body["ups"]["ups"][0])
-    assert set(body) == {"ts", "system", "llm", "ups", "recent_events"}
+    assert set(body) == {"ts", "system", "llm", "ups", "recent_events", "alerts"}
    # ...and the new keys are there beside the old ones, not instead of them
    assert "severity" in body["system"]["fleet"]
    assert "severity" in body["system"]["hosts"][0]
    assert "severity" in body["ups"]["ups"][0]
 # `/` paints two panels; the payload is still the whole-fleet contract. Every path here is
 # read by something that is not the overview page: the live suite, an external probe, a
 # notifier later. Deleting a card must never delete a key.
 SUMMARY_CONTRACT = ("system.enabled", "system.fleet", "system.hosts",
                    "llm.engines", "ups.ups", "recent_events", "alerts")
 def dig(body: dict[str, Any], path: str) -> Any:
    """`body["a"]["b"]` that names the missing path when it is not there."""
    node: Any = body
    for part in path.split("."):
        assert isinstance(node, dict) and part in node, f"/api/summary lost `{path}`"
        node = node[part]
    return node
@pytest.mark.parametrize("fixture", ["client", "data_client"])
 def test_summary_serves_every_path_another_consumer_reads(request, fixture: str) -> None:
    """The keys survive the overview cleanup whether or not anything is configured.
    Run twice on purpose: `client` has no targets at all, so it proves the keys are present
    as `None`/`[]` rather than omitted when a source is unconfigured (invariant 3 is about
    the payload shape as much as about colour); `data_client` proves the same paths carry
    the fleet once there is one.
    """
    test_client = request.getfixturevalue(fixture)
    body = test_client.get("/api/summary").json()
    populated = fixture == "data_client"
    for path in SUMMARY_CONTRACT:
        dig(body, path)  # presence is the assertion; a missing path raises with its name
    assert dig(body, "system.enabled") is populated
    assert (dig(body, "system.fleet") is not None) is populated
    names = {host["name"] for host in dig(body, "system.hosts")}
    assert names == ({"driveripper"} if populated else set())
    batteries = {item["name"] for item in dig(body, "ups.ups")}
    assert batteries == ({"apc", "cyberpower"} if populated else set())
    assert dig(body, "llm.engines") == [], "neither fixture configures an engine"
    assert isinstance(dig(body, "recent_events"), list)
    assert isinstance(dig(body, "alerts"), list)
    # and the payload as a whole still carries every top-level section it always did
    assert set(body) >= {"ts", "system", "llm", "ups", "recent_events", "alerts"}
 # ---- alerts: the landing page's answer to "what is wrong" ------------------- #
 def test_summary_is_empty_of_alerts_when_everything_is_healthy(healthy_client) -> None:
    """`[]`, not a green banner: an empty list is the clearest possible all-clear."""
    body = healthy_client.get("/api/summary").json()
    assert body["alerts"] == []
    assert body["system"]["fleet"]["hosts_live"] == 1
 def test_summary_raises_one_alert_for_the_host_that_stopped_answering(data_client) -> None:
    """One dead host is one line naming it, with the link to the page that explains it."""
    body = data_client.get("/api/summary").json()
    down = [item for item in body["alerts"] if item["source"] == "driveripper"]
    assert len(down) == 1, "its stale payload must not add amber lines on top of the red one"
    assert down[0]["severity"] == "fail"
    assert down[0]["href"] == "/system"
    assert "unreachable" in down[0]["message"]
 def test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet(
        tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
    """A dead agent is a red row in the fleet, not a fleet that quietly got smaller.
    `/api/system/snapshot` already stubs a job with no payload; `/api/summary` has to agree
    with it, or a target that never answered since startup leaves the table at "2/2 hosts"
    while the panel above says a third box is down (`AGENTS.md` invariants 3 and 4).
    """
    with build_client(tmp_path, monkeypatch,
                      host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
        body = test_client.get("/api/summary").json()
        rows = body["system"]["hosts"]
        assert [row["name"] for row in rows] == ["driveripper"]
        assert rows[0]["unreachable"] is True
        assert rows[0]["cpu_usage_pct"] is None, "n/a, never a 0.0 nobody measured"
        assert body["system"]["fleet"]["hosts_total"] == 1
        assert body["system"]["fleet"]["hosts_live"] == 0
        # and the alert list stays exactly one line for it, not a row plus a stale payload
        assert [item["source"] for item in body["alerts"]] == ["driveripper"]
 def test_summary_alerts_are_worst_first_and_carry_the_contract(data_client) -> None:
    body = data_client.get("/api/summary").json()
    alerts_list = body["alerts"]
    assert alerts_list, "a dead host plus simulated UPS data cannot read as all-clear"
    for item in alerts_list:
        assert set(item) == {"id", "severity", "source", "message", "href", "at"}
        assert item["severity"] in ("warn", "fail")
    ranks = [0 if item["severity"] == "fail" else 1 for item in alerts_list]
    assert ranks == sorted(ranks), "fails before warns"
    assert any("simulat" in item["message"].lower() and item["href"] == "/ups"
               for item in alerts_list), "NUT_SIMULATE must be said out loud on the landing page"
 def test_summary_reuses_the_alert_builder_verbatim(data_client) -> None:
    """The endpoint is a thin call: what `alerts.build` says over the same state is the body."""
    body = data_client.get("/api/summary").json()
    hub = data_client.app.state.ctx.hub  # type: ignore[attr-defined]
    settings = data_client.app.state.ctx.settings  # type: ignore[attr-defined]
    hosts = [job.state.payload for job in hub.jobs.values()
             if job.dashboard == "system" and job.state.payload]
    ups_payload = hub.jobs["nut"].state.payload
    expected = alerts.build(settings, hub.states(), hosts, [], ups_payload, [])
    assert [item["id"] for item in body["alerts"]] == [item["id"] for item in expected]
 def test_snapshot_payloads_carry_their_own_severity_sections(data_client) -> None:
    """/api/system/snapshot and /api/ups/snapshot ship the same verdicts the summary reuses."""
    system = data_client.get("/api/system/snapshot").json()
@@ -371,14 +520,103 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
 def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
-    """The old call sites still work until phases 02-05 convert each page."""
+    """The old call sites still work until the remaining phases convert each page.
    `/` was converted in phase 02: the overview colours from the `severity` verdicts
    `/api/summary` already sends, so it no longer carries a threshold of its own. The
    three pages still to come must keep compiling against the fallback until they are.
    """
    js = client.get("/static/app.js").text
    assert "FALLBACK ONLY" in js
-    for page in ("/static/page-index.js", "/static/page-system.js",
+    for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"):
                 "/static/page-llm.js", "/static/page-ups.js"):
        assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site"
 def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> None:
    """`page-index.js` reads verdicts; it does not compare a number to a bound."""
    js = client.get("/static/page-index.js").text
    assert "severityFrom" in js
    assert "severityFor" not in js
 def test_overview_is_exactly_two_panels(client) -> None:
    """The landing page answers one question: what is wrong, and what is the fleet doing.
    Asserted on the served bytes because there is no JS test runner (a locked anchor):
    the summary cards and the raw event feed are what made this page busy, and once a
    panel is gone a browser is the only thing that would notice its return.
    """
    html = client.get("/").text
    assert html.count("<section") == 2, "the overview grew a third panel"
    assert 'id="alerts"' in html and 'id="alerts-clear"' in html
    assert 'aria-live="polite"' in html
    assert 'id="fleet-table"' in html
    for gone in ("sys-cpu", "sys-mem", "sys-disk", "sys-gpus", "llm-decode", "llm-kv",
                 "ups-list", 'id="events"'):
        assert gone not in html, f"the overview still renders {gone}"
    # the header's source dots are chrome owned by app.js, not the deleted card dots
    assert 'id="dot-system"' not in html and 'id="nav-status"' in html
 def test_overview_alert_rows_link_to_the_dashboard_that_explains_them(client) -> None:
    """Every alert carries the `href` its row turns into a link; the page invents none."""
    page_js = client.get("/static/page-index.js").text
    assert "alert.href" in page_js
    css = client.get("/static/dash.css").text
    for selector in ("ul.alerts li.ok .src", "ul.alerts li.warn .src", "ul.alerts li.fail .src",
                     "table.data td.warn", "table.data td.fail"):
        assert selector in css, f"dash.css lost {selector}"
    # the swap that puts a live problem above the fleet table
    assert 'data-state="alerting"' in css
 def test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering(client) -> None:
    """A dead host keeps its last good numbers, so the row has to say they are stale.
    `AGENTS.md` invariant 4: keep the payload and mark it stale, never zero it. The
    overview's table is the one place those last good numbers still print, so it reads the
    per-source verdicts `/api/summary` sends (`system.states`) and applies the shared
    `.stale` class the other pages already use — no new palette, no JS threshold.
    """
    page_js = client.get("/static/page-index.js").text
    assert "system.states" in page_js, "the fleet table stopped reading reachability"
    assert 'class="stale"' in page_js
    assert ".stale" in client.get("/static/dash.css").text
 # Which script owns which page, for the orphan check below. `charts.js` and `app.js` are
 # loaded by all four pages, so an id either page code or shared chrome may read.
 PAGE_SCRIPTS = {
    "/": "page-index.js",
    "/system": "page-system.js",
    "/llm": "page-llm.js",
    "/ups": "page-ups.js",
 }
 SHARED_SCRIPTS = ("app.js", "charts.js")
 def test_no_page_carries_an_id_nothing_reads(client) -> None:
    """An id no script queries, no stylesheet styles and no attribute points at is dead markup.
    The summary cards were retired this way — a `<span id="sys-cpu">` left behind after its
    writer was deleted is the same mistake in slower motion. Asserted on served bytes: the
    browser console is otherwise the only thing that would notice.
    """
    scripts = {name: client.get(f"/static/{name}").text for name in
               (*SHARED_SCRIPTS, *PAGE_SCRIPTS.values(), "dash.css")}
    for path, own_script in PAGE_SCRIPTS.items():
        html = _code(client.get(path).text)
        corpus = "".join(_code(scripts[name])
                         for name in (*SHARED_SCRIPTS, own_script, "dash.css"))
        # ids the markup uses itself: labels, targets, `<label for>`
        anchors = set(re.findall(
            r'(?:aria-labelledby|aria-describedby|aria-controls|for)="([^"]+)"', html))
        for element_id in re.findall(r'id="([^"]+)"', html):
            quoted = (f'"{element_id}"' in corpus or f"'{element_id}'" in corpus
                      or f"#{element_id}" in corpus or element_id in anchors)
            assert quoted, f"{path} has id=\"{element_id}\" that nothing reads"
 def test_api_responses_are_not_cacheable(client) -> None:
    for path in ("/api/config", "/api/health", "/api/summary"):
        assert "no-store" in client.get(path).headers.get("cache-control", "")
@@ -413,6 +651,87 @@ def test_history_requires_the_series_coordinates(client) -> None:
    assert client.get("/api/history").status_code == 422
@pytest.mark.parametrize(("raw", "seconds"), [
    *[(key, float(seconds)) for key, seconds in RANGE_SECONDS.items()],
    ("45s", 45.0), ("13h", 46_800.0), ("7200", 7_200.0), ("  2D  ", 172_800.0),
    (None, 3_600.0), ("", 3_600.0),
 ])
 def test_parse_range_reads_every_window_the_ui_offers(raw: Any, seconds: float) -> None:
    """Every chip key (5m…90d), any `<n><s|m|h|d>`, raw seconds, and the 1h default.
    `parse_range` is the only thing between a typo'd query string and a chart of the wrong
    window, and a wrong window is worse than an error: it still looks like data.
    """
    assert parse_range(raw) == seconds
@pytest.mark.parametrize("raw", ["banana", "1w", "h", "-4h", "one"])
 def test_parse_range_refuses_a_window_it_cannot_interpret(raw: str) -> None:
    """A 400 that names the value, never a silent fallback to 1h."""
    with pytest.raises(HTTPException) as excinfo:
        parse_range(raw)
    assert excinfo.value.status_code == 400
    assert raw in str(excinfo.value.detail)
 # ---- the routes the *other* pages chart from -------------------------------- #
 #
 # `/` renders two panels now; `/system`, `/llm` and `/ups` still need these, and so does
 # anything outside the browser. Retiring overview code must not retire them, so the shapes
 # are pinned here rather than discovered by a page three phases from now.
 def test_system_history_charts_every_configured_host_in_one_request(
        tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
    """N hosts x M metrics, one round trip — the chart endpoint's whole reason to exist."""
    with build_client(tmp_path, monkeypatch,
                      host_targets=(Target("driveripper", DEAD_URL),
                                    Target("deskwork", DEAD_URL))) as test_client:
        body = test_client.get("/api/system/history",
                               params={"metrics": "cpu.usage_pct,mem.used_pct",
                                       "range": "1h"}).json()
        assert body["hosts"] == ["driveripper", "deskwork"]
        assert body["range_seconds"] == 3_600.0
        assert len(body["series"]) == 4
        assert {(item["host"], item["metric"]) for item in body["series"]} == {
            ("driveripper", "cpu.usage_pct"), ("driveripper", "mem.used_pct"),
            ("deskwork", "cpu.usage_pct"), ("deskwork", "mem.used_pct")}
        # an explicit `hosts` list narrows it instead of adding to the default
        narrowed = test_client.get("/api/system/history",
                                   params={"metrics": "cpu.usage_pct",
                                           "hosts": "deskwork"}).json()
        assert narrowed["hosts"] == ["deskwork"]
 def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
        tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
    """A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
    # a host target too, so the snapshot's job filter has to skip a non-LLM job
    with build_client(tmp_path, monkeypatch, vllm_url=DEAD_URL,
                      host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
        body = test_client.get("/api/llm/snapshot").json()
        assert [engine["engine"] for engine in body["engines"]] == ["vLLM"]
        assert body["engines"][0]["unreachable"] is True
        assert body["fleet"]["engines_total"] == 1
        assert body["fleet"]["engines_live"] == 0
        assert [state["key"] for state in body["states"]] == ["llm:vllm"]
 def test_llm_history_charts_both_engines_by_default(
        tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
    with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
                      vllm_url=DEAD_URL) as test_client:
        body = test_client.get("/api/llm/history",
                               params={"metrics": "decode_tokens_per_second,kv_cache_usage_ratio",
                                       "range": "15m"}).json()
        assert body["engines"] == ["gufo", "vllm"]
        assert body["range_seconds"] == 900.0
        assert {(item["host"], item["metric"]) for item in body["series"]} == {
            ("gufo", "llm.decode_tokens_per_second"),
            ("gufo", "llm.kv_cache_usage_ratio"),
            ("vllm", "llm.decode_tokens_per_second"),
            ("vllm", "llm.kv_cache_usage_ratio")}
 def test_metrics_endpoint_lists_what_actually_has_history(client) -> None:
    body = client.get("/api/metrics").json()
    assert body["metrics"] == []
@@ -474,6 +793,153 @@ def test_a_dangerous_command_needs_the_confirmation_word(ups_client) -> None:
                           json={"ups": name, "command": deep, "confirm": "DANGER"}).status_code == 200
 def test_a_command_the_driver_does_not_advertise_is_refused_before_it_is_sent(
        ups_client) -> None:
    """Being in the risk catalog is not permission; `LIST CMD` from the driver is (invariant 1).
    `shutdown.return` is a real NUT command the catalog knows and this UPS does not offer,
    so the refusal must come from the advertisement check and not from the confirmation gate.
    """
    name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
    response = ups_client.post("/api/ups/command",
                               json={"ups": name, "command": "shutdown.return"})
    assert response.status_code == 400
    assert "does not advertise" in response.json()["detail"]
 def test_cutting_the_output_is_refused_while_the_ups_is_already_on_battery(
        tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
    """Invariant 1, the one that matters most: no power cut while the pack is carrying the load.
    Driven through the simulator's own state machine — a quick battery test puts the pack on
    battery — so the 409 comes from the same poll the buttons are drawn from, with a 0.5s
    interval to keep the wait short. Nothing here can touch real hardware: `NUT_SIMULATE=1`.
    """
    with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
                      nut_allow_actions=True, nut_interval=0.5) as test_client:
        name = wait_for_payload(test_client, "/api/ups/snapshot")["ups"][0]["name"]
        start = test_client.post("/api/ups/command",
                                 json={"ups": name, "command": "test.battery.start.quick"})
        assert start.status_code == 200
        deadline = time.time() + 6.0
        card: dict[str, Any] = {}
        while time.time() < deadline:
            card = next(item for item in test_client.get("/api/ups/snapshot").json()["ups"]
                        if item["name"] == name)
            if card["on_battery"]:
                break
            time.sleep(0.05)
        assert card.get("on_battery") is True, "the quick test must put the simulated pack on battery"
        refusal = test_client.post("/api/ups/command",
                                   json={"ups": name, "command": "load.off",
                                         "confirm": "DANGER"})
        assert refusal.status_code == 409
        assert "on battery" in refusal.json()["detail"]
        # and the same command is still offered once the pack is back on line power
        assert not any(item["command"] == "load.off" and item["enabled"]
                       for item in card["commands"]), "the panel must not offer a blocked button"
 class ScriptedNutClient:
    """`nut.NutClient` with the socket left out: the real code path, no battery attached.
    `AGENTS.md` keeps live tests read-only, so this is where the one write endpoint's real
    branch gets proven. The variable names are the ones `VAR_MAP` already pins from real
    upsd captures; only the replies are scripted (a fake for behaviour, per the test policy).
    """
    #: set per test: an exception instance to raise from `instcmd`, e.g. a refusal
    refusal: Exception | None = None
    sent: list[tuple[str, str]]
    logged_in: list[str]
    def __init__(self, host: str, port: int, *, timeout: float = 4.0,
                 username: str | None = None, password: str | None = None) -> None:
        self.credentials = (username, password)
    async def __aenter__(self) -> ScriptedNutClient:
        return self
    async def __aexit__(self, *_exc: object) -> None:
        return None
    async def list_ups(self) -> list[dict[str, str]]:
        return [{"name": "apc", "description": "APC Back-UPS RS 1500MS"}]
    async def list_vars(self, ups: str) -> dict[str, str]:
        return {"ups.mfr": "APC", "ups.model": "Back-UPS RS 1500MS",
                "ups.status": "OL", "battery.charge": "100", "ups.load": "18",
                "battery.voltage": "13.6", "input.voltage": "230",
                "battery.mfr.date": "2022-04", "ups.test.result": "Ok last test passed"}
    async def list_cmds(self, ups: str) -> list[str]:
        return ["test.panel.start", "beeper.mute", "load.off"]
    async def list_rw_vars(self, ups: str) -> list[str]:
        return []
    async def login(self) -> None:
        assert self.credentials[0], "a command send must authenticate as a named user"
        ScriptedNutClient.logged_in.append(str(self.credentials[0]))
    async def instcmd(self, ups: str, command: str) -> str:
        if ScriptedNutClient.refusal is not None:
            raise ScriptedNutClient.refusal
        ScriptedNutClient.sent.append((ups, command))
        return "OK Start Panel Test"
@pytest.fixture
 def real_upsd_client(monkeypatch: pytest.MonkeyPatch) -> Iterator[type[ScriptedNutClient]]:
    """Swap the module-level `NutClient` so the collector and the send use the same stand-in."""
    ScriptedNutClient.sent = []
    ScriptedNutClient.logged_in = []
    ScriptedNutClient.refusal = None
    monkeypatch.setattr(nut, "NutClient", ScriptedNutClient)
    yield ScriptedNutClient
@pytest.fixture
 def nut_client_app(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
                   real_upsd_client: type[ScriptedNutClient]) -> Iterator[TestClient]:
    """A non-simulated UPS app: real code paths, scripted upsd, 0.5s polls."""
    with build_client(tmp_path, monkeypatch, nut_host="127.0.0.1", nut_user="dashboards",
                      nut_password="not-the-real-one", nut_allow_actions=True,
                      nut_interval=0.5) as test_client:
        yield test_client
 def test_a_real_upsd_send_authenticates_sends_and_audits(nut_client_app,
                                                         real_upsd_client) -> None:
    body = wait_for_payload(nut_client_app, "/api/ups/snapshot")
    assert body["simulated"] is False, "this test is about the branch that talks to upsd"
    name = body["ups"][0]["name"]
    response = nut_client_app.post("/api/ups/command",
                                   json={"ups": name, "command": "test.panel.start"})
    assert response.status_code == 200, response.text
    assert response.json()["simulated"] is False
    assert real_upsd_client.sent == [(name, "test.panel.start")]
    assert real_upsd_client.logged_in, "instcmd requires a LOGIN first (upsd refuses otherwise)"
    events = nut_client_app.get("/api/events").json()["events"]
    assert any(event["kind"] == "action" and name in event["message"] for event in events)
 def test_a_refusal_from_upsd_is_a_502_and_an_error_event(nut_client_app,
                                                         real_upsd_client) -> None:
    real_upsd_client.refusal = nut.NutError("Access denied", code="ACCESS-DENIED")
    name = wait_for_payload(nut_client_app, "/api/ups/snapshot")["ups"][0]["name"]
    response = nut_client_app.post("/api/ups/command",
                                   json={"ups": name, "command": "beeper.mute"})
    assert response.status_code == 502
    detail = response.json()["detail"]
    assert "ACCESS-DENIED" in detail or "Access denied" in detail, detail
    assert real_upsd_client.sent == [], "a refused command was never accepted"
    events = nut_client_app.get("/api/events").json()["events"]
    failures = [event for event in events if event["kind"] == "action"
                and event["severity"] == "error"]
    assert failures, "an upsd refusal has to leave a trace, not just an HTTP status"
 def test_unknown_commands_are_refused_even_with_actions_on(ups_client) -> None:
    name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
    for command in ("load.format", "rm -rf /", "INSTCMD x y"):
@@ -490,6 +490,19 @@ async def test_simulated_ups_cards_carry_severity_too() -> None:
        assert card["severity"]["charge_pct"] == "ok", "the sim starts at float charge"
 async def test_a_fresh_simulator_reads_healthy_so_amber_still_means_something() -> None:
    """The dev model must not train people to ignore the UPS badge.
    Its battery dates are fiction, so they are computed relative to today: a hard-coded
    2019 manufacture date turns every simulated card amber forever, and an amber nobody
    can clear is the same failure `assess()` avoids for a UPS that floats at 90 %.
    """
    for ups in await NutSimulator().read():
        card = nut.payload(ups)
        assert card["health"]["overall"] == "healthy", card["health"]["checks"]
        assert not [c for c in card["health"]["checks"] if c["status"] == "fail"]
 def test_history_samples_only_include_real_numbers() -> None:
    values = {key: value for key, value in VAR_ROWS.items() if key != "battery.runtime"}
    samples = nut.history_samples(nut.UpsReading(name="cyberpower", vars=values, commands=[]))