phase: 02_overview_fleet_and_alerts
**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**
- Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
This commit is contained in:
+12
@@ -0,0 +1,12 @@
|
||||
**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**
|
||||
|
||||
- Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
|
||||
- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
|
||||
- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
|
||||
- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
|
||||
- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
|
||||
- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
|
||||
- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
|
||||
- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
|
||||
- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
|
||||
- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 48%]
|
||||
.......ssssss........................................................... [ 64%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 97%]
|
||||
............. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 175 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
|
||||
src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115
|
||||
src/dashboards/config.py 110 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 139 1 99% 226
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 2 99% 122, 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 226
|
||||
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
|
||||
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2104 67 97%
|
||||
Required test coverage of 94% reached. Total coverage: 96.82%
|
||||
439 passed, 6 skipped, 1 warning in 43.39s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
**Implemented**
|
||||
- `src/dashboards/alerts.py` (new, 100% covered): pure `build(settings, hub_states, host_payloads, engine_payloads, ups_payload, recent_events, *, now)` → `[{id, severity, source, message, href, at}]`, fail→warn then by source. Rules: unconfigured never alerts; `ok=false` → `fail` (collectors' name prefix de-duplicated, `at` from the poller's `unreachable` event); `ok` but older than 3× its own poll interval → `warn`; host/engine/UPS threshold breaches read only phase-01 `severity` dicts + `health.checks[]` (no new thresholds; bound quoted back from `severity.THRESHOLDS`/`INVERTED`); degraded-agent sections → `warn`; `HOST_TARGETS` problems → `warn`; dedupe by (source, metric) keeping the worst verdict; capped at `MAX_ALERTS=12` with a marker stating how many were hidden; a dead source yields exactly one line (stale payload suppressed).
|
||||
- `src/dashboards/api.py`: `/api/summary` calls it and adds `"alerts"`; `system`/`llm`/`ups`/`recent_events` byte-identical otherwise (additive-only).
|
||||
- Tests: `tests/test_alerts.py` (40 tests incl. all-clear `[]`, unconfigured, stale-vs-dead, per-source intervals, KV pressure, UPS detail strings, simulated-only, dedupe/cap/sort/shape), 4 new `/api/summary` tests + `healthy_client` fixture in `test_api.py`, shared `host_scrape` fixture in `conftest.py`.
|
||||
- Defects fixed in passing: `nut_sim` hard-coded `battery.mfr.date` (2019/2022) aged into permanent amber → computed relative to today (+ test); `build_client()` leaked its `SampleStore` (28 ResourceWarnings under `--cov`) → now a context manager that closes it.
|
||||
|
||||
**Gate** (`bash .agents/validate.sh`, exit 0): `uv run pytest --cov --cov-fail-under=94` → **400 passed, 6 skipped, 1 warning; total coverage 95.18%**, `alerts.py` 100% (changed part of `api.py` fully covered); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; repo hygiene clean. `uv run pytest -m live --live` → 6 passed (unchanged). Dev server restarted: all 6 sources `ok`; `curl :8090/api/summary` keys `['alerts','llm','recent_events','system','ts','ups']` with live alerts (e.g. `fail deskwork /system deskwork: GPU utilisation 100% … (fail ≥95)`, `warn apc /ups Battery age: 5.3 years — past the 4-year replacement window`).
|
||||
|
||||
**Docs**: AGENTS.md — `alerts.py` responsibility bullet, `/api/summary` additive note, nut_sim date behaviour, refreshed test/coverage counts (400 / 95.2%); README.md — `alerts.py` bullet in Architecture.
|
||||
|
||||
**Decisions / deviations**
|
||||
- LLM alerts come from the cells `llm.SEVERITY_CELLS` already rates (KV pressure only) so no thresholds are invented here; `engine_payloads` therefore has a real purpose beyond reachability.
|
||||
- The cap keeps the list at exactly `MAX_ALERTS` by replacing the last slot with an `alerts:suppressed` marker whose severity is the worst item it hid (tested both directions).
|
||||
- Live fleet today is *not* all-clear: the real UPSes emit `warn` checks (battery age, "No test initiated"), so the `[]`-when-healthy and "NUT_SIMULATE=1 → exactly the simulated warning" criteria are asserted hermetically instead; I did not change `nut.assess()` semantics.
|
||||
- No live infra change was made (no tunnel was killed); "one dead host → exactly one alert with `/system`" is covered by `test_summary_raises_one_alert_for_the_host_that_stopped_answering`, and the browser proof belongs to task 02.
|
||||
|
||||
**Next pending task**: `.agents/phases/todo/02_overview_fleet_and_alerts/02_rewrite_overview_page.md` (reduce `index.html` + `page-index.js` to fleet table + alerts panel).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 35%]
|
||||
........................................ssssss.......................... [ 53%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 88%]
|
||||
.............................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 166 0 100%
|
||||
src/dashboards/api.py 193 28 85% 60-61, 63, 294-300, 317-325, 332-336, 420, 425-426, 432-443
|
||||
src/dashboards/app.py 109 7 94% 50-51, 56, 58, 60, 97, 131
|
||||
src/dashboards/collectors.py 140 9 94% 78-79, 92-93, 97-98, 115, 232, 236
|
||||
src/dashboards/config.py 110 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 139 1 99% 226
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 2 99% 122, 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 226
|
||||
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
|
||||
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
|
||||
src/dashboards/store.py 178 3 98% 204, 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2095 101 95%
|
||||
Required test coverage of 94% reached. Total coverage: 95.18%
|
||||
400 passed, 6 skipped, 1 warning in 40.23s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
**Task 02 complete — all green.**
|
||||
|
||||
**Implemented**
|
||||
- `web/pages/index.html`: deleted the three summary cards, card dots, and events panel; page is now exactly two `<section>`s — **Warnings & errors** (`#alerts`, `aria-live="polite"`, "All clear" empty state) and **Fleet** (`#fleet-table` columns unchanged); masthead/`#nav-status`/`#clock`/`#theme-toggle`/`#config-chip`/`#banners`/`#foot-versions` kept.
|
||||
- `web/static/page-index.js`: `renderEvents()` + all `#sys-*`/`#llm-*`/`#ups-list` code deleted; new `renderAlerts()` (row = verdict class via `Dash.severityFrom`, source linked via the API's `href`, message, age via `Dash.age()`) and `renderFleet()` (cells coloured from `/api/summary` row `severity` verdicts only — no JS thresholds); `#summary-line` is "3/3 hosts · 2/2 engines · 2 UPS · N warnings / all clear"; one `/api/summary` call + poll loop kept.
|
||||
- `dash.css`: `ul.alerts` rows + verdict colours, `table.data td.warn/.fail`, panel swap via `#overview-panels[data-state="alerting"]` (alerts above fleet when problems exist, all-clear below when empty).
|
||||
- Defect fixed in passing: `Dash.age()` printed "2$ ago" (`replace(/…$/, '$')` — lone `$` is a literal); now "2m ago".
|
||||
- `tests/test_api.py`: 3 new served-HTML tests (two-panel shape, removed ids, alert href/CSS contract); fallback test updated — `/` is converted, 03–05 remain.
|
||||
- Screenshots in `data/logs/` (live data): `overview-alert.png` (driveripper tunnel stopped → one red row naming it, link to `/system`, alerts panel floated above fleet) and `overview-clear.png` (live gateway-only instance, `alerts: []` → "All clear" below fleet). Tunnel restored; all 6 sources `ok`.
|
||||
|
||||
**Gate**: `bash .agents/validate.sh` exit 0 — `uv run pytest --cov --cov-fail-under=94` → **403 passed, 6 skipped, 95.18%**; ruff clean; pyright 0 errors; `node --check` clean. `uv run pytest -m live --live` → 6 passed (unchanged).
|
||||
|
||||
**Docs**: AGENTS.md (severityFor fallback now phases 03–05; UI-conventions paragraph pinning the two-panel overview), README.md (`web/` bullet describes the two panels), `app.js` fallback docstring.
|
||||
|
||||
**Decisions**: all-clear screenshot uses a throwaway 8091 instance (gateway only) because the real UPSes permanently warn — a full live config can never show `[]`; fleet table colours only warn/fail (ok stays plain text) so a healthy table is quiet.
|
||||
|
||||
**Next pending task**: `.agents/phases/todo/02_overview_fleet_and_alerts/03_retire_unused_overview_code.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 35%]
|
||||
...........................................ssssss....................... [ 52%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 88%]
|
||||
................................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 166 0 100%
|
||||
src/dashboards/api.py 193 28 85% 60-61, 63, 294-300, 317-325, 332-336, 420, 425-426, 432-443
|
||||
src/dashboards/app.py 109 7 94% 50-51, 56, 58, 60, 97, 131
|
||||
src/dashboards/collectors.py 140 9 94% 78-79, 92-93, 97-98, 115, 232, 236
|
||||
src/dashboards/config.py 110 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 139 1 99% 226
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 2 99% 122, 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 226
|
||||
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
|
||||
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
|
||||
src/dashboards/store.py 178 3 98% 204, 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2095 101 95%
|
||||
Required test coverage of 94% reached. Total coverage: 95.18%
|
||||
403 passed, 6 skipped, 1 warning in 45.80s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
All green. Report:
|
||||
|
||||
**Implemented**
|
||||
- `index.html`: removed the inert `#config-chip`, `#fleet-note` and the unread `#alerts-panel`/`#fleet-panel` ids; `system.html`: dropped the unread `#f-power` id (element kept, still renders `n/a`); `app.js`: deleted the dead `getElementById('sim-flag')` + `void target` in the clock tick.
|
||||
- `dash.css`: removed `.grid.wide` + its 1150px media query (the only rule the deleted fleet+events row made dead) plus `.cardset`/`.stale-flag` (duplicates of the live `.stack`/`.stale`); `.metric*`, `.events`, `.hero-bar` stay — still used/pinned.
|
||||
- `api.py`: `/api/summary` docstring now states the page renders only `alerts` + `system.fleet`/`hosts`, the rest is programmatic (additive-only).
|
||||
- Tests (`tests/test_api.py`): `test_summary_serves_every_path_another_consumer_reads` pins `system.enabled|fleet|hosts`, `llm.engines`, `ups.ups`, `recent_events`, `alerts` with **and** without targets (mutation-checked: renaming `llm.engines` fails it); `test_no_page_carries_an_id_nothing_reads` (orphan markup tripwire, all four pages); `parse_range` accept/refuse matrix; `/api/system/history`, `/api/llm/snapshot`, `/api/llm/history` contracts; UPS gates — driver-does-not-advertise 400, `load.off`+`DANGER` → **409 while on battery** (via the labelled simulator), and a scripted `nut.NutClient` proving login→INSTCMD→audit event and 502+error-event on refusal.
|
||||
- Docs: README `/` row → “fleet + alerts in one request (`/api/summary`)”; AGENTS.md — `/api/summary` render-vs-programmatic bullet, orphan-id/CSS convention, counts (435 tests, 96.8%), `ScriptedNutClient` note.
|
||||
|
||||
**Gate** — `bash .agents/validate.sh` **exit 0**: `pytest --cov --cov-fail-under=94` → **435 passed, 6 skipped, coverage 96.80%** (`api.py` 100%, `alerts.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean. `uv run pytest -m live --live` → **6 passed** (read-only). `uv pip list --outdated` empty.
|
||||
**Browser** — `data/logs/retire-index.png` (live fleet, 6 alerts, alerts above Fleet, 2 `<section>`s, zero console/request errors) and `data/logs/retire-system.png` (Power still `n/a`). `curl :8090/api/summary` returns every pinned key.
|
||||
**Grep evidence** — `grep -rn "sys-cpu\|llm-kv\|ups-list\|renderEvents" src/dashboards/web/` → *no output*; `grep -rn "cardset\|stale-flag\|grid wide" src/dashboards/web/` → *no output*.
|
||||
|
||||
**Decisions** — Removed 3 pre-existing dead CSS rules that duplicated a live mechanism; left the remaining unused-but-intentional vocabulary (`.hero-bar` is pinned by phase 01's test, `.cols-3`/`.vh`/`.bar.tall`/`table.data.scroll` are components phases 03–05 adopt), and wrote that line into AGENTS.md rather than deleting rework. No live infrastructure was touched (no tunnel/host changed); the on-battery refusal is proven through the labelled simulator only.
|
||||
|
||||
**Next pending task**: `.agents/phases/todo/03_system_per_host_panels/01_host_card_payload_contract.md` (this was phase 02's last task).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 48%]
|
||||
...ssssss............................................................... [ 65%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 97%]
|
||||
......... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 166 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
|
||||
src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115
|
||||
src/dashboards/config.py 110 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 139 1 99% 226
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 2 99% 122, 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 226
|
||||
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
|
||||
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2095 67 97%
|
||||
Required test coverage of 94% reached. Total coverage: 96.80%
|
||||
435 passed, 6 skipped, 1 warning in 44.68s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
@@ -54,8 +54,10 @@ page still renders.
|
||||
panels that matter.
|
||||
4. **One dead source must not blink another.** Each source is its own asyncio task with its
|
||||
own timeout and health record (`poller.py`). A failed poll keeps the last good payload and
|
||||
marks it stale in the UI instead of zeroing it. Transitions write events; repeated failure
|
||||
writes one line, not 500.
|
||||
marks it stale in the UI instead of zeroing it — on `/` too, where a host that has never
|
||||
answered still gets a fleet row (`unreachable: true`, dimmed, every number `n/a`) because
|
||||
dropping the row would turn a dead agent into "never configured". Transitions write
|
||||
events; repeated failure writes one line, not 500.
|
||||
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
|
||||
charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
|
||||
no outbound access — a CDN reference is a request that never comes back.
|
||||
@@ -113,6 +115,8 @@ Responsibilities, one line each:
|
||||
kelvin-vs-celsius normalisation.
|
||||
- `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a
|
||||
battery. It must never be the default, and every payload it produces says `simulated`.
|
||||
Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that
|
||||
ages), so a fresh simulator reads healthy and amber keeps meaning something.
|
||||
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
|
||||
- `rates.py` — Δ/window estimator, keyed by resolved series name.
|
||||
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
|
||||
@@ -123,8 +127,29 @@ Responsibilities, one line each:
|
||||
(charge/load/runtime), `severity` inside rated LLM cells (KV pressure, classified on a
|
||||
percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
|
||||
row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
|
||||
- `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload,
|
||||
recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts`
|
||||
(`{id, severity, source, message, href, at}`, fail first then warn, then by source). A
|
||||
pure projection: no I/O, no store, no thresholds of its own — it reads job states for
|
||||
reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions.
|
||||
Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but
|
||||
older than 3× its own poll interval is `warn` while a source that stopped answering is
|
||||
`fail`; a dead source gets exactly one line (its stale payload adds nothing) that names it
|
||||
once — the poller's `"UpstreamError: "` wrapper and the duplicated host name are stripped,
|
||||
because a message that says `gateway` twice reads as two problems; `unknown` and missing
|
||||
numbers are silence; one alert per (source, metric) and at most 12, the last being a marker
|
||||
that says how many it hid. This is why the overview can be two panels.
|
||||
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
|
||||
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
|
||||
`/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
|
||||
the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `llm.engines`,
|
||||
`ups.ups`, the `states` maps and `recent_events` stay for programmatic consumers (live
|
||||
tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events` are unchanged
|
||||
and `alerts` is additive — `tests/test_api.py` pins every one of those key paths, with and
|
||||
without targets configured, so retiring a panel can never shrink the payload. A configured
|
||||
job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
|
||||
`system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
|
||||
counts the machines there are and `hosts_live` the ones answering.
|
||||
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
|
||||
with an in-memory DB and no targets.
|
||||
- `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the
|
||||
@@ -133,10 +158,12 @@ Responsibilities, one line each:
|
||||
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
|
||||
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
|
||||
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
|
||||
values the API does not rate; converting its remaining call sites is phases 02–05's job.
|
||||
values the API does not rate; converting its remaining call sites is phases 03–05's job
|
||||
(`/` was converted in phase 02: the overview now colours only from `/api/summary` verdicts).
|
||||
|
||||
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
|
||||
`gpu.busy_pct@0`, `llm.decode_tps`, `ups.charge_pct`), `@stream` for per-device series.
|
||||
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
|
||||
series — `/api/metrics` answers with exactly these names.
|
||||
That scheme is the history API's key — treat it as a public interface.
|
||||
|
||||
## The fleet (real environment, verified 2026-10)
|
||||
@@ -178,8 +205,8 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
|
||||
./scripts/dev-server.sh status # per-source ok/DOWN
|
||||
./scripts/dev-server.sh logs
|
||||
|
||||
uv run pytest # hermetic: 356 tests, no network
|
||||
uv run pytest --cov # coverage gate (94.75%; floor 94%)
|
||||
uv run pytest # hermetic: 439 passed + 6 live-skipped, no network
|
||||
uv run pytest --cov # coverage gate (96.8%; floor 94%)
|
||||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||||
uv run ruff check src tests agent && uv run pyright
|
||||
./scripts/capture-fixtures.sh # re-record ground truth
|
||||
@@ -215,18 +242,39 @@ service here.
|
||||
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
|
||||
so it stays in the default run and is *not* marked `live`.
|
||||
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
|
||||
94% (`--cov-fail-under`), and ~95% is where we are; keep it there. New/modified Python
|
||||
94% (`--cov-fail-under`), and ~97% is where we are; keep it there. New/modified Python
|
||||
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
|
||||
never a real target.
|
||||
never a real target, and never a real upstream: `/api/ups/command`'s send is proven against
|
||||
a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live
|
||||
suite is read-only — that fake is the only place the write reaches a "driver".
|
||||
|
||||
## UI conventions
|
||||
|
||||
Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
|
||||
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
|
||||
dots in the header, window chips (1h…90d) on history pages. A panel that cannot get data says
|
||||
dots in the header, window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API
|
||||
accepts 5m…90d). A panel that cannot get data says
|
||||
why in the panel — the operator should never have to open devtools to learn that a source is
|
||||
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
||||
|
||||
The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per
|
||||
`/api/summary` alert: verdict class, source linked to its dashboard via the alert's own
|
||||
`href`, message, age) and **Fleet**. A non-empty list floats the alerts panel above the
|
||||
fleet table (`#overview-panels[data-state]` swaps them); an empty one is an explicit
|
||||
"all clear" state below it, because an empty list is a message, not a blank box. A host that
|
||||
is not answering keeps its row, dimmed with the shared `.stale` class and reading `n/a`, so
|
||||
the table never implies last good numbers are live and never silently loses a machine.
|
||||
No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the
|
||||
two-panel shape against the served HTML.
|
||||
|
||||
Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an
|
||||
`id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for`
|
||||
attribute references — the orphan a deleted renderer leaves behind, caught on served bytes
|
||||
because there is no JS test runner (a locked anchor). Same rule for `dash.css`: delete the
|
||||
rules a removed panel made dead (`.grid.wide` went with the overview's fleet+events row);
|
||||
component vocabulary the pages are still being converted to (`.hero-bar`, the grid and bar
|
||||
variants) stays even while only one page uses it.
|
||||
|
||||
A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
|
||||
muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's
|
||||
`severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*`
|
||||
|
||||
@@ -4,7 +4,7 @@ Three dashboards, one domain, one process, one SQLite file:
|
||||
|
||||
| Page | What it answers | Source |
|
||||
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
||||
| `/` | Is anything wrong, and which dashboard do I need? | all of the below, one request |
|
||||
| `/` | Is anything wrong, and which dashboard do I need? | fleet + alerts in one request (`/api/summary`) |
|
||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
||||
@@ -136,8 +136,23 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
|
||||
colours the same way and a value nobody reported renders `n/a` instead of green. A metric
|
||||
published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays
|
||||
what the upstream published.
|
||||
* `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states,
|
||||
host_payloads, engine_payloads, ups_payload, recent_events)` is a pure projection of the
|
||||
state the poller already keeps plus the verdicts `severity.py` and the UPS health checks
|
||||
already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as
|
||||
`alerts`, worst first: `{id, severity, source, message, href, at}`. A configured source
|
||||
that stopped answering is `fail`; one that is stale but still serving its last good value
|
||||
is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than
|
||||
one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a
|
||||
marker that says how many it hid.
|
||||
* `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No
|
||||
framework, no CDN, no bundler. A big colour-coded number is `Dash.hero(el, {...})` on the
|
||||
framework, no CDN, no bundler. The overview (`/`) is exactly two panels — a
|
||||
**Warnings & errors** list built from `/api/summary`'s `alerts` (each row links to the
|
||||
dashboard that explains it) and the **Fleet** table — and nothing else: an empty alert
|
||||
list reads as an explicit "all clear", and a live problem moves the panel above the
|
||||
table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the
|
||||
fleet never quietly gets smaller and last good values are never mistaken for live ones.
|
||||
A big colour-coded number is `Dash.hero(el, {...})` on the
|
||||
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
|
||||
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
|
||||
with no verdict renders `n/a` with no colour at all.
|
||||
|
||||
@@ -0,0 +1,414 @@
|
||||
"""What is wrong right now, derived once on the server instead of on every page.
|
||||
|
||||
The landing page has one question — *is anything wrong, and where do I look?* — and the
|
||||
answer must be identical in the browser, in a future mobile view, and in whatever
|
||||
notifies someone later. So the answer is a function of the state the poller already
|
||||
keeps and the verdicts `severity.py` / `sources/nut.assess()` already produced, and this
|
||||
module is the only place that assembles it.
|
||||
|
||||
It is a pure projection by design: no I/O, no store, no HTTP, no clock it cannot be given.
|
||||
That is what makes every rule below unit-testable without a hub, and what keeps a page
|
||||
from growing its own opinion about what red means.
|
||||
|
||||
The rules, and the reason each exists:
|
||||
|
||||
* **Never alert for an unconfigured source.** `AGENTS.md` invariant 3: a server the
|
||||
operator never installed must not raise a red flag, because that destroys the signal
|
||||
from the panels that matter. `SourceState.configured` is the gate.
|
||||
* **Stale is not dead.** A source that stopped answering *and* has no last good value is
|
||||
`fail`. A source that is still serving its last good value but has not refreshed in
|
||||
three of its own poll intervals is `warn` — saying "dead" while the page shows numbers
|
||||
would be a lie. Per-source intervals because system polls at 10s and UPS at 5s; one
|
||||
global number would be either noise or too late.
|
||||
* **Verdicts come from elsewhere.** Host/LLM breaches read the `severity` dicts phase 01
|
||||
added; UPS breaches read `health.checks[].status`. Nothing here compares a number to a
|
||||
threshold of its own — the message *quotes* the bound from `severity.THRESHOLDS` so the
|
||||
operator can see which line was crossed, and the numbers still live in one file.
|
||||
* **A dead source is one alert.** Its last good payload is still in the hub, so its stale
|
||||
threshold breaches would pile three more amber lines on top of the red one that already
|
||||
names it. Payload-derived alerts are skipped for a source that is not answering.
|
||||
* **Bounded and deduped.** One alert per (source, metric) — the `id` is that pair — and at
|
||||
most :data:`MAX_ALERTS` of them, the last being a marker that says how many it hid. A
|
||||
40-disk fleet must not turn the landing page into a log tail.
|
||||
* **Sorted fail → warn, then by source**, so the thing that needs action is at the top
|
||||
without the operator scanning.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import time
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from .config import Settings
|
||||
from .severity import INVERTED, THRESHOLDS
|
||||
from .sources.llm import SEVERITY_CELLS
|
||||
|
||||
#: The most recent verdicts the landing page shows. Beyond this it is a wall of text and
|
||||
#: the question it answers becomes "read this list", which is a different page.
|
||||
MAX_ALERTS = 12
|
||||
|
||||
FAIL = "fail"
|
||||
WARN = "warn"
|
||||
|
||||
#: Only these two are alerts. `ok` is not news, and `None`/`unknown` means nothing honest
|
||||
#: could be said — `AGENTS.md` rule 2, no measurement, no verdict, no colour.
|
||||
_RANK = {FAIL: 0, WARN: 1}
|
||||
|
||||
#: Where each dashboard lives, so every alert can say where to look.
|
||||
HREF = {"system": "/system", "llm": "/llm", "ups": "/ups"}
|
||||
DEFAULT_HREF = "/"
|
||||
|
||||
#: A source whose last good value is older than this many of its own poll intervals is
|
||||
#: serving data the operator should not act on. It is still on screen, hence amber.
|
||||
STALE_FACTOR = 3.0
|
||||
|
||||
#: How to say a metric out loud: label, unit, decimals. Keyed by the dotted metric name
|
||||
#: from `severity.THRESHOLDS`; an unlisted metric prints its own name rather than nothing.
|
||||
_DISPLAY: dict[str, tuple[str, str, int]] = {
|
||||
"cpu.usage_pct": ("CPU", "%", 0),
|
||||
"cpu.temp_c": ("CPU temperature", "°C", 0),
|
||||
"mem.used_pct": ("memory", "%", 0),
|
||||
"disk.used_pct": ("disk", "%", 0),
|
||||
"gpu.busy_pct": ("GPU utilisation", "%", 0),
|
||||
"gpu.mem_used_pct": ("GPU memory", "%", 0),
|
||||
"llm.kv_cache_usage_pct": ("KV cache", "%", 0),
|
||||
}
|
||||
|
||||
#: host payload section -> (field, severity metric). Mirrors the `severity` sub-dicts that
|
||||
#: `sources/host.py` attaches, so a new rated field is one row here.
|
||||
_HOST_CELLS: tuple[tuple[str, str, str], ...] = (
|
||||
("cpu", "usage_pct", "cpu.usage_pct"),
|
||||
("cpu", "temp_c", "cpu.temp_c"),
|
||||
("memory", "used_pct", "mem.used_pct"),
|
||||
("disks", "worst_used_pct", "disk.used_pct"),
|
||||
)
|
||||
_GPU_CELLS: tuple[tuple[str, str], ...] = (
|
||||
("busy_pct", "gpu.busy_pct"),
|
||||
("mem_used_pct", "gpu.mem_used_pct"),
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# formatting helpers (words, not thresholds)
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _value(metric: str, number: float, scale: float = 1.0) -> str:
|
||||
"""`CPU 92%` — the metric said out loud, with the value the upstream published.
|
||||
|
||||
Callers only get here with a real number (no measurement, no alert), so there is no
|
||||
`n/a` case. ``scale`` is a unit for the *display* of a ratio the threshold table keeps
|
||||
in percent (`llamacpp:kv_cache_usage_ratio` is 0-1); it is the same conversion
|
||||
`with_severity()` applies to the verdict, so the number and the colour cannot disagree.
|
||||
"""
|
||||
label, unit, digits = _DISPLAY.get(metric, (metric, "", 1))
|
||||
return f"{label} {number * scale:.{digits}f}{unit}"
|
||||
|
||||
|
||||
def _trip(metric: str, verdict: Any) -> str:
|
||||
"""` (fail ≥90)` — which bound was crossed, quoted from `severity.THRESHOLDS`.
|
||||
|
||||
The threshold still lives in one place; this only reads it back so the alert is
|
||||
actionable without opening the source. Direction comes from `severity.INVERTED`, so a
|
||||
low battery charge prints `≤` instead of an arithmetic-looking `≥20`.
|
||||
"""
|
||||
if verdict not in _RANK:
|
||||
return ""
|
||||
base = metric.split("@", 1)[0]
|
||||
bounds = THRESHOLDS.get(base)
|
||||
if bounds is None:
|
||||
return ""
|
||||
warn, danger = bounds
|
||||
low_is_worse = base in INVERTED
|
||||
chosen = danger if verdict == FAIL else warn
|
||||
return f" ({verdict} {'≤' if low_is_worse else '≥'}{chosen:g})"
|
||||
|
||||
|
||||
#: `poller.Job` records a failure as `"<ExceptionClass>: <message>"`, and collectors build
|
||||
#: their message as `"<name>: <what happened>"`. Left alone, a dead host prints
|
||||
#: `gateway: UpstreamError: gateway: http://… unreachable` — the name twice and the one
|
||||
#: exception class this codebase guarantees everything else raises (`AGENTS.md`: collectors
|
||||
#: surface a single `UpstreamError` type, so its name is never information). Other class
|
||||
#: names stay: an unexpected `ValueError` is a bug worth reading about. The full string
|
||||
#: survives in `/api/health` and the event feed.
|
||||
_POLLER_CLASS = re.compile(r"^UpstreamError: ")
|
||||
|
||||
|
||||
def _clean_error(label: str, error: str) -> str:
|
||||
"""`http://h:9121/snapshot unreachable (ConnectError)` from the poller's wrapped string."""
|
||||
text = error.strip()
|
||||
while True:
|
||||
stripped = _POLLER_CLASS.sub("", text, count=1).strip()
|
||||
if stripped.startswith(f"{label}: "):
|
||||
stripped = stripped[len(label) + 2:].strip()
|
||||
if stripped == text:
|
||||
return text
|
||||
text = stripped
|
||||
|
||||
|
||||
def _sections(degraded: Any) -> str:
|
||||
"""The agent's `errors` map (or list) as words: `temperatures (no hwmon sensors)`."""
|
||||
if isinstance(degraded, Mapping):
|
||||
return ", ".join(f"{name} ({detail})" if detail else str(name)
|
||||
for name, detail in degraded.items()) or "nothing"
|
||||
if isinstance(degraded, (list, tuple)):
|
||||
return ", ".join(str(item) for item in degraded) or "nothing"
|
||||
return str(degraded)
|
||||
|
||||
|
||||
def _interval(settings: Settings, dashboard: str) -> float:
|
||||
"""The source's own poll interval — the unit staleness is measured in."""
|
||||
if dashboard == "llm":
|
||||
return float(settings.llm_interval)
|
||||
if dashboard == "ups":
|
||||
return float(settings.nut_interval)
|
||||
if dashboard == "system":
|
||||
return float(settings.host_interval)
|
||||
return 60.0
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# the sink: dedupe, bound, sort
|
||||
# --------------------------------------------------------------------------- #
|
||||
class _Sink:
|
||||
"""Collects alerts keyed by (source, metric), keeping the worst verdict per key."""
|
||||
|
||||
def __init__(self, now: float) -> None:
|
||||
self.now = now
|
||||
self.items: dict[str, dict[str, Any]] = {}
|
||||
|
||||
def add(self, verdict: Any, source: str, key: str, message: str, href: str,
|
||||
at: float | None = None) -> None:
|
||||
"""Record one verdict — unless it is `ok`, `unknown`, or `None`, which are not alerts."""
|
||||
if verdict not in _RANK:
|
||||
return
|
||||
alert_id = f"{source}:{key}"
|
||||
existing = self.items.get(alert_id)
|
||||
if existing is not None:
|
||||
# same source + same metric twice (two UPS cards answering as one name, a
|
||||
# payload re-projected after a state change): one line, and the worse verdict
|
||||
if _RANK[str(verdict)] < _RANK[existing["severity"]]:
|
||||
existing["severity"] = str(verdict)
|
||||
existing["message"] = message
|
||||
existing["at"] = self.now if at is None else at
|
||||
return
|
||||
self.items[alert_id] = {
|
||||
"id": alert_id,
|
||||
"severity": str(verdict),
|
||||
"source": source,
|
||||
"message": message,
|
||||
"href": href,
|
||||
# when this became true, best effort: the last good poll, the outage event, or
|
||||
# now. The page prints it as "since", so an alert that resets its clock every
|
||||
# refresh is an alert whose duration nobody can read.
|
||||
"at": self.now if at is None else at,
|
||||
}
|
||||
|
||||
def result(self) -> list[dict[str, Any]]:
|
||||
ordered = sorted(self.items.values(),
|
||||
key=lambda item: (_RANK.get(item["severity"], 9), item["source"],
|
||||
item["id"]))
|
||||
if len(ordered) <= MAX_ALERTS:
|
||||
return ordered
|
||||
visible, hidden = ordered[:MAX_ALERTS - 1], ordered[MAX_ALERTS - 1:]
|
||||
failing = sum(1 for item in hidden if item["severity"] == FAIL)
|
||||
visible.append({
|
||||
"id": "alerts:suppressed",
|
||||
# the marker inherits the worst thing it hid: a list that ends in amber while
|
||||
# three reds are behind it would be a lie of the same shape as hiding them
|
||||
"severity": hidden[0]["severity"],
|
||||
"source": "alerts",
|
||||
"message": (f"{len(hidden)} more not shown "
|
||||
f"({failing} failing, {len(hidden) - failing} warning)"),
|
||||
"href": DEFAULT_HREF,
|
||||
"at": self.now,
|
||||
})
|
||||
return visible
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# the rules
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _state_for(states: Sequence[Mapping[str, Any]], name: str) -> Mapping[str, Any] | None:
|
||||
"""The health record for the job that produced `name` (a host or engine label)."""
|
||||
for state in states:
|
||||
if state.get("label") == name or str(state.get("key") or "").endswith(f":{name}"):
|
||||
return state
|
||||
return None
|
||||
|
||||
|
||||
def _answering(states: Sequence[Mapping[str, Any]], name: str) -> bool:
|
||||
"""Is that source currently answering? Unknown (no state at all) counts as yes.
|
||||
|
||||
A payload that exists is data; inventing a reachability verdict for a source with no
|
||||
health record would be this module's opinion, not the poller's.
|
||||
"""
|
||||
state = _state_for(states, name)
|
||||
return True if state is None else bool(state.get("ok"))
|
||||
|
||||
|
||||
def _dashboard_answering(states: Sequence[Mapping[str, Any]], dashboard: str) -> bool:
|
||||
mine = [state for state in states if state.get("dashboard") == dashboard]
|
||||
return any(bool(state.get("ok")) for state in mine) if mine else True
|
||||
|
||||
|
||||
def _outage_started(events: Sequence[Mapping[str, Any]], dashboard: str,
|
||||
label: str) -> float | None:
|
||||
"""When the poller first said this source was down, from its own transition event.
|
||||
|
||||
Best effort by construction: `recent_events` is the short list the summary already
|
||||
carries, so an outage older than that reads as `at: now`. The alert itself never
|
||||
depends on the event being there.
|
||||
"""
|
||||
matches = [float(event["ts"]) for event in events
|
||||
if event.get("kind") == "unreachable" and event.get("host") == label
|
||||
and (event.get("source") == dashboard or event.get("source") is None)
|
||||
and isinstance(event.get("ts"), (int, float))]
|
||||
return max(matches) if matches else None
|
||||
|
||||
|
||||
def _source_alerts(sink: _Sink, settings: Settings, states: Sequence[Mapping[str, Any]],
|
||||
events: Sequence[Mapping[str, Any]]) -> None:
|
||||
for state in states:
|
||||
if not state.get("configured"):
|
||||
continue # invariant 3: never red for a source nobody installed
|
||||
label = str(state.get("label") or state.get("key") or "source")
|
||||
dashboard = str(state.get("dashboard") or "")
|
||||
href = HREF.get(dashboard, DEFAULT_HREF)
|
||||
key = str(state.get("key") or label)
|
||||
if not state.get("ok"):
|
||||
# the alert starts with the name, so the name inside the message is removed —
|
||||
# "driveripper: driveripper: ..." reads like two problems instead of one
|
||||
error = _clean_error(label, str(state.get("error") or "")) or "is not answering"
|
||||
sink.add(FAIL, label, f"{key}:down", f"{label}: {error}", href,
|
||||
_outage_started(events, dashboard, label) or sink.now)
|
||||
continue
|
||||
age = state.get("age_seconds")
|
||||
interval = _interval(settings, dashboard)
|
||||
if isinstance(age, (int, float)) and age > interval * STALE_FACTOR:
|
||||
sink.add(WARN, label, f"{key}:stale",
|
||||
f"{label} last answered {age:.0f}s ago (every {interval:g}s): the page "
|
||||
f"is showing the last good value, not a live one",
|
||||
href, sink.now - age)
|
||||
|
||||
|
||||
def _host_alerts(sink: _Sink, host: Mapping[str, Any],
|
||||
states: Sequence[Mapping[str, Any]]) -> None:
|
||||
name = str(host.get("name") or "host")
|
||||
if host.get("unreachable") or not _answering(states, name):
|
||||
return # the job-level alert already names this host; one line, not four
|
||||
at = host.get("ts") if isinstance(host.get("ts"), (int, float)) else None
|
||||
for section, field, metric in _HOST_CELLS:
|
||||
block = host.get(section) or {}
|
||||
number = block.get(field)
|
||||
verdict = (block.get("severity") or {}).get(field)
|
||||
# no number, no alert: `n/a` is not a breach (and the verdict is None anyway)
|
||||
if number is None or verdict not in _RANK:
|
||||
continue
|
||||
mount = block.get("worst_mount")
|
||||
where = f" on {mount}" if mount else ""
|
||||
sink.add(verdict, name, metric,
|
||||
f"{name}: {_value(metric, number)}{where}{_trip(metric, verdict)}",
|
||||
HREF["system"], at)
|
||||
for gpu in host.get("gpus") or []:
|
||||
card = str(gpu.get("model") or f"GPU {gpu.get('index', 0)}")
|
||||
for field, metric in _GPU_CELLS:
|
||||
number = gpu.get(field)
|
||||
verdict = (gpu.get("severity") or {}).get(field)
|
||||
if number is None or verdict not in _RANK:
|
||||
continue
|
||||
sink.add(verdict, name, f"{metric}@{gpu.get('index', 0)}",
|
||||
f"{name}: {_value(metric, number)} on {card}{_trip(metric, verdict)}",
|
||||
HREF["system"], at)
|
||||
degraded = host.get("degraded")
|
||||
if degraded:
|
||||
# a /proc read failed on the box: the numbers that did come back are real, but the
|
||||
# snapshot is not the whole picture and the page must not imply that it is
|
||||
sink.add(WARN, name, "agent.sections",
|
||||
f"{name}: the agent could not read {_sections(degraded)}", HREF["system"], at)
|
||||
|
||||
|
||||
def _engine_alerts(sink: _Sink, engine: Mapping[str, Any],
|
||||
states: Sequence[Mapping[str, Any]]) -> None:
|
||||
name = str(engine.get("engine") or "engine")
|
||||
if engine.get("unreachable") or not _answering(states, name):
|
||||
return
|
||||
at = engine.get("scraped_at") if isinstance(engine.get("scraped_at"), (int, float)) else None
|
||||
metrics = engine.get("metrics") or {}
|
||||
# the rated cells are declared next to the field maps they come from (`SEVERITY_CELLS`),
|
||||
# so a new one is rated by severity.py and alerted here without touching this loop
|
||||
for key, (metric, scale) in SEVERITY_CELLS.items():
|
||||
cell = metrics.get(key)
|
||||
if not isinstance(cell, Mapping):
|
||||
continue
|
||||
value, verdict = cell.get("value"), cell.get("severity")
|
||||
# no number in the cell, no claim (rule 2); no verdict, no colour to justify one
|
||||
if not isinstance(value, (int, float)) or verdict not in _RANK:
|
||||
continue
|
||||
sink.add(verdict, name, f"{metric}@{key}",
|
||||
f"{name}: {_value(metric, float(value), scale)}{_trip(metric, verdict)}",
|
||||
HREF["llm"], at)
|
||||
|
||||
|
||||
def _ups_alerts(sink: _Sink, payload: Mapping[str, Any],
|
||||
states: Sequence[Mapping[str, Any]]) -> None:
|
||||
if payload.get("simulated"):
|
||||
# `AGENTS.md`: the simulator must never be silent about what it is. The UPS page
|
||||
# badges it; the landing page has to say it too, or a modelled battery reads as a
|
||||
# healthy real one from two clicks away.
|
||||
sink.add(WARN, "ups", "simulated",
|
||||
"UPS numbers are simulated — NUT_SIMULATE=1 is set, so these are modelled "
|
||||
"figures and not your batteries", HREF["ups"])
|
||||
if not _dashboard_answering(states, "ups"):
|
||||
return
|
||||
for card in payload.get("ups") or []:
|
||||
name = str(card.get("name") or "ups")
|
||||
at = card.get("ts") if isinstance(card.get("ts"), (int, float)) else None
|
||||
for check in (card.get("health") or {}).get("checks") or []:
|
||||
label = str(check.get("name") or "check")
|
||||
detail = str(check.get("detail") or "").strip()
|
||||
# `unknown` stays out: a driver that publishes no battery temperature is not a
|
||||
# fault, and the UPS page already says "not reported by this driver"
|
||||
sink.add(check.get("status"), name, f"health:{label}",
|
||||
f"{label}: {detail}" if detail else label, HREF["ups"], at)
|
||||
|
||||
|
||||
def _config_alerts(sink: _Sink, settings: Settings) -> None:
|
||||
for problem in settings.host_target_problems:
|
||||
# a typo'd entry is not a dead host: the operator needs to know it is not being
|
||||
# polled at all, which is otherwise invisible (the host simply is not in the list)
|
||||
sink.add(WARN, "config", f"HOST_TARGETS:{problem}",
|
||||
f"HOST_TARGETS: {problem} — that entry is not being polled", HREF["system"])
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# entry point
|
||||
# --------------------------------------------------------------------------- #
|
||||
def build(settings: Settings, hub_states: Sequence[Mapping[str, Any]] = (),
|
||||
host_payloads: Sequence[Mapping[str, Any]] = (),
|
||||
engine_payloads: Sequence[Mapping[str, Any]] = (),
|
||||
ups_payload: Mapping[str, Any] | None = None,
|
||||
recent_events: Sequence[Mapping[str, Any]] = (), *,
|
||||
now: float | None = None) -> list[dict[str, Any]]:
|
||||
"""Everything that is wrong right now, worst first.
|
||||
|
||||
Each alert is `{"id", "severity", "source", "message", "href", "at"}`; `id` is
|
||||
`"<source>:<metric>"` and is stable across refreshes so a page can key DOM off it, and
|
||||
`at` is a unix timestamp for when the condition was first observed (best effort).
|
||||
Empty means all clear — there is no `ok` alert, because an empty list is the clearest
|
||||
possible answer to "what is wrong".
|
||||
|
||||
Inputs are the same objects `/api/summary` already assembles (job states, last good
|
||||
payloads, the UPS payload, the short event list), so answering costs nothing extra and
|
||||
the answer cannot disagree with the panels underneath it.
|
||||
"""
|
||||
sink = _Sink(time.time() if now is None else now)
|
||||
_source_alerts(sink, settings, hub_states, recent_events)
|
||||
for host in host_payloads:
|
||||
_host_alerts(sink, host, hub_states)
|
||||
for engine in engine_payloads:
|
||||
_engine_alerts(sink, engine, hub_states)
|
||||
if ups_payload is not None:
|
||||
_ups_alerts(sink, ups_payload, hub_states)
|
||||
_config_alerts(sink, settings)
|
||||
return sink.result()
|
||||
+47
-14
@@ -18,6 +18,7 @@ from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Body, Depends, HTTPException, Query, Request
|
||||
|
||||
from . import alerts as alert_rules
|
||||
from .collectors import NutMonitor
|
||||
from .config import Settings
|
||||
from .poller import Hub
|
||||
@@ -190,11 +191,30 @@ def common_router() -> APIRouter:
|
||||
|
||||
@router.get("/summary")
|
||||
async def summary(ctx: AppContext = Depends(get_context)) -> dict[str, Any]:
|
||||
"""One call for the landing page: is each dashboard alive, and what stands out."""
|
||||
hosts = [job.state.payload for job in ctx.hub.jobs.values()
|
||||
if job.dashboard == "system" and job.state.payload]
|
||||
engines = [job.state.payload for job in ctx.hub.jobs.values()
|
||||
if job.dashboard == "llm" and job.state.payload]
|
||||
"""One call for the landing page: is anything wrong, and what is the fleet doing.
|
||||
|
||||
What `/` renders is `alerts` plus `system.fleet`/`system.hosts` and nothing else.
|
||||
Everything else here is for **programmatic use**: this is the cheapest whole-fleet
|
||||
probe that exists, so the live tests, an external check and a notifier later read
|
||||
`llm.engines`, `ups.ups` and the `states`/`recent_events` no panel paints any more.
|
||||
Keys are additive-only — `alerts` was added, nothing was renamed, dropped, or given
|
||||
a new meaning. `tests/test_api.py` pins each path below against a future cleanup.
|
||||
"""
|
||||
# A configured job that has never answered gets the same stub `/api/*/snapshot`
|
||||
# hands out: a host in `HOST_TARGETS` with no payload yet is still in the fleet, and
|
||||
# dropping its row would turn "dead agent" into "never configured" (invariants 3
|
||||
# and 4) while quietly shrinking `fleet.hosts_total` to the answering subset.
|
||||
hosts: list[dict[str, Any]] = [
|
||||
job.state.payload or {"name": job.label, "ts": job.state.last_attempt,
|
||||
"unreachable": True}
|
||||
for job in ctx.hub.jobs.values() if job.dashboard == "system"
|
||||
]
|
||||
engines: list[dict[str, Any]] = [
|
||||
job.state.payload or {"engine": job.label, "metrics": {},
|
||||
"sessions": {"active": None, "list": []},
|
||||
"unreachable": True}
|
||||
for job in ctx.hub.jobs.values() if job.dashboard == "llm"
|
||||
]
|
||||
ups_payload = None
|
||||
nut_job = ctx.hub.jobs.get("nut")
|
||||
if nut_job and nut_job.state.payload:
|
||||
@@ -202,28 +222,40 @@ def common_router() -> APIRouter:
|
||||
fleet = fleet_summary(hosts) if hosts else None
|
||||
if fleet is not None:
|
||||
fleet["severity"] = _fleet_severity(fleet)
|
||||
# one list of events, reused twice: the feed itself, and the timestamp for a source
|
||||
# that is currently failing (alerts.py reads `kind: unreachable` out of it)
|
||||
recent_events = [event.as_json() for event in ctx.store.recent_events(12)]
|
||||
# "what is wrong right now" is a server-side answer (see alerts.py) so the overview,
|
||||
# a mobile view, and a notifier later all agree on the same verdicts
|
||||
summary_alerts = alert_rules.build(ctx.settings, ctx.hub.states(), hosts, engines,
|
||||
ups_payload, recent_events)
|
||||
return {
|
||||
"ts": time.time(),
|
||||
"system": {"enabled": bool(ctx.settings.host_targets),
|
||||
"fleet": fleet,
|
||||
"hosts": [{"name": h["name"], "os": (h.get("info") or {}).get("os_pretty"),
|
||||
"cpu_usage_pct": h["cpu"]["usage_pct"],
|
||||
"load1": h["cpu"].get("load1"),
|
||||
# every read is `.get()` because a stub has none of these sections —
|
||||
# an unreachable agent renders as `n/a` (rule 2), never as a 500
|
||||
"hosts": [{"name": h["name"],
|
||||
"os": (h.get("info") or {}).get("os_pretty"),
|
||||
"cpu_usage_pct": (h.get("cpu") or {}).get("usage_pct"),
|
||||
"load1": (h.get("cpu") or {}).get("load1"),
|
||||
"cores": (h.get("info") or {}).get("cpu_cores_logical"),
|
||||
"mem_used_pct": h["memory"]["used_pct"],
|
||||
"disk_worst_used_pct": h["disks"]["worst_used_pct"],
|
||||
"mem_used_pct": (h.get("memory") or {}).get("used_pct"),
|
||||
"disk_worst_used_pct": (h.get("disks") or {}).get("worst_used_pct"),
|
||||
"gpu_busy_pct": [g.get("busy_pct") for g in h.get("gpus") or []],
|
||||
"temp_c": h["cpu"].get("temp_c"),
|
||||
"temp_c": (h.get("cpu") or {}).get("temp_c"),
|
||||
"uptime_seconds": (h.get("info") or {}).get("uptime_seconds"),
|
||||
"unreachable": bool(h.get("unreachable")),
|
||||
"severity": _host_severity(h)}
|
||||
for h in hosts],
|
||||
"states": ctx.hub.states("system")},
|
||||
"llm": {"enabled": bool(ctx.settings.gufo_url or ctx.settings.vllm_url),
|
||||
"fleet": fleet_totals(engines) if engines else None,
|
||||
"engines": [{"engine": e["engine"],
|
||||
"decode_tokens_per_second": e["metrics"].get(
|
||||
"decode_tokens_per_second": (e.get("metrics") or {}).get(
|
||||
"decode_tokens_per_second", {}).get("value"),
|
||||
"active": e["sessions"].get("active")}
|
||||
"active": (e.get("sessions") or {}).get("active"),
|
||||
"unreachable": bool(e.get("unreachable"))}
|
||||
for e in engines],
|
||||
"states": ctx.hub.states("llm")},
|
||||
"ups": {"enabled": ctx.settings.nut_enabled,
|
||||
@@ -236,7 +268,8 @@ def common_router() -> APIRouter:
|
||||
"severity": _ups_severity(card)}
|
||||
for card in (ups_payload or {}).get("ups", [])],
|
||||
"states": ctx.hub.states("ups")},
|
||||
"recent_events": [event.as_json() for event in ctx.store.recent_events(12)],
|
||||
"recent_events": recent_events,
|
||||
"alerts": summary_alerts,
|
||||
}
|
||||
|
||||
return router
|
||||
|
||||
@@ -23,6 +23,18 @@ from typing import Any
|
||||
from .nut import UpsReading
|
||||
|
||||
|
||||
def _mfr_date(years_ago: float) -> str:
|
||||
"""A battery manufacture date `years_ago` years back, computed instead of hard-coded.
|
||||
|
||||
A fixed date is fine for a year or two and then quietly turns every development page
|
||||
amber forever (`Battery age: 7.0 years — past the 4-year replacement window`), which is
|
||||
the same failure `assess()` refuses for a UPS that floats at 90 %: a warning nobody can
|
||||
clear teaches people to ignore the warnings that matter. The simulator stands in for a
|
||||
working UPS, so its pack stays a fixed age behind today.
|
||||
"""
|
||||
return time.strftime("%Y-%m-%d", time.gmtime(time.time() - years_ago * 365.25 * 86_400))
|
||||
|
||||
|
||||
@dataclass
|
||||
class SimUps:
|
||||
"""One fake UPS: nameplate values plus the state a driver would report."""
|
||||
@@ -180,10 +192,10 @@ class NutSimulator:
|
||||
self.ups = [
|
||||
SimUps("cyberpower", "PR1500LCDRT2U", "Cyber Power Systems", "CPS-04213",
|
||||
nominal_va=1500.0, nominal_w=900.0, load_w=210.0, capacity_ah=17.0,
|
||||
mfr_date="2022-04-15"),
|
||||
mfr_date=_mfr_date(2.0)),
|
||||
SimUps("apc", "Smart-UPS 1500", "APC", "AS-11902",
|
||||
nominal_va=1500.0, nominal_w=865.0, load_w=140.0, capacity_ah=8.0,
|
||||
mfr_date="2019-11-02", driver="usbhid-ups",
|
||||
mfr_date=_mfr_date(3.0), driver="usbhid-ups",
|
||||
test_result="PASSED", test_date="2026-06-14"),
|
||||
]
|
||||
self._last = time.time()
|
||||
|
||||
@@ -30,70 +30,29 @@
|
||||
<div class="sub" id="summary-line">reading fleet…</div>
|
||||
</div>
|
||||
<div class="spacer"></div>
|
||||
<span class="chip" id="config-chip" hidden>loading config</span>
|
||||
</div>
|
||||
|
||||
<div id="banners" class="stack" style="margin-bottom:.85rem"></div>
|
||||
|
||||
<section class="grid auto" aria-label="Dashboards">
|
||||
<article class="card">
|
||||
<header><h2>System resources</h2><span class="spacer"></span><span class="dot idle" id="dot-system"></span></header>
|
||||
<!-- Exactly two panels: what is wrong, and what the machines are doing. Their order is
|
||||
the page's only trick: a problem moves alerts above the fleet table, an empty list
|
||||
leaves the morning calm. `data-state` is set by page-index.js from /api/summary. -->
|
||||
<div class="stack" id="overview-panels" data-state="clear">
|
||||
<section class="card panel-alerts" aria-labelledby="alerts-title">
|
||||
<header><h2 id="alerts-title">Warnings & errors</h2><span class="spacer"></span>
|
||||
<span class="faint mono" id="alerts-count"></span></header>
|
||||
<div class="body">
|
||||
<div class="metrics">
|
||||
<div class="metric"><div class="label">Hosts answering</div>
|
||||
<div class="value"><span id="sys-hosts-live">n/a</span><span class="unit" id="sys-hosts-total"></span></div></div>
|
||||
<div class="metric"><div class="label">CPU average</div>
|
||||
<div class="value"><span id="sys-cpu">n/a</span><span class="unit">%</span></div>
|
||||
<div class="bar" style="margin-top:.3rem"><i></i></div></div>
|
||||
<div class="metric"><div class="label">Memory</div>
|
||||
<div class="value"><span id="sys-mem">n/a</span><span class="unit" id="sys-mem-unit"></span></div>
|
||||
<div class="bar" style="margin-top:.3rem"><i></i></div></div>
|
||||
<div class="metric"><div class="label">Worst disk</div>
|
||||
<div class="value"><span id="sys-disk">n/a</span><span class="unit">%</span></div>
|
||||
<div class="bar" style="margin-top:.3rem"><i></i></div></div>
|
||||
<div class="metric"><div class="label">GPUs</div>
|
||||
<div class="value sm" id="sys-gpus">n/a</div></div>
|
||||
<ul class="alerts" id="alerts" aria-live="polite"></ul>
|
||||
<div class="empty-state" id="alerts-clear" hidden>
|
||||
<h3>All clear</h3>
|
||||
<p>Nothing is failing and no threshold is breached right now. Sources the operator
|
||||
never configured are not problems, so they are not here either.</p>
|
||||
</div>
|
||||
<div class="gap" style="margin-top:.7rem"><a class="action" href="/system" style="text-decoration:none">Open system dashboard →</a></div>
|
||||
</div>
|
||||
</article>
|
||||
|
||||
<article class="card">
|
||||
<header><h2>LLM engines</h2><span class="spacer"></span><span class="dot idle" id="dot-llm"></span></header>
|
||||
<div class="body">
|
||||
<div class="metrics">
|
||||
<div class="metric"><div class="label">Engines answering</div>
|
||||
<div class="value"><span id="llm-live">n/a</span><span class="unit" id="llm-total"></span></div></div>
|
||||
<div class="metric"><div class="label">Decode</div>
|
||||
<div class="value"><span id="llm-decode">n/a</span><span class="unit">tok/s</span></div></div>
|
||||
<div class="metric"><div class="label">Prompt</div>
|
||||
<div class="value"><span id="llm-prompt">n/a</span><span class="unit">tok/s</span></div></div>
|
||||
<div class="metric"><div class="label">Active sessions</div>
|
||||
<div class="value" id="llm-active">n/a</div></div>
|
||||
<div class="metric"><div class="label">Queued</div>
|
||||
<div class="value" id="llm-waiting">n/a</div></div>
|
||||
<div class="metric"><div class="label" title="highest KV-cache use across engines right now">KV cache now</div>
|
||||
<div class="value"><span id="llm-kv">n/a</span><span class="unit">%</span></div>
|
||||
<div class="bar" style="margin-top:.3rem"><i></i></div></div>
|
||||
</div>
|
||||
<div class="gap" style="margin-top:.7rem"><a class="action" href="/llm" style="text-decoration:none">Open LLM dashboard →</a></div>
|
||||
</div>
|
||||
</article>
|
||||
|
||||
<article class="card">
|
||||
<header><h2>UPS / battery</h2><span class="spacer"></span><span class="dot idle" id="dot-ups"></span></header>
|
||||
<div class="body">
|
||||
<div id="ups-list" class="stack">
|
||||
<div class="muted">loading…</div>
|
||||
</div>
|
||||
<div class="gap" style="margin-top:.7rem"><a class="action" href="/ups" style="text-decoration:none">Open UPS dashboard →</a></div>
|
||||
</div>
|
||||
</article>
|
||||
</section>
|
||||
|
||||
<section class="grid wide" style="margin-top:.85rem">
|
||||
<article class="card">
|
||||
<header><h2>Fleet</h2><span class="spacer"></span><span class="faint mono" id="fleet-note"></span></header>
|
||||
<section class="card panel-fleet" aria-labelledby="fleet-title">
|
||||
<header><h2 id="fleet-title">Fleet</h2></header>
|
||||
<div class="body tight">
|
||||
<table class="data" id="fleet-table">
|
||||
<thead><tr><th>Host</th><th>CPU</th><th>Load 1</th><th>Mem used</th><th>Worst disk</th><th>GPU busy</th><th>Temp</th><th>Up</th></tr></thead>
|
||||
@@ -105,14 +64,8 @@
|
||||
<code>HOST_TARGETS</code>.</p>
|
||||
</div>
|
||||
</div>
|
||||
</article>
|
||||
|
||||
<article class="card">
|
||||
<header><h2>Recent events</h2><span class="spacer"></span>
|
||||
<span class="faint">transitions only</span></header>
|
||||
<div class="body"><ul class="events" id="events"><li class="muted">loading…</li></ul></div>
|
||||
</article>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<footer class="pagefoot">
|
||||
<span id="foot-versions"></span>
|
||||
|
||||
@@ -57,7 +57,7 @@
|
||||
<div class="metric"><div class="label">GPU busy (avg)</div>
|
||||
<div class="value"><span id="f-gpu">n/a</span><span class="unit">%</span></div></div>
|
||||
<div class="metric"><div class="label">Power</div>
|
||||
<div class="value sm" id="f-power" title="No whole-host power source is wired up yet; this stays n/a rather than guessing.">n/a</div></div>
|
||||
<div class="value sm" title="No whole-host power source is wired up yet; no script writes this, so it stays n/a rather than guessing.">n/a</div></div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -62,7 +62,10 @@ const Dash = (() => {
|
||||
if (ts == null) return 'never';
|
||||
const seconds = Math.max(0, Date.now() / 1000 - ts);
|
||||
if (seconds < 2) return 'now';
|
||||
return duration(seconds).replace(/(h|m|s|d|y)$/, '$') + ' ago';
|
||||
// `duration()` already carries the unit: "2m ago", "5h ago". (It used to be stripped
|
||||
// with `replace(/(h|m|s|d|y)$/, '$')` — a lone `$` is a literal in a JS replacement,
|
||||
// so every age printed "2$ ago". The unit is what makes the number mean anything.)
|
||||
return `${duration(seconds)} ago`;
|
||||
}
|
||||
|
||||
const naText = 'n/a';
|
||||
@@ -107,10 +110,10 @@ const Dash = (() => {
|
||||
*
|
||||
* Thresholds live in `src/dashboards/severity.py` and travel in the payload as
|
||||
* `severity`; `severityFrom()` below is how a page reads them. This function survives
|
||||
* only because `page-index.js`, `page-system.js`, `page-llm.js` and `page-ups.js` still
|
||||
* call it with their own numbers (phases 02–05 convert them page by page, so each page's
|
||||
* change is reviewable on its own). Do not add a new call site: a threshold written in
|
||||
* JS is a threshold that can disagree with the other three pages.
|
||||
* only because `page-system.js`, `page-llm.js` and `page-ups.js` still call it with
|
||||
* their own numbers (phases 03–05 convert them page by page, so each page's change is
|
||||
* reviewable on its own — `/` was already converted in phase 02). Do not add a new call
|
||||
* site: a threshold written in JS is a threshold that can disagree with the other pages.
|
||||
*/
|
||||
function severityFor(value, warnAt, failAt) {
|
||||
const warn = warnAt == null ? 75 : warnAt;
|
||||
@@ -312,10 +315,7 @@ const Dash = (() => {
|
||||
const clock = $('#clock');
|
||||
if (clock) {
|
||||
const tickClock = () => {
|
||||
const now = new Date();
|
||||
const target = document.getElementById('sim-flag');
|
||||
clock.textContent = now.toLocaleTimeString(undefined, { hour12: false });
|
||||
void target;
|
||||
clock.textContent = new Date().toLocaleTimeString(undefined, { hour12: false });
|
||||
};
|
||||
tickClock();
|
||||
setInterval(tickClock, 1000);
|
||||
|
||||
@@ -179,8 +179,6 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
|
||||
.grid.auto { grid-template-columns: repeat(auto-fit, minmax(min(360px, 100%), 1fr)); }
|
||||
.grid.cols-2 { grid-template-columns: repeat(auto-fit, minmax(min(480px, 100%), 1fr)); }
|
||||
.grid.cols-3 { grid-template-columns: repeat(auto-fit, minmax(min(300px, 100%), 1fr)); }
|
||||
.grid.wide { grid-template-columns: minmax(0, 2fr) minmax(0, 1fr); }
|
||||
@media (max-width: 1150px) { .grid.wide { grid-template-columns: 1fr; } }
|
||||
.stack { display: grid; gap: .85rem; align-content: start; }
|
||||
|
||||
/* ---- cards -------------------------------------------------------------- */
|
||||
@@ -208,8 +206,6 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
|
||||
.card > .body.tight { padding: .45rem .8rem .6rem; }
|
||||
.card.flat { box-shadow: none; }
|
||||
|
||||
.cardset { display: grid; gap: .85rem; }
|
||||
|
||||
/* ---- metrics ------------------------------------------------------------ */
|
||||
.metrics { display: grid; gap: .7rem .9rem; grid-template-columns: repeat(auto-fit, minmax(104px, 1fr)); }
|
||||
.metric { min-width: 0; }
|
||||
@@ -353,6 +349,11 @@ table.data tbody tr:hover td { background: var(--bg-sunken); }
|
||||
table.data td.name { font-weight: 600; }
|
||||
table.data .num { font-family: var(--mono); }
|
||||
table.data .mute { color: var(--text-muted); }
|
||||
/* verdict colour on a table cell (overview fleet rows): only a breach paints, `ok` stays
|
||||
plain text so a healthy table is quiet — the words are the same `ok|warn|fail` the API
|
||||
sends, never a JS-side threshold */
|
||||
table.data td.warn { color: var(--warn); }
|
||||
table.data td.fail { color: var(--fail); }
|
||||
table.data.scroll { display: block; overflow: auto; max-height: 320px; }
|
||||
|
||||
/* ---- charts ------------------------------------------------------------- */
|
||||
@@ -404,6 +405,29 @@ ul.events .sev.info { color: var(--text-muted); }
|
||||
ul.events .sev.warn { color: var(--warn); }
|
||||
ul.events .sev.error { color: var(--fail); }
|
||||
|
||||
/* overview alerts: source (link to its dashboard) · message · age. The class on the row
|
||||
is the server's verdict — `alerts.py` decided, the page only paints it. */
|
||||
ul.alerts { list-style: none; margin: 0; padding: 0; display: grid; gap: .1rem; }
|
||||
ul.alerts li {
|
||||
display: grid; grid-template-columns: max-content 1fr max-content; gap: .7rem; align-items: baseline;
|
||||
padding: .4rem .1rem; border-bottom: 1px solid color-mix(in srgb, var(--border) 50%, transparent);
|
||||
font-size: 13px;
|
||||
}
|
||||
ul.alerts li:last-child { border-bottom: 0; }
|
||||
ul.alerts .src { font-weight: 650; white-space: nowrap; }
|
||||
ul.alerts .msg { color: var(--text-muted); overflow-wrap: anywhere; }
|
||||
ul.alerts time { font-family: var(--mono); font-size: 11px; color: var(--text-faint); white-space: nowrap; }
|
||||
ul.alerts li.ok .src { color: var(--ok); }
|
||||
ul.alerts li.warn .src { color: var(--warn); }
|
||||
ul.alerts li.fail .src { color: var(--fail); }
|
||||
|
||||
/* the overview's two panels swap by verdict, not by re-rendering: an active problem reads
|
||||
first, an empty list leaves the fleet table on top and the all-clear state below it */
|
||||
#overview-panels .panel-fleet { order: 1; }
|
||||
#overview-panels .panel-alerts { order: 2; }
|
||||
#overview-panels[data-state="alerting"] .panel-alerts { order: 1; }
|
||||
#overview-panels[data-state="alerting"] .panel-fleet { order: 2; }
|
||||
|
||||
ul.checks { list-style: none; margin: 0; padding: 0; display: grid; gap: .4rem; }
|
||||
ul.checks li { display: grid; grid-template-columns: 18px 1fr; gap: .5rem; align-items: start; }
|
||||
ul.checks .what { font-weight: 600; font-size: 12.5px; }
|
||||
@@ -472,7 +496,6 @@ dialog footer { padding: .7rem .9rem; border-top: 1px solid var(--border); displ
|
||||
.empty-state ul { text-align: left; display: inline-block; margin: .6rem 0 0; }
|
||||
|
||||
.stale { opacity: .62; }
|
||||
.stale-flag { color: var(--warn); font-size: 11px; }
|
||||
|
||||
select, input[type="text"] {
|
||||
background: var(--bg-sunken); color: var(--text); border: 1px solid var(--border);
|
||||
|
||||
@@ -1,7 +1,13 @@
|
||||
/* Overview page: one /api/summary call, plus the config-driven setup hints. */
|
||||
/* Overview page: exactly two answers — what needs attention, and what the fleet is doing.
|
||||
*
|
||||
* One `/api/summary` call feeds both panels; "what needs attention" is the server's list
|
||||
* (`alerts.py`), never a browser-side opinion. The page owns no thresholds: cells are
|
||||
* coloured from the `severity` verdicts in the payload via `Dash.severityFrom`, and an
|
||||
* unrated value gets no colour (which is also what keeps `n/a` from looking green).
|
||||
*/
|
||||
'use strict';
|
||||
|
||||
const { $, fetchJSON, setTxt, setValue, setBar, setDot, bytes, pct, num, duration, poll } = Dash;
|
||||
const { $, fetchJSON, num, duration, age, poll } = Dash;
|
||||
|
||||
async function loadConfig() {
|
||||
try {
|
||||
@@ -48,101 +54,84 @@ async function loadConfig() {
|
||||
}
|
||||
}
|
||||
|
||||
function renderEvents(events) {
|
||||
const list = $('#events');
|
||||
if (!events || !events.length) {
|
||||
list.innerHTML = '<li class="muted" style="grid-template-columns:1fr">Nothing has changed ' +
|
||||
'since this server started — the log only records transitions.</li>';
|
||||
/**
|
||||
* The warnings/errors panel: one row per alert from `/api/summary`'s `alerts`.
|
||||
*
|
||||
* The row is the source (a link to the dashboard that explains it), the server's message,
|
||||
* and the age. Colour comes from `alert.severity` through `Dash.severityFrom` — the page
|
||||
* never re-derives a verdict. An empty list is a message ("all clear"), not a blank box,
|
||||
* and it also moves the panel below the fleet table so a calm morning reads calm.
|
||||
*/
|
||||
function renderAlerts(alerts) {
|
||||
const list = $('#alerts');
|
||||
const clear = $('#alerts-clear');
|
||||
const panels = $('#overview-panels');
|
||||
const items = Array.isArray(alerts) ? alerts : [];
|
||||
panels.dataset.state = items.length ? 'alerting' : 'clear';
|
||||
$('#alerts-count').textContent = items.length
|
||||
? `${items.length} active`
|
||||
: '';
|
||||
clear.hidden = items.length > 0;
|
||||
list.hidden = items.length === 0;
|
||||
if (!items.length) {
|
||||
list.innerHTML = '';
|
||||
return;
|
||||
}
|
||||
list.innerHTML = events.map((event) => {
|
||||
const when = new Date(event.ts * 1000).toLocaleTimeString(undefined,
|
||||
{ hour: '2-digit', minute: '2-digit', hour12: false });
|
||||
return `<li><time>${when}</time><span><span class="sev ${event.severity}">${event.host}</span> ` +
|
||||
`${DashCharts.escapeHtml(event.message.replace(event.host + ':', '').trim() || event.kind)}</span></li>`;
|
||||
list.innerHTML = items.map((alert) => {
|
||||
// alerts.py names the source at the head of most messages; the link already says it,
|
||||
// and "deskwork: deskwork: CPU 92%" reads like two problems instead of one
|
||||
const source = String(alert.source || 'fleet');
|
||||
const raw = String(alert.message || '');
|
||||
const prefix = `${source}:`;
|
||||
const message = raw.startsWith(prefix) ? raw.slice(prefix.length).trim() : raw;
|
||||
const sev = Dash.severityFrom(alert.severity);
|
||||
return `<li${sev ? ` class="${sev}"` : ''}>
|
||||
<a class="src" href="${DashCharts.escapeHtml(alert.href || '/')}">${DashCharts.escapeHtml(source)}</a>
|
||||
<span class="msg">${DashCharts.escapeHtml(message)}</span>
|
||||
<time>${age(alert.at)}</time>
|
||||
</li>`;
|
||||
}).join('');
|
||||
}
|
||||
|
||||
function renderSummary(summary) {
|
||||
const system = summary.system || {};
|
||||
const fleet = system.fleet || {};
|
||||
setDot($('#dot-system'), system.enabled ? (fleet.hosts_live === fleet.hosts_total ? 'ok'
|
||||
: fleet.hosts_live ? 'warn' : 'fail') : 'idle');
|
||||
setTxt($('#sys-hosts-live'), fleet.hosts_live == null ? 'n/a' : String(fleet.hosts_live));
|
||||
setTxt($('#sys-hosts-total'), fleet.hosts_total == null ? '' : `of ${fleet.hosts_total}`);
|
||||
setValue($('#sys-cpu'), fleet.cpu_usage_pct_avg, 0);
|
||||
setBar($('#sys-cpu').closest('.metric').querySelector('.bar'), fleet.cpu_usage_pct_avg);
|
||||
if (fleet.memory_used_bytes != null) {
|
||||
const used = Dash.fmtBytes(fleet.memory_used_bytes, 0);
|
||||
const total = Dash.fmtBytes(fleet.memory_total_bytes, 0);
|
||||
// "200 / 253 GiB", not "200 GiB of 253 GiB": the card column is narrow and the long form
|
||||
// ran into the worst-disk number next to it
|
||||
setTxt($('#sys-mem'), total ? `${used.text}/${total.text}` : used.text);
|
||||
setTxt($('#sys-mem-unit'), used.unit);
|
||||
} else {
|
||||
setTxt($('#sys-mem'), 'n/a');
|
||||
setTxt($('#sys-mem-unit'), '');
|
||||
/**
|
||||
* The fleet table: one row per host, columns unchanged.
|
||||
*
|
||||
* Cell colour comes from the row's `severity` verdicts (`/api/summary`, phase 01). `ok`
|
||||
* prints as plain text on purpose — an all-green table is noise; only a breach earns a
|
||||
* colour here, and the alerts panel is where anything rated gets its line.
|
||||
*
|
||||
* A host whose agent stopped answering keeps its last good numbers rather than being
|
||||
* zeroed, so its row is dimmed with the shared `.stale` class (and says so in its
|
||||
* tooltip): the table must never imply dead numbers are live while the panel above
|
||||
* says the box is down.
|
||||
*/
|
||||
function renderFleet(rows, states) {
|
||||
const answering = {};
|
||||
for (const state of states || []) {
|
||||
const name = String(state.label || state.key || '');
|
||||
if (name) answering[name] = Boolean(state.ok);
|
||||
}
|
||||
setBar($('#sys-mem').closest('.metric').querySelector('.bar'), fleet.memory_used_pct);
|
||||
setValue($('#sys-disk'), fleet.disk_worst_used_pct, 0);
|
||||
setBar($('#sys-disk').closest('.metric').querySelector('.bar'), fleet.disk_worst_used_pct,
|
||||
Dash.severityFor(fleet.disk_worst_used_pct, 80, 92));
|
||||
setTxt($('#sys-gpus'), fleet.gpu_count == null ? 'n/a'
|
||||
: `${fleet.gpu_count}${fleet.gpu_busy_pct_avg != null ? ' · ' + num(fleet.gpu_busy_pct_avg, 0) + '%' : ''}`);
|
||||
|
||||
const llm = summary.llm || {};
|
||||
const engines = llm.fleet || {};
|
||||
setDot($('#dot-llm'), !llm.enabled ? 'idle'
|
||||
: (engines.engines_live === engines.engines_total ? 'ok' : engines.engines_live ? 'warn' : 'fail'));
|
||||
setTxt($('#llm-live'), engines.engines_live == null ? 'n/a' : String(engines.engines_live));
|
||||
setTxt($('#llm-total'), engines.engines_total == null ? '' : `of ${engines.engines_total}`);
|
||||
setValue($('#llm-decode'), engines.decode_tokens_per_second, 1);
|
||||
setValue($('#llm-prompt'), engines.prompt_tokens_per_second, 0);
|
||||
setTxt($('#llm-active'), engines.requests_active == null ? 'n/a' : num(engines.requests_active, 0));
|
||||
setTxt($('#llm-waiting'), engines.requests_waiting == null ? 'n/a' : num(engines.requests_waiting, 0));
|
||||
setValue($('#llm-kv'), engines.kv_cache_usage_max == null ? null : engines.kv_cache_usage_max * 100, 0);
|
||||
setBar($('#llm-kv').closest('.metric').querySelector('.bar'),
|
||||
engines.kv_cache_usage_max == null ? null : engines.kv_cache_usage_max * 100,
|
||||
Dash.severityFor(engines.kv_cache_usage_max == null ? null : engines.kv_cache_usage_max * 100, 75, 90));
|
||||
|
||||
const ups = summary.ups || {};
|
||||
setDot($('#dot-ups'), !ups.enabled ? 'idle'
|
||||
: (ups.ups || []).some((u) => u.overall === 'at risk') ? 'fail'
|
||||
: (ups.ups || []).some((u) => u.overall !== 'healthy') ? 'warn' : 'ok');
|
||||
const list = $('#ups-list');
|
||||
if (!ups.enabled) {
|
||||
list.innerHTML = '<div class="muted">No NUT server configured.</div>';
|
||||
} else if (!(ups.ups || []).length) {
|
||||
list.innerHTML = '<div class="muted">No UPS answering from this upsd yet.</div>';
|
||||
} else {
|
||||
list.innerHTML = ups.ups.map((item) => {
|
||||
const kind = item.overall === 'healthy' ? 'ok' : item.overall === 'at risk' ? 'fail'
|
||||
: item.overall === 'degraded' ? 'warn' : '';
|
||||
return `<div class="gap" style="justify-content:space-between">
|
||||
<span class="gap"><span class="chip ${kind}">${DashCharts.escapeHtml(item.name)}</span>
|
||||
<span class="mono faint">${DashCharts.escapeHtml(item.status || 'no state')}</span>
|
||||
${ups.simulated ? '<span class="sim-badge">sim</span>' : ''}</span>
|
||||
<span class="mono">${item.charge_pct == null ? 'n/a' : num(item.charge_pct, 0) + '%'} ·
|
||||
${item.load_pct == null ? 'n/a' : num(item.load_pct, 0) + '% load'} ·
|
||||
${item.runtime_seconds == null ? 'n/a' : duration(item.runtime_seconds) + ' runtime'}</span>
|
||||
</div>`;
|
||||
}).join('');
|
||||
}
|
||||
|
||||
const rows = system.hosts || [];
|
||||
const tbody = $('#fleet-table tbody');
|
||||
$('#fleet-empty').hidden = rows.length > 0;
|
||||
if (!rows.length) { tbody.innerHTML = ''; return; }
|
||||
const verdict = (severity, key) => {
|
||||
const kind = Dash.severityFrom((severity || {})[key]);
|
||||
return kind === 'ok' || kind === '' ? '' : ` ${kind}`;
|
||||
};
|
||||
tbody.innerHTML = rows.map((host) => {
|
||||
const sev = host.severity || {};
|
||||
const gpus = (host.gpu_busy_pct || []).filter((v) => v != null);
|
||||
return `<tr>
|
||||
// the row's own flag (a host that has never answered has no payload to be stale about)
|
||||
// or the job state: either way the numbers below are the last good ones, not live ones
|
||||
const stale = host.unreachable === true || answering[host.name] === false;
|
||||
return `<tr${stale ? ' class="stale" title="Last good values — this host is not answering"' : ''}>
|
||||
<td class="name">${DashCharts.escapeHtml(host.name)}</td>
|
||||
<td class="num">${host.cpu_usage_pct == null ? 'n/a' : num(host.cpu_usage_pct, 0) + '%'}</td>
|
||||
<td class="num${verdict(sev, 'cpu_usage_pct')}">${host.cpu_usage_pct == null ? 'n/a' : num(host.cpu_usage_pct, 0) + '%'}</td>
|
||||
<td class="num mute">${host.load1 == null ? 'n/a' : num(host.load1, 2)}</td>
|
||||
<td class="num">${host.mem_used_pct == null ? 'n/a' : num(host.mem_used_pct, 0) + '%'}</td>
|
||||
<td class="num">${host.disk_worst_used_pct == null ? 'n/a' : num(host.disk_worst_used_pct, 0) + '%'}</td>
|
||||
<td class="num${verdict(sev, 'mem_used_pct')}">${host.mem_used_pct == null ? 'n/a' : num(host.mem_used_pct, 0) + '%'}</td>
|
||||
<td class="num${verdict(sev, 'disk_worst_used_pct')}">${host.disk_worst_used_pct == null ? 'n/a' : num(host.disk_worst_used_pct, 0) + '%'}</td>
|
||||
<td class="num">${gpus.length ? num(gpus[0], 0) + '%' : '<span class="mute">—</span>'}</td>
|
||||
<td class="num">${host.temp_c == null ? '<span class="mute">n/a</span>' : num(host.temp_c, 0) + '°C'}</td>
|
||||
<td class="num${verdict(sev, 'temp_c')}">${host.temp_c == null ? '<span class="mute">n/a</span>' : num(host.temp_c, 0) + '°C'}</td>
|
||||
<td class="num mute">${host.uptime_seconds == null ? 'n/a' : duration(host.uptime_seconds)}</td>
|
||||
</tr>`;
|
||||
}).join('');
|
||||
@@ -151,14 +140,24 @@ function renderSummary(summary) {
|
||||
async function refresh() {
|
||||
try {
|
||||
const summary = await fetchJSON('/api/summary');
|
||||
const total = (summary.system.hosts || []).length;
|
||||
const live = summary.system.fleet ? summary.system.fleet.hosts_live : 0;
|
||||
const engines = summary.llm.fleet ? `${summary.llm.fleet.engines_live}/${summary.llm.fleet.engines_total} engines` : 'no engines';
|
||||
const ups = summary.ups.ups ? `${summary.ups.ups.length} UPS` : 'no UPS';
|
||||
$('#summary-line').textContent = `${live}/${total} hosts · ${engines} · ${ups} · ` +
|
||||
`${(summary.recent_events || []).length} recent events`;
|
||||
renderSummary(summary);
|
||||
renderEvents(summary.recent_events);
|
||||
const system = summary.system || {};
|
||||
const fleet = system.fleet || {};
|
||||
const hosts = system.hosts || [];
|
||||
const live = fleet.hosts_live == null ? hosts.length : fleet.hosts_live;
|
||||
const total = fleet.hosts_total == null ? hosts.length : fleet.hosts_total;
|
||||
const engines = summary.llm && summary.llm.fleet
|
||||
? `${summary.llm.fleet.engines_live}/${summary.llm.fleet.engines_total} engines`
|
||||
: 'no engines';
|
||||
const ups = summary.ups && (summary.ups.ups || []).length
|
||||
? `${summary.ups.ups.length} UPS`
|
||||
: 'no UPS';
|
||||
const alerts = summary.alerts || [];
|
||||
const problems = alerts.length ? `${alerts.length} warning${alerts.length === 1 ? '' : 's'}`
|
||||
: 'all clear';
|
||||
$('#summary-line').textContent =
|
||||
`${live}/${total} hosts · ${engines} · ${ups} · ${problems}`;
|
||||
renderAlerts(alerts);
|
||||
renderFleet(hosts, system.states);
|
||||
} catch (err) {
|
||||
$('#summary-line').textContent = `summary failed: ${err.message}`;
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@ letting `uv run pytest -m live --live` catch an upstream that renamed something.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
@@ -42,6 +43,7 @@ if str(SRC) not in sys.path:
|
||||
sys.path.insert(0, str(SRC))
|
||||
|
||||
from dashboards.rates import RateEstimator # noqa: E402
|
||||
from dashboards.sources.host import HostCollector, parse_host_snapshot # noqa: E402
|
||||
from dashboards.store import SampleStore # noqa: E402
|
||||
|
||||
FIXTURES = Path(__file__).resolve().parent / "fixtures"
|
||||
@@ -77,6 +79,34 @@ def agent_snapshots() -> dict[str, dict[str, Any]]:
|
||||
return out
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def host_scrape() -> Any:
|
||||
"""`host_scrape(snapshot, cpu_busy=10.0, mutate=None)` -> one parsed host payload.
|
||||
|
||||
Two scrapes are needed before any rate exists (the agent reports counters; the
|
||||
dashboard owns the window), and the payload the API serves is the second one. `mutate`
|
||||
edits the snapshot before both scrapes so a test can put a value on a chosen side of
|
||||
`severity.THRESHOLDS`; `cpu_busy` sets the CPU percentage over the window directly.
|
||||
Driven from the real capture so a schema drift breaks here rather than in a browser.
|
||||
"""
|
||||
def scrape(snapshot: dict[str, Any], *, name: str = "driveripper", cpu_busy: float = 10.0,
|
||||
mutate: Any = None) -> dict[str, Any]:
|
||||
snap = copy.deepcopy(snapshot)
|
||||
if mutate is not None:
|
||||
mutate(snap)
|
||||
collector = HostCollector(RateEstimator(span=60.0))
|
||||
parse_host_snapshot(name, snap, collector, now=1_000.0)
|
||||
second = copy.deepcopy(snap)
|
||||
second["ts"] = 1_010.0
|
||||
modes = second["cpu"]["modes"]
|
||||
# busy = (Δtotal - Δidle)/Δtotal, and iowait is left untouched so it rates at 0
|
||||
modes["total"] += 1_000.0
|
||||
modes["idle"] += 1_000.0 - (cpu_busy * 10.0)
|
||||
return parse_host_snapshot(name, second, collector, now=1_010.0).payload
|
||||
|
||||
return scrape
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def metrics_text() -> Any:
|
||||
"""metrics_text('gufo') / metrics_text('vllm') -> the captured exposition verbatim."""
|
||||
|
||||
@@ -0,0 +1,476 @@
|
||||
"""`alerts.build`: the whole "what is wrong right now" answer, with no I/O anywhere.
|
||||
|
||||
The builder is a pure projection of state the poller already keeps and verdicts
|
||||
`severity.py` / `sources/nut.assess()` already reached, so every rule can be tested by
|
||||
handing it a dict — no hub, no DB, no sockets, no clock it was not given. The cases below
|
||||
are the ones that make the landing page trustworthy rather than merely busy:
|
||||
|
||||
* a source nobody configured is **never** an alert (`AGENTS.md` invariant 3);
|
||||
* a source that is stale but still serving is amber, not red;
|
||||
* a dead source is **one** line, not one per stale metric in its last good payload;
|
||||
* only `fail` and `warn` come through — `ok`, `unknown`, and a missing number are silence;
|
||||
* the list dedupes per (source, metric), is bounded, and says how much it hid.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from dashboards import alerts
|
||||
from dashboards.config import Settings
|
||||
from dashboards.sources import nut
|
||||
from dashboards.sources.nut_sim import NutSimulator
|
||||
|
||||
SETTINGS = Settings() # defaults: no targets, intervals 10s system / 2s llm / 5s ups
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# state shapers
|
||||
# --------------------------------------------------------------------------- #
|
||||
def state(key: str = "host:driveripper", dashboard: str = "system",
|
||||
label: str = "driveripper", *, ok: bool = True, configured: bool = True,
|
||||
error: str | None = None, age: float | None = 0.0) -> dict[str, Any]:
|
||||
"""A `SourceState.as_json()`-shaped health record — the only shape the builder reads."""
|
||||
return {"key": key, "dashboard": dashboard, "label": label, "configured": configured,
|
||||
"ok": ok, "error": error, "age_seconds": age, "attempts": 4, "successes": 3,
|
||||
"consecutive_failures": 0 if ok else 1, "last_ok": None, "latency_ms": None}
|
||||
|
||||
|
||||
def check(name: str, status: str, detail: str = "") -> dict[str, Any]:
|
||||
return {"name": name, "status": status, "detail": detail, "metric": None, "value": None}
|
||||
|
||||
|
||||
def ups_payload(*cards: dict[str, Any], simulated: bool = False) -> dict[str, Any]:
|
||||
return {"ts": 1_000.0, "simulated": simulated, "ups": list(cards)}
|
||||
|
||||
|
||||
def ups_card(name: str = "apc", *checks: dict[str, Any]) -> dict[str, Any]:
|
||||
return {"name": name, "ts": 1_000.0, "health": {"overall": "healthy", "checks": list(checks)}}
|
||||
|
||||
|
||||
def engine(name: str = "gufo", **cells: Any) -> dict[str, Any]:
|
||||
return {"engine": name, "scraped_at": 1_000.0, "metrics": dict(cells)}
|
||||
|
||||
|
||||
def ratio(value: float | None, verdict: str | None) -> dict[str, Any]:
|
||||
"""A rated LLM cell, exactly as `sources/llm.py` publishes it (0-1 ratio + verdict)."""
|
||||
return {"value": value, "source": "llamacpp:kv_cache_usage_ratio", "derived": False,
|
||||
"severity": verdict}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# the all-clear, and the things that must not look like a problem
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_a_healthy_fleet_alerts_nothing(host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=10.0)
|
||||
assert alerts.build(SETTINGS, [state()], [payload], [], ups_payload(), [], now=2_000.0) == []
|
||||
|
||||
|
||||
def test_no_inputs_at_all_is_all_clear_rather_than_an_error() -> None:
|
||||
"""A fresh install with nothing wired up has nothing wrong with it."""
|
||||
assert alerts.build(SETTINGS) == []
|
||||
|
||||
|
||||
def test_an_unconfigured_source_never_alerts() -> None:
|
||||
"""`AGENTS.md` invariant 3: not installed is not broken.
|
||||
|
||||
The health record for a disabled source says `ok: false` exactly like a dead one, so
|
||||
`configured` is the only thing that tells those two apart — and the state map having
|
||||
nothing for a source must not be read as a failure either.
|
||||
"""
|
||||
out = alerts.build(SETTINGS, [
|
||||
state(ok=False, configured=False, error="Connection refused"),
|
||||
state(key="nut", dashboard="ups", label="NUT server", ok=False, configured=False),
|
||||
], [], [], ups_payload(), now=2_000.0)
|
||||
assert out == []
|
||||
|
||||
|
||||
def test_a_card_with_no_checks_and_a_source_with_no_state_say_nothing() -> None:
|
||||
assert alerts.build(SETTINGS, [], [], [], ups_payload(ups_card("apc"))) == []
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# reachability
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_a_source_that_stopped_answering_is_a_fail_naming_it_and_linking_its_page() -> None:
|
||||
out = alerts.build(SETTINGS, [state(ok=False, error="Connection refused")], now=2_000.0)
|
||||
assert len(out) == 1
|
||||
alert = out[0]
|
||||
assert alert["severity"] == "fail"
|
||||
assert alert["source"] == "driveripper"
|
||||
assert alert["href"] == "/system"
|
||||
assert alert["message"] == "driveripper: Connection refused"
|
||||
assert alert["at"] == 2_000.0
|
||||
|
||||
|
||||
def test_the_collectors_host_prefix_is_not_doubled() -> None:
|
||||
"""Collectors already say `driveripper: …`; the alert starts with the name too."""
|
||||
out = alerts.build(SETTINGS, [state(ok=False,
|
||||
error="driveripper: http://h:9121/snapshot unreachable")])
|
||||
assert out[0]["message"] == "driveripper: http://h:9121/snapshot unreachable"
|
||||
|
||||
|
||||
def test_the_pollers_exception_prefix_is_not_printed_either() -> None:
|
||||
"""The live shape is `poller`'s `UpstreamError: name: …`; one name, no class noise.
|
||||
|
||||
Verified against a real stopped agent tunnel: the health record reads
|
||||
`UpstreamError: gateway: http://127.0.0.1:9123/snapshot unreachable (ConnectError)`,
|
||||
and printing that verbatim put the host name on the page twice.
|
||||
"""
|
||||
out = alerts.build(SETTINGS, [state(key="host:gateway", label="gateway", ok=False,
|
||||
error="UpstreamError: gateway: http://h:9123/snapshot "
|
||||
"unreachable (ConnectError)")])
|
||||
assert out[0]["message"] == ("gateway: http://h:9123/snapshot unreachable (ConnectError)")
|
||||
|
||||
|
||||
def test_an_unexpected_exception_class_is_still_named() -> None:
|
||||
"""Only `UpstreamError` is noise; a bug in our own code is worth reading about."""
|
||||
out = alerts.build(SETTINGS, [state(ok=False, error="ValueError: bad snapshot")])
|
||||
assert out[0]["message"] == "driveripper: ValueError: bad snapshot"
|
||||
|
||||
|
||||
def test_a_failing_source_with_no_recorded_error_still_says_so() -> None:
|
||||
out = alerts.build(SETTINGS, [state(key="llm:gufo", dashboard="llm", label="gufo",
|
||||
ok=False, error=None)])
|
||||
assert out[0]["message"] == "gufo: is not answering"
|
||||
assert out[0]["href"] == "/llm"
|
||||
|
||||
|
||||
def test_a_stale_but_serving_source_is_a_warn_because_the_page_still_shows_numbers() -> None:
|
||||
"""Last good value on screen + no refresh for 3 intervals = amber, not `dead`."""
|
||||
out = alerts.build(SETTINGS, [state(age=45.0)], now=2_000.0)
|
||||
assert [item["severity"] for item in out] == ["warn"]
|
||||
assert "45s ago" in out[0]["message"]
|
||||
assert "last good value" in out[0]["message"]
|
||||
assert out[0]["at"] == 2_000.0 - 45.0
|
||||
# inside the grace period (3 x 10s) the same source is not news at all
|
||||
assert alerts.build(SETTINGS, [state(age=25.0)]) == []
|
||||
|
||||
|
||||
def test_staleness_is_measured_in_each_sources_own_poll_interval() -> None:
|
||||
"""system polls every 10s and the UPS every 5s — one global number is wrong for one."""
|
||||
llm = state(key="llm:gufo", dashboard="llm", label="gufo", age=5.0) # 3 x 2s = 6s
|
||||
ups = state(key="nut", dashboard="ups", label="NUT server", age=20.0) # 3 x 5s = 15s
|
||||
assert alerts.build(SETTINGS, [llm]) == []
|
||||
out = alerts.build(SETTINGS, [ups])
|
||||
assert out[0]["severity"] == "warn" and out[0]["href"] == "/ups"
|
||||
|
||||
|
||||
def test_a_dashboard_the_builder_does_not_know_about_still_gets_a_verdict_and_a_home() -> None:
|
||||
"""A new dashboard must not be silently un-alertable while its interval is unknown."""
|
||||
unknown_down = {"key": "probe:0", "dashboard": "telemetry", "label": "probe",
|
||||
"configured": True, "ok": False, "error": "no route"}
|
||||
assert alerts.build(SETTINGS, [unknown_down])[0]["href"] == "/"
|
||||
unknown_stale = {**unknown_down, "ok": True, "error": None, "age_seconds": 200.0}
|
||||
out = alerts.build(SETTINGS, [unknown_stale])
|
||||
assert "every 60s" in out[0]["message"]
|
||||
|
||||
|
||||
def test_a_failing_source_says_when_the_poller_first_noticed() -> None:
|
||||
events = [{"ts": 1_500.0, "source": "system", "host": "driveripper", "kind": "unreachable",
|
||||
"severity": "error", "message": "ConnectError", "data": None},
|
||||
{"ts": 1_600.0, "source": "host", "host": "deskwork", "kind": "reboot",
|
||||
"severity": "warn", "message": "deskwork rebooted", "data": None}]
|
||||
out = alerts.build(SETTINGS, [state(ok=False, error="nope")], [], [], None, events,
|
||||
now=2_000.0)
|
||||
assert out[0]["at"] == 1_500.0, "the outage clock must not reset on every refresh"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# host thresholds, degraded agents
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_a_threshold_breach_names_the_host_the_metric_and_the_value(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=95.0)
|
||||
out = alerts.build(SETTINGS, [state()], [payload], now=2_000.0)
|
||||
assert out[0]["id"] == "driveripper:cpu.usage_pct"
|
||||
assert out[0]["message"] == "driveripper: CPU 95% (fail ≥90)"
|
||||
assert out[0]["href"] == "/system"
|
||||
assert out[0]["at"] == 1_010.0, "`at` is the poll that measured it"
|
||||
|
||||
|
||||
def test_every_rated_host_field_alerts_at_its_own_verdict(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
def nudge(snap: dict[str, Any]) -> None:
|
||||
snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.85 # warn ≥80
|
||||
snap["cpu_temperature_celsius"] = 80.0 # warn ≥75
|
||||
for fs in snap["filesystems"]:
|
||||
if fs["mountpoint"] == "/btrfs/pool1":
|
||||
fs["used_pct"] = 91.0 # fail ≥90
|
||||
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=10.0, mutate=nudge)
|
||||
out = {item["id"]: item for item in alerts.build(SETTINGS, [state()], [payload])}
|
||||
assert out["driveripper:mem.used_pct"]["message"] == "driveripper: memory 85% (warn ≥80)"
|
||||
assert out["driveripper:cpu.temp_c"]["message"] == (
|
||||
"driveripper: CPU temperature 80°C (warn ≥75)")
|
||||
assert out["driveripper:disk.used_pct"]["message"] == (
|
||||
"driveripper: disk 91% on /btrfs/pool1 (fail ≥90)")
|
||||
assert "driveripper:cpu.usage_pct" not in out, "10% busy is not a problem"
|
||||
|
||||
|
||||
def test_a_number_the_upstream_does_not_publish_is_never_a_breach(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
"""No measurement, no verdict, no alert — the same rule `severity.classify` upholds."""
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=10.0,
|
||||
mutate=lambda snap: snap.pop("cpu_temperature_celsius"))
|
||||
assert alerts.build(SETTINGS, [state()], [payload]) == []
|
||||
|
||||
|
||||
def test_a_gpu_breach_names_the_card_and_dedupes_per_metric(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
def fill_the_card(snap: dict[str, Any]) -> None:
|
||||
gpu = snap["gpus"][0]
|
||||
gpu["busy_pct"] = 97.0
|
||||
for used, total in (("vram_used_bytes", "vram_total_bytes"),
|
||||
("gtt_used_bytes", "gtt_total_bytes")):
|
||||
if gpu.get(total) is not None:
|
||||
gpu[used] = gpu[total]
|
||||
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=10.0, mutate=fill_the_card)
|
||||
messages = {item["id"]: item["message"] for item in alerts.build(SETTINGS, [state()], [payload])}
|
||||
assert any("GPU utilisation 97% on Radeon AI PRO R9700" in text
|
||||
for text in messages.values()), messages
|
||||
assert any("GPU memory" in text and "R9700" in text for text in messages.values())
|
||||
|
||||
|
||||
def test_a_degraded_agent_warns_and_names_the_sections_that_failed(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=10.0)
|
||||
payload["degraded"] = {"temperatures": "no hwmon sensors", "disk_io": ""}
|
||||
out = alerts.build(SETTINGS, [state()], [payload])
|
||||
assert len(out) == 1, "a partial snapshot is one warning, not one per number that is missing"
|
||||
assert out[0]["id"] == "driveripper:agent.sections"
|
||||
assert out[0]["severity"] == "warn"
|
||||
assert "temperatures (no hwmon sensors)" in out[0]["message"]
|
||||
assert "disk_io" in out[0]["message"]
|
||||
|
||||
|
||||
def test_a_degraded_list_from_an_older_agent_is_readable_too(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=10.0)
|
||||
payload["degraded"] = ["network"]
|
||||
out = alerts.build(SETTINGS, [state()], [payload])
|
||||
assert out[0]["message"] == "driveripper: the agent could not read network"
|
||||
|
||||
|
||||
def test_a_degraded_note_that_is_a_plain_string_is_still_reported(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
"""An agent (or a future collector) that reports one string instead of a map."""
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=10.0)
|
||||
payload["degraded"] = "hwmon"
|
||||
out = alerts.build(SETTINGS, [state()], [payload])
|
||||
assert out[0]["message"] == "driveripper: the agent could not read hwmon"
|
||||
|
||||
|
||||
def test_an_unreachable_host_is_one_alert_even_with_its_last_good_payload(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
"""The exact completion criterion for the phase: one line naming the dead host."""
|
||||
def nudge(snap: dict[str, Any]) -> None:
|
||||
snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.99
|
||||
snap["cpu_temperature_celsius"] = 95.0
|
||||
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=96.0, mutate=nudge)
|
||||
out = alerts.build(SETTINGS, [state(ok=False, error="timed out")], [payload])
|
||||
assert len(out) == 1
|
||||
assert out[0]["severity"] == "fail" and "timed out" in out[0]["message"]
|
||||
|
||||
|
||||
def test_a_stub_payload_for_a_host_that_never_answered_alerts_once() -> None:
|
||||
"""/api/system/snapshot represents that host as {name, unreachable}; still one line."""
|
||||
stub = {"name": "gateway", "ts": 1_000.0, "unreachable": True}
|
||||
out = alerts.build(SETTINGS, [state(key="host:gateway", label="gateway", ok=False,
|
||||
error="unreachable")], [stub])
|
||||
assert [item["id"] for item in out] == ["gateway:host:gateway:down"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# LLM engines
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_kv_pressure_comes_from_the_rated_cell_and_reads_as_a_percent() -> None:
|
||||
"""The engine publishes 0-1; the operator reads percent, matching the verdict's scale."""
|
||||
out = alerts.build(SETTINGS, [state(key="llm:gufo", dashboard="llm", label="gufo")], [],
|
||||
[engine(kv_cache_usage_ratio=ratio(0.95, "fail"))])
|
||||
assert out[0]["message"] == "gufo: KV cache 95% (fail ≥90)"
|
||||
assert out[0]["href"] == "/llm"
|
||||
assert out[0]["id"] == "gufo:llm.kv_cache_usage_pct@kv_cache_usage_ratio"
|
||||
|
||||
|
||||
def test_vllm_publishes_the_same_verdict_under_its_own_series_name() -> None:
|
||||
out = alerts.build(SETTINGS, [], [], [engine("vllm", gpu_cache_usage=ratio(0.8, "warn"))])
|
||||
assert out[0]["message"] == "vllm: KV cache 80% (warn ≥75)"
|
||||
|
||||
|
||||
def test_a_cool_or_absent_kv_cell_is_not_an_alert() -> None:
|
||||
for cell in (ratio(0.10, "ok"), ratio(None, None), ratio(0.95, None), {}, "not a cell"):
|
||||
out = alerts.build(SETTINGS, [], [], [engine(kv_cache_usage_ratio=cell)])
|
||||
assert out == [], cell
|
||||
|
||||
|
||||
def test_an_engine_that_is_not_answering_alerts_only_its_own_job() -> None:
|
||||
dead = {"engine": "vllm", "metrics": {}, "sessions": {}, "unreachable": True}
|
||||
out = alerts.build(SETTINGS, [state(key="llm:vllm", dashboard="llm", label="vLLM",
|
||||
ok=False, error="502 Bad Gateway")], [], [dead])
|
||||
assert [item["id"] for item in out] == ["vLLM:llm:vllm:down"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# UPS
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_a_failing_ups_check_is_a_fail_carrying_the_check_and_its_detail() -> None:
|
||||
payload = ups_payload(ups_card("apc",
|
||||
check("Power path", "fail", "The UPS is on battery"),
|
||||
check("Charge", "warn", "42% and discharging")))
|
||||
out = alerts.build(SETTINGS, [state(key="nut", dashboard="ups", label="NUT server")],
|
||||
[], [], payload)
|
||||
by_id = {item["id"]: item for item in out}
|
||||
assert by_id["apc:health:Power path"]["severity"] == "fail"
|
||||
assert by_id["apc:health:Power path"]["message"] == "Power path: The UPS is on battery"
|
||||
assert by_id["apc:health:Charge"]["severity"] == "warn"
|
||||
assert all(item["href"] == "/ups" for item in out)
|
||||
|
||||
|
||||
def test_the_landing_page_is_where_battery_age_belongs() -> None:
|
||||
"""The task's own example: a sentence an operator can act on, not a score."""
|
||||
detail = "5.3 years — past the 4-year replacement window"
|
||||
out = alerts.build(SETTINGS, [], [], [], ups_payload(ups_card("apc",
|
||||
check("Battery age", "warn", detail))))
|
||||
assert out[0]["message"] == f"Battery age: {detail}"
|
||||
assert out[0]["source"] == "apc"
|
||||
|
||||
|
||||
def test_an_unknown_check_is_silence_and_a_check_with_no_detail_is_still_readable() -> None:
|
||||
payload = ups_payload(ups_card("apc", check("Battery temperature", "unknown",
|
||||
"not reported by this driver"),
|
||||
check("Alarms", "fail")))
|
||||
out = alerts.build(SETTINGS, [], [], [], payload)
|
||||
assert [item["message"] for item in out] == ["Alarms"]
|
||||
|
||||
|
||||
def test_simulated_ups_data_is_announced_once_and_loudly() -> None:
|
||||
"""With NUT_SIMULATE=1 and otherwise-healthy cards, that warning is the whole list."""
|
||||
payload = ups_payload(ups_card("apc", check("Charge", "ok", "100%")),
|
||||
ups_card("cyberpower", check("Charge", "ok", "100%")),
|
||||
simulated=True)
|
||||
out = alerts.build(SETTINGS, [state(key="nut", dashboard="ups", label="NUT server")],
|
||||
[], [], payload)
|
||||
assert len(out) == 1
|
||||
assert out[0]["severity"] == "warn"
|
||||
assert "NUT_SIMULATE=1" in out[0]["message"]
|
||||
assert out[0]["href"] == "/ups"
|
||||
|
||||
|
||||
async def test_the_simulator_on_its_own_produces_exactly_that_warning() -> None:
|
||||
"""The task's acceptance line, through the real simulator and the real payload builder."""
|
||||
readings = await NutSimulator().read()
|
||||
payload = {"ts": 1_000.0, "simulated": True,
|
||||
"ups": [nut.payload(reading) for reading in readings]}
|
||||
out = alerts.build(SETTINGS,
|
||||
[state(key="nut", dashboard="ups", label="NUT server")], [], [], payload)
|
||||
assert [item["id"] for item in out] == ["ups:simulated"], out
|
||||
|
||||
|
||||
def test_a_real_upsd_does_not_claim_to_be_simulated() -> None:
|
||||
out = alerts.build(SETTINGS, [], [], [], ups_payload(ups_card("apc")))
|
||||
assert out == []
|
||||
|
||||
|
||||
def test_a_dead_upsd_is_one_alert_and_not_its_stale_cards_too() -> None:
|
||||
payload = ups_payload(ups_card("apc", check("Power path", "fail", "on battery")))
|
||||
out = alerts.build(SETTINGS, [state(key="nut", dashboard="ups", label="NUT server",
|
||||
ok=False, error="upsd closed the connection")],
|
||||
[], [], payload)
|
||||
assert [item["source"] for item in out] == ["NUT server"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# configuration
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_a_bad_target_entry_is_a_warn_that_says_it_is_not_being_polled() -> None:
|
||||
settings = Settings(host_target_problems=("nope is not name=url",
|
||||
"deskwork: url must start with http://"))
|
||||
out = alerts.build(settings)
|
||||
assert len(out) == 2
|
||||
assert all(item["severity"] == "warn" and item["source"] == "config" for item in out)
|
||||
assert all(item["href"] == "/system" for item in out)
|
||||
assert "HOST_TARGETS" in out[0]["message"] and "not being polled" in out[0]["message"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# dedupe, bound, order, shape
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_one_alert_per_source_and_metric_with_the_worst_verdict_winning() -> None:
|
||||
"""Two cards answering as the same name (a re-exported UPS) must not double the alarm."""
|
||||
payload = ups_payload(ups_card("apc", check("Charge", "warn", "60%")),
|
||||
ups_card("apc", check("Charge", "fail", "12%")))
|
||||
out = alerts.build(SETTINGS, [], [], [], payload)
|
||||
assert len(out) == 1
|
||||
assert out[0]["severity"] == "fail"
|
||||
assert out[0]["message"] == "Charge: 12%"
|
||||
|
||||
|
||||
def test_the_list_is_bounded_and_says_how_many_it_hid() -> None:
|
||||
cards = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery"))
|
||||
for i in range(alerts.MAX_ALERTS + 5)]
|
||||
out = alerts.build(SETTINGS, [], [], [], ups_payload(*cards))
|
||||
assert len(out) == alerts.MAX_ALERTS
|
||||
marker = out[-1]
|
||||
assert marker["id"] == "alerts:suppressed"
|
||||
assert marker["severity"] == "fail", "hiding reds behind an amber marker is a lie"
|
||||
hidden = len(cards) - (alerts.MAX_ALERTS - 1) # one slot is the marker itself
|
||||
assert marker["message"] == f"{hidden} more not shown ({hidden} failing, 0 warning)"
|
||||
|
||||
|
||||
def test_the_marker_is_the_worst_verdict_it_did_not_show() -> None:
|
||||
cards = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery"))
|
||||
for i in range(alerts.MAX_ALERTS)]
|
||||
cards += [ups_card(f"warn{i:02d}", check("Charge", "warn", "40%")) for i in range(3)]
|
||||
out = alerts.build(SETTINGS, [], [], [], ups_payload(*cards))
|
||||
assert len(out) == alerts.MAX_ALERTS
|
||||
assert out[-1]["id"] == "alerts:suppressed"
|
||||
assert out[-1]["severity"] == "fail", "a red is behind the marker, so the marker is red"
|
||||
# and when only warnings are hidden, the marker says amber — it describes what it hid
|
||||
mild = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery"))
|
||||
for i in range(3)]
|
||||
mild += [ups_card(f"warn{i:02d}", check("Charge", "warn", "40%"))
|
||||
for i in range(alerts.MAX_ALERTS)]
|
||||
out = alerts.build(SETTINGS, [], [], [], ups_payload(*mild))
|
||||
assert out[-1]["severity"] == "warn"
|
||||
|
||||
|
||||
def test_exactly_the_cap_adds_no_marker() -> None:
|
||||
cards = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery"))
|
||||
for i in range(alerts.MAX_ALERTS)]
|
||||
out = alerts.build(SETTINGS, [], [], [], ups_payload(*cards))
|
||||
assert len(out) == alerts.MAX_ALERTS
|
||||
assert all(item["id"] != "alerts:suppressed" for item in out)
|
||||
|
||||
|
||||
def test_fails_sort_before_warns_and_then_by_source() -> None:
|
||||
payload = ups_payload(ups_card("zeta", check("Power path", "fail", "on battery")),
|
||||
ups_card("alpha", check("Charge", "warn", "40%")),
|
||||
ups_card("beta", check("Load", "warn", "82%")))
|
||||
out = alerts.build(SETTINGS, [], [], [], payload)
|
||||
assert [(item["severity"], item["source"]) for item in out] == [
|
||||
("fail", "zeta"), ("warn", "alpha"), ("warn", "beta")]
|
||||
|
||||
|
||||
def test_the_trip_note_quotes_the_threshold_table_in_the_right_direction() -> None:
|
||||
"""`_trip` reads `severity.THRESHOLDS` back instead of carrying its own numbers.
|
||||
|
||||
Reached through `build()` only by metrics that are not inverted today, so this checks
|
||||
the helper directly: a low-is-worse metric must print `≤` (an `alert` for a battery is
|
||||
exactly where a wrong sign would be published), and a metric with no threshold gets no
|
||||
note rather than a made-up one.
|
||||
"""
|
||||
assert alerts._trip("cpu.usage_pct", "fail") == " (fail ≥90)"
|
||||
assert alerts._trip("ups.battery_runtime", "warn") == " (warn ≤600)"
|
||||
assert alerts._trip("ups.charge_pct", "fail") == " (fail ≤20)"
|
||||
assert alerts._trip("cpu.usage_pct", "ok") == ""
|
||||
assert alerts._trip("mystery.metric", "fail") == ""
|
||||
|
||||
|
||||
def test_the_alert_shape_is_the_contract_the_pages_render() -> None:
|
||||
out = alerts.build(SETTINGS, [state(ok=False, error="nope")], now=2_000.0)
|
||||
assert out[0] == {"id": "driveripper:host:driveripper:down", "severity": "fail",
|
||||
"source": "driveripper", "message": "driveripper: nope",
|
||||
"href": "/system", "at": 2_000.0}
|
||||
+490
-24
@@ -7,30 +7,38 @@ asset the network can fail to deliver.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import re
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from dashboards import alerts
|
||||
from dashboards.api import RANGE_SECONDS, parse_range
|
||||
from dashboards.app import create_app
|
||||
from dashboards.config import Settings, Target
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources.host import HostCollector, parse_host_snapshot
|
||||
from dashboards.sources import nut
|
||||
from dashboards.store import Event, Sample, SampleStore
|
||||
|
||||
# refused immediately, and never dials out of the test environment
|
||||
DEAD_URL = "http://127.0.0.1:1"
|
||||
|
||||
|
||||
@contextmanager
|
||||
def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: bool = False,
|
||||
**overrides: Any) -> TestClient:
|
||||
"""A TestClient whose lifespan runs, against a private DB and no real network."""
|
||||
**overrides: Any) -> Iterator[TestClient]:
|
||||
"""A TestClient whose lifespan runs, against a private DB and no real network.
|
||||
|
||||
The store is injected, so `create_app()` deliberately does not close it (production
|
||||
owns its own); the caller here does, or every test leaks a SQLite handle and the
|
||||
`--cov` log fills with ResourceWarnings that bury the warnings worth reading.
|
||||
"""
|
||||
if simulate:
|
||||
monkeypatch.setenv("NUT_SIMULATE", "1")
|
||||
else:
|
||||
@@ -38,10 +46,13 @@ def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: b
|
||||
db = str(tmp_path / "api.sqlite3")
|
||||
settings = Settings(db_path=db, **overrides)
|
||||
store = SampleStore(db)
|
||||
app = create_app(settings, store)
|
||||
client = TestClient(app)
|
||||
client = TestClient(create_app(settings, store))
|
||||
client.__enter__()
|
||||
return client
|
||||
try:
|
||||
yield client
|
||||
finally:
|
||||
client.__exit__(None, None, None)
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -68,29 +79,23 @@ def wait_for_payload(client: TestClient, path: str, *, timeout: float = 5.0) ->
|
||||
pytest.fail(f"{path} never became available")
|
||||
|
||||
|
||||
def host_payload(agent_snapshot: dict[str, Any]) -> dict[str, Any]:
|
||||
def failing_host_payload(scrape: Any, agent_snapshot: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Two scrapes of the real driveripper capture, nudged onto known sides of the table.
|
||||
|
||||
Driven from the fixture rather than hand-written so a schema drift breaks here first.
|
||||
"""
|
||||
snap = copy.deepcopy(agent_snapshot)
|
||||
def nudge(snap: dict[str, Any]) -> None:
|
||||
snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95 # mem.used_pct -> fail
|
||||
snap["cpu_temperature_celsius"] = 91.0 # cpu.temp_c -> fail
|
||||
for fs in snap["filesystems"]:
|
||||
fs["used_pct"] = 93.0 # disk -> fail
|
||||
collector = HostCollector(RateEstimator(span=60.0))
|
||||
parse_host_snapshot("driveripper", snap, collector, now=1_000.0)
|
||||
second = copy.deepcopy(snap)
|
||||
second["ts"] = 1_010.0
|
||||
modes = second["cpu"]["modes"]
|
||||
modes["total"] += 1_000.0
|
||||
modes["idle"] += 900.0 # 10 % busy over the window -> ok
|
||||
return parse_host_snapshot("driveripper", second, collector, now=1_010.0).payload
|
||||
|
||||
return scrape(agent_snapshot, cpu_busy=10.0, mutate=nudge)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
agent_snapshot: dict[str, Any]) -> Iterator[TestClient]:
|
||||
agent_snapshot: dict[str, Any], host_scrape: Any) -> Iterator[TestClient]:
|
||||
"""Overview-shaped app: one host payload in the hub plus the simulated UPSes.
|
||||
|
||||
The host job points at a dead port, so a poll can never overwrite the injected payload
|
||||
@@ -104,7 +109,8 @@ def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
test_client = TestClient(app)
|
||||
test_client.__enter__()
|
||||
try:
|
||||
app.state.ctx.hub.jobs["host:driveripper"].state.payload = host_payload(agent_snapshot)
|
||||
app.state.ctx.hub.jobs["host:driveripper"].state.payload = failing_host_payload(
|
||||
host_scrape, agent_snapshot)
|
||||
wait_for_payload(test_client, "/api/ups/snapshot")
|
||||
yield test_client
|
||||
finally:
|
||||
@@ -112,6 +118,41 @@ def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def healthy_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
agent_snapshot: dict[str, Any],
|
||||
host_scrape: Any) -> Iterator[TestClient]:
|
||||
"""The all-clear app: one host answering, nothing past a threshold, no UPS configured.
|
||||
|
||||
The job's collector is swapped for one that hands back the injected payload, so the
|
||||
health record says `ok` truthfully instead of being overridden behind the poller.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
|
||||
from dashboards.poller import Result # noqa: PLC0415 - only this fixture needs it
|
||||
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=10.0)
|
||||
|
||||
async def collect() -> Result:
|
||||
return Result(payload=payload, samples=[])
|
||||
|
||||
job = test_client.app.state.ctx.hub.jobs["host:driveripper"] # type: ignore[attr-defined]
|
||||
job.fn = collect
|
||||
# let the poll already in flight against the dead port land first (it fails in
|
||||
# microseconds), then take the health record over: the alerts must describe a host
|
||||
# that is answering, not the half-second of startup where it was not
|
||||
deadline = time.time() + 2.0
|
||||
while time.time() < deadline and job.state.attempts == 0:
|
||||
time.sleep(0.02)
|
||||
time.sleep(0.05)
|
||||
job.state.payload = payload
|
||||
job.state.ok = True
|
||||
job.state.error = None
|
||||
job.state.consecutive_failures = 0
|
||||
job.state.last_ok = time.time()
|
||||
yield test_client
|
||||
|
||||
|
||||
# ---- severity travels with the value --------------------------------------- #
|
||||
|
||||
def test_summary_carries_the_verdict_for_every_column_the_page_colours(data_client) -> None:
|
||||
@@ -147,13 +188,121 @@ def test_summary_shape_did_not_shrink(data_client) -> None:
|
||||
assert fleet_keys <= set(body["system"]["fleet"]), fleet_keys - set(body["system"]["fleet"])
|
||||
assert host_keys <= set(body["system"]["hosts"][0]), host_keys - set(body["system"]["hosts"][0])
|
||||
assert ups_keys <= set(body["ups"]["ups"][0]), ups_keys - set(body["ups"]["ups"][0])
|
||||
assert set(body) == {"ts", "system", "llm", "ups", "recent_events"}
|
||||
assert set(body) == {"ts", "system", "llm", "ups", "recent_events", "alerts"}
|
||||
# ...and the new keys are there beside the old ones, not instead of them
|
||||
assert "severity" in body["system"]["fleet"]
|
||||
assert "severity" in body["system"]["hosts"][0]
|
||||
assert "severity" in body["ups"]["ups"][0]
|
||||
|
||||
|
||||
# `/` paints two panels; the payload is still the whole-fleet contract. Every path here is
|
||||
# read by something that is not the overview page: the live suite, an external probe, a
|
||||
# notifier later. Deleting a card must never delete a key.
|
||||
SUMMARY_CONTRACT = ("system.enabled", "system.fleet", "system.hosts",
|
||||
"llm.engines", "ups.ups", "recent_events", "alerts")
|
||||
|
||||
|
||||
def dig(body: dict[str, Any], path: str) -> Any:
|
||||
"""`body["a"]["b"]` that names the missing path when it is not there."""
|
||||
node: Any = body
|
||||
for part in path.split("."):
|
||||
assert isinstance(node, dict) and part in node, f"/api/summary lost `{path}`"
|
||||
node = node[part]
|
||||
return node
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fixture", ["client", "data_client"])
|
||||
def test_summary_serves_every_path_another_consumer_reads(request, fixture: str) -> None:
|
||||
"""The keys survive the overview cleanup whether or not anything is configured.
|
||||
|
||||
Run twice on purpose: `client` has no targets at all, so it proves the keys are present
|
||||
as `None`/`[]` rather than omitted when a source is unconfigured (invariant 3 is about
|
||||
the payload shape as much as about colour); `data_client` proves the same paths carry
|
||||
the fleet once there is one.
|
||||
"""
|
||||
test_client = request.getfixturevalue(fixture)
|
||||
body = test_client.get("/api/summary").json()
|
||||
populated = fixture == "data_client"
|
||||
for path in SUMMARY_CONTRACT:
|
||||
dig(body, path) # presence is the assertion; a missing path raises with its name
|
||||
assert dig(body, "system.enabled") is populated
|
||||
assert (dig(body, "system.fleet") is not None) is populated
|
||||
names = {host["name"] for host in dig(body, "system.hosts")}
|
||||
assert names == ({"driveripper"} if populated else set())
|
||||
batteries = {item["name"] for item in dig(body, "ups.ups")}
|
||||
assert batteries == ({"apc", "cyberpower"} if populated else set())
|
||||
assert dig(body, "llm.engines") == [], "neither fixture configures an engine"
|
||||
assert isinstance(dig(body, "recent_events"), list)
|
||||
assert isinstance(dig(body, "alerts"), list)
|
||||
# and the payload as a whole still carries every top-level section it always did
|
||||
assert set(body) >= {"ts", "system", "llm", "ups", "recent_events", "alerts"}
|
||||
|
||||
|
||||
# ---- alerts: the landing page's answer to "what is wrong" ------------------- #
|
||||
|
||||
def test_summary_is_empty_of_alerts_when_everything_is_healthy(healthy_client) -> None:
|
||||
"""`[]`, not a green banner: an empty list is the clearest possible all-clear."""
|
||||
body = healthy_client.get("/api/summary").json()
|
||||
assert body["alerts"] == []
|
||||
assert body["system"]["fleet"]["hosts_live"] == 1
|
||||
|
||||
|
||||
def test_summary_raises_one_alert_for_the_host_that_stopped_answering(data_client) -> None:
|
||||
"""One dead host is one line naming it, with the link to the page that explains it."""
|
||||
body = data_client.get("/api/summary").json()
|
||||
down = [item for item in body["alerts"] if item["source"] == "driveripper"]
|
||||
assert len(down) == 1, "its stale payload must not add amber lines on top of the red one"
|
||||
assert down[0]["severity"] == "fail"
|
||||
assert down[0]["href"] == "/system"
|
||||
assert "unreachable" in down[0]["message"]
|
||||
|
||||
|
||||
def test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A dead agent is a red row in the fleet, not a fleet that quietly got smaller.
|
||||
|
||||
`/api/system/snapshot` already stubs a job with no payload; `/api/summary` has to agree
|
||||
with it, or a target that never answered since startup leaves the table at "2/2 hosts"
|
||||
while the panel above says a third box is down (`AGENTS.md` invariants 3 and 4).
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
|
||||
body = test_client.get("/api/summary").json()
|
||||
rows = body["system"]["hosts"]
|
||||
assert [row["name"] for row in rows] == ["driveripper"]
|
||||
assert rows[0]["unreachable"] is True
|
||||
assert rows[0]["cpu_usage_pct"] is None, "n/a, never a 0.0 nobody measured"
|
||||
assert body["system"]["fleet"]["hosts_total"] == 1
|
||||
assert body["system"]["fleet"]["hosts_live"] == 0
|
||||
# and the alert list stays exactly one line for it, not a row plus a stale payload
|
||||
assert [item["source"] for item in body["alerts"]] == ["driveripper"]
|
||||
|
||||
|
||||
def test_summary_alerts_are_worst_first_and_carry_the_contract(data_client) -> None:
|
||||
body = data_client.get("/api/summary").json()
|
||||
alerts_list = body["alerts"]
|
||||
assert alerts_list, "a dead host plus simulated UPS data cannot read as all-clear"
|
||||
for item in alerts_list:
|
||||
assert set(item) == {"id", "severity", "source", "message", "href", "at"}
|
||||
assert item["severity"] in ("warn", "fail")
|
||||
ranks = [0 if item["severity"] == "fail" else 1 for item in alerts_list]
|
||||
assert ranks == sorted(ranks), "fails before warns"
|
||||
assert any("simulat" in item["message"].lower() and item["href"] == "/ups"
|
||||
for item in alerts_list), "NUT_SIMULATE must be said out loud on the landing page"
|
||||
|
||||
|
||||
def test_summary_reuses_the_alert_builder_verbatim(data_client) -> None:
|
||||
"""The endpoint is a thin call: what `alerts.build` says over the same state is the body."""
|
||||
body = data_client.get("/api/summary").json()
|
||||
hub = data_client.app.state.ctx.hub # type: ignore[attr-defined]
|
||||
settings = data_client.app.state.ctx.settings # type: ignore[attr-defined]
|
||||
hosts = [job.state.payload for job in hub.jobs.values()
|
||||
if job.dashboard == "system" and job.state.payload]
|
||||
ups_payload = hub.jobs["nut"].state.payload
|
||||
expected = alerts.build(settings, hub.states(), hosts, [], ups_payload, [])
|
||||
assert [item["id"] for item in body["alerts"]] == [item["id"] for item in expected]
|
||||
|
||||
|
||||
def test_snapshot_payloads_carry_their_own_severity_sections(data_client) -> None:
|
||||
"""/api/system/snapshot and /api/ups/snapshot ship the same verdicts the summary reuses."""
|
||||
system = data_client.get("/api/system/snapshot").json()
|
||||
@@ -371,14 +520,103 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
|
||||
|
||||
|
||||
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
|
||||
"""The old call sites still work until phases 02-05 convert each page."""
|
||||
"""The old call sites still work until the remaining phases convert each page.
|
||||
|
||||
`/` was converted in phase 02: the overview colours from the `severity` verdicts
|
||||
`/api/summary` already sends, so it no longer carries a threshold of its own. The
|
||||
three pages still to come must keep compiling against the fallback until they are.
|
||||
"""
|
||||
js = client.get("/static/app.js").text
|
||||
assert "FALLBACK ONLY" in js
|
||||
for page in ("/static/page-index.js", "/static/page-system.js",
|
||||
"/static/page-llm.js", "/static/page-ups.js"):
|
||||
for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"):
|
||||
assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site"
|
||||
|
||||
|
||||
def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> None:
|
||||
"""`page-index.js` reads verdicts; it does not compare a number to a bound."""
|
||||
js = client.get("/static/page-index.js").text
|
||||
assert "severityFrom" in js
|
||||
assert "severityFor" not in js
|
||||
|
||||
|
||||
def test_overview_is_exactly_two_panels(client) -> None:
|
||||
"""The landing page answers one question: what is wrong, and what is the fleet doing.
|
||||
|
||||
Asserted on the served bytes because there is no JS test runner (a locked anchor):
|
||||
the summary cards and the raw event feed are what made this page busy, and once a
|
||||
panel is gone a browser is the only thing that would notice its return.
|
||||
"""
|
||||
html = client.get("/").text
|
||||
assert html.count("<section") == 2, "the overview grew a third panel"
|
||||
assert 'id="alerts"' in html and 'id="alerts-clear"' in html
|
||||
assert 'aria-live="polite"' in html
|
||||
assert 'id="fleet-table"' in html
|
||||
for gone in ("sys-cpu", "sys-mem", "sys-disk", "sys-gpus", "llm-decode", "llm-kv",
|
||||
"ups-list", 'id="events"'):
|
||||
assert gone not in html, f"the overview still renders {gone}"
|
||||
# the header's source dots are chrome owned by app.js, not the deleted card dots
|
||||
assert 'id="dot-system"' not in html and 'id="nav-status"' in html
|
||||
|
||||
|
||||
def test_overview_alert_rows_link_to_the_dashboard_that_explains_them(client) -> None:
|
||||
"""Every alert carries the `href` its row turns into a link; the page invents none."""
|
||||
page_js = client.get("/static/page-index.js").text
|
||||
assert "alert.href" in page_js
|
||||
css = client.get("/static/dash.css").text
|
||||
for selector in ("ul.alerts li.ok .src", "ul.alerts li.warn .src", "ul.alerts li.fail .src",
|
||||
"table.data td.warn", "table.data td.fail"):
|
||||
assert selector in css, f"dash.css lost {selector}"
|
||||
# the swap that puts a live problem above the fleet table
|
||||
assert 'data-state="alerting"' in css
|
||||
|
||||
|
||||
def test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering(client) -> None:
|
||||
"""A dead host keeps its last good numbers, so the row has to say they are stale.
|
||||
|
||||
`AGENTS.md` invariant 4: keep the payload and mark it stale, never zero it. The
|
||||
overview's table is the one place those last good numbers still print, so it reads the
|
||||
per-source verdicts `/api/summary` sends (`system.states`) and applies the shared
|
||||
`.stale` class the other pages already use — no new palette, no JS threshold.
|
||||
"""
|
||||
page_js = client.get("/static/page-index.js").text
|
||||
assert "system.states" in page_js, "the fleet table stopped reading reachability"
|
||||
assert 'class="stale"' in page_js
|
||||
assert ".stale" in client.get("/static/dash.css").text
|
||||
|
||||
|
||||
# Which script owns which page, for the orphan check below. `charts.js` and `app.js` are
|
||||
# loaded by all four pages, so an id either page code or shared chrome may read.
|
||||
PAGE_SCRIPTS = {
|
||||
"/": "page-index.js",
|
||||
"/system": "page-system.js",
|
||||
"/llm": "page-llm.js",
|
||||
"/ups": "page-ups.js",
|
||||
}
|
||||
SHARED_SCRIPTS = ("app.js", "charts.js")
|
||||
|
||||
|
||||
def test_no_page_carries_an_id_nothing_reads(client) -> None:
|
||||
"""An id no script queries, no stylesheet styles and no attribute points at is dead markup.
|
||||
|
||||
The summary cards were retired this way — a `<span id="sys-cpu">` left behind after its
|
||||
writer was deleted is the same mistake in slower motion. Asserted on served bytes: the
|
||||
browser console is otherwise the only thing that would notice.
|
||||
"""
|
||||
scripts = {name: client.get(f"/static/{name}").text for name in
|
||||
(*SHARED_SCRIPTS, *PAGE_SCRIPTS.values(), "dash.css")}
|
||||
for path, own_script in PAGE_SCRIPTS.items():
|
||||
html = _code(client.get(path).text)
|
||||
corpus = "".join(_code(scripts[name])
|
||||
for name in (*SHARED_SCRIPTS, own_script, "dash.css"))
|
||||
# ids the markup uses itself: labels, targets, `<label for>`
|
||||
anchors = set(re.findall(
|
||||
r'(?:aria-labelledby|aria-describedby|aria-controls|for)="([^"]+)"', html))
|
||||
for element_id in re.findall(r'id="([^"]+)"', html):
|
||||
quoted = (f'"{element_id}"' in corpus or f"'{element_id}'" in corpus
|
||||
or f"#{element_id}" in corpus or element_id in anchors)
|
||||
assert quoted, f"{path} has id=\"{element_id}\" that nothing reads"
|
||||
|
||||
|
||||
def test_api_responses_are_not_cacheable(client) -> None:
|
||||
for path in ("/api/config", "/api/health", "/api/summary"):
|
||||
assert "no-store" in client.get(path).headers.get("cache-control", "")
|
||||
@@ -413,6 +651,87 @@ def test_history_requires_the_series_coordinates(client) -> None:
|
||||
assert client.get("/api/history").status_code == 422
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("raw", "seconds"), [
|
||||
*[(key, float(seconds)) for key, seconds in RANGE_SECONDS.items()],
|
||||
("45s", 45.0), ("13h", 46_800.0), ("7200", 7_200.0), (" 2D ", 172_800.0),
|
||||
(None, 3_600.0), ("", 3_600.0),
|
||||
])
|
||||
def test_parse_range_reads_every_window_the_ui_offers(raw: Any, seconds: float) -> None:
|
||||
"""Every chip key (5m…90d), any `<n><s|m|h|d>`, raw seconds, and the 1h default.
|
||||
|
||||
`parse_range` is the only thing between a typo'd query string and a chart of the wrong
|
||||
window, and a wrong window is worse than an error: it still looks like data.
|
||||
"""
|
||||
assert parse_range(raw) == seconds
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw", ["banana", "1w", "h", "-4h", "one"])
|
||||
def test_parse_range_refuses_a_window_it_cannot_interpret(raw: str) -> None:
|
||||
"""A 400 that names the value, never a silent fallback to 1h."""
|
||||
with pytest.raises(HTTPException) as excinfo:
|
||||
parse_range(raw)
|
||||
assert excinfo.value.status_code == 400
|
||||
assert raw in str(excinfo.value.detail)
|
||||
|
||||
|
||||
# ---- the routes the *other* pages chart from -------------------------------- #
|
||||
#
|
||||
# `/` renders two panels now; `/system`, `/llm` and `/ups` still need these, and so does
|
||||
# anything outside the browser. Retiring overview code must not retire them, so the shapes
|
||||
# are pinned here rather than discovered by a page three phases from now.
|
||||
|
||||
def test_system_history_charts_every_configured_host_in_one_request(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""N hosts x M metrics, one round trip — the chart endpoint's whole reason to exist."""
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
host_targets=(Target("driveripper", DEAD_URL),
|
||||
Target("deskwork", DEAD_URL))) as test_client:
|
||||
body = test_client.get("/api/system/history",
|
||||
params={"metrics": "cpu.usage_pct,mem.used_pct",
|
||||
"range": "1h"}).json()
|
||||
assert body["hosts"] == ["driveripper", "deskwork"]
|
||||
assert body["range_seconds"] == 3_600.0
|
||||
assert len(body["series"]) == 4
|
||||
assert {(item["host"], item["metric"]) for item in body["series"]} == {
|
||||
("driveripper", "cpu.usage_pct"), ("driveripper", "mem.used_pct"),
|
||||
("deskwork", "cpu.usage_pct"), ("deskwork", "mem.used_pct")}
|
||||
# an explicit `hosts` list narrows it instead of adding to the default
|
||||
narrowed = test_client.get("/api/system/history",
|
||||
params={"metrics": "cpu.usage_pct",
|
||||
"hosts": "deskwork"}).json()
|
||||
assert narrowed["hosts"] == ["deskwork"]
|
||||
|
||||
|
||||
def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
|
||||
# a host target too, so the snapshot's job filter has to skip a non-LLM job
|
||||
with build_client(tmp_path, monkeypatch, vllm_url=DEAD_URL,
|
||||
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
|
||||
body = test_client.get("/api/llm/snapshot").json()
|
||||
assert [engine["engine"] for engine in body["engines"]] == ["vLLM"]
|
||||
assert body["engines"][0]["unreachable"] is True
|
||||
assert body["fleet"]["engines_total"] == 1
|
||||
assert body["fleet"]["engines_live"] == 0
|
||||
assert [state["key"] for state in body["states"]] == ["llm:vllm"]
|
||||
|
||||
|
||||
def test_llm_history_charts_both_engines_by_default(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
|
||||
vllm_url=DEAD_URL) as test_client:
|
||||
body = test_client.get("/api/llm/history",
|
||||
params={"metrics": "decode_tokens_per_second,kv_cache_usage_ratio",
|
||||
"range": "15m"}).json()
|
||||
assert body["engines"] == ["gufo", "vllm"]
|
||||
assert body["range_seconds"] == 900.0
|
||||
assert {(item["host"], item["metric"]) for item in body["series"]} == {
|
||||
("gufo", "llm.decode_tokens_per_second"),
|
||||
("gufo", "llm.kv_cache_usage_ratio"),
|
||||
("vllm", "llm.decode_tokens_per_second"),
|
||||
("vllm", "llm.kv_cache_usage_ratio")}
|
||||
|
||||
|
||||
def test_metrics_endpoint_lists_what_actually_has_history(client) -> None:
|
||||
body = client.get("/api/metrics").json()
|
||||
assert body["metrics"] == []
|
||||
@@ -474,6 +793,153 @@ def test_a_dangerous_command_needs_the_confirmation_word(ups_client) -> None:
|
||||
json={"ups": name, "command": deep, "confirm": "DANGER"}).status_code == 200
|
||||
|
||||
|
||||
def test_a_command_the_driver_does_not_advertise_is_refused_before_it_is_sent(
|
||||
ups_client) -> None:
|
||||
"""Being in the risk catalog is not permission; `LIST CMD` from the driver is (invariant 1).
|
||||
|
||||
`shutdown.return` is a real NUT command the catalog knows and this UPS does not offer,
|
||||
so the refusal must come from the advertisement check and not from the confirmation gate.
|
||||
"""
|
||||
name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
|
||||
response = ups_client.post("/api/ups/command",
|
||||
json={"ups": name, "command": "shutdown.return"})
|
||||
assert response.status_code == 400
|
||||
assert "does not advertise" in response.json()["detail"]
|
||||
|
||||
|
||||
def test_cutting_the_output_is_refused_while_the_ups_is_already_on_battery(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Invariant 1, the one that matters most: no power cut while the pack is carrying the load.
|
||||
|
||||
Driven through the simulator's own state machine — a quick battery test puts the pack on
|
||||
battery — so the 409 comes from the same poll the buttons are drawn from, with a 0.5s
|
||||
interval to keep the wait short. Nothing here can touch real hardware: `NUT_SIMULATE=1`.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
|
||||
nut_allow_actions=True, nut_interval=0.5) as test_client:
|
||||
name = wait_for_payload(test_client, "/api/ups/snapshot")["ups"][0]["name"]
|
||||
start = test_client.post("/api/ups/command",
|
||||
json={"ups": name, "command": "test.battery.start.quick"})
|
||||
assert start.status_code == 200
|
||||
deadline = time.time() + 6.0
|
||||
card: dict[str, Any] = {}
|
||||
while time.time() < deadline:
|
||||
card = next(item for item in test_client.get("/api/ups/snapshot").json()["ups"]
|
||||
if item["name"] == name)
|
||||
if card["on_battery"]:
|
||||
break
|
||||
time.sleep(0.05)
|
||||
assert card.get("on_battery") is True, "the quick test must put the simulated pack on battery"
|
||||
refusal = test_client.post("/api/ups/command",
|
||||
json={"ups": name, "command": "load.off",
|
||||
"confirm": "DANGER"})
|
||||
assert refusal.status_code == 409
|
||||
assert "on battery" in refusal.json()["detail"]
|
||||
# and the same command is still offered once the pack is back on line power
|
||||
assert not any(item["command"] == "load.off" and item["enabled"]
|
||||
for item in card["commands"]), "the panel must not offer a blocked button"
|
||||
|
||||
|
||||
class ScriptedNutClient:
|
||||
"""`nut.NutClient` with the socket left out: the real code path, no battery attached.
|
||||
|
||||
`AGENTS.md` keeps live tests read-only, so this is where the one write endpoint's real
|
||||
branch gets proven. The variable names are the ones `VAR_MAP` already pins from real
|
||||
upsd captures; only the replies are scripted (a fake for behaviour, per the test policy).
|
||||
"""
|
||||
|
||||
#: set per test: an exception instance to raise from `instcmd`, e.g. a refusal
|
||||
refusal: Exception | None = None
|
||||
sent: list[tuple[str, str]]
|
||||
logged_in: list[str]
|
||||
|
||||
def __init__(self, host: str, port: int, *, timeout: float = 4.0,
|
||||
username: str | None = None, password: str | None = None) -> None:
|
||||
self.credentials = (username, password)
|
||||
|
||||
async def __aenter__(self) -> ScriptedNutClient:
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *_exc: object) -> None:
|
||||
return None
|
||||
|
||||
async def list_ups(self) -> list[dict[str, str]]:
|
||||
return [{"name": "apc", "description": "APC Back-UPS RS 1500MS"}]
|
||||
|
||||
async def list_vars(self, ups: str) -> dict[str, str]:
|
||||
return {"ups.mfr": "APC", "ups.model": "Back-UPS RS 1500MS",
|
||||
"ups.status": "OL", "battery.charge": "100", "ups.load": "18",
|
||||
"battery.voltage": "13.6", "input.voltage": "230",
|
||||
"battery.mfr.date": "2022-04", "ups.test.result": "Ok last test passed"}
|
||||
|
||||
async def list_cmds(self, ups: str) -> list[str]:
|
||||
return ["test.panel.start", "beeper.mute", "load.off"]
|
||||
|
||||
async def list_rw_vars(self, ups: str) -> list[str]:
|
||||
return []
|
||||
|
||||
async def login(self) -> None:
|
||||
assert self.credentials[0], "a command send must authenticate as a named user"
|
||||
ScriptedNutClient.logged_in.append(str(self.credentials[0]))
|
||||
|
||||
async def instcmd(self, ups: str, command: str) -> str:
|
||||
if ScriptedNutClient.refusal is not None:
|
||||
raise ScriptedNutClient.refusal
|
||||
ScriptedNutClient.sent.append((ups, command))
|
||||
return "OK Start Panel Test"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def real_upsd_client(monkeypatch: pytest.MonkeyPatch) -> Iterator[type[ScriptedNutClient]]:
|
||||
"""Swap the module-level `NutClient` so the collector and the send use the same stand-in."""
|
||||
ScriptedNutClient.sent = []
|
||||
ScriptedNutClient.logged_in = []
|
||||
ScriptedNutClient.refusal = None
|
||||
monkeypatch.setattr(nut, "NutClient", ScriptedNutClient)
|
||||
yield ScriptedNutClient
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def nut_client_app(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
real_upsd_client: type[ScriptedNutClient]) -> Iterator[TestClient]:
|
||||
"""A non-simulated UPS app: real code paths, scripted upsd, 0.5s polls."""
|
||||
with build_client(tmp_path, monkeypatch, nut_host="127.0.0.1", nut_user="dashboards",
|
||||
nut_password="not-the-real-one", nut_allow_actions=True,
|
||||
nut_interval=0.5) as test_client:
|
||||
yield test_client
|
||||
|
||||
|
||||
def test_a_real_upsd_send_authenticates_sends_and_audits(nut_client_app,
|
||||
real_upsd_client) -> None:
|
||||
body = wait_for_payload(nut_client_app, "/api/ups/snapshot")
|
||||
assert body["simulated"] is False, "this test is about the branch that talks to upsd"
|
||||
name = body["ups"][0]["name"]
|
||||
response = nut_client_app.post("/api/ups/command",
|
||||
json={"ups": name, "command": "test.panel.start"})
|
||||
assert response.status_code == 200, response.text
|
||||
assert response.json()["simulated"] is False
|
||||
assert real_upsd_client.sent == [(name, "test.panel.start")]
|
||||
assert real_upsd_client.logged_in, "instcmd requires a LOGIN first (upsd refuses otherwise)"
|
||||
events = nut_client_app.get("/api/events").json()["events"]
|
||||
assert any(event["kind"] == "action" and name in event["message"] for event in events)
|
||||
|
||||
|
||||
def test_a_refusal_from_upsd_is_a_502_and_an_error_event(nut_client_app,
|
||||
real_upsd_client) -> None:
|
||||
real_upsd_client.refusal = nut.NutError("Access denied", code="ACCESS-DENIED")
|
||||
name = wait_for_payload(nut_client_app, "/api/ups/snapshot")["ups"][0]["name"]
|
||||
response = nut_client_app.post("/api/ups/command",
|
||||
json={"ups": name, "command": "beeper.mute"})
|
||||
assert response.status_code == 502
|
||||
detail = response.json()["detail"]
|
||||
assert "ACCESS-DENIED" in detail or "Access denied" in detail, detail
|
||||
assert real_upsd_client.sent == [], "a refused command was never accepted"
|
||||
events = nut_client_app.get("/api/events").json()["events"]
|
||||
failures = [event for event in events if event["kind"] == "action"
|
||||
and event["severity"] == "error"]
|
||||
assert failures, "an upsd refusal has to leave a trace, not just an HTTP status"
|
||||
|
||||
|
||||
def test_unknown_commands_are_refused_even_with_actions_on(ups_client) -> None:
|
||||
name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
|
||||
for command in ("load.format", "rm -rf /", "INSTCMD x y"):
|
||||
|
||||
@@ -490,6 +490,19 @@ async def test_simulated_ups_cards_carry_severity_too() -> None:
|
||||
assert card["severity"]["charge_pct"] == "ok", "the sim starts at float charge"
|
||||
|
||||
|
||||
async def test_a_fresh_simulator_reads_healthy_so_amber_still_means_something() -> None:
|
||||
"""The dev model must not train people to ignore the UPS badge.
|
||||
|
||||
Its battery dates are fiction, so they are computed relative to today: a hard-coded
|
||||
2019 manufacture date turns every simulated card amber forever, and an amber nobody
|
||||
can clear is the same failure `assess()` avoids for a UPS that floats at 90 %.
|
||||
"""
|
||||
for ups in await NutSimulator().read():
|
||||
card = nut.payload(ups)
|
||||
assert card["health"]["overall"] == "healthy", card["health"]["checks"]
|
||||
assert not [c for c in card["health"]["checks"] if c["status"] == "fail"]
|
||||
|
||||
|
||||
def test_history_samples_only_include_real_numbers() -> None:
|
||||
values = {key: value for key, value in VAR_ROWS.items() if key != "battery.runtime"}
|
||||
samples = nut.history_samples(nut.UpsReading(name="cyberpower", vars=values, commands=[]))
|
||||
|
||||
Reference in New Issue
Block a user