diff --git a/.agents/phases/todo/02_overview_fleet_and_alerts/00_phase.md b/.agents/phases/complete/02_overview_fleet_and_alerts/00_phase.md similarity index 100% rename from .agents/phases/todo/02_overview_fleet_and_alerts/00_phase.md rename to .agents/phases/complete/02_overview_fleet_and_alerts/00_phase.md diff --git a/.agents/phases/todo/02_overview_fleet_and_alerts/01_alerts_builder.md b/.agents/phases/complete/02_overview_fleet_and_alerts/01_alerts_builder.md similarity index 100% rename from .agents/phases/todo/02_overview_fleet_and_alerts/01_alerts_builder.md rename to .agents/phases/complete/02_overview_fleet_and_alerts/01_alerts_builder.md diff --git a/.agents/phases/todo/02_overview_fleet_and_alerts/02_rewrite_overview_page.md b/.agents/phases/complete/02_overview_fleet_and_alerts/02_rewrite_overview_page.md similarity index 100% rename from .agents/phases/todo/02_overview_fleet_and_alerts/02_rewrite_overview_page.md rename to .agents/phases/complete/02_overview_fleet_and_alerts/02_rewrite_overview_page.md diff --git a/.agents/phases/todo/02_overview_fleet_and_alerts/03_retire_unused_overview_code.md b/.agents/phases/complete/02_overview_fleet_and_alerts/03_retire_unused_overview_code.md similarity index 100% rename from .agents/phases/todo/02_overview_fleet_and_alerts/03_retire_unused_overview_code.md rename to .agents/phases/complete/02_overview_fleet_and_alerts/03_retire_unused_overview_code.md diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.err b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.md b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.md new file mode 100644 index 0000000..7f94d81 --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.md @@ -0,0 +1,12 @@ +**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.** + +- Verified: `/` = exactly 2 `
`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live). +- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`. +- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`. +- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`. +- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**. +- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`. +- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅. +- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent. +- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward. +- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`. diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.md.status b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.validate b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.validate new file mode 100644 index 0000000..ef7dd13 --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__00_phase.a1.validate @@ -0,0 +1,46 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 16%] +........................................................................ [ 32%] +........................................................................ [ 48%] +.......ssssss........................................................... [ 64%] +........................................................................ [ 80%] +........................................................................ [ 97%] +............. [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 175 0 100% +src/dashboards/api.py 193 0 100% +src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131 +src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115 +src/dashboards/config.py 110 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 139 1 99% 226 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/host.py 199 2 99% 122, 147 +src/dashboards/sources/llm.py 150 1 99% 226 +src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617 +src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226 +src/dashboards/store.py 178 2 99% 243-244 +------------------------------------------------------------------ +TOTAL 2104 67 97% +Required test coverage of 94% reached. Total coverage: 96.82% +439 passed, 6 skipped, 1 warning in 43.39s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.err b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.md b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.md new file mode 100644 index 0000000..3f6e142 --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.md @@ -0,0 +1,17 @@ +**Implemented** +- `src/dashboards/alerts.py` (new, 100% covered): pure `build(settings, hub_states, host_payloads, engine_payloads, ups_payload, recent_events, *, now)` → `[{id, severity, source, message, href, at}]`, fail→warn then by source. Rules: unconfigured never alerts; `ok=false` → `fail` (collectors' name prefix de-duplicated, `at` from the poller's `unreachable` event); `ok` but older than 3× its own poll interval → `warn`; host/engine/UPS threshold breaches read only phase-01 `severity` dicts + `health.checks[]` (no new thresholds; bound quoted back from `severity.THRESHOLDS`/`INVERTED`); degraded-agent sections → `warn`; `HOST_TARGETS` problems → `warn`; dedupe by (source, metric) keeping the worst verdict; capped at `MAX_ALERTS=12` with a marker stating how many were hidden; a dead source yields exactly one line (stale payload suppressed). +- `src/dashboards/api.py`: `/api/summary` calls it and adds `"alerts"`; `system`/`llm`/`ups`/`recent_events` byte-identical otherwise (additive-only). +- Tests: `tests/test_alerts.py` (40 tests incl. all-clear `[]`, unconfigured, stale-vs-dead, per-source intervals, KV pressure, UPS detail strings, simulated-only, dedupe/cap/sort/shape), 4 new `/api/summary` tests + `healthy_client` fixture in `test_api.py`, shared `host_scrape` fixture in `conftest.py`. +- Defects fixed in passing: `nut_sim` hard-coded `battery.mfr.date` (2019/2022) aged into permanent amber → computed relative to today (+ test); `build_client()` leaked its `SampleStore` (28 ResourceWarnings under `--cov`) → now a context manager that closes it. + +**Gate** (`bash .agents/validate.sh`, exit 0): `uv run pytest --cov --cov-fail-under=94` → **400 passed, 6 skipped, 1 warning; total coverage 95.18%**, `alerts.py` 100% (changed part of `api.py` fully covered); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; repo hygiene clean. `uv run pytest -m live --live` → 6 passed (unchanged). Dev server restarted: all 6 sources `ok`; `curl :8090/api/summary` keys `['alerts','llm','recent_events','system','ts','ups']` with live alerts (e.g. `fail deskwork /system deskwork: GPU utilisation 100% … (fail ≥95)`, `warn apc /ups Battery age: 5.3 years — past the 4-year replacement window`). + +**Docs**: AGENTS.md — `alerts.py` responsibility bullet, `/api/summary` additive note, nut_sim date behaviour, refreshed test/coverage counts (400 / 95.2%); README.md — `alerts.py` bullet in Architecture. + +**Decisions / deviations** +- LLM alerts come from the cells `llm.SEVERITY_CELLS` already rates (KV pressure only) so no thresholds are invented here; `engine_payloads` therefore has a real purpose beyond reachability. +- The cap keeps the list at exactly `MAX_ALERTS` by replacing the last slot with an `alerts:suppressed` marker whose severity is the worst item it hid (tested both directions). +- Live fleet today is *not* all-clear: the real UPSes emit `warn` checks (battery age, "No test initiated"), so the `[]`-when-healthy and "NUT_SIMULATE=1 → exactly the simulated warning" criteria are asserted hermetically instead; I did not change `nut.assess()` semantics. +- No live infra change was made (no tunnel was killed); "one dead host → exactly one alert with `/system`" is covered by `test_summary_raises_one_alert_for_the_host_that_stopped_answering`, and the browser proof belongs to task 02. + +**Next pending task**: `.agents/phases/todo/02_overview_fleet_and_alerts/02_rewrite_overview_page.md` (reduce `index.html` + `page-index.js` to fleet table + alerts panel). diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.md.status b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.validate b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.validate new file mode 100644 index 0000000..1f7073c --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__01_alerts_builder.a1.validate @@ -0,0 +1,45 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 17%] +........................................................................ [ 35%] +........................................ssssss.......................... [ 53%] +........................................................................ [ 70%] +........................................................................ [ 88%] +.............................................. [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 166 0 100% +src/dashboards/api.py 193 28 85% 60-61, 63, 294-300, 317-325, 332-336, 420, 425-426, 432-443 +src/dashboards/app.py 109 7 94% 50-51, 56, 58, 60, 97, 131 +src/dashboards/collectors.py 140 9 94% 78-79, 92-93, 97-98, 115, 232, 236 +src/dashboards/config.py 110 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 139 1 99% 226 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/host.py 199 2 99% 122, 147 +src/dashboards/sources/llm.py 150 1 99% 226 +src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617 +src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226 +src/dashboards/store.py 178 3 98% 204, 243-244 +------------------------------------------------------------------ +TOTAL 2095 101 95% +Required test coverage of 94% reached. Total coverage: 95.18% +400 passed, 6 skipped, 1 warning in 40.23s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.err b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.md b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.md new file mode 100644 index 0000000..c0de198 --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.md @@ -0,0 +1,17 @@ +**Task 02 complete — all green.** + +**Implemented** +- `web/pages/index.html`: deleted the three summary cards, card dots, and events panel; page is now exactly two `
`s — **Warnings & errors** (`#alerts`, `aria-live="polite"`, "All clear" empty state) and **Fleet** (`#fleet-table` columns unchanged); masthead/`#nav-status`/`#clock`/`#theme-toggle`/`#config-chip`/`#banners`/`#foot-versions` kept. +- `web/static/page-index.js`: `renderEvents()` + all `#sys-*`/`#llm-*`/`#ups-list` code deleted; new `renderAlerts()` (row = verdict class via `Dash.severityFrom`, source linked via the API's `href`, message, age via `Dash.age()`) and `renderFleet()` (cells coloured from `/api/summary` row `severity` verdicts only — no JS thresholds); `#summary-line` is "3/3 hosts · 2/2 engines · 2 UPS · N warnings / all clear"; one `/api/summary` call + poll loop kept. +- `dash.css`: `ul.alerts` rows + verdict colours, `table.data td.warn/.fail`, panel swap via `#overview-panels[data-state="alerting"]` (alerts above fleet when problems exist, all-clear below when empty). +- Defect fixed in passing: `Dash.age()` printed "2$ ago" (`replace(/…$/, '$')` — lone `$` is a literal); now "2m ago". +- `tests/test_api.py`: 3 new served-HTML tests (two-panel shape, removed ids, alert href/CSS contract); fallback test updated — `/` is converted, 03–05 remain. +- Screenshots in `data/logs/` (live data): `overview-alert.png` (driveripper tunnel stopped → one red row naming it, link to `/system`, alerts panel floated above fleet) and `overview-clear.png` (live gateway-only instance, `alerts: []` → "All clear" below fleet). Tunnel restored; all 6 sources `ok`. + +**Gate**: `bash .agents/validate.sh` exit 0 — `uv run pytest --cov --cov-fail-under=94` → **403 passed, 6 skipped, 95.18%**; ruff clean; pyright 0 errors; `node --check` clean. `uv run pytest -m live --live` → 6 passed (unchanged). + +**Docs**: AGENTS.md (severityFor fallback now phases 03–05; UI-conventions paragraph pinning the two-panel overview), README.md (`web/` bullet describes the two panels), `app.js` fallback docstring. + +**Decisions**: all-clear screenshot uses a throwaway 8091 instance (gateway only) because the real UPSes permanently warn — a full live config can never show `[]`; fleet table colours only warn/fail (ok stays plain text) so a healthy table is quiet. + +**Next pending task**: `.agents/phases/todo/02_overview_fleet_and_alerts/03_retire_unused_overview_code.md`. diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.md.status b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.validate b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.validate new file mode 100644 index 0000000..ef9221b --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__02_rewrite_overview_page.a1.validate @@ -0,0 +1,45 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 17%] +........................................................................ [ 35%] +...........................................ssssss....................... [ 52%] +........................................................................ [ 70%] +........................................................................ [ 88%] +................................................. [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 166 0 100% +src/dashboards/api.py 193 28 85% 60-61, 63, 294-300, 317-325, 332-336, 420, 425-426, 432-443 +src/dashboards/app.py 109 7 94% 50-51, 56, 58, 60, 97, 131 +src/dashboards/collectors.py 140 9 94% 78-79, 92-93, 97-98, 115, 232, 236 +src/dashboards/config.py 110 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 139 1 99% 226 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/host.py 199 2 99% 122, 147 +src/dashboards/sources/llm.py 150 1 99% 226 +src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617 +src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226 +src/dashboards/store.py 178 3 98% 204, 243-244 +------------------------------------------------------------------ +TOTAL 2095 101 95% +Required test coverage of 94% reached. Total coverage: 95.18% +403 passed, 6 skipped, 1 warning in 45.80s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.err b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.md b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.md new file mode 100644 index 0000000..af0b07d --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.md @@ -0,0 +1,16 @@ +All green. Report: + +**Implemented** +- `index.html`: removed the inert `#config-chip`, `#fleet-note` and the unread `#alerts-panel`/`#fleet-panel` ids; `system.html`: dropped the unread `#f-power` id (element kept, still renders `n/a`); `app.js`: deleted the dead `getElementById('sim-flag')` + `void target` in the clock tick. +- `dash.css`: removed `.grid.wide` + its 1150px media query (the only rule the deleted fleet+events row made dead) plus `.cardset`/`.stale-flag` (duplicates of the live `.stack`/`.stale`); `.metric*`, `.events`, `.hero-bar` stay — still used/pinned. +- `api.py`: `/api/summary` docstring now states the page renders only `alerts` + `system.fleet`/`hosts`, the rest is programmatic (additive-only). +- Tests (`tests/test_api.py`): `test_summary_serves_every_path_another_consumer_reads` pins `system.enabled|fleet|hosts`, `llm.engines`, `ups.ups`, `recent_events`, `alerts` with **and** without targets (mutation-checked: renaming `llm.engines` fails it); `test_no_page_carries_an_id_nothing_reads` (orphan markup tripwire, all four pages); `parse_range` accept/refuse matrix; `/api/system/history`, `/api/llm/snapshot`, `/api/llm/history` contracts; UPS gates — driver-does-not-advertise 400, `load.off`+`DANGER` → **409 while on battery** (via the labelled simulator), and a scripted `nut.NutClient` proving login→INSTCMD→audit event and 502+error-event on refusal. +- Docs: README `/` row → “fleet + alerts in one request (`/api/summary`)”; AGENTS.md — `/api/summary` render-vs-programmatic bullet, orphan-id/CSS convention, counts (435 tests, 96.8%), `ScriptedNutClient` note. + +**Gate** — `bash .agents/validate.sh` **exit 0**: `pytest --cov --cov-fail-under=94` → **435 passed, 6 skipped, coverage 96.80%** (`api.py` 100%, `alerts.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean. `uv run pytest -m live --live` → **6 passed** (read-only). `uv pip list --outdated` empty. +**Browser** — `data/logs/retire-index.png` (live fleet, 6 alerts, alerts above Fleet, 2 `
`s, zero console/request errors) and `data/logs/retire-system.png` (Power still `n/a`). `curl :8090/api/summary` returns every pinned key. +**Grep evidence** — `grep -rn "sys-cpu\|llm-kv\|ups-list\|renderEvents" src/dashboards/web/` → *no output*; `grep -rn "cardset\|stale-flag\|grid wide" src/dashboards/web/` → *no output*. + +**Decisions** — Removed 3 pre-existing dead CSS rules that duplicated a live mechanism; left the remaining unused-but-intentional vocabulary (`.hero-bar` is pinned by phase 01's test, `.cols-3`/`.vh`/`.bar.tall`/`table.data.scroll` are components phases 03–05 adopt), and wrote that line into AGENTS.md rather than deleting rework. No live infrastructure was touched (no tunnel/host changed); the on-battery refusal is proven through the labelled simulator only. + +**Next pending task**: `.agents/phases/todo/03_system_per_host_panels/01_host_card_payload_contract.md` (this was phase 02's last task). diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.md.status b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.validate b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.validate new file mode 100644 index 0000000..93d1b2d --- /dev/null +++ b/.agents/reports/02_overview_fleet_and_alerts/02_overview_fleet_and_alerts__03_retire_unused_overview_code.a1.validate @@ -0,0 +1,46 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 16%] +........................................................................ [ 32%] +........................................................................ [ 48%] +...ssssss............................................................... [ 65%] +........................................................................ [ 81%] +........................................................................ [ 97%] +......... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/alerts.py 166 0 100% +src/dashboards/api.py 193 0 100% +src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131 +src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115 +src/dashboards/config.py 110 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 139 1 99% 226 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/host.py 199 2 99% 122, 147 +src/dashboards/sources/llm.py 150 1 99% 226 +src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617 +src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226 +src/dashboards/store.py 178 2 99% 243-244 +------------------------------------------------------------------ +TOTAL 2095 67 97% +Required test coverage of 94% reached. Total coverage: 96.80% +435 passed, 6 skipped, 1 warning in 44.68s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/AGENTS.md b/AGENTS.md index 4e46fff..68c8931 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,8 +54,10 @@ page still renders. panels that matter. 4. **One dead source must not blink another.** Each source is its own asyncio task with its own timeout and health record (`poller.py`). A failed poll keeps the last good payload and - marks it stale in the UI instead of zeroing it. Transitions write events; repeated failure - writes one line, not 500. + marks it stale in the UI instead of zeroing it — on `/` too, where a host that has never + answered still gets a fleet row (`unreachable: true`, dimmed, every number `n/a`) because + dropping the row would turn a dead agent into "never configured". Transitions write + events; repeated failure writes one line, not 500. 5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app; charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with no outbound access — a CDN reference is a request that never comes back. @@ -113,6 +115,8 @@ Responsibilities, one line each: kelvin-vs-celsius normalisation. - `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a battery. It must never be the default, and every payload it produces says `simulated`. + Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that + ages), so a fresh simulator reads healthy and amber keeps meaning something. - `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own). - `rates.py` — Δ/window estimator, keyed by resolved series name. - `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric, @@ -123,8 +127,29 @@ Responsibilities, one line each: (charge/load/runtime), `severity` inside rated LLM cells (KV pressure, classified on a percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS row in `/api/summary`. Pages render the class; they never compare a number to a threshold. +- `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload, + recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts` + (`{id, severity, source, message, href, at}`, fail first then warn, then by source). A + pure projection: no I/O, no store, no thresholds of its own — it reads job states for + reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions. + Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but + older than 3× its own poll interval is `warn` while a source that stopped answering is + `fail`; a dead source gets exactly one line (its stale payload adds nothing) that names it + once — the poller's `"UpstreamError: "` wrapper and the duplicated host name are stripped, + because a message that says `gateway` twice reads as two problems; `unknown` and missing + numbers are silence; one alert per (source, metric) and at most 12, the last being a marker + that says how many it hid. This is why the overview can be two panels. - `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning. - `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary. + `/api/summary` is the overview's single call and the cheapest whole-fleet probe there is: + the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `llm.engines`, + `ups.ups`, the `states` maps and `recent_events` stay for programmatic consumers (live + tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events` are unchanged + and `alerts` is additive — `tests/test_api.py` pins every one of those key paths, with and + without targets configured, so retiring a panel can never shrink the payload. A configured + job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in + `system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total` + counts the machines there are and `hosts_live` the ones answering. - `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app with an in-memory DB and no targets. - `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the @@ -133,10 +158,12 @@ Responsibilities, one line each: `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a `.hero` big number (plus an optional `.bar`) from that class — it never compares a number to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for - values the API does not rate; converting its remaining call sites is phases 02–05's job. + values the API does not rate; converting its remaining call sites is phases 03–05's job + (`/` was converted in phase 02: the overview now colours only from `/api/summary` verdicts). Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`, -`gpu.busy_pct@0`, `llm.decode_tps`, `ups.charge_pct`), `@stream` for per-device series. +`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device +series — `/api/metrics` answers with exactly these names. That scheme is the history API's key — treat it as a public interface. ## The fleet (real environment, verified 2026-10) @@ -178,8 +205,8 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets ./scripts/dev-server.sh status # per-source ok/DOWN ./scripts/dev-server.sh logs -uv run pytest # hermetic: 356 tests, no network -uv run pytest --cov # coverage gate (94.75%; floor 94%) +uv run pytest # hermetic: 439 passed + 6 live-skipped, no network +uv run pytest --cov # coverage gate (96.8%; floor 94%) uv run pytest -m live --live # opt-in: real fleet / engines / upsd uv run ruff check src tests agent && uv run pyright ./scripts/capture-fixtures.sh # re-record ground truth @@ -215,18 +242,39 @@ service here. - The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest, so it stays in the default run and is *not* marked `live`. - Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces - 94% (`--cov-fail-under`), and ~95% is where we are; keep it there. New/modified Python + 94% (`--cov-fail-under`), and ~97% is where we are; keep it there. New/modified Python still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB, - never a real target. + never a real target, and never a real upstream: `/api/ups/command`'s send is proven against + a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live + suite is read-only — that fake is the only place the write reaches a "driver". ## UI conventions Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ` labels on measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status -dots in the header, window chips (1h…90d) on history pages. A panel that cannot get data says +dots in the header, window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API +accepts 5m…90d). A panel that cannot get data says why in the panel — the operator should never have to open devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`. +The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per +`/api/summary` alert: verdict class, source linked to its dashboard via the alert's own +`href`, message, age) and **Fleet**. A non-empty list floats the alerts panel above the +fleet table (`#overview-panels[data-state]` swaps them); an empty one is an explicit +"all clear" state below it, because an empty list is a message, not a blank box. A host that +is not answering keeps its row, dimmed with the shared `.stale` class and reading `n/a`, so +the table never implies last good numbers are live and never silently loses a machine. +No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the +two-panel shape against the served HTML. + +Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an +`id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for` +attribute references — the orphan a deleted renderer leaves behind, caught on served bytes +because there is no JS test runner (a locked anchor). Same rule for `dash.css`: delete the +rules a removed panel made dead (`.grid.wide` went with the overview's fleet+events row); +component vocabulary the pages are still being converted to (`.hero-bar`, the grid and bar +variants) stays even while only one page uses it. + A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals, muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's `severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*` diff --git a/README.md b/README.md index 8ff748a..96694fd 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Three dashboards, one domain, one process, one SQLite file: | Page | What it answers | Source | | ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- | -| `/` | Is anything wrong, and which dashboard do I need? | all of the below, one request | +| `/` | Is anything wrong, and which dashboard do I need? | fleet + alerts in one request (`/api/summary`) | | `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host | | `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only | | `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) | @@ -136,8 +136,23 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ── colours the same way and a value nobody reported renders `n/a` instead of green. A metric published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays what the upstream published. +* `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states, + host_payloads, engine_payloads, ups_payload, recent_events)` is a pure projection of the + state the poller already keeps plus the verdicts `severity.py` and the UPS health checks + already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as + `alerts`, worst first: `{id, severity, source, message, href, at}`. A configured source + that stopped answering is `fail`; one that is stale but still serving its last good value + is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than + one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a + marker that says how many it hid. * `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No - framework, no CDN, no bundler. A big colour-coded number is `Dash.hero(el, {...})` on the + framework, no CDN, no bundler. The overview (`/`) is exactly two panels — a + **Warnings & errors** list built from `/api/summary`'s `alerts` (each row links to the + dashboard that explains it) and the **Fleet** table — and nothing else: an empty alert + list reads as an explicit "all clear", and a live problem moves the panel above the + table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the + fleet never quietly gets smaller and last good values are never mistaken for live ones. + A big colour-coded number is `Dash.hero(el, {...})` on the `.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via `Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value with no verdict renders `n/a` with no colour at all. diff --git a/src/dashboards/alerts.py b/src/dashboards/alerts.py new file mode 100644 index 0000000..0baa78b --- /dev/null +++ b/src/dashboards/alerts.py @@ -0,0 +1,414 @@ +"""What is wrong right now, derived once on the server instead of on every page. + +The landing page has one question — *is anything wrong, and where do I look?* — and the +answer must be identical in the browser, in a future mobile view, and in whatever +notifies someone later. So the answer is a function of the state the poller already +keeps and the verdicts `severity.py` / `sources/nut.assess()` already produced, and this +module is the only place that assembles it. + +It is a pure projection by design: no I/O, no store, no HTTP, no clock it cannot be given. +That is what makes every rule below unit-testable without a hub, and what keeps a page +from growing its own opinion about what red means. + +The rules, and the reason each exists: + +* **Never alert for an unconfigured source.** `AGENTS.md` invariant 3: a server the + operator never installed must not raise a red flag, because that destroys the signal + from the panels that matter. `SourceState.configured` is the gate. +* **Stale is not dead.** A source that stopped answering *and* has no last good value is + `fail`. A source that is still serving its last good value but has not refreshed in + three of its own poll intervals is `warn` — saying "dead" while the page shows numbers + would be a lie. Per-source intervals because system polls at 10s and UPS at 5s; one + global number would be either noise or too late. +* **Verdicts come from elsewhere.** Host/LLM breaches read the `severity` dicts phase 01 + added; UPS breaches read `health.checks[].status`. Nothing here compares a number to a + threshold of its own — the message *quotes* the bound from `severity.THRESHOLDS` so the + operator can see which line was crossed, and the numbers still live in one file. +* **A dead source is one alert.** Its last good payload is still in the hub, so its stale + threshold breaches would pile three more amber lines on top of the red one that already + names it. Payload-derived alerts are skipped for a source that is not answering. +* **Bounded and deduped.** One alert per (source, metric) — the `id` is that pair — and at + most :data:`MAX_ALERTS` of them, the last being a marker that says how many it hid. A + 40-disk fleet must not turn the landing page into a log tail. +* **Sorted fail → warn, then by source**, so the thing that needs action is at the top + without the operator scanning. +""" + +from __future__ import annotations + +import re +import time +from collections.abc import Mapping, Sequence +from typing import Any + +from .config import Settings +from .severity import INVERTED, THRESHOLDS +from .sources.llm import SEVERITY_CELLS + +#: The most recent verdicts the landing page shows. Beyond this it is a wall of text and +#: the question it answers becomes "read this list", which is a different page. +MAX_ALERTS = 12 + +FAIL = "fail" +WARN = "warn" + +#: Only these two are alerts. `ok` is not news, and `None`/`unknown` means nothing honest +#: could be said — `AGENTS.md` rule 2, no measurement, no verdict, no colour. +_RANK = {FAIL: 0, WARN: 1} + +#: Where each dashboard lives, so every alert can say where to look. +HREF = {"system": "/system", "llm": "/llm", "ups": "/ups"} +DEFAULT_HREF = "/" + +#: A source whose last good value is older than this many of its own poll intervals is +#: serving data the operator should not act on. It is still on screen, hence amber. +STALE_FACTOR = 3.0 + +#: How to say a metric out loud: label, unit, decimals. Keyed by the dotted metric name +#: from `severity.THRESHOLDS`; an unlisted metric prints its own name rather than nothing. +_DISPLAY: dict[str, tuple[str, str, int]] = { + "cpu.usage_pct": ("CPU", "%", 0), + "cpu.temp_c": ("CPU temperature", "°C", 0), + "mem.used_pct": ("memory", "%", 0), + "disk.used_pct": ("disk", "%", 0), + "gpu.busy_pct": ("GPU utilisation", "%", 0), + "gpu.mem_used_pct": ("GPU memory", "%", 0), + "llm.kv_cache_usage_pct": ("KV cache", "%", 0), +} + +#: host payload section -> (field, severity metric). Mirrors the `severity` sub-dicts that +#: `sources/host.py` attaches, so a new rated field is one row here. +_HOST_CELLS: tuple[tuple[str, str, str], ...] = ( + ("cpu", "usage_pct", "cpu.usage_pct"), + ("cpu", "temp_c", "cpu.temp_c"), + ("memory", "used_pct", "mem.used_pct"), + ("disks", "worst_used_pct", "disk.used_pct"), +) +_GPU_CELLS: tuple[tuple[str, str], ...] = ( + ("busy_pct", "gpu.busy_pct"), + ("mem_used_pct", "gpu.mem_used_pct"), +) + + +# --------------------------------------------------------------------------- # +# formatting helpers (words, not thresholds) +# --------------------------------------------------------------------------- # +def _value(metric: str, number: float, scale: float = 1.0) -> str: + """`CPU 92%` — the metric said out loud, with the value the upstream published. + + Callers only get here with a real number (no measurement, no alert), so there is no + `n/a` case. ``scale`` is a unit for the *display* of a ratio the threshold table keeps + in percent (`llamacpp:kv_cache_usage_ratio` is 0-1); it is the same conversion + `with_severity()` applies to the verdict, so the number and the colour cannot disagree. + """ + label, unit, digits = _DISPLAY.get(metric, (metric, "", 1)) + return f"{label} {number * scale:.{digits}f}{unit}" + + +def _trip(metric: str, verdict: Any) -> str: + """` (fail ≥90)` — which bound was crossed, quoted from `severity.THRESHOLDS`. + + The threshold still lives in one place; this only reads it back so the alert is + actionable without opening the source. Direction comes from `severity.INVERTED`, so a + low battery charge prints `≤` instead of an arithmetic-looking `≥20`. + """ + if verdict not in _RANK: + return "" + base = metric.split("@", 1)[0] + bounds = THRESHOLDS.get(base) + if bounds is None: + return "" + warn, danger = bounds + low_is_worse = base in INVERTED + chosen = danger if verdict == FAIL else warn + return f" ({verdict} {'≤' if low_is_worse else '≥'}{chosen:g})" + + +#: `poller.Job` records a failure as `": "`, and collectors build +#: their message as `": "`. Left alone, a dead host prints +#: `gateway: UpstreamError: gateway: http://… unreachable` — the name twice and the one +#: exception class this codebase guarantees everything else raises (`AGENTS.md`: collectors +#: surface a single `UpstreamError` type, so its name is never information). Other class +#: names stay: an unexpected `ValueError` is a bug worth reading about. The full string +#: survives in `/api/health` and the event feed. +_POLLER_CLASS = re.compile(r"^UpstreamError: ") + + +def _clean_error(label: str, error: str) -> str: + """`http://h:9121/snapshot unreachable (ConnectError)` from the poller's wrapped string.""" + text = error.strip() + while True: + stripped = _POLLER_CLASS.sub("", text, count=1).strip() + if stripped.startswith(f"{label}: "): + stripped = stripped[len(label) + 2:].strip() + if stripped == text: + return text + text = stripped + + +def _sections(degraded: Any) -> str: + """The agent's `errors` map (or list) as words: `temperatures (no hwmon sensors)`.""" + if isinstance(degraded, Mapping): + return ", ".join(f"{name} ({detail})" if detail else str(name) + for name, detail in degraded.items()) or "nothing" + if isinstance(degraded, (list, tuple)): + return ", ".join(str(item) for item in degraded) or "nothing" + return str(degraded) + + +def _interval(settings: Settings, dashboard: str) -> float: + """The source's own poll interval — the unit staleness is measured in.""" + if dashboard == "llm": + return float(settings.llm_interval) + if dashboard == "ups": + return float(settings.nut_interval) + if dashboard == "system": + return float(settings.host_interval) + return 60.0 + + +# --------------------------------------------------------------------------- # +# the sink: dedupe, bound, sort +# --------------------------------------------------------------------------- # +class _Sink: + """Collects alerts keyed by (source, metric), keeping the worst verdict per key.""" + + def __init__(self, now: float) -> None: + self.now = now + self.items: dict[str, dict[str, Any]] = {} + + def add(self, verdict: Any, source: str, key: str, message: str, href: str, + at: float | None = None) -> None: + """Record one verdict — unless it is `ok`, `unknown`, or `None`, which are not alerts.""" + if verdict not in _RANK: + return + alert_id = f"{source}:{key}" + existing = self.items.get(alert_id) + if existing is not None: + # same source + same metric twice (two UPS cards answering as one name, a + # payload re-projected after a state change): one line, and the worse verdict + if _RANK[str(verdict)] < _RANK[existing["severity"]]: + existing["severity"] = str(verdict) + existing["message"] = message + existing["at"] = self.now if at is None else at + return + self.items[alert_id] = { + "id": alert_id, + "severity": str(verdict), + "source": source, + "message": message, + "href": href, + # when this became true, best effort: the last good poll, the outage event, or + # now. The page prints it as "since", so an alert that resets its clock every + # refresh is an alert whose duration nobody can read. + "at": self.now if at is None else at, + } + + def result(self) -> list[dict[str, Any]]: + ordered = sorted(self.items.values(), + key=lambda item: (_RANK.get(item["severity"], 9), item["source"], + item["id"])) + if len(ordered) <= MAX_ALERTS: + return ordered + visible, hidden = ordered[:MAX_ALERTS - 1], ordered[MAX_ALERTS - 1:] + failing = sum(1 for item in hidden if item["severity"] == FAIL) + visible.append({ + "id": "alerts:suppressed", + # the marker inherits the worst thing it hid: a list that ends in amber while + # three reds are behind it would be a lie of the same shape as hiding them + "severity": hidden[0]["severity"], + "source": "alerts", + "message": (f"{len(hidden)} more not shown " + f"({failing} failing, {len(hidden) - failing} warning)"), + "href": DEFAULT_HREF, + "at": self.now, + }) + return visible + + +# --------------------------------------------------------------------------- # +# the rules +# --------------------------------------------------------------------------- # +def _state_for(states: Sequence[Mapping[str, Any]], name: str) -> Mapping[str, Any] | None: + """The health record for the job that produced `name` (a host or engine label).""" + for state in states: + if state.get("label") == name or str(state.get("key") or "").endswith(f":{name}"): + return state + return None + + +def _answering(states: Sequence[Mapping[str, Any]], name: str) -> bool: + """Is that source currently answering? Unknown (no state at all) counts as yes. + + A payload that exists is data; inventing a reachability verdict for a source with no + health record would be this module's opinion, not the poller's. + """ + state = _state_for(states, name) + return True if state is None else bool(state.get("ok")) + + +def _dashboard_answering(states: Sequence[Mapping[str, Any]], dashboard: str) -> bool: + mine = [state for state in states if state.get("dashboard") == dashboard] + return any(bool(state.get("ok")) for state in mine) if mine else True + + +def _outage_started(events: Sequence[Mapping[str, Any]], dashboard: str, + label: str) -> float | None: + """When the poller first said this source was down, from its own transition event. + + Best effort by construction: `recent_events` is the short list the summary already + carries, so an outage older than that reads as `at: now`. The alert itself never + depends on the event being there. + """ + matches = [float(event["ts"]) for event in events + if event.get("kind") == "unreachable" and event.get("host") == label + and (event.get("source") == dashboard or event.get("source") is None) + and isinstance(event.get("ts"), (int, float))] + return max(matches) if matches else None + + +def _source_alerts(sink: _Sink, settings: Settings, states: Sequence[Mapping[str, Any]], + events: Sequence[Mapping[str, Any]]) -> None: + for state in states: + if not state.get("configured"): + continue # invariant 3: never red for a source nobody installed + label = str(state.get("label") or state.get("key") or "source") + dashboard = str(state.get("dashboard") or "") + href = HREF.get(dashboard, DEFAULT_HREF) + key = str(state.get("key") or label) + if not state.get("ok"): + # the alert starts with the name, so the name inside the message is removed — + # "driveripper: driveripper: ..." reads like two problems instead of one + error = _clean_error(label, str(state.get("error") or "")) or "is not answering" + sink.add(FAIL, label, f"{key}:down", f"{label}: {error}", href, + _outage_started(events, dashboard, label) or sink.now) + continue + age = state.get("age_seconds") + interval = _interval(settings, dashboard) + if isinstance(age, (int, float)) and age > interval * STALE_FACTOR: + sink.add(WARN, label, f"{key}:stale", + f"{label} last answered {age:.0f}s ago (every {interval:g}s): the page " + f"is showing the last good value, not a live one", + href, sink.now - age) + + +def _host_alerts(sink: _Sink, host: Mapping[str, Any], + states: Sequence[Mapping[str, Any]]) -> None: + name = str(host.get("name") or "host") + if host.get("unreachable") or not _answering(states, name): + return # the job-level alert already names this host; one line, not four + at = host.get("ts") if isinstance(host.get("ts"), (int, float)) else None + for section, field, metric in _HOST_CELLS: + block = host.get(section) or {} + number = block.get(field) + verdict = (block.get("severity") or {}).get(field) + # no number, no alert: `n/a` is not a breach (and the verdict is None anyway) + if number is None or verdict not in _RANK: + continue + mount = block.get("worst_mount") + where = f" on {mount}" if mount else "" + sink.add(verdict, name, metric, + f"{name}: {_value(metric, number)}{where}{_trip(metric, verdict)}", + HREF["system"], at) + for gpu in host.get("gpus") or []: + card = str(gpu.get("model") or f"GPU {gpu.get('index', 0)}") + for field, metric in _GPU_CELLS: + number = gpu.get(field) + verdict = (gpu.get("severity") or {}).get(field) + if number is None or verdict not in _RANK: + continue + sink.add(verdict, name, f"{metric}@{gpu.get('index', 0)}", + f"{name}: {_value(metric, number)} on {card}{_trip(metric, verdict)}", + HREF["system"], at) + degraded = host.get("degraded") + if degraded: + # a /proc read failed on the box: the numbers that did come back are real, but the + # snapshot is not the whole picture and the page must not imply that it is + sink.add(WARN, name, "agent.sections", + f"{name}: the agent could not read {_sections(degraded)}", HREF["system"], at) + + +def _engine_alerts(sink: _Sink, engine: Mapping[str, Any], + states: Sequence[Mapping[str, Any]]) -> None: + name = str(engine.get("engine") or "engine") + if engine.get("unreachable") or not _answering(states, name): + return + at = engine.get("scraped_at") if isinstance(engine.get("scraped_at"), (int, float)) else None + metrics = engine.get("metrics") or {} + # the rated cells are declared next to the field maps they come from (`SEVERITY_CELLS`), + # so a new one is rated by severity.py and alerted here without touching this loop + for key, (metric, scale) in SEVERITY_CELLS.items(): + cell = metrics.get(key) + if not isinstance(cell, Mapping): + continue + value, verdict = cell.get("value"), cell.get("severity") + # no number in the cell, no claim (rule 2); no verdict, no colour to justify one + if not isinstance(value, (int, float)) or verdict not in _RANK: + continue + sink.add(verdict, name, f"{metric}@{key}", + f"{name}: {_value(metric, float(value), scale)}{_trip(metric, verdict)}", + HREF["llm"], at) + + +def _ups_alerts(sink: _Sink, payload: Mapping[str, Any], + states: Sequence[Mapping[str, Any]]) -> None: + if payload.get("simulated"): + # `AGENTS.md`: the simulator must never be silent about what it is. The UPS page + # badges it; the landing page has to say it too, or a modelled battery reads as a + # healthy real one from two clicks away. + sink.add(WARN, "ups", "simulated", + "UPS numbers are simulated — NUT_SIMULATE=1 is set, so these are modelled " + "figures and not your batteries", HREF["ups"]) + if not _dashboard_answering(states, "ups"): + return + for card in payload.get("ups") or []: + name = str(card.get("name") or "ups") + at = card.get("ts") if isinstance(card.get("ts"), (int, float)) else None + for check in (card.get("health") or {}).get("checks") or []: + label = str(check.get("name") or "check") + detail = str(check.get("detail") or "").strip() + # `unknown` stays out: a driver that publishes no battery temperature is not a + # fault, and the UPS page already says "not reported by this driver" + sink.add(check.get("status"), name, f"health:{label}", + f"{label}: {detail}" if detail else label, HREF["ups"], at) + + +def _config_alerts(sink: _Sink, settings: Settings) -> None: + for problem in settings.host_target_problems: + # a typo'd entry is not a dead host: the operator needs to know it is not being + # polled at all, which is otherwise invisible (the host simply is not in the list) + sink.add(WARN, "config", f"HOST_TARGETS:{problem}", + f"HOST_TARGETS: {problem} — that entry is not being polled", HREF["system"]) + + +# --------------------------------------------------------------------------- # +# entry point +# --------------------------------------------------------------------------- # +def build(settings: Settings, hub_states: Sequence[Mapping[str, Any]] = (), + host_payloads: Sequence[Mapping[str, Any]] = (), + engine_payloads: Sequence[Mapping[str, Any]] = (), + ups_payload: Mapping[str, Any] | None = None, + recent_events: Sequence[Mapping[str, Any]] = (), *, + now: float | None = None) -> list[dict[str, Any]]: + """Everything that is wrong right now, worst first. + + Each alert is `{"id", "severity", "source", "message", "href", "at"}`; `id` is + `":"` and is stable across refreshes so a page can key DOM off it, and + `at` is a unix timestamp for when the condition was first observed (best effort). + Empty means all clear — there is no `ok` alert, because an empty list is the clearest + possible answer to "what is wrong". + + Inputs are the same objects `/api/summary` already assembles (job states, last good + payloads, the UPS payload, the short event list), so answering costs nothing extra and + the answer cannot disagree with the panels underneath it. + """ + sink = _Sink(time.time() if now is None else now) + _source_alerts(sink, settings, hub_states, recent_events) + for host in host_payloads: + _host_alerts(sink, host, hub_states) + for engine in engine_payloads: + _engine_alerts(sink, engine, hub_states) + if ups_payload is not None: + _ups_alerts(sink, ups_payload, hub_states) + _config_alerts(sink, settings) + return sink.result() diff --git a/src/dashboards/api.py b/src/dashboards/api.py index 1c89709..9f0b934 100644 --- a/src/dashboards/api.py +++ b/src/dashboards/api.py @@ -18,6 +18,7 @@ from typing import Any from fastapi import APIRouter, Body, Depends, HTTPException, Query, Request +from . import alerts as alert_rules from .collectors import NutMonitor from .config import Settings from .poller import Hub @@ -190,11 +191,30 @@ def common_router() -> APIRouter: @router.get("/summary") async def summary(ctx: AppContext = Depends(get_context)) -> dict[str, Any]: - """One call for the landing page: is each dashboard alive, and what stands out.""" - hosts = [job.state.payload for job in ctx.hub.jobs.values() - if job.dashboard == "system" and job.state.payload] - engines = [job.state.payload for job in ctx.hub.jobs.values() - if job.dashboard == "llm" and job.state.payload] + """One call for the landing page: is anything wrong, and what is the fleet doing. + + What `/` renders is `alerts` plus `system.fleet`/`system.hosts` and nothing else. + Everything else here is for **programmatic use**: this is the cheapest whole-fleet + probe that exists, so the live tests, an external check and a notifier later read + `llm.engines`, `ups.ups` and the `states`/`recent_events` no panel paints any more. + Keys are additive-only — `alerts` was added, nothing was renamed, dropped, or given + a new meaning. `tests/test_api.py` pins each path below against a future cleanup. + """ + # A configured job that has never answered gets the same stub `/api/*/snapshot` + # hands out: a host in `HOST_TARGETS` with no payload yet is still in the fleet, and + # dropping its row would turn "dead agent" into "never configured" (invariants 3 + # and 4) while quietly shrinking `fleet.hosts_total` to the answering subset. + hosts: list[dict[str, Any]] = [ + job.state.payload or {"name": job.label, "ts": job.state.last_attempt, + "unreachable": True} + for job in ctx.hub.jobs.values() if job.dashboard == "system" + ] + engines: list[dict[str, Any]] = [ + job.state.payload or {"engine": job.label, "metrics": {}, + "sessions": {"active": None, "list": []}, + "unreachable": True} + for job in ctx.hub.jobs.values() if job.dashboard == "llm" + ] ups_payload = None nut_job = ctx.hub.jobs.get("nut") if nut_job and nut_job.state.payload: @@ -202,28 +222,40 @@ def common_router() -> APIRouter: fleet = fleet_summary(hosts) if hosts else None if fleet is not None: fleet["severity"] = _fleet_severity(fleet) + # one list of events, reused twice: the feed itself, and the timestamp for a source + # that is currently failing (alerts.py reads `kind: unreachable` out of it) + recent_events = [event.as_json() for event in ctx.store.recent_events(12)] + # "what is wrong right now" is a server-side answer (see alerts.py) so the overview, + # a mobile view, and a notifier later all agree on the same verdicts + summary_alerts = alert_rules.build(ctx.settings, ctx.hub.states(), hosts, engines, + ups_payload, recent_events) return { "ts": time.time(), "system": {"enabled": bool(ctx.settings.host_targets), "fleet": fleet, - "hosts": [{"name": h["name"], "os": (h.get("info") or {}).get("os_pretty"), - "cpu_usage_pct": h["cpu"]["usage_pct"], - "load1": h["cpu"].get("load1"), + # every read is `.get()` because a stub has none of these sections — + # an unreachable agent renders as `n/a` (rule 2), never as a 500 + "hosts": [{"name": h["name"], + "os": (h.get("info") or {}).get("os_pretty"), + "cpu_usage_pct": (h.get("cpu") or {}).get("usage_pct"), + "load1": (h.get("cpu") or {}).get("load1"), "cores": (h.get("info") or {}).get("cpu_cores_logical"), - "mem_used_pct": h["memory"]["used_pct"], - "disk_worst_used_pct": h["disks"]["worst_used_pct"], + "mem_used_pct": (h.get("memory") or {}).get("used_pct"), + "disk_worst_used_pct": (h.get("disks") or {}).get("worst_used_pct"), "gpu_busy_pct": [g.get("busy_pct") for g in h.get("gpus") or []], - "temp_c": h["cpu"].get("temp_c"), + "temp_c": (h.get("cpu") or {}).get("temp_c"), "uptime_seconds": (h.get("info") or {}).get("uptime_seconds"), + "unreachable": bool(h.get("unreachable")), "severity": _host_severity(h)} for h in hosts], "states": ctx.hub.states("system")}, "llm": {"enabled": bool(ctx.settings.gufo_url or ctx.settings.vllm_url), "fleet": fleet_totals(engines) if engines else None, "engines": [{"engine": e["engine"], - "decode_tokens_per_second": e["metrics"].get( + "decode_tokens_per_second": (e.get("metrics") or {}).get( "decode_tokens_per_second", {}).get("value"), - "active": e["sessions"].get("active")} + "active": (e.get("sessions") or {}).get("active"), + "unreachable": bool(e.get("unreachable"))} for e in engines], "states": ctx.hub.states("llm")}, "ups": {"enabled": ctx.settings.nut_enabled, @@ -236,7 +268,8 @@ def common_router() -> APIRouter: "severity": _ups_severity(card)} for card in (ups_payload or {}).get("ups", [])], "states": ctx.hub.states("ups")}, - "recent_events": [event.as_json() for event in ctx.store.recent_events(12)], + "recent_events": recent_events, + "alerts": summary_alerts, } return router diff --git a/src/dashboards/sources/nut_sim.py b/src/dashboards/sources/nut_sim.py index ab77331..47c92b9 100644 --- a/src/dashboards/sources/nut_sim.py +++ b/src/dashboards/sources/nut_sim.py @@ -23,6 +23,18 @@ from typing import Any from .nut import UpsReading +def _mfr_date(years_ago: float) -> str: + """A battery manufacture date `years_ago` years back, computed instead of hard-coded. + + A fixed date is fine for a year or two and then quietly turns every development page + amber forever (`Battery age: 7.0 years — past the 4-year replacement window`), which is + the same failure `assess()` refuses for a UPS that floats at 90 %: a warning nobody can + clear teaches people to ignore the warnings that matter. The simulator stands in for a + working UPS, so its pack stays a fixed age behind today. + """ + return time.strftime("%Y-%m-%d", time.gmtime(time.time() - years_ago * 365.25 * 86_400)) + + @dataclass class SimUps: """One fake UPS: nameplate values plus the state a driver would report.""" @@ -180,10 +192,10 @@ class NutSimulator: self.ups = [ SimUps("cyberpower", "PR1500LCDRT2U", "Cyber Power Systems", "CPS-04213", nominal_va=1500.0, nominal_w=900.0, load_w=210.0, capacity_ah=17.0, - mfr_date="2022-04-15"), + mfr_date=_mfr_date(2.0)), SimUps("apc", "Smart-UPS 1500", "APC", "AS-11902", nominal_va=1500.0, nominal_w=865.0, load_w=140.0, capacity_ah=8.0, - mfr_date="2019-11-02", driver="usbhid-ups", + mfr_date=_mfr_date(3.0), driver="usbhid-ups", test_result="PASSED", test_date="2026-06-14"), ] self._last = time.time() diff --git a/src/dashboards/web/pages/index.html b/src/dashboards/web/pages/index.html index 863bfd5..6f464e1 100644 --- a/src/dashboards/web/pages/index.html +++ b/src/dashboards/web/pages/index.html @@ -30,70 +30,29 @@
reading fleet…
-
-
-
-

System resources

+ +
+
+

Warnings & errors

+
-
-
Hosts answering
-
n/a
-
CPU average
-
n/a%
-
-
Memory
-
n/a
-
-
Worst disk
-
n/a%
-
-
GPUs
-
n/a
+
    + -
    Open system dashboard →
    -
    +
    -
    -

    LLM engines

    -
    -
    -
    Engines answering
    -
    n/a
    -
    Decode
    -
    n/atok/s
    -
    Prompt
    -
    n/atok/s
    -
    Active sessions
    -
    n/a
    -
    Queued
    -
    n/a
    -
    KV cache now
    -
    n/a%
    -
    -
    -
    Open LLM dashboard →
    -
    -
    - - -
    - -
    -
    -

    Fleet

    +
    +

    Fleet

    @@ -105,14 +64,8 @@ HOST_TARGETS.

    - - -
    -

    Recent events

    - transitions only
    -
    • loading…
    -
    - + +
    diff --git a/src/dashboards/web/pages/system.html b/src/dashboards/web/pages/system.html index ffb604d..d952b44 100644 --- a/src/dashboards/web/pages/system.html +++ b/src/dashboards/web/pages/system.html @@ -57,7 +57,7 @@
    GPU busy (avg)
    n/a%
    Power
    -
    n/a
    +
    n/a
    diff --git a/src/dashboards/web/static/app.js b/src/dashboards/web/static/app.js index 7e8bd69..2d56d2e 100644 --- a/src/dashboards/web/static/app.js +++ b/src/dashboards/web/static/app.js @@ -62,7 +62,10 @@ const Dash = (() => { if (ts == null) return 'never'; const seconds = Math.max(0, Date.now() / 1000 - ts); if (seconds < 2) return 'now'; - return duration(seconds).replace(/(h|m|s|d|y)$/, '$') + ' ago'; + // `duration()` already carries the unit: "2m ago", "5h ago". (It used to be stripped + // with `replace(/(h|m|s|d|y)$/, '$')` — a lone `$` is a literal in a JS replacement, + // so every age printed "2$ ago". The unit is what makes the number mean anything.) + return `${duration(seconds)} ago`; } const naText = 'n/a'; @@ -107,10 +110,10 @@ const Dash = (() => { * * Thresholds live in `src/dashboards/severity.py` and travel in the payload as * `severity`; `severityFrom()` below is how a page reads them. This function survives - * only because `page-index.js`, `page-system.js`, `page-llm.js` and `page-ups.js` still - * call it with their own numbers (phases 02–05 convert them page by page, so each page's - * change is reviewable on its own). Do not add a new call site: a threshold written in - * JS is a threshold that can disagree with the other three pages. + * only because `page-system.js`, `page-llm.js` and `page-ups.js` still call it with + * their own numbers (phases 03–05 convert them page by page, so each page's change is + * reviewable on its own — `/` was already converted in phase 02). Do not add a new call + * site: a threshold written in JS is a threshold that can disagree with the other pages. */ function severityFor(value, warnAt, failAt) { const warn = warnAt == null ? 75 : warnAt; @@ -312,10 +315,7 @@ const Dash = (() => { const clock = $('#clock'); if (clock) { const tickClock = () => { - const now = new Date(); - const target = document.getElementById('sim-flag'); - clock.textContent = now.toLocaleTimeString(undefined, { hour12: false }); - void target; + clock.textContent = new Date().toLocaleTimeString(undefined, { hour12: false }); }; tickClock(); setInterval(tickClock, 1000); diff --git a/src/dashboards/web/static/dash.css b/src/dashboards/web/static/dash.css index 147ee6c..96b11d6 100644 --- a/src/dashboards/web/static/dash.css +++ b/src/dashboards/web/static/dash.css @@ -179,8 +179,6 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; } .grid.auto { grid-template-columns: repeat(auto-fit, minmax(min(360px, 100%), 1fr)); } .grid.cols-2 { grid-template-columns: repeat(auto-fit, minmax(min(480px, 100%), 1fr)); } .grid.cols-3 { grid-template-columns: repeat(auto-fit, minmax(min(300px, 100%), 1fr)); } -.grid.wide { grid-template-columns: minmax(0, 2fr) minmax(0, 1fr); } -@media (max-width: 1150px) { .grid.wide { grid-template-columns: 1fr; } } .stack { display: grid; gap: .85rem; align-content: start; } /* ---- cards -------------------------------------------------------------- */ @@ -208,8 +206,6 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; } .card > .body.tight { padding: .45rem .8rem .6rem; } .card.flat { box-shadow: none; } -.cardset { display: grid; gap: .85rem; } - /* ---- metrics ------------------------------------------------------------ */ .metrics { display: grid; gap: .7rem .9rem; grid-template-columns: repeat(auto-fit, minmax(104px, 1fr)); } .metric { min-width: 0; } @@ -353,6 +349,11 @@ table.data tbody tr:hover td { background: var(--bg-sunken); } table.data td.name { font-weight: 600; } table.data .num { font-family: var(--mono); } table.data .mute { color: var(--text-muted); } +/* verdict colour on a table cell (overview fleet rows): only a breach paints, `ok` stays + plain text so a healthy table is quiet — the words are the same `ok|warn|fail` the API + sends, never a JS-side threshold */ +table.data td.warn { color: var(--warn); } +table.data td.fail { color: var(--fail); } table.data.scroll { display: block; overflow: auto; max-height: 320px; } /* ---- charts ------------------------------------------------------------- */ @@ -404,6 +405,29 @@ ul.events .sev.info { color: var(--text-muted); } ul.events .sev.warn { color: var(--warn); } ul.events .sev.error { color: var(--fail); } +/* overview alerts: source (link to its dashboard) · message · age. The class on the row + is the server's verdict — `alerts.py` decided, the page only paints it. */ +ul.alerts { list-style: none; margin: 0; padding: 0; display: grid; gap: .1rem; } +ul.alerts li { + display: grid; grid-template-columns: max-content 1fr max-content; gap: .7rem; align-items: baseline; + padding: .4rem .1rem; border-bottom: 1px solid color-mix(in srgb, var(--border) 50%, transparent); + font-size: 13px; +} +ul.alerts li:last-child { border-bottom: 0; } +ul.alerts .src { font-weight: 650; white-space: nowrap; } +ul.alerts .msg { color: var(--text-muted); overflow-wrap: anywhere; } +ul.alerts time { font-family: var(--mono); font-size: 11px; color: var(--text-faint); white-space: nowrap; } +ul.alerts li.ok .src { color: var(--ok); } +ul.alerts li.warn .src { color: var(--warn); } +ul.alerts li.fail .src { color: var(--fail); } + +/* the overview's two panels swap by verdict, not by re-rendering: an active problem reads + first, an empty list leaves the fleet table on top and the all-clear state below it */ +#overview-panels .panel-fleet { order: 1; } +#overview-panels .panel-alerts { order: 2; } +#overview-panels[data-state="alerting"] .panel-alerts { order: 1; } +#overview-panels[data-state="alerting"] .panel-fleet { order: 2; } + ul.checks { list-style: none; margin: 0; padding: 0; display: grid; gap: .4rem; } ul.checks li { display: grid; grid-template-columns: 18px 1fr; gap: .5rem; align-items: start; } ul.checks .what { font-weight: 600; font-size: 12.5px; } @@ -472,7 +496,6 @@ dialog footer { padding: .7rem .9rem; border-top: 1px solid var(--border); displ .empty-state ul { text-align: left; display: inline-block; margin: .6rem 0 0; } .stale { opacity: .62; } -.stale-flag { color: var(--warn); font-size: 11px; } select, input[type="text"] { background: var(--bg-sunken); color: var(--text); border: 1px solid var(--border); diff --git a/src/dashboards/web/static/page-index.js b/src/dashboards/web/static/page-index.js index a28d424..010046e 100644 --- a/src/dashboards/web/static/page-index.js +++ b/src/dashboards/web/static/page-index.js @@ -1,7 +1,13 @@ -/* Overview page: one /api/summary call, plus the config-driven setup hints. */ +/* Overview page: exactly two answers — what needs attention, and what the fleet is doing. + * + * One `/api/summary` call feeds both panels; "what needs attention" is the server's list + * (`alerts.py`), never a browser-side opinion. The page owns no thresholds: cells are + * coloured from the `severity` verdicts in the payload via `Dash.severityFrom`, and an + * unrated value gets no colour (which is also what keeps `n/a` from looking green). + */ 'use strict'; -const { $, fetchJSON, setTxt, setValue, setBar, setDot, bytes, pct, num, duration, poll } = Dash; +const { $, fetchJSON, num, duration, age, poll } = Dash; async function loadConfig() { try { @@ -48,101 +54,84 @@ async function loadConfig() { } } -function renderEvents(events) { - const list = $('#events'); - if (!events || !events.length) { - list.innerHTML = '
  • Nothing has changed ' + - 'since this server started — the log only records transitions.
    • '; +/** + * The warnings/errors panel: one row per alert from `/api/summary`'s `alerts`. + * + * The row is the source (a link to the dashboard that explains it), the server's message, + * and the age. Colour comes from `alert.severity` through `Dash.severityFrom` — the page + * never re-derives a verdict. An empty list is a message ("all clear"), not a blank box, + * and it also moves the panel below the fleet table so a calm morning reads calm. + */ +function renderAlerts(alerts) { + const list = $('#alerts'); + const clear = $('#alerts-clear'); + const panels = $('#overview-panels'); + const items = Array.isArray(alerts) ? alerts : []; + panels.dataset.state = items.length ? 'alerting' : 'clear'; + $('#alerts-count').textContent = items.length + ? `${items.length} active` + : ''; + clear.hidden = items.length > 0; + list.hidden = items.length === 0; + if (!items.length) { + list.innerHTML = ''; return; } - list.innerHTML = events.map((event) => { - const when = new Date(event.ts * 1000).toLocaleTimeString(undefined, - { hour: '2-digit', minute: '2-digit', hour12: false }); - return `
  • ${event.host} ` + - `${DashCharts.escapeHtml(event.message.replace(event.host + ':', '').trim() || event.kind)}
    • `; + list.innerHTML = items.map((alert) => { + // alerts.py names the source at the head of most messages; the link already says it, + // and "deskwork: deskwork: CPU 92%" reads like two problems instead of one + const source = String(alert.source || 'fleet'); + const raw = String(alert.message || ''); + const prefix = `${source}:`; + const message = raw.startsWith(prefix) ? raw.slice(prefix.length).trim() : raw; + const sev = Dash.severityFrom(alert.severity); + return ` + ${DashCharts.escapeHtml(source)} + ${DashCharts.escapeHtml(message)} + + `; }).join(''); } -function renderSummary(summary) { - const system = summary.system || {}; - const fleet = system.fleet || {}; - setDot($('#dot-system'), system.enabled ? (fleet.hosts_live === fleet.hosts_total ? 'ok' - : fleet.hosts_live ? 'warn' : 'fail') : 'idle'); - setTxt($('#sys-hosts-live'), fleet.hosts_live == null ? 'n/a' : String(fleet.hosts_live)); - setTxt($('#sys-hosts-total'), fleet.hosts_total == null ? '' : `of ${fleet.hosts_total}`); - setValue($('#sys-cpu'), fleet.cpu_usage_pct_avg, 0); - setBar($('#sys-cpu').closest('.metric').querySelector('.bar'), fleet.cpu_usage_pct_avg); - if (fleet.memory_used_bytes != null) { - const used = Dash.fmtBytes(fleet.memory_used_bytes, 0); - const total = Dash.fmtBytes(fleet.memory_total_bytes, 0); - // "200 / 253 GiB", not "200 GiB of 253 GiB": the card column is narrow and the long form - // ran into the worst-disk number next to it - setTxt($('#sys-mem'), total ? `${used.text}/${total.text}` : used.text); - setTxt($('#sys-mem-unit'), used.unit); - } else { - setTxt($('#sys-mem'), 'n/a'); - setTxt($('#sys-mem-unit'), ''); +/** + * The fleet table: one row per host, columns unchanged. + * + * Cell colour comes from the row's `severity` verdicts (`/api/summary`, phase 01). `ok` + * prints as plain text on purpose — an all-green table is noise; only a breach earns a + * colour here, and the alerts panel is where anything rated gets its line. + * + * A host whose agent stopped answering keeps its last good numbers rather than being + * zeroed, so its row is dimmed with the shared `.stale` class (and says so in its + * tooltip): the table must never imply dead numbers are live while the panel above + * says the box is down. + */ +function renderFleet(rows, states) { + const answering = {}; + for (const state of states || []) { + const name = String(state.label || state.key || ''); + if (name) answering[name] = Boolean(state.ok); } - setBar($('#sys-mem').closest('.metric').querySelector('.bar'), fleet.memory_used_pct); - setValue($('#sys-disk'), fleet.disk_worst_used_pct, 0); - setBar($('#sys-disk').closest('.metric').querySelector('.bar'), fleet.disk_worst_used_pct, - Dash.severityFor(fleet.disk_worst_used_pct, 80, 92)); - setTxt($('#sys-gpus'), fleet.gpu_count == null ? 'n/a' - : `${fleet.gpu_count}${fleet.gpu_busy_pct_avg != null ? ' · ' + num(fleet.gpu_busy_pct_avg, 0) + '%' : ''}`); - - const llm = summary.llm || {}; - const engines = llm.fleet || {}; - setDot($('#dot-llm'), !llm.enabled ? 'idle' - : (engines.engines_live === engines.engines_total ? 'ok' : engines.engines_live ? 'warn' : 'fail')); - setTxt($('#llm-live'), engines.engines_live == null ? 'n/a' : String(engines.engines_live)); - setTxt($('#llm-total'), engines.engines_total == null ? '' : `of ${engines.engines_total}`); - setValue($('#llm-decode'), engines.decode_tokens_per_second, 1); - setValue($('#llm-prompt'), engines.prompt_tokens_per_second, 0); - setTxt($('#llm-active'), engines.requests_active == null ? 'n/a' : num(engines.requests_active, 0)); - setTxt($('#llm-waiting'), engines.requests_waiting == null ? 'n/a' : num(engines.requests_waiting, 0)); - setValue($('#llm-kv'), engines.kv_cache_usage_max == null ? null : engines.kv_cache_usage_max * 100, 0); - setBar($('#llm-kv').closest('.metric').querySelector('.bar'), - engines.kv_cache_usage_max == null ? null : engines.kv_cache_usage_max * 100, - Dash.severityFor(engines.kv_cache_usage_max == null ? null : engines.kv_cache_usage_max * 100, 75, 90)); - - const ups = summary.ups || {}; - setDot($('#dot-ups'), !ups.enabled ? 'idle' - : (ups.ups || []).some((u) => u.overall === 'at risk') ? 'fail' - : (ups.ups || []).some((u) => u.overall !== 'healthy') ? 'warn' : 'ok'); - const list = $('#ups-list'); - if (!ups.enabled) { - list.innerHTML = '
    No NUT server configured.
    '; - } else if (!(ups.ups || []).length) { - list.innerHTML = '
    No UPS answering from this upsd yet.
    '; - } else { - list.innerHTML = ups.ups.map((item) => { - const kind = item.overall === 'healthy' ? 'ok' : item.overall === 'at risk' ? 'fail' - : item.overall === 'degraded' ? 'warn' : ''; - return `
    - ${DashCharts.escapeHtml(item.name)} - ${DashCharts.escapeHtml(item.status || 'no state')} - ${ups.simulated ? 'sim' : ''} - ${item.charge_pct == null ? 'n/a' : num(item.charge_pct, 0) + '%'} · - ${item.load_pct == null ? 'n/a' : num(item.load_pct, 0) + '% load'} · - ${item.runtime_seconds == null ? 'n/a' : duration(item.runtime_seconds) + ' runtime'} -
    `; - }).join(''); - } - - const rows = system.hosts || []; const tbody = $('#fleet-table tbody'); $('#fleet-empty').hidden = rows.length > 0; if (!rows.length) { tbody.innerHTML = ''; return; } + const verdict = (severity, key) => { + const kind = Dash.severityFrom((severity || {})[key]); + return kind === 'ok' || kind === '' ? '' : ` ${kind}`; + }; tbody.innerHTML = rows.map((host) => { + const sev = host.severity || {}; const gpus = (host.gpu_busy_pct || []).filter((v) => v != null); - return `
    + // the row's own flag (a host that has never answered has no payload to be stale about) + // or the job state: either way the numbers below are the last good ones, not live ones + const stale = host.unreachable === true || answering[host.name] === false; + return ` - + - - + + - + `; }).join(''); @@ -151,14 +140,24 @@ function renderSummary(summary) { async function refresh() { try { const summary = await fetchJSON('/api/summary'); - const total = (summary.system.hosts || []).length; - const live = summary.system.fleet ? summary.system.fleet.hosts_live : 0; - const engines = summary.llm.fleet ? `${summary.llm.fleet.engines_live}/${summary.llm.fleet.engines_total} engines` : 'no engines'; - const ups = summary.ups.ups ? `${summary.ups.ups.length} UPS` : 'no UPS'; - $('#summary-line').textContent = `${live}/${total} hosts · ${engines} · ${ups} · ` + - `${(summary.recent_events || []).length} recent events`; - renderSummary(summary); - renderEvents(summary.recent_events); + const system = summary.system || {}; + const fleet = system.fleet || {}; + const hosts = system.hosts || []; + const live = fleet.hosts_live == null ? hosts.length : fleet.hosts_live; + const total = fleet.hosts_total == null ? hosts.length : fleet.hosts_total; + const engines = summary.llm && summary.llm.fleet + ? `${summary.llm.fleet.engines_live}/${summary.llm.fleet.engines_total} engines` + : 'no engines'; + const ups = summary.ups && (summary.ups.ups || []).length + ? `${summary.ups.ups.length} UPS` + : 'no UPS'; + const alerts = summary.alerts || []; + const problems = alerts.length ? `${alerts.length} warning${alerts.length === 1 ? '' : 's'}` + : 'all clear'; + $('#summary-line').textContent = + `${live}/${total} hosts · ${engines} · ${ups} · ${problems}`; + renderAlerts(alerts); + renderFleet(hosts, system.states); } catch (err) { $('#summary-line').textContent = `summary failed: ${err.message}`; } diff --git a/tests/conftest.py b/tests/conftest.py index bf63241..b6277c4 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -11,6 +11,7 @@ letting `uv run pytest -m live --live` catch an upstream that renamed something. """ from __future__ import annotations +import copy import json import sys from pathlib import Path @@ -42,6 +43,7 @@ if str(SRC) not in sys.path: sys.path.insert(0, str(SRC)) from dashboards.rates import RateEstimator # noqa: E402 +from dashboards.sources.host import HostCollector, parse_host_snapshot # noqa: E402 from dashboards.store import SampleStore # noqa: E402 FIXTURES = Path(__file__).resolve().parent / "fixtures" @@ -77,6 +79,34 @@ def agent_snapshots() -> dict[str, dict[str, Any]]: return out +@pytest.fixture +def host_scrape() -> Any: + """`host_scrape(snapshot, cpu_busy=10.0, mutate=None)` -> one parsed host payload. + + Two scrapes are needed before any rate exists (the agent reports counters; the + dashboard owns the window), and the payload the API serves is the second one. `mutate` + edits the snapshot before both scrapes so a test can put a value on a chosen side of + `severity.THRESHOLDS`; `cpu_busy` sets the CPU percentage over the window directly. + Driven from the real capture so a schema drift breaks here rather than in a browser. + """ + def scrape(snapshot: dict[str, Any], *, name: str = "driveripper", cpu_busy: float = 10.0, + mutate: Any = None) -> dict[str, Any]: + snap = copy.deepcopy(snapshot) + if mutate is not None: + mutate(snap) + collector = HostCollector(RateEstimator(span=60.0)) + parse_host_snapshot(name, snap, collector, now=1_000.0) + second = copy.deepcopy(snap) + second["ts"] = 1_010.0 + modes = second["cpu"]["modes"] + # busy = (Δtotal - Δidle)/Δtotal, and iowait is left untouched so it rates at 0 + modes["total"] += 1_000.0 + modes["idle"] += 1_000.0 - (cpu_busy * 10.0) + return parse_host_snapshot(name, second, collector, now=1_010.0).payload + + return scrape + + @pytest.fixture def metrics_text() -> Any: """metrics_text('gufo') / metrics_text('vllm') -> the captured exposition verbatim.""" diff --git a/tests/test_alerts.py b/tests/test_alerts.py new file mode 100644 index 0000000..0e88794 --- /dev/null +++ b/tests/test_alerts.py @@ -0,0 +1,476 @@ +"""`alerts.build`: the whole "what is wrong right now" answer, with no I/O anywhere. + +The builder is a pure projection of state the poller already keeps and verdicts +`severity.py` / `sources/nut.assess()` already reached, so every rule can be tested by +handing it a dict — no hub, no DB, no sockets, no clock it was not given. The cases below +are the ones that make the landing page trustworthy rather than merely busy: + +* a source nobody configured is **never** an alert (`AGENTS.md` invariant 3); +* a source that is stale but still serving is amber, not red; +* a dead source is **one** line, not one per stale metric in its last good payload; +* only `fail` and `warn` come through — `ok`, `unknown`, and a missing number are silence; +* the list dedupes per (source, metric), is bounded, and says how much it hid. +""" + +from __future__ import annotations + +from typing import Any + +from dashboards import alerts +from dashboards.config import Settings +from dashboards.sources import nut +from dashboards.sources.nut_sim import NutSimulator + +SETTINGS = Settings() # defaults: no targets, intervals 10s system / 2s llm / 5s ups + + +# --------------------------------------------------------------------------- # +# state shapers +# --------------------------------------------------------------------------- # +def state(key: str = "host:driveripper", dashboard: str = "system", + label: str = "driveripper", *, ok: bool = True, configured: bool = True, + error: str | None = None, age: float | None = 0.0) -> dict[str, Any]: + """A `SourceState.as_json()`-shaped health record — the only shape the builder reads.""" + return {"key": key, "dashboard": dashboard, "label": label, "configured": configured, + "ok": ok, "error": error, "age_seconds": age, "attempts": 4, "successes": 3, + "consecutive_failures": 0 if ok else 1, "last_ok": None, "latency_ms": None} + + +def check(name: str, status: str, detail: str = "") -> dict[str, Any]: + return {"name": name, "status": status, "detail": detail, "metric": None, "value": None} + + +def ups_payload(*cards: dict[str, Any], simulated: bool = False) -> dict[str, Any]: + return {"ts": 1_000.0, "simulated": simulated, "ups": list(cards)} + + +def ups_card(name: str = "apc", *checks: dict[str, Any]) -> dict[str, Any]: + return {"name": name, "ts": 1_000.0, "health": {"overall": "healthy", "checks": list(checks)}} + + +def engine(name: str = "gufo", **cells: Any) -> dict[str, Any]: + return {"engine": name, "scraped_at": 1_000.0, "metrics": dict(cells)} + + +def ratio(value: float | None, verdict: str | None) -> dict[str, Any]: + """A rated LLM cell, exactly as `sources/llm.py` publishes it (0-1 ratio + verdict).""" + return {"value": value, "source": "llamacpp:kv_cache_usage_ratio", "derived": False, + "severity": verdict} + + +# --------------------------------------------------------------------------- # +# the all-clear, and the things that must not look like a problem +# --------------------------------------------------------------------------- # +def test_a_healthy_fleet_alerts_nothing(host_scrape: Any, agent_snapshot: dict[str, Any]) -> None: + payload = host_scrape(agent_snapshot, cpu_busy=10.0) + assert alerts.build(SETTINGS, [state()], [payload], [], ups_payload(), [], now=2_000.0) == [] + + +def test_no_inputs_at_all_is_all_clear_rather_than_an_error() -> None: + """A fresh install with nothing wired up has nothing wrong with it.""" + assert alerts.build(SETTINGS) == [] + + +def test_an_unconfigured_source_never_alerts() -> None: + """`AGENTS.md` invariant 3: not installed is not broken. + + The health record for a disabled source says `ok: false` exactly like a dead one, so + `configured` is the only thing that tells those two apart — and the state map having + nothing for a source must not be read as a failure either. + """ + out = alerts.build(SETTINGS, [ + state(ok=False, configured=False, error="Connection refused"), + state(key="nut", dashboard="ups", label="NUT server", ok=False, configured=False), + ], [], [], ups_payload(), now=2_000.0) + assert out == [] + + +def test_a_card_with_no_checks_and_a_source_with_no_state_say_nothing() -> None: + assert alerts.build(SETTINGS, [], [], [], ups_payload(ups_card("apc"))) == [] + + +# --------------------------------------------------------------------------- # +# reachability +# --------------------------------------------------------------------------- # +def test_a_source_that_stopped_answering_is_a_fail_naming_it_and_linking_its_page() -> None: + out = alerts.build(SETTINGS, [state(ok=False, error="Connection refused")], now=2_000.0) + assert len(out) == 1 + alert = out[0] + assert alert["severity"] == "fail" + assert alert["source"] == "driveripper" + assert alert["href"] == "/system" + assert alert["message"] == "driveripper: Connection refused" + assert alert["at"] == 2_000.0 + + +def test_the_collectors_host_prefix_is_not_doubled() -> None: + """Collectors already say `driveripper: …`; the alert starts with the name too.""" + out = alerts.build(SETTINGS, [state(ok=False, + error="driveripper: http://h:9121/snapshot unreachable")]) + assert out[0]["message"] == "driveripper: http://h:9121/snapshot unreachable" + + +def test_the_pollers_exception_prefix_is_not_printed_either() -> None: + """The live shape is `poller`'s `UpstreamError: name: …`; one name, no class noise. + + Verified against a real stopped agent tunnel: the health record reads + `UpstreamError: gateway: http://127.0.0.1:9123/snapshot unreachable (ConnectError)`, + and printing that verbatim put the host name on the page twice. + """ + out = alerts.build(SETTINGS, [state(key="host:gateway", label="gateway", ok=False, + error="UpstreamError: gateway: http://h:9123/snapshot " + "unreachable (ConnectError)")]) + assert out[0]["message"] == ("gateway: http://h:9123/snapshot unreachable (ConnectError)") + + +def test_an_unexpected_exception_class_is_still_named() -> None: + """Only `UpstreamError` is noise; a bug in our own code is worth reading about.""" + out = alerts.build(SETTINGS, [state(ok=False, error="ValueError: bad snapshot")]) + assert out[0]["message"] == "driveripper: ValueError: bad snapshot" + + +def test_a_failing_source_with_no_recorded_error_still_says_so() -> None: + out = alerts.build(SETTINGS, [state(key="llm:gufo", dashboard="llm", label="gufo", + ok=False, error=None)]) + assert out[0]["message"] == "gufo: is not answering" + assert out[0]["href"] == "/llm" + + +def test_a_stale_but_serving_source_is_a_warn_because_the_page_still_shows_numbers() -> None: + """Last good value on screen + no refresh for 3 intervals = amber, not `dead`.""" + out = alerts.build(SETTINGS, [state(age=45.0)], now=2_000.0) + assert [item["severity"] for item in out] == ["warn"] + assert "45s ago" in out[0]["message"] + assert "last good value" in out[0]["message"] + assert out[0]["at"] == 2_000.0 - 45.0 + # inside the grace period (3 x 10s) the same source is not news at all + assert alerts.build(SETTINGS, [state(age=25.0)]) == [] + + +def test_staleness_is_measured_in_each_sources_own_poll_interval() -> None: + """system polls every 10s and the UPS every 5s — one global number is wrong for one.""" + llm = state(key="llm:gufo", dashboard="llm", label="gufo", age=5.0) # 3 x 2s = 6s + ups = state(key="nut", dashboard="ups", label="NUT server", age=20.0) # 3 x 5s = 15s + assert alerts.build(SETTINGS, [llm]) == [] + out = alerts.build(SETTINGS, [ups]) + assert out[0]["severity"] == "warn" and out[0]["href"] == "/ups" + + +def test_a_dashboard_the_builder_does_not_know_about_still_gets_a_verdict_and_a_home() -> None: + """A new dashboard must not be silently un-alertable while its interval is unknown.""" + unknown_down = {"key": "probe:0", "dashboard": "telemetry", "label": "probe", + "configured": True, "ok": False, "error": "no route"} + assert alerts.build(SETTINGS, [unknown_down])[0]["href"] == "/" + unknown_stale = {**unknown_down, "ok": True, "error": None, "age_seconds": 200.0} + out = alerts.build(SETTINGS, [unknown_stale]) + assert "every 60s" in out[0]["message"] + + +def test_a_failing_source_says_when_the_poller_first_noticed() -> None: + events = [{"ts": 1_500.0, "source": "system", "host": "driveripper", "kind": "unreachable", + "severity": "error", "message": "ConnectError", "data": None}, + {"ts": 1_600.0, "source": "host", "host": "deskwork", "kind": "reboot", + "severity": "warn", "message": "deskwork rebooted", "data": None}] + out = alerts.build(SETTINGS, [state(ok=False, error="nope")], [], [], None, events, + now=2_000.0) + assert out[0]["at"] == 1_500.0, "the outage clock must not reset on every refresh" + + +# --------------------------------------------------------------------------- # +# host thresholds, degraded agents +# --------------------------------------------------------------------------- # +def test_a_threshold_breach_names_the_host_the_metric_and_the_value( + host_scrape: Any, agent_snapshot: dict[str, Any]) -> None: + payload = host_scrape(agent_snapshot, cpu_busy=95.0) + out = alerts.build(SETTINGS, [state()], [payload], now=2_000.0) + assert out[0]["id"] == "driveripper:cpu.usage_pct" + assert out[0]["message"] == "driveripper: CPU 95% (fail ≥90)" + assert out[0]["href"] == "/system" + assert out[0]["at"] == 1_010.0, "`at` is the poll that measured it" + + +def test_every_rated_host_field_alerts_at_its_own_verdict( + host_scrape: Any, agent_snapshot: dict[str, Any]) -> None: + def nudge(snap: dict[str, Any]) -> None: + snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.85 # warn ≥80 + snap["cpu_temperature_celsius"] = 80.0 # warn ≥75 + for fs in snap["filesystems"]: + if fs["mountpoint"] == "/btrfs/pool1": + fs["used_pct"] = 91.0 # fail ≥90 + + payload = host_scrape(agent_snapshot, cpu_busy=10.0, mutate=nudge) + out = {item["id"]: item for item in alerts.build(SETTINGS, [state()], [payload])} + assert out["driveripper:mem.used_pct"]["message"] == "driveripper: memory 85% (warn ≥80)" + assert out["driveripper:cpu.temp_c"]["message"] == ( + "driveripper: CPU temperature 80°C (warn ≥75)") + assert out["driveripper:disk.used_pct"]["message"] == ( + "driveripper: disk 91% on /btrfs/pool1 (fail ≥90)") + assert "driveripper:cpu.usage_pct" not in out, "10% busy is not a problem" + + +def test_a_number_the_upstream_does_not_publish_is_never_a_breach( + host_scrape: Any, agent_snapshot: dict[str, Any]) -> None: + """No measurement, no verdict, no alert — the same rule `severity.classify` upholds.""" + payload = host_scrape(agent_snapshot, cpu_busy=10.0, + mutate=lambda snap: snap.pop("cpu_temperature_celsius")) + assert alerts.build(SETTINGS, [state()], [payload]) == [] + + +def test_a_gpu_breach_names_the_card_and_dedupes_per_metric( + host_scrape: Any, agent_snapshot: dict[str, Any]) -> None: + def fill_the_card(snap: dict[str, Any]) -> None: + gpu = snap["gpus"][0] + gpu["busy_pct"] = 97.0 + for used, total in (("vram_used_bytes", "vram_total_bytes"), + ("gtt_used_bytes", "gtt_total_bytes")): + if gpu.get(total) is not None: + gpu[used] = gpu[total] + + payload = host_scrape(agent_snapshot, cpu_busy=10.0, mutate=fill_the_card) + messages = {item["id"]: item["message"] for item in alerts.build(SETTINGS, [state()], [payload])} + assert any("GPU utilisation 97% on Radeon AI PRO R9700" in text + for text in messages.values()), messages + assert any("GPU memory" in text and "R9700" in text for text in messages.values()) + + +def test_a_degraded_agent_warns_and_names_the_sections_that_failed( + host_scrape: Any, agent_snapshot: dict[str, Any]) -> None: + payload = host_scrape(agent_snapshot, cpu_busy=10.0) + payload["degraded"] = {"temperatures": "no hwmon sensors", "disk_io": ""} + out = alerts.build(SETTINGS, [state()], [payload]) + assert len(out) == 1, "a partial snapshot is one warning, not one per number that is missing" + assert out[0]["id"] == "driveripper:agent.sections" + assert out[0]["severity"] == "warn" + assert "temperatures (no hwmon sensors)" in out[0]["message"] + assert "disk_io" in out[0]["message"] + + +def test_a_degraded_list_from_an_older_agent_is_readable_too( + host_scrape: Any, agent_snapshot: dict[str, Any]) -> None: + payload = host_scrape(agent_snapshot, cpu_busy=10.0) + payload["degraded"] = ["network"] + out = alerts.build(SETTINGS, [state()], [payload]) + assert out[0]["message"] == "driveripper: the agent could not read network" + + +def test_a_degraded_note_that_is_a_plain_string_is_still_reported( + host_scrape: Any, agent_snapshot: dict[str, Any]) -> None: + """An agent (or a future collector) that reports one string instead of a map.""" + payload = host_scrape(agent_snapshot, cpu_busy=10.0) + payload["degraded"] = "hwmon" + out = alerts.build(SETTINGS, [state()], [payload]) + assert out[0]["message"] == "driveripper: the agent could not read hwmon" + + +def test_an_unreachable_host_is_one_alert_even_with_its_last_good_payload( + host_scrape: Any, agent_snapshot: dict[str, Any]) -> None: + """The exact completion criterion for the phase: one line naming the dead host.""" + def nudge(snap: dict[str, Any]) -> None: + snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.99 + snap["cpu_temperature_celsius"] = 95.0 + + payload = host_scrape(agent_snapshot, cpu_busy=96.0, mutate=nudge) + out = alerts.build(SETTINGS, [state(ok=False, error="timed out")], [payload]) + assert len(out) == 1 + assert out[0]["severity"] == "fail" and "timed out" in out[0]["message"] + + +def test_a_stub_payload_for_a_host_that_never_answered_alerts_once() -> None: + """/api/system/snapshot represents that host as {name, unreachable}; still one line.""" + stub = {"name": "gateway", "ts": 1_000.0, "unreachable": True} + out = alerts.build(SETTINGS, [state(key="host:gateway", label="gateway", ok=False, + error="unreachable")], [stub]) + assert [item["id"] for item in out] == ["gateway:host:gateway:down"] + + +# --------------------------------------------------------------------------- # +# LLM engines +# --------------------------------------------------------------------------- # +def test_kv_pressure_comes_from_the_rated_cell_and_reads_as_a_percent() -> None: + """The engine publishes 0-1; the operator reads percent, matching the verdict's scale.""" + out = alerts.build(SETTINGS, [state(key="llm:gufo", dashboard="llm", label="gufo")], [], + [engine(kv_cache_usage_ratio=ratio(0.95, "fail"))]) + assert out[0]["message"] == "gufo: KV cache 95% (fail ≥90)" + assert out[0]["href"] == "/llm" + assert out[0]["id"] == "gufo:llm.kv_cache_usage_pct@kv_cache_usage_ratio" + + +def test_vllm_publishes_the_same_verdict_under_its_own_series_name() -> None: + out = alerts.build(SETTINGS, [], [], [engine("vllm", gpu_cache_usage=ratio(0.8, "warn"))]) + assert out[0]["message"] == "vllm: KV cache 80% (warn ≥75)" + + +def test_a_cool_or_absent_kv_cell_is_not_an_alert() -> None: + for cell in (ratio(0.10, "ok"), ratio(None, None), ratio(0.95, None), {}, "not a cell"): + out = alerts.build(SETTINGS, [], [], [engine(kv_cache_usage_ratio=cell)]) + assert out == [], cell + + +def test_an_engine_that_is_not_answering_alerts_only_its_own_job() -> None: + dead = {"engine": "vllm", "metrics": {}, "sessions": {}, "unreachable": True} + out = alerts.build(SETTINGS, [state(key="llm:vllm", dashboard="llm", label="vLLM", + ok=False, error="502 Bad Gateway")], [], [dead]) + assert [item["id"] for item in out] == ["vLLM:llm:vllm:down"] + + +# --------------------------------------------------------------------------- # +# UPS +# --------------------------------------------------------------------------- # +def test_a_failing_ups_check_is_a_fail_carrying_the_check_and_its_detail() -> None: + payload = ups_payload(ups_card("apc", + check("Power path", "fail", "The UPS is on battery"), + check("Charge", "warn", "42% and discharging"))) + out = alerts.build(SETTINGS, [state(key="nut", dashboard="ups", label="NUT server")], + [], [], payload) + by_id = {item["id"]: item for item in out} + assert by_id["apc:health:Power path"]["severity"] == "fail" + assert by_id["apc:health:Power path"]["message"] == "Power path: The UPS is on battery" + assert by_id["apc:health:Charge"]["severity"] == "warn" + assert all(item["href"] == "/ups" for item in out) + + +def test_the_landing_page_is_where_battery_age_belongs() -> None: + """The task's own example: a sentence an operator can act on, not a score.""" + detail = "5.3 years — past the 4-year replacement window" + out = alerts.build(SETTINGS, [], [], [], ups_payload(ups_card("apc", + check("Battery age", "warn", detail)))) + assert out[0]["message"] == f"Battery age: {detail}" + assert out[0]["source"] == "apc" + + +def test_an_unknown_check_is_silence_and_a_check_with_no_detail_is_still_readable() -> None: + payload = ups_payload(ups_card("apc", check("Battery temperature", "unknown", + "not reported by this driver"), + check("Alarms", "fail"))) + out = alerts.build(SETTINGS, [], [], [], payload) + assert [item["message"] for item in out] == ["Alarms"] + + +def test_simulated_ups_data_is_announced_once_and_loudly() -> None: + """With NUT_SIMULATE=1 and otherwise-healthy cards, that warning is the whole list.""" + payload = ups_payload(ups_card("apc", check("Charge", "ok", "100%")), + ups_card("cyberpower", check("Charge", "ok", "100%")), + simulated=True) + out = alerts.build(SETTINGS, [state(key="nut", dashboard="ups", label="NUT server")], + [], [], payload) + assert len(out) == 1 + assert out[0]["severity"] == "warn" + assert "NUT_SIMULATE=1" in out[0]["message"] + assert out[0]["href"] == "/ups" + + +async def test_the_simulator_on_its_own_produces_exactly_that_warning() -> None: + """The task's acceptance line, through the real simulator and the real payload builder.""" + readings = await NutSimulator().read() + payload = {"ts": 1_000.0, "simulated": True, + "ups": [nut.payload(reading) for reading in readings]} + out = alerts.build(SETTINGS, + [state(key="nut", dashboard="ups", label="NUT server")], [], [], payload) + assert [item["id"] for item in out] == ["ups:simulated"], out + + +def test_a_real_upsd_does_not_claim_to_be_simulated() -> None: + out = alerts.build(SETTINGS, [], [], [], ups_payload(ups_card("apc"))) + assert out == [] + + +def test_a_dead_upsd_is_one_alert_and_not_its_stale_cards_too() -> None: + payload = ups_payload(ups_card("apc", check("Power path", "fail", "on battery"))) + out = alerts.build(SETTINGS, [state(key="nut", dashboard="ups", label="NUT server", + ok=False, error="upsd closed the connection")], + [], [], payload) + assert [item["source"] for item in out] == ["NUT server"] + + +# --------------------------------------------------------------------------- # +# configuration +# --------------------------------------------------------------------------- # +def test_a_bad_target_entry_is_a_warn_that_says_it_is_not_being_polled() -> None: + settings = Settings(host_target_problems=("nope is not name=url", + "deskwork: url must start with http://")) + out = alerts.build(settings) + assert len(out) == 2 + assert all(item["severity"] == "warn" and item["source"] == "config" for item in out) + assert all(item["href"] == "/system" for item in out) + assert "HOST_TARGETS" in out[0]["message"] and "not being polled" in out[0]["message"] + + +# --------------------------------------------------------------------------- # +# dedupe, bound, order, shape +# --------------------------------------------------------------------------- # +def test_one_alert_per_source_and_metric_with_the_worst_verdict_winning() -> None: + """Two cards answering as the same name (a re-exported UPS) must not double the alarm.""" + payload = ups_payload(ups_card("apc", check("Charge", "warn", "60%")), + ups_card("apc", check("Charge", "fail", "12%"))) + out = alerts.build(SETTINGS, [], [], [], payload) + assert len(out) == 1 + assert out[0]["severity"] == "fail" + assert out[0]["message"] == "Charge: 12%" + + +def test_the_list_is_bounded_and_says_how_many_it_hid() -> None: + cards = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery")) + for i in range(alerts.MAX_ALERTS + 5)] + out = alerts.build(SETTINGS, [], [], [], ups_payload(*cards)) + assert len(out) == alerts.MAX_ALERTS + marker = out[-1] + assert marker["id"] == "alerts:suppressed" + assert marker["severity"] == "fail", "hiding reds behind an amber marker is a lie" + hidden = len(cards) - (alerts.MAX_ALERTS - 1) # one slot is the marker itself + assert marker["message"] == f"{hidden} more not shown ({hidden} failing, 0 warning)" + + +def test_the_marker_is_the_worst_verdict_it_did_not_show() -> None: + cards = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery")) + for i in range(alerts.MAX_ALERTS)] + cards += [ups_card(f"warn{i:02d}", check("Charge", "warn", "40%")) for i in range(3)] + out = alerts.build(SETTINGS, [], [], [], ups_payload(*cards)) + assert len(out) == alerts.MAX_ALERTS + assert out[-1]["id"] == "alerts:suppressed" + assert out[-1]["severity"] == "fail", "a red is behind the marker, so the marker is red" + # and when only warnings are hidden, the marker says amber — it describes what it hid + mild = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery")) + for i in range(3)] + mild += [ups_card(f"warn{i:02d}", check("Charge", "warn", "40%")) + for i in range(alerts.MAX_ALERTS)] + out = alerts.build(SETTINGS, [], [], [], ups_payload(*mild)) + assert out[-1]["severity"] == "warn" + + +def test_exactly_the_cap_adds_no_marker() -> None: + cards = [ups_card(f"ups{i:02d}", check("Power path", "fail", "on battery")) + for i in range(alerts.MAX_ALERTS)] + out = alerts.build(SETTINGS, [], [], [], ups_payload(*cards)) + assert len(out) == alerts.MAX_ALERTS + assert all(item["id"] != "alerts:suppressed" for item in out) + + +def test_fails_sort_before_warns_and_then_by_source() -> None: + payload = ups_payload(ups_card("zeta", check("Power path", "fail", "on battery")), + ups_card("alpha", check("Charge", "warn", "40%")), + ups_card("beta", check("Load", "warn", "82%"))) + out = alerts.build(SETTINGS, [], [], [], payload) + assert [(item["severity"], item["source"]) for item in out] == [ + ("fail", "zeta"), ("warn", "alpha"), ("warn", "beta")] + + +def test_the_trip_note_quotes_the_threshold_table_in_the_right_direction() -> None: + """`_trip` reads `severity.THRESHOLDS` back instead of carrying its own numbers. + + Reached through `build()` only by metrics that are not inverted today, so this checks + the helper directly: a low-is-worse metric must print `≤` (an `alert` for a battery is + exactly where a wrong sign would be published), and a metric with no threshold gets no + note rather than a made-up one. + """ + assert alerts._trip("cpu.usage_pct", "fail") == " (fail ≥90)" + assert alerts._trip("ups.battery_runtime", "warn") == " (warn ≤600)" + assert alerts._trip("ups.charge_pct", "fail") == " (fail ≤20)" + assert alerts._trip("cpu.usage_pct", "ok") == "" + assert alerts._trip("mystery.metric", "fail") == "" + + +def test_the_alert_shape_is_the_contract_the_pages_render() -> None: + out = alerts.build(SETTINGS, [state(ok=False, error="nope")], now=2_000.0) + assert out[0] == {"id": "driveripper:host:driveripper:down", "severity": "fail", + "source": "driveripper", "message": "driveripper: nope", + "href": "/system", "at": 2_000.0} diff --git a/tests/test_api.py b/tests/test_api.py index bf76430..4c4b4ed 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -7,30 +7,38 @@ asset the network can fail to deliver. """ from __future__ import annotations -import copy import re import time from collections.abc import Iterator +from contextlib import contextmanager from pathlib import Path from typing import Any import pytest +from fastapi import HTTPException from fastapi.testclient import TestClient +from dashboards import alerts +from dashboards.api import RANGE_SECONDS, parse_range from dashboards.app import create_app from dashboards.config import Settings, Target -from dashboards.rates import RateEstimator from dashboards.severity import classify -from dashboards.sources.host import HostCollector, parse_host_snapshot +from dashboards.sources import nut from dashboards.store import Event, Sample, SampleStore # refused immediately, and never dials out of the test environment DEAD_URL = "http://127.0.0.1:1" +@contextmanager def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: bool = False, - **overrides: Any) -> TestClient: - """A TestClient whose lifespan runs, against a private DB and no real network.""" + **overrides: Any) -> Iterator[TestClient]: + """A TestClient whose lifespan runs, against a private DB and no real network. + + The store is injected, so `create_app()` deliberately does not close it (production + owns its own); the caller here does, or every test leaks a SQLite handle and the + `--cov` log fills with ResourceWarnings that bury the warnings worth reading. + """ if simulate: monkeypatch.setenv("NUT_SIMULATE", "1") else: @@ -38,10 +46,13 @@ def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: b db = str(tmp_path / "api.sqlite3") settings = Settings(db_path=db, **overrides) store = SampleStore(db) - app = create_app(settings, store) - client = TestClient(app) + client = TestClient(create_app(settings, store)) client.__enter__() - return client + try: + yield client + finally: + client.__exit__(None, None, None) + store.close() @pytest.fixture @@ -68,29 +79,23 @@ def wait_for_payload(client: TestClient, path: str, *, timeout: float = 5.0) -> pytest.fail(f"{path} never became available") -def host_payload(agent_snapshot: dict[str, Any]) -> dict[str, Any]: +def failing_host_payload(scrape: Any, agent_snapshot: dict[str, Any]) -> dict[str, Any]: """Two scrapes of the real driveripper capture, nudged onto known sides of the table. Driven from the fixture rather than hand-written so a schema drift breaks here first. """ - snap = copy.deepcopy(agent_snapshot) - snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95 # mem.used_pct -> fail - snap["cpu_temperature_celsius"] = 91.0 # cpu.temp_c -> fail - for fs in snap["filesystems"]: - fs["used_pct"] = 93.0 # disk -> fail - collector = HostCollector(RateEstimator(span=60.0)) - parse_host_snapshot("driveripper", snap, collector, now=1_000.0) - second = copy.deepcopy(snap) - second["ts"] = 1_010.0 - modes = second["cpu"]["modes"] - modes["total"] += 1_000.0 - modes["idle"] += 900.0 # 10 % busy over the window -> ok - return parse_host_snapshot("driveripper", second, collector, now=1_010.0).payload + def nudge(snap: dict[str, Any]) -> None: + snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95 # mem.used_pct -> fail + snap["cpu_temperature_celsius"] = 91.0 # cpu.temp_c -> fail + for fs in snap["filesystems"]: + fs["used_pct"] = 93.0 # disk -> fail + + return scrape(agent_snapshot, cpu_busy=10.0, mutate=nudge) @pytest.fixture def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, - agent_snapshot: dict[str, Any]) -> Iterator[TestClient]: + agent_snapshot: dict[str, Any], host_scrape: Any) -> Iterator[TestClient]: """Overview-shaped app: one host payload in the hub plus the simulated UPSes. The host job points at a dead port, so a poll can never overwrite the injected payload @@ -104,7 +109,8 @@ def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, test_client = TestClient(app) test_client.__enter__() try: - app.state.ctx.hub.jobs["host:driveripper"].state.payload = host_payload(agent_snapshot) + app.state.ctx.hub.jobs["host:driveripper"].state.payload = failing_host_payload( + host_scrape, agent_snapshot) wait_for_payload(test_client, "/api/ups/snapshot") yield test_client finally: @@ -112,6 +118,41 @@ def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, store.close() +@pytest.fixture +def healthy_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, + agent_snapshot: dict[str, Any], + host_scrape: Any) -> Iterator[TestClient]: + """The all-clear app: one host answering, nothing past a threshold, no UPS configured. + + The job's collector is swapped for one that hands back the injected payload, so the + health record says `ok` truthfully instead of being overridden behind the poller. + """ + with build_client(tmp_path, monkeypatch, + host_targets=(Target("driveripper", DEAD_URL),)) as test_client: + from dashboards.poller import Result # noqa: PLC0415 - only this fixture needs it + + payload = host_scrape(agent_snapshot, cpu_busy=10.0) + + async def collect() -> Result: + return Result(payload=payload, samples=[]) + + job = test_client.app.state.ctx.hub.jobs["host:driveripper"] # type: ignore[attr-defined] + job.fn = collect + # let the poll already in flight against the dead port land first (it fails in + # microseconds), then take the health record over: the alerts must describe a host + # that is answering, not the half-second of startup where it was not + deadline = time.time() + 2.0 + while time.time() < deadline and job.state.attempts == 0: + time.sleep(0.02) + time.sleep(0.05) + job.state.payload = payload + job.state.ok = True + job.state.error = None + job.state.consecutive_failures = 0 + job.state.last_ok = time.time() + yield test_client + + # ---- severity travels with the value --------------------------------------- # def test_summary_carries_the_verdict_for_every_column_the_page_colours(data_client) -> None: @@ -147,13 +188,121 @@ def test_summary_shape_did_not_shrink(data_client) -> None: assert fleet_keys <= set(body["system"]["fleet"]), fleet_keys - set(body["system"]["fleet"]) assert host_keys <= set(body["system"]["hosts"][0]), host_keys - set(body["system"]["hosts"][0]) assert ups_keys <= set(body["ups"]["ups"][0]), ups_keys - set(body["ups"]["ups"][0]) - assert set(body) == {"ts", "system", "llm", "ups", "recent_events"} + assert set(body) == {"ts", "system", "llm", "ups", "recent_events", "alerts"} # ...and the new keys are there beside the old ones, not instead of them assert "severity" in body["system"]["fleet"] assert "severity" in body["system"]["hosts"][0] assert "severity" in body["ups"]["ups"][0] +# `/` paints two panels; the payload is still the whole-fleet contract. Every path here is +# read by something that is not the overview page: the live suite, an external probe, a +# notifier later. Deleting a card must never delete a key. +SUMMARY_CONTRACT = ("system.enabled", "system.fleet", "system.hosts", + "llm.engines", "ups.ups", "recent_events", "alerts") + + +def dig(body: dict[str, Any], path: str) -> Any: + """`body["a"]["b"]` that names the missing path when it is not there.""" + node: Any = body + for part in path.split("."): + assert isinstance(node, dict) and part in node, f"/api/summary lost `{path}`" + node = node[part] + return node + + +@pytest.mark.parametrize("fixture", ["client", "data_client"]) +def test_summary_serves_every_path_another_consumer_reads(request, fixture: str) -> None: + """The keys survive the overview cleanup whether or not anything is configured. + + Run twice on purpose: `client` has no targets at all, so it proves the keys are present + as `None`/`[]` rather than omitted when a source is unconfigured (invariant 3 is about + the payload shape as much as about colour); `data_client` proves the same paths carry + the fleet once there is one. + """ + test_client = request.getfixturevalue(fixture) + body = test_client.get("/api/summary").json() + populated = fixture == "data_client" + for path in SUMMARY_CONTRACT: + dig(body, path) # presence is the assertion; a missing path raises with its name + assert dig(body, "system.enabled") is populated + assert (dig(body, "system.fleet") is not None) is populated + names = {host["name"] for host in dig(body, "system.hosts")} + assert names == ({"driveripper"} if populated else set()) + batteries = {item["name"] for item in dig(body, "ups.ups")} + assert batteries == ({"apc", "cyberpower"} if populated else set()) + assert dig(body, "llm.engines") == [], "neither fixture configures an engine" + assert isinstance(dig(body, "recent_events"), list) + assert isinstance(dig(body, "alerts"), list) + # and the payload as a whole still carries every top-level section it always did + assert set(body) >= {"ts", "system", "llm", "ups", "recent_events", "alerts"} + + +# ---- alerts: the landing page's answer to "what is wrong" ------------------- # + +def test_summary_is_empty_of_alerts_when_everything_is_healthy(healthy_client) -> None: + """`[]`, not a green banner: an empty list is the clearest possible all-clear.""" + body = healthy_client.get("/api/summary").json() + assert body["alerts"] == [] + assert body["system"]["fleet"]["hosts_live"] == 1 + + +def test_summary_raises_one_alert_for_the_host_that_stopped_answering(data_client) -> None: + """One dead host is one line naming it, with the link to the page that explains it.""" + body = data_client.get("/api/summary").json() + down = [item for item in body["alerts"] if item["source"] == "driveripper"] + assert len(down) == 1, "its stale payload must not add amber lines on top of the red one" + assert down[0]["severity"] == "fail" + assert down[0]["href"] == "/system" + assert "unreachable" in down[0]["message"] + + +def test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """A dead agent is a red row in the fleet, not a fleet that quietly got smaller. + + `/api/system/snapshot` already stubs a job with no payload; `/api/summary` has to agree + with it, or a target that never answered since startup leaves the table at "2/2 hosts" + while the panel above says a third box is down (`AGENTS.md` invariants 3 and 4). + """ + with build_client(tmp_path, monkeypatch, + host_targets=(Target("driveripper", DEAD_URL),)) as test_client: + body = test_client.get("/api/summary").json() + rows = body["system"]["hosts"] + assert [row["name"] for row in rows] == ["driveripper"] + assert rows[0]["unreachable"] is True + assert rows[0]["cpu_usage_pct"] is None, "n/a, never a 0.0 nobody measured" + assert body["system"]["fleet"]["hosts_total"] == 1 + assert body["system"]["fleet"]["hosts_live"] == 0 + # and the alert list stays exactly one line for it, not a row plus a stale payload + assert [item["source"] for item in body["alerts"]] == ["driveripper"] + + +def test_summary_alerts_are_worst_first_and_carry_the_contract(data_client) -> None: + body = data_client.get("/api/summary").json() + alerts_list = body["alerts"] + assert alerts_list, "a dead host plus simulated UPS data cannot read as all-clear" + for item in alerts_list: + assert set(item) == {"id", "severity", "source", "message", "href", "at"} + assert item["severity"] in ("warn", "fail") + ranks = [0 if item["severity"] == "fail" else 1 for item in alerts_list] + assert ranks == sorted(ranks), "fails before warns" + assert any("simulat" in item["message"].lower() and item["href"] == "/ups" + for item in alerts_list), "NUT_SIMULATE must be said out loud on the landing page" + + +def test_summary_reuses_the_alert_builder_verbatim(data_client) -> None: + """The endpoint is a thin call: what `alerts.build` says over the same state is the body.""" + body = data_client.get("/api/summary").json() + hub = data_client.app.state.ctx.hub # type: ignore[attr-defined] + settings = data_client.app.state.ctx.settings # type: ignore[attr-defined] + hosts = [job.state.payload for job in hub.jobs.values() + if job.dashboard == "system" and job.state.payload] + ups_payload = hub.jobs["nut"].state.payload + expected = alerts.build(settings, hub.states(), hosts, [], ups_payload, []) + assert [item["id"] for item in body["alerts"]] == [item["id"] for item in expected] + + def test_snapshot_payloads_carry_their_own_severity_sections(data_client) -> None: """/api/system/snapshot and /api/ups/snapshot ship the same verdicts the summary reuses.""" system = data_client.get("/api/system/snapshot").json() @@ -371,14 +520,103 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None: def test_severity_for_is_kept_as_the_documented_fallback(client) -> None: - """The old call sites still work until phases 02-05 convert each page.""" + """The old call sites still work until the remaining phases convert each page. + + `/` was converted in phase 02: the overview colours from the `severity` verdicts + `/api/summary` already sends, so it no longer carries a threshold of its own. The + three pages still to come must keep compiling against the fallback until they are. + """ js = client.get("/static/app.js").text assert "FALLBACK ONLY" in js - for page in ("/static/page-index.js", "/static/page-system.js", - "/static/page-llm.js", "/static/page-ups.js"): + for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"): assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site" +def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> None: + """`page-index.js` reads verdicts; it does not compare a number to a bound.""" + js = client.get("/static/page-index.js").text + assert "severityFrom" in js + assert "severityFor" not in js + + +def test_overview_is_exactly_two_panels(client) -> None: + """The landing page answers one question: what is wrong, and what is the fleet doing. + + Asserted on the served bytes because there is no JS test runner (a locked anchor): + the summary cards and the raw event feed are what made this page busy, and once a + panel is gone a browser is the only thing that would notice its return. + """ + html = client.get("/").text + assert html.count(" None: + """Every alert carries the `href` its row turns into a link; the page invents none.""" + page_js = client.get("/static/page-index.js").text + assert "alert.href" in page_js + css = client.get("/static/dash.css").text + for selector in ("ul.alerts li.ok .src", "ul.alerts li.warn .src", "ul.alerts li.fail .src", + "table.data td.warn", "table.data td.fail"): + assert selector in css, f"dash.css lost {selector}" + # the swap that puts a live problem above the fleet table + assert 'data-state="alerting"' in css + + +def test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering(client) -> None: + """A dead host keeps its last good numbers, so the row has to say they are stale. + + `AGENTS.md` invariant 4: keep the payload and mark it stale, never zero it. The + overview's table is the one place those last good numbers still print, so it reads the + per-source verdicts `/api/summary` sends (`system.states`) and applies the shared + `.stale` class the other pages already use — no new palette, no JS threshold. + """ + page_js = client.get("/static/page-index.js").text + assert "system.states" in page_js, "the fleet table stopped reading reachability" + assert 'class="stale"' in page_js + assert ".stale" in client.get("/static/dash.css").text + + +# Which script owns which page, for the orphan check below. `charts.js` and `app.js` are +# loaded by all four pages, so an id either page code or shared chrome may read. +PAGE_SCRIPTS = { + "/": "page-index.js", + "/system": "page-system.js", + "/llm": "page-llm.js", + "/ups": "page-ups.js", +} +SHARED_SCRIPTS = ("app.js", "charts.js") + + +def test_no_page_carries_an_id_nothing_reads(client) -> None: + """An id no script queries, no stylesheet styles and no attribute points at is dead markup. + + The summary cards were retired this way — a `` left behind after its + writer was deleted is the same mistake in slower motion. Asserted on served bytes: the + browser console is otherwise the only thing that would notice. + """ + scripts = {name: client.get(f"/static/{name}").text for name in + (*SHARED_SCRIPTS, *PAGE_SCRIPTS.values(), "dash.css")} + for path, own_script in PAGE_SCRIPTS.items(): + html = _code(client.get(path).text) + corpus = "".join(_code(scripts[name]) + for name in (*SHARED_SCRIPTS, own_script, "dash.css")) + # ids the markup uses itself: labels, targets, `
    HostCPULoad 1Mem usedWorst diskGPU busyTempUp
    ${DashCharts.escapeHtml(host.name)}${host.cpu_usage_pct == null ? 'n/a' : num(host.cpu_usage_pct, 0) + '%'}${host.cpu_usage_pct == null ? 'n/a' : num(host.cpu_usage_pct, 0) + '%'} ${host.load1 == null ? 'n/a' : num(host.load1, 2)}${host.mem_used_pct == null ? 'n/a' : num(host.mem_used_pct, 0) + '%'}${host.disk_worst_used_pct == null ? 'n/a' : num(host.disk_worst_used_pct, 0) + '%'}${host.mem_used_pct == null ? 'n/a' : num(host.mem_used_pct, 0) + '%'}${host.disk_worst_used_pct == null ? 'n/a' : num(host.disk_worst_used_pct, 0) + '%'} ${gpus.length ? num(gpus[0], 0) + '%' : '—'}${host.temp_c == null ? 'n/a' : num(host.temp_c, 0) + '°C'}${host.temp_c == null ? 'n/a' : num(host.temp_c, 0) + '°C'} ${host.uptime_seconds == null ? 'n/a' : duration(host.uptime_seconds)}