phase: 03_system_per_host_panels
**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**
- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
- **Next pending phase**: `04_llm_chart_readability`.
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
# Verification notes — `/system` on every host shape
|
||||
|
||||
**Date:** 2026-10-17 (dev loop: tunnels 9121/9122/9123 + upsd 3494 all OK; dev-server
|
||||
`/api/health` all six sources `ok` before and after verification)
|
||||
|
||||
## What was checked, on live data
|
||||
|
||||
Verified against `http://127.0.0.1:8090/system` with all three real agents answering, in a
|
||||
real browser (headless Chromium/Playwright), DOM assertions plus screenshots.
|
||||
|
||||
| Host (shape) | Result | Screenshot |
|
||||
| --- | --- | --- |
|
||||
| `driveripper` (discrete Radeon AI PRO R9700) | GPU hero: `3 %` ok, `.sub` = "Radeon AI PRO R9700 · VRAM 30.5 GiB / 31.9 GiB · pool 33 %"; util/net/GPU charts all present inside the panel | `system-driveripper.png` (dark), `system-driveripper-light.png` |
|
||||
| `deskwork` (Strix Halo APU, two memory pools) | GPU hero busy went 82 % ok → 85 % warn → 99 % fail across the session, so threshold colour visibly moves; `.sub` = "Radeon 8060S Graphics · VRAM 172.7 MiB / 512.0 MiB · pool 78 %" — small dedicated VRAM and high shared pool read as two facts, not a contradiction; MEM 91 % and TEMP ~83 °C amber | `system-deskwork.png` |
|
||||
| `gateway` (no GPU, no CPU temp sensor) | CPU TEMP `n/a` with **no** severity class; GPU slot = `n/a` + "No GPU reported by this agent", uncoloured; **no** GPU chart and no hole where one would be | `system-gateway.png` (dark), `system-gateway-light.png` |
|
||||
|
||||
Full-page shapes: `system-shapes-dark.png`, `system-shapes-light.png`.
|
||||
|
||||
Failure path: stopped only the gateway tunnel (same commands `dev-tunnels.sh stop` runs,
|
||||
scoped to the 9123 entry; remote agent stopped too). After 15 poll failures the panel kept
|
||||
its last-good heroes (MEM 32 %, DISK 61 %, NET rates — nothing zeroed, nothing `n/a`), its
|
||||
dot turned red, the header read `stale 2.1m`, and the card dimmed via `.stale`. The other
|
||||
two panels stayed green ("one dead source must not blink another"). Store recorded exactly
|
||||
one `unreachable` and, after `./scripts/dev-tunnels.sh start`, one `reachable` event.
|
||||
Screenshot: `system-stale-gateway.png` (+ `system-stale-overview.png` full page). Restored;
|
||||
`dev-tunnels.sh status` and `/api/health` back to all-OK.
|
||||
|
||||
Accessibility pass:
|
||||
|
||||
- hero values are real selectable text (`div.value > span.num/span.unit`, no SVG paths) —
|
||||
asserted in-page for every hero on all three panels;
|
||||
- colour is never the only signal: each hero prints the number, the NET slot is
|
||||
deliberately colourless, staleness reads as text (`updated Ns` / `stale 2.1m`);
|
||||
- charts — **defect found**: the SVGs were anonymous (`aria-label` absent on container and
|
||||
svg). Fixed: `LineChart` gained an `ariaLabel` option that sets `role="img"` +
|
||||
`aria-label` on the `<svg>`, and every `/system` chart now names its host and metrics
|
||||
(e.g. "driveripper — utilisation over the selected window: CPU, memory and worst disk in
|
||||
per cent"). Pinned by `tests/test_api.py::test_every_system_chart_names_itself_for_assistive_tech`.
|
||||
The `/llm` and `/ups` charts can adopt the same option in phases 04/05.
|
||||
|
||||
Both themes checked via the header toggle (severity colours come from theme tokens; the
|
||||
light-theme deskwork fail-red and warn-amber read clearly — see
|
||||
`system-shapes-light.png`, `system-gateway-light.png`). Note for the next shooter: set
|
||||
`prefers-color-scheme` or seed `localStorage['dash-theme']`; Playwright defaults to light,
|
||||
which silently mislabels a "dark" screenshot.
|
||||
|
||||
`curl "localhost:8090/api/system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h"`
|
||||
→ 354 points: dropping the per-core plot did not drop the data.
|
||||
|
||||
## Defects found and fixed during verification
|
||||
|
||||
1. **Unlabelled charts** (above) — fixed in `charts.js` + `page-system.js`, test-pinned,
|
||||
re-screenshotted.
|
||||
2. **`/system` Events panel was permanently empty** — the poller stores transition events
|
||||
under the *dashboard* name (`system`) while the page queried `/api/events?source=host`,
|
||||
so the feed said "No host transitions recorded yet" even with the deliberate gateway
|
||||
outage in the store. Fixed to `source=system` (with a comment on the naming trap);
|
||||
`tests/test_api.py::test_system_event_feed_queries_the_source_transitions_are_stored_under`
|
||||
pins it; the re-shot full-page screenshots show the feed populated with the outage and
|
||||
recovery. Pre-existing since the first commit, not introduced by phase 03.
|
||||
|
||||
## After the fixes
|
||||
|
||||
`bash .agents/validate.sh` → exit 0: 460 passed + 7 skipped, coverage 96.86 % (≥ 94),
|
||||
ruff + pyright clean, `node --check` clean. `uv run pytest -m live --live` → 7 passed.
|
||||
Console stayed error-free throughout.
|
||||
@@ -0,0 +1,10 @@
|
||||
**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**
|
||||
|
||||
- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
|
||||
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
|
||||
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
|
||||
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
|
||||
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
|
||||
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
|
||||
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
|
||||
- **Next pending phase**: `04_llm_chart_readability`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 45%]
|
||||
...........................sssssss...................................... [ 60%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 90%]
|
||||
............................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 175 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
|
||||
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
|
||||
src/dashboards/config.py 113 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 139 1 99% 226
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 226
|
||||
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
|
||||
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2109 66 97%
|
||||
Required test coverage of 94% reached. Total coverage: 96.87%
|
||||
472 passed, 7 skipped, 1 warning in 43.59s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
**Implemented**
|
||||
- `tests/card_contract.py` (new): the single `/system` hero-card payload table — `CARD_METRICS`, `GPU_METRICS`, `SEVERITY_FREE_SECTIONS`, `HOST_SHAPES`, `dig()`, `assert_card_contract()`. Documents the two recorded choices: card temp comes from `cpu.temp_c` (identical to `temperatures.cpu_c`, and what `/api/summary` flattens), disk verdict is `disks.severity.worst_used_pct`; `network` deliberately has no severity.
|
||||
- `tests/test_system_contract.py` (new, 13 tests): drives the real captures through `parse_host_snapshot()` for all three host shapes (discrete GPU / Strix Halo APU / no GPU), checks the paths over HTTP via `/api/system/snapshot`, pins `gpus == []` (not a missing key), `severity: null` for unknowns, absence of `network.severity`, and that dropped visuals (`cpu.usage_pct@coreN`, per-mount, per-interface, sensors, processes) are **still sampled** and `cpu.usage_pct@core7` still queryable from `/api/system/history`.
|
||||
- `tests/test_live_fleet.py`: added `test_every_host_serves_the_hero_card_contract` — same table against the real agents.
|
||||
- `src/dashboards/sources/host.py`: **comments only** (card-temp field choice; why `network` has no severity). No field renamed/removed; no page code touched.
|
||||
- Verified mutation-sensitivity: renaming `worst_mount` or adding `network.severity` fails 10 contract tests.
|
||||
|
||||
**Tests / lint / coverage**
|
||||
- `uv run pytest tests/test_system_contract.py` → 13 passed
|
||||
- `bash .agents/validate.sh` (pytest `--cov --cov-fail-under=94`, ruff, pyright, node --check, hygiene) → exit 0: **452 passed, 7 skipped**, coverage **96.86%** (`sources/host.py` 99%), ruff clean, pyright 0 errors
|
||||
- `uv run pytest -m live --live` (fleet + engines + upsd reachable) → **7 passed** (was 6)
|
||||
- `uv pip list --outdated` → empty; dev server restarted, `/api/health` ok; live `/api/system/snapshot` passes the contract for deskwork/driveripper/gateway
|
||||
|
||||
**Docs**: AGENTS.md (test count 439→452 + new testing-policy bullet on the card-contract table) and README.md (Testing section notes `tests/card_contract.py`).
|
||||
|
||||
**Decisions / deviations**
|
||||
- Contract table lives in `tests/card_contract.py` so hermetic and live tests share one list instead of duplicating paths.
|
||||
- Task text's `disks.worst.severity` is realised as phase 01's sibling form `disks.severity.worst_used_pct` (no nested `disks.worst` object exists); recorded in the test docstring, no payload change.
|
||||
- Phase-file curl example for per-core history uses `?host=&metric=`, but the API takes `hosts=`/`metrics=`; the test uses the real names (phase file left untouched).
|
||||
|
||||
**Next pending task**: `03_system_per_host_panels/02_hero_cards_without_per_core.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 47%]
|
||||
.......sssssss.......................................................... [ 62%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 94%]
|
||||
........................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 175 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
|
||||
src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115
|
||||
src/dashboards/config.py 110 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 139 1 99% 226
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 226
|
||||
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
|
||||
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2104 66 97%
|
||||
Required test coverage of 94% reached. Total coverage: 96.86%
|
||||
452 passed, 7 skipped, 1 warning in 41.93s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
All green. Task complete.
|
||||
|
||||
**Report**
|
||||
|
||||
**Implemented**
|
||||
- `page-system.js`: `hostCard()` rewritten — one `.heroes` row of six slots in fixed order (CPU, MEM, TEMP, DISK, GPU, NET), each with a `data-sev` hook naming its payload verdict; per-core block (`data-k="cores"`, `renderCores`) deleted; Load/Clock/Disk-read/write/busy/Up moved into the collapsed "More detail" `<details>` (with GPU power/temp table, filesystems, processes).
|
||||
- `renderHostCard()` now drives colour via `Dash.hero()` from `cpu.severity.usage_pct/temp_c`, `memory.severity.used_pct`, `disks.severity.worst_used_pct`, `gpus[].severity.busy_pct`; zero `severityFor(` calls left. MEM shows `%` big + `113.5 / 125 GiB` sub; DISK sub is `worst_mount`; GPU = one hero per card (model, VRAM used/total, APU `pool %` in `.sub`) or uncoloured "No GPU reported by this agent"; NET is one slot, two lines via `Dash.rate()`, `n/a` (never `0 B/s`) when null, colourless by design.
|
||||
- `dash.css`: added `.heroes` (auto-fit → 3×2/2×3 wrap), `.hero-slot`, `.hero .hero-lines` — no new colour vocabulary, tokens only.
|
||||
- `app.js`: fallback docstring updated (phases 04–05 remain).
|
||||
- Tests: 2 new byte-pinned tests in `tests/test_api.py` (six heroes in order, no per-core, no JS thresholds, NET has no verdict, detail keys survive); `test_severity_for_is_kept_as_the_documented_fallback` narrowed to `/llm` + `/ups` (same pattern phase 02 applied to `/`).
|
||||
|
||||
**Verification**
|
||||
- `bash .agents/validate.sh` → exit 0: `454 passed, 7 skipped`, coverage 96.86% (floor 94), ruff clean, pyright 0 errors, `node --check` clean.
|
||||
- `uv run pytest -m live --live` → 7 passed (real fleet, incl. hero-card contract on all three agents).
|
||||
- Live browser (dev server restarted, port 8090): `data/logs/system-heroes-live.png`, `-warn-fail-deskwork.png` (91% amber, 95% red GPU), `-nogpu-gateway.png` (TEMP/GPU `n/a` uncoloured), `-driveripper-card.png`, `-more-detail.png`. Colours tracked live threshold crossings (deskwork GPU 95→fail, 94→warn between polls). `cpu.usage_pct@core7` still queryable (355 pts/1h).
|
||||
|
||||
**Docs**: AGENTS.md (`web/` fallback status, new UI-conventions paragraph for the six-hero card) and README.md (`web/` bullet) updated.
|
||||
|
||||
**Decisions**: `.cores` CSS + `renderCores` in `charts.js` kept (phase sanctions charts.js-only; task 03 confirms); fleet sub-line "cores" kept (core count, not per-core visual); hero-row test scopes the per-core grep to the card functions for the same reason.
|
||||
|
||||
**Next pending task**: `03_system_per_host_panels/03_per_host_charts_replace_shared.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 46%]
|
||||
.........sssssss........................................................ [ 62%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 93%]
|
||||
............................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 175 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
|
||||
src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115
|
||||
src/dashboards/config.py 110 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 139 1 99% 226
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 226
|
||||
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
|
||||
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2104 66 97%
|
||||
Required test coverage of 94% reached. Total coverage: 96.86%
|
||||
454 passed, 7 skipped, 1 warning in 42.22s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
**Task complete** — `03_per_host_charts_replace_shared.md` (phase 03). Working tree only, no git operations.
|
||||
|
||||
**Deleted from `/system`**: the fleet totals card, the six fleet-wide charts (`#chart-cpu|mem|disk|net|gpu|temp` + legends), the Series-detail picker (`#detail-host|metric|chart|note`), `<select>`s, `renderCores()` in `charts.js`, and the `select`/`label.field` CSS they alone used. `main` now holds exactly two sections: host panels + events.
|
||||
|
||||
**Each host panel** stacks its own 120px charts under the hero row — util (CPU/mem/worst-disk, one 0-100 axis), net (↓in/↑out), GPU busy per card — GPU slot created and destroyed with `host.gpus`, legends naming metrics (or the GPU model) never hosts, torn down by `pruneHosts` when a host stops reporting.
|
||||
|
||||
**One request per host per refresh**: `/api/system/history?hosts=<host>&metrics=<that panel's series>&range=…`; measured live at 5 XHRs per poll for 3 hosts (1 snapshot + 3 history + events) regardless of chart count. `/api/metrics` discovery no longer used by the page.
|
||||
|
||||
**Tests** (`tests/test_api.py`, +3): `test_system_page_keeps_no_fleet_chart_or_detail_picker` (deleted ids/labels/`<select>` gone, 2 sections), `test_page_system_script_no_longer_references_the_deleted_ui`, `test_every_system_chart_is_drawn_inside_its_own_host_panel` (all 3 `LineChart` constructions take a card/slot `[data-chart]`, scoped host query, GPU metric built from `gpus[].index`, dynamic GPU add+remove, `pruneHosts` drops card and chart state), `test_one_request_per_host_answers_every_per_host_chart` (integration: the exact per-host query returns all 6 series, unknown `gpu.busy_pct@1` returns empty not 404), `test_render_cores_is_gone_and_the_chart_primitives_remain`.
|
||||
|
||||
**Live browser verification** (`http://127.0.0.1:8090`, 3 real hosts): no console/page/request errors, no NaN/undefined, 8 charts all inside panels, legends never list two hosts, each range chip re-fires one request per host (verified 5m→30d), and a host dropped from `/api/system/snapshot` lost its panel *and* its charts. Screenshots: `data/logs/per-host-charts-system-dark.png`, `-light.png`.
|
||||
|
||||
**Two defects fixed in passing**: `.chart .grid text` had no `fill`, so every y-axis label rendered black (invisible on the dark theme) — now `var(--text-faint)`, confirmed in both themes; and `drawHostChart()` now shades a chart only when one series has points, because three 0-100 area fills stacked into one block and hid the CPU line behind memory.
|
||||
|
||||
**Docs**: README `/system` bullet and AGENTS.md "Charts belong to the host they chart" rewritten, including correcting the stale claim that `renderCores` survives as unused library vocabulary.
|
||||
|
||||
**Gate**: `bash .agents/validate.sh` exit 0 — 458 passed, 7 skipped, coverage 96.86% (floor 94), ruff clean, pyright 0 errors, `node --check` clean. Phase/task files untouched.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 46%]
|
||||
.............sssssss.................................................... [ 61%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 92%]
|
||||
................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 175 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
|
||||
src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115
|
||||
src/dashboards/config.py 110 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 139 1 99% 226
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 226
|
||||
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
|
||||
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2104 66 97%
|
||||
Required test coverage of 94% reached. Total coverage: 96.86%
|
||||
458 passed, 7 skipped, 1 warning in 43.12s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
**Implemented (task 04 — verify every host shape on live `/system`)**
|
||||
- Browser-verified all three host shapes live: driveripper (R9700, VRAM 30.5/31.9 GiB + pool line), deskwork (APU — VRAM vs `pool 78%` both legible; GPU visibly crossed ok→warn→fail), gateway (GPU/temp `n/a` uncoloured, no GPU chart).
|
||||
- Failure path proven: stopped only the gateway tunnel → panel kept last-good heroes (nothing zeroed), dot red, `stale 2.1m` text, dimmed; other hosts unaffected; exactly one unreachable + one reachable event; tunnel restored, fleet all-ok.
|
||||
- **Fixed 2 defects found**: (1) anonymous chart SVGs — `LineChart` gained an `ariaLabel` option (`role="img"` + `aria-label`), all three `/system` slots name host+metrics; (2) `/system` Events feed queried `source=host` while the poller stores transitions under `system` — feed was empty since the first commit; now populated.
|
||||
- New pins: `test_every_system_chart_names_itself_for_assistive_tech`, `test_system_event_feed_queries_the_source_transitions_are_stored_under` (tests/test_api.py).
|
||||
- `NOTES-verify.md` written in the phase directory; both themes checked via header toggle; per-core history still queryable (354 points for `cpu.usage_pct@core7`).
|
||||
- Screenshots opened/read: `system-driveripper.png`, `system-deskwork.png`, `system-gateway.png`, `system-stale-gateway.png`, `system-shapes-dark/light.png`, `system-driveripper-light.png`, `system-gateway-light.png` (all in `data/logs/`).
|
||||
|
||||
**Tests / lint**: `bash .agents/validate.sh` → exit 0 — **460 passed, 7 skipped**, coverage **96.86%** (≥94), ruff clean, pyright 0 errors, `node --check` clean · `uv run pytest -m live --live` (with `.env.dev`) → **7 passed** · `uv pip list --outdated` empty.
|
||||
|
||||
**Docs**: AGENTS.md (poller event-source trap; chart `ariaLabel` convention), README.md (chart aria naming in the `/system` paragraph).
|
||||
|
||||
**Notable**: Playwright defaults to light `prefers-color-scheme` — screenshots now seed `localStorage['dash-theme']`; noted in NOTES-verify.md for later phases. `/llm`/`/ups` charts can adopt the new `ariaLabel` option in phases 04/05 (not touched here).
|
||||
|
||||
**Next pending task**: `04_llm_chart_readability` (phase 03 fully complete after this task).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 46%]
|
||||
...............sssssss.................................................. [ 61%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 92%]
|
||||
................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 175 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
|
||||
src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115
|
||||
src/dashboards/config.py 110 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 139 1 99% 226
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 226
|
||||
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
|
||||
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2104 66 97%
|
||||
Required test coverage of 94% reached. Total coverage: 96.86%
|
||||
460 passed, 7 skipped, 1 warning in 43.42s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
@@ -61,9 +61,13 @@ page still renders.
|
||||
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
|
||||
charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
|
||||
no outbound access — a CDN reference is a request that never comes back.
|
||||
6. **Never leak a credential.** Target URLs may carry a userinfo token; `/api/config`
|
||||
exposes them only through `Target.display_url`, which strips userinfo. No token,
|
||||
password, or personal hostname goes into tracked files, fixtures, or the browser.
|
||||
6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets`
|
||||
moves it into an `Authorization` header, so `Target.url` ends up clean). Nothing that leaves
|
||||
the process may carry one: `/api/config` serves `Target.display_url` and
|
||||
`config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message
|
||||
is redacted too — those strings go into the health record *and* the `event` table, which
|
||||
`/api/events` serves to the browser and `page-system.js` prints. No token, password, or
|
||||
personal hostname goes into tracked files, fixtures, or the browser.
|
||||
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
|
||||
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
|
||||
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
|
||||
@@ -104,7 +108,12 @@ Responsibilities, one line each:
|
||||
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
|
||||
env var always wins. Defaults are safe except the sources, which default to unset.
|
||||
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
|
||||
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping.
|
||||
Events are stored under the **dashboard** name (`system`/`llm`/`ups`), not the sample
|
||||
source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
|
||||
(found on `/system` during phase 03 browser verification).
|
||||
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping. Failure
|
||||
messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one
|
||||
that was dialled, because they become health records and event rows.
|
||||
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
|
||||
(`gpu.busy_pct@<index>`).
|
||||
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
|
||||
@@ -158,8 +167,9 @@ Responsibilities, one line each:
|
||||
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
|
||||
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
|
||||
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
|
||||
values the API does not rate; converting its remaining call sites is phases 03–05's job
|
||||
(`/` was converted in phase 02: the overview now colours only from `/api/summary` verdicts).
|
||||
values the API does not rate; converting its remaining call sites is phases 04–05's job
|
||||
(`/` was converted in phase 02, `/system` in phase 03 — those pages colour only from the
|
||||
`severity` verdicts the API sends, and no `severityFor(` call survives in their scripts).
|
||||
|
||||
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
|
||||
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
|
||||
@@ -205,8 +215,8 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
|
||||
./scripts/dev-server.sh status # per-source ok/DOWN
|
||||
./scripts/dev-server.sh logs
|
||||
|
||||
uv run pytest # hermetic: 439 passed + 6 live-skipped, no network
|
||||
uv run pytest --cov # coverage gate (96.8%; floor 94%)
|
||||
uv run pytest # hermetic: 472 passed + 7 live-skipped, no network
|
||||
uv run pytest --cov # coverage gate (96.9%; floor 94%)
|
||||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||||
uv run ruff check src tests agent && uv run pyright
|
||||
./scripts/capture-fixtures.sh # re-record ground truth
|
||||
@@ -235,6 +245,14 @@ service here.
|
||||
fixture test fails before a dashboard does.
|
||||
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
|
||||
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
|
||||
- **Page payloads that a rewrite must not break are pinned as a table.**
|
||||
`tests/card_contract.py` holds the dotted `/api/system/snapshot` paths the `/system` hero
|
||||
cards read (`CARD_METRICS`, `GPU_METRICS`, `HOST_SHAPES`); `tests/test_system_contract.py`
|
||||
asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and
|
||||
`test_every_host_serves_the_hero_card_contract` asserts the same table against the live
|
||||
agents — so a rename fails a test, and a card that shows `n/a` cannot be "fixed" by
|
||||
inventing a field. `network` carries no `severity` on purpose (throughput has no good/bad),
|
||||
and the contract pins its absence.
|
||||
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
|
||||
fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
|
||||
unset, so partial fleet availability is normal. **Live tests must not write** — a battery
|
||||
@@ -252,8 +270,11 @@ service here.
|
||||
|
||||
Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
|
||||
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
|
||||
dots in the header, window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API
|
||||
accepts 5m…90d). A panel that cannot get data says
|
||||
dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
|
||||
`LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
|
||||
silence, and the legend is a sibling that labels nothing) — `/system` does this for all three
|
||||
slots, pinned by `tests/test_api.py`. Window chips (5m…30d on `/system` and `/llm`, 1h…90d on
|
||||
`/ups`; the API accepts 5m…90d). A panel that cannot get data says
|
||||
why in the panel — the operator should never have to open devtools to learn that a source is
|
||||
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
||||
|
||||
@@ -281,6 +302,41 @@ muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the
|
||||
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
|
||||
rule reaches for a token that `:root` does not declare.
|
||||
|
||||
The `/system` host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the
|
||||
`.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script).
|
||||
Colour comes from the payload's `severity` dicts and nowhere else; a missing number or
|
||||
verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**. NET is
|
||||
one slot with two lines (`.hero-lines`, `↓` in / `↑` out) and is deliberately colourless:
|
||||
throughput has no good/bad, so `network` carries no severity. The GPU slot holds one hero
|
||||
per card (model, VRAM used/total and the honest APU `pool` view in the `.sub`), and
|
||||
everything not glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU
|
||||
power/temp, filesystems, processes — folds into a collapsed "More detail" `<details>`: a
|
||||
simpler surface, not a less complete page.
|
||||
|
||||
A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
|
||||
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
|
||||
those (a range when the hosts disagree) instead of the poll interval, which only coincides with
|
||||
them at the defaults.
|
||||
|
||||
**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
|
||||
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
|
||||
panel and the host/metric detail picker are deleted, along with the `#chart-*` / `#detail-*` ids and
|
||||
the `select` / `label.field` styling only those pickers used. Each panel stacks its own
|
||||
`.host-charts` — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where
|
||||
the GPU slot is created and destroyed with `host.gpus`, so gateway shows neither a phantom card
|
||||
nor a hole. **One `/api/system/history` request per host per refresh**
|
||||
(`hosts=<that host>&metrics=<that panel's series>&range=…`, pinned by
|
||||
`test_every_system_chart_is_drawn_inside_its_own_host_panel`): unknown metrics come back as
|
||||
empty series on purpose, so a panel asks for every series it could draw and asks once. A legend
|
||||
here names metrics (or one GPU's model), never hosts — the panel heading already says who this
|
||||
is, and a legend that grew a second host is the bug this replaced. A host that stops reporting
|
||||
loses its panel *and* its charts (`pruneHosts`), never a chart of nothing. Per-core visuals are
|
||||
gone from `/system` and `renderCores` is **deleted** from `charts.js` with its last caller.
|
||||
`drawHostChart()` shades a chart only when one series has points — two or three filled areas
|
||||
stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy
|
||||
memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryable from
|
||||
`/api/system/history` — removing the plot is not removing the data.
|
||||
|
||||
## Known gaps / next work
|
||||
|
||||
1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see
|
||||
|
||||
@@ -54,8 +54,10 @@ that already listens there (upsd) — that is how the UPS page reads a real batt
|
||||
|
||||
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
|
||||
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
|
||||
JSON document), `GET /info` (slow-changing identity: OS, kernel, CPU model, GPU list), and
|
||||
`GET /health` (the one route that never needs the token) as JSON.
|
||||
JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS,
|
||||
kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and
|
||||
`GET /health` (the one route that never needs the token) as JSON, with a plain HTML status
|
||||
page at `GET /` so a browser pointed at an agent learns what it is looking at.
|
||||
|
||||
```bash
|
||||
python3 host_agent.py --once # print one snapshot, exit
|
||||
@@ -89,7 +91,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
||||
| ------------------------------------- | -------------------------------------------------------------- |
|
||||
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
|
||||
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
||||
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) |
|
||||
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and `/system`'s footnote prints that, not this default |
|
||||
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
|
||||
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
|
||||
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
|
||||
@@ -108,11 +110,16 @@ A source that is not configured answers `404 {"configured": false}` **without di
|
||||
and the page says "not configured" instead of showing a red panel for something you never
|
||||
installed.
|
||||
|
||||
Nothing that leaves the app carries a credential. A `HOST_TARGETS` userinfo token is parsed
|
||||
into an `Authorization` header, and every URL the browser can see — `/api/config`, a source's
|
||||
error string, the event feed — goes through `config.strip_userinfo` first, because an
|
||||
unreachable host writes its failure message into the event table and `/api/events` serves it.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
||||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR
|
||||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
|
||||
▼ ▼ ▼
|
||||
src/dashboards/collectors.py ────────────────────────────────────────────┐
|
||||
│ parse + Δcounter/Δt (rates.py) │
|
||||
@@ -155,7 +162,19 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
|
||||
A big colour-coded number is `Dash.hero(el, {...})` on the
|
||||
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
|
||||
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
|
||||
with no verdict renders `n/a` with no colour at all.
|
||||
with no verdict renders `n/a` with no colour at all. Each `/system` host is one panel led
|
||||
by exactly six of them — CPU %, MEM %, TEMP, DISK % (mount underneath), GPU busy (VRAM and
|
||||
the APU pool in the footnote), and NET on two lines (↓ in / ↑ out, never coloured: a byte
|
||||
rate has no good/bad). Under the numbers, **each panel charts only its own host**: utilisation
|
||||
(CPU, memory, worst disk), network (↓ in / ↑ out), and GPU busy for a host that reports
|
||||
cards — three 120px SVG charts, each with its own legend and its own `aria-label` naming the
|
||||
host and the plotted metrics (`LineChart`'s `ariaLabel` option), never a second host in sight. The
|
||||
fleet-wide charts, the nine-figure fleet-totals panel and the host/metric detail picker are gone, and
|
||||
so are the per-core charts, while `cpu.usage_pct@coreN` stays in SQLite. Everything else —
|
||||
load, clock, disk I/O, uptime, GPU power/temp, filesystems, processes — folds into a collapsed
|
||||
"More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request
|
||||
(`hosts=<that host>&metrics=…&range=…`), and a host that stops reporting takes its panel and
|
||||
its charts off the page.
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -176,6 +195,12 @@ also skips when its source is unset, they drive the production code paths (`coll
|
||||
`NutMonitor.collect`) against whatever hardware is reachable, and they never write — a
|
||||
battery test on live hardware is not a unit test.
|
||||
|
||||
One more file is worth knowing about: `tests/card_contract.py` is the table of
|
||||
`/api/system/snapshot` paths the system page's per-host cards are allowed to read. It is
|
||||
checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and
|
||||
again against the live fleet, so a card can never be "fixed" by inventing a number the agent
|
||||
does not report.
|
||||
|
||||
## Battery-test buttons
|
||||
|
||||
Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`)
|
||||
|
||||
+10
-4
@@ -5,7 +5,8 @@ hot path) plus SQLite for anything older than a few minutes. Two rules from llm_
|
||||
are worth restating here because they are easy to break by accident:
|
||||
|
||||
* `/api/config` never leaks a token or password — target URLs are passed through
|
||||
`Target.display_url`, which strips userinfo.
|
||||
`Target.display_url`, which strips userinfo, and so does every URL that reaches an error
|
||||
message, because those travel into `/api/health` and the event feed too.
|
||||
* an unconfigured source answers 404 `{"configured": false}` without dialling anything, so
|
||||
the page can tell "not deployed here" apart from "deployed and broken".
|
||||
"""
|
||||
@@ -20,7 +21,7 @@ from fastapi import APIRouter, Body, Depends, HTTPException, Query, Request
|
||||
|
||||
from . import alerts as alert_rules
|
||||
from .collectors import NutMonitor
|
||||
from .config import Settings
|
||||
from .config import Settings, strip_userinfo
|
||||
from .poller import Hub
|
||||
from .severity import classify
|
||||
from .sources import nut
|
||||
@@ -133,9 +134,14 @@ def common_router() -> APIRouter:
|
||||
"target_problems": list(settings.host_target_problems),
|
||||
},
|
||||
"llm": {
|
||||
"gufo": {"enabled": bool(settings.gufo_url), "base_url": settings.gufo_url,
|
||||
# `strip_userinfo` on the engine URLs too: `GUFO_API_KEY` is the documented
|
||||
# way to authenticate, but a URL that carries credentials must never be echoed
|
||||
# to a browser whatever the config says (invariant 6).
|
||||
"gufo": {"enabled": bool(settings.gufo_url),
|
||||
"base_url": strip_userinfo(settings.gufo_url),
|
||||
"auth": bool(settings.gufo_api_key)},
|
||||
"vllm": {"enabled": bool(settings.vllm_url), "base_url": settings.vllm_url,
|
||||
"vllm": {"enabled": bool(settings.vllm_url),
|
||||
"base_url": strip_userinfo(settings.vllm_url),
|
||||
"auth": bool(settings.vllm_api_key)},
|
||||
"interval_seconds": settings.llm_interval,
|
||||
"poll_ms": settings.poll_ms_llm,
|
||||
|
||||
@@ -13,7 +13,7 @@ from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from .config import Target
|
||||
from .config import Target, strip_userinfo
|
||||
from .poller import Job, Result
|
||||
from .rates import RateEstimator
|
||||
from .sources import llm as llm_source
|
||||
@@ -37,17 +37,21 @@ class UpstreamError(Exception):
|
||||
async def fetch_host_snapshot(client: httpx.AsyncClient, target: Target,
|
||||
host_collector: HostCollector) -> Result:
|
||||
url = f"{target.url}/snapshot"
|
||||
# Every message below uses the redacted URL on purpose: an UpstreamError string is kept
|
||||
# in the job's health record and written to the event table, both of which the browser
|
||||
# reads, and `target.url` may carry the agent's token as userinfo.
|
||||
shown = f"{target.display_url}/snapshot"
|
||||
try:
|
||||
response = await client.get(url, headers=target.headers)
|
||||
except httpx.TimeoutException as exc:
|
||||
raise UpstreamError(f"{target.name}: {url} timed out", status=504) from exc
|
||||
raise UpstreamError(f"{target.name}: {shown} timed out", status=504) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise UpstreamError(f"{target.name}: {url} unreachable ({type(exc).__name__})") from exc
|
||||
raise UpstreamError(f"{target.name}: {shown} unreachable ({type(exc).__name__})") from exc
|
||||
if response.status_code == 401:
|
||||
raise UpstreamError(f"{target.name}: agent rejected the token (401) — check the "
|
||||
f"userinfo token in HOST_TARGETS")
|
||||
if response.status_code >= 400:
|
||||
raise UpstreamError(f"{target.name}: {url} -> HTTP {response.status_code}")
|
||||
raise UpstreamError(f"{target.name}: {shown} -> HTTP {response.status_code}")
|
||||
try:
|
||||
body = response.json()
|
||||
except ValueError as exc:
|
||||
@@ -71,14 +75,15 @@ async def fetch_engine(client: httpx.AsyncClient, engine: str, base_url: str,
|
||||
with_slots: bool = False) -> Result:
|
||||
headers = {"Authorization": f"Bearer {api_key}"} if api_key else {}
|
||||
url = f"{base_url}/metrics"
|
||||
shown = strip_userinfo(url) # same rule as the host fetch above: no credential in a message
|
||||
try:
|
||||
response = await client.get(url, headers=headers)
|
||||
except httpx.TimeoutException as exc:
|
||||
raise UpstreamError(f"{engine}: {url} timed out", status=504) from exc
|
||||
raise UpstreamError(f"{engine}: {shown} timed out", status=504) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise UpstreamError(f"{engine}: {url} unreachable ({type(exc).__name__})") from exc
|
||||
raise UpstreamError(f"{engine}: {shown} unreachable ({type(exc).__name__})") from exc
|
||||
if response.status_code >= 400:
|
||||
raise UpstreamError(f"{engine}: {url} -> HTTP {response.status_code} "
|
||||
raise UpstreamError(f"{engine}: {shown} -> HTTP {response.status_code} "
|
||||
f"{response.reason_phrase}")
|
||||
text = response.text
|
||||
|
||||
|
||||
@@ -75,6 +75,24 @@ def _base_url(name: str) -> str:
|
||||
# variable; the token never leaves the backend, it is stripped from every /api payload.
|
||||
_TARGET_SPLIT = re.compile(r"[,\s]+")
|
||||
_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_.-]*$")
|
||||
_USERINFO_RE = re.compile(r"^(https?://)[^/@]+@", re.IGNORECASE)
|
||||
|
||||
|
||||
def strip_userinfo(url: str) -> str:
|
||||
"""`url` with any userinfo (`http://TOKEN@host` -> `http://host`) removed.
|
||||
|
||||
The only form a URL may take on its way out of the process. Two things make this needed
|
||||
rather than belt-and-braces: a URL does not travel only in the fetch — an `UpstreamError`
|
||||
message repeats the URL it was dialling, and that string lands in the job's health record
|
||||
(`/api/health`) and in the `event` table (`/api/events`), both of which a browser reads —
|
||||
and the engine URLs (`GUFO_BASE_URL`, `VLLM_BASE_URL`) are used verbatim, with no
|
||||
`parse_targets`-style move of userinfo into a header, so a credential written into one of
|
||||
them would otherwise be served by `/api/config` and echoed by every failed poll.
|
||||
`HOST_TARGETS` strips its token into a header, so this is the backstop for a `Target` that
|
||||
still carries one. One rule, stated once: AGENTS.md invariant 6, no credential in anything
|
||||
a browser can read.
|
||||
"""
|
||||
return _USERINFO_RE.sub(r"\1", url)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -88,7 +106,7 @@ class Target:
|
||||
@property
|
||||
def display_url(self) -> str:
|
||||
"""URL with any userinfo removed — safe to show in the browser."""
|
||||
return re.sub(r"^(https?://)[^/@]+@", r"\1", self.url)
|
||||
return strip_userinfo(self.url)
|
||||
|
||||
|
||||
def parse_targets(raw: str) -> tuple[list[Target], list[str]]:
|
||||
|
||||
@@ -328,6 +328,10 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl
|
||||
"forks_per_s": forks_rate,
|
||||
"procs_running": cpu_raw.get("procs_running"),
|
||||
"procs_blocked": cpu_raw.get("procs_blocked"),
|
||||
# The card metric for "cpu temp" is this one; `temperatures.cpu_c` below is the
|
||||
# same measurement (the agent's own `cpu_temperature_celsius`) and `/api/summary`
|
||||
# flattens this field, so the card and the overview cannot disagree. Pinned by
|
||||
# tests/test_system_contract.py.
|
||||
"temp_c": cpu_temp,
|
||||
# Additive: the pages colour from this verdict instead of comparing the number
|
||||
# to a threshold of their own. Keys mirror the rated fields in this section.
|
||||
@@ -364,6 +368,10 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl
|
||||
"rx_bytes_per_s": rx_total if any_net else None,
|
||||
"tx_bytes_per_s": tx_total if any_net else None,
|
||||
"interfaces": net_rows,
|
||||
# Deliberately no `severity` section here, unlike cpu/memory/disks/gpus: a byte
|
||||
# rate has no good/bad (900 MB/s is idle on 10 GbE, 2 KB/s can be an incident),
|
||||
# so `severity.THRESHOLDS` has no `net.*` row and a colour here would be a
|
||||
# verdict nobody measured. tests/test_system_contract.py pins its absence.
|
||||
},
|
||||
"gpus": gpus,
|
||||
"temperatures": {"cpu_c": cpu_temp, "max_c": max_temp, "nvme_max_c": nvme_max,
|
||||
|
||||
@@ -37,83 +37,19 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<section class="card flat" aria-label="Fleet totals">
|
||||
<div class="body">
|
||||
<div class="metrics">
|
||||
<div class="metric"><div class="label">Hosts answering</div>
|
||||
<div class="value"><span id="f-hosts">n/a</span><span class="unit" id="f-hosts-of"></span></div></div>
|
||||
<div class="metric"><div class="label">CPU average</div>
|
||||
<div class="value"><span id="f-cpu">n/a</span><span class="unit">%</span></div></div>
|
||||
<div class="metric"><div class="label">Load 1 (sum)</div>
|
||||
<div class="value" id="f-load">n/a</div></div>
|
||||
<div class="metric"><div class="label">Memory in use</div>
|
||||
<div class="value"><span id="f-mem">n/a</span><span class="unit" id="f-mem-of"></span></div></div>
|
||||
<div class="metric"><div class="label">Worst filesystem</div>
|
||||
<div class="value"><span id="f-disk">n/a</span><span class="unit">%</span></div></div>
|
||||
<div class="metric"><div class="label">Network in</div>
|
||||
<div class="value"><span id="f-rx">n/a</span><span class="unit" id="f-rx-unit"></span></div></div>
|
||||
<div class="metric"><div class="label">Network out</div>
|
||||
<div class="value"><span id="f-tx">n/a</span><span class="unit" id="f-tx-unit"></span></div></div>
|
||||
<div class="metric"><div class="label">GPU busy (avg)</div>
|
||||
<div class="value"><span id="f-gpu">n/a</span><span class="unit">%</span></div></div>
|
||||
<div class="metric"><div class="label">Power</div>
|
||||
<div class="value sm" title="No whole-host power source is wired up yet; no script writes this, so it stays n/a rather than guessing.">n/a</div></div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="grid cols-2" style="margin-top:.85rem" aria-label="Fleet charts">
|
||||
<article class="card">
|
||||
<header><h3>CPU utilisation</h3><span class="spacer"></span><span class="faint">% · per host</span></header>
|
||||
<div class="body"><div class="chart" id="chart-cpu"></div><div class="legend" id="legend-cpu"></div></div>
|
||||
</article>
|
||||
<article class="card">
|
||||
<header><h3>Memory used</h3><span class="spacer"></span><span class="faint">% of MemAvailable basis</span></header>
|
||||
<div class="body"><div class="chart" id="chart-mem"></div><div class="legend" id="legend-mem"></div></div>
|
||||
</article>
|
||||
<article class="card">
|
||||
<header><h3>Disk I/O throughput</h3><span class="spacer"></span>
|
||||
<span class="faint">selected host · all physical devices summed</span></header>
|
||||
<div class="body"><div class="chart" id="chart-disk"></div><div class="legend" id="legend-disk"></div></div>
|
||||
</article>
|
||||
<article class="card">
|
||||
<header><h3>Network throughput</h3><span class="spacer"></span><span class="faint">selected host · physical interfaces</span></header>
|
||||
<div class="body"><div class="chart" id="chart-net"></div><div class="legend" id="legend-net"></div></div>
|
||||
</article>
|
||||
<article class="card">
|
||||
<header><h3>GPU utilisation</h3><span class="spacer"></span><span class="faint">one series per host GPU</span></header>
|
||||
<div class="body"><div class="chart" id="chart-gpu"></div><div class="legend" id="legend-gpu"></div></div>
|
||||
</article>
|
||||
<article class="card">
|
||||
<header><h3>Temperature</h3><span class="spacer"></span><span class="faint">CPU package · °C</span></header>
|
||||
<div class="body"><div class="chart" id="chart-temp"></div><div class="legend" id="legend-temp"></div></div>
|
||||
</article>
|
||||
</section>
|
||||
|
||||
<section class="card" style="margin-top:.85rem" aria-label="Series detail">
|
||||
<header>
|
||||
<h3>Detail</h3>
|
||||
<span class="spacer"></span>
|
||||
<label class="field" for="detail-host">host</label>
|
||||
<select id="detail-host"></select>
|
||||
<label class="field" for="detail-metric">metric</label>
|
||||
<select id="detail-metric"></select>
|
||||
</header>
|
||||
<div class="body">
|
||||
<div class="chart" id="chart-detail" style="height:200px"></div>
|
||||
<div class="legend" id="legend-detail"></div>
|
||||
<div class="faint" id="detail-note" style="margin-top:.4rem;font-size:11.5px"></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section style="margin-top:.85rem">
|
||||
<!-- No fleet-wide chart and no host/metric picker survives here (phase 03 task 03): every
|
||||
chart is drawn inside one host panel below, and the question "which machine stands
|
||||
out?" belongs to the overview's alerts and fleet table. -->
|
||||
<section>
|
||||
<div class="page-head"><h2 style="font-size:.95rem;margin:0">Per host</h2>
|
||||
<span class="sub">agent-reported values; rates measured over the window in each header</span></div>
|
||||
<span class="sub">agent-reported values; rates are measured over the Δ window in the
|
||||
footer, and each panel's charts cover only that host</span></div>
|
||||
<div class="grid auto" id="hosts"></div>
|
||||
<div class="empty-state" id="hosts-empty" hidden>
|
||||
<h3>No hosts reporting</h3>
|
||||
<p>Start the agent on a machine and add it to <code>HOST_TARGETS</code>:</p>
|
||||
<p class="mono">python3 agent/host_agent.py --host 0.0.0.0 --port 9120 --token …</p>
|
||||
<p class="mono">python3 agent/host_agent.py --host 0.0.0.0 --port 9120 \
|
||||
--token-file /etc/dashboards-agent.token</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -123,7 +59,7 @@
|
||||
</section>
|
||||
|
||||
<footer class="pagefoot">
|
||||
<span>numbers are measured, not inferred: rates are Δcounter/Δt over the poll window</span>
|
||||
<span>numbers are measured, not inferred: rates are Δcounter/Δt over the window below</span>
|
||||
<span class="spacer" style="margin-inline-start:auto"></span>
|
||||
<span class="mono" id="foot-note"></span>
|
||||
</footer>
|
||||
|
||||
@@ -110,9 +110,9 @@ const Dash = (() => {
|
||||
*
|
||||
* Thresholds live in `src/dashboards/severity.py` and travel in the payload as
|
||||
* `severity`; `severityFrom()` below is how a page reads them. This function survives
|
||||
* only because `page-system.js`, `page-llm.js` and `page-ups.js` still call it with
|
||||
* their own numbers (phases 03–05 convert them page by page, so each page's change is
|
||||
* reviewable on its own — `/` was already converted in phase 02). Do not add a new call
|
||||
* only because `page-llm.js` and `page-ups.js` still call it with their own numbers
|
||||
* (phases 04–05 convert them page by page, so each page's change is reviewable on its
|
||||
* own — `/` was converted in phase 02, `/system` in phase 03). Do not add a new call
|
||||
* site: a threshold written in JS is a threshold that can disagree with the other pages.
|
||||
*/
|
||||
function severityFor(value, warnAt, failAt) {
|
||||
|
||||
@@ -43,7 +43,7 @@ const DEFAULT_FORMAT = (v) => (Math.abs(v) >= 100 ? v.toFixed(0) : v.toFixed(1))
|
||||
class LineChart {
|
||||
/**
|
||||
* container: element to fill (its width is used; height comes from opts.height)
|
||||
* opts: {height, yFormat, xSpan, yMin, yMax, area, legend, showAxis, min, unit}
|
||||
* opts: {height, yFormat, xSpan, yMin, yMax, area, legend, showAxis, min, unit, ariaLabel}
|
||||
*/
|
||||
constructor(container, opts) {
|
||||
this.el = typeof container === 'string' ? document.querySelector(container) : container;
|
||||
@@ -69,6 +69,13 @@ class LineChart {
|
||||
this.svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
|
||||
this.svg.setAttribute('width', '100%');
|
||||
this.svg.setAttribute('height', this.opts.height);
|
||||
// A chart is an image to assistive tech, which means silence unless the page names it —
|
||||
// the legend below is a sibling, not a label. `ariaLabel` is optional so pages adopt it
|
||||
// one at a time (`/system` names every chart it draws; see page-system.js).
|
||||
if (this.opts.ariaLabel) {
|
||||
this.svg.setAttribute('role', 'img');
|
||||
this.svg.setAttribute('aria-label', String(this.opts.ariaLabel));
|
||||
}
|
||||
this.el.appendChild(this.svg);
|
||||
this.empty = document.createElement('div');
|
||||
this.empty.className = 'empty';
|
||||
@@ -337,17 +344,6 @@ function renderLegend(container, series, format, showPeak) {
|
||||
}).join('');
|
||||
}
|
||||
|
||||
/** Column heat bars for per-core usage: 48 cores read as a pattern, not a table. */
|
||||
function renderCores(container, values, format) {
|
||||
const f = format || ((v) => (v == null ? 'n/a' : v.toFixed(0) + '%'));
|
||||
container.innerHTML = (values || []).map((v, i) => {
|
||||
const pct = v == null ? 0 : Math.max(0, Math.min(100, v));
|
||||
const hue = pct > 85 ? 'var(--fail)' : pct > 60 ? 'var(--warn)' : 'var(--accent)';
|
||||
return `<span class="core" title="core ${i}: ${escapeHtml(f(v))}">` +
|
||||
`<i style="height:${pct.toFixed(1)}%;background:${hue}"></i></span>`;
|
||||
}).join('');
|
||||
}
|
||||
|
||||
/** Small non-interactive sparkline for cards where a full chart would shout. */
|
||||
function sparkline(container, points, opts) {
|
||||
const o = Object.assign({ height: 34, color: cssVar('--series-1'), min: null, max: null }, opts || {});
|
||||
@@ -374,5 +370,5 @@ function sparkline(container, points, opts) {
|
||||
container.innerHTML = svg;
|
||||
}
|
||||
|
||||
window.DashCharts = { LineChart, niceTicks, renderLegend, renderCores, sparkline, cssVar,
|
||||
window.DashCharts = { LineChart, niceTicks, renderLegend, sparkline, cssVar,
|
||||
CHART_COLORS, DEFAULT_FORMAT, escapeHtml };
|
||||
|
||||
@@ -282,6 +282,27 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
|
||||
/* with no verdict on either the hero or the bar the fill stays the neutral accent: the
|
||||
bar shows *how much*, never a judgement nobody measured. */
|
||||
|
||||
/* The per-host hero row: six glanceable numbers — CPU, MEM, TEMP, DISK, GPU, NET — in one
|
||||
line on a wide card, wrapping to 3×2 and then 2×3 as the card narrows. Tracks are a
|
||||
`minmax` of the widest plain hero value (`100 %`), so a row never squints into ellipsis. */
|
||||
.heroes {
|
||||
display: grid; gap: .9rem 1rem;
|
||||
grid-template-columns: repeat(auto-fit, minmax(min(118px, 100%), 1fr));
|
||||
}
|
||||
/* a slot that holds more than one hero (one per GPU card) without breaking the row */
|
||||
.hero-slot { display: grid; gap: .7rem; min-width: 0; align-content: start; }
|
||||
/* Network is one hero with two lines: in and out are one wire seen in two directions, and
|
||||
two big numbers side by side read as a competition. It is deliberately colourless —
|
||||
a byte rate has no good/bad, so no verdict ever paints here (severity.py has no net.*
|
||||
row); `n/a` comes from `.na` and is as faint as any other unmeasured value. */
|
||||
.hero .hero-lines {
|
||||
display: grid; gap: .15rem;
|
||||
font-family: var(--mono); font-size: 1.28rem; font-weight: 650; line-height: 1.25;
|
||||
font-variant-numeric: tabular-nums; /* same fixed digit width as .hero .value */
|
||||
color: var(--text);
|
||||
}
|
||||
.hero .hero-lines .dir { color: var(--text-muted); }
|
||||
|
||||
/* ---- bars, dots, chips -------------------------------------------------- */
|
||||
.bar {
|
||||
position: relative; height: 7px; border-radius: 999px; background: var(--bg-sunken);
|
||||
@@ -357,9 +378,15 @@ table.data td.fail { color: var(--fail); }
|
||||
table.data.scroll { display: block; overflow: auto; max-height: 320px; }
|
||||
|
||||
/* ---- charts ------------------------------------------------------------- */
|
||||
/* The chart vocabulary every page draws with. `LineChart` puts the y tick labels in the
|
||||
`grid` group and the x labels in `axis`, so both groups need the same text style — the
|
||||
grid text used to inherit SVG's black fill, which meant the numbers beside a utilisation
|
||||
chart were invisible on the dark theme. */
|
||||
.chart { position: relative; width: 100%; }
|
||||
.chart svg { display: block; width: 100%; height: 100%; overflow: visible; }
|
||||
.chart .axis text { fill: var(--text-faint); font-family: var(--mono); font-size: 10px; }
|
||||
.chart .axis text, .chart .grid text {
|
||||
fill: var(--text-faint); font-family: var(--mono); font-size: 10px;
|
||||
}
|
||||
.chart .axis line, .chart .grid line { stroke: var(--border); stroke-width: 1; }
|
||||
.chart .grid line { stroke-dasharray: 2 4; opacity: .75; }
|
||||
.chart .zero line { stroke: var(--border-strong); }
|
||||
@@ -375,6 +402,20 @@ table.data.scroll { display: block; overflow: auto; max-height: 320px; }
|
||||
.legend .swatch { width: 9px; height: 9px; border-radius: 2px; }
|
||||
.legend .lv { font-family: var(--mono); color: var(--text); }
|
||||
|
||||
/* The per-host chart stack inside a `/system` panel: supporting evidence under a number,
|
||||
not the main event. One utilisation chart, one network chart, and a GPU chart only for a
|
||||
host that reports cards (`page-system.js` adds and removes that slot with `host.gpus`),
|
||||
each with its legend glued underneath. They are stacked rather than side by side because
|
||||
a 200px-wide plot cannot show an hour of CPU without its axis labels colliding — and no
|
||||
chart in here ever contains a second host. */
|
||||
.host-charts { display: grid; gap: .7rem; }
|
||||
.host-chart { display: grid; gap: .15rem; min-width: 0; }
|
||||
/* `sm` is the slot's floor before `LineChart` writes its measured height inline, so an
|
||||
empty panel is the height it is about to be rather than a collapsing card. */
|
||||
.chart.sm { min-height: 120px; }
|
||||
.host-chart .legend { font-size: 10.5px; gap: .2rem .6rem; }
|
||||
.host-chart .legend .swatch { width: 7px; height: 7px; }
|
||||
|
||||
.tooltip {
|
||||
position: absolute; z-index: 30; pointer-events: none; opacity: 0;
|
||||
background: color-mix(in srgb, var(--bg-raised) 95%, transparent);
|
||||
@@ -386,11 +427,6 @@ table.data.scroll { display: block; overflow: auto; max-height: 320px; }
|
||||
.tooltip .row { display: flex; align-items: center; gap: .35rem; }
|
||||
.tooltip .row .val { margin-inline-start: auto; font-family: var(--mono); }
|
||||
|
||||
/* per-core usage heat bars */
|
||||
.cores { display: grid; grid-template-columns: repeat(auto-fill, minmax(9px, 1fr)); gap: 2px; }
|
||||
.cores .core { height: 22px; border-radius: 2px; background: var(--bg-sunken); position: relative; }
|
||||
.cores .core > i { position: absolute; bottom: 0; left: 0; right: 0; background: var(--accent); border-radius: 2px; }
|
||||
|
||||
/* ---- lists -------------------------------------------------------------- */
|
||||
ul.events { list-style: none; margin: 0; padding: 0; display: grid; gap: .1rem; }
|
||||
ul.events li {
|
||||
@@ -497,11 +533,13 @@ dialog footer { padding: .7rem .9rem; border-top: 1px solid var(--border); displ
|
||||
|
||||
.stale { opacity: .62; }
|
||||
|
||||
select, input[type="text"] {
|
||||
/* Only the UPS confirmation dialog takes text now — `/system`'s host/metric pickers were the
|
||||
last `select` on the app and left with the shared charts, along with the `label.field` caption
|
||||
that sat above them. */
|
||||
input[type="text"] {
|
||||
background: var(--bg-sunken); color: var(--text); border: 1px solid var(--border);
|
||||
border-radius: var(--radius-sm); padding: .3rem .45rem; font: 500 12.5px var(--sans);
|
||||
}
|
||||
label.field { font-size: 11px; text-transform: uppercase; letter-spacing: .05em; color: var(--text-faint); }
|
||||
|
||||
footer.pagefoot {
|
||||
margin-top: 1.6rem; padding-top: .8rem; border-top: 1px solid var(--border);
|
||||
|
||||
@@ -2,71 +2,43 @@
|
||||
*
|
||||
* Host cards are created once and patched in place (keyed by host name) — rebuilding the
|
||||
* DOM every 10s kills hover, transitions, and any scroll position inside the tables.
|
||||
* Charts get their history from /api/system/history: one request per chart, N hosts each.
|
||||
*
|
||||
* Every chart on this page belongs to exactly one host and is drawn inside that host's own
|
||||
* panel (`/api/system/history`, one request per host per refresh). Nothing here plots two
|
||||
* machines on one axis and there is no host/metric picker: "which machine stands out?" is
|
||||
* the overview's job (alerts + fleet table), and `/system` answers "what is this machine
|
||||
* doing?". See the per-host chart section below.
|
||||
*/
|
||||
'use strict';
|
||||
|
||||
const { $, $$, fetchJSON, setTxt, setValue, setBar, setDot, bytes, pct, num, duration, poll,
|
||||
staleness, markStale, severityFor } = Dash;
|
||||
const { $, fetchJSON, setTxt, setDot, bytes, num, duration, poll,
|
||||
markStale } = Dash;
|
||||
|
||||
const state = {
|
||||
range: 3600,
|
||||
detailHost: null,
|
||||
detailMetric: 'cpu.usage_pct',
|
||||
hosts: new Map(), // name -> card element
|
||||
charts: {},
|
||||
charts: new Map(), // name -> { util, net, gpu|null } LineChart instances
|
||||
poller: null,
|
||||
pollMs: 10000,
|
||||
metricList: [],
|
||||
};
|
||||
|
||||
const DETAIL_METRICS = [
|
||||
['cpu.usage_pct', 'CPU utilisation', 'pct'],
|
||||
['cpu.user_pct', 'CPU user', 'pct'],
|
||||
['cpu.system_pct', 'CPU system', 'pct'],
|
||||
['cpu.iowait_pct', 'CPU iowait', 'pct'],
|
||||
['cpu.steal_pct', 'CPU steal', 'pct'],
|
||||
['cpu.load1', 'Load average (1m)', 'num'],
|
||||
['cpu.load15', 'Load average (15m)', 'num'],
|
||||
['cpu.freq_mhz', 'CPU frequency', 'mhz'],
|
||||
['cpu.context_switches_per_s', 'Context switches', 'num'],
|
||||
['mem.used_pct', 'Memory used', 'pct'],
|
||||
['mem.used_bytes', 'Memory used', 'bytes'],
|
||||
['mem.available_bytes', 'Memory available', 'bytes'],
|
||||
['mem.swap_used_pct', 'Swap used', 'pct'],
|
||||
['disk.used_pct', 'Filesystem used (worst)', 'pct'],
|
||||
['disk.read_bytes_per_s', 'Disk read', 'bytes'],
|
||||
['disk.write_bytes_per_s', 'Disk write', 'bytes'],
|
||||
['disk.util_pct', 'Disk busy', 'pct'],
|
||||
['net.rx_bytes_per_s', 'Network in', 'bytes'],
|
||||
['net.tx_bytes_per_s', 'Network out', 'bytes'],
|
||||
['temp.cpu_c', 'CPU temperature', 'temp'],
|
||||
['temp.nvme_max_c', 'NVMe temperature', 'temp'],
|
||||
['host.uptime_seconds', 'Uptime', 'dur'],
|
||||
['host.processes', 'Process count', 'num'],
|
||||
];
|
||||
|
||||
const FORMATS = {
|
||||
pct: { fn: (v) => num(v, 1), unit: '%', axis: (v) => num(v, 0) },
|
||||
bytes: { fn: (v) => bytes(v, 1), unit: 'B/s', axis: (v) => (Dash.fmtBytes(v, 0) || { text: '0' }).text +
|
||||
((Dash.fmtBytes(v, 0) || { unit: '' }).unit) },
|
||||
mhz: { fn: (v) => num(v, 0) + ' MHz', unit: 'MHz', axis: (v) => num(v, 0) },
|
||||
temp: { fn: (v) => num(v, 1) + ' °C', unit: '°C', axis: (v) => num(v, 0) },
|
||||
dur: { fn: (v) => duration(v), unit: '', axis: (v) => duration(v) },
|
||||
num: { fn: (v) => num(v, 1), unit: '', axis: (v) => num(v, 0) },
|
||||
};
|
||||
|
||||
function formatFor(kind) { return FORMATS[kind] || FORMATS.num; }
|
||||
|
||||
function metricKind(name) {
|
||||
if (name.endsWith('_pct')) return 'pct';
|
||||
if (name.endsWith('_bytes') || name.includes('bytes_per_s')) return 'bytes';
|
||||
if (name.includes('_c') && name.startsWith('temp.')) return 'temp';
|
||||
if (name === 'host.uptime_seconds') return 'dur';
|
||||
return 'num';
|
||||
}
|
||||
|
||||
// ---- host cards -----------------------------------------------------------
|
||||
//
|
||||
// Six heroes per host, in one fixed order: CPU, MEM, TEMP, DISK, GPU, NET. Every colour
|
||||
// comes from the payload's `severity` dicts — `severity.py` owns the thresholds, this file
|
||||
// compares no number to any bound — and `Dash.hero()` prints `n/a` with **no** colour
|
||||
// class when the number or its verdict is missing, so gateway's absent CPU sensor cannot
|
||||
// borrow last poll's green.
|
||||
//
|
||||
// `data-sev` names the verdict each slot paints; tests/test_api.py pins the six slots and
|
||||
// their order on the served bytes. The NET slot carries none on purpose: `network` has no
|
||||
// `severity` (a byte rate has no good/bad), so nothing here may colour it.
|
||||
//
|
||||
// Per-core usage is not plotted here or anywhere on this page — the series is still
|
||||
// stored as `cpu.usage_pct@coreN`, the page just stopped drawing a pattern nobody reads.
|
||||
// Everything that is not glance-worthy (load averages, clock, disk read/write/busy,
|
||||
// uptime, GPU power and temperature, filesystems, top processes) lives in "More detail",
|
||||
// collapsed by default: the TODO asked for a simpler surface, not a less complete page.
|
||||
function hostCard(name) {
|
||||
const card = document.createElement('article');
|
||||
card.className = 'card';
|
||||
@@ -80,38 +52,110 @@ function hostCard(name) {
|
||||
<span class="faint mono updated"></span>
|
||||
</header>
|
||||
<div class="body stack">
|
||||
<div class="metrics">
|
||||
<div class="metric"><div class="label">CPU</div><div class="value"><span data-k="cpu">n/a</span><span class="unit">%</span></div>
|
||||
<div class="bar" data-bar="cpu"><i></i></div></div>
|
||||
<div class="metric"><div class="label">Load 1 / 15</div><div class="value sm" data-k="load">n/a</div></div>
|
||||
<div class="metric"><div class="label">CPU temp</div><div class="value sm"><span data-k="temp">n/a</span><span class="unit">°C</span></div></div>
|
||||
<div class="metric"><div class="label">Clock</div><div class="value sm"><span data-k="freq">n/a</span><span class="unit">GHz</span></div></div>
|
||||
<div class="metric"><div class="label">Memory</div><div class="value sm"><span data-k="mem">n/a</span><span class="unit"></span></div>
|
||||
<div class="bar" data-bar="mem"><i></i></div></div>
|
||||
<div class="metric"><div class="label">Worst fs</div><div class="value sm"><span data-k="disk">n/a</span><span class="unit"></span></div>
|
||||
<div class="bar" data-bar="disk"><i></i></div></div>
|
||||
<div class="heroes">
|
||||
<div class="hero hero-bar" data-k="cpu" data-sev="cpu.severity.usage_pct"></div>
|
||||
<div class="hero hero-bar" data-k="mem" data-sev="memory.severity.used_pct"></div>
|
||||
<div class="hero" data-k="temp" data-sev="cpu.severity.temp_c"></div>
|
||||
<div class="hero hero-bar" data-k="disk" data-sev="disks.severity.worst_used_pct"></div>
|
||||
<div class="hero-slot" data-k="gpus" data-sev="gpus[].severity.busy_pct"></div>
|
||||
<div class="hero" data-k="net">
|
||||
<div class="label">Net</div>
|
||||
<div class="hero-lines">
|
||||
<div class="line"><span class="dir">↓</span><span data-k="rx">n/a</span></div>
|
||||
<div class="line"><span class="dir">↑</span><span data-k="tx">n/a</span></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<div class="label faint" style="font-size:10.5px;letter-spacing:.06em;text-transform:uppercase">Per-core</div>
|
||||
<div class="cores" data-k="cores" style="margin-top:.25rem"></div>
|
||||
<div class="host-charts">
|
||||
<div class="host-chart">
|
||||
<div class="chart sm" data-chart="util"></div>
|
||||
<div class="legend" data-legend="util"></div>
|
||||
</div>
|
||||
<div class="host-chart">
|
||||
<div class="chart sm" data-chart="net"></div>
|
||||
<div class="legend" data-legend="net"></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="metrics">
|
||||
<div class="metric"><div class="label">Disk read</div><div class="value sm"><span data-k="dr">n/a</span><span class="unit"></span></div></div>
|
||||
<div class="metric"><div class="label">Disk write</div><div class="value sm"><span data-k="dw">n/a</span><span class="unit"></span></div></div>
|
||||
<div class="metric"><div class="label">Disk busy</div><div class="value sm"><span data-k="dutil">n/a</span><span class="unit">%</span></div></div>
|
||||
<div class="metric"><div class="label">Net in</div><div class="value sm"><span data-k="rx">n/a</span><span class="unit"></span></div></div>
|
||||
<div class="metric"><div class="label">Net out</div><div class="value sm"><span data-k="tx">n/a</span><span class="unit"></span></div></div>
|
||||
<div class="metric"><div class="label">Up</div><div class="value sm" data-k="uptime">n/a</div></div>
|
||||
</div>
|
||||
<div data-k="gpus"></div>
|
||||
<details>
|
||||
<summary class="faint" style="cursor:pointer;font-size:12px">Filesystems & processes</summary>
|
||||
<summary class="faint" style="cursor:pointer;font-size:12px">More detail</summary>
|
||||
<div class="metrics" style="margin-top:.6rem">
|
||||
<div class="metric"><div class="label">Load 1 / 5 / 15</div><div class="value sm" data-k="load">n/a</div></div>
|
||||
<div class="metric"><div class="label">Clock</div><div class="value sm" data-k="freq">n/a</div></div>
|
||||
<div class="metric"><div class="label">Disk read</div><div class="value sm" data-k="dr">n/a</div></div>
|
||||
<div class="metric"><div class="label">Disk write</div><div class="value sm" data-k="dw">n/a</div></div>
|
||||
<div class="metric"><div class="label">Disk busy</div><div class="value sm" data-k="dutil">n/a</div></div>
|
||||
<div class="metric"><div class="label">Up</div><div class="value sm" data-k="uptime">n/a</div></div>
|
||||
</div>
|
||||
<div data-k="gpu-extra"></div>
|
||||
<div style="margin-top:.6rem" data-k="details"></div>
|
||||
</details>
|
||||
</div>`;
|
||||
return card;
|
||||
}
|
||||
|
||||
/** `200 / 253 GiB` — the percentage is the verdict, the bytes are the detail. */
|
||||
function memorySub(mem) {
|
||||
const used = Dash.fmtBytes(mem.used_bytes, 1);
|
||||
const total = Dash.fmtBytes(mem.total_bytes, 0);
|
||||
if (!used || !total) return null;
|
||||
if (used.unit === total.unit) return `${used.text} / ${total.text} ${total.unit}`;
|
||||
return `${bytes(mem.used_bytes, 1)} of ${bytes(mem.total_bytes, 0)}`;
|
||||
}
|
||||
|
||||
/** Model, VRAM used / total, and the honest APU pool view: a Strix Halo's model lives in
|
||||
* GTT, so "VRAM 3%" and "pool 78%" are both true and the card shows both, not whichever
|
||||
* flatters the machine. */
|
||||
function gpuSub(gpu) {
|
||||
const bits = [gpu.model || 'GPU'];
|
||||
bits.push(gpu.vram_used_bytes == null || gpu.vram_total_bytes == null
|
||||
? 'VRAM n/a'
|
||||
: `VRAM ${bytes(gpu.vram_used_bytes, 1)} / ${bytes(gpu.vram_total_bytes, 1)}`);
|
||||
if (gpu.mem_used_pct != null) bits.push(`pool ${num(gpu.mem_used_pct, 0)}%`);
|
||||
return bits.join(' · ');
|
||||
}
|
||||
|
||||
/** The GPU slot of the hero row: one hero per card, or the honest no-GPU state. A box
|
||||
* with no card is not a box at 0% busy, and gateway must not look broken for it. */
|
||||
function renderGpuSlot(cell, gpus) {
|
||||
if (cell.dataset.count !== String(gpus.length)) {
|
||||
cell.innerHTML = '';
|
||||
for (let i = 0; i < Math.max(gpus.length, 1); i += 1) {
|
||||
const row = document.createElement('div');
|
||||
row.className = 'hero hero-bar';
|
||||
cell.appendChild(row);
|
||||
}
|
||||
cell.dataset.count = String(gpus.length);
|
||||
}
|
||||
const rows = Array.from(cell.children);
|
||||
if (!gpus.length) {
|
||||
Dash.hero(rows[0], { label: 'GPU', value: null, sub: 'No GPU reported by this agent' });
|
||||
return;
|
||||
}
|
||||
gpus.forEach((gpu, i) => {
|
||||
Dash.hero(rows[i], {
|
||||
label: gpus.length > 1 ? `GPU ${i}` : 'GPU',
|
||||
value: gpu.busy_pct, unit: '%', digits: 0,
|
||||
severity: (gpu.severity || {}).busy_pct, bar: true,
|
||||
sub: gpuSub(gpu),
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/** What the GPU hero left out, in "More detail": power, temperature, both memory pools. */
|
||||
function renderGpuExtra(cell, gpus) {
|
||||
if (!gpus.length) { cell.innerHTML = ''; return; }
|
||||
cell.innerHTML = `
|
||||
<div class="label faint" style="font-size:10.5px;letter-spacing:.06em;text-transform:uppercase;margin-top:.6rem">GPU detail</div>
|
||||
<table class="data" style="margin-top:.25rem"><thead><tr>
|
||||
<th>Card</th><th>Power</th><th>Temp</th><th>VRAM</th><th>GTT</th></tr></thead>
|
||||
<tbody>${gpus.map((gpu) => `<tr>
|
||||
<td class="name mono" style="font-size:11.5px">${DashCharts.escapeHtml(gpu.model || 'GPU')}</td>
|
||||
<td class="num">${gpu.power_watts == null ? 'n/a' : `${num(gpu.power_watts, 0)} W`}</td>
|
||||
<td class="num">${gpu.temp_c == null ? 'n/a' : `${num(gpu.temp_c, 0)} °C`}</td>
|
||||
<td class="num mute">${bytes(gpu.vram_used_bytes, 1)} / ${bytes(gpu.vram_total_bytes, 1)}</td>
|
||||
<td class="num mute">${bytes(gpu.gtt_used_bytes, 1)}</td></tr>`).join('')}</tbody></table>`;
|
||||
}
|
||||
|
||||
function renderHostCard(card, host) {
|
||||
const q = (key) => card.querySelector(`[data-k="${key}"]`);
|
||||
const stateRow = host.state || {};
|
||||
@@ -125,65 +169,41 @@ function renderHostCard(card, host) {
|
||||
markStale(card, stateRow.ok === false || Dash.ageSeconds(stateRow.last_ok) > 5 * (host.window || 10));
|
||||
|
||||
const cpu = host.cpu || {};
|
||||
setValue(q('cpu'), cpu.usage_pct, 0);
|
||||
setBar(card.querySelector('[data-bar="cpu"]'), cpu.usage_pct, severityFor(cpu.usage_pct, 75, 90));
|
||||
setTxt(q('load'), cpu.load1 == null ? 'n/a' : `${num(cpu.load1, 2)} / ${num(cpu.load15, 2)}`);
|
||||
setValue(q('temp'), cpu.temp_c, 1);
|
||||
setValue(q('freq'), cpu.freq_mhz == null ? null : cpu.freq_mhz / 1000, 2);
|
||||
|
||||
const mem = host.memory || {};
|
||||
const memText = Dash.fmtBytes(mem.used_bytes, 1);
|
||||
const memTotal = Dash.fmtBytes(mem.total_bytes, 0);
|
||||
setTxt(q('mem'), memText ? memText.text : 'n/a');
|
||||
setTxt(q('mem').nextElementSibling, memText ? `${memText.unit}/${memTotal ? memTotal.text + memTotal.unit : ''}` : '');
|
||||
setBar(card.querySelector('[data-bar="mem"]'), mem.used_pct, severityFor(mem.used_pct, 80, 92));
|
||||
|
||||
const disks = host.disks || {};
|
||||
setTxt(q('disk'), disks.worst_used_pct == null ? 'n/a' : num(disks.worst_used_pct, 0) + '%');
|
||||
setTxt(q('disk').nextElementSibling, disks.worst_mount ? ' ' + disks.worst_mount : '');
|
||||
setBar(card.querySelector('[data-bar="disk"]'), disks.worst_used_pct, severityFor(disks.worst_used_pct, 80, 92));
|
||||
const net = host.network || {};
|
||||
|
||||
DashCharts.renderCores(q('cores'), cpu.core_usage_pct || []);
|
||||
// The six heroes. The verdict is whatever the payload carried (`severity` dicts written
|
||||
// by severity.py, pinned by tests/test_system_contract.py); a missing number or a
|
||||
// missing verdict prints `n/a` with no colour class at all.
|
||||
Dash.hero(q('cpu'), { label: 'CPU', value: cpu.usage_pct, unit: '%', digits: 0,
|
||||
severity: (cpu.severity || {}).usage_pct, bar: true });
|
||||
Dash.hero(q('mem'), { label: 'Memory', value: mem.used_pct, unit: '%', digits: 0,
|
||||
severity: (mem.severity || {}).used_pct, bar: true,
|
||||
sub: memorySub(mem) });
|
||||
Dash.hero(q('temp'), { label: 'CPU temp', value: cpu.temp_c, unit: '°C', digits: 1,
|
||||
severity: (cpu.severity || {}).temp_c });
|
||||
// the mount is the `.sub` because "80 %" without "of what" is not actionable
|
||||
Dash.hero(q('disk'), { label: 'Disk', value: disks.worst_used_pct, unit: '%', digits: 0,
|
||||
severity: (disks.severity || {}).worst_used_pct, bar: true,
|
||||
sub: disks.worst_mount });
|
||||
renderGpuSlot(q('gpus'), host.gpus || []);
|
||||
// One wire seen in two directions, two lines, never two competing numbers — and a null
|
||||
// (no physical interface reported) stays `n/a` rather than becoming a confident 0 B/s.
|
||||
// No severity by design: `network` carries none, so this slot never colours.
|
||||
setTxt(q('rx'), net.rx_bytes_per_s == null ? null : Dash.rate(net.rx_bytes_per_s));
|
||||
setTxt(q('tx'), net.tx_bytes_per_s == null ? null : Dash.rate(net.tx_bytes_per_s));
|
||||
|
||||
setValue(q('dr'), disks.read_bytes_per_s == null ? null : disks.read_bytes_per_s, null);
|
||||
if (disks.read_bytes_per_s != null) {
|
||||
const parts = Dash.fmtBytes(disks.read_bytes_per_s, 1);
|
||||
setTxt(q('dr'), parts.text); setTxt(q('dr').nextElementSibling, parts.unit + '/s');
|
||||
const w = Dash.fmtBytes(disks.write_bytes_per_s, 1);
|
||||
setTxt(q('dw'), w ? w.text : 'n/a'); setTxt(q('dw').nextElementSibling, w ? w.unit + '/s' : '');
|
||||
} else { setTxt(q('dr'), 'n/a'); setTxt(q('dw'), 'n/a'); }
|
||||
setValue(q('dutil'), disks.util_pct, 0);
|
||||
const rx = Dash.fmtBytes((host.network || {}).rx_bytes_per_s, 1);
|
||||
const tx = Dash.fmtBytes((host.network || {}).tx_bytes_per_s, 1);
|
||||
setTxt(q('rx'), rx ? rx.text : 'n/a'); setTxt(q('rx').nextElementSibling, rx ? rx.unit + '/s' : '');
|
||||
setTxt(q('tx'), tx ? tx.text : 'n/a'); setTxt(q('tx').nextElementSibling, tx ? tx.unit + '/s' : '');
|
||||
setTxt(q('uptime'), duration((host.info || {}).uptime_seconds));
|
||||
|
||||
// GPUs: one row each, with both memory pools named (an APU's model lives in GTT)
|
||||
const gpus = host.gpus || [];
|
||||
const gpuHost = q('gpus');
|
||||
if (!gpus.length) {
|
||||
gpuHost.innerHTML = '<div class="faint" style="font-size:12px">No GPU reported by this agent</div>';
|
||||
} else {
|
||||
gpuHost.innerHTML = gpus.map((gpu) => `
|
||||
<div class="stack" style="gap:.2rem;margin-top:.35rem">
|
||||
<div class="gap" style="justify-content:space-between">
|
||||
<span class="mono" style="font-size:12px">${DashCharts.escapeHtml(gpu.model || 'GPU')}</span>
|
||||
<span class="mono" style="font-size:12px">
|
||||
${gpu.busy_pct == null ? 'n/a' : num(gpu.busy_pct, 0) + '% busy'} ·
|
||||
${gpu.power_watts == null ? 'power n/a' : num(gpu.power_watts, 0) + ' W'} ·
|
||||
${gpu.temp_c == null ? 'temp n/a' : num(gpu.temp_c, 0) + '°C'}</span>
|
||||
</div>
|
||||
<div class="bar thin"><i style="width:${(gpu.busy_pct || 0).toFixed(1)}%"></i></div>
|
||||
<div class="gap" style="justify-content:space-between;font-size:11.5px" class="faint">
|
||||
<span class="faint">VRAM ${gpu.vram_used_bytes == null ? 'n/a' : bytes(gpu.vram_used_bytes, 0)}
|
||||
${gpu.vram_total_bytes ? 'of ' + bytes(gpu.vram_total_bytes, 0) : ''}</span>
|
||||
<span class="faint">shared (GTT) ${gpu.gtt_used_bytes == null ? 'n/a' : bytes(gpu.gtt_used_bytes, 0)}
|
||||
${gpu.mem_used_pct == null ? '' : ' · pool ' + num(gpu.mem_used_pct, 0) + '%'}</span>
|
||||
</div>
|
||||
<div class="bar thin"><i style="width:${(gpu.mem_used_pct || 0).toFixed(1)}%;background:var(--series-2)"></i></div>
|
||||
</div>`).join('');
|
||||
}
|
||||
// "More detail": everything the glance does not need but the page still owes.
|
||||
setTxt(q('load'), cpu.load1 == null && cpu.load5 == null && cpu.load15 == null
|
||||
? null : `${num(cpu.load1, 2)} / ${num(cpu.load5, 2)} / ${num(cpu.load15, 2)}`);
|
||||
setTxt(q('freq'), cpu.freq_mhz == null ? null : `${num(cpu.freq_mhz / 1000, 2)} GHz`);
|
||||
setTxt(q('dr'), disks.read_bytes_per_s == null ? null : Dash.rate(disks.read_bytes_per_s));
|
||||
setTxt(q('dw'), disks.write_bytes_per_s == null ? null : Dash.rate(disks.write_bytes_per_s));
|
||||
setTxt(q('dutil'), disks.util_pct == null ? null : `${num(disks.util_pct, 0)}%`);
|
||||
const up = (host.info || {}).uptime_seconds;
|
||||
setTxt(q('uptime'), up == null ? null : duration(up));
|
||||
renderGpuExtra(q('gpu-extra'), host.gpus || []);
|
||||
|
||||
const filesystems = (disks.filesystems || []).filter((fs) => fs.used_pct != null);
|
||||
const procs = host.processes || {};
|
||||
@@ -226,180 +246,175 @@ function stalenessText(row) {
|
||||
return seconds > 30 ? `stale ${duration(seconds)}` : `updated ${seconds < 2 ? 'now' : duration(seconds)}`;
|
||||
}
|
||||
|
||||
// ---- charts ---------------------------------------------------------------
|
||||
function makeCharts() {
|
||||
const common = { height: 165, xSpan: state.range };
|
||||
state.charts.cpu = new DashCharts.LineChart('#chart-cpu', Object.assign({}, common, {
|
||||
yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) }));
|
||||
state.charts.mem = new DashCharts.LineChart('#chart-mem', Object.assign({}, common, {
|
||||
yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) }));
|
||||
state.charts.disk = new DashCharts.LineChart('#chart-disk', Object.assign({}, common, {
|
||||
yMin: 0, yFormat: (v) => (Dash.fmtBytes(v, 0) || { text: '0' }).text }));
|
||||
state.charts.net = new DashCharts.LineChart('#chart-net', Object.assign({}, common, {
|
||||
yMin: 0, yFormat: (v) => (Dash.fmtBytes(v, 0) || { text: '0' }).text }));
|
||||
state.charts.gpu = new DashCharts.LineChart('#chart-gpu', Object.assign({}, common, {
|
||||
yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) }));
|
||||
state.charts.temp = new DashCharts.LineChart('#chart-temp', Object.assign({}, common, {
|
||||
yFormat: (v) => num(v, 0) }));
|
||||
state.charts.detail = new DashCharts.LineChart('#chart-detail', Object.assign({}, common, {
|
||||
height: 200, yFormat: (v) => num(v, 1) }));
|
||||
}
|
||||
// ---- per-host charts ------------------------------------------------------
|
||||
//
|
||||
// One host, one panel, at most three charts — and no chart on this page contains a second
|
||||
// host. Six metrics x three hosts as six separate charts is the unreadable version of
|
||||
// "per-host charts", so the metrics that share a unit share an axis:
|
||||
//
|
||||
// util cpu.usage_pct + mem.used_pct + disk.used_pct one 0-100 % axis
|
||||
// net net.rx_bytes_per_s + net.tx_bytes_per_s one byte-rate axis
|
||||
// gpu gpu.busy_pct@<index> per card the agent reports (no chart when there is none)
|
||||
//
|
||||
// Each host costs **one** `/api/system/history` request per refresh: the endpoint takes a
|
||||
// `hosts=` and a `metrics=` list, so the XHR budget for a three-host fleet is one snapshot
|
||||
// plus three history calls plus the event feed, whatever the chart count. GPU stream names
|
||||
// come from the payload's own `gpus[].index` (`sources/host.py` writes `gpu.busy_pct@<i>`),
|
||||
// so nothing here has to discover them through /api/metrics the way the deleted
|
||||
// fleet-wide GPU chart did.
|
||||
//
|
||||
// The charts live in a Map keyed by host name and are torn down with the card: a panel for
|
||||
// a host that has left the payload would otherwise keep drawing its last series forever.
|
||||
const CHART_HEIGHT = 120;
|
||||
const UTIL_SERIES = [
|
||||
['cpu.usage_pct', 'CPU'],
|
||||
['mem.used_pct', 'Mem'],
|
||||
['disk.used_pct', 'Disk (worst)'],
|
||||
];
|
||||
const NET_SERIES = [
|
||||
['net.rx_bytes_per_s', '↓ in'],
|
||||
['net.tx_bytes_per_s', '↑ out'],
|
||||
];
|
||||
const PCT_FORMAT = (v) => `${num(v, 1)}%`;
|
||||
const RATE_FORMAT = (v) => `${bytes(v, 1)}/s`;
|
||||
const AXIS_PCT = (v) => num(v, 0);
|
||||
const AXIS_RATE = (v) => {
|
||||
const parts = Dash.fmtBytes(v, 0) || { text: '0', unit: '' };
|
||||
return `${parts.text}${parts.unit}`;
|
||||
};
|
||||
|
||||
function setChartSpan(span) {
|
||||
Object.values(state.charts).forEach((chart) => { chart.opts.xSpan = span; });
|
||||
}
|
||||
|
||||
function seriesFor(response, metric) {
|
||||
return (response.series || []).filter((s) => s.metric === metric).map((s) => ({
|
||||
name: s.host, points: s.points.map((p) => ({ ts: p.ts, value: p.value })),
|
||||
}));
|
||||
}
|
||||
|
||||
async function refreshCharts(snapshot) {
|
||||
const hostNames = (snapshot.hosts || []).map((h) => h.name);
|
||||
const detailHost = state.detailHost || hostNames[0] || null;
|
||||
const span = state.range;
|
||||
const requests = {
|
||||
cpu: fetchJSON(`/api/system/history?metrics=cpu.usage_pct&range=${span}`).catch(() => null),
|
||||
mem: fetchJSON(`/api/system/history?metrics=mem.used_pct&range=${span}`).catch(() => null),
|
||||
temp: fetchJSON(`/api/system/history?metrics=temp.cpu_c&range=${span}`).catch(() => null),
|
||||
disk: detailHost ? fetchJSON(`/api/system/history?hosts=${encodeURIComponent(detailHost)}&` +
|
||||
`metrics=disk.read_bytes_per_s,disk.write_bytes_per_s&range=${span}`).catch(() => null) : null,
|
||||
net: detailHost ? fetchJSON(`/api/system/history?hosts=${encodeURIComponent(detailHost)}&` +
|
||||
`metrics=net.rx_bytes_per_s,net.tx_bytes_per_s&range=${span}`).catch(() => null) : null,
|
||||
};
|
||||
const [cpu, mem, temp, disk, net] = await Promise.all([requests.cpu, requests.mem,
|
||||
requests.temp, requests.disk, requests.net]);
|
||||
|
||||
if (cpu) {
|
||||
const series = seriesFor(cpu, 'cpu.usage_pct');
|
||||
state.charts.cpu.setData(series);
|
||||
DashCharts.renderLegend($('#legend-cpu'), series, (v) => num(v, 1) + '%', true);
|
||||
}
|
||||
if (mem) {
|
||||
const series = seriesFor(mem, 'mem.used_pct');
|
||||
state.charts.mem.setData(series);
|
||||
DashCharts.renderLegend($('#legend-mem'), series, (v) => num(v, 1) + '%');
|
||||
}
|
||||
if (temp) {
|
||||
const series = seriesFor(temp, 'temp.cpu_c');
|
||||
state.charts.temp.setData(series);
|
||||
DashCharts.renderLegend($('#legend-temp'), series, (v) => num(v, 1) + '°C', true);
|
||||
}
|
||||
if (disk) {
|
||||
const read = (disk.series.find((s) => s.metric === 'disk.read_bytes_per_s') || { points: [] })
|
||||
.points.map((p) => ({ ts: p.ts, value: p.value }));
|
||||
const write = (disk.series.find((s) => s.metric === 'disk.write_bytes_per_s') || { points: [] })
|
||||
.points.map((p) => ({ ts: p.ts, value: p.value }));
|
||||
const series = [{ name: 'read', points: read }, { name: 'write', points: write }];
|
||||
state.charts.disk.setData(series);
|
||||
DashCharts.renderLegend($('#legend-disk'), series, (v) => bytes(v, 1) + '/s', true);
|
||||
}
|
||||
if (net) {
|
||||
const rx = (net.series.find((s) => s.metric === 'net.rx_bytes_per_s') || { points: [] })
|
||||
.points.map((p) => ({ ts: p.ts, value: p.value }));
|
||||
const tx = (net.series.find((s) => s.metric === 'net.tx_bytes_per_s') || { points: [] })
|
||||
.points.map((p) => ({ ts: p.ts, value: p.value }));
|
||||
const series = [{ name: 'in', points: rx }, { name: 'out', points: tx }];
|
||||
state.charts.net.setData(series);
|
||||
DashCharts.renderLegend($('#legend-net'), series, (v) => bytes(v, 1) + '/s', true);
|
||||
}
|
||||
await refreshGpuChart(hostNames);
|
||||
await refreshDetailChart();
|
||||
}
|
||||
|
||||
async function refreshGpuChart(hostNames) {
|
||||
if (!hostNames.length) {
|
||||
state.charts.gpu.setData([]);
|
||||
$('#legend-gpu').innerHTML = '<span class="faint">no hosts</span>';
|
||||
return;
|
||||
}
|
||||
const series = [];
|
||||
for (const host of hostNames) {
|
||||
// per-stream names are discovered, not guessed: gpu.busy_pct@0, @1, ...
|
||||
const found = await fetchJSON(`/api/metrics?source=host&host=${encodeURIComponent(host)}`)
|
||||
.catch(() => ({ metrics: [] }));
|
||||
const names = (found.metrics || []).filter((m) => m.startsWith('gpu.busy_pct@'));
|
||||
for (const name of names) {
|
||||
const data = await fetchJSON(`/api/history?source=host&host=${encodeURIComponent(host)}` +
|
||||
`&metric=${encodeURIComponent(name)}&range=${state.range}`).catch(() => null);
|
||||
if (data) series.push({ name: `${host} ${name.split('@')[1]}`,
|
||||
points: data.points.map((p) => ({ ts: p.ts, value: p.value })) });
|
||||
}
|
||||
}
|
||||
state.charts.gpu.setData(series);
|
||||
DashCharts.renderLegend($('#legend-gpu'), series, (v) => num(v, 0) + '%', true);
|
||||
}
|
||||
|
||||
async function refreshDetailChart() {
|
||||
const host = state.detailHost;
|
||||
if (!host) { state.charts.detail.setData([]); return; }
|
||||
const kind = metricKind(state.detailMetric);
|
||||
const format = formatFor(kind);
|
||||
const data = await fetchJSON(`/api/history?source=host&host=${encodeURIComponent(host)}` +
|
||||
`&metric=${encodeURIComponent(state.detailMetric)}&range=${state.range}`).catch(() => null);
|
||||
if (!data) return;
|
||||
const series = [{ name: state.detailMetric,
|
||||
points: data.points.map((p) => ({ ts: p.ts, value: p.value })) }];
|
||||
state.charts.detail.opts.yFormat = format.axis;
|
||||
state.charts.detail.opts.yMin = state.detailMetric.endsWith('_pct') ? 0 : null;
|
||||
state.charts.detail.opts.yMax = state.detailMetric.endsWith('_pct') ? 100 : null;
|
||||
state.charts.detail.setData(series);
|
||||
DashCharts.renderLegend($('#legend-detail'), series, format.fn);
|
||||
const last = series[0].points.slice(-1)[0];
|
||||
$('#detail-note').textContent = `${series[0].points.length} samples over ${duration(state.range)} · ` +
|
||||
`last ${last ? format.fn(last.value) : 'n/a'} · ${state.range <= 86400 ? 'raw resolution' : 'rolled up'}`;
|
||||
}
|
||||
|
||||
async function populateDetailMetrics(host) {
|
||||
const select = $('#detail-metric');
|
||||
const found = await fetchJSON(`/api/metrics?source=host&host=${encodeURIComponent(host)}`)
|
||||
.catch(() => ({ metrics: [] }));
|
||||
const curated = new Set(DETAIL_METRICS.map(([name]) => name));
|
||||
const extras = (found.metrics || []).filter((m) => !curated.has(m)).sort();
|
||||
select.innerHTML = '';
|
||||
const groupMain = document.createElement('optgroup');
|
||||
groupMain.label = 'Common';
|
||||
DETAIL_METRICS.forEach(([name, label]) => {
|
||||
const option = document.createElement('option');
|
||||
option.value = name; option.textContent = label;
|
||||
groupMain.appendChild(option);
|
||||
/** The GPU streams one host reports, named from the payload rather than guessed. */
|
||||
function gpuStreams(host) {
|
||||
return (host.gpus || []).map((gpu, i) => {
|
||||
const index = gpu.index == null ? i : gpu.index;
|
||||
return { metric: `gpu.busy_pct@${index}`, label: gpu.model || `GPU ${index}` };
|
||||
});
|
||||
select.appendChild(groupMain);
|
||||
if (extras.length) {
|
||||
const groupExtra = document.createElement('optgroup');
|
||||
groupExtra.label = `Per-device (${extras.length})`;
|
||||
extras.forEach((name) => {
|
||||
const option = document.createElement('option');
|
||||
option.value = name; option.textContent = name;
|
||||
groupExtra.appendChild(option);
|
||||
});
|
||||
select.appendChild(groupExtra);
|
||||
}
|
||||
|
||||
/**
|
||||
* The chart objects inside one host panel, built once and reused every poll.
|
||||
*
|
||||
* The GPU chart is added the first time the host reports a card and removed the moment it
|
||||
* stops reporting one — a machine whose GPU went missing must not keep a plot of it.
|
||||
*/
|
||||
function hostChartSet(card, host) {
|
||||
let set = state.charts.get(host.name);
|
||||
if (!set) {
|
||||
// no `area` here on purpose — `drawHostChart()` decides the fill per draw, from how many
|
||||
// series actually have points, which is the only question worth asking
|
||||
const common = { height: CHART_HEIGHT, xSpan: state.range };
|
||||
// Every chart names itself for assistive tech: the panel heading says *who*, the
|
||||
// aria-label says what is plotted. An unlabelled SVG is silence, and a screen reader
|
||||
// must not need the colour-coded legend to know which machine it is listening to.
|
||||
set = {
|
||||
util: new DashCharts.LineChart(card.querySelector('[data-chart="util"]'),
|
||||
Object.assign({}, common, { yMin: 0, yMax: 100, yFormat: AXIS_PCT,
|
||||
ariaLabel: `${host.name} — utilisation over the selected window: CPU, memory and worst disk in per cent` })),
|
||||
net: new DashCharts.LineChart(card.querySelector('[data-chart="net"]'),
|
||||
Object.assign({}, common, { yMin: 0, yFormat: AXIS_RATE,
|
||||
ariaLabel: `${host.name} — network traffic over the selected window: bytes in and bytes out` })),
|
||||
gpu: null,
|
||||
};
|
||||
state.charts.set(host.name, set);
|
||||
}
|
||||
select.value = DETAIL_METRICS.some(([name]) => name === state.detailMetric)
|
||||
? state.detailMetric : (extras[0] || DETAIL_METRICS[0][0]);
|
||||
state.detailMetric = select.value || DETAIL_METRICS[0][0];
|
||||
const wantsGpu = gpuStreams(host).length > 0;
|
||||
if (wantsGpu && !set.gpu) {
|
||||
const slot = document.createElement('div');
|
||||
slot.className = 'host-chart';
|
||||
slot.dataset.chartSlot = 'gpu';
|
||||
slot.innerHTML = '<div class="chart sm" data-chart="gpu"></div>' +
|
||||
'<div class="legend" data-legend="gpu"></div>';
|
||||
card.querySelector('.host-charts').appendChild(slot);
|
||||
set.gpu = new DashCharts.LineChart(slot.querySelector('[data-chart="gpu"]'), {
|
||||
height: CHART_HEIGHT, xSpan: state.range, yMin: 0, yMax: 100, yFormat: AXIS_PCT,
|
||||
ariaLabel: `${host.name} — GPU busy per cent over the selected window: ` +
|
||||
gpuStreams(host).map((s) => s.label).join(', ') });
|
||||
}
|
||||
if (!wantsGpu && set.gpu) {
|
||||
// the whole slot goes: an empty wrapper or a plot of a card that is no longer
|
||||
// reported is a machine's panel lying about hardware it does not have
|
||||
card.querySelectorAll('[data-chart-slot="gpu"]').forEach((node) => node.remove());
|
||||
set.gpu = null;
|
||||
}
|
||||
return set;
|
||||
}
|
||||
|
||||
/**
|
||||
* `[metric, label][]` -> LineChart series, from one host's history response.
|
||||
*
|
||||
* The legend names the *metric*, never the host: a chart here can only ever hold one
|
||||
* machine, so repeating the host name three times under a chart is ink with no information.
|
||||
*/
|
||||
function seriesFor(response, definitions) {
|
||||
return definitions.map(([metric, label]) => {
|
||||
const found = (response.series || []).find((s) => s.metric === metric);
|
||||
return { name: label,
|
||||
points: ((found || {}).points || []).map((p) => ({ ts: p.ts, value: p.value })) };
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Plot one series set into one panel chart and repaint its legend.
|
||||
*
|
||||
* The fill is decided per draw on purpose: a single series reads as a shape worth shading
|
||||
* (GPU busy), while two or three filled areas stack into one block and bury the smaller
|
||||
* line — which is how a quiet CPU hides behind a busy memory plot.
|
||||
*/
|
||||
function drawHostChart(chart, card, key, series, format) {
|
||||
chart.opts.xSpan = state.range;
|
||||
chart.opts.area = series.filter((s) => s.points.length).length < 2;
|
||||
chart.setData(series);
|
||||
DashCharts.renderLegend(card.querySelector(`[data-legend="${key}"]`), series, format, true);
|
||||
}
|
||||
|
||||
/** One host's charts, from one request. Called for every host on every refresh. */
|
||||
async function renderHostCharts(card, host) {
|
||||
const charts = hostChartSet(card, host);
|
||||
const gpu = gpuStreams(host);
|
||||
const wanted = UTIL_SERIES.concat(NET_SERIES, gpu.map((s) => [s.metric, s.label]));
|
||||
const query = `/api/system/history?hosts=${encodeURIComponent(host.name)}` +
|
||||
`&metrics=${encodeURIComponent(wanted.map(([metric]) => metric).join(','))}` +
|
||||
`&range=${state.range}`;
|
||||
const response = await fetchJSON(query).catch(() => null);
|
||||
if (!response || !state.charts.has(host.name)) return; // the panel went away mid-flight
|
||||
|
||||
drawHostChart(charts.util, card, 'util', seriesFor(response, UTIL_SERIES), PCT_FORMAT);
|
||||
drawHostChart(charts.net, card, 'net', seriesFor(response, NET_SERIES), RATE_FORMAT);
|
||||
if (charts.gpu) {
|
||||
drawHostChart(charts.gpu, card, 'gpu',
|
||||
seriesFor(response, gpu.map((s) => [s.metric, s.label])), PCT_FORMAT);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The Δ window the rates were actually measured over, for the page footnote.
|
||||
*
|
||||
* `window_seconds` beside the fleet is the *poll interval*; the window behind a rate is the
|
||||
* estimator's (`RATE_WINDOW_SECONDS`) and the two need not agree, so the footnote reads each
|
||||
* host's own measured `window_seconds` instead of repeating a config number as a measurement
|
||||
* (invariant 2: a rate says how it was obtained). One poll loop drives every host, so the
|
||||
* windows differ by milliseconds; a real spread prints as a range rather than picking a winner.
|
||||
*/
|
||||
function windowLabel(values, fallback) {
|
||||
const nums = values.filter((value) => value != null && Number.isFinite(value));
|
||||
if (!nums.length) return fallback;
|
||||
const lo = Math.min(...nums);
|
||||
const hi = Math.max(...nums);
|
||||
return lo.toFixed(1) === hi.toFixed(1) ? `Δ${hi.toFixed(1)}s`
|
||||
: `Δ${lo.toFixed(1)}–${hi.toFixed(1)}s`;
|
||||
}
|
||||
|
||||
/** Drop cards (and charts) for hosts that are no longer in the payload. */
|
||||
function pruneHosts(living) {
|
||||
Array.from(state.hosts.keys()).forEach((name) => {
|
||||
if (living.includes(name)) return;
|
||||
state.hosts.get(name).remove();
|
||||
state.hosts.delete(name);
|
||||
state.charts.delete(name);
|
||||
});
|
||||
}
|
||||
|
||||
// ---- snapshot -------------------------------------------------------------
|
||||
function renderFleet(fleet) {
|
||||
if (!fleet) return;
|
||||
setTxt($('#f-hosts'), fleet.hosts_live == null ? 'n/a' : String(fleet.hosts_live));
|
||||
setTxt($('#f-hosts-of'), fleet.hosts_total == null ? '' : `of ${fleet.hosts_total}`);
|
||||
setValue($('#f-cpu'), fleet.cpu_usage_pct_avg, 0);
|
||||
setValue($('#f-load'), fleet.load1_total, 2);
|
||||
const used = Dash.fmtBytes(fleet.memory_used_bytes, 0);
|
||||
const total = Dash.fmtBytes(fleet.memory_total_bytes, 0);
|
||||
setTxt($('#f-mem'), used ? used.text : 'n/a');
|
||||
setTxt($('#f-mem-of'), used && total ? `${used.unit} of ${total.text} ${total.unit}` : '');
|
||||
setValue($('#f-disk'), fleet.disk_worst_used_pct, 0);
|
||||
const rx = Dash.fmtBytes(fleet.net_rx_bytes_per_s, 1);
|
||||
const tx = Dash.fmtBytes(fleet.net_tx_bytes_per_s, 1);
|
||||
setTxt($('#f-rx'), rx ? rx.text : 'n/a'); setTxt($('#f-rx-unit'), rx ? rx.unit + '/s' : '');
|
||||
setTxt($('#f-tx'), tx ? tx.text : 'n/a'); setTxt($('#f-tx-unit'), tx ? tx.unit + '/s' : '');
|
||||
setValue($('#f-gpu'), fleet.gpu_busy_pct_avg, 0);
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
let snapshot;
|
||||
try {
|
||||
@@ -411,6 +426,7 @@ async function refresh() {
|
||||
const states = {};
|
||||
(snapshot.states || []).forEach((row) => { states[row.key.split(':')[1]] = row; });
|
||||
const hostNames = [];
|
||||
const chartJobs = [];
|
||||
(snapshot.hosts || []).forEach((host) => {
|
||||
hostNames.push(host.name);
|
||||
host.state = states[host.name];
|
||||
@@ -420,43 +436,33 @@ async function refresh() {
|
||||
card = hostCard(host.name);
|
||||
state.hosts.set(host.name, card);
|
||||
$('#hosts').appendChild(card);
|
||||
card.addEventListener('click', () => {
|
||||
if (state.detailHost !== host.name) {
|
||||
state.detailHost = host.name;
|
||||
$('#detail-host').value = host.name;
|
||||
populateDetailMetrics(host.name).then(refreshDetailChart);
|
||||
}
|
||||
});
|
||||
}
|
||||
renderHostCard(card, host);
|
||||
// one history request per host, all of its charts fed from the same response
|
||||
chartJobs.push(renderHostCharts(card, host));
|
||||
});
|
||||
// a host that left HOST_TARGETS must take its panel — and its live charts — with it
|
||||
pruneHosts(hostNames);
|
||||
$('#hosts-empty').hidden = hostNames.length > 0;
|
||||
|
||||
const hostSelect = $('#detail-host');
|
||||
if (hostSelect.dataset.names !== hostNames.join(',')) {
|
||||
hostSelect.dataset.names = hostNames.join(',');
|
||||
hostSelect.innerHTML = hostNames.map((name) =>
|
||||
`<option value="${DashCharts.escapeHtml(name)}">${DashCharts.escapeHtml(name)}</option>`).join('');
|
||||
if (!state.detailHost || !hostNames.includes(state.detailHost)) {
|
||||
state.detailHost = hostNames[0] || null;
|
||||
}
|
||||
if (state.detailHost) hostSelect.value = state.detailHost;
|
||||
if (state.detailHost) await populateDetailMetrics(state.detailHost);
|
||||
}
|
||||
|
||||
renderFleet(snapshot.fleet);
|
||||
const fleet = snapshot.fleet || {};
|
||||
$('#sub').textContent = `${fleet.hosts_live || 0}/${fleet.hosts_total || 0} hosts answering · ` +
|
||||
`${fleet.cpu_cores || 0} cores · ${fleet.gpu_count || 0} GPUs · ` +
|
||||
`window ${duration(state.range)}`;
|
||||
$('#foot-note').textContent = `rates over Δ${snapshot.window_seconds}s · ` +
|
||||
`poll ${Math.round(state.pollMs / 1000)}s`;
|
||||
await refreshCharts(snapshot);
|
||||
// the Δ comes from the payloads, never from the interval the fleet happens to answer on
|
||||
$('#foot-note').textContent = `rates over ${windowLabel(
|
||||
(snapshot.hosts || []).map((host) => host.window_seconds),
|
||||
`Δ${snapshot.window_seconds}s`, // nothing measured yet — the configured interval
|
||||
)} · poll ${Math.round(state.pollMs / 1000)}s`;
|
||||
await Promise.all(chartJobs);
|
||||
}
|
||||
|
||||
async function refreshEvents() {
|
||||
try {
|
||||
const data = await fetchJSON('/api/events?limit=25&source=host');
|
||||
// `system`, not `host`: reachability/reboot events are logged under the *dashboard* name
|
||||
// (`poller.py` writes `Event(ts, self.dashboard, …)`; samples use `host`, events do not).
|
||||
// Querying `source=host` matched nothing and left this feed forever empty.
|
||||
const data = await fetchJSON('/api/events?limit=25&source=system');
|
||||
const list = $('#events');
|
||||
if (!(data.events || []).length) {
|
||||
list.innerHTML = '<li class="muted" style="grid-template-columns:1fr">No host transitions ' +
|
||||
@@ -471,7 +477,6 @@ async function refreshEvents() {
|
||||
}
|
||||
|
||||
document.addEventListener('DOMContentLoaded', async () => {
|
||||
makeCharts();
|
||||
let pollMs = 10000;
|
||||
try {
|
||||
const config = await fetchJSON('/api/config');
|
||||
@@ -482,22 +487,12 @@ document.addEventListener('DOMContentLoaded', async () => {
|
||||
} catch (err) { /* config is nice-to-have */ }
|
||||
state.pollMs = pollMs;
|
||||
|
||||
// the window chips re-fetch every panel's history: the range is per page, not per chart
|
||||
Dash.rangeControl($('#range'), state.range, (seconds) => {
|
||||
state.range = seconds;
|
||||
setChartSpan(seconds);
|
||||
refresh();
|
||||
});
|
||||
|
||||
$('#detail-host').addEventListener('change', async (event) => {
|
||||
state.detailHost = event.target.value;
|
||||
await populateDetailMetrics(state.detailHost);
|
||||
await refreshDetailChart();
|
||||
});
|
||||
$('#detail-metric').addEventListener('change', async (event) => {
|
||||
state.detailMetric = event.target.value;
|
||||
await refreshDetailChart();
|
||||
});
|
||||
|
||||
state.poller = poll(async () => { await refresh(); await refreshEvents(); }, state.pollMs);
|
||||
Dash.bindPause(state.poller);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
"""The `/system` hero-card payload contract, shared by the hermetic and the live tests.
|
||||
|
||||
Phase 03 makes `/system` one panel per host with six big colour-coded numbers. The paths
|
||||
below are the payload fields those cards are allowed to read; they were read out of
|
||||
`parse_host_snapshot()` and are pinned hermetically in `tests/test_system_contract.py`
|
||||
(driven from `tests/fixtures/agent_*.json`) and against the real fleet in
|
||||
`tests/test_live_fleet.py::test_every_host_serves_the_hero_card_contract`.
|
||||
|
||||
Keeping the list here rather than in one test file means the live assertions cannot drift
|
||||
away from the fixture assertions: there is one table, and it is the thing under test.
|
||||
|
||||
=========================== ==========================================================
|
||||
Card Path in `/api/system/snapshot` `hosts[]`
|
||||
=========================== ==========================================================
|
||||
cpu % ``cpu.usage_pct`` (+ ``cpu.severity.usage_pct``)
|
||||
mem % ``memory.used_pct`` (+ ``memory.severity.used_pct``)
|
||||
cpu temp ``cpu.temp_c`` (+ ``cpu.severity.temp_c``)
|
||||
disk usage % ``disks.worst_used_pct`` + ``disks.worst_mount`` (+ verdict)
|
||||
graphics card ``gpus[].model/busy_pct/vram_*/mem_used_pct`` (+ verdict)
|
||||
network down/up ``network.rx_bytes_per_s`` / ``tx_bytes_per_s`` (no verdict)
|
||||
=========================== ==========================================================
|
||||
|
||||
CPU temp comes from ``cpu.temp_c`` rather than ``temperatures.cpu_c``: both are the agent's
|
||||
own ``cpu_temperature_celsius``, and ``cpu.temp_c`` is the field ``/api/summary`` flattens
|
||||
into its fleet row, so the card and the overview cannot drift apart.
|
||||
|
||||
Deliberately *not* card metrics (legibility, not missing data — the series stay in SQLite):
|
||||
per-core usage (``cpu.usage_pct@coreN``), swap, the process table, the raw sensor list, and
|
||||
per-interface rows. Network carries no ``severity`` on purpose; throughput has no good/bad,
|
||||
so ``severity.THRESHOLDS`` has no ``net.*`` row.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
#: Card metric -> dotted payload paths. Present is the contract; a value may be ``None``
|
||||
#: on a host that genuinely does not have the number (gateway reports no CPU temperature).
|
||||
CARD_METRICS: dict[str, tuple[str, ...]] = {
|
||||
"cpu %": ("cpu.usage_pct", "cpu.severity.usage_pct"),
|
||||
"mem %": ("memory.used_pct", "memory.severity.used_pct"),
|
||||
"cpu temp": ("cpu.temp_c", "cpu.severity.temp_c", "temperatures.cpu_c"),
|
||||
# the mount is what makes the percentage actionable: 88 % used — of which disk?
|
||||
"disk %": ("disks.worst_used_pct", "disks.worst_mount", "disks.severity.worst_used_pct"),
|
||||
"network down/up": ("network.rx_bytes_per_s", "network.tx_bytes_per_s"),
|
||||
}
|
||||
|
||||
#: Per-GPU paths, checked on every row of `gpus[]` — and on none of them when the list is
|
||||
#: empty, which is how a host without a GPU stays honest (no card, not a card at 0 % busy).
|
||||
GPU_METRICS: tuple[str, ...] = (
|
||||
"model",
|
||||
"busy_pct",
|
||||
"vram_used_bytes",
|
||||
"vram_total_bytes",
|
||||
"vram_used_pct",
|
||||
# the honest APU view: vram + GTT, because a Strix Halo keeps the model in GTT
|
||||
"mem_used_pct",
|
||||
"severity.busy_pct",
|
||||
"severity.mem_used_pct",
|
||||
)
|
||||
|
||||
#: `network` must never grow a `severity` dict (see the module docstring).
|
||||
SEVERITY_FREE_SECTIONS: tuple[str, ...] = ("network",)
|
||||
|
||||
#: The host shapes the captures in `tests/fixtures/` cover, and therefore the shapes a card
|
||||
#: must handle: a discrete GPU, an APU (VRAM *and* GTT), and no GPU at all. A new capture
|
||||
#: has to be added here — `test_the_contract_covers_every_captured_host` fails otherwise.
|
||||
HOST_SHAPES: tuple[str, ...] = ("driveripper", "deskwork", "gateway")
|
||||
|
||||
|
||||
def dig(node: Any, path: str) -> Any:
|
||||
"""Follow a dotted payload path, failing with the path in the message.
|
||||
|
||||
A missing key is a contract break; a ``None`` value is not (that renders `n/a`), which is
|
||||
why this returns the value instead of asserting on it.
|
||||
"""
|
||||
walked = ""
|
||||
for part in path.split("."):
|
||||
assert isinstance(node, dict), f"{path}: hit a non-dict at {walked or 'payload'}"
|
||||
assert part in node, (
|
||||
f"{path}: no key {part!r} under {walked or 'payload'} — a hero card would print "
|
||||
f"n/a for a number the API used to serve"
|
||||
)
|
||||
node = node[part]
|
||||
walked = f"{walked}.{part}" if walked else part
|
||||
return node
|
||||
|
||||
|
||||
def assert_card_contract(payload: Any) -> None:
|
||||
"""Every card path exists on this host payload, GPU rows included."""
|
||||
for paths in CARD_METRICS.values():
|
||||
for path in paths:
|
||||
dig(payload, path)
|
||||
gpus = payload.get("gpus")
|
||||
assert isinstance(gpus, list), "gpus must be a list — [] when the agent reports no GPU"
|
||||
for gpu in gpus:
|
||||
for path in GPU_METRICS:
|
||||
dig(gpu, path)
|
||||
for section in SEVERITY_FREE_SECTIONS:
|
||||
block = payload.get(section)
|
||||
assert isinstance(block, dict), f"{section} must be a dict"
|
||||
assert "severity" not in block, (
|
||||
f"{section}.severity would be a colour nobody measured — throughput has no "
|
||||
f"good/bad and severity.THRESHOLDS has no {section}.* row"
|
||||
)
|
||||
+292
-5
@@ -388,6 +388,60 @@ def test_agent_tokens_never_reach_the_browser(tmp_path: Path, monkeypatch: pytes
|
||||
store.close()
|
||||
|
||||
|
||||
def test_a_dead_host_never_writes_its_token_into_the_feed(tmp_path: Path) -> None:
|
||||
"""The second path a credential can take to a browser: the failure message.
|
||||
|
||||
`parse_targets` moves a userinfo token out of the URL and into a bearer header, so the
|
||||
documented `HOST_TARGETS` path never puts one in `Target.url` — this is the backstop for
|
||||
any `Target` that does (a hand-built one, a future parser), and the message path is where
|
||||
it would matter: the poller keeps an `UpstreamError` string in the health record and logs
|
||||
it as an event, so a userinfo-bearing URL there would sit in SQLite and be served on
|
||||
`/api/events` forever (AGENTS.md invariant 6). Asserted on every endpoint a page reads,
|
||||
not just `/api/config`.
|
||||
"""
|
||||
db = str(tmp_path / "dead-host-token.sqlite3")
|
||||
store = SampleStore(db)
|
||||
settings = Settings(
|
||||
db_path=db,
|
||||
host_interval=0.05,
|
||||
host_targets=(Target(name="driveripper", url="http://SEKRIT-AGENT-TOKEN@127.0.0.1:1"),),
|
||||
)
|
||||
with TestClient(create_app(settings, store)) as client:
|
||||
deadline = time.time() + 10
|
||||
while time.time() < deadline:
|
||||
health = client.get("/api/health").json()
|
||||
if health["sources"] and health["sources"][0]["error"]:
|
||||
break
|
||||
time.sleep(0.05)
|
||||
else:
|
||||
pytest.fail("the dead host never recorded an error")
|
||||
for path in ("/api/health", "/api/events", "/api/system/snapshot", "/api/summary"):
|
||||
body = client.get(path).text
|
||||
assert "SEKRIT-AGENT-TOKEN" not in body, f"{path} leaked the agent token"
|
||||
# and the outage is still legible — redaction is not the same as silence
|
||||
assert "driveripper" in client.get("/api/health").text
|
||||
assert "unreachable" in client.get("/api/events").text
|
||||
store.close()
|
||||
|
||||
|
||||
def test_engine_urls_are_redacted_the_same_way_hosts_are(tmp_path: Path) -> None:
|
||||
"""`GUFO_BASE_URL`/`VLLM_BASE_URL` go to the browser verbatim in `/api/config`, and unlike
|
||||
`HOST_TARGETS` nothing parses a userinfo token out of them into a header — so a URL that
|
||||
happens to carry credentials has to be redacted where it is served (invariant 6).
|
||||
"""
|
||||
db = str(tmp_path / "engine-redact.sqlite3")
|
||||
settings = Settings(db_path=db, gufo_url="http://SEKRIT-ENGINE-TOKEN@deskwork:8731",
|
||||
vllm_url="http://driveripper:8000")
|
||||
store = SampleStore(db)
|
||||
with TestClient(create_app(settings, store)) as client:
|
||||
body = client.get("/api/config")
|
||||
assert "SEKRIT-ENGINE-TOKEN" not in body.text
|
||||
llm = body.json()["llm"]
|
||||
assert llm["gufo"]["base_url"] == "http://deskwork:8731"
|
||||
assert llm["vllm"]["base_url"] == "http://driveripper:8000" # untouched when clean
|
||||
store.close()
|
||||
|
||||
|
||||
def wait_for_state(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
@@ -522,13 +576,14 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
|
||||
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
|
||||
"""The old call sites still work until the remaining phases convert each page.
|
||||
|
||||
`/` was converted in phase 02: the overview colours from the `severity` verdicts
|
||||
`/api/summary` already sends, so it no longer carries a threshold of its own. The
|
||||
three pages still to come must keep compiling against the fallback until they are.
|
||||
`/` was converted in phase 02 and `/system` in phase 03: both colour themselves from
|
||||
the `severity` verdicts the API already sends, so they carry no threshold of their
|
||||
own. The two pages still to come must keep compiling against the fallback until they
|
||||
are converted — deleting it early would break a page mid-migration.
|
||||
"""
|
||||
js = client.get("/static/app.js").text
|
||||
assert "FALLBACK ONLY" in js
|
||||
for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"):
|
||||
for page in ("/static/page-llm.js", "/static/page-ups.js"):
|
||||
assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site"
|
||||
|
||||
|
||||
@@ -539,6 +594,198 @@ def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> Non
|
||||
assert "severityFor" not in js
|
||||
|
||||
|
||||
def test_system_page_colours_from_the_payload_not_from_js_thresholds(client) -> None:
|
||||
"""`page-system.js` hero cards paint payload verdicts, not thresholds of their own.
|
||||
|
||||
Phase 03 task 02's promise: every colour on a host card comes from a `severity` dict
|
||||
`severity.py` produced (`cpu.severity.usage_pct`, `memory.severity.used_pct`,
|
||||
`disks.severity.worst_used_pct`, `gpus[].severity.busy_pct`, `cpu.severity.temp_c`).
|
||||
A threshold written here could disagree with the overview's alerts; a hero with no
|
||||
verdict prints `n/a` uncoloured, which is `Dash.hero()`'s contract.
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
assert "severityFor(" not in js, "the system page re-invented a threshold"
|
||||
assert "Dash.hero(" in js
|
||||
for hook in ("cpu.severity.usage_pct", "memory.severity.used_pct",
|
||||
"disks.severity.worst_used_pct", "gpus[].severity.busy_pct",
|
||||
"cpu.severity.temp_c"):
|
||||
assert hook in js, f"the card no longer names {hook} as a colour source"
|
||||
|
||||
|
||||
def test_system_host_card_is_six_heroes_without_per_core(client) -> None:
|
||||
"""Each host card leads with exactly six heroes: CPU, MEM, TEMP, DISK, GPU, NET.
|
||||
|
||||
Asserted on the served script because there is no JS test runner (a locked anchor).
|
||||
The two dense `.metrics` rows and the per-core heat bars are exactly what made the
|
||||
card unreadable, and "no per-core anywhere on /system" is a promise a stray
|
||||
`renderCores` call would silently break — the series stays stored, the page just
|
||||
stopped drawing it (PLAN.md *Dropped visuals*).
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
card = js[js.index("function hostCard("):js.index("function renderHostCard(")]
|
||||
slots = re.findall(r'<div class="(?:hero|hero-slot)[^"]*" data-k="([^"]+)"', card)
|
||||
assert slots == ["cpu", "mem", "temp", "disk", "gpus", "net"], \
|
||||
"the hero row is no longer the six numbers in the agreed order"
|
||||
# per-core visualisation is gone from the card code entirely
|
||||
assert "cores" not in card, "a per-core visual came back to the card"
|
||||
assert "renderCores" not in js and "core_usage" not in js
|
||||
# the NET slot is a slot with two lines and no verdict: a byte rate has no good/bad
|
||||
assert 'data-k="net"' in card
|
||||
assert not re.search(r'data-k="net"[^>]*data-sev', card), "the NET slot invented a colour"
|
||||
# what the glance leaves out stays reachable, collapsed by default
|
||||
assert "More detail" in card
|
||||
for key in ("load", "freq", "dr", "dw", "dutil", "uptime", "details"):
|
||||
assert f'data-k="{key}"' in card, f"{key} left the page instead of the glance"
|
||||
# the panel owns its charts: the slots live inside the card, not on the page
|
||||
for chart in ("util", "net"):
|
||||
assert f'data-chart="{chart}"' in card, f"the panel lost its {chart} chart slot"
|
||||
|
||||
|
||||
def test_system_page_keeps_no_fleet_wide_chart_or_metric_picker(client) -> None:
|
||||
"""Nothing on `/system` plots two hosts, and nothing needs a selection to make sense.
|
||||
|
||||
TODO.md L4 is the instruction ("The system page doesn't need shared charts for
|
||||
anything") and the overview's alerts + fleet table are what made removing them safe
|
||||
(phase 02). Asserted on the served bytes because the only thing that would otherwise
|
||||
notice a resurrected `<select>` is a browser: there is no JS test runner (a locked
|
||||
anchor), and `test_no_page_carries_an_id_nothing_reads` cannot see ids that were deleted
|
||||
correctly from the page but are still queried by the script.
|
||||
"""
|
||||
html = _code(client.get("/system").text)
|
||||
for gone in ('id="chart-cpu"', 'id="chart-mem"', 'id="chart-disk"', 'id="chart-net"',
|
||||
'id="chart-gpu"', 'id="chart-temp"', 'id="chart-detail"', 'id="detail-host"',
|
||||
'id="detail-metric"', 'id="f-cpu"', 'id="f-load"', 'id="f-gpu"',
|
||||
"Fleet totals", "Fleet charts", "Series detail", "<select"):
|
||||
assert gone not in html, f"/system still carries {gone}"
|
||||
# the two panels that are left are the per-host grid and the event feed
|
||||
assert html.count("<section") == 2, "/system grew a panel back on top of the host grid"
|
||||
assert 'id="hosts"' in html and 'id="events"' in html
|
||||
|
||||
# and the script must not reach for the deleted endpoints/ids either — a `#detail-host`
|
||||
# query against a removed node is a silent null, so the picker would just never appear
|
||||
js = client.get("/static/page-system.js").text
|
||||
for gone in ("#chart-", "detail-host", "detail-metric", "detail-note",
|
||||
"renderFleet", "makeCharts", "refreshGpuChart", "populateDetailMetrics"):
|
||||
assert gone not in js, f"page-system.js still references {gone}"
|
||||
|
||||
|
||||
def test_every_system_chart_is_drawn_inside_its_own_host_panel(client) -> None:
|
||||
"""One host per chart: the legends of `/system` can never list two machine names.
|
||||
|
||||
The construction is a factory that takes the card, so a chart cannot be built against a
|
||||
page-level id; the metric list is per host, so the GPU streams cannot leak from one
|
||||
panel into another; and a panel is torn down (with its charts) when its host leaves the
|
||||
payload, which is the bug class a keyed-by-host chart map exists to avoid.
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
builds = re.findall(r"new DashCharts\.LineChart\(([^,]+),", js)
|
||||
assert builds, "the system page builds no charts at all"
|
||||
# every chart is built from a `[data-chart]` slot inside the host card, never from a
|
||||
# page-level id — which is what makes "one host per chart" structural, not a convention
|
||||
assert all(re.search(r"(?:card|slot)\.querySelector\(.\[data-chart=", b) for b in builds), \
|
||||
f"a chart is built outside a host panel: {builds}"
|
||||
# three construction sites, one per slot (util, net, gpu) — never one per metric
|
||||
assert len(builds) == 3, f"charts are not built one per panel slot: {builds}"
|
||||
assert sorted(re.findall(r'data-chart=\\?"(\w+)"', " ".join(builds))) == ["gpu", "net", "util"]
|
||||
|
||||
factory = js[js.index("async function renderHostCharts("):js.index("function pruneHosts(")]
|
||||
assert "hosts=${encodeURIComponent(host.name)}" in factory, \
|
||||
"the per-host chart query is not scoped to one host"
|
||||
assert "&metrics=" in factory and "state.range" in factory, \
|
||||
"a host's charts need one metric list and the selected window"
|
||||
# GPU series are named from the payload's own `gpus[].index`, never discovered/guessed
|
||||
assert "gpu.busy_pct@" in js and "gpu.index" in js
|
||||
# the GPU chart exists only while the host reports cards
|
||||
gpu_slot = js[js.index("function hostChartSet("):js.index("function seriesFor(")]
|
||||
assert re.search(r"if \(!wantsGpu && set\.gpu\)", gpu_slot), \
|
||||
"a host that stops reporting GPUs keeps a GPU chart it has no data for"
|
||||
|
||||
pruner = js[js.index("function pruneHosts("):js.index("// ---- snapshot")]
|
||||
assert "state.hosts.delete(name)" in pruner and "state.charts.delete(name)" in pruner, \
|
||||
"a host that left the payload keeps a live panel and drawing charts"
|
||||
|
||||
|
||||
def test_every_system_chart_names_itself_for_assistive_tech(client) -> None:
|
||||
"""A 120px SVG is silence to a screen reader unless the page names it.
|
||||
|
||||
The phase-03 accessibility pass (task 04) found the per-host charts drawing anonymous
|
||||
plots: the legend is a sibling element and labels nothing, so the `<svg>` itself must
|
||||
carry `role="img"` + `aria-label` that says which host and which metrics. Pinned on the
|
||||
served bytes because there is no JS test runner (a locked anchor).
|
||||
"""
|
||||
charts_js = client.get("/static/charts.js").text
|
||||
# the shared LineChart applies the name to the svg, and only when the caller opted in
|
||||
assert "if (this.opts.ariaLabel)" in charts_js, "LineChart stopped honouring ariaLabel"
|
||||
assert "setAttribute('role', 'img')" in charts_js
|
||||
assert "setAttribute('aria-label', String(this.opts.ariaLabel))" in charts_js
|
||||
|
||||
js = client.get("/static/page-system.js").text
|
||||
# all three slots name themselves, and each name starts with the host the panel belongs
|
||||
# to — an aria-label that cannot say *which machine* is not an accessible name here
|
||||
assert js.count("ariaLabel: `") == 3, "a per-host chart slot lost its accessible name"
|
||||
for slot in ("utilisation", "network traffic", "GPU busy"):
|
||||
assert f"${{host.name}} \u2014 {slot}" in js or f"${{host.name}} — {slot}" in js, \
|
||||
f"the {slot} chart has no host-scoped accessible name"
|
||||
|
||||
|
||||
def test_system_event_feed_queries_the_source_transitions_are_stored_under(client) -> None:
|
||||
"""Host transitions are stored under the *dashboard* name, not the sample source.
|
||||
|
||||
`poller.py` logs `Event(ts, self.dashboard, …)` — host jobs write `system`, the way
|
||||
the UPS job writes `ups` — while *samples* use `host`. The `/system` feed queried
|
||||
`source=host` since the first commit and therefore always rendered "no transitions
|
||||
yet", even with a real outage (a gateway disconnect) sitting in the store. Pinned on
|
||||
the served bytes because there is no JS test runner (a locked anchor).
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
feed = js[js.index("async function refreshEvents("):]
|
||||
url = re.search(r"fetchJSON\('(/api/events[^']*)'\)", feed)
|
||||
assert url, "the /system page no longer fetches the event feed"
|
||||
assert "source=system" in url.group(1), \
|
||||
"the /system event feed must ask for the source the poller stores events under"
|
||||
assert "source=host" not in url.group(1), \
|
||||
"source=host matches no stored event — the feed is empty forever"
|
||||
|
||||
|
||||
def test_system_footnote_prints_the_window_it_actually_measured(client) -> None:
|
||||
"""`Δ<window>` is the page's promise that a rate is a measurement (invariant 2), so the
|
||||
number behind it has to come from the measurement.
|
||||
|
||||
The `window_seconds` beside the fleet is `HOST_INTERVAL_SECONDS`, while the window behind a
|
||||
rate is the estimator's `RATE_WINDOW_SECONDS` — equal at the defaults, which is why the
|
||||
footnote could quote the poll interval and look right. Set `RATE_WINDOW_SECONDS=30` and it
|
||||
would have labelled every rate on the page with a window it never used. The footnote now
|
||||
reads each host's own measured `window_seconds`, and prints a range when they disagree.
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
assert "function windowLabel(" in js, "the footnote lost its measured-window helper"
|
||||
footnote = js[js.index("$('#foot-note').textContent"):js.index("await Promise.all(chartJobs)")]
|
||||
assert "windowLabel(" in footnote
|
||||
assert "host.window_seconds" in footnote, \
|
||||
"the footnote is quoting the poll interval as a measurement again"
|
||||
# a footnote that says "window" and a page that says "poll" must stay two separate claims
|
||||
assert "poll ${Math.round(state.pollMs / 1000)}s" in footnote
|
||||
|
||||
|
||||
def test_no_per_core_visual_library_survives_the_page(client) -> None:
|
||||
"""The per-core heat bars left `/system` and `charts.js` with them (PLAN.md *Dropped
|
||||
visuals* is about what the pages draw, not what the store keeps).
|
||||
|
||||
`renderCores` had one caller — the hero-card per-core block deleted in task 02 — so
|
||||
keeping it would be dead markup generating unstyled `.core` spans; `.cores` was the CSS
|
||||
that block needed. Both are checked absent here so neither can come back as a
|
||||
half-change, while `sparkline`/`LineChart` stay for phase 04 and the UPS page.
|
||||
"""
|
||||
assert "renderCores" not in client.get("/static/charts.js").text
|
||||
css = client.get("/static/dash.css").text
|
||||
assert ".cores" not in css
|
||||
for asset in ASSETS:
|
||||
assert "core_usage" not in _code(client.get(asset).text), f"{asset} plots per-core"
|
||||
# what phase 04 and the other pages still chart from
|
||||
for name in ("LineChart", "sparkline", "renderLegend"):
|
||||
assert name in client.get("/static/charts.js").text
|
||||
|
||||
|
||||
def test_overview_is_exactly_two_panels(client) -> None:
|
||||
"""The landing page answers one question: what is wrong, and what is the fleet doing.
|
||||
|
||||
@@ -682,7 +929,14 @@ def test_parse_range_refuses_a_window_it_cannot_interpret(raw: str) -> None:
|
||||
|
||||
def test_system_history_charts_every_configured_host_in_one_request(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""N hosts x M metrics, one round trip — the chart endpoint's whole reason to exist."""
|
||||
"""N hosts x M metrics, one round trip.
|
||||
|
||||
`/system` no longer uses the N-hosts half of this (every chart belongs to one host and
|
||||
the page issues one request per host, pinned by
|
||||
`test_every_system_chart_is_drawn_inside_its_own_host_panel`); the shape is kept because
|
||||
the metric-list half is exactly what the per-host query costs one request with, and
|
||||
because the endpoint is a public interface for programmatic consumers.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
host_targets=(Target("driveripper", DEAD_URL),
|
||||
Target("deskwork", DEAD_URL))) as test_client:
|
||||
@@ -702,6 +956,39 @@ def test_system_history_charts_every_configured_host_in_one_request(
|
||||
assert narrowed["hosts"] == ["deskwork"]
|
||||
|
||||
|
||||
def test_one_request_per_host_answers_every_per_host_chart(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The exact query `/system` issues per host returns every series that panel draws.
|
||||
|
||||
`page-system.js` sends one request per host carrying the utilisation trio
|
||||
(`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct`), both network rates, and one
|
||||
`gpu.busy_pct@<index>` per card the payload reported — so a name missing here is a chart
|
||||
that silently reads "no samples yet" in the browser, and a renamed series in
|
||||
`sources/host.py` fails here rather than in a screenshot. This is the metric-name half of
|
||||
the contract; `tests/test_system_contract.py` holds the payload-path half.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
|
||||
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
|
||||
now = time.time()
|
||||
metrics = ["cpu.usage_pct", "mem.used_pct", "disk.used_pct",
|
||||
"net.rx_bytes_per_s", "net.tx_bytes_per_s", "gpu.busy_pct@0"]
|
||||
for metric in metrics:
|
||||
store.add("host", "driveripper", metric, 12.5, ts=now - 60)
|
||||
store.add("host", "driveripper", metric, 25.0, ts=now - 30)
|
||||
|
||||
body = test_client.get("/api/system/history", params={
|
||||
"hosts": "driveripper", "metrics": ",".join(metrics), "range": "1h"}).json()
|
||||
assert body["hosts"] == ["driveripper"], "one panel, one host"
|
||||
assert [item["metric"] for item in body["series"]] == metrics
|
||||
assert [len(item["points"]) for item in body["series"]] == [2] * len(metrics)
|
||||
|
||||
# a second card the host does not have answers empty, never zero and never 404
|
||||
gap = test_client.get("/api/system/history",
|
||||
params={"hosts": "driveripper", "metrics": "gpu.busy_pct@1"}).json()
|
||||
assert gap["series"][0]["points"] == []
|
||||
|
||||
|
||||
def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
|
||||
|
||||
@@ -120,6 +120,44 @@ async def test_a_refused_connection_is_an_upstream_error_not_a_traceback() -> No
|
||||
HostCollector(RateEstimator()))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", ["refused", "timeout", "http-500"])
|
||||
async def test_a_failing_host_never_echoes_its_token_in_the_message(failure: str) -> None:
|
||||
"""An `UpstreamError` message is not just a log line: the poller keeps it in the health
|
||||
record (`/api/health`) and writes it to the `event` table (`/api/events`), both of which
|
||||
the browser reads — so the URL it names has to be the redacted one, on every failure path
|
||||
(AGENTS.md invariant 6). `parse_targets` normally moves a userinfo token into a header, so
|
||||
this is the backstop that keeps the guarantee true for any `Target` that still carries one.
|
||||
"""
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if failure == "refused":
|
||||
raise httpx.ConnectError("connection refused")
|
||||
if failure == "timeout":
|
||||
raise httpx.ReadTimeout("too slow")
|
||||
return httpx.Response(500, text="boom")
|
||||
|
||||
target = Target("gw", "http://SEKRIT-AGENT-TOKEN@agent.test")
|
||||
async with client_for(handler) as client:
|
||||
with pytest.raises(UpstreamError) as excinfo:
|
||||
await fetch_host_snapshot(client, target, HostCollector(RateEstimator()))
|
||||
message = excinfo.value.message
|
||||
assert "SEKRIT-AGENT-TOKEN" not in message, f"the token leaked through {failure}"
|
||||
# …and the message still names the machine it failed to reach
|
||||
assert "http://agent.test/snapshot" in message
|
||||
|
||||
|
||||
async def test_a_failing_engine_never_echoes_credentials_in_its_url() -> None:
|
||||
"""Same rule for the engines: the URL in the message is the redacted one."""
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
raise httpx.ConnectError("connection refused")
|
||||
|
||||
async with client_for(handler) as client:
|
||||
with pytest.raises(UpstreamError, match="unreachable") as excinfo:
|
||||
await fetch_engine(client, "gufo", "http://SEKRIT-ENGINE-TOKEN@agent.test", "",
|
||||
RateEstimator())
|
||||
assert "SEKRIT-ENGINE-TOKEN" not in excinfo.value.message
|
||||
assert "http://agent.test/metrics" in excinfo.value.message
|
||||
|
||||
|
||||
async def test_a_slow_agent_is_a_gateway_timeout() -> None:
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
raise httpx.ReadTimeout("too slow")
|
||||
|
||||
@@ -47,6 +47,21 @@ def test_token_in_url_becomes_a_bearer_header_and_stays_out_of_the_display_url()
|
||||
assert "SEKRIT" not in target.display_url
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw,shown", [
|
||||
("http://SEKRIT@host:9120", "http://host:9120"),
|
||||
("https://user:pass@host:8000/metrics", "https://host:8000/metrics"),
|
||||
("HTTP://SEKRIT@host:1", "HTTP://host:1"),
|
||||
("http://host:9120", "http://host:9120"), # clean URLs pass through untouched
|
||||
("", ""),
|
||||
])
|
||||
def test_strip_userinfo_is_the_one_redaction_every_url_leaves_by(raw: str, shown: str) -> None:
|
||||
"""`strip_userinfo` is what `/api/config` and every `UpstreamError` message use, so it has
|
||||
to survive the shapes a URL actually comes in: both schemes, `user:pass@`, a path, and the
|
||||
common no-credentials case where it must change nothing (invariant 6).
|
||||
"""
|
||||
assert config.strip_userinfo(raw) == shown
|
||||
|
||||
|
||||
def test_target_names_are_restricted_because_they_are_db_keys() -> None:
|
||||
"""Names key the history DB and appear in URLs, so they are validated, not guessed.
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@ import os
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from card_contract import assert_card_contract
|
||||
from dashboards import collectors
|
||||
from dashboards.config import Settings, load
|
||||
from dashboards.rates import RateEstimator
|
||||
@@ -70,6 +71,27 @@ async def test_every_host_agent_answers_and_parses(settings: Settings,
|
||||
)
|
||||
|
||||
|
||||
async def test_every_host_serves_the_hero_card_contract(settings: Settings,
|
||||
host_collector: HostCollector) -> None:
|
||||
"""The phase-03 `/system` card paths exist on the **real** agents, not just the captures.
|
||||
|
||||
Values may honestly be `n/a` on real hardware (gateway reports no CPU sensor and no GPU
|
||||
at all); the *shape* is what phase 03 renders, so this is the test that catches an agent
|
||||
upgrade renaming a field. Hermetic twin: `tests/test_system_contract.py`.
|
||||
"""
|
||||
if not settings.host_targets:
|
||||
pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh")
|
||||
async with httpx.AsyncClient(timeout=settings.host_timeout) as client:
|
||||
for target in settings.host_targets:
|
||||
await collectors.fetch_host_snapshot(client, target, host_collector)
|
||||
await asyncio.sleep(1.2)
|
||||
result = await collectors.fetch_host_snapshot(client, target, host_collector)
|
||||
assert_card_contract(result.payload)
|
||||
assert result.payload["cpu"]["temp_c"] == result.payload["temperatures"]["cpu_c"], (
|
||||
f"{target.name}: the card and /api/summary would report different CPU temps"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("engine", ["gufo", "vllm"])
|
||||
async def test_engine_still_publishes_the_series_we_parse(engine: str,
|
||||
settings: Settings) -> None:
|
||||
|
||||
@@ -0,0 +1,288 @@
|
||||
"""The payload contract behind the `/system` hero cards (phase 03, task 01).
|
||||
|
||||
Phase 03 turns `/system` into one panel per host with six big colour-coded numbers. Before
|
||||
any of it is rendered, the data has to be proven present — otherwise a card that shows `n/a`
|
||||
gets "fixed" by inventing a field, and the field it invents is a number the agent never
|
||||
reported. This file drives the verbatim captures in `tests/fixtures/` through
|
||||
`parse_host_snapshot()` (and through a real app), so a rename breaks here rather than in a
|
||||
browser at 2am (`AGENTS.md` invariant 7).
|
||||
|
||||
The path table itself lives in `tests/card_contract.py` so the live suite asserts the same
|
||||
list against real agents (`tests/test_live_fleet.py::test_every_host_serves_the_hero_card_contract`)
|
||||
and the two cannot drift apart.
|
||||
|
||||
What is **deliberately not on the card** — a legibility decision, not a missing feature:
|
||||
per-core usage (still stored per logical CPU as `cpu.usage_pct@coreN`, see
|
||||
`test_visuals_the_page_drops_are_still_stored`), swap, the process table, the raw sensor
|
||||
list, and per-interface rows. Removing a plot is not removing data (PLAN.md *Dropped
|
||||
visuals*), so the decision stays reversible.
|
||||
|
||||
Two choices are recorded here because the task asked for them in writing:
|
||||
|
||||
* **CPU temp comes from `cpu.temp_c`**, not `temperatures.cpu_c`. They are the same
|
||||
measurement (the agent's own `cpu_temperature_celsius`) on every capture, `cpu.temp_c` is
|
||||
what `/api/summary` flattens into its fleet row, and a test below keeps them equal.
|
||||
* **The disk verdict lives at `disks.severity.worst_used_pct`** (phase 01 attaches severity
|
||||
as a sibling dict per section, so there is no nested `disks.worst` object to hang it on).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import math
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from card_contract import GPU_METRICS, HOST_SHAPES, assert_card_contract, dig
|
||||
from dashboards.app import create_app
|
||||
from dashboards.config import Settings, Target
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources.host import HostCollector, parse_host_snapshot
|
||||
from dashboards.store import SampleStore
|
||||
|
||||
#: One scrape is not a rate, so the API-level app points at a port that refuses instantly.
|
||||
DEAD_URL = "http://127.0.0.1:1"
|
||||
|
||||
|
||||
def make_collector() -> HostCollector:
|
||||
return HostCollector(RateEstimator(span=60.0))
|
||||
|
||||
|
||||
def _bump_modes(modes: dict[str, Any], delta: float) -> None:
|
||||
"""Move a /proc/stat mode set so the window reads `delta` ticks, 10 % of them busy."""
|
||||
if not isinstance(modes.get("total"), (int, float)):
|
||||
return
|
||||
modes["total"] += delta
|
||||
for idle_mode, share in (("idle", 0.8), ("iowait", 0.1)):
|
||||
if isinstance(modes.get(idle_mode), (int, float)):
|
||||
modes[idle_mode] += delta * share
|
||||
|
||||
|
||||
def _advance(snapshot: dict[str, Any], dt: float = 10.0) -> dict[str, Any]:
|
||||
"""The same scrape `dt` later with counters moved — the second point every rate needs."""
|
||||
moved = copy.deepcopy(snapshot)
|
||||
moved["ts"] = float(snapshot["ts"]) + dt
|
||||
_bump_modes(moved["cpu"]["modes"], 1_000.0)
|
||||
for core in moved["cpu"].get("cores") or []:
|
||||
_bump_modes(core.get("modes") or {}, 1_000.0)
|
||||
for iface in (moved.get("network") or {}).get("interfaces") or []:
|
||||
for key, value in list(iface.items()):
|
||||
if key.endswith("_total") and isinstance(value, (int, float)):
|
||||
iface[key] = value + 50_000.0
|
||||
return moved
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def payloads(agent_snapshots: dict[str, dict[str, Any]]) -> dict[str, dict[str, Any]]:
|
||||
"""Each capture parsed once, first scrape — the shape the cards must tolerate."""
|
||||
out: dict[str, dict[str, Any]] = {}
|
||||
for name, snapshot in agent_snapshots.items():
|
||||
out[name] = parse_host_snapshot(name, snapshot, make_collector(), now=1_000.0).payload
|
||||
return out
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def api_client(tmp_path: Any, agent_snapshot: dict[str, Any],
|
||||
host_scrape: Any) -> Iterator[TestClient]:
|
||||
"""A running app serving one host payload, so the contract is checked over HTTP too."""
|
||||
db = str(tmp_path / "contract.sqlite3")
|
||||
store = SampleStore(db)
|
||||
app = create_app(Settings(db_path=db, host_targets=(Target("driveripper", DEAD_URL),)),
|
||||
store)
|
||||
client = TestClient(app)
|
||||
client.__enter__()
|
||||
try:
|
||||
# the job's port refuses, so a poll can never overwrite what is injected here
|
||||
app.state.ctx.hub.jobs["host:driveripper"].state.payload = host_scrape(
|
||||
agent_snapshot, cpu_busy=10.0)
|
||||
yield client
|
||||
finally:
|
||||
client.__exit__(None, None, None)
|
||||
store.close()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Every card metric has a path, on every host shape
|
||||
# --------------------------------------------------------------------------- #
|
||||
@pytest.mark.parametrize("name", HOST_SHAPES)
|
||||
def test_every_card_metric_has_a_payload_path_on_every_host(
|
||||
payloads: dict[str, dict[str, Any]], name: str) -> None:
|
||||
assert_card_contract(payloads[name])
|
||||
|
||||
|
||||
def test_the_contract_covers_every_captured_host(agent_snapshots: dict[str, dict[str, Any]],
|
||||
payloads: dict[str, dict[str, Any]]) -> None:
|
||||
"""A fourth capture must join `HOST_SHAPES`, or its host shape goes uncontracted."""
|
||||
assert set(HOST_SHAPES) == set(payloads) == set(agent_snapshots)
|
||||
|
||||
|
||||
def test_the_discrete_gpu_capture_supplies_real_numbers(
|
||||
payloads: dict[str, dict[str, Any]], agent_snapshots: dict[str, dict[str, Any]]) -> None:
|
||||
"""Not just present: correct, on the host that has all six metrics."""
|
||||
payload = payloads["driveripper"]
|
||||
snapshot = agent_snapshots["driveripper"]
|
||||
|
||||
assert payload["cpu"]["temp_c"] == snapshot["cpu_temperature_celsius"]
|
||||
# the two places the CPU temperature is exposed are the same measurement, always
|
||||
assert payload["temperatures"]["cpu_c"] == payload["cpu"]["temp_c"]
|
||||
|
||||
memory = snapshot["memory"]
|
||||
assert payload["memory"]["used_pct"] == pytest.approx(
|
||||
memory["used_bytes"] / memory["total_bytes"] * 100.0)
|
||||
|
||||
worst = max(snapshot["filesystems"], key=lambda fs: fs["used_pct"])
|
||||
assert payload["disks"]["worst_mount"] == worst["mountpoint"]
|
||||
assert payload["disks"]["worst_used_pct"] == pytest.approx(worst["used_pct"])
|
||||
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["model"]
|
||||
assert gpu["busy_pct"] == pytest.approx(snapshot["gpus"][0]["busy_pct"])
|
||||
assert gpu["vram_used_pct"] == pytest.approx(
|
||||
gpu["vram_used_bytes"] / gpu["vram_total_bytes"] * 100.0)
|
||||
|
||||
# one scrape is not a rate: usage and throughput are honestly n/a until the second
|
||||
assert payload["cpu"]["usage_pct"] is None
|
||||
assert payload["network"]["rx_bytes_per_s"] is None
|
||||
|
||||
|
||||
def test_measured_values_appear_on_the_second_scrape(host_scrape: Any,
|
||||
agent_snapshot: dict[str, Any]) -> None:
|
||||
"""The payload the API actually serves (after two scrapes) has numbers in all six places."""
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=33.0)
|
||||
assert payload["cpu"]["usage_pct"] == pytest.approx(33.0)
|
||||
assert math.isfinite(payload["network"]["rx_bytes_per_s"])
|
||||
assert math.isfinite(payload["network"]["tx_bytes_per_s"])
|
||||
assert payload["memory"]["used_pct"] is not None
|
||||
assert payload["disks"]["worst_used_pct"] is not None
|
||||
assert payload["cpu"]["temp_c"] is not None
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["busy_pct"] is not None
|
||||
assert gpu["vram_used_pct"] is not None
|
||||
assert gpu["mem_used_pct"] is not None
|
||||
|
||||
|
||||
def test_each_card_number_travels_with_its_own_verdict(host_scrape: Any,
|
||||
agent_snapshot: dict[str, Any]) -> None:
|
||||
"""The card colours a verdict from the payload; it never holds a threshold (phase 01)."""
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=33.0)
|
||||
assert payload["cpu"]["severity"] == {
|
||||
"usage_pct": classify("cpu.usage_pct", payload["cpu"]["usage_pct"]),
|
||||
"temp_c": classify("cpu.temp_c", payload["cpu"]["temp_c"]),
|
||||
}
|
||||
assert payload["memory"]["severity"]["used_pct"] == classify(
|
||||
"mem.used_pct", payload["memory"]["used_pct"])
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] == classify(
|
||||
"disk.used_pct", payload["disks"]["worst_used_pct"])
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["severity"] == {
|
||||
"busy_pct": classify("gpu.busy_pct", gpu["busy_pct"]),
|
||||
"mem_used_pct": classify("gpu.mem_used_pct", gpu["mem_used_pct"]),
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The two host shapes that are not "a box with a discrete GPU"
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_the_box_with_no_gpu_reports_an_empty_list_not_a_missing_key(
|
||||
payloads: dict[str, dict[str, Any]]) -> None:
|
||||
"""gateway has no GPU: `gpus == []`, never a missing key and never a card at 0 % busy."""
|
||||
payload = payloads["gateway"]
|
||||
assert "gpus" in payload, "a host with no GPU still has the key; the page says 'no GPU'"
|
||||
assert payload["gpus"] == []
|
||||
# the other five numbers keep their paths, with the honest None where nothing is measured
|
||||
assert_card_contract(payload)
|
||||
assert payload["cpu"]["temp_c"] is None, "this agent reports no CPU sensor"
|
||||
assert payload["cpu"]["severity"]["temp_c"] is None, "unknown is not ok"
|
||||
|
||||
|
||||
def test_the_apu_reports_vram_and_the_whole_addressable_pool(
|
||||
payloads: dict[str, dict[str, Any]]) -> None:
|
||||
"""deskwork (Strix Halo) is why `mem_used_pct` exists: 3 % VRAM and 78 % memory, both true."""
|
||||
gpu = payloads["deskwork"]["gpus"][0]
|
||||
assert gpu["vram_total_bytes"] < gpu["mem_total_bytes"]
|
||||
assert gpu["vram_used_pct"] < gpu["mem_used_pct"]
|
||||
assert gpu["model"], "the card needs a name to put beside the numbers"
|
||||
|
||||
|
||||
def test_paths_survive_a_host_that_reports_almost_nothing(agent_snapshot: dict[str, Any]) -> None:
|
||||
"""Keys present, values None: the card must render `n/a` without a KeyError."""
|
||||
snapshot = copy.deepcopy(agent_snapshot)
|
||||
snapshot["cpu_temperature_celsius"] = None
|
||||
snapshot["filesystems"] = []
|
||||
snapshot["network"]["interfaces"] = []
|
||||
snapshot["gpus"][0]["busy_pct"] = None
|
||||
snapshot["gpus"][0]["vram_total_bytes"] = None
|
||||
|
||||
payload = parse_host_snapshot("driveripper", snapshot, make_collector(),
|
||||
now=1_000.0).payload
|
||||
assert_card_contract(payload)
|
||||
for path in GPU_METRICS:
|
||||
dig(payload["gpus"][0], path)
|
||||
|
||||
assert payload["cpu"]["temp_c"] is None and payload["cpu"]["severity"]["temp_c"] is None
|
||||
assert payload["disks"]["worst_used_pct"] is None
|
||||
assert payload["disks"]["worst_mount"] is None
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] is None
|
||||
assert payload["network"]["rx_bytes_per_s"] is None
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["busy_pct"] is None and gpu["severity"]["busy_pct"] is None
|
||||
assert gpu["vram_used_pct"] is None, "no total, so no percentage — not 0 %"
|
||||
assert gpu["mem_used_pct"] is None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# What the cards drop stays in the database
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_visuals_the_page_drops_are_still_stored(agent_snapshot: dict[str, Any]) -> None:
|
||||
"""Per-core / per-mount / per-interface / sensor / process data leaves the page.
|
||||
|
||||
It must still be *sampled*: PLAN.md's *Dropped visuals* anchor says the page stops
|
||||
drawing what the operator stopped asking for while history keeps the series, so the
|
||||
decision stays reversible.
|
||||
"""
|
||||
collector = make_collector()
|
||||
parse_host_snapshot("driveripper", agent_snapshot, collector, now=1_000.0)
|
||||
result = parse_host_snapshot("driveripper", _advance(agent_snapshot), collector,
|
||||
now=1_010.0)
|
||||
stored = {sample.metric for sample in result.samples}
|
||||
|
||||
assert result.payload["cpu"]["usage_pct"] == pytest.approx(10.0), "the counters moved"
|
||||
cores = {metric for metric in stored if metric.startswith("cpu.usage_pct@core")}
|
||||
logical_cpus = result.payload["info"]["cpu_cores_logical"]
|
||||
assert len(cores) == logical_cpus, "one series per logical CPU, plotted or not"
|
||||
assert "cpu.usage_pct@core0" in stored and f"cpu.usage_pct@core{logical_cpus - 1}" in stored
|
||||
assert "disk.used_pct@/" in stored, "per-mount usage is still stored; the card shows worst"
|
||||
assert any(metric.startswith("net.rx_bytes_per_s@") for metric in stored), (
|
||||
"per-interface throughput is still stored; the card shows the totals")
|
||||
assert "temp.max_c" in stored, "the sensor list is not on the card but stays in history"
|
||||
assert "host.processes" in stored
|
||||
# swap is not a card metric either; it is absent here only because this capture has no
|
||||
# swap device (swap_total_bytes == 0), not because storing it stopped
|
||||
assert result.payload["memory"]["swap_used_pct"] is None
|
||||
assert "mem.swap_used_pct" not in stored
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The same contract over HTTP
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_snapshot_endpoint_serves_the_card_paths(api_client: TestClient) -> None:
|
||||
"""The cards read `/api/system/snapshot`, so the contract is checked on that body too."""
|
||||
hosts = api_client.get("/api/system/snapshot").json()["hosts"]
|
||||
assert len(hosts) == 1
|
||||
assert_card_contract(hosts[0])
|
||||
|
||||
|
||||
def test_history_endpoint_still_answers_for_a_per_core_metric(api_client: TestClient) -> None:
|
||||
"""`cpu.usage_pct@core7` is off the page but still queryable — dropping the plot is
|
||||
not dropping the series."""
|
||||
store = api_client.app.state.ctx.store # type: ignore[attr-defined]
|
||||
store.add("host", "driveripper", "cpu.usage_pct@core7", 42.0, time.time())
|
||||
body = api_client.get("/api/system/history",
|
||||
params={"metrics": "cpu.usage_pct@core7", "hosts": "driveripper",
|
||||
"range": "1h"}).json()
|
||||
assert body["series"], "the per-core series must still be retrievable"
|
||||
assert [point["value"] for point in body["series"][0]["points"]] == [42.0]
|
||||
Reference in New Issue
Block a user