phase: 03_system_per_host_panels

**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**

- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
- **Next pending phase**: `04_llm_chart_readability`.
This commit is contained in:
2026-10-17 04:42:09 -04:00
parent 8b048599f3
commit 9757e3c947
43 changed files with 1686 additions and 464 deletions
@@ -0,0 +1,66 @@
# Verification notes — `/system` on every host shape
**Date:** 2026-10-17 (dev loop: tunnels 9121/9122/9123 + upsd 3494 all OK; dev-server
`/api/health` all six sources `ok` before and after verification)
## What was checked, on live data
Verified against `http://127.0.0.1:8090/system` with all three real agents answering, in a
real browser (headless Chromium/Playwright), DOM assertions plus screenshots.
| Host (shape) | Result | Screenshot |
| --- | --- | --- |
| `driveripper` (discrete Radeon AI PRO R9700) | GPU hero: `3 %` ok, `.sub` = "Radeon AI PRO R9700 · VRAM 30.5 GiB / 31.9 GiB · pool 33 %"; util/net/GPU charts all present inside the panel | `system-driveripper.png` (dark), `system-driveripper-light.png` |
| `deskwork` (Strix Halo APU, two memory pools) | GPU hero busy went 82 % ok → 85 % warn → 99 % fail across the session, so threshold colour visibly moves; `.sub` = "Radeon 8060S Graphics · VRAM 172.7 MiB / 512.0 MiB · pool 78 %" — small dedicated VRAM and high shared pool read as two facts, not a contradiction; MEM 91 % and TEMP ~83 °C amber | `system-deskwork.png` |
| `gateway` (no GPU, no CPU temp sensor) | CPU TEMP `n/a` with **no** severity class; GPU slot = `n/a` + "No GPU reported by this agent", uncoloured; **no** GPU chart and no hole where one would be | `system-gateway.png` (dark), `system-gateway-light.png` |
Full-page shapes: `system-shapes-dark.png`, `system-shapes-light.png`.
Failure path: stopped only the gateway tunnel (same commands `dev-tunnels.sh stop` runs,
scoped to the 9123 entry; remote agent stopped too). After 15 poll failures the panel kept
its last-good heroes (MEM 32 %, DISK 61 %, NET rates — nothing zeroed, nothing `n/a`), its
dot turned red, the header read `stale 2.1m`, and the card dimmed via `.stale`. The other
two panels stayed green ("one dead source must not blink another"). Store recorded exactly
one `unreachable` and, after `./scripts/dev-tunnels.sh start`, one `reachable` event.
Screenshot: `system-stale-gateway.png` (+ `system-stale-overview.png` full page). Restored;
`dev-tunnels.sh status` and `/api/health` back to all-OK.
Accessibility pass:
- hero values are real selectable text (`div.value > span.num/span.unit`, no SVG paths) —
asserted in-page for every hero on all three panels;
- colour is never the only signal: each hero prints the number, the NET slot is
deliberately colourless, staleness reads as text (`updated Ns` / `stale 2.1m`);
- charts — **defect found**: the SVGs were anonymous (`aria-label` absent on container and
svg). Fixed: `LineChart` gained an `ariaLabel` option that sets `role="img"` +
`aria-label` on the `<svg>`, and every `/system` chart now names its host and metrics
(e.g. "driveripper — utilisation over the selected window: CPU, memory and worst disk in
per cent"). Pinned by `tests/test_api.py::test_every_system_chart_names_itself_for_assistive_tech`.
The `/llm` and `/ups` charts can adopt the same option in phases 04/05.
Both themes checked via the header toggle (severity colours come from theme tokens; the
light-theme deskwork fail-red and warn-amber read clearly — see
`system-shapes-light.png`, `system-gateway-light.png`). Note for the next shooter: set
`prefers-color-scheme` or seed `localStorage['dash-theme']`; Playwright defaults to light,
which silently mislabels a "dark" screenshot.
`curl "localhost:8090/api/system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h"`
→ 354 points: dropping the per-core plot did not drop the data.
## Defects found and fixed during verification
1. **Unlabelled charts** (above) — fixed in `charts.js` + `page-system.js`, test-pinned,
re-screenshotted.
2. **`/system` Events panel was permanently empty** — the poller stores transition events
under the *dashboard* name (`system`) while the page queried `/api/events?source=host`,
so the feed said "No host transitions recorded yet" even with the deliberate gateway
outage in the store. Fixed to `source=system` (with a comment on the naming trap);
`tests/test_api.py::test_system_event_feed_queries_the_source_transitions_are_stored_under`
pins it; the re-shot full-page screenshots show the feed populated with the outage and
recovery. Pre-existing since the first commit, not introduced by phase 03.
## After the fixes
`bash .agents/validate.sh` → exit 0: 460 passed + 7 skipped, coverage 96.86 % (≥ 94),
ruff + pyright clean, `node --check` clean. `uv run pytest -m live --live` → 7 passed.
Console stayed error-free throughout.
@@ -0,0 +1,10 @@
**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**
- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
- **Next pending phase**: `04_llm_chart_readability`.
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 30%]
........................................................................ [ 45%]
...........................sssssss...................................... [ 60%]
........................................................................ [ 75%]
........................................................................ [ 90%]
............................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
472 passed, 7 skipped, 1 warning in 43.59s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1,21 @@
**Implemented**
- `tests/card_contract.py` (new): the single `/system` hero-card payload table — `CARD_METRICS`, `GPU_METRICS`, `SEVERITY_FREE_SECTIONS`, `HOST_SHAPES`, `dig()`, `assert_card_contract()`. Documents the two recorded choices: card temp comes from `cpu.temp_c` (identical to `temperatures.cpu_c`, and what `/api/summary` flattens), disk verdict is `disks.severity.worst_used_pct`; `network` deliberately has no severity.
- `tests/test_system_contract.py` (new, 13 tests): drives the real captures through `parse_host_snapshot()` for all three host shapes (discrete GPU / Strix Halo APU / no GPU), checks the paths over HTTP via `/api/system/snapshot`, pins `gpus == []` (not a missing key), `severity: null` for unknowns, absence of `network.severity`, and that dropped visuals (`cpu.usage_pct@coreN`, per-mount, per-interface, sensors, processes) are **still sampled** and `cpu.usage_pct@core7` still queryable from `/api/system/history`.
- `tests/test_live_fleet.py`: added `test_every_host_serves_the_hero_card_contract` — same table against the real agents.
- `src/dashboards/sources/host.py`: **comments only** (card-temp field choice; why `network` has no severity). No field renamed/removed; no page code touched.
- Verified mutation-sensitivity: renaming `worst_mount` or adding `network.severity` fails 10 contract tests.
**Tests / lint / coverage**
- `uv run pytest tests/test_system_contract.py` → 13 passed
- `bash .agents/validate.sh` (pytest `--cov --cov-fail-under=94`, ruff, pyright, node --check, hygiene) → exit 0: **452 passed, 7 skipped**, coverage **96.86%** (`sources/host.py` 99%), ruff clean, pyright 0 errors
- `uv run pytest -m live --live` (fleet + engines + upsd reachable) → **7 passed** (was 6)
- `uv pip list --outdated` → empty; dev server restarted, `/api/health` ok; live `/api/system/snapshot` passes the contract for deskwork/driveripper/gateway
**Docs**: AGENTS.md (test count 439→452 + new testing-policy bullet on the card-contract table) and README.md (Testing section notes `tests/card_contract.py`).
**Decisions / deviations**
- Contract table lives in `tests/card_contract.py` so hermetic and live tests share one list instead of duplicating paths.
- Task text's `disks.worst.severity` is realised as phase 01's sibling form `disks.severity.worst_used_pct` (no nested `disks.worst` object exists); recorded in the test docstring, no payload change.
- Phase-file curl example for per-core history uses `?host=&metric=`, but the API takes `hosts=`/`metrics=`; the test uses the real names (phase file left untouched).
**Next pending task**: `03_system_per_host_panels/02_hero_cards_without_per_core.md`.
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 31%]
........................................................................ [ 47%]
.......sssssss.......................................................... [ 62%]
........................................................................ [ 78%]
........................................................................ [ 94%]
........................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115
src/dashboards/config.py 110 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2104 66 97%
Required test coverage of 94% reached. Total coverage: 96.86%
452 passed, 7 skipped, 1 warning in 41.93s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1,21 @@
All green. Task complete.
**Report**
**Implemented**
- `page-system.js`: `hostCard()` rewritten — one `.heroes` row of six slots in fixed order (CPU, MEM, TEMP, DISK, GPU, NET), each with a `data-sev` hook naming its payload verdict; per-core block (`data-k="cores"`, `renderCores`) deleted; Load/Clock/Disk-read/write/busy/Up moved into the collapsed "More detail" `<details>` (with GPU power/temp table, filesystems, processes).
- `renderHostCard()` now drives colour via `Dash.hero()` from `cpu.severity.usage_pct/temp_c`, `memory.severity.used_pct`, `disks.severity.worst_used_pct`, `gpus[].severity.busy_pct`; zero `severityFor(` calls left. MEM shows `%` big + `113.5 / 125 GiB` sub; DISK sub is `worst_mount`; GPU = one hero per card (model, VRAM used/total, APU `pool %` in `.sub`) or uncoloured "No GPU reported by this agent"; NET is one slot, two lines via `Dash.rate()`, `n/a` (never `0 B/s`) when null, colourless by design.
- `dash.css`: added `.heroes` (auto-fit → 3×2/2×3 wrap), `.hero-slot`, `.hero .hero-lines` — no new colour vocabulary, tokens only.
- `app.js`: fallback docstring updated (phases 04–05 remain).
- Tests: 2 new byte-pinned tests in `tests/test_api.py` (six heroes in order, no per-core, no JS thresholds, NET has no verdict, detail keys survive); `test_severity_for_is_kept_as_the_documented_fallback` narrowed to `/llm` + `/ups` (same pattern phase 02 applied to `/`).
**Verification**
- `bash .agents/validate.sh` → exit 0: `454 passed, 7 skipped`, coverage 96.86% (floor 94), ruff clean, pyright 0 errors, `node --check` clean.
- `uv run pytest -m live --live` → 7 passed (real fleet, incl. hero-card contract on all three agents).
- Live browser (dev server restarted, port 8090): `data/logs/system-heroes-live.png`, `-warn-fail-deskwork.png` (91% amber, 95% red GPU), `-nogpu-gateway.png` (TEMP/GPU `n/a` uncoloured), `-driveripper-card.png`, `-more-detail.png`. Colours tracked live threshold crossings (deskwork GPU 95→fail, 94→warn between polls). `cpu.usage_pct@core7` still queryable (355 pts/1h).
**Docs**: AGENTS.md (`web/` fallback status, new UI-conventions paragraph for the six-hero card) and README.md (`web/` bullet) updated.
**Decisions**: `.cores` CSS + `renderCores` in `charts.js` kept (phase sanctions charts.js-only; task 03 confirms); fleet sub-line "cores" kept (core count, not per-core visual); hero-row test scopes the per-core grep to the card functions for the same reason.
**Next pending task**: `03_system_per_host_panels/03_per_host_charts_replace_shared.md`.
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 31%]
........................................................................ [ 46%]
.........sssssss........................................................ [ 62%]
........................................................................ [ 78%]
........................................................................ [ 93%]
............................. [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115
src/dashboards/config.py 110 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2104 66 97%
Required test coverage of 94% reached. Total coverage: 96.86%
454 passed, 7 skipped, 1 warning in 42.22s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1,17 @@
**Task complete** — `03_per_host_charts_replace_shared.md` (phase 03). Working tree only, no git operations.
**Deleted from `/system`**: the fleet totals card, the six fleet-wide charts (`#chart-cpu|mem|disk|net|gpu|temp` + legends), the Series-detail picker (`#detail-host|metric|chart|note`), `<select>`s, `renderCores()` in `charts.js`, and the `select`/`label.field` CSS they alone used. `main` now holds exactly two sections: host panels + events.
**Each host panel** stacks its own 120px charts under the hero row — util (CPU/mem/worst-disk, one 0-100 axis), net (↓in/↑out), GPU busy per card — GPU slot created and destroyed with `host.gpus`, legends naming metrics (or the GPU model) never hosts, torn down by `pruneHosts` when a host stops reporting.
**One request per host per refresh**: `/api/system/history?hosts=<host>&metrics=<that panel's series>&range=…`; measured live at 5 XHRs per poll for 3 hosts (1 snapshot + 3 history + events) regardless of chart count. `/api/metrics` discovery no longer used by the page.
**Tests** (`tests/test_api.py`, +3): `test_system_page_keeps_no_fleet_chart_or_detail_picker` (deleted ids/labels/`<select>` gone, 2 sections), `test_page_system_script_no_longer_references_the_deleted_ui`, `test_every_system_chart_is_drawn_inside_its_own_host_panel` (all 3 `LineChart` constructions take a card/slot `[data-chart]`, scoped host query, GPU metric built from `gpus[].index`, dynamic GPU add+remove, `pruneHosts` drops card and chart state), `test_one_request_per_host_answers_every_per_host_chart` (integration: the exact per-host query returns all 6 series, unknown `gpu.busy_pct@1` returns empty not 404), `test_render_cores_is_gone_and_the_chart_primitives_remain`.
**Live browser verification** (`http://127.0.0.1:8090`, 3 real hosts): no console/page/request errors, no NaN/undefined, 8 charts all inside panels, legends never list two hosts, each range chip re-fires one request per host (verified 5m→30d), and a host dropped from `/api/system/snapshot` lost its panel *and* its charts. Screenshots: `data/logs/per-host-charts-system-dark.png`, `-light.png`.
**Two defects fixed in passing**: `.chart .grid text` had no `fill`, so every y-axis label rendered black (invisible on the dark theme) — now `var(--text-faint)`, confirmed in both themes; and `drawHostChart()` now shades a chart only when one series has points, because three 0-100 area fills stacked into one block and hid the CPU line behind memory.
**Docs**: README `/system` bullet and AGENTS.md "Charts belong to the host they chart" rewritten, including correcting the stale claim that `renderCores` survives as unused library vocabulary.
**Gate**: `bash .agents/validate.sh` exit 0 — 458 passed, 7 skipped, coverage 96.86% (floor 94), ruff clean, pyright 0 errors, `node --check` clean. Phase/task files untouched.
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 30%]
........................................................................ [ 46%]
.............sssssss.................................................... [ 61%]
........................................................................ [ 77%]
........................................................................ [ 92%]
................................. [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115
src/dashboards/config.py 110 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2104 66 97%
Required test coverage of 94% reached. Total coverage: 96.86%
458 passed, 7 skipped, 1 warning in 43.12s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1,15 @@
**Implemented (task 04 — verify every host shape on live `/system`)**
- Browser-verified all three host shapes live: driveripper (R9700, VRAM 30.5/31.9 GiB + pool line), deskwork (APU — VRAM vs `pool 78%` both legible; GPU visibly crossed ok→warn→fail), gateway (GPU/temp `n/a` uncoloured, no GPU chart).
- Failure path proven: stopped only the gateway tunnel → panel kept last-good heroes (nothing zeroed), dot red, `stale 2.1m` text, dimmed; other hosts unaffected; exactly one unreachable + one reachable event; tunnel restored, fleet all-ok.
- **Fixed 2 defects found**: (1) anonymous chart SVGs — `LineChart` gained an `ariaLabel` option (`role="img"` + `aria-label`), all three `/system` slots name host+metrics; (2) `/system` Events feed queried `source=host` while the poller stores transitions under `system` — feed was empty since the first commit; now populated.
- New pins: `test_every_system_chart_names_itself_for_assistive_tech`, `test_system_event_feed_queries_the_source_transitions_are_stored_under` (tests/test_api.py).
- `NOTES-verify.md` written in the phase directory; both themes checked via header toggle; per-core history still queryable (354 points for `cpu.usage_pct@core7`).
- Screenshots opened/read: `system-driveripper.png`, `system-deskwork.png`, `system-gateway.png`, `system-stale-gateway.png`, `system-shapes-dark/light.png`, `system-driveripper-light.png`, `system-gateway-light.png` (all in `data/logs/`).
**Tests / lint**: `bash .agents/validate.sh` → exit 0 — **460 passed, 7 skipped**, coverage **96.86%** (≥94), ruff clean, pyright 0 errors, `node --check` clean · `uv run pytest -m live --live` (with `.env.dev`) → **7 passed** · `uv pip list --outdated` empty.
**Docs**: AGENTS.md (poller event-source trap; chart `ariaLabel` convention), README.md (chart aria naming in the `/system` paragraph).
**Notable**: Playwright defaults to light `prefers-color-scheme` — screenshots now seed `localStorage['dash-theme']`; noted in NOTES-verify.md for later phases. `/llm`/`/ups` charts can adopt the new `ariaLabel` option in phases 04/05 (not touched here).
**Next pending task**: `04_llm_chart_readability` (phase 03 fully complete after this task).
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 30%]
........................................................................ [ 46%]
...............sssssss.................................................. [ 61%]
........................................................................ [ 77%]
........................................................................ [ 92%]
................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 140 5 96% 92-93, 97-98, 115
src/dashboards/config.py 110 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2104 66 97%
Required test coverage of 94% reached. Total coverage: 96.86%
460 passed, 7 skipped, 1 warning in 43.42s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
+66 -10
View File
@@ -61,9 +61,13 @@ page still renders.
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app; 5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
no outbound access — a CDN reference is a request that never comes back. no outbound access — a CDN reference is a request that never comes back.
6. **Never leak a credential.** Target URLs may carry a userinfo token; `/api/config` 6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets`
exposes them only through `Target.display_url`, which strips userinfo. No token, moves it into an `Authorization` header, so `Target.url` ends up clean). Nothing that leaves
password, or personal hostname goes into tracked files, fixtures, or the browser. the process may carry one: `/api/config` serves `Target.display_url` and
`config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message
is redacted too — those strings go into the health record *and* the `event` table, which
`/api/events` serves to the browser and `page-system.js` prints. No token, password, or
personal hostname goes into tracked files, fixtures, or the browser.
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in 7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT `sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with 2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
@@ -104,7 +108,12 @@ Responsibilities, one line each:
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real - `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
env var always wins. Defaults are safe except the sources, which default to unset. env var always wins. Defaults are safe except the sources, which default to unset.
- `poller.py` — job loop, health records, stale/last-good semantics, transition events. - `poller.py` — job loop, health records, stale/last-good semantics, transition events.
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping. Events are stored under the **dashboard** name (`system`/`llm`/`ups`), not the sample
source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
(found on `/system` during phase 03 browser verification).
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping. Failure
messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one
that was dialled, because they become health records and event rows.
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card - `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
(`gpu.busy_pct@<index>`). (`gpu.busy_pct@<index>`).
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`, - `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
@@ -158,8 +167,9 @@ Responsibilities, one line each:
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number `.hero` big number (plus an optional `.bar`) from that class — it never compares a number
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
values the API does not rate; converting its remaining call sites is phases 03–05's job values the API does not rate; converting its remaining call sites is phases 04–05's job
(`/` was converted in phase 02: the overview now colours only from `/api/summary` verdicts). (`/` was converted in phase 02, `/system` in phase 03 — those pages colour only from the
`severity` verdicts the API sends, and no `severityFor(` call survives in their scripts).
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`, Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device `gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
@@ -205,8 +215,8 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
./scripts/dev-server.sh status # per-source ok/DOWN ./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh logs ./scripts/dev-server.sh logs
uv run pytest # hermetic: 439 passed + 6 live-skipped, no network uv run pytest # hermetic: 472 passed + 7 live-skipped, no network
uv run pytest --cov # coverage gate (96.8%; floor 94%) uv run pytest --cov # coverage gate (96.9%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth ./scripts/capture-fixtures.sh # re-record ground truth
@@ -235,6 +245,14 @@ service here.
fixture test fails before a dashboard does. fixture test fails before a dashboard does.
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host - Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`. rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
- **Page payloads that a rewrite must not break are pinned as a table.**
`tests/card_contract.py` holds the dotted `/api/system/snapshot` paths the `/system` hero
cards read (`CARD_METRICS`, `GPU_METRICS`, `HOST_SHAPES`); `tests/test_system_contract.py`
asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and
`test_every_host_serves_the_hero_card_contract` asserts the same table against the live
agents — so a rename fails a test, and a card that shows `n/a` cannot be "fixed" by
inventing a field. `network` carries no `severity` on purpose (throughput has no good/bad),
and the contract pins its absence.
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real - `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
unset, so partial fleet availability is normal. **Live tests must not write** — a battery unset, so partial fleet availability is normal. **Live tests must not write** — a battery
@@ -252,8 +270,11 @@ service here.
Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
dots in the header, window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
accepts 5m…90d). A panel that cannot get data says `LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
silence, and the legend is a sibling that labels nothing) — `/system` does this for all three
slots, pinned by `tests/test_api.py`. Window chips (5m…30d on `/system` and `/llm`, 1h…90d on
`/ups`; the API accepts 5m…90d). A panel that cannot get data says
why in the panel — the operator should never have to open devtools to learn that a source is why in the panel — the operator should never have to open devtools to learn that a source is
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`. unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
@@ -281,6 +302,41 @@ muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare. rule reaches for a token that `:root` does not declare.
The `/system` host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the
`.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script).
Colour comes from the payload's `severity` dicts and nowhere else; a missing number or
verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**. NET is
one slot with two lines (`.hero-lines`, `↓` in / `↑` out) and is deliberately colourless:
throughput has no good/bad, so `network` carries no severity. The GPU slot holds one hero
per card (model, VRAM used/total and the honest APU `pool` view in the `.sub`), and
everything not glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU
power/temp, filesystems, processes — folds into a collapsed "More detail" `<details>`: a
simpler surface, not a less complete page.
A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
those (a range when the hosts disagree) instead of the poll interval, which only coincides with
them at the defaults.
**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
panel and the host/metric detail picker are deleted, along with the `#chart-*` / `#detail-*` ids and
the `select` / `label.field` styling only those pickers used. Each panel stacks its own
`.host-charts` — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where
the GPU slot is created and destroyed with `host.gpus`, so gateway shows neither a phantom card
nor a hole. **One `/api/system/history` request per host per refresh**
(`hosts=<that host>&metrics=<that panel's series>&range=…`, pinned by
`test_every_system_chart_is_drawn_inside_its_own_host_panel`): unknown metrics come back as
empty series on purpose, so a panel asks for every series it could draw and asks once. A legend
here names metrics (or one GPU's model), never hosts — the panel heading already says who this
is, and a legend that grew a second host is the bug this replaced. A host that stops reporting
loses its panel *and* its charts (`pruneHosts`), never a chart of nothing. Per-core visuals are
gone from `/system` and `renderCores` is **deleted** from `charts.js` with its last caller.
`drawHostChart()` shades a chart only when one series has points — two or three filled areas
stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy
memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryable from
`/api/system/history` — removing the plot is not removing the data.
## Known gaps / next work ## Known gaps / next work
1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see 1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see
+30 -5
View File
@@ -54,8 +54,10 @@ that already listens there (upsd) — that is how the UPS page reads a real batt
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`, `agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one `/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
JSON document), `GET /info` (slow-changing identity: OS, kernel, CPU model, GPU list), and JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS,
`GET /health` (the one route that never needs the token) as JSON. kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and
`GET /health` (the one route that never needs the token) as JSON, with a plain HTML status
page at `GET /` so a browser pointed at an agent learns what it is looking at.
```bash ```bash
python3 host_agent.py --once # print one snapshot, exit python3 host_agent.py --once # print one snapshot, exit
@@ -89,7 +91,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
| ------------------------------------- | -------------------------------------------------------------- | | ------------------------------------- | -------------------------------------------------------------- |
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) | | `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears | | `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) | | `RATE_WINDOW_SECONDS` | the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and `/system`'s footnote prints that, not this default |
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) | | `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) | | `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others | | `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
@@ -108,11 +110,16 @@ A source that is not configured answers `404 {"configured": false}` **without di
and the page says "not configured" instead of showing a red panel for something you never and the page says "not configured" instead of showing a red panel for something you never
installed. installed.
Nothing that leaves the app carries a credential. A `HOST_TARGETS` userinfo token is parsed
into an `Authorization` header, and every URL the browser can see — `/api/config`, a source's
error string, the event feed — goes through `config.strip_userinfo` first, because an
unreachable host writes its failure message into the event table and `/api/events` serves it.
## Architecture ## Architecture
``` ```
agent/host_agent.py (each host) gufo / vLLM upsd (NUT) agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR │ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
▼ ▼ ▼ ▼ ▼ ▼
src/dashboards/collectors.py ────────────────────────────────────────────┐ src/dashboards/collectors.py ────────────────────────────────────────────┐
│ parse + Δcounter/Δt (rates.py) │ │ parse + Δcounter/Δt (rates.py) │
@@ -155,7 +162,19 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
A big colour-coded number is `Dash.hero(el, {...})` on the A big colour-coded number is `Dash.hero(el, {...})` on the
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via `.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value `Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
with no verdict renders `n/a` with no colour at all. with no verdict renders `n/a` with no colour at all. Each `/system` host is one panel led
by exactly six of them — CPU %, MEM %, TEMP, DISK % (mount underneath), GPU busy (VRAM and
the APU pool in the footnote), and NET on two lines (↓ in / ↑ out, never coloured: a byte
rate has no good/bad). Under the numbers, **each panel charts only its own host**: utilisation
(CPU, memory, worst disk), network (↓ in / ↑ out), and GPU busy for a host that reports
cards — three 120px SVG charts, each with its own legend and its own `aria-label` naming the
host and the plotted metrics (`LineChart`'s `ariaLabel` option), never a second host in sight. The
fleet-wide charts, the nine-figure fleet-totals panel and the host/metric detail picker are gone, and
so are the per-core charts, while `cpu.usage_pct@coreN` stays in SQLite. Everything else —
load, clock, disk I/O, uptime, GPU power/temp, filesystems, processes — folds into a collapsed
"More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request
(`hosts=<that host>&metrics=…&range=…`), and a host that stops reporting takes its panel and
its charts off the page.
## Testing ## Testing
@@ -176,6 +195,12 @@ also skips when its source is unset, they drive the production code paths (`coll
`NutMonitor.collect`) against whatever hardware is reachable, and they never write — a `NutMonitor.collect`) against whatever hardware is reachable, and they never write — a
battery test on live hardware is not a unit test. battery test on live hardware is not a unit test.
One more file is worth knowing about: `tests/card_contract.py` is the table of
`/api/system/snapshot` paths the system page's per-host cards are allowed to read. It is
checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and
again against the live fleet, so a card can never be "fixed" by inventing a number the agent
does not report.
## Battery-test buttons ## Battery-test buttons
Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`) Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`)
+10 -4
View File
@@ -5,7 +5,8 @@ hot path) plus SQLite for anything older than a few minutes. Two rules from llm_
are worth restating here because they are easy to break by accident: are worth restating here because they are easy to break by accident:
* `/api/config` never leaks a token or password — target URLs are passed through * `/api/config` never leaks a token or password — target URLs are passed through
`Target.display_url`, which strips userinfo. `Target.display_url`, which strips userinfo, and so does every URL that reaches an error
message, because those travel into `/api/health` and the event feed too.
* an unconfigured source answers 404 `{"configured": false}` without dialling anything, so * an unconfigured source answers 404 `{"configured": false}` without dialling anything, so
the page can tell "not deployed here" apart from "deployed and broken". the page can tell "not deployed here" apart from "deployed and broken".
""" """
@@ -20,7 +21,7 @@ from fastapi import APIRouter, Body, Depends, HTTPException, Query, Request
from . import alerts as alert_rules from . import alerts as alert_rules
from .collectors import NutMonitor from .collectors import NutMonitor
from .config import Settings from .config import Settings, strip_userinfo
from .poller import Hub from .poller import Hub
from .severity import classify from .severity import classify
from .sources import nut from .sources import nut
@@ -133,9 +134,14 @@ def common_router() -> APIRouter:
"target_problems": list(settings.host_target_problems), "target_problems": list(settings.host_target_problems),
}, },
"llm": { "llm": {
"gufo": {"enabled": bool(settings.gufo_url), "base_url": settings.gufo_url, # `strip_userinfo` on the engine URLs too: `GUFO_API_KEY` is the documented
# way to authenticate, but a URL that carries credentials must never be echoed
# to a browser whatever the config says (invariant 6).
"gufo": {"enabled": bool(settings.gufo_url),
"base_url": strip_userinfo(settings.gufo_url),
"auth": bool(settings.gufo_api_key)}, "auth": bool(settings.gufo_api_key)},
"vllm": {"enabled": bool(settings.vllm_url), "base_url": settings.vllm_url, "vllm": {"enabled": bool(settings.vllm_url),
"base_url": strip_userinfo(settings.vllm_url),
"auth": bool(settings.vllm_api_key)}, "auth": bool(settings.vllm_api_key)},
"interval_seconds": settings.llm_interval, "interval_seconds": settings.llm_interval,
"poll_ms": settings.poll_ms_llm, "poll_ms": settings.poll_ms_llm,
+12 -7
View File
@@ -13,7 +13,7 @@ from typing import Any
import httpx import httpx
from .config import Target from .config import Target, strip_userinfo
from .poller import Job, Result from .poller import Job, Result
from .rates import RateEstimator from .rates import RateEstimator
from .sources import llm as llm_source from .sources import llm as llm_source
@@ -37,17 +37,21 @@ class UpstreamError(Exception):
async def fetch_host_snapshot(client: httpx.AsyncClient, target: Target, async def fetch_host_snapshot(client: httpx.AsyncClient, target: Target,
host_collector: HostCollector) -> Result: host_collector: HostCollector) -> Result:
url = f"{target.url}/snapshot" url = f"{target.url}/snapshot"
# Every message below uses the redacted URL on purpose: an UpstreamError string is kept
# in the job's health record and written to the event table, both of which the browser
# reads, and `target.url` may carry the agent's token as userinfo.
shown = f"{target.display_url}/snapshot"
try: try:
response = await client.get(url, headers=target.headers) response = await client.get(url, headers=target.headers)
except httpx.TimeoutException as exc: except httpx.TimeoutException as exc:
raise UpstreamError(f"{target.name}: {url} timed out", status=504) from exc raise UpstreamError(f"{target.name}: {shown} timed out", status=504) from exc
except httpx.HTTPError as exc: except httpx.HTTPError as exc:
raise UpstreamError(f"{target.name}: {url} unreachable ({type(exc).__name__})") from exc raise UpstreamError(f"{target.name}: {shown} unreachable ({type(exc).__name__})") from exc
if response.status_code == 401: if response.status_code == 401:
raise UpstreamError(f"{target.name}: agent rejected the token (401) — check the " raise UpstreamError(f"{target.name}: agent rejected the token (401) — check the "
f"userinfo token in HOST_TARGETS") f"userinfo token in HOST_TARGETS")
if response.status_code >= 400: if response.status_code >= 400:
raise UpstreamError(f"{target.name}: {url} -> HTTP {response.status_code}") raise UpstreamError(f"{target.name}: {shown} -> HTTP {response.status_code}")
try: try:
body = response.json() body = response.json()
except ValueError as exc: except ValueError as exc:
@@ -71,14 +75,15 @@ async def fetch_engine(client: httpx.AsyncClient, engine: str, base_url: str,
with_slots: bool = False) -> Result: with_slots: bool = False) -> Result:
headers = {"Authorization": f"Bearer {api_key}"} if api_key else {} headers = {"Authorization": f"Bearer {api_key}"} if api_key else {}
url = f"{base_url}/metrics" url = f"{base_url}/metrics"
shown = strip_userinfo(url) # same rule as the host fetch above: no credential in a message
try: try:
response = await client.get(url, headers=headers) response = await client.get(url, headers=headers)
except httpx.TimeoutException as exc: except httpx.TimeoutException as exc:
raise UpstreamError(f"{engine}: {url} timed out", status=504) from exc raise UpstreamError(f"{engine}: {shown} timed out", status=504) from exc
except httpx.HTTPError as exc: except httpx.HTTPError as exc:
raise UpstreamError(f"{engine}: {url} unreachable ({type(exc).__name__})") from exc raise UpstreamError(f"{engine}: {shown} unreachable ({type(exc).__name__})") from exc
if response.status_code >= 400: if response.status_code >= 400:
raise UpstreamError(f"{engine}: {url} -> HTTP {response.status_code} " raise UpstreamError(f"{engine}: {shown} -> HTTP {response.status_code} "
f"{response.reason_phrase}") f"{response.reason_phrase}")
text = response.text text = response.text
+19 -1
View File
@@ -75,6 +75,24 @@ def _base_url(name: str) -> str:
# variable; the token never leaves the backend, it is stripped from every /api payload. # variable; the token never leaves the backend, it is stripped from every /api payload.
_TARGET_SPLIT = re.compile(r"[,\s]+") _TARGET_SPLIT = re.compile(r"[,\s]+")
_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_.-]*$") _NAME_RE = re.compile(r"^[a-z0-9][a-z0-9_.-]*$")
_USERINFO_RE = re.compile(r"^(https?://)[^/@]+@", re.IGNORECASE)
def strip_userinfo(url: str) -> str:
"""`url` with any userinfo (`http://TOKEN@host` -> `http://host`) removed.
The only form a URL may take on its way out of the process. Two things make this needed
rather than belt-and-braces: a URL does not travel only in the fetch — an `UpstreamError`
message repeats the URL it was dialling, and that string lands in the job's health record
(`/api/health`) and in the `event` table (`/api/events`), both of which a browser reads —
and the engine URLs (`GUFO_BASE_URL`, `VLLM_BASE_URL`) are used verbatim, with no
`parse_targets`-style move of userinfo into a header, so a credential written into one of
them would otherwise be served by `/api/config` and echoed by every failed poll.
`HOST_TARGETS` strips its token into a header, so this is the backstop for a `Target` that
still carries one. One rule, stated once: AGENTS.md invariant 6, no credential in anything
a browser can read.
"""
return _USERINFO_RE.sub(r"\1", url)
@dataclass(frozen=True) @dataclass(frozen=True)
@@ -88,7 +106,7 @@ class Target:
@property @property
def display_url(self) -> str: def display_url(self) -> str:
"""URL with any userinfo removed — safe to show in the browser.""" """URL with any userinfo removed — safe to show in the browser."""
return re.sub(r"^(https?://)[^/@]+@", r"\1", self.url) return strip_userinfo(self.url)
def parse_targets(raw: str) -> tuple[list[Target], list[str]]: def parse_targets(raw: str) -> tuple[list[Target], list[str]]:
+8
View File
@@ -328,6 +328,10 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl
"forks_per_s": forks_rate, "forks_per_s": forks_rate,
"procs_running": cpu_raw.get("procs_running"), "procs_running": cpu_raw.get("procs_running"),
"procs_blocked": cpu_raw.get("procs_blocked"), "procs_blocked": cpu_raw.get("procs_blocked"),
# The card metric for "cpu temp" is this one; `temperatures.cpu_c` below is the
# same measurement (the agent's own `cpu_temperature_celsius`) and `/api/summary`
# flattens this field, so the card and the overview cannot disagree. Pinned by
# tests/test_system_contract.py.
"temp_c": cpu_temp, "temp_c": cpu_temp,
# Additive: the pages colour from this verdict instead of comparing the number # Additive: the pages colour from this verdict instead of comparing the number
# to a threshold of their own. Keys mirror the rated fields in this section. # to a threshold of their own. Keys mirror the rated fields in this section.
@@ -364,6 +368,10 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl
"rx_bytes_per_s": rx_total if any_net else None, "rx_bytes_per_s": rx_total if any_net else None,
"tx_bytes_per_s": tx_total if any_net else None, "tx_bytes_per_s": tx_total if any_net else None,
"interfaces": net_rows, "interfaces": net_rows,
# Deliberately no `severity` section here, unlike cpu/memory/disks/gpus: a byte
# rate has no good/bad (900 MB/s is idle on 10 GbE, 2 KB/s can be an incident),
# so `severity.THRESHOLDS` has no `net.*` row and a colour here would be a
# verdict nobody measured. tests/test_system_contract.py pins its absence.
}, },
"gpus": gpus, "gpus": gpus,
"temperatures": {"cpu_c": cpu_temp, "max_c": max_temp, "nvme_max_c": nvme_max, "temperatures": {"cpu_c": cpu_temp, "max_c": max_temp, "nvme_max_c": nvme_max,
+9 -73
View File
@@ -37,83 +37,19 @@
</div> </div>
</div> </div>
<section class="card flat" aria-label="Fleet totals"> <!-- No fleet-wide chart and no host/metric picker survives here (phase 03 task 03): every
<div class="body"> chart is drawn inside one host panel below, and the question "which machine stands
<div class="metrics"> out?" belongs to the overview's alerts and fleet table. -->
<div class="metric"><div class="label">Hosts answering</div> <section>
<div class="value"><span id="f-hosts">n/a</span><span class="unit" id="f-hosts-of"></span></div></div>
<div class="metric"><div class="label">CPU average</div>
<div class="value"><span id="f-cpu">n/a</span><span class="unit">%</span></div></div>
<div class="metric"><div class="label">Load 1 (sum)</div>
<div class="value" id="f-load">n/a</div></div>
<div class="metric"><div class="label">Memory in use</div>
<div class="value"><span id="f-mem">n/a</span><span class="unit" id="f-mem-of"></span></div></div>
<div class="metric"><div class="label">Worst filesystem</div>
<div class="value"><span id="f-disk">n/a</span><span class="unit">%</span></div></div>
<div class="metric"><div class="label">Network in</div>
<div class="value"><span id="f-rx">n/a</span><span class="unit" id="f-rx-unit"></span></div></div>
<div class="metric"><div class="label">Network out</div>
<div class="value"><span id="f-tx">n/a</span><span class="unit" id="f-tx-unit"></span></div></div>
<div class="metric"><div class="label">GPU busy (avg)</div>
<div class="value"><span id="f-gpu">n/a</span><span class="unit">%</span></div></div>
<div class="metric"><div class="label">Power</div>
<div class="value sm" title="No whole-host power source is wired up yet; no script writes this, so it stays n/a rather than guessing.">n/a</div></div>
</div>
</div>
</section>
<section class="grid cols-2" style="margin-top:.85rem" aria-label="Fleet charts">
<article class="card">
<header><h3>CPU utilisation</h3><span class="spacer"></span><span class="faint">% · per host</span></header>
<div class="body"><div class="chart" id="chart-cpu"></div><div class="legend" id="legend-cpu"></div></div>
</article>
<article class="card">
<header><h3>Memory used</h3><span class="spacer"></span><span class="faint">% of MemAvailable basis</span></header>
<div class="body"><div class="chart" id="chart-mem"></div><div class="legend" id="legend-mem"></div></div>
</article>
<article class="card">
<header><h3>Disk I/O throughput</h3><span class="spacer"></span>
<span class="faint">selected host · all physical devices summed</span></header>
<div class="body"><div class="chart" id="chart-disk"></div><div class="legend" id="legend-disk"></div></div>
</article>
<article class="card">
<header><h3>Network throughput</h3><span class="spacer"></span><span class="faint">selected host · physical interfaces</span></header>
<div class="body"><div class="chart" id="chart-net"></div><div class="legend" id="legend-net"></div></div>
</article>
<article class="card">
<header><h3>GPU utilisation</h3><span class="spacer"></span><span class="faint">one series per host GPU</span></header>
<div class="body"><div class="chart" id="chart-gpu"></div><div class="legend" id="legend-gpu"></div></div>
</article>
<article class="card">
<header><h3>Temperature</h3><span class="spacer"></span><span class="faint">CPU package · °C</span></header>
<div class="body"><div class="chart" id="chart-temp"></div><div class="legend" id="legend-temp"></div></div>
</article>
</section>
<section class="card" style="margin-top:.85rem" aria-label="Series detail">
<header>
<h3>Detail</h3>
<span class="spacer"></span>
<label class="field" for="detail-host">host</label>
<select id="detail-host"></select>
<label class="field" for="detail-metric">metric</label>
<select id="detail-metric"></select>
</header>
<div class="body">
<div class="chart" id="chart-detail" style="height:200px"></div>
<div class="legend" id="legend-detail"></div>
<div class="faint" id="detail-note" style="margin-top:.4rem;font-size:11.5px"></div>
</div>
</section>
<section style="margin-top:.85rem">
<div class="page-head"><h2 style="font-size:.95rem;margin:0">Per host</h2> <div class="page-head"><h2 style="font-size:.95rem;margin:0">Per host</h2>
<span class="sub">agent-reported values; rates measured over the window in each header</span></div> <span class="sub">agent-reported values; rates are measured over the Δ window in the
footer, and each panel's charts cover only that host</span></div>
<div class="grid auto" id="hosts"></div> <div class="grid auto" id="hosts"></div>
<div class="empty-state" id="hosts-empty" hidden> <div class="empty-state" id="hosts-empty" hidden>
<h3>No hosts reporting</h3> <h3>No hosts reporting</h3>
<p>Start the agent on a machine and add it to <code>HOST_TARGETS</code>:</p> <p>Start the agent on a machine and add it to <code>HOST_TARGETS</code>:</p>
<p class="mono">python3 agent/host_agent.py --host 0.0.0.0 --port 9120 --token …</p> <p class="mono">python3 agent/host_agent.py --host 0.0.0.0 --port 9120 \
--token-file /etc/dashboards-agent.token</p>
</div> </div>
</section> </section>
@@ -123,7 +59,7 @@
</section> </section>
<footer class="pagefoot"> <footer class="pagefoot">
<span>numbers are measured, not inferred: rates are Δcounter/Δt over the poll window</span> <span>numbers are measured, not inferred: rates are Δcounter/Δt over the window below</span>
<span class="spacer" style="margin-inline-start:auto"></span> <span class="spacer" style="margin-inline-start:auto"></span>
<span class="mono" id="foot-note"></span> <span class="mono" id="foot-note"></span>
</footer> </footer>
+3 -3
View File
@@ -110,9 +110,9 @@ const Dash = (() => {
* *
* Thresholds live in `src/dashboards/severity.py` and travel in the payload as * Thresholds live in `src/dashboards/severity.py` and travel in the payload as
* `severity`; `severityFrom()` below is how a page reads them. This function survives * `severity`; `severityFrom()` below is how a page reads them. This function survives
* only because `page-system.js`, `page-llm.js` and `page-ups.js` still call it with * only because `page-llm.js` and `page-ups.js` still call it with their own numbers
* their own numbers (phases 03–05 convert them page by page, so each page's change is * (phases 04–05 convert them page by page, so each page's change is reviewable on its
* reviewable on its own — `/` was already converted in phase 02). Do not add a new call * own — `/` was converted in phase 02, `/system` in phase 03). Do not add a new call
* site: a threshold written in JS is a threshold that can disagree with the other pages. * site: a threshold written in JS is a threshold that can disagree with the other pages.
*/ */
function severityFor(value, warnAt, failAt) { function severityFor(value, warnAt, failAt) {
+9 -13
View File
@@ -43,7 +43,7 @@ const DEFAULT_FORMAT = (v) => (Math.abs(v) >= 100 ? v.toFixed(0) : v.toFixed(1))
class LineChart { class LineChart {
/** /**
* container: element to fill (its width is used; height comes from opts.height) * container: element to fill (its width is used; height comes from opts.height)
* opts: {height, yFormat, xSpan, yMin, yMax, area, legend, showAxis, min, unit} * opts: {height, yFormat, xSpan, yMin, yMax, area, legend, showAxis, min, unit, ariaLabel}
*/ */
constructor(container, opts) { constructor(container, opts) {
this.el = typeof container === 'string' ? document.querySelector(container) : container; this.el = typeof container === 'string' ? document.querySelector(container) : container;
@@ -69,6 +69,13 @@ class LineChart {
this.svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg'); this.svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
this.svg.setAttribute('width', '100%'); this.svg.setAttribute('width', '100%');
this.svg.setAttribute('height', this.opts.height); this.svg.setAttribute('height', this.opts.height);
// A chart is an image to assistive tech, which means silence unless the page names it —
// the legend below is a sibling, not a label. `ariaLabel` is optional so pages adopt it
// one at a time (`/system` names every chart it draws; see page-system.js).
if (this.opts.ariaLabel) {
this.svg.setAttribute('role', 'img');
this.svg.setAttribute('aria-label', String(this.opts.ariaLabel));
}
this.el.appendChild(this.svg); this.el.appendChild(this.svg);
this.empty = document.createElement('div'); this.empty = document.createElement('div');
this.empty.className = 'empty'; this.empty.className = 'empty';
@@ -337,17 +344,6 @@ function renderLegend(container, series, format, showPeak) {
}).join(''); }).join('');
} }
/** Column heat bars for per-core usage: 48 cores read as a pattern, not a table. */
function renderCores(container, values, format) {
const f = format || ((v) => (v == null ? 'n/a' : v.toFixed(0) + '%'));
container.innerHTML = (values || []).map((v, i) => {
const pct = v == null ? 0 : Math.max(0, Math.min(100, v));
const hue = pct > 85 ? 'var(--fail)' : pct > 60 ? 'var(--warn)' : 'var(--accent)';
return `<span class="core" title="core ${i}: ${escapeHtml(f(v))}">` +
`<i style="height:${pct.toFixed(1)}%;background:${hue}"></i></span>`;
}).join('');
}
/** Small non-interactive sparkline for cards where a full chart would shout. */ /** Small non-interactive sparkline for cards where a full chart would shout. */
function sparkline(container, points, opts) { function sparkline(container, points, opts) {
const o = Object.assign({ height: 34, color: cssVar('--series-1'), min: null, max: null }, opts || {}); const o = Object.assign({ height: 34, color: cssVar('--series-1'), min: null, max: null }, opts || {});
@@ -374,5 +370,5 @@ function sparkline(container, points, opts) {
container.innerHTML = svg; container.innerHTML = svg;
} }
window.DashCharts = { LineChart, niceTicks, renderLegend, renderCores, sparkline, cssVar, window.DashCharts = { LineChart, niceTicks, renderLegend, sparkline, cssVar,
CHART_COLORS, DEFAULT_FORMAT, escapeHtml }; CHART_COLORS, DEFAULT_FORMAT, escapeHtml };
+46 -8
View File
@@ -282,6 +282,27 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
/* with no verdict on either the hero or the bar the fill stays the neutral accent: the /* with no verdict on either the hero or the bar the fill stays the neutral accent: the
bar shows *how much*, never a judgement nobody measured. */ bar shows *how much*, never a judgement nobody measured. */
/* The per-host hero row: six glanceable numbers — CPU, MEM, TEMP, DISK, GPU, NET — in one
line on a wide card, wrapping to 3×2 and then 2×3 as the card narrows. Tracks are a
`minmax` of the widest plain hero value (`100 %`), so a row never squints into ellipsis. */
.heroes {
display: grid; gap: .9rem 1rem;
grid-template-columns: repeat(auto-fit, minmax(min(118px, 100%), 1fr));
}
/* a slot that holds more than one hero (one per GPU card) without breaking the row */
.hero-slot { display: grid; gap: .7rem; min-width: 0; align-content: start; }
/* Network is one hero with two lines: in and out are one wire seen in two directions, and
two big numbers side by side read as a competition. It is deliberately colourless —
a byte rate has no good/bad, so no verdict ever paints here (severity.py has no net.*
row); `n/a` comes from `.na` and is as faint as any other unmeasured value. */
.hero .hero-lines {
display: grid; gap: .15rem;
font-family: var(--mono); font-size: 1.28rem; font-weight: 650; line-height: 1.25;
font-variant-numeric: tabular-nums; /* same fixed digit width as .hero .value */
color: var(--text);
}
.hero .hero-lines .dir { color: var(--text-muted); }
/* ---- bars, dots, chips -------------------------------------------------- */ /* ---- bars, dots, chips -------------------------------------------------- */
.bar { .bar {
position: relative; height: 7px; border-radius: 999px; background: var(--bg-sunken); position: relative; height: 7px; border-radius: 999px; background: var(--bg-sunken);
@@ -357,9 +378,15 @@ table.data td.fail { color: var(--fail); }
table.data.scroll { display: block; overflow: auto; max-height: 320px; } table.data.scroll { display: block; overflow: auto; max-height: 320px; }
/* ---- charts ------------------------------------------------------------- */ /* ---- charts ------------------------------------------------------------- */
/* The chart vocabulary every page draws with. `LineChart` puts the y tick labels in the
`grid` group and the x labels in `axis`, so both groups need the same text style — the
grid text used to inherit SVG's black fill, which meant the numbers beside a utilisation
chart were invisible on the dark theme. */
.chart { position: relative; width: 100%; } .chart { position: relative; width: 100%; }
.chart svg { display: block; width: 100%; height: 100%; overflow: visible; } .chart svg { display: block; width: 100%; height: 100%; overflow: visible; }
.chart .axis text { fill: var(--text-faint); font-family: var(--mono); font-size: 10px; } .chart .axis text, .chart .grid text {
fill: var(--text-faint); font-family: var(--mono); font-size: 10px;
}
.chart .axis line, .chart .grid line { stroke: var(--border); stroke-width: 1; } .chart .axis line, .chart .grid line { stroke: var(--border); stroke-width: 1; }
.chart .grid line { stroke-dasharray: 2 4; opacity: .75; } .chart .grid line { stroke-dasharray: 2 4; opacity: .75; }
.chart .zero line { stroke: var(--border-strong); } .chart .zero line { stroke: var(--border-strong); }
@@ -375,6 +402,20 @@ table.data.scroll { display: block; overflow: auto; max-height: 320px; }
.legend .swatch { width: 9px; height: 9px; border-radius: 2px; } .legend .swatch { width: 9px; height: 9px; border-radius: 2px; }
.legend .lv { font-family: var(--mono); color: var(--text); } .legend .lv { font-family: var(--mono); color: var(--text); }
/* The per-host chart stack inside a `/system` panel: supporting evidence under a number,
not the main event. One utilisation chart, one network chart, and a GPU chart only for a
host that reports cards (`page-system.js` adds and removes that slot with `host.gpus`),
each with its legend glued underneath. They are stacked rather than side by side because
a 200px-wide plot cannot show an hour of CPU without its axis labels colliding — and no
chart in here ever contains a second host. */
.host-charts { display: grid; gap: .7rem; }
.host-chart { display: grid; gap: .15rem; min-width: 0; }
/* `sm` is the slot's floor before `LineChart` writes its measured height inline, so an
empty panel is the height it is about to be rather than a collapsing card. */
.chart.sm { min-height: 120px; }
.host-chart .legend { font-size: 10.5px; gap: .2rem .6rem; }
.host-chart .legend .swatch { width: 7px; height: 7px; }
.tooltip { .tooltip {
position: absolute; z-index: 30; pointer-events: none; opacity: 0; position: absolute; z-index: 30; pointer-events: none; opacity: 0;
background: color-mix(in srgb, var(--bg-raised) 95%, transparent); background: color-mix(in srgb, var(--bg-raised) 95%, transparent);
@@ -386,11 +427,6 @@ table.data.scroll { display: block; overflow: auto; max-height: 320px; }
.tooltip .row { display: flex; align-items: center; gap: .35rem; } .tooltip .row { display: flex; align-items: center; gap: .35rem; }
.tooltip .row .val { margin-inline-start: auto; font-family: var(--mono); } .tooltip .row .val { margin-inline-start: auto; font-family: var(--mono); }
/* per-core usage heat bars */
.cores { display: grid; grid-template-columns: repeat(auto-fill, minmax(9px, 1fr)); gap: 2px; }
.cores .core { height: 22px; border-radius: 2px; background: var(--bg-sunken); position: relative; }
.cores .core > i { position: absolute; bottom: 0; left: 0; right: 0; background: var(--accent); border-radius: 2px; }
/* ---- lists -------------------------------------------------------------- */ /* ---- lists -------------------------------------------------------------- */
ul.events { list-style: none; margin: 0; padding: 0; display: grid; gap: .1rem; } ul.events { list-style: none; margin: 0; padding: 0; display: grid; gap: .1rem; }
ul.events li { ul.events li {
@@ -497,11 +533,13 @@ dialog footer { padding: .7rem .9rem; border-top: 1px solid var(--border); displ
.stale { opacity: .62; } .stale { opacity: .62; }
select, input[type="text"] { /* Only the UPS confirmation dialog takes text now — `/system`'s host/metric pickers were the
last `select` on the app and left with the shared charts, along with the `label.field` caption
that sat above them. */
input[type="text"] {
background: var(--bg-sunken); color: var(--text); border: 1px solid var(--border); background: var(--bg-sunken); color: var(--text); border: 1px solid var(--border);
border-radius: var(--radius-sm); padding: .3rem .45rem; font: 500 12.5px var(--sans); border-radius: var(--radius-sm); padding: .3rem .45rem; font: 500 12.5px var(--sans);
} }
label.field { font-size: 11px; text-transform: uppercase; letter-spacing: .05em; color: var(--text-faint); }
footer.pagefoot { footer.pagefoot {
margin-top: 1.6rem; padding-top: .8rem; border-top: 1px solid var(--border); margin-top: 1.6rem; padding-top: .8rem; border-top: 1px solid var(--border);
+322 -327
View File
@@ -2,71 +2,43 @@
* *
* Host cards are created once and patched in place (keyed by host name) — rebuilding the * Host cards are created once and patched in place (keyed by host name) — rebuilding the
* DOM every 10s kills hover, transitions, and any scroll position inside the tables. * DOM every 10s kills hover, transitions, and any scroll position inside the tables.
* Charts get their history from /api/system/history: one request per chart, N hosts each. *
* Every chart on this page belongs to exactly one host and is drawn inside that host's own
* panel (`/api/system/history`, one request per host per refresh). Nothing here plots two
* machines on one axis and there is no host/metric picker: "which machine stands out?" is
* the overview's job (alerts + fleet table), and `/system` answers "what is this machine
* doing?". See the per-host chart section below.
*/ */
'use strict'; 'use strict';
const { $, $$, fetchJSON, setTxt, setValue, setBar, setDot, bytes, pct, num, duration, poll, const { $, fetchJSON, setTxt, setDot, bytes, num, duration, poll,
staleness, markStale, severityFor } = Dash; markStale } = Dash;
const state = { const state = {
range: 3600, range: 3600,
detailHost: null,
detailMetric: 'cpu.usage_pct',
hosts: new Map(), // name -> card element hosts: new Map(), // name -> card element
charts: {}, charts: new Map(), // name -> { util, net, gpu|null } LineChart instances
poller: null, poller: null,
pollMs: 10000, pollMs: 10000,
metricList: [],
}; };
const DETAIL_METRICS = [
['cpu.usage_pct', 'CPU utilisation', 'pct'],
['cpu.user_pct', 'CPU user', 'pct'],
['cpu.system_pct', 'CPU system', 'pct'],
['cpu.iowait_pct', 'CPU iowait', 'pct'],
['cpu.steal_pct', 'CPU steal', 'pct'],
['cpu.load1', 'Load average (1m)', 'num'],
['cpu.load15', 'Load average (15m)', 'num'],
['cpu.freq_mhz', 'CPU frequency', 'mhz'],
['cpu.context_switches_per_s', 'Context switches', 'num'],
['mem.used_pct', 'Memory used', 'pct'],
['mem.used_bytes', 'Memory used', 'bytes'],
['mem.available_bytes', 'Memory available', 'bytes'],
['mem.swap_used_pct', 'Swap used', 'pct'],
['disk.used_pct', 'Filesystem used (worst)', 'pct'],
['disk.read_bytes_per_s', 'Disk read', 'bytes'],
['disk.write_bytes_per_s', 'Disk write', 'bytes'],
['disk.util_pct', 'Disk busy', 'pct'],
['net.rx_bytes_per_s', 'Network in', 'bytes'],
['net.tx_bytes_per_s', 'Network out', 'bytes'],
['temp.cpu_c', 'CPU temperature', 'temp'],
['temp.nvme_max_c', 'NVMe temperature', 'temp'],
['host.uptime_seconds', 'Uptime', 'dur'],
['host.processes', 'Process count', 'num'],
];
const FORMATS = {
pct: { fn: (v) => num(v, 1), unit: '%', axis: (v) => num(v, 0) },
bytes: { fn: (v) => bytes(v, 1), unit: 'B/s', axis: (v) => (Dash.fmtBytes(v, 0) || { text: '0' }).text +
((Dash.fmtBytes(v, 0) || { unit: '' }).unit) },
mhz: { fn: (v) => num(v, 0) + ' MHz', unit: 'MHz', axis: (v) => num(v, 0) },
temp: { fn: (v) => num(v, 1) + ' °C', unit: '°C', axis: (v) => num(v, 0) },
dur: { fn: (v) => duration(v), unit: '', axis: (v) => duration(v) },
num: { fn: (v) => num(v, 1), unit: '', axis: (v) => num(v, 0) },
};
function formatFor(kind) { return FORMATS[kind] || FORMATS.num; }
function metricKind(name) {
if (name.endsWith('_pct')) return 'pct';
if (name.endsWith('_bytes') || name.includes('bytes_per_s')) return 'bytes';
if (name.includes('_c') && name.startsWith('temp.')) return 'temp';
if (name === 'host.uptime_seconds') return 'dur';
return 'num';
}
// ---- host cards ----------------------------------------------------------- // ---- host cards -----------------------------------------------------------
//
// Six heroes per host, in one fixed order: CPU, MEM, TEMP, DISK, GPU, NET. Every colour
// comes from the payload's `severity` dicts — `severity.py` owns the thresholds, this file
// compares no number to any bound — and `Dash.hero()` prints `n/a` with **no** colour
// class when the number or its verdict is missing, so gateway's absent CPU sensor cannot
// borrow last poll's green.
//
// `data-sev` names the verdict each slot paints; tests/test_api.py pins the six slots and
// their order on the served bytes. The NET slot carries none on purpose: `network` has no
// `severity` (a byte rate has no good/bad), so nothing here may colour it.
//
// Per-core usage is not plotted here or anywhere on this page — the series is still
// stored as `cpu.usage_pct@coreN`, the page just stopped drawing a pattern nobody reads.
// Everything that is not glance-worthy (load averages, clock, disk read/write/busy,
// uptime, GPU power and temperature, filesystems, top processes) lives in "More detail",
// collapsed by default: the TODO asked for a simpler surface, not a less complete page.
function hostCard(name) { function hostCard(name) {
const card = document.createElement('article'); const card = document.createElement('article');
card.className = 'card'; card.className = 'card';
@@ -80,38 +52,110 @@ function hostCard(name) {
<span class="faint mono updated"></span> <span class="faint mono updated"></span>
</header> </header>
<div class="body stack"> <div class="body stack">
<div class="metrics"> <div class="heroes">
<div class="metric"><div class="label">CPU</div><div class="value"><span data-k="cpu">n/a</span><span class="unit">%</span></div> <div class="hero hero-bar" data-k="cpu" data-sev="cpu.severity.usage_pct"></div>
<div class="bar" data-bar="cpu"><i></i></div></div> <div class="hero hero-bar" data-k="mem" data-sev="memory.severity.used_pct"></div>
<div class="metric"><div class="label">Load 1 / 15</div><div class="value sm" data-k="load">n/a</div></div> <div class="hero" data-k="temp" data-sev="cpu.severity.temp_c"></div>
<div class="metric"><div class="label">CPU temp</div><div class="value sm"><span data-k="temp">n/a</span><span class="unit">°C</span></div></div> <div class="hero hero-bar" data-k="disk" data-sev="disks.severity.worst_used_pct"></div>
<div class="metric"><div class="label">Clock</div><div class="value sm"><span data-k="freq">n/a</span><span class="unit">GHz</span></div></div> <div class="hero-slot" data-k="gpus" data-sev="gpus[].severity.busy_pct"></div>
<div class="metric"><div class="label">Memory</div><div class="value sm"><span data-k="mem">n/a</span><span class="unit"></span></div> <div class="hero" data-k="net">
<div class="bar" data-bar="mem"><i></i></div></div> <div class="label">Net</div>
<div class="metric"><div class="label">Worst fs</div><div class="value sm"><span data-k="disk">n/a</span><span class="unit"></span></div> <div class="hero-lines">
<div class="bar" data-bar="disk"><i></i></div></div> <div class="line"><span class="dir">↓</span><span data-k="rx">n/a</span></div>
<div class="line"><span class="dir">↑</span><span data-k="tx">n/a</span></div>
</div> </div>
<div>
<div class="label faint" style="font-size:10.5px;letter-spacing:.06em;text-transform:uppercase">Per-core</div>
<div class="cores" data-k="cores" style="margin-top:.25rem"></div>
</div> </div>
<div class="metrics"> </div>
<div class="metric"><div class="label">Disk read</div><div class="value sm"><span data-k="dr">n/a</span><span class="unit"></span></div></div> <div class="host-charts">
<div class="metric"><div class="label">Disk write</div><div class="value sm"><span data-k="dw">n/a</span><span class="unit"></span></div></div> <div class="host-chart">
<div class="metric"><div class="label">Disk busy</div><div class="value sm"><span data-k="dutil">n/a</span><span class="unit">%</span></div></div> <div class="chart sm" data-chart="util"></div>
<div class="metric"><div class="label">Net in</div><div class="value sm"><span data-k="rx">n/a</span><span class="unit"></span></div></div> <div class="legend" data-legend="util"></div>
<div class="metric"><div class="label">Net out</div><div class="value sm"><span data-k="tx">n/a</span><span class="unit"></span></div></div> </div>
<div class="host-chart">
<div class="chart sm" data-chart="net"></div>
<div class="legend" data-legend="net"></div>
</div>
</div>
<details>
<summary class="faint" style="cursor:pointer;font-size:12px">More detail</summary>
<div class="metrics" style="margin-top:.6rem">
<div class="metric"><div class="label">Load 1 / 5 / 15</div><div class="value sm" data-k="load">n/a</div></div>
<div class="metric"><div class="label">Clock</div><div class="value sm" data-k="freq">n/a</div></div>
<div class="metric"><div class="label">Disk read</div><div class="value sm" data-k="dr">n/a</div></div>
<div class="metric"><div class="label">Disk write</div><div class="value sm" data-k="dw">n/a</div></div>
<div class="metric"><div class="label">Disk busy</div><div class="value sm" data-k="dutil">n/a</div></div>
<div class="metric"><div class="label">Up</div><div class="value sm" data-k="uptime">n/a</div></div> <div class="metric"><div class="label">Up</div><div class="value sm" data-k="uptime">n/a</div></div>
</div> </div>
<div data-k="gpus"></div> <div data-k="gpu-extra"></div>
<details>
<summary class="faint" style="cursor:pointer;font-size:12px">Filesystems &amp; processes</summary>
<div style="margin-top:.6rem" data-k="details"></div> <div style="margin-top:.6rem" data-k="details"></div>
</details> </details>
</div>`; </div>`;
return card; return card;
} }
/** `200 / 253 GiB` — the percentage is the verdict, the bytes are the detail. */
function memorySub(mem) {
const used = Dash.fmtBytes(mem.used_bytes, 1);
const total = Dash.fmtBytes(mem.total_bytes, 0);
if (!used || !total) return null;
if (used.unit === total.unit) return `${used.text} / ${total.text} ${total.unit}`;
return `${bytes(mem.used_bytes, 1)} of ${bytes(mem.total_bytes, 0)}`;
}
/** Model, VRAM used / total, and the honest APU pool view: a Strix Halo's model lives in
* GTT, so "VRAM 3%" and "pool 78%" are both true and the card shows both, not whichever
* flatters the machine. */
function gpuSub(gpu) {
const bits = [gpu.model || 'GPU'];
bits.push(gpu.vram_used_bytes == null || gpu.vram_total_bytes == null
? 'VRAM n/a'
: `VRAM ${bytes(gpu.vram_used_bytes, 1)} / ${bytes(gpu.vram_total_bytes, 1)}`);
if (gpu.mem_used_pct != null) bits.push(`pool ${num(gpu.mem_used_pct, 0)}%`);
return bits.join(' · ');
}
/** The GPU slot of the hero row: one hero per card, or the honest no-GPU state. A box
* with no card is not a box at 0% busy, and gateway must not look broken for it. */
function renderGpuSlot(cell, gpus) {
if (cell.dataset.count !== String(gpus.length)) {
cell.innerHTML = '';
for (let i = 0; i < Math.max(gpus.length, 1); i += 1) {
const row = document.createElement('div');
row.className = 'hero hero-bar';
cell.appendChild(row);
}
cell.dataset.count = String(gpus.length);
}
const rows = Array.from(cell.children);
if (!gpus.length) {
Dash.hero(rows[0], { label: 'GPU', value: null, sub: 'No GPU reported by this agent' });
return;
}
gpus.forEach((gpu, i) => {
Dash.hero(rows[i], {
label: gpus.length > 1 ? `GPU ${i}` : 'GPU',
value: gpu.busy_pct, unit: '%', digits: 0,
severity: (gpu.severity || {}).busy_pct, bar: true,
sub: gpuSub(gpu),
});
});
}
/** What the GPU hero left out, in "More detail": power, temperature, both memory pools. */
function renderGpuExtra(cell, gpus) {
if (!gpus.length) { cell.innerHTML = ''; return; }
cell.innerHTML = `
<div class="label faint" style="font-size:10.5px;letter-spacing:.06em;text-transform:uppercase;margin-top:.6rem">GPU detail</div>
<table class="data" style="margin-top:.25rem"><thead><tr>
<th>Card</th><th>Power</th><th>Temp</th><th>VRAM</th><th>GTT</th></tr></thead>
<tbody>${gpus.map((gpu) => `<tr>
<td class="name mono" style="font-size:11.5px">${DashCharts.escapeHtml(gpu.model || 'GPU')}</td>
<td class="num">${gpu.power_watts == null ? 'n/a' : `${num(gpu.power_watts, 0)} W`}</td>
<td class="num">${gpu.temp_c == null ? 'n/a' : `${num(gpu.temp_c, 0)} °C`}</td>
<td class="num mute">${bytes(gpu.vram_used_bytes, 1)} / ${bytes(gpu.vram_total_bytes, 1)}</td>
<td class="num mute">${bytes(gpu.gtt_used_bytes, 1)}</td></tr>`).join('')}</tbody></table>`;
}
function renderHostCard(card, host) { function renderHostCard(card, host) {
const q = (key) => card.querySelector(`[data-k="${key}"]`); const q = (key) => card.querySelector(`[data-k="${key}"]`);
const stateRow = host.state || {}; const stateRow = host.state || {};
@@ -125,65 +169,41 @@ function renderHostCard(card, host) {
markStale(card, stateRow.ok === false || Dash.ageSeconds(stateRow.last_ok) > 5 * (host.window || 10)); markStale(card, stateRow.ok === false || Dash.ageSeconds(stateRow.last_ok) > 5 * (host.window || 10));
const cpu = host.cpu || {}; const cpu = host.cpu || {};
setValue(q('cpu'), cpu.usage_pct, 0);
setBar(card.querySelector('[data-bar="cpu"]'), cpu.usage_pct, severityFor(cpu.usage_pct, 75, 90));
setTxt(q('load'), cpu.load1 == null ? 'n/a' : `${num(cpu.load1, 2)} / ${num(cpu.load15, 2)}`);
setValue(q('temp'), cpu.temp_c, 1);
setValue(q('freq'), cpu.freq_mhz == null ? null : cpu.freq_mhz / 1000, 2);
const mem = host.memory || {}; const mem = host.memory || {};
const memText = Dash.fmtBytes(mem.used_bytes, 1);
const memTotal = Dash.fmtBytes(mem.total_bytes, 0);
setTxt(q('mem'), memText ? memText.text : 'n/a');
setTxt(q('mem').nextElementSibling, memText ? `${memText.unit}/${memTotal ? memTotal.text + memTotal.unit : ''}` : '');
setBar(card.querySelector('[data-bar="mem"]'), mem.used_pct, severityFor(mem.used_pct, 80, 92));
const disks = host.disks || {}; const disks = host.disks || {};
setTxt(q('disk'), disks.worst_used_pct == null ? 'n/a' : num(disks.worst_used_pct, 0) + '%'); const net = host.network || {};
setTxt(q('disk').nextElementSibling, disks.worst_mount ? ' ' + disks.worst_mount : '');
setBar(card.querySelector('[data-bar="disk"]'), disks.worst_used_pct, severityFor(disks.worst_used_pct, 80, 92));
DashCharts.renderCores(q('cores'), cpu.core_usage_pct || []); // The six heroes. The verdict is whatever the payload carried (`severity` dicts written
// by severity.py, pinned by tests/test_system_contract.py); a missing number or a
// missing verdict prints `n/a` with no colour class at all.
Dash.hero(q('cpu'), { label: 'CPU', value: cpu.usage_pct, unit: '%', digits: 0,
severity: (cpu.severity || {}).usage_pct, bar: true });
Dash.hero(q('mem'), { label: 'Memory', value: mem.used_pct, unit: '%', digits: 0,
severity: (mem.severity || {}).used_pct, bar: true,
sub: memorySub(mem) });
Dash.hero(q('temp'), { label: 'CPU temp', value: cpu.temp_c, unit: '°C', digits: 1,
severity: (cpu.severity || {}).temp_c });
// the mount is the `.sub` because "80 %" without "of what" is not actionable
Dash.hero(q('disk'), { label: 'Disk', value: disks.worst_used_pct, unit: '%', digits: 0,
severity: (disks.severity || {}).worst_used_pct, bar: true,
sub: disks.worst_mount });
renderGpuSlot(q('gpus'), host.gpus || []);
// One wire seen in two directions, two lines, never two competing numbers — and a null
// (no physical interface reported) stays `n/a` rather than becoming a confident 0 B/s.
// No severity by design: `network` carries none, so this slot never colours.
setTxt(q('rx'), net.rx_bytes_per_s == null ? null : Dash.rate(net.rx_bytes_per_s));
setTxt(q('tx'), net.tx_bytes_per_s == null ? null : Dash.rate(net.tx_bytes_per_s));
setValue(q('dr'), disks.read_bytes_per_s == null ? null : disks.read_bytes_per_s, null); // "More detail": everything the glance does not need but the page still owes.
if (disks.read_bytes_per_s != null) { setTxt(q('load'), cpu.load1 == null && cpu.load5 == null && cpu.load15 == null
const parts = Dash.fmtBytes(disks.read_bytes_per_s, 1); ? null : `${num(cpu.load1, 2)} / ${num(cpu.load5, 2)} / ${num(cpu.load15, 2)}`);
setTxt(q('dr'), parts.text); setTxt(q('dr').nextElementSibling, parts.unit + '/s'); setTxt(q('freq'), cpu.freq_mhz == null ? null : `${num(cpu.freq_mhz / 1000, 2)} GHz`);
const w = Dash.fmtBytes(disks.write_bytes_per_s, 1); setTxt(q('dr'), disks.read_bytes_per_s == null ? null : Dash.rate(disks.read_bytes_per_s));
setTxt(q('dw'), w ? w.text : 'n/a'); setTxt(q('dw').nextElementSibling, w ? w.unit + '/s' : ''); setTxt(q('dw'), disks.write_bytes_per_s == null ? null : Dash.rate(disks.write_bytes_per_s));
} else { setTxt(q('dr'), 'n/a'); setTxt(q('dw'), 'n/a'); } setTxt(q('dutil'), disks.util_pct == null ? null : `${num(disks.util_pct, 0)}%`);
setValue(q('dutil'), disks.util_pct, 0); const up = (host.info || {}).uptime_seconds;
const rx = Dash.fmtBytes((host.network || {}).rx_bytes_per_s, 1); setTxt(q('uptime'), up == null ? null : duration(up));
const tx = Dash.fmtBytes((host.network || {}).tx_bytes_per_s, 1); renderGpuExtra(q('gpu-extra'), host.gpus || []);
setTxt(q('rx'), rx ? rx.text : 'n/a'); setTxt(q('rx').nextElementSibling, rx ? rx.unit + '/s' : '');
setTxt(q('tx'), tx ? tx.text : 'n/a'); setTxt(q('tx').nextElementSibling, tx ? tx.unit + '/s' : '');
setTxt(q('uptime'), duration((host.info || {}).uptime_seconds));
// GPUs: one row each, with both memory pools named (an APU's model lives in GTT)
const gpus = host.gpus || [];
const gpuHost = q('gpus');
if (!gpus.length) {
gpuHost.innerHTML = '<div class="faint" style="font-size:12px">No GPU reported by this agent</div>';
} else {
gpuHost.innerHTML = gpus.map((gpu) => `
<div class="stack" style="gap:.2rem;margin-top:.35rem">
<div class="gap" style="justify-content:space-between">
<span class="mono" style="font-size:12px">${DashCharts.escapeHtml(gpu.model || 'GPU')}</span>
<span class="mono" style="font-size:12px">
${gpu.busy_pct == null ? 'n/a' : num(gpu.busy_pct, 0) + '% busy'} ·
${gpu.power_watts == null ? 'power n/a' : num(gpu.power_watts, 0) + ' W'} ·
${gpu.temp_c == null ? 'temp n/a' : num(gpu.temp_c, 0) + '°C'}</span>
</div>
<div class="bar thin"><i style="width:${(gpu.busy_pct || 0).toFixed(1)}%"></i></div>
<div class="gap" style="justify-content:space-between;font-size:11.5px" class="faint">
<span class="faint">VRAM ${gpu.vram_used_bytes == null ? 'n/a' : bytes(gpu.vram_used_bytes, 0)}
${gpu.vram_total_bytes ? 'of ' + bytes(gpu.vram_total_bytes, 0) : ''}</span>
<span class="faint">shared (GTT) ${gpu.gtt_used_bytes == null ? 'n/a' : bytes(gpu.gtt_used_bytes, 0)}
${gpu.mem_used_pct == null ? '' : ' · pool ' + num(gpu.mem_used_pct, 0) + '%'}</span>
</div>
<div class="bar thin"><i style="width:${(gpu.mem_used_pct || 0).toFixed(1)}%;background:var(--series-2)"></i></div>
</div>`).join('');
}
const filesystems = (disks.filesystems || []).filter((fs) => fs.used_pct != null); const filesystems = (disks.filesystems || []).filter((fs) => fs.used_pct != null);
const procs = host.processes || {}; const procs = host.processes || {};
@@ -226,180 +246,175 @@ function stalenessText(row) {
return seconds > 30 ? `stale ${duration(seconds)}` : `updated ${seconds < 2 ? 'now' : duration(seconds)}`; return seconds > 30 ? `stale ${duration(seconds)}` : `updated ${seconds < 2 ? 'now' : duration(seconds)}`;
} }
// ---- charts --------------------------------------------------------------- // ---- per-host charts ------------------------------------------------------
function makeCharts() { //
const common = { height: 165, xSpan: state.range }; // One host, one panel, at most three charts — and no chart on this page contains a second
state.charts.cpu = new DashCharts.LineChart('#chart-cpu', Object.assign({}, common, { // host. Six metrics x three hosts as six separate charts is the unreadable version of
yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) })); // "per-host charts", so the metrics that share a unit share an axis:
state.charts.mem = new DashCharts.LineChart('#chart-mem', Object.assign({}, common, { //
yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) })); // util cpu.usage_pct + mem.used_pct + disk.used_pct one 0-100 % axis
state.charts.disk = new DashCharts.LineChart('#chart-disk', Object.assign({}, common, { // net net.rx_bytes_per_s + net.tx_bytes_per_s one byte-rate axis
yMin: 0, yFormat: (v) => (Dash.fmtBytes(v, 0) || { text: '0' }).text })); // gpu gpu.busy_pct@<index> per card the agent reports (no chart when there is none)
state.charts.net = new DashCharts.LineChart('#chart-net', Object.assign({}, common, { //
yMin: 0, yFormat: (v) => (Dash.fmtBytes(v, 0) || { text: '0' }).text })); // Each host costs **one** `/api/system/history` request per refresh: the endpoint takes a
state.charts.gpu = new DashCharts.LineChart('#chart-gpu', Object.assign({}, common, { // `hosts=` and a `metrics=` list, so the XHR budget for a three-host fleet is one snapshot
yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) })); // plus three history calls plus the event feed, whatever the chart count. GPU stream names
state.charts.temp = new DashCharts.LineChart('#chart-temp', Object.assign({}, common, { // come from the payload's own `gpus[].index` (`sources/host.py` writes `gpu.busy_pct@<i>`),
yFormat: (v) => num(v, 0) })); // so nothing here has to discover them through /api/metrics the way the deleted
state.charts.detail = new DashCharts.LineChart('#chart-detail', Object.assign({}, common, { // fleet-wide GPU chart did.
height: 200, yFormat: (v) => num(v, 1) })); //
// The charts live in a Map keyed by host name and are torn down with the card: a panel for
// a host that has left the payload would otherwise keep drawing its last series forever.
const CHART_HEIGHT = 120;
const UTIL_SERIES = [
['cpu.usage_pct', 'CPU'],
['mem.used_pct', 'Mem'],
['disk.used_pct', 'Disk (worst)'],
];
const NET_SERIES = [
['net.rx_bytes_per_s', '↓ in'],
['net.tx_bytes_per_s', '↑ out'],
];
const PCT_FORMAT = (v) => `${num(v, 1)}%`;
const RATE_FORMAT = (v) => `${bytes(v, 1)}/s`;
const AXIS_PCT = (v) => num(v, 0);
const AXIS_RATE = (v) => {
const parts = Dash.fmtBytes(v, 0) || { text: '0', unit: '' };
return `${parts.text}${parts.unit}`;
};
/** The GPU streams one host reports, named from the payload rather than guessed. */
function gpuStreams(host) {
return (host.gpus || []).map((gpu, i) => {
const index = gpu.index == null ? i : gpu.index;
return { metric: `gpu.busy_pct@${index}`, label: gpu.model || `GPU ${index}` };
});
} }
function setChartSpan(span) { /**
Object.values(state.charts).forEach((chart) => { chart.opts.xSpan = span; }); * The chart objects inside one host panel, built once and reused every poll.
} *
* The GPU chart is added the first time the host reports a card and removed the moment it
function seriesFor(response, metric) { * stops reporting one — a machine whose GPU went missing must not keep a plot of it.
return (response.series || []).filter((s) => s.metric === metric).map((s) => ({ */
name: s.host, points: s.points.map((p) => ({ ts: p.ts, value: p.value })), function hostChartSet(card, host) {
})); let set = state.charts.get(host.name);
} if (!set) {
// no `area` here on purpose — `drawHostChart()` decides the fill per draw, from how many
async function refreshCharts(snapshot) { // series actually have points, which is the only question worth asking
const hostNames = (snapshot.hosts || []).map((h) => h.name); const common = { height: CHART_HEIGHT, xSpan: state.range };
const detailHost = state.detailHost || hostNames[0] || null; // Every chart names itself for assistive tech: the panel heading says *who*, the
const span = state.range; // aria-label says what is plotted. An unlabelled SVG is silence, and a screen reader
const requests = { // must not need the colour-coded legend to know which machine it is listening to.
cpu: fetchJSON(`/api/system/history?metrics=cpu.usage_pct&range=${span}`).catch(() => null), set = {
mem: fetchJSON(`/api/system/history?metrics=mem.used_pct&range=${span}`).catch(() => null), util: new DashCharts.LineChart(card.querySelector('[data-chart="util"]'),
temp: fetchJSON(`/api/system/history?metrics=temp.cpu_c&range=${span}`).catch(() => null), Object.assign({}, common, { yMin: 0, yMax: 100, yFormat: AXIS_PCT,
disk: detailHost ? fetchJSON(`/api/system/history?hosts=${encodeURIComponent(detailHost)}&` + ariaLabel: `${host.name} — utilisation over the selected window: CPU, memory and worst disk in per cent` })),
`metrics=disk.read_bytes_per_s,disk.write_bytes_per_s&range=${span}`).catch(() => null) : null, net: new DashCharts.LineChart(card.querySelector('[data-chart="net"]'),
net: detailHost ? fetchJSON(`/api/system/history?hosts=${encodeURIComponent(detailHost)}&` + Object.assign({}, common, { yMin: 0, yFormat: AXIS_RATE,
`metrics=net.rx_bytes_per_s,net.tx_bytes_per_s&range=${span}`).catch(() => null) : null, ariaLabel: `${host.name} — network traffic over the selected window: bytes in and bytes out` })),
gpu: null,
}; };
const [cpu, mem, temp, disk, net] = await Promise.all([requests.cpu, requests.mem, state.charts.set(host.name, set);
requests.temp, requests.disk, requests.net]);
if (cpu) {
const series = seriesFor(cpu, 'cpu.usage_pct');
state.charts.cpu.setData(series);
DashCharts.renderLegend($('#legend-cpu'), series, (v) => num(v, 1) + '%', true);
} }
if (mem) { const wantsGpu = gpuStreams(host).length > 0;
const series = seriesFor(mem, 'mem.used_pct'); if (wantsGpu && !set.gpu) {
state.charts.mem.setData(series); const slot = document.createElement('div');
DashCharts.renderLegend($('#legend-mem'), series, (v) => num(v, 1) + '%'); slot.className = 'host-chart';
slot.dataset.chartSlot = 'gpu';
slot.innerHTML = '<div class="chart sm" data-chart="gpu"></div>' +
'<div class="legend" data-legend="gpu"></div>';
card.querySelector('.host-charts').appendChild(slot);
set.gpu = new DashCharts.LineChart(slot.querySelector('[data-chart="gpu"]'), {
height: CHART_HEIGHT, xSpan: state.range, yMin: 0, yMax: 100, yFormat: AXIS_PCT,
ariaLabel: `${host.name} — GPU busy per cent over the selected window: ` +
gpuStreams(host).map((s) => s.label).join(', ') });
} }
if (temp) { if (!wantsGpu && set.gpu) {
const series = seriesFor(temp, 'temp.cpu_c'); // the whole slot goes: an empty wrapper or a plot of a card that is no longer
state.charts.temp.setData(series); // reported is a machine's panel lying about hardware it does not have
DashCharts.renderLegend($('#legend-temp'), series, (v) => num(v, 1) + '°C', true); card.querySelectorAll('[data-chart-slot="gpu"]').forEach((node) => node.remove());
set.gpu = null;
} }
if (disk) { return set;
const read = (disk.series.find((s) => s.metric === 'disk.read_bytes_per_s') || { points: [] })
.points.map((p) => ({ ts: p.ts, value: p.value }));
const write = (disk.series.find((s) => s.metric === 'disk.write_bytes_per_s') || { points: [] })
.points.map((p) => ({ ts: p.ts, value: p.value }));
const series = [{ name: 'read', points: read }, { name: 'write', points: write }];
state.charts.disk.setData(series);
DashCharts.renderLegend($('#legend-disk'), series, (v) => bytes(v, 1) + '/s', true);
}
if (net) {
const rx = (net.series.find((s) => s.metric === 'net.rx_bytes_per_s') || { points: [] })
.points.map((p) => ({ ts: p.ts, value: p.value }));
const tx = (net.series.find((s) => s.metric === 'net.tx_bytes_per_s') || { points: [] })
.points.map((p) => ({ ts: p.ts, value: p.value }));
const series = [{ name: 'in', points: rx }, { name: 'out', points: tx }];
state.charts.net.setData(series);
DashCharts.renderLegend($('#legend-net'), series, (v) => bytes(v, 1) + '/s', true);
}
await refreshGpuChart(hostNames);
await refreshDetailChart();
} }
async function refreshGpuChart(hostNames) { /**
if (!hostNames.length) { * `[metric, label][]` -> LineChart series, from one host's history response.
state.charts.gpu.setData([]); *
$('#legend-gpu').innerHTML = '<span class="faint">no hosts</span>'; * The legend names the *metric*, never the host: a chart here can only ever hold one
return; * machine, so repeating the host name three times under a chart is ink with no information.
} */
const series = []; function seriesFor(response, definitions) {
for (const host of hostNames) { return definitions.map(([metric, label]) => {
// per-stream names are discovered, not guessed: gpu.busy_pct@0, @1, ... const found = (response.series || []).find((s) => s.metric === metric);
const found = await fetchJSON(`/api/metrics?source=host&host=${encodeURIComponent(host)}`) return { name: label,
.catch(() => ({ metrics: [] })); points: ((found || {}).points || []).map((p) => ({ ts: p.ts, value: p.value })) };
const names = (found.metrics || []).filter((m) => m.startsWith('gpu.busy_pct@'));
for (const name of names) {
const data = await fetchJSON(`/api/history?source=host&host=${encodeURIComponent(host)}` +
`&metric=${encodeURIComponent(name)}&range=${state.range}`).catch(() => null);
if (data) series.push({ name: `${host} ${name.split('@')[1]}`,
points: data.points.map((p) => ({ ts: p.ts, value: p.value })) });
}
}
state.charts.gpu.setData(series);
DashCharts.renderLegend($('#legend-gpu'), series, (v) => num(v, 0) + '%', true);
}
async function refreshDetailChart() {
const host = state.detailHost;
if (!host) { state.charts.detail.setData([]); return; }
const kind = metricKind(state.detailMetric);
const format = formatFor(kind);
const data = await fetchJSON(`/api/history?source=host&host=${encodeURIComponent(host)}` +
`&metric=${encodeURIComponent(state.detailMetric)}&range=${state.range}`).catch(() => null);
if (!data) return;
const series = [{ name: state.detailMetric,
points: data.points.map((p) => ({ ts: p.ts, value: p.value })) }];
state.charts.detail.opts.yFormat = format.axis;
state.charts.detail.opts.yMin = state.detailMetric.endsWith('_pct') ? 0 : null;
state.charts.detail.opts.yMax = state.detailMetric.endsWith('_pct') ? 100 : null;
state.charts.detail.setData(series);
DashCharts.renderLegend($('#legend-detail'), series, format.fn);
const last = series[0].points.slice(-1)[0];
$('#detail-note').textContent = `${series[0].points.length} samples over ${duration(state.range)} · ` +
`last ${last ? format.fn(last.value) : 'n/a'} · ${state.range <= 86400 ? 'raw resolution' : 'rolled up'}`;
}
async function populateDetailMetrics(host) {
const select = $('#detail-metric');
const found = await fetchJSON(`/api/metrics?source=host&host=${encodeURIComponent(host)}`)
.catch(() => ({ metrics: [] }));
const curated = new Set(DETAIL_METRICS.map(([name]) => name));
const extras = (found.metrics || []).filter((m) => !curated.has(m)).sort();
select.innerHTML = '';
const groupMain = document.createElement('optgroup');
groupMain.label = 'Common';
DETAIL_METRICS.forEach(([name, label]) => {
const option = document.createElement('option');
option.value = name; option.textContent = label;
groupMain.appendChild(option);
}); });
select.appendChild(groupMain); }
if (extras.length) {
const groupExtra = document.createElement('optgroup'); /**
groupExtra.label = `Per-device (${extras.length})`; * Plot one series set into one panel chart and repaint its legend.
extras.forEach((name) => { *
const option = document.createElement('option'); * The fill is decided per draw on purpose: a single series reads as a shape worth shading
option.value = name; option.textContent = name; * (GPU busy), while two or three filled areas stack into one block and bury the smaller
groupExtra.appendChild(option); * line — which is how a quiet CPU hides behind a busy memory plot.
}); */
select.appendChild(groupExtra); function drawHostChart(chart, card, key, series, format) {
chart.opts.xSpan = state.range;
chart.opts.area = series.filter((s) => s.points.length).length < 2;
chart.setData(series);
DashCharts.renderLegend(card.querySelector(`[data-legend="${key}"]`), series, format, true);
}
/** One host's charts, from one request. Called for every host on every refresh. */
async function renderHostCharts(card, host) {
const charts = hostChartSet(card, host);
const gpu = gpuStreams(host);
const wanted = UTIL_SERIES.concat(NET_SERIES, gpu.map((s) => [s.metric, s.label]));
const query = `/api/system/history?hosts=${encodeURIComponent(host.name)}` +
`&metrics=${encodeURIComponent(wanted.map(([metric]) => metric).join(','))}` +
`&range=${state.range}`;
const response = await fetchJSON(query).catch(() => null);
if (!response || !state.charts.has(host.name)) return; // the panel went away mid-flight
drawHostChart(charts.util, card, 'util', seriesFor(response, UTIL_SERIES), PCT_FORMAT);
drawHostChart(charts.net, card, 'net', seriesFor(response, NET_SERIES), RATE_FORMAT);
if (charts.gpu) {
drawHostChart(charts.gpu, card, 'gpu',
seriesFor(response, gpu.map((s) => [s.metric, s.label])), PCT_FORMAT);
} }
select.value = DETAIL_METRICS.some(([name]) => name === state.detailMetric) }
? state.detailMetric : (extras[0] || DETAIL_METRICS[0][0]);
state.detailMetric = select.value || DETAIL_METRICS[0][0]; /**
* The Δ window the rates were actually measured over, for the page footnote.
*
* `window_seconds` beside the fleet is the *poll interval*; the window behind a rate is the
* estimator's (`RATE_WINDOW_SECONDS`) and the two need not agree, so the footnote reads each
* host's own measured `window_seconds` instead of repeating a config number as a measurement
* (invariant 2: a rate says how it was obtained). One poll loop drives every host, so the
* windows differ by milliseconds; a real spread prints as a range rather than picking a winner.
*/
function windowLabel(values, fallback) {
const nums = values.filter((value) => value != null && Number.isFinite(value));
if (!nums.length) return fallback;
const lo = Math.min(...nums);
const hi = Math.max(...nums);
return lo.toFixed(1) === hi.toFixed(1) ? `Δ${hi.toFixed(1)}s`
: `Δ${lo.toFixed(1)}–${hi.toFixed(1)}s`;
}
/** Drop cards (and charts) for hosts that are no longer in the payload. */
function pruneHosts(living) {
Array.from(state.hosts.keys()).forEach((name) => {
if (living.includes(name)) return;
state.hosts.get(name).remove();
state.hosts.delete(name);
state.charts.delete(name);
});
} }
// ---- snapshot ------------------------------------------------------------- // ---- snapshot -------------------------------------------------------------
function renderFleet(fleet) {
if (!fleet) return;
setTxt($('#f-hosts'), fleet.hosts_live == null ? 'n/a' : String(fleet.hosts_live));
setTxt($('#f-hosts-of'), fleet.hosts_total == null ? '' : `of ${fleet.hosts_total}`);
setValue($('#f-cpu'), fleet.cpu_usage_pct_avg, 0);
setValue($('#f-load'), fleet.load1_total, 2);
const used = Dash.fmtBytes(fleet.memory_used_bytes, 0);
const total = Dash.fmtBytes(fleet.memory_total_bytes, 0);
setTxt($('#f-mem'), used ? used.text : 'n/a');
setTxt($('#f-mem-of'), used && total ? `${used.unit} of ${total.text} ${total.unit}` : '');
setValue($('#f-disk'), fleet.disk_worst_used_pct, 0);
const rx = Dash.fmtBytes(fleet.net_rx_bytes_per_s, 1);
const tx = Dash.fmtBytes(fleet.net_tx_bytes_per_s, 1);
setTxt($('#f-rx'), rx ? rx.text : 'n/a'); setTxt($('#f-rx-unit'), rx ? rx.unit + '/s' : '');
setTxt($('#f-tx'), tx ? tx.text : 'n/a'); setTxt($('#f-tx-unit'), tx ? tx.unit + '/s' : '');
setValue($('#f-gpu'), fleet.gpu_busy_pct_avg, 0);
}
async function refresh() { async function refresh() {
let snapshot; let snapshot;
try { try {
@@ -411,6 +426,7 @@ async function refresh() {
const states = {}; const states = {};
(snapshot.states || []).forEach((row) => { states[row.key.split(':')[1]] = row; }); (snapshot.states || []).forEach((row) => { states[row.key.split(':')[1]] = row; });
const hostNames = []; const hostNames = [];
const chartJobs = [];
(snapshot.hosts || []).forEach((host) => { (snapshot.hosts || []).forEach((host) => {
hostNames.push(host.name); hostNames.push(host.name);
host.state = states[host.name]; host.state = states[host.name];
@@ -420,43 +436,33 @@ async function refresh() {
card = hostCard(host.name); card = hostCard(host.name);
state.hosts.set(host.name, card); state.hosts.set(host.name, card);
$('#hosts').appendChild(card); $('#hosts').appendChild(card);
card.addEventListener('click', () => {
if (state.detailHost !== host.name) {
state.detailHost = host.name;
$('#detail-host').value = host.name;
populateDetailMetrics(host.name).then(refreshDetailChart);
}
});
} }
renderHostCard(card, host); renderHostCard(card, host);
// one history request per host, all of its charts fed from the same response
chartJobs.push(renderHostCharts(card, host));
}); });
// a host that left HOST_TARGETS must take its panel — and its live charts — with it
pruneHosts(hostNames);
$('#hosts-empty').hidden = hostNames.length > 0; $('#hosts-empty').hidden = hostNames.length > 0;
const hostSelect = $('#detail-host');
if (hostSelect.dataset.names !== hostNames.join(',')) {
hostSelect.dataset.names = hostNames.join(',');
hostSelect.innerHTML = hostNames.map((name) =>
`<option value="${DashCharts.escapeHtml(name)}">${DashCharts.escapeHtml(name)}</option>`).join('');
if (!state.detailHost || !hostNames.includes(state.detailHost)) {
state.detailHost = hostNames[0] || null;
}
if (state.detailHost) hostSelect.value = state.detailHost;
if (state.detailHost) await populateDetailMetrics(state.detailHost);
}
renderFleet(snapshot.fleet);
const fleet = snapshot.fleet || {}; const fleet = snapshot.fleet || {};
$('#sub').textContent = `${fleet.hosts_live || 0}/${fleet.hosts_total || 0} hosts answering · ` + $('#sub').textContent = `${fleet.hosts_live || 0}/${fleet.hosts_total || 0} hosts answering · ` +
`${fleet.cpu_cores || 0} cores · ${fleet.gpu_count || 0} GPUs · ` + `${fleet.cpu_cores || 0} cores · ${fleet.gpu_count || 0} GPUs · ` +
`window ${duration(state.range)}`; `window ${duration(state.range)}`;
$('#foot-note').textContent = `rates over Δ${snapshot.window_seconds}s · ` + // the Δ comes from the payloads, never from the interval the fleet happens to answer on
`poll ${Math.round(state.pollMs / 1000)}s`; $('#foot-note').textContent = `rates over ${windowLabel(
await refreshCharts(snapshot); (snapshot.hosts || []).map((host) => host.window_seconds),
`Δ${snapshot.window_seconds}s`, // nothing measured yet — the configured interval
)} · poll ${Math.round(state.pollMs / 1000)}s`;
await Promise.all(chartJobs);
} }
async function refreshEvents() { async function refreshEvents() {
try { try {
const data = await fetchJSON('/api/events?limit=25&source=host'); // `system`, not `host`: reachability/reboot events are logged under the *dashboard* name
// (`poller.py` writes `Event(ts, self.dashboard, …)`; samples use `host`, events do not).
// Querying `source=host` matched nothing and left this feed forever empty.
const data = await fetchJSON('/api/events?limit=25&source=system');
const list = $('#events'); const list = $('#events');
if (!(data.events || []).length) { if (!(data.events || []).length) {
list.innerHTML = '<li class="muted" style="grid-template-columns:1fr">No host transitions ' + list.innerHTML = '<li class="muted" style="grid-template-columns:1fr">No host transitions ' +
@@ -471,7 +477,6 @@ async function refreshEvents() {
} }
document.addEventListener('DOMContentLoaded', async () => { document.addEventListener('DOMContentLoaded', async () => {
makeCharts();
let pollMs = 10000; let pollMs = 10000;
try { try {
const config = await fetchJSON('/api/config'); const config = await fetchJSON('/api/config');
@@ -482,22 +487,12 @@ document.addEventListener('DOMContentLoaded', async () => {
} catch (err) { /* config is nice-to-have */ } } catch (err) { /* config is nice-to-have */ }
state.pollMs = pollMs; state.pollMs = pollMs;
// the window chips re-fetch every panel's history: the range is per page, not per chart
Dash.rangeControl($('#range'), state.range, (seconds) => { Dash.rangeControl($('#range'), state.range, (seconds) => {
state.range = seconds; state.range = seconds;
setChartSpan(seconds);
refresh(); refresh();
}); });
$('#detail-host').addEventListener('change', async (event) => {
state.detailHost = event.target.value;
await populateDetailMetrics(state.detailHost);
await refreshDetailChart();
});
$('#detail-metric').addEventListener('change', async (event) => {
state.detailMetric = event.target.value;
await refreshDetailChart();
});
state.poller = poll(async () => { await refresh(); await refreshEvents(); }, state.pollMs); state.poller = poll(async () => { await refresh(); await refreshEvents(); }, state.pollMs);
Dash.bindPause(state.poller); Dash.bindPause(state.poller);
}); });
+104
View File
@@ -0,0 +1,104 @@
"""The `/system` hero-card payload contract, shared by the hermetic and the live tests.
Phase 03 makes `/system` one panel per host with six big colour-coded numbers. The paths
below are the payload fields those cards are allowed to read; they were read out of
`parse_host_snapshot()` and are pinned hermetically in `tests/test_system_contract.py`
(driven from `tests/fixtures/agent_*.json`) and against the real fleet in
`tests/test_live_fleet.py::test_every_host_serves_the_hero_card_contract`.
Keeping the list here rather than in one test file means the live assertions cannot drift
away from the fixture assertions: there is one table, and it is the thing under test.
=========================== ==========================================================
Card Path in `/api/system/snapshot` `hosts[]`
=========================== ==========================================================
cpu % ``cpu.usage_pct`` (+ ``cpu.severity.usage_pct``)
mem % ``memory.used_pct`` (+ ``memory.severity.used_pct``)
cpu temp ``cpu.temp_c`` (+ ``cpu.severity.temp_c``)
disk usage % ``disks.worst_used_pct`` + ``disks.worst_mount`` (+ verdict)
graphics card ``gpus[].model/busy_pct/vram_*/mem_used_pct`` (+ verdict)
network down/up ``network.rx_bytes_per_s`` / ``tx_bytes_per_s`` (no verdict)
=========================== ==========================================================
CPU temp comes from ``cpu.temp_c`` rather than ``temperatures.cpu_c``: both are the agent's
own ``cpu_temperature_celsius``, and ``cpu.temp_c`` is the field ``/api/summary`` flattens
into its fleet row, so the card and the overview cannot drift apart.
Deliberately *not* card metrics (legibility, not missing data — the series stay in SQLite):
per-core usage (``cpu.usage_pct@coreN``), swap, the process table, the raw sensor list, and
per-interface rows. Network carries no ``severity`` on purpose; throughput has no good/bad,
so ``severity.THRESHOLDS`` has no ``net.*`` row.
"""
from __future__ import annotations
from typing import Any
#: Card metric -> dotted payload paths. Present is the contract; a value may be ``None``
#: on a host that genuinely does not have the number (gateway reports no CPU temperature).
CARD_METRICS: dict[str, tuple[str, ...]] = {
"cpu %": ("cpu.usage_pct", "cpu.severity.usage_pct"),
"mem %": ("memory.used_pct", "memory.severity.used_pct"),
"cpu temp": ("cpu.temp_c", "cpu.severity.temp_c", "temperatures.cpu_c"),
# the mount is what makes the percentage actionable: 88 % used — of which disk?
"disk %": ("disks.worst_used_pct", "disks.worst_mount", "disks.severity.worst_used_pct"),
"network down/up": ("network.rx_bytes_per_s", "network.tx_bytes_per_s"),
}
#: Per-GPU paths, checked on every row of `gpus[]` — and on none of them when the list is
#: empty, which is how a host without a GPU stays honest (no card, not a card at 0 % busy).
GPU_METRICS: tuple[str, ...] = (
"model",
"busy_pct",
"vram_used_bytes",
"vram_total_bytes",
"vram_used_pct",
# the honest APU view: vram + GTT, because a Strix Halo keeps the model in GTT
"mem_used_pct",
"severity.busy_pct",
"severity.mem_used_pct",
)
#: `network` must never grow a `severity` dict (see the module docstring).
SEVERITY_FREE_SECTIONS: tuple[str, ...] = ("network",)
#: The host shapes the captures in `tests/fixtures/` cover, and therefore the shapes a card
#: must handle: a discrete GPU, an APU (VRAM *and* GTT), and no GPU at all. A new capture
#: has to be added here — `test_the_contract_covers_every_captured_host` fails otherwise.
HOST_SHAPES: tuple[str, ...] = ("driveripper", "deskwork", "gateway")
def dig(node: Any, path: str) -> Any:
"""Follow a dotted payload path, failing with the path in the message.
A missing key is a contract break; a ``None`` value is not (that renders `n/a`), which is
why this returns the value instead of asserting on it.
"""
walked = ""
for part in path.split("."):
assert isinstance(node, dict), f"{path}: hit a non-dict at {walked or 'payload'}"
assert part in node, (
f"{path}: no key {part!r} under {walked or 'payload'} — a hero card would print "
f"n/a for a number the API used to serve"
)
node = node[part]
walked = f"{walked}.{part}" if walked else part
return node
def assert_card_contract(payload: Any) -> None:
"""Every card path exists on this host payload, GPU rows included."""
for paths in CARD_METRICS.values():
for path in paths:
dig(payload, path)
gpus = payload.get("gpus")
assert isinstance(gpus, list), "gpus must be a list — [] when the agent reports no GPU"
for gpu in gpus:
for path in GPU_METRICS:
dig(gpu, path)
for section in SEVERITY_FREE_SECTIONS:
block = payload.get(section)
assert isinstance(block, dict), f"{section} must be a dict"
assert "severity" not in block, (
f"{section}.severity would be a colour nobody measured — throughput has no "
f"good/bad and severity.THRESHOLDS has no {section}.* row"
)
+292 -5
View File
@@ -388,6 +388,60 @@ def test_agent_tokens_never_reach_the_browser(tmp_path: Path, monkeypatch: pytes
store.close() store.close()
def test_a_dead_host_never_writes_its_token_into_the_feed(tmp_path: Path) -> None:
"""The second path a credential can take to a browser: the failure message.
`parse_targets` moves a userinfo token out of the URL and into a bearer header, so the
documented `HOST_TARGETS` path never puts one in `Target.url` — this is the backstop for
any `Target` that does (a hand-built one, a future parser), and the message path is where
it would matter: the poller keeps an `UpstreamError` string in the health record and logs
it as an event, so a userinfo-bearing URL there would sit in SQLite and be served on
`/api/events` forever (AGENTS.md invariant 6). Asserted on every endpoint a page reads,
not just `/api/config`.
"""
db = str(tmp_path / "dead-host-token.sqlite3")
store = SampleStore(db)
settings = Settings(
db_path=db,
host_interval=0.05,
host_targets=(Target(name="driveripper", url="http://SEKRIT-AGENT-TOKEN@127.0.0.1:1"),),
)
with TestClient(create_app(settings, store)) as client:
deadline = time.time() + 10
while time.time() < deadline:
health = client.get("/api/health").json()
if health["sources"] and health["sources"][0]["error"]:
break
time.sleep(0.05)
else:
pytest.fail("the dead host never recorded an error")
for path in ("/api/health", "/api/events", "/api/system/snapshot", "/api/summary"):
body = client.get(path).text
assert "SEKRIT-AGENT-TOKEN" not in body, f"{path} leaked the agent token"
# and the outage is still legible — redaction is not the same as silence
assert "driveripper" in client.get("/api/health").text
assert "unreachable" in client.get("/api/events").text
store.close()
def test_engine_urls_are_redacted_the_same_way_hosts_are(tmp_path: Path) -> None:
"""`GUFO_BASE_URL`/`VLLM_BASE_URL` go to the browser verbatim in `/api/config`, and unlike
`HOST_TARGETS` nothing parses a userinfo token out of them into a header — so a URL that
happens to carry credentials has to be redacted where it is served (invariant 6).
"""
db = str(tmp_path / "engine-redact.sqlite3")
settings = Settings(db_path=db, gufo_url="http://SEKRIT-ENGINE-TOKEN@deskwork:8731",
vllm_url="http://driveripper:8000")
store = SampleStore(db)
with TestClient(create_app(settings, store)) as client:
body = client.get("/api/config")
assert "SEKRIT-ENGINE-TOKEN" not in body.text
llm = body.json()["llm"]
assert llm["gufo"]["base_url"] == "http://deskwork:8731"
assert llm["vllm"]["base_url"] == "http://driveripper:8000" # untouched when clean
store.close()
def wait_for_state(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]: def wait_for_state(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
deadline = time.time() + timeout deadline = time.time() + timeout
while time.time() < deadline: while time.time() < deadline:
@@ -522,13 +576,14 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None: def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
"""The old call sites still work until the remaining phases convert each page. """The old call sites still work until the remaining phases convert each page.
`/` was converted in phase 02: the overview colours from the `severity` verdicts `/` was converted in phase 02 and `/system` in phase 03: both colour themselves from
`/api/summary` already sends, so it no longer carries a threshold of its own. The the `severity` verdicts the API already sends, so they carry no threshold of their
three pages still to come must keep compiling against the fallback until they are. own. The two pages still to come must keep compiling against the fallback until they
are converted — deleting it early would break a page mid-migration.
""" """
js = client.get("/static/app.js").text js = client.get("/static/app.js").text
assert "FALLBACK ONLY" in js assert "FALLBACK ONLY" in js
for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"): for page in ("/static/page-llm.js", "/static/page-ups.js"):
assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site" assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site"
@@ -539,6 +594,198 @@ def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> Non
assert "severityFor" not in js assert "severityFor" not in js
def test_system_page_colours_from_the_payload_not_from_js_thresholds(client) -> None:
"""`page-system.js` hero cards paint payload verdicts, not thresholds of their own.
Phase 03 task 02's promise: every colour on a host card comes from a `severity` dict
`severity.py` produced (`cpu.severity.usage_pct`, `memory.severity.used_pct`,
`disks.severity.worst_used_pct`, `gpus[].severity.busy_pct`, `cpu.severity.temp_c`).
A threshold written here could disagree with the overview's alerts; a hero with no
verdict prints `n/a` uncoloured, which is `Dash.hero()`'s contract.
"""
js = client.get("/static/page-system.js").text
assert "severityFor(" not in js, "the system page re-invented a threshold"
assert "Dash.hero(" in js
for hook in ("cpu.severity.usage_pct", "memory.severity.used_pct",
"disks.severity.worst_used_pct", "gpus[].severity.busy_pct",
"cpu.severity.temp_c"):
assert hook in js, f"the card no longer names {hook} as a colour source"
def test_system_host_card_is_six_heroes_without_per_core(client) -> None:
"""Each host card leads with exactly six heroes: CPU, MEM, TEMP, DISK, GPU, NET.
Asserted on the served script because there is no JS test runner (a locked anchor).
The two dense `.metrics` rows and the per-core heat bars are exactly what made the
card unreadable, and "no per-core anywhere on /system" is a promise a stray
`renderCores` call would silently break — the series stays stored, the page just
stopped drawing it (PLAN.md *Dropped visuals*).
"""
js = client.get("/static/page-system.js").text
card = js[js.index("function hostCard("):js.index("function renderHostCard(")]
slots = re.findall(r'<div class="(?:hero|hero-slot)[^"]*" data-k="([^"]+)"', card)
assert slots == ["cpu", "mem", "temp", "disk", "gpus", "net"], \
"the hero row is no longer the six numbers in the agreed order"
# per-core visualisation is gone from the card code entirely
assert "cores" not in card, "a per-core visual came back to the card"
assert "renderCores" not in js and "core_usage" not in js
# the NET slot is a slot with two lines and no verdict: a byte rate has no good/bad
assert 'data-k="net"' in card
assert not re.search(r'data-k="net"[^>]*data-sev', card), "the NET slot invented a colour"
# what the glance leaves out stays reachable, collapsed by default
assert "More detail" in card
for key in ("load", "freq", "dr", "dw", "dutil", "uptime", "details"):
assert f'data-k="{key}"' in card, f"{key} left the page instead of the glance"
# the panel owns its charts: the slots live inside the card, not on the page
for chart in ("util", "net"):
assert f'data-chart="{chart}"' in card, f"the panel lost its {chart} chart slot"
def test_system_page_keeps_no_fleet_wide_chart_or_metric_picker(client) -> None:
"""Nothing on `/system` plots two hosts, and nothing needs a selection to make sense.
TODO.md L4 is the instruction ("The system page doesn't need shared charts for
anything") and the overview's alerts + fleet table are what made removing them safe
(phase 02). Asserted on the served bytes because the only thing that would otherwise
notice a resurrected `<select>` is a browser: there is no JS test runner (a locked
anchor), and `test_no_page_carries_an_id_nothing_reads` cannot see ids that were deleted
correctly from the page but are still queried by the script.
"""
html = _code(client.get("/system").text)
for gone in ('id="chart-cpu"', 'id="chart-mem"', 'id="chart-disk"', 'id="chart-net"',
'id="chart-gpu"', 'id="chart-temp"', 'id="chart-detail"', 'id="detail-host"',
'id="detail-metric"', 'id="f-cpu"', 'id="f-load"', 'id="f-gpu"',
"Fleet totals", "Fleet charts", "Series detail", "<select"):
assert gone not in html, f"/system still carries {gone}"
# the two panels that are left are the per-host grid and the event feed
assert html.count("<section") == 2, "/system grew a panel back on top of the host grid"
assert 'id="hosts"' in html and 'id="events"' in html
# and the script must not reach for the deleted endpoints/ids either — a `#detail-host`
# query against a removed node is a silent null, so the picker would just never appear
js = client.get("/static/page-system.js").text
for gone in ("#chart-", "detail-host", "detail-metric", "detail-note",
"renderFleet", "makeCharts", "refreshGpuChart", "populateDetailMetrics"):
assert gone not in js, f"page-system.js still references {gone}"
def test_every_system_chart_is_drawn_inside_its_own_host_panel(client) -> None:
"""One host per chart: the legends of `/system` can never list two machine names.
The construction is a factory that takes the card, so a chart cannot be built against a
page-level id; the metric list is per host, so the GPU streams cannot leak from one
panel into another; and a panel is torn down (with its charts) when its host leaves the
payload, which is the bug class a keyed-by-host chart map exists to avoid.
"""
js = client.get("/static/page-system.js").text
builds = re.findall(r"new DashCharts\.LineChart\(([^,]+),", js)
assert builds, "the system page builds no charts at all"
# every chart is built from a `[data-chart]` slot inside the host card, never from a
# page-level id — which is what makes "one host per chart" structural, not a convention
assert all(re.search(r"(?:card|slot)\.querySelector\(.\[data-chart=", b) for b in builds), \
f"a chart is built outside a host panel: {builds}"
# three construction sites, one per slot (util, net, gpu) — never one per metric
assert len(builds) == 3, f"charts are not built one per panel slot: {builds}"
assert sorted(re.findall(r'data-chart=\\?"(\w+)"', " ".join(builds))) == ["gpu", "net", "util"]
factory = js[js.index("async function renderHostCharts("):js.index("function pruneHosts(")]
assert "hosts=${encodeURIComponent(host.name)}" in factory, \
"the per-host chart query is not scoped to one host"
assert "&metrics=" in factory and "state.range" in factory, \
"a host's charts need one metric list and the selected window"
# GPU series are named from the payload's own `gpus[].index`, never discovered/guessed
assert "gpu.busy_pct@" in js and "gpu.index" in js
# the GPU chart exists only while the host reports cards
gpu_slot = js[js.index("function hostChartSet("):js.index("function seriesFor(")]
assert re.search(r"if \(!wantsGpu && set\.gpu\)", gpu_slot), \
"a host that stops reporting GPUs keeps a GPU chart it has no data for"
pruner = js[js.index("function pruneHosts("):js.index("// ---- snapshot")]
assert "state.hosts.delete(name)" in pruner and "state.charts.delete(name)" in pruner, \
"a host that left the payload keeps a live panel and drawing charts"
def test_every_system_chart_names_itself_for_assistive_tech(client) -> None:
"""A 120px SVG is silence to a screen reader unless the page names it.
The phase-03 accessibility pass (task 04) found the per-host charts drawing anonymous
plots: the legend is a sibling element and labels nothing, so the `<svg>` itself must
carry `role="img"` + `aria-label` that says which host and which metrics. Pinned on the
served bytes because there is no JS test runner (a locked anchor).
"""
charts_js = client.get("/static/charts.js").text
# the shared LineChart applies the name to the svg, and only when the caller opted in
assert "if (this.opts.ariaLabel)" in charts_js, "LineChart stopped honouring ariaLabel"
assert "setAttribute('role', 'img')" in charts_js
assert "setAttribute('aria-label', String(this.opts.ariaLabel))" in charts_js
js = client.get("/static/page-system.js").text
# all three slots name themselves, and each name starts with the host the panel belongs
# to — an aria-label that cannot say *which machine* is not an accessible name here
assert js.count("ariaLabel: `") == 3, "a per-host chart slot lost its accessible name"
for slot in ("utilisation", "network traffic", "GPU busy"):
assert f"${{host.name}} \u2014 {slot}" in js or f"${{host.name}} — {slot}" in js, \
f"the {slot} chart has no host-scoped accessible name"
def test_system_event_feed_queries_the_source_transitions_are_stored_under(client) -> None:
"""Host transitions are stored under the *dashboard* name, not the sample source.
`poller.py` logs `Event(ts, self.dashboard, …)` — host jobs write `system`, the way
the UPS job writes `ups` — while *samples* use `host`. The `/system` feed queried
`source=host` since the first commit and therefore always rendered "no transitions
yet", even with a real outage (a gateway disconnect) sitting in the store. Pinned on
the served bytes because there is no JS test runner (a locked anchor).
"""
js = client.get("/static/page-system.js").text
feed = js[js.index("async function refreshEvents("):]
url = re.search(r"fetchJSON\('(/api/events[^']*)'\)", feed)
assert url, "the /system page no longer fetches the event feed"
assert "source=system" in url.group(1), \
"the /system event feed must ask for the source the poller stores events under"
assert "source=host" not in url.group(1), \
"source=host matches no stored event — the feed is empty forever"
def test_system_footnote_prints_the_window_it_actually_measured(client) -> None:
"""`Δ<window>` is the page's promise that a rate is a measurement (invariant 2), so the
number behind it has to come from the measurement.
The `window_seconds` beside the fleet is `HOST_INTERVAL_SECONDS`, while the window behind a
rate is the estimator's `RATE_WINDOW_SECONDS` — equal at the defaults, which is why the
footnote could quote the poll interval and look right. Set `RATE_WINDOW_SECONDS=30` and it
would have labelled every rate on the page with a window it never used. The footnote now
reads each host's own measured `window_seconds`, and prints a range when they disagree.
"""
js = client.get("/static/page-system.js").text
assert "function windowLabel(" in js, "the footnote lost its measured-window helper"
footnote = js[js.index("$('#foot-note').textContent"):js.index("await Promise.all(chartJobs)")]
assert "windowLabel(" in footnote
assert "host.window_seconds" in footnote, \
"the footnote is quoting the poll interval as a measurement again"
# a footnote that says "window" and a page that says "poll" must stay two separate claims
assert "poll ${Math.round(state.pollMs / 1000)}s" in footnote
def test_no_per_core_visual_library_survives_the_page(client) -> None:
"""The per-core heat bars left `/system` and `charts.js` with them (PLAN.md *Dropped
visuals* is about what the pages draw, not what the store keeps).
`renderCores` had one caller — the hero-card per-core block deleted in task 02 — so
keeping it would be dead markup generating unstyled `.core` spans; `.cores` was the CSS
that block needed. Both are checked absent here so neither can come back as a
half-change, while `sparkline`/`LineChart` stay for phase 04 and the UPS page.
"""
assert "renderCores" not in client.get("/static/charts.js").text
css = client.get("/static/dash.css").text
assert ".cores" not in css
for asset in ASSETS:
assert "core_usage" not in _code(client.get(asset).text), f"{asset} plots per-core"
# what phase 04 and the other pages still chart from
for name in ("LineChart", "sparkline", "renderLegend"):
assert name in client.get("/static/charts.js").text
def test_overview_is_exactly_two_panels(client) -> None: def test_overview_is_exactly_two_panels(client) -> None:
"""The landing page answers one question: what is wrong, and what is the fleet doing. """The landing page answers one question: what is wrong, and what is the fleet doing.
@@ -682,7 +929,14 @@ def test_parse_range_refuses_a_window_it_cannot_interpret(raw: str) -> None:
def test_system_history_charts_every_configured_host_in_one_request( def test_system_history_charts_every_configured_host_in_one_request(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""N hosts x M metrics, one round trip — the chart endpoint's whole reason to exist.""" """N hosts x M metrics, one round trip.
`/system` no longer uses the N-hosts half of this (every chart belongs to one host and
the page issues one request per host, pinned by
`test_every_system_chart_is_drawn_inside_its_own_host_panel`); the shape is kept because
the metric-list half is exactly what the per-host query costs one request with, and
because the endpoint is a public interface for programmatic consumers.
"""
with build_client(tmp_path, monkeypatch, with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL), host_targets=(Target("driveripper", DEAD_URL),
Target("deskwork", DEAD_URL))) as test_client: Target("deskwork", DEAD_URL))) as test_client:
@@ -702,6 +956,39 @@ def test_system_history_charts_every_configured_host_in_one_request(
assert narrowed["hosts"] == ["deskwork"] assert narrowed["hosts"] == ["deskwork"]
def test_one_request_per_host_answers_every_per_host_chart(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The exact query `/system` issues per host returns every series that panel draws.
`page-system.js` sends one request per host carrying the utilisation trio
(`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct`), both network rates, and one
`gpu.busy_pct@<index>` per card the payload reported — so a name missing here is a chart
that silently reads "no samples yet" in the browser, and a renamed series in
`sources/host.py` fails here rather than in a screenshot. This is the metric-name half of
the contract; `tests/test_system_contract.py` holds the payload-path half.
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
now = time.time()
metrics = ["cpu.usage_pct", "mem.used_pct", "disk.used_pct",
"net.rx_bytes_per_s", "net.tx_bytes_per_s", "gpu.busy_pct@0"]
for metric in metrics:
store.add("host", "driveripper", metric, 12.5, ts=now - 60)
store.add("host", "driveripper", metric, 25.0, ts=now - 30)
body = test_client.get("/api/system/history", params={
"hosts": "driveripper", "metrics": ",".join(metrics), "range": "1h"}).json()
assert body["hosts"] == ["driveripper"], "one panel, one host"
assert [item["metric"] for item in body["series"]] == metrics
assert [len(item["points"]) for item in body["series"]] == [2] * len(metrics)
# a second card the host does not have answers empty, never zero and never 404
gap = test_client.get("/api/system/history",
params={"hosts": "driveripper", "metrics": "gpu.busy_pct@1"}).json()
assert gap["series"][0]["points"] == []
def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list( def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A configured-but-dead engine is listed as unreachable, not dropped (invariant 4).""" """A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
+38
View File
@@ -120,6 +120,44 @@ async def test_a_refused_connection_is_an_upstream_error_not_a_traceback() -> No
HostCollector(RateEstimator())) HostCollector(RateEstimator()))
@pytest.mark.parametrize("failure", ["refused", "timeout", "http-500"])
async def test_a_failing_host_never_echoes_its_token_in_the_message(failure: str) -> None:
"""An `UpstreamError` message is not just a log line: the poller keeps it in the health
record (`/api/health`) and writes it to the `event` table (`/api/events`), both of which
the browser reads — so the URL it names has to be the redacted one, on every failure path
(AGENTS.md invariant 6). `parse_targets` normally moves a userinfo token into a header, so
this is the backstop that keeps the guarantee true for any `Target` that still carries one.
"""
def handler(request: httpx.Request) -> httpx.Response:
if failure == "refused":
raise httpx.ConnectError("connection refused")
if failure == "timeout":
raise httpx.ReadTimeout("too slow")
return httpx.Response(500, text="boom")
target = Target("gw", "http://SEKRIT-AGENT-TOKEN@agent.test")
async with client_for(handler) as client:
with pytest.raises(UpstreamError) as excinfo:
await fetch_host_snapshot(client, target, HostCollector(RateEstimator()))
message = excinfo.value.message
assert "SEKRIT-AGENT-TOKEN" not in message, f"the token leaked through {failure}"
# …and the message still names the machine it failed to reach
assert "http://agent.test/snapshot" in message
async def test_a_failing_engine_never_echoes_credentials_in_its_url() -> None:
"""Same rule for the engines: the URL in the message is the redacted one."""
def handler(request: httpx.Request) -> httpx.Response:
raise httpx.ConnectError("connection refused")
async with client_for(handler) as client:
with pytest.raises(UpstreamError, match="unreachable") as excinfo:
await fetch_engine(client, "gufo", "http://SEKRIT-ENGINE-TOKEN@agent.test", "",
RateEstimator())
assert "SEKRIT-ENGINE-TOKEN" not in excinfo.value.message
assert "http://agent.test/metrics" in excinfo.value.message
async def test_a_slow_agent_is_a_gateway_timeout() -> None: async def test_a_slow_agent_is_a_gateway_timeout() -> None:
def handler(request: httpx.Request) -> httpx.Response: def handler(request: httpx.Request) -> httpx.Response:
raise httpx.ReadTimeout("too slow") raise httpx.ReadTimeout("too slow")
+15
View File
@@ -47,6 +47,21 @@ def test_token_in_url_becomes_a_bearer_header_and_stays_out_of_the_display_url()
assert "SEKRIT" not in target.display_url assert "SEKRIT" not in target.display_url
@pytest.mark.parametrize("raw,shown", [
("http://SEKRIT@host:9120", "http://host:9120"),
("https://user:pass@host:8000/metrics", "https://host:8000/metrics"),
("HTTP://SEKRIT@host:1", "HTTP://host:1"),
("http://host:9120", "http://host:9120"), # clean URLs pass through untouched
("", ""),
])
def test_strip_userinfo_is_the_one_redaction_every_url_leaves_by(raw: str, shown: str) -> None:
"""`strip_userinfo` is what `/api/config` and every `UpstreamError` message use, so it has
to survive the shapes a URL actually comes in: both schemes, `user:pass@`, a path, and the
common no-credentials case where it must change nothing (invariant 6).
"""
assert config.strip_userinfo(raw) == shown
def test_target_names_are_restricted_because_they_are_db_keys() -> None: def test_target_names_are_restricted_because_they_are_db_keys() -> None:
"""Names key the history DB and appear in URLs, so they are validated, not guessed. """Names key the history DB and appear in URLs, so they are validated, not guessed.
+22
View File
@@ -26,6 +26,7 @@ import os
import httpx import httpx
import pytest import pytest
from card_contract import assert_card_contract
from dashboards import collectors from dashboards import collectors
from dashboards.config import Settings, load from dashboards.config import Settings, load
from dashboards.rates import RateEstimator from dashboards.rates import RateEstimator
@@ -70,6 +71,27 @@ async def test_every_host_agent_answers_and_parses(settings: Settings,
) )
async def test_every_host_serves_the_hero_card_contract(settings: Settings,
host_collector: HostCollector) -> None:
"""The phase-03 `/system` card paths exist on the **real** agents, not just the captures.
Values may honestly be `n/a` on real hardware (gateway reports no CPU sensor and no GPU
at all); the *shape* is what phase 03 renders, so this is the test that catches an agent
upgrade renaming a field. Hermetic twin: `tests/test_system_contract.py`.
"""
if not settings.host_targets:
pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh")
async with httpx.AsyncClient(timeout=settings.host_timeout) as client:
for target in settings.host_targets:
await collectors.fetch_host_snapshot(client, target, host_collector)
await asyncio.sleep(1.2)
result = await collectors.fetch_host_snapshot(client, target, host_collector)
assert_card_contract(result.payload)
assert result.payload["cpu"]["temp_c"] == result.payload["temperatures"]["cpu_c"], (
f"{target.name}: the card and /api/summary would report different CPU temps"
)
@pytest.mark.parametrize("engine", ["gufo", "vllm"]) @pytest.mark.parametrize("engine", ["gufo", "vllm"])
async def test_engine_still_publishes_the_series_we_parse(engine: str, async def test_engine_still_publishes_the_series_we_parse(engine: str,
settings: Settings) -> None: settings: Settings) -> None:
+288
View File
@@ -0,0 +1,288 @@
"""The payload contract behind the `/system` hero cards (phase 03, task 01).
Phase 03 turns `/system` into one panel per host with six big colour-coded numbers. Before
any of it is rendered, the data has to be proven present — otherwise a card that shows `n/a`
gets "fixed" by inventing a field, and the field it invents is a number the agent never
reported. This file drives the verbatim captures in `tests/fixtures/` through
`parse_host_snapshot()` (and through a real app), so a rename breaks here rather than in a
browser at 2am (`AGENTS.md` invariant 7).
The path table itself lives in `tests/card_contract.py` so the live suite asserts the same
list against real agents (`tests/test_live_fleet.py::test_every_host_serves_the_hero_card_contract`)
and the two cannot drift apart.
What is **deliberately not on the card** — a legibility decision, not a missing feature:
per-core usage (still stored per logical CPU as `cpu.usage_pct@coreN`, see
`test_visuals_the_page_drops_are_still_stored`), swap, the process table, the raw sensor
list, and per-interface rows. Removing a plot is not removing data (PLAN.md *Dropped
visuals*), so the decision stays reversible.
Two choices are recorded here because the task asked for them in writing:
* **CPU temp comes from `cpu.temp_c`**, not `temperatures.cpu_c`. They are the same
measurement (the agent's own `cpu_temperature_celsius`) on every capture, `cpu.temp_c` is
what `/api/summary` flattens into its fleet row, and a test below keeps them equal.
* **The disk verdict lives at `disks.severity.worst_used_pct`** (phase 01 attaches severity
as a sibling dict per section, so there is no nested `disks.worst` object to hang it on).
"""
from __future__ import annotations
import copy
import math
import time
from collections.abc import Iterator
from typing import Any
import pytest
from fastapi.testclient import TestClient
from card_contract import GPU_METRICS, HOST_SHAPES, assert_card_contract, dig
from dashboards.app import create_app
from dashboards.config import Settings, Target
from dashboards.rates import RateEstimator
from dashboards.severity import classify
from dashboards.sources.host import HostCollector, parse_host_snapshot
from dashboards.store import SampleStore
#: One scrape is not a rate, so the API-level app points at a port that refuses instantly.
DEAD_URL = "http://127.0.0.1:1"
def make_collector() -> HostCollector:
return HostCollector(RateEstimator(span=60.0))
def _bump_modes(modes: dict[str, Any], delta: float) -> None:
"""Move a /proc/stat mode set so the window reads `delta` ticks, 10 % of them busy."""
if not isinstance(modes.get("total"), (int, float)):
return
modes["total"] += delta
for idle_mode, share in (("idle", 0.8), ("iowait", 0.1)):
if isinstance(modes.get(idle_mode), (int, float)):
modes[idle_mode] += delta * share
def _advance(snapshot: dict[str, Any], dt: float = 10.0) -> dict[str, Any]:
"""The same scrape `dt` later with counters moved — the second point every rate needs."""
moved = copy.deepcopy(snapshot)
moved["ts"] = float(snapshot["ts"]) + dt
_bump_modes(moved["cpu"]["modes"], 1_000.0)
for core in moved["cpu"].get("cores") or []:
_bump_modes(core.get("modes") or {}, 1_000.0)
for iface in (moved.get("network") or {}).get("interfaces") or []:
for key, value in list(iface.items()):
if key.endswith("_total") and isinstance(value, (int, float)):
iface[key] = value + 50_000.0
return moved
@pytest.fixture
def payloads(agent_snapshots: dict[str, dict[str, Any]]) -> dict[str, dict[str, Any]]:
"""Each capture parsed once, first scrape — the shape the cards must tolerate."""
out: dict[str, dict[str, Any]] = {}
for name, snapshot in agent_snapshots.items():
out[name] = parse_host_snapshot(name, snapshot, make_collector(), now=1_000.0).payload
return out
@pytest.fixture
def api_client(tmp_path: Any, agent_snapshot: dict[str, Any],
host_scrape: Any) -> Iterator[TestClient]:
"""A running app serving one host payload, so the contract is checked over HTTP too."""
db = str(tmp_path / "contract.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db, host_targets=(Target("driveripper", DEAD_URL),)),
store)
client = TestClient(app)
client.__enter__()
try:
# the job's port refuses, so a poll can never overwrite what is injected here
app.state.ctx.hub.jobs["host:driveripper"].state.payload = host_scrape(
agent_snapshot, cpu_busy=10.0)
yield client
finally:
client.__exit__(None, None, None)
store.close()
# --------------------------------------------------------------------------- #
# Every card metric has a path, on every host shape
# --------------------------------------------------------------------------- #
@pytest.mark.parametrize("name", HOST_SHAPES)
def test_every_card_metric_has_a_payload_path_on_every_host(
payloads: dict[str, dict[str, Any]], name: str) -> None:
assert_card_contract(payloads[name])
def test_the_contract_covers_every_captured_host(agent_snapshots: dict[str, dict[str, Any]],
payloads: dict[str, dict[str, Any]]) -> None:
"""A fourth capture must join `HOST_SHAPES`, or its host shape goes uncontracted."""
assert set(HOST_SHAPES) == set(payloads) == set(agent_snapshots)
def test_the_discrete_gpu_capture_supplies_real_numbers(
payloads: dict[str, dict[str, Any]], agent_snapshots: dict[str, dict[str, Any]]) -> None:
"""Not just present: correct, on the host that has all six metrics."""
payload = payloads["driveripper"]
snapshot = agent_snapshots["driveripper"]
assert payload["cpu"]["temp_c"] == snapshot["cpu_temperature_celsius"]
# the two places the CPU temperature is exposed are the same measurement, always
assert payload["temperatures"]["cpu_c"] == payload["cpu"]["temp_c"]
memory = snapshot["memory"]
assert payload["memory"]["used_pct"] == pytest.approx(
memory["used_bytes"] / memory["total_bytes"] * 100.0)
worst = max(snapshot["filesystems"], key=lambda fs: fs["used_pct"])
assert payload["disks"]["worst_mount"] == worst["mountpoint"]
assert payload["disks"]["worst_used_pct"] == pytest.approx(worst["used_pct"])
gpu = payload["gpus"][0]
assert gpu["model"]
assert gpu["busy_pct"] == pytest.approx(snapshot["gpus"][0]["busy_pct"])
assert gpu["vram_used_pct"] == pytest.approx(
gpu["vram_used_bytes"] / gpu["vram_total_bytes"] * 100.0)
# one scrape is not a rate: usage and throughput are honestly n/a until the second
assert payload["cpu"]["usage_pct"] is None
assert payload["network"]["rx_bytes_per_s"] is None
def test_measured_values_appear_on_the_second_scrape(host_scrape: Any,
agent_snapshot: dict[str, Any]) -> None:
"""The payload the API actually serves (after two scrapes) has numbers in all six places."""
payload = host_scrape(agent_snapshot, cpu_busy=33.0)
assert payload["cpu"]["usage_pct"] == pytest.approx(33.0)
assert math.isfinite(payload["network"]["rx_bytes_per_s"])
assert math.isfinite(payload["network"]["tx_bytes_per_s"])
assert payload["memory"]["used_pct"] is not None
assert payload["disks"]["worst_used_pct"] is not None
assert payload["cpu"]["temp_c"] is not None
gpu = payload["gpus"][0]
assert gpu["busy_pct"] is not None
assert gpu["vram_used_pct"] is not None
assert gpu["mem_used_pct"] is not None
def test_each_card_number_travels_with_its_own_verdict(host_scrape: Any,
agent_snapshot: dict[str, Any]) -> None:
"""The card colours a verdict from the payload; it never holds a threshold (phase 01)."""
payload = host_scrape(agent_snapshot, cpu_busy=33.0)
assert payload["cpu"]["severity"] == {
"usage_pct": classify("cpu.usage_pct", payload["cpu"]["usage_pct"]),
"temp_c": classify("cpu.temp_c", payload["cpu"]["temp_c"]),
}
assert payload["memory"]["severity"]["used_pct"] == classify(
"mem.used_pct", payload["memory"]["used_pct"])
assert payload["disks"]["severity"]["worst_used_pct"] == classify(
"disk.used_pct", payload["disks"]["worst_used_pct"])
gpu = payload["gpus"][0]
assert gpu["severity"] == {
"busy_pct": classify("gpu.busy_pct", gpu["busy_pct"]),
"mem_used_pct": classify("gpu.mem_used_pct", gpu["mem_used_pct"]),
}
# --------------------------------------------------------------------------- #
# The two host shapes that are not "a box with a discrete GPU"
# --------------------------------------------------------------------------- #
def test_the_box_with_no_gpu_reports_an_empty_list_not_a_missing_key(
payloads: dict[str, dict[str, Any]]) -> None:
"""gateway has no GPU: `gpus == []`, never a missing key and never a card at 0 % busy."""
payload = payloads["gateway"]
assert "gpus" in payload, "a host with no GPU still has the key; the page says 'no GPU'"
assert payload["gpus"] == []
# the other five numbers keep their paths, with the honest None where nothing is measured
assert_card_contract(payload)
assert payload["cpu"]["temp_c"] is None, "this agent reports no CPU sensor"
assert payload["cpu"]["severity"]["temp_c"] is None, "unknown is not ok"
def test_the_apu_reports_vram_and_the_whole_addressable_pool(
payloads: dict[str, dict[str, Any]]) -> None:
"""deskwork (Strix Halo) is why `mem_used_pct` exists: 3 % VRAM and 78 % memory, both true."""
gpu = payloads["deskwork"]["gpus"][0]
assert gpu["vram_total_bytes"] < gpu["mem_total_bytes"]
assert gpu["vram_used_pct"] < gpu["mem_used_pct"]
assert gpu["model"], "the card needs a name to put beside the numbers"
def test_paths_survive_a_host_that_reports_almost_nothing(agent_snapshot: dict[str, Any]) -> None:
"""Keys present, values None: the card must render `n/a` without a KeyError."""
snapshot = copy.deepcopy(agent_snapshot)
snapshot["cpu_temperature_celsius"] = None
snapshot["filesystems"] = []
snapshot["network"]["interfaces"] = []
snapshot["gpus"][0]["busy_pct"] = None
snapshot["gpus"][0]["vram_total_bytes"] = None
payload = parse_host_snapshot("driveripper", snapshot, make_collector(),
now=1_000.0).payload
assert_card_contract(payload)
for path in GPU_METRICS:
dig(payload["gpus"][0], path)
assert payload["cpu"]["temp_c"] is None and payload["cpu"]["severity"]["temp_c"] is None
assert payload["disks"]["worst_used_pct"] is None
assert payload["disks"]["worst_mount"] is None
assert payload["disks"]["severity"]["worst_used_pct"] is None
assert payload["network"]["rx_bytes_per_s"] is None
gpu = payload["gpus"][0]
assert gpu["busy_pct"] is None and gpu["severity"]["busy_pct"] is None
assert gpu["vram_used_pct"] is None, "no total, so no percentage — not 0 %"
assert gpu["mem_used_pct"] is None
# --------------------------------------------------------------------------- #
# What the cards drop stays in the database
# --------------------------------------------------------------------------- #
def test_visuals_the_page_drops_are_still_stored(agent_snapshot: dict[str, Any]) -> None:
"""Per-core / per-mount / per-interface / sensor / process data leaves the page.
It must still be *sampled*: PLAN.md's *Dropped visuals* anchor says the page stops
drawing what the operator stopped asking for while history keeps the series, so the
decision stays reversible.
"""
collector = make_collector()
parse_host_snapshot("driveripper", agent_snapshot, collector, now=1_000.0)
result = parse_host_snapshot("driveripper", _advance(agent_snapshot), collector,
now=1_010.0)
stored = {sample.metric for sample in result.samples}
assert result.payload["cpu"]["usage_pct"] == pytest.approx(10.0), "the counters moved"
cores = {metric for metric in stored if metric.startswith("cpu.usage_pct@core")}
logical_cpus = result.payload["info"]["cpu_cores_logical"]
assert len(cores) == logical_cpus, "one series per logical CPU, plotted or not"
assert "cpu.usage_pct@core0" in stored and f"cpu.usage_pct@core{logical_cpus - 1}" in stored
assert "disk.used_pct@/" in stored, "per-mount usage is still stored; the card shows worst"
assert any(metric.startswith("net.rx_bytes_per_s@") for metric in stored), (
"per-interface throughput is still stored; the card shows the totals")
assert "temp.max_c" in stored, "the sensor list is not on the card but stays in history"
assert "host.processes" in stored
# swap is not a card metric either; it is absent here only because this capture has no
# swap device (swap_total_bytes == 0), not because storing it stopped
assert result.payload["memory"]["swap_used_pct"] is None
assert "mem.swap_used_pct" not in stored
# --------------------------------------------------------------------------- #
# The same contract over HTTP
# --------------------------------------------------------------------------- #
def test_snapshot_endpoint_serves_the_card_paths(api_client: TestClient) -> None:
"""The cards read `/api/system/snapshot`, so the contract is checked on that body too."""
hosts = api_client.get("/api/system/snapshot").json()["hosts"]
assert len(hosts) == 1
assert_card_contract(hosts[0])
def test_history_endpoint_still_answers_for_a_per_core_metric(api_client: TestClient) -> None:
"""`cpu.usage_pct@core7` is off the page but still queryable — dropping the plot is
not dropping the series."""
store = api_client.app.state.ctx.store # type: ignore[attr-defined]
store.add("host", "driveripper", "cpu.usage_pct@core7", 42.0, time.time())
body = api_client.get("/api/system/history",
params={"metrics": "cpu.usage_pct@core7", "hosts": "driveripper",
"range": "1h"}).json()
assert body["series"], "the per-core series must still be retrievable"
assert [point["value"] for point in body["series"][0]["points"]] == [42.0]