diff --git a/.agents/phases/todo/03_system_per_host_panels/00_phase.md b/.agents/phases/complete/03_system_per_host_panels/00_phase.md similarity index 100% rename from .agents/phases/todo/03_system_per_host_panels/00_phase.md rename to .agents/phases/complete/03_system_per_host_panels/00_phase.md diff --git a/.agents/phases/todo/03_system_per_host_panels/01_host_card_payload_contract.md b/.agents/phases/complete/03_system_per_host_panels/01_host_card_payload_contract.md similarity index 100% rename from .agents/phases/todo/03_system_per_host_panels/01_host_card_payload_contract.md rename to .agents/phases/complete/03_system_per_host_panels/01_host_card_payload_contract.md diff --git a/.agents/phases/todo/03_system_per_host_panels/02_hero_cards_without_per_core.md b/.agents/phases/complete/03_system_per_host_panels/02_hero_cards_without_per_core.md similarity index 100% rename from .agents/phases/todo/03_system_per_host_panels/02_hero_cards_without_per_core.md rename to .agents/phases/complete/03_system_per_host_panels/02_hero_cards_without_per_core.md diff --git a/.agents/phases/todo/03_system_per_host_panels/03_per_host_charts_replace_shared.md b/.agents/phases/complete/03_system_per_host_panels/03_per_host_charts_replace_shared.md similarity index 100% rename from .agents/phases/todo/03_system_per_host_panels/03_per_host_charts_replace_shared.md rename to .agents/phases/complete/03_system_per_host_panels/03_per_host_charts_replace_shared.md diff --git a/.agents/phases/todo/03_system_per_host_panels/04_verify_every_host_shape.md b/.agents/phases/complete/03_system_per_host_panels/04_verify_every_host_shape.md similarity index 100% rename from .agents/phases/todo/03_system_per_host_panels/04_verify_every_host_shape.md rename to .agents/phases/complete/03_system_per_host_panels/04_verify_every_host_shape.md diff --git a/.agents/phases/todo/03_system_per_host_panels/NOTES-verify.md b/.agents/phases/todo/03_system_per_host_panels/NOTES-verify.md new file mode 100644 index 0000000..e6c4862 --- /dev/null +++ b/.agents/phases/todo/03_system_per_host_panels/NOTES-verify.md @@ -0,0 +1,66 @@ +# Verification notes — `/system` on every host shape + +**Date:** 2026-10-17 (dev loop: tunnels 9121/9122/9123 + upsd 3494 all OK; dev-server +`/api/health` all six sources `ok` before and after verification) + +## What was checked, on live data + +Verified against `http://127.0.0.1:8090/system` with all three real agents answering, in a +real browser (headless Chromium/Playwright), DOM assertions plus screenshots. + +| Host (shape) | Result | Screenshot | +| --- | --- | --- | +| `driveripper` (discrete Radeon AI PRO R9700) | GPU hero: `3 %` ok, `.sub` = "Radeon AI PRO R9700 · VRAM 30.5 GiB / 31.9 GiB · pool 33 %"; util/net/GPU charts all present inside the panel | `system-driveripper.png` (dark), `system-driveripper-light.png` | +| `deskwork` (Strix Halo APU, two memory pools) | GPU hero busy went 82 % ok → 85 % warn → 99 % fail across the session, so threshold colour visibly moves; `.sub` = "Radeon 8060S Graphics · VRAM 172.7 MiB / 512.0 MiB · pool 78 %" — small dedicated VRAM and high shared pool read as two facts, not a contradiction; MEM 91 % and TEMP ~83 °C amber | `system-deskwork.png` | +| `gateway` (no GPU, no CPU temp sensor) | CPU TEMP `n/a` with **no** severity class; GPU slot = `n/a` + "No GPU reported by this agent", uncoloured; **no** GPU chart and no hole where one would be | `system-gateway.png` (dark), `system-gateway-light.png` | + +Full-page shapes: `system-shapes-dark.png`, `system-shapes-light.png`. + +Failure path: stopped only the gateway tunnel (same commands `dev-tunnels.sh stop` runs, +scoped to the 9123 entry; remote agent stopped too). After 15 poll failures the panel kept +its last-good heroes (MEM 32 %, DISK 61 %, NET rates — nothing zeroed, nothing `n/a`), its +dot turned red, the header read `stale 2.1m`, and the card dimmed via `.stale`. The other +two panels stayed green ("one dead source must not blink another"). Store recorded exactly +one `unreachable` and, after `./scripts/dev-tunnels.sh start`, one `reachable` event. +Screenshot: `system-stale-gateway.png` (+ `system-stale-overview.png` full page). Restored; +`dev-tunnels.sh status` and `/api/health` back to all-OK. + +Accessibility pass: + +- hero values are real selectable text (`div.value > span.num/span.unit`, no SVG paths) — + asserted in-page for every hero on all three panels; +- colour is never the only signal: each hero prints the number, the NET slot is + deliberately colourless, staleness reads as text (`updated Ns` / `stale 2.1m`); +- charts — **defect found**: the SVGs were anonymous (`aria-label` absent on container and + svg). Fixed: `LineChart` gained an `ariaLabel` option that sets `role="img"` + + `aria-label` on the ``, and every `/system` chart now names its host and metrics + (e.g. "driveripper — utilisation over the selected window: CPU, memory and worst disk in + per cent"). Pinned by `tests/test_api.py::test_every_system_chart_names_itself_for_assistive_tech`. + The `/llm` and `/ups` charts can adopt the same option in phases 04/05. + +Both themes checked via the header toggle (severity colours come from theme tokens; the +light-theme deskwork fail-red and warn-amber read clearly — see +`system-shapes-light.png`, `system-gateway-light.png`). Note for the next shooter: set +`prefers-color-scheme` or seed `localStorage['dash-theme']`; Playwright defaults to light, +which silently mislabels a "dark" screenshot. + +`curl "localhost:8090/api/system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h"` +→ 354 points: dropping the per-core plot did not drop the data. + +## Defects found and fixed during verification + +1. **Unlabelled charts** (above) — fixed in `charts.js` + `page-system.js`, test-pinned, + re-screenshotted. +2. **`/system` Events panel was permanently empty** — the poller stores transition events + under the *dashboard* name (`system`) while the page queried `/api/events?source=host`, + so the feed said "No host transitions recorded yet" even with the deliberate gateway + outage in the store. Fixed to `source=system` (with a comment on the naming trap); + `tests/test_api.py::test_system_event_feed_queries_the_source_transitions_are_stored_under` + pins it; the re-shot full-page screenshots show the feed populated with the outage and + recovery. Pre-existing since the first commit, not introduced by phase 03. + +## After the fixes + +`bash .agents/validate.sh` → exit 0: 460 passed + 7 skipped, coverage 96.86 % (≥ 94), +ruff + pyright clean, `node --check` clean. `uv run pytest -m live --live` → 7 passed. +Console stayed error-free throughout. diff --git a/.agents/reports/03_system_per_host_panels/03_system_per_host_panels__00_phase.a1.err b/.agents/reports/03_system_per_host_panels/03_system_per_host_panels__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/03_system_per_host_panels/03_system_per_host_panels__00_phase.a1.md b/.agents/reports/03_system_per_host_panels/03_system_per_host_panels__00_phase.a1.md new file mode 100644 index 0000000..512cb26 --- /dev/null +++ b/.agents/reports/03_system_per_host_panels/03_system_per_host_panels__00_phase.a1.md @@ -0,0 +1,10 @@ +**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.** + +- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `
-
-
Hosts answering
-
n/a
-
CPU average
-
n/a%
-
Load 1 (sum)
-
n/a
-
Memory in use
-
n/a
-
Worst filesystem
-
n/a%
-
Network in
-
n/a
-
Network out
-
n/a
-
GPU busy (avg)
-
n/a%
-
Power
-
n/a
-
-
- - -
-
-

CPU utilisation

% · per host
-
-
-
-

Memory used

% of MemAvailable basis
-
-
-
-

Disk I/O throughput

- selected host · all physical devices summed
-
-
-
-

Network throughput

selected host · physical interfaces
-
-
-
-

GPU utilisation

one series per host GPU
-
-
-
-

Temperature

CPU package · °C
-
-
-
- -
-
-

Detail

- - - - - -
-
-
-
-
-
-
- -
+ +

Per host

- agent-reported values; rates measured over the window in each header
+ agent-reported values; rates are measured over the Δ window in the + footer, and each panel's charts cover only that host
@@ -123,7 +59,7 @@
diff --git a/src/dashboards/web/static/app.js b/src/dashboards/web/static/app.js index 2d56d2e..659507f 100644 --- a/src/dashboards/web/static/app.js +++ b/src/dashboards/web/static/app.js @@ -110,9 +110,9 @@ const Dash = (() => { * * Thresholds live in `src/dashboards/severity.py` and travel in the payload as * `severity`; `severityFrom()` below is how a page reads them. This function survives - * only because `page-system.js`, `page-llm.js` and `page-ups.js` still call it with - * their own numbers (phases 03–05 convert them page by page, so each page's change is - * reviewable on its own — `/` was already converted in phase 02). Do not add a new call + * only because `page-llm.js` and `page-ups.js` still call it with their own numbers + * (phases 04–05 convert them page by page, so each page's change is reviewable on its + * own — `/` was converted in phase 02, `/system` in phase 03). Do not add a new call * site: a threshold written in JS is a threshold that can disagree with the other pages. */ function severityFor(value, warnAt, failAt) { diff --git a/src/dashboards/web/static/charts.js b/src/dashboards/web/static/charts.js index 6ac1c4b..0f0ba10 100644 --- a/src/dashboards/web/static/charts.js +++ b/src/dashboards/web/static/charts.js @@ -43,7 +43,7 @@ const DEFAULT_FORMAT = (v) => (Math.abs(v) >= 100 ? v.toFixed(0) : v.toFixed(1)) class LineChart { /** * container: element to fill (its width is used; height comes from opts.height) - * opts: {height, yFormat, xSpan, yMin, yMax, area, legend, showAxis, min, unit} + * opts: {height, yFormat, xSpan, yMin, yMax, area, legend, showAxis, min, unit, ariaLabel} */ constructor(container, opts) { this.el = typeof container === 'string' ? document.querySelector(container) : container; @@ -69,6 +69,13 @@ class LineChart { this.svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg'); this.svg.setAttribute('width', '100%'); this.svg.setAttribute('height', this.opts.height); + // A chart is an image to assistive tech, which means silence unless the page names it — + // the legend below is a sibling, not a label. `ariaLabel` is optional so pages adopt it + // one at a time (`/system` names every chart it draws; see page-system.js). + if (this.opts.ariaLabel) { + this.svg.setAttribute('role', 'img'); + this.svg.setAttribute('aria-label', String(this.opts.ariaLabel)); + } this.el.appendChild(this.svg); this.empty = document.createElement('div'); this.empty.className = 'empty'; @@ -337,17 +344,6 @@ function renderLegend(container, series, format, showPeak) { }).join(''); } -/** Column heat bars for per-core usage: 48 cores read as a pattern, not a table. */ -function renderCores(container, values, format) { - const f = format || ((v) => (v == null ? 'n/a' : v.toFixed(0) + '%')); - container.innerHTML = (values || []).map((v, i) => { - const pct = v == null ? 0 : Math.max(0, Math.min(100, v)); - const hue = pct > 85 ? 'var(--fail)' : pct > 60 ? 'var(--warn)' : 'var(--accent)'; - return `` + - ``; - }).join(''); -} - /** Small non-interactive sparkline for cards where a full chart would shout. */ function sparkline(container, points, opts) { const o = Object.assign({ height: 34, color: cssVar('--series-1'), min: null, max: null }, opts || {}); @@ -374,5 +370,5 @@ function sparkline(container, points, opts) { container.innerHTML = svg; } -window.DashCharts = { LineChart, niceTicks, renderLegend, renderCores, sparkline, cssVar, +window.DashCharts = { LineChart, niceTicks, renderLegend, sparkline, cssVar, CHART_COLORS, DEFAULT_FORMAT, escapeHtml }; diff --git a/src/dashboards/web/static/dash.css b/src/dashboards/web/static/dash.css index 96b11d6..f3017d6 100644 --- a/src/dashboards/web/static/dash.css +++ b/src/dashboards/web/static/dash.css @@ -282,6 +282,27 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; } /* with no verdict on either the hero or the bar the fill stays the neutral accent: the bar shows *how much*, never a judgement nobody measured. */ +/* The per-host hero row: six glanceable numbers — CPU, MEM, TEMP, DISK, GPU, NET — in one + line on a wide card, wrapping to 3×2 and then 2×3 as the card narrows. Tracks are a + `minmax` of the widest plain hero value (`100 %`), so a row never squints into ellipsis. */ +.heroes { + display: grid; gap: .9rem 1rem; + grid-template-columns: repeat(auto-fit, minmax(min(118px, 100%), 1fr)); +} +/* a slot that holds more than one hero (one per GPU card) without breaking the row */ +.hero-slot { display: grid; gap: .7rem; min-width: 0; align-content: start; } +/* Network is one hero with two lines: in and out are one wire seen in two directions, and + two big numbers side by side read as a competition. It is deliberately colourless — + a byte rate has no good/bad, so no verdict ever paints here (severity.py has no net.* + row); `n/a` comes from `.na` and is as faint as any other unmeasured value. */ +.hero .hero-lines { + display: grid; gap: .15rem; + font-family: var(--mono); font-size: 1.28rem; font-weight: 650; line-height: 1.25; + font-variant-numeric: tabular-nums; /* same fixed digit width as .hero .value */ + color: var(--text); +} +.hero .hero-lines .dir { color: var(--text-muted); } + /* ---- bars, dots, chips -------------------------------------------------- */ .bar { position: relative; height: 7px; border-radius: 999px; background: var(--bg-sunken); @@ -357,9 +378,15 @@ table.data td.fail { color: var(--fail); } table.data.scroll { display: block; overflow: auto; max-height: 320px; } /* ---- charts ------------------------------------------------------------- */ +/* The chart vocabulary every page draws with. `LineChart` puts the y tick labels in the + `grid` group and the x labels in `axis`, so both groups need the same text style — the + grid text used to inherit SVG's black fill, which meant the numbers beside a utilisation + chart were invisible on the dark theme. */ .chart { position: relative; width: 100%; } .chart svg { display: block; width: 100%; height: 100%; overflow: visible; } -.chart .axis text { fill: var(--text-faint); font-family: var(--mono); font-size: 10px; } +.chart .axis text, .chart .grid text { + fill: var(--text-faint); font-family: var(--mono); font-size: 10px; +} .chart .axis line, .chart .grid line { stroke: var(--border); stroke-width: 1; } .chart .grid line { stroke-dasharray: 2 4; opacity: .75; } .chart .zero line { stroke: var(--border-strong); } @@ -375,6 +402,20 @@ table.data.scroll { display: block; overflow: auto; max-height: 320px; } .legend .swatch { width: 9px; height: 9px; border-radius: 2px; } .legend .lv { font-family: var(--mono); color: var(--text); } +/* The per-host chart stack inside a `/system` panel: supporting evidence under a number, + not the main event. One utilisation chart, one network chart, and a GPU chart only for a + host that reports cards (`page-system.js` adds and removes that slot with `host.gpus`), + each with its legend glued underneath. They are stacked rather than side by side because + a 200px-wide plot cannot show an hour of CPU without its axis labels colliding — and no + chart in here ever contains a second host. */ +.host-charts { display: grid; gap: .7rem; } +.host-chart { display: grid; gap: .15rem; min-width: 0; } +/* `sm` is the slot's floor before `LineChart` writes its measured height inline, so an + empty panel is the height it is about to be rather than a collapsing card. */ +.chart.sm { min-height: 120px; } +.host-chart .legend { font-size: 10.5px; gap: .2rem .6rem; } +.host-chart .legend .swatch { width: 7px; height: 7px; } + .tooltip { position: absolute; z-index: 30; pointer-events: none; opacity: 0; background: color-mix(in srgb, var(--bg-raised) 95%, transparent); @@ -386,11 +427,6 @@ table.data.scroll { display: block; overflow: auto; max-height: 320px; } .tooltip .row { display: flex; align-items: center; gap: .35rem; } .tooltip .row .val { margin-inline-start: auto; font-family: var(--mono); } -/* per-core usage heat bars */ -.cores { display: grid; grid-template-columns: repeat(auto-fill, minmax(9px, 1fr)); gap: 2px; } -.cores .core { height: 22px; border-radius: 2px; background: var(--bg-sunken); position: relative; } -.cores .core > i { position: absolute; bottom: 0; left: 0; right: 0; background: var(--accent); border-radius: 2px; } - /* ---- lists -------------------------------------------------------------- */ ul.events { list-style: none; margin: 0; padding: 0; display: grid; gap: .1rem; } ul.events li { @@ -497,11 +533,13 @@ dialog footer { padding: .7rem .9rem; border-top: 1px solid var(--border); displ .stale { opacity: .62; } -select, input[type="text"] { +/* Only the UPS confirmation dialog takes text now — `/system`'s host/metric pickers were the + last `select` on the app and left with the shared charts, along with the `label.field` caption + that sat above them. */ +input[type="text"] { background: var(--bg-sunken); color: var(--text); border: 1px solid var(--border); border-radius: var(--radius-sm); padding: .3rem .45rem; font: 500 12.5px var(--sans); } -label.field { font-size: 11px; text-transform: uppercase; letter-spacing: .05em; color: var(--text-faint); } footer.pagefoot { margin-top: 1.6rem; padding-top: .8rem; border-top: 1px solid var(--border); diff --git a/src/dashboards/web/static/page-system.js b/src/dashboards/web/static/page-system.js index 207ceb5..58c271f 100644 --- a/src/dashboards/web/static/page-system.js +++ b/src/dashboards/web/static/page-system.js @@ -2,71 +2,43 @@ * * Host cards are created once and patched in place (keyed by host name) — rebuilding the * DOM every 10s kills hover, transitions, and any scroll position inside the tables. - * Charts get their history from /api/system/history: one request per chart, N hosts each. + * + * Every chart on this page belongs to exactly one host and is drawn inside that host's own + * panel (`/api/system/history`, one request per host per refresh). Nothing here plots two + * machines on one axis and there is no host/metric picker: "which machine stands out?" is + * the overview's job (alerts + fleet table), and `/system` answers "what is this machine + * doing?". See the per-host chart section below. */ 'use strict'; -const { $, $$, fetchJSON, setTxt, setValue, setBar, setDot, bytes, pct, num, duration, poll, - staleness, markStale, severityFor } = Dash; +const { $, fetchJSON, setTxt, setDot, bytes, num, duration, poll, + markStale } = Dash; const state = { range: 3600, - detailHost: null, - detailMetric: 'cpu.usage_pct', hosts: new Map(), // name -> card element - charts: {}, + charts: new Map(), // name -> { util, net, gpu|null } LineChart instances poller: null, pollMs: 10000, - metricList: [], }; -const DETAIL_METRICS = [ - ['cpu.usage_pct', 'CPU utilisation', 'pct'], - ['cpu.user_pct', 'CPU user', 'pct'], - ['cpu.system_pct', 'CPU system', 'pct'], - ['cpu.iowait_pct', 'CPU iowait', 'pct'], - ['cpu.steal_pct', 'CPU steal', 'pct'], - ['cpu.load1', 'Load average (1m)', 'num'], - ['cpu.load15', 'Load average (15m)', 'num'], - ['cpu.freq_mhz', 'CPU frequency', 'mhz'], - ['cpu.context_switches_per_s', 'Context switches', 'num'], - ['mem.used_pct', 'Memory used', 'pct'], - ['mem.used_bytes', 'Memory used', 'bytes'], - ['mem.available_bytes', 'Memory available', 'bytes'], - ['mem.swap_used_pct', 'Swap used', 'pct'], - ['disk.used_pct', 'Filesystem used (worst)', 'pct'], - ['disk.read_bytes_per_s', 'Disk read', 'bytes'], - ['disk.write_bytes_per_s', 'Disk write', 'bytes'], - ['disk.util_pct', 'Disk busy', 'pct'], - ['net.rx_bytes_per_s', 'Network in', 'bytes'], - ['net.tx_bytes_per_s', 'Network out', 'bytes'], - ['temp.cpu_c', 'CPU temperature', 'temp'], - ['temp.nvme_max_c', 'NVMe temperature', 'temp'], - ['host.uptime_seconds', 'Uptime', 'dur'], - ['host.processes', 'Process count', 'num'], -]; - -const FORMATS = { - pct: { fn: (v) => num(v, 1), unit: '%', axis: (v) => num(v, 0) }, - bytes: { fn: (v) => bytes(v, 1), unit: 'B/s', axis: (v) => (Dash.fmtBytes(v, 0) || { text: '0' }).text + - ((Dash.fmtBytes(v, 0) || { unit: '' }).unit) }, - mhz: { fn: (v) => num(v, 0) + ' MHz', unit: 'MHz', axis: (v) => num(v, 0) }, - temp: { fn: (v) => num(v, 1) + ' °C', unit: '°C', axis: (v) => num(v, 0) }, - dur: { fn: (v) => duration(v), unit: '', axis: (v) => duration(v) }, - num: { fn: (v) => num(v, 1), unit: '', axis: (v) => num(v, 0) }, -}; - -function formatFor(kind) { return FORMATS[kind] || FORMATS.num; } - -function metricKind(name) { - if (name.endsWith('_pct')) return 'pct'; - if (name.endsWith('_bytes') || name.includes('bytes_per_s')) return 'bytes'; - if (name.includes('_c') && name.startsWith('temp.')) return 'temp'; - if (name === 'host.uptime_seconds') return 'dur'; - return 'num'; -} - // ---- host cards ----------------------------------------------------------- +// +// Six heroes per host, in one fixed order: CPU, MEM, TEMP, DISK, GPU, NET. Every colour +// comes from the payload's `severity` dicts — `severity.py` owns the thresholds, this file +// compares no number to any bound — and `Dash.hero()` prints `n/a` with **no** colour +// class when the number or its verdict is missing, so gateway's absent CPU sensor cannot +// borrow last poll's green. +// +// `data-sev` names the verdict each slot paints; tests/test_api.py pins the six slots and +// their order on the served bytes. The NET slot carries none on purpose: `network` has no +// `severity` (a byte rate has no good/bad), so nothing here may colour it. +// +// Per-core usage is not plotted here or anywhere on this page — the series is still +// stored as `cpu.usage_pct@coreN`, the page just stopped drawing a pattern nobody reads. +// Everything that is not glance-worthy (load averages, clock, disk read/write/busy, +// uptime, GPU power and temperature, filesystems, top processes) lives in "More detail", +// collapsed by default: the TODO asked for a simpler surface, not a less complete page. function hostCard(name) { const card = document.createElement('article'); card.className = 'card'; @@ -80,38 +52,110 @@ function hostCard(name) {
-
-
CPU
n/a%
-
-
Load 1 / 15
n/a
-
CPU temp
n/a°C
-
Clock
n/aGHz
-
Memory
n/a
-
-
Worst fs
n/a
-
+
+
+
+
+
+
+
+
Net
+
+
↓n/a
+
↑n/a
+
+
-
-
Per-core
-
+
+
+
+
+
+
+
+
+
-
-
Disk read
n/a
-
Disk write
n/a
-
Disk busy
n/a%
-
Net in
n/a
-
Net out
n/a
-
Up
n/a
-
-
- Filesystems & processes + More detail +
+
Load 1 / 5 / 15
n/a
+
Clock
n/a
+
Disk read
n/a
+
Disk write
n/a
+
Disk busy
n/a
+
Up
n/a
+
+
`; return card; } +/** `200 / 253 GiB` — the percentage is the verdict, the bytes are the detail. */ +function memorySub(mem) { + const used = Dash.fmtBytes(mem.used_bytes, 1); + const total = Dash.fmtBytes(mem.total_bytes, 0); + if (!used || !total) return null; + if (used.unit === total.unit) return `${used.text} / ${total.text} ${total.unit}`; + return `${bytes(mem.used_bytes, 1)} of ${bytes(mem.total_bytes, 0)}`; +} + +/** Model, VRAM used / total, and the honest APU pool view: a Strix Halo's model lives in + * GTT, so "VRAM 3%" and "pool 78%" are both true and the card shows both, not whichever + * flatters the machine. */ +function gpuSub(gpu) { + const bits = [gpu.model || 'GPU']; + bits.push(gpu.vram_used_bytes == null || gpu.vram_total_bytes == null + ? 'VRAM n/a' + : `VRAM ${bytes(gpu.vram_used_bytes, 1)} / ${bytes(gpu.vram_total_bytes, 1)}`); + if (gpu.mem_used_pct != null) bits.push(`pool ${num(gpu.mem_used_pct, 0)}%`); + return bits.join(' · '); +} + +/** The GPU slot of the hero row: one hero per card, or the honest no-GPU state. A box + * with no card is not a box at 0% busy, and gateway must not look broken for it. */ +function renderGpuSlot(cell, gpus) { + if (cell.dataset.count !== String(gpus.length)) { + cell.innerHTML = ''; + for (let i = 0; i < Math.max(gpus.length, 1); i += 1) { + const row = document.createElement('div'); + row.className = 'hero hero-bar'; + cell.appendChild(row); + } + cell.dataset.count = String(gpus.length); + } + const rows = Array.from(cell.children); + if (!gpus.length) { + Dash.hero(rows[0], { label: 'GPU', value: null, sub: 'No GPU reported by this agent' }); + return; + } + gpus.forEach((gpu, i) => { + Dash.hero(rows[i], { + label: gpus.length > 1 ? `GPU ${i}` : 'GPU', + value: gpu.busy_pct, unit: '%', digits: 0, + severity: (gpu.severity || {}).busy_pct, bar: true, + sub: gpuSub(gpu), + }); + }); +} + +/** What the GPU hero left out, in "More detail": power, temperature, both memory pools. */ +function renderGpuExtra(cell, gpus) { + if (!gpus.length) { cell.innerHTML = ''; return; } + cell.innerHTML = ` +
GPU detail
+ + + ${gpus.map((gpu) => ` + + + + + `).join('')}
CardPowerTempVRAMGTT
${DashCharts.escapeHtml(gpu.model || 'GPU')}${gpu.power_watts == null ? 'n/a' : `${num(gpu.power_watts, 0)} W`}${gpu.temp_c == null ? 'n/a' : `${num(gpu.temp_c, 0)} °C`}${bytes(gpu.vram_used_bytes, 1)} / ${bytes(gpu.vram_total_bytes, 1)}${bytes(gpu.gtt_used_bytes, 1)}
`; +} + function renderHostCard(card, host) { const q = (key) => card.querySelector(`[data-k="${key}"]`); const stateRow = host.state || {}; @@ -125,65 +169,41 @@ function renderHostCard(card, host) { markStale(card, stateRow.ok === false || Dash.ageSeconds(stateRow.last_ok) > 5 * (host.window || 10)); const cpu = host.cpu || {}; - setValue(q('cpu'), cpu.usage_pct, 0); - setBar(card.querySelector('[data-bar="cpu"]'), cpu.usage_pct, severityFor(cpu.usage_pct, 75, 90)); - setTxt(q('load'), cpu.load1 == null ? 'n/a' : `${num(cpu.load1, 2)} / ${num(cpu.load15, 2)}`); - setValue(q('temp'), cpu.temp_c, 1); - setValue(q('freq'), cpu.freq_mhz == null ? null : cpu.freq_mhz / 1000, 2); - const mem = host.memory || {}; - const memText = Dash.fmtBytes(mem.used_bytes, 1); - const memTotal = Dash.fmtBytes(mem.total_bytes, 0); - setTxt(q('mem'), memText ? memText.text : 'n/a'); - setTxt(q('mem').nextElementSibling, memText ? `${memText.unit}/${memTotal ? memTotal.text + memTotal.unit : ''}` : ''); - setBar(card.querySelector('[data-bar="mem"]'), mem.used_pct, severityFor(mem.used_pct, 80, 92)); - const disks = host.disks || {}; - setTxt(q('disk'), disks.worst_used_pct == null ? 'n/a' : num(disks.worst_used_pct, 0) + '%'); - setTxt(q('disk').nextElementSibling, disks.worst_mount ? ' ' + disks.worst_mount : ''); - setBar(card.querySelector('[data-bar="disk"]'), disks.worst_used_pct, severityFor(disks.worst_used_pct, 80, 92)); + const net = host.network || {}; - DashCharts.renderCores(q('cores'), cpu.core_usage_pct || []); + // The six heroes. The verdict is whatever the payload carried (`severity` dicts written + // by severity.py, pinned by tests/test_system_contract.py); a missing number or a + // missing verdict prints `n/a` with no colour class at all. + Dash.hero(q('cpu'), { label: 'CPU', value: cpu.usage_pct, unit: '%', digits: 0, + severity: (cpu.severity || {}).usage_pct, bar: true }); + Dash.hero(q('mem'), { label: 'Memory', value: mem.used_pct, unit: '%', digits: 0, + severity: (mem.severity || {}).used_pct, bar: true, + sub: memorySub(mem) }); + Dash.hero(q('temp'), { label: 'CPU temp', value: cpu.temp_c, unit: '°C', digits: 1, + severity: (cpu.severity || {}).temp_c }); + // the mount is the `.sub` because "80 %" without "of what" is not actionable + Dash.hero(q('disk'), { label: 'Disk', value: disks.worst_used_pct, unit: '%', digits: 0, + severity: (disks.severity || {}).worst_used_pct, bar: true, + sub: disks.worst_mount }); + renderGpuSlot(q('gpus'), host.gpus || []); + // One wire seen in two directions, two lines, never two competing numbers — and a null + // (no physical interface reported) stays `n/a` rather than becoming a confident 0 B/s. + // No severity by design: `network` carries none, so this slot never colours. + setTxt(q('rx'), net.rx_bytes_per_s == null ? null : Dash.rate(net.rx_bytes_per_s)); + setTxt(q('tx'), net.tx_bytes_per_s == null ? null : Dash.rate(net.tx_bytes_per_s)); - setValue(q('dr'), disks.read_bytes_per_s == null ? null : disks.read_bytes_per_s, null); - if (disks.read_bytes_per_s != null) { - const parts = Dash.fmtBytes(disks.read_bytes_per_s, 1); - setTxt(q('dr'), parts.text); setTxt(q('dr').nextElementSibling, parts.unit + '/s'); - const w = Dash.fmtBytes(disks.write_bytes_per_s, 1); - setTxt(q('dw'), w ? w.text : 'n/a'); setTxt(q('dw').nextElementSibling, w ? w.unit + '/s' : ''); - } else { setTxt(q('dr'), 'n/a'); setTxt(q('dw'), 'n/a'); } - setValue(q('dutil'), disks.util_pct, 0); - const rx = Dash.fmtBytes((host.network || {}).rx_bytes_per_s, 1); - const tx = Dash.fmtBytes((host.network || {}).tx_bytes_per_s, 1); - setTxt(q('rx'), rx ? rx.text : 'n/a'); setTxt(q('rx').nextElementSibling, rx ? rx.unit + '/s' : ''); - setTxt(q('tx'), tx ? tx.text : 'n/a'); setTxt(q('tx').nextElementSibling, tx ? tx.unit + '/s' : ''); - setTxt(q('uptime'), duration((host.info || {}).uptime_seconds)); - - // GPUs: one row each, with both memory pools named (an APU's model lives in GTT) - const gpus = host.gpus || []; - const gpuHost = q('gpus'); - if (!gpus.length) { - gpuHost.innerHTML = '
No GPU reported by this agent
'; - } else { - gpuHost.innerHTML = gpus.map((gpu) => ` -
-
- ${DashCharts.escapeHtml(gpu.model || 'GPU')} - - ${gpu.busy_pct == null ? 'n/a' : num(gpu.busy_pct, 0) + '% busy'} · - ${gpu.power_watts == null ? 'power n/a' : num(gpu.power_watts, 0) + ' W'} · - ${gpu.temp_c == null ? 'temp n/a' : num(gpu.temp_c, 0) + '°C'} -
-
-
- VRAM ${gpu.vram_used_bytes == null ? 'n/a' : bytes(gpu.vram_used_bytes, 0)} - ${gpu.vram_total_bytes ? 'of ' + bytes(gpu.vram_total_bytes, 0) : ''} - shared (GTT) ${gpu.gtt_used_bytes == null ? 'n/a' : bytes(gpu.gtt_used_bytes, 0)} - ${gpu.mem_used_pct == null ? '' : ' · pool ' + num(gpu.mem_used_pct, 0) + '%'} -
-
-
`).join(''); - } + // "More detail": everything the glance does not need but the page still owes. + setTxt(q('load'), cpu.load1 == null && cpu.load5 == null && cpu.load15 == null + ? null : `${num(cpu.load1, 2)} / ${num(cpu.load5, 2)} / ${num(cpu.load15, 2)}`); + setTxt(q('freq'), cpu.freq_mhz == null ? null : `${num(cpu.freq_mhz / 1000, 2)} GHz`); + setTxt(q('dr'), disks.read_bytes_per_s == null ? null : Dash.rate(disks.read_bytes_per_s)); + setTxt(q('dw'), disks.write_bytes_per_s == null ? null : Dash.rate(disks.write_bytes_per_s)); + setTxt(q('dutil'), disks.util_pct == null ? null : `${num(disks.util_pct, 0)}%`); + const up = (host.info || {}).uptime_seconds; + setTxt(q('uptime'), up == null ? null : duration(up)); + renderGpuExtra(q('gpu-extra'), host.gpus || []); const filesystems = (disks.filesystems || []).filter((fs) => fs.used_pct != null); const procs = host.processes || {}; @@ -226,180 +246,175 @@ function stalenessText(row) { return seconds > 30 ? `stale ${duration(seconds)}` : `updated ${seconds < 2 ? 'now' : duration(seconds)}`; } -// ---- charts --------------------------------------------------------------- -function makeCharts() { - const common = { height: 165, xSpan: state.range }; - state.charts.cpu = new DashCharts.LineChart('#chart-cpu', Object.assign({}, common, { - yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) })); - state.charts.mem = new DashCharts.LineChart('#chart-mem', Object.assign({}, common, { - yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) })); - state.charts.disk = new DashCharts.LineChart('#chart-disk', Object.assign({}, common, { - yMin: 0, yFormat: (v) => (Dash.fmtBytes(v, 0) || { text: '0' }).text })); - state.charts.net = new DashCharts.LineChart('#chart-net', Object.assign({}, common, { - yMin: 0, yFormat: (v) => (Dash.fmtBytes(v, 0) || { text: '0' }).text })); - state.charts.gpu = new DashCharts.LineChart('#chart-gpu', Object.assign({}, common, { - yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) })); - state.charts.temp = new DashCharts.LineChart('#chart-temp', Object.assign({}, common, { - yFormat: (v) => num(v, 0) })); - state.charts.detail = new DashCharts.LineChart('#chart-detail', Object.assign({}, common, { - height: 200, yFormat: (v) => num(v, 1) })); -} +// ---- per-host charts ------------------------------------------------------ +// +// One host, one panel, at most three charts — and no chart on this page contains a second +// host. Six metrics x three hosts as six separate charts is the unreadable version of +// "per-host charts", so the metrics that share a unit share an axis: +// +// util cpu.usage_pct + mem.used_pct + disk.used_pct one 0-100 % axis +// net net.rx_bytes_per_s + net.tx_bytes_per_s one byte-rate axis +// gpu gpu.busy_pct@ per card the agent reports (no chart when there is none) +// +// Each host costs **one** `/api/system/history` request per refresh: the endpoint takes a +// `hosts=` and a `metrics=` list, so the XHR budget for a three-host fleet is one snapshot +// plus three history calls plus the event feed, whatever the chart count. GPU stream names +// come from the payload's own `gpus[].index` (`sources/host.py` writes `gpu.busy_pct@`), +// so nothing here has to discover them through /api/metrics the way the deleted +// fleet-wide GPU chart did. +// +// The charts live in a Map keyed by host name and are torn down with the card: a panel for +// a host that has left the payload would otherwise keep drawing its last series forever. +const CHART_HEIGHT = 120; +const UTIL_SERIES = [ + ['cpu.usage_pct', 'CPU'], + ['mem.used_pct', 'Mem'], + ['disk.used_pct', 'Disk (worst)'], +]; +const NET_SERIES = [ + ['net.rx_bytes_per_s', '↓ in'], + ['net.tx_bytes_per_s', '↑ out'], +]; +const PCT_FORMAT = (v) => `${num(v, 1)}%`; +const RATE_FORMAT = (v) => `${bytes(v, 1)}/s`; +const AXIS_PCT = (v) => num(v, 0); +const AXIS_RATE = (v) => { + const parts = Dash.fmtBytes(v, 0) || { text: '0', unit: '' }; + return `${parts.text}${parts.unit}`; +}; -function setChartSpan(span) { - Object.values(state.charts).forEach((chart) => { chart.opts.xSpan = span; }); -} - -function seriesFor(response, metric) { - return (response.series || []).filter((s) => s.metric === metric).map((s) => ({ - name: s.host, points: s.points.map((p) => ({ ts: p.ts, value: p.value })), - })); -} - -async function refreshCharts(snapshot) { - const hostNames = (snapshot.hosts || []).map((h) => h.name); - const detailHost = state.detailHost || hostNames[0] || null; - const span = state.range; - const requests = { - cpu: fetchJSON(`/api/system/history?metrics=cpu.usage_pct&range=${span}`).catch(() => null), - mem: fetchJSON(`/api/system/history?metrics=mem.used_pct&range=${span}`).catch(() => null), - temp: fetchJSON(`/api/system/history?metrics=temp.cpu_c&range=${span}`).catch(() => null), - disk: detailHost ? fetchJSON(`/api/system/history?hosts=${encodeURIComponent(detailHost)}&` + - `metrics=disk.read_bytes_per_s,disk.write_bytes_per_s&range=${span}`).catch(() => null) : null, - net: detailHost ? fetchJSON(`/api/system/history?hosts=${encodeURIComponent(detailHost)}&` + - `metrics=net.rx_bytes_per_s,net.tx_bytes_per_s&range=${span}`).catch(() => null) : null, - }; - const [cpu, mem, temp, disk, net] = await Promise.all([requests.cpu, requests.mem, - requests.temp, requests.disk, requests.net]); - - if (cpu) { - const series = seriesFor(cpu, 'cpu.usage_pct'); - state.charts.cpu.setData(series); - DashCharts.renderLegend($('#legend-cpu'), series, (v) => num(v, 1) + '%', true); - } - if (mem) { - const series = seriesFor(mem, 'mem.used_pct'); - state.charts.mem.setData(series); - DashCharts.renderLegend($('#legend-mem'), series, (v) => num(v, 1) + '%'); - } - if (temp) { - const series = seriesFor(temp, 'temp.cpu_c'); - state.charts.temp.setData(series); - DashCharts.renderLegend($('#legend-temp'), series, (v) => num(v, 1) + '°C', true); - } - if (disk) { - const read = (disk.series.find((s) => s.metric === 'disk.read_bytes_per_s') || { points: [] }) - .points.map((p) => ({ ts: p.ts, value: p.value })); - const write = (disk.series.find((s) => s.metric === 'disk.write_bytes_per_s') || { points: [] }) - .points.map((p) => ({ ts: p.ts, value: p.value })); - const series = [{ name: 'read', points: read }, { name: 'write', points: write }]; - state.charts.disk.setData(series); - DashCharts.renderLegend($('#legend-disk'), series, (v) => bytes(v, 1) + '/s', true); - } - if (net) { - const rx = (net.series.find((s) => s.metric === 'net.rx_bytes_per_s') || { points: [] }) - .points.map((p) => ({ ts: p.ts, value: p.value })); - const tx = (net.series.find((s) => s.metric === 'net.tx_bytes_per_s') || { points: [] }) - .points.map((p) => ({ ts: p.ts, value: p.value })); - const series = [{ name: 'in', points: rx }, { name: 'out', points: tx }]; - state.charts.net.setData(series); - DashCharts.renderLegend($('#legend-net'), series, (v) => bytes(v, 1) + '/s', true); - } - await refreshGpuChart(hostNames); - await refreshDetailChart(); -} - -async function refreshGpuChart(hostNames) { - if (!hostNames.length) { - state.charts.gpu.setData([]); - $('#legend-gpu').innerHTML = 'no hosts'; - return; - } - const series = []; - for (const host of hostNames) { - // per-stream names are discovered, not guessed: gpu.busy_pct@0, @1, ... - const found = await fetchJSON(`/api/metrics?source=host&host=${encodeURIComponent(host)}`) - .catch(() => ({ metrics: [] })); - const names = (found.metrics || []).filter((m) => m.startsWith('gpu.busy_pct@')); - for (const name of names) { - const data = await fetchJSON(`/api/history?source=host&host=${encodeURIComponent(host)}` + - `&metric=${encodeURIComponent(name)}&range=${state.range}`).catch(() => null); - if (data) series.push({ name: `${host} ${name.split('@')[1]}`, - points: data.points.map((p) => ({ ts: p.ts, value: p.value })) }); - } - } - state.charts.gpu.setData(series); - DashCharts.renderLegend($('#legend-gpu'), series, (v) => num(v, 0) + '%', true); -} - -async function refreshDetailChart() { - const host = state.detailHost; - if (!host) { state.charts.detail.setData([]); return; } - const kind = metricKind(state.detailMetric); - const format = formatFor(kind); - const data = await fetchJSON(`/api/history?source=host&host=${encodeURIComponent(host)}` + - `&metric=${encodeURIComponent(state.detailMetric)}&range=${state.range}`).catch(() => null); - if (!data) return; - const series = [{ name: state.detailMetric, - points: data.points.map((p) => ({ ts: p.ts, value: p.value })) }]; - state.charts.detail.opts.yFormat = format.axis; - state.charts.detail.opts.yMin = state.detailMetric.endsWith('_pct') ? 0 : null; - state.charts.detail.opts.yMax = state.detailMetric.endsWith('_pct') ? 100 : null; - state.charts.detail.setData(series); - DashCharts.renderLegend($('#legend-detail'), series, format.fn); - const last = series[0].points.slice(-1)[0]; - $('#detail-note').textContent = `${series[0].points.length} samples over ${duration(state.range)} · ` + - `last ${last ? format.fn(last.value) : 'n/a'} · ${state.range <= 86400 ? 'raw resolution' : 'rolled up'}`; -} - -async function populateDetailMetrics(host) { - const select = $('#detail-metric'); - const found = await fetchJSON(`/api/metrics?source=host&host=${encodeURIComponent(host)}`) - .catch(() => ({ metrics: [] })); - const curated = new Set(DETAIL_METRICS.map(([name]) => name)); - const extras = (found.metrics || []).filter((m) => !curated.has(m)).sort(); - select.innerHTML = ''; - const groupMain = document.createElement('optgroup'); - groupMain.label = 'Common'; - DETAIL_METRICS.forEach(([name, label]) => { - const option = document.createElement('option'); - option.value = name; option.textContent = label; - groupMain.appendChild(option); +/** The GPU streams one host reports, named from the payload rather than guessed. */ +function gpuStreams(host) { + return (host.gpus || []).map((gpu, i) => { + const index = gpu.index == null ? i : gpu.index; + return { metric: `gpu.busy_pct@${index}`, label: gpu.model || `GPU ${index}` }; }); - select.appendChild(groupMain); - if (extras.length) { - const groupExtra = document.createElement('optgroup'); - groupExtra.label = `Per-device (${extras.length})`; - extras.forEach((name) => { - const option = document.createElement('option'); - option.value = name; option.textContent = name; - groupExtra.appendChild(option); - }); - select.appendChild(groupExtra); +} + +/** + * The chart objects inside one host panel, built once and reused every poll. + * + * The GPU chart is added the first time the host reports a card and removed the moment it + * stops reporting one — a machine whose GPU went missing must not keep a plot of it. + */ +function hostChartSet(card, host) { + let set = state.charts.get(host.name); + if (!set) { + // no `area` here on purpose — `drawHostChart()` decides the fill per draw, from how many + // series actually have points, which is the only question worth asking + const common = { height: CHART_HEIGHT, xSpan: state.range }; + // Every chart names itself for assistive tech: the panel heading says *who*, the + // aria-label says what is plotted. An unlabelled SVG is silence, and a screen reader + // must not need the colour-coded legend to know which machine it is listening to. + set = { + util: new DashCharts.LineChart(card.querySelector('[data-chart="util"]'), + Object.assign({}, common, { yMin: 0, yMax: 100, yFormat: AXIS_PCT, + ariaLabel: `${host.name} — utilisation over the selected window: CPU, memory and worst disk in per cent` })), + net: new DashCharts.LineChart(card.querySelector('[data-chart="net"]'), + Object.assign({}, common, { yMin: 0, yFormat: AXIS_RATE, + ariaLabel: `${host.name} — network traffic over the selected window: bytes in and bytes out` })), + gpu: null, + }; + state.charts.set(host.name, set); } - select.value = DETAIL_METRICS.some(([name]) => name === state.detailMetric) - ? state.detailMetric : (extras[0] || DETAIL_METRICS[0][0]); - state.detailMetric = select.value || DETAIL_METRICS[0][0]; + const wantsGpu = gpuStreams(host).length > 0; + if (wantsGpu && !set.gpu) { + const slot = document.createElement('div'); + slot.className = 'host-chart'; + slot.dataset.chartSlot = 'gpu'; + slot.innerHTML = '
' + + '
'; + card.querySelector('.host-charts').appendChild(slot); + set.gpu = new DashCharts.LineChart(slot.querySelector('[data-chart="gpu"]'), { + height: CHART_HEIGHT, xSpan: state.range, yMin: 0, yMax: 100, yFormat: AXIS_PCT, + ariaLabel: `${host.name} — GPU busy per cent over the selected window: ` + + gpuStreams(host).map((s) => s.label).join(', ') }); + } + if (!wantsGpu && set.gpu) { + // the whole slot goes: an empty wrapper or a plot of a card that is no longer + // reported is a machine's panel lying about hardware it does not have + card.querySelectorAll('[data-chart-slot="gpu"]').forEach((node) => node.remove()); + set.gpu = null; + } + return set; +} + +/** + * `[metric, label][]` -> LineChart series, from one host's history response. + * + * The legend names the *metric*, never the host: a chart here can only ever hold one + * machine, so repeating the host name three times under a chart is ink with no information. + */ +function seriesFor(response, definitions) { + return definitions.map(([metric, label]) => { + const found = (response.series || []).find((s) => s.metric === metric); + return { name: label, + points: ((found || {}).points || []).map((p) => ({ ts: p.ts, value: p.value })) }; + }); +} + +/** + * Plot one series set into one panel chart and repaint its legend. + * + * The fill is decided per draw on purpose: a single series reads as a shape worth shading + * (GPU busy), while two or three filled areas stack into one block and bury the smaller + * line — which is how a quiet CPU hides behind a busy memory plot. + */ +function drawHostChart(chart, card, key, series, format) { + chart.opts.xSpan = state.range; + chart.opts.area = series.filter((s) => s.points.length).length < 2; + chart.setData(series); + DashCharts.renderLegend(card.querySelector(`[data-legend="${key}"]`), series, format, true); +} + +/** One host's charts, from one request. Called for every host on every refresh. */ +async function renderHostCharts(card, host) { + const charts = hostChartSet(card, host); + const gpu = gpuStreams(host); + const wanted = UTIL_SERIES.concat(NET_SERIES, gpu.map((s) => [s.metric, s.label])); + const query = `/api/system/history?hosts=${encodeURIComponent(host.name)}` + + `&metrics=${encodeURIComponent(wanted.map(([metric]) => metric).join(','))}` + + `&range=${state.range}`; + const response = await fetchJSON(query).catch(() => null); + if (!response || !state.charts.has(host.name)) return; // the panel went away mid-flight + + drawHostChart(charts.util, card, 'util', seriesFor(response, UTIL_SERIES), PCT_FORMAT); + drawHostChart(charts.net, card, 'net', seriesFor(response, NET_SERIES), RATE_FORMAT); + if (charts.gpu) { + drawHostChart(charts.gpu, card, 'gpu', + seriesFor(response, gpu.map((s) => [s.metric, s.label])), PCT_FORMAT); + } +} + +/** + * The Δ window the rates were actually measured over, for the page footnote. + * + * `window_seconds` beside the fleet is the *poll interval*; the window behind a rate is the + * estimator's (`RATE_WINDOW_SECONDS`) and the two need not agree, so the footnote reads each + * host's own measured `window_seconds` instead of repeating a config number as a measurement + * (invariant 2: a rate says how it was obtained). One poll loop drives every host, so the + * windows differ by milliseconds; a real spread prints as a range rather than picking a winner. + */ +function windowLabel(values, fallback) { + const nums = values.filter((value) => value != null && Number.isFinite(value)); + if (!nums.length) return fallback; + const lo = Math.min(...nums); + const hi = Math.max(...nums); + return lo.toFixed(1) === hi.toFixed(1) ? `Δ${hi.toFixed(1)}s` + : `Δ${lo.toFixed(1)}–${hi.toFixed(1)}s`; +} + +/** Drop cards (and charts) for hosts that are no longer in the payload. */ +function pruneHosts(living) { + Array.from(state.hosts.keys()).forEach((name) => { + if (living.includes(name)) return; + state.hosts.get(name).remove(); + state.hosts.delete(name); + state.charts.delete(name); + }); } // ---- snapshot ------------------------------------------------------------- -function renderFleet(fleet) { - if (!fleet) return; - setTxt($('#f-hosts'), fleet.hosts_live == null ? 'n/a' : String(fleet.hosts_live)); - setTxt($('#f-hosts-of'), fleet.hosts_total == null ? '' : `of ${fleet.hosts_total}`); - setValue($('#f-cpu'), fleet.cpu_usage_pct_avg, 0); - setValue($('#f-load'), fleet.load1_total, 2); - const used = Dash.fmtBytes(fleet.memory_used_bytes, 0); - const total = Dash.fmtBytes(fleet.memory_total_bytes, 0); - setTxt($('#f-mem'), used ? used.text : 'n/a'); - setTxt($('#f-mem-of'), used && total ? `${used.unit} of ${total.text} ${total.unit}` : ''); - setValue($('#f-disk'), fleet.disk_worst_used_pct, 0); - const rx = Dash.fmtBytes(fleet.net_rx_bytes_per_s, 1); - const tx = Dash.fmtBytes(fleet.net_tx_bytes_per_s, 1); - setTxt($('#f-rx'), rx ? rx.text : 'n/a'); setTxt($('#f-rx-unit'), rx ? rx.unit + '/s' : ''); - setTxt($('#f-tx'), tx ? tx.text : 'n/a'); setTxt($('#f-tx-unit'), tx ? tx.unit + '/s' : ''); - setValue($('#f-gpu'), fleet.gpu_busy_pct_avg, 0); -} - async function refresh() { let snapshot; try { @@ -411,6 +426,7 @@ async function refresh() { const states = {}; (snapshot.states || []).forEach((row) => { states[row.key.split(':')[1]] = row; }); const hostNames = []; + const chartJobs = []; (snapshot.hosts || []).forEach((host) => { hostNames.push(host.name); host.state = states[host.name]; @@ -420,43 +436,33 @@ async function refresh() { card = hostCard(host.name); state.hosts.set(host.name, card); $('#hosts').appendChild(card); - card.addEventListener('click', () => { - if (state.detailHost !== host.name) { - state.detailHost = host.name; - $('#detail-host').value = host.name; - populateDetailMetrics(host.name).then(refreshDetailChart); - } - }); } renderHostCard(card, host); + // one history request per host, all of its charts fed from the same response + chartJobs.push(renderHostCharts(card, host)); }); + // a host that left HOST_TARGETS must take its panel — and its live charts — with it + pruneHosts(hostNames); $('#hosts-empty').hidden = hostNames.length > 0; - const hostSelect = $('#detail-host'); - if (hostSelect.dataset.names !== hostNames.join(',')) { - hostSelect.dataset.names = hostNames.join(','); - hostSelect.innerHTML = hostNames.map((name) => - ``).join(''); - if (!state.detailHost || !hostNames.includes(state.detailHost)) { - state.detailHost = hostNames[0] || null; - } - if (state.detailHost) hostSelect.value = state.detailHost; - if (state.detailHost) await populateDetailMetrics(state.detailHost); - } - - renderFleet(snapshot.fleet); const fleet = snapshot.fleet || {}; $('#sub').textContent = `${fleet.hosts_live || 0}/${fleet.hosts_total || 0} hosts answering · ` + `${fleet.cpu_cores || 0} cores · ${fleet.gpu_count || 0} GPUs · ` + `window ${duration(state.range)}`; - $('#foot-note').textContent = `rates over Δ${snapshot.window_seconds}s · ` + - `poll ${Math.round(state.pollMs / 1000)}s`; - await refreshCharts(snapshot); + // the Δ comes from the payloads, never from the interval the fleet happens to answer on + $('#foot-note').textContent = `rates over ${windowLabel( + (snapshot.hosts || []).map((host) => host.window_seconds), + `Δ${snapshot.window_seconds}s`, // nothing measured yet — the configured interval + )} · poll ${Math.round(state.pollMs / 1000)}s`; + await Promise.all(chartJobs); } async function refreshEvents() { try { - const data = await fetchJSON('/api/events?limit=25&source=host'); + // `system`, not `host`: reachability/reboot events are logged under the *dashboard* name + // (`poller.py` writes `Event(ts, self.dashboard, …)`; samples use `host`, events do not). + // Querying `source=host` matched nothing and left this feed forever empty. + const data = await fetchJSON('/api/events?limit=25&source=system'); const list = $('#events'); if (!(data.events || []).length) { list.innerHTML = '
  • No host transitions ' + @@ -471,7 +477,6 @@ async function refreshEvents() { } document.addEventListener('DOMContentLoaded', async () => { - makeCharts(); let pollMs = 10000; try { const config = await fetchJSON('/api/config'); @@ -482,22 +487,12 @@ document.addEventListener('DOMContentLoaded', async () => { } catch (err) { /* config is nice-to-have */ } state.pollMs = pollMs; + // the window chips re-fetch every panel's history: the range is per page, not per chart Dash.rangeControl($('#range'), state.range, (seconds) => { state.range = seconds; - setChartSpan(seconds); refresh(); }); - $('#detail-host').addEventListener('change', async (event) => { - state.detailHost = event.target.value; - await populateDetailMetrics(state.detailHost); - await refreshDetailChart(); - }); - $('#detail-metric').addEventListener('change', async (event) => { - state.detailMetric = event.target.value; - await refreshDetailChart(); - }); - state.poller = poll(async () => { await refresh(); await refreshEvents(); }, state.pollMs); Dash.bindPause(state.poller); }); diff --git a/tests/card_contract.py b/tests/card_contract.py new file mode 100644 index 0000000..efcebe3 --- /dev/null +++ b/tests/card_contract.py @@ -0,0 +1,104 @@ +"""The `/system` hero-card payload contract, shared by the hermetic and the live tests. + +Phase 03 makes `/system` one panel per host with six big colour-coded numbers. The paths +below are the payload fields those cards are allowed to read; they were read out of +`parse_host_snapshot()` and are pinned hermetically in `tests/test_system_contract.py` +(driven from `tests/fixtures/agent_*.json`) and against the real fleet in +`tests/test_live_fleet.py::test_every_host_serves_the_hero_card_contract`. + +Keeping the list here rather than in one test file means the live assertions cannot drift +away from the fixture assertions: there is one table, and it is the thing under test. + +=========================== ========================================================== +Card Path in `/api/system/snapshot` `hosts[]` +=========================== ========================================================== +cpu % ``cpu.usage_pct`` (+ ``cpu.severity.usage_pct``) +mem % ``memory.used_pct`` (+ ``memory.severity.used_pct``) +cpu temp ``cpu.temp_c`` (+ ``cpu.severity.temp_c``) +disk usage % ``disks.worst_used_pct`` + ``disks.worst_mount`` (+ verdict) +graphics card ``gpus[].model/busy_pct/vram_*/mem_used_pct`` (+ verdict) +network down/up ``network.rx_bytes_per_s`` / ``tx_bytes_per_s`` (no verdict) +=========================== ========================================================== + +CPU temp comes from ``cpu.temp_c`` rather than ``temperatures.cpu_c``: both are the agent's +own ``cpu_temperature_celsius``, and ``cpu.temp_c`` is the field ``/api/summary`` flattens +into its fleet row, so the card and the overview cannot drift apart. + +Deliberately *not* card metrics (legibility, not missing data — the series stay in SQLite): +per-core usage (``cpu.usage_pct@coreN``), swap, the process table, the raw sensor list, and +per-interface rows. Network carries no ``severity`` on purpose; throughput has no good/bad, +so ``severity.THRESHOLDS`` has no ``net.*`` row. +""" +from __future__ import annotations + +from typing import Any + +#: Card metric -> dotted payload paths. Present is the contract; a value may be ``None`` +#: on a host that genuinely does not have the number (gateway reports no CPU temperature). +CARD_METRICS: dict[str, tuple[str, ...]] = { + "cpu %": ("cpu.usage_pct", "cpu.severity.usage_pct"), + "mem %": ("memory.used_pct", "memory.severity.used_pct"), + "cpu temp": ("cpu.temp_c", "cpu.severity.temp_c", "temperatures.cpu_c"), + # the mount is what makes the percentage actionable: 88 % used — of which disk? + "disk %": ("disks.worst_used_pct", "disks.worst_mount", "disks.severity.worst_used_pct"), + "network down/up": ("network.rx_bytes_per_s", "network.tx_bytes_per_s"), +} + +#: Per-GPU paths, checked on every row of `gpus[]` — and on none of them when the list is +#: empty, which is how a host without a GPU stays honest (no card, not a card at 0 % busy). +GPU_METRICS: tuple[str, ...] = ( + "model", + "busy_pct", + "vram_used_bytes", + "vram_total_bytes", + "vram_used_pct", + # the honest APU view: vram + GTT, because a Strix Halo keeps the model in GTT + "mem_used_pct", + "severity.busy_pct", + "severity.mem_used_pct", +) + +#: `network` must never grow a `severity` dict (see the module docstring). +SEVERITY_FREE_SECTIONS: tuple[str, ...] = ("network",) + +#: The host shapes the captures in `tests/fixtures/` cover, and therefore the shapes a card +#: must handle: a discrete GPU, an APU (VRAM *and* GTT), and no GPU at all. A new capture +#: has to be added here — `test_the_contract_covers_every_captured_host` fails otherwise. +HOST_SHAPES: tuple[str, ...] = ("driveripper", "deskwork", "gateway") + + +def dig(node: Any, path: str) -> Any: + """Follow a dotted payload path, failing with the path in the message. + + A missing key is a contract break; a ``None`` value is not (that renders `n/a`), which is + why this returns the value instead of asserting on it. + """ + walked = "" + for part in path.split("."): + assert isinstance(node, dict), f"{path}: hit a non-dict at {walked or 'payload'}" + assert part in node, ( + f"{path}: no key {part!r} under {walked or 'payload'} — a hero card would print " + f"n/a for a number the API used to serve" + ) + node = node[part] + walked = f"{walked}.{part}" if walked else part + return node + + +def assert_card_contract(payload: Any) -> None: + """Every card path exists on this host payload, GPU rows included.""" + for paths in CARD_METRICS.values(): + for path in paths: + dig(payload, path) + gpus = payload.get("gpus") + assert isinstance(gpus, list), "gpus must be a list — [] when the agent reports no GPU" + for gpu in gpus: + for path in GPU_METRICS: + dig(gpu, path) + for section in SEVERITY_FREE_SECTIONS: + block = payload.get(section) + assert isinstance(block, dict), f"{section} must be a dict" + assert "severity" not in block, ( + f"{section}.severity would be a colour nobody measured — throughput has no " + f"good/bad and severity.THRESHOLDS has no {section}.* row" + ) diff --git a/tests/test_api.py b/tests/test_api.py index 4c4b4ed..5ff80fe 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -388,6 +388,60 @@ def test_agent_tokens_never_reach_the_browser(tmp_path: Path, monkeypatch: pytes store.close() +def test_a_dead_host_never_writes_its_token_into_the_feed(tmp_path: Path) -> None: + """The second path a credential can take to a browser: the failure message. + + `parse_targets` moves a userinfo token out of the URL and into a bearer header, so the + documented `HOST_TARGETS` path never puts one in `Target.url` — this is the backstop for + any `Target` that does (a hand-built one, a future parser), and the message path is where + it would matter: the poller keeps an `UpstreamError` string in the health record and logs + it as an event, so a userinfo-bearing URL there would sit in SQLite and be served on + `/api/events` forever (AGENTS.md invariant 6). Asserted on every endpoint a page reads, + not just `/api/config`. + """ + db = str(tmp_path / "dead-host-token.sqlite3") + store = SampleStore(db) + settings = Settings( + db_path=db, + host_interval=0.05, + host_targets=(Target(name="driveripper", url="http://SEKRIT-AGENT-TOKEN@127.0.0.1:1"),), + ) + with TestClient(create_app(settings, store)) as client: + deadline = time.time() + 10 + while time.time() < deadline: + health = client.get("/api/health").json() + if health["sources"] and health["sources"][0]["error"]: + break + time.sleep(0.05) + else: + pytest.fail("the dead host never recorded an error") + for path in ("/api/health", "/api/events", "/api/system/snapshot", "/api/summary"): + body = client.get(path).text + assert "SEKRIT-AGENT-TOKEN" not in body, f"{path} leaked the agent token" + # and the outage is still legible — redaction is not the same as silence + assert "driveripper" in client.get("/api/health").text + assert "unreachable" in client.get("/api/events").text + store.close() + + +def test_engine_urls_are_redacted_the_same_way_hosts_are(tmp_path: Path) -> None: + """`GUFO_BASE_URL`/`VLLM_BASE_URL` go to the browser verbatim in `/api/config`, and unlike + `HOST_TARGETS` nothing parses a userinfo token out of them into a header — so a URL that + happens to carry credentials has to be redacted where it is served (invariant 6). + """ + db = str(tmp_path / "engine-redact.sqlite3") + settings = Settings(db_path=db, gufo_url="http://SEKRIT-ENGINE-TOKEN@deskwork:8731", + vllm_url="http://driveripper:8000") + store = SampleStore(db) + with TestClient(create_app(settings, store)) as client: + body = client.get("/api/config") + assert "SEKRIT-ENGINE-TOKEN" not in body.text + llm = body.json()["llm"] + assert llm["gufo"]["base_url"] == "http://deskwork:8731" + assert llm["vllm"]["base_url"] == "http://driveripper:8000" # untouched when clean + store.close() + + def wait_for_state(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]: deadline = time.time() + timeout while time.time() < deadline: @@ -522,13 +576,14 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None: def test_severity_for_is_kept_as_the_documented_fallback(client) -> None: """The old call sites still work until the remaining phases convert each page. - `/` was converted in phase 02: the overview colours from the `severity` verdicts - `/api/summary` already sends, so it no longer carries a threshold of its own. The - three pages still to come must keep compiling against the fallback until they are. + `/` was converted in phase 02 and `/system` in phase 03: both colour themselves from + the `severity` verdicts the API already sends, so they carry no threshold of their + own. The two pages still to come must keep compiling against the fallback until they + are converted — deleting it early would break a page mid-migration. """ js = client.get("/static/app.js").text assert "FALLBACK ONLY" in js - for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"): + for page in ("/static/page-llm.js", "/static/page-ups.js"): assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site" @@ -539,6 +594,198 @@ def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> Non assert "severityFor" not in js +def test_system_page_colours_from_the_payload_not_from_js_thresholds(client) -> None: + """`page-system.js` hero cards paint payload verdicts, not thresholds of their own. + + Phase 03 task 02's promise: every colour on a host card comes from a `severity` dict + `severity.py` produced (`cpu.severity.usage_pct`, `memory.severity.used_pct`, + `disks.severity.worst_used_pct`, `gpus[].severity.busy_pct`, `cpu.severity.temp_c`). + A threshold written here could disagree with the overview's alerts; a hero with no + verdict prints `n/a` uncoloured, which is `Dash.hero()`'s contract. + """ + js = client.get("/static/page-system.js").text + assert "severityFor(" not in js, "the system page re-invented a threshold" + assert "Dash.hero(" in js + for hook in ("cpu.severity.usage_pct", "memory.severity.used_pct", + "disks.severity.worst_used_pct", "gpus[].severity.busy_pct", + "cpu.severity.temp_c"): + assert hook in js, f"the card no longer names {hook} as a colour source" + + +def test_system_host_card_is_six_heroes_without_per_core(client) -> None: + """Each host card leads with exactly six heroes: CPU, MEM, TEMP, DISK, GPU, NET. + + Asserted on the served script because there is no JS test runner (a locked anchor). + The two dense `.metrics` rows and the per-core heat bars are exactly what made the + card unreadable, and "no per-core anywhere on /system" is a promise a stray + `renderCores` call would silently break — the series stays stored, the page just + stopped drawing it (PLAN.md *Dropped visuals*). + """ + js = client.get("/static/page-system.js").text + card = js[js.index("function hostCard("):js.index("function renderHostCard(")] + slots = re.findall(r'
    ]*data-sev', card), "the NET slot invented a colour" + # what the glance leaves out stays reachable, collapsed by default + assert "More detail" in card + for key in ("load", "freq", "dr", "dw", "dutil", "uptime", "details"): + assert f'data-k="{key}"' in card, f"{key} left the page instead of the glance" + # the panel owns its charts: the slots live inside the card, not on the page + for chart in ("util", "net"): + assert f'data-chart="{chart}"' in card, f"the panel lost its {chart} chart slot" + + +def test_system_page_keeps_no_fleet_wide_chart_or_metric_picker(client) -> None: + """Nothing on `/system` plots two hosts, and nothing needs a selection to make sense. + + TODO.md L4 is the instruction ("The system page doesn't need shared charts for + anything") and the overview's alerts + fleet table are what made removing them safe + (phase 02). Asserted on the served bytes because the only thing that would otherwise + notice a resurrected `