phase: 03_system_per_host_panels

**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**

- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
- **Next pending phase**: `04_llm_chart_readability`.
This commit is contained in:
2026-10-17 04:42:09 -04:00
parent 8b048599f3
commit 9757e3c947
43 changed files with 1686 additions and 464 deletions
+66 -10
View File
@@ -61,9 +61,13 @@ page still renders.
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
no outbound access — a CDN reference is a request that never comes back.
6. **Never leak a credential.** Target URLs may carry a userinfo token; `/api/config`
exposes them only through `Target.display_url`, which strips userinfo. No token,
password, or personal hostname goes into tracked files, fixtures, or the browser.
6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets`
moves it into an `Authorization` header, so `Target.url` ends up clean). Nothing that leaves
the process may carry one: `/api/config` serves `Target.display_url` and
`config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message
is redacted too — those strings go into the health record *and* the `event` table, which
`/api/events` serves to the browser and `page-system.js` prints. No token, password, or
personal hostname goes into tracked files, fixtures, or the browser.
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
@@ -104,7 +108,12 @@ Responsibilities, one line each:
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
env var always wins. Defaults are safe except the sources, which default to unset.
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping.
Events are stored under the **dashboard** name (`system`/`llm`/`ups`), not the sample
source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
(found on `/system` during phase 03 browser verification).
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping. Failure
messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one
that was dialled, because they become health records and event rows.
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
(`gpu.busy_pct@<index>`).
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
@@ -158,8 +167,9 @@ Responsibilities, one line each:
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
values the API does not rate; converting its remaining call sites is phases 03–05's job
(`/` was converted in phase 02: the overview now colours only from `/api/summary` verdicts).
values the API does not rate; converting its remaining call sites is phases 04–05's job
(`/` was converted in phase 02, `/system` in phase 03 — those pages colour only from the
`severity` verdicts the API sends, and no `severityFor(` call survives in their scripts).
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
@@ -205,8 +215,8 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh logs
uv run pytest # hermetic: 439 passed + 6 live-skipped, no network
uv run pytest --cov # coverage gate (96.8%; floor 94%)
uv run pytest # hermetic: 472 passed + 7 live-skipped, no network
uv run pytest --cov # coverage gate (96.9%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth
@@ -235,6 +245,14 @@ service here.
fixture test fails before a dashboard does.
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
- **Page payloads that a rewrite must not break are pinned as a table.**
`tests/card_contract.py` holds the dotted `/api/system/snapshot` paths the `/system` hero
cards read (`CARD_METRICS`, `GPU_METRICS`, `HOST_SHAPES`); `tests/test_system_contract.py`
asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and
`test_every_host_serves_the_hero_card_contract` asserts the same table against the live
agents — so a rename fails a test, and a card that shows `n/a` cannot be "fixed" by
inventing a field. `network` carries no `severity` on purpose (throughput has no good/bad),
and the contract pins its absence.
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
unset, so partial fleet availability is normal. **Live tests must not write** — a battery
@@ -252,8 +270,11 @@ service here.
Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
dots in the header, window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API
accepts 5m…90d). A panel that cannot get data says
dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
`LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
silence, and the legend is a sibling that labels nothing) — `/system` does this for all three
slots, pinned by `tests/test_api.py`. Window chips (5m…30d on `/system` and `/llm`, 1h…90d on
`/ups`; the API accepts 5m…90d). A panel that cannot get data says
why in the panel — the operator should never have to open devtools to learn that a source is
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
@@ -281,6 +302,41 @@ muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare.
The `/system` host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the
`.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script).
Colour comes from the payload's `severity` dicts and nowhere else; a missing number or
verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**. NET is
one slot with two lines (`.hero-lines`, `↓` in / `↑` out) and is deliberately colourless:
throughput has no good/bad, so `network` carries no severity. The GPU slot holds one hero
per card (model, VRAM used/total and the honest APU `pool` view in the `.sub`), and
everything not glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU
power/temp, filesystems, processes — folds into a collapsed "More detail" `<details>`: a
simpler surface, not a less complete page.
A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
those (a range when the hosts disagree) instead of the poll interval, which only coincides with
them at the defaults.
**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
panel and the host/metric detail picker are deleted, along with the `#chart-*` / `#detail-*` ids and
the `select` / `label.field` styling only those pickers used. Each panel stacks its own
`.host-charts` — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where
the GPU slot is created and destroyed with `host.gpus`, so gateway shows neither a phantom card
nor a hole. **One `/api/system/history` request per host per refresh**
(`hosts=<that host>&metrics=<that panel's series>&range=…`, pinned by
`test_every_system_chart_is_drawn_inside_its_own_host_panel`): unknown metrics come back as
empty series on purpose, so a panel asks for every series it could draw and asks once. A legend
here names metrics (or one GPU's model), never hosts — the panel heading already says who this
is, and a legend that grew a second host is the bug this replaced. A host that stops reporting
loses its panel *and* its charts (`pruneHosts`), never a chart of nothing. Per-core visuals are
gone from `/system` and `renderCores` is **deleted** from `charts.js` with its last caller.
`drawHostChart()` shades a chart only when one series has points — two or three filled areas
stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy
memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryable from
`/api/system/history` — removing the plot is not removing the data.
## Known gaps / next work
1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see