phase: 03_system_per_host_panels

**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**

- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
- **Next pending phase**: `04_llm_chart_readability`.
This commit is contained in:
2026-10-17 04:42:09 -04:00
parent 8b048599f3
commit 9757e3c947
43 changed files with 1686 additions and 464 deletions
+30 -5
View File
@@ -54,8 +54,10 @@ that already listens there (upsd) — that is how the UPS page reads a real batt
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
JSON document), `GET /info` (slow-changing identity: OS, kernel, CPU model, GPU list), and
`GET /health` (the one route that never needs the token) as JSON.
JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS,
kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and
`GET /health` (the one route that never needs the token) as JSON, with a plain HTML status
page at `GET /` so a browser pointed at an agent learns what it is looking at.
```bash
python3 host_agent.py --once # print one snapshot, exit
@@ -89,7 +91,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
| ------------------------------------- | -------------------------------------------------------------- |
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) |
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and `/system`'s footnote prints that, not this default |
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
@@ -108,11 +110,16 @@ A source that is not configured answers `404 {"configured": false}` **without di
and the page says "not configured" instead of showing a red panel for something you never
installed.
Nothing that leaves the app carries a credential. A `HOST_TARGETS` userinfo token is parsed
into an `Authorization` header, and every URL the browser can see — `/api/config`, a source's
error string, the event feed — goes through `config.strip_userinfo` first, because an
unreachable host writes its failure message into the event table and `/api/events` serves it.
## Architecture
```
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
▼ ▼ ▼
src/dashboards/collectors.py ────────────────────────────────────────────┐
│ parse + Δcounter/Δt (rates.py) │
@@ -155,7 +162,19 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
A big colour-coded number is `Dash.hero(el, {...})` on the
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
with no verdict renders `n/a` with no colour at all.
with no verdict renders `n/a` with no colour at all. Each `/system` host is one panel led
by exactly six of them — CPU %, MEM %, TEMP, DISK % (mount underneath), GPU busy (VRAM and
the APU pool in the footnote), and NET on two lines (↓ in / ↑ out, never coloured: a byte
rate has no good/bad). Under the numbers, **each panel charts only its own host**: utilisation
(CPU, memory, worst disk), network (↓ in / ↑ out), and GPU busy for a host that reports
cards — three 120px SVG charts, each with its own legend and its own `aria-label` naming the
host and the plotted metrics (`LineChart`'s `ariaLabel` option), never a second host in sight. The
fleet-wide charts, the nine-figure fleet-totals panel and the host/metric detail picker are gone, and
so are the per-core charts, while `cpu.usage_pct@coreN` stays in SQLite. Everything else —
load, clock, disk I/O, uptime, GPU power/temp, filesystems, processes — folds into a collapsed
"More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request
(`hosts=<that host>&metrics=…&range=…`), and a host that stops reporting takes its panel and
its charts off the page.
## Testing
@@ -176,6 +195,12 @@ also skips when its source is unset, they drive the production code paths (`coll
`NutMonitor.collect`) against whatever hardware is reachable, and they never write — a
battery test on live hardware is not a unit test.
One more file is worth knowing about: `tests/card_contract.py` is the table of
`/api/system/snapshot` paths the system page's per-host cards are allowed to read. It is
checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and
again against the live fleet, so a card can never be "fixed" by inventing a number the agent
does not report.
## Battery-test buttons
Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`)