# Homelab Dashboards Four dashboards, one domain, one process, one SQLite file: | Page | What it answers | Source | | ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- | | `/` | Is anything wrong, and which dashboard do I need? | all four, in one request (`/api/summary`) | | `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host | | `/disks` | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount) | | `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only | | `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) | No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at runtime; every asset is served by this app. Two more pages are not dashboards: `/login` (the one thing an anonymous visitor sees) and `/settings` — `⚙ Account` in the masthead of every dashboard — which shows this browser's session, signs it out, and changes the admin password. Working on this repo? **[`AGENTS.md`](AGENTS.md)** holds the invariants (read-only by construction, never invent a number, unconfigured ≠ broken), the locked decisions, the real fleet facts, and the known gaps. ## Quickstart ```bash uv sync --python 3.14 # deps + the package, editable cp .env.example .env # then edit — including DASH_ADMIN_PASSWORD uv run dashboards # http://127.0.0.1:8090 uv run uvicorn dashboards.app:app --port 8090 # equivalent uv run pytest # unit tests, no network needed ``` The first start needs a password to seed: with no `DASH_ADMIN_PASSWORD` and no verifier stored in `DASH_DB` yet, the process prints the reason and exits non-zero rather than serving the fleet open (see *Signing in*). The dev password this project uses locally is `testpass`; production gets a real one. The first thing a browser meets is `/login` — every page and every `/api` route except the four listed under *What the session covers* asks for it, while `/api/health` stays open for the container healthcheck and `./scripts/dev-server.sh status`. Developing against the real fleet from a laptop — dev dials exactly the addresses the deployed dashboard dials, so there is nothing to tunnel and no port to forward: ```bash cp .env.dev.example .env.dev # your host addresses + agent tokens (chmod 0600) sudo deploy/install-agent.sh --bind --port 9120 # once per monitored host: the agent as a systemd # service, printing its bearer token exactly once set -a; . ./.env.dev; set +a uv run dashboards # foreground ./scripts/dev-server.sh restart # or backgrounded, with a pidfile + logs ``` Local dev signs in with the `DASH_ADMIN_PASSWORD` in that file — the word the template's comment names (`testpass`, what the hermetic suite uses) — and without it a fresh database exits non-zero for the same reason production does. `.env.dev` is gitignored and `0600`: it holds the agent tokens and the upsd password too. Each `HOST_TARGETS` entry carries that token as URL userinfo (`driveripper=http://@:9120`); `parse_targets` moves it into an `Authorization` header, so `Target.url`, `/api/config` and every error message stay credential-free. No root shell on a host? `deploy/agent-sudoers` is the one-line grant that lets `ops` run the installer, and running the installer without root prints that exact line. upsd is dialled at `NUT_HOST`/`NUT_PORT` on the port it really answers on (3051 on driveripper, not the stock 3493) — see the NUT section below for who may connect. ## The host agent `agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`, `/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS, kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and `GET /health` (the one route that never needs the token) as JSON, with a plain HTML status page at `GET /` so a browser pointed at an agent learns what it is looking at. ```bash python3 host_agent.py --once # print one snapshot, exit python3 host_agent.py --host 0.0.0.0 --port 9120 --token-file /etc/dashboards-agent.token sudo deploy/install-agent.sh # systemd unit + user + token deploy/install-agent.sh --status # unit state and /health sudo deploy/install-agent.sh --uninstall ``` No root shell on the monitored host? `deploy/agent-sudoers` is the single sudoers line that lets the fleet's unprivileged `ops` account run the installer; run it without root and it prints that exact line with the script's own resolved path instead of saying "run as root". It is read-only by construction: there is no code path that writes, and the HTTP server only answers `GET`. Counters are reported as counters — the dashboard computes rates over a labelled window, so the agent never has to remember who polled it last. Per-disk `smart` health is a best-effort add-on on the whole-disk `/sys/block` list (no partitions): the agent runs `smartctl -i -A -H` per physical disk — 5 s bound per device, 20 s per pass, results cached 4 minutes — and parses health, temperature, power-on hours, the sector alarms and SSD wear into `disk_io.devices[].smart`. Refreshing an expired cache happens on a background thread while the request is served the previous result, so a wedged disk can never cost a poll more than its own timeout (only the very first pass after the agent starts runs inline). A host without `smartmontools`, or whose agent user cannot read the block devices, is *not broken*: every disk reports `smart: null` with one explanatory `errors` line, and that line never makes the system page amber. Enable SMART with `smartmontools` plus the `disk` group (which `deploy/install-agent.sh` grants when it exists) — the agent shells `smartctl` as itself and never calls `sudo`, so a sudoers rule alone is not enough — and **redeploy the agent**, since one installed before phase 09 has no SMART pass to run. Opt out silently with `--no-smart`. The drive truth this is all developed against lives in `tests/fixtures/smartctl_driveripper_*.txt`: 16 real drives, recorded with serials scrubbed, each one parsed by a test. Binding off-loopback without a token is refused (exit 2) rather than warned about. Prefer `--token-file` over `--token`: argv is world-readable through `ps`, and this token guards hostname, mounts, disk and GPU models, and process names. `deploy/install-agent.sh` creates a system user, generates a token once (re-runs reuse it; `--rotate-token` for a new one), stores it `0640 root:dashboards-agent` — the unit hands the agent the *path*, and the agent opens it as its own user, so the file must be readable by that user and by nobody else — and runs the agent under `ProtectSystem=strict`, `DevicePolicy=strict` with only the disk nodes and the `/dev/null` family reopened (SMART needs the former; without `/dev/null` in the allow list systemd cannot start the service at all), with no capabilities. ## Signing in One admin, one password, and the server **refuses to start** when it has neither a stored verifier nor `DASH_ADMIN_PASSWORD` — the domain is public and the fleet's hostnames and UPS controls sit behind this login, so there is no way to run it open by accident. There is no second account and no per-user view: the five pages (`/`, `/system`, `/disks`, `/llm`, `/ups`), the `/settings` page and every data route belong to that one operator. A session is an **HMAC-signed cookie** (`dash_session`), not a row in a table: it carries the signed-in name and its own expiry, is valid for `DASH_SESSION_HOURS` (default `720` = 30 days), and is verified with a signing key that also lives in `DASH_DB`. That is why a password change signs out every other browser at once — it rotates the key, so every cookie minted under the old one stops verifying on its next request — and why nothing has to be pruned. `DASH_ADMIN_PASSWORD` is a *seed*, not the credential. The first start hashes it (PBKDF2-HMAC-SHA256, 600 000 rounds) into the `meta` table of `DASH_DB` as `auth.verifier`, and from then on the stored verifier wins: remove the variable once the server is up and the same password still works, change it with `POST /api/auth/password` — or in a browser, on the `/settings` page (*The account page* below) — with no redeploy, and know that a container env can never roll a changed password back. If `DASH_ADMIN_PASSWORD` is still set on a box that already has a verifier, it loses — and the server says so once at startup, in words, without quoting the value. Forgotten password, or a database you want to hand over? Delete the rows and seed again: ```bash sqlite3 "$DASH_DB" "DELETE FROM meta WHERE key IN ('auth.verifier','auth.session_secret')" ``` Deleting `auth.session_secret` — which that command does — signs out every browser already signed in, which is also exactly what changing the password does on purpose. Only the hash and the signing key are ever written: `grep -c "your password" "$DASH_DB"` comes back `0`, and no log line, event row, or API response carries the password, the verifier, or a session token. Nothing that leaves the app carries a credential. A `HOST_TARGETS` userinfo token is parsed into an `Authorization` header, and every URL the browser can see — `/api/config`, a source's error string, the event feed — goes through `config.strip_userinfo` first, because an unreachable host writes its failure message into the event table and `/api/events` serves it. ### What the session covers The gate is default-deny: **every page and every `/api` route** requires a signed-in admin except four entries, named next to the middleware they guard in `app.py`: | Public | Why it has to be | | -------------------------- | ------------------------------------------------------------------ | | `/login` | the form itself | | `POST /api/auth/login` | you cannot be signed in to sign in | | `/api/health` | the Containerfile `HEALTHCHECK` and `./scripts/dev-server.sh status` poll it unsigned, and it publishes no fleet data | | `/static/…` | the login page is styled by the same stylesheet as the dashboards | Adding a path to that list is a security decision, and a route a future phase forgets fails `tests/test_auth_api.py::test_every_route_that_is_not_public_requires_a_session`, which walks the served route table rather than a list written by hand. What a refusal looks like depends on who asked, so that neither caller has to guess: ```bash curl -si http://127.0.0.1:8090/system | head -3 # HTTP/1.1 303 See Other → Location: /login?next=/system (a browser ends up back here) curl -s http://127.0.0.1:8090/api/summary # {"ok":false,"detail":"sign in required","login":"/login"} (a script never gets HTML) curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8090/api/health # 200 ``` The `?next=` value keeps the query string too (`/llm?range=1h` survives the round trip), and it is judged before it is ever followed: a `next` that leaves the site (`//evil.example`, `http://evil.example`, the percent-encoded and control-character spellings alike) is dropped to `/`. `auth.safe_next` is the one function that decides both halves. And every answer the gate produces — `303`, `401`, `403`, `429`, the route's own `404` — carries `no-store`, because a cached redirect would park a browser on the login page after it had already signed in. ### The sign-in page `/login` is one card with one password field — no username, because there is one admin and a field that is always the same word teaches you that half the form is theatre. It carries the masthead's brand link and the theme toggle and nothing else from it: no tabs (each would bounce straight back here) and no status dots, because there is nothing signed in for them to report. What it does with an answer is the part worth knowing: | Response | On screen | | -------- | -------------------------------------------------------------------- | | `200` | straight to `?next=`, or to `/` | | `401` | `Wrong password.` under the field, and the text you typed is selected | | `429` | the refusal plus the lockout counting down, button disabled to zero | | other | the server's own `detail`, printed verbatim | The countdown is the server's `Retry-After`, not a guess, and a refusal is written as text rather than HTML — this is the one page a stranger reaches first. A browser that already holds a session never sees the form: the server `303`s it to the dashboard, and the script checks `/api/auth/status` first so the form does not even flash. `?next=` is followed only when it is an in-app path (`/ups`, not `//evil.example`), judged on the server and again in the page. ### The account page `/settings` — reached by `⚙ Account` in the masthead of every dashboard — is the session facts and the one form that changes the password. | Panel | What it says, and what the button does | | ----- | ------------------------------------------------------------------------ | | Session | the signed-in name, the expiry as a local timestamp **and** as time left, and **Sign out** (`POST /api/auth/logout`, then `/login`) | | Change password | current + new + confirm, then `POST /api/auth/password`; the outcome is one line under the form | The form checks what it can check itself (a current password typed, at least 8 new characters, the two new boxes agreeing) so a typo is answered without a round trip, and then reports whatever the server said: `403` is *the current password is wrong*, `422` names the rule the new password broke, `429` is the lockout with its own seconds. The page also states the precedence rule in its own words — the env var seeded the password once, a change here wins, resetting from the environment means deleting `auth.verifier` and restarting — because that is the sentence that prevents a support call about an edited `.env` that did nothing. One detail makes the page more than a form. A change rotates the session-signing secret, which is what signs out every *other* browser, and it invalidates the CSRF token this browser cached — so `Dash.resetAuth()` clears that cache after the change (and after any `401` bounce). Without it the operator changes the password, then every UPS button 403s until they reload. **Sign out deletes this browser's cookie; it cannot revoke a copy of the token.** A session is signed rather than stored, so nothing is left to mark as dead — the one thing that reaches every browser at once is the signing key, and that is what a password change rotates. If a machine is lost or a session is suspected, change the password here: that is the revocation, and it is why the same page carries both buttons. ### Writes need the CSRF token A write authorised by the **cookie** must carry `X-CSRF-Token`, validated against that same session (`GET /api/auth/status` is where a page that is already signed in reads it — the login and change-password responses carry it too, minted for the session they just created — and it sits behind the session because a token is worthless to anyone who cannot read that response). `Dash.fetchJSON` fetches it once per browser and attaches it to every non-`GET`/`HEAD` request, so page code does not think about it — and `POST /api/auth/logout` is gated like any other write, because an `` planted on a page you read mid-outage is a CSRF that signs you out. Mismatched `Origin` or a `Sec-Fetch-Site` that is not `same-origin`/`none` is refused before the token is even looked at, so a cross-site login post cannot spend your throttle either. ### Changing the password `POST /api/auth/password` with `{"current": "…", "new": "…"}` — a protected write, so it needs the session and the CSRF token like everything else: ```bash curl -s -X POST -H 'content-type: application/json' -H "Cookie: dash_session=$SESSION" \ -H "X-CSRF-Token: $CSRF" -d '{"current":"…","new":"…"}' \ http://127.0.0.1:8090/api/auth/password # {"ok":true,"changed":true,"hint":"other browsers are now signed out", # "expires_at":1794982214.0,"csrf":"…"} ``` Nothing forces a change: no expiry, no first-login interstitial, and `DASH_ADMIN_PASSWORD` is never re-applied after the first boot of a database. The only composition rule is length — at least `api.MIN_PASSWORD_LENGTH` (8) characters, not empty once stripped, and different from the current one — each refused with a `422` that names the rule. A symbol/digit mandate produces sticky notes rather than passwords, and the cost of a guess here is PBKDF2 plus the backoff, not the character classes. A wrong `current` is `403 {"ok":false,"detail":"current password is wrong"}` and a `password-denied` event, and guesses at that field are throttled on their **own** key (`password:
`) — the same arithmetic as the login form, a separate bucket, so this form is never an unthrottled oracle for the old password once a cookie is stolen and one form's lockout does not lock out the other. A right one rewrites `auth.verifier` **and** `auth.session_secret`, which is the whole revocation story: a session is a signed cookie with no server-side record, so rotating the signing key is the only way a password change reaches the browsers that are not in front of the form (without it a stolen cookie would stay valid for another `DASH_SESSION_HOURS`). The response carries a fresh cookie and a fresh CSRF token minted from the **new** secret, so the caller who just changed the password is not signed out by their own form, while every other browser gets a `401` on its next request and signs back in with the new password. `password-changed` lands in the event feed — an event, not an overview alert, and not a field on `/api/health` (which is public) or `/api/config`. ### Scripting the API `Authorization: Bearer ` is the scripted caller, and it needs no CSRF token — another site can make your browser send its cookie but cannot read a token out of your shell. The session token is in the login response's `Set-Cookie`, not in its body: ```bash SESSION=$(curl -si -X POST -H 'content-type: application/json' \ -d '{"password":"…"}' http://127.0.0.1:8090/api/auth/login \ | tr -d '\r' | sed -n 's/^set-cookie: dash_session=\([^;]*\).*/\1/p' | head -1) curl -s -H "Authorization: Bearer $SESSION" http://127.0.0.1:8090/api/summary # 200 curl -s -H "Authorization: Bearer $SESSION" -X POST -H 'content-type: application/json' \ -d '{"ups":"apc","command":"test.panel.start"}' http://127.0.0.1:8090/api/ups/command ``` That is the app's one write endpoint **that dials anything** (the `/api/auth/*` writes change only this process and its SQLite `meta` table — invariant 1 is about the fleet), and the bearer header changes nothing about its gates: a command that can cut output still needs `"confirm":"DANGER"`, and is still refused while the UPS is on battery. ### Lockout Five wrong passwords from one address starts a backoff (`429` with `Retry-After`, exponential from 60 s, capped at an hour; a good sign-in clears it, and a restart resets the counters). The change-password form keeps its **own** counter under `password:
`, so hammering one form cannot lock you out of the other and neither form is an unthrottled oracle for the same secret. The refusal is the same sentence for every kind of failure — one account, so the response must not rank reasons — and each refused attempt writes a `login-denied` or `password-denied` event you can read at `/api/events?source=auth`. Neither ever becomes an overview alert: a password guess is not an outage, and a page that blinks for both trains you to ignore the ones that matter. ## Configuration Everything comes from the environment (see `.env.example` for the annotated list). At startup a `.env` next to the checkout is loaded first if it exists — as a *seed*: a variable the container, systemd unit, or your shell already set always wins. Point `DASH_ENV_FILE` somewhere else to keep several installs on one box. `.env.example` is documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev` itself (see the quickstart) because dev settings must not leak into a production process. | Variable | Meaning | | ------------------------------------- | -------------------------------------------------------------- | | `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) | | `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears | | `RATE_WINDOW_SECONDS` | the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and `/system`'s footnote prints that, not this default | | `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) | | `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) | | `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:` jobs exist exactly when `HOST_TARGETS` does | | `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others | | `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and 20 s of wedged disk must not read as a dead host | | `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server | | `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT | | `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) | | `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app | | `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) | | `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags | | `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) | | `DASH_ADMIN_PASSWORD` | the admin password **seed**, needed only on the first boot of a database — see *Signing in* above | | `DASH_SESSION_HOURS` | session lifetime (default `720` = 30 days) | | `DASH_COOKIE_SECURE` | `1` marks the session cookie `Secure`; off because dev is `http://127.0.0.1`, and a request that arrived over `X-Forwarded-Proto: https` is treated as secure either way | | `DASH_PBKDF2_ITERATIONS` | verifier cost (default `600000`). The knob exists for the test suite; leave it alone in a deployment. Raising it on a real box does take effect: the next successful sign-in re-hashes a cheaper stored verifier and keeps the signing key | | `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer. `/disks` has no `POLL_MS_*` of its own and polls at a fixed 30 s: its payload only moves when the `disks` job re-reads the agents | | `DASH_RELOAD` | `1` makes `uv run dashboards` watch the source and restart on a change — development only, and every poll loop restarts with it | | `DEBUGPY` / `DEBUGPY_PORT` | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup | | `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning | A source that is not configured answers `404 {"configured": false}` **without dialling**, and the page says "not configured" instead of showing a red panel for something you never installed. ## Architecture ``` agent/host_agent.py (each host) gufo / vLLM upsd (NUT) │ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD │ (twice: system 10s, disks 5m) │ + /slots (gufo) │ ▼ ▼ ▼ src/dashboards/collectors.py ────────────────────────────────────────────┐ │ parse + Δcounter/Δt (rates.py) │ ▼ ▼ sources/host.py · disks.py · llm.py · nut.py poller.py (one task per source) │ samples events │ ▼ ▼ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event + meta ── api.py → web/ ▲ auth.py ── verifier + session key, loaded once at startup (no row and no seed → no server) ``` * `poller.py` — every source gets its own asyncio task, its own health record, and its own error handling. A dead host keeps its last good values and is labelled stale; it never blanks or blocks another source. Disk health is a source of its own (`disks:` in `/api/health`, 5-minute cadence, its own fetch timeout) reading the *same* agent `/snapshot` the system page reads — a separate task is what keeps a SMART poll out of the 10-second utilisation poll and lets drive history go stale without blinking `/system`, and `alerts.py` is why one dead agent is still one line on the overview rather than two. And it never quietly *stops*: the task is that source's only poller, so a SQLite write that gets refused is reported (`ok: false`, the real reason, the numbers aging, one line in the event log) instead of freezing the source under a green dot. * `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are pruned, so a year of history is a few hundred MB rather than a disk emergency. Its `meta` table is the one place state that must outlive a restart goes (`get_meta`/`set_meta`/ `delete_meta`), and its first user is the admin credential below. * `auth.py` — the admin login's decisions, in stdlib and without HTTP: a self-describing PBKDF2-HMAC-SHA256 verifier, an HMAC-signed stateless session cookie (no session table, and rotating the stored key is what evicts every other browser), a session-bound CSRF token, and an in-process login backoff. `load_auth_state()` is the startup half: the environment seeds a password **once**, the stored verifier wins from then on, and a server with neither refuses to start (see *Signing in*). * `severity.py` — the single green/amber/red table (`THRESHOLDS`, plus `INVERTED` for the metrics where lower is worse). Rated values in the API payloads travel with a sibling `severity` (`cpu.severity.usage_pct`, `memory.severity`, `disks.severity`, a UPS card's `severity.charge_pct`, `severity` inside the rated LLM cells such as KV pressure, `/api/summary` rows), computed server-side, so every page colours the same way and a value nobody reported renders `n/a` instead of green. A metric published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays what the upstream published. There is no `gpu.*` row: these are LLM servers, a busy GPU is the job, so the GPU card (and the fleet GPU column) renders uncoloured at every utilisation — the `gpu.*` history series are still stored and charted. Drive health gets two rows: `disk.temp_c` (amber 50 °C, red 70 °C) and `disk.percentage_used` (amber 80 %, red 100 % — an SSD that has spent its rated wear is a replacement). The sector counters (`disk.reallocated`, `disk.pending`, `disk.uncorrectable`) and `disk.power_on_hours` have **no row on purpose**: a non-zero counter is an alarm chip on presence, zero is the absence of a chip and never a green "healthy drive", and an odometer has no bad value. * `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states, host_payloads, engine_payloads, ups_payload, recent_events)` is a pure projection of the state the poller already keeps plus the verdicts `severity.py` and the UPS health checks already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as `alerts`, worst first: `{id, severity, source, message, href, at}`. A configured source that stopped answering is `fail`; one that is stale but still serving its last good value is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a marker that says how many it hid. * `api.py` — the JSON every page reads: `/api/config`, `/api/health`, `/api/events`, `/api/history`, `/api/metrics`, `/api/summary`, and one `snapshot` + `history` pair per dashboard (`/api/system/*`, `/api/disks/*`, `/api/llm/*`, `/api/ups/*`). GETs everywhere except the single `POST /api/ups/command`. Two rules make the pages legible: a source with no env config answers `404 {"configured": false}` **without dialling** (so a page says "not configured" instead of going red), and a history query for a series that does not exist answers an empty series instead of an error (so a card asks for every plot it could draw and reads `n/a` for the ones the drive never reported). `/api/disks/snapshot` is the drive-health surface: one entry per host with its physical disk rows, the agent's `smart` block verbatim, `smartctl_available` (`true` / `false` = install smartmontools / `null` = the agent never said), a `severity` beside each row's rated numbers, and a `fleet` block summed here so the page does no arithmetic — a host that has not answered yet is listed as an `unreachable` stub of `n/a`s and counted, never summed. `/api/config` carries `disks.interval_seconds`, which is why a drive card four minutes old is behaving, not broken. `/api/summary` grew a `disks` block with that page (the same `fleet` sums plus one row per host — counts, `smartctl_available`, and `failed_devices`, so a notifier learns *which* drive the hardware called `FAILED` without a second call); it is additive, exactly as `alerts` was, and the key paths of both are pinned with and without targets configured. * `web/` — five static pages (`/`, `/system`, `/disks`, `/llm`, `/ups`), one stylesheet, hand-rolled SVG charts (`charts.js`), and a masthead status dot per dashboard job (`disks` has its own: the same agent can answer the 10-second utilisation poll and have a wedged SMART pass). No framework, no CDN, no bundler, and no narration: a page never explains its own pipeline, so the words on screen are the ones you act on — a unit, a `Δ10.0s` provenance tag, a `SIMULATED` badge, or the sentence beside an empty panel. The overview (`/`) is exactly two panels — a **Warnings & errors** list built from `/api/summary`'s `alerts` (each row links to the dashboard that explains it) and the **Fleet** table — and nothing else: an empty alert list reads as an explicit "all clear", and a live problem moves the panel above the table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the fleet never quietly gets smaller and last good values are never mistaken for live ones. A big colour-coded number is `Dash.hero(el, {...})` on the `.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via `Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value with no verdict renders `n/a` with no colour at all. A battery is `DashCharts.battery(el, {charge, severity, state, label})`: one inline SVG whose fill *is* the charge — clamped to 0–100, never rounded up, clipped to the plate so 0 % is an empty shell — coloured only by the payload verdict, with a shape per state (solid shell on line, bolt charging, dashed outline on battery, a dash through the body when there is no reading) and a `` / `aria-label` that names it ("apc: Charge 100 percent, on line") unless the number beside it already does. Each `/system` host is one panel led by exactly six of them — CPU %, MEM %, TEMP, DISK % (mount underneath), GPU busy (VRAM, the APU pool and the card's temperature in the footnote — no temperature is printed when the driver exposes no sensor, never `0 °C`; and likewise never coloured and with no bar: a busy GPU is the job on an LLM server), and NET on two lines (↓ in / ↑ out, never coloured: a byte rate has no good/bad). Under the numbers, **each panel charts only its own host** — behind a collapsed **Charts** toggle, because the default view is current stats and a fleet of ten hosts each drawing three plots is noise: the summary reads `▸ Charts` with the window in it (`1h`, `24h` …), and opening it is what builds and paints the charts — instantly, from the response the last refresh already fetched, and repainted on every refresh after that while it stays open. Expanding one panel never stretches its collapsed siblings (`.grid.items-start`). The stack is utilisation (CPU, memory, worst disk), network (↓ in / ↑ out), and GPU busy for a host that reports cards — three 120px SVG charts, each with its own legend and its own `aria-label` naming the host and the plotted metrics (`LineChart`'s `ariaLabel` option), never a second host in sight. The fleet-wide charts, the nine-figure fleet-totals panel and the host/metric detail picker are gone, and so are the per-core charts, while `cpu.usage_pct@coreN` stays in SQLite. Everything else — load, clock, disk I/O, uptime, GPU power and the per-label temp breakdown, filesystems, processes — folds into a collapsed "More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request (`hosts=<that host>&metrics=…&range=…`) every refresh, stack open or not, and a host that stops reporting takes its panel, its Charts toggle and its charts off the page. `/disks` is the drive-health page: **one card per physical drive** (the agent's whole-`/sys/block` list — never a partition, never a mount), led by the device path and the drive's own verdict word (`OK` / `FAILED` / `unknown`, coloured only by `severity.health`), then three equal cells — **Temp**, **Power-on** hours, **Wear** — coloured by `disk.temp_c` and `disk.percentage_used` from the API, with the odometer deliberately uncoloured because `severity.py` rates no power-on hours. The sector counters appear as **red chips that exist only while non-zero**, so a healthy drive shows no alarm row at all rather than a wall of zeroes. A host whose SMART cannot be read keeps its cards, dimmed with every number `n/a` and one `.banner` line saying why — `smartctl not available on <host>` when the agent reported the binary missing, its own sentence when it reported something else, and "not answering" when the agent is down — never a red panel, because a missing binary is unconfigured, not broken. The header's drive counts are the API's `fleet` block (hidden below two hosts, so one box's sums never duplicate its own cards), and each host's **Charts** disclosure behaves exactly like `/system`'s: collapsed, built only while open, and fed by one `/api/disks/history` request per host per refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive it has. `/llm` opens with the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, the note *sums only engines answering now*, then **running / waiting / prompt tok/s / decode tok/s / total tokens** — and every number on it is `fleet_totals()`, so the page sums nothing itself. A dead engine keeps its last-good numbers in its own (stale-labelled) panel but is dropped from a total presented as current: the tag counts what answered, names the missing engine in its tooltip and dims while the sums are partial. Nothing on the block is coloured — `severity.py` rates no throughput and no queue depth, because 0 tok/s on an idle fleet is healthy — the dot is reachability, and `fleet.seams` (how the two engines count prompt tokens differently) is a tooltip on the prompt sum rather than another line of ink. Below two configured engines the block is hidden: a cross-engine sum of one engine duplicates that engine's panel. Below the fleet block sits the **stats strip**: one card per configured engine — reachability dot, name, age stamp, the **KV-cache bar** (the one rated cell: the fill and a breach's text take their colour from the payload's `severity` and nothing else), then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ10.0s`, scraped, estimated), `total tokens`, the sessions this engine reports (`slots busy` counted from gufo's `/slots`, or `running`/`waiting` for vLLM, plus `waiting`), `prefix cache hit`, `spec accept` (gufo), `queue time` with its `p95 …` foot (vLLM) — and red alarm chips for preemptions / GPU-context losses that exist only while non-zero. Then the engine panels hold the detail the strip does not print: two big throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is healthy) with their provenance tag, and the latency rows (TTFT, TPOT) each with a `p95 …` foot line and its own sparkline over the selected window. Under the gufo panel's **`sessions (GET /slots)`** label sits the **slot grid**: one card per parallel slot, keyed by the slot's own id, created once and patched in place so a busy slot never flickers — request id, prompt tokens (with `processed …` while prefill runs), cache hit and its share of the prompt, generated tokens, remaining, context, and the model beside the slot number. A busy card is a border, a glyph and the word `processing`, never a green/amber/red verdict (a slot generating tokens is the job). Every cell reads exactly one key of the captured `/slots` shape — no `a ?? b` fallback chains — so a field rename fails a contract test instead of printing dashes, and `-1` sentinels print `idle`/`n/a`, never a negative countdown. When `/slots` does not answer, the panel says so with the API's own sentence rather than rendering a card of nothing, and vLLM's panel carries one line (*exposes no per-slot JSON*) and no grid. No row appears in both places — one value, one home — and there is no peak-context cell, because this page prints a number the upstream published or `n/a`. The four trend charts sit below the panels at half height, and one `/api/llm/history` request per refresh feeds all of it. With a single engine configured the page is one clean card in the strip and one clean panel — never a red half for a server you do not run. Each UPS card (`/ups`) opens with its battery glyph and three big numbers — **charge %**, **load %**, **runtime left** — coloured from the card's `severity` dict, with the electrical readings (line in, output, battery volts and amps, battery °C) kept below them in smaller muted type because they are what you read *while* the UPS is on battery. Health checks, the battery-test buttons (with their `DANGER` confirmation) and the NUT variable dump are unchanged, just second. The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event` table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath. One helper, `Dash.eventsPager()`, renders every feed and each keeps its own row markup; the fetch asks `/api/events` for 200 so there is a page 2 to reach, and the endpoint itself is unchanged. Reading page 3 survives a poll; a refresh that brings new events goes back to page 1, because page 3 of a list that just changed is not the history you were reading. ## Testing ```bash uv run pytest # unit + API tests, fully mocked, no network uv run pytest --cov # coverage gate set -a; . ./.env.dev; set +a # the live suite reads the same env as the app uv run pytest -m live --live # optional: real fleet / engines / upsd uv run ruff check src tests agent && uv run pyright # lint + types ./scripts/capture-fixtures.sh # re-record ground truth: host agents, UPS replies, gufo /slots ./scripts/capture-fixtures.sh --with-llm # ... and the engines' /metrics while they are up SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh # ... and smartctl per physical disk: plain smartctl, # `sudo -n` if that is denied; scrubs serials/WWNs bash .agents/validate.sh # all of the above + `node --check` on every page # script + repo hygiene, with the 94% coverage floor ``` Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must match an upstream (`/proc` layouts, vLLM metric names, NUT protocol framing) is asserted against a verbatim capture in `tests/fixtures/`; behaviour that needs a failure — upsd refusing a command, an engine returning 500, a host rebooting — is a fake. The `live` tests (`tests/test_live_fleet.py`) are the bridge: they SKIP unless you pass `--live`, each one also skips when its source is unset (**source `.env.dev` first** — run bare and they all skip quietly, which reads exactly like a pass), they drive the production code paths (`collectors.*`, `NutMonitor.collect`) against whatever hardware is reachable, and they never write — a battery test on live hardware is not a unit test. `tests/auth_contract.py` is two constants and the reason for them: every test that starts a server seeds the admin password `testpass` (a server with no stored verifier and no seed refuses to start, so a test that wants a running app has to give it a credential the way `.env.dev` does) and hashes it at 1 000 rounds through that same `pbkdf2_iterations` setting instead of the real 600 000 — ~0.3 s per hash times a hundred-odd servers is not a slower suite, it is a suite nobody runs. The real cost is pinned where it belongs (`test_the_default_cost_is_the_owasp_figure` and the config default test), so lowering it here cannot lower it there. Its partner is `tests/auth_client.py`: `sign_in(client)` posts that password once, asserts the sign-in worked, and puts the returned CSRF token on the client's default headers — which is what keeps the ~200 pre-existing HTTP assertions asserting the behaviour they were written for instead of the new gate. `tests/test_auth_api.py` is the perimeter, and it is deliberately awkward in one place: every anonymous request runs with `follow_redirects=False`, because a client that follows them would fetch the login page and report `200` for a route that refused it — the way a gated suite ends up passing while testing nothing. It also walks the *served* route table (`test_every_route_that_is_not_public_requires_a_session`), so a route a later phase forgets to protect fails there rather than in a browser, and it proves the one refusal that matters most on the wire rather than on a status code: a `POST /api/ups/command` without the CSRF token answers `403` **and the scripted upsd records nothing**, while the same request with the token still reaches it — and `driver.killpower` still needs the word `DANGER` behind the session, because a fifth gate that quietly replaced the fourth would be the worst kind of regression. `tests/test_auth_docs.py` tests the paperwork, because the paperwork is what an operator holds when the dashboard will not come up. It reads `.env.example` and `.env.dev.example` and demands that every `DASH_*` auth setting is declared there and that no secret-shaped **assignment** carries anything but a placeholder (the same allow-list `bash .agents/validate.sh` enforces through `git ls-files`, restated so a plain `uv run pytest` catches it on the working tree) — while the local-testing word stays legal in prose, which is the only place it can be useful. It demands that `scripts/dev-server.sh` still gets its readiness probe and its `status` from the public `/api/health` and has grown no session, token or password of its own, and that both docs still name the four public paths, the reset command, the four variables and the `DASH_COOKIE_SECURE=1` deployment note. Nothing starts a server there; a sentence that quietly disappears is the bug. One more file is worth knowing about: `tests/card_contract.py` is the table of `/api/system/snapshot` paths the system page's per-host cards are allowed to read. It is checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and again against the live fleet, so a card can never be "fixed" by inventing a number the agent does not report. `tests/ups_contract.py` does the same job for the UPS page's hero row and battery glyph (`charge %`, `load %`, `runtime left`, `battery_voltage`, the glyph's `charger_status` + `on_battery`, the card header), against the two real UPSes — recorded verbatim as `tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, with the device serials scrubbed — and again against live upsd. The captures earn their place: the CyberPower answers `battery.mfr.date: CPS`, the APC's `battery.date` is a factory placeholder, and **neither** publishes `battery.temperature` or `battery.charger.status`, so a hand-written variable table would have tested nobody's memory. `tests/slots_contract.py` pins gufo's `/slots` rows the same way — hermetically from `tests/fixtures/slots_gufo.json` (the verbatim `/slots` reply, captured whenever `GUFO_BASE_URL` is set; the idle-slot rows are the shape the grid must handle) and again against the running engine. It exists because that table was once guessed: the old slot table read field names the server never published and rendered dashes forever. It also picks the **canonical name of each fact** — the live server answers `id_task` *and* `task_id` for one request — and the page renders only the canonical key, so a rename fails a test instead of quietly becoming a fallback chain. A busy slot's `prompt` is the user's in-flight request text, so the capture script blanks it and the page never reads it. ## Battery-test buttons Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`) and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users` account that actually has the command, and a confirmation in the UI — destructive ones (deep test, calibration, `load.off`) require typing `DANGER`. The server re-checks every gate; hiding a button is not a safety control. Commands that would cut power are refused outright while the UPS is already on battery. Give the app a narrowly scoped NUT user instead of a god-mode one — per-command `instcmds`, not `actions = -`: ```ini # /etc/ups/upsd.users [dashboards] password = "..." instcmds = test.battery.start.quick instcmds = test.battery.start.deep instcmds = test.battery.stop ``` One command per `instcmds` line — that is what `man 5 upsd.users` specifies ("to specify multiple commands, use multiple instances of the instcmds field"), and a comma-separated list on one line is not that. `instcmds = all` is the alternative, and the fleet runs that: every command the drivers advertise, including the ones that cut output power. Nothing about the gates changes — the risk tier, the typed `DANGER`, `NUT_ALLOW_ACTIONS` and the on-battery refusal are what prevent a click from becoming an outage, because upsd gives the dashboard no way to ask what an account is allowed to send. upsd authenticates with `USERNAME` + `PASSWORD` (see `HELP` on your own server for the verbs it accepts — `USER`/`PASS` are not among them in 2.8.5); a user needs `instcmds`, never `actions = -`. `deploy/nut-sudoers` is the ruleset that lets the `ops` account read (and scoped-write) those files: `sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut`. Read `upsd.users` before you write it — `upsmon` authenticates from the same file, so overwriting it blind can take the box down at the wrong moment. Bind upsd to loopback (`LISTEN 127.0.0.1 3051` in `upsd.conf`) when the dashboard runs on the same box, and to a real interface plus a source-scoped firewall rule when it runs elsewhere — and do not try to scope it by client inside `upsd.conf` instead: **NUT 2.8 removed the `ACL`/`ACCEPT`/`REJECT` directives** (a build without TCP Wrappers logs `ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and ignores the line), so `LISTEN` plus a host firewall rule is the only access control `upsd.conf` offers, and an unauthenticated `LIST VAR` from any routed source is the normal state of a NUT server. `NUT_SIMULATE=1` stands in for upsd during development; everything it produces is badged `SIMULATED`, and it is never the default. ## Deployment `Containerfile` is a two-stage build: `uv.lock` is resolved in the builder (exactly the versions the tests ran against) and the runtime stage is slim Python with nothing but the venv in it. The host agent is copied in too, so one image can monitor the box it runs on. In the deployment repo the pattern is the same as every other service: quadlet container + env template + nginx/SNI on the single domain. ```bash podman build -t dashboards . podman run --rm -p 8090:8090 --env-file dashboards.env \ -v dashboards-data:/var/lib/dashboards dashboards ``` It listens on `0.0.0.0:8090` as an unprivileged user, and the SQLite history lives in `/var/lib/dashboards` — put a volume there or every restart forgets the charts **and** the admin password, because the verifier and the session key are rows in that database. `dashboards.env` therefore needs `DASH_ADMIN_PASSWORD` for the first start of a new volume; with neither the seed nor a stored verifier the container exits non-zero with the reason instead of coming up open, and the same seed is ignored once the row exists, so it can be dropped from the env file afterwards. With no sources configured `/api/health` answers `200 {"status": "empty"}`, so the container passes its health probe before its env file is right instead of crash-looping. Two more lines belong in that quadlet env template because this service faces the public domain: `DASH_COOKIE_SECURE=1`, so the session cookie never rides plain HTTP (the app also treats a request that arrived with `X-Forwarded-Proto: https` as secure, so a misconfigured flag degrades rather than leaks), and the password itself — set once, then removable, since the stored verifier wins. The nginx server block, its `X-Forwarded-*` headers and HSTS live in the deployment repo like every other service here, so a change to how the edge forwards TLS is a change in that repo, not in this one; the only thing this app reads off the proxy is that one scheme header.