dashboards/README.md

Homelab Dashboards

Three dashboards, one domain, one process, one SQLite file:

Page	What it answers	Source
/	Is anything wrong, and which dashboard do I need?	fleet + alerts in one request (/api/summary)
/system	What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it	agent/host_agent.py, our own stdlib agent per host
/llm	Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles	llama.cpp-family (gufo) /metrics + /slots, vLLM /metrics — read-only
/ups	Battery health, runtime left, power events, and battery-test buttons	NUT upsd, on whatever port it really listens on (its own wire-protocol client)

No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at runtime; every asset is served by this app.

Working on this repo? AGENTS.md holds the invariants (read-only by construction, never invent a number, unconfigured ≠ broken), the locked decisions, the real fleet facts, and the known gaps.

Quickstart

uv sync --python 3.14                       # deps + the package, editable
cp .env.example .env                        # then edit
uv run dashboards                           # http://127.0.0.1:8090
uv run uvicorn dashboards.app:app --port 8090   # equivalent
uv run pytest                               # unit tests, no network needed

Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards them to localhost):

cp .env.dev.example .env.dev                # your ssh aliases + which ports
python3 - <<'PY'                            # push the agent to each host (once)
import shlex, subprocess, pathlib
for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
    subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True)
    subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True)
PY
./scripts/dev-tunnels.sh start              # 9121/9122/9123 -> each host's agent,
                                            # + DEV_FORWARDS -> upsd on 127.0.0.1:3494
                                            # (the remote port is upsd's real one — 3051
                                            #  on driveripper, not the 3493 default)
set -a; . ./.env.dev; set +a
uv run dashboards                           # foreground
./scripts/dev-server.sh restart             # or backgrounded, with a pidfile + logs

DEV_TUNNELS starts the agent remotely and forwards it; DEV_FORWARDS forwards a service that already listens there (upsd) — that is how the UPS page reads a real battery without 3493 being reachable from anywhere but this laptop.

The host agent

agent/host_agent.py is one file, stdlib only, Python 3.10+. It reads /proc, /sys/block, /sys/class/drm and statvfs and answers GET /snapshot (everything, one JSON document; ?top=N trims the process lists), GET /info (slow-changing identity: OS, kernel, CPU model, GPU list), GET /schema (the schema version plus this document) and GET /health (the one route that never needs the token) as JSON, with a plain HTML status page at GET / so a browser pointed at an agent learns what it is looking at.

python3 host_agent.py --once                                # print one snapshot, exit
python3 host_agent.py --host 0.0.0.0 --port 9120 --token-file /etc/dashboards-agent.token
sudo deploy/install-agent.sh                                # systemd unit + user + token
deploy/install-agent.sh --status                            # unit state and /health
sudo deploy/install-agent.sh --uninstall

It is read-only by construction: there is no code path that writes, and the HTTP server only answers GET. Counters are reported as counters — the dashboard computes rates over a labelled window, so the agent never has to remember who polled it last.

Binding off-loopback without a token is refused (exit 2) rather than warned about. Prefer --token-file over --token: argv is world-readable through ps, and this token guards hostname, mounts, disk and GPU models, and process names. deploy/install-agent.sh creates a system user, generates a token once (re-runs reuse it; --rotate-token for a new one), stores it 0600 root-owned, and runs the agent under ProtectSystem=strict with no capabilities — enough to read /proc, /sys and statvfs() and nothing else.

Configuration

Everything comes from the environment (see .env.example for the annotated list). At startup a .env next to the checkout is loaded first if it exists — as a seed: a variable the container, systemd unit, or your shell already set always wins. Point DASH_ENV_FILE somewhere else to keep several installs on one box. .env.example is documentation and is never loaded, and scripts/dev-server.sh exports .env.dev itself (see the quickstart) because dev settings must not leak into a production process.

Variable	Meaning
HOST_TARGETS	name=url per host running the agent; a bearer token may ride in the URL userinfo (http://TOKEN@host:9120)
GUFO_BASE_URL / VLLM_BASE_URL	LLM engines; unset means not deployed here, and its half of the page disappears
RATE_WINDOW_SECONDS	the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and /system's footnote prints that, not this default
DASH_HOST / DASH_PORT	bind address for the dashboard itself (default 127.0.0.1:8090; the container sets 0.0.0.0)
HOST_INTERVAL_SECONDS / LLM_INTERVAL_SECONDS / NUT_INTERVAL_SECONDS	how often each source is polled (10 / 2 / 5 s)
HOST_TIMEOUT_SECONDS / LLM_TIMEOUT_SECONDS / NUT_TIMEOUT_SECONDS	per-source fetch timeout — one slow source never holds up the others
GUFO_API_KEY / VLLM_API_KEY	only if an engine is behind a bearer token; they never leave the server
NUT_HOST / NUT_PORT	upsd address — whatever port it really listens on, not always 3493. Reads are unauthenticated in stock NUT
NUT_USER / NUT_PASSWORD	only needed for instant commands (battery tests)
NUT_ALLOW_ACTIONS	1 enables the one write endpoint in the app
NUT_SIMULATE	1 replaces upsd with a labelled simulator (development only)
DASH_READ_ONLY	1 disables every mutating endpoint regardless of other flags
DASH_DB	SQLite path (default data/dashboards.sqlite3)
POLL_MS_SYSTEM / POLL_MS_LLM / POLL_MS_UPS	how often the browser refetches each page (10 s / 2 s / 5 s) — shown in the page footer
DEBUGPY / DEBUGPY_PORT	DEBUGPY=1 opens a debugpy listener (default 5678) without blocking startup
RETAIN_RAW_HOURS / RETAIN_MINUTE_DAYS / RETAIN_HOUR_DAYS	history rollup and pruning

A source that is not configured answers 404 {"configured": false} without dialling, and the page says "not configured" instead of showing a red panel for something you never installed.

Nothing that leaves the app carries a credential. A HOST_TARGETS userinfo token is parsed into an Authorization header, and every URL the browser can see — /api/config, a source's error string, the event feed — goes through config.strip_userinfo first, because an unreachable host writes its failure message into the event table and /api/events serves it.

Architecture

agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        │                                  │ + /slots (gufo)        │
        ▼                                  ▼                        ▼
src/dashboards/collectors.py ────────────────────────────────────────────┐
        │  parse + Δcounter/Δt (rates.py)                                │
        ▼                                                                ▼
sources/host.py · sources/llm.py · sources/nut.py          poller.py (one task per source)
        │        samples                    events                      │
        ▼                                                                 ▼
store.py  ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──  api.py  → web/

• poller.py — every source gets its own asyncio task, its own health record, and its own error handling. A dead host keeps its last good values and is labelled stale; it never blanks or blocks another source. And it never quietly stops: the task is that source's only poller, so a SQLite write that gets refused is reported (ok: false, the real reason, the numbers aging, one line in the event log) instead of freezing the source under a green dot.
• store.py — raw samples roll up to 1-minute and 1-hour averages and the raw rows are pruned, so a year of history is a few hundred MB rather than a disk emergency.
• severity.py — the single green/amber/red table (THRESHOLDS, plus INVERTED for the metrics where lower is worse). Rated values in the API payloads travel with a sibling severity (cpu.severity.usage_pct, memory.severity, disks.severity, a UPS card's severity.charge_pct, severity inside the rated LLM cells such as KV pressure, /api/summary rows), computed server-side, so every page colours the same way and a value nobody reported renders n/a instead of green. A metric published as a 0-1 ratio is rescaled for the verdict only — the number printed stays what the upstream published. There is no gpu.* row: these are LLM servers, a busy GPU is the job, so the GPU card (and the fleet GPU column) renders uncoloured at every utilisation — the gpu.* history series are still stored and charted.
• alerts.py — "what is wrong right now", as data. alerts.build(settings, job_states, host_payloads, engine_payloads, ups_payload, recent_events) is a pure projection of the state the poller already keeps plus the verdicts severity.py and the UPS health checks already reached — no I/O, no thresholds of its own — and /api/summary returns it as alerts, worst first: {id, severity, source, message, href, at}. A configured source that stopped answering is fail; one that is stale but still serving its last good value is warn; an unconfigured source never alerts; a dead source is one line rather than one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a marker that says how many it hid.
• web/ — four static pages, one stylesheet, hand-rolled SVG charts (charts.js). No framework, no CDN, no bundler. The overview (/) is exactly two panels — a Warnings & errors list built from /api/summary's alerts (each row links to the dashboard that explains it) and the Fleet table — and nothing else: an empty alert list reads as an explicit "all clear", and a live problem moves the panel above the table. A host that is not answering keeps its row — dimmed, every number n/a — so the fleet never quietly gets smaller and last good values are never mistaken for live ones. A big colour-coded number is Dash.hero(el, {...}) on the .hero / .hero-bar styles, coloured by the severity in the payload via Dash.severityFrom() — the only colour vocabulary is ok / warn / fail, and a value with no verdict renders n/a with no colour at all. A battery is DashCharts.battery(el, {charge, severity, state, label}): one inline SVG whose fill is the charge — clamped to 0–100, never rounded up, clipped to the plate so 0 % is an empty shell — coloured only by the payload verdict, with a shape per state (solid shell on line, bolt charging, dashed outline on battery, a dash through the body when there is no reading) and a <title> / aria-label that names it ("apc: Charge 100 percent, on line") unless the number beside it already does. Each /system host is one panel led by exactly six of them — CPU %, MEM %, TEMP, DISK % (mount underneath), GPU busy (VRAM, the APU pool and the card's temperature in the footnote — no temperature is printed when the driver exposes no sensor, never 0 °C; and likewise never coloured and with no bar: a busy GPU is the job on an LLM server), and NET on two lines (↓ in / ↑ out, never coloured: a byte rate has no good/bad). Under the numbers, each panel charts only its own host — behind a collapsed Charts toggle, because the default view is current stats and a fleet of ten hosts each drawing three plots is noise: the summary reads ▸ Charts with the window in it (1h, 24h …), and opening it is what builds and paints the charts — instantly, from the response the last refresh already fetched, and repainted on every refresh after that while it stays open. Expanding one panel never stretches its collapsed siblings (.grid.items-start). The stack is utilisation (CPU, memory, worst disk), network (↓ in / ↑ out), and GPU busy for a host that reports cards — three 120px SVG charts, each with its own legend and its own aria-label naming the host and the plotted metrics (LineChart's ariaLabel option), never a second host in sight. The fleet-wide charts, the nine-figure fleet-totals panel and the host/metric detail picker are gone, and so are the per-core charts, while cpu.usage_pct@coreN stays in SQLite. Everything else — load, clock, disk I/O, uptime, GPU power and the per-label temp breakdown, filesystems, processes — folds into a collapsed "More detail" disclosure. Each panel pulls its series in one /api/system/history request (hosts=<that host>&metrics=…&range=…) every refresh, stack open or not, and a host that stops reporting takes its panel, its Charts toggle and its charts off the page. /llm opens with the fleet: one block above the panels — ALL ENGINES, an N of M engines tag, the note sums only engines answering now, then running / waiting / prompt tok/s / decode tok/s / total tokens — and every number on it is fleet_totals(), so the page sums nothing itself. A dead engine keeps its last-good numbers in its own (stale-labelled) panel but is dropped from a total presented as current: the tag counts what answered, names the missing engine in its tooltip and dims while the sums are partial. Nothing on the block is coloured — severity.py rates no throughput and no queue depth, because 0 tok/s on an idle fleet is healthy — the dot is reachability, and fleet.seams (how the two engines count prompt tokens differently) is a tooltip on the prompt sum rather than another line of ink. Below two configured engines the block is hidden: a cross-engine sum of one engine duplicates that engine's panel. Below the fleet block sits the stats strip: one card per configured engine — reachability dot, name, age stamp, the KV-cache bar (the one rated cell: the fill and a breach's text take their colour from the payload's severity and nothing else), then prompt tok/s and decode tok/s with their provenance tag (Δ10.0s, scraped, estimated), total tokens, the sessions this engine reports (slots busy counted from gufo's /slots, or running/waiting for vLLM, plus waiting), prefix cache hit, spec accept (gufo), queue time with its p95 … foot (vLLM) — and red alarm chips for preemptions / GPU-context losses that exist only while non-zero. Then the engine panels hold the detail the strip does not print: two big throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is healthy) with their provenance tag, and the latency rows (TTFT, TPOT) each with a p95 … foot line and its own sparkline over the selected window. Under the gufo panel's sessions (GET /slots) label sits the slot grid: one card per parallel slot, keyed by the slot's own id, created once and patched in place so a busy slot never flickers — request id, prompt tokens (with processed … while prefill runs), cache hit and its share of the prompt, generated tokens, remaining, context, and the model beside the slot number. A busy card is a border, a glyph and the word processing, never a green/amber/red verdict (a slot generating tokens is the job). Every cell reads exactly one key of the captured /slots shape — no a ?? b fallback chains — so a field rename fails a contract test instead of printing dashes, and -1 sentinels print idle/n/a, never a negative countdown. When /slots does not answer, the panel says so with the API's own sentence rather than rendering a card of nothing, and vLLM's panel carries one line (exposes no per-slot JSON) and no grid. No row appears in both places — one value, one home — and there is no peak-context cell, because this page prints a number the upstream published or n/a. The four trend charts sit below the panels at half height, and one /api/llm/history request per refresh feeds all of it. With a single engine configured the page is one clean card in the strip and one clean panel — never a red half for a server you do not run. Each UPS card (/ups) opens with its battery glyph and three big numbers — charge %, load %, runtime left — coloured from the card's severity dict, with the electrical readings (line in, output, battery volts and amps, battery °C) kept below them in smaller muted type because they are what you read while the UPS is on battery. Health checks, the battery-test buttons (with their DANGER confirmation) and the NUT variable dump are unchanged, just second.

Testing

uv run pytest                                  # unit + API tests, fully mocked, no network
uv run pytest --cov                            # coverage gate
set -a; . ./.env.dev; set +a                 # the live suite reads the same env as the app
uv run pytest -m live --live                   # optional: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright            # lint + types
./scripts/capture-fixtures.sh                  # re-record ground truth: host agents, UPS replies, gufo /slots
./scripts/capture-fixtures.sh --with-llm        # ... and the engines' /metrics while they are up
bash .agents/validate.sh                       # all of the above + `node --check` on every page
                                               # script + repo hygiene, with the 94% coverage floor

Test philosophy: real captures as fixtures, mocks for behaviour. Anything that must match an upstream (/proc layouts, vLLM metric names, NUT protocol framing) is asserted against a verbatim capture in tests/fixtures/; behaviour that needs a failure — upsd refusing a command, an engine returning 500, a host rebooting — is a fake. The live tests (tests/test_live_fleet.py) are the bridge: they SKIP unless you pass --live, each one also skips when its source is unset (source .env.dev first — run bare and they all skip quietly, which reads exactly like a pass), they drive the production code paths (collectors.*, NutMonitor.collect) against whatever hardware is reachable, and they never write — a battery test on live hardware is not a unit test.

One more file is worth knowing about: tests/card_contract.py is the table of /api/system/snapshot paths the system page's per-host cards are allowed to read. It is checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and again against the live fleet, so a card can never be "fixed" by inventing a number the agent does not report. tests/ups_contract.py does the same job for the UPS page's hero row and battery glyph (charge %, load %, runtime left, battery_voltage, the glyph's charger_status + on_battery, the card header), against the two real UPSes — recorded verbatim as tests/fixtures/nut_<ups>.vars by ./scripts/capture-fixtures.sh, with the device serials scrubbed — and again against live upsd. The captures earn their place: the CyberPower answers battery.mfr.date: CPS, the APC's battery.date is a factory placeholder, and neither publishes battery.temperature or battery.charger.status, so a hand-written variable table would have tested nobody's memory. tests/slots_contract.py pins gufo's /slots rows the same way — hermetically from tests/fixtures/slots_gufo.json (the verbatim /slots reply, captured whenever GUFO_BASE_URL is set; the idle-slot rows are the shape the grid must handle) and again against the running engine. It exists because that table was once guessed: the old slot table read field names the server never published and rendered dashes forever. It also picks the canonical name of each fact — the live server answers id_task and task_id for one request — and the page renders only the canonical key, so a rename fails a test instead of quietly becoming a fallback chain. A busy slot's prompt is the user's in-flight request text, so the capture script blanks it and the page never reads it.

Battery-test buttons

Instant commands are the only mutating endpoint in the application (POST /api/ups/command) and they are gated four times: NUT_ALLOW_ACTIONS=1, not DASH_READ_ONLY, an upsd.users account that actually has the command, and a confirmation in the UI — destructive ones (deep test, calibration, load.off) require typing DANGER. The server re-checks every gate; hiding a button is not a safety control. Commands that would cut power are refused outright while the UPS is already on battery.

Give the app a narrowly scoped NUT user instead of a god-mode one — per-command instcmds, not actions = -:

# /etc/ups/upsd.users
[dashboards]
    password = "..."
    instcmds = test.battery.start.quick
    instcmds = test.battery.start.deep
    instcmds = test.battery.stop

One command per instcmds line — that is what man 5 upsd.users specifies ("to specify multiple commands, use multiple instances of the instcmds field"), and a comma-separated list on one line is not that. instcmds = all is the alternative, and the fleet runs that: every command the drivers advertise, including the ones that cut output power. Nothing about the gates changes — the risk tier, the typed DANGER, NUT_ALLOW_ACTIONS and the on-battery refusal are what prevent a click from becoming an outage, because upsd gives the dashboard no way to ask what an account is allowed to send.

upsd authenticates with USERNAME + PASSWORD (see HELP on your own server for the verbs it accepts — USER/PASS are not among them in 2.8.5); a user needs instcmds, never actions = -.

deploy/nut-sudoers is the ruleset that lets the ops account read (and scoped-write) those files: sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut. Read upsd.users before you write it — upsmon authenticates from the same file, so overwriting it blind can take the box down at the wrong moment.

Bind upsd to loopback (LISTEN 127.0.0.1 in upsd.conf) and reach it with an ssh forward (DEV_FORWARDS) or a tailnet address. NUT_SIMULATE=1 stands in for upsd during development; everything it produces is badged SIMULATED, and it is never the default.

Deployment

Containerfile is a two-stage build: uv.lock is resolved in the builder (exactly the versions the tests ran against) and the runtime stage is slim Python with nothing but the venv in it. The host agent is copied in too, so one image can monitor the box it runs on. In the deployment repo the pattern is the same as every other service: quadlet container + env template + nginx/SNI on the single domain.

podman build -t dashboards .
podman run --rm -p 8090:8090 --env-file dashboards.env \
    -v dashboards-data:/var/lib/dashboards dashboards

It listens on 0.0.0.0:8090 as an unprivileged user, and the SQLite history lives in /var/lib/dashboards — put a volume there or every restart forgets the charts. With no sources configured /api/health answers 200 {"status": "empty"}, so the container passes its health probe before its env file is right instead of crash-looping.