dashboards/README.md

Homelab Dashboards

Three dashboards, one domain, one process, one SQLite file:

Page	What it answers	Source
/	Is anything wrong, and which dashboard do I need?	all of the below, one request
/system	What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it	agent/host_agent.py, our own stdlib agent per host
/llm	Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles	llama.cpp-family (gufo) + vLLM /metrics, read-only
/ups	Battery health, runtime left, power events, and battery-test buttons	NUT upsd on 3493 (its own wire protocol client)

No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at runtime; every asset is served by this app.

Working on this repo? AGENTS.md holds the invariants (read-only by construction, never invent a number, unconfigured ≠ broken), the locked decisions, the real fleet facts, and the known gaps.

Quickstart

uv sync --python 3.14                       # deps + the package, editable
cp .env.example .env                        # then edit
uv run dashboards                           # http://127.0.0.1:8090
uv run uvicorn dashboards.app:app --port 8090   # equivalent
uv run pytest                               # unit tests, no network needed

Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards them to localhost):

cp .env.dev.example .env.dev                # your ssh aliases + which ports
python3 - <<'PY'                            # push the agent to each host (once)
import shlex, subprocess, pathlib
for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
    subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True)
    subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True)
PY
./scripts/dev-tunnels.sh start              # 9121/9122/9123 -> each host's agent,
                                            # + DEV_FORWARDS -> upsd on 127.0.0.1:3494
set -a; . ./.env.dev; set +a
uv run dashboards                           # foreground
./scripts/dev-server.sh restart             # or backgrounded, with a pidfile + logs

DEV_TUNNELS starts the agent remotely and forwards it; DEV_FORWARDS forwards a service that already listens there (upsd) — that is how the UPS page reads a real battery without 3493 being reachable from anywhere but this laptop.

The host agent

agent/host_agent.py is one file, stdlib only, Python 3.10+. It reads /proc, /sys/block, /sys/class/drm and statvfs and answers GET /snapshot as JSON.

python3 host_agent.py --once                                # print one snapshot, exit
python3 host_agent.py --host 0.0.0.0 --port 9120 --token-file /etc/dashboards-agent.token
sudo deploy/install-agent.sh                                # systemd unit + user + token
deploy/install-agent.sh --status                            # unit state and /health
sudo deploy/install-agent.sh --uninstall

It is read-only by construction: there is no code path that writes, and the HTTP server only answers GET. Counters are reported as counters — the dashboard computes rates over a labelled window, so the agent never has to remember who polled it last.

Binding off-loopback without a token is refused (exit 2) rather than warned about. Prefer --token-file over --token: argv is world-readable through ps, and this token guards hostname, mounts, disk and GPU models, and process names. deploy/install-agent.sh creates a system user, generates a token once (re-runs reuse it; --rotate-token for a new one), stores it 0600 root-owned, and runs the agent under ProtectSystem=strict with no capabilities — enough to read /proc, /sys and statvfs() and nothing else.

Configuration

Everything comes from the environment (see .env.example for the annotated list). At startup a .env next to the checkout is loaded first if it exists — as a seed: a variable the container, systemd unit, or your shell already set always wins. Point DASH_ENV_FILE somewhere else to keep several installs on one box. .env.example is documentation and is never loaded, and scripts/dev-server.sh exports .env.dev itself (see the quickstart) because dev settings must not leak into a production process.

Variable	Meaning
HOST_TARGETS	name=url per host running the agent; a bearer token may ride in the URL userinfo (http://TOKEN@host:9120)
HOST_INTERVAL_SECONDS	poll interval for hosts (default 10)
GUFO_BASE_URL / VLLM_BASE_URL	LLM engines; unset means not deployed here, and its half of the page disappears
RATE_WINDOW_SECONDS	the Δ window behind every measured token rate (default 10)
NUT_HOST / NUT_PORT	upsd address. Reads work unauthenticated in stock NUT
NUT_USER / NUT_PASSWORD	only needed for instant commands (battery tests)
NUT_ALLOW_ACTIONS	1 enables the one write endpoint in the app
NUT_SIMULATE	1 replaces upsd with a labelled simulator (development only)
DASH_READ_ONLY	1 disables every mutating endpoint regardless of other flags
DASH_DB	SQLite path (default data/dashboards.sqlite3)
RETAIN_RAW_HOURS / RETAIN_MINUTE_DAYS / RETAIN_HOUR_DAYS	history rollup and pruning

A source that is not configured answers 404 {"configured": false} without dialling, and the page says "not configured" instead of showing a red panel for something you never installed.

Architecture

agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR
        ▼                                  ▼                        ▼
src/dashboards/collectors.py ────────────────────────────────────────────┐
        │  parse + Δcounter/Δt (rates.py)                                │
        ▼                                                                ▼
sources/host.py · sources/llm.py · sources/nut.py          poller.py (one task per source)
        │        samples                    events                      │
        ▼                                                                 ▼
store.py  ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──  api.py  → web/

• poller.py — every source gets its own asyncio task, its own health record, and its own error handling. A dead host keeps its last good values and is labelled stale; it never blanks or blocks another source.
• store.py — raw samples roll up to 1-minute and 1-hour averages and the raw rows are pruned, so a year of history is a few hundred MB rather than a disk emergency.
• web/ — four static pages, one stylesheet, hand-rolled SVG charts (charts.js). No framework, no CDN, no bundler.

Testing

uv run pytest                                  # unit + API tests, fully mocked, no network
uv run pytest --cov                            # coverage gate
uv run pytest -m live --live                   # optional: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright            # lint + types
./scripts/capture-fixtures.sh                  # re-record ground-truth captures

Test philosophy: real captures as fixtures, mocks for behaviour. Anything that must match an upstream (/proc layouts, vLLM metric names, NUT protocol framing) is asserted against a verbatim capture in tests/fixtures/; behaviour that needs a failure — upsd refusing a command, an engine returning 500, a host rebooting — is a fake. The live tests (tests/test_live_fleet.py) are the bridge: they SKIP unless you pass --live, each one also skips when its source is unset, they drive the production code paths (collectors.*, NutMonitor.collect) against whatever hardware is reachable, and they never write — a battery test on live hardware is not a unit test.

Battery-test buttons

Instant commands are the only mutating endpoint in the application (POST /api/ups/command) and they are gated four times: NUT_ALLOW_ACTIONS=1, not DASH_READ_ONLY, an upsd.users account that actually has the command, and a confirmation in the UI — destructive ones (deep test, calibration, load.off) require typing DANGER. The server re-checks every gate; hiding a button is not a safety control. Commands that would cut power are refused outright while the UPS is already on battery.

Give the app a narrowly scoped NUT user instead of a god-mode one — per-command instcmds, not actions = -:

# /etc/ups/upsd.users
[dashboards]
    password = "..."
    instcmds = test.battery.start.quick test.battery.start.deep test.battery.stop

Bind upsd to loopback (LISTEN 127.0.0.1 in upsd.conf) and reach it with an ssh forward (DEV_FORWARDS) or a tailnet address. NUT_SIMULATE=1 stands in for upsd during development; everything it produces is badged SIMULATED, and it is never the default.

Deployment

Containerfile is a two-stage build: uv.lock is resolved in the builder (exactly the versions the tests ran against) and the runtime stage is slim Python with nothing but the venv in it. The host agent is copied in too, so one image can monitor the box it runs on. In the deployment repo the pattern is the same as every other service: quadlet container + env template + nginx/SNI on the single domain.

podman build -t dashboards .
podman run --rm -p 8090:8090 --env-file dashboards.env \
    -v dashboards-data:/var/lib/dashboards dashboards

It listens on 0.0.0.0:8090 as an unprivileged user, and the SQLite history lives in /var/lib/dashboards — put a volume there or every restart forgets the charts. With no sources configured /api/health answers 200 {"status": "empty"}, so the container passes its health probe before its env file is right instead of crash-looping.