# Homelab Dashboards Three dashboards, one domain, one process, one SQLite file: | Page | What it answers | Source | | ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- | | `/` | Is anything wrong, and which dashboard do I need? | fleet + alerts in one request (`/api/summary`) | | `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host | | `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only | | `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) | No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at runtime; every asset is served by this app. Working on this repo? **[`AGENTS.md`](AGENTS.md)** holds the invariants (read-only by construction, never invent a number, unconfigured ≠ broken), the locked decisions, the real fleet facts, and the known gaps. ## Quickstart ```bash uv sync --python 3.14 # deps + the package, editable cp .env.example .env # then edit uv run dashboards # http://127.0.0.1:8090 uv run uvicorn dashboards.app:app --port 8090 # equivalent uv run pytest # unit tests, no network needed ``` Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards them to localhost): ```bash cp .env.dev.example .env.dev # your ssh aliases + which ports python3 - <<'PY' # push the agent to each host (once) import shlex, subprocess, pathlib for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"): subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True) subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True) PY ./scripts/dev-tunnels.sh start # 9121/9122/9123 -> each host's agent, # + DEV_FORWARDS -> upsd on 127.0.0.1:3494 # (the remote port is upsd's real one — 3051 # on driveripper, not the 3493 default) set -a; . ./.env.dev; set +a uv run dashboards # foreground ./scripts/dev-server.sh restart # or backgrounded, with a pidfile + logs ``` `DEV_TUNNELS` starts the agent remotely and forwards it; `DEV_FORWARDS` forwards a service that already listens there (upsd) — that is how the UPS page reads a real battery without 3493 being reachable from anywhere but this laptop. ## The host agent `agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`, `/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one JSON document), `GET /info` (slow-changing identity: OS, kernel, CPU model, GPU list), and `GET /health` (the one route that never needs the token) as JSON. ```bash python3 host_agent.py --once # print one snapshot, exit python3 host_agent.py --host 0.0.0.0 --port 9120 --token-file /etc/dashboards-agent.token sudo deploy/install-agent.sh # systemd unit + user + token deploy/install-agent.sh --status # unit state and /health sudo deploy/install-agent.sh --uninstall ``` It is read-only by construction: there is no code path that writes, and the HTTP server only answers `GET`. Counters are reported as counters — the dashboard computes rates over a labelled window, so the agent never has to remember who polled it last. Binding off-loopback without a token is refused (exit 2) rather than warned about. Prefer `--token-file` over `--token`: argv is world-readable through `ps`, and this token guards hostname, mounts, disk and GPU models, and process names. `deploy/install-agent.sh` creates a system user, generates a token once (re-runs reuse it; `--rotate-token` for a new one), stores it `0600` root-owned, and runs the agent under `ProtectSystem=strict` with no capabilities — enough to read `/proc`, `/sys` and `statvfs()` and nothing else. ## Configuration Everything comes from the environment (see `.env.example` for the annotated list). At startup a `.env` next to the checkout is loaded first if it exists — as a *seed*: a variable the container, systemd unit, or your shell already set always wins. Point `DASH_ENV_FILE` somewhere else to keep several installs on one box. `.env.example` is documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev` itself (see the quickstart) because dev settings must not leak into a production process. | Variable | Meaning | | ------------------------------------- | -------------------------------------------------------------- | | `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) | | `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears | | `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) | | `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) | | `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) | | `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others | | `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server | | `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT | | `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) | | `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app | | `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) | | `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags | | `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) | | `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer | | `DEBUGPY` / `DEBUGPY_PORT` | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup | | `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning | A source that is not configured answers `404 {"configured": false}` **without dialling**, and the page says "not configured" instead of showing a red panel for something you never installed. ## Architecture ``` agent/host_agent.py (each host) gufo / vLLM upsd (NUT) │ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR ▼ ▼ ▼ src/dashboards/collectors.py ────────────────────────────────────────────┐ │ parse + Δcounter/Δt (rates.py) │ ▼ ▼ sources/host.py · sources/llm.py · sources/nut.py poller.py (one task per source) │ samples events │ ▼ ▼ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ── api.py → web/ ``` * `poller.py` — every source gets its own asyncio task, its own health record, and its own error handling. A dead host keeps its last good values and is labelled stale; it never blanks or blocks another source. * `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are pruned, so a year of history is a few hundred MB rather than a disk emergency. * `severity.py` — the single green/amber/red table (`THRESHOLDS`, plus `INVERTED` for the metrics where lower is worse). Rated values in the API payloads travel with a sibling `severity` (`cpu.severity.usage_pct`, `memory.severity`, `disks.severity`, `gpus[].severity`, a UPS card's `severity.charge_pct`, `severity` inside the rated LLM cells such as KV pressure, `/api/summary` rows), computed server-side, so every page colours the same way and a value nobody reported renders `n/a` instead of green. A metric published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays what the upstream published. * `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states, host_payloads, engine_payloads, ups_payload, recent_events)` is a pure projection of the state the poller already keeps plus the verdicts `severity.py` and the UPS health checks already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as `alerts`, worst first: `{id, severity, source, message, href, at}`. A configured source that stopped answering is `fail`; one that is stale but still serving its last good value is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a marker that says how many it hid. * `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No framework, no CDN, no bundler. The overview (`/`) is exactly two panels — a **Warnings & errors** list built from `/api/summary`'s `alerts` (each row links to the dashboard that explains it) and the **Fleet** table — and nothing else: an empty alert list reads as an explicit "all clear", and a live problem moves the panel above the table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the fleet never quietly gets smaller and last good values are never mistaken for live ones. A big colour-coded number is `Dash.hero(el, {...})` on the `.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via `Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value with no verdict renders `n/a` with no colour at all. ## Testing ```bash uv run pytest # unit + API tests, fully mocked, no network uv run pytest --cov # coverage gate uv run pytest -m live --live # optional: real fleet / engines / upsd uv run ruff check src tests agent && uv run pyright # lint + types ./scripts/capture-fixtures.sh # re-record ground-truth captures ``` Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must match an upstream (`/proc` layouts, vLLM metric names, NUT protocol framing) is asserted against a verbatim capture in `tests/fixtures/`; behaviour that needs a failure — upsd refusing a command, an engine returning 500, a host rebooting — is a fake. The `live` tests (`tests/test_live_fleet.py`) are the bridge: they SKIP unless you pass `--live`, each one also skips when its source is unset, they drive the production code paths (`collectors.*`, `NutMonitor.collect`) against whatever hardware is reachable, and they never write — a battery test on live hardware is not a unit test. ## Battery-test buttons Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`) and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users` account that actually has the command, and a confirmation in the UI — destructive ones (deep test, calibration, `load.off`) require typing `DANGER`. The server re-checks every gate; hiding a button is not a safety control. Commands that would cut power are refused outright while the UPS is already on battery. Give the app a narrowly scoped NUT user instead of a god-mode one — per-command `instcmds`, not `actions = -`: ```ini # /etc/ups/upsd.users [dashboards] password = "..." instcmds = test.battery.start.quick test.battery.start.deep test.battery.stop ``` Bind upsd to loopback (`LISTEN 127.0.0.1` in `upsd.conf`) and reach it with an ssh forward (`DEV_FORWARDS`) or a tailnet address. `NUT_SIMULATE=1` stands in for upsd during development; everything it produces is badged `SIMULATED`, and it is never the default. ## Deployment `Containerfile` is a two-stage build: `uv.lock` is resolved in the builder (exactly the versions the tests ran against) and the runtime stage is slim Python with nothing but the venv in it. The host agent is copied in too, so one image can monitor the box it runs on. In the deployment repo the pattern is the same as every other service: quadlet container + env template + nginx/SNI on the single domain. ```bash podman build -t dashboards . podman run --rm -p 8090:8090 --env-file dashboards.env \ -v dashboards-data:/var/lib/dashboards dashboards ``` It listens on `0.0.0.0:8090` as an unprivileged user, and the SQLite history lives in `/var/lib/dashboards` — put a volume there or every restart forgets the charts. With no sources configured `/api/health` answers `200 {"status": "empty"}`, so the container passes its health probe before its env file is right instead of crash-looping.