Add homelab dashboards: system, LLM, and UPS on one domain
Three dashboards served by one FastAPI app with a SQLite history store
(raw / 1m / 1h) and hand-written SVG charts: no Prometheus, no Grafana,
no CDN, no build step. Nothing is fetched from the internet at runtime.
* system: a stdlib-only agent (agent/host_agent.py) reports /proc and
/sys counters; the dashboard turns them into rates over a labelled
window, so every number says how it was measured.
* llm: gufo/llama.cpp and vLLM /metrics are parsed by an in-house
exposition parser. Measured delta/elapsed throughput beats a native
per-request gauge, and each value carries its provenance.
* ups: a NUT wire-protocol client plus health checks that name what
they looked at. Commands are gated by NUT_ALLOW_ACTIONS,
DASH_READ_ONLY and a DANGER confirmation, re-checked server-side.
Missing values render as n/a rather than 0, and an unconfigured source
answers 404 {"configured": false} without dialling anything, so "not
deployed here" never looks like an outage.
253 tests at 94% coverage against captured real fixtures; ruff and
pyright clean.
LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,142 @@
|
||||
# Homelab Dashboards
|
||||
|
||||
Three dashboards, one domain, one process, one SQLite file:
|
||||
|
||||
| Page | What it answers | Source |
|
||||
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
||||
| `/` | Is anything wrong, and which dashboard do I need? | all of the below, one request |
|
||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd` on 3493 (its own wire protocol client) |
|
||||
|
||||
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
||||
runtime; every asset is served by this app.
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
uv sync --python 3.14 # deps + the package, editable
|
||||
cp .env.example .env # then edit
|
||||
uv run dashboards # http://127.0.0.1:8090
|
||||
uv run uvicorn dashboards.app:app --port 8090 # equivalent
|
||||
uv run pytest # unit tests, no network needed
|
||||
```
|
||||
|
||||
Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards
|
||||
them to localhost):
|
||||
|
||||
```bash
|
||||
cp .env.dev.example .env.dev # your ssh aliases + which ports
|
||||
python3 - <<'PY' # push the agent to each host (once)
|
||||
import shlex, subprocess, pathlib
|
||||
for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
|
||||
subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True)
|
||||
subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True)
|
||||
PY
|
||||
./scripts/dev-tunnels.sh start # 9121/9122/9123 -> each host's agent
|
||||
set -a; . ./.env.dev; set +a
|
||||
uv run dashboards
|
||||
```
|
||||
|
||||
## The host agent
|
||||
|
||||
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
|
||||
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` as JSON.
|
||||
|
||||
```bash
|
||||
python3 host_agent.py --once # print one snapshot, exit
|
||||
python3 host_agent.py --host 0.0.0.0 --port 9120 --token … # serve, token required
|
||||
deploy/install-agent.sh # systemd unit (run with sudo)
|
||||
```
|
||||
|
||||
It is read-only by construction: there is no code path that writes, and the HTTP server
|
||||
only answers `GET`. Counters are reported as counters — the dashboard computes rates over
|
||||
a labelled window, so the agent never has to remember who polled it last.
|
||||
|
||||
## Configuration
|
||||
|
||||
Everything comes from the environment (see `.env.example` for the annotated list). At
|
||||
startup a `.env` next to the checkout is loaded first if it exists — as a *seed*: a
|
||||
variable the container, systemd unit, or your shell already set always wins. Point
|
||||
`DASH_ENV_FILE` somewhere else to keep several installs on one box. `.env.example` is
|
||||
documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev` itself
|
||||
(see the quickstart) because dev settings must not leak into a production process.
|
||||
|
||||
| Variable | Meaning |
|
||||
| ------------------------------------- | -------------------------------------------------------------- |
|
||||
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
|
||||
| `HOST_INTERVAL_SECONDS` | poll interval for hosts (default 10) |
|
||||
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
||||
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) |
|
||||
| `NUT_HOST` / `NUT_PORT` | upsd address. Reads work unauthenticated in stock NUT |
|
||||
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
||||
| `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app |
|
||||
| `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |
|
||||
| `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags |
|
||||
| `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) |
|
||||
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |
|
||||
|
||||
A source that is not configured answers `404 {"configured": false}` **without dialling**,
|
||||
and the page says "not configured" instead of showing a red panel for something you never
|
||||
installed.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
||||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR
|
||||
▼ ▼ ▼
|
||||
src/dashboards/collectors.py ────────────────────────────────────────────┐
|
||||
│ parse + Δcounter/Δt (rates.py) │
|
||||
▼ ▼
|
||||
sources/host.py · sources/llm.py · sources/nut.py poller.py (one task per source)
|
||||
│ samples events │
|
||||
▼ ▼
|
||||
store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ── api.py → web/
|
||||
```
|
||||
|
||||
* `poller.py` — every source gets its own asyncio task, its own health record, and its own
|
||||
error handling. A dead host keeps its last good values and is labelled stale; it never
|
||||
blanks or blocks another source.
|
||||
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
|
||||
pruned, so a year of history is a few hundred MB rather than a disk emergency.
|
||||
* `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No
|
||||
framework, no CDN, no bundler.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
uv run pytest # unit + API tests, fully mocked
|
||||
uv run pytest --cov # coverage gate
|
||||
uv run pytest -m live --live # optional: hit the real fleet/engines
|
||||
uv run ruff check src tests agent && uv run pyright # lint + types
|
||||
./scripts/capture-fixtures.sh # re-record ground-truth captures
|
||||
```
|
||||
|
||||
Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must
|
||||
match an upstream (`/proc` layouts, vLLM metric names, NUT protocol framing) is asserted
|
||||
against a verbatim capture in `tests/fixtures/`; behaviour that needs a failure — upsd
|
||||
refusing a command, an engine returning 500, a host rebooting — is a fake. `-m live`
|
||||
opt-in tests are the bridge: they run only when you point them at real hardware.
|
||||
|
||||
## Battery-test buttons
|
||||
|
||||
Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`)
|
||||
and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
|
||||
account that actually has `actions`, and a confirmation in the UI — destructive ones
|
||||
(deep test, calibration, `load.off`) require typing `DANGER`. The server re-checks every
|
||||
gate; hiding a button is not a safety control. Commands that would cut power are refused
|
||||
outright while the UPS is already on battery.
|
||||
|
||||
## Deployment
|
||||
|
||||
`Containerfile` builds a slim image (the agent is copied in too, so one image can serve a
|
||||
host if you want). In the deployment repo the pattern is the same as every other service:
|
||||
quadlet container + env template + nginx/SNI on the single domain.
|
||||
|
||||
```bash
|
||||
podman build -t dashboards .
|
||||
podman run --rm -p 8090:8090 --env-file dashboards.env dashboards
|
||||
```
|
||||
|
||||
Keep the SQLite file on persistent storage (`-v dashboards-data:/var/lib/dashboards`).
|
||||
Reference in New Issue
Block a user