ducoterra
c4bb374ccc
Three dashboards served by one FastAPI app with a SQLite history store
(raw / 1m / 1h) and hand-written SVG charts: no Prometheus, no Grafana,
no CDN, no build step. Nothing is fetched from the internet at runtime.
* system: a stdlib-only agent (agent/host_agent.py) reports /proc and
/sys counters; the dashboard turns them into rates over a labelled
window, so every number says how it was measured.
* llm: gufo/llama.cpp and vLLM /metrics are parsed by an in-house
exposition parser. Measured delta/elapsed throughput beats a native
per-request gauge, and each value carries its provenance.
* ups: a NUT wire-protocol client plus health checks that name what
they looked at. Commands are gated by NUT_ALLOW_ACTIONS,
DASH_READ_ONLY and a DANGER confirmation, re-checked server-side.
Missing values render as n/a rather than 0, and an unconfigured source
answers 404 {"configured": false} without dialling anything, so "not
deployed here" never looks like an outage.
253 tests at 94% coverage against captured real fixtures; ruff and
pyright clean.
LLM-Generated: true
143 lines
8.2 KiB
Markdown
143 lines
8.2 KiB
Markdown
# Homelab Dashboards
|
|
|
|
Three dashboards, one domain, one process, one SQLite file:
|
|
|
|
| Page | What it answers | Source |
|
|
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
|
| `/` | Is anything wrong, and which dashboard do I need? | all of the below, one request |
|
|
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
|
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
|
|
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd` on 3493 (its own wire protocol client) |
|
|
|
|
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
|
runtime; every asset is served by this app.
|
|
|
|
## Quickstart
|
|
|
|
```bash
|
|
uv sync --python 3.14 # deps + the package, editable
|
|
cp .env.example .env # then edit
|
|
uv run dashboards # http://127.0.0.1:8090
|
|
uv run uvicorn dashboards.app:app --port 8090 # equivalent
|
|
uv run pytest # unit tests, no network needed
|
|
```
|
|
|
|
Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards
|
|
them to localhost):
|
|
|
|
```bash
|
|
cp .env.dev.example .env.dev # your ssh aliases + which ports
|
|
python3 - <<'PY' # push the agent to each host (once)
|
|
import shlex, subprocess, pathlib
|
|
for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
|
|
subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True)
|
|
subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True)
|
|
PY
|
|
./scripts/dev-tunnels.sh start # 9121/9122/9123 -> each host's agent
|
|
set -a; . ./.env.dev; set +a
|
|
uv run dashboards
|
|
```
|
|
|
|
## The host agent
|
|
|
|
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
|
|
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` as JSON.
|
|
|
|
```bash
|
|
python3 host_agent.py --once # print one snapshot, exit
|
|
python3 host_agent.py --host 0.0.0.0 --port 9120 --token … # serve, token required
|
|
deploy/install-agent.sh # systemd unit (run with sudo)
|
|
```
|
|
|
|
It is read-only by construction: there is no code path that writes, and the HTTP server
|
|
only answers `GET`. Counters are reported as counters — the dashboard computes rates over
|
|
a labelled window, so the agent never has to remember who polled it last.
|
|
|
|
## Configuration
|
|
|
|
Everything comes from the environment (see `.env.example` for the annotated list). At
|
|
startup a `.env` next to the checkout is loaded first if it exists — as a *seed*: a
|
|
variable the container, systemd unit, or your shell already set always wins. Point
|
|
`DASH_ENV_FILE` somewhere else to keep several installs on one box. `.env.example` is
|
|
documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev` itself
|
|
(see the quickstart) because dev settings must not leak into a production process.
|
|
|
|
| Variable | Meaning |
|
|
| ------------------------------------- | -------------------------------------------------------------- |
|
|
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
|
|
| `HOST_INTERVAL_SECONDS` | poll interval for hosts (default 10) |
|
|
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
|
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) |
|
|
| `NUT_HOST` / `NUT_PORT` | upsd address. Reads work unauthenticated in stock NUT |
|
|
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
|
| `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app |
|
|
| `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |
|
|
| `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags |
|
|
| `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) |
|
|
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |
|
|
|
|
A source that is not configured answers `404 {"configured": false}` **without dialling**,
|
|
and the page says "not configured" instead of showing a red panel for something you never
|
|
installed.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
|
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR
|
|
▼ ▼ ▼
|
|
src/dashboards/collectors.py ────────────────────────────────────────────┐
|
|
│ parse + Δcounter/Δt (rates.py) │
|
|
▼ ▼
|
|
sources/host.py · sources/llm.py · sources/nut.py poller.py (one task per source)
|
|
│ samples events │
|
|
▼ ▼
|
|
store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ── api.py → web/
|
|
```
|
|
|
|
* `poller.py` — every source gets its own asyncio task, its own health record, and its own
|
|
error handling. A dead host keeps its last good values and is labelled stale; it never
|
|
blanks or blocks another source.
|
|
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
|
|
pruned, so a year of history is a few hundred MB rather than a disk emergency.
|
|
* `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No
|
|
framework, no CDN, no bundler.
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
uv run pytest # unit + API tests, fully mocked
|
|
uv run pytest --cov # coverage gate
|
|
uv run pytest -m live --live # optional: hit the real fleet/engines
|
|
uv run ruff check src tests agent && uv run pyright # lint + types
|
|
./scripts/capture-fixtures.sh # re-record ground-truth captures
|
|
```
|
|
|
|
Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must
|
|
match an upstream (`/proc` layouts, vLLM metric names, NUT protocol framing) is asserted
|
|
against a verbatim capture in `tests/fixtures/`; behaviour that needs a failure — upsd
|
|
refusing a command, an engine returning 500, a host rebooting — is a fake. `-m live`
|
|
opt-in tests are the bridge: they run only when you point them at real hardware.
|
|
|
|
## Battery-test buttons
|
|
|
|
Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`)
|
|
and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
|
|
account that actually has `actions`, and a confirmation in the UI — destructive ones
|
|
(deep test, calibration, `load.off`) require typing `DANGER`. The server re-checks every
|
|
gate; hiding a button is not a safety control. Commands that would cut power are refused
|
|
outright while the UPS is already on battery.
|
|
|
|
## Deployment
|
|
|
|
`Containerfile` builds a slim image (the agent is copied in too, so one image can serve a
|
|
host if you want). In the deployment repo the pattern is the same as every other service:
|
|
quadlet container + env template + nginx/SNI on the single domain.
|
|
|
|
```bash
|
|
podman build -t dashboards .
|
|
podman run --rm -p 8090:8090 --env-file dashboards.env dashboards
|
|
```
|
|
|
|
Keep the SQLite file on persistent storage (`-v dashboards-data:/var/lib/dashboards`).
|