# Homelab Dashboards

Three dashboards, one domain, one process, one SQLite file:

| Page        | What it answers                                                                 | Source                                             |
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
| `/`         | Is anything wrong, and which dashboard do I need?                               | all of the below, one request                      |
| `/system`   | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
| `/llm`      | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
| `/ups`      | Battery health, runtime left, power events, and battery-test buttons             | NUT `upsd` on 3493 (its own wire protocol client)   |

No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
runtime; every asset is served by this app.

## Quickstart

```bash
uv sync --python 3.14                       # deps + the package, editable
cp .env.example .env                        # then edit
uv run dashboards                           # http://127.0.0.1:8090
uv run uvicorn dashboards.app:app --port 8090   # equivalent
uv run pytest                               # unit tests, no network needed
```

Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards
them to localhost):

```bash
cp .env.dev.example .env.dev                # your ssh aliases + which ports
python3 - <<'PY'                            # push the agent to each host (once)
import shlex, subprocess, pathlib
for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
    subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True)
    subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True)
PY
./scripts/dev-tunnels.sh start              # 9121/9122/9123 -> each host's agent
set -a; . ./.env.dev; set +a
uv run dashboards
```

## The host agent

`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` as JSON.

```bash
python3 host_agent.py --once                                # print one snapshot, exit
python3 host_agent.py --host 0.0.0.0 --port 9120 --token …  # serve, token required
deploy/install-agent.sh                                     # systemd unit (run with sudo)
```

It is read-only by construction: there is no code path that writes, and the HTTP server
only answers `GET`. Counters are reported as counters — the dashboard computes rates over
a labelled window, so the agent never has to remember who polled it last.

## Configuration

Everything comes from the environment (see `.env.example` for the annotated list). At
startup a `.env` next to the checkout is loaded first if it exists — as a *seed*: a
variable the container, systemd unit, or your shell already set always wins. Point
`DASH_ENV_FILE` somewhere else to keep several installs on one box. `.env.example` is
documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev` itself
(see the quickstart) because dev settings must not leak into a production process.

| Variable                              | Meaning                                                        |
| ------------------------------------- | -------------------------------------------------------------- |
| `HOST_TARGETS`                        | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
| `HOST_INTERVAL_SECONDS`               | poll interval for hosts (default 10)                            |
| `GUFO_BASE_URL` / `VLLM_BASE_URL`     | LLM engines; unset means *not deployed here*, and its half of the page disappears |
| `RATE_WINDOW_SECONDS`                 | the Δ window behind every measured token rate (default 10)      |
| `NUT_HOST` / `NUT_PORT`               | upsd address. Reads work unauthenticated in stock NUT           |
| `NUT_USER` / `NUT_PASSWORD`           | only needed for instant commands (battery tests)                |
| `NUT_ALLOW_ACTIONS`                   | `1` enables the one write endpoint in the app                   |
| `NUT_SIMULATE`                        | `1` replaces upsd with a **labelled** simulator (development only) |
| `DASH_READ_ONLY`                      | `1` disables every mutating endpoint regardless of other flags  |
| `DASH_DB`                             | SQLite path (default `data/dashboards.sqlite3`)                 |
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |

A source that is not configured answers `404 {"configured": false}` **without dialling**,
and the page says "not configured" instead of showing a red panel for something you never
installed.

## Architecture

```
agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR
        ▼                                  ▼                        ▼
src/dashboards/collectors.py ────────────────────────────────────────────┐
        │  parse + Δcounter/Δt (rates.py)                                │
        ▼                                                                ▼
sources/host.py · sources/llm.py · sources/nut.py          poller.py (one task per source)
        │        samples                    events                      │
        ▼                                                                 ▼
store.py  ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──  api.py  → web/
```

* `poller.py` — every source gets its own asyncio task, its own health record, and its own
  error handling. A dead host keeps its last good values and is labelled stale; it never
  blanks or blocks another source.
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
  pruned, so a year of history is a few hundred MB rather than a disk emergency.
* `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No
  framework, no CDN, no bundler.

## Testing

```bash
uv run pytest                                  # unit + API tests, fully mocked
uv run pytest --cov                            # coverage gate
uv run pytest -m live --live                   # optional: hit the real fleet/engines
uv run ruff check src tests agent && uv run pyright            # lint + types
./scripts/capture-fixtures.sh                  # re-record ground-truth captures
```

Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must
match an upstream (`/proc` layouts, vLLM metric names, NUT protocol framing) is asserted
against a verbatim capture in `tests/fixtures/`; behaviour that needs a failure — upsd
refusing a command, an engine returning 500, a host rebooting — is a fake. `-m live`
opt-in tests are the bridge: they run only when you point them at real hardware.

## Battery-test buttons

Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`)
and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
account that actually has `actions`, and a confirmation in the UI — destructive ones
(deep test, calibration, `load.off`) require typing `DANGER`. The server re-checks every
gate; hiding a button is not a safety control. Commands that would cut power are refused
outright while the UPS is already on battery.

## Deployment

`Containerfile` builds a slim image (the agent is copied in too, so one image can serve a
host if you want). In the deployment repo the pattern is the same as every other service:
quadlet container + env template + nginx/SNI on the single domain.

```bash
podman build -t dashboards .
podman run --rm -p 8090:8090 --env-file dashboards.env dashboards
```

Keep the SQLite file on persistent storage (`-v dashboards-data:/var/lib/dashboards`).