ducoterra
8b048599f3
**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**
- Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
220 lines
14 KiB
Markdown
220 lines
14 KiB
Markdown
# Homelab Dashboards
|
|
|
|
Three dashboards, one domain, one process, one SQLite file:
|
|
|
|
| Page | What it answers | Source |
|
|
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
|
| `/` | Is anything wrong, and which dashboard do I need? | fleet + alerts in one request (`/api/summary`) |
|
|
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
|
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
|
|
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
|
|
|
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
|
runtime; every asset is served by this app.
|
|
|
|
Working on this repo? **[`AGENTS.md`](AGENTS.md)** holds the invariants (read-only by
|
|
construction, never invent a number, unconfigured ≠ broken), the locked decisions, the real
|
|
fleet facts, and the known gaps.
|
|
|
|
## Quickstart
|
|
|
|
```bash
|
|
uv sync --python 3.14 # deps + the package, editable
|
|
cp .env.example .env # then edit
|
|
uv run dashboards # http://127.0.0.1:8090
|
|
uv run uvicorn dashboards.app:app --port 8090 # equivalent
|
|
uv run pytest # unit tests, no network needed
|
|
```
|
|
|
|
Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards
|
|
them to localhost):
|
|
|
|
```bash
|
|
cp .env.dev.example .env.dev # your ssh aliases + which ports
|
|
python3 - <<'PY' # push the agent to each host (once)
|
|
import shlex, subprocess, pathlib
|
|
for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
|
|
subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True)
|
|
subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True)
|
|
PY
|
|
./scripts/dev-tunnels.sh start # 9121/9122/9123 -> each host's agent,
|
|
# + DEV_FORWARDS -> upsd on 127.0.0.1:3494
|
|
# (the remote port is upsd's real one — 3051
|
|
# on driveripper, not the 3493 default)
|
|
set -a; . ./.env.dev; set +a
|
|
uv run dashboards # foreground
|
|
./scripts/dev-server.sh restart # or backgrounded, with a pidfile + logs
|
|
```
|
|
|
|
`DEV_TUNNELS` starts the agent remotely and forwards it; `DEV_FORWARDS` forwards a service
|
|
that already listens there (upsd) — that is how the UPS page reads a real battery without
|
|
3493 being reachable from anywhere but this laptop.
|
|
|
|
## The host agent
|
|
|
|
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
|
|
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
|
|
JSON document), `GET /info` (slow-changing identity: OS, kernel, CPU model, GPU list), and
|
|
`GET /health` (the one route that never needs the token) as JSON.
|
|
|
|
```bash
|
|
python3 host_agent.py --once # print one snapshot, exit
|
|
python3 host_agent.py --host 0.0.0.0 --port 9120 --token-file /etc/dashboards-agent.token
|
|
sudo deploy/install-agent.sh # systemd unit + user + token
|
|
deploy/install-agent.sh --status # unit state and /health
|
|
sudo deploy/install-agent.sh --uninstall
|
|
```
|
|
|
|
It is read-only by construction: there is no code path that writes, and the HTTP server
|
|
only answers `GET`. Counters are reported as counters — the dashboard computes rates over
|
|
a labelled window, so the agent never has to remember who polled it last.
|
|
|
|
Binding off-loopback without a token is refused (exit 2) rather than warned about. Prefer
|
|
`--token-file` over `--token`: argv is world-readable through `ps`, and this token guards
|
|
hostname, mounts, disk and GPU models, and process names. `deploy/install-agent.sh` creates
|
|
a system user, generates a token once (re-runs reuse it; `--rotate-token` for a new one),
|
|
stores it `0600` root-owned, and runs the agent under `ProtectSystem=strict` with no
|
|
capabilities — enough to read `/proc`, `/sys` and `statvfs()` and nothing else.
|
|
|
|
## Configuration
|
|
|
|
Everything comes from the environment (see `.env.example` for the annotated list). At
|
|
startup a `.env` next to the checkout is loaded first if it exists — as a *seed*: a
|
|
variable the container, systemd unit, or your shell already set always wins. Point
|
|
`DASH_ENV_FILE` somewhere else to keep several installs on one box. `.env.example` is
|
|
documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev` itself
|
|
(see the quickstart) because dev settings must not leak into a production process.
|
|
|
|
| Variable | Meaning |
|
|
| ------------------------------------- | -------------------------------------------------------------- |
|
|
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
|
|
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
|
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) |
|
|
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
|
|
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
|
|
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
|
|
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
|
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
|
|
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
|
| `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app |
|
|
| `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |
|
|
| `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags |
|
|
| `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) |
|
|
| `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer |
|
|
| `DEBUGPY` / `DEBUGPY_PORT` | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup |
|
|
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |
|
|
|
|
A source that is not configured answers `404 {"configured": false}` **without dialling**,
|
|
and the page says "not configured" instead of showing a red panel for something you never
|
|
installed.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
|
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR
|
|
▼ ▼ ▼
|
|
src/dashboards/collectors.py ────────────────────────────────────────────┐
|
|
│ parse + Δcounter/Δt (rates.py) │
|
|
▼ ▼
|
|
sources/host.py · sources/llm.py · sources/nut.py poller.py (one task per source)
|
|
│ samples events │
|
|
▼ ▼
|
|
store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ── api.py → web/
|
|
```
|
|
|
|
* `poller.py` — every source gets its own asyncio task, its own health record, and its own
|
|
error handling. A dead host keeps its last good values and is labelled stale; it never
|
|
blanks or blocks another source.
|
|
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
|
|
pruned, so a year of history is a few hundred MB rather than a disk emergency.
|
|
* `severity.py` — the single green/amber/red table (`THRESHOLDS`, plus `INVERTED` for the
|
|
metrics where lower is worse). Rated values in the API payloads travel with a sibling
|
|
`severity` (`cpu.severity.usage_pct`, `memory.severity`, `disks.severity`,
|
|
`gpus[].severity`, a UPS card's `severity.charge_pct`, `severity` inside the rated LLM
|
|
cells such as KV pressure, `/api/summary` rows), computed server-side, so every page
|
|
colours the same way and a value nobody reported renders `n/a` instead of green. A metric
|
|
published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays
|
|
what the upstream published.
|
|
* `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states,
|
|
host_payloads, engine_payloads, ups_payload, recent_events)` is a pure projection of the
|
|
state the poller already keeps plus the verdicts `severity.py` and the UPS health checks
|
|
already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as
|
|
`alerts`, worst first: `{id, severity, source, message, href, at}`. A configured source
|
|
that stopped answering is `fail`; one that is stale but still serving its last good value
|
|
is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than
|
|
one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a
|
|
marker that says how many it hid.
|
|
* `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No
|
|
framework, no CDN, no bundler. The overview (`/`) is exactly two panels — a
|
|
**Warnings & errors** list built from `/api/summary`'s `alerts` (each row links to the
|
|
dashboard that explains it) and the **Fleet** table — and nothing else: an empty alert
|
|
list reads as an explicit "all clear", and a live problem moves the panel above the
|
|
table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the
|
|
fleet never quietly gets smaller and last good values are never mistaken for live ones.
|
|
A big colour-coded number is `Dash.hero(el, {...})` on the
|
|
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
|
|
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
|
|
with no verdict renders `n/a` with no colour at all.
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
uv run pytest # unit + API tests, fully mocked, no network
|
|
uv run pytest --cov # coverage gate
|
|
uv run pytest -m live --live # optional: real fleet / engines / upsd
|
|
uv run ruff check src tests agent && uv run pyright # lint + types
|
|
./scripts/capture-fixtures.sh # re-record ground-truth captures
|
|
```
|
|
|
|
Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must
|
|
match an upstream (`/proc` layouts, vLLM metric names, NUT protocol framing) is asserted
|
|
against a verbatim capture in `tests/fixtures/`; behaviour that needs a failure — upsd
|
|
refusing a command, an engine returning 500, a host rebooting — is a fake. The `live` tests
|
|
(`tests/test_live_fleet.py`) are the bridge: they SKIP unless you pass `--live`, each one
|
|
also skips when its source is unset, they drive the production code paths (`collectors.*`,
|
|
`NutMonitor.collect`) against whatever hardware is reachable, and they never write — a
|
|
battery test on live hardware is not a unit test.
|
|
|
|
## Battery-test buttons
|
|
|
|
Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`)
|
|
and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
|
|
account that actually has the command, and a confirmation in the UI — destructive ones
|
|
(deep test, calibration, `load.off`) require typing `DANGER`. The server re-checks every
|
|
gate; hiding a button is not a safety control. Commands that would cut power are refused
|
|
outright while the UPS is already on battery.
|
|
|
|
Give the app a narrowly scoped NUT user instead of a god-mode one — per-command `instcmds`,
|
|
not `actions = -`:
|
|
|
|
```ini
|
|
# /etc/ups/upsd.users
|
|
[dashboards]
|
|
password = "..."
|
|
instcmds = test.battery.start.quick test.battery.start.deep test.battery.stop
|
|
```
|
|
|
|
Bind upsd to loopback (`LISTEN 127.0.0.1` in `upsd.conf`) and reach it with an ssh forward
|
|
(`DEV_FORWARDS`) or a tailnet address. `NUT_SIMULATE=1` stands in for upsd during
|
|
development; everything it produces is badged `SIMULATED`, and it is never the default.
|
|
|
|
## Deployment
|
|
|
|
`Containerfile` is a two-stage build: `uv.lock` is resolved in the builder (exactly the
|
|
versions the tests ran against) and the runtime stage is slim Python with nothing but the
|
|
venv in it. The host agent is copied in too, so one image can monitor the box it runs on.
|
|
In the deployment repo the pattern is the same as every other service: quadlet container +
|
|
env template + nginx/SNI on the single domain.
|
|
|
|
```bash
|
|
podman build -t dashboards .
|
|
podman run --rm -p 8090:8090 --env-file dashboards.env \
|
|
-v dashboards-data:/var/lib/dashboards dashboards
|
|
```
|
|
|
|
It listens on `0.0.0.0:8090` as an unprivileged user, and the SQLite history lives in
|
|
`/var/lib/dashboards` — put a volume there or every restart forgets the charts. With no
|
|
sources configured `/api/health` answers `200 {"status": "empty"}`, so the container passes
|
|
its health probe before its env file is right instead of crash-looping.
|