dashboards/README.md

# Homelab Dashboards

Four dashboards, one domain, one process, one SQLite file:

| Page        | What it answers                                                                 | Source                                             |
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
| `/`         | Is anything wrong, and which dashboard do I need?                               | all four, in one request (`/api/summary`)          |
| `/system`   | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
| `/disks`    | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount) |
| `/llm`      | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
| `/ups`      | Battery health, runtime left, power events, and battery-test buttons             | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |

No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
runtime; every asset is served by this app.

Working on this repo? **[`AGENTS.md`](AGENTS.md)** holds the invariants (read-only by
construction, never invent a number, unconfigured ≠ broken), the locked decisions, the real
fleet facts, and the known gaps.

## Quickstart

```bash
uv sync --python 3.14                       # deps + the package, editable
cp .env.example .env                        # then edit
uv run dashboards                           # http://127.0.0.1:8090
uv run uvicorn dashboards.app:app --port 8090   # equivalent
uv run pytest                               # unit tests, no network needed
```

Developing against the real fleet from a laptop (agents stay unexposed — ssh forwards
them to localhost):

```bash
cp .env.dev.example .env.dev                # your ssh aliases + which ports
python3 - <<'PY'                            # push the agent to each host (once)
import shlex, subprocess, pathlib
for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
    subprocess.run(["ssh", host, "mkdir -p ~/lab-agent"], check=True)
    subprocess.run(["scp", "agent/host_agent.py", f"{host}:~/lab-agent/"], check=True)
PY
./scripts/dev-tunnels.sh start              # 9121/9122/9123 -> each host's agent,
                                            # + DEV_FORWARDS -> upsd on 127.0.0.1:3494
                                            # (the remote port is upsd's real one — 3051
                                            #  on driveripper, not the 3493 default)
set -a; . ./.env.dev; set +a
uv run dashboards                           # foreground
./scripts/dev-server.sh restart             # or backgrounded, with a pidfile + logs
```

`DEV_TUNNELS` starts the agent remotely and forwards it; `DEV_FORWARDS` forwards a service
that already listens there (upsd) — that is how the UPS page reads a real battery without
3493 being reachable from anywhere but this laptop.

## The host agent

`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS,
kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and
`GET /health` (the one route that never needs the token) as JSON, with a plain HTML status
page at `GET /` so a browser pointed at an agent learns what it is looking at.

```bash
python3 host_agent.py --once                                # print one snapshot, exit
python3 host_agent.py --host 0.0.0.0 --port 9120 --token-file /etc/dashboards-agent.token
sudo deploy/install-agent.sh                                # systemd unit + user + token
deploy/install-agent.sh --status                            # unit state and /health
sudo deploy/install-agent.sh --uninstall
```

It is read-only by construction: there is no code path that writes, and the HTTP server
only answers `GET`. Counters are reported as counters — the dashboard computes rates over
a labelled window, so the agent never has to remember who polled it last.

Per-disk `smart` health is a best-effort add-on on the whole-disk `/sys/block` list (no
partitions): the agent runs `smartctl -i -A -H` per physical disk — 5 s bound per device,
20 s per pass, results cached 4 minutes — and parses health, temperature, power-on hours,
the sector alarms and SSD wear into `disk_io.devices[].smart`. Refreshing an expired cache
happens on a background thread while the request is served the previous result, so a wedged
disk can never cost a poll more than its own timeout (only the very first pass after the
agent starts runs inline). A host without
`smartmontools`, or whose agent user cannot read the block devices, is *not broken*: every
disk reports `smart: null` with one explanatory `errors` line, and that line never makes
the system page amber. Enable SMART with `smartmontools` plus the `disk` group (which
`deploy/install-agent.sh` grants when it exists) — the agent shells `smartctl` as itself and
never calls `sudo`, so a sudoers rule alone is not enough — and **redeploy the agent**, since
one installed before phase 09 has no SMART pass to run. Opt out silently with `--no-smart`.
The drive truth this is all developed against lives in `tests/fixtures/smartctl_driveripper_*.txt`:
16 real drives, recorded with serials scrubbed, each one parsed by a test.

Binding off-loopback without a token is refused (exit 2) rather than warned about. Prefer
`--token-file` over `--token`: argv is world-readable through `ps`, and this token guards
hostname, mounts, disk and GPU models, and process names. `deploy/install-agent.sh` creates
a system user, generates a token once (re-runs reuse it; `--rotate-token` for a new one),
stores it `0600` root-owned, and runs the agent under `ProtectSystem=strict` with no
capabilities — enough to read `/proc`, `/sys` and `statvfs()` and nothing else.

## Configuration

Everything comes from the environment (see `.env.example` for the annotated list). At
startup a `.env` next to the checkout is loaded first if it exists — as a *seed*: a
variable the container, systemd unit, or your shell already set always wins. Point
`DASH_ENV_FILE` somewhere else to keep several installs on one box. `.env.example` is
documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev` itself
(see the quickstart) because dev settings must not leak into a production process.

| Variable                              | Meaning                                                        |
| ------------------------------------- | -------------------------------------------------------------- |
| `HOST_TARGETS`                        | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
| `GUFO_BASE_URL` / `VLLM_BASE_URL`     | LLM engines; unset means *not deployed here*, and its half of the page disappears |
| `RATE_WINDOW_SECONDS`                 | the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and `/system`'s footnote prints that, not this default |
| `DASH_HOST` / `DASH_PORT`             | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
| `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:<host>` jobs exist exactly when `HOST_TARGETS` does |
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and 20 s of wedged disk must not read as a dead host |
| `GUFO_API_KEY` / `VLLM_API_KEY`       | only if an engine is behind a bearer token; they never leave the server |
| `NUT_HOST` / `NUT_PORT`               | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
| `NUT_USER` / `NUT_PASSWORD`           | only needed for instant commands (battery tests)                |
| `NUT_ALLOW_ACTIONS`                   | `1` enables the one write endpoint in the app                   |
| `NUT_SIMULATE`                        | `1` replaces upsd with a **labelled** simulator (development only) |
| `DASH_READ_ONLY`                      | `1` disables every mutating endpoint regardless of other flags  |
| `DASH_DB`                             | SQLite path (default `data/dashboards.sqlite3`)                 |
| `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer. `/disks` has no `POLL_MS_*` of its own and polls at a fixed 30 s: its payload only moves when the `disks` job re-reads the agents |
| `DEBUGPY` / `DEBUGPY_PORT`            | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup |
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |

A source that is not configured answers `404 {"configured": false}` **without dialling**,
and the page says "not configured" instead of showing a red panel for something you never
installed.

Nothing that leaves the app carries a credential. A `HOST_TARGETS` userinfo token is parsed
into an `Authorization` header, and every URL the browser can see — `/api/config`, a source's
error string, the event feed — goes through `config.strip_userinfo` first, because an
unreachable host writes its failure message into the event table and `/api/events` serves it.

## Architecture

```
agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        │  (twice: system 10s, disks 5m)   │ + /slots (gufo)        │
        ▼                                  ▼                        ▼
src/dashboards/collectors.py ────────────────────────────────────────────┐
        │  parse + Δcounter/Δt (rates.py)                                │
        ▼                                                                ▼
sources/host.py · disks.py · llm.py · nut.py               poller.py (one task per source)
        │        samples                    events                      │
        ▼                                                                 ▼
store.py  ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──  api.py  → web/
```

* `poller.py` — every source gets its own asyncio task, its own health record, and its own
  error handling. A dead host keeps its last good values and is labelled stale; it never
  blanks or blocks another source. Disk health is a source of its own (`disks:<host>` in
  `/api/health`, 5-minute cadence, its own fetch timeout) reading the *same* agent
  `/snapshot` the system page reads — a separate task is what keeps a SMART poll out of the
  10-second utilisation poll and lets drive history go stale without blinking `/system`, and
  `alerts.py` is why one dead agent is still one line on the overview rather than two. And it never quietly *stops*: the task is that source's only
  poller, so a SQLite write that gets refused is reported (`ok: false`, the real reason, the
  numbers aging, one line in the event log) instead of freezing the source under a green dot.
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
  pruned, so a year of history is a few hundred MB rather than a disk emergency.
* `severity.py` — the single green/amber/red table (`THRESHOLDS`, plus `INVERTED` for the
  metrics where lower is worse). Rated values in the API payloads travel with a sibling
  `severity` (`cpu.severity.usage_pct`, `memory.severity`, `disks.severity`, a UPS card's
  `severity.charge_pct`, `severity` inside the rated LLM
  cells such as KV pressure, `/api/summary` rows), computed server-side, so every page
  colours the same way and a value nobody reported renders `n/a` instead of green. A metric
  published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays
  what the upstream published. There is no `gpu.*` row: these are LLM servers, a busy GPU
  is the job, so the GPU card (and the fleet GPU column) renders uncoloured at every
  utilisation — the `gpu.*` history series are still stored and charted. Drive health gets two
  rows: `disk.temp_c` (amber 50 °C, red 70 °C) and `disk.percentage_used` (amber 80 %, red
  100 % — an SSD that has spent its rated wear is a replacement). The sector counters
  (`disk.reallocated`, `disk.pending`, `disk.uncorrectable`) and `disk.power_on_hours` have
  **no row on purpose**: a non-zero counter is an alarm chip on presence, zero is the absence
  of a chip and never a green "healthy drive", and an odometer has no bad value.
* `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states,
  host_payloads, engine_payloads, ups_payload, recent_events)` is a pure projection of the
  state the poller already keeps plus the verdicts `severity.py` and the UPS health checks
  already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as
  `alerts`, worst first: `{id, severity, source, message, href, at}`. A configured source
  that stopped answering is `fail`; one that is stale but still serving its last good value
  is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than
  one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a
  marker that says how many it hid.
* `api.py` — the JSON every page reads: `/api/config`, `/api/health`, `/api/events`,
  `/api/history`, `/api/metrics`, `/api/summary`, and one `snapshot` + `history` pair per
  dashboard (`/api/system/*`, `/api/disks/*`, `/api/llm/*`, `/api/ups/*`). GETs everywhere
  except the single `POST /api/ups/command`. Two rules make the pages legible: a source with
  no env config answers `404 {"configured": false}` **without dialling** (so a page says
  "not configured" instead of going red), and a history query for a series that does not
  exist answers an empty series instead of an error (so a card asks for every plot it could
  draw and reads `n/a` for the ones the drive never reported). `/api/disks/snapshot` is the
  drive-health surface: one entry per host with its physical disk rows, the agent's `smart`
  block verbatim, `smartctl_available` (`true` / `false` = install smartmontools / `null` =
  the agent never said), a `severity` beside each row's rated numbers, and a `fleet` block
  summed here so the page does no arithmetic — a host that has not answered yet is listed as
  an `unreachable` stub of `n/a`s and counted, never summed. `/api/config` carries
  `disks.interval_seconds`, which is why a drive card four minutes old is behaving, not
  broken. `/api/summary` grew a `disks` block with that page (the same `fleet` sums plus one row
  per host — counts, `smartctl_available`, and `failed_devices`, so a notifier learns *which*
  drive the hardware called `FAILED` without a second call); it is additive, exactly as `alerts`
  was, and the key paths of both are pinned with and without targets configured.
* `web/` — five static pages (`/`, `/system`, `/disks`, `/llm`, `/ups`), one stylesheet, hand-rolled
  SVG charts (`charts.js`), and a masthead status dot per dashboard job (`disks` has its own:
  the same agent can answer the 10-second utilisation poll and have a wedged SMART pass). No
  framework, no CDN, no bundler, and no narration: a page never explains its own pipeline, so
  the words on screen are the ones you act on — a unit, a `Δ10.0s` provenance tag, a
  `SIMULATED` badge, or the sentence beside an empty panel. The overview (`/`) is exactly two panels — a
  **Warnings & errors** list built from `/api/summary`'s `alerts` (each row links to the
  dashboard that explains it) and the **Fleet** table — and nothing else: an empty alert
  list reads as an explicit "all clear", and a live problem moves the panel above the
  table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the
  fleet never quietly gets smaller and last good values are never mistaken for live ones.
  A big colour-coded number is `Dash.hero(el, {...})` on the
  `.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
  `Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
  with no verdict renders `n/a` with no colour at all. A battery is
  `DashCharts.battery(el, {charge, severity, state, label})`: one inline SVG whose fill *is* the
  charge — clamped to 0–100, never rounded up, clipped to the plate so 0 % is an empty shell —
  coloured only by the payload verdict, with a shape per state (solid shell on line, bolt
  charging, dashed outline on battery, a dash through the body when there is no reading) and a
  `<title>` / `aria-label` that names it ("apc: Charge 100 percent, on line") unless the number
  beside it already does. Each `/system` host is one panel led
  by exactly six of them — CPU %, MEM %, TEMP, DISK % (mount underneath), GPU busy (VRAM, the
  APU pool and the card's temperature in the footnote — no temperature is printed when the
  driver exposes no sensor, never `0 °C`; and likewise never coloured and with no bar: a busy
  GPU is the job on an LLM server), and NET on two lines (↓ in / ↑ out, never coloured: a byte
  rate has no good/bad). Under the numbers, **each panel charts only its own host** — behind a
  collapsed **Charts** toggle, because the default view is current stats and a fleet of ten
  hosts each drawing three plots is noise: the summary reads `▸ Charts` with the window in it
  (`1h`, `24h` …), and opening it is what builds and paints the charts — instantly, from the
  response the last refresh already fetched, and repainted on every refresh after that while it
  stays open. Expanding one panel never stretches its collapsed siblings
  (`.grid.items-start`). The stack is utilisation
  (CPU, memory, worst disk), network (↓ in / ↑ out), and GPU busy for a host that reports
  cards — three 120px SVG charts, each with its own legend and its own `aria-label` naming the
  host and the plotted metrics (`LineChart`'s `ariaLabel` option), never a second host in sight. The
  fleet-wide charts, the nine-figure fleet-totals panel and the host/metric detail picker are gone, and
  so are the per-core charts, while `cpu.usage_pct@coreN` stays in SQLite. Everything else —
  load, clock, disk I/O, uptime, GPU power and the per-label temp breakdown, filesystems,
  processes — folds into a collapsed
  "More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request
  (`hosts=<that host>&metrics=…&range=…`) every refresh, stack open or not, and a host that stops
  reporting takes its panel, its Charts toggle and its charts off the page.
  `/disks` is the drive-health page: **one card per physical drive** (the agent's
  whole-`/sys/block` list — never a partition, never a mount), led by the device path and the
  drive's own verdict word (`OK` / `FAILED` / `unknown`, coloured only by `severity.health`), then
  three equal cells — **Temp**, **Power-on** hours, **Wear** — coloured by `disk.temp_c` and
  `disk.percentage_used` from the API, with the odometer deliberately uncoloured because
  `severity.py` rates no power-on hours. The sector counters appear as **red chips that exist only
  while non-zero**, so a healthy drive shows no alarm row at all rather than a wall of zeroes.
  A host whose SMART cannot be read keeps its cards, dimmed with every number `n/a` and one
  `.banner` line saying why — `smartctl not available on <host>` when the agent reported the binary
  missing, its own sentence when it reported something else, and "not answering" when the agent is
  down — never a red panel, because a missing binary is unconfigured, not broken. The header's
  drive counts are the API's `fleet` block (hidden below two hosts, so one box's sums never
  duplicate its own cards), and each host's **Charts** disclosure behaves exactly like `/system`'s:
  collapsed, built only while open, and fed by one `/api/disks/history` request per host per
  refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive it has.
  `/llm` opens with
  the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, the note
  *sums only engines answering now*, then **running / waiting / prompt tok/s / decode tok/s /
  total tokens** — and every number on it is `fleet_totals()`, so the page sums nothing itself.
  A dead engine keeps its last-good numbers in its own (stale-labelled) panel but is dropped
  from a total presented as current: the tag counts what answered, names the missing engine in
  its tooltip and dims while the sums are partial. Nothing on the block is coloured —
  `severity.py` rates no throughput and no queue depth, because 0 tok/s on an idle fleet is
  healthy — the dot is reachability, and `fleet.seams` (how the two engines count prompt
  tokens differently) is a tooltip on the prompt sum rather than another line of ink. Below
  two configured engines the block is hidden: a cross-engine sum of one engine duplicates that
  engine's panel. Below the fleet block sits the **stats strip**: one card per configured
  engine — reachability dot, name, age stamp, the **KV-cache bar** (the one rated cell: the
  fill and a breach's text take their colour from the payload's `severity` and nothing else),
  then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ10.0s`, scraped,
  estimated), `total tokens`, the sessions this engine reports (`slots busy` counted from
  gufo's `/slots`, or `running`/`waiting` for vLLM, plus `waiting`), `prefix cache hit`,
  `spec accept` (gufo), `queue time` with its `p95 …` foot (vLLM) — and red alarm chips for
  preemptions / GPU-context losses that exist only while non-zero. Then the engine panels
  hold the detail the strip does not print: two big
  throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is
  healthy) with their provenance tag, and the latency rows (TTFT, TPOT) each with a `p95 …`
  foot line and its own sparkline over the selected window. Under the gufo panel's
  **`sessions (GET /slots)`** label sits the **slot grid**: one card per parallel slot, keyed by
  the slot's own id, created once and patched in place so a busy slot never flickers — request
  id, prompt tokens (with `processed …` while prefill runs), cache hit and its share of the
  prompt, generated tokens, remaining, context, and the model beside the slot number. A busy
  card is a border, a glyph and the word `processing`, never a green/amber/red verdict (a slot
  generating tokens is the job). Every cell reads exactly one key of the captured `/slots`
  shape — no `a ?? b` fallback chains — so a field rename fails a contract test instead of
  printing dashes, and `-1` sentinels print `idle`/`n/a`, never a negative countdown. When
  `/slots` does not answer, the panel says so with the API's own sentence rather than rendering
  a card of nothing, and vLLM's panel carries one line (*exposes no per-slot JSON*) and no grid.
  No row appears in both places —
  one value, one home — and there is no peak-context cell, because this page prints a number
  the upstream published or `n/a`. The four trend charts sit below the panels
  at half height, and one `/api/llm/history` request per refresh feeds all of it. With a
  single engine configured the page is one clean card in the strip and one clean panel —
  never a red half for a server you do not run. Each UPS card (`/ups`) opens with its battery glyph and
  three big numbers — **charge %**, **load %**, **runtime left** — coloured from the card's
  `severity` dict, with the electrical readings (line in, output, battery volts and amps,
  battery °C) kept below them in smaller muted type because they are what you read *while* the
  UPS is on battery. Health checks, the battery-test buttons (with their `DANGER`
  confirmation) and the NUT variable dump are unchanged, just second.
  The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event`
  table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a
  page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath.
  One helper, `Dash.eventsPager()`, renders every feed and each keeps its own row markup; the fetch
  asks `/api/events` for 200 so there is a page 2 to reach, and the endpoint itself is
  unchanged. Reading page 3 survives a poll; a refresh that brings new events goes back to
  page 1, because page 3 of a list that just changed is not the history you were reading.

## Testing

```bash
uv run pytest                                  # unit + API tests, fully mocked, no network
uv run pytest --cov                            # coverage gate
set -a; . ./.env.dev; set +a                 # the live suite reads the same env as the app
uv run pytest -m live --live                   # optional: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright            # lint + types
./scripts/capture-fixtures.sh                  # re-record ground truth: host agents, UPS replies, gufo /slots
./scripts/capture-fixtures.sh --with-llm        # ... and the engines' /metrics while they are up
SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh
                                               # ... and smartctl per physical disk: plain smartctl,
                                               # `sudo -n` if that is denied; scrubs serials/WWNs
bash .agents/validate.sh                       # all of the above + `node --check` on every page
                                               # script + repo hygiene, with the 94% coverage floor
```

Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must
match an upstream (`/proc` layouts, vLLM metric names, NUT protocol framing) is asserted
against a verbatim capture in `tests/fixtures/`; behaviour that needs a failure — upsd
refusing a command, an engine returning 500, a host rebooting — is a fake. The `live` tests
(`tests/test_live_fleet.py`) are the bridge: they SKIP unless you pass `--live`, each one
also skips when its source is unset (**source `.env.dev` first** — run bare and they all skip
quietly, which reads exactly like a pass), they drive the production code paths
(`collectors.*`, `NutMonitor.collect`) against whatever hardware is reachable, and they never
write — a battery test on live hardware is not a unit test.

One more file is worth knowing about: `tests/card_contract.py` is the table of
`/api/system/snapshot` paths the system page's per-host cards are allowed to read. It is
checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and
again against the live fleet, so a card can never be "fixed" by inventing a number the agent
does not report. `tests/ups_contract.py` does the same job for the UPS page's hero row and
battery glyph (`charge %`, `load %`, `runtime left`, `battery_voltage`, the glyph's
`charger_status` + `on_battery`, the card header), against the two real UPSes — recorded
verbatim as `tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, with the
device serials scrubbed — and again against live upsd. The captures earn their place: the
CyberPower answers `battery.mfr.date: CPS`, the APC's `battery.date` is a factory
placeholder, and **neither** publishes `battery.temperature` or `battery.charger.status`, so
a hand-written variable table would have tested nobody's memory. `tests/slots_contract.py`
pins gufo's `/slots` rows the same way — hermetically from
`tests/fixtures/slots_gufo.json` (the verbatim `/slots` reply, captured whenever
`GUFO_BASE_URL` is set; the idle-slot rows are the shape the grid must handle) and again
against the running engine. It exists because that table was once guessed: the old slot
table read field names the server never published and rendered dashes forever. It also picks
the **canonical name of each fact** — the live server answers `id_task` *and* `task_id` for
one request — and the page renders only the canonical key, so a rename fails a test instead of
quietly becoming a fallback chain. A busy
slot's `prompt` is the user's in-flight request text, so the capture script blanks it and
the page never reads it.

## Battery-test buttons

Instant commands are the only mutating endpoint in the application (`POST /api/ups/command`)
and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
account that actually has the command, and a confirmation in the UI — destructive ones
(deep test, calibration, `load.off`) require typing `DANGER`. The server re-checks every
gate; hiding a button is not a safety control. Commands that would cut power are refused
outright while the UPS is already on battery.

Give the app a narrowly scoped NUT user instead of a god-mode one — per-command `instcmds`,
not `actions = -`:

```ini
# /etc/ups/upsd.users
[dashboards]
    password = "..."
    instcmds = test.battery.start.quick
    instcmds = test.battery.start.deep
    instcmds = test.battery.stop
```

One command per `instcmds` line — that is what `man 5 upsd.users` specifies ("to specify
multiple commands, use multiple instances of the instcmds field"), and a comma-separated list
on one line is not that. `instcmds = all` is the alternative, and the fleet runs that: every
command the drivers advertise, including the ones that cut output power. Nothing about the
gates changes — the risk tier, the typed `DANGER`, `NUT_ALLOW_ACTIONS` and the on-battery
refusal are what prevent a click from becoming an outage, because upsd gives the dashboard no
way to ask what an account is allowed to send.

upsd authenticates with `USERNAME` + `PASSWORD` (see `HELP` on your own server for the verbs
it accepts — `USER`/`PASS` are not among them in 2.8.5); a user needs `instcmds`, never
`actions = -`.

`deploy/nut-sudoers` is the ruleset that lets the `ops` account read (and scoped-write) those
files: `sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut`.
Read `upsd.users` before you write it — `upsmon` authenticates from the same file, so
overwriting it blind can take the box down at the wrong moment.

Bind upsd to loopback (`LISTEN 127.0.0.1` in `upsd.conf`) and reach it with an ssh forward
(`DEV_FORWARDS`) or a tailnet address. `NUT_SIMULATE=1` stands in for upsd during
development; everything it produces is badged `SIMULATED`, and it is never the default.

## Deployment

`Containerfile` is a two-stage build: `uv.lock` is resolved in the builder (exactly the
versions the tests ran against) and the runtime stage is slim Python with nothing but the
venv in it. The host agent is copied in too, so one image can monitor the box it runs on.
In the deployment repo the pattern is the same as every other service: quadlet container +
env template + nginx/SNI on the single domain.

```bash
podman build -t dashboards .
podman run --rm -p 8090:8090 --env-file dashboards.env \
    -v dashboards-data:/var/lib/dashboards dashboards
```

It listens on `0.0.0.0:8090` as an unprivileged user, and the SQLite history lives in
`/var/lib/dashboards` — put a volume there or every restart forgets the charts. With no
sources configured `/api/health` answers `200 {"status": "empty"}`, so the container passes
its health probe before its env file is right instead of crash-looping.