dashboards/AGENTS.md

# AGENTS.md — homelab dashboards

The plan, the rules, and the environment facts an agent needs to work on this repo without
re-deriving them. `README.md` is the operator-facing how-to; this file is the why, the
invariants, and the state of the fleet.

## What this is

Three dashboards for one homelab, behind one domain, in one process, backed by one SQLite
file — plus an overview page that says which of the three you need:

| Route     | Question it answers                                                            | Source                                            |
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
| `/`       | Is anything wrong, and where do I look first?                                  | all three, in one request                         |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history   | `agent/host_agent.py`, our own stdlib agent/host  |
| `/llm`    | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles      | gufo (llama.cpp family) + vLLM `/metrics`         |
| `/ups`    | Battery health, runtime left, power events, battery-test buttons                | NUT `upsd` on 3493 (our own wire-protocol client) |

History is a first-class feature, not a sparkline afterthought: every number worth graphing
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.

## Non-negotiable rules

These are the things that make the numbers trustworthy. Breaking one is a bug even if the
page still renders.

1. **Read-only by construction, with exactly one exception.** Nothing in this app writes to
   an upstream except `POST /api/ups/command`. That endpoint is gated four times
   (`NUT_ALLOW_ACTIONS`, not `DASH_READ_ONLY`, the upsd.users account actually having the
   command, and a UI confirmation — `DANGER` for anything that can cut power) and the server
   re-checks every gate on every request. Hiding a button is not a safety control.
   Commands that would cut output are refused outright while the UPS is already on battery.
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
   as `0`. Every derived value carries the `source` string it came from and `derived: true`
   so the UI can print *how* it was obtained (`computed:Δtokens/Δt`, `Δ10.0s`). Aggregating
   two different series into one rate is the failure mode this rule exists to catch — the
   rate estimator is keyed by the *resolved* upstream series name for exactly that reason.
3. **Unconfigured is not broken.** A source with no env config answers
   `404 {"configured": false}` **without dialling**, and the page says "not configured".
   A red panel for a server the operator never installed destroys the signal from the
   panels that matter.
4. **One dead source must not blink another.** Each source is its own asyncio task with its
   own timeout and health record (`poller.py`). A failed poll keeps the last good payload and
   marks it stale in the UI instead of zeroing it. Transitions write events; repeated failure
   writes one line, not 500.
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
   charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
   no outbound access — a CDN reference is a request that never comes back.
6. **Never leak a credential.** Target URLs may carry a userinfo token; `/api/config`
   exposes them only through `Target.display_url`, which strips userinfo. No token,
   password, or personal hostname goes into tracked files, fixtures, or the browser.
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
   `sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
   2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
   `scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
   exporters) rather than trusting a README or a memory of an upstream.
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
   Check after touching dependencies: `uv pip list --outdated` must come back empty.
   Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
   `python3.14`) — do not lower these to make something else work.
9. **SQLite owns the history and must stay bounded.** Writes go through `store.py`; new
   metrics need a rollup path and a retention story, or a year of charts becomes a disk
   emergency. Put the DB on a volume in deployment.
10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`,
    http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot
    before they are called done. `data/logs/*.png` is where those land.

## Architecture

```
agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        ▼                                  ▼                        ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
        │  parse + Δcounter/Δt                                                     │
        ▼                                                                           ▼
sources/host.py · sources/llm.py · sources/nut.py                    poller.py (task per source)
        │        Result(payload, samples, events)                                    │
        ▼                                                                             ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/
```

Responsibilities, one line each:

- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
  `statvfs`; serves `GET /snapshot` + `GET /info` + `GET /health`. Refuses to bind
  off-loopback without a token (exit 2). Counters stay counters: the dashboard computes
  rates so the agent never remembers who polled it last.
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
  env var always wins. Defaults are safe except the sources, which default to unset.
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping.
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
  (`gpu.busy_pct@<index>`).
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
  `VLLM_HISTOGRAMS`), measured Δ rates, prefill speed as
  `mean(prompt tokens)/mean(prefill time)`, session list from gufo `/slots`.
- `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not
  a mystery score), `COMMANDS` catalog with risk tiers, `available_commands()` filtering,
  kelvin-vs-celsius normalisation.
- `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a
  battery. It must never be the default, and every payload it produces says `simulated`.
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
- `rates.py` — Δ/window estimator, keyed by resolved series name.
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
  with an in-memory DB and no targets.
- `web/` — four static pages, one stylesheet, `charts.js`. No framework.

Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tps`, `ups.charge_pct`), `@stream` for per-device series.
That scheme is the history API's key — treat it as a public interface.

## The fleet (real environment, verified 2026-10)

| Host              | ssh alias          | What it gives                                                        |
| ----------------- | ------------------ | -------------------------------------------------------------------- |
| `driveripper`     | `driveripper-ops`  | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards |
| `deskwork`        | `deskwork-ops`     | Strix Halo (Radeon 8060S iGPU), gufo `:8731`                          |
| `gateway`         | `gateway-ops`      | router, no GPU                                                       |

- Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123`
  by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev.
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
  on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
  *Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
  | nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are
  exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`:
  `apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U.
  Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date
  (`battery.date` is often an unset placeholder like `2001-09-25`), CyberPower reports
  `battery.mfr.date: CPS` (not a date → `unknown`, never a fake age), neither driver reports
  a battery temperature, and `ups.test.result: No test initiated` is the factory state.
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
  set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
  work that is still blocked).

## Commands

```bash
uv sync --python 3.14                     # deps + editable install
cp .env.example .env                      # production-shaped config
cp .env.dev.example .env.dev              # dev-only: ssh tunnels + live targets

./scripts/dev-tunnels.sh start            # agents (9121-9123) + DEV_FORWARDS (upsd 3494)
./scripts/dev-server.sh restart           # keep http://127.0.0.1:8090 up
./scripts/dev-server.sh status            # per-source ok/DOWN
./scripts/dev-server.sh logs

uv run pytest                             # hermetic: 258 tests, no network
uv run pytest --cov                       # coverage gate (~94%)
uv run pytest -m live --live              # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh             # re-record ground truth
```

Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--token-file`, `ProtectSystem=strict`).
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
service here.

## Testing policy

**Real captures for shape, fakes for behaviour, `live` for drift.**

- Anything that must match an upstream — `/proc` layouts, vLLM series names, NUT framing —
  is asserted against a verbatim capture in `tests/fixtures/`. If the schema drifts, a
  fixture test fails before a dashboard does.
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
  rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
  fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
  unset, so partial fleet availability is normal. **Live tests must not write** — a battery
  test on hardware something is plugged into is not a test.
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
  so it stays in the default run and is *not* marked `live`.
- Coverage gate: 80% is the floor, ~94% is where we are; keep it there. API tests use
  `create_app()` with an in-memory DB, never a real target.

## UI conventions

Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
dots in the header, window chips (1h…90d) on history pages. A panel that cannot get data says
why in the panel — the operator should never have to open devtools to learn that a source is
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.

## Known gaps / next work

1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see
   above), but every button is gated off with "NUT_USER / NUT_PASSWORD must name a user with
   `actions` in upsd.users" because no such user exists yet. Creating one safely needs a
   *read* of `/etc/ups/upsd.users` first — `sudo tee` can write the file but overwriting it
   blind would delete the users upsmon already authenticates as, which can take the box down
   at the wrong moment. `deploy/nut-sudoers` lists the read rules; once they are in, add
   `[dashboards]` with `instcmds` limited to the battery tests, put the credentials in
   `.env.dev`, and the buttons come alive. Never grant `actions = -`.
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
   (`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
   fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
   all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the
   field map is written.
3. **No remote yet.** Two commits, no `origin` — the user is providing it.
4. **UPS history depth.** Events and the numeric series are stored; long-window battery
   health (capacity fade over months) has the rollups but no dedicated view.

## Locked decisions (and why)

- **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10s resolution, one file the
  container owns. Rollups keep a year of charts at a few hundred MB.
- **Our own stdlib host agent, not node_exporter.** One JSON snapshot with exactly the
  fields the page needs (GPU model strings, per-mount usage, temperature labels) versus
  ~1500 series and a re-labeling layer; no dependency to install, `ProtectSystem=strict`
  friendly, and read-only by construction. `scripts/capture-node-exporter.sh` remains to
  compare ground truth if that is ever revisited.
- **Our own NUT client, not `pynut`/`upsc`.** The protocol is ~10 line commands, the
  third-party libs are subprocess-based or stalled, and we need instant commands with
  explicit gating.
- **In-house Prometheus parser.** Only two upstreams use the format; keeps `prometheus_client`
  out of the dependency set.
- **Port 8090**, inherited from the `llm_monitor` prototype this started from: 8000 is vLLM
  and 8731 is gufo in this fleet.
- **`.env` is a seed, never an override** — container/systemd env must win.

## Definition of done (per change)

`uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated`
empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted
in a browser against **live** data · fixtures re-captured if an upstream contract changed ·
this file and `README.md` updated when a rule, command, gap, or fleet fact changed.