dashboards/AGENTS.md

AGENTS.md — homelab dashboards

The plan, the rules, and the environment facts an agent needs to work on this repo without re-deriving them. README.md is the operator-facing how-to; this file is the why, the invariants, and the state of the fleet.

What this is

Three dashboards for one homelab, behind one domain, in one process, backed by one SQLite file — plus an overview page that says which of the three you need:

Route	Question it answers	Source
/	Is anything wrong, and where do I look first?	all three, in one request
/system	What are my machines doing — CPU, memory, disk, GPU, network, and the history	agent/host_agent.py, our own stdlib agent/host
/llm	Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles	gufo (llama.cpp family) + vLLM /metrics
/ups	Battery health, runtime left, power events, battery-test buttons	NUT upsd on 3493 (our own wire-protocol client)

History is a first-class feature, not a sparkline afterthought: every number worth graphing is stored, rolled up (raw → 1m → 1h), and served from /api/*/history.

Non-negotiable rules

These are the things that make the numbers trustworthy. Breaking one is a bug even if the page still renders.

1. Read-only by construction, with exactly one exception. Nothing in this app writes to an upstream except POST /api/ups/command. That endpoint is gated four times (NUT_ALLOW_ACTIONS, not DASH_READ_ONLY, the upsd.users account actually having the command, and a UI confirmation — DANGER for anything that can cut power) and the server re-checks every gate on every request. Hiding a button is not a safety control. Commands that would cut output are refused outright while the UPS is already on battery.
2. Never invent a number. A value the upstream does not publish renders as n/a, never as 0. Every derived value carries the source string it came from and derived: true so the UI can print how it was obtained (computed:Δtokens/Δt, Δ10.0s). Aggregating two different series into one rate is the failure mode this rule exists to catch — the rate estimator is keyed by the resolved upstream series name for exactly that reason.
3. Unconfigured is not broken. A source with no env config answers 404 {"configured": false} without dialling, and the page says "not configured". A red panel for a server the operator never installed destroys the signal from the panels that matter.
4. One dead source must not blink another. Each source is its own asyncio task with its own timeout and health record (poller.py). A failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it. Transitions write events; repeated failure writes one line, not 500.
5. No CDN, no build step, no internet at runtime. Every asset is served by this app; charts are hand-rolled SVG (web/static/charts.js). The engines live on a network with no outbound access — a CDN reference is a request that never comes back.
6. Never leak a credential. Target URLs may carry a userinfo token; /api/config exposes them only through Target.display_url, which strips userinfo. No token, password, or personal hostname goes into tracked files, fixtures, or the browser.
7. Verify against the deployed upstream; never guess a metric name. Field maps in sources/llm.py and sources/nut.py were captured from the running engines and NUT 2.8.5, and are pinned by fixtures in tests/fixtures/. Re-capture with scripts/capture-fixtures.sh (and scripts/capture-node-exporter.sh when comparing exporters) rather than trusting a README or a memory of an upstream.
8. Keep dependencies current on purpose. LLM-written version floors are usually stale. Check after touching dependencies: uv pip list --outdated must come back empty. Python is 3.14 (requires-python >=3.14, ruff py314, pyright 3.14, Containerfile python3.14) — do not lower these to make something else work.
9. SQLite owns the history and must stay bounded. Writes go through store.py; new metrics need a rollup path and a retention story, or a year of charts becomes a disk emergency. Put the DB on a volume in deployment.
10. The dev server stays up while work happens (./scripts/dev-server.sh restart, http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot before they are called done. data/logs/*.png is where those land.

Architecture

agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        ▼                                  ▼                        ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
        │  parse + Δcounter/Δt                                                     │
        ▼                                                                           ▼
sources/host.py · sources/llm.py · sources/nut.py                    poller.py (task per source)
        │        Result(payload, samples, events)                                    │
        ▼                                                                             ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/

Responsibilities, one line each:

• agent/host_agent.py — stdlib-only (3.10+), reads /proc, /sys/block, /sys/class/drm, statvfs; serves GET /snapshot + GET /info + GET /health. Refuses to bind off-loopback without a token (exit 2). Counters stay counters: the dashboard computes rates so the agent never remembers who polled it last.
• config.py — env only, no config file format, no CLI flags. .env is a seed; a real env var always wins. Defaults are safe except the sources, which default to unset.
• poller.py — job loop, health records, stale/last-good semantics, transition events.
• collectors.py — the fetch half. NutMonitor owns per-UPS transition bookkeeping.
• sources/host.py — snapshot → payload + samples; reboot detection; GPU series per card (gpu.busy_pct@<index>).
• sources/llm.py — gufo + vLLM field maps (GUFO_SERIES, VLLM_SERIES, VLLM_HISTOGRAMS), measured Δ rates, prefill speed as mean(prompt tokens)/mean(prefill time), session list from gufo /slots.
• sources/nut.py — upsd client, UpsReading, VAR_MAP, assess() (explicit checks, not a mystery score), COMMANDS catalog with risk tiers, available_commands() filtering, kelvin-vs-celsius normalisation.
• sources/nut_sim.py — a labelled simulator so the UPS page is developable without a battery. It must never be the default, and every payload it produces says simulated.
• prom.py — Prometheus text-exposition parser (in-house; only for engines we don't own).
• rates.py — Δ/window estimator, keyed by resolved series name.
• store.py — sample(source,host,metric,res,ts) + event + meta, rollups, pruning.
• api.py — GETs everywhere plus the one POST. /api/config is redaction boundary.
• app.py — assembly; create_app() takes injectable settings/store so tests build an app with an in-memory DB and no targets.
• web/ — four static pages, one stylesheet, charts.js. No framework.

Metric naming: dotted (cpu.usage_pct, mem.used_pct, disk.used_pct@/srv, gpu.busy_pct@0, llm.decode_tps, ups.charge_pct), @stream for per-device series. That scheme is the history API's key — treat it as a public interface.

The fleet (real environment, verified 2026-10)

Host	ssh alias	What it gives
driveripper	driveripper-ops	Threadripper 24c/48t + Radeon AI PRO R9700, vLLM :8000, upsd, the dashboards
deskwork	deskwork-ops	Strix Halo (Radeon 8060S iGPU), gufo :8731
gateway	gateway-ops	router, no GPU

• Agents run from ~/lab-agent/host_agent.py on each host, forwarded to 127.0.0.1:9121/9122/9123 by scripts/dev-tunnels.sh (DEV_TUNNELS). Nothing listens on the hosts in dev.
• Engines are reached directly: gufo deskwork.reeselink.com:8731, vLLM driveripper.reeselink.com:8000.
• NUT 2.8.5 on driveripper: configs live in /etc/ups/, not /etc/nut/. nut-server is active and nut-driver@apc + nut-driver@cyberpower are both running, so ups.conf already exports two driver instances; a CyberPower PR1500LCDRT2U is attached on USB (0764:0601). upsd does not bind 3493, so reads fail with Connection refused and the dashboard falls back to the labelled simulator. A restart of nut-server does not fix it. See Known gaps — this needs root logs and a config read.
• Host access is via ssh keys as ops. Passwordless sudo exists but is command-scoped; the set we rely on is documented in deploy/nut-sudoers (additions are listed there for the work that is still blocked).

Commands

uv sync --python 3.14                     # deps + editable install
cp .env.example .env                      # production-shaped config
cp .env.dev.example .env.dev              # dev-only: ssh tunnels + live targets

./scripts/dev-tunnels.sh start            # agents (9121-9123) + DEV_FORWARDS (upsd 3494)
./scripts/dev-server.sh restart           # keep http://127.0.0.1:8090 up
./scripts/dev-server.sh status            # per-source ok/DOWN
./scripts/dev-server.sh logs

uv run pytest                             # hermetic: 258 tests, no network
uv run pytest --cov                       # coverage gate (~94%)
uv run pytest -m live --live              # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh             # re-record ground truth

Deployment: Containerfile (two-stage, uv sync --frozen, runs as uid 10001 on 0.0.0.0:8090, SQLite under /var/lib/dashboards — put a volume there) and deploy/install-agent.sh per monitored host (systemd unit, own user, --token-file, ProtectSystem=strict). The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other service here.

Testing policy

Real captures for shape, fakes for behaviour, live for drift.

• Anything that must match an upstream — /proc layouts, vLLM series names, NUT framing — is asserted against a verbatim capture in tests/fixtures/. If the schema drifts, a fixture test fails before a dashboard does.
• Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host rebooting mid-poll, a counter reset — is a fake in tests/conftest.py.
• live (tests/test_live_fleet.py) re-runs the production code paths against the real fleet, engines, and upsd. Skipped unless --live; each test skips when its source is unset, so partial fleet availability is normal. Live tests must not write — a battery test on hardware something is plugged into is not a test.
• The agent's real-/proc test (tests/test_agent.py) touches only the box running pytest, so it stays in the default run and is not marked live.
• Coverage gate: 80% is the floor, ~94% is where we are; keep it there. API tests use create_app() with an in-memory DB, never a real target.

UI conventions

Hand-rolled SVG (charts.js), one stylesheet, dark/light toggle, Δ<window> labels on measured rates, n/a for the unknown, SIMULATED badges on modelled data, per-source status dots in the header, window chips (1h…90d) on history pages. A panel that cannot get data says why in the panel — the operator should never have to open devtools to learn that a source is unconfigured. Escape anything upstream-provided through DashCharts.escapeHtml.

Known gaps / next work

1. Real UPS data (blocking issue, host-side). upsd on driveripper runs but binds nothing. Needed: root journal for nut-server (journalctl -u nut-server), a read of /etc/ups/{upsd.conf,upsd.users,ups.conf}, then LISTEN 127.0.0.1 (plus a dashboards user with scoped instcmds). Then: DEV_FORWARDS="3494:driveripper-ops:3493", NUT_HOST=127.0.0.1 NUT_PORT=3494, unset NUT_SIMULATE, and re-verify /ups against a real battery. Until then every UPS number is modelled, and the page says so.
2. GPU coverage is AMD only. host_agent.gpus() reads amdgpu DRM sysfs (gpu_busy_percent, mem_info_vram_*, hwmon temps, clocks). That covers the current fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at all — needs a nvidia-smi/NVML or DCGM path, and a fixture from a real card before the field map is written.
3. No remote yet. Two commits, no origin — the user is providing it.
4. UPS history depth. Events and the numeric series are stored; long-window battery health (capacity fade over months) has the rollups but no dedicated view.

Locked decisions (and why)

• SQLite, not Prometheus/Grafana. Handful of boxes, 1–10s resolution, one file the container owns. Rollups keep a year of charts at a few hundred MB.
• Our own stdlib host agent, not node_exporter. One JSON snapshot with exactly the fields the page needs (GPU model strings, per-mount usage, temperature labels) versus ~1500 series and a re-labeling layer; no dependency to install, ProtectSystem=strict friendly, and read-only by construction. scripts/capture-node-exporter.sh remains to compare ground truth if that is ever revisited.
• Our own NUT client, not pynut/upsc. The protocol is ~10 line commands, the third-party libs are subprocess-based or stalled, and we need instant commands with explicit gating.
• In-house Prometheus parser. Only two upstreams use the format; keeps prometheus_client out of the dependency set.
• Port 8090, inherited from the llm_monitor prototype this started from: 8000 is vLLM and 8731 is gufo in this fleet.
• .env is a seed, never an override — container/systemd env must win.

Definition of done (per change)

uv run pytest green · coverage not down · ruff + pyright clean · uv pip list --outdated empty if deps moved · dev server restarted and /api/health sane · UI change screenshotted in a browser against live data · fixtures re-captured if an upstream contract changed · this file and README.md updated when a rule, command, gap, or fleet fact changed.