dashboards/AGENTS.md

AGENTS.md — homelab dashboards

The plan, the rules, and the environment facts an agent needs to work on this repo without re-deriving them. README.md is the operator-facing how-to; this file is the why, the invariants, and the state of the fleet.

Working protocol

This project runs the .agents/phases/ phased-execution pipeline.

1. Always read .agents/PLAN.md first to understand the project context and goals.
2. Follow the phased execution protocol in .agents/phases/.
3. Never modify .agents/PLAN.md or any files in .agents/phases/complete/.
4. If you need to update any file in .agents/phases/todo/, you must ask the user for permission first.
5. Strictly adhere to the LOCKED DECISIONS listed in .agents/PLAN.md.

Everything below this section is the project's own content: what the app is, the rules that keep its numbers honest, and the state of the fleet it watches.

What this is

Three dashboards for one homelab, behind one domain, in one process, backed by one SQLite file — plus an overview page that says which of the three you need:

Route	Question it answers	Source
/	Is anything wrong, and where do I look first?	all three, in one request
/system	What are my machines doing — CPU, memory, disk, GPU, network, and the history	agent/host_agent.py, our own stdlib agent/host
/llm	Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles	gufo (llama.cpp family) + vLLM /metrics
/ups	Battery health, runtime left, power events, battery-test buttons	NUT upsd (our own wire-protocol client; 3051 here, not 3493)

History is a first-class feature, not a sparkline afterthought: every number worth graphing is stored, rolled up (raw → 1m → 1h), and served from /api/*/history.

Non-negotiable rules

These are the things that make the numbers trustworthy. Breaking one is a bug even if the page still renders.

1. Read-only by construction, with exactly one exception. Nothing in this app writes to an upstream except POST /api/ups/command. That endpoint is gated four times (NUT_ALLOW_ACTIONS, not DASH_READ_ONLY, the upsd.users account actually having the command, and a UI confirmation — DANGER for anything that can cut power) and the server re-checks every gate on every request. Hiding a button is not a safety control. Commands that would cut output are refused outright while the UPS is already on battery.
2. Never invent a number. A value the upstream does not publish renders as n/a, never as 0. Every derived value carries the source string it came from and derived: true so the UI can print how it was obtained (computed:Δtokens/Δt, Δ10.0s). Aggregating two different series into one rate is the failure mode this rule exists to catch — the rate estimator is keyed by the resolved upstream series name for exactly that reason.
3. Unconfigured is not broken. A source with no env config answers 404 {"configured": false} without dialling, and the page says "not configured". A red panel for a server the operator never installed destroys the signal from the panels that matter.
4. One dead source must not blink another. Each source is its own asyncio task with its own timeout and health record (poller.py). A failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it. Transitions write events; repeated failure writes one line, not 500.
5. No CDN, no build step, no internet at runtime. Every asset is served by this app; charts are hand-rolled SVG (web/static/charts.js). The engines live on a network with no outbound access — a CDN reference is a request that never comes back.
6. Never leak a credential. Target URLs may carry a userinfo token; /api/config exposes them only through Target.display_url, which strips userinfo. No token, password, or personal hostname goes into tracked files, fixtures, or the browser.
7. Verify against the deployed upstream; never guess a metric name. Field maps in sources/llm.py and sources/nut.py were captured from the running engines and NUT 2.8.5, and are pinned by fixtures in tests/fixtures/. Re-capture with scripts/capture-fixtures.sh (and scripts/capture-node-exporter.sh when comparing exporters) rather than trusting a README or a memory of an upstream.
8. Keep dependencies current on purpose. LLM-written version floors are usually stale. Check after touching dependencies: uv pip list --outdated must come back empty. Python is 3.14 (requires-python >=3.14, ruff py314, pyright 3.14, Containerfile python3.14) — do not lower these to make something else work.
9. SQLite owns the history and must stay bounded. Writes go through store.py; new metrics need a rollup path and a retention story, or a year of charts becomes a disk emergency. Put the DB on a volume in deployment.
10. The dev server stays up while work happens (./scripts/dev-server.sh restart, http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot before they are called done. data/logs/*.png is where those land.

Architecture

agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        ▼                                  ▼                        ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
        │  parse + Δcounter/Δt                                                     │
        ▼                                                                           ▼
sources/host.py · sources/llm.py · sources/nut.py                    poller.py (task per source)
        │        Result(payload, samples, events)                                    │
        ▼                                                                             ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/

Responsibilities, one line each:

• agent/host_agent.py — stdlib-only (3.10+), reads /proc, /sys/block, /sys/class/drm, statvfs; serves GET /snapshot + GET /info + GET /health. Refuses to bind off-loopback without a token (exit 2). Counters stay counters: the dashboard computes rates so the agent never remembers who polled it last.
• config.py — env only, no config file format, no CLI flags. .env is a seed; a real env var always wins. Defaults are safe except the sources, which default to unset.
• poller.py — job loop, health records, stale/last-good semantics, transition events.
• collectors.py — the fetch half. NutMonitor owns per-UPS transition bookkeeping.
• sources/host.py — snapshot → payload + samples; reboot detection; GPU series per card (gpu.busy_pct@<index>).
• sources/llm.py — gufo + vLLM field maps (GUFO_SERIES, VLLM_SERIES, VLLM_HISTOGRAMS), measured Δ rates, prefill speed as mean(prompt tokens)/mean(prefill time), session list from gufo /slots.
• sources/nut.py — upsd client, UpsReading, VAR_MAP, assess() (explicit checks, not a mystery score), COMMANDS catalog with risk tiers, available_commands() filtering, kelvin-vs-celsius normalisation.
• sources/nut_sim.py — a labelled simulator so the UPS page is developable without a battery. It must never be the default, and every payload it produces says simulated.
• prom.py — Prometheus text-exposition parser (in-house; only for engines we don't own).
• rates.py — Δ/window estimator, keyed by resolved series name.
• severity.py — the only place thresholds live: THRESHOLDS/INVERTED per metric, classify() → ok/warn/fail/None, plus attach()/with_severity() to put the verdict beside a value. No measurement, no verdict: None in → None out, never a green pass for an unknown. The verdicts travel additively in payloads — cpu.severity, memory.severity, disks.severity, gpus[].severity, each UPS card's severity (charge/load/runtime), severity inside rated LLM cells (KV pressure, classified on a percent scale because the engines publish a 0-1 ratio), and severity per host/fleet/UPS row in /api/summary. Pages render the class; they never compare a number to a threshold.
• store.py — sample(source,host,metric,res,ts) + event + meta, rollups, pruning.
• api.py — GETs everywhere plus the one POST. /api/config is redaction boundary.
• app.py — assembly; create_app() takes injectable settings/store so tests build an app with an in-memory DB and no targets.
• web/ — four static pages, one stylesheet, charts.js. No framework. app.js owns the colour vocabulary: Dash.severityFrom(cellOrSeverity) turns a payload verdict into an ok|warn|fail class ('' for anything unrated, so unknown never looks green) and Dash.hero(el, {label, value, unit, digits, severity, sub, bar}) renders/patches a .hero big number (plus an optional .bar) from that class — it never compares a number to a threshold. Dash.severityFor(value, warnAt, failAt) is the documented fallback for values the API does not rate; converting its remaining call sites is phases 02–05's job.

Metric naming: dotted (cpu.usage_pct, mem.used_pct, disk.used_pct@/srv, gpu.busy_pct@0, llm.decode_tps, ups.charge_pct), @stream for per-device series. That scheme is the history API's key — treat it as a public interface.

The fleet (real environment, verified 2026-10)

Host	ssh alias	What it gives
driveripper	driveripper-ops	Threadripper 24c/48t + Radeon AI PRO R9700, vLLM :8000, upsd, the dashboards
deskwork	deskwork-ops	Strix Halo (Radeon 8060S iGPU), gufo :8731
gateway	gateway-ops	router, no GPU

• Agents run from ~/lab-agent/host_agent.py on each host, forwarded to 127.0.0.1:9121/9122/9123 by scripts/dev-tunnels.sh (DEV_TUNNELS). Nothing listens on the hosts in dev.
• Engines are reached directly: gufo deskwork.reeselink.com:8731, vLLM driveripper.reeselink.com:8000.
• NUT 2.8.5 on driveripper: configs live in /etc/ups/, not /etc/nut/, and upsd is on port 3051, not 3493. Nothing listens on 3493 at all, so a default client reports Connection refused and it looks like a dead service — ss -ltnH plus printf 'LIST UPS\n' | nc 127.0.0.1 3051 (answers BEGIN LIST UPS) is how that was pinned down. Two UPSes are exported over USB and read live through DEV_FORWARDS="3494:driveripper-ops:3051": apc = APC Back-UPS RS 1500MS, cyberpower = CyberPower PR1500LCDRT2U (that is ups.description, the operator's label — the driver's own identity.model answers GX1500U, so the description is what identifies the box). Real-hardware quirks the parsers already handle: battery.mfr.date is the useful date (battery.date is often an unset placeholder like 2001-09-25), CyberPower reports battery.mfr.date: CPS (not a date → unknown, never a fake age), neither driver reports a battery temperature, and ups.test.result: No test initiated is the factory state on both.
• Host access is via ssh keys as ops. Passwordless sudo exists but is command-scoped; the set we rely on is documented in deploy/nut-sudoers (additions are listed there for the work that is still blocked).

Commands

uv sync --python 3.14                     # deps + editable install
cp .env.example .env                      # production-shaped config
cp .env.dev.example .env.dev              # dev-only: ssh tunnels + live targets

./scripts/dev-tunnels.sh start            # agents (9121-9123) + DEV_FORWARDS (upsd 3494)
./scripts/dev-server.sh restart           # keep http://127.0.0.1:8090 up
./scripts/dev-server.sh status            # per-source ok/DOWN
./scripts/dev-server.sh logs

uv run pytest                             # hermetic: 356 tests, no network
uv run pytest --cov                       # coverage gate (94.75%; floor 94%)
uv run pytest -m live --live              # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh             # re-record ground truth

bash .agents/validate.sh                  # the phase gate: the four checks above (pytest
                                          # --cov --cov-fail-under=94, ruff, pyright, plus a
                                          # node --check of web/static) + repo hygiene.
                                          # Always `uv run` — a bare `python3`/`ruff` on PATH
                                          # tests an interpreter nobody deploys.

Deployment: Containerfile (two-stage, uv sync --frozen, runs as uid 10001 on 0.0.0.0:8090, SQLite under /var/lib/dashboards — put a volume there) and deploy/install-agent.sh per monitored host (systemd unit, own user, --bind/--port/--token/--rotate-token/ --status/--uninstall; it stores the token in a root-owned 0600 file and hands the agent --token-file, never argv, and runs under ProtectSystem=strict). The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other service here.

Testing policy

Real captures for shape, fakes for behaviour, live for drift.

• Anything that must match an upstream — /proc layouts, vLLM series names, NUT framing — is asserted against a verbatim capture in tests/fixtures/. If the schema drifts, a fixture test fails before a dashboard does.
• Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host rebooting mid-poll, a counter reset — is a fake in tests/conftest.py.
• live (tests/test_live_fleet.py) re-runs the production code paths against the real fleet, engines, and upsd. Skipped unless --live; each test skips when its source is unset, so partial fleet availability is normal. Live tests must not write — a battery test on hardware something is plugged into is not a test.
• The agent's real-/proc test (tests/test_agent.py) touches only the box running pytest, so it stays in the default run and is not marked live.
• Coverage gate: 80% is the repo floor, the phase gate in .agents/validate.sh enforces 94% (--cov-fail-under), and ~95% is where we are; keep it there. New/modified Python still has to clear >90% on its own. API tests use create_app() with an in-memory DB, never a real target.

UI conventions

Hand-rolled SVG (charts.js), one stylesheet, dark/light toggle, Δ<window> labels on measured rates, n/a for the unknown, SIMULATED badges on modelled data, per-source status dots in the header, window chips (1h…90d) on history pages. A panel that cannot get data says why in the panel — the operator should never have to open devtools to learn that a source is unconfigured. Escape anything upstream-provided through DashCharts.escapeHtml.

A big number is Dash.hero() on .hero / .hero-bar (dash.css): 2.4rem tabular numerals, muted unit, mono .sub footnote (Δ10.0s, "at current load"), coloured by the payload's severity through the existing .ok / .warn / .fail classes — there is no .sev-* family and no second palette, and tests/test_api.py fails if either appears or if a CSS rule reaches for a token that :root does not declare.

Known gaps / next work

1. UPS instant commands are read-only in practice. Reads work against real upsd (see above), but every button is gated off with "NUT_USER / NUT_PASSWORD must name a user with actions in upsd.users" because no such user exists yet. Creating one safely needs a read of /etc/ups/upsd.users first — sudo tee can write the file but overwriting it blind would delete the users upsmon already authenticates as, which can take the box down at the wrong moment. deploy/nut-sudoers is the installable ruleset that makes that read (and only a scoped write + reload) possible — it carries the ops ALL=(root) NOPASSWD: NUT_READ, NUT_WRITE grant and is installed with sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut, but it is repo-side only so far: the live UPS cards still answer "NUT_USER / NUT_PASSWORD must name a user with actions in upsd.users" (verified through /api/ups/snapshot). Once it is installed: read upsd.users, add [dashboards] with instcmds limited to the battery tests, put the credentials in .env.dev, and the buttons come alive. Never grant actions = -, and never write the file blind.
2. GPU coverage is AMD only. host_agent.gpus() reads amdgpu DRM sysfs (gpu_busy_percent, mem_info_vram_*, hwmon temps, clocks). That covers the current fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at all — needs a nvidia-smi/NVML or DCGM path, and a fixture from a real card before the field map is written.
3. No remote yet. No origin is configured (git remote -v is empty) — the user is providing it, so every phase so far is local-only history.
4. UPS history depth. Events and the numeric series are stored; long-window battery health (capacity fade over months) has the rollups but no dedicated view.

Locked decisions (and why)

• SQLite, not Prometheus/Grafana. Handful of boxes, 1–10s resolution, one file the container owns. Rollups keep a year of charts at a few hundred MB.
• Our own stdlib host agent, not node_exporter. One JSON snapshot with exactly the fields the page needs (GPU model strings, per-mount usage, temperature labels) versus ~1500 series and a re-labeling layer; no dependency to install, ProtectSystem=strict friendly, and read-only by construction. scripts/capture-node-exporter.sh remains to compare ground truth if that is ever revisited.
• Our own NUT client, not pynut/upsc. The protocol is ~10 line commands, the third-party libs are subprocess-based or stalled, and we need instant commands with explicit gating.
• In-house Prometheus parser. Only two upstreams use the format; keeps prometheus_client out of the dependency set.
• Port 8090, inherited from the llm_monitor prototype this started from: 8000 is vLLM and 8731 is gufo in this fleet.
• .env is a seed, never an override — container/systemd env must win.

Definition of done (per change)

The first three are exactly what bash .agents/validate.sh (the phase gate) checks, so no phase may be called done on a subset of them.

uv run pytest green · coverage not down · ruff + pyright clean · uv pip list --outdated empty if deps moved · dev server restarted and /api/health sane · UI change screenshotted in a browser against live data · fixtures re-captured if an upstream contract changed · this file and README.md updated when a rule, command, gap, or fleet fact changed.