dashboards/AGENTS.md

AGENTS.md — homelab dashboards

The plan, the rules, and the environment facts an agent needs to work on this repo without re-deriving them. README.md is the operator-facing how-to; this file is the why, the invariants, and the state of the fleet.

Working protocol

This project runs the .agents/phases/ phased-execution pipeline.

1. Always read .agents/PLAN.md first to understand the project context and goals.
2. Follow the phased execution protocol in .agents/phases/.
3. Never modify .agents/PLAN.md or any files in .agents/phases/complete/.
4. If you need to update any file in .agents/phases/todo/, you must ask the user for permission first.
5. Strictly adhere to the LOCKED DECISIONS listed in .agents/PLAN.md.

Everything below this section is the project's own content: what the app is, the rules that keep its numbers honest, and the state of the fleet it watches.

What this is

Three dashboards for one homelab, behind one domain, in one process, backed by one SQLite file — plus an overview page that says which of the three you need:

Route	Question it answers	Source
/	Is anything wrong, and where do I look first?	all three, in one request
/system	What are my machines doing — CPU, memory, disk, GPU, network, and the history	agent/host_agent.py, our own stdlib agent/host
/llm	Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles	gufo (llama.cpp family) + vLLM /metrics
/ups	Battery health, runtime left, power events, battery-test buttons	NUT upsd (our own wire-protocol client; 3051 here, not 3493)

History is a first-class feature, not a sparkline afterthought: every number worth graphing is stored, rolled up (raw → 1m → 1h), and served from /api/*/history.

Non-negotiable rules

These are the things that make the numbers trustworthy. Breaking one is a bug even if the page still renders.

1. Read-only by construction, with exactly one exception. Nothing in this app writes to an upstream except POST /api/ups/command. That endpoint is gated four times (NUT_ALLOW_ACTIONS, not DASH_READ_ONLY, the upsd.users account actually having the command, and a UI confirmation — DANGER for anything that can cut power) and the server re-checks every gate on every request. Hiding a button is not a safety control. Commands that would cut output are refused outright while the UPS is already on battery.
2. Never invent a number. A value the upstream does not publish renders as n/a, never as 0. Every derived value carries the source string it came from and derived: true so the UI can print how it was obtained (computed:Δtokens/Δt, Δ10.0s). Aggregating two different series into one rate is the failure mode this rule exists to catch — the rate estimator is keyed by the resolved upstream series name for exactly that reason.
3. Unconfigured is not broken. A source with no env config answers 404 {"configured": false} without dialling, and the page says "not configured". A red panel for a server the operator never installed destroys the signal from the panels that matter.
4. One dead source must not blink another. Each source is its own asyncio task with its own timeout and health record (poller.py). A failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it — on / too, where a host that has never answered still gets a fleet row (unreachable: true, dimmed, every number n/a) because dropping the row would turn a dead agent into "never configured". Transitions write events; repeated failure writes one line, not 500.
5. No CDN, no build step, no internet at runtime. Every asset is served by this app; charts are hand-rolled SVG (web/static/charts.js). The engines live on a network with no outbound access — a CDN reference is a request that never comes back.
6. Never leak a credential. Target URLs may carry a userinfo token (parse_targets moves it into an Authorization header, so Target.url ends up clean). Nothing that leaves the process may carry one: /api/config serves Target.display_url and config.strip_userinfo-redacted engine URLs, and every URL inside an UpstreamError message is redacted too — those strings go into the health record and the event table, which /api/events serves to the browser and page-system.js prints. No token, password, or personal hostname goes into tracked files, fixtures, or the browser.
7. Verify against the deployed upstream; never guess a metric name. Field maps in sources/llm.py and sources/nut.py were captured from the running engines and NUT 2.8.5, and are pinned by fixtures in tests/fixtures/. Re-capture with scripts/capture-fixtures.sh (and scripts/capture-node-exporter.sh when comparing exporters) rather than trusting a README or a memory of an upstream.
8. Keep dependencies current on purpose. LLM-written version floors are usually stale. Check after touching dependencies: uv pip list --outdated must come back empty. Python is 3.14 (requires-python >=3.14, ruff py314, pyright 3.14, Containerfile python3.14) — do not lower these to make something else work.
9. SQLite owns the history and must stay bounded. Writes go through store.py; new metrics need a rollup path and a retention story, or a year of charts becomes a disk emergency. Put the DB on a volume in deployment.
10. The dev server stays up while work happens (./scripts/dev-server.sh restart, http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot before they are called done. data/logs/*.png is where those land.

Architecture

agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        ▼                                  ▼                        ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
        │  parse + Δcounter/Δt                                                     │
        ▼                                                                           ▼
sources/host.py · sources/llm.py · sources/nut.py                    poller.py (task per source)
        │        Result(payload, samples, events)                                    │
        ▼                                                                             ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/

Responsibilities, one line each:

• agent/host_agent.py — stdlib-only (3.10+), reads /proc, /sys/block, /sys/class/drm, statvfs; serves GET /snapshot + GET /info + GET /health. Refuses to bind off-loopback without a token (exit 2). Counters stay counters: the dashboard computes rates so the agent never remembers who polled it last.
• config.py — env only, no config file format, no CLI flags. .env is a seed; a real env var always wins. Defaults are safe except the sources, which default to unset.
• poller.py — job loop, health records, stale/last-good semantics, transition events. Events are stored under the dashboard name (system/llm/ups), not the sample source (host) — a page filtering /api/events by host gets an empty feed forever (found on /system during phase 03 browser verification).
• collectors.py — the fetch half. NutMonitor owns per-UPS transition bookkeeping. Failure messages name the redacted URL (strip_userinfo / Target.display_url), never the one that was dialled, because they become health records and event rows.
• sources/host.py — snapshot → payload + samples; reboot detection; GPU series per card (gpu.busy_pct@<index>).
• sources/llm.py — gufo + vLLM field maps (GUFO_SERIES, VLLM_SERIES, VLLM_HISTOGRAMS), measured Δ rates, prefill speed as mean(prompt tokens)/mean(prefill time), session list from gufo /slots.
• sources/nut.py — upsd client, UpsReading, VAR_MAP, assess() (explicit checks, not a mystery score), COMMANDS catalog with risk tiers, available_commands() filtering, kelvin-vs-celsius normalisation.
• sources/nut_sim.py — a labelled simulator so the UPS page is developable without a battery. It must never be the default, and every payload it produces says simulated. Its modelled battery.mfr.date is computed relative to today (not a fixed string that ages), so a fresh simulator reads healthy and amber keeps meaning something.
• prom.py — Prometheus text-exposition parser (in-house; only for engines we don't own).
• rates.py — Δ/window estimator, keyed by resolved series name.
• severity.py — the only place thresholds live: THRESHOLDS/INVERTED per metric, classify() → ok/warn/fail/None, plus attach()/with_severity() to put the verdict beside a value. No measurement, no verdict: None in → None out, never a green pass for an unknown. The verdicts travel additively in payloads — cpu.severity, memory.severity, disks.severity, gpus[].severity, each UPS card's severity (charge/load/runtime), severity inside rated LLM cells (KV pressure, classified on a percent scale because the engines publish a 0-1 ratio), and severity per host/fleet/UPS row in /api/summary. Pages render the class; they never compare a number to a threshold.
• alerts.py — build(settings, job_states, host_payloads, engine_payloads, ups_payload, recent_events) → the "what is wrong right now" list /api/summary serves as alerts ({id, severity, source, message, href, at}, fail first then warn, then by source). A pure projection: no I/O, no store, no thresholds of its own — it reads job states for reachability, the severity dicts for breaches, health.checks[] for UPS conditions. Rules: an unconfigured source never alerts (invariant 3); a source that is ok but older than 3× its own poll interval is warn while a source that stopped answering is fail; a dead source gets exactly one line (its stale payload adds nothing) that names it once — the poller's "UpstreamError: " wrapper and the duplicated host name are stripped, because a message that says gateway twice reads as two problems; unknown and missing numbers are silence; one alert per (source, metric) and at most 12, the last being a marker that says how many it hid. This is why the overview can be two panels.
• store.py — sample(source,host,metric,res,ts) + event + meta, rollups, pruning.
• api.py — GETs everywhere plus the one POST. /api/config is redaction boundary. /api/summary is the overview's single call and the cheapest whole-fleet probe there is: the page renders only alerts plus system.fleet/system.hosts, while llm.engines, ups.ups, the states maps and recent_events stay for programmatic consumers (live tests, an external check, a notifier). system/llm/ups/recent_events are unchanged and alerts is additive — tests/test_api.py pins every one of those key paths, with and without targets configured, so retiring a panel can never shrink the payload. A configured job with no payload yet is listed as a stub (unreachable: true, every value null) in system.hosts/llm.engines exactly as /api/system/snapshot does, so hosts_total counts the machines there are and hosts_live the ones answering.
• app.py — assembly; create_app() takes injectable settings/store so tests build an app with an in-memory DB and no targets.
• web/ — four static pages, one stylesheet, charts.js. No framework. app.js owns the colour vocabulary: Dash.severityFrom(cellOrSeverity) turns a payload verdict into an ok|warn|fail class ('' for anything unrated, so unknown never looks green) and Dash.hero(el, {label, value, unit, digits, severity, sub, bar}) renders/patches a .hero big number (plus an optional .bar) from that class — it never compares a number to a threshold. Dash.severityFor(value, warnAt, failAt) is the documented fallback for values the API does not rate; converting its remaining call sites is phases 04–05's job (/ was converted in phase 02, /system in phase 03 — those pages colour only from the severity verdicts the API sends, and no severityFor( call survives in their scripts).

Metric naming: dotted (cpu.usage_pct, mem.used_pct, disk.used_pct@/srv, gpu.busy_pct@0, llm.decode_tokens_per_second, ups.charge_pct), @stream for per-device series — /api/metrics answers with exactly these names. That scheme is the history API's key — treat it as a public interface.

The fleet (real environment, verified 2026-10)

Host	ssh alias	What it gives
driveripper	driveripper-ops	Threadripper 24c/48t + Radeon AI PRO R9700, vLLM :8000, upsd, the dashboards
deskwork	deskwork-ops	Strix Halo (Radeon 8060S iGPU), gufo :8731
gateway	gateway-ops	router, no GPU

• Agents run from ~/lab-agent/host_agent.py on each host, forwarded to 127.0.0.1:9121/9122/9123 by scripts/dev-tunnels.sh (DEV_TUNNELS). Nothing listens on the hosts in dev.
• Engines are reached directly: gufo deskwork.reeselink.com:8731, vLLM driveripper.reeselink.com:8000.
• NUT 2.8.5 on driveripper: configs live in /etc/ups/, not /etc/nut/, and upsd is on port 3051, not 3493. Nothing listens on 3493 at all, so a default client reports Connection refused and it looks like a dead service — ss -ltnH plus printf 'LIST UPS\n' | nc 127.0.0.1 3051 (answers BEGIN LIST UPS) is how that was pinned down. Two UPSes are exported over USB and read live through DEV_FORWARDS="3494:driveripper-ops:3051": apc = APC Back-UPS RS 1500MS, cyberpower = CyberPower PR1500LCDRT2U (that is ups.description, the operator's label — the driver's own identity.model answers GX1500U, so the description is what identifies the box). Real-hardware quirks the parsers already handle: battery.mfr.date is the useful date (battery.date is often an unset placeholder like 2001-09-25), CyberPower reports battery.mfr.date: CPS (not a date → unknown, never a fake age), neither driver reports a battery temperature, and ups.test.result: No test initiated is the factory state on both.
• Host access is via ssh keys as ops. Passwordless sudo exists but is command-scoped; the set we rely on is documented in deploy/nut-sudoers (additions are listed there for the work that is still blocked).

Commands

uv sync --python 3.14                     # deps + editable install
cp .env.example .env                      # production-shaped config
cp .env.dev.example .env.dev              # dev-only: ssh tunnels + live targets

./scripts/dev-tunnels.sh start            # agents (9121-9123) + DEV_FORWARDS (upsd 3494)
./scripts/dev-server.sh restart           # keep http://127.0.0.1:8090 up
./scripts/dev-server.sh status            # per-source ok/DOWN
./scripts/dev-server.sh logs

uv run pytest                             # hermetic: 472 passed + 7 live-skipped, no network
uv run pytest --cov                       # coverage gate (96.9%; floor 94%)
uv run pytest -m live --live              # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh             # re-record ground truth

bash .agents/validate.sh                  # the phase gate: the four checks above (pytest
                                          # --cov --cov-fail-under=94, ruff, pyright, plus a
                                          # node --check of web/static) + repo hygiene.
                                          # Always `uv run` — a bare `python3`/`ruff` on PATH
                                          # tests an interpreter nobody deploys.

Deployment: Containerfile (two-stage, uv sync --frozen, runs as uid 10001 on 0.0.0.0:8090, SQLite under /var/lib/dashboards — put a volume there) and deploy/install-agent.sh per monitored host (systemd unit, own user, --bind/--port/--token/--rotate-token/ --status/--uninstall; it stores the token in a root-owned 0600 file and hands the agent --token-file, never argv, and runs under ProtectSystem=strict). The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other service here.

Testing policy

Real captures for shape, fakes for behaviour, live for drift.

• Anything that must match an upstream — /proc layouts, vLLM series names, NUT framing — is asserted against a verbatim capture in tests/fixtures/. If the schema drifts, a fixture test fails before a dashboard does.
• Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host rebooting mid-poll, a counter reset — is a fake in tests/conftest.py.
• Page payloads that a rewrite must not break are pinned as a table. tests/card_contract.py holds the dotted /api/system/snapshot paths the /system hero cards read (CARD_METRICS, GPU_METRICS, HOST_SHAPES); tests/test_system_contract.py asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and test_every_host_serves_the_hero_card_contract asserts the same table against the live agents — so a rename fails a test, and a card that shows n/a cannot be "fixed" by inventing a field. network carries no severity on purpose (throughput has no good/bad), and the contract pins its absence.
• live (tests/test_live_fleet.py) re-runs the production code paths against the real fleet, engines, and upsd. Skipped unless --live; each test skips when its source is unset, so partial fleet availability is normal. Live tests must not write — a battery test on hardware something is plugged into is not a test.
• The agent's real-/proc test (tests/test_agent.py) touches only the box running pytest, so it stays in the default run and is not marked live.
• Coverage gate: 80% is the repo floor, the phase gate in .agents/validate.sh enforces 94% (--cov-fail-under), and ~97% is where we are; keep it there. New/modified Python still has to clear >90% on its own. API tests use create_app() with an in-memory DB, never a real target, and never a real upstream: /api/ups/command's send is proven against a scripted nut.NutClient (ScriptedNutClient in tests/test_api.py) because the live suite is read-only — that fake is the only place the write reaches a "driver".

UI conventions

Hand-rolled SVG (charts.js), one stylesheet, dark/light toggle, Δ<window> labels on measured rates, n/a for the unknown, SIMULATED badges on modelled data, per-source status dots in the header. Every chart names itself for assistive tech: pages pass ariaLabel to LineChart, which puts role="img" + aria-label on the <svg> (an unlabelled plot is silence, and the legend is a sibling that labels nothing) — /system does this for all three slots, pinned by tests/test_api.py. Window chips (5m…30d on /system and /llm, 1h…90d on /ups; the API accepts 5m…90d). A panel that cannot get data says why in the panel — the operator should never have to open devtools to learn that a source is unconfigured. Escape anything upstream-provided through DashCharts.escapeHtml.

The overview is exactly two panels — Warnings & errors (ul.alerts, one row per /api/summary alert: verdict class, source linked to its dashboard via the alert's own href, message, age) and Fleet. A non-empty list floats the alerts panel above the fleet table (#overview-panels[data-state] swaps them); an empty one is an explicit "all clear" state below it, because an empty list is a message, not a blank box. A host that is not answering keeps its row, dimmed with the shared .stale class and reading n/a, so the table never implies last good numbers are live and never silently loses a machine. No summary cards and no raw event feed survive on / — tests/test_api.py pins the two-panel shape against the served HTML.

Nothing waits for a writer that never came: tests/test_api.py fails if a page keeps an id that no page script, no shared script, no stylesheet rule and no aria-*/for attribute references — the orphan a deleted renderer leaves behind, caught on served bytes because there is no JS test runner (a locked anchor). Same rule for dash.css: delete the rules a removed panel made dead (.grid.wide went with the overview's fleet+events row); component vocabulary the pages are still being converted to (.hero-bar, the grid and bar variants) stays even while only one page uses it.

A big number is Dash.hero() on .hero / .hero-bar (dash.css): 2.4rem tabular numerals, muted unit, mono .sub footnote (Δ10.0s, "at current load"), coloured by the payload's severity through the existing .ok / .warn / .fail classes — there is no .sev-* family and no second palette, and tests/test_api.py fails if either appears or if a CSS rule reaches for a token that :root does not declare.

The /system host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the .heroes grid; tests/test_api.py pins the slots and their order on the served script). Colour comes from the payload's severity dicts and nowhere else; a missing number or verdict — gateway's CPU temp, any host's absent GPU — prints n/a uncoloured. NET is one slot with two lines (.hero-lines, ↓ in / ↑ out) and is deliberately colourless: throughput has no good/bad, so network carries no severity. The GPU slot holds one hero per card (model, VRAM used/total and the honest APU pool view in the .sub), and everything not glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU power/temp, filesystems, processes — folds into a collapsed "More detail" <details>: a simpler surface, not a less complete page.

A Δ<window> label is a measurement, never a config value: sources/host.py sends the window the estimator actually used as the payload's window_seconds, and /system's footnote prints those (a range when the hosts disagree) instead of the poll interval, which only coincides with them at the defaults.

Charts belong to the host they chart. /system has two sections — the host panels and the transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals panel and the host/metric detail picker are deleted, along with the #chart-* / #detail-* ids and the select / label.field styling only those pickers used. Each panel stacks its own .host-charts — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where the GPU slot is created and destroyed with host.gpus, so gateway shows neither a phantom card nor a hole. One /api/system/history request per host per refresh (hosts=<that host>&metrics=<that panel's series>&range=…, pinned by test_every_system_chart_is_drawn_inside_its_own_host_panel): unknown metrics come back as empty series on purpose, so a panel asks for every series it could draw and asks once. A legend here names metrics (or one GPU's model), never hosts — the panel heading already says who this is, and a legend that grew a second host is the bug this replaced. A host that stops reporting loses its panel and its charts (pruneHosts), never a chart of nothing. Per-core visuals are gone from /system and renderCores is deleted from charts.js with its last caller. drawHostChart() shades a chart only when one series has points — two or three filled areas stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy memory plot; the cpu.usage_pct@coreN series is still stored and still queryable from /api/system/history — removing the plot is not removing the data.

Known gaps / next work

1. UPS instant commands are read-only in practice. Reads work against real upsd (see above), but every button is gated off with "NUT_USER / NUT_PASSWORD must name a user with actions in upsd.users" because no such user exists yet. Creating one safely needs a read of /etc/ups/upsd.users first — sudo tee can write the file but overwriting it blind would delete the users upsmon already authenticates as, which can take the box down at the wrong moment. deploy/nut-sudoers is the installable ruleset that makes that read (and only a scoped write + reload) possible — it carries the ops ALL=(root) NOPASSWD: NUT_READ, NUT_WRITE grant and is installed with sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut, but it is repo-side only so far: the live UPS cards still answer "NUT_USER / NUT_PASSWORD must name a user with actions in upsd.users" (verified through /api/ups/snapshot). Once it is installed: read upsd.users, add [dashboards] with instcmds limited to the battery tests, put the credentials in .env.dev, and the buttons come alive. Never grant actions = -, and never write the file blind.
2. GPU coverage is AMD only. host_agent.gpus() reads amdgpu DRM sysfs (gpu_busy_percent, mem_info_vram_*, hwmon temps, clocks). That covers the current fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at all — needs a nvidia-smi/NVML or DCGM path, and a fixture from a real card before the field map is written.
3. No remote yet. No origin is configured (git remote -v is empty) — the user is providing it, so every phase so far is local-only history.
4. UPS history depth. Events and the numeric series are stored; long-window battery health (capacity fade over months) has the rollups but no dedicated view.

Locked decisions (and why)

• SQLite, not Prometheus/Grafana. Handful of boxes, 1–10s resolution, one file the container owns. Rollups keep a year of charts at a few hundred MB.
• Our own stdlib host agent, not node_exporter. One JSON snapshot with exactly the fields the page needs (GPU model strings, per-mount usage, temperature labels) versus ~1500 series and a re-labeling layer; no dependency to install, ProtectSystem=strict friendly, and read-only by construction. scripts/capture-node-exporter.sh remains to compare ground truth if that is ever revisited.
• Our own NUT client, not pynut/upsc. The protocol is ~10 line commands, the third-party libs are subprocess-based or stalled, and we need instant commands with explicit gating.
• In-house Prometheus parser. Only two upstreams use the format; keeps prometheus_client out of the dependency set.
• Port 8090, inherited from the llm_monitor prototype this started from: 8000 is vLLM and 8731 is gufo in this fleet.
• .env is a seed, never an override — container/systemd env must win.

Definition of done (per change)

The first three are exactly what bash .agents/validate.sh (the phase gate) checks, so no phase may be called done on a subset of them.

uv run pytest green · coverage not down · ruff + pyright clean · uv pip list --outdated empty if deps moved · dev server restarted and /api/health sane · UI change screenshotted in a browser against live data · fixtures re-captured if an upstream contract changed · this file and README.md updated when a rule, command, gap, or fleet fact changed.