Homelab Dashboards

Five dashboards and the overview page that points at them — one domain, one process, one SQLite file:

Page	What it answers	Source
/	Is anything wrong, and which dashboard do I need?	all five, in one request (/api/summary)
/system	What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it	agent/host_agent.py, our own stdlib agent per host
/disks	Are any of my physical disks failing, and which one — health, temperature, power-on hours, wear, sector alarms	the same agents, running smartctl -i -A -H per whole drive (never a partition or a mount); a host whose agent has no smartctl is not a host on this page
/llm	Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles	llama.cpp-family (gufo) /metrics + /slots, vLLM /metrics, radiance /metrics plus its /stats as a best-effort sidecar (its answer to /slots) — read-only
/ups	Battery health, runtime left, power events, and battery-test buttons	NUT upsd, on whatever port it really listens on (its own wire-protocol client)
/gateway	Which domains are coming in on which port, and where each one goes — two connection tables, one per proxy config file (nginx, then haproxy): a trunk per bound port with its configured routes branching off it, each line reading the config's own inbound word (server_name, map key, backend name) on line 1 over the counterpart it routes to on line 2 (→ host:port), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server	the agent on one host, GATEWAY_HOST (default gateway), reading its haproxy.cfg + nginx tree and /proc/net/tcp{,6} — so it needs agent 1.2.0, plus 1.3.0's resolver section to put a number on a route whose target the config writes as a name (without it those lines are drawn and dashed), plus 1.4.0's #off marker read to show a route the switch has turned off or back on (without it an off route is drawn on, because the marker is a comment a 1.3.0 agent does not read) — an off route stays drawn and counted, its row simply reads enabled: false (its id gains a #off suffix), and the config-integrity warnings the marker caused (a backend emptied of its last server, a map entry pointed at the switch's sink) are read as a chosen state while the same shape from a hand-deleted line is still a warning — and the whole page is unconfigured (never red) unless GATEWAY_HOST names a HOST_TARGETS entry

No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at runtime; every asset is served by this app.

Three more pages are not dashboards: /login (the one thing an anonymous visitor sees), /settings — ⚙ Account, in the masthead of every signed-in page — which shows this browser's session, signs it out, and changes the admin password, and /certs — ⚷ Certs, beside it — which is the certificate authority the agents are dialled with: the root CA, this dashboard's own client certificate, and one row per host with its rotate, revoke and bundle download.

Working on this repo? AGENTS.md holds the invariants (read-only by construction, never invent a number, unconfigured ≠ broken), the locked decisions, the real fleet facts, and the known gaps.

Quickstart

uv sync --python 3.14                       # deps + the package, editable
cp .env.example .env                        # then edit — including DASH_ADMIN_PASSWORD
uv run dashboards                           # http://127.0.0.1:8090
uv run uvicorn dashboards.app:app --port 8090   # equivalent
uv run pytest                               # unit tests, no network needed

The first start needs a password to seed: with no DASH_ADMIN_PASSWORD and no verifier stored in DASH_DB yet, the process prints the reason and exits non-zero rather than serving the fleet open (see Signing in). The dev password this project uses locally is testpass; production gets a real one. The first thing a browser meets is /login — every page and every /api route except the four listed under What the session covers asks for it, while /api/health stays open for the container healthcheck and ./scripts/dev-server.sh status.

Developing against the real fleet from a laptop — dev dials exactly the addresses the deployed dashboard dials, so there is nothing to tunnel and no port to forward:

cp .env.dev.example .env.dev                # your host addresses (chmod 0600); this fleet's
                                            # entries are `https://` and carry no userinfo
sudo deploy/install-agent.sh --bind <addr> --port 9120 --fetch-bundle <dashboard-url> \
     --cert-name <name> --session <signed-in token>
                                            # once per monitored host: the agent as a systemd
                                            # service, registered with the bundle this dashboard
                                            # signs for it (the token-mode install — no
                                            # `--cert-bundle` — still works and prints its bearer
                                            # token exactly once; see *Protecting an agent*)
set -a; . ./.env.dev; set +a
uv run dashboards                           # foreground
./scripts/dev-server.sh restart             # or backgrounded, with a pidfile + logs
./scripts/dev-server.sh bundles             # the bundles it exported, with sizes and modes

Local dev signs in with the DASH_ADMIN_PASSWORD in that file — the word the template's comment names (testpass, what the hermetic suite uses) — and without it a fresh database exits non-zero for the same reason production does. .env.dev is gitignored and 0600: it holds the upsd password, and it held the agent bearer tokens until every host of this fleet moved onto an installed bundle — a TLS HOST_TARGETS entry carries no userinfo, so the credential that used to sit in the URL is now a certificate on the host and a meta row here.

dev-server.sh restart also exports the agent bundles while it is up: one line per HOST_TARGETS name, and each host's stored bundle written to data/certs/<name>.bundle at 0600 (data/ is gitignored, so no key this writes can be committed). It is a read of GET /api/certs/<name>/bundle and nothing else — the authority stays the one in the database — so the file is byte-identical to the row and ready to scp into install-agent.sh --cert-bundle — though the normal path now needs neither: install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token> downloads that same route on the host during its own install (see Protecting an agent, host by host). A host with nothing issued writes no file and prints the route that issues one. See Managing the certificates.

Each token-mode HOST_TARGETS entry carries its token as URL userinfo (driveripper=http://<token>@<addr>:9120); parse_targets moves it into an Authorization header, so Target.url, /api/config and every error message stay credential-free. A host that runs from an installed bundle is https://<addr>:9120 with no userinfo at all — the client certificate is the credential there, and this fleet is all https:// today (see Protecting an agent, host by host). No root shell on a host? deploy/agent-sudoers is the one-line grant that lets ops run the installer, and running the installer without root prints that exact line. upsd is dialled at NUT_HOST/NUT_PORT on the port it really answers on (3051 on driveripper, not the stock 3493) — see the NUT section below for who may connect.

The host agent

agent/host_agent.py is one file, stdlib only, Python 3.10+. It reads /proc, /sys/block, /sys/class/drm and statvfs — and, on the host that fronts the fleet, the haproxy/nginx configs and /proc/net/tcp{,6} — and answers GET /snapshot (everything, one JSON document; ?top=N trims the process lists), GET /info (slow-changing identity: OS, kernel, CPU model, GPU list), GET /schema (the schema version plus this document) and GET /health (the one route that needs no credential) as JSON, with a plain HTML status page at GET / so a browser pointed at an agent learns what it is looking at.

python3 host_agent.py --once                                # print one snapshot, exit
python3 host_agent.py --host 0.0.0.0 --port 9120 --token-file /etc/dashboards-agent.token
python3 host_agent.py --host 0.0.0.0 --port 9120 --cert-bundle /etc/dashboards-agent.certs
sudo deploy/install-agent.sh                                # systemd unit + user + token
sudo deploy/install-agent.sh --cert-bundle agent.bundle     # ...or the TLS install (bundle from /certs)
sudo deploy/install-agent.sh --fetch-bundle https://dashboards.example.com \
     --cert-name deskwork --session <signed-in token> --bind <addr> --port 9120
                                            # ...the same TLS install, bundle fetched in the run
sudo deploy/install-agent.sh --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN
                                            # ...or a kernel-gated host (SMART passthrough, below)
sudo deploy/install-agent.sh --force        # ...only when it must overwrite a unit it would change
sudo deploy/install-agent.sh --with-routectl # ...plus the route-switch helper (the gateway, below)
sudo deploy/install-agent.sh --allow-writes --with-routectl  # the gateway's full write install:
                                            # the flag on the agent unit + the helper beside it
deploy/install-agent.sh --status                            # unit state and /health (https when TLS)
sudo deploy/install-agent.sh --uninstall
sudo deploy/install-agent.sh --without-routectl   # remove helper + units, keep config and state

Re-running the installer is the silent, idempotent path it has always been as long as the unit it would write is the unit already on disk. When the two differ the run stops: it prints refusing to overwrite … with the unified diff on stdout and exits 1 before writing the unit, before daemon-reload and before the service is re-enabled, so a routine redeploy can never undo a fix somebody made by hand on the host (--as-root/--caps on a host already running the dedicated user is exactly such a change, and installing a TLS bundle onto a token-mode host is another). --force prints the same diff and then applies it.

No root shell on the monitored host? deploy/agent-sudoers is the single sudoers line that lets the fleet's unprivileged ops account run the installer; run it without root and it prints that exact line with the script's own resolved path instead of saying "run as root".

It is read-only by construction except the one write below: every GET path answers a read, and the HTTP server's one other path is the gated POST /route. Counters are reported as counters — the dashboard computes rates over a labelled window, so the agent never has to remember who polled it last.

The one write (1.4.0, the gateway). 1.4.0 adds a single POST /route, reachable only with a verified client certificate and the agent's own --allow-writes flag, and it dials a socket-activated root service rather than giving the agent any privilege. The full write install is --allow-writes --with-routectl: the first adds exactly one argument to the agent unit's ExecStart (a re-run diff that shows a security directive moving is a stop, not a --force), and the second installs deploy/dashboards-routectl.py to /usr/local/sbin/ (0755, root) and the two units dashboards-routectl.socket (a 0600 stream socket at /run/dashboards-routectl.sock, owned by the agent's user) and dashboards-routectl@.service (the helper, root, one instance per connection, StandardInput=socket, ReadWritePaths limited to /etc/haproxy, /etc/nginx/conf.d and /etc/nginx/stream.d), then verifies that the agent's own uid can connect before reporting success. No sudoers line is involved: the grant is a file mode systemd enforces, the helper's SO_PEERCRED check is the second look, and with --with-routectl alone the rendered dashboards-agent.service is byte-identical. The helper edits exactly one route (a #off <name> marker), validates the file with the proxy's own checker before it means anything, restores the previous bytes on a checker failure, and reloads, never restarts; --without-routectl removes it again, printing the list of still-off routes first. deploy/gateway-routectl-README.md is the operator's page for all of it.

Per-disk smart health is a best-effort add-on on the whole-disk /sys/block list (no partitions): the agent runs smartctl -i -A -H per physical disk — 5 s bound per device, a pass budget of 20 s as a floor that scales to 5 s × the disk count, results cached 4 minutes — and parses health, temperature, power-on hours, the sector alarms and SSD wear into disk_io.devices[].smart. Refreshing an expired cache happens on a background thread while the request is served the previous result, so a wedged disk can never cost a poll more than its own timeout (only the very first pass after the agent starts runs inline). A host without smartmontools, or whose agent user cannot read the block devices, is not broken: every disk reports smart: null with one explanatory errors line, and that line never makes the system page amber. The two cases differ on /disks: a host that only cannot read its drives keeps its (dimmed, all-n/a) cards, while one whose agent says smartctl is not installed is not listed on that page at all. Enable SMART with smartmontools plus the disk group (which deploy/install-agent.sh grants when it exists) — the agent shells smartctl as itself and never calls sudo, so a sudoers rule alone is not enough — and redeploy the agent, since one installed before phase 09 has no SMART pass to run. Opt out silently with --no-smart. The drive truth this is all developed against lives in tests/fixtures/smartctl_driveripper_*.txt: 16 real drives, recorded with serials scrubbed, each one parsed by a test.

The haproxy section is the same kind of best-effort add-on, for the host that fronts the fleet. The agent reads /etc/haproxy/haproxy.cfg (--haproxy-config names another path) and reports what the config says: every frontend block with its name, each bind address:port exactly as written (:::22 is the address :: on port 22) and the backend names it routes to (default_backend, plus every named use_backend), and every backend block with each server's name, target and port — port: null when the line names none, never 0. It interprets nothing: the routing between the two lists is the names, so joining them is the gateway page's job (sources/gateway.py) and the agent never evaluates a rule. available answers three questions the way smartctl_available does — true read and parsed, false no config at that path (this host runs no haproxy), null a file is there but could not be read or understood — and each non-true answer costs exactly one errors line for the pass. Like the SMART lines, those lines never make the system page amber: a host that never installed haproxy is unconfigured, not broken, and the page that owns the add-on reads the sentence out of agent.errors. A failing config read cannot fail /snapshot or /health, and --no-proxy drops the section and the line both. The parser is pinned to tests/fixtures/gateway_haproxy.cfg, the verbatim capture from gateway. Neither path is an env var and deploy/install-agent.sh passes neither flag — the unit it writes runs on the defaults — so a box whose configs live elsewhere means adding the flag to $UNIT after installing.

The nginx section is that add-on's other half, and it reads more than one file: the main config (/etc/nginx/nginx.conf, --nginx-config names another) plus the files its own include lines name, one level deep, because that is where nginx keeps its vhosts (include /etc/nginx/conf.d/*.conf) and its SNI routing (/etc/nginx/stream.d/*.conf). It reports every server block with its listen address:port pairs ([::]:8443 is the address :: on port 8443; a bare listen 80 has no address, and the ssl/proxy_protocol flags are not this section's business), its server_name list (the catch-all _ included verbatim — "any domain" is what the page says about it), and every proxy_pass; every upstream block with its server targets; and every map block with its keys. A proxy_pass $var is served with the variable's name and target: null: nginx resolves that per connection from the map that defines it, so the agent reports both facts unmerged and leaves the joining to a page — a merged target would be a number the wire never published. files names the main config and the included documents actually merged, and the include pass is bounded (8 files / 1 MB): what it could not take is said in one errors line naming the count, and the section still reads true. The same three-way available, the same one line per arm, the same rule that a missing, unreadable or unparseable config degrades only this section — and --no-proxy drops the whole family with it, in silence.

The sockets section is that add-on's live half: what is actually open and carrying traffic, as opposed to what a config claims. The agent reads /proc/net/tcp and /proc/net/tcp6 and keeps only the two states a gateway page can draw pipes from — kernel state 0A LISTEN into listens ({family, addr, port}) and 01 ESTABLISHED into established ({family, local, remote}, each end an {addr, port}). Every other state is a socket in transit: a TIME_WAIT row is a connection that stopped being true before the page painted, and a proxy accumulates thousands of them, so they are skipped without a word. Each table is read through at most 4096 lines (/proc's own header among them, so 4095 sockets per table); a longer one reports the rows it read and costs one errors line naming the bound, with available still true. addr is the address itself, unrolled from /proc's per-32-bit-word little-endian spelling — 0.0.0.0 and :: keep the literal wildcard form, because "open on every address" is the fact the inbound join keys on — and a word that is not an address is null, never a guess. The tables are read independently: a kernel without IPv6 has no tcp6 file, which costs nothing and no line, while a table that is there and denied costs one. available is true when either table answered and null when neither did, with one line saying why; there is no false, because a table that did not answer is "no answer" and not "this host has no sockets". Nothing here is joined to a config: a socket does not know its domain or its backend, so the gateway page is what joins these ports to the haproxy and nginx sections, and a socket no config claims stays visible as one. The join lives in sources/gateway.py, never in the agent: /snapshot keeps the three sections as the agent read them, /api/gateway/snapshot serves the joined topology, and only the counts that join produces are stored — as the gateway.in_conns@<port> / gateway.out_conns@<host>:<port> gauges the page charts (since phase 30 task 02 the outbound gauge exists only for an endpoint the config wrote as an address: the name-keyed ones were a chart of the port-only guess the join no longer makes, so they would be permanently empty). Nothing in /api/system/snapshot grows a proxy or socket field, and no sample row is written for a config.

The resolver section is the fourth and the only one that leaves the host. The two configs above name their targets two ways — an address (proxy_pass http://127.0.0.1:8081) and a name (server mc1 minecraft.reeselink.com:25565) — and the kernel's socket table speaks addresses, so a name is the half a page could never count. The agent therefore takes every (name, port) the two parsed sections write as a target (haproxy server lines, then nginx upstream members, proxy_pass targets and map entry targets, deduplicated in config order) and asks libc what each one is: getaddrinfo over SOCK_STREAM, published as via: "libc" because that is the resolution haproxy's own libc path performs on this box — the box that opens the connections. The nginx config's own resolver directive travels beside it verbatim as config_directive (10.1.0.1 here, read out of stream.d/stream.conf), so a divergence between the two vantages is something an operator can see rather than something this file guesses; the agent does not query that resolver directly, and the dashboard never resolves anything at all (sources/gateway.py has a test that fails if it ever calls a resolver). An address literal is reported status: "literal" with its address and no lookup, and a name that does not answer right now is nxdomain or error with addresses: [] — the route keeps its row and carries no count, never a 0.

That pass is bounded and cached exactly the way the SMART one is, for the SMART reason: at most 64 lookups inside 3 s of wall clock, cached 300 s, the first pass inline so a freshly started agent answers with real addresses and every later expiry refreshed on a single background thread while the request is served the previous result. getaddrinfo takes no timeout argument, so stopping between lookups is the only defence a black-holing resolver leaves — and the cost of not stopping is the system page's poll timeout, spent on an add-on that page never draws. Each entry carries the time it was answered, so a cache-served entry cannot pretend to be current; the budget and the name bound each report themselves in one errors line and leave available at true. available is the family's three-way answer (false means there was nothing to resolve because no proxy config was read, and its one line says not found or unreadable according to which of the two it was), and --no-resolve removes the section in silence the way --no-smart removes SMART — while --no-proxy removes it too, because a box that collects no proxy has no declared names to look up.

All four parsers are pinned to captures rather than to descriptions, and ./scripts/capture-gateway-config.sh re-records them over the GATEWAY_SSH alias (default gateway-ops): /etc/haproxy/haproxy.cfg, /etc/nginx/nginx.conf plus every document its own glob include lines name (gateway_nginx_conf_d_*.conf, gateway_nginx_stream_d_*.conf), /proc/net/tcp{,6}, and tests/fixtures/gateway_resolver.json — the resolver section itself, so the name→address join is testable with no network. No sudo and no DNS client: those paths are world-readable, which is exactly what the agent reads as itself, and the resolver is recorded with getent ahostsv4 / getent ahostsv6, which is libc on the box that dials (the same vantage via: "libc" claims). A read that fails, or answers empty, leaves the existing fixture alone and exits 1, so a laptop with no route to the gateway cannot turn seven captures into seven empty files.

The resolver document is derived, never listed: the script feeds its own freshly captured configs through agent.host_agent's parsers and target_names(), so it asks about exactly the targets the agent will ask about, an address literal costs no remote read, and a route added to the gateway moves the fixture with it. getent answering rc 2 is a fact — that name has no record and becomes status: "nxdomain", kept beside the names that did answer — while a dead transport, a missing getent, or not one name answering is a broken vantage: the run refuses, because a document of nothing but nxdomain would make every route on /gateway a permanent dash and every join test a test of nothing. Re-capturing a gateway nobody changed is a no-op (fixed ts, deduplicated, sorted with the agent's own sort_resolver_addresses), so a diff in that file means the ingress changed.

The configs are not rewritten (their domains are the ingress map, and key material appears only as the paths nginx is told to read); the socket tables and the resolver's answers lose only their identifiers, in line with the rule that scrubs a drive's serial: a globally routable address becomes a documentation one (RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI-64 and an inbound visitor's address is not ours to publish. Every port, state, column and width in those tables is verbatim, and those files are volatile by nature — a proxy in service has a different connection set every second — so the tests assert shape and the ports the configs claim, never a particular row. AGENT_VERSION is 1.4.0 — the resolver section (1.3.0) and the #off marker read plus the one write (below) on top of the three phase 28 shipped; GET /schema names all four, each with its available flag, and --no-proxy and --no-resolve are listed there too. A host still running 1.2.0 answers with a snapshot that simply has no resolver key — which is why the version spread costs nothing: a section an agent did not collect is absence, and the gateway page counts the address-written routes it has and draws the name-written ones with a dash.

The dashboard reads that section and never produces one. sources/gateway.py indexes the entries by (name, port), stamps each route's out with resolved / via / resolved_ts — the address the gateway dialled, the agent's own string for how it got it, and the moment that entry was answered — and lets the socket join reach a name-written route through it. Which is the whole of phase 32: the outbound endpoint of a server line, a proxy_pass or a map entry is now an address the box that opens the connection published, so the gateway.out_conns@<domain>:<port> gauges phase 30 task 02 stopped writing are written again, under the config's own spelling, with a figure that belongs to the domain instead of to whatever else was listening on its port. Four rules keep the join honest, each pinned in tests/test_gateway_source.py: a name nobody resolved keeps resolved: null, connections: null, metric: null and no stored sample (never 0, never an empty list); matching is by normalised address, so ::ffff:10.1.2.64 in /proc and 10.1.2.64 from getaddrinfo are one endpoint while a zero-padded 010.1.2.64 — which is no one's address notation — matches nothing; two names resolving to one address, or two route lines reaching it, are one count held by every line that reaches it; and the module still never calls a resolver, a spy pin covering the parse, the gauge name and the capture alike. An address: true route claims none of the three provenance fields, because the config wrote that address and no lookup happened.

deploy/install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token> is the same TLS install with the download done inside the registration run: it asks GET <url>/api/certs/<name>/bundle, writes the answer to a temp file, and hands it to the identical validator + 0640 install + unit path. The three flags are all-or-nothing, they are refused together with a local --cert-bundle (one install, one bundle source) and with --status/--uninstall, and the fetch and the shape check both run before the root boundary — a dashboard that answers 401, or that has issued nothing for that name, stops the run with its own one-line sentence and no partial install. The session token is a fetch credential only: it is sent on that one request, never printed (the sudoers hint re-quotes the command without it), never written to the token file, the unit, the bundle or the log, and unset the moment the bytes are in hand. The temp copy is removed on every exit path, so the installed /etc/dashboards-agent.certs is the only place the fetched key survives.

TLS mode is --cert-bundle (env AGENT_CERT_BUNDLE): one installed PEM the dashboard signed — this host's server certificate and key, the dashboard's root CA, a # BUNDLE-REV <n> line and the # REVOKED <serial> denylist. The agent then serves https:// and answers every path but /health only to a client certificate signed by that root and not on the list: 403 client certificate required for a peer that proved nothing, 403 client certificate revoked for one the operator withdrew (a revoked identity is refused on /health too — that path is open to the installer's probe, which holds the CA and no certificate, not to a certificate that was taken away). The denylist is re-read when the bundle file's stamp changes, so a re-deployed bundle takes effect on the next request rather than the next restart; the certificate itself needs a restart, because ssl cannot swap a chain under a live socket. The unpacked PEMs exist only long enough to load into the context and are then deleted, and the private key is written 0600 in the unit's RuntimeDirectory when it has one. In this mode /snapshot gains two additive fields — tls: true and the bundle_rev the bundle carried, which is how the dashboard tells a stale install from a current one (they reach /api/system/snapshot and /api/disks/snapshot as agent.tls / agent.bundle_rev) — and the bearer token is not read: the handshake is the auth, and a token that kept working after a revocation would defeat the list. The dashboard's half is collectors.AgentTls: point that host's HOST_TARGETS entry at https:// and the fetch presents the dashboard's client certificate and verifies the agent against the same root, and a host revoked in certs.agents is refused before a socket is opened. A bundle that will not load is exit 2 naming the file; there is no "fall back to plain HTTP" path.

deploy/install-agent.sh --cert-bundle <file> is how that file gets there. It validates the bundle first by the same text shape (two certificates and exactly one key, plus the # BUNDLE-REV line) — before touching anything, before the root check, and without ever dialling the dashboard — then installs it to /etc/dashboards-agent.certs as 0640 root:dashboards-agent, the token-file idiom: the agent opens the bundle itself at startup and re-reads it when its stamp changes, so the service user must be able to read it and no other account may. The unit gains --cert-bundle and a private RuntimeDirectory for the PEM blocks, --status probes https://…/health with the bundle's CA and no client certificate (python3 ssl, never an external TLS binary), --uninstall removes the bundle with the token, and a re-run without the flag leaves an installed bundle in place — the token-reuse idiom.

Binding off-loopback without a token or a bundle is refused (exit 2) rather than warned about. Prefer --token-file over --token: argv is world-readable through ps, and this token guards hostname, mounts, disk and GPU models, and process names. deploy/install-agent.sh creates a system user, generates a token once (re-runs reuse it; --rotate-token for a new one), stores it 0640 root:dashboards-agent — the unit hands the agent the path, and the agent opens it as its own user, so the file must be readable by that user and by nobody else — and runs the agent under ProtectSystem=strict with an empty CapabilityBoundingSet (unless --caps is asked for, below) and no device filter. SMART device access is DAC: the installer adds the service user to the disk group and SELinux stays in front of it. The strict device policy this script used to write is what locked driveripper's whole device namespace — systemd matches those globs by name against /proc/devices, where kernel 7.2 lists NVMe as blkext, so /dev/nvme* matched nothing and a strict policy denies whatever a list fails to match. PrivateDevices=false is still written out explicitly, because a private /dev with no device list is what dies at spawn (status=208/STDIN).

When DAC is not enough — a kernel gate — --as-root and --caps write the two unit lines that gate checks accept. Kernel 7.2 gates smartctl's raw passthrough behind capabilities, not uid: an NVMe SMART-log read through the block node needs CAP_SYS_ADMIN unless the drive's firmware advertises the page, and SATA SMART behind a SAT layer is an ATA pass-through the unprivileged SG_IO allow-list does not include (CAP_SYS_RAWIO is what the kernel names for it). So --as-root writes User=root and --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN writes CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN (comma-separated in, space-separated out); both default off, because a host whose drives answer through the disk group needs neither. --caps without --as-root is refused with exit 1 — a capability grant on the dedicated service user is not the kernel-gate fix — and --as-root alone proceeds with a note, because a root process under the default empty bounding set is capability-less (CapEff 0) and reads no more than the unprivileged service user did. The token file, its 0640 root:dashboards-agent mode and the --status probe are deliberately unchanged by these flags: root reads the token either way, and a probe of a port cannot see who owns the process behind it.

Managing the certificates

The dashboard is the authority, and /certs (⚷ Certs in the masthead) is its page: the Root CA block (CN, serial, expiry as a local timestamp and as time left, and Rotate root CA), the Dashboard client line (the one certificate every agent verifies the dial against — display-only, because it is minted with the root and has no route of its own), and one row per host with its status chip, serial, expiry, issued-vs-deployed bundle revision, the addresses its certificate is valid for, and Rotate / Revoke / bundle. The agents table carries the stylesheet's own scroll modifier, so it is a box on both axes with a 320 px height cap at every width and its right-hand columns — the serial, the bundle revisions, the addresses and the controls — are reachable by scrolling instead of clipped. Since phase 27 it is no longer the only thing that scrolls (at ≤640px every table.data does, sideways only, no height cap); .scroll out-ranks that rule, so this table keeps its own box on a phone exactly as on a desktop.

Nothing there is coloured: severity.py rates no certificate, so an expiry is a date and a bundle revision is a count. The one verdict on the page is the API's status word (active green, revoked red, unissued grey — a certificate that does not exist never borrows a green pass). Every write asks a confirm() first and prints the server's own sentence afterwards; a root rotation answers restart_required, because the CA and the client pair are startup material — the table is new at once, the running process keeps dialling with what it loaded, and every host then needs its new bundle installed. A row whose host has left HOST_TARGETS is listed with no buttons at all, because every per-host route refuses it.

Under the page is the same API a script would call. Every route below is signed-in (and every write is CSRF-gated like any other), and none of them dials a host — the effect of a revocation is that the next poll refuses to dial (see collectors.AgentTls above):

Route	What it does
GET /api/certs	root and client with serial and expiry, plus one row per host the operator could act on: every HOST_TARGETS name and every certificate still in the table. A configured host nobody has installed yet reads status: "unissued" with null serial rather than going missing; bundle_rev is what this dashboard issued and deployed_bundle_rev is what that host's agent reported back (so "the box is one bundle behind" is visible, n/a when nothing answered). Reads are pure — no page refresh ever mints a certificate
GET /api/certs/<name>/bundle	the exact bytes of the stored bundle as application/x-pem-file with Content-Disposition: attachment; filename="agent-<name>.bundle". That byte-equality is the point: install-agent.sh is handed the file and compares nothing else, so a route that re-signed on each read would make "is this the bundle I installed?" unanswerable. It carries that host's private key, which is why it is a download behind the session and no-store
POST /api/certs/<name>/rotate	fresh key, fresh serial, bundle_rev +1, status: active — and it is also how a host gets its first bundle (bundle_rev 1). The SAN comes from the current HOST_TARGETS URLs plus the name, not from the old certificate, because the address a host is dialled by is a deployment fact
POST /api/certs/<name>/revoke	marks the identity revoked, and re-seals every other bundle because the # REVOKED denylist lives inside each of them — so the fleet has new bundles to collect, while their bundle_rev does not move (their identity did not change). There is no unrevoke: the fix is a rotate
POST /api/certs/root/rotate	a new root, a new client certificate, every agent re-issued, and {"ok": true, "restart_required": true} — the CA and the client pair are startup material, so the table is new at once while the running process keeps dialling with what it loaded. Saying so beats pretending a hot-swap that does not exist

During development there is no second authority. The CA the first boot minted into the meta table is the one everything reads, and scripts/dev-certs-export.py is the dev loop's half of that: ./scripts/dev-server.sh restart runs it after the readiness probe, it signs in to the local server, downloads every HOST_TARGETS name's bundle through GET /api/certs/<name>/bundle, and writes each one to data/certs/<name>.bundle at 0600 (os.open, so the mode is the file's birth mode) in gitignored data/. Per host it prints exactly one line — the file it wrote with its size, or no bundle issued yet — issue one on /certs — and nothing at all is written for a host with nothing issued. It is a read of the same route the /certs download uses, never the startup path, so a restart cannot rotate an identity: the bytes on disk are the bytes in the row. A sign-in that fails is exit 2 and a ! line (a 429 says so with its wait), and dev-server.sh prints that as a warning rather than failing start, because the server is provably up and the authority is untouched. status — or dev-server.sh bundles — lists what is there with sizes and modes.

Protecting an agent, host by host

Four steps, and they are order-dependent because the dashboard picks its dial from the URL. A URL flipped before its bundle is installed turns that host red by construction, so the URL flips last, after the probe answers.

# 0. issue the host a bundle if it has none: POST /api/certs/<name>/rotate (the Rotate button on
#    /certs) is also how a host gets its *first* one, and `GET …/bundle` then answers the bytes.
#    An unissued name is what makes step 1 say "no bundle issued for <name>".

# 1. register the host: one run installs the agent *and* fetches its bundle from the dashboard
#    that owns the authority. The name is the HOST_TARGETS name; the token is a signed-in
#    session and it buys this one GET — it is not stored on the host. As root, 0640
#    root:dashboards-agent, after a shape check on the fetched bytes that runs before anything
#    is written, so a 401 or an unissued name leaves this host exactly as it was.
#    Stage the repo's script and agent first (`../agent/host_agent.py` is where the installer
#    reads the agent from, and the fleet's sudo grant names a path, not a checkout):
scp deploy/install-agent.sh <host>-ops:/home/ops/agent-install/deploy/
scp agent/host_agent.py     <host>-ops:/home/ops/agent-install/agent/
ssh <host>-ops sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120 \
     --fetch-bundle https://dashboards.example.com --cert-name deskwork --session "$SESSION" \
     --force

#    the fallback, for a host that cannot reach the dashboard: the signed-in GET by hand (its
#    bytes are identical to the stored row and it carries that host's private key, so `curl -o`
#    it somewhere private rather than `cat`ting it into a shell history — and against a local dev
#    server this file already exists, written by `./scripts/dev-server.sh restart`), then
#    --cert-bundle <file> instead of --fetch-bundle.
#      curl -s -H "Authorization: Bearer $SESSION" -o agent-deskwork.bundle \
#           http://127.0.0.1:8090/api/certs/deskwork/bundle
#      scp agent-deskwork.bundle <host>-ops:/tmp/
#    A *dev* dashboard on a laptop is usually unreachable from a monitored host in the other
#    direction too (the tunnel route is one-way). An SSH reverse tunnel keeps the registration
#    path rather than the scp: `ssh -R 18099:127.0.0.1:8090 <host>-ops` and then
#    `--fetch-bundle http://127.0.0.1:18099`.

# 2. restart it as root: the same run copies the repo's `agent/host_agent.py`, which is how a
#    pre-1.1.0 agent gains TLS mode at all (an older file does not know the `--cert-bundle`
#    flag), and the unit gains `--cert-bundle` and its RuntimeDirectory — but `systemctl enable
#    --now` only *starts* a unit, so an already-running agent keeps serving its old mode until
#    something restarts it.
ssh <host>-ops sudo systemctl restart dashboards-agent
#    the CA-only https probe — pass the SAME --bind the unit uses, because the probe takes its
#    address from the flags and would otherwise dial 127.0.0.1 and report "no answer" for an
#    agent bound to a LAN address that is in fact healthy.
ssh <host>-ops sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120 --status

# 3. point that host's HOST_TARGETS entry at https with NO userinfo token — the certificate is
#    the auth now — and restart the dashboard. `/certs` should then read issued == deployed rev.
export HOST_TARGETS="deskwork=https://<addr>:9120 …"

--force in step 1 is not a shortcut: installing a bundle is a change to that host's unit (the --cert-bundle flag and its RuntimeDirectory), so without it the run prints the diff and stops by design — read it, then re-run with the flag. Reading it first is cheap and worth doing on a host with a hand-fixed unit (run the same command once without the flag, study the diff, run it again with it). The same rule covers every change the installer applies, which is why the --status line above comes after the restart: the probe reads the unit file to decide between http and https, so pointing it at a process that has not been restarted yet reports "no answer" for a bundle that installed perfectly. install-agent.sh --bind … --port 9120 prints the https://<addr>:9120 line for a bundle install, so step 3 is a copy rather than a guess. A host left on http:// keeps working over its bearer token — the rollout is per host, and one revoked or expired certificate never blinks the others.

Upgrading an already-registered host to a newer agent is one file, not a rollout. Re-stage agent/host_agent.py into /home/ops/agent-install/agent/ (the snapshot-first rule applies: a path grant is only as current as the file it names, so copy what you are replacing to host_agent.py-<date>-backup first — a name the sudo grant does not name, so sudo will not run it), run the granted path with the unit's own arguments and nothing else (sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120), and restart. No --cert-bundle, no --fetch-bundle, no --force: the bundle already on disk is what the unit render reads — the flag only replaces the file — so the render equals the installed unit, the re-run guard prints nothing, and /etc/dashboards-agent.certs and the token file keep their bytes. Then the same two checks the registration uses: the CA-only --status probe (with --bind) answers {"ok": true, …, "version": "1.3.0"} on /health, and the dashboard's own https:// dial answers whatever the new version added — agent_version plus the haproxy / nginx / sockets sections (1.2.0) and the resolver section (1.3.0) — beside the sections it always had. That is exactly how gateway moved 1.1.0 → 1.2.0 (phase 28) and 1.2.0 → 1.3.0 (phase 32, 2026-10-25: the same unit sha256 before and after, the bundle untouched, the guard silent, one file and one restart), and the /gateway page counted 33 of 33 routes afterwards where it had counted 2 before.

Two things the first full rollout learned, both cheap to hit and expensive to guess at:

• A CA row older than the identifier fix must be rotated before any bundle is installed. A root minted without a SubjectKeyIdentifier is still readable, still starts, and still signs — but ssl.create_default_context() (the strict mode every dial here builds) rejects every chain it produces, with CERTIFICATE_VERIFY_FAILED: Missing Subject Key Identifier, so the host comes up healthy on https and reads unreachable (ConnectError) from the dashboard. POST /api/certs/root/rotate is the repair — a new root with its identifiers, a new client identity, every agent re-issued, then a dashboard restart and a fresh bundle per host. Never a DELETE FROM meta for this: the rotate button exists precisely so the operator keeps the authority in one place.
• Rolling back is not just the URL. A TLS-mode agent serves https only, and the unit keeps its --token-file line (so the token survives on the box, ready for this). Restoring an old http://<token>@… entry does not bring the host back on its own: the --cert-bundle line has to come out of the unit and /etc/dashboards-agent.certs has to go, because the installer reuses an installed bundle on a run that was not given the flag.

A name that is not in HOST_TARGETS answers 404 {"configured": false} before any signing, any dial, and any meta write: a typo that minted a key would leave a certificate nobody can install in the authority forever. Each accepted write leaves one fixed-text event — cert-rotated, cert-revoked, cert-root-rotated under source: "certs", readable at /api/events?source=certs — and a certificate serial never leaves a meta row. Serials are served as decimal strings for that reason: a 64-bit serial through a JS number is a serial the page cannot match against a denylist.

Signing in

One admin, one password, and the server refuses to start when it has neither a stored verifier nor DASH_ADMIN_PASSWORD — the domain is public and the fleet's hostnames and UPS controls sit behind this login, so there is no way to run it open by accident. There is no second account and no per-user view: the six dashboard pages (/, /system, /disks, /llm, /ups, /gateway), the two admin pages (/settings, /certs) and every data route belong to that one operator.

A session is an HMAC-signed cookie (dash_session), not a row in a table: it carries the signed-in name and its own expiry, is valid for DASH_SESSION_HOURS (default 720 = 30 days), and is verified with a signing key that also lives in DASH_DB. That is why a password change signs out every other browser at once — it rotates the key, so every cookie minted under the old one stops verifying on its next request — and why nothing has to be pruned.

DASH_ADMIN_PASSWORD is a seed, not the credential. The first start hashes it (PBKDF2-HMAC-SHA256, 600 000 rounds) into the meta table of DASH_DB as auth.verifier, and from then on the stored verifier wins: remove the variable once the server is up and the same password still works, change it with POST /api/auth/password — or in a browser, on the /settings page (The account page below) — with no redeploy, and know that a container env can never roll a changed password back. If DASH_ADMIN_PASSWORD is still set on a box that already has a verifier, it loses — and the server says so once at startup, in words, without quoting the value.

Forgotten password, or a database you want to hand over? Delete the rows and seed again:

sqlite3 "$DASH_DB" "DELETE FROM meta WHERE key IN ('auth.verifier','auth.session_secret')"

Deleting auth.session_secret — which that command does — signs out every browser already signed in, which is also exactly what changing the password does on purpose. Only the hash and the signing key are ever written: grep -c "your password" "$DASH_DB" comes back 0, and no log line, event row, or API response carries the password, the verifier, or a session token.

Nothing that leaves the app carries a credential. A HOST_TARGETS userinfo token is parsed into an Authorization header, and every URL the browser can see — /api/config, a source's error string, the event feed — goes through config.strip_userinfo first, because an unreachable host writes its failure message into the event table and /api/events serves it.

What the session covers

The gate is default-deny: every page and every /api route requires a signed-in admin except four entries, named next to the middleware they guard in app.py:

Public	Why it has to be
/login	the form itself
POST /api/auth/login	you cannot be signed in to sign in
/api/health	the Containerfile HEALTHCHECK and ./scripts/dev-server.sh status poll it unsigned, and it publishes no fleet data
/static/…	the login page is styled by the same stylesheet as the dashboards

Adding a path to that list is a security decision, and a route a future phase forgets fails tests/test_auth_api.py::test_every_route_that_is_not_public_requires_a_session, which walks the served route table rather than a list written by hand.

What a refusal looks like depends on who asked, so that neither caller has to guess:

curl -si http://127.0.0.1:8090/system | head -3
# HTTP/1.1 303 See Other        → Location: /login?next=/system  (a browser ends up back here)
curl -s http://127.0.0.1:8090/api/summary
# {"ok":false,"detail":"sign in required","login":"/login"}   (a script never gets HTML)
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8090/api/health
# 200

The ?next= value keeps the query string too (/llm?range=1h survives the round trip), and it is judged before it is ever followed: a next that leaves the site (//evil.example, http://evil.example, the percent-encoded and control-character spellings alike) is dropped to /. auth.safe_next is the one function that decides both halves. And every answer the gate produces — 303, 401, 403, 429, the route's own 404 — carries no-store, because a cached redirect would park a browser on the login page after it had already signed in.

The sign-in page

/login is one card with one password field — no username, because there is one admin and a field that is always the same word teaches you that half the form is theatre. It carries the masthead's brand link and the theme toggle and nothing else from it: no tabs (each would bounce straight back here) and no status dots, because there is nothing signed in for them to report.

What it does with an answer is the part worth knowing:

Response	On screen
200	straight to ?next=, or to /
401	Wrong password. under the field, and the text you typed is selected
429	the refusal plus the lockout counting down, button disabled to zero
other	the server's own detail, printed verbatim

The countdown is the server's Retry-After, not a guess, and a refusal is written as text rather than HTML — this is the one page a stranger reaches first. A browser that already holds a session never sees the form: the server 303s it to the dashboard, and the script checks /api/auth/status first so the form does not even flash. ?next= is followed only when it is an in-app path (/ups, not //evil.example), judged on the server and again in the page.

The account page

/settings — reached by ⚙ Account, which sits in the masthead of every signed-in page including this one — is the session facts and the one form that changes the password.

Panel	What it says, and what the button does
Session	the signed-in name, the expiry as a local timestamp and as time left, and Sign out (POST /api/auth/logout, then /login)
Change password	current + new + confirm, then POST /api/auth/password; the outcome is one line under the form

The form checks what it can check itself (a current password typed, at least 8 new characters, the two new boxes agreeing) so a typo is answered without a round trip, and then reports whatever the server said: 403 is the current password is wrong, 422 names the rule the new password broke, 429 is the lockout with its own seconds. The page also states the precedence rule in its own words — the env var seeded the password once, a change here wins, resetting from the environment means deleting auth.verifier and restarting — because that is the sentence that prevents a support call about an edited .env that did nothing.

One detail makes the page more than a form. A change rotates the session-signing secret, which is what signs out every other browser, and it invalidates the CSRF token this browser cached — so Dash.resetAuth() clears that cache after the change (and after any 401 bounce). Without it the operator changes the password, then every UPS button 403s until they reload.

Sign out deletes this browser's cookie; it cannot revoke a copy of the token. A session is signed rather than stored, so nothing is left to mark as dead — the one thing that reaches every browser at once is the signing key, and that is what a password change rotates. If a machine is lost or a session is suspected, change the password here: that is the revocation, and it is why the same page carries both buttons.

Writes need the CSRF token

A write authorised by the cookie must carry X-CSRF-Token, validated against that same session (GET /api/auth/status is where a page that is already signed in reads it — the login and change-password responses carry it too, minted for the session they just created — and it sits behind the session because a token is worthless to anyone who cannot read that response). Dash.fetchJSON fetches it once per browser and attaches it to every non-GET/HEAD request, so page code does not think about it — and POST /api/auth/logout is gated like any other write, because an <a href="/logout"> planted on a page you read mid-outage is a CSRF that signs you out. Mismatched Origin or a Sec-Fetch-Site that is not same-origin/none is refused before the token is even looked at, so a cross-site login post cannot spend your throttle either.

Changing the password

POST /api/auth/password with {"current": "…", "new": "…"} — a protected write, so it needs the session and the CSRF token like everything else:

curl -s -X POST -H 'content-type: application/json' -H "Cookie: dash_session=$SESSION" \
     -H "X-CSRF-Token: $CSRF" -d '{"current":"…","new":"…"}' \
     http://127.0.0.1:8090/api/auth/password
# {"ok":true,"changed":true,"hint":"other browsers are now signed out",
#  "expires_at":1794982214.0,"csrf":"…"}

Nothing forces a change: no expiry, no first-login interstitial, and DASH_ADMIN_PASSWORD is never re-applied after the first boot of a database. The only composition rule is length — at least api.MIN_PASSWORD_LENGTH (8) characters, not empty once stripped, and different from the current one — each refused with a 422 that names the rule. A symbol/digit mandate produces sticky notes rather than passwords, and the cost of a guess here is PBKDF2 plus the backoff, not the character classes.

A wrong current is 403 {"ok":false,"detail":"current password is wrong"} and a password-denied event, and guesses at that field are throttled on their own key (password:<address>) — the same arithmetic as the login form, a separate bucket, so this form is never an unthrottled oracle for the old password once a cookie is stolen and one form's lockout does not lock out the other.

A right one rewrites auth.verifier and auth.session_secret, which is the whole revocation story: a session is a signed cookie with no server-side record, so rotating the signing key is the only way a password change reaches the browsers that are not in front of the form (without it a stolen cookie would stay valid for another DASH_SESSION_HOURS). The response carries a fresh cookie and a fresh CSRF token minted from the new secret, so the caller who just changed the password is not signed out by their own form, while every other browser gets a 401 on its next request and signs back in with the new password. password-changed lands in the event feed — an event, not an overview alert, and not a field on /api/health (which is public) or /api/config.

Scripting the API

Authorization: Bearer <session token> is the scripted caller, and it needs no CSRF token — another site can make your browser send its cookie but cannot read a token out of your shell. The session token is in the login response's Set-Cookie, not in its body:

SESSION=$(curl -si -X POST -H 'content-type: application/json' \
              -d '{"password":"…"}' http://127.0.0.1:8090/api/auth/login \
          | tr -d '\r' | sed -n 's/^set-cookie: dash_session=\([^;]*\).*/\1/p' | head -1)
curl -s -H "Authorization: Bearer $SESSION" http://127.0.0.1:8090/api/summary   # 200
curl -s -H "Authorization: Bearer $SESSION" -X POST -H 'content-type: application/json' \
     -d '{"ups":"apc","command":"test.panel.start"}' http://127.0.0.1:8090/api/ups/command

That is the UPS half of the app's two write endpoints that dial anything (the /api/auth/* and /api/certs/* writes change only this process and its SQLite meta table — invariant 1 is about the fleet; the other half is the /gateway route switch, GATEWAY_ALLOW_WRITES-gated), and the bearer header changes nothing about its gates: a command that can cut output still needs "confirm":"DANGER", and is still refused while the UPS is on battery.

Lockout

Five wrong passwords from one address starts a backoff (429 with Retry-After, exponential from 60 s, capped at an hour; a good sign-in clears it, and a restart resets the counters). The change-password form keeps its own counter under password:<address>, so hammering one form cannot lock you out of the other and neither form is an unthrottled oracle for the same secret. The refusal is the same sentence for every kind of failure — one account, so the response must not rank reasons — and each refused attempt writes a login-denied or password-denied event you can read at /api/events?source=auth. Neither ever becomes an overview alert: a password guess is not an outage, and a page that blinks for both trains you to ignore the ones that matter.

Configuration

Everything comes from the environment (see .env.example for the annotated list). At startup a .env next to the checkout is loaded first if it exists — as a seed: a variable the container, systemd unit, or your shell already set always wins. Point DASH_ENV_FILE somewhere else to keep several installs on one box. .env.example is documentation and is never loaded, and scripts/dev-server.sh exports .env.dev itself (see the quickstart) because dev settings must not leak into a production process.

Variable	Meaning
HOST_TARGETS	name=url per host running the agent. The shape a new host gets is https://host:9120 with no userinfo — a certificate-authenticated agent, verified against this app's own root CA and presented with its client certificate, with a revoked host refused before the dial (certs.py, collectors.AgentTls). The compatibility shape is a token-mode agent's URL userinfo (http://TOKEN@host:9120), moved into an Authorization header by parse_targets so it never reaches /api/config or an error message
GUFO_BASE_URL / VLLM_BASE_URL / RADIANCE_BASE_URL	LLM engines; unset means not deployed here, and its half of the page disappears
RATE_WINDOW_SECONDS	the host's Δ window (default 10) — bytes/sec and jiffies→CPU %, measured over what a 10 s poll can fill; each value carries the window it actually got, and /system's footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one n/a instead of a spike, and a rate 8× above the last one is held until a window measured from a different sample confirms it (phase 13). The LLM rates do not use it: they are measured over LLM_INTERVAL_SECONDS (phase 23)
LLM_INTERVAL_SECONDS	the /llm poll cadence (default 2 s) and the Δ window behind every token rate — one estimator per measuring cadence, so a slot that starts reads steady within two polls instead of ramping over five. Slowing the poll slows the window with it; there is no separate LLM window knob, and a window narrower than the interval would answer n/a every poll
DASH_HOST / DASH_PORT	bind address for the dashboard itself (default 127.0.0.1:8090; the container sets 0.0.0.0)
HOST_INTERVAL_SECONDS / NUT_INTERVAL_SECONDS	how often the host agents and upsd are polled (10 / 5 s). GATEWAY_INTERVAL_SECONDS (30) and GATEWAY_TIMEOUT_SECONDS (10) are the gateway source's own pair: a connection count is the shape of half a minute, and the fetch is the plain agent one with headroom for a handler that may be mid-smartctl pass
DISKS_INTERVAL_SECONDS	how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the disks:<host> jobs exist exactly when HOST_TARGETS does
HOST_TIMEOUT_SECONDS / LLM_TIMEOUT_SECONDS / NUT_TIMEOUT_SECONDS	per-source fetch timeout — one slow source never holds up the others
DISKS_TIMEOUT_SECONDS	the disks fetch's own budget (default 30 s), because the agent runs smartctl inside its snapshot handler and a wedged SMART pass must not read as a dead host
GATEWAY_HOST	which HOST_TARGETS name is the ingress the /gateway page watches (default gateway). It has no URL of its own — the source reads the same agent /snapshot the system page reads, projected onto agent 1.2.0's haproxy/nginx/sockets sections — so a name that is not in HOST_TARGETS means "no ingress to watch", and the source is unconfigured rather than red. The page refetches itself every 15 s in the browser and has no POLL_MS_* knob, because the payload only moves when that one job re-reads the ingress
GUFO_API_KEY / VLLM_API_KEY / RADIANCE_API_KEY	only if an engine is behind a bearer token; they never leave the server
NUT_HOST / NUT_PORT	upsd address — whatever port it really listens on, not always 3493. Reads are unauthenticated in stock NUT
NUT_USER / NUT_PASSWORD	only needed for instant commands (battery tests)
NUT_ALLOW_ACTIONS	1 enables the UPS write endpoint that dials anything — the /api/auth/* and /api/certs/* writes touch only this process and its meta table
GATEWAY_ALLOW_WRITES	1 enables the second write endpoint, the /gateway route switch (POST /api/gateway/route); off by default, the same truthy rule as NUT_ALLOW_ACTIONS, and the two writes are two decisions — one flag does not carry the other
GATEWAY_WRITE_TIMEOUT_SECONDS	the route write's own budget (default 35 s), deliberately longer than the agent's 30 s peer timeout so the dashboard never times out first and leaves you guessing whether the reload happened; the read's 10 s stays the read's own
NUT_SIMULATE	1 replaces upsd with a labelled simulator (development only)
DASH_READ_ONLY	1 disables every mutating endpoint regardless of other flags
DASH_DB	SQLite path (default data/dashboards.sqlite3)
DASH_ADMIN_PASSWORD	the admin password seed, needed only on the first boot of a database — see Signing in above
DASH_SESSION_HOURS	session lifetime (default 720 = 30 days)
DASH_COOKIE_SECURE	1 marks the session cookie Secure; off because dev is http://127.0.0.1, and a request that arrived over X-Forwarded-Proto: https is treated as secure either way
DASH_PBKDF2_ITERATIONS	verifier cost (default 600000). The knob exists for the test suite; leave it alone in a deployment. Raising it on a real box does take effect: the next successful sign-in re-hashes a cheaper stored verifier and keeps the signing key
POLL_MS_SYSTEM / POLL_MS_LLM / POLL_MS_UPS	how often the browser refetches each page (10 s / 2 s / 5 s) — shown in the page footer. /disks and /gateway have no POLL_MS_* of their own and poll at a fixed 30 s / 15 s: the first payload only moves when the disks job re-reads the agents, the second when the one gateway job re-reads the ingress. A backgrounded tab idles its polls and fires one on return; nothing an operator presses stops them (phase 21)
DASH_RELOAD	1 makes uv run dashboards watch the source and restart on a change — development only, and every poll loop restarts with it
DEBUGPY / DEBUGPY_PORT	DEBUGPY=1 opens a debugpy listener (default 5678) without blocking startup
RETAIN_RAW_HOURS / RETAIN_MINUTE_DAYS / RETAIN_HOUR_DAYS	history rollup and pruning

A source that is not configured answers 404 {"configured": false} without dialling, and the page says "not configured" instead of showing a red panel for something you never installed.

Architecture

agent/host_agent.py (each host)      gufo / vLLM / radiance  upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        │  (three readers: system 10s,     │ + /slots (gufo)        │
        │   disks 5m, gateway 30s)         │ + /stats (radiance)    │
        ▼                                  ▼                        ▼
src/dashboards/collectors.py ────────────────────────────────────────────┐
        │  parse + Δcounter/Δt (rates.py)                                │
        ▼                                                                ▼
sources/host.py · disks.py · gateway.py · llm.py · nut.py   poller.py (one task per source)
        │        samples                    events                      │
        ▼                                                                 ▼
store.py  ── SQLite (WAL): sample(source,host,metric,res,ts) + event + meta ──  api.py  → web/
                                                    ▲
auth.py ── verifier + session key, loaded once at startup (no row and no seed → no server)
certs.py ─ root CA + client cert, minted once at startup (unreadable row → no server)

• poller.py — every source gets its own asyncio task, its own health record, and its own error handling. A dead host keeps its last good values and is labelled stale; it never blanks or blocks another source. Disk health is a source of its own (disks:<host> in /api/health, 5-minute cadence, its own fetch timeout) reading the same agent /snapshot the system page reads — a separate task is what keeps a SMART poll out of the 10-second utilisation poll and lets drive history go stale without blinking /system, and alerts.py is why one dead agent is still one line on the overview rather than two. The gateway page's job (phase 29) is the third reader of that same /snapshot, and one job for the whole deployment rather than one per host — GATEWAY_HOST names the single ingress, and a job per host would ask every machine a question only one of them can answer — on its own 30 s cadence with no RateEstimator, because a connection count is not a rate. And it never quietly stops: the task is that source's only poller, so a SQLite write that gets refused is reported (ok: false, the real reason, the numbers aging, one line in the event log) instead of freezing the source under a green dot.
• store.py — raw samples roll up to 1-minute and 1-hour averages and the raw rows are pruned, so a year of history is a few hundred MB rather than a disk emergency. Nothing a per-request path asks is answered by a whole-table scan on every call (phase 14): the two answers that used to be scanned on every call — series_names() behind /api/metrics and the row counts behind /api/health — come from a 60 s / 30 s cache, so the scan is paid once per window (its cost is the honest remainder in the known-gaps list) instead of once per request. series_names' is ended only by a write that can change it: a (source, host, metric) the store has never written, or any DELETE. A poll of an existing series leaves it alone, which is what makes the cache serve at all on a store every poller writes to (task 04's live watch is the proof: 535 ms per call before, 1.8 ms median after), and the counts age on the clock. A chart's resolution comes from one capped compound statement instead of three full COUNTs per series. last_maintain and size_bytes are deliberately never cached — the first is how an operator tells a stalled rollup loop from a healthy one. The write side is windowed too: maintain() rolls up only the buckets that became due since its last run (read from last_maintain, overlapped by one bucket and snapped down to a bucket boundary, so a gap in polling is a catch-up scan rather than missing charts) and prunes on its full retention bound, one statement per resolution. It ends with PRAGMA optimize — the store's only statistics refresh, deliberately not inside a poller's commit — and the WAL is kept honest by wal_autocheckpoint plus a 64 MB journal_size_limit cap. On a 6.4 M-row store that took the once-a-minute write transaction from 5.4 s to a few milliseconds, which is what stopped a poller's insert losing its 5 s busy_timeout (known gap 4). Its meta table is the one place state that must outlive a restart goes (get_meta/ set_meta/delete_meta). Its users are the admin credential below and, since phase 15, the certificate authority in certs.py — five certs.* rows and nothing else.
• auth.py — the admin login's decisions, in stdlib and without HTTP: a self-describing PBKDF2-HMAC-SHA256 verifier, an HMAC-signed stateless session cookie (no session table, and rotating the stored key is what evicts every other browser), a session-bound CSRF token, and an in-process login backoff. load_auth_state() is the startup half: the environment seeds a password once, the stored verifier wins from then on, and a server with neither refuses to start (see Signing in).
• certs.py — the fleet's certificate authority, in the same shape as auth.py: a pure lifecycle over the meta table, no clock it cannot be handed, no logging, and one fixed-text raise. The first boot of a database mints a root CA (certs.root.key/.cert, self-signed CN=dashboards-root, 10 years) and the dashboard's own client certificate (certs.client.key/.cert, CN=dashboards, 1 year); every later boot reads those rows and regenerates nothing, because a root that changes on restart untrusts every certificate already installed on a host. issue_agent_cert/reissue_agent_cert sign one server certificate per HOST_TARGETS name (CN = the host, SAN = the addresses it is dialled by) into certs.agents, and each entry's bundle — the agent's cert, the agent's key, the root cert (never the root key), a # BUNDLE-REV <n> line and the current # REVOKED <serial> lines — is stored verbatim so the bytes an installer places are the bytes the row holds. revoke marks the host and re-seals every bundle, because the denylist lives inside each of them; rotate_root replaces the root, the client, and every agent at once. A stored PEM that will not parse (or a client leaf the stored root did not sign) is a fail-closed startup, the admin-password rule: the message is sqlite3 "<db>" "DELETE FROM meta WHERE key LIKE 'certs.%'", after which every agent needs a fresh bundle. Every certificate carries a SubjectKeyIdentifier and every leaf an AuthorityKeyIdentifier naming that root, because OpenSSL's strict mode — the one Python's ssl.create_default_context() turns on, which is what the dial below builds — refuses a chain without them. The dial is collectors.AgentTls (task 03): with an https:// HOST_TARGETS entry, the fetch verifies the agent against this root and presents this dashboard's own client certificate, so only this dashboard can read the host; a host marked revoked fails its host: and disks: jobs with agent certificate revoked before any socket opens, and because the status is a fresh certs.agents read on every poll, revoking takes effect on the next poll with no restart. An http:// entry is untouched — same shared client, same bearer token — so the rollout is per host. The private keys are credentials like any password: never served, never logged, and the root key never leaves the row that holds it.
• severity.py — the single green/amber/red table (THRESHOLDS, plus INVERTED for the metrics where lower is worse). Rated values in the API payloads travel with a sibling severity (cpu.severity.usage_pct, memory.severity, disks.severity, a UPS card's severity.charge_pct, severity inside the rated LLM cells such as KV pressure, /api/summary rows), computed server-side, so every page colours the same way and a value nobody reported renders n/a instead of green. A metric published as a 0-1 ratio is rescaled for the verdict only — the number printed stays what the upstream published. There is no gpu.* row: these are LLM servers, a busy GPU is the job, so the GPU card (and the fleet GPU column) renders uncoloured at every utilisation — the gpu.* history series are still stored and charted. Drive health gets two rows: disk.temp_c (amber 50 °C, red 70 °C) and disk.percentage_used (amber 80 %, red 100 % — an SSD that has spent its rated wear is a replacement). The sector counters (disk.reallocated, disk.pending, disk.uncorrectable) and disk.power_on_hours have no row on purpose: a non-zero counter is an alarm chip on presence, zero is the absence of a chip and never a green "healthy drive", and an odometer has no bad value.
• alerts.py — "what is wrong right now", as data. alerts.build(settings, job_states, host_payloads, engine_payloads, ups_payload, recent_events, *, gateway_payload=None) is a pure projection of the state the poller already keeps plus the verdicts severity.py and the UPS health checks already reached — no I/O, no thresholds of its own — and /api/summary returns it as alerts, worst first: {id, severity, source, message, href, at}. A configured source that stopped answering is fail; one that is stale but still serving its last good value is warn; an unconfigured source never alerts; a dead source is one line rather than one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a marker that says how many it hid. A job that dials an agent another job already named adds no line either (AGENT_ECHO_DASHBOARDS: disks and, since phase 29, gateway — three jobs read one /snapshot, so one dead box is one overview row), while a job failing while its host answers keeps its own voice and links to the page that explains it (/disks for a wedged SMART pass, /gateway for a proxy or socket section that went quiet). Staleness is measured in each source's own interval, so the gateway page goes amber after 90 quiet seconds and not after 30. The keyword-only gateway_payload (phase 30 task 03) is the one non-reachability voice this module has for that source: a non-empty warnings list — the two config files not connecting to themselves — is one warn row (N proxy config problem(s) to address — <first message>, href /gateway), because a dangling backend is a structural fact and not a graded number, and it is emitted only while the job is configured, ok and not stale, so a router that has stopped answering keeps its single reachability line instead of growing a second one about a reading the operator cannot refresh. No count of connections can produce a row here at all.
• api.py — the JSON every page reads: /api/config, /api/health, /api/events, /api/history, /api/metrics, /api/summary, one snapshot + history pair per dashboard (/api/system/*, /api/disks/*, /api/gateway/*, /api/llm/*, /api/ups/* — including /api/ups/commands, which answers what may be sent to each UPS right now and why not when it may not), and the certificate authority's own surface (/api/certs, /api/certs/<name>/bundle, POST …/rotate, POST …/revoke, POST /api/certs/root/rotate). GETs everywhere except POST /api/ups/command, which stays the only route that dials anything: the /api/certs/* writes are meta row writes through certs.py, the same class of write POST /api/auth/password is, and a revocation's whole effect is that the next poll refuses to dial (see certs.py below). Reads there are pure — certs.read_state is the request-time read and certs.ensure_certs the startup bootstrap — so refreshing the list cannot mint a certificate, and it reports the stored authority even after a root rotation the running process has not picked up yet. Two rules make the pages legible: a source with no env config answers 404 {"configured": false} without dialling (so a page says "not configured" instead of going red), and a history query for a series that does not exist answers an empty series instead of an error (so a card asks for every plot it could draw and reads n/a for the ones the drive never reported). The five history routes take one batched door into the store (api._history_batch → store.history_many), so a page's chart request is one grouped resolution query plus one index seek per series rather than four queries per series — and the JSON it answers with is the same document the per-series loop produced, series for series and in the same order (tests/test_api.py pins that against points read straight out of the store with query()). /api/disks/snapshot is the drive-health surface: one entry per host with drives to show — its physical disk rows, the agent's smart block verbatim, smartctl_available (true / false = install smartmontools / null = the agent never said), a severity beside each row's rated numbers, and a fleet block summed here so the page does no arithmetic — a host that has not answered yet is listed as an unreachable stub of n/as and counted, never summed. The one host that is not listed is the one whose agent said false: a box with no smartmontools (a VM with nothing behind vda) has no card, no fleet contribution and no /api/summary row, because the only thing its card could say is "install it". null and an unreachable stub both stay — an agent that cannot say is not an agent that said no (api._disk_visible is the single predicate, pinned both ways in tests/test_api.py; the disks:<host> job, its health record and its stored disk.* series still see the host). /api/config carries disks.interval_seconds, which is why a drive card four minutes old is behaving, not broken. /api/summary grew a disks block with that page (the same fleet sums plus one row per host — counts, smartctl_available, and failed_devices, so a notifier learns which drive the hardware called FAILED without a second call); it is additive, exactly as alerts was, and the key paths of both are pinned with and without targets configured. Since phase 22 that block lists the same hosts /api/disks/snapshot does, and fleet.hosts_total is that row count, so / and /disks can never describe two different fleets. /api/summary grew a gateway block the same way in phase 29, reshaped by phase 30 task 02 (configured, host, href, supported, unreachable, listeners, connections, live_in, live_out, map_targets, states) and given warnings by task 03: what the configs declare (listeners, connections, map_targets, warnings) beside what the sockets are doing (live_in, live_out — the latter summed over distinct endpoints, since several routes reaching one address is one count and not two). null and never 0 in every arm that did not measure — except warnings, which is a config fact and so reads a real 0 when the two files connect to themselves. No severity anywhere: a connection count is not a graded number, and a structural fault needs no threshold because the hardware of a config file either names a thing that exists or it does not. The unmatched / unmatched_listeners / partial fields went out with the guessing rule they existed to explain. /api/config carries the matching row, including gateway.interval_seconds/timeout_seconds and the named target's redacted url. /api/gateway/* (phase 29 task 03) is that page's pair. /snapshot is one host's whole topology — the haproxy frontends and the nginx listen/map echo the pipes were built from, the listeners (port, its domains, its live connection count, up to 32 client addresses), the connections graph (phase 30: one row per route the two config files declare, each naming its inbound port, its outbound endpoint, and the config's own selector for it — a haproxy backend name, an nginx server_name, a stream map key) with its live count — the socket table's ESTABLISHED rows, inbound by the socket's local port and outbound where a remote (address, port) equals an endpoint the join can name — a target the config wrote as an address, or (phase 32 task 03) one of the addresses the gateway's own resolver section published for the name it wrote. Each row carries the metric it is stored under, and shared: true when more than one route reaches its endpoint (one count, shown twice, never split). supported/sections/why cover the degradation arms. warnings (phase 30 task 03) is the list of the two config files not connecting to themselves — seven kinds, each one row of {id, kind, subject, message} where id is "<kind>:<subject>" and the message is the sentence the page prints: haproxy_route_dangling, haproxy_backend_orphan, haproxy_backend_empty, nginx_server_unlistened, nginx_pool_undeclared, nginx_variable_unmapped, nginx_map_route_missing. Nothing is inferred to fill a gap, so three shapes are deliberately silent: a vhost that listens and passes nothing (a return 301 looks identical here), a frontend that routes by a fetch expression (agent 1.2.0 reports "no name", which is the same value "no routing directive" has), and an entry of a map no proxy_pass names (the $connection_upgrade map's upgrade/close are header words, not missing routes). An unreadable or absent section warns about nothing — a host that cannot be read has not been found guilty of anything — and a warnings: [] is therefore the truth, not a shrug. Non-zero, / gets one amber row (N proxy config problems to address — <first message>) and this page gets the list. Nothing reconciles to sockets.established and that is the design: a connection to a target nothing can name (a name with no resolver entry behind it) or a socket on a port no config names is ignored — not counted, not totalled, not warned about — so a pipe count means "this route has N live connections" and nothing more. The payload also carries the agent's resolver block (available, via, and the projected entries: name, port, status, addresses, ts — nothing an operator has not reviewed) beside the sections pair that already says which halves answered, so a dash on one line is explainable from the document the page already holds. On the recorded ingress that section places every declared route, which makes the dash the exception rather than the normal look of this page, and its meaning is this name has no record — and on the live fleet it is the exception too: the gateway has run agent 1.3.0 since 2026-10-25, resolver.available answers true with one entry per declared target, and that host's page counted 33 of 33 routes with no dash at all. GATEWAY_HOST naming no HOST_TARGETS entry is a 404 {"configured": false} answered before any client is built (both arms spy-pinned), a host that has never answered is the unreachable stub whose every count is null, and a 1.1.0 agent is supported: false plus its one why sentence — three arms, no red among them (invariant 3). /history serves the gateway.in_conns@<port> and gateway.out_conns@<host>:<port> gauges — the port in the outbound name because two services on one host are two facts, and only ever for an endpoint whose address is known: one the config wrote, or (since phase 32 task 03) one the gateway resolved the name to — with hosts= defaulting to the ingress alone, unknown names answering as empty series, and no dial either way because history reads only SQLite.
• web/ — six static dashboard pages (/, /system, /disks, /llm, /ups, /gateway) plus the three admin pages (/login, /settings, /certs), one stylesheet, hand-rolled SVG charts (charts.js), and a masthead status dot per dashboard job on every signed-in page — the two admin pages included (disks has its own: the same agent can answer the 10-second utilisation poll and have a wedged SMART pass, and gateway has one because its job is its own poll of one ingress). The masthead is three rows at every width, the two control rows flush right: the brand and the six dashboard tabs, then the per-source indicators on their own row, then the ⚷ Certs / ⚙ Account / ☾ Dark / clock controls on a row under those, in that order, so the time is the farthest-right thing on the bar (phase 25: the badges and the cluster were re-justified right, then the clock moved to the end of the markup — flex-end makes the last child the rightmost item, and it stays a markup fact rather than a CSS order: so the pin can read it; the clock is hidden below 900 px as it has always been, being a readout and not a control). Only /login renders no indicators, and so two rows, and a control is never hidden to make a narrower width work. There is no control that stops the polling (phase 21): the bar used to carry a ⏸ Pause button with a Space shortcut, and it left the numbers on screen looking current with no stale label on them. Both admin links are on all eight signed-in pages — the six dashboards plus /settings and /certs, the two they name included — and not on /login, the ninth page and the only anonymous one: the cluster is the same wherever you are, and a link to the page you are already on is a harmless no-op rather than chrome that vanishes. The bar is not sticky: it scrolls away with the page, so every pixel of height goes to the data (it stays a positioned element, which is what keeps the mobile dropdown anchored under it). At ≤640px the six tab links leave the bar and come back as a dropdown under the whole of it, opened by the ☰ button (/login renders no tabs, so it has no button either), and that menu slides: 180 ms of fade plus an 8 px rise on the way out and the same on the way back in, with nothing hidden from a keyboard or a screen reader while it is shut, and no motion at all when the system asks for none (prefers-reduced-motion). The same ≤640px block is what makes every table scroll sideways inside its own card (phase 27): table.data becomes a horizontal scroll container, so the columns past a phone's 320 px — Worst disk, GPU busy, Temp, Up on the fleet table — are reachable by dragging the table instead of being clipped by the card, while the document itself still never scrolls sideways and a desktop table that fits shows no scrollbar. Two more things were clipped by that same card edge and are fixed at every width: a card header wraps onto a second line when its facts do not fit on one (the OS/model chip and the updated 5s age stamp used to be painted past the card and eaten), and a collapsed Charts / More detail disclosure scrolls whatever it reveals sideways, which is what puts /system's GPU VRAM/GTT columns inside their card instead of under the neighbouring one. Nothing is hidden for the arithmetic: the row wraps, the disclosure scrolls, and every number stays on screen. No framework, no CDN, no bundler, and no narration: a page never explains its own pipeline, so the words on screen are the ones you act on — a unit, a Δ<window> provenance tag, a SIMULATED badge, or the sentence beside an empty panel. The overview (/) is exactly two panels — a Warnings & errors list built from /api/summary's alerts (each row links to the dashboard that explains it) and the Fleet table — and nothing else: an empty alert list reads as an explicit "all clear", and a live problem moves the panel above the table. A host that is not answering keeps its row — dimmed, every number n/a — so the fleet never quietly gets smaller and last good values are never mistaken for live ones. Under that table is the ingress's one line (gateway · 10 listeners · 33 connections · 17 live in · 2 live out as the live router answered on 2026-10-23 — a reading, the same way a listener count is, and only the field names behind it are a contract), linked to /gateway: every figure is /api/summary's own, a count the source did not measure prints n/a rather than 0, and a deployment with no GATEWAY_HOST renders no line at all. A non-zero warnings appends · N to address and nothing else — 0 and null both print nothing, so a clean ingress never grows a permanent sentence about the checks that did not fire (phase 30 task 03). A big colour-coded number is Dash.hero(el, {...}) on the .hero / .hero-bar styles, coloured by the severity in the payload via Dash.severityFrom() — the only colour vocabulary is ok / warn / fail, and a value with no verdict renders n/a with no colour at all. A battery is DashCharts.battery(el, {charge, severity, state, label}): one inline SVG whose fill is the charge — clamped to 0–100, never rounded up, clipped to the plate so 0 % is an empty shell — coloured only by the payload verdict, with a shape per state (solid shell on line, bolt charging, dashed outline on battery, a dash through the body when there is no reading) and a <title> / aria-label that names it ("apc: Charge 100 percent, on line") unless the number beside it already does. Each /system host is one panel led by six hero slots — CPU %, MEM %, TEMP, DISK % (mount underneath), GPU busy, and NET on two lines (↓ in / ↑ out, never coloured: a byte rate has no good/bad). A host with no GPU renders five: the GPU slot is omitted rather than filled with n/a, because a router is not broken for the card it never had. A GPU is one busy hero plus four labelled values under it — Model, VRAM (used / total), Pool (the honest APU view) and Temp — instead of the dense model · VRAM · pool · °C footnote that was 11px mono; a field the agent did not publish prints n/a in its own cell, never 0 °C, and nothing in the slot is coloured or barred (a busy GPU is the job on an LLM server). Missing numbers still print n/a uncoloured — gateway's CPU temp — which is a different fact from a missing device. Under the numbers, each panel charts only its own host — behind a collapsed Charts toggle, because the default view is current stats and a fleet of ten hosts each drawing three plots is noise: the summary reads ▸ Charts with the window in it (1h, 24h …), and opening it is what builds and paints the charts — instantly, from the response the last refresh already fetched, and repainted on every refresh after that while it stays open. Expanding one panel never stretches its collapsed siblings (.grid.items-start). The stack is utilisation (CPU, memory, worst disk), network (↓ in / ↑ out), and GPU busy for a host that reports cards — three 120px SVG charts, each with its own legend and its own aria-label naming the host and the plotted metrics (LineChart's ariaLabel option), never a second host in sight. The fleet-wide charts, the nine-figure fleet-totals panel and the host/metric detail picker are gone, and so are the per-core charts, while cpu.usage_pct@coreN stays in SQLite. Everything else — load, clock, disk I/O, uptime, GPU power and the per-label temp breakdown, filesystems, processes — folds into a collapsed "More detail" disclosure. Each panel pulls its series in one /api/system/history request (hosts=<that host>&metrics=…&range=…) every refresh, stack open or not, and a host that stops reporting takes its panel, its Charts toggle and its charts off the page. /disks is the drive-health page: one card per physical drive (the agent's whole-/sys/block list — never a partition, never a mount), led by the device path and the drive's own verdict word (OK / FAILED / unknown, coloured only by severity.health), then three equal cells — Temp, Power-on hours, Wear — coloured by disk.temp_c and disk.percentage_used from the API, with the odometer deliberately uncoloured because severity.py rates no power-on hours. The sector counters appear as red chips that exist only while non-zero, so a healthy drive shows no alarm row at all rather than a wall of zeroes. A host whose SMART cannot be read keeps its cards, dimmed with every number n/a and one .banner line saying why — the agent's own sentence when it reported something, and "not answering" when the agent is down — never a red panel, because unreadable drives are unconfigured, not broken. The one case that gets no card at all is the agent that said smartctl is not installed: such a host is absent from the page's host cards (and from the API's host lists) rather than a card whose only content is "install smartmontools" — its disks:<host> job still polls, still reports ok, and its transitions still reach the drive events feed below the cards, because that feed is the job's history and not a host list. The header's drive counts are the API's fleet block (hidden below two hosts, so one box's sums never duplicate its own cards), and each host's Charts disclosure behaves exactly like /system's: collapsed, built only while open, and fed by one /api/disks/history request per host per refresh naming disk.temp_c@dev and disk.percentage_used@dev for every drive it has. /llm watches up to three engines, each configured by its own *_BASE_URL (unset means not deployed here, and its card and panel disappear — never a dead half): gufo (a llama.cpp-family server — /metrics plus the per-slot /slots JSON), vLLM (/metrics only, which is why its panel says exposes no per-slot JSON), and radiance — a vLLM-derived engine (this fleet's driveripper:8000) that answers /metrics (its radiance: native windowed gauges win the throughput cells when present, falling back to the measured Δ<window> of its vLLM-shaped counters when absent), /stats — its answer to /slots: fetched best-effort beside the /metrics poll, so a dead sidecar degrades the session arm and the queue cap to n/a, never 0, and never fails the poll — and /v1/models. /llm opens with the fleet: one block above the panels — ALL ENGINES, an N of M engines tag, a note that is the API's own fleet.seams sentence (the shipped line, sums only engines answering now, is the fallback for a payload that predates it), then running / waiting / prompt tok/s / decode tok/s / total tokens — and every number on it is fleet_totals(), so the page sums nothing itself. A dead engine keeps its last-good numbers in its own (stale-labelled) panel but is dropped from a total presented as current: the tag counts what answered, names the missing engine in its tooltip and dims while the sums are partial. Nothing on the block is coloured — severity.py rates no throughput and no queue depth, because 0 tok/s on an idle fleet is healthy — the dot is reachability, and fleet.seams (how the engines count prompt tokens and where their throughputs come from) is the note under the count and a tooltip on the prompt sum. Below two configured engines the block is hidden: a cross-engine sum of one engine duplicates that engine's panel. Below the fleet block — with the page's shared .85rem gap, as everywhere else in the section stack — sits the stats strip: one card per configured engine — reachability dot, name, age stamp, the KV-cache bar (the one rated cell: the fill and a breach's text take their colour from the payload's severity and nothing else), then prompt tok/s and decode tok/s with their provenance tag (Δ2.0s at the default cadence — the LLM window is one poll interval, phase 23 — scraped, estimated), total tokens, the sessions this engine reports (slots busy counted from gufo's /slots, or running/waiting for vLLM and radiance, plus waiting), prefix cache hit, spec accept (gufo and radiance — radiance's gauge reuses gufo's key, the same 0-1 scale), queue time with its p95 … foot (vLLM; the dev radiance build publishes no queue-time histogram, so its cell is n/a by absence, never back-filled) — and red alarm chips for preemptions / GPU-context losses that exist only while non-zero. The radiance card adds its own four cells — queue cap (the admission ceiling, n/a while /stats is not answering, never 0), build (the engine's own version string), uptime, and VRAM (card 0) (the engine's own view of the card in GiB = 2³⁰; /system's GPU row is the agent's view of the same card) — and two more things: a 7-bar spec-acceptance curve (one bar per position the engine published, in order, height ∝ the 0-1 ratio, drawn in neutral ink because an efficiency number is not a verdict) and its per-request rows beside the counts (the engine's own fields, as-is — at rest the table is hidden and the API's own count-otherwise sentence takes their place). Then the engine panels hold the detail the strip does not print: two big throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is healthy) with their provenance tag, and the latency rows (TTFT, TPOT — vLLM and radiance; gufo's build does not publish them) each with a p95 … foot line and its own sparkline over the selected window. Under the gufo panel's sessions (GET /slots) label sits the slot grid: one card per parallel slot, keyed by the slot's own id, created once and patched in place so a busy slot never flickers — request id, prompt tokens (with processed … while prefill runs), cache hit and its share of the prompt, generated tokens, remaining, context, and the model beside the slot number. A busy card is a border, a glyph and the word processing, never a green/amber/red verdict (a slot generating tokens is the job). Every cell reads exactly one key of the captured /slots shape — no a ?? b fallback chains — so a field rename fails a contract test instead of printing dashes, and -1 sentinels print idle/n/a, never a negative countdown. When /slots does not answer, the panel says so with the API's own sentence rather than rendering a card of nothing, and vLLM's panel carries one line (exposes no per-slot JSON) and no grid. No row appears in both places — one value, one home — and there is no peak-context cell, because this page prints a number the upstream published or n/a. The four trend charts sit below the panels at half height, and one /api/llm/history request per refresh feeds all of it. With a single engine configured the page is one clean card in the strip and one clean panel — never a red half for a server you do not run. Each UPS card (/ups) opens with its battery glyph and three big numbers — charge %, load %, runtime left — coloured from the card's severity dict, with the electrical readings (line in, output, battery volts and amps, battery °C) kept below them in smaller muted type because they are what you read while the UPS is on battery. Health checks, the battery-test buttons (with their DANGER confirmation) and the NUT variable dump are unchanged, just second. The Last self-test check grades the outcome of the driver's word and not its phrasing: a pass (Done and passed, the sentence this fleet's CyberPower answers) reads ok and raises no overview warning, a fail reads fail, and a pack that has never been tested — No test initiated or no variable at all — says the same "run a battery test" sentence as unknown. Only a word the map has never seen stays a warning, printed verbatim so it can be added. /gateway answers the topology question /system cannot: which ports on the ingress are open, what routes out of each, and how many connections are live on each. It is two panels — one per config file, nginx then haproxy — each hidden until a poll actually has rows for it, so a router with no nginx in it spends no panel saying so. Inside a panel the rows are groups: a trunk is one port that file binds, and hanging off it are the leaves — the routes that port's config names, each reading inbound first: line 1 is the config's own word for who reaches the endpoint (a server_name, a haproxy backend name, a stream map key, the literal default), and line 2 is the counterpart it flows to, exactly as the config wrote it, marked → host:port with the kind of route after it. Both ends are verbatim, and both are carried whole on the row's tooltip and the line's accessible name whichever one CSS has to ellipsise. That is the shape of the file, which is why the stream map is not a third panel: its entries are the nginx panel's :443 leaves, so every SNI domain reads beside the port it reaches. A trunk that names no target — this fleet's :80 return 301 — prints "no target named in the config" instead of drawing nothing. Each line is joined to its counterparts: nginx's ports branch — a dashed spine down the fan-out with a tick into every endpoint — and haproxy's one-server frontends run straight across, because a port with one route behind it is not a branch. A route is green because the config declares it, an existence colour of the page's own, and that green covers every segment of the row — spine, tick, line and switch all read one class on the row, so no part of an active route is drawn as grey plumbing — while the line animates only while it carries at least one live connection: motion is the socket, colour is the config, and a resting route keeps the same green as a busy one. The trunk carries a short line of its own for that reason, and it reads from a different field: a leaf moves on its endpoint's count, the trunk's stub moves on listeners[].connections — the sockets whose local end is that port, which /proc states whether or not the route's target is a resolvable address. So a busy port is visibly busy even when every leaf under it carries a —, which is the normal look of this fleet's :443. That line's claim is about the port and never about one route: its accessible name and tooltip say :4443 — 9 connections are open on this port right now (or no connections…, or connections on this port are unknown: the socket table did not answer) and name no endpoint, and the unknown arm prints n/a where the count is and no motion at all — never 0, and never a dash, because "the socket table went quiet" is a different fact from "this target is a name". Above the map, one filter box finds a domain among 33 routes (stackexpected, brain, 4443, minecraft) because the rows are in config order and the question is always about one name. It hides what the payload already sent and asks for nothing: no request, no re-derivation, no new route. It matches the strings the payload carries — the inbound word(s), the endpoint, the port, the haproxy server name, the addresses the gateway resolved for a name target, and a trunk's own domains and client addresses — case-insensitively, so a label CSS ellipsised still matches its whole text and typing 10.1.2.64 finds the route whose target that address belongs to. A trunk that matches keeps its whole fan-out: typing 4443 asks for the port, and every route hanging off it is the answer. Escape clears it, a chip beside the field counts N of M routes, and a query that matches nothing leaves both panels standing with no route matches "…" in each. What it never touches is the header's four tallies, the warnings panel, or the charts — those describe the ingress, not the operator's current view of it — and it never groups, sorts or colours by domain suffix: reeseapps.com versus stackexpected.com is a fact neither config file states, so the operator types the suffix and the page hides the rest rather than inventing a zone no proxy declared. Nothing on the page is coloured by a verdict — a busy ingress is the job, so severity.py rates no gateway.* metric and no number gets a green/amber/red tint. A line prints —, never 0, and its tooltip and its accessible name say which unknown it is: the gateway could not resolve this name in this reading when nothing is known for the endpoint, or the socket table did not answer this reading when an address is known and nothing was measured — two different things to go and fix. A resolved endpoint with nothing open is neither of them: it prints 0, because that is a measurement. A counted line carries its provenance on hover (… · address from libc, resolved 3m), and a line the config wrote as an address claims no resolution at all. A dash on a domain-named route now means the gateway had no record for that name in this reading (nxdomain/error, or no resolver section at all on a pre-1.3.0 agent) rather than "this page cannot count names": the ingress here runs 1.3.0, so all 33 of its declared routes carry a number. The footer says the same thing once, in plain terms, and points at the hover. The header's four tallies are ports, connections, live and warnings, every one of them the payload's own figure. A warnings count above zero raises a panel above the map with the config-integrity rows (a backend no frontend routes to, a proxy_pass with no upstream behind it, an SNI entry that reaches no address) in the overview's alert vocabulary; the live ingress here is clean, so that panel is off. Each panel's trend is behind a collapsed Charts toggle like /system's, one /api/gateway/history request per panel per refresh. With no GATEWAY_HOST the page is the setup sentence rather than a red panel, and an ingress running an agent older than 1.2.0 says so in the agent's own words with every count n/a and nothing coloured. Each line's head carries the switch — a real role="switch" button, in the on position and tinted --gate-on because the route is in the config, named in full for a screen reader (endpoint, reached by <selector>, in the nginx config, on) — and it has two arms on GATEWAY_ALLOW_WRITES. With the flag off — the default — it is unwired on purpose: aria-disabled with a tooltip that says so, and a route changes only by editing haproxy.cfg or an nginx server block, never in a browser. With GATEWAY_ALLOW_WRITES=1 the same control is live: activation opens one confirm dialog that first dials the helper as a dry run (dry_run: true — it validates and answers with the one-line diff without writing a byte), then asks the operator to type DANGER, then sends exactly once. The helper edits the route in its config file, validates the file with the proxy's own checker and reloads that proxy — nothing is restarted — and the switch flips the moment the helper answers applied: true, with the row re-read from the next gateway poll. The write takes the read's own revocation gate on the same dial, and its gates (read-only, the flag, a per-address throttle, the route's state in the current payload, the read's freshness, the typed DANGER for both verbs) all run before anything reaches the host. What runs today is agent 1.4.0 — name resolution on the gateway, counts on every declared route, the #off marker read, and the switch when the flag is set. The four event feeds (/system, /disks, /llm, /ups) are a paged window over the event table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a page, with 1–50 of 200 between ‹ prev and next › (disabled at the ends) underneath. One helper, Dash.eventsPager(), renders every feed — five lists today, because /ups splits its one fetch in two — and each keeps its own row markup; the fetch asks /api/events for 200 so there is a page 2 to reach, and the endpoint itself is unchanged. Reading page 3 survives a poll; a refresh that brings new events goes back to page 1, because page 3 of a list that just changed is not the history you were reading. The /ups feed is also expandable: its card is a Power events summary line, closed until you open it, and the list inside is the same bounded, paged window. What sits behind that summary is only the power half of the feed — status transitions (the power loss and its return), self-test results, and the commands you sent — because the rest of it is connection trouble (NUT server NutError: cannot reach upsd … and its recovery line), which is not history to tuck away. That error half is its own Connection & errors card above the summary — the same bounded, paged window, the same row shape — hidden while it has nothing to say and appearing with its first unreachable/reachable line, so a poll failure is readable without opening anything. That is also why the feed is fetched on every poll rather than only while the summary is open: half of it is always on screen, and the two lists page independently of each other.

Testing

uv run pytest                                  # unit + API tests, fully mocked, no network
uv run pytest --cov                            # coverage gate
set -a; . ./.env.dev; set +a                 # the live suite reads the same env as the app
uv run pytest -m live --live                   # optional: real fleet / engines / upsd — an
                                               # `https://` HOST_TARGETS entry is dialled with the
                                               # authority in DASH_DB (read-only), so a box with no
                                               # store skips the host tests instead of failing them
uv run pytest -m store_perf --store-perf -s    # optional: the store's hot-path timings (phase 14)
uv run python scripts/store_perf_probe.py      # the same measurements as a report in data/logs/
DASH_WATCH_SECONDS=1500 uv run python scripts/watch-store-perf.py
                                               # ... and the live watch: 25 signed-in minutes against
                                               # the running dev server, counting store-error and WAL
DASH_LLM_WATCH_SECONDS=780 uv run python scripts/watch-llm-window.py
                                               # ... and the LLM Δ-window watch (phase 23): signed-in
                                               # reads of /api/llm/snapshot at the app's own cadence,
                                               # judging every slot start/end against "settled within
                                               # two polls", the Δ<window> labels and the 1 m rollups.
                                               # Read-only: it starts no workload and writes nothing.
DASH_LLM_WATCH_REPLAY=data/logs/llm-window-watch.tsv \
  uv run python scripts/watch-llm-window.py    # ... or re-derive that verdict from a captured pass,
                                               # over the same samples, with no traffic sent
uv run ruff check src tests agent && uv run pyright            # lint + types
./scripts/capture-fixtures.sh                  # re-record ground truth: host agents, UPS replies, gufo /slots
./scripts/capture-fixtures.sh --with-llm        # ... and the engines' /metrics while they are up
                                               # (radiance also records /stats + /v1/models in the same pass)
SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh
                                               # ... and smartctl per physical disk: plain smartctl,
                                               # `sudo -n` if that is denied; scrubs serials/WWNs
./scripts/capture-gateway-config.sh            # ... and the gateway's proxy tree + socket tables:
                                               # haproxy.cfg, nginx.conf + its glob includes,
                                               # /proc/net/tcp{,6} over $GATEWAY_SSH (gateway-ops).
                                               # World-readable paths only; a failed read never
                                               # replaces a good fixture
bash .agents/validate.sh                       # all of the above + `node --check` on every page
                                               # script + repo hygiene, with the 94% coverage floor

Test philosophy: real captures as fixtures, mocks for behaviour. Anything that must match an upstream (/proc layouts, vLLM metric names, NUT protocol framing) is asserted against a verbatim capture in tests/fixtures/; behaviour that needs a failure — upsd refusing a command, an engine returning 500, a host rebooting — is a fake. The live tests (tests/test_live_fleet.py) are the bridge: they SKIP unless you pass --live, each one also skips when its source is unset (source .env.dev first — run bare and they all skip quietly, which reads exactly like a pass), they drive the production code paths (collectors.*, NutMonitor.collect) against whatever hardware is reachable, and they never write — a battery test on live hardware is not a unit test.

tests/auth_contract.py is two constants and the reason for them: every test that starts a server seeds the admin password testpass (a server with no stored verifier and no seed refuses to start, so a test that wants a running app has to give it a credential the way .env.dev does) and hashes it at 1 000 rounds through that same pbkdf2_iterations setting instead of the real 600 000 — ~0.3 s per hash times a hundred-odd servers is not a slower suite, it is a suite nobody runs. The real cost is pinned where it belongs (test_the_default_cost_is_the_owasp_figure and the config default test), so lowering it here cannot lower it there. Its partner is tests/auth_client.py: sign_in(client) posts that password once, asserts the sign-in worked, and puts the returned CSRF token on the client's default headers — which is what keeps the ~200 pre-existing HTTP assertions asserting the behaviour they were written for instead of the new gate.

tests/test_auth_api.py is the perimeter, and it is deliberately awkward in one place: every anonymous request runs with follow_redirects=False, because a client that follows them would fetch the login page and report 200 for a route that refused it — the way a gated suite ends up passing while testing nothing. It also walks the served route table (test_every_route_that_is_not_public_requires_a_session), so a route a later phase forgets to protect fails there rather than in a browser, and it proves the one refusal that matters most on the wire rather than on a status code: a POST /api/ups/command without the CSRF token answers 403 and the scripted upsd records nothing, while the same request with the token still reaches it — and driver.killpower still needs the word DANGER behind the session, because a fifth gate that quietly replaced the fourth would be the worst kind of regression.

tests/test_auth_docs.py tests the paperwork, because the paperwork is what an operator holds when the dashboard will not come up. It reads .env.example and .env.dev.example and demands that every DASH_* auth setting is declared there and that no secret-shaped assignment carries anything but a placeholder (the same allow-list bash .agents/validate.sh enforces through git ls-files, restated so a plain uv run pytest catches it on the working tree) — while the local-testing word stays legal in prose, which is the only place it can be useful. It demands that scripts/dev-server.sh still gets its readiness probe and its status from the public /api/health and has grown no session, token or password of its own, and that both docs still name the four public paths, the reset command, the four variables and the DASH_COOKIE_SECURE=1 deployment note. Nothing starts a server there; a sentence that quietly disappears is the bug.

tests/test_certs.py is the same kind of file for the certificate authority, and two of its tests are structural rather than behavioural on purpose: they read certs.py itself and fail if it gains a logging import or a raise that interpolates anything, because a stored PEM is a private key and this app turns log lines and error strings into rows a browser reads. The rest runs the whole lifecycle over a dict store — first boot persists exactly five rows, a second boot changes no byte, rotate/revoke/root-rotate and their denylists, the bundle never carrying the root key — and one test completes a real client-certificate handshake with the material a startup would write. tests/test_agent.py runs the matching TLS matrix against the real agent handler, and tests/test_deploy_scripts.py runs the installer in a sandbox where install/chown are faked, so the 0640 mode, the byte-identical bundle and the --status probe (against a live TLS agent) are proven without a root shell. The same file runs scripts/dev-certs-export.py against a real create_app(...) served by uvicorn on a loopback port (phase 24) — a thread and a socket rather than TestClient, because the unit under test is an HTTP client that reads its session out of a Set-Cookie header — and pins the export against the route: 0600, byte-identical, one line per host, no file for an unissued one, no mint and no PEM in the output, and a wrong password exiting non-zero having written nothing.

tests/store_perf.py is the second opt-in marker's machinery (live is the other). It seeds a production-shaped SQLite file — SampleStore's own retention (raw 48 h / 1m 30 d / 1h 400 d), the fleet's real metric names, the real 10 s cadence: ~6.4 M rows and ~720 MB, the dev DB's order of magnitude — and times every hot path of store.py/api.py against it, with EXPLAIN QUERY PLAN for the queries phase 14 changes and a two-connection race that reproduces the database is locked line in the known-gaps list — beside the pre-fix control of the same race, because a clean race that never met a lock would prove nothing. Every pass that writes (a rollup catch-up, a race) runs on a scratch copy of the seeded file, so the baseline the next measurement compares against comes back exactly as it went in. The default suite builds that store and asserts its shape and determinism (row counts, retention windows, rollups that are the mean of the raw rows they came from, a second maintain() that changes nothing) but never its wall time — except behind the marker, where store_perf.BUDGETS is the phase's pinned regression table (14 ceilings, 2-3x the measured "after", maintain() < 1 s, a racing insert at zero database is locked) and a stale budget pointer fails a default-run test. The cached reads are reported twice — cold (the cache invalidated before every run) and cached — because comparing a cache hit against the baseline's full scan would book a 1 000x speed-up that is really just a cache. Only -m store_perf --store-perf runs the timing passes, and scripts/store_perf_probe.py is the CLI form that writes data/logs/store-perf-baseline.{md,json} (task 03's write-side run is store-perf-task03.*, task 04's budget run store-perf-task04.*) for data/logs/NOTES-store-perf.md. Two rules keep it honest: no test ever opens data/dashboards.sqlite3 (the --dev-db flag exists for a human running the probe read-only), and the seeded file is a cache under data/perf/ (DASH_PERF_STORE_DIR relocates it) because a before/after has to measure the same file — pytest's tmp_path on this box is tmpfs, i.e. RAM, which times a store nobody deploys.

The non-timing half of that lives in tests/test_store.py and runs every time: a cache hit issues no SQL at all (counted through sqlite3's trace callback, never through a wall clock), the series list is invalidated by the write that introduces a name and not by the poll that repeats one (and the tracking that decides has a cap, below which it falls back to invalidating), last_maintain is never served from a cache, and pick_res_many()/history_many() answer exactly what the per-series calls answer — on a small store and on the seeded one, at every window.

One more file is worth knowing about: tests/card_contract.py is the table of /api/system/snapshot paths the system page's per-host cards are allowed to read. It is checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and again against the live fleet, so a card can never be "fixed" by inventing a number the agent does not report. tests/ups_contract.py does the same job for the UPS page's hero row and battery glyph (charge %, load %, runtime left, battery_voltage, the glyph's charger_status + on_battery, the card header), against the two real UPSes — recorded verbatim as tests/fixtures/nut_<ups>.vars by ./scripts/capture-fixtures.sh, with the device serials scrubbed — and again against live upsd. The captures earn their place: the CyberPower answers battery.mfr.date: CPS, the APC's battery.date is a factory placeholder, neither publishes battery.temperature or battery.charger.status, and the two disagree on ups.test.result (the CyberPower has run a test and answers Done and passed, the APC still answers the factory No test initiated), so a hand-written variable table would have tested nobody's memory. tests/slots_contract.py pins gufo's /slots rows the same way — hermetically from tests/fixtures/slots_gufo.json (the verbatim /slots reply, captured whenever GUFO_BASE_URL is set; the idle-slot rows are the shape the grid must handle) and again against the running engine. It exists because that table was once guessed: the old slot table read field names the server never published and rendered dashes forever. It also picks the canonical name of each fact — the live server answers id_task and task_id for one request — and the page renders only the canonical key, so a rename fails a test instead of quietly becoming a fallback chain. A busy slot's prompt is the user's in-flight request text, so the capture script blanks it and the page never reads it.

tests/test_gateway_source.py is where the ingress join's honesty lives, and it pins that module from both ends. The input is the captured gateway — gateway_haproxy.cfg, the nginx main config plus its included documents, and the two /proc/net/tcp{,6} captures — run through the real host_agent parsers, so the test compares the join against the agent's own document rather than against a copy of its field names. The assertions are the join's two rules and the limits around them: the graph carries exactly the ports, server_names and targets those documents declare (all 33 routes on this capture, as a set, so a missing or invented route fails by name); a socket is inbound by its local port and outbound only when its remote (address, port) equals a target the config wrote as an address; every other socket is ignored — established, live_in and live_out are all printed and deliberately never reconciled, because a connection to a name, or on a port nobody configured, is not a fact this page may claim; several routes reaching one endpoint carry one count repeated with shared: true rather than a split nobody measured (so the totals, in the source and on /2, sum distinct endpoints); a name target is never credited, and 0 never stands for "not counted"; and no join ever asks a resolver — socket.getaddrinfo is spied on and fails the test, because deciding "already an address, or a name?" is the only question that module may ask, and a DNS lookup in the poll loop would be rule 2 with a round trip attached.

The seventh capture, gateway_resolver.json, is in that suite's reach too: it is the agent's own resolver document from the real gateway, so tests/test_agent.py proves it has the section's shape (one entry per name target_names() derives, via: "libc", the config's own resolver line, no globally routable address) and tests/test_gateway_source.py proves it reaches every route the configs write as a name and shares addresses with the recorded socket table — the capability the join needs, pinned before the join exists to use it.

tests/rate_window_contract.py is the same device for one number rather than a field table: the Δ window each source measures its rates over — RATE_WINDOW_SECONDS for the host agents, LLM_INTERVAL_SECONDS for the engines, one estimator per measuring cadence (phase 23). It holds the two spans, the Δ<window> rounding the pages print, and a range assertion — the estimator's own floor up to its 2 × span ceiling, plus the printed label — shared by four files: the build_hub wiring spy in test_api.py, both throughput cells of both engines in test_llm_source.py, the host payload's window_seconds in test_host_source.py, and the estimator's floor / 2 × span ceiling / 8× jump gate run at both spans in test_rates.py. The two payloads are the same shape and only the number differs, so a pin naming one source's window is exactly how the other could quietly keep advertising a window no poll ever measured.

Battery-test buttons

Instant commands are one of exactly two endpoints in the application that change anything outside this process (the other is the route switch, in the next section; the session and certificate-authority writes change only this app's own rows), and they are gated four times: NUT_ALLOW_ACTIONS=1, not DASH_READ_ONLY, an upsd.users account that actually has the command, and a confirmation in the UI — destructive ones (deep test, calibration, load.off) require typing DANGER. Two more sit in front of those (since the login landed): a session, and this session's CSRF token when the write arrived on the cookie — additive, never a replacement, so the typed DANGER is still what cuts power. The server re-checks every gate on every request; hiding a button is not a safety control. Commands that would cut power are refused outright while the UPS is already on battery.

Give the app a narrowly scoped NUT user instead of a god-mode one — per-command instcmds, not actions = -:

# /etc/ups/upsd.users
[dashboards]
    password = "..."
    instcmds = test.battery.start.quick
    instcmds = test.battery.start.deep
    instcmds = test.battery.stop

One command per instcmds line — that is what man 5 upsd.users specifies ("to specify multiple commands, use multiple instances of the instcmds field"), and a comma-separated list on one line is not that. instcmds = all is the alternative, and the fleet runs that: every command the drivers advertise, including the ones that cut output power. Nothing about the gates changes — the risk tier, the typed DANGER, NUT_ALLOW_ACTIONS and the on-battery refusal are what prevent a click from becoming an outage, because upsd gives the dashboard no way to ask what an account is allowed to send.

upsd authenticates with USERNAME + PASSWORD (see HELP on your own server for the verbs it accepts — USER/PASS are not among them in 2.8.5); a user needs instcmds, never actions = -.

deploy/nut-sudoers is the ruleset that lets the ops account read (and scoped-write) those files: sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut. Read upsd.users before you write it — upsmon authenticates from the same file, so overwriting it blind can take the box down at the wrong moment.

Bind upsd to loopback (LISTEN 127.0.0.1 3051 in upsd.conf) when the dashboard runs on the same box, and to a real interface plus a source-scoped firewall rule when it runs elsewhere — and do not try to scope it by client inside upsd.conf instead: NUT 2.8 removed the ACL/ACCEPT/REJECT directives (a build without TCP Wrappers logs ACL in upsd.conf is no longer supported - switch to LISTEN once per line and ignores the line), so LISTEN plus a host firewall rule is the only access control upsd.conf offers, and an unauthenticated LIST VAR from any routed source is the normal state of a NUT server. NUT_SIMULATE=1 stands in for upsd during development; everything it produces is badged SIMULATED, and it is never the default.

Route switching

/gateway can change its own subject: each row's switch turns a route off and, the other way around, back on. Off is the helper's #off marker on the config line (a switched-off map entry is pointed at the helper's own sink server instead); on is the marker removed and the line's original text restored — but only if the line is unedited since (an edited line is a drift refusal and no byte moves). The switch is a decision of its own, off by default, and it is the application's second write path (the first is the UPS command buttons above).

Enabling it

The write is the flag GATEWAY_ALLOW_WRITES (the same truthy rule as NUT_ALLOW_ACTIONS, and one flag does not carry the other) and it only exists on a host the operator installed --allow-writes --with-routectl on — the certificate-first gate on the agent and the socket-activated root helper beside it (The host agent section above). The install is the agent unit's one extra argument plus the helper and its two units, dashboards-routectl.socket and dashboards-routectl@.service; to undo everything, sudo ./deploy/install-agent.sh --without-routectl removes the helper and its units, leaves the proxy configs and the state directory alone, and prints the still-off routes first — those are yours to switch back on. With the flag off, every switch is a disabled reading with the flag's own sentence, and a click does nothing: no wire is touched, no request is sent.

Switching a route

The press runs the flow the operator can see: the dry-run diff first — the helper's own word for what the change will do, before the operator is asked for the word — then the confirmation that carries the diff and a typed DANGER (both verbs ask for it: turning a route back on is a write to the fleet's only ingress, just as turning it off is), then exactly one send — one connection, one request, one answer, no retry (a retry of a write is a second reload). The answer's words are printed verbatim: applied: true flips the switch and the row's colour follows the payload's word on the next gateway poll — never the click's; applied: false flips nothing. Every call stores one audit event — route-disabled, route-enabled, or route-denied for the gates' and the helper's refusals (a dry run stores none: it changed nothing) — so the audit trail is one line per call with who and when — and a switch outages the route on purpose: it will appear in the event feed.

The honest limits: the switch is one admin (the sign-in's one factor, no more), with no per-host kill switch and no scheduler; and there is no runtime disable server state — the marker is a config line, so a hand reload or a reboot does not undo it (a runtime switch would be quietly undone by the next reload).

When the dashboard is the thing that is broken at 03:00, the recovery is on the box: /usr/local/sbin/dashboards-routectl.py (its operator page is deploy/gateway-routectl-README.md). sudo /usr/local/sbin/dashboards-routectl.py list says what is off (the state sidecar), sudo … verify checks the sidecar against the files (drift is reported, never repaired), and sudo … enable --id <id> puts one route back on through the same protocol — the write, the proxy's own checker, restore on a checker failure, the reload (never a restart), the state record.

Deployment

Containerfile is a two-stage build: uv.lock is resolved in the builder (exactly the versions the tests ran against) and the runtime stage is slim Python with nothing but the venv in it. The host agent is copied in too, so one image can monitor the box it runs on. In the deployment repo the pattern is the same as every other service: quadlet container + env template + nginx/SNI on the single domain.

podman build -t dashboards .
podman run --rm -p 8090:8090 --env-file dashboards.env \
    -v dashboards-data:/var/lib/dashboards dashboards

It listens on 0.0.0.0:8090 as an unprivileged user, and the SQLite history lives in /var/lib/dashboards — put a volume there or every restart forgets the charts and the admin password, because the verifier and the session key are rows in that database. dashboards.env therefore needs DASH_ADMIN_PASSWORD for the first start of a new volume; with neither the seed nor a stored verifier the container exits non-zero with the reason instead of coming up open, and the same seed is ignored once the row exists, so it can be dropped from the env file afterwards. With no sources configured /api/health answers 200 {"status": "empty"}, so the container passes its health probe before its env file is right instead of crash-looping.

Two more lines belong in that quadlet env template because this service faces the public domain: DASH_COOKIE_SECURE=1, so the session cookie never rides plain HTTP (the app also treats a request that arrived with X-Forwarded-Proto: https as secure, so a misconfigured flag degrades rather than leaks), and the password itself — set once, then removable, since the stored verifier wins. The nginx server block, its X-Forwarded-* headers and HSTS live in the deployment repo like every other service here, so a change to how the edge forwards TLS is a change in that repo, not in this one; the only thing this app reads off the proxy is that one scheme header.