dashboards/README.md

# Homelab Dashboards

Five dashboards and the overview page that points at them — one domain, one process, one SQLite
file:

| Page         | What it answers                                                                 | Source                                             |
| ------------ | ------------------------------------------------------------------------------- | -------------------------------------------------- |
| `/`          | Is anything wrong, and which dashboard do I need?                               | all five, in one request (`/api/summary`)          |
| `/system`    | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
| `/disks`     | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount); a host whose agent has no `smartctl` is not a host on this page |
| `/llm`       | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
| `/ups`       | Battery health, runtime left, power events, and battery-test buttons             | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
| `/gateway`   | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, plus **1.3.0**'s `resolver` section to put a number on a route whose target the config writes as a *name* (without it those lines are drawn and dashed), and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |

No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
runtime; every asset is served by this app.

Three more pages are not dashboards: `/login` (the one thing an anonymous visitor sees),
`/settings` — `⚙ Account`, in the masthead of every signed-in page — which shows this browser's
session, signs it out, and changes the admin password, and `/certs` — `⚷ Certs`, beside it —
which is the certificate authority the agents are dialled with: the root CA, this dashboard's
own client certificate, and one row per host with its rotate, revoke and bundle download.

Working on this repo? **[`AGENTS.md`](AGENTS.md)** holds the invariants (read-only by
construction, never invent a number, unconfigured ≠ broken), the locked decisions, the real
fleet facts, and the known gaps.

## Quickstart

```bash
uv sync --python 3.14                       # deps + the package, editable
cp .env.example .env                        # then edit — including DASH_ADMIN_PASSWORD
uv run dashboards                           # http://127.0.0.1:8090
uv run uvicorn dashboards.app:app --port 8090   # equivalent
uv run pytest                               # unit tests, no network needed
```

The first start needs a password to seed: with no `DASH_ADMIN_PASSWORD` and no verifier stored
in `DASH_DB` yet, the process prints the reason and exits non-zero rather than serving the fleet
open (see *Signing in*). The dev password this project uses locally is
`testpass`; production gets a real one. The first thing a browser meets is `/login` — every page
and every `/api` route except the four listed under *What the session covers* asks for it, while
`/api/health` stays open for the container healthcheck and `./scripts/dev-server.sh status`.

Developing against the real fleet from a laptop — dev dials exactly the addresses the deployed
dashboard dials, so there is nothing to tunnel and no port to forward:

```bash
cp .env.dev.example .env.dev                # your host addresses (chmod 0600); this fleet's
                                            # entries are `https://` and carry no userinfo
sudo deploy/install-agent.sh --bind <addr> --port 9120 --fetch-bundle <dashboard-url> \
     --cert-name <name> --session <signed-in token>
                                            # once per monitored host: the agent as a systemd
                                            # service, registered with the bundle this dashboard
                                            # signs for it (the token-mode install — no
                                            # `--cert-bundle` — still works and prints its bearer
                                            # token exactly once; see *Protecting an agent*)
set -a; . ./.env.dev; set +a
uv run dashboards                           # foreground
./scripts/dev-server.sh restart             # or backgrounded, with a pidfile + logs
./scripts/dev-server.sh bundles             # the bundles it exported, with sizes and modes
```

Local dev signs in with the `DASH_ADMIN_PASSWORD` in that file — the word the template's comment
names (`testpass`, what the hermetic suite uses) — and without it a fresh database exits non-zero
for the same reason production does. `.env.dev` is gitignored and `0600`: it holds the upsd
password, and it held the agent bearer tokens until every host of this fleet moved onto an
installed bundle — a TLS `HOST_TARGETS` entry carries no userinfo, so the credential that used to
sit in the URL is now a certificate on the host and a `meta` row here.

`dev-server.sh restart` also **exports the agent bundles** while it is up: one line per
`HOST_TARGETS` name, and each host's stored bundle written to `data/certs/<name>.bundle` at `0600`
(`data/` is gitignored, so no key this writes can be committed). It is a read of
`GET /api/certs/<name>/bundle` and nothing else — the authority stays the one in the database — so
the file is byte-identical to the row and ready to `scp` into `install-agent.sh --cert-bundle` —
though the normal path now needs neither: `install-agent.sh --fetch-bundle <url> --cert-name <name>
--session <token>` downloads that same route on the host during its own install (see
*Protecting an agent, host by host*). A host with nothing issued writes no file and prints the route
that issues one. See *Managing the certificates*.

Each **token-mode** `HOST_TARGETS` entry carries its token as URL userinfo
(`driveripper=http://<token>@<addr>:9120`); `parse_targets` moves it into an `Authorization`
header, so `Target.url`, `/api/config` and every error message stay credential-free. A host that
runs from an installed bundle is `https://<addr>:9120` with **no userinfo at all** — the client
certificate is the credential there, and this fleet is all `https://` today (see
*Protecting an agent, host by host*). No root shell on a host? `deploy/agent-sudoers` is the
one-line grant that lets `ops` run the installer, and running the installer without root prints
that exact line. upsd is dialled at
`NUT_HOST`/`NUT_PORT` on the port it really answers on (3051 on driveripper, not the stock
3493) — see the NUT section below for who may connect.

## The host agent

`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
`/sys/block`, `/sys/class/drm` and `statvfs` — and, on the host that fronts the fleet, the
haproxy/nginx configs and `/proc/net/tcp{,6}` — and answers `GET /snapshot` (everything, one
JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS,
kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and
`GET /health` (the one route that needs no credential) as JSON, with a plain HTML status
page at `GET /` so a browser pointed at an agent learns what it is looking at.

```bash
python3 host_agent.py --once                                # print one snapshot, exit
python3 host_agent.py --host 0.0.0.0 --port 9120 --token-file /etc/dashboards-agent.token
python3 host_agent.py --host 0.0.0.0 --port 9120 --cert-bundle /etc/dashboards-agent.certs
sudo deploy/install-agent.sh                                # systemd unit + user + token
sudo deploy/install-agent.sh --cert-bundle agent.bundle     # ...or the TLS install (bundle from /certs)
sudo deploy/install-agent.sh --fetch-bundle https://dashboards.example.com \
     --cert-name deskwork --session <signed-in token> --bind <addr> --port 9120
                                            # ...the same TLS install, bundle fetched in the run
sudo deploy/install-agent.sh --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN
                                            # ...or a kernel-gated host (SMART passthrough, below)
sudo deploy/install-agent.sh --force        # ...only when it must overwrite a unit it would change
deploy/install-agent.sh --status                            # unit state and /health (https when TLS)
sudo deploy/install-agent.sh --uninstall
```

Re-running the installer is the silent, idempotent path it has always been **as long as the unit
it would write is the unit already on disk**. When the two differ the run stops: it prints
`refusing to overwrite …` with the unified diff on stdout and exits 1 before writing the unit,
before `daemon-reload` and before the service is re-enabled, so a routine redeploy can never undo
a fix somebody made by hand on the host (`--as-root`/`--caps` on a host already running the
dedicated user is exactly such a change, and installing a TLS bundle onto a token-mode host is
another). `--force` prints the same diff and then applies it.

No root shell on the monitored host? `deploy/agent-sudoers` is the single sudoers line that
lets the fleet's unprivileged `ops` account run the installer; run it without root and it
prints that exact line with the script's own resolved path instead of saying "run as root".

It is read-only by construction: there is no code path that writes, and the HTTP server
only answers `GET`. Counters are reported as counters — the dashboard computes rates over
a labelled window, so the agent never has to remember who polled it last.

Per-disk `smart` health is a best-effort add-on on the whole-disk `/sys/block` list (no
partitions): the agent runs `smartctl -i -A -H` per physical disk — 5 s bound per device,
a pass budget of 20 s as a floor that scales to 5 s × the disk count, results cached 4 minutes —
and parses health, temperature, power-on hours,
the sector alarms and SSD wear into `disk_io.devices[].smart`. Refreshing an expired cache
happens on a background thread while the request is served the previous result, so a wedged
disk can never cost a poll more than its own timeout (only the very first pass after the
agent starts runs inline). A host without
`smartmontools`, or whose agent user cannot read the block devices, is *not broken*: every
disk reports `smart: null` with one explanatory `errors` line, and that line never makes
the system page amber. The two cases differ on `/disks`: a host that only cannot **read** its
drives keeps its (dimmed, all-`n/a`) cards, while one whose agent says `smartctl` is not
installed is not listed on that page at all. Enable SMART with `smartmontools` plus the `disk` group (which
`deploy/install-agent.sh` grants when it exists) — the agent shells `smartctl` as itself and
never calls `sudo`, so a sudoers rule alone is not enough — and **redeploy the agent**, since
one installed before phase 09 has no SMART pass to run. Opt out silently with `--no-smart`.
The drive truth this is all developed against lives in `tests/fixtures/smartctl_driveripper_*.txt`:
16 real drives, recorded with serials scrubbed, each one parsed by a test.

The `haproxy` section is the same kind of best-effort add-on, for the host that fronts the fleet.
The agent reads `/etc/haproxy/haproxy.cfg` (`--haproxy-config` names another path) and reports
what the config says: every `frontend` block with its name, each `bind` address:port exactly as
written (`:::22` is the address `::` on port `22`) and the backend names it routes to
(`default_backend`, plus every named `use_backend`), and every `backend` block with each `server`'s
name, target and port — `port: null` when the line names none, never `0`. It interprets nothing:
the routing between the two lists is the **names**, so joining them is the gateway page's job
(`sources/gateway.py`) and the agent never evaluates a rule. `available` answers three questions the way `smartctl_available` does —
`true` read and parsed, `false` no config at that path (this host runs no haproxy), `null` a file
is there but could not be read or understood — and each non-`true` answer costs exactly **one**
`errors` line for the pass. Like the SMART lines, those lines never make the *system* page amber:
a host that never installed haproxy is unconfigured, not broken, and the page that owns the add-on
reads the sentence out of `agent.errors`. A failing config read cannot fail `/snapshot` or
`/health`, and `--no-proxy` drops the section and the line both. The parser is pinned to
`tests/fixtures/gateway_haproxy.cfg`, the verbatim capture from gateway. Neither path is an env
var and `deploy/install-agent.sh` passes neither flag — the unit it writes runs on the defaults —
so a box whose configs live elsewhere means adding the flag to `$UNIT` after installing.

The `nginx` section is that add-on's other half, and it reads more than one file: the main config
(`/etc/nginx/nginx.conf`, `--nginx-config` names another) **plus the files its own `include` lines
name**, one level deep, because that is where nginx keeps its vhosts
(`include /etc/nginx/conf.d/*.conf`) and its SNI routing (`/etc/nginx/stream.d/*.conf`). It
reports every `server` block with its `listen` address:port pairs (`[::]:8443` is the address `::`
on port `8443`; a bare `listen 80` has no address, and the `ssl`/`proxy_protocol` flags are not
this section's business), its `server_name` list (the catch-all `_` included verbatim — "any
domain" is what the page says about it), and every `proxy_pass`; every `upstream` block with its
`server` targets; and every `map` block with its keys. A `proxy_pass $var` is served with the
variable's name and `target: null`: nginx resolves that per connection from the `map` that defines
it, so the agent reports both facts **unmerged** and leaves the joining to a page — a merged target
would be a number the wire never published. `files` names the main config and the included
documents actually merged, and the include pass is bounded (8 files / 1 MB): what it could not
take is said in one `errors` line naming the count, and the section still reads `true`. The same
three-way `available`, the same one line per arm, the same rule that a missing, unreadable or
unparseable config degrades only this section — and `--no-proxy` drops the whole family with it,
in silence.

The `sockets` section is that add-on's live half: what is actually open and carrying traffic, as
opposed to what a config claims. The agent reads `/proc/net/tcp` and `/proc/net/tcp6` and keeps
only the two states a gateway page can draw pipes from — kernel state `0A` LISTEN into `listens`
(`{family, addr, port}`) and `01` ESTABLISHED into `established` (`{family, local, remote}`, each
end an `{addr, port}`). Every other state is a socket in transit: a `TIME_WAIT` row is a
connection that stopped being true before the page painted, and a proxy accumulates thousands of
them, so they are skipped without a word. Each table is read through **at most 4096 lines**
(`/proc`'s own header among them, so 4095 sockets per table); a longer one reports the rows it read
and costs one `errors` line naming the bound, with `available` still `true`. `addr` is the address
itself, unrolled from `/proc`'s per-32-bit-word little-endian spelling — `0.0.0.0` and `::` keep
the literal wildcard form, because "open on every address" is the fact the inbound join keys on —
and a word that is not an address is `null`, never a guess. The tables are read independently: a
kernel without IPv6 has no `tcp6` file, which costs nothing and no line, while a table that is
there and denied costs one. `available` is `true` when either table answered and `null` when
neither did, with one line saying why; there is no `false`, because a table that did not answer is
"no answer" and not "this host has no sockets". Nothing here is joined to a config: a socket does
not know its domain or its backend, so the **gateway page** is what joins these ports to the
`haproxy` and `nginx` sections, and a socket no config claims stays visible as one. The join lives
in `sources/gateway.py`, never in the agent: `/snapshot` keeps the three sections as the agent read
them, `/api/gateway/snapshot` serves the joined topology, and only the counts that join produces are
stored — as the `gateway.in_conns@<port>` / `gateway.out_conns@<host>:<port>` gauges the page
charts (since phase 30 task 02 the outbound gauge exists **only** for an endpoint the config wrote as
an address: the name-keyed ones were a chart of the port-only guess the join no longer makes, so they
would be permanently empty). Nothing in `/api/system/snapshot` grows a proxy or socket field, and no
`sample` row is written for a config.

The `resolver` section is the fourth and the only one that leaves the host. The two configs above
name their targets two ways — an address (`proxy_pass http://127.0.0.1:8081`) and a name
(`server mc1 minecraft.reeselink.com:25565`) — and the kernel's socket table speaks addresses, so a
name is the half a page could never count. The agent therefore takes every `(name, port)` the two
**parsed** sections write as a target (haproxy `server` lines, then nginx `upstream` members,
`proxy_pass` targets and `map` entry targets, deduplicated in config order) and asks libc what each
one is: `getaddrinfo` over `SOCK_STREAM`, published as `via: "libc"` because that is the resolution
haproxy's own libc path performs **on this box** — the box that opens the connections. The nginx
config's own `resolver` directive travels beside it verbatim as `config_directive` (`10.1.0.1`
here, read out of `stream.d/stream.conf`), so a divergence between the two vantages is something an
operator can see rather than something this file guesses; the agent does not query that resolver
directly, and the dashboard never resolves anything at all (`sources/gateway.py` has a test that
fails if it ever calls a resolver). An address literal is reported `status: "literal"` with its
address and **no** lookup, and a name that does not answer right now is `nxdomain` or `error` with
`addresses: []` — the route keeps its row and carries no count, never a `0`.

That pass is bounded and cached exactly the way the SMART one is, for the SMART reason: at most
**64** lookups inside **3 s** of wall clock, cached **300 s**, the first pass inline so a freshly
started agent answers with real addresses and every later expiry refreshed on a single background
thread while the request is served the previous result. `getaddrinfo` takes no timeout argument, so
stopping between lookups is the only defence a black-holing resolver leaves — and the cost of not
stopping is the *system* page's poll timeout, spent on an add-on that page never draws. Each entry
carries the time **it** was answered, so a cache-served entry cannot pretend to be current; the
budget and the name bound each report themselves in one `errors` line and leave `available` at
`true`. `available` is the family's three-way answer (`false` means there was nothing to resolve
because no proxy config was read, and its one line says **not found** or **unreadable** according to
which of the two it was), and
`--no-resolve` removes the section in silence the way `--no-smart` removes SMART — while `--no-proxy`
removes it too, because a box that collects no proxy has no declared names to look up.

All four parsers are pinned to captures rather than to descriptions, and
`./scripts/capture-gateway-config.sh` re-records them over the `GATEWAY_SSH` alias (default
`gateway-ops`): `/etc/haproxy/haproxy.cfg`, `/etc/nginx/nginx.conf` plus every document its own
glob `include` lines name (`gateway_nginx_conf_d_*.conf`, `gateway_nginx_stream_d_*.conf`),
`/proc/net/tcp{,6}`, and `tests/fixtures/gateway_resolver.json` — the `resolver` section itself, so
the name→address join is testable with no network. No sudo and no DNS client: those paths are
world-readable, which is exactly what the agent reads as itself, and the resolver is recorded with
`getent ahostsv4` / `getent ahostsv6`, which is libc on the box that dials (the same vantage
`via: "libc"` claims). A read that fails, or answers empty, leaves the existing fixture alone and
exits 1, so a laptop with no route to the gateway cannot turn seven captures into seven empty files.

The resolver document is **derived, never listed**: the script feeds its own freshly captured
configs through `agent.host_agent`'s parsers and `target_names()`, so it asks about exactly the
targets the agent will ask about, an address literal costs no remote read, and a route added to the
gateway moves the fixture with it. `getent` answering `rc 2` is a fact — that name has no record and
becomes `status: "nxdomain"`, kept beside the names that did answer — while a dead transport, a
missing `getent`, or **not one name answering** is a broken vantage: the run refuses, because a
document of nothing but `nxdomain` would make every route on `/gateway` a permanent dash and every
join test a test of nothing. Re-capturing a gateway nobody changed is a no-op (fixed `ts`,
deduplicated, sorted with the agent's own `sort_resolver_addresses`), so a diff in that file means
the ingress changed.

The configs are not rewritten (their domains are the ingress map, and key material appears only
as the paths nginx is told to read); the socket tables and the resolver's answers lose only their
identifiers, in line with the rule that scrubs a drive's serial: a globally routable address becomes
a documentation one (RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI-64 and an inbound
visitor's address is not ours to publish. Every port, state, column and width in those tables is
verbatim, and those files are volatile by nature — a proxy in service has a different connection set
every second — so the tests assert shape and the ports the configs claim, never a particular row.
`AGENT_VERSION` is **1.3.0**, the version that adds the `resolver` section on top of the three
phase 28 shipped; `GET /schema` names all four, each with its `available` flag, and `--no-proxy`
and `--no-resolve` are listed there too. A host still running 1.2.0 answers with a snapshot that
simply has no `resolver` key — which is why the version spread costs nothing: a section an agent
did not collect is absence, and the gateway page counts the address-written routes it has and draws
the name-written ones with a dash.

**The dashboard reads that section and never produces one.** `sources/gateway.py` indexes the
entries by `(name, port)`, stamps each route's `out` with `resolved` / `via` / `resolved_ts` — the
address the gateway dialled, the agent's own string for how it got it, and the moment that **entry**
was answered — and lets the socket join reach a name-written route through it. Which is the whole
of phase 32: the outbound endpoint of a `server` line, a `proxy_pass` or a `map` entry is now an
address the box that opens the connection published, so the `gateway.out_conns@<domain>:<port>`
gauges phase 30 task 02 stopped writing are written again, under the config's own spelling, with a
figure that belongs to the domain instead of to whatever else was listening on its port. Four rules
keep the join honest, each pinned in `tests/test_gateway_source.py`: a name nobody resolved keeps
`resolved: null`, `connections: null`, `metric: null` and **no stored sample** (never `0`, never an
empty list); matching is by normalised address, so `::ffff:10.1.2.64` in `/proc` and `10.1.2.64`
from `getaddrinfo` are one endpoint while a zero-padded `010.1.2.64` — which is no one's address
notation — matches nothing; two names resolving to one address, or two route lines reaching it, are
**one** count held by every line that reaches it; and the module still never calls a resolver, a spy
pin covering the parse, the gauge name and the capture alike. An `address: true` route claims none
of the three provenance fields, because the config wrote that address and no lookup happened.

`deploy/install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token>` is the same
TLS install with the download done inside the registration run: it asks
`GET <url>/api/certs/<name>/bundle`, writes the answer to a temp file, and hands it to the
identical validator + `0640` install + unit path. The three flags are all-or-nothing, they are
refused together with a local `--cert-bundle` (one install, one bundle source) and with
`--status`/`--uninstall`, and the fetch and the shape check both run **before** the root
boundary — a dashboard that answers 401, or that has issued nothing for that name, stops the run
with its own one-line sentence and no partial install. The session token is a *fetch credential
only*: it is sent on that one request, never printed (the sudoers hint re-quotes the command
without it), never written to the token file, the unit, the bundle or the log, and unset the
moment the bytes are in hand. The temp copy is removed on every exit path, so the installed
`/etc/dashboards-agent.certs` is the only place the fetched key survives.

**TLS mode is `--cert-bundle` (env `AGENT_CERT_BUNDLE`)**: one installed PEM the dashboard
signed — this host's server certificate and key, the dashboard's root CA, a `# BUNDLE-REV <n>`
line and the `# REVOKED <serial>` denylist. The agent then serves `https://` and answers every
path but `/health` only to a **client certificate signed by that root and not on the list**: 403
`client certificate required` for a peer that proved nothing, 403 `client certificate revoked`
for one the operator withdrew (a revoked identity is refused on `/health` too — that path is open
to the installer's probe, which holds the CA and no certificate, not to a certificate that was
taken away). The denylist is re-read when the bundle file's stamp changes, so a re-deployed
bundle takes effect on the next request rather than the next restart; the certificate itself needs
a restart, because `ssl` cannot swap a chain under a live socket. The unpacked PEMs exist only
long enough to load into the context and are then deleted, and the private key is written `0600`
in the unit's `RuntimeDirectory` when it has one. In this mode `/snapshot` gains two additive
fields — `tls: true` and the `bundle_rev` the bundle carried, which is how the dashboard tells a
stale install from a current one (they reach `/api/system/snapshot` and `/api/disks/snapshot` as
`agent.tls` / `agent.bundle_rev`) — and **the bearer token is not read**: the handshake is the
auth, and a token that kept working after a revocation would defeat the list. The dashboard's half
is `collectors.AgentTls`: point that host's `HOST_TARGETS` entry at `https://` and the fetch
presents the dashboard's client certificate and verifies the agent against the same root, and a
host revoked in `certs.agents` is refused before a socket is opened. A bundle that will
not load is exit 2 naming the file; there is no "fall back to plain HTTP" path.

`deploy/install-agent.sh --cert-bundle <file>` is how that file gets there. It validates the
bundle first by the same text shape (two certificates and exactly one key, plus the
`# BUNDLE-REV` line) — before touching anything, before the root check, and without ever
dialling the dashboard — then installs it to `/etc/dashboards-agent.certs` as
`0640 root:dashboards-agent`, the token-file idiom: the agent opens the bundle *itself* at
startup and re-reads it when its stamp changes, so the service user must be able to read it
and no other account may. The unit gains `--cert-bundle` and a private `RuntimeDirectory` for
the PEM blocks, `--status` probes `https://…/health` with the bundle's CA and no client
certificate (python3 `ssl`, never an external TLS binary), `--uninstall` removes the bundle
with the token, and a re-run without the flag leaves an installed bundle in place — the
token-reuse idiom.

Binding off-loopback without a token *or* a bundle is refused (exit 2) rather than warned about.
Prefer `--token-file` over `--token`: argv is world-readable through `ps`, and this token guards
hostname, mounts, disk and GPU models, and process names. `deploy/install-agent.sh` creates
a system user, generates a token once (re-runs reuse it; `--rotate-token` for a new one),
stores it `0640 root:dashboards-agent` — the unit hands the agent the *path*, and the agent
opens it as its own user, so the file must be readable by that user and by nobody else —
and runs the agent under `ProtectSystem=strict` with an empty `CapabilityBoundingSet` (unless
`--caps` is asked for, below) and **no device filter**.
SMART device access is DAC: the installer adds the service user to the `disk` group and SELinux
stays in front of it. The strict device policy this script used to write is what locked
`driveripper`'s whole device namespace — systemd matches those globs by name against
`/proc/devices`, where kernel 7.2 lists NVMe as `blkext`, so `/dev/nvme*` matched nothing and a
strict policy denies whatever a list fails to match. `PrivateDevices=false` is still written out
explicitly, because a private `/dev` with no device list is what dies at spawn
(`status=208/STDIN`).

When DAC is not enough — a **kernel gate** — `--as-root` and `--caps` write the two unit lines
that gate checks accept. Kernel 7.2 gates smartctl's raw passthrough behind *capabilities*, not
uid: an NVMe SMART-log read through the block node needs `CAP_SYS_ADMIN` unless the drive's
firmware advertises the page, and SATA SMART behind a SAT layer is an ATA pass-through the
unprivileged `SG_IO` allow-list does not include (`CAP_SYS_RAWIO` is what the kernel names for
it). So `--as-root` writes `User=root` and `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` writes
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (comma-separated in, space-separated out);
both default off, because a host whose drives answer through the `disk` group needs neither.
`--caps` without `--as-root` is refused with exit 1 — a capability grant on the dedicated
service user is not the kernel-gate fix — and `--as-root` alone proceeds with a note, because a
root process under the default empty bounding set is capability-less (`CapEff 0`) and reads no
more than the unprivileged service user did. The token file, its `0640 root:dashboards-agent`
mode and the `--status` probe are deliberately unchanged by these flags: root reads the token
either way, and a probe of a port cannot see who owns the process behind it.

### Managing the certificates

The dashboard is the authority, and `/certs` (`⚷ Certs` in the masthead) is its page: the **Root
CA** block (CN, serial, expiry as a local timestamp *and* as time left, and **Rotate root CA**),
the **Dashboard client** line (the one certificate every agent verifies the dial against —
display-only, because it is minted with the root and has no route of its own), and one row per
host with its status chip, serial, expiry, issued-vs-deployed bundle revision, the addresses its
certificate is valid for, and **Rotate** / **Revoke** / **bundle**. The agents table carries the
stylesheet's own `scroll` modifier, so it is a box on **both** axes with a 320 px height cap at
every width and its right-hand columns — the serial, the bundle revisions, the addresses and the
controls — are reachable by scrolling instead of clipped. Since phase 27 it is no longer the only
thing that scrolls (at ≤640px every `table.data` does, sideways only, no height cap); `.scroll`
out-ranks that rule, so this table keeps its own box on a phone exactly as on a desktop.

Nothing there is coloured: `severity.py` rates no certificate, so an expiry is a date and a
bundle revision is a count. The one verdict on the page is the API's `status` word (`active` green,
`revoked` red, `unissued` grey — a certificate that does not exist never borrows a green pass).
Every write asks a `confirm()` first and prints the server's own sentence afterwards; a root
rotation answers `restart_required`, because the CA and the client pair are startup material — the
table is new at once, the running process keeps dialling with what it loaded, and every host then
needs its new bundle installed. A row whose host has left `HOST_TARGETS` is listed with no buttons
at all, because every per-host route refuses it.

Under the page is the same API a script would call. Every route below is signed-in (and every write
is CSRF-gated like any other), and **none of them dials a host** — the effect of a revocation is
that the *next poll* refuses to dial (see `collectors.AgentTls` above):

| Route | What it does |
| ----- | ------------ |
| `GET /api/certs` | `root` and `client` with serial and expiry, plus one row per host the operator could act on: every `HOST_TARGETS` name and every certificate still in the table. A configured host nobody has installed yet reads `status: "unissued"` with `null` serial rather than going missing; `bundle_rev` is what this dashboard issued and `deployed_bundle_rev` is what that host's agent reported back (so "the box is one bundle behind" is visible, `n/a` when nothing answered). Reads are pure — no page refresh ever mints a certificate |
| `GET /api/certs/<name>/bundle` | the exact bytes of the stored bundle as `application/x-pem-file` with `Content-Disposition: attachment; filename="agent-<name>.bundle"`. That byte-equality is the point: `install-agent.sh` is handed the file and compares nothing else, so a route that re-signed on each read would make "is this the bundle I installed?" unanswerable. It carries that host's **private key**, which is why it is a download behind the session and `no-store` |
| `POST /api/certs/<name>/rotate` | fresh key, fresh serial, `bundle_rev` +1, `status: active` — and it is also how a host gets its **first** bundle (`bundle_rev` 1). The SAN comes from the current `HOST_TARGETS` URLs plus the name, not from the old certificate, because the address a host is dialled by is a deployment fact |
| `POST /api/certs/<name>/revoke` | marks the identity revoked, and **re-seals every other bundle** because the `# REVOKED` denylist lives inside each of them — so the fleet has new bundles to collect, while their `bundle_rev` does not move (their identity did not change). There is no unrevoke: the fix is a rotate |
| `POST /api/certs/root/rotate` | a new root, a new client certificate, every agent re-issued, and `{"ok": true, "restart_required": true}` — the CA and the client pair are startup material, so the table is new at once while the running process keeps dialling with what it loaded. Saying so beats pretending a hot-swap that does not exist |

**During development there is no second authority.** The CA the first boot minted into the
`meta` table is the one everything reads, and `scripts/dev-certs-export.py` is the dev loop's half
of that: `./scripts/dev-server.sh restart` runs it after the readiness probe, it signs in to the
local server, downloads every `HOST_TARGETS` name's bundle through `GET /api/certs/<name>/bundle`,
and writes each one to `data/certs/<name>.bundle` at `0600` (`os.open`, so the mode is the file's
birth mode) in gitignored `data/`. Per host it prints exactly one line — the file it wrote with its
size, or `no bundle issued yet — issue one on /certs` — and nothing at all is written for a host
with nothing issued. It is a **read** of the same route the `/certs` download uses, never the
startup path, so a restart cannot rotate an identity: the bytes on disk are the bytes in the row.
A sign-in that fails is exit 2 and a `!` line (a `429` says so with its wait), and
`dev-server.sh` prints that as a warning rather than failing `start`, because the server is
provably up and the authority is untouched. `status` — or `dev-server.sh bundles` — lists what is
there with sizes and modes.

### Protecting an agent, host by host

Four steps, and they are order-dependent because the dashboard picks its dial from the URL. **A URL
flipped before its bundle is installed turns that host red by construction**, so the URL flips
last, after the probe answers.

```bash
# 0. issue the host a bundle if it has none: POST /api/certs/<name>/rotate (the Rotate button on
#    /certs) is also how a host gets its *first* one, and `GET …/bundle` then answers the bytes.
#    An unissued name is what makes step 1 say "no bundle issued for <name>".

# 1. register the host: one run installs the agent *and* fetches its bundle from the dashboard
#    that owns the authority. The name is the HOST_TARGETS name; the token is a signed-in
#    session and it buys this one GET — it is not stored on the host. As root, 0640
#    root:dashboards-agent, after a shape check on the fetched bytes that runs before anything
#    is written, so a 401 or an unissued name leaves this host exactly as it was.
#    Stage the repo's script and agent first (`../agent/host_agent.py` is where the installer
#    reads the agent from, and the fleet's sudo grant names a path, not a checkout):
scp deploy/install-agent.sh <host>-ops:/home/ops/agent-install/deploy/
scp agent/host_agent.py     <host>-ops:/home/ops/agent-install/agent/
ssh <host>-ops sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120 \
     --fetch-bundle https://dashboards.example.com --cert-name deskwork --session "$SESSION" \
     --force

#    the fallback, for a host that cannot reach the dashboard: the signed-in GET by hand (its
#    bytes are identical to the stored row and it carries that host's private key, so `curl -o`
#    it somewhere private rather than `cat`ting it into a shell history — and against a local dev
#    server this file already exists, written by `./scripts/dev-server.sh restart`), then
#    --cert-bundle <file> instead of --fetch-bundle.
#      curl -s -H "Authorization: Bearer $SESSION" -o agent-deskwork.bundle \
#           http://127.0.0.1:8090/api/certs/deskwork/bundle
#      scp agent-deskwork.bundle <host>-ops:/tmp/
#    A *dev* dashboard on a laptop is usually unreachable from a monitored host in the other
#    direction too (the tunnel route is one-way). An SSH reverse tunnel keeps the registration
#    path rather than the scp: `ssh -R 18099:127.0.0.1:8090 <host>-ops` and then
#    `--fetch-bundle http://127.0.0.1:18099`.

# 2. restart it as root: the same run copies the repo's `agent/host_agent.py`, which is how a
#    pre-1.1.0 agent gains TLS mode at all (an older file does not know the `--cert-bundle`
#    flag), and the unit gains `--cert-bundle` and its RuntimeDirectory — but `systemctl enable
#    --now` only *starts* a unit, so an already-running agent keeps serving its old mode until
#    something restarts it.
ssh <host>-ops sudo systemctl restart dashboards-agent
#    the CA-only https probe — pass the SAME --bind the unit uses, because the probe takes its
#    address from the flags and would otherwise dial 127.0.0.1 and report "no answer" for an
#    agent bound to a LAN address that is in fact healthy.
ssh <host>-ops sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120 --status

# 3. point that host's HOST_TARGETS entry at https with NO userinfo token — the certificate is
#    the auth now — and restart the dashboard. `/certs` should then read issued == deployed rev.
export HOST_TARGETS="deskwork=https://<addr>:9120 …"
```

`--force` in step 1 is not a shortcut: installing a bundle *is* a change to that host's unit (the
`--cert-bundle` flag and its `RuntimeDirectory`), so without it the run prints the diff and stops
by design — read it, then re-run with the flag. Reading it first is cheap and worth doing on a host
with a hand-fixed unit (run the same command once without the flag, study the diff, run it again
with it). The same rule covers every change the installer applies, which is why the `--status` line
above comes *after* the restart: the probe reads the unit file to decide between http and https, so
pointing it at a process that has not been restarted yet reports "no answer" for a bundle that
installed perfectly. `install-agent.sh --bind … --port 9120` prints the `https://<addr>:9120` line
for a bundle install, so step 3 is a copy rather than a guess. A host left on `http://` keeps
working over its bearer token — the rollout is per host, and one revoked or expired certificate
never blinks the others.

**Upgrading an already-registered host to a newer agent is one file, not a rollout.** Re-stage
`agent/host_agent.py` into `/home/ops/agent-install/agent/` (the snapshot-first rule applies: a path
grant is only as current as the file it names, so copy what you are replacing to
`host_agent.py-<date>-backup` first — a name the sudo grant does not name, so sudo will not run it),
run the granted path with **the unit's own arguments and nothing else**
(`sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120`), and restart.
No `--cert-bundle`, no `--fetch-bundle`, no `--force`: the bundle already on disk is what the unit
render reads — the flag only replaces the file — so the render equals the installed unit, the re-run
guard prints nothing, and `/etc/dashboards-agent.certs` and the token file keep their bytes. Then
the same two checks the registration uses: the CA-only `--status` probe (with `--bind`) answers
`{"ok": true, …, "version": "1.3.0"}` on `/health`, and the dashboard's own `https://` dial answers
whatever the new version added — `agent_version` plus the `haproxy` / `nginx` / `sockets` sections
(1.2.0) and the `resolver` section (1.3.0) — beside the sections it always had. That is exactly how
`gateway` moved 1.1.0 → 1.2.0 (phase 28) and 1.2.0 → **1.3.0** (phase 32, 2026-10-25: the same unit
sha256 before and after, the bundle untouched, the guard silent, one file and one restart), and the
`/gateway` page counted **33 of 33** routes afterwards where it had counted 2 before.

Two things the first full rollout learned, both cheap to hit and expensive to guess at:

* **A CA row older than the identifier fix must be rotated before any bundle is installed.** A
  root minted without a `SubjectKeyIdentifier` is still readable, still starts, and still signs —
  but `ssl.create_default_context()` (the strict mode every dial here builds) rejects every chain
  it produces, with `CERTIFICATE_VERIFY_FAILED: Missing Subject Key Identifier`, so the host comes
  up healthy on `https` and reads `unreachable (ConnectError)` from the dashboard. `POST
  /api/certs/root/rotate` is the repair — a new root with its identifiers, a new client identity,
  every agent re-issued, then a dashboard restart and a fresh bundle per host. Never a
  `DELETE FROM meta` for this: the rotate button exists precisely so the operator keeps the
  authority in one place.
* **Rolling back is not just the URL.** A TLS-mode agent serves `https` only, and the unit keeps
  its `--token-file` line (so the token survives on the box, ready for this). Restoring an old
  `http://<token>@…` entry does not bring the host back on its own: the `--cert-bundle` line has to
  come out of the unit *and* `/etc/dashboards-agent.certs` has to go, because the installer reuses
  an installed bundle on a run that was not given the flag.

A name that is not in `HOST_TARGETS` answers `404 {"configured": false}` **before** any signing, any
dial, and any `meta` write: a typo that minted a key would leave a certificate nobody can install
in the authority forever. Each accepted write leaves one fixed-text event —
`cert-rotated`, `cert-revoked`, `cert-root-rotated` under `source: "certs"`, readable at
`/api/events?source=certs` — and a certificate serial never leaves a `meta` row. Serials are served
as decimal **strings** for that reason: a 64-bit serial through a JS number is a serial the page
cannot match against a denylist.

## Signing in

One admin, one password, and the server **refuses to start** when it has neither a stored
verifier nor `DASH_ADMIN_PASSWORD` — the domain is public and the fleet's hostnames and UPS
controls sit behind this login, so there is no way to run it open by accident. There is no
second account and no per-user view: the six dashboard pages (`/`, `/system`, `/disks`, `/llm`,
`/ups`, `/gateway`), the two admin pages (`/settings`, `/certs`) and every data route belong to
that one operator.

A session is an **HMAC-signed cookie** (`dash_session`), not a row in a table: it carries the
signed-in name and its own expiry, is valid for `DASH_SESSION_HOURS` (default `720` = 30 days),
and is verified with a signing key that also lives in `DASH_DB`. That is why a password change
signs out every other browser at once — it rotates the key, so every cookie minted under the old
one stops verifying on its next request — and why nothing has to be pruned.

`DASH_ADMIN_PASSWORD` is a *seed*, not the credential. The first start hashes it
(PBKDF2-HMAC-SHA256, 600 000 rounds) into the `meta` table of `DASH_DB` as `auth.verifier`, and
from then on the stored verifier wins: remove the variable once the server is up and the same
password still works, change it with `POST /api/auth/password` — or in a browser, on the
`/settings` page (*The account page* below) — with no redeploy, and know that a container env can
never roll a changed password back. If `DASH_ADMIN_PASSWORD` is still set on a box that already has a
verifier, it loses — and the server says so once at startup, in words, without quoting the value.

Forgotten password, or a database you want to hand over? Delete the rows and seed again:

```bash
sqlite3 "$DASH_DB" "DELETE FROM meta WHERE key IN ('auth.verifier','auth.session_secret')"
```

Deleting `auth.session_secret` — which that command does — signs out every browser already
signed in, which is also exactly what changing the password does on purpose. Only the hash and
the signing key are ever written: `grep -c "your password" "$DASH_DB"` comes back `0`, and no log
line, event row, or API response carries the password, the verifier, or a session token.

Nothing that leaves the app carries a credential. A `HOST_TARGETS` userinfo token is parsed
into an `Authorization` header, and every URL the browser can see — `/api/config`, a source's
error string, the event feed — goes through `config.strip_userinfo` first, because an
unreachable host writes its failure message into the event table and `/api/events` serves it.

### What the session covers

The gate is default-deny: **every page and every `/api` route** requires a signed-in admin
except four entries, named next to the middleware they guard in `app.py`:

| Public                     | Why it has to be                                                   |
| -------------------------- | ------------------------------------------------------------------ |
| `/login`                   | the form itself                                                     |
| `POST /api/auth/login`     | you cannot be signed in to sign in                                  |
| `/api/health`              | the Containerfile `HEALTHCHECK` and `./scripts/dev-server.sh status` poll it unsigned, and it publishes no fleet data |
| `/static/…`                | the login page is styled by the same stylesheet as the dashboards    |

Adding a path to that list is a security decision, and a route a future phase forgets fails
`tests/test_auth_api.py::test_every_route_that_is_not_public_requires_a_session`, which walks
the served route table rather than a list written by hand.

What a refusal looks like depends on who asked, so that neither caller has to guess:

```bash
curl -si http://127.0.0.1:8090/system | head -3
# HTTP/1.1 303 See Other        → Location: /login?next=/system  (a browser ends up back here)
curl -s http://127.0.0.1:8090/api/summary
# {"ok":false,"detail":"sign in required","login":"/login"}   (a script never gets HTML)
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8090/api/health
# 200
```

The `?next=` value keeps the query string too (`/llm?range=1h` survives the round trip), and it
is judged before it is ever followed: a `next` that leaves the site (`//evil.example`,
`http://evil.example`, the percent-encoded and control-character spellings alike) is dropped to
`/`. `auth.safe_next` is the one function that decides both halves. And every answer the gate
produces — `303`, `401`, `403`, `429`, the route's own `404` — carries `no-store`, because a
cached redirect would park a browser on the login page after it had already signed in.

### The sign-in page

`/login` is one card with one password field — no username, because there is one admin and a field
that is always the same word teaches you that half the form is theatre. It carries the masthead's
brand link and the theme toggle and nothing else from it: no tabs (each would bounce straight back
here) and no status dots, because there is nothing signed in for them to report.

What it does with an answer is the part worth knowing:

| Response | On screen                                                            |
| -------- | -------------------------------------------------------------------- |
| `200`    | straight to `?next=`, or to `/`                                        |
| `401`    | `Wrong password.` under the field, and the text you typed is selected   |
| `429`    | the refusal plus the lockout counting down, button disabled to zero    |
| other    | the server's own `detail`, printed verbatim                            |

The countdown is the server's `Retry-After`, not a guess, and a refusal is written as text rather
than HTML — this is the one page a stranger reaches first. A browser that already holds a session
never sees the form: the server `303`s it to the dashboard, and the script checks
`/api/auth/status` first so the form does not even flash. `?next=` is followed only when it is an
in-app path (`/ups`, not `//evil.example`), judged on the server and again in the page.

### The account page

`/settings` — reached by `⚙ Account`, which sits in the masthead of every signed-in page including
this one — is the session facts and the one form that changes the password.

| Panel | What it says, and what the button does |
| ----- | ------------------------------------------------------------------------ |
| Session | the signed-in name, the expiry as a local timestamp **and** as time left, and **Sign out** (`POST /api/auth/logout`, then `/login`) |
| Change password | current + new + confirm, then `POST /api/auth/password`; the outcome is one line under the form |

The form checks what it can check itself (a current password typed, at least 8 new characters,
the two new boxes agreeing) so a typo is answered without a round trip, and then reports whatever
the server said: `403` is *the current password is wrong*, `422` names the rule the new password
broke, `429` is the lockout with its own seconds. The page also states the precedence rule in its
own words — the env var seeded the password once, a change here wins, resetting from the
environment means deleting `auth.verifier` and restarting — because that is the sentence that
prevents a support call about an edited `.env` that did nothing.

One detail makes the page more than a form. A change rotates the session-signing secret, which is
what signs out every *other* browser, and it invalidates the CSRF token this browser cached — so
`Dash.resetAuth()` clears that cache after the change (and after any `401` bounce). Without it the
operator changes the password, then every UPS button 403s until they reload.

**Sign out deletes this browser's cookie; it cannot revoke a copy of the token.** A session is
signed rather than stored, so nothing is left to mark as dead — the one thing that reaches every
browser at once is the signing key, and that is what a password change rotates. If a machine is
lost or a session is suspected, change the password here: that is the revocation, and it is why
the same page carries both buttons.

### Writes need the CSRF token

A write authorised by the **cookie** must carry `X-CSRF-Token`, validated against that same
session (`GET /api/auth/status` is where a page that is already signed in reads it — the login and
change-password responses carry it too, minted for the session they just created — and it sits
behind the session because a token is worthless to anyone who cannot read that response). `Dash.fetchJSON`
fetches it once per browser and attaches it to every non-`GET`/`HEAD` request, so page code does
not think about it — and `POST /api/auth/logout` is gated like any other write, because an
`<a href="/logout">` planted on a page you read mid-outage is a CSRF that signs you out.
Mismatched `Origin` or a `Sec-Fetch-Site` that is not `same-origin`/`none` is refused before the
token is even looked at, so a cross-site login post cannot spend your throttle either.

### Changing the password

`POST /api/auth/password` with `{"current": "…", "new": "…"}` — a protected write, so it needs the
session and the CSRF token like everything else:

```bash
curl -s -X POST -H 'content-type: application/json' -H "Cookie: dash_session=$SESSION" \
     -H "X-CSRF-Token: $CSRF" -d '{"current":"…","new":"…"}' \
     http://127.0.0.1:8090/api/auth/password
# {"ok":true,"changed":true,"hint":"other browsers are now signed out",
#  "expires_at":1794982214.0,"csrf":"…"}
```

Nothing forces a change: no expiry, no first-login interstitial, and `DASH_ADMIN_PASSWORD` is never
re-applied after the first boot of a database. The only composition rule is length — at least
`api.MIN_PASSWORD_LENGTH` (8) characters, not empty once stripped, and different from the current
one — each refused with a `422` that names the rule. A symbol/digit mandate produces sticky notes
rather than passwords, and the cost of a guess here is PBKDF2 plus the backoff, not the character
classes.

A wrong `current` is `403 {"ok":false,"detail":"current password is wrong"}` and a
`password-denied` event, and guesses at that field are throttled on their **own** key
(`password:<address>`) — the same arithmetic as the login form, a separate bucket, so this form is
never an unthrottled oracle for the old password once a cookie is stolen and one form's lockout
does not lock out the other.

A right one rewrites `auth.verifier` **and** `auth.session_secret`, which is the whole revocation
story: a session is a signed cookie with no server-side record, so rotating the signing key is the
only way a password change reaches the browsers that are not in front of the form (without it a
stolen cookie would stay valid for another `DASH_SESSION_HOURS`). The response carries a fresh
cookie and a fresh CSRF token minted from the **new** secret, so the caller who just changed the
password is not signed out by their own form, while every other browser gets a `401` on its next
request and signs back in with the new password. `password-changed` lands in the event feed — an
event, not an overview alert, and not a field on `/api/health` (which is public) or `/api/config`.

### Scripting the API

`Authorization: Bearer <session token>` is the scripted caller, and it needs no CSRF token —
another site can make your browser send its cookie but cannot read a token out of your shell.
The session token is in the login response's `Set-Cookie`, not in its body:

```bash
SESSION=$(curl -si -X POST -H 'content-type: application/json' \
              -d '{"password":"…"}' http://127.0.0.1:8090/api/auth/login \
          | tr -d '\r' | sed -n 's/^set-cookie: dash_session=\([^;]*\).*/\1/p' | head -1)
curl -s -H "Authorization: Bearer $SESSION" http://127.0.0.1:8090/api/summary   # 200
curl -s -H "Authorization: Bearer $SESSION" -X POST -H 'content-type: application/json' \
     -d '{"ups":"apc","command":"test.panel.start"}' http://127.0.0.1:8090/api/ups/command
```

That is the app's one write endpoint **that dials anything** (the `/api/auth/*` and `/api/certs/*`
writes change only this process and its SQLite `meta` table — invariant 1 is about the fleet), and
the bearer header changes nothing about its gates: a command that can cut output still needs
`"confirm":"DANGER"`, and is still refused while the UPS is on battery.

### Lockout

Five wrong passwords from one address starts a backoff (`429` with `Retry-After`, exponential
from 60 s, capped at an hour; a good sign-in clears it, and a restart resets the counters). The
change-password form keeps its **own** counter under `password:<address>`, so hammering one form
cannot lock you out of the other and neither form is an unthrottled oracle for the same secret.
The refusal is the same sentence for every kind of failure — one account, so the response must
not rank reasons — and each refused attempt writes a `login-denied` or `password-denied` event you
can read at `/api/events?source=auth`. Neither ever becomes an overview alert: a password guess is
not an outage, and a page that blinks for both trains you to ignore the ones that matter.

## Configuration

Everything comes from the environment (see `.env.example` for the annotated list). At
startup a `.env` next to the checkout is loaded first if it exists — as a *seed*: a
variable the container, systemd unit, or your shell already set always wins. Point
`DASH_ENV_FILE` somewhere else to keep several installs on one box. `.env.example` is
documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev` itself
(see the quickstart) because dev settings must not leak into a production process.

| Variable                              | Meaning                                                        |
| ------------------------------------- | -------------------------------------------------------------- |
| `HOST_TARGETS`                        | `name=url` per host running the agent. The shape a new host gets is `https://host:9120` with **no userinfo** — a certificate-authenticated agent, verified against this app's own root CA and presented with its client certificate, with a revoked host refused before the dial (`certs.py`, `collectors.AgentTls`). The compatibility shape is a token-mode agent's URL userinfo (`http://TOKEN@host:9120`), moved into an `Authorization` header by `parse_targets` so it never reaches `/api/config` or an error message |
| `GUFO_BASE_URL` / `VLLM_BASE_URL`     | LLM engines; unset means *not deployed here*, and its half of the page disappears |
| `RATE_WINDOW_SECONDS`                 | the **host's** Δ window (default 10) — bytes/sec and jiffies→CPU %, measured over what a 10 s poll can fill; each value carries the window it actually got, and `/system`'s footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one `n/a` instead of a spike, and a rate 8× above the last one is held until a window measured from a **different** sample confirms it (phase 13). The LLM rates do **not** use it: they are measured over `LLM_INTERVAL_SECONDS` (phase 23) |
| `LLM_INTERVAL_SECONDS`                | the `/llm` poll cadence (default 2 s) **and** the Δ window behind every token rate — one estimator per measuring cadence, so a slot that starts reads steady within two polls instead of ramping over five. Slowing the poll slows the window with it; there is no separate LLM window knob, and a window narrower than the interval would answer `n/a` every poll |
| `DASH_HOST` / `DASH_PORT`             | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
| `HOST_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often the host agents and upsd are polled (10 / 5 s). `GATEWAY_INTERVAL_SECONDS` (30) and `GATEWAY_TIMEOUT_SECONDS` (10) are the gateway source's own pair: a connection count is the shape of half a minute, and the fetch is the plain agent one with headroom for a handler that may be mid-smartctl pass |
| `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:<host>` jobs exist exactly when `HOST_TARGETS` does |
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and a wedged SMART pass must not read as a dead host |
| `GATEWAY_HOST`            | which `HOST_TARGETS` name is the ingress the `/gateway` page watches (default `gateway`). It has no URL of its own — the source reads the same agent `/snapshot` the system page reads, projected onto agent 1.2.0's `haproxy`/`nginx`/`sockets` sections — so a name that is not in `HOST_TARGETS` means "no ingress to watch", and the source is unconfigured rather than red. The page refetches itself every 15 s in the browser and has no `POLL_MS_*` knob, because the payload only moves when that one job re-reads the ingress |
| `GUFO_API_KEY` / `VLLM_API_KEY`       | only if an engine is behind a bearer token; they never leave the server |
| `NUT_HOST` / `NUT_PORT`               | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
| `NUT_USER` / `NUT_PASSWORD`           | only needed for instant commands (battery tests)                |
| `NUT_ALLOW_ACTIONS`                   | `1` enables the one write endpoint that dials **anything** — the `/api/auth/*` and `/api/certs/*` writes touch only this process and its `meta` table |
| `NUT_SIMULATE`                        | `1` replaces upsd with a **labelled** simulator (development only) |
| `DASH_READ_ONLY`                      | `1` disables every mutating endpoint regardless of other flags  |
| `DASH_DB`                             | SQLite path (default `data/dashboards.sqlite3`)                 |
| `DASH_ADMIN_PASSWORD`                 | the admin password **seed**, needed only on the first boot of a database — see *Signing in* above |
| `DASH_SESSION_HOURS`                  | session lifetime (default `720` = 30 days)                       |
| `DASH_COOKIE_SECURE`                  | `1` marks the session cookie `Secure`; off because dev is `http://127.0.0.1`, and a request that arrived over `X-Forwarded-Proto: https` is treated as secure either way |
| `DASH_PBKDF2_ITERATIONS`              | verifier cost (default `600000`). The knob exists for the test suite; leave it alone in a deployment. Raising it on a real box does take effect: the next successful sign-in re-hashes a cheaper stored verifier and keeps the signing key |
| `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer. `/disks` and `/gateway` have no `POLL_MS_*` of their own and poll at a fixed 30 s / 15 s: the first payload only moves when the `disks` job re-reads the agents, the second when the one `gateway` job re-reads the ingress. A backgrounded tab idles its polls and fires one on return; nothing an operator presses stops them (phase 21) |
| `DASH_RELOAD`                         | `1` makes `uv run dashboards` watch the source and restart on a change — development only, and every poll loop restarts with it |
| `DEBUGPY` / `DEBUGPY_PORT`            | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup |
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |

A source that is not configured answers `404 {"configured": false}` **without dialling**,
and the page says "not configured" instead of showing a red panel for something you never
installed.

## Architecture

```
agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        │  (three readers: system 10s,     │ + /slots (gufo)        │
        │   disks 5m, gateway 30s)         │                        │
        ▼                                  ▼                        ▼
src/dashboards/collectors.py ────────────────────────────────────────────┐
        │  parse + Δcounter/Δt (rates.py)                                │
        ▼                                                                ▼
sources/host.py · disks.py · gateway.py · llm.py · nut.py   poller.py (one task per source)
        │        samples                    events                      │
        ▼                                                                 ▼
store.py  ── SQLite (WAL): sample(source,host,metric,res,ts) + event + meta ──  api.py  → web/
                                                    ▲
auth.py ── verifier + session key, loaded once at startup (no row and no seed → no server)
certs.py ─ root CA + client cert, minted once at startup (unreadable row → no server)
```

* `poller.py` — every source gets its own asyncio task, its own health record, and its own
  error handling. A dead host keeps its last good values and is labelled stale; it never
  blanks or blocks another source. Disk health is a source of its own (`disks:<host>` in
  `/api/health`, 5-minute cadence, its own fetch timeout) reading the *same* agent
  `/snapshot` the system page reads — a separate task is what keeps a SMART poll out of the
  10-second utilisation poll and lets drive history go stale without blinking `/system`, and
  `alerts.py` is why one dead agent is still one line on the overview rather than two. The gateway
  page's job (phase 29) is the **third** reader of that same `/snapshot`, and one job for the whole
  deployment rather than one per host — `GATEWAY_HOST` names the single ingress, and a job per host
  would ask every machine a question only one of them can answer — on its own 30 s cadence with no
  `RateEstimator`, because a connection count is not a rate. And it never quietly *stops*: the task
  is that source's only poller, so a SQLite write that gets refused is reported (`ok: false`, the
  real reason, the numbers aging, one line in the event log) instead of freezing the source under a
  green dot.
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
  pruned, so a year of history is a few hundred MB rather than a disk emergency. Nothing a
  per-request path asks is answered by a whole-table scan **on every call** (phase 14): the two
  answers that used to be scanned on every call — `series_names()` behind `/api/metrics` and the
  row counts behind `/api/health` — come from a 60 s / 30 s cache, so the scan is paid once per
  window (its cost is the honest remainder in the known-gaps list) instead of once per request. `series_names`' is ended only by a write that can
  change it: a `(source, host, metric)` the store has never written, or any `DELETE`. A poll of an
  existing series leaves it alone, which is what makes the cache serve at all on a store every
  poller writes to (task 04's live watch is the proof: 535 ms per call before, 1.8 ms median after), and the
  counts age on the clock. A chart's resolution comes from one capped compound statement instead of
  three full `COUNT`s per series. `last_maintain` and `size_bytes` are deliberately never cached —
  the first is how an operator tells a stalled rollup loop from a healthy one.
  The write side is windowed too: `maintain()` rolls up only the buckets that became due since its
  last run (read from `last_maintain`, overlapped by one bucket and snapped down to a bucket
  boundary, so a gap in polling is a catch-up scan rather than missing charts) and prunes on its
  full retention bound, one statement per resolution. It ends with `PRAGMA optimize` — the store's
  only statistics refresh, deliberately not inside a poller's commit — and the WAL is kept honest by
  `wal_autocheckpoint` plus a 64 MB `journal_size_limit` cap. On a 6.4 M-row store that took the
  once-a-minute write transaction from 5.4 s to a few milliseconds, which is what stopped a poller's
  insert losing its 5 s `busy_timeout` (known gap 4).
  Its `meta` table is the one place state that must outlive a restart goes (`get_meta`/
  `set_meta`/`delete_meta`). Its users are the admin credential below and, since phase 15, the
  certificate authority in `certs.py` — five `certs.*` rows and nothing else.
* `auth.py` — the admin login's decisions, in stdlib and without HTTP: a self-describing
  PBKDF2-HMAC-SHA256 verifier, an HMAC-signed stateless session cookie (no session table, and
  rotating the stored key is what evicts every other browser), a session-bound CSRF token, and
  an in-process login backoff. `load_auth_state()` is the startup half: the environment seeds a
  password **once**, the stored verifier wins from then on, and a server with neither refuses to
  start (see *Signing in*).
* `certs.py` — the fleet's certificate authority, in the same shape as `auth.py`: a pure lifecycle
  over the `meta` table, no clock it cannot be handed, no logging, and one fixed-text raise. The
  first boot of a database mints a **root CA** (`certs.root.key`/`.cert`, self-signed
  `CN=dashboards-root`, 10 years) and the dashboard's own **client** certificate
  (`certs.client.key`/`.cert`, `CN=dashboards`, 1 year); every later boot reads those rows and
  regenerates nothing, because a root that changes on restart untrusts every certificate already
  installed on a host. `issue_agent_cert`/`reissue_agent_cert` sign one server certificate per
  `HOST_TARGETS` name (`CN` = the host, SAN = the addresses it is dialled by) into
  `certs.agents`, and each entry's **bundle** — the agent's cert, the agent's key, the root *cert*
  (never the root key), a `# BUNDLE-REV <n>` line and the current `# REVOKED <serial>` lines — is
  stored verbatim so the bytes an installer places are the bytes the row holds. `revoke` marks the
  host and re-seals every bundle, because the denylist lives inside each of them; `rotate_root`
  replaces the root, the client, and every agent at once. A stored PEM that will not parse (or a
  client leaf the stored root did not sign) is a **fail-closed startup**, the admin-password rule:
  the message is `sqlite3 "<db>" "DELETE FROM meta WHERE key LIKE 'certs.%'"`, after which every
  agent needs a fresh bundle. Every certificate carries a `SubjectKeyIdentifier` and every leaf an
  `AuthorityKeyIdentifier` naming that root, because OpenSSL's strict mode — the one Python's
  `ssl.create_default_context()` turns on, which is what the dial below builds — refuses a chain
  without them. The **dial** is `collectors.AgentTls` (task 03): with an `https://` `HOST_TARGETS`
  entry, the fetch verifies the agent against this root and presents this dashboard's own client
  certificate, so only this dashboard can read the host; a host marked `revoked` fails its `host:`
  and `disks:` jobs with `agent certificate revoked` *before any socket opens*, and because the
  status is a fresh `certs.agents` read on every poll, revoking takes effect on the next poll with
  no restart. An `http://` entry is untouched — same shared client, same bearer token — so the
  rollout is per host. The private keys are credentials like any password: never served, never
  logged, and the root key never leaves the row that holds it.
* `severity.py` — the single green/amber/red table (`THRESHOLDS`, plus `INVERTED` for the
  metrics where lower is worse). Rated values in the API payloads travel with a sibling
  `severity` (`cpu.severity.usage_pct`, `memory.severity`, `disks.severity`, a UPS card's
  `severity.charge_pct`, `severity` inside the rated LLM
  cells such as KV pressure, `/api/summary` rows), computed server-side, so every page
  colours the same way and a value nobody reported renders `n/a` instead of green. A metric
  published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays
  what the upstream published. There is no `gpu.*` row: these are LLM servers, a busy GPU
  is the job, so the GPU card (and the fleet GPU column) renders uncoloured at every
  utilisation — the `gpu.*` history series are still stored and charted. Drive health gets two
  rows: `disk.temp_c` (amber 50 °C, red 70 °C) and `disk.percentage_used` (amber 80 %, red
  100 % — an SSD that has spent its rated wear is a replacement). The sector counters
  (`disk.reallocated`, `disk.pending`, `disk.uncorrectable`) and `disk.power_on_hours` have
  **no row on purpose**: a non-zero counter is an alarm chip on presence, zero is the absence
  of a chip and never a green "healthy drive", and an odometer has no bad value.
* `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states,
  host_payloads, engine_payloads, ups_payload, recent_events, *, gateway_payload=None)` is a pure
  projection of the state the poller already keeps plus the verdicts `severity.py` and the UPS health
  checks already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as
  `alerts`, worst first: `{id, severity, source, message, href, at}`. A configured source
  that stopped answering is `fail`; one that is stale but still serving its last good value
  is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than
  one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a
  marker that says how many it hid. A job that dials an agent another job already named adds no
  line either (`AGENT_ECHO_DASHBOARDS`: `disks` and, since phase 29, `gateway` — three jobs read
  one `/snapshot`, so one dead box is one overview row), while a job failing *while its host
  answers* keeps its own voice and links to the page that explains it (`/disks` for a wedged SMART
  pass, `/gateway` for a proxy or socket section that went quiet). Staleness is measured in each
  source's own interval, so the gateway page goes amber after 90 quiet seconds and not after 30.
  The keyword-only `gateway_payload` (phase 30 task 03) is the one non-reachability voice this
  module has for that source: a non-empty `warnings` list — the two config files not connecting to
  themselves — is **one** `warn` row (`N proxy config problem(s) to address — <first message>`,
  `href` `/gateway`), because a dangling backend is a structural fact and not a graded number, and
  it is emitted only while the job is configured, `ok` and not stale, so a router that has stopped
  answering keeps its single reachability line instead of growing a second one about a reading the
  operator cannot refresh. No count of connections can produce a row here at all.
* `api.py` — the JSON every page reads: `/api/config`, `/api/health`, `/api/events`,
  `/api/history`, `/api/metrics`, `/api/summary`, one `snapshot` + `history` pair per
  dashboard (`/api/system/*`, `/api/disks/*`, `/api/gateway/*`, `/api/llm/*`, `/api/ups/*` —
  including `/api/ups/commands`, which answers what may be sent to each UPS right now and why not
  when it may not), and the certificate authority's own surface (`/api/certs`,
  `/api/certs/<name>/bundle`, `POST …/rotate`, `POST
  …/revoke`, `POST /api/certs/root/rotate`). GETs everywhere except `POST /api/ups/command`,
  which stays the **only** route that dials anything: the `/api/certs/*` writes are `meta` row
  writes through `certs.py`, the same class of write `POST /api/auth/password` is, and a
  revocation's whole effect is that the next poll refuses to dial (see `certs.py` below).
  Reads there are pure — `certs.read_state` is the request-time read and `certs.ensure_certs` the
  startup bootstrap — so refreshing the list cannot mint a certificate, and it reports the
  **stored** authority even after a root rotation the running process has not picked up yet. Two rules make the pages legible: a source with
  no env config answers `404 {"configured": false}` **without dialling** (so a page says
  "not configured" instead of going red), and a history query for a series that does not
  exist answers an empty series instead of an error (so a card asks for every plot it could
  draw and reads `n/a` for the ones the drive never reported). The five `history` routes take
  one batched door into the store (`api._history_batch` → `store.history_many`), so a page's
  chart request is one grouped resolution query plus one index seek per series rather than four
  queries per series — and the JSON it answers with is the same document the per-series loop
  produced, series for series and in the same order (`tests/test_api.py` pins that against
  points read straight out of the store with `query()`). `/api/disks/snapshot` is the
  drive-health surface: one entry per host **with drives to show** — its physical disk rows,
  the agent's `smart` block verbatim, `smartctl_available` (`true` / `false` = install
  smartmontools / `null` = the agent never said), a `severity` beside each row's rated
  numbers, and a `fleet` block summed here so the page does no arithmetic — a host that has
  not answered yet is listed as an `unreachable` stub of `n/a`s and counted, never summed.
  The one host that is **not** listed is the one whose agent said `false`: a box with no
  smartmontools (a VM with nothing behind `vda`) has no card, no fleet contribution and no
  `/api/summary` row, because the only thing its card could say is "install it". `null` and
  an `unreachable` stub both stay — an agent that cannot say is not an agent that said *no*
  (`api._disk_visible` is the single predicate, pinned both ways in `tests/test_api.py`; the
  `disks:<host>` job, its health record and its stored `disk.*` series still see the host).
  `/api/config` carries
  `disks.interval_seconds`, which is why a drive card four minutes old is behaving, not
  broken. `/api/summary` grew a `disks` block with that page (the same `fleet` sums plus one row
  per host — counts, `smartctl_available`, and `failed_devices`, so a notifier learns *which*
  drive the hardware called `FAILED` without a second call); it is additive, exactly as `alerts`
  was, and the key paths of both are pinned with and without targets configured. Since phase 22
  that block lists the **same hosts** `/api/disks/snapshot` does, and `fleet.hosts_total` is
  that row count, so `/` and `/disks` can never describe two different fleets. `/api/summary` grew a
  `gateway` block the same way in phase 29, reshaped by phase 30 task 02 (`configured`, `host`,
  `href`, `supported`, `unreachable`, `listeners`, `connections`, `live_in`, `live_out`,
  `map_targets`, `states`) and given `warnings` by task 03: what the configs declare (`listeners`,
  `connections`, `map_targets`, `warnings`) beside what the sockets are doing (`live_in`, `live_out`
  — the latter summed over **distinct endpoints**, since several routes reaching one address is one
  count and not two). `null` and never `0` in every arm that did not measure — **except**
  `warnings`, which is a config fact and so reads a real `0` when the two files connect to
  themselves. No `severity` anywhere: a connection count is not a graded number, and a structural
  fault needs no threshold because the hardware of a config file either names a thing that exists or
  it does not. The `unmatched` / `unmatched_listeners` / `partial` fields went out with the
  guessing rule they existed to explain. `/api/config` carries the matching row, including
  `gateway.interval_seconds`/`timeout_seconds` and the named target's redacted `url`.
  `/api/gateway/*` (phase 29 task 03) is that page's pair. `/snapshot` is one host's whole
  topology — the haproxy frontends and the nginx `listen`/`map` echo the pipes were built from,
  the `listeners` (port, its domains, its live connection count, up to 32 client addresses), the
  `connections` graph (phase 30: one row per route the two config files declare, each naming its
  inbound port, its outbound endpoint, and the config's own selector for it — a haproxy backend
  name, an nginx `server_name`, a stream `map` key) with its **live count** — the socket table's
  ESTABLISHED rows, inbound by the socket's local port and outbound where a remote
  `(address, port)` equals an endpoint the join can name — a target the config wrote as an address, or
  (phase 32 task 03) one of the addresses the gateway's own `resolver` section published for the name
  it wrote. Each row carries the `metric` it
  is stored under, and `shared: true` when more than one route reaches its endpoint (one count, shown
  twice, never split). `supported`/`sections`/`why` cover the degradation arms. `warnings` (phase 30
  task 03) is the list of the two config files **not connecting to themselves** — seven kinds, each
  one row of `{id, kind, subject, message}` where `id` is `"<kind>:<subject>"` and the message is the
  sentence the page prints: `haproxy_route_dangling`, `haproxy_backend_orphan`,
  `haproxy_backend_empty`, `nginx_server_unlistened`, `nginx_pool_undeclared`,
  `nginx_variable_unmapped`, `nginx_map_route_missing`. Nothing is inferred to fill a gap, so three
  shapes are deliberately **silent**: a vhost that listens and passes nothing (a `return 301` looks
  identical here), a frontend that routes by a fetch expression (agent 1.2.0 reports "no name", which
  is the same value "no routing directive" has), and an entry of a `map` no `proxy_pass` names (the
  `$connection_upgrade` map's `upgrade`/`close` are header words, not missing routes). An unreadable
  or absent section warns about nothing — a host that cannot be read has not been found guilty of
  anything — and a `warnings: []` is therefore the truth, not a shrug. Non-zero, `/` gets **one**
  amber row (`N proxy config problems to address — <first message>`) and this page gets the list. **Nothing reconciles
  to `sockets.established` and that is the design**: a connection to a target nothing can name (a
  name with no resolver entry behind it) or a socket on a port no config names is ignored — not
  counted, not totalled, not warned about — so a pipe count means "this route has N live connections"
  and nothing more. The payload also carries the agent's `resolver` block (`available`, `via`, and
  the projected `entries`: name, port, status, addresses, ts — nothing an operator has not reviewed)
  beside the `sections` pair that already says which halves answered, so a dash on one line is
  explainable from the document the page already holds. On the recorded
  ingress that section places every declared route, which makes the dash the exception rather than
  the normal look of this page, and its meaning is *this name has no record* — and on the live fleet
  it is the exception too: the gateway has run agent **1.3.0** since 2026-10-25, `resolver.available`
  answers `true` with one entry per declared target, and that host's page counted **33 of 33** routes
  with no dash at all. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
  a `404 {"configured": false}` answered before any client is built (both arms spy-pinned), a host
  that has never answered is the `unreachable` stub whose every count is `null`, and a 1.1.0 agent
  is `supported: false` plus its one `why` sentence — three arms, no red among them (invariant 3).
  `/history` serves the `gateway.in_conns@<port>` and `gateway.out_conns@<host>:<port>` gauges —
  the port in the outbound name because two services on one host are two facts, and only ever for an
  endpoint whose address is known: one the config wrote, or (since phase 32 task 03) one the gateway
  resolved the name to — with `hosts=`
  defaulting to the ingress alone, unknown names answering as empty series, and no dial either way
  because history reads only SQLite.
* `web/` — six static dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/gateway`) plus
  the three admin pages (`/login`, `/settings`, `/certs`), one stylesheet, hand-rolled
  SVG charts (`charts.js`), and a masthead status dot per dashboard job on **every** signed-in
  page — the two admin pages included (`disks` has its own:
  the same agent can answer the 10-second utilisation poll and have a wedged SMART pass, and
  `gateway` has one because its job is its own poll of one ingress).
  The masthead is **three rows at every width, the two control rows flush right**: the brand and the six dashboard
  tabs, then the per-source indicators on their own row, then the `⚷ Certs` / `⚙ Account` /
  `☾ Dark` / clock controls on a row under those, in that order, so the **time is the farthest-right
  thing on the bar** (phase 25: the badges and the cluster were re-justified right, then the clock
  moved to the end of the markup — `flex-end` makes the last child the rightmost item, and it stays
  a markup fact rather than a CSS `order:` so the pin can read it; the clock is hidden below 900 px
  as it has always been, being a readout and not a control). Only `/login` renders no
  indicators, and so two rows, and a control is never hidden to make a
  narrower width work. There is no control that stops the polling (phase 21): the bar used to carry
  a `⏸ Pause` button with a `Space` shortcut, and it left the numbers on screen looking current with
  no `stale` label on them. Both admin links are on all eight
  signed-in pages — the six dashboards plus `/settings` and `/certs`, the two they name included —
  and not on `/login`, the ninth page and the only anonymous one: the cluster is the same wherever
  you are, and a link to the page you are already on is a harmless no-op rather than chrome that
  vanishes. The bar is **not sticky**: it scrolls away
  with the page, so every pixel of height goes to the data (it stays a positioned element, which is
  what keeps the mobile dropdown anchored under it). At ≤640px the six tab links leave the bar
  and come back as a dropdown under the whole of it, opened by the `☰` button (`/login` renders no
  tabs, so it has no button either), and that menu **slides**: 180 ms of fade plus an 8 px rise on
  the way out and the same on the way back in, with nothing hidden from a keyboard or a screen
  reader while it is shut, and no motion at all when the system asks for none
  (`prefers-reduced-motion`). The same ≤640px block is what makes **every** table scroll sideways
  inside its own card (phase 27): `table.data` becomes a horizontal scroll container, so the
  columns past a phone's 320 px — `Worst disk`, `GPU busy`, `Temp`, `Up` on the fleet table — are
  reachable by dragging the table instead of being clipped by the card, while the document itself
  still never scrolls sideways and a desktop table that fits shows no scrollbar. Two more things
  were clipped by that same card edge and are fixed at **every** width: a card header wraps onto a
  second line when its facts do not fit on one (the OS/model chip and the `updated 5s` age stamp
  used to be painted past the card and eaten), and a collapsed `Charts` / `More detail` disclosure
  scrolls whatever it reveals sideways, which is what puts `/system`'s GPU `VRAM`/`GTT` columns
  inside their card instead of under the neighbouring one. Nothing is hidden for the arithmetic:
  the row wraps, the disclosure scrolls, and every number stays on screen.
  No framework, no CDN, no bundler, and no narration: a page never
  explains its own pipeline, so the words on screen are the ones you act on — a unit, a `Δ<window>` provenance tag, a
  `SIMULATED` badge, or the sentence beside an empty panel. The overview (`/`) is exactly two panels — a
  **Warnings & errors** list built from `/api/summary`'s `alerts` (each row links to the
  dashboard that explains it) and the **Fleet** table — and nothing else: an empty alert
  list reads as an explicit "all clear", and a live problem moves the panel above the
  table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the
  fleet never quietly gets smaller and last good values are never mistaken for live ones.
  Under that table is the ingress's **one line** (`gateway · 10 listeners · 33 connections · 17 live
  in · 2 live out` as the live router answered on 2026-10-23 — a reading, the same way a listener count
  is, and only the field names behind it are a contract), linked to `/gateway`: every figure is `/api/summary`'s own, a count the source
  did not measure prints `n/a` rather than `0`, and a deployment with no `GATEWAY_HOST` renders no
  line at all. A non-zero `warnings` appends ` · N to address` and nothing else — `0` and `null` both
  print nothing, so a clean ingress never grows a permanent sentence about the checks that did not
  fire (phase 30 task 03).
  A big colour-coded number is `Dash.hero(el, {...})` on the
  `.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
  `Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
  with no verdict renders `n/a` with no colour at all. A battery is
  `DashCharts.battery(el, {charge, severity, state, label})`: one inline SVG whose fill *is* the
  charge — clamped to 0–100, never rounded up, clipped to the plate so 0 % is an empty shell —
  coloured only by the payload verdict, with a shape per state (solid shell on line, bolt
  charging, dashed outline on battery, a dash through the body when there is no reading) and a
  `<title>` / `aria-label` that names it ("apc: Charge 100 percent, on line") unless the number
  beside it already does. Each `/system` host is one panel led
  by six hero slots — CPU %, MEM %, TEMP, DISK % (mount underneath), GPU busy, and NET on two
  lines (↓ in / ↑ out, never coloured: a byte rate has no good/bad). A host with **no GPU**
  renders **five**: the GPU slot is omitted rather than filled with `n/a`, because a router is
  not broken for the card it never had. A GPU is one busy hero plus **four labelled values**
  under it — **Model**, **VRAM** (`used / total`), **Pool** (the honest APU view) and **Temp**
  — instead of the dense `model · VRAM · pool · °C` footnote that was 11px mono; a field the
  agent did not publish prints `n/a` in its own cell, never `0 °C`, and nothing in the slot is
  coloured or barred (a busy GPU is the job on an LLM server). Missing *numbers* still print
  `n/a` uncoloured — gateway's CPU temp — which is a different fact from a missing *device*.
  Under the numbers, **each panel charts only its own host** — behind a
  collapsed **Charts** toggle, because the default view is current stats and a fleet of ten
  hosts each drawing three plots is noise: the summary reads `▸ Charts` with the window in it
  (`1h`, `24h` …), and opening it is what builds and paints the charts — instantly, from the
  response the last refresh already fetched, and repainted on every refresh after that while it
  stays open. Expanding one panel never stretches its collapsed siblings
  (`.grid.items-start`). The stack is utilisation
  (CPU, memory, worst disk), network (↓ in / ↑ out), and GPU busy for a host that reports
  cards — three 120px SVG charts, each with its own legend and its own `aria-label` naming the
  host and the plotted metrics (`LineChart`'s `ariaLabel` option), never a second host in sight. The
  fleet-wide charts, the nine-figure fleet-totals panel and the host/metric detail picker are gone, and
  so are the per-core charts, while `cpu.usage_pct@coreN` stays in SQLite. Everything else —
  load, clock, disk I/O, uptime, GPU power and the per-label temp breakdown, filesystems,
  processes — folds into a collapsed
  "More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request
  (`hosts=<that host>&metrics=…&range=…`) every refresh, stack open or not, and a host that stops
  reporting takes its panel, its Charts toggle and its charts off the page.
  `/disks` is the drive-health page: **one card per physical drive** (the agent's
  whole-`/sys/block` list — never a partition, never a mount), led by the device path and the
  drive's own verdict word (`OK` / `FAILED` / `unknown`, coloured only by `severity.health`), then
  three equal cells — **Temp**, **Power-on** hours, **Wear** — coloured by `disk.temp_c` and
  `disk.percentage_used` from the API, with the odometer deliberately uncoloured because
  `severity.py` rates no power-on hours. The sector counters appear as **red chips that exist only
  while non-zero**, so a healthy drive shows no alarm row at all rather than a wall of zeroes.
  A host whose SMART cannot be read keeps its cards, dimmed with every number `n/a` and one
  `.banner` line saying why — the agent's own sentence when it reported something, and "not
  answering" when the agent is down — never a red panel, because unreadable drives are
  unconfigured, not broken. The one case that gets no card at all is the agent that said
  `smartctl` is **not installed**: such a host is absent from the page's host cards (and from
  the API's host lists) rather than a card whose only content is "install smartmontools" — its
  `disks:<host>` job still polls, still reports `ok`, and its transitions still reach the drive
  events feed below the cards, because that feed is the job's history and not a host list. The header's
  drive counts are the API's `fleet` block (hidden below two hosts, so one box's sums never
  duplicate its own cards), and each host's **Charts** disclosure behaves exactly like `/system`'s:
  collapsed, built only while open, and fed by one `/api/disks/history` request per host per
  refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive it has.
  `/llm` opens with
  the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, the note
  *sums only engines answering now*, then **running / waiting / prompt tok/s / decode tok/s /
  total tokens** — and every number on it is `fleet_totals()`, so the page sums nothing itself.
  A dead engine keeps its last-good numbers in its own (stale-labelled) panel but is dropped
  from a total presented as current: the tag counts what answered, names the missing engine in
  its tooltip and dims while the sums are partial. Nothing on the block is coloured —
  `severity.py` rates no throughput and no queue depth, because 0 tok/s on an idle fleet is
  healthy — the dot is reachability, and `fleet.seams` (how the two engines count prompt
  tokens differently) is a tooltip on the prompt sum rather than another line of ink. Below
  two configured engines the block is hidden: a cross-engine sum of one engine duplicates that
  engine's panel. Below the fleet block — with the page's shared `.85rem` gap, as everywhere else
  in the section stack — sits the **stats strip**: one card per configured
  engine — reachability dot, name, age stamp, the **KV-cache bar** (the one rated cell: the
  fill and a breach's text take their colour from the payload's `severity` and nothing else),
  then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ2.0s` at the default cadence —
  the LLM window is one poll interval, phase 23 — scraped, estimated), `total tokens`, the sessions this engine reports (`slots busy` counted from
  gufo's `/slots`, or `running`/`waiting` for vLLM, plus `waiting`), `prefix cache hit`,
  `spec accept` (gufo), `queue time` with its `p95 …` foot (vLLM) — and red alarm chips for
  preemptions / GPU-context losses that exist only while non-zero. Then the engine panels
  hold the detail the strip does not print: two big
  throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is
  healthy) with their provenance tag, and the latency rows (TTFT, TPOT) each with a `p95 …`
  foot line and its own sparkline over the selected window. Under the gufo panel's
  **`sessions (GET /slots)`** label sits the **slot grid**: one card per parallel slot, keyed by
  the slot's own id, created once and patched in place so a busy slot never flickers — request
  id, prompt tokens (with `processed …` while prefill runs), cache hit and its share of the
  prompt, generated tokens, remaining, context, and the model beside the slot number. A busy
  card is a border, a glyph and the word `processing`, never a green/amber/red verdict (a slot
  generating tokens is the job). Every cell reads exactly one key of the captured `/slots`
  shape — no `a ?? b` fallback chains — so a field rename fails a contract test instead of
  printing dashes, and `-1` sentinels print `idle`/`n/a`, never a negative countdown. When
  `/slots` does not answer, the panel says so with the API's own sentence rather than rendering
  a card of nothing, and vLLM's panel carries one line (*exposes no per-slot JSON*) and no grid.
  No row appears in both places —
  one value, one home — and there is no peak-context cell, because this page prints a number
  the upstream published or `n/a`. The four trend charts sit below the panels
  at half height, and one `/api/llm/history` request per refresh feeds all of it. With a
  single engine configured the page is one clean card in the strip and one clean panel —
  never a red half for a server you do not run. Each UPS card (`/ups`) opens with its battery glyph and
  three big numbers — **charge %**, **load %**, **runtime left** — coloured from the card's
  `severity` dict, with the electrical readings (line in, output, battery volts and amps,
  battery °C) kept below them in smaller muted type because they are what you read *while* the
  UPS is on battery. Health checks, the battery-test buttons (with their `DANGER`
  confirmation) and the NUT variable dump are unchanged, just second. The `Last self-test`
  check grades the **outcome** of the driver's word and not its phrasing: a pass
  (`Done and passed`, the sentence this fleet's CyberPower answers) reads `ok` and raises no
  overview warning, a fail reads `fail`, and a pack that has never been tested —
  `No test initiated` or no variable at all — says the same "run a battery test" sentence as
  `unknown`. Only a word the map has never seen stays a warning, printed verbatim so it can
  be added.
  `/gateway` answers the topology question `/system` cannot: which ports on the ingress are open,
  what routes out of each, and how many connections are live on each. It is **two panels** — one per
  config file, `nginx` then `haproxy` — each hidden until a poll actually has rows for it, so a router
  with no nginx in it spends no panel saying so. Inside a panel the rows are **groups**: a *trunk* is
  one port that file binds, and hanging off it are the *leaves* — the routes that port's config names,
  each reading **inbound first**: line 1 is the config's own word for who reaches the endpoint (a
  `server_name`, a haproxy `backend` name, a stream `map` key, the literal `default`), and line 2 is
  the counterpart it flows to, **exactly as the config wrote it**, marked `→ host:port` with the kind
  of route after it. Both ends are verbatim, and both are carried whole on the row's tooltip and the
  line's accessible name whichever one CSS has to ellipsise. That is the shape of the file, which is
  why the stream
  `map` is not a third panel: its entries are the nginx panel's `:443` leaves, so every SNI domain
  reads beside the port it reaches. A trunk that names no target — this fleet's `:80` `return 301` —
  prints "no target named in the config" instead of drawing nothing. Each line is joined to its
  counterparts: nginx's ports **branch** — a dashed spine down the fan-out with a tick into every
  endpoint — and haproxy's one-`server` frontends run **straight across**, because a port with one
  route behind it is not a branch. A route is **green because the config declares it**, an existence
  colour of the page's own, and that green covers **every segment of the row** — spine, tick, line and
  switch all read one class on the row, so no part of an active route is drawn as grey plumbing — while
  the line **animates only while it carries at least one live connection**: motion is the socket,
  colour is the config, and a resting route keeps the same green as a busy one. The trunk carries a
  **short line of its own** for that reason, and it reads from a *different* field: a leaf moves on its
  endpoint's count, the trunk's stub moves on `listeners[].connections` — the sockets whose **local**
  end is that port, which `/proc` states whether or not the route's target is a resolvable address. So
  a busy port is visibly busy even when every leaf under it carries a `—`, which is the normal look of
  this fleet's `:443`. That line's claim is about **the port and never about one route**: its
  accessible name and tooltip say `:4443 — 9 connections are open on this port right now` (or
  `no connections…`, or `connections on this port are unknown: the socket table did not answer`) and
  name no endpoint, and the unknown arm prints `n/a` where the count is and no motion at all — never
  `0`, and never a dash, because "the socket table went quiet" is a different fact from "this target
  is a name".
  Above the map, one **filter box** finds a domain among 33 routes (`stackexpected`, `brain`, `4443`,
  `minecraft`) because the rows are in config order and the question is always about one name. It hides
  what the payload already sent and asks for **nothing**: no request, no re-derivation, no new route.
  It matches the strings the payload carries — the inbound word(s), the endpoint, the port, the haproxy
  `server` name, the **addresses the gateway resolved** for a name target, and a trunk's own domains
  and client addresses — case-insensitively, so a label CSS ellipsised still matches its whole text and
  typing `10.1.2.64` finds the route whose target that address belongs to. A trunk that matches keeps its **whole fan-out**: typing
  `4443` asks for the port, and every route hanging off it is the answer. `Escape` clears it, a chip
  beside the field counts `N of M routes`, and a query that matches nothing leaves both panels standing
  with `no route matches "…"` in each. What it never touches is the header's four tallies, the warnings
  panel, or the charts — those describe the ingress, not the operator's current view of it — and it
  never **groups, sorts or colours** by domain suffix: `reeseapps.com` versus `stackexpected.com` is a
  fact neither config file states, so the operator types the suffix and the page hides the rest rather
  than inventing a zone no proxy declared.
  **Nothing on the page
  is coloured by a verdict** — a busy ingress is the job, so `severity.py` rates no `gateway.*` metric
  and no number gets a green/amber/red tint. A line prints `—`, never `0`, and its tooltip and its
  accessible name say **which** unknown it is: *the gateway could not resolve this name in this
  reading* when nothing is known for the endpoint, or *the socket table did not answer this reading*
  when an address is known and nothing was measured — two different things to go and fix. A resolved
  endpoint with nothing open is neither of them: it prints `0`, because that is a measurement. A
  counted line carries its provenance on hover (`… · address from libc, resolved 3m`), and a line the
  config wrote as an address claims no resolution at all. A dash on a domain-named route now means the
  gateway had **no record** for that name in this reading (`nxdomain`/`error`, or no `resolver` section
  at all on a pre-1.3.0 agent) rather than "this page cannot count names": the ingress here runs
  **1.3.0**, so all 33 of its declared routes carry a number. The
  footer says the same thing once, in plain terms, and points at the hover. The header's four tallies are `ports`, `connections`, `live` and
  `warnings`, every one of them the payload's own figure. A `warnings` count above zero raises a panel
  above the map with the config-integrity rows (`a backend no frontend routes to`, a `proxy_pass` with
  no `upstream` behind it, an SNI entry that reaches no address) in the overview's alert vocabulary;
  the live ingress here is clean, so that panel is off. Each panel's trend is
  behind a collapsed **Charts** toggle like `/system`'s, one `/api/gateway/history` request per
  panel per refresh. With no `GATEWAY_HOST` the page is the setup sentence rather than a red panel,
  and an ingress running an agent older than **1.2.0** says so in the agent's own words with every
  count `n/a` and nothing coloured. Each line's head carries the sketched **switch** — a real
  `role="switch"` button, green and to the position because the route is in the config, named in full
  for a screen reader (`endpoint, reached by <selector>, in the nginx config, on`) — and it is
  **unwired on purpose**: `aria-disabled` with a tooltip that says so, and this app writes to no
  upstream, so a route is disabled by editing `haproxy.cfg` or an nginx `server` block and by nothing
  in a browser. Giving either of those two dashes or knobs a real backing store is deliberately **not**
  documented here as a thing you can have: name-based liveness needs the agent to report resolved
  targets (a future agent version), and switching a route needs a second gated write path in front of
  the proxy configs, which is a change to this app's read-only promise rather than a feature. What runs
  today is **agent 1.3.0** — name resolution on the gateway, counts on every declared route, and no
  switching.
  The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event`
  table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a
  page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath.
  One helper, `Dash.eventsPager()`, renders every feed — five lists today, because `/ups` splits
  its one fetch in two — and each keeps its own row markup; the fetch
  asks `/api/events` for 200 so there is a page 2 to reach, and the endpoint itself is
  unchanged. Reading page 3 survives a poll; a refresh that brings new events goes back to
  page 1, because page 3 of a list that just changed is not the history you were reading.
  The `/ups` feed is also **expandable**: its card is a `Power events` summary line, closed until
  you open it, and the list inside is the same bounded, paged window. What sits behind that
  summary is only the *power* half of the feed — `status` transitions (the power loss and its
  return), self-test results, and the commands you sent — because the rest of it is connection
  trouble (`NUT server NutError: cannot reach upsd …` and its recovery line), which is not
  history to tuck away. That error half is its own **Connection & errors** card above the
  summary — the same bounded, paged window, the same row shape — hidden while it has nothing to
  say and appearing with its first unreachable/reachable line, so a poll failure is readable
  without opening anything. That is also why the feed is fetched on every poll rather than only
  while the summary is open: half of it is always on screen, and the two lists page
  independently of each other.

## Testing

```bash
uv run pytest                                  # unit + API tests, fully mocked, no network
uv run pytest --cov                            # coverage gate
set -a; . ./.env.dev; set +a                 # the live suite reads the same env as the app
uv run pytest -m live --live                   # optional: real fleet / engines / upsd — an
                                               # `https://` HOST_TARGETS entry is dialled with the
                                               # authority in DASH_DB (read-only), so a box with no
                                               # store skips the host tests instead of failing them
uv run pytest -m store_perf --store-perf -s    # optional: the store's hot-path timings (phase 14)
uv run python scripts/store_perf_probe.py      # the same measurements as a report in data/logs/
DASH_WATCH_SECONDS=1500 uv run python scripts/watch-store-perf.py
                                               # ... and the live watch: 25 signed-in minutes against
                                               # the running dev server, counting store-error and WAL
DASH_LLM_WATCH_SECONDS=780 uv run python scripts/watch-llm-window.py
                                               # ... and the LLM Δ-window watch (phase 23): signed-in
                                               # reads of /api/llm/snapshot at the app's own cadence,
                                               # judging every slot start/end against "settled within
                                               # two polls", the Δ<window> labels and the 1 m rollups.
                                               # Read-only: it starts no workload and writes nothing.
DASH_LLM_WATCH_REPLAY=data/logs/llm-window-watch.tsv \
  uv run python scripts/watch-llm-window.py    # ... or re-derive that verdict from a captured pass,
                                               # over the same samples, with no traffic sent
uv run ruff check src tests agent && uv run pyright            # lint + types
./scripts/capture-fixtures.sh                  # re-record ground truth: host agents, UPS replies, gufo /slots
./scripts/capture-fixtures.sh --with-llm        # ... and the engines' /metrics while they are up
SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh
                                               # ... and smartctl per physical disk: plain smartctl,
                                               # `sudo -n` if that is denied; scrubs serials/WWNs
./scripts/capture-gateway-config.sh            # ... and the gateway's proxy tree + socket tables:
                                               # haproxy.cfg, nginx.conf + its glob includes,
                                               # /proc/net/tcp{,6} over $GATEWAY_SSH (gateway-ops).
                                               # World-readable paths only; a failed read never
                                               # replaces a good fixture
bash .agents/validate.sh                       # all of the above + `node --check` on every page
                                               # script + repo hygiene, with the 94% coverage floor
```

Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must
match an upstream (`/proc` layouts, vLLM metric names, NUT protocol framing) is asserted
against a verbatim capture in `tests/fixtures/`; behaviour that needs a failure — upsd
refusing a command, an engine returning 500, a host rebooting — is a fake. The `live` tests
(`tests/test_live_fleet.py`) are the bridge: they SKIP unless you pass `--live`, each one
also skips when its source is unset (**source `.env.dev` first** — run bare and they all skip
quietly, which reads exactly like a pass), they drive the production code paths
(`collectors.*`, `NutMonitor.collect`) against whatever hardware is reachable, and they never
write — a battery test on live hardware is not a unit test.

`tests/auth_contract.py` is two constants and the reason for them: every test that starts a
server seeds the admin password `testpass` (a server with no stored verifier and no seed
refuses to start, so a test that wants a running app has to give it a credential the way
`.env.dev` does) and hashes it at 1 000 rounds through that same `pbkdf2_iterations` setting
instead of the real 600 000 — ~0.3 s per hash times a hundred-odd servers is not a slower suite,
it is a suite nobody runs. The real cost is pinned where it belongs
(`test_the_default_cost_is_the_owasp_figure` and the config default test), so lowering it here
cannot lower it there. Its partner is `tests/auth_client.py`: `sign_in(client)` posts that
password once, asserts the sign-in worked, and puts the returned CSRF token on the client's
default headers — which is what keeps the ~200 pre-existing HTTP assertions asserting the
behaviour they were written for instead of the new gate.

`tests/test_auth_api.py` is the perimeter, and it is deliberately awkward in one place: every
anonymous request runs with `follow_redirects=False`, because a client that follows them would
fetch the login page and report `200` for a route that refused it — the way a gated suite ends
up passing while testing nothing. It also walks the *served* route table
(`test_every_route_that_is_not_public_requires_a_session`), so a route a later phase forgets to
protect fails there rather than in a browser, and it proves the one refusal that matters most on
the wire rather than on a status code: a `POST /api/ups/command` without the CSRF token answers
`403` **and the scripted upsd records nothing**, while the same request with the token still
reaches it — and `driver.killpower` still needs the word `DANGER` behind the session, because a
fifth gate that quietly replaced the fourth would be the worst kind of regression.

`tests/test_auth_docs.py` tests the paperwork, because the paperwork is what an operator holds
when the dashboard will not come up. It reads `.env.example` and `.env.dev.example` and demands
that every `DASH_*` auth setting is declared there and that no secret-shaped **assignment** carries
anything but a placeholder (the same allow-list `bash .agents/validate.sh` enforces through
`git ls-files`, restated so a plain `uv run pytest` catches it on the working tree) — while the
local-testing word stays legal in prose, which is the only place it can be useful. It demands that
`scripts/dev-server.sh` still gets its readiness probe and its `status` from the public
`/api/health` and has grown no session, token or password of its own, and that both docs still name
the four public paths, the reset command, the four variables and the `DASH_COOKIE_SECURE=1`
deployment note. Nothing starts a server there; a sentence that quietly disappears is the bug.

`tests/test_certs.py` is the same kind of file for the certificate authority, and two of its tests
are structural rather than behavioural on purpose: they read `certs.py` itself and fail if it gains
a `logging` import or a raise that interpolates anything, because a stored PEM is a private key and
this app turns log lines and error strings into rows a browser reads. The rest runs the whole
lifecycle over a dict store — first boot persists exactly five rows, a second boot changes no byte,
rotate/revoke/root-rotate and their denylists, the bundle never carrying the root key — and one test
completes a real client-certificate handshake with the material a startup would write.
`tests/test_agent.py` runs the matching TLS matrix against the real agent handler, and
`tests/test_deploy_scripts.py` runs the installer in a sandbox where `install`/`chown` are faked, so
the `0640` mode, the byte-identical bundle and the `--status` probe (against a live TLS agent) are
proven without a root shell. The same file runs `scripts/dev-certs-export.py` against a real
`create_app(...)` served by uvicorn on a loopback port (phase 24) — a thread and a socket rather
than `TestClient`, because the unit under test is an HTTP client that reads its session out of a
`Set-Cookie` header — and pins the export against the route: `0600`, byte-identical, one line per
host, no file for an unissued one, no mint and no PEM in the output, and a wrong password exiting
non-zero having written nothing.

`tests/store_perf.py` is the second opt-in marker's machinery (`live` is the other). It seeds a
**production-shaped
SQLite file** — `SampleStore`'s own retention (`raw 48 h / 1m 30 d / 1h 400 d`), the fleet's real
metric names, the real 10 s cadence: ~6.4 M rows and ~720 MB, the dev DB's order of magnitude — and
times every hot path of `store.py`/`api.py` against it, with `EXPLAIN QUERY PLAN` for the queries
phase 14 changes and a two-connection race that reproduces the `database is locked` line in the
known-gaps list — beside the **pre-fix control** of the same race, because a clean race that never
met a lock would prove nothing. Every pass that *writes* (a rollup catch-up, a race) runs on a
scratch copy of the seeded file, so the baseline the next measurement compares against comes back
exactly as it went in. The default suite builds that store and asserts its **shape and determinism**
(row counts, retention windows, rollups that are the mean of the raw rows they came from, a second
`maintain()` that changes nothing) but never its wall time — except behind the marker, where
`store_perf.BUDGETS` is the phase's pinned regression table (14 ceilings, 2-3x the measured "after",
`maintain()` < 1 s, a racing insert at zero `database is locked`) and a stale budget pointer fails a
default-run test. The cached reads are reported twice —
**cold** (the cache invalidated before every run) and **cached** — because comparing a cache hit
against the baseline's full scan would book a 1 000x speed-up that is really just a cache. Only
`-m store_perf --store-perf` runs the timing passes, and `scripts/store_perf_probe.py` is the CLI
form that writes `data/logs/store-perf-baseline.{md,json}` (task 03's write-side run is
`store-perf-task03.*`, task 04's budget run `store-perf-task04.*`) for `data/logs/NOTES-store-perf.md`. Two rules keep it
honest: **no test ever opens `data/dashboards.sqlite3`** (the `--dev-db` flag exists for a human
running the probe read-only), and the seeded file is a cache under `data/perf/`
(`DASH_PERF_STORE_DIR` relocates it) because a before/after has to measure the same file — pytest's
`tmp_path` on this box is `tmpfs`, i.e. RAM, which times a store nobody deploys.

The non-timing half of that lives in `tests/test_store.py` and runs every time: a cache hit
issues **no** SQL at all (counted through `sqlite3`'s trace callback, never through a wall clock),
the series list is invalidated by the write that introduces a name and not by the poll that repeats
one (and the tracking that decides has a cap, below which it falls back to invalidating),
`last_maintain` is never served from a cache, and `pick_res_many()`/`history_many()` answer exactly
what the per-series calls answer — on a small store and on the seeded one, at every window.

One more file is worth knowing about: `tests/card_contract.py` is the table of
`/api/system/snapshot` paths the system page's per-host cards are allowed to read. It is
checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and
again against the live fleet, so a card can never be "fixed" by inventing a number the agent
does not report. `tests/ups_contract.py` does the same job for the UPS page's hero row and
battery glyph (`charge %`, `load %`, `runtime left`, `battery_voltage`, the glyph's
`charger_status` + `on_battery`, the card header), against the two real UPSes — recorded
verbatim as `tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, with the
device serials scrubbed — and again against live upsd. The captures earn their place: the
CyberPower answers `battery.mfr.date: CPS`, the APC's `battery.date` is a factory
placeholder, **neither** publishes `battery.temperature` or `battery.charger.status`, and the
two disagree on `ups.test.result` (the CyberPower has run a test and answers `Done and passed`,
the APC still answers the factory `No test initiated`), so a hand-written variable table would
have tested nobody's memory. `tests/slots_contract.py`
pins gufo's `/slots` rows the same way — hermetically from
`tests/fixtures/slots_gufo.json` (the verbatim `/slots` reply, captured whenever
`GUFO_BASE_URL` is set; the idle-slot rows are the shape the grid must handle) and again
against the running engine. It exists because that table was once guessed: the old slot
table read field names the server never published and rendered dashes forever. It also picks
the **canonical name of each fact** — the live server answers `id_task` *and* `task_id` for
one request — and the page renders only the canonical key, so a rename fails a test instead of
quietly becoming a fallback chain. A busy
slot's `prompt` is the user's in-flight request text, so the capture script blanks it and
the page never reads it.

`tests/test_gateway_source.py` is where the ingress join's honesty lives, and it pins that module
from both ends. The input is the **captured** gateway — `gateway_haproxy.cfg`, the nginx main config
plus its included documents, and the two `/proc/net/tcp{,6}` captures — run through the real
`host_agent` parsers, so the test compares the join against the agent's own document rather than
against a copy of its field names. The assertions are the join's two rules and the limits around
them: the graph carries exactly the ports, `server_name`s and targets those documents declare (all
33 routes on this capture, as a **set**, so a missing or invented route fails by name); a socket is
inbound by its local port and outbound only when its remote `(address, port)` equals a target the
config wrote as an address; **every other socket is ignored** — `established`, `live_in` and
`live_out` are all printed and deliberately never reconciled, because a connection to a name, or on
a port nobody configured, is not a fact this page may claim; several routes reaching one endpoint
carry one count repeated with `shared: true` rather than a split nobody measured (so the totals, in
the source and on `/2`, sum **distinct endpoints**); a name target is never credited, and `0` never
stands for "not counted"; and **no join ever asks a resolver** — `socket.getaddrinfo` is spied on and
fails the test, because deciding "already an address, or a name?" is the only question that module
may ask, and a DNS lookup in the poll loop would be rule 2 with a round trip attached.

The seventh capture, `gateway_resolver.json`, is in that suite's reach too: it is the agent's own
`resolver` document from the real gateway, so `tests/test_agent.py` proves it has the section's shape
(one entry per name `target_names()` derives, `via: "libc"`, the config's own `resolver` line, no
globally routable address) and `tests/test_gateway_source.py` proves it reaches every route the
configs write as a **name** and shares addresses with the recorded socket table — the capability the
join needs, pinned before the join exists to use it.

`tests/rate_window_contract.py` is the same device for one number rather than a field table: the Δ
window each source measures its rates over — `RATE_WINDOW_SECONDS` for the host agents,
`LLM_INTERVAL_SECONDS` for the engines, one estimator per measuring cadence (phase 23). It holds
the two spans, the `Δ<window>` rounding the pages print, and a range assertion — the estimator's
own floor up to its `2 × span` ceiling, plus the printed label — shared by four files: the
`build_hub` wiring spy in
`test_api.py`, both throughput cells of both engines in `test_llm_source.py`, the host payload's
`window_seconds` in `test_host_source.py`, and the estimator's floor / `2 × span` ceiling / 8×
jump gate run at both spans in `test_rates.py`. The two payloads are the same shape and only the
number differs, so a pin naming one source's window is exactly how the other could quietly keep
advertising a window no poll ever measured.

## Battery-test buttons

Instant commands are the only endpoint in the application that changes anything **outside this
process** (`POST /api/ups/command`; the session and certificate-authority writes change only this
app's own rows), and they are gated four times: `NUT_ALLOW_ACTIONS=1`, not `DASH_READ_ONLY`, an `upsd.users`
account that actually has the command, and a confirmation in the UI — destructive ones
(deep test, calibration, `load.off`) require typing `DANGER`. Two more sit in front of those
(since the login landed): a session, and this session's CSRF token when the write arrived on the
cookie — additive, never a replacement, so the typed `DANGER` is still what cuts power. The server
re-checks every gate on every request; hiding a button is not a safety control. Commands that would cut power are refused
outright while the UPS is already on battery.

Give the app a narrowly scoped NUT user instead of a god-mode one — per-command `instcmds`,
not `actions = -`:

```ini
# /etc/ups/upsd.users
[dashboards]
    password = "..."
    instcmds = test.battery.start.quick
    instcmds = test.battery.start.deep
    instcmds = test.battery.stop
```

One command per `instcmds` line — that is what `man 5 upsd.users` specifies ("to specify
multiple commands, use multiple instances of the instcmds field"), and a comma-separated list
on one line is not that. `instcmds = all` is the alternative, and the fleet runs that: every
command the drivers advertise, including the ones that cut output power. Nothing about the
gates changes — the risk tier, the typed `DANGER`, `NUT_ALLOW_ACTIONS` and the on-battery
refusal are what prevent a click from becoming an outage, because upsd gives the dashboard no
way to ask what an account is allowed to send.

upsd authenticates with `USERNAME` + `PASSWORD` (see `HELP` on your own server for the verbs
it accepts — `USER`/`PASS` are not among them in 2.8.5); a user needs `instcmds`, never
`actions = -`.

`deploy/nut-sudoers` is the ruleset that lets the `ops` account read (and scoped-write) those
files: `sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut`.
Read `upsd.users` before you write it — `upsmon` authenticates from the same file, so
overwriting it blind can take the box down at the wrong moment.

Bind upsd to loopback (`LISTEN 127.0.0.1 3051` in `upsd.conf`) when the dashboard runs on the
same box, and to a real interface plus a source-scoped firewall rule when it runs elsewhere —
and do not try to scope it by client inside `upsd.conf` instead: **NUT 2.8
removed the `ACL`/`ACCEPT`/`REJECT` directives** (a build without TCP Wrappers logs
`ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and ignores the
line), so `LISTEN` plus a host firewall rule is the only access control `upsd.conf` offers,
and an unauthenticated `LIST VAR` from any routed source is the normal state of a NUT server.
`NUT_SIMULATE=1` stands in for upsd during
development; everything it produces is badged `SIMULATED`, and it is never the default.

## Deployment

`Containerfile` is a two-stage build: `uv.lock` is resolved in the builder (exactly the
versions the tests ran against) and the runtime stage is slim Python with nothing but the
venv in it. The host agent is copied in too, so one image can monitor the box it runs on.
In the deployment repo the pattern is the same as every other service: quadlet container +
env template + nginx/SNI on the single domain.

```bash
podman build -t dashboards .
podman run --rm -p 8090:8090 --env-file dashboards.env \
    -v dashboards-data:/var/lib/dashboards dashboards
```

It listens on `0.0.0.0:8090` as an unprivileged user, and the SQLite history lives in
`/var/lib/dashboards` — put a volume there or every restart forgets the charts **and** the admin
password, because the verifier and the session key are rows in that database. `dashboards.env`
therefore needs `DASH_ADMIN_PASSWORD` for the first start of a new volume; with neither the seed
nor a stored verifier the container exits non-zero with the reason instead of coming up open, and
the same seed is ignored once the row exists, so it can be dropped from the env file afterwards.
With no sources configured `/api/health` answers `200 {"status": "empty"}`, so the container
passes its health probe before its env file is right instead of crash-looping.

Two more lines belong in that quadlet env template because this service faces the public domain:
`DASH_COOKIE_SECURE=1`, so the session cookie never rides plain HTTP (the app also treats a
request that arrived with `X-Forwarded-Proto: https` as secure, so a misconfigured flag degrades
rather than leaks), and the password itself — set once, then removable, since the stored verifier
wins. The nginx server block, its `X-Forwarded-*` headers and HSTS live in the deployment repo
like every other service here, so a change to how the edge forwards TLS is a change in that repo,
not in this one; the only thing this app reads off the proxy is that one scheme header.