dashboards/AGENTS.md

# AGENTS.md — homelab dashboards

Orientation for an agent starting work in this repo: what the app is, the rules that keep
its numbers honest, where things live, the fleet it watches, and the commands. `README.md`
is the operator-facing how-to (quickstart, agent install, signing in, configuration,
deployment) — read it for those; this file does not repeat it. This file is **orientation,
not a changelog**: phase history lives in `.agents/phases/` and `data/logs/`. When a phase
closes, update the sections below only if a *rule, invariant, command, or fleet fact*
changed — never append the phase's narrative.

## Working protocol

1. Read `.agents/PLAN.md` first; follow the phased execution protocol in `.agents/phases/`.
2. Never modify `.agents/PLAN.md` or files in `.agents/phases/complete/`; ask the user
   before updating anything in `.agents/phases/todo/`.
3. Strictly adhere to the **LOCKED DECISIONS** in `.agents/PLAN.md` and the section at the
   bottom of this file.

## What this is

Five dashboards for one homelab, behind one domain, in one process, backed by one SQLite
file — plus an overview page:

| Route      | Question it answers                                            | Source                                                        |
| ---------- | -------------------------------------------------------------- | ------------------------------------------------------------- |
| `/`        | Is anything wrong, and where do I look first?                  | all five, in one request                                       |
| `/system`  | CPU, memory, disk, GPU, network, and the history               | `agent/host_agent.py`, our own stdlib agent per host           |
| `/disks`   | Are any physical disks failing, and which — health, temp, wear | the same agents, `smartctl -i -A -H` per whole drive          |
| `/llm`     | Tokens/s, active sessions, queue depth, KV pressure, latency   | gufo `/metrics` + `/slots`, vLLM `/metrics`, radiance `/metrics` + `/stats`  |
| `/ups`     | Battery health, runtime left, power events, test buttons       | NUT `upsd` (our own wire client; port **3051**, not 3493)    |
| `/gateway` | Which domains on which port, and where each goes               | the same agent's `haproxy` + `nginx` sections joined to `/proc/net/tcp{,6}` on the one `GATEWAY_HOST` host (the page needs agent **≥ 1.2.0**; the route switch's `#off` marker read is **1.4.0**) |

History is a first-class feature: every number worth graphing is stored, rolled up
(raw → 1m → 1h) and served from `/api/*/history`.

None of it is public. One admin signs in at `/login` (the only page an anonymous visitor
reaches); `/settings` (⚙ Account) and `/certs` (⚷ Certs) live behind it. Every page and
every `/api` route requires the session except the public entries in invariant 12.

## Non-negotiable rules

Breaking one is a bug even if the page still renders.

1. **Read-only by construction, with exactly two exceptions, and both are gated on purpose.**
   Nothing writes to an upstream except `POST /api/ups/command` and
   `POST /api/gateway/route`, and each is re-checked on every request by its own gates:
   the UPS one is gated four times (env allow, not `DASH_READ_ONLY`, the upsd account
   actually having the command, a UI confirmation — typed `DANGER` for anything that can
   cut power) and power-cutting commands are refused outright while on battery; the gateway
   one is gated ten (not `DASH_READ_ONLY`, `GATEWAY_ALLOW_WRITES` — off by default — a
   per-address write throttle, a hand-parsed body, the route's state in the current
   payload, the freshness of the read end to end, and a typed `danger="DANGER"` for
   **both** verbs — a `dry_run` preview asks nothing and writes nothing — for anything
   that can cut a route through the fleet's only ingress), and it dials once through
   the same revocation gate the reads take. Hiding a button is not a safety control.
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
   as `0`. Every derived value carries its `source` string and `derived: true`; the rate
   estimator is keyed by the *resolved* series name **plus its label set** and refuses a
   window too short or too stale. Aggregating two different series into one rate is the
   failure mode this rule exists to catch.
3. **Unconfigured is not broken.** A source with no env config answers
   `404 {"configured": false}` **without dialling**, and the page says "not configured".
   A red panel for a server the operator never installed destroys the signal.
4. **One dead source must not blink another — and nothing may stop polling in silence.**
   Each source is its own asyncio task with its own timeout and health record. A failed
   poll keeps the last good payload and marks it stale instead of zeroing it; a host that
   never answered still gets its row (`unreachable: true`, dimmed, `n/a`), because dropping
   it would turn a dead agent into "never configured". **The task is the source's only
   poller, so no exception may end it**: a store that refuses an otherwise good poll is a
   *reported* failure (`ok: false`, the real reason, `last_ok` frozen, one event then
   `recording` on recovery), and anything that escapes the poll body is logged and the loop
   continues. A frozen source that looks healthy is the one failure mode worse than an
   outage.
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
   charts are hand-rolled SVG (`src/dashboards/web/static/charts.js`). The engines live on
   a network with no outbound access — a CDN reference is a request that never comes back.
6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets`
   moves it into `Authorization`, so `Target.url` ends up clean), but nothing that leaves
   the process may: URLs in `UpstreamError` messages, health records, `event` rows
   (`/api/events` serves them to the browser) and `/api/config` are redacted. **The admin's
   credential, the private keys, and session tokens are the same case**: `auth.py` records
   nothing and answers `False`/`None` to every rejection, `certs.py` has no `logging`
   import, and both modules' `raise` sites and `auth.py`'s one `logging` call carry
   **fixed** text with nothing interpolated — pinned structurally by `tests/test_auth.py`
   and `tests/test_certs.py`. No token, password, or personal hostname goes into tracked
   files or fixtures; captured socket tables and resolver answers substitute globally
   routable addresses for documentation space (RFC 5737 / RFC 3849), idempotently.
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
   `sources/llm.py` and `sources/nut.py` were captured from the running engines and pinned
   by `tests/fixtures/`. Re-capture with `scripts/capture-fixtures.sh` (and
   `scripts/capture-gateway-config.sh` for the gateway's proxy tree, socket tables and
   resolved names) rather than trusting a README or a memory of an upstream.
8. **Keep dependencies current on purpose.** After touching dependencies, `uv pip list
   --outdated` must come back empty. Python is 3.14 everywhere (`requires-python`, ruff
   `py314`, pyright, Containerfile) — do not lower it to make something else work.
9. **SQLite owns the history and must stay bounded.** Writes go through `store.py`; new
   metrics need a rollup path and a retention story, or a year of charts becomes a disk
   emergency. Put the DB on a volume in deployment.
10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`,
    http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot
    (`data/logs/*.png`) before they are called done.
11. **The dashboard never starts open, and the stored password beats the environment.** The
    admin verifier and the session-signing key are durable `meta` rows
    (`auth.verifier`, `auth.session_secret`) and the only place either secret lives;
    `DASH_ADMIN_PASSWORD` is a **seed** that hashes into the row on the first boot of a
    database and is ignored thereafter. A server with neither a stored verifier nor the seed
    **refuses to start** (a `RuntimeError` out of the lifespan, non-zero exit). `testpass`
    is the local-testing password (dev config and the hermetic suite), never a production
    fallback.
12. **The gate is default-deny and its public surface is four entries** (`/login`,
    `/api/auth/login`, `/api/health`, `/static/`). A page `GET` gets `303` to
    `/login?next=…` (query string included, checked by `auth.safe_next`); everything else
    the one JSON refusal; every answer carries `no-store`; a write additionally needs a
    matching `Origin`/`Sec-Fetch-Site` and — when it came in on the **cookie** — this
    session's `X-CSRF-Token`. A process whose lifespan never produced `ctx.auth` answers
    `503` for everything, public paths included. `tests/test_auth_api.py` walks the
    **served** route table so a route that forgets the gate fails a test instead of
    shipping — the middleware guards by default. `/api/health` is a contract with
    deployment, which means **nothing may be added to it without re-deciding that**: it is
    public, and a new field there is a field the whole internet reads.

## Architecture

```
agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / VAR / CMD
        ▼                                  ▼                        ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command and post_route
        │  parse + Δcounter/Δt
        ▼
sources/host.py · disks.py · llm.py · nut.py · gateway.py          poller.py (task per source)
        │        Result(payload, samples, events)
        ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/
```

One line each — the load-bearing behaviour is in the bullet, the rest is in the module:

- `agent/host_agent.py` — stdlib-only (3.10+), one per host; serves `GET /snapshot`,
  `/info`, `/schema`, `/health`. Reads `/proc`, `/sys/block`, amdgpu DRM for GPUs,
  `statvfs`; `smartctl -i -A -H` per whole `/sys/block` disk with a scaled pass budget
  (20 s floor, 5 s per disk) whose refresh runs on a background thread — never inline on
  a later poll. Refuses to bind off-loopback without a token **or** a certificate bundle
  (exit 2). **TLS mode (≥ 1.1.0)**: the bundle is the auth (the bearer token is the
  compatibility path), the `# REVOKED` denylist in the bundle is re-read when the bundle
  changes, and a serial on it is refused on every path including `/health`. **The proxy
  add-on (≥ 1.2.0)**: the `haproxy`, `nginx` (plus its one-level `include`s) and `sockets`
  sections; **the resolver section (1.3.0)** looks up *only* the names the two config
  files declare, through libc on the box that dials, cached and single-flight. All four
  sections are additive, `available` is three-way (`true`/`false`/`null`) with exactly one
  error line per non-`true` arm — the missing one saying **not found**, the pinned wording
  the dashboard reads to answer "is it installed?" — and `--no-proxy` / `--no-resolve`
  remove their keys in silence. Nothing in a section can fail `/snapshot` or `/health`.
  Counters stay counters: the dashboard computes rates. **The one write (1.4.0, gateway
  hosts):** a single `POST /route`, refused without a verified, non-revoked client
  certificate **and** the agent's own `--allow-writes` flag (certificate first, in that
  order), forwarding exactly `{id, action, dry_run}` to the socket-activated root helper
  `dashboards-routectl` and answering its JSON verbatim — **one connection, one
  request, one answer, no retry**, because a retry of a write is a second reload. The
  privilege is the helper's socket, not the agent's: the unit keeps
  `NoNewPrivileges=true` + `ProtectSystem=strict` byte-identical with and without
  `--with-routectl`, and no sudoers line is involved. Every `GET` path is untouched —
  `#off` markers are read on the normal proxy sections (the marker's own grammar is the
  agent's: `ROUTE_MARKER` / `split_marker` / `config_line_index` — `disabled` on
  `server`/`proxy_pass`/`map` nodes; a marked line stays in the document).
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
  env var always wins. Safe defaults except the sources, which default to unset
  (`HOST_TARGETS` and the engine URLs `GUFO_BASE_URL` / `VLLM_BASE_URL` /
  `RADIANCE_BASE_URL`, each with its optional `*_API_KEY`). The
  gateway source is enabled only when `GATEWAY_HOST` names a `HOST_TARGETS` entry
  (default `gateway`). The write's two settings are beside it: `GATEWAY_ALLOW_WRITES`
  (default **off**, the NUT flag's truthy rule) and `GATEWAY_WRITE_TIMEOUT_SECONDS`
  (default 35 s, deliberately longer than the agent's 30 s peer timeout so the dashboard
  never times out first); the read's 10 s budget stays the read's alone.
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
  `disks:<host>` is a source of its own (300 s / 30 s) and `gateway` is **one** job (30 s
  / 10 s) that exists only when configured, because the page is about the single ingress;
  it is handed no `RateEstimator` (a count is not a rate). The success path records
  (payload → samples → events) and *then* moves `last_ok`, so a refused SQLite write ages
  the source and alerts instead of leaving it current with charts that silently stop
  drawing. Events are stored under the **dashboard** name (`system`/`disks`/`gateway`/
  `llm`/`ups`), not the sample source. The job list is built once (`app.build_hub()`)
  because it is wired twice (production and the injected-store test path).
- `collectors.py` — the fetch half. `AgentTls` makes an `https://` `HOST_TARGETS` entry a
  certificate-authenticated dial (per-dial client, this app's client cert verified against
  its own root; the revocation answer is a fresh read per poll, so a `/certs` revoke bites
  on the next poll with no restart, before any socket opens). `post_route` is the write's
  dial beside `fetch_snapshot`: the same per-dial client and the same revocation check
  before any socket opens, but **`Authorization` is never sent on the certificate path** —
  the bundle is the credential there, and the token travels only on the plain token path.
  The answer is the helper's own object, verbatim. Failure messages name the
  **redacted** URL. Agent `errors` become `degraded` **except** the add-on lines
  (`ADDON_ERROR_PREFIXES`: `smart`, `haproxy`, `nginx`, `sockets`, `resolver`) — a host
  without smartctl or a proxy config is unconfigured, not broken (rule 3); those lines
  stay verbatim for the page that owns the add-on. **A new agent add-on section adds its
  prefix here in the same commit.**
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card.
  Carries the agent's `tls`/`bundle_rev` passthrough (present only when published — rule 2).
- `sources/disks.py` — physical drives only (never a partition, never a mount), the agent's
  `smart` block verbatim, `smartctl_available` three-way on purpose, per-row `severity`
  (`temp_c`, `percentage_used`, the drive's own word for `health`); sector counters are
  alarm chips, never graded. Nothing here is a rate.
- `sources/gateway.py` — the snapshot projected onto the connection topology. The join is
  **two attributions and nothing else**: a socket whose local port names a configured
  listener is inbound; a socket whose remote `(address, port)` equals a target written as
  an **address** in a config is outbound; everything else is ignored. A name target is
  credited **only through an address the gateway's own `resolver` section published** —
  the module never calls a resolver itself (spy-pinned). Pipes are `null`, never `0`, when
  unmeasured. `warnings()` judges the two config files against themselves (seven kinds,
  each `{id, kind, subject, message}`) and never joins a warning to a row. Each row's
  `enabled` carries the config's own state (the 1.4.0 `#off` marker, or a map entry
  pointed at the helper's sink); a merge that would mix states splits into two rows
  keyed by `enabled` (a union of selectors with one disabled line is two facts; the
  `#off` suffix keeps the halves distinct); the marker-caused warning shapes — an
  emptied backend, a targetless route, a sink-pointed entry — warn nothing, while the
  same shape from a hand-deleted line still warns; the switch-off tally is published
  as `/api/summary`'s `disabled` field. No `severity` anywhere in the payload (a busy
  proxy is the job).
- `sources/llm.py` — the gufo / vLLM / radiance field maps, measured Δ rates **diffed per label set**,
  prefill speed, the session list from gufo `/slots` (radiance's from its `/stats`
  sidecar). A refused estimator window is still
  an answer: the cell keeps its `source` and an honest `Δ<window>` beside a `null`.
- `sources/nut.py` — the NUT wire client, `assess()` (explicit checks, not a mystery
  score), `SELF_TEST_OUTCOMES`, the `COMMANDS` catalog with risk tiers and
  `available_commands()` filtering, kelvin/celsius and charger-status normalisation.
- `sources/nut_sim.py` — a **labelled** simulator; never the default, every payload says
  `simulated`; its modelled battery date ages relative to today.
- `prom.py` — in-house Prometheus text parser (only for engines we don't own);
  `values_by_series()` is the per-label-set view counter deltas need.
- `rates.py` — the Δ/window estimator: keyed by series name + label set, window bounded
  both sides (floor; `2 × span` ceiling), a rate ≥ 8× the previous one held one poll and
  confirmed only by an **independent anchor**. `app.build_hub` builds one estimator per
  measuring cadence — host at `RATE_WINDOW_SECONDS` (10 s), LLM at `LLM_INTERVAL_SECONDS`
  (2 s) — because one estimator cannot be right about two cadences.
- `severity.py` — the only place thresholds live. `classify()` → `ok`/`warn`/`fail`/None;
  verdicts travel **additively** in payloads; pages render the class and never compare a
  number to a threshold; unknown in → unknown out, never a green pass. There is no
  `gpu.*` row and no `gateway.*` row: a busy GPU or a busy proxy is the job at every
  level.
- `alerts.py` — the pure "what is wrong right now" projection `/api/summary` serves:
  reachability, severity breaches, UPS checks. An unconfigured source never alerts; a dead
  source gets exactly one line; one alert per (source, metric), at most 12. It has no
  thresholds of its own.
- `store.py` — `sample` + `event` + `meta`, rollups, pruning. `meta` is the only durable
  key/value state, reached through `get_meta`/`set_meta`/`delete_meta` only. **No
  per-request path may pay a whole-table scan**: `series_names()` and `stats()` answer
  from short-TTL caches. `maintain()` is windowed (only what became due since the last
  pass, snapped down to a bucket boundary) and its prune is one `DELETE` per resolution;
  it ends with `PRAGMA optimize` (auto-analyze is off, so no poller's commit pays for it).
- `auth.py` — every auth decision as a pure function, stdlib only: PBKDF2 verifier
  (self-describing, so the cost can rise without a migration), HMAC-signed stateless
  session cookie, CSRF bound to the session, per-address login throttle, `safe_next()`,
  and the verifier bootstrap (env seeds once; the stored row wins; no row and no seed
  raises). The primitives **never raise and record nothing** — the reason an input was
  rejected is the credential (rule 6).
- `certs.py` — the fleet's certificate authority over the same three-method `MetaStore`
  protocol: root CA, dashboard client cert, and per-host agent certs, all in `meta`; the
  bundle is the install artifact (agent cert + key + root *certificate*, never the root
  key, + `# BUNDLE-REV` + the denylist). **A stored row always wins** — the root is
  generated on the first boot of a database and never on a later one. `ensure_certs` is
  the startup read (it mints; nothing else may call it); `read_state` is the request-time
  read (same validation, never a write). `cryptography` is the app's only new dependency,
  used here alone; the agent stays stdlib.
- `api.py` — GETs everywhere plus two POSTs that dial an upstream (`/api/ups/command` and
  `/api/gateway/route`, invariant 1's two exceptions). Every source's
  configured gate runs **before** anything that could reach the network (spy-pinned: no
  client, no fetch). `/api/config` is a redaction boundary and serves
  `gateway.writes_enabled` so the page can render the honest arm. All five history routes go
  through one door (`history_many`). The upstream write set is exactly those two (pinned by
  enumerating every served route's verbs; a third fails by name). The route write's gates
  (invariant 1) run in a pinned order — read-only, flag, per-address throttle, body
  shape, the id's state in the current payload, end-to-end freshness, typed `DANGER` for
  both verbs — and every answered call stores one audit event under its own kind
  (`route-disabled`/`route-enabled`/`route-denied`; a dry run stores none) in a fixed
  shape: the id, the action, the helper's words — **never the config text**.
- `app.py` — assembly and the **session gate** middleware (invariant 12), answering in
  this order: no `AuthState` → `503` for everything; no session and no such route → the
  router's `404`; public → through (the origin check still applies to a public write);
  no session → `303` for a page / JSON for the rest; a write → origin, then CSRF.
  `build_hub` is called in **both** the owned-store and injected-store branches; the
  lifespan builds `ctx.auth` before the first poll, and a `RuntimeError` there leaves the
  process non-zero.
- `web/` — the six dashboards plus `/login`, `/settings`, `/certs`; one stylesheet,
  `charts.js`, no framework. Colour comes only from the payload `severity` verdict
  (`/gateway` colours nothing by verdict — its only pair is existence); `n/a` for the
  unknown, `SIMULATED` badges, per-source status dots, `Δ<window>` labels on measured
  rates. **The served-byte pins in `tests/test_api.py` are the UI contract** — when a UI
  rule is in doubt, read that test.

Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `ups.charge_pct`), `@stream` for per-device series. `sample.source` is
the dashboard family — `host`, `llm`, `ups`, `disks` (the SMART series) and `gateway`
(the connection gauges: `gateway.in_conns@<listen-port>`, `gateway.out_conns@<host>:<port>`
— the port belongs in the outbound name because two services on one host are two facts) —
so `GET /api/metrics?source=…` answers "what does *this page* have history for".
Treat the scheme as a public interface of the history API.

## The fleet (real environment)

| Host          | ssh           | What it gives                                                             |
| ------------- | ------------- | ------------------------------------------------------------------------- |
| `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, radiance `:8000` (replaced vLLM there, 2026-10-24; the vLLM code and env var stay for other deployments), **upsd**, the dashboards. 16 whole disks; the agent unit runs `User=root` with `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's unprivileged-passthrough gate; an empty bounding set is capability-less, `CapEff 0`) |
| `deskwork`    | `deskwork-ops`  | Strix Halo (Radeon 8060S iGPU), gufo `:8731`                               |
| `gateway`     | `gateway-ops` | the router, the fleet's ingress: haproxy (`/etc/haproxy/haproxy.cfg`, six frontends → six backends) + nginx (`/etc/nginx/nginx.conf` plus `conf.d/` vhosts and `stream.d/`'s `map` on `:443`); runs agent **1.4.0** with the `dashboards-routectl` helper units |

- **Transport is direct dial; the fleet is mTLS.** `HOST_TARGETS` are `https://` with no
  userinfo — the certificate bundle is the credential (the token in a URL is the
  compatibility path for a host nobody has bundled). All three hosts run the installed
  `dashboards-agent` systemd service, installed by `deploy/install-agent.sh` under the
  sudo grant — registration is the install: `--fetch-bundle <url> --cert-name <name>
  --session <token>` pulls the host's own bundle from this dashboard during the install, so a
  host is never registered onto a token URL because a `curl` was forgotten. **Re-stage
  `/home/ops/agent-install/` first — the grant names a path, not a checkout —** and a re-run
  whose rendered unit differs from the installed one prints the diff and refuses without
  `--force`. `deskwork`/`driveripper` still run **1.1.0**: the
  1.2.0/1.3.0/1.4.0 sections are purely additive (a 1.1.0 snapshot has none of the keys,
  which `sources/gateway.py` answers as `supported: false`), so they move at their next
  re-stage.
  A TLS agent serves https only — rolling a host back means removing the bundle lines too,
  not just the URL. **The gateway's write (1.4.0) installs beside the agent, never into
  it**: `install-agent.sh --allow-writes --with-routectl` adds one argument to the agent
  unit's `ExecStart` (`--allow-writes` — a diff that shows a security directive moving
  is a stop, not a `--force`) and installs `dashboards-routectl.py` (0755, root,
  `/usr/local/sbin/`) plus the two units **`dashboards-routectl.socket`** (`0600`, owned
  by the agent's user) and **`dashboards-routectl@.service`** (root, one instance per
  connection, `ReadWritePaths` = `/etc/haproxy` + `/etc/nginx/conf.d` +
  `/etc/nginx/stream.d` only), verifying the agent's uid can connect before reporting
  success. **No sudoers line is involved** — the grant is a file mode systemd enforces,
  and `--with-routectl` alone leaves the agent's unit byte-identical; the helper edits
  one `#off` route (the marker carries the line's original text after its `was:`, a
  per-route state sidecar records who last changed it and the file's hash, an `enable`
  restores only an unedited line — an edited one is a `drift` refusal, no byte moves —
  and a switched-off `map` entry points at the helper's own sink `server`), validates
  with the proxy's own checker (restore on failure) and **reloads, never restarts**.
  `--without-routectl` removes it, printing the still-off routes first;
  `deploy/gateway-routectl-README.md` is the operator's page. The live **drill has not
  run**: the install is verified end to end (task 07), but this fleet's proxies run in
  podman containers, so the host has no `nginx`/`haproxy` binary for the checker —
  `data/logs/NOTES-33-switching.md` records it, and a real `DANGER` is still the
  operator's call.
- **NUT 2.8.5 on driveripper: configs in `/etc/ups/` (not `/etc/nut/`), upsd on port
  3051 (not 3493 — nothing listens on 3493).** NUT 2.8 has no `ACL` directive;
  per-source restriction is a **firewall** rule, never an `upsd.conf` line. Two UPSes:
  `apc` (APC Back-UPS RS 1500MS) and `cyberpower` (CyberPower PR1500LCDRT2U — that is
  `ups.description`; the driver's own `identity.model` answers `GX1500U`). Real driver
  quirks the parsers already handle: CyberPower's `battery.mfr.date` is `CPS` (not a date
  → `unknown`, never a fake age); **neither driver** reports a battery temperature or
  `battery.charger.status`; `ups.test.result` is `No test initiated` (APC) vs `Done and
  passed` (CyberPower) — both words re-read from the wire and pinned in `tests/test_nut.py`.
- **SMART**: `smartctl` 7.5 on deskwork and driveripper; the agent shells it **as
  itself** (never sudo). This binary annotates the attribute table's RAW_VALUE column
  (`35 (Min/Max 34/44)`, `43894h+43m+32.519s`), so the parser reads that column, not the
  end of the line. `gateway` has no smartmontools (a VM) → `smartctl_available: false` →
  no `/disks` card at all (the visibility gate, `api._disk_visible`), while the
  `disks:gateway` job still polls and its events still reach the page.
- Ground truth: `./scripts/dev-server.sh status` (per-source ok/DOWN, unauthenticated) and
  `uv run pytest -m live --live` (opt-in, **read-only**). The per-host install record is
  in `data/logs/` (gitignored) — re-run the probes rather than trusting a note.

## Commands

```bash
uv sync --python 3.14                      # deps + editable install
set -a; . ./.env.dev; set +a              # dev dials the fleet the prod way, no tunnels
./scripts/dev-server.sh restart           # keep http://127.0.0.1:8090 up; sign in with testpass
./scripts/dev-server.sh status            # per-source ok/DOWN (unauthenticated)
./scripts/dev-server.sh bundles           # the bundles the restart exported to data/certs/
./scripts/dev-server.sh logs
uv run pytest                             # hermetic: no network (skips 12 live + 4 store_perf)
uv run pytest --cov                       # coverage gate (floor 94%)
uv run pytest -m live --live              # opt-in: real fleet / engines / upsd — read-only
uv run pytest -m store_perf --store-perf  # opt-in: store hot-path timings vs store_perf.BUDGETS
uv run ruff check src tests agent && uv run pyright
bash .agents/validate.sh                  # the gate: the four checks above + repo hygiene
./scripts/capture-fixtures.sh             # re-record ground truth (UPS + gufo /slots + smartctl)
./scripts/capture-gateway-config.sh       # the gateway's haproxy/nginx tree, /proc/net/tcp{,6},
                                         # and the recorded resolver answers (via $GATEWAY_SSH)
```

Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on
`0.0.0.0:8090`, SQLite under `/var/lib/dashboards` — put a volume there) and
`deploy/install-agent.sh` per monitored host. The README's *Deployment* section is the
operator's how-to for both.

## Testing policy

**Real captures for shape, fakes for behaviour, `live` for drift.**

- Anything that must match an upstream (`/proc` layouts, vLLM series names, NUT framing,
  smartctl output, the gateway's configs) is asserted against a **verbatim capture** in
  `tests/fixtures/`. If the schema drifts, a fixture test fails before a dashboard does.
- Failure modes (a dead upsd, an engine 500ing, a host rebooting mid-poll, a counter
  reset) are fakes in `tests/conftest.py`.
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
  fleet. Skipped unless `--live`; each test skips when its source is unset. **Live tests
  must not write** — a battery test on hardware something is plugged into is not a test.
  An `https://` agent is dialled through `fetch_snapshot` with the stored authority,
  never a hand-rolled `client.get`.
- **A verdict the operator sees is pinned on the surface that shows it, in both
  directions** — an absence-only assertion is satisfied by deleting the check.
- API tests use `create_app()` with an in-memory DB and the `testpass` seed at
  `pbkdf2_iterations=1000` (the real 600 000 cost is pinned separately in
  `tests/test_auth.py` and `tests/test_config.py`, so the cheap cost cannot become the
  default there).
- `tests/test_auth_api.py` is the perimeter: it walks the **served** route table and
  demands a session for every path outside the four public entries, and proves the UPS
  CSRF refusal on the wire — 403 after `instcmd` would be a power cut with an error
  message attached.
- **A cost claim is pinned as a shape and a control, never as a wall clock** (timings
  live behind `store_perf` and in the probe scripts, with budgets in a table).
- Coverage: 80 % is the repo floor, `bash .agents/validate.sh` enforces 94 %; new/modified
  Python must still clear > 90 % on its own.

## UI conventions (short)

- Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, no framework, no CDN,
  no build step.
- Colour comes **only** from the payload `severity` verdict, through
  `Dash.severityFrom()`; a missing number or verdict prints `n/a` uncoloured — never `0`,
  never green. A value the API should rate gets a row in `severity.py`, never a bound in
  JS.
- `Δ<window>` labels on measured rates print the window the estimator actually used (the
  payload's `window_seconds`), never the poll interval; `SIMULATED` badges on modelled
  data; per-source status dots in the header.
- **Flavour text is not helpful information**: copy survives only where the operator would
  act differently without it — a why-this-panel-is-empty sentence, a provenance tag, a
  measured `Δ` label, a unit. No pipeline narration.
- A wide table scrolls sideways **inside its own card** at ≤ 640 px; card headers wrap
  their facts; disclosures scroll what they reveal; nothing is painted past a card edge
  and eaten. The masthead is three rows, the two control rows flush right, and the bar
  is **not sticky**.
- An event feed is a **window** (50 per page, `Dash.eventsPager`), not a printout; `/ups`
  splits power events (behind a collapsed disclosure) from connection errors (a card of
  their own, hidden at rest).
- `/gateway` is two panels (one per config file), trunks per bound port with their
  configured routes branching off; its only colour pair is existence
  (`--gate-on`/`--gate-off`) — no verdict colour anywhere on the page; its filter box is
  an **eye, not a query** (it hides rows whose own payload text does not contain what was
  typed; no DNS, no re-ranking). Each row's switch runs on `GATEWAY_ALLOW_WRITES`: off
  is a `aria-disabled` reading with the flag's own sentence (no hover, no listener, no
  wire); on is a real control whose press is the dry-run diff → typed `DANGER` → one
  send, and the row's colour is the payload's word from the next poll — never the
  click's — with `off` reading `--gate-off` end to end (row, spine, the dialog's diff
  tints) and the diff scrolling in its own `<pre>`.
- Every chart names itself for assistive tech (`role="img"` + `aria-label`); every chart
  on every page has one, pinned in `tests/test_api.py`.

## Known gaps (the current state, one line each)

1. The UPS gates are the control: the upsd account has `instcmds = all`, and NUT has no
   call for a user's privileges — `enabled: true` cannot mean "may send", so the risk
   tiers and typed `DANGER` are load-bearing, not decoration.
2. One admin, one factor — no second factor and no per-user accounts; a logout ends only
   this browser's cookie, and the only revocation is a password change (which rotates the
   session secret). The throttle is per-process and keys on the socket peer address
   (behind nginx every login shares one bucket; a restart clears it).
3. GPU coverage is AMD only — an NVIDIA host would report no GPU at all.
4. UPS history depth: events and numeric series are stored; long-window capacity fade has
   the rollups but no dedicated view.
5. The store's cold-cache tail (a ~1.3 s whole-table re-derivation once per cache window)
   is a known cost, not a bug; the `database is locked` cascade is fixed by the windowed
   `maintain()`.
6. `driveripper`'s drives read live only because the unit runs as root with the bounded
   gate capabilities (kernel 7.2's unprivileged-passthrough gate); the installer
   template reproduces it with `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force`.
7. The fleet trusts **this** deployment's CA — a second deployment with its own
   `certs.*` rows cannot dial these agents until it carries the same root or re-issues
   and re-installs every bundle.
8. Route switching is closed, and its honest limits are the gap list's:
   `GATEWAY_ALLOW_WRITES` is off by default (a fresh deployment sees no switch arm at
   all); the switch is one admin (gap 2) — whoever holds the session can switch a route
   — with no per-host kill switch and no scheduler; and there is no runtime `disable
   server` state: the marker is a config line, so a hand reload or a reboot does not
   undo it (a runtime switch would be quietly undone by the next reload).

## Locked decisions (and why)

- **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10 s resolution, one file the
  container owns; rollups keep a year of charts at a few hundred MB.
- **Our own stdlib host agent, not node_exporter.** One JSON snapshot with exactly the
  fields the page needs versus ~1500 series and a re-labeling layer; no dependency to
  install, `ProtectSystem=strict` friendly.
- **Our own NUT client, not `pynut`/`upsc`.** The protocol is ~10-line commands; the
  third-party libs are subprocess-based or stalled; we need instant commands with
  explicit gating.
- **In-house Prometheus parser.** Only two upstreams use the format; keeps
  `prometheus_client` out of the dependency set.
- **`cryptography` for X.509, dashboard-side only.** Minting a CA is not something to
  hand-roll; the agent stays stdlib (it loads the PEMs with `ssl`), which is why the
  bundle is plain text and the agent runs on a host's Python 3.10.
- **Port 8090**, inherited from the `llm_monitor` prototype: 8000 is vLLM and 8731 is
  gufo in this fleet.
- **`.env` is a seed, never an override** — container/systemd env must win.

## Definition of done (per change)

`uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list
--outdated` empty if deps moved · dev server restarted and `/api/health` sane · UI change
screenshotted in a browser against **live** data (in `data/logs/`) · fixtures
re-captured if an upstream contract changed · this file and `README.md` updated when a
rule, command, or fleet fact changed.