dashboards/AGENTS.md

# AGENTS.md — homelab dashboards

The plan, the rules, and the environment facts an agent needs to work on this repo without
re-deriving them. `README.md` is the operator-facing how-to; this file is the why, the
invariants, and the state of the fleet.

## Working protocol

This project runs the `.agents/phases/` phased-execution pipeline.

1. Always read `.agents/PLAN.md` first to understand the project context and goals.
2. Follow the phased execution protocol in `.agents/phases/`.
3. Never modify `.agents/PLAN.md` or any files in `.agents/phases/complete/`.
4. If you need to update any file in `.agents/phases/todo/`, you must ask the user for permission first.
5. Strictly adhere to the **LOCKED DECISIONS** listed in `.agents/PLAN.md`.

Everything below this section is the project's own content: what the app is, the rules that
keep its numbers honest, and the state of the fleet it watches.

## What this is

Four dashboards for one homelab, behind one domain, in one process, backed by one SQLite
file — plus an overview page that says which of the four you need:

| Route     | Question it answers                                                            | Source                                            |
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
| `/`       | Is anything wrong, and where do I look first?                                  | all four, in one request                          |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history   | `agent/host_agent.py`, our own stdlib agent per host  |
| `/disks`  | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) |
| `/llm`    | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles      | gufo `/metrics` + `/slots`, vLLM `/metrics`      |
| `/ups`    | Battery health, runtime left, power events, battery-test buttons                | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |

History is a first-class feature, not a sparkline afterthought: every number worth graphing
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.

None of it is public. One admin signs in at `/login` (the only page an anonymous visitor
reaches), and the two admin pages live behind it: `/settings` — `⚙ Account` in the masthead —
where that session is read, ended and re-credentialled, and `/certs` — `⚷ Certs`, next to it —
where the fleet's certificate authority is listed, rotated, revoked and downloaded. Every page
above, `/settings`, `/certs`, and every `/api` route but the four named in invariant 12 require
the session.

## Non-negotiable rules

These are the things that make the numbers trustworthy. Breaking one is a bug even if the
page still renders.

1. **Read-only by construction, with exactly one exception.** Nothing in this app writes to
   an upstream except `POST /api/ups/command`. That endpoint is gated four times
   (`NUT_ALLOW_ACTIONS`, not `DASH_READ_ONLY`, the upsd.users account actually having the
   command, and a UI confirmation — `DANGER` for anything that can cut power) and the server
   re-checks every gate on every request. Hiding a button is not a safety control.
   Commands that would cut output are refused outright while the UPS is already on battery. Since
   phase 11 there are two more in front of it — a session, and this session's CSRF token for a
   cookie write (invariant 12) — and they are additive: `tests/test_auth_api.py` proves the
   `DANGER` requirement and the wire send still behave exactly as they did without them.
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
   as `0`. Every derived value carries the `source` string it came from and `derived: true`
   so the UI can print *how* it was obtained (`computed:Δtokens/Δt` labelled `Δ2.0s`, the LLM's
   own poll interval since phase 23). Aggregating
   two different series into one rate is the failure mode this rule exists to catch — the
   rate estimator is keyed by the *resolved* upstream series name **plus its label set**
   for exactly that reason (phase 13: a label set joining a summed counter mid-window
   contributes nothing until it has its own baseline, never its carried cumulative value),
   and the estimator refuses a window too short or too stale to carry the claim.
3. **Unconfigured is not broken.** A source with no env config answers
   `404 {"configured": false}` **without dialling**, and the page says "not configured".
   A red panel for a server the operator never installed destroys the signal from the
   panels that matter.
4. **One dead source must not blink another — and nothing may stop polling in silence.** Each
   source is its own asyncio task with its own timeout and health record (`poller.py`). A
   failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it —
   on `/` too, where a host that has never answered still gets a fleet row
   (`unreachable: true`, dimmed, every number `n/a`) because dropping the row would turn a dead
   agent into "never configured". Transitions write events; repeated failure writes one line,
   not 500. **The task is the source's only poller, so no exception may end it**: a store that
   refuses an otherwise good poll is a *reported* failure (`ok: false`, the real reason in
   `error`, `last_ok` frozen so the page ages the numbers, one `store-error` event then one
   `recording` event on recovery), and anything that escapes the poll body is logged and the
   loop continues. Four dev-server sources once froze behind an unretrieved SQLite exception
   while `/api/health` went on calling them `ok` — a frozen source that looks healthy is the
   one failure mode worse than an outage.
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
   charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
   no outbound access — a CDN reference is a request that never comes back.
6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets`
   moves it into an `Authorization` header, so `Target.url` ends up clean). Nothing that leaves
   the process may carry one: `/api/config` serves `Target.display_url` and
   `config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message
   is redacted too — those strings go into the health record *and* the `event` table, which
   `/api/events` serves to the browser and `page-system.js` prints. No token, password, or
   personal hostname goes into tracked files, fixtures, or the browser. **The admin's own
   credential is the same case** (phase 11): `auth.py` records nothing and answers
   `False`/`None` to every rejection, because the *reason* a login was refused is a verifier, a
   salt, or a session token — and this app turns reasons into health records and `event` rows
   that a browser reads. Its one `logging` call and its two `raise` sites (the fail-closed
   startup and the empty-new-password refusal) carry **fixed** operator text with nothing
   interpolated, not even a prefix of a hash; `tests/test_auth.py` pins that structurally, so
   the day someone adds `logger.warning(f"bad verifier {v}")` a test fails instead of a log line.
   **A private key is the same case** (phase 15): the root CA key and the dashboard's client key
   live only in `meta`, `certs.py` has no `logging` import and one fixed-text raise naming the rows
   to delete (pinned structurally by `tests/test_certs.py`, which also asserts the refusal quotes
   no line of the stored key), the root key is never written to the temp dir the client material
   lives in, and a bundle carries the root *certificate* — never the key — because an agent only
   has to verify its clients and a stolen agent install must not be a stolen CA.
   **A session token used to fetch a bundle is the same case** (phase 24): the installer's
   `--fetch-bundle <url> --cert-name <name> --session <token>` sends that token on exactly one
   request, hands it to its `python3` child through the **environment** rather than argv, prints
   only the URL in its log line, `unset`s it before anything is written, and never repeats it in
   the sudoers hint's re-quoted command (which names `--bind`/`--port` and nothing else).
   `tests/test_deploy_scripts.py` signs in for a real token and then greps every file the run
   could have touched plus both output streams.
   **A bundle exported for the dev loop is the same case** (phase 24):
   `scripts/dev-certs-export.py` writes only into the gitignored `data/certs/` tree, creates each
   file with `os.open` so `0600` is its *birth* mode (and re-chmods a file that already existed
   looser), dials `GET /api/certs/<name>/bundle` over HTTP instead of importing `ensure_certs` — so
   it cannot mint even by accident — and prints one line per host with no PEM, no token and no
   password in it (`tests/test_deploy_scripts.py` greps the output for all three).
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
   `sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
   2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
   `scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
   exporters) rather than trusting a README or a memory of an upstream.
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
   Check after touching dependencies: `uv pip list --outdated` must come back empty.
   Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
   `python3.14`) — do not lower these to make something else work.
9. **SQLite owns the history and must stay bounded.** Writes go through `store.py`; new
   metrics need a rollup path and a retention story, or a year of charts becomes a disk
   emergency. Put the DB on a volume in deployment.
10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`,
    http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot
    before they are called done. `data/logs/*.png` is where those land.
11. **The dashboard never starts open, and the stored password beats the environment**
    (phase 11). The admin verifier and the session-signing key are durable rows in the SQLite
    `meta` table (`auth.verifier`, `auth.session_secret`) and are the only place either secret
    lives; `DASH_ADMIN_PASSWORD` is a **seed** that hashes into the row on the first boot of a
    database and is ignored thereafter (one warning line, value never quoted), so a password is
    changeable without a redeploy and no container env can roll it back. A server with neither a
    stored verifier nor the seed **refuses to start** — a `RuntimeError` out of the lifespan, a
    non-zero exit, and a message naming the variable, the meta key, and the
    `sqlite3 <db> "DELETE FROM meta WHERE key IN ('auth.verifier','auth.session_secret')"`
    reset command. "Start open with a warning" is not an option: the domain is public and the
    fleet's hostnames and UPS controls are behind that login. `testpass` is the local-testing
    password (dev config and the hermetic suite) and never a production fallback. Invariant 12 is
    the request-time half; this one is why there is always a credential to check against.
12. **The gate is default-deny and its public surface is four entries** (phase 11). `app.py`'s
    `PUBLIC_PATHS` (`/login`, `/api/auth/login`, `/api/health`) plus `PUBLIC_PREFIXES`
    (`/static/`) are the only things an anonymous caller can reach — every page, every `/api`
    data route, and `/api/auth/status`/`logout` require a session read from the
    `dash_session` cookie or an `Authorization: Bearer` token. `/api/health` is the one data
    route that stays open on purpose (the Containerfile `HEALTHCHECK` and
    `./scripts/dev-server.sh status` poll it unsigned, and it leaks nothing); its shape is now a
    contract with deployment, which means **nothing may be added to it without re-deciding that**
    — a new field there is a field the whole internet reads. Adding a path to that list is a
    security decision, and
    `tests/test_auth_api.py::test_every_route_that_is_not_public_requires_a_session` walks the
    **served** route table so a route that forgets the gate fails a test instead of shipping —
    the middleware guards by default, which is why no new route has to remember to. Rules the
    gate keeps and each has a test: a page `GET` gets `303` to `/login?next=…` (query string
    included) and everything else the one JSON refusal, because a redirect a script cannot render
    is not an answer; every answer carries `no-store`; `?next=` passes `auth.safe_next` and is
    never followed raw (an open redirect on a login page is a phishing kit with a homelab theme);
    a write additionally needs a matching `Origin`/`Sec-Fetch-Site` and — when it came in on the
    **cookie** — this session's `X-CSRF-Token`, because a bearer caller is something no other site
    can forge. A process whose lifespan never produced `ctx.auth` answers `503` for **everything,
    public paths included**: "healthy" from a server that cannot say who its admin is is the
    invariant-4 failure mode, and a login form that cannot verify a password is a worse promise
    than no form. `POST /api/ups/command` keeps all four of its own gates behind this one (the
    `DANGER` word is still what cuts power), and `POST /api/auth/logout` is gated like any other
    write because a sign-you-out link is a CSRF. `POST /api/auth/password` (task 04) is the third
    gated write and the app's only revocation: guesses at the current password spend their own
    `password:<address>` bucket, and a right one replaces `auth.verifier` **and**
    `auth.session_secret` so every browser that is not in front of the form dies on its next
    request — which is why the response is minted from the new state and never the one being
    replaced.

## Architecture

```
agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        │                                  │ + /slots (gufo)        │
        ▼                                  ▼                        ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
        │  parse + Δcounter/Δt                                                     │
        ▼                                                                           ▼
sources/host.py · disks.py · llm.py · nut.py                         poller.py (task per source)
        │        Result(payload, samples, events)                                    │
        ▼                                                                             ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/
```

Responsibilities, one line each:

- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
  `statvfs`; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an
  HTML status page at `GET /`. Refuses to bind off-loopback without a token **or** a certificate
  bundle (exit 2). Counters stay counters: the dashboard computes
  rates so the agent never remembers who polled it last.
  **TLS mode (phase 15 task 02, `AGENT_VERSION` 1.1.0):** `--cert-bundle`/`AGENT_CERT_BUNDLE`
  loads the one PEM the dashboard signs (`resolve_bundle`, the `resolve_token` idiom: server cert
  + server key + root CA + `# BUNDLE-REV <n>` + `# REVOKED <serial>` lines, grepped by text
  because the agent has no `cryptography`), unpacks it into the unit's `RuntimeDirectory` when
  there is one else a temp dir, builds a `PROTOCOL_TLS_SERVER` context with
  `verify_mode = CERT_OPTIONAL`, and **deletes the PEMs once loaded** (the key then lives only in
  this process, so a `kill -9` has no cleanup to forget). `CERT_OPTIONAL` is the assumption that
  keeps `/health` probeable: the *handler* enforces, refusing a peer with no verified certificate
  403 `client certificate required` on every path but `/health`, and refusing a serial on the
  bundle's denylist 403 `client certificate revoked` on **every** path including `/health` (the
  exemption is for a probe holding the CA and no certificate, not for an identity the operator
  withdrew). The denylist is re-read when the bundle's `(mtime_ns, size)` stamp changes — that is
  the whole reason the `# REVOKED` lines live in the bundle — while a new certificate needs a
  restart, because `ssl` cannot swap a chain under a live socket. Serial matching accepts the hex
  `getpeercert()` renders *and* the decimal `certs.py` writes, because comparing one spelling to
  the other accepts a revoked client forever. On the TLS path the bearer token is **not** read (the
  handshake is the auth, and a second credential that outlives a revocation defeats the list),
  `/snapshot` gains additive `tls: true` + `bundle_rev` only in TLS mode, and token mode is
  byte-identical to before. Task 03 has since flipped the collector (see the `collectors.py`
  bullet), so those two fields are how the dashboard answers "is this host running the bundle
  `/certs` issued?" — `sources/host.py` and `sources/disks.py` carry them into both payloads as
  `agent.tls` / `agent.bundle_rev`, present only when the agent published them.
  SMART (phase 09): `smart()` runs `smartctl -i -A -H` per whole `/sys/block` disk
  (`parse_smart_output` covers ATA + NVMe; NVMe media errors map to
  `offline_uncorrectable`, hex-packed vendor raws are never decoded, and an attribute row is
  read from its **RAW_VALUE column** rather than the end of the line because this binary
  annotates that column — `35 (Min/Max 34/44)`, `43894h+43m+32.519s` — so its last token is
  not the value), 5 s per device / a pass budget whose 20 s floor scales to 5 s × the disk count
  / 240 s cache, attached as `disk_io.devices[].smart` (gap 10: a 16-drive shelf takes longer
  than the floor, and truncating the pass would null the tail drives). Missing binary →
  **one** `errors` line for the pass (the word `not found` in it is what
  `sources/disks.py` reads to answer "is smartctl installed", and
  `tests/test_agent.py` pins that the line still says it); one denied/wedged device → that
  disk's `smart` is `null` with its own line; `--no-smart` opts out silently. Never fails
  the snapshot. **An expired cache refreshes on a background thread while the request is
  served the previous result** — only the first pass after startup runs inline — because a wedged
  pass on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the
  *system* source red every four minutes over an add-on that page never draws. The refresher
  is single-flight and keeps the old cache on failure, so a bad pass retries on the next
  expiry instead of hammering the disk.
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
  env var always wins. Defaults are safe except the sources, which default to unset. The auth
  block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means
  *unset* rather than "the password is empty"), `DASH_SESSION_HOURS` (720 = 30 days: a dashboard
  that logs the operator out mid-outage gets bypassed), `DASH_COOKIE_SECURE` (off, because dev is
  `http://127.0.0.1`; task 03 also treats `X-Forwarded-Proto: https` as secure), and
  `DASH_PBKDF2_ITERATIONS` (default `auth.PBKDF2_ITERATIONS`, imported so the primitive and the
  setting cannot drift) — that knob exists so the hermetic suite hashes in microseconds instead
  of 0.3 s per client and is not a production tuning parameter.
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
  `/disks` is a **source of its own** (`disks:<host>`, `DISKS_INTERVAL_SECONDS` 300 s,
  `DISKS_TIMEOUT_SECONDS` 30 s, events under the dashboard name `disks`) reading the same
  agents the `system` jobs read: the separate task is what keeps a SMART poll out of the
  10-second utilisation loop and lets `/disks` age on its own instead of blinking `/system`.
  The job list is built once (`app.build_hub()`) because it is wired twice — production and
  the injected-store test path — and two hand-written copies drifting is how a source ends up
  polling in tests and not in the container. The
  success path records (payload → samples → events) and only *then* moves `last_ok`, so a
  refused SQLite write ages the source and alerts instead of leaving it current with charts
  that have silently stopped drawing.
  Events are stored under the **dashboard** name (`system`/`disks`/`llm`/`ups`), not the sample
  source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
  (found on `/system` during phase 03 browser verification).
- `collectors.py` — the fetch half. `AgentTls` (phase 15 task 03) is what makes an `https://`
  `HOST_TARGETS` entry a **certificate-authenticated** dial: `verify` and `cert` are the paths
  `certs.write_material` wrote at startup, and the dial is verified against this app's own root
  and presents this app's own client certificate. Two things about it are load-bearing. (a) The
  credential travels in a **client built per dial**, because `httpx` resolves TLS material when a
  client is *constructed* (there is no per-request `verify`/`cert`, and `verify=<str>`/`cert=` are
  deprecated anyway — the supported form is an `ssl.SSLContext`, which is also the honest shape:
  trust in one named CA, hostname checks on) while one shared client dials the plain agents and
  both engines — a `verify` on *that* would make every engine read depend on a CA nobody issued
  their cert with. (b) `status` is a **callable** doing a fresh `certs.agents` read, so a
  revocation clicked in the browser bites on the next poll with no restart, and a revoked host is
  refused **before any socket opens** — invariant 3's shape: a source the app knows it must not
  talk to says so without a network round trip, and the poller's stale/last-good semantics do the
  rest. `refusal_detail()` quotes the agent's own 403 sentence (`client certificate required` /
  `client certificate revoked`) because those are two different operator actions, and the
  certificate wording is used only for a dial where a certificate was presented — a 401/403 from a
  token-mode host keeps the token/HTTP wording, or the operator is sent to `/certs` for a token
  problem. `agent_tls_map()` builds one entry per configured host (the CA and client pair are
  shared — the dashboard has one identity — only the revocation answer is per name, and an absent
  entry is *active*, never-issued being a different fact from revoked); `build_hub` takes that map
  and hands the **same** entry to the `host:` and `disks:` jobs of one host, so a revocation
  fails both sources with one sentence and neither of them dials. `fetch_snapshot()` keeps
  `http://` targets on the shared client with their bearer token, unchanged from before the phase,
  so a rollout is per host.
  `NutMonitor` owns per-UPS transition bookkeeping (its
  self-test event takes its severity from `nut.self_test_outcome()` and not its own word list,
  so a passed test is never an amber line in the feed while the card beside it reads green).
  Its per-UPS tolerance for a failed `LIST CMD`/`LIST RW` is for a *complete* error reply only:
  one battery whose command list is denied loses its buttons and the poll goes on, while a
  **stalled** reply fails the poll, because `NutClient` marks a socket it could not read to its
  `END` as unusable (a reply still arriving would otherwise be taken for the *next* command's
  answer — the real cause of the `expected 'BEGIN LIST VAR', got 'RW apc …'` lines this fleet's
  event table used to fill with, and the same hazard on `INSTCMD`, where a late `OK` read as a
  different command's success would be a gate that lies).
  Failure messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one
  that was dialled, because they become health records and event rows. `fetch_snapshot()` is
  the one agent `/snapshot` read that `system` and `disks` share, so a dead host says the
  same sentence on both pages and only the `timeout` differs (the disks poll has to outlast
  the agent's smartctl pass, and `None` is translated into httpx's sentinel because a
  literal `None` means *no timeout at all*). Agent `errors`
  become the payload's `degraded` **except** the `smart:` add-on lines — a host without
  smartmontools or device read access is unconfigured, not broken (invariant 3); they stay
  in `agent.errors` for the `/disks` page to explain — which since phase 22 is only the
  device-read case, because a host whose agent said `smartctl` is *missing* is not listed on
  that page at all (`api._disk_visible`).
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
  (`gpu.busy_pct@<index>`). `agent_tls_fields()` is the phase-15 passthrough: a TLS agent's
  `tls`/`bundle_rev` ride into the payload's existing `agent` block, and a token-mode snapshot
  grows nothing (no `tls: false`, no `bundle_rev: null` — rule 2), so the fields are how the API
  distinguishes a stale bundle from an agent that never had one.
- `sources/disks.py` — the same snapshot projected onto **physical** drives for `/disks`
  (phase 09): one row per whole-`/sys/block` disk (never a partition, never a mount), the
  agent's `smart` block carried through verbatim, `health` the drive's own word normalised to
  `ok`/`failed`/`None`, and `smartctl_available` three-way on purpose — `True` (a disk
  answered), `False` (the agent said the binary is not installed), `None` (an agent predating
  the pass, or `--no-smart`: nobody knows, so `n/a` plus the agent's own `smart_errors` line
  and never a guessed verdict). Since phase 22 the flag's *page-side* consequence is not only
  the why-line: `api._disk_visible` reads `False` as "not a host on `/disks` at all", while
  `None` keeps its dimmed row — the computation here is unchanged and the agent still reports
  all three arms whatever the API decides to list. **Nothing here
  is a rate**: power-on hours and the sector counters are stored counters-as-reported, so the
  source takes no `RateEstimator` and its payload carries no `window_seconds` (rule 2 — a rate
  with no estimator, no `Δ<window>` label and no `source` string is an invented number).
  `disk.health_ok@dev` is not a series either: a verdict is not a number, so the drive's word
  travels in the payload only. Each row does carry a `severity` dict (`SEVERITY_FIELDS` = the
  two rated numbers, `temp_c`/`percentage_used`, plus `health` through `HEALTH_SEVERITY`,
  which translates the drive's own word instead of grading it), and `UNGRADED_FIELDS` names
  the numbers that must never be graded. `fleet_disk_summary()` sums the per-host `summary`
  counts for the page header and drops a host with no payload from the sums while still
  counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does
  arithmetic and a dead agent never shrinks a fleet total.
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
  `VLLM_HISTOGRAMS`), measured Δ rates **diffed per label set** (`rate_key()` keys the
  estimator on the resolved name plus its labels; the cell is the Σ of the per-series
  rates over the widest window one of them actually used — phase 13, so a joining
  `{engine="1"}` series can never lend its carried counter to one interval's delta),
  prefill speed as `mean(prompt tokens)/mean(prefill time)`, session list from gufo
  `/slots`. A refused window (the estimator's floor) is still an answer: the cell keeps
  its `computed:Δ<series>/Δt` source and its honest `Δ<window>` beside a `null` value.
- `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not
  a mystery score), `SELF_TEST_OUTCOMES` + `self_test_outcome()` (the `Last self-test` verdict
  follows the *outcome* of the driver's `*.test.result` word, not its phrasing: pass words →
  `ok` (including `Done and passed`, what the CyberPower here answers), fail words → `fail`,
  `No test initiated` / empty / no variable → `unknown` printing `NEVER_TESTED_DETAIL` so both
  "never tested" states say one sentence, and an unrecognised word → `warn`, because a driver
  inventing a verdict is drift worth noticing and the check prints its own word to be added),
  `COMMANDS` catalog with risk tiers, `COMMAND_ALIASES` + `resolve_command()`
  (drivers answer two names for the same action — `beeper.enable`/`beeper.on`,
  `beeper.disable`/`beeper.off` — so one entry is offered per action and any of its names hits
  that entry's verdict, never a gate the rendered name missed), `available_commands()` filtering,
  kelvin-vs-celsius normalisation, and `charger_status` normalised to the `CHARGER_STATUS`
  keys (the same lowercase token `states()` and `on_battery` already compare), and `states()`
  publishes one chip per fact — a driver exporting both `battery.charger.status` and the
  legacy flag that says the same word gets "charging, online", not "charging, online,
  charging". `battery_runtime`
  comes from `battery.runtime` **only** — `battery.runtime.low` is a shutdown threshold, not
  time left, so it lives at `battery_runtime_low` and never becomes the hero's "runtime left".
- `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a
  battery. It must never be the default, and every payload it produces says `simulated`.
  Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that
  ages), so a fresh simulator reads healthy and amber keeps meaning something. Its status
  follows the real drivers: a below-float pack on mains reports `OL CHRG`/`chg` (the glyph's
  bolt state is keyed on that token, and the model has to be able to produce it) and a battery
  being drawn from is `OB`/`dchg`, never `ol`. Every command it advertises is implemented and
  visible in a variable a real driver publishes — the beeper ones set `ups.beeper.status`
  (`enabled`/`muted`/`disabled`), and a test window that ends writes `ups.test.result: PASSED`
  while an aborted one writes nothing; the load/shutdown commands answer `OK` without modelling
  an outlet, because a faked power cut in the history is indistinguishable from a real one.
  `from_env()` is the single place the `NUT_SIMULATE` truthy rule lives — `app.py` builds the
  monitor through it, so the assembly and the model cannot disagree about what is enabled.
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
  `scalar(agg="sum")` stays right for **gauges**; `values_by_series()` is the per-label-set
  view **counter** deltas need, because a summed counter mixes whatever label sets exist now.
- `rates.py` — Δ/window estimator, keyed by resolved series name + label set. Its effective
  window is `now - base[0]` **bounded on both sides** (phase 13): under
  `MIN_WINDOW_SECONDS` (the poll loop's 0.2 s catch-up gap after any slow fetch) it answers
  `(None, dt)` — one `n/a` with the window it refused — and drops that base so the polls
  *after* the stall cannot re-diff it; the window-empty fallback accepts a base no older
  than `MAX_WINDOW_RATIO × span` (the `2 × span` ceiling the docstring always claimed, now
  enforced). A rate ≥ 8× the previously published one is held for one
  poll (the previous value repeats, label unchanged) and publishes only if the next interval
  confirms it (`JUMP_CONFIRM_RATIO`) — and **confirmation means a different anchor**: a held
  jump remembers the sample its window was diffed from, and stays held while that same sample
  still anchors the window. A stalled fetch stamps its body with the time the poll *finished*,
  so that sample is the stall older than its own timestamp, and every later window off it is
  the same skewed delta re-diffed over a longer span — measured twice from one sample is one
  measurement, and the hold simply continues until the stale anchor ages out of the span.
  `min_window=0.0` exists for tests that scrape faster than any real loop. **The span belongs to
  the cadence that fills it** (phase 23): `app.build_hub` builds one estimator per measuring
  cadence — the host's at `RATE_WINDOW_SECONDS` (10), the LLM's at `LLM_INTERVAL_SECONDS` (2) —
  because a window wider than the poll averages bursts across the polls the operator is watching
  and a window narrower than it has nothing inside it to diff. Nothing else about the estimator
  moved: the floor, the ceiling, the jump gate and the label are the same code, and a 2 s window
  is 4× above the floor, so a healthy 2 s cadence is never refused.
- `auth.py` — every authentication decision as a pure function, stdlib only (phase 11):
  `hash_password`/`verify_password`/`needs_rehash` over a self-describing
  `pbkdf2_sha256$<iterations>$<salt_hex>$<hash_hex>` verifier (so the cost can rise without a
  migration), `mint_session`/`read_session` for an HMAC-signed stateless cookie (no session
  table and nothing to prune; rotating the stored secret signs every cookie ever issued out,
  which is how changing the password evicts every other browser), `mint_csrf`/`verify_csrf`
  bound to the session token instead of stored, `LoginThrottle` (in-process exponential
  backoff), and `origin_matches`/`sec_fetch_site_allowed` as string functions so the gate is
  testable without a `Request`. The primitives **never raise and record nothing**: a malformed
  verifier, a tampered cookie, and a rotated secret all answer `False`/`None`, because the
  reason an input was rejected is the credential (invariant 6). No HTTP, no SQLite driver, no
  new dependency — `app.py`'s middleware and `api.py`'s `/api/auth/*` are the callers, so a
  decision made here is made once and is unit-testable without a `Request`. It also carries the
  two strings the gate and the pages share (`LOGIN_PATH`, `LOGIN_NEXT`) and `safe_next()`, the
  only judge of a `?next=` value: kept verbatim (query string and all, because the range chips
  live there) when it stays on this app, dropped otherwise — scheme-prefixed, scheme-relative,
  the percent-encoded spellings of both, anything with a control character (not a path but a
  second `Location` header), and anything without a leading `/`.
  The **verifier bootstrap** (task 02) is the one deliberate exception to "never raises":
  `load_auth_state(store, admin_password=…, iterations=…, session_hours=…, db_path=…)` reads the
  `meta` rows `auth.verifier` / `auth.session_secret` through a three-method `MetaStore`
  `Protocol` (so this module still imports nothing in the package, and its tests use a dict)
  and returns a frozen `AuthState` — the environment **seeds** a password once and the stored
  verifier wins thereafter (`env_ignored` and one fixed-line warning when both are present, so
  `/settings` is the real answer and no redeploy can roll a changed password back); no row and
  no seed raises a `RuntimeError` naming `DASH_ADMIN_PASSWORD`, the meta key, and the
  `sqlite3 … DELETE FROM meta …` reset command, because "start open with a warning" is the
  failure this phase exists to prevent; a stored secret that is not 256 bits of hex is
  regenerated and persisted (the effect is one login prompt, never a lockout). Nothing here
  reads a clock, so the state carries no expiry — a session's is inside its signed payload.
  `update_verifier(store, state, new_password)` writes the new verifier **and** a fresh secret
  (secret first, so a crash cannot leave sessions outliving the password they were issued
  under) and refuses an empty/whitespace password; it is what `POST /api/auth/password` calls,
  because rotating the secret **is** this app's revocation list — a stateless cookie has no row to
  delete, so without it a stolen session would outlive the password that "revoked" it for another
  `DASH_SESSION_HOURS`. Its two raises, like the loader's, carry
  fixed text — `tests/test_auth.py` pins the file's raise sites and its one logging call
  structurally, so no future edit can interpolate a credential into either.
- `certs.py` — the fleet's certificate authority as the same shape of module (phase 15): a pure
  lifecycle over the three-method `MetaStore` protocol, no clock it cannot be handed, no `logging`,
  and one fixed-text `RuntimeError` (`tests/test_certs.py` pins all three structurally, the way
  `tests/test_auth.py` pins `auth.py`). Five `meta` rows and nothing else:
  `certs.root.key`/`certs.root.cert` (self-signed `CN=dashboards-root`, 10 years),
  `certs.client.key`/`certs.client.cert` (`CN=dashboards`, 1 year — the identity every agent will
  check), and `certs.agents`, one JSON object mapping a `HOST_TARGETS` name to
  `{serial, not_after, status, bundle_rev, bundle, san}`. **A stored row always wins:** the root is
  generated on the first boot of a database and never on a later one, because a CA that rotates
  with the process untrusts every bundle already installed on a host — and a newly generated root
  takes its client certificate with it, since a leaf signed by the dead root is a credential
  nothing can verify. `issue_agent_cert`/`reissue_agent_cert` (the API's "rotate") sign a fresh
  `CN=<host>` leaf whose SAN is the addresses that host is dialled by — the name is always in the
  list, and an IP goes in as an `iPAddress` or hostname verification of an IP dial fails — bump
  `bundle_rev`, and reset `status` to `active`. `revoke` marks the host and **re-seals every
  bundle**, because the `# REVOKED` denylist lives inside each of them and an agent that cannot see
  a serial keeps accepting that client; there is no "unrevoke", and revoking a never-issued host
  still records the revocation (what a revoke enforces is the dashboard refusing to dial). The
  **bundle** is the install artifact — agent cert, agent key, the root *cert* (never the root key,
  asserted by header count and by parsing the one key against both keypairs), `# BUNDLE-REV <n>`,
  then the denylist — stored verbatim so what an operator downloads is byte-identical to what the
  row holds. `rotate_root` mints a new root plus a new client and reissues every agent
  (`bundle_rev` +1 each, denylist reset with the generation); the running process keeps the
  material it loaded at startup, which is the honest `restart_required` task 04 answers with.
  Every certificate carries `SubjectKeyIdentifier` and every leaf an `AuthorityKeyIdentifier`
  naming it, non-critical: OpenSSL's **strict** mode refuses a leaf with no AKI
  (`CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier`) and Python's
  `ssl.create_default_context()` — the context the dashboard's own https dial builds — turns that
  mode on. The agent's server-side check is not strict, so nothing on the agent side notices;
  `tests/test_certs.py` pins the identifiers instead, and a bundle recorded without them is one a
  strict client rejects. `write_material` puts the CA and the client pair in a `0700` temp dir with
  `0600` files for the dialing — the root key is never written to disk, so its only copy stays the
  `meta` row — and the lifespan removes the directory on shutdown. Unreadable PEM, a key that does
  not match its certificate, or a leaf the stored root did not sign is a **fail-closed startup** (the
  `ctx.auth` rule), and the message is the `sqlite3 "<db>" "DELETE FROM meta WHERE key LIKE
  'certs.%'"` reset. `cryptography` is this module's dependency and the app's only new one;
  `agent/host_agent.py` stays stdlib and loads the PEMs with `ssl`. There are **two reads** and the
  difference is a side effect: `ensure_certs` is the startup one (it mints what the table is missing,
  which is why it is what `app.build_certs` calls and nothing else may), and `read_state` is the
  request-time one task 04's `GET /api/certs` uses — same validation, same fixed-text refusal, never
  a write, so a page refresh cannot bootstrap an authority the operator deleted. The material is
  read with (task 03): `collectors.agent_tls_map` turns `AppContext.certs` into the per-host dial
  pair, and because the revocation answer is `agent_status()` called per poll, a revoke in this
  module is what stops the dial — no restart, no cache, no second list to keep in sync. There is a
  **third door and it is deliberately outside this module** (phase 24):
  `scripts/dev-certs-export.py` is the dev loop's exporter — stdlib `urllib`, signs in through
  `POST /api/auth/login`, reads the token out of the `Set-Cookie`, and downloads each
  `HOST_TARGETS` name's bundle into gitignored `data/certs/` at `0600`. It cannot mint because it
  imports neither read; it dials the route the `/certs` page downloads from, which is what makes the
  exported file byte-identical to the row. `scripts/dev-server.sh` calls it after the readiness
  probe and treats a failure as a warning line — the server being up is the thing the probe proved.
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
  `classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
  beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
  an unknown. The verdicts travel **additively** in payloads — `cpu.severity`,
  `memory.severity`, `disks.severity`, each UPS card's `severity` (charge/load/runtime),
  `severity` inside rated LLM cells (KV pressure, classified on a
  percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
  row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
  There is **no `gpu.*` row and no `gpus[].severity`** (phase 06): these are LLM servers,
  a busy GPU is the job at every level, so the GPU card and the fleet GPU column render
  uncoloured — like `net.*`, whose colourlessness predates it. Removing the verdict is not
  removing the data: `gpu.busy_pct@N` / `gpu.mem_used_pct@N` are still stored and charted.
  The `/disks` rows (phase 09) are `disk.temp_c` (50/70) and `disk.percentage_used` (80/100)
  — an SSD at 100 % of its rated wear is a replacement, not a warning — attached per row by
  `sources/disks.py`; the sector counters and `disk.power_on_hours` deliberately have **no
  row**, because a non-zero counter is an alarm chip on presence and zero is not an
  achievement (the `/llm` preemption-chip rule), so `classify("disk.reallocated", 1)` is
  `None` and the page shows a chip instead of a colour.
- `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload,
  recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts`
  (`{id, severity, source, message, href, at}`, fail first then warn, then by source). A
  pure projection: no I/O, no store, no thresholds of its own — it reads job states for
  reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions.
  Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but
  older than 3× its own poll interval is `warn` while a source that stopped answering is
  `fail` (the interval is the job's own dashboard's, so `disks` goes amber after 15 quiet
  minutes rather than 30 seconds); a dead source gets exactly one line (its stale payload
  adds nothing) that names it once — the poller's `"UpstreamError: "` wrapper and the
  duplicated host name are stripped, because a message that says `gateway` twice reads as two
  problems, and the same rule silences the **`disks` echo** of a dead agent (the disk job
  dials the same `/snapshot`, so its failure carries no new information — while a `disks` job
  failing *while its host answers*, SMART wedged inside the handler, is the only voice that
  fault has); `unknown` and missing numbers are silence; one alert per (source, metric) and at most 12, the last being a marker
  that says how many it hid. This is why the overview can be two panels. There is **no
  disk-health rule and none is wanted**: a drive the hardware itself called `FAILED` is a red
  chip on its `/disks` card and a line in that page's event feed, not an overview alert —
  `severity.py` rates no sector counter and this module may not have a threshold of its own, so
  grading a drive here would be the bug, not the fix (phase 09). Its `HREF` table does carry
  `/disks`, so the one alert a disk job *can* raise — SMART failing while its host answers —
  links to the page that names the drive.
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning. The
  `meta` table is the only durable key/value state in the app and is reached through three
  named helpers — `get_meta`/`set_meta`/`delete_meta` (the last returns its rowcount, because
  the password-reset procedure has to be able to say whether a row was there, and one key per
  call, because a reset that deletes a row it did not name deletes the operator's credential).
  `stats()` reads `last_maintain` through `get_meta()` too — outside its own lock, which
  `get_meta` takes itself and `threading.Lock` is not reentrant — while `maintain()`'s write
  stays inline inside that method's transaction. **No per-request path may pay a whole-table
  scan** (phase 14 task 02): `series_names()` (1.4 s of `SCAN sample` on a production-sized store,
  behind `/api/metrics`) and the two `COUNT`s behind `stats()` (1.25 s, run by `app.js`'s
  `/api/health` poll on every tab every 15 s) answer from short-TTL caches — 60 s and 30 s, the
  series list additionally invalidated **only by a write that can change it** (task 04: an insert
  cannot remove a name, so `_note_series_write()` ends the cache on a `(source, host, metric)` the
  store has never written and leaves a poll of an existing series alone — invalidating on every
  sample was task 02's defect, and it put the 572 ms scan back on nearly every `/api/metrics` call
  on a store nine pollers write to, behind a cache that never served; `delete_series()`/`maintain()`
  invalidate unconditionally because a `DELETE` can take a name, `delete_series()` also forgetting
  the triples it emptied so a re-created series counts as new again), through a generation counter so
  a write racing a read cannot park a stale answer under a fresh timestamp, with the name cache
  bounded to 256 `(source, host)` keys because `/api/metrics` takes both as free text and the
  existence tracking capped at `SERIES_TRACKED_SERIES_MAX`, below which it falls back to
  invalidating on every write. A tracking set made wrong by *another* process deleting rows costs at
  most one TTL window, which is the reason the cache has a TTL and not only the tracking. `last_maintain` and `size_bytes` are never
  cached: a `last_maintain` that stops moving is how an operator spots the invariant-4 failure
  mode, and it costs a three-row lookup. The resolution decision is one compound statement of
  per-`(host, metric, res)` branches whose `COUNT` is capped by `LIMIT max_points + 1` — exact,
  because the decision only ever asks whether a count is `0`, `≤ max_points`, or neither — and
  `pick_res_many()`/`history_many()` are the batched door the N × M endpoints go through. The rows
  stay a per-series seek **measuredly**: a `ROW_NUMBER()` window or a compound of the 27 tail
  selects both cost more than 27 index seeks on the seeded store (`data/logs/NOTES-store-perf.md`
  records the table), so batching the *decision* is the fix and batching the *fetch* would be a
  round trip traded for work.
  **The write side is windowed too** (phase 14 task 03): `maintain()` reads `last_maintain` through
  `get_meta()` *before* its transaction (same non-reentrant-lock rule as `stats()`) and bounds each
  rollup step to what became due since, instead of averaging every row the resolution retains —
  3.97 M rows of 1-minute history per minute, a 5.4 s write transaction that made a poller's
  `add_many` outlive `busy_timeout=5000` (gap 4's `store-error` line). Two properties make that a
  cost bound and not a data change, and both are pinned: the lower bound reaches **past** the
  previous pass (one bucket plus slack, so ten quiet minutes are a catch-up scan rather than ten
  minutes of empty charts), and it is snapped **down to a bucket boundary** — the step's ceiling is
  a row bound, so the newest bucket is partial and a later pass re-rolls it as its rows land, and a
  floor that cut into a bucket would overwrite the fuller average with the mean of its tail rows
  (2.5 becoming 4.5 on a six-sample minute: rule 2 with a chart's clothes on). A window is only a
  cost bound if the plan can seek to it, and two things were in the way. The prune keeps its full
  retention bound — a rollup may window because it only adds rows, a prune must reach anything that
  expired — but it is **one `DELETE` per resolution**, because one
  `WHERE (res=? AND ts<?) OR (res=? AND ts<?) OR (…)` is unreadable to the planner and read the
  whole 6.4 M-row table to delete nothing (554 ms of a 562 ms pass; 0.3 ms per statement now). And
  the store had no query statistics, so every window was priced as a scan however few rows were
  inside it: `maintain()` ends with `PRAGMA optimize` and `__init__` sets `automatic_analyze=0`,
  because SQLite's own refresh lands inside whichever poller's commit trips its counter — the exact
  path this shortens — while a tick a minute costs ~100 ms once and 0.0 ms afterwards. Measured on
  the seeded store: **5,368 ms → 2.7 ms** steady, 48 ms for a ten-minute catch-up, and the
  two-connection race at the app's cadence goes from 5 of 9 inserts raised to 100 of 100 landing
  with a 21 ms worst wait. `__init__` also sets `wal_autocheckpoint` (SQLite's own 1 000 pages,
  written down so the knob is visible) plus `journal_size_limit = 64 MB`, the cap that truncates a
  runaway WAL at the next checkpoint. `busy_timeout=5000` is unchanged: the fix is a shorter write,
  not a longer wait.
- `api.py` — GETs everywhere plus the one POST that dials an upstream, and the `auth_router()`
  that creates, describes, **re-credentials**, and ends a session (`POST /api/auth/login` public;
  `GET /api/auth/status`, `POST /api/auth/logout` and `POST /api/auth/password` protected). Those
  three, and the `certs_router()` of phase 15 (`GET /api/certs`, `GET /api/certs/<name>/bundle`,
  `POST /api/certs/<name>/rotate`, `POST /api/certs/<name>/revoke`,
  `POST /api/certs/root/rotate`), touch this process and its `meta` table and dial
  nothing — invariant 1 is about the fleet, and `tests/test_api.py` pins the three lists separately
  rather than widening the one that names `/api/ups/command`.
  The **`/api/certs` surface** (phase 15 task 04) is the CA's own admin half. The list reads the
  **stored** state through `certs.read_state` — the request-time twin of `ensure_certs`, which
  mints and must only ever be called at startup, because a `GET` that bootstrapped a root CA would
  replace the authority nobody asked it to replace — and its rows are the union of `HOST_TARGETS`
  and `certs.agents`, so a configured-but-uninstalled host appears as `status: "unissued"` with
  nulls (invariant 3, rule 2) and an entry whose host left the config is listed but not writable.
  `deployed_bundle_rev` comes from that host's `system` payload `agent` block (task 03's
  passthrough), never from what the CA hopes. `bundle_rev` is what was issued; the two numbers side
  by side are the "install the new bundle" sentence, and neither carries a `severity` — an expiry
  and an identity counter are not graded numbers. A `<name>` outside `HOST_TARGETS` is refused in
  the route's first line with `404 {"configured": false}` **before** any signing, dial, or store
  write (spy-pinned both ways, and the same gate makes `GET …/bundle` answer
  `{"configured": true, "issued": false}` for a host that is configured and has nothing issued —
  two different facts, two different bodies). The bundle download is the stored row **verbatim** as
  `application/x-pem-file` with an `attachment` filename, because it is `install-agent.sh`'s input
  and it carries that host's private key (session-gated, `no-store`). `POST /api/certs/root/rotate`
  answers `restart_required: true` rather than pretending a hot-swap: the CA and the client pair are
  startup material, so the table is new at once while the process keeps dialling with the old one.
  Every write logs one fixed-text `certs`-source event naming host and action, and serials travel as
  **decimal strings** — a 64-bit certificate serial through a JS number is a serial a page cannot
  match against a bundle's `# REVOKED` line (invariant 6: no PEM reaches an event, a health record,
  or any payload but the bundle route's).
  The login route is where a *sign-in* password is compared: it
  throttles before it hashes, refuses an absent/empty/non-object body with `422` so `""` never
  reaches `verify_password`, answers every failure with the one sentence `wrong password`, and
  answers `429` + `Retry-After` while the per-address backoff runs (the only extra fact it adds,
  because a wait is something the caller has to act on). After a good sign-in it also upgrades a
  verifier minted at a lower `DASH_PBKDF2_ITERATIONS` (`auth.needs_rehash`), keeping the signing
  secret — that knob has to actually buy something, and a login is the one moment the plaintext is
  in hand. A refused upgrade write is swallowed with one fixed-text `logging` line (this module's
  only one, carrying no credential: invariant 6) because `database is locked` on a bookkeeping row
  must not turn a correct password into no login at all. `POST /api/auth/password` is task 04: it
  validates `new` first (`MIN_PASSWORD_LENGTH = 8`, non-empty once stripped, different from
  `current` — length is the whole policy, because composition mandates produce sticky notes),
  throttles the *current* password on its own key `password:<address>`, refuses a wrong one with
  `403` + a `password-denied` event, and on the right one calls `auth.update_verifier` and answers
  from the **new** state with a cookie and CSRF token minted from the rotated secret. Building that
  response from the old `ctx.auth` is the bug the order exists to prevent: the operator would be
  signed out by their own form.
  `AppContext` carries `auth` and the one
  `LoginThrottle` per process — a throttle built per request is a throttle that never trips. A
  refused sign-in writes a `login-denied` **event** and nothing else, and so do the two password
  outcomes (`password-denied`, `password-changed`): `alerts.py` turns only
  `kind: "unreachable"` into an overview row and every feed filters by `source`, so
  `/api/events?source=auth` is the audit trail and the overview stays about outages (do not
  "fix" that by hiding the event). Neither fact appears on `/api/health` — which is public — or
  `/api/config`. `/api/config` is a redaction boundary.
  `/api/disks/*` (phase 09) is the drive-health pair: `/snapshot` answers one entry per
  configured host **with drives to show** (disk rows + `smart` + `smartctl_available` +
  per-row `severity` + a `fleet` block summed by the API, with a configured-but-never-answered
  host as an `unreachable` stub whose counts are `None`, not `0`), and `/history` serves the
  `disk.*@dev` series with unknown names answering as empty series like the other two
  history endpoints. The configured gate runs **before** anything that could reach the
  network — `HOST_TARGETS` empty is `404 {"configured": false}` with no client, no fetch and
  no `disks` job in the hub at all (invariant 3), and `tests/test_api.py` proves it with
  spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot`. Configured-ness and
  visibility are two different gates, and only one helper decides the second:
  `api._disk_visible(host)` drops a host whose payload's `smartctl_available` is **explicitly**
  `False` ("no smartmontools, no drives to report" — the VM case) from `hosts` and therefore
  from `fleet` (`fleet_disk_summary` sums whatever it is handed, so `hosts_total` is the row
  count), while `True`, `None` and an `unreachable` stub all stay — an agent that cannot say is
  not an agent that said *no*, and a dead agent must not read as "never configured"
  (invariant 3). The check is `is not False`, not falsy, and it runs in `/api/summary`'s disks
  block too, before both the fleet sum and the per-host rows, so the two endpoints cannot list
  different machines; `states` still names every `disks:<host>` job (hidden included, still
  polling, still `ok`) and the source, the store and the health records see no change at all —
  which is also why an overview alert from a hidden host's `disks` job still names that host
  (`alerts.py` reads job states, not this list): hiding a card must never hide a fault
  (phase 22). The new surface grows no
  write of its own — a fourth dashboard must not quietly become a second one, so the app's
  only POST that **dials an upstream** stays `/api/ups/command` (invariant 1, pinned by enumerating
  every served route's verbs and sorting the writes into fleet / session / certificate-authority
  lists — the third is phase 15's, and it writes `meta` rows and nothing else).
  All four `history` routes reach the store through one door, `api._history_batch()` →
  `store.history_many()` (phase 14 task 02). That is a cost change and nothing else: the series
  list is host-major and metric-minor in request order, each entry the same
  `{source, host, metric, range_seconds, points}` document the per-series loop built, with the
  same empty-series answer for a name that has no rows. `tests/test_api.py` pins it for all four
  routes against points read straight out of the store with `query()` — a third path, in neither
  the loop nor the batch — so a future "optimisation" that moves a point, an order, or a key
  fails a test instead of redrawing a chart wrong.
  `/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
  the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `disks.hosts`,
  `llm.engines`, `ups.ups`, the `states` maps and `recent_events` stay for programmatic
  consumers (live tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events`
  are unchanged and `alerts` (phase 02) and `disks` (phase 09) are additive — the `disks`
  block is `fleet_disk_summary` verbatim plus one row per host (`disks`/`with_smart`/
  `health_failed`/`alarm_disks`/`smartctl_available`/`failed_devices`), so a notifier learns
  *which* drive the hardware called `FAILED` without a second call, and
  `tests/test_api.py` pins every one of those key paths, with and without targets configured,
  so retiring a panel can never shrink the payload. Since phase 22 those rows are the **visible**
  hosts — the same `_disk_visible` predicate `/api/disks/snapshot` applies, over the same
  entries, before the fleet sum — so `disks.fleet` equals `/api/disks/snapshot`'s `fleet` and a
  host with no smartmontools is missing from both, while `system.hosts` still lists it (a missing
  binary is a fact about the drive page and nothing else) and `disks.states` still lists its job.
  A configured
  job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
  `system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
  counts the machines there are and `hosts_live` the ones answering.
- `app.py` — assembly, and the **session gate** (`session_gate` middleware, invariant 12) whose
  `PUBLIC_PATHS`/`PUBLIC_PREFIXES` are the whole anonymous surface. It answers in this order and
  the order is the design: no `AuthState` → `503` for everything; no session and no such route →
  the router's `404` (a typo is not a sign-in problem, and "sign in required" would otherwise be
  the only answer anyone outside ever gets for a wrong URL); public → through, except that the
  origin check still applies to a public write (a cross-site login post spends the throttle of
  whoever is typing) and `GET /login` for someone already signed in goes to the dashboard, not
  back to the form; no session → `303` to `/login?next=…` for a page and the JSON refusal for
  everything else; a write → origin, then this session's CSRF token when it arrived on the
  cookie. `_own_origin` reads `X-Forwarded-Proto` for the same reason the cookie's `Secure` flag
  does, so behind nginx a real same-origin write is not a cross-site refusal — the bug that only
  appears in production, which is why it has a test. `create_app()` takes injectable settings/store so tests build an app
  with an in-memory DB and no targets. The lifespan builds `ctx.auth`
  (`auth.load_auth_state`) after the context exists and **before** the first poll, for the
  owned-store and injected-store branches alike — one call site, because two copies drift — and
  a `RuntimeError` from it leaves the process non-zero with the operator's fix in the output.
  That failure is never caught here: a dashboard that cannot say who its admin is does not start
  the loops either (`__main__.py` passes `SETTINGS` to `create_app()` explicitly so the served
  app is configured by the same object the bind address came from). `build_certs(store)` is the
  same rule for phase 15: one helper, called **before** `build_hub` in the owned-store
  (`build_context`) and injected-store (lifespan) branches, whose `CertMaterial` (the CA + client
  paths in a private temp dir) hangs off `AppContext.certs` and whose
  directory the lifespan's `finally` removes. What hands that material to the loops is
  `agent_tls_map(material, store, targets)` → `build_hub(..., tls=...)`, called in **both**
  branches the same way: no material (or no targets) means no TLS map, which is the plain
  `http://`/token path rather than an error, and one map covers both of a host's jobs. A corrupt stored certificate raises out of the same
  place — a dashboard that cannot present its own client certificate would otherwise spend the
  fleet's TLS history reporting an upstream problem it caused itself.
  `build_hub` also builds **two** `RateEstimator`s, because a window is only a claim about the
  polls that fill it (phase 23, `TODO.md:8`'s "ramp up/ramp down"): the `host` jobs get
  `RateEstimator(span=settings.rate_window)` (10 s — what a 10 s cadence can fill) and the `llm`
  jobs `RateEstimator(span=settings.llm_interval)` (2 s, one poll interval, so a slot that starts
  reads steady on the next poll instead of climbing one fifth of the true rate per poll), and
  there is no third knob — the LLM window **is** `LLM_INTERVAL_SECONDS`, which is the only way
  the pair cannot drift. One estimator for both could only be right about one of them: a 2 s span
  over a 10 s poll sits the window-empty fallback against the `2 × span` ceiling and answers
  `n/a` every poll, and a 10 s span over a 2 s poll smooths the work the operator is watching.
  The estimator's own gates (the `MIN_WINDOW_SECONDS` floor, the ceiling, the 8× jump
  confirmation with its independent anchor) are unchanged and apply to the smaller window, whose
  `Δ2.0s` label the payload carries with the value.
- `web/` — five dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`) plus the login page
  (`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`. No
  framework. `/login` renders the masthead's **brand link and theme toggle** and nothing else
  from it: no tabs (every tab bounces straight back here, and a nav of links that all redirect is
  a lie about what is available), no status dots and no clock (there is nothing signed in for them
  to say). It posts through `Dash.fetchJSON` like every other script, which is only safe because
  `bounceToLogin()` does nothing when the browser is already on `LOGIN_PATH` — without that guard
  a `401` would reload the form instead of printing `Wrong password.`, and the page that exists to
  end a bounce loop would be one. So the form branches on `err.status`: `401` is the inline
  sentence plus `field.select()` (fixing a typo without retyping is the whole UX here), `429`
  shows the server's `detail` and **counts down its `Retry-After`** with the button disabled until
  the wait is over — `fetchJSON` therefore carries `response.headers` on the `Error` it throws,
  because a wait this page made up would be an invented number (rule 2) — and anything else prints
  the server's `detail` verbatim. Every refusal goes out through `textContent`, never
  `innerHTML`: this is the one page a stranger reaches first. Its own copy of `auth.safe_next`
  guards the one redirect followed on client strength (already-signed-in `?next=`), and
  `Dash.ensureAuth()` up front stops a flash of the form for a browser that is already in. The
  login page is audited for orphan `id`s and CDN references too (`tests/test_auth_api.py`), fetched
  unsigned because a signed-in `GET /login` is a `303` and would audit `/`'s markup under the wrong
  name. `/settings` (`pages/settings.html` + `static/page-settings.js`) is a **page and not a
  sixth dashboard**: same `<head>` and masthead, the five tabs with **no** `aria-current` (it is
  not one of them), no status dots, and it is reached from the `⚙ Account` link in
  `.masthead-right` of every dashboard **including itself** — outside `.tabs`, so the pinned tab
  structure is untouched (`tests/test_api.py::test_every_page_links_to_both_admin_pages`). It prints
  `/api/auth/status` (name, expiry as a local timestamp *and* as time left), signs out through
  `POST /api/auth/logout`, and changes the password through `POST /api/auth/password` after
  checking locally what it can judge itself (non-empty current, ≥ 8 new characters —
  `tests/test_api.py` pins that number against `api.MIN_PASSWORD_LENGTH` so the two cannot drift —
  new matching confirm), printing `403`/`422`/`429` through `textContent` as the server words them,
  then re-reads the status so the panel cannot show a session the rotation ended. Its static
  sub-line is the precedence rule (env seed → stored verifier wins → reset by deleting
  `auth.verifier`), because that sentence is what stops the "I edited the env file and nothing
  happened" call. So `app.js` exports **`resetAuth()`**: it clears the memoised `authStatus` and its
  in-flight promise, and must be called after a password change (the secret rotated, so the cached
  `csrf` is now invalid — every later write, the UPS buttons included, 403s until a reload) and on
  any `401` bounce.
  `/certs` (`pages/certs.html` + `static/page-certs.js`) is the second of those admin pages, and it
  is the same shape for the same reasons: five tabs with **no** `aria-current`, no `#nav-status`,
  and both masthead links — `⚷ Certs` beside `⚙ Account` — exactly as the five dashboards render
  them. **Both links are on both admin pages** (phase 20 task 04, `TODO.md:4`), which retires phase
  15's "no self-link" precedent: chrome that changes shape depending on where you already are reads
  as a broken button, while a link to the page you are on is a no-op that answers with the page you
  were looking at (`data/logs/20-task04-admin-links.cjs` measures exactly that — one document
  response, no redirect, the same cluster afterwards). The two links therefore sit on all seven
  signed-in pages (the five dashboards plus `/settings` and `/certs`) in one order
  (clock → `⚷ Certs` → `⚙ Account` → theme toggle), which is what lets
  the pinned audit be one loop over `MARKUP_PAGES` instead of two over different tuples; `/login`
  is the eighth page and the one with neither. Everything on it is
  `GET /api/certs`: the Root CA block, the display-only Dashboard client line (that certificate is
  minted with the root and has no route of its own), and one row per host with its status chip,
  serial, expiry (local timestamp *and* `Dash.duration` left), issued-vs-deployed `bundle_rev`,
  `san` list, and the three controls. Two rules it exists to keep. (1) **No colour the API did not
  send**: `/api/certs` publishes no `severity` — an expiry is a date and a revision count is not a
  graded number — so the page has no `severity` token at all, the one chip class map is
  `status` word → `ok|fail` with `unissued` mapping to *nothing* (a certificate that does not exist
  never borrows green), and a behind bundle is a neutral `stale` chip. (2) **No control the API
  would refuse**: a row with `configured: false` prints why (`not in HOST_TARGETS`) instead of
  three buttons that all answer `404`, a never-issued host has no **Revoke** and no **bundle** link
  (that route answers `issued: false`), and the download is a plain `GET` link because a session
  cookie is all a signed `GET` needs. Writes are `confirm()`d, not typed-`DANGER`d — that gate is
  for cutting power and this page cannot — and the outcome line prints the route's own `hint`, so
  `restart_required`'s restart sentence lives in `api.py` and nowhere else. Its `COLUMNS` table
  names every payload key it indexes, and `tests/test_api.py` resolves that table against a live
  `GET /api/certs` row in both directions (the `SLOT_FIELDS` device), so a rename on either side
  fails a test instead of printing `n/a` beside a healthy fleet. The two CNs are markup, and pinned
  to `certs.ROOT_CN`/`certs.CLIENT_CN` the way `/settings` pins its password floor. The agents
  table is `table.data.scroll` — the repo's one scroll mechanism (`TODO.md:9`: at phone width the
  serial / bundle / `san` / controls tail was clipped by `.card { overflow: hidden }` with no way
  to reach it), and the class on the served table and the served `table.data.scroll` rule are
  pinned **together** in `tests/test_api.py`, because the mechanism exists only while both name
  it (measured at 360 px and 1280 px by `data/logs/21-task03-scroll.cjs`).
  `app.js`'s masthead status dots are one per dashboard job —
  including `disks`, because the same agent can answer the 10-second utilisation poll and
  have a wedged SMART pass, and one dot for both would be right about neither. `app.js` owns the
  colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
  `ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
  `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
  `.hero` big number (plus an optional `.bar`) from that class — it never compares a number
  to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
  values the API does not rate, and **no page script calls it any more** — `/` was converted
  in phase 02, `/system` in phase 03, `/llm` in phase 04, `/ups` in phase 05 and `/disks`
  shipped that way from its first commit, so every colour on all five pages comes from a
  `severity` verdict the API sent
  (`tests/test_api.py` fails if a page script names it again). A value the API should rate
  gets a row in `severity.py`, never a bound in JS.
  **The masthead is three left-justified rows at every width** (phase 20 task 01, `TODO.md:5-6`):
  row 1 the brand, `#nav-toggle` and the five tabs; row 2 the `#nav-status` per-source dots; row 3
  the `.masthead-right` controls (clock, `⚷ Certs`, `⚙ Account`, theme — phase 21 task 01 deleted
  the `⏸ Pause` that used to sit between the clock and the two links, button, `Space` shortcut and
  `setPaused`/`_paused` plumbing alike: it froze the numbers on screen without growing a `stale`
  label, and a backgrounded tab already idles its polls, which is the only stop this app needs).
  That is CSS and
  no markup change, which decides the mechanism: `#nav-status` is a **child** of
  `.masthead-right`, so there is no wrapper to promote, and the rows are line breaks — `.masthead`
  is `flex-wrap: wrap`, and both `#nav-status` and `.masthead-right` carry `flex-basis: 100%` with
  `justify-content: flex-start` (line breaking uses a flex item's **base** size, before shrink, so
  a 100 % item is alone on its line: the div's break starts row 2 of the bar and the dots' starts
  row 2 inside it, leaving the controls to wrap to row 3). `.masthead-right` lost the
  `margin-inline-start: auto` that pinned the cluster to the right edge of the one-row bar.
  `flex-direction: column` was the other option the task allowed and is the wrong one: it puts the
  brand and the tabs on separate rows. `height: var(--header-h)` became **`min-height`** (three rows
  do not fit in 58 px and a fixed height clips the controls) and is declared **once** — the ≤640px
  block no longer restates `height`/`min-height`/`flex-wrap`, and that pair drifting is exactly what
  would clip the bar on a phone only. A page with no dots (`/certs`, `/settings`) renders two rows,
  `/login` two (brand, then the theme toggle). Pinned on the served bytes by the three phase-20
  tests in `tests/test_api.py` (the three rules' declarations, the mobile block's silence about
  the bar's height, and the bar's `position`), and measured in a browser at 1280 px and 360 px by
  `data/logs/20-task01-rows.cjs` (seven pages × two widths plus the dropdown; screenshots
  `data/logs/masthead-rows*.png`).
  **The bar is not sticky** (phase 20 task 02, `TODO.md:10` — "It shouldn't be sticky at all. We
  want to maximize the amount of screen space for data viewing"): `position: sticky; top: 0` became
  **`position: relative`** with no offset, so the whole masthead scrolls away with the page and the
  phone's "sticky only part way down" quirk — the fixed-height sticky bar fighting the wrapped
  rows — went with it. `relative` rather than the default `static` because `.masthead` is still the
  positioned ancestor the ≤640px dropdown's `top: 100%` resolves against (a `static` bar moves the
  menu to the top of the document), and `z-index: 40` stays so the menu's `41` still means one
  above the bar; the consequence is that the open menu scrolls with the bar, which is what an
  unpinned bar means. The only `position: sticky` left in the sheet is a table's own `thead`, inside
  its own scroll container, and this task is not it. Proven by `data/logs/20-task02-unsticky.cjs`
  (seven pages scrolled at 1280 px and 360 px — with a squeezed-height leg for the pages that fit
  a tall viewport — plus the 360 px dropdown still landing directly under the bar and still hanging
  off it after a scroll; screenshots `data/logs/masthead-unsticky*.png`). Note that a browser cannot
  tell `relative` from `relative; top: 0` — Chrome computes `top: 0px` for either — so the absence
  of the offset is a served-CSS pin, not a measurement.
  **The tab nav collapses into a hamburger at ≤640px** (phase 18): the seven tabbed pages each
  carry `#nav-toggle` (`class="ghost nav-toggle"`, `type="button"`, `aria-controls="nav-tabs"`,
  `aria-label="Toggle navigation"`, `☰`) between `.brand` and `<nav class="tabs" id="nav-tabs">`,
  and `/login` carries neither — it renders the brand link and the theme toggle only. Its base
  rule is `#nav-toggle { display: var(--nav-toggle-display) }` with `--nav-toggle-display: none`
  in `:root`, and that token is a specificity ladder, not decoration: the button also carries
  `button.ghost { display: inline-flex }`, so a `.nav-toggle { display: none }` loses and ships a
  visible button into the desktop masthead. The `@media (max-width: 640px)` block reveals it
  through the token, hides `nav.tabs`, and tightens the bar's `row-gap` plus the cluster's gap (the
  cluster still wraps inside its own row — hiding a control to make the width work is not the deal)
  and pads the bar down by
  `env(safe-area-inset-top, 0px)` (the block's `padding-block` value restated on the top side so
  the two cannot drift), because a real device's top-edge zone — Android Chrome is edge-to-edge
  by default, and iOS does the same once the page opts into `viewport-fit=cover` — does not
  deliver taps to the page: the first-row button in it is *unreachable*, not unbound, and no
  headless emulation has such a zone, so a fully green emulated pass still ships a dead button
  (the same symptom brain of reese fixed in production; `env()` resolves to 0 on every non-notch
  surface, which render byte-identical, and the dropdown's `top: 100%` follows the taller bar);
  it renders the dropdown as the `nav` element itself — since phase 20 task 03 that element is
  **one box in two states**: `nav.tabs` carries the geometry (`position: absolute; top: 100%;
  z-index: 41`) under the **whole** bar — three rows since task 01, which is the intended look: a
  menu below the masthead, not wedged between its rows — *and* the closed state
  (`visibility: hidden; opacity: 0; transform: translateY(-8px); pointer-events: none`), while
  `nav.tabs.is-open` flips only those four properties to their visible counterparts. That shape is
  what the slide needs, not a stylistic choice: the block used to close the menu with
  `display: none`, and a box that is not rendered has no intermediate state, so the 180 ms
  `opacity`/`transform` transition the operator asked for (`TODO.md:11`, "a slide out and slide
  back in", after `brain_of_reese`'s `.app-nav`) could never have run. `visibility` is delayed by
  exactly the slide on the closed rule (so the box fades out before leaving the tree) and
  undelayed on `.is-open` (or the menu would be invisible for the first 180 ms of its own
  opening); `visibility: hidden` + `pointer-events: none` keep the closed menu out of the a11y
  tree, out of the tab order and unhittable, exactly as `display: none` did, and the closed state
  costs no scroll area (measured). With no second menu element, no `body` marker class, no colour
  outside the tokens and **no JS change at all**, both states are named in the file's one
  `@media (prefers-reduced-motion: reduce)` block — naming only the closed one would leave the
  higher-specificity open transition animating (brain of reese's phase-46 lesson). One binding in
  `app.js`'s `initChrome()` owns the state through a `setNavMenu(open)` funnel (`.is-open` on the
  nav and `aria-expanded` on the button, always together) with **four close paths**: the button
  again, a tab click delegated on the nav, `Esc` with focus returned to the button, and
  `matchMedia` turning false on resize back above the breakpoint — which drops a stale `.is-open`
  so the next shrink starts closed and `aria-expanded` never claims a menu that is not on screen.
  The `if (navToggle && appNav)` guard is the whole `/login` story: the shared script loads there
  and does nothing. The `640px` literal lives in two files — the stylesheet block and that media
  query — and has to move in both, or the menu opens into a hidden nav. The contract is
  pinned on the **served** bytes by three `tests/test_api.py` pins — the button on all seven pages
  and nowhere on `/login`, and one test that **compares** the CSS media query with the JS
  `matchMedia` instead of grepping for `640px` twice, so a drift fails with both numbers in the
  message — and proven in a browser at 360px by `data/logs/nav-task04-verify.cjs` (the `/` +
  `/system` interaction, 24 checks) and `data/logs/nav-final-pass.cjs` (the phase-completion
  sweep: all seven tabbed pages at 360px and at 1280px, `/login` at both, 45 checks — its desktop
  leg now expects the three-row bar's height, not the old fixed 58 px, and still demands inline
  tabs, no button and a dropped `.is-open`). Phase 20 task 03 adds three served-CSS pins (the
  closed state's `visibility`/`opacity`/`transform`/`pointer-events` and its delayed-`visibility`
  transition, the open state's visible counterparts with the delay dropped, and the both-states
  reduced-motion rule inside the file's only such block) plus the guard that no `display: none`
  comes back to that block, and the browser half is `data/logs/20-task03-slide.cjs` (the computed
  closed state, a mid-transition frame at ≈60 ms in both directions, the four close paths with
  `aria-expanded`, the snap under emulated `prefers-reduced-motion: reduce`, the untouched desktop
  tabs and the no-phantom-scroll measurement; screenshots `data/logs/slide-*.png`). The
  safe-area line is pinned in the same test's rule loop; it was verified on the operator's
  mobile Chrome rather than in an emulator, because CDP's `safeAreaInsets` override does not
  feed `env(safe-area-inset-top)` — the one half of this contract a headless pass cannot prove.

Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
series — `/api/metrics` answers with exactly these names. The `sample.source` column is the
dashboard family — `host`, `llm`, `ups`, and `disks` for the SMART series
(`disk.temp_c@nvme0n1`, `disk.power_on_hours@…`, `disk.reallocated@…`, `disk.pending@…`,
`disk.uncorrectable@…`, `disk.percentage_used@…`). Those keep their own family instead of
sitting beside `host`'s `disk.used_pct@/mount` because the cadences differ thirty-fold and
`GET /api/metrics?source=disks` has to answer "what does *this page* have history for".
That scheme is the history API's key — treat it as a public interface.

## The fleet (real environment, verified 2026-10)

| Host              | ssh alias          | What it gives                                                        |
| ----------------- | ------------------ | -------------------------------------------------------------------- |
| `driveripper`     | `driveripper-ops`  | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards |
| `deskwork`        | `deskwork-ops`     | Strix Halo (Radeon 8060S iGPU), gufo `:8731`                          |
| `gateway`         | `gateway-ops`      | router, no GPU                                                       |

- **Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05).** The
  designed shape is one `dashboards-agent` **systemd service per host**, installed by `ops`
  through the `deploy/agent-sudoers` grant with `deploy/install-agent.sh`, bound to
  the address the dashboard dials — since phase 24 registered **with a certificate bundle**
  (`--fetch-bundle`, so its `HOST_TARGETS` URL is `https://` and carries no userinfo); what was
  once the token in the URL is the compatibility path for a host nobody has bundled. driveripper's
  upsd serves the LAN on 3051 with the host and laptop ranges allowed
  by the **firewall**, never by a `upsd.conf` line. The old ssh-tunnel script is deleted and
  neither `.env.dev` nor `.env.dev.example` declares a forwarded port — dev dials the same
  addresses the deployed dashboard does, and `scripts/capture-fixtures.sh` reads
  `HOST_TARGETS` from `.env.dev` instead of naming hosts of its own
  (`tests/test_deploy_scripts.py` fails the day any tunnel name or port returns).
  **Fleet transport, re-verified from the laptop on 2026-10-19.** The port-selective inter-VLAN
  drop phase 10 recorded on the lab gateway `10.2.0.1` is **gone**: `gateway` (`10.1.1.27:9120`)
  and `deskwork` (`10.1.0.51:9120`) both take a plain TCP connect and answer `/snapshot`, and
  `driveripper`'s upsd answers `LIST UPS` on `10.1.0.126:3051` from this same laptop. All three
  hosts now run the **installed** service — `systemctl is-active dashboards-agent` is `active` on
  each, with `/usr/local/lib/dashboards-agent/host_agent.py` and a `--token-file` unit bound to
  the address the dashboard dials. **`driveripper` joined during phase 13 on 2026-10-19, in two
  operator steps.** It went from *no agent at all* (unit `inactive`, nothing listening →
  `ConnectionRefused`, an `unreachable` stub row) to **`active`** with
  `/usr/local/lib/dashboards-agent/host_agent.py` installed at 13:27, and then its bearer token
  went into `HOST_TARGETS`' URL userinfo — so `host:driveripper` and `disks:driveripper` are
  **`ok`** and the `live` suite is **11 passed / 0 failed** (it was 9/2 on the missing token that
  morning, and 8/3 on the ConnectError before that). The unit reads
  `--host 10.1.0.126 --port 9120 --token-file /etc/dashboards-agent.token` (`640
  root:dashboards-agent`, the mode `deploy/install-agent.sh` writes) and is still in the `disk`
  group (`getent group disk` → `disk:x:6:dashboards-agent`) — but it does **not** run as that
  user: gap 10's hand-fix set `User=root`, and re-read from the host on 2026-10-20 the installed
  unit is `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` with no device filter.
  **Those two extra lines are now template output, and the unit's rewrite is no longer
  silent**: phase 17 took the
  `DevicePolicy`/`DeviceAllow` lines out of the template (task 01), gave it `--as-root` and
  `--caps` (task 02), and made a re-run whose generated unit differs from the installed one print
  the diff and exit 1 unless `--force` was passed (task 03), so
  `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force`
  reproduces it and a re-run that would quietly replace a hand-fixed unit is now a refusal with a
  diff attached rather than a clobber (gap 10).
  `/api/health` today reads **all nine sources `ok`**: `host:driveripper`, `host:deskwork`,
  `host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
  `llm:vllm`.
  **The fleet is mTLS since 2026-10-21 (phase 24 task 03), so the bearer-token transport is now
  history rather than the designed shape.** All three hosts run `agent/host_agent.py` **1.1.0**
  from an installed bundle, every unit carries `--cert-bundle /etc/dashboards-agent.certs` (plus
  `RuntimeDirectory=dashboards-agent`, and still its `--token-file` line), and `HOST_TARGETS` is
  `"driveripper=https://10.1.0.126:9120 deskwork=https://10.1.0.51:9120
  gateway=https://10.1.1.27:9120"` — **no `http://` entry and no userinfo anywhere**: the
  certificate is the credential (the pre-flip `.env.dev` is backed up in `data/env-backup/`). `agent.tls: true` + `agent.bundle_rev: 2` in
  every payload, `deployed_bundle_rev == bundle_rev` for all three on `/certs`, and
  `./scripts/dev-server.sh status` reads the nine sources `ok`. The per-host order was the safety:
  install agent 1.1.0 + bundle → `systemctl restart` → the CA-only `--status` probe answers →
  *then* that host's URL flips, so no source was red longer than its own host's restart.
  `data/logs/NOTES-24-rollout.md` is the transcript (sha256 of what was staged where, the unit
  diffs the re-run guard printed, the before/after SMART tables, the screenshots). Two facts in it
  are load-bearing for the next host: the fetch needed an **SSH reverse tunnel** (no monitored host
  can route back to a dev box, so `ssh -R 18099:127.0.0.1:8090 <host>` +
  `--fetch-bundle http://127.0.0.1:18099` is how registration reaches the dashboard), and
  `--status` takes its address from `--bind`, so run bare it dials `127.0.0.1` and reports "no
  answer" about a healthy host. **Rolling a host back is not just the URL**: a TLS-mode agent
  serves https only, so the `--cert-bundle` line and `/etc/dashboards-agent.certs` both have to go
  with it. And the fleet now trusts **this** dashboard's CA — a second deployment with its own
  `certs.*` rows cannot dial these agents until it carries the same authority or re-issues and
  re-installs every bundle.
  Phase 10's proof that the old drop was not this repo's firewall stands as history in
  `data/logs/NOTES-verify.md` — do not re-add a tunnel on the strength of it, and re-run the
  probe rather than trusting either note.
  The per-host record and the exact root commands are in `data/logs/NOTES-agent-install.md`
  (its firewall lines are the phase-10 state, not today's), how far the NUT wire reaches from each
  network is in `data/logs/NOTES-upsd-acl.md`, and the end-to-end direct-dial verification
  (health table, no-tunnel-port proof, live-suite tally, screenshots, and the probes that put
  the block on the lab gateway's port allow-list rather than in this app) is in
  `data/logs/NOTES-verify.md`, and the phase-11 login verification (the `curl` refusal codes, the
  six screenshots, the two `meta` keys the database actually holds, and the operator's
  deployment-repo actions) is in `data/logs/NOTES-auth.md`. Those notes are local evidence, not
  tracked files: `data/` is gitignored (invariant 10's `data/logs/*.png` included), so a fresh
  clone has the conclusions in the prose above and the captures live on the box that ran them —
  re-run the probe rather than trusting either.
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
  on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
  *Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
  | nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down.
  **NUT 2.8 has no `ACL` directive** (this Fedora build is `--without-wrap`; upsd logs
  `ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and then ignores
  it), so per-source restriction is a **firewall** rule and never an `upsd.conf` line — and
  2.8 does not refuse non-local clients the way older docs suggest: `LISTEN 0.0.0.0 3051` +
  `LISTEN :: 3051` with `3051/tcp` in the `FedoraServer` zone answers any client routed to it
  (reads are unauthenticated; `SET`/`INSTCMD` still need an `upsd.users` account). The host
  range `10.1.0.0/21` reaches it, and **since 2026-10-19 so does the laptop**: a raw
  `LIST UPS` to `10.1.0.126:3051` from the dev box answers `BEGIN LIST UPS`, which is why
  `nut` reads `ok` in `./scripts/dev-server.sh status` with two real UPSes and no
  `NUT_SIMULATE`. Phase 10 recorded `10.2.0.1` dropping 3051/3493 for this laptop; that is
  history now, and the probes behind both statements are in `data/logs/NOTES-upsd-acl.md`.
  Two UPSes are exported over USB and dialled directly as `NUT_HOST` + `NUT_PORT=3051`, and that
  reads live from every vantage the dashboard has ever used — the host itself, the deployed
  dashboard, and since 2026-10-19 the laptop (`/ups` shows both boxes at 100 % charge, `OL`,
  with no `SIMULATED` badge anywhere):
  `apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
  `ups.description`, the operator's label — the driver's own `identity.model` answers
  `GX1500U`, so the description is what identifies the box).
  Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date
  (`battery.date` is often an unset placeholder — `2001/09/25`, slashes and all, on the APC),
  CyberPower reports `battery.mfr.date: CPS` (not a date → `unknown`, never a fake age),
  **neither driver reports a battery temperature or the NUT 2.8 `battery.charger.status`**
  (so the battery glyph's state has to come from `on_battery`, and `charger_status` must stay
  optional), and **`ups.test.result` is where the two boxes disagree**: the APC still answers
  the factory `No test initiated` while the CyberPower answers `Done and passed` (a test has
  run on it since the phase-05 capture; both words re-read from the wire on 2026-10-19 and
  pinned as `LIVE_SELF_TEST_WORDS` in `tests/test_nut.py`, which fails if the fixture and the
  capture ever drift).
  Both UPSes are recorded verbatim (`LIST UPS`/`LIST VAR`/`LIST CMD`, serials scrubbed) as
  `tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, and the wire quotes a
  field only when it must — `VAR apc ups.status "OL"`, not three quoted fields.
- Host access is via ssh keys as `ops`, and **passwordless sudo is different on every host** —
  that is what decides how much of the fleet an install can finish unattended. `gateway` grants
  `(ALL) NOPASSWD: ALL`, which is why its agent installed in phase 10 with no
  `deploy/agent-sudoers` line present; `driveripper` grants a command-scoped set (`smartctl`,
  `upsc`, `udevadm`, the `/etc/ups` reads, and the `tee /etc/ups/upsd.users|upsd.conf` +
  `systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork`
  grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`).
  Neither `deskwork` nor `driveripper` grants `visudo`, so a sudoers change is the operator's
  action. **`deskwork` and `driveripper` both grant the installer** —
  `(root) NOPASSWD: /home/ops/agent-install/deploy/install-agent.sh` — and both also grant
  `(root) NOPASSWD: /usr/bin/systemctl restart *`. The earlier record
  here said deskwork's `ops` had no installer grant and that its install went through the
  operator's root; that is out of date — `sudo -n -l` on 2026-10-21 lists the grant, and phase 24
  task 03 installed deskwork's bundle as `ops`. The grant names a **path**, and until phase 24
  that path held an Oct-18 snapshot of the script which carried neither the later flags, nor the
  filter removal, nor the re-run guard — running it would have replaced driveripper's `User=root`
  + `CapabilityBoundingSet` unit with the snapshot's capability-less one, **silently**. That trap
  is now closed on all three hosts: `/home/ops/agent-install/` is `ops`-owned, so the repo's
  `deploy/install-agent.sh` (sha256 `ffbcd91f…`) and `agent/host_agent.py` (**1.1.0**) were staged
  over it without root, and each snapshot was left beside the grant as
  `install-agent.sh.oct18-backup`, which sudo will not run. **Re-staging is therefore the first
  step of any install, not an optional one** — the grant is only as current as the file it points
  at. The `systemctl restart *` grant on those two hosts is the other half of an unattended
  install: a bundle changes the unit, and the unit only takes effect on a restart that
  `systemctl enable --now` will not perform.
  The template now writes the live unit's shape: no device filter at all
  (phase 17 task 01 — systemd's default `auto`, with `PrivateDevices=false`) and `--as-root`
  plus `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` for the `User=root` and
  `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` lines (task 02). The old allow-list's
  missing nodes are no longer a concern for the next install: with no filter, `/dev/ng*n*` and
  `/dev/bsg/*` are reachable the same way `/dev/nvme*` and `/dev/sd*` are, subject to DAC and
  SELinux.
- **SMART: real captures, and one live host finally reads them.** `smartctl` 7.5 is installed on
  deskwork and driveripper. The `ops` account (which is who runs a hand-started dev agent) is in
  no `disk` group, driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and deskwork
  grants nothing, and `host_agent.py` shells the binary **as itself** — it never calls sudo — so
  an agent run as `ops` still answers *Permission denied* (exit 2) and reports `smart: null`.
  What changed by 2026-10-19 is that `deskwork`'s **installed** service runs as
  `dashboards-agent`, which is in the `disk` group (`getent group disk` → `disk:dashboards-agent`)
  on an agent installed after the SMART pass: `/disks` shows real numbers there —
  `smartctl_available: true`, `nvme0n1` healthy at 55 °C that day — amber, past `disk.temp_c`'s
  50 °C threshold, and a plain 48 °C `ok` on the 2026-10-20 re-read — 19 166 power-on hours (an
  odometer, so that figure is a capture date, not a fact about the drive), 11 % of rated
  wear — so the wear bar and the alarm chips are drawn from live hardware, not only fixtures.
  `gateway` answers `smartctl_available: false` (no smartmontools in its VM: the *install it*
  half of the three-way flag — and, since phase 22, the arm that keeps it off the `/disks` page
  entirely, so `/api/disks/snapshot` there lists deskwork + driveripper and `fleet.hosts_total`
  is 2 while `host:gateway`/`disks:gateway` both stay `ok` — re-verified live read-only on
  2026-10-21, where the two cards still draw their pre-phase numbers (`deskwork nvme0n1` 48 °C /
  19 201 h / 11 %, `driveripper` 16 drives with the `nvme5n1` `13 uncorrectable` chip, `/`
  unchanged at three fleet rows; `data/logs/NOTES-22-disks-live.md`), and **`driveripper` is the fourth
  case the flag was built for:
  `smartctl_available: true` with almost no numbers.** Its agent answers (16 whole-disk rows,
  `nvme0n1`…`nvme7n1` + `sda`…`sdh`) and `smart` is non-null for the 8 NVMes, but every one of
  them carries only identity — `health`, `temp_c`, `power_on_hours`, `percentage_used` and the
  counters are all `null` — while the 8 spinning drives come back `smart: null` each with the
  agent's own line `smart: sda: Try an additional '-d ata' or '-d sat' argument.` Pinned on
  2026-10-19 with read-only probes (`data/logs/NOTES-driveripper-agent.md`): the `disk`-group
  **open of the block nodes works** (agent user, `O_RDWR`), there is no SELinux denier (zero
  AVCs with `auditd` running), and the install's `DevicePolicy=strict`/`DeviceAllow` lines were
  removed from the unit (they matched nothing useful anyway — systemd resolves `DeviceAllow`
  globs by name against `/proc/devices`, where this kernel lists NVMe as `blkext`, so
  `nvme*` matches nothing). What locks the agent out is **kernel 7.2's unprivileged
  passthrough gate**: NVMe admin via the block node is unprivileged-only for five identify
  types; the SMART-log read needs the drive's ACEL to mark it supported (the Samsung 980
  PRO's 2020-era `5B2QGXA7` does not — `NVME_IOCTL_ADMIN_CMD: Permission denied`) or
  `CAP_SYS_ADMIN`, and SATA SMART behind a SAT layer is ATA pass-through, which the
  unprivileged `SG_IO` allow-list (`scsi_cmd_allowed()`: read-safe opcodes only) does not
  include — explicit `-d sat`/`-d ata` fails identically, so no `host_agent.smart()` retry
  form can reach it. `deskwork` reads live with the same user/unit because its drive's
  firmware does advertise the page. So `/disks` shows driveripper's drive inventory with
  `n/a` values and a why-line — the designed degradation, **resolved the same day (gap 10):
  the unit now runs `User=root` with
  `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, and all 16 drives read live** (the
  980 PRO's 59 °C / 6328 h / 14 % row, `nvme1n1` at 58 % wear, the red `13 uncorrectable`
  alarm chip on `nvme5n1`; verified in `data/logs/disks-driveripper-live.png`). A root
  process with an **empty** capability bounding set is still capability-less (`CapEff: 0`)
  because the kernel gates check capabilities, not uid — the named bounding set is what
  unblocked it.
  **Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
  drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
  every one — which is what found the parser's one real bug: this binary annotates the
  attribute table's RAW_VALUE column (`35 (Min/Max 34/44)` on every Seagate, and one Western
  Digital writes its hours as `43894h+43m+32.519s`), so reading the *end* of the row lost the
  temperature of all eight HDDs and one power-on count. The captures are also the first
  honest look at this fleet's drives: `nvme5n1` reports 13 media-and-data-integrity errors and
  `nvme1n1` is at 58 % of its rated wear, so the alarm chip and the amber wear bar on `/disks`
  are real features with real data behind them, not demo furniture.
  **Lighting up the rest of it was the same two operator actions that travel together** — install
  the agent on `driveripper`, and give its agent user device read access: the `disk` group
  (`sudo usermod -aG disk <agent-user>`, which is what `deploy/install-agent.sh` does on install),
  *not* a sudo rule, because the agent must read the device itself — on an `agent/host_agent.py`
  from phase 09 or later (never redeploy without the grant: 16 permission-denied lines would age
  nothing but are pure noise, while with the grant they are 16 drives' worth of temperature,
  hours and sector counters). **The agent and the `disk` group are both in place on
  `driveripper`** — the `disk` group alone is not enough on kernel 7.2 (the passthrough gate, see above), so the unit additionally runs as root with the two gate capabilities bounded (gap 10) — all 16 drives now read live. `gateway` needs smartmontools first and will still read no SMART
  after that, being a VM with no physical disk behind `vda` — and it is the host phase 22's
  visibility gate exists for: its `smartctl_available: false` means `/disks` draws **no gateway
  card at all** and `/api/disks/snapshot` lists two hosts, not three — while its `disks:gateway`
  job still polls, still reads `ok`, and its transitions still reach that page's drive-events
  feed, because the gate hides a host row and not the job's history. That is a different
  designed degradation from a host that cannot *read* its drives for any other reason (a denied
  device, `--no-smart`, an agent predating the pass): that one keeps its dimmed cards, every
  number `n/a`, and the one-line why — **no host here is in that `None` arm today** (all three
  agents post-date the SMART pass), so the arm that stays visible is proven by
  `tests/test_api.py`'s pins and by nothing on the fleet.
  `tests/fixtures/smartctl_reference_*.txt` stay as what they are: **reference shapes** (labels
  verbatim from the deployed binary, values synthetic), kept beside the real captures so the
  synthetic edge cases stay pinned. Re-record the real ones with
  `SMART_HOSTS="driveripper-ops" ./scripts/capture-fixtures.sh`: it tries plain `smartctl`
  first (so a fixture can never read healthier than the agent does), falls back to `sudo -n`,
  refuses to overwrite good fixtures with unreadable output, and scrubs
  Serial/WWN/EUI-64 (`test_smartctl_captures_carry_no_unscrubbed_serials` fails the day it
  stops).

## Commands

```bash
uv sync --python 3.14                     # deps + editable install
cp .env.example .env                      # production-shaped config (DASH_ADMIN_PASSWORD seeds
                                          # the login on the first boot of a database — with no
                                          # seed and no stored verifier the server exits non-zero)
cp .env.dev.example .env.dev              # dev-only: the real hosts (this fleet's URLs are
                                          # `https://` with no userinfo — the bundle is the cred)

set -a; . ./.env.dev; set +a              # dev dials the fleet the prod way, no tunnels
./scripts/dev-server.sh restart           # keep http://127.0.0.1:8090 up; sign in with the
                                          # DASH_ADMIN_PASSWORD from .env.dev (`testpass`), then
                                          # export every issued HOST_TARGETS bundle to data/certs/
./scripts/dev-server.sh status            # per-source ok/DOWN — unauthenticated: /api/health
./scripts/dev-server.sh bundles           # what that export left: names, sizes, modes
./scripts/dev-server.sh logs

uv run python scripts/dev-certs-export.py --url http://127.0.0.1:8090 \
        --password "$DASH_ADMIN_PASSWORD" --names <the HOST_TARGETS names>
                                          # the export on its own (phase 24): reads
                                          # GET /api/certs/<name>/bundle and writes
                                          # data/certs/<name>.bundle at 0600 — mints nothing

uv run pytest                             # hermetic: 1542 passed + 15 skipped (11 live,
                                          # 4 store_perf), no network
uv run pytest --cov                       # coverage gate (98.94%; floor 94%)
uv run pytest -m live --live              # opt-in: real fleet / engines / upsd
uv run pytest -m store_perf --store-perf  # opt-in: the store hot-path timings over the seeded
                                          # store, priced against `store_perf.BUDGETS` (the cached
                                          # reads report cold *and* warm, so a before/after never
                                          # books a cache hit as a speed-up)
uv run python scripts/watch-store-perf.py
                                          # ... and the live proof: 25 signed-in minutes against the
                                          # running dev server, timing the page paths while counting
                                          # `store-error` rows and the WAL size (DASH_WATCH_SECONDS
                                          # shortens it; writes store-perf-watch.{log,tsv,json})
uv run python scripts/watch-llm-window.py
                                          # ... and the phase-23 proof: signed-in reads of
                                          # /api/llm/snapshot at the app's own cadence, judging every
                                          # slot start/end against "settled within two polls", the
                                          # Δ<window> labels and the 1 m rollups — read-only, it
                                          # starts no workload (DASH_LLM_WATCH_SECONDS shortens it;
                                          # DASH_LLM_WATCH_REPLAY=<tsv> re-derives the verdict from a
                                          # captured pass over the same samples, sending nothing)
uv run python scripts/store_perf_probe.py # ... as a report (data/logs/store-perf-baseline.{md,json},
                                          # then store-perf-task03.* for the write side and
                                          # store-perf-task04.* for the budget table);
                                          # --rebuild re-seeds, --dev-db <file> also times a real
                                          # store read-only, DASH_PERF_STORE_DIR moves the cache, and
                                          # every pass that *writes* runs on a scratch copy so the
                                          # seeded baseline never moves under the next measurement
# the bundle for one host: the stored bytes (that host's private key included) and
# `install-agent.sh --cert-bundle`'s only input — signed-in GET, see README's rollout
#   curl -s -H "Authorization: Bearer $SESSION" -o /tmp/agent-<host>.bundle \
#        http://127.0.0.1:8090/api/certs/<host>/bundle
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh             # re-record ground truth (agents + UPS + gufo /slots)
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
                                          # ... plus smartctl per physical disk (needs device read
                                          # access: plain smartctl, else `sudo -n`; scrubs serials)

bash .agents/validate.sh                  # the phase gate: the four checks above (pytest
                                          # --cov --cov-fail-under=94, ruff, pyright, plus a
                                          # node --check of web/static) + repo hygiene.
                                          # Always `uv run` — a bare `python3`/`ruff` on PATH
                                          # tests an interpreter nobody deploys.
```

Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
`--cert-bundle`/`--fetch-bundle`/`--cert-name`/`--session`/`--as-root`/`--caps`/`--force`/
`--status`/`--uninstall`; it stores the token
`0640 root:dashboards-agent` — the
unit hands the agent the *path* and the agent opens it as its own user, so that user must be able
to read it and no other account may, and a `--cert-bundle` file is installed to
`/etc/dashboards-agent.certs` at that same mode after a shape check that runs before anything is
touched (with a local file this script dials nothing; the fetched path is the one dial, below), so
0600 would fail the unit exactly like it fails the token file; the task file's `0600` is the
intent — *root and the agent's own user only* — and 0640 is the mode that achieves it) — never
puts a token on argv, and runs under `ProtectSystem=strict` with **no device
filter** — SMART device access is DAC via the `disk` group + SELinux, because kernel 7.2 matches
a device-filter glob by name against `/proc/devices` (where NVMe is `blkext`), so the old
allow-list denied the whole namespace; `PrivateDevices=false` stays explicit because a private
`/dev` with no device list kills the unit at spawn (`status=208/STDIN`). Where DAC is not enough,
`--as-root` writes `User=root` and `--caps CAP1,CAP2` writes that `CapabilityBoundingSet` line
(space-separated, as systemd spells it): kernel 7.2's unprivileged-passthrough gate checks
capabilities rather than uid, so `--as-root` alone is capability-less (`CapEff 0`) and prints a
note saying so, while `--caps` without `--as-root` is refused with exit 1 before `need_root` and
before any write. Both default off, and neither touches the token file's mode or the `--status`
probe. A bundle-installed
unit also gets `--cert-bundle` plus `RuntimeDirectory=dashboards-agent` (mode 0700 — the bundle unpacks its
PEM blocks there and the agent deletes the key once the SSL context holds it), keeps the
`--token-file` line for a host-by-host rollout, and `--status` then probes `https` on the
bundle's CA with **no** client certificate via a `python3` `ssl` one-liner — task 02's
`CERT_OPTIONAL` + `/health` exemption is what makes that probeable, and there is no external TLS
binary. A re-run without the flag leaves an installed bundle in place, and `--uninstall` removes
it with the token — and neither is guarded, because both are deliberate. **Registration can fetch
that bundle** (phase 24 task 02): `--fetch-bundle <dashboard url> --cert-name <HOST_TARGETS name>
--session <signed-in token>` dials `GET /api/certs/<name>/bundle` with `python3` `urllib` (no
`curl` assumption on a monitored host) in the install run and hands the bytes to that same
`--cert-bundle` path, so the shape check, the `0640` install, the unit's TLS lines and the
`--status` probe behave identically for a fetched and a downloaded bundle. The three flags are
all-or-nothing, are refused together with a local `--cert-bundle` and with `--status`/`--uninstall`,
and the fetch plus the shape check run **before** `need_root`: a 401, an unissued name, a proxy's
200-with-HTML or a dashboard that never answers is one line naming the status and no partial
install. The session is a **fetch credential only** — it travels to python in the environment (argv
is world-readable through `ps`, which is why the agent's token is a file), is `unset` as soon as
the bytes are in hand, appears in no written artefact and not even in the sudoers hint's re-quoted
command, and the fetched temp file is removed by an EXIT trap on every path. What is guarded is the
unit itself (phase 17 task 03): it is rendered to a temp file first, and if an installed `$UNIT`
differs the run prints `refusing to overwrite …` plus the `diff -u` on stdout and exits 1 before
the unit write, the `daemon-reload` and the `enable`, so a redeploy cannot undo a hand-fix and an
identical re-run stays the silent zero-noise path. `--force` prints the same diff and applies it —
which is what the TLS rollout's `--cert-bundle` step needs on a token-mode host, because that step
*is* a unit change. What it applies is the **file**: `systemctl enable --now` only *starts* a unit,
so a run that rewrote `ExecStart` leaves the already-running agent on its old settings until
`systemctl restart dashboards-agent` — and `--status` chooses http from https by reading the unit
file, so probing before that restart reports "no answer" for a change that landed correctly (the
rollout in README restarts for exactly this reason). It takes its **address** from the flags rather
than the unit, so a bare `--status` probes `127.0.0.1:9120` and answers "no answer" about an agent
bound to a LAN address that is healthy — always pass the same `--bind`/`--port` the install did. No root shell? The one sudoers
line that lets `ops` run the installer is `deploy/agent-sudoers`; run the script without root and
it prints that line with its own resolved path.)
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
service here. The source itself is `git@git.reeseapps.com:services/dashboards.git` (`origin`,
branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only gate.

## Testing policy

**Real captures for shape, fakes for behaviour, `live` for drift.**

- Anything that must match an upstream — `/proc` layouts, vLLM series names, NUT framing —
  is asserted against a verbatim capture in `tests/fixtures/`. If the schema drifts, a
  fixture test fails before a dashboard does. `smartctl_driveripper_<dev>.txt` is the whole
  16-drive fleet read through a `smartctl` sudo rule (Serial/WWN/EUI-64 scrubbed), and
  `test_the_real_drives_answer_every_number_the_disks_page_prints` asserts every number
  `/disks` prints comes out of real drive output — that test is what caught the parser
  reading the end of an attribute row instead of its annotated RAW_VALUE column.
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
  rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
- **A verdict the operator sees is pinned on the surface that shows it, in both directions.**
  `tests/test_nut.py` grades the word; `tests/test_api.py`'s `self_test_app` harness holds the
  `nut` job's own payload (its collector is swapped, so a poll cannot overwrite what the test
  mutated) and rewrites the `Last self-test` check in the object `/api/ups/snapshot` serves:
  a pass (`Done and passed`) and a never-tested pack produce **no** `/api/summary` alert row,
  a fail produces exactly one `fail` row per UPS linking to `/ups`. The two directions ship
  together on purpose — an absence-only assertion is satisfied by deleting the check, which is
  not the fix the report asked for (phase 12). The same file pins that the `/ups` check list
  needs **no** markup for a new verdict: `<li class="${check.status}">` plus
  `ul.checks li.<verdict> .mark` in `dash.css` is the whole render path, and no page asset or
  stylesheet rule names one check (a second opinion about the driver's word is the bug).
- **A measurement window is pinned from both sources, through one contract module** (phase 23).
  `tests/rate_window_contract.py` names the two Δ windows (`HOST_WINDOW` = `RATE_WINDOW_SECONDS`,
  `LLM_WINDOW` = `LLM_INTERVAL_SECONDS`), the `delta_label()` rounding the pages use to print
  `Δ2.0s` / `Δ10.0s`, and `assert_window()` — the estimator's own bounds (`MIN_WINDOW_SECONDS` up
  to `2 × span`, because a span equal to the cadence means the base comes from the window-empty
  fallback and a real window is the poll gap: 2.009-2.026 s live on `/llm`) **plus** the printed
  label, so raising a cadence knob moves the window and the pin together instead of rotting it.
  Four files read it: `test_api.py` spies the estimator object each source's jobs receive and pins that
  `/api/config`'s LLM window is the LLM's own span; `test_llm_source.py` walks both throughput
  cells of **both** engines at 2 s and re-runs the phase-13 floor refusal through the production
  parse path at that span; `test_host_source.py` keeps the host payload's `window_seconds` at 10 s
  and shows the same 10 s cadence starved to `n/a` by a 2 s span (why one estimator cannot serve
  both); `test_rates.py` runs the floor, the `2 × span` ceiling and the 8× jump gate at both
  spans, because those are properties of the estimator and not of the window size. A pin naming
  one source's window is what this prevents — the two payloads are the same shape and only the
  number differs, so half the pin is how a quiet regression would look.
- **Page payloads that a rewrite must not break are pinned as a table.**
  `tests/card_contract.py` holds the dotted `/api/system/snapshot` paths the `/system` hero
  cards read (`CARD_METRICS`, `GPU_METRICS`, `HOST_SHAPES`); `tests/test_system_contract.py`
  asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and
  `test_every_host_serves_the_hero_card_contract` asserts the same table against the live
  agents — so a rename fails a test, and a card that shows `n/a` cannot be "fixed" by
  inventing a field. `network` carries no `severity` on purpose (throughput has no good/bad)
  and neither does a GPU row (phase 06: a busy GPU is the job on an LLM server) — the
  contract pins both absences. `tests/ups_contract.py` is the same device for the `/ups`
  hero row and battery glyph (`HERO_FIELDS`, `UPS_SHAPES`, `assert_ups_hero_contract`):
  `tests/test_ups_contract.py` drives it from `tests/fixtures/nut_<ups>.vars` — verbatim
  `LIST UPS`/`LIST VAR`/`LIST CMD` replies from the two real UPSes — and
  `test_ups_hero_fields_are_present_on_the_real_batteries` drives it against live upsd. The
  captures are what make the driver disagreements testable: one of them answers
  `battery.mfr.date: CPS`, neither exports `battery.temperature` or
  `battery.charger.status`. `tests/slots_contract.py` is the same device for gufo's `/slots`
  rows, which the `/llm` slot grid (phase 07) reads (`assert_slot_shape`, required/optional
  field tables, duplicate slot ids, the `next_token[0]` entries): hermetically
  from `tests/fixtures/slots_gufo.json` — the verbatim `/slots` reply, idle slots and all,
  captured whenever `GUFO_BASE_URL` is set — and live via
  `test_gufo_slots_answer_the_pinned_shape`. That fixture exists because its absence is how
  the deleted slot table rendered dashes forever: its field names (`slot.state`,
  `slot.n_prompt`, `slot.decode_per_second`) were a memory of an upstream, not a capture.
  **The contract names one canonical key per fact and the grid renders that one**: the capture
  answers `id_task` *and* `task_id` for the same request, so `CANONICAL_TASK_ID_FIELD` decides
  and the alias is captured but rendered by nothing — a rename fails a test instead of becoming
  a silent `a ?? b` chain in the page. `tests/test_api.py` reads the served `SLOT_FIELDS` table
  and resolves every key it names against the capture, in both directions.
  A busy slot's `prompt` is the user's in-flight request text, so the capture script blanks
  it and the page never reads it.
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
  fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
  unset, so partial fleet availability is normal. **Live tests must not write** — a battery
  test on hardware something is plugged into is not a test.
  **An `https://` agent is dialled with the stored authority, through `fetch_snapshot` and never a
  hand-rolled `client.get`** (phase 24 task 03). The module-scoped `agent_tls` fixture reads the
  `certs.*` rows out of the same SQLite file the app uses — through a read-only `sqlite3` URI whose
  `set_meta`/`delete_meta` **raise**, because "live tests do not write" includes not minting or
  repairing an authority from a test — and hands `collectors.agent_tls_map` the material
  `certs.write_material` laid down, exactly as `app.build_hub` does. The failure this removes is the
  misleading kind: a raw `get` against a healthy mTLS agent dies with `CERTIFICATE_VERIFY_FAILED:
  self-signed certificate in certificate chain`, which reads like a dead host and is really a test
  that forgot to present a certificate. No store file, or no authority in it, skips — the same
  partial-environment rule as an unreachable host.
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
  so it stays in the default run and is *not* marked `live`.
- `tests/auth_contract.py` is why the HTTP suite can start a server at all: every app in a test
  is seeded with `admin_password="testpass"` (the local-testing password — a server with no
  stored verifier and no seed refuses to start, so an app under test needs a credential exactly
  like a deployment does) and hashed at `pbkdf2_iterations=1000`, because the real 600 000 costs
  ~0.3 s per client and the suite starts a hundred-odd of them. The real figure stays pinned in
  `test_auth.py` and `test_config.py`, so the cheap cost here cannot become the default there.
- **The perimeter is its own file, and it is paranoid in two specific ways.**
  `tests/auth_client.py::sign_in(client)` is the one way a test gets a session: it posts
  `testpass`, asserts the sign-in worked (so a suite cannot silently start from a 401 and fail
  three lines later on something that looks like an unrelated bug), and puts the returned CSRF
  token on the client's default headers — which is what let the ~200 pre-existing HTTP assertions
  in `tests/test_api.py` stay untouched, asserting UPS behaviour rather than the new gate.
  `tests/test_auth_api.py` then runs the gate through a real ASGI request with
  `follow_redirects=False` on every anonymous call: `TestClient` follows redirects by default, so
  an unsigned `GET /api/config` would fetch the login page and report `200` — the way a gated
  suite ends up passing while testing nothing. Two of its tests are the ones to keep if you keep
  none: `test_every_route_that_is_not_public_requires_a_session` walks the **served** route table
  (`walk_routes` flattens pages, `include_router` wrappers and the `/static` mount, and raises on
  anything it does not recognise, so the audit cannot silently shrink) and demands `401`/`303` for
  every path outside `PUBLIC_PATHS`, GET and POST — the test that protects the next phase rather
  than this one; and the UPS pair prove the CSRF refusal on **the wire** (a scripted
  `nut.NutClient` records nothing for a tokenless write and does record it once the header is
  there, and `driver.killpower` still needs `DANGER` behind the session), because 403 after
  `instcmd` would be a power cut with an error message attached.
- **The setup surface has its own pins, because prose rots quietly.**
  `tests/test_auth_docs.py` reads the artefacts an operator actually holds when a dashboard will
  not come up: both tracked env templates (every `DASH_*` auth setting declared; every
  secret-shaped *assignment* a placeholder — the same allow-list `.agents/validate.sh` 2b uses,
  so `uv run pytest` catches a committed password on the working tree and not only in the gate;
  the local-testing word allowed in prose and forbidden as a value), `scripts/dev-server.sh` (its
  probe and its `status` ride `/api/health`, no `dash_session`/`/api/auth/`/`Authorization:`
  anywhere, the password named but never assigned), and `README.md`/`AGENTS.md` (the *Signing in*
  section before *Configuration*, the four public entries, the reset `DELETE FROM meta …` command
  verbatim, the four config rows, the `DASH_COOKIE_SECURE=1` deployment note, the three auth
  gaps). No server starts and no request is made: a doc that stops naming the reset command is a
  locked-out admin, and that is worth a test even though it asserts nothing about Python.
- **A module that holds a private key is pinned structurally, not only behaviourally**
  (phase 15). `tests/test_certs.py` reads `src/dashboards/certs.py` as text and fails if it grows a
  `logging` import, a raise carrying an interpolated value, or an import from the rest of the
  package — the same three pins `tests/test_auth.py` holds over `auth.py` — because this app turns
  log lines and error strings into `event` rows and `/api/health` fields that a browser reads, and a
  stored PEM is a key. Its behavioural half runs the whole lifecycle over a dict `MetaStore` (five
  rows on first boot, no byte changed on the second, rotate/revoke/root-rotate and their denylists,
  a bundle that carries the agent key and never the root key), and one test completes a real
  client-certificate handshake through the material `write_material` lays down. The other two halves
  run real code rather than a model: `tests/test_agent.py` drives the real `host_agent` handler over
  a TLS-wrapped loopback port (the handshake matrix, the live denylist re-read, token mode unchanged)
  and `tests/test_deploy_scripts.py` runs `install-agent.sh` in a sandbox whose `install`/`chown` are
  faked, which is how a `0640` mode, a byte-identical bundle and the https `--status` probe are proven
  without a root shell or a host. **A `0600` in a task file is not a fact about this repo**: the agent
  opens its own bundle and re-reads it for the denylist, so the group bit is load-bearing.
- **A bundle source is validated once, whichever source it is** (phase 24 task 02). The
  installer's `--fetch-bundle` is tested through the same `served_dashboard` socket as the export,
  because the claim is the same claim: the file a host installs is the route's bytes. The pins that
  go with it are the ones a second install path would break — byte-identical `0640` install with
  the unit's TLS lines; `issued: false`, a 401 and a 200-with-HTML each refused with their own
  sentence, no stub call logged and no file written (the stub log is the proof the refusal came
  *before* the root boundary); a temp `TMPDIR` left empty on the success path and on every refusal
  path, including the one where the shape check is what stopped the run; the three flags
  all-or-nothing and refused beside a local `--cert-bundle`, beside `--status`/`--uninstall`; and
  one label-aware validator, pinned by demanding that the local path still reads `--cert-bundle …`
  and never names the flag that was not used.
- **A script that is an HTTP client is tested over a real socket** (phase 24). The dev bundle
  exporter is run as a subprocess against a real `create_app(...)` served by uvicorn on
  `127.0.0.1:0` in a thread, not through `TestClient`: what has to be proven is that it signs in
  through the public route, reads the session token back out of a `Set-Cookie` header, and sends it
  as `Authorization: Bearer` — none of which a transport-level test client has to serialise. The
  pins that go with it are the ones that keep the artifact honest: the file byte-identical to the
  route's bytes, `0600` (including the re-`chmod` of a file somebody loosened), no file and one
  note for `issued: false` and for `configured: false`, exit 2 with nothing written for a refused
  sign-in, **no mint** (the serials, revisions and root are the same after the run), and no PEM,
  token or password in the output. The dev-server composition is pinned on the served text too —
  export after the readiness probe, failure as `|| echo` and no `return 1` — because "it runs
  later, and never fails `start`" is the part a refactor loses.
- **A live measurement is re-derivable, so a verdict can be re-checked without re-watching**
  (phase 23). `scripts/watch-llm-window.py` writes the series it judged (value, the `Δ<window>`
  the payload carried, the engine's own busy count, the raw token counter) beside its verdict, and
  `DASH_LLM_WATCH_REPLAY=<tsv>` re-computes that verdict from the file with no traffic sent. The
  pass line is read on the same samples re-diffed over the estimator's own span *and* on the
  published value, because the phase-13 jump gate can legitimately hold a correct number for a
  poll — judging only the published column would call a designed hold a ramp.
- **A cost claim is pinned as a shape and a control, never as a wall clock** (phase 14). The default
  run keeps the seeded-store tests hermetic and asserts **shape and determinism** — counts,
  retention windows, the statements a plan is taken from, idempotence — while the timings live behind
  `-m store_perf --store-perf` and in `scripts/store_perf_probe.py`, because a budget asserted on
  first writing is a number fitted to the machine it was written on. Two rules came out of it.
  (1) A cheaper query is proven *equal*, not just faster:
  `test_windowed_rollups_answer_exactly_what_one_full_window_rollup_answers` compares every stored
  row of a windowed store against a second store rolled the old way, and
  `test_a_re_rolled_bucket_only_ever_gains_rows` pins the one bug that equivalence alone misses (a
  window floor that cuts into a bucket re-rolls it from its tail rows). (2) A race that comes back
  clean has to be shown capable of going dirty: `store_perf` runs the pre-fix whole-window
  `maintain()` as a **control** beside the windowed one and `tests/test_store.py` fails if the control
  stops contending, because "0 `database is locked`" from a race that never met is the invariant-4
  failure mode with a green checkmark attached. (3) **A budget is a table, not a test body**
  (task 04): the 14 ceilings live in `store_perf.BUDGETS` with the measured figure and the margin in
  each row's `basis`, the probe renders that table, and two default-run tests fail if a budget stops
  being priced or a ceiling contradicts its own ratio partner — a ceiling nobody measures is a
  comment. (4) A cache's cost claim needs the *write* cadence in the measurement: task 02's series
  cache reported 0.0 ms on a quiet seeded store and served almost nothing on a live one, because
  every sample write invalidated it. The pinned version is statement-counting, not timing — a poll of
  existing series issues no `SELECT`, a write of a name nobody wrote ends the cache — and the live
  version is `scripts/watch-store-perf.py`, which is where the defect showed up.
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
  94% (`--cov-fail-under`), and ~98% is where we are; keep it there. New/modified Python
  still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
  never a real target, and never a real upstream: `/api/ups/command`'s send is proven against
  a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live
  suite is read-only — that fake is the only place the write reaches a "driver".

## UI conventions

Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
`LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
silence, and the legend is a sibling that labels nothing). Every `LineChart` on every page has
one — `/system`'s three per-host slots, `/disks`'s two per-host plots, `/llm`'s four trend
plots, `/ups`'s four battery plots — each pinned by `tests/test_api.py`, so a chart that ships
anonymous fails a test.
Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups` and `/disks`; the API accepts
5m…90d — SMART lands every 300 s, so a 5-minute window on `/disks` is one dot). A
panel that cannot get data says why in the panel — the operator should never have to open
devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.

**Flavour text is not helpful information.** A page does not narrate its own data pipeline or
its own design; copy exists only where the operator needs it to act — a why-this-panel-is-empty
sentence, a provenance tag, a measured `Δ<window>` label, a chart's unit, a `SIMULATED` badge,
an error that names what failed. The audit is one question per string: *would the operator act
differently without it?* No → delete it, container and all. Deleted in phase 08 (the cited
examples, from `TODO.md:7`): "every number is scraped from the box it belongs to · `null`
renders as `n/a`" (`/`'s footer), "agent-reported values; rates are measured over the Δ window
in the footer, and each panel's charts cover only that host" (`/system`'s *Per host* sub-line),
and the same genre — `/system`'s "numbers are measured, not inferred", `/llm`'s "each throughput
number carries its provenance", `/ups`'s "commands are the only write path in this app", each
event feed's content caption (the empty-state sentence already says it), and the design
clauses inside empty-state prose. What survived the audit did so because it is a measurement or
a why: every `Δ<window>` footnote, `provenance()` on each rate, `fleet.seams`, *sums only
engines answering now*, the queued-request note under the slot grid, the SIMULATED banners, and
the setup instructions in every empty state. `tests/test_api.py` pins the deleted sentences
**absent** from every served page and script (a sentence that migrates to another page is the
same slop), the kept functional strings **present**, and that no removed sub-line left an empty
`.sub`/`.faint` container behind.

The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per
`/api/summary` alert: verdict class, source linked to its dashboard via the alert's own
`href`, message, age) and **Fleet**. A non-empty list floats the alerts panel above the
fleet table (`#overview-panels[data-state]` swaps them); an empty one is an explicit
"all clear" state below it, because an empty list is a message, not a blank box. A host that
is not answering keeps its row, dimmed with the shared `.stale` class and reading `n/a`, so
the table never implies last good numbers are live and never silently loses a machine.
No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the
two-panel shape against the served HTML.

Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an
`id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for`
attribute references — the orphan a deleted renderer leaves behind, caught on served bytes
because there is no JS test runner (a locked anchor). Same rule for `dash.css`: delete the
rules a removed panel made dead (`.grid.wide` went with the overview's fleet+events row);
component vocabulary the pages are still being converted to (`.hero-bar`, the grid and bar
variants) stays even while only one page uses it.

A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
muted unit, mono `.sub` footnote (`Δ2.0s` on `/llm`'s rates since phase 23, "at the current load"
on `/ups`), coloured by the payload's `severity` through the existing `.ok` / `.warn` / `.fail`
classes — there is no `.sev-*` family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare.

**The `/disks` card is one panel per *physical* drive, led by identity and verdict, not by a
hero** (phase 09): a drive has no one-number story the way a UPS has charge %, so the card opens
with the device path (`/dev/nvme0n1`) and the drive's own word (`OK` / `FAILED` / `unknown`, a
`.chip` coloured only by `severity.health` — an unreadable drive gets grey `unknown`, never a
green pass), then the identity line (`model · size · SSD|HDD`, nothing invented where a field is
absent) and three equal `.metric` cells: **Temp**, **Power-on**, **Wear**. Colour comes from
`severity.temp_c` / `severity.percentage_used` and nowhere else; `power_on_hours` is an odometer,
written with a `null` verdict, so it cannot read green at 100 h or red at 60 000 h. The sector
counters are **`.chip.fail` alarm chips that exist only while non-zero** (the `/llm` preemption
rule, and why `severity.py` rates no counter): a drive at zero draws no chip row at all, because a
permanent "0 pending sectors" line trains the eye to ignore it and a green one would call a quiet
drive an achievement. Per host the three-way `smartctl_available` decides one `.banner.info`
why-line and nothing else — `null` prints the agent's own `smart_errors` line or names both fixes
without claiming to know which half is missing, and an agent that is not answering says that
instead — and in either case the drive grid takes the shared `.stale` dim while the host card
stays uncoloured (invariant 3: unreadable drives are unconfigured, not broken, and a red panel
here would train the operator to ignore red on the drives that do report). The `false` arm — the
agent said `smartctl` is not installed — no longer reaches the page at all: `api._disk_visible`
drops that host from `/api/disks/snapshot`, from the fleet sums and from `/api/summary`'s disks
rows (phase 22), because a card whose only content is *install smartmontools and grant the disk
group* is not a host on a drive page; `whyLine()`'s `false` sentence stays as the renderer for a
payload that says it, so relaxing the gate later cannot turn those hosts into silent blank cards.
Nothing on the page is per
partition or per mount: the rows are the agent's whole-`/sys/block` list, and a filesystem field
naming its way into `page-disks.js` fails a test. The fleet header renders
`fleet_disk_summary()` — hidden below two hosts (one box's sums duplicate its own cards) and
coloured nowhere, because a count of drives is not a graded number — and the per-host `Charts`
disclosure follows `/system` exactly: collapsed, lazy-built, **one** `/api/disks/history` request
per host per refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive (the
endpoint answers unknown names as empty series on purpose), the wear plot created only once some
drive publishes the field, legends naming drives because the card heading already says who, and
`pruneHosts` taking the cards, the toggle and the plots off the page together.

A cell-scale trend is `DashCharts.sparkline()` — the `llm_monitor` `drawSpark` idiom, so a
percent metric is plotted against `fixedMax` (**never** autoscaled: an 8 % KV window that once
held a 92 % spike must not read flat, and the axis gets top/bottom scale marks so "a third of
the way up" says a third of what). `align: 'right'` pins the newest sample right and grows the
line leftwards inside a real `span` in seconds — x stays time, not sample index. Stroke colour
comes from the payload `severity` via the `--ok`/`--warn`/`--fail` tokens only; an unrated
series draws a faint `--text-faint` stroke with no fill, because a shape is not a verdict.
Below `minPoints` (2) it prints `no history` rather than drawing a confident line through one
point, `labels: {current, max}` renders the `now / max` foot through the caller's `format`, and
there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the
other half of the same honesty for count metrics: history arrives as rollups, so eight seconds
of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for
a queue nobody was in. `/llm`'s count rows moved to the phase-07 stats strip (numbers, not
sparks), so no page passes `minSpan` today and `drawSparks()` still plumbs it through — a count
sparkline that comes back must set it, and the component's own contract test keeps the option
honest (`test_sparkline_ships_the_fixed_scale_and_verdict_options`).

A battery is `DashCharts.battery(container, {charge, severity, state, label})` — one inline SVG
whose **fill *is* the charge**: 1 viewBox unit per percent, clamped to 0–100 and never rounded up
(a 2 % cell must look nearly empty, because a reassuring battery above a flat one is a lie),
clipped to the plate so 0 % leaves an empty shell rather than a stray pixel. Colour comes only from
the payload `severity` token; a measured charge with no verdict draws faint grey at half strength
(`sparkline`'s rule — a shape is not a verdict) and never an empty shell ("0 %" and "not rated" are
different sentences). Every state has a **shape** as well as a colour, so the glyph survives
monochrome: solid shell `online`, bolt `charging`, dashed outline `onbattery`, no fill with a dash
pattern *through* the body `unknown`; a state word it does not know degrades to `unknown`, never to
a confident "on line". Accessible **by default** — `role="img"` + `<title>` + `aria-label` built
from the inputs ("apc: Charge 100 percent, on line"), because the glyph carries information the
markup does not say in words — and a page that prints that sentence next to it passes
`decorative: true` to make the `<svg>` `aria-hidden` rather than say it twice. No animation and no
`ResizeObserver`: one `viewBox` + `preserveAspectRatio` scaled by `.battery` in `dash.css`, so the
hero row gets a big cell on a desktop and a small one on a phone.

**The `/ups` card is led by the glyph and three heroes** (phase 05): the battery on the left, then
**Charge %**, **Load %** and **Runtime left** on `.heroes` at the shared 2.4rem, coloured from
`severity.charge_pct` / `severity.load_pct` / `severity.battery_runtime` and nothing else. Charge
carries **no** `.bar` — the glyph beside it is that value drawn as a shape, and one value drawn
twice in one row is the pattern this app avoids (Load keeps its bar; nothing else draws it). Its
`.sub` lines are the context each number needs and nothing more: `24 V nominal · 27.3 V` for the
pack, `128 W of 900 W` when the driver publishes both `realpower` figures (a wattage inferred from
a percent against another model's rating would be an invented number), `at the current load` for
runtime. Runtime prose is `runtimeText()`'s (`1h 32m`, `10m 21s`) and an estimate the driver never
made goes into `Dash.hero()` as `null`, so it prints faint and uncoloured rather than as `0m`. The
readings the hero row displaced — line in, output, battery volts and amps, battery °C — stay on the
card, demoted to `.ups-electrical`: muted, `.95rem`, never folded away, because they are what you
reach for *while* the UPS is on battery, and battery °C reads `n/a` on both real drivers here.
`state` is derived in the page from `on_battery` → `charger_status` (`chg` charges; `dchg`/`rec` are
the server's own on-battery tokens; `full` is **on line, not charging** — a rested floating battery
reports it) with one fallback: a state chip the API rated `ok`, because `nut.states()` reserves
`ok` for on-line/fully-charged and neither real driver here publishes `battery.charger.status` at
all. `ups.status` itself is never re-parsed in JS (`tests/test_ups_contract.py`). `/ups` passes
`decorative: true`, because the charge hero and the state chips already say the sentence. The glyph
repaints only when charge / verdict / state change (`drawGlyph()` keys a `data-drawn` triple): the
charge float jitters every `NUT_INTERVAL_SECONDS`, and rebuilding the `<svg>` on that re-announces
an accessible name for a difference no eye can see.

An event row names its subject **once**. Every feed (`/system`, `/disks`, `/llm`, `/ups`) renders the
message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a
*leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py`
messages start with their own label, and each row already prints that label as a chip, so
"gufo gufo is answering again" reads as two problems. A name in the middle of a message stays:
the UPS rows used to `replace()` it anywhere and mangled "Quick battery test sent to apc: OK".
This is presentation only — the stored event and `/api/events` keep every word, and the raw
error string still surfaces in `/api/health`.

An event feed is a **window**, not a printout (`Dash.eventsPager(el, events, opts)` in
`app.js`, one helper with four call sites (`/system`, `/disks`, `/llm`, `/ups`) — a pager that
works on one page is the bug this
shape prevents). `ul.events` is a bounded 320px scroll container (`overscroll-behavior:
contain`) and the pager puts `EVENT_PAGE_SIZE` = **50** events in it, with a `.events-pager`
strip under the list: `1–50 of 200` between `‹ prev` and `next ›`, both `aria-label`led for
direction (the feed is newest-first, so `prev` walks towards now) and `disabled` at the ends.
The strip is `hidden` when the feed fits on one page — every row on screen *is* the whole
story, and a disabled arrow says "more history" when there is none. Each page keeps its own
`row(event)` (so `/ups` keeps its longer timestamp and runtime footnote, and escaping stays
the page's job — the pager has no default renderer and says so in a row when a caller forgets
one) and its own `empty` sentence, so the `loading…` and empty states read exactly as they
did before paging existed. Two numbers are load-bearing and pinned together: the fetch
(`EVENT_FEED_LIMIT` = 200, `/api/events?limit=…`) must exceed one page, because a limit at or
below `EVENT_PAGE_SIZE` is a pager that silently never appears; `/api/events` itself is
**unchanged** — no `page`/`limit` vocabulary grew for one page's layout. Paging state lives in
a `WeakMap` keyed by the `<ul>`: it survives a poll (so page 3 is still there two seconds
later — resetting every refresh would make page 2 unreadable) and resets to page 1 when the
event *set* changes (an operator reading page 3 of yesterday's history must not land on page 3
of a list that just changed). Rows are re-written only when their markup changed, which is
what keeps the reader's scroll position inside the container. The `/ups` feed is also
**expandable** (phase 19, `TODO.md:4`): its card body is a `details.events-box` disclosure with
no `open` attribute, so that card is one `Power events` summary line until the operator opens it
— a closed `<details>` renders no content, which is `/system`'s Charts rule applied to a feed,
and the `#events` id stays *inside* the disclosure so the pager and its bounded scroll keep their
target (pinned on the served bytes). The disclosure is also what makes it **lazy**: `refresh()`
fetches the feed only while `state.events.open` is true, so a card nobody is looking at costs the
poll loop nothing (a hidden-but-still-polled list is the half-measure this is). The first expand
fetches; a re-expand paints the rows already in the `<ul>` while they are younger than one poll
interval (`state.pollMs`, the page's own cadence) and re-fetches only past it — a `<details>`
keeps its children, so a fresh re-open costs no request and loses neither the pager's page nor
the reader's scroll. `painted`/`fetchedAt` are stamped on the success path alone, which is the
invariant-4 shape here: a failed fetch leaves the rows on screen, leaves the stamp where it was,
and the next open re-fetches. Its styling is `details.charts-box`'s shape with one structural
difference: that toggle sits *inside* a card body, while this `<summary>` **replaces** the card's
`header` — so it carries the header's type, padding, sunken ground and bottom rule (a closed card
reads as its heading, an open one puts the feed under the same rule every other card draws) and
no `margin-bottom` when open, because the `.body` below carries that spacing. That is also why
`details.events-box > .body` restates the `.8rem`: `.card > .body` is a **child** selector, so a
body moved inside a disclosure stops being padded until something says so — while `ul.events`
and `.events-pager` are class-based and needed no new rule to keep the 320px scroll and the
`1–50 of N` strip working inside it.

The `/system` host card is one row of six hero slots — CPU, MEM, TEMP, DISK, GPU, NET (the
`.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script).
Colour comes from the payload's `severity` dicts and nowhere else, and a missing **number or
verdict** prints `n/a` **uncoloured** — gateway's CPU temp, a driver with no temperature
input, a rate with no second sample yet. What a machine does not **have** is a different
fact and prints nothing at all: the GPU slot ships `hidden` and `renderGpuSlot()` un-hides
it only for a host whose payload carries a card, so gateway's card is **five** heroes with
no GPU slot in the row (`TODO.md:12` — "Boxes without a GPU should not report N/A but
instead should omit these values"), and the sheet's `[hidden] { display: none !important }`
is what takes the wrapper out of the grid so the row reflows instead of leaving a wide gap.
The two rules are one sentence each in the page and one test each; collapsing them into "any
missing thing is `n/a`" is what made a router look broken over the GPU it never had
(invariant 3). NET is one slot with two lines (`.hero-lines`, `↓` in / `↑` out) and is
deliberately colourless: throughput has no good/bad, so `network` carries no severity.

The GPU slot is one busy hero per card plus **four labelled values** under it — **Model**
(mono, `gpus[].model`), **VRAM** (`used / total`), **Pool** (`mem_used_pct`, the honest APU
view a Strix Halo needs because its model lives in GTT) and **Temp** (`gpus[].temp_c` — the
agent's max over that card's hwmon labels, the same value stored as `gpu.temp_c@N`, so the
cell and the history cannot disagree; the per-label breakdown stays in "More detail"). They
replaced one ` · `-joined 11px `.sub` — `TODO.md:12`: "`Radeon 8060S Graphics · VRAM 180.2
MiB / 512.0 MiB · pool 79% · 69 °C` is too much info for such small text and should be its
own values" — and they are `.metric` cells in a `.gpu-values` grid, so a 20-character model
string wraps instead of truncating. The four cells come from one `GPU_VALUE_FIELDS` table,
which `tests/test_api.py` resolves against every capture **and** against the served
`/api/system/snapshot` (`SLOT_FIELDS`' device): a rename on either side fails a test instead
of printing `n/a` beside a healthy fleet. A field the agent did not publish prints `n/a` in
its own cell (the card is real, so it is a number the card is supposed to have) and never
`0 °C`. Nothing in this slot is coloured and nothing draws a bar — no `severity`, no
`ok|warn|fail` token in the markup or the grid's rules (phase 06: a 95 %-busy card on an LLM
server is the job, and a bar under a verdictless number would be a verdict by shape).
Everything not glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU power
and the per-label temp breakdown, filesystems, processes — folds into a collapsed "More
detail" `<details>`: a simpler surface, not a less complete page (the table keeps its Temp
column; this was a promotion, not a de-duplication).

A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
those (a range when the hosts disagree) instead of the poll interval, which only coincides with
them at the defaults (since phase 23 the `/llm` rates are measured over one poll interval, so
their tag reads `Δ2.0s` while `/system`'s stays `Δ10.0s` — each page naming the window its own
estimator used). `/llm`'s hero `.sub` prints the rate cell's own `window` through the same
`provenance()` tag, and the hero's `title` carries the exact `source` string — "78 tok/s" must
always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_monitor`'s
`setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification).

**The `/llm` page reads per engine** (phase 04, decided in
`.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`, laid out by phase
07): a row of two `Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**,
because 0 tok/s on an idle server is healthy and `severity.py` has no throughput row) and the
latency rows, each a mean with a mono `p95 …` foot (no hover; p95 needs no stored series) and a
`sparkline()` fixed at 160px so rows line up. Rows are engine-scoped by the `only` flag on both
components: gufo never grows a TTFT mystery, vLLM never a spec-accept hole.

**The stats strip (`#strip`, phase 07 task 03) is the summary; the panels are the detail, and no
row is printed in both** — the UPS hero row's "one value drawn twice" rule, widened to two
components. One `.card.stat` per *configured* engine between the fleet block and the panels: the
reachability dot, the engine name and the age stamp in the head (same distinction every card
draws — the dot speaks reachability, the numbers never do), then the **KV-cache bar** — the one
cell on the page with a verdict, so the fill and a breach's text take their colour from the
payload's `severity` through `.bar.ok/.warn/.fail` and the ratio→percent conversion the panel's
cell used to make, with `ok` staying quiet ink and a missing gauge an empty bar plus `n/a` — then
`prompt tok/s` / `decode tok/s` with their `Δ<window>` tags, `total tokens` (lifetime, so its
source is the tooltip), the sessions this engine reports, `prefix cache hit`, `spec accept`
(gufo) and `queue time` with its `p95 …` foot (vLLM). `STRIP_ROWS` is the whole table and its
`key` **is** the `cellValue()` key, so every number on it is a cell of `/api/llm/snapshot`: no
field was invented in `sources/llm.py` for this card, and there is deliberately **no peak-ctx
row** (phase assumption). The session row is engine-shaped — vLLM publishes `running`/`waiting`
counts, gufo's is `slots busy`, counted from the `sessions.list` rows `/slots` answered
(`is_processing`, the shape pinned by `tests/slots_contract.py`); when `/slots` did not answer
there are no rows to count, so it falls back to the published `sessions.active` carrying the
API's own `sessions.detail` sentence rather than reporting 0 busy slots from no data (rule 2).
`preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded
threshold) and they live on this card, not in a permanent cell; the lifetime token counter left
the panel's footnote line for the strip's row. **One `/api/llm/history` request per refresh**
carries every chart *and* spark series (`HISTORY_SERIES` = the four chart series +
`SPARK_CELLS`; the endpoint answers unknown names as empty series on purpose), and the two series
phase 04 added (`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) stay stored
additively in `sources/llm.py` HISTORY_METRICS. Above the strip sits **the fleet block**
(`#fleet`, phase 07: "I'm missing the aggregate throughput at the top") — `ALL ENGINES`, a
reachability dot, an
`N of M engines` tag, the note *sums only engines answering now*, then running / waiting /
prompt tok/s / decode tok/s / total tokens. `FLEET_FIELDS` in `page-llm.js` is a list of
`fleet_totals()` keys and nothing else: the page sums no number of its own, a dead engine is
dropped from every total presented as current (its own stale-labelled panel still shows its
last good values), and the tag counts the answering engines, names the missing one in its
tooltip and takes the shared `.stale` dim while the sums are partial. It **hides below two
engines** — a cross-engine sum of one engine duplicates that engine's own panel — and it
carries **no severity colour anywhere**: `fleet_totals` has no `severity` to render (the
summed-KV bar is gone for that reason) and `severity.py` rates no throughput or queue depth,
so the numbers stay ink and only the dot speaks reachability. `fleet.seams` — the API's own
note on how the two engines count prompt tokens differently — is the prompt sum's tooltip
rather than another line of ink. **The block, the strip and the panels keep the page's shared
`.85rem` gap** (phase 13, `TODO.md:4`: fleet and strip rendered flush into one wall):
`#strip { margin-top: .85rem }` is unconditional because below two engines `#fleet` is `[hidden]`
and the strip *is* the first visible section, and `#banners:empty ~ #fleet { margin-top: .85rem }`
gives the block its top gap only when the banner slot renders nothing, so a banner's own
`margin-bottom:.85rem` never stacks a second gap under it. `tests/test_api.py` pins both rules on
the served stylesheet, pins that **exactly one** rule styles each of the two ids (a second,
unguarded one is the double gap), and pins that `#banners`/`#fleet`/`#strip` are still direct
siblings of `<main>` — the `~` combinator resolves between siblings, so re-flowing that markup has
to rewrite the selectors instead of losing the gap. The
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
sit at their own height (`#engines` carries `.items-start`): a two-hero gufo card next to a
latency-row vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". An
engine that leaves the payload leaves both components (`pruneEngines`), never a stale card in
either grid.

**The slot grid (`llm_monitor`'s, phase 07 task 04) belongs to the gufo panel, under its
`sessions (GET /slots)` label.** One card per row of `sessions.list`, **keyed by the row's own
`id`** in `state.slots`: built once by `slotCard()`, patched in place afterwards, and reordered
by `appendChild` of the existing node so DOM order stays slot order — rebuilding a busy slot's
card every 2 s poll re-announces the whole card to assistive tech for a difference of a few
tokens. `SLOT_FIELDS` is the whole table and each cell names **exactly one** `/slots` key
(`path: 'next_token'` reads `next_token[0]`, where the capture puts `n_remain`); `slotValue()`
is the only place a row is indexed, so `tests/test_api.py` can resolve every printed field
against `tests/fixtures/slots_gufo.json` and a rename fails there rather than printing dashes
or a plausible wrong number. No per-slot speed cell: the capture publishes no per-slot rate key
and a page-side Δn_decoded/Δt would be a measurement with no estimator, no `Δ<window>` label and
no `source` string (rule 2) — the engine's rate lives on its strip card and panel. llama.cpp's
`-1` sentinels print a word or nothing: `id_task: -1` is **`idle`**, `n_remain: -1` is **`n/a`**,
never a negative countdown and never the `0` that reads as one token from done. The busy marker
is a **border, a glyph (`○`/`●`) and the word `processing`** in the brand `--accent`, never an
`ok|warn|fail` token — a slot generating tokens is the job (phase 06's GPU reasoning), so
`severity.py` rates no slot and the stylesheet section contains no verdict token. The empty
state is a sentence, never a card of nothing, and it distinguishes the two cases: no `/slots`
answer prints the API's own `sessions.detail`, an answer with no rows prints *engine reports 0
slots*; `sessions.source` is the gate. The note under the grid stays because a **queued
(deferred) request is not in `/slots`** — the waiting row counts those. vLLM's panel gets one
line (*exposes no per-slot JSON*) with `sessions.detail` as its tooltip and **no grid**: a
component of permanent `n/a`s for a server that cannot answer is the dead-cell anti-pattern.
The gate is the engine name (`SLOT_ENGINES`), not row count, so an engine that has not answered
yet still shows its label and its honest empty sentence.

**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
panel and the host/metric detail picker are deleted, along with the `#chart-*` / `#detail-*` ids and
the `select` / `label.field` styling only those pickers used. Each panel stacks its own
`.host-charts` — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where
the GPU **chart** slot is created and destroyed with `host.gpus` (its own thing from the hero
slot above it, which is hidden instead), so gateway shows neither a phantom card nor a hole. **That stack is collapsed behind a per-host `Charts` `<details>`** (phase 06: "only
visible if expanded… there will be lots of hosts"): the default `/system` view is the hero row,
the summary carries the window label the stack will draw (the chips themselves stay in the page
head, one control per page), and `drawHostCharts()` is the single place that decides whether to
paint — so nothing is built or drawn while the stack is shut (`LineChart` inside a
`display:none` container measures 0 px, and rebuilding three charts per host every poll for
plots nobody sees is the waste this removes). The **history request is deliberately unchanged**:
one per host per refresh, naming every series the panel could draw, and its response is cached
per host so the first expand paints from data in hand rather than waiting a poll; a cached
response from another window is re-fetched instead of being drawn under the wrong label.
Expanding one panel must not stretch its collapsed siblings into blank cards, so `#hosts`
carries the existing `.grid.items-start`. **One `/api/system/history` request per host per refresh**
(`hosts=<that host>&metrics=<that panel's series>&range=…`, pinned by
`test_every_system_chart_is_drawn_inside_its_own_host_panel` and the collapse by
`test_system_history_charts_are_collapsed_behind_a_toggle`): unknown metrics come back as
empty series on purpose, so a panel asks for every series it could draw and asks once. A legend
here names metrics (or one GPU's model), never hosts — the panel heading already says who this
is, and a legend that grew a second host is the bug this replaced. A host that stops reporting
loses its panel, its `Charts` toggle *and* its charts (`pruneHosts` deletes the card plus the
payload/chart/history maps), never a chart of nothing or an orphan control in the grid. Per-core
visuals are gone from `/system` and `renderCores` is **deleted** from `charts.js` with its last
caller.
`drawHostChart()` shades a chart only when one series has points — two or three filled areas
stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy
memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryable from
`/api/system/history` — removing the plot is not removing the data.

## Known gaps / next work

1. **UPS commands work, and the account can now cut power — the app's gates are the control.**
   Three defects were cleared on 2026-10-17, each verified against real hardware. (a) `NutClient.login()` sent `USER`/`PASS`, which the deployed
   upsd 2.8.5 does not implement — each answered `ERR UNKNOWN-COMMAND`, so `_read_line()`
   raised and `POST /api/ups/command` could only 502 before `instcmd()`, credentials or not.
   It now sends `USERNAME`/`PASSWORD`, the pair the server's own `HELP` advertises
   (`HELP VER PROTVER GET LIST SET INSTCMD LOGIN LOGOUT USERNAME PASSWORD STARTTLS`), pinned
   by `test_login_uses_the_verbs_upsd_2_8_5_answers_to` — the fake upsd has no `USER`/`PASS`
   branch, so a regression answers `ERR UNKNOWN-COMMAND` exactly like the real thing.
   (b) 12 of the 23 commands the two real UPSes advertise had **no `COMMANDS` entry**, and
   `available_commands()` drops anything the catalog misses — the page silently offered less
   than the hardware does. Every advertised command is catalogued now, each with a risk tier:
   `test_every_command_the_hardware_advertises_is_catalogued` fails if a capture gains a
   command the catalog does not know, and
   `test_every_command_that_can_cut_power_is_in_the_dangerous_tier` keeps `load.off`,
   `driver.killpower` and the `shutdown.*` sequence in the tier that types `DANGER` and is
   refused on battery. `shutdown.stop` is the exception the name hides — it *cancels* a
   shutdown, so it is `safe`, and the test says so in words.
   (c) `[dashboards]` in `/etc/ups/upsd.users` is `instcmds = all` at the operator's request,
   so this app may send every advertised command on both UPSes, power-cutting ones included.
   `.env.dev` (mode `0600`, gitignored) carries the credentials.
   What remains to know:
   - **Proven live, and reversibly.** `beeper.enable` returned `OK` and `ups.beeper.status`
     read `enabled` on the APC afterwards; `driver.killpower` without the word `DANGER` still
     answers 422 and never reaches the wire. A battery test has deliberately not been run —
     it puts real load on a battery things are plugged into, which stays the operator's call.
   - **`enabled: true` cannot mean "this account may send it".** `available_commands()`
     filters on `LIST CMD`, which upsd serves **unfiltered**, and NUT has no call for a
     user's privileges — so the wire cannot tell the page what the account is granted. With
     `instcmds = all` the two agree by configuration rather than by check, which is precisely
     why the risk tiers and the typed `DANGER` are load-bearing and not decoration.
   - **Pre-existing upsd log noise:** every upsd start logs `Unknown upsmon type =` (20
     occurrences over 12 starts, oldest before this work). Harmless — `upsmon` logs into both
     UPSes right after it — but unexplained.
   - **One connection every few minutes answers nothing for >4 s.** Live upsd here stalls a
     single poll past `NUT_TIMEOUT_SECONDS` roughly every 2–4 minutes and then answers normally
     again; `nut` reads `ok` throughout, because the poller's job is one amber line plus the
     recovery, not a frozen page (invariant 4). What phase 12's verification pass fixed on our
     side was the **cascade** that used to follow it — the event table filling with
     `expected 'BEGIN LIST VAR', got 'RW apc battery.charge.low "10"'`, which blamed the wire
     protocol for a stall two commands earlier (see the `NutClient` bullet above). The stall
     itself is an upsd/driver-side question with no repo bug in it, and a battery test is not
     how to chase it.
   Editing `upsd.users` at all: never grant `actions = -`, and never write it blind — the
   running `upsmon` authenticates as its own account from that same file for both UPSes, so
   dropping it means no orderly power-down. Copy the untouched sections from the live bytes
   and `cmp` them before writing; `tee` truncates in place, so owner and mode survive.
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
   (`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
   fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
   all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the
   field map is written.
3. **UPS history depth.** Events and the numeric series are stored; long-window battery
   health (capacity fade over months) has the rollups but no dedicated view.
4. **The dev DB's `database is locked` line is fixed (phase 14); a cold-cache tail is left.** The dev store (~640 MB on
   2026-10-19, its WAL grown past 140 MB at its worst) logged `store-error: OperationalError:
   database is locked` a few times an hour across `system`/`llm`/`ups` through 2026-10-18; the
   poller handles it exactly as invariant 4 requires — the source goes red with the real reason,
   ages, and writes `recording` when the next poll lands — so nothing freezes, but those polls'
   samples are lost. The one cause this repo can prove is two servers on one file, which is why
   `scripts/dev-server.sh stop` waits for the old pid to die (its comment records the incident).
   A controlled 25 s external read transaction during the phase-09 pass produced none, and
   phase 12's re-check on 2026-10-19 found the 190 stored `store-error` rows all older than
   2026-10-18 06:33 with none in the previous 24 h of the running dev store — the absence of a
   sample, not the absence of the pressure, since the file only keeps growing.
   Retention/`wal_checkpoint` tuning for a long-running store was its own phase rather than a
   phase-09 follow-up — that phase is `14_store_performance`, and the two paragraphs below are
   what it measured and changed.
   **Phase 14 has since split this in two, and the read half is closed.** The in-process half was
   never a SQLite lock at all: `SampleStore._lock` serialises everything, so a 1.25 s `stats()`
   from a browser tab's `/api/health` poll was a poller waiting up to 1.45 s to insert 30 rows
   (measured, `data/logs/NOTES-store-perf.md`). Task 02 took the two whole-table reads off the
   per-request path, and `/api/health` now answers in ~1.2 ms instead of ~1.27 s, so the store
   mutex is no longer held by a reader for seconds at a time. The **write half is closed too**
   (task 03): `maintain()` rolls up only what became due since `last_maintain`, its prune is one
   `DELETE` per resolution instead of one `OR` the planner could not use, and `PRAGMA optimize` runs
   there (with SQLite's automatic ANALYZE switched off, so no poller's commit pays for it). On the
   seeded 6.4 M-row store that is 5,368 ms → **2.7 ms** steady / 48 ms for a ten-minute catch-up, and
   the two-connection race at the app's cadence is **100 of 100 inserts, zero `database is locked`,
   max wait 21 ms** where the pre-fix control still waits 4.76 s (task 04's re-run of the same
   budget: 6.8 ms steady / 46 ms catch-up / 8.5 ms worst wait, so the row is a range, not a
   digit). The WAL follows the same shape: a steady pass leaves ~0 MB and a catch-up tick 4 MB
   instead of 38 MB that never comes back down, and `journal_size_limit=64 MB` caps what a
   long-lived reader can make it grow to (`data/logs/NOTES-store-perf.md` has the tables).
   **The live proof is in** (task 04, 25 minutes / 74 passes / 296 signed-in requests against the
   running dev server on the operator's own 636 MB store): **0 new `store-error` rows** (still 190,
   newest 2026-10-18 06:33:30, i.e. 39 h before the watch), **WAL 6.5 → 7.77 MB** against the 67.1 MB
   cap and a main file that did not move, 0 non-200 responses, and page paths at 1.8-3.3 ms median
   (`/api/health`, `/api/summary`, `/api/metrics`, a 3-host × 9-metric 90-day `/api/system/history`).
   The watch also caught a defect in task 02's fix — the series cache was invalidated by *every*
   sample write, so on a store the hub writes to every few seconds it never served and `/api/metrics`
   paid the 572 ms scan on nearly every call (measured: 535 ms per call pre-fix, 1.8 ms median
   after) — and that is what the store bullet above now describes.
   What is **left** is the cost of the two cold answers themselves, measured on that store: 599 ms
   for `series_names('host')`, 729 ms for `stats()`' counts, now once per cache window (~1.3 s of
   store mutex per minute) instead of once per call — which is why the watch's medians are 1.8-3.3 ms
   and its p95s are 620/729 ms: the p95 is the window boundary, and 37 of 74 passes had one request
   over 250 ms, every one of them a cold re-derivation. No cache policy reaches *that*; the fix is
   the bounded `(source, metric)` structure or a counts table named in task 03's note. Two cheaper
   knobs exist and were deliberately not pulled, because both change what `/api/health` answers and
   move rather than remove the mutex hold: raising `STATS_CACHE_TTL_SECONDS`, and refreshing both
   caches from the maintenance tick so the miss lands on a poller instead of a page load. That is a
   phase, not a cleanup. Two servers on one file can still contend, and `busy_timeout=5000` is
   unchanged on purpose so a second one stays visible as a slower write; what they contend with is
   now milliseconds instead of 5.4 s.
   One consequence worth naming, because two other gaps pointed here for it: the **~10 s
   process-wide stall** (gap 9's `Δ`-window dilution, `poller`'s catch-up gaps) is gone with the
   mutex holds — coincident multi-second gaps in the 2 s `llm` and 5 s `ups` raw streams went from
   393 and 198 in the two halves of 2026-10-19 to **0** across the 110 minutes the phase-14 store
   has run (re-measured in this phase's verification pass from the `sample` table; the signature
   separates a process stall from one slow engine).
5. **The login throttle keys on the socket peer address** (phase 11), and the one
   `LoginThrottle` instance now holds two keys per address: the bare one for
   `POST /api/auth/login` and `password:<address>` for `POST /api/auth/password`. Behind nginx
   every login shares one bucket, so anyone who can reach the login form can lock the operator out
   of the *dashboard* — not the UPS, not the data, not a host — for at most
   `LoginThrottle.max_seconds` (1 h by default) at a time, and a process restart clears the
   counters. The two keys are separate on purpose, so neither form can be hammered at the other's
   rate and a stolen cookie plus the change form is not an unthrottled oracle for the old
   password; the cost of that choice is that one address can be blocked twice over. Accepted for a
   single-admin box rather than becoming an `X-Forwarded-For` knob:
   that alternative is spoofable by anyone the moment somebody forgets to configure which proxy
   to trust, which would hand the attacker the throttle key instead of leaving it on the
   network layer. PBKDF2-HMAC-SHA256 at `PBKDF2_ITERATIONS` (600 000) stays the real cost of
   every guess; the throttle only makes hammering the form slower than useless.
6. **One admin, one factor — no second factor and no per-user accounts, by design** (phase 11).
   The operator asked for a single admin, and that is what the model is: one verifier row, one
   username in every signed session, one name in every `password-changed` event, no read-only
   role and no per-viewer view. The consequence is worth stating out loud, because it is the
   whole security budget of a public domain: the PBKDF2 cost plus the backoff is the only thing
   between a stranger and the fleet's hostnames and the UPS command buttons, so there is no
   defence in depth behind it — no TOTP, no WebAuthn, no per-IP allow-list, no "this login is
   from somewhere new" notice. Adding a second human means more than a second row: the gate has
   no notion of scope, so a colleague would be able to cut power unless somebody designs a
   read-only session first. That is a phase, not an edit.
7. **The throttle is per-process, so a second uvicorn worker multiplies it** (phase 11).
   `LoginThrottle` is a dict in one process's memory: `uvicorn --workers 4` hands anyone who
   reaches the form four independent buckets and four times the attempts before a `429`, and a
   restart clears all of them (the gap-5 note above). Nothing else in the auth design has that
   problem — the session is an HMAC-signed cookie and the CSRF token derives from it, so both
   are worker-safe by construction with no store to share and nothing to prune. The reason this
   stays unfixed is that the app is single-process by nature anyway: every worker would run its
   own poller loops against the same SQLite file (AGENTS.md invariant 4, and the `database is
   locked` incident in gap 4 is what two processes on one store already looks like). If this is
   ever revisited, the fix is a shared counter (the `meta` table would do, at login rates), not
   a bigger number in one process.
8. **A logout ends this browser's cookie, not a copy of it** (phase 11). The session is an
   HMAC-signed cookie with no server-side record — that is the design invariant 12 and gap 7 rest
   on — so `POST /api/auth/logout` can only tell *this* browser to drop the cookie (verified
   against a running server: a replayed token from a logged-out jar still authenticates until it
   expires). The same is true of an `Authorization: Bearer` token, which was never in a cookie to
   begin with. The only revocation this app has is rotating `auth.session_secret`, which
   `POST /api/auth/password` does on purpose, and the reset `DELETE FROM meta …` does as the
   lockout escape hatch. Accepted for a single-admin box on a domain nobody else signs in to —
   the operator's own machine is the threat model here, not a shared kiosk — and the reason a
   lost laptop is a password change and not a sign-out. A real revocation list would be a row per
   session with a prune rule, which is the thing this design traded away.
9. **The `/llm` `decode tok/s` spike is fixed, and verified live** (phase 13, 2026-10-19;
   the diagnosis is in the git-ignored `data/logs/NOTES-decode-spike.md`, the live proof in
   `data/logs/NOTES-decode-stable.md`). **Task 04's sustained watch passed the ceiling test:**
   11 minutes / 285 signed-in `/api/llm/snapshot` polls against the live engines, worst value
   **117.57 tok/s** (gufo) and **98.00** (vLLM) — both in the payload and in the stored raw
   series, 2.6× under the 300 tok/s pass line, with **zero** values printed without their
   `Δ<window>` label. The stall is still there (38 catch-up gaps < 1.5 s in 11 minutes, a
   10.6 s gap between two samples) and the estimator now absorbs it: three refusals each
   labelled with the sub-0.5 s window it refused (`Δ0.387s`, `Δ0.399s`, `Δ0.363s`), two ≥ 8×
   steps held for one poll and published only when the next interval confirmed them, and — the
   point of the whole fix — **1 m rollups topping out at 51.9 / 34.1 tok/s** where task 02 found
   a 222.7 tok/s minute, so the trend chart's y-scale stops being set by arithmetic. Fixtures
   were **not** re-captured: task 02 compared the live exposure with `tests/fixtures/` series
   by series (gufo 15/15, vLLM 128/128, identical sets), task 13 changed no parsing, and the
   page still reads `15 series · 0 unparsed` / `128 series · 0 unparsed`. Screenshots:
   `data/logs/llm-decode-stable.png` (+ `-top`, the fleet-block gap and both `Δ` tags) and
   `data/logs/llm-decode-burst.png` (a real two-slot burst: 62.1 tok/s `Δ8.5s`, fleet sum 102.4).
   The confirmed mechanism was neither a counter bug nor a label-set sum: `poller.Job._loop`
   sleeps `max(0.2, interval - elapsed)`, so any fetch slower than `LLM_INTERVAL_SECONDS`
   is followed by a poll **0.2 s later**, and the estimator happily diffed a whole stall's
   tokens over that gap — the 2026-10-18 09:16 gufo row read **3284.59 tok/s for an engine
   doing 53**, with an inflated tail because the stale sample stayed `inside[0]`. Task 03
   made that impossible by construction (see the `rates.py` bullet above): the window floor
   turns the catch-up poll into one `n/a` **and drops the stale base** so the tail re-bases;
   the fallback's `2 × span` ceiling is enforced (the honest 15.25 s diluted row still
   publishes); jumps ≥ 8× hold one poll and publish only on confirmation; and
   `values_by_series()` + `rate_key()` diff every counter
   per label set, so the latent joining-series defect (candidate (a)) contributes 0 tokens
   until it has its own baseline. The four task-02 `xfail(strict=True)` tests are green with
   their markers removed and every assertion kept. Spike-free across two days of raw history
   the fleet's real maximum is **119.9 tok/s** (gufo / deskwork) and **171.7**
   (vLLM / driveripper) — anything above ~300 is arithmetic, not work, which was task 04's
   pass line and what the watch above stayed under. The `Δ<window>` label stayed honest throughout —
   both before and after the fix it names the window actually used; do not "fix" anything
   here by hiding or rewriting it. The ~10–15 s process-wide **stall** that triggers a
   refused poll was gap 4's and `14_store_performance`'s problem, and **phase 14 closed it**
   (measured on the dev store from the sample table: coincident multi-second gaps in the 2 s `llm`
   and 5 s `ups` raw streams — the signature of a process-wide stall, not a slow engine — were
   393 in 00:00–12:00 and 198 in 12:00–21:36 on 2026-10-19, and **0** across the 110 minutes the
   phase-14 store has been running, worst `llm` gap 7.1 s and that one the recorded gufo timeout).
   The mechanism is the one phase 14 removed: a whole-table `stats()` count held the store mutex
   for ~1.25 s on every browser tab's `/api/health` poll, and `maintain()` held the write lock for
   5.4 s. The estimator no longer
   turns a slow fetch into 0.2 s arithmetic, and one `n/a` per stall is the designed degradation
   (the residual the verification pass measured is the paragraph below).
   **The verification pass found one residual path and closed it** (same day, live rows in the
   store the fix was running): a stalled fetch stamps its body with the time the poll
   *finished*, so the sample it stores is the stall older than its own timestamp. The floor
   cannot see that (the window is honest, the value is old), the sample then anchors every
   window for the rest of the span, and the jump gate published the second one because it was
   measured from the same anchor — gufo 12:42:10 read **232.28 tok/s** with exactly one request
   running at ~50, followed by the tell-tale decaying tail 146.4 / 114.1 / 94.8 and the true
   47.5 (a 13:40:58 row repeats it at 200.6). `rates.py` now requires an **independent anchor**
   for a confirmation (`_pending` stores the base sample's timestamp and the hold continues
   while the window is still diffed from it), so a skewed sample is held until it ages out and
   the honest rate publishes — the reconstruction of both rows is pinned by
   `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` (estimator) and
   `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (through
   `parse_gufo_metrics`), each of which fails on the pre-fix gate and prints the live shape.
   **That closes the gate's path, not the skew itself, and the verification pass measured the
   difference.** A slow reply's counter is true somewhere between the request and the reply and
   nothing in the payload says where, so when the resulting jump is *under* 8× the last published
   rate the gate never engages: the post-fix watch (10 min / 540 reads, `data/logs/
   llm-decode-verify-pass.tsv`) caught gufo at **203.05 tok/s labelled `Δ2.595s`** where two busy
   slots were really doing ~110, then 127.4 / 100.2 / 89.6 as the window grew back to the span and
   35.6 once the freed slot settled — every value labelled, **0 of 540 above the 300 pass line**,
   and no stored raw row over it since the restart either. A collapsed window is what makes a
   burst *look* like a rate, so the fix for this family is the ~10 s stall itself (gap 4 /
   `14_store_performance` — which has since removed it, see gap 4 and the paragraph above, so what
   is left of this family is the sub-8× skew itself), not a fresh threshold invented in the
   estimator: task 03 pinned
   `MIN_WINDOW_SECONDS` as an absolute floor and pinned that an in-bounds diluted row publishes
   (`test_a_fallback_inside_the_documented_ceiling_still_publishes`), so a span-relative floor
   would be a re-decision of that phase, not a bug fix. Do not "fix" the residual by hiding the
   short window or its label.
   **Phase 23 changed the size of the window, not the gates** (`TODO.md:8`: "a strange delay and a
   'ramp up/ramp down' effect"). What was left after phases 13/14 was the span itself: one
   10 s-window estimator served both cadences, so on the 2 s `llm` poll a slot's first seconds
   were spread over five polls — the capture replay in
   `tests/test_llm_source.py::test_gufo_the_window_the_llm_jobs_are_spanned_by_is_one_that_fills`
   prints 12.5 → 20 → 30 tok/s for an engine steady at 50, and the same five polls backwards when
   the slot ends. `build_hub` now builds **one estimator per measuring cadence** (the `llm` jobs
   spanned by `LLM_INTERVAL_SECONDS`, the host jobs by `RATE_WINDOW_SECONDS`), which reads 50 on
   the first poll after the change and 0 on the first after it ends. Every phase-13 protection
   still applies to the smaller window — `Δ2.0s` is 4× above `MIN_WINDOW_SECONDS`, the ceiling is
   relative to the span, and the 8× jump gate is the estimator's own — so this is the smoothing
   dial turned down, not a second definition of a rate.
   **The live half has landed** (task 03, 2026-10-21, read-only: nothing was written and no
   workload was started — the busy periods were the operator's own requests). A 13-minute
   signed-in pass of `/api/llm/snapshot` at the app's own 2 s cadence
   (`scripts/watch-llm-window.py`; series `data/logs/llm-window-watch.tsv`, verdict
   `data/logs/llm-window-watch.{json,log}`) recorded **zero** values printed without their
   `Δ<window>` label across 780 rows, with every window in 0.61-3.38 s and never 10 s. It found
   17 busy-edges on gufo and judged the 5 starts + 5 ends that held long enough (an edge followed
   by the next request within a poll is not a slot ending, and prefill polls genuinely decode
   ~0 — both are reported, neither is judged): **every** one was inside the 8× pass line at poll 2
   on the ungated Δ. Re-diffing the *same samples* over the span this phase replaced is the
   comparison the operator's complaint was about: median |ln(value/steady)| at poll 2 was **0.037**
   with this window against **0.924** with the 10 s one, whose median edge needed **poll 3** to
   settle (the five starts read 0.98 / 1.04 / 0.95 / 1.00 / 1.27 of steady at poll 2 on the
   ungated 2 s Δ where those same five read 0.25 / 0.34 / 0.96 / 0.40 / 0.40 on the 10 s one; ends
   took polls 4-5 to fall away against poll 2 now).
   One start had its first correct value held for a poll by the phase-13 jump gate and published
   on the next — designed behaviour, and the reason the pass line is read on the ungated re-diff
   as well as on the published column. The stored **1 m rollups** for the window topped out at
   **42.3 tok/s** (raw max 201.6 — that row's own `Δ0.610s` label says so, gap 9's sub-8× skew,
   far under the 300 pass line). A second pass over the same engines an hour later
   (`data/logs/llm-window-watch-pass2.log`) reproduced it: 3 starts + 2 ends, 0 unsettled, 0
   unlabelled, 1 m rollups max 24.7 tok/s.
   The browser half (`data/logs/23-live-window.cjs`, transcript `data/logs/23-live-window.out`,
   screenshots `data/logs/23-live-llm-{opening,burst-settled,midburst,post-free,gufo-idle,vllm-idle,final}.png`)
   checked the same claims as rendered: **866** decode figures, **0** without a tag in either
   component that prints the number (the panel hero's `.sub` and the strip card's `.prov`), every
   tag a Δ inside the estimator's own bounds — `MIN_WINDOW_SECONDS` to `2 × span`, the modal one
   `Δ2.0s` on both the page and the API (833 of 866) and never the host's `Δ10s`. Its own edge
   checks (isolated only, for the same reason) read inside the 8× line at poll 2 on two starts and
   exactly `0.0 tok/s` at poll 2 on two ends — the frame named post-free is only taken once the
   engine is provably idle, because the busy signal flickers between back-to-back requests.
10. **`driveripper`'s 16 drives read `n/a`: kernel 7.2's unprivileged-passthrough gate
    (2026-10-19, resolved the same day; the installer template that shipped the broken device
    filter was cleared in phase 17).**
    The agent is installed, the `disk` group grant is in place, the block nodes open, and the
    page renders all 16 real drive rows — but kernel 7.2.11 blocks the *agent user's* `smartctl`
    at the command level (not the open; the probe matrix is in
    `data/logs/NOTES-driveripper-agent.md`): the 8 SATA drives' SMART is ATA pass-through
    behind a SAT layer, and the kernel's unprivileged `SG_IO` allow-list
    (`scsi_cmd_allowed()`, read-safe opcodes only) does not include it — `CAP_SYS_RAWIO` or
    root unblocks it, with or without `-d sat`/`-d ata`; the 8 Samsung/WD NVMes' SMART-log
    read needs the drive's ACEL to advertise the page (the 980 PRO's `5B2QGXA7`, 2020-era,
    does not: `NVME_IOCTL_ADMIN_CMD: Permission denied`) or `CAP_SYS_ADMIN`; `deskwork` reads
    live because its firmware does. Options, in order of narrowness: (a) accept the designed
    degradation (what `/disks` rendered then — the identity is real, the values are `n/a` with a
    why-line); (b) `AmbientCapabilities=CAP_SYS_RAWIO` (+ `CapabilityBoundingSet`) on the unit
    — unblocks exactly the 8 HDDs, is the capability the kernel names for the gate, but a
    compromised agent could then write raw block I/O — test it in a `systemd-run --scope`
    before committing; (c) `User=root` with the empty capability set — unblocks all 16 but
    hands a token-guarded process root's file-read DAC (not recommended); (d) a Samsung
    firmware update — the only clean fix for the NVMe side. **Resolved the same day with a
    tighter variant of (c): `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`
    — all 16 drives read live** (root DAC is the price; the bounding set keeps the grant
    named rather than full). Two empirical notes: a root process under the unit's *empty*
    bounding set is capability-less (`CapEff: 0`), because the kernel gates check
    capabilities, not uid — that is why the first `User=root` attempt still read `n/a`; and
    `CapEff` came back `0000000000220000` (SYS_ADMIN + systemd's auto CHROOT; RAWIO in the
    bounding set but not effective) — the 8 SATA drives' reads were unblocked by SYS_ADMIN
    alone, so the effective grant is {SYS_ADMIN}; the RAWIO in the line stays as the named
    intent for the `SG_IO` gate. The agent also gained the pass-budget scale-up — 20 s floor, 5 s ×
    disk count, an explicit override still winning — because a 16-disk pass takes more than 20 s
    when the drives answer, and truncating it would null the tail drives. Phase 16 pinned that
    default in `tests/test_agent.py` (`test_the_pass_budget_scales_with_the_disk_count`,
    `test_the_pass_budget_floor_is_twenty_seconds`,
    `test_the_pass_budget_error_line_names_the_scaled_figure` — which demands the line read `80s`
    for 16 disks, not the floor; the override test stays). Re-read live on 2026-10-20: the installed
    agent answers 16/16 drives with temperatures and no `pass budget … exhausted` line.
    The `deploy/install-agent.sh` template no longer ships the device-filter lines that
    misbehaved on this host — phase 17 task 01 removed them (the `nvme*` glob matches nothing
    on this kernel because `/proc/devices` lists NVMe as `blkext`, and a strict device policy
    denies everything a list fails to match), and the unit pins in `tests/test_deploy_scripts.py`
    now assert their absence. Task 02 added the two flags that reproduce this host's remaining
    lines — `--as-root` → `User=root`, `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` →
    `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, both off by default, `--caps` refused
    without `--as-root`, `--as-root` alone accepted with the CapEff-0 note — pinned by
    `tests/test_deploy_scripts.py` (exact `User=` / `CapabilityBoundingSet=` lines in a fake-root
    sandbox, the refusal before any write, the `--help` recipe). Task 03 removed the unconditional
    rewrite: the unit is rendered to a temp file, and a re-run whose render differs from the
    installed unit prints the diff on stdout and exits 1 unless `--force` is passed, so the flag
    line above now needs `--force` to land and a routine re-run can no longer clobber this host's
    unit (pinned by the refuse/force/silent sequence in `tests/test_deploy_scripts.py`, including
    the `--force` leg of this host's own recipe — the same
    `--bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` line the
    fleet record above prints). Task 04 verified the reproduction **read-only** rather
    than by running it: the unit installed on `driveripper` (`systemctl cat`, 2026-10-20) differs
    from what this template renders with that exact flag line in **comment lines only** — every
    directive matches, `User=root`, `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, no device
    filter, `PrivateDevices=false`, the same `ExecStart` on the token file — and the host still
    answers `active` with `CapEff: 0000000000220000` and 16/16 drives with temperatures in
    `/snapshot`. So the operator's `--force` run lands the new rationale prose and changes nothing
    systemd acts on (the three captures and the exact probe list are in
    `data/logs/NOTES-17-installer-repro.md`; no live change was made while pinning this phase). Two facts that came out of reading the live host, both load-bearing for the
    next install: the sudo-granted `/home/ops/agent-install` path held the Oct-18 file (no
    `--as-root`, no `--caps`, no `--force`, no `--cert-bundle`, still nine `DeviceAllow` lines) and
    it **refused** the recipe with `unknown argument: --as-root` and exit 2 before `need_root`, so
    that path could not apply this recipe at all — while a re-run of it *without* the flags was
    still the silent clobber the guard exists for. **Resolved on 2026-10-21:** phase 24 task 03
    staged the repo's script and agent 1.1.0 into `/home/ops/agent-install/` on all three hosts
    (the directory is `ops`-owned, so no root needed) before running the granted path, and left the
    snapshot beside it as a file sudo will not execute. The rule that came out of it is in the
    fleet record above: re-stage first, because a path grant is only as current as the file. (The other
    fact from that read — an applied change reaches the file but not the running process until
    something restarts it — is in the Deployment paragraph, and README's rollout now restarts.)
    `tests/fixtures/smartctl_driveripper_*.txt` (captured via `sudo`) remain the ground truth
    for the drives' shape.
11. **The fleet runs on installed bundles; what is left is rotation hygiene**
    (phase 15 tasks 01-05 landed 2026-10-20; phase 24 task 03 rolled the fleet out 2026-10-21).
    The rollout is done and verified: gateway, deskwork and driveripper each run `agent/host_agent.py`
    **1.1.0** with `/etc/dashboards-agent.certs` installed `0640 root:dashboards-agent`, their units
    carry `--cert-bundle` + `RuntimeDirectory=dashboards-agent` (and keep `--token-file`),
    `HOST_TARGETS` is `https://` for all three with no userinfo, `agent.bundle_rev` reads `2` on
    every host and equals what `/api/certs` issued, and the nine sources are `ok`. The transcript is
    `data/logs/NOTES-24-rollout.md`, and task 04 re-read the whole end state **read-only** afterwards
    (`data/logs/verify-24-mtls.py` → `24-task04-endstate.txt`, plus the rendered proof
    `data/logs/24-task04-verify.cjs` → `24-task04-{certs,disks,system}.png`): nine sources `ok`,
    `status: active` with `deployed_bundle_rev == bundle_rev` on all three rows, `agent.tls` +
    `agent.bundle_rev` + `agent.version 1.1.0` in every host payload, driveripper's 16 drives and
    deskwork's `nvme0n1` reporting temperature, hours and wear, deskwork's GPU card drawn on
    `/system`, gateway still absent from `/disks` with `disks:gateway` still `ok`, no `http://` and
    no userinfo in any URL the dashboard serves, and **no transport or certificate row in
    `/api/summary`'s alerts**. It revoked nothing: the standing negative control for "a revoked
    serial is refused" remains phase 15's `data/logs/verify-15-tls-e2e.py` (transcript
    `verify-15-tls-e2e.out`) plus the two suite pins — `tests/test_collectors.py`'s
    `test_a_revoked_agent_is_refused_before_anything_is_dialled` and `tests/test_api.py`'s
    `test_a_revoked_agent_fails_both_of_its_sources_without_a_dial`.

    The **docs** are part of this end state, because a fleet comes back to `http://` through a
    setup file somebody copies: `tests/test_deploy_scripts.py` now pins that `README.md`,
    `AGENTS.md` and both tracked env templates name the registration trio (`--fetch-bundle`,
    `--cert-name`, `--session`), and that every `HOST_TARGETS` example line in the templates is
    `https://` with no userinfo (an empty `HOST_TARGETS=` stays allowed — unconfigured is not
    misconfigured). The token-mode dial itself is unchanged and still documented as the
    compatibility path; what may not come back is it being the shape a template seeds.

    Two things that gap learned belong with this record, because both bite before anybody reads a
    dashboard:

    * **A `certs.root.cert` row older than the SKI/AKI fix signs chains nobody can verify.** "A
      stored row always wins" is the invariant that keeps a CA from rotating with the process, and
      its cost is that a root minted before the extension keeps being used forever: a leaf issued
      afterwards carries a correct AKI pointing at an identifier its issuer does not have, and
      `ssl.create_default_context()` — strict mode, the context `AgentTls.ssl_context()` builds —
      answers `CERTIFICATE_VERIFY_FAILED: Missing Subject Key Identifier`. The agent is healthy,
      `--status` answers over TLS, and the dashboard calls the host unreachable. The repair is
      `POST /api/certs/root/rotate`, a dashboard restart (the CA and client pair are startup
      material) and a fresh bundle per host — never `DELETE FROM meta`, which throws away the
      authority instead of replacing it. Pinned by
      `tests/test_certs.py::test_a_stored_root_without_an_identifier_still_boots_and_root_rotate_repairs_it`,
      which also pins the other half: reading such a table must **not** fail closed, because
      `/certs` is the page that fixes it.
    * **These bundles came from the dev-box authority.** The fleet trusts the `certs.*` rows in the
      database `./scripts/dev-server.sh` runs against, and nothing else dials these agents (a
      `ss` sweep on each host saw only the dev box on 9120). A second deployment with its own
      authority cannot read this fleet until it carries the same root or re-issues and reinstalls;
      that is a deployment-repo decision, and the honest version is one authority per fleet.

    Everything below is the phase-15/24 history that led here.
    `certs.py` mints the CA and the dashboard's client certificate into `meta` on the first boot of
    a database — verified against the running dev store: the five `certs.*` rows exist
    and did not change across two restarts — signs per-host server certificates into
    `certs.agents`, and does reissue, revocation, and root rotation. Task 02 gave
    `agent/host_agent.py` its TLS mode (see the agent bullet above: one installed bundle, the
    per-path client-cert check, the live `# REVOKED` denylist, `bundle_rev` in the snapshot) and
    task 03 gave the collector the matching half: an `https://` `HOST_TARGETS` entry is fetched
    with this app's client certificate and verified against its own root, and a host marked
    `revoked` fails its `host:` and `disks:` jobs with `agent certificate revoked` **before any
    socket opens** — spy-pinned in `tests/test_collectors.py` and again through the real
    `build_hub` wiring in `tests/test_api.py`, where the revocation is written into the same `meta`
    table the running app reads. So both halves are proven against each other with no network and
    no hardware. What phase 15 left missing — an **installed** bundle, which is what makes
    `DELETE FROM meta` the expensive command the reset message warns about — is what task 03
    supplied: deleting `certs.*` now costs the fleet's identity, and every host has to be
    re-bundled. Phase 24
    task 01 has since closed the **dev-side** half of that sentence: `./scripts/dev-server.sh
    restart` exports every issued bundle into gitignored `data/certs/<name>.bundle` at `0600`, so
    putting one on a host is an `scp` plus `install-agent.sh --cert-bundle` rather than a hand-run
    signed `curl` (`scripts/dev-certs-export.py`, invariant 6's export rules, pinned by
    `tests/test_deploy_scripts.py` against a real uvicorn-served `create_app`). Task 02 has since
    removed even that hand-run from the rollout: `install-agent.sh --fetch-bundle <url> --cert-name
    <name> --session <token>` fetches `GET /api/certs/<name>/bundle` during the install and feeds
    the bytes to the unchanged validator + `0640` install + TLS-unit path, refusing a 401, an
    unissued name or a non-bundle body before it writes anything, with the session token absent from
    every artefact the run leaves (pinned the same way, over the same socket) — which is the path
    all three hosts were installed through, over an SSH reverse tunnel because no monitored host
    can route back to a laptop. Proving
    the dial also found one defect in task 01's output: the leaves carried no
    `AuthorityKeyIdentifier`, and OpenSSL's strict mode — which is what Python's
    `ssl.create_default_context()` turns on, and that is the context this dial builds — refuses
    such a chain with `CERTIFICATE_VERIFY_FAILED: Missing Authority Key Identifier`. Fixed in
    `certs.py` (SKI on the root and every leaf, AKI on every leaf, non-critical), pinned in
    `tests/test_certs.py`, and the reason a bundle recorded before it must not be installed. Task
    04 put the same lifecycle behind the gate: `/api/certs` lists the **stored** authority (through
    `certs.read_state`, so a page refresh cannot mint anything) with one row per configured host and
    per existing entry, `GET /api/certs/<name>/bundle` serves the stored bytes as an `attachment`,
    and the three writes rotate, revoke, and rotate the root with `restart_required: true` said out
    loud — each leaving one fixed-text `certs`-source event and no PEM outside `meta` (pinned by a
    grep of the stored rows and of every payload a browser reads). Task 05 put the page on top of it:
    `/certs` (`⚷ Certs`, in every signed-in masthead — its own included, since phase 20 task 04)
    renders that document — the root
    CN/serial/expiry, the display-only client line, one row per host with its status chip and
    issued-vs-deployed bundle rev and its **Rotate** / **Revoke** / **bundle** controls — and keeps
    two rules the API already keeps: no colour the payload did not send (no `severity` token on the
    page at all; `unissued` is a grey chip, a behind bundle a neutral `stale` one) and no control
    the API would refuse (no buttons on a `configured: false` row, no revoke or download on a
    never-issued host, the download a plain session `GET`). Writes are `confirm()`d rather than
    typed-`DANGER`d, and the restart sentence is the route's own `hint`. Task 06 has since landed
    the installer half: `install-agent.sh --cert-bundle <file>` validates the bundle by the
    agent's own text shape *before* `need_root` (a bad file says "wrong file", not "add a sudoers
    line"; exit 2), installs it `0640 root:dashboards-agent` byte-identically — the task file's
    literal `0600` cannot work, because the agent opens the bundle as itself and re-reads the
    denylist live (the exact incident the token file's comment records), and the task's own
    intent is "only root and the agent's own service user may read it", which is what 0640
    achieves — re-runs without the flag leave the installed bundle in place, the unit gains
    `--cert-bundle` + `RuntimeDirectory=dashboards-agent` (`0700`) from one conditional branch
    while keeping the token line, `--status` probes `https` with the bundle's CA and no client
    certificate (`python3` `ssl`, never an external TLS binary; an unreadable bundle says *run
    --status as root* instead of pretending), `--uninstall` sweeps the bundle with the token, and
    `deploy/agent-sudoers` gained no line (the bundle travels through the same root invocation).
    `tests/test_deploy_scripts.py` pins all of it, including the probe run end-to-end against a
    real in-process TLS agent: CA-only reads `/health`, a foreign root is refused, token mode
    keeps the plain probe. The final pass proved the composition rather than the halves: a real
    `host_agent.py --cert-bundle` on loopback and a real dashboard process over a scratch database
    (`HOST_TARGETS=loop=https://127.0.0.1:<port>`) answer 27 checks — `host:loop`/`disks:loop` `ok`
    through the certificate dial, `agent.tls`/`agent.bundle_rev` in both snapshots, the bundle
    download byte-identical to the `meta` row, an unsigned `/certs` a `303` with `?next=`, a revoke
    followed by `agent certificate revoked` on both jobs with no dial, the dashboard's own client
    identity untouched by a *host* revoke (the list is serials, not hosts), and a revoked **client**
    serial refused 403 by the running agent on `/health` too the moment the re-sealed bundle is in
    place (`data/logs/verify-15-tls-e2e.py`, transcript `verify-15-tls-e2e.out`).
    What was left was the live rollout, and phase 24 task 03 ran it on 2026-10-21, host by host, in
    the order that keeps every number honest: install **1.1.0 together with the bundle** (never the
    bundle alone — the installed agents were 1.0.0 and do not know `--cert-bundle`), restart the
    unit, let the CA-only `--status` probe answer, and only *then* flip that host's URL, because a
    URL flipped before its bundle is installed turns the host red by construction. All three: the
    trivial `gateway`, the drop-in-carrying `deskwork`, and `driveripper` with the gap-10 line
    (`--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force`), whose dry-run diff moved comments and
    the TLS lines only — `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` came
    through byte-identical, and all 16 drives read live on both sides of the rewrite (same single
    `nvme5n1` alarm chip). Every host ran the installer **twice**, once without `--force` to read
    what task 17's guard would refuse before letting it apply. Staging the repo's script and agent
    into the sudo-granted `/home/ops/agent-install/` path was step one on all three (gap 10), and
    fetching was reached through an SSH reverse tunnel because no monitored host can route back to
    the dev box. `data/logs/NOTES-24-rollout.md` is the transcript: commands, unit diffs, the
    before/after SMART tables, the screenshots, and the two operational traps named at the head of
    this gap. What the fleet now *never* does is answer an agent over plain `http` or accept a
    bearer token on the certificate path — the TODO's "the client should never allow non-mtls
    communication" is the shipped state, and a new host has to be registered the same way.
12. **The nav collapses on a phone; the page content below it does not** (phase 18's verification
    pass, 2026-10-20). At 360px `/`'s fleet table measures 450-510 px inside a 320 px
    `section.card`, and `.card { overflow: hidden }` clips the tail columns (`WORST DISK`,
    `GPU BUSY`, `TEMP`, `UP`) with no scroll container to reach them — `table.data` only scrolls
    with its `.scroll` modifier, which that table does not carry. Measured as pre-existing by
    serving the `git show HEAD:` bytes of both `dash.css` and `pages/index.html` over a live
    session: same clipping (a clipped card never un-clips when the document scrolls sideways).
    What phase 18 did change is that the document no longer scrolls sideways at a phone width at
    all — 360 px where it was 750 px — because the bar that overflowed its fixed 58 px row now
    breaks into rows, so nothing is lost and nothing gained: the masthead is complete at 360 px and
    the table tail is still out of reach. The fix is a responsive-table pass (`.scroll` on the wide
    tables, or a per-host card layout at ≤640px) and it touches pinned page shape on `/`, so it is
    a phase rather than a cleanup — `21_page_cleanups`' `.scroll` on the `/certs` table is the same
    shape applied to one table and does not reach this one; `27_mobile_tables_and_audit` is that
    pass, still in `todo/`. Of the two operator TODO items that sat
    beside it, `20_masthead_nav_polish` has landed in full (now complete): the bar is three
    left-justified rows at every width, it is no longer sticky, the phone menu slides, and both
    `⚙ Account` and `⚷ Certs` render on all seven signed-in pages including the two they name
    (task 04, reversing phase 15's "no self-link" rule at the operator's request). The other,
    `24_dev_mtls_rollout` (never accepting a non-mTLS agent) is closed: its fleet half landed on
    2026-10-21 and task 04 re-read the end state read-only the same day — gap 11. No `http://`
    agent URL and no token-mode agent is left on this fleet, and the setup templates no longer
    seed one (`tests/test_deploy_scripts.py` keeps it that way). What remains of phase 24 is
    ordinary rotation hygiene, not a missing step.

## Locked decisions (and why)

- **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10s resolution, one file the
  container owns. Rollups keep a year of charts at a few hundred MB.
- **Our own stdlib host agent, not node_exporter.** One JSON snapshot with exactly the
  fields the page needs (GPU model strings, per-mount usage, temperature labels) versus
  ~1500 series and a re-labeling layer; no dependency to install, `ProtectSystem=strict`
  friendly, and read-only by construction. `scripts/capture-node-exporter.sh` remains to
  compare ground truth if that is ever revisited.
- **Our own NUT client, not `pynut`/`upsc`.** The protocol is ~10 line commands, the
  third-party libs are subprocess-based or stalled, and we need instant commands with
  explicit gating.
- **In-house Prometheus parser.** Only two upstreams use the format; keeps `prometheus_client`
  out of the dependency set.
- **`cryptography` for X.509, dashboard-side only** (phase 15). Minting a CA and signing leaves is
  not something to hand-roll, and there is exactly one place in this app that does it
  (`certs.py`). The agent gets no new dependency: it loads the PEMs the dashboard signed with stdlib
  `ssl`, which is why the bundle is plain text and why the agent still runs on a host's Python 3.10.
- **Port 8090**, inherited from the `llm_monitor` prototype this started from: 8000 is vLLM
  and 8731 is gufo in this fleet.
- **`.env` is a seed, never an override** — container/systemd env must win.

## Definition of done (per change)

The first three are exactly what `bash .agents/validate.sh` (the phase gate) checks, so no
phase may be called done on a subset of them.

`uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated`
empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted
in a browser against **live** data · fixtures re-captured if an upstream contract changed ·
this file and `README.md` updated when a rule, command, gap, or fleet fact changed.