dashboards/AGENTS.md

# AGENTS.md — homelab dashboards

The plan, the rules, and the environment facts an agent needs to work on this repo without
re-deriving them. `README.md` is the operator-facing how-to; this file is the why, the
invariants, and the state of the fleet.

## Working protocol

This project runs the `.agents/phases/` phased-execution pipeline.

1. Always read `.agents/PLAN.md` first to understand the project context and goals.
2. Follow the phased execution protocol in `.agents/phases/`.
3. Never modify `.agents/PLAN.md` or any files in `.agents/phases/complete/`.
4. If you need to update any file in `.agents/phases/todo/`, you must ask the user for permission first.
5. Strictly adhere to the **LOCKED DECISIONS** listed in `.agents/PLAN.md`.

Everything below this section is the project's own content: what the app is, the rules that
keep its numbers honest, and the state of the fleet it watches.

## What this is

Three dashboards for one homelab, behind one domain, in one process, backed by one SQLite
file — plus an overview page that says which of the three you need:

| Route     | Question it answers                                                            | Source                                            |
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
| `/`       | Is anything wrong, and where do I look first?                                  | all three, in one request                         |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history   | `agent/host_agent.py`, our own stdlib agent/host  |
| `/llm`    | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles      | gufo (llama.cpp family) + vLLM `/metrics`         |
| `/ups`    | Battery health, runtime left, power events, battery-test buttons                | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |

History is a first-class feature, not a sparkline afterthought: every number worth graphing
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.

## Non-negotiable rules

These are the things that make the numbers trustworthy. Breaking one is a bug even if the
page still renders.

1. **Read-only by construction, with exactly one exception.** Nothing in this app writes to
   an upstream except `POST /api/ups/command`. That endpoint is gated four times
   (`NUT_ALLOW_ACTIONS`, not `DASH_READ_ONLY`, the upsd.users account actually having the
   command, and a UI confirmation — `DANGER` for anything that can cut power) and the server
   re-checks every gate on every request. Hiding a button is not a safety control.
   Commands that would cut output are refused outright while the UPS is already on battery.
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
   as `0`. Every derived value carries the `source` string it came from and `derived: true`
   so the UI can print *how* it was obtained (`computed:Δtokens/Δt`, `Δ10.0s`). Aggregating
   two different series into one rate is the failure mode this rule exists to catch — the
   rate estimator is keyed by the *resolved* upstream series name for exactly that reason.
3. **Unconfigured is not broken.** A source with no env config answers
   `404 {"configured": false}` **without dialling**, and the page says "not configured".
   A red panel for a server the operator never installed destroys the signal from the
   panels that matter.
4. **One dead source must not blink another — and nothing may stop polling in silence.** Each
   source is its own asyncio task with its own timeout and health record (`poller.py`). A
   failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it —
   on `/` too, where a host that has never answered still gets a fleet row
   (`unreachable: true`, dimmed, every number `n/a`) because dropping the row would turn a dead
   agent into "never configured". Transitions write events; repeated failure writes one line,
   not 500. **The task is the source's only poller, so no exception may end it**: a store that
   refuses an otherwise good poll is a *reported* failure (`ok: false`, the real reason in
   `error`, `last_ok` frozen so the page ages the numbers, one `store-error` event then one
   `recording` event on recovery), and anything that escapes the poll body is logged and the
   loop continues. Four dev-server sources once froze behind an unretrieved SQLite exception
   while `/api/health` went on calling them `ok` — a frozen source that looks healthy is the
   one failure mode worse than an outage.
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
   charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
   no outbound access — a CDN reference is a request that never comes back.
6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets`
   moves it into an `Authorization` header, so `Target.url` ends up clean). Nothing that leaves
   the process may carry one: `/api/config` serves `Target.display_url` and
   `config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message
   is redacted too — those strings go into the health record *and* the `event` table, which
   `/api/events` serves to the browser and `page-system.js` prints. No token, password, or
   personal hostname goes into tracked files, fixtures, or the browser.
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
   `sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
   2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
   `scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
   exporters) rather than trusting a README or a memory of an upstream.
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
   Check after touching dependencies: `uv pip list --outdated` must come back empty.
   Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
   `python3.14`) — do not lower these to make something else work.
9. **SQLite owns the history and must stay bounded.** Writes go through `store.py`; new
   metrics need a rollup path and a retention story, or a year of charts becomes a disk
   emergency. Put the DB on a volume in deployment.
10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`,
    http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot
    before they are called done. `data/logs/*.png` is where those land.

## Architecture

```
agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        ▼                                  ▼                        ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
        │  parse + Δcounter/Δt                                                     │
        ▼                                                                           ▼
sources/host.py · sources/llm.py · sources/nut.py                    poller.py (task per source)
        │        Result(payload, samples, events)                                    │
        ▼                                                                             ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/
```

Responsibilities, one line each:

- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
  `statvfs`; serves `GET /snapshot` + `GET /info` + `GET /health`. Refuses to bind
  off-loopback without a token (exit 2). Counters stay counters: the dashboard computes
  rates so the agent never remembers who polled it last.
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
  env var always wins. Defaults are safe except the sources, which default to unset.
- `poller.py` — job loop, health records, stale/last-good semantics, transition events. The
  success path records (payload → samples → events) and only *then* moves `last_ok`, so a
  refused SQLite write ages the source and alerts instead of leaving it current with charts
  that have silently stopped drawing.
  Events are stored under the **dashboard** name (`system`/`llm`/`ups`), not the sample
  source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
  (found on `/system` during phase 03 browser verification).
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping. Failure
  messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one
  that was dialled, because they become health records and event rows.
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
  (`gpu.busy_pct@<index>`).
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
  `VLLM_HISTOGRAMS`), measured Δ rates, prefill speed as
  `mean(prompt tokens)/mean(prefill time)`, session list from gufo `/slots`.
- `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not
  a mystery score), `COMMANDS` catalog with risk tiers, `COMMAND_ALIASES` + `resolve_command()`
  (drivers answer two names for the same action — `beeper.enable`/`beeper.on`,
  `beeper.disable`/`beeper.off` — so one entry is offered per action and any of its names hits
  that entry's verdict, never a gate the rendered name missed), `available_commands()` filtering,
  kelvin-vs-celsius normalisation, and `charger_status` normalised to the `CHARGER_STATUS`
  keys (the same lowercase token `states()` and `on_battery` already compare), and `states()`
  publishes one chip per fact — a driver exporting both `battery.charger.status` and the
  legacy flag that says the same word gets "charging, online", not "charging, online,
  charging". `battery_runtime`
  comes from `battery.runtime` **only** — `battery.runtime.low` is a shutdown threshold, not
  time left, so it lives at `battery_runtime_low` and never becomes the hero's "runtime left".
- `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a
  battery. It must never be the default, and every payload it produces says `simulated`.
  Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that
  ages), so a fresh simulator reads healthy and amber keeps meaning something. Its status
  follows the real drivers: a below-float pack on mains reports `OL CHRG`/`chg` (the glyph's
  bolt state is keyed on that token, and the model has to be able to produce it) and a battery
  being drawn from is `OB`/`dchg`, never `ol`. Every command it advertises is implemented and
  visible in a variable a real driver publishes — the beeper ones set `ups.beeper.status`
  (`enabled`/`muted`/`disabled`), and a test window that ends writes `ups.test.result: PASSED`
  while an aborted one writes nothing; the load/shutdown commands answer `OK` without modelling
  an outlet, because a faked power cut in the history is indistinguishable from a real one.
  `from_env()` is the single place the `NUT_SIMULATE` truthy rule lives — `app.py` builds the
  monitor through it, so the assembly and the model cannot disagree about what is enabled.
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
- `rates.py` — Δ/window estimator, keyed by resolved series name.
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
  `classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
  beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
  an unknown. The verdicts travel **additively** in payloads — `cpu.severity`,
  `memory.severity`, `disks.severity`, each UPS card's `severity` (charge/load/runtime),
  `severity` inside rated LLM cells (KV pressure, classified on a
  percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
  row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
  There is **no `gpu.*` row and no `gpus[].severity`** (phase 06): these are LLM servers,
  a busy GPU is the job at every level, so the GPU card and the fleet GPU column render
  uncoloured — like `net.*`, whose colourlessness predates it. Removing the verdict is not
  removing the data: `gpu.busy_pct@N` / `gpu.mem_used_pct@N` are still stored and charted.
- `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload,
  recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts`
  (`{id, severity, source, message, href, at}`, fail first then warn, then by source). A
  pure projection: no I/O, no store, no thresholds of its own — it reads job states for
  reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions.
  Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but
  older than 3× its own poll interval is `warn` while a source that stopped answering is
  `fail`; a dead source gets exactly one line (its stale payload adds nothing) that names it
  once — the poller's `"UpstreamError: "` wrapper and the duplicated host name are stripped,
  because a message that says `gateway` twice reads as two problems; `unknown` and missing
  numbers are silence; one alert per (source, metric) and at most 12, the last being a marker
  that says how many it hid. This is why the overview can be two panels.
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
  `/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
  the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `llm.engines`,
  `ups.ups`, the `states` maps and `recent_events` stay for programmatic consumers (live
  tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events` are unchanged
  and `alerts` is additive — `tests/test_api.py` pins every one of those key paths, with and
  without targets configured, so retiring a panel can never shrink the payload. A configured
  job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
  `system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
  counts the machines there are and `hosts_live` the ones answering.
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
  with an in-memory DB and no targets.
- `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the
  colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
  `ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
  `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
  `.hero` big number (plus an optional `.bar`) from that class — it never compares a number
  to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
  values the API does not rate, and **no page script calls it any more** — `/` was converted
  in phase 02, `/system` in phase 03, `/llm` in phase 04 and `/ups` in phase 05, so every
  colour on all four pages comes from a `severity` verdict the API sent
  (`tests/test_api.py` fails if a page script names it again). A value the API should rate
  gets a row in `severity.py`, never a bound in JS.

Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
series — `/api/metrics` answers with exactly these names.
That scheme is the history API's key — treat it as a public interface.

## The fleet (real environment, verified 2026-10)

| Host              | ssh alias          | What it gives                                                        |
| ----------------- | ------------------ | -------------------------------------------------------------------- |
| `driveripper`     | `driveripper-ops`  | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards |
| `deskwork`        | `deskwork-ops`     | Strix Halo (Radeon 8060S iGPU), gufo `:8731`                          |
| `gateway`         | `gateway-ops`      | router, no GPU                                                       |

- Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123`
  by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev.
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
  on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
  *Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
  | nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are
  exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`:
  `apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
  `ups.description`, the operator's label — the driver's own `identity.model` answers
  `GX1500U`, so the description is what identifies the box).
  Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date
  (`battery.date` is often an unset placeholder — `2001/09/25`, slashes and all, on the APC),
  CyberPower reports `battery.mfr.date: CPS` (not a date → `unknown`, never a fake age),
  **neither driver reports a battery temperature or the NUT 2.8 `battery.charger.status`**
  (so the battery glyph's state has to come from `on_battery`, and `charger_status` must stay
  optional), and `ups.test.result: No test initiated` is the factory state on both.
  Both UPSes are recorded verbatim (`LIST UPS`/`LIST VAR`/`LIST CMD`, serials scrubbed) as
  `tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, and the wire quotes a
  field only when it must — `VAR apc ups.status "OL"`, not three quoted fields.
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
  set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
  work that is still blocked).

## Commands

```bash
uv sync --python 3.14                     # deps + editable install
cp .env.example .env                      # production-shaped config
cp .env.dev.example .env.dev              # dev-only: ssh tunnels + live targets

./scripts/dev-tunnels.sh start            # agents (9121-9123) + DEV_FORWARDS (upsd 3494)
./scripts/dev-server.sh restart           # keep http://127.0.0.1:8090 up
./scripts/dev-server.sh status            # per-source ok/DOWN
./scripts/dev-server.sh logs

uv run pytest                             # hermetic: 550 passed + 8 live-skipped, no network
uv run pytest --cov                       # coverage gate (97.9%; floor 94%)
uv run pytest -m live --live              # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh             # re-record ground truth (agents + UPS captures)

bash .agents/validate.sh                  # the phase gate: the four checks above (pytest
                                          # --cov --cov-fail-under=94, ruff, pyright, plus a
                                          # node --check of web/static) + repo hygiene.
                                          # Always `uv run` — a bare `python3`/`ruff` on PATH
                                          # tests an interpreter nobody deploys.
```

Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
`--status`/`--uninstall`; it stores the token in a root-owned 0600 file and hands the agent
`--token-file`, never argv, and runs under `ProtectSystem=strict`).
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
service here. The source itself is `git@git.reeseapps.com:services/dashboards.git` (`origin`,
branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only gate.

## Testing policy

**Real captures for shape, fakes for behaviour, `live` for drift.**

- Anything that must match an upstream — `/proc` layouts, vLLM series names, NUT framing —
  is asserted against a verbatim capture in `tests/fixtures/`. If the schema drifts, a
  fixture test fails before a dashboard does.
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
  rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
- **Page payloads that a rewrite must not break are pinned as a table.**
  `tests/card_contract.py` holds the dotted `/api/system/snapshot` paths the `/system` hero
  cards read (`CARD_METRICS`, `GPU_METRICS`, `HOST_SHAPES`); `tests/test_system_contract.py`
  asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and
  `test_every_host_serves_the_hero_card_contract` asserts the same table against the live
  agents — so a rename fails a test, and a card that shows `n/a` cannot be "fixed" by
  inventing a field. `network` carries no `severity` on purpose (throughput has no good/bad)
  and neither does a GPU row (phase 06: a busy GPU is the job on an LLM server) — the
  contract pins both absences. `tests/ups_contract.py` is the same device for the `/ups`
  hero row and battery glyph (`HERO_FIELDS`, `UPS_SHAPES`, `assert_ups_hero_contract`):
  `tests/test_ups_contract.py` drives it from `tests/fixtures/nut_<ups>.vars` — verbatim
  `LIST UPS`/`LIST VAR`/`LIST CMD` replies from the two real UPSes — and
  `test_ups_hero_fields_are_present_on_the_real_batteries` drives it against live upsd. The
  captures are what make the driver disagreements testable: one of them answers
  `battery.mfr.date: CPS`, neither exports `battery.temperature` or
  `battery.charger.status`.
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
  fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
  unset, so partial fleet availability is normal. **Live tests must not write** — a battery
  test on hardware something is plugged into is not a test.
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
  so it stays in the default run and is *not* marked `live`.
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
  94% (`--cov-fail-under`), and ~98% is where we are; keep it there. New/modified Python
  still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
  never a real target, and never a real upstream: `/api/ups/command`'s send is proven against
  a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live
  suite is read-only — that fake is the only place the write reaches a "driver".

## UI conventions

Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
`LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
silence, and the legend is a sibling that labels nothing) — `/system` does this for all three
slots, pinned by `tests/test_api.py`. Window chips (5m…30d on `/system` and `/llm`, 1h…90d on
`/ups`; the API accepts 5m…90d). A panel that cannot get data says
why in the panel — the operator should never have to open devtools to learn that a source is
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.

The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per
`/api/summary` alert: verdict class, source linked to its dashboard via the alert's own
`href`, message, age) and **Fleet**. A non-empty list floats the alerts panel above the
fleet table (`#overview-panels[data-state]` swaps them); an empty one is an explicit
"all clear" state below it, because an empty list is a message, not a blank box. A host that
is not answering keeps its row, dimmed with the shared `.stale` class and reading `n/a`, so
the table never implies last good numbers are live and never silently loses a machine.
No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the
two-panel shape against the served HTML.

Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an
`id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for`
attribute references — the orphan a deleted renderer leaves behind, caught on served bytes
because there is no JS test runner (a locked anchor). Same rule for `dash.css`: delete the
rules a removed panel made dead (`.grid.wide` went with the overview's fleet+events row);
component vocabulary the pages are still being converted to (`.hero-bar`, the grid and bar
variants) stays even while only one page uses it.

A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's
`severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*`
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare.

A cell-scale trend is `DashCharts.sparkline()` — the `llm_monitor` `drawSpark` idiom, so a
percent metric is plotted against `fixedMax` (**never** autoscaled: an 8 % KV window that once
held a 92 % spike must not read flat, and the axis gets top/bottom scale marks so "a third of
the way up" says a third of what). `align: 'right'` pins the newest sample right and grows the
line leftwards inside a real `span` in seconds — x stays time, not sample index. Stroke colour
comes from the payload `severity` via the `--ok`/`--warn`/`--fail` tokens only; an unrated
series draws a faint `--text-faint` stroke with no fill, because a shape is not a verdict.
Below `minPoints` (2) it prints `no history` rather than drawing a confident line through one
point, `labels: {current, max}` renders the `now / max` foot through the caller's `format`, and
there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the
other half of the same honesty for count metrics: history arrives as rollups, so eight seconds
of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for
a queue nobody was in — `/llm` passes `minSpan: 1` on sessions and queued so its axis is never
finer than one request.

A battery is `DashCharts.battery(container, {charge, severity, state, label})` — one inline SVG
whose **fill *is* the charge**: 1 viewBox unit per percent, clamped to 0–100 and never rounded up
(a 2 % cell must look nearly empty, because a reassuring battery above a flat one is a lie),
clipped to the plate so 0 % leaves an empty shell rather than a stray pixel. Colour comes only from
the payload `severity` token; a measured charge with no verdict draws faint grey at half strength
(`sparkline`'s rule — a shape is not a verdict) and never an empty shell ("0 %" and "not rated" are
different sentences). Every state has a **shape** as well as a colour, so the glyph survives
monochrome: solid shell `online`, bolt `charging`, dashed outline `onbattery`, no fill with a dash
pattern *through* the body `unknown`; a state word it does not know degrades to `unknown`, never to
a confident "on line". Accessible **by default** — `role="img"` + `<title>` + `aria-label` built
from the inputs ("apc: Charge 100 percent, on line"), because the glyph carries information the
markup does not say in words — and a page that prints that sentence next to it passes
`decorative: true` to make the `<svg>` `aria-hidden` rather than say it twice. No animation and no
`ResizeObserver`: one `viewBox` + `preserveAspectRatio` scaled by `.battery` in `dash.css`, so the
hero row gets a big cell on a desktop and a small one on a phone.

**The `/ups` card is led by the glyph and three heroes** (phase 05): the battery on the left, then
**Charge %**, **Load %** and **Runtime left** on `.heroes` at the shared 2.4rem, coloured from
`severity.charge_pct` / `severity.load_pct` / `severity.battery_runtime` and nothing else. Charge
carries **no** `.bar` — the glyph beside it is that value drawn as a shape, and one value drawn
twice in one row is the pattern this app avoids (Load keeps its bar; nothing else draws it). Its
`.sub` lines are the context each number needs and nothing more: `24 V nominal · 27.3 V` for the
pack, `128 W of 900 W` when the driver publishes both `realpower` figures (a wattage inferred from
a percent against another model's rating would be an invented number), `at the current load` for
runtime. Runtime prose is `runtimeText()`'s (`1h 32m`, `10m 21s`) and an estimate the driver never
made goes into `Dash.hero()` as `null`, so it prints faint and uncoloured rather than as `0m`. The
readings the hero row displaced — line in, output, battery volts and amps, battery °C — stay on the
card, demoted to `.ups-electrical`: muted, `.95rem`, never folded away, because they are what you
reach for *while* the UPS is on battery, and battery °C reads `n/a` on both real drivers here.
`state` is derived in the page from `on_battery` → `charger_status` (`chg` charges; `dchg`/`rec` are
the server's own on-battery tokens; `full` is **on line, not charging** — a rested floating battery
reports it) with one fallback: a state chip the API rated `ok`, because `nut.states()` reserves
`ok` for on-line/fully-charged and neither real driver here publishes `battery.charger.status` at
all. `ups.status` itself is never re-parsed in JS (`tests/test_ups_contract.py`). `/ups` passes
`decorative: true`, because the charge hero and the state chips already say the sentence. The glyph
repaints only when charge / verdict / state change (`drawGlyph()` keys a `data-drawn` triple): the
charge float jitters every `NUT_INTERVAL_SECONDS`, and rebuilding the `<svg>` on that re-announces
an accessible name for a difference no eye can see.

An event row names its subject **once**. Every feed (`/system`, `/llm`, `/ups`) renders the
message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a
*leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py`
messages start with their own label, and each row already prints that label as a chip, so
"gufo gufo is answering again" reads as two problems. A name in the middle of a message stays:
the UPS rows used to `replace()` it anywhere and mangled "Quick battery test sent to apc: OK".
This is presentation only — the stored event and `/api/events` keep every word, and the raw
error string still surfaces in `/api/health`.

The `/system` host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the
`.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script).
Colour comes from the payload's `severity` dicts and nowhere else; a missing number or
verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**. NET is
one slot with two lines (`.hero-lines`, `↓` in / `↑` out) and is deliberately colourless:
throughput has no good/bad, so `network` carries no severity. The GPU slot holds one hero
per card whose `.sub` carries model, VRAM used/total, the honest APU `pool` view and the
card's temperature (`gpus[].temp_c` — the agent's max over that card's hwmon labels, the
same value stored as `gpu.temp_c@N`, so the hero and the history cannot disagree), and it is
colourless for the same reason (phase 06): a 95 %-busy card on an LLM server is the job, so
the hero carries no `severity` and no utilisation bar — a bar under a verdictless number
would be a verdict by shape. A card whose driver exposes no temperature input prints **no**
temperature in that footnote rather than `n/a` or `0 °C`: "where you can" is the operator's
wording, and `n/a` is reserved for numbers the card is supposed to have. Everything not
glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU power and the
per-label temp breakdown, filesystems, processes — folds into a collapsed "More detail"
`<details>`: a simpler surface, not a less complete page (the table keeps its Temp column;
this was a promotion, not a de-duplication).

A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
those (a range when the hosts disagree) instead of the poll interval, which only coincides with
them at the defaults. `/llm`'s hero `.sub` prints the rate cell's own `window` through the same
`provenance()` tag, and the hero's `title` carries the exact `source` string — "78 tok/s" must
always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_monitor`'s
`setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification).

**The `/llm` page reads per engine** (phase 04, decided in
`.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`): a row of two
`Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**, because 0 tok/s
on an idle server is healthy and `severity.py` has no throughput row), then a compact `.cells`
list — label left, tabular value right, a `sparkline()` fixed at 160px so rows line up, and a
mono `p95 …` foot under every latency mean (no hover; p95 needs no stored series). Rows are
engine-scoped by the `only` flag: gufo never grows a TTFT mystery, vLLM never a spec-accept
hole. `preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded
threshold); lifetime token counters are a quiet footnote line. **One `/api/llm/history` request
per refresh** carries every chart *and* spark series (`HISTORY_SERIES`; the endpoint answers
unknown names as empty series on purpose), and the two series that addition required
(`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) are stored additively in
`sources/llm.py` HISTORY_METRICS. The engine-totals card kept only engines-live/active/queued
and **hides below two engines** — a cross-engine sum of one engine duplicates that engine's own
panel, and the summed-KV bar went because `fleet_totals` carries no `severity` to render. The
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
sit at their own height (`#engines` carries `.items-start`): a 5-row gufo panel next to a 7-row
vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data".

**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
panel and the host/metric detail picker are deleted, along with the `#chart-*` / `#detail-*` ids and
the `select` / `label.field` styling only those pickers used. Each panel stacks its own
`.host-charts` — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where
the GPU slot is created and destroyed with `host.gpus`, so gateway shows neither a phantom card
nor a hole. **That stack is collapsed behind a per-host `Charts` `<details>`** (phase 06: "only
visible if expanded… there will be lots of hosts"): the default `/system` view is the hero row,
the summary carries the window label the stack will draw (the chips themselves stay in the page
head, one control per page), and `drawHostCharts()` is the single place that decides whether to
paint — so nothing is built or drawn while the stack is shut (`LineChart` inside a
`display:none` container measures 0 px, and rebuilding three charts per host every poll for
plots nobody sees is the waste this removes). The **history request is deliberately unchanged**:
one per host per refresh, naming every series the panel could draw, and its response is cached
per host so the first expand paints from data in hand rather than waiting a poll; a cached
response from another window is re-fetched instead of being drawn under the wrong label.
Expanding one panel must not stretch its collapsed siblings into blank cards, so `#hosts`
carries the existing `.grid.items-start`. **One `/api/system/history` request per host per refresh**
(`hosts=<that host>&metrics=<that panel's series>&range=…`, pinned by
`test_every_system_chart_is_drawn_inside_its_own_host_panel` and the collapse by
`test_system_history_charts_are_collapsed_behind_a_toggle`): unknown metrics come back as
empty series on purpose, so a panel asks for every series it could draw and asks once. A legend
here names metrics (or one GPU's model), never hosts — the panel heading already says who this
is, and a legend that grew a second host is the bug this replaced. A host that stops reporting
loses its panel, its `Charts` toggle *and* its charts (`pruneHosts` deletes the card plus the
payload/chart/history maps), never a chart of nothing or an orphan control in the grid. Per-core
visuals are gone from `/system` and `renderCores` is **deleted** from `charts.js` with its last
caller.
`drawHostChart()` shades a chart only when one series has points — two or three filled areas
stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy
memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryable from
`/api/system/history` — removing the plot is not removing the data.

## Known gaps / next work

1. **UPS commands work, and the account can now cut power — the app's gates are the control.**
   Three defects were cleared on 2026-10-17, each verified against real hardware. (a) `NutClient.login()` sent `USER`/`PASS`, which the deployed
   upsd 2.8.5 does not implement — each answered `ERR UNKNOWN-COMMAND`, so `_read_line()`
   raised and `POST /api/ups/command` could only 502 before `instcmd()`, credentials or not.
   It now sends `USERNAME`/`PASSWORD`, the pair the server's own `HELP` advertises
   (`HELP VER PROTVER GET LIST SET INSTCMD LOGIN LOGOUT USERNAME PASSWORD STARTTLS`), pinned
   by `test_login_uses_the_verbs_upsd_2_8_5_answers_to` — the fake upsd has no `USER`/`PASS`
   branch, so a regression answers `ERR UNKNOWN-COMMAND` exactly like the real thing.
   (b) 12 of the 23 commands the two real UPSes advertise had **no `COMMANDS` entry**, and
   `available_commands()` drops anything the catalog misses — the page silently offered less
   than the hardware does. Every advertised command is catalogued now, each with a risk tier:
   `test_every_command_the_hardware_advertises_is_catalogued` fails if a capture gains a
   command the catalog does not know, and
   `test_every_command_that_can_cut_power_is_in_the_dangerous_tier` keeps `load.off`,
   `driver.killpower` and the `shutdown.*` sequence in the tier that types `DANGER` and is
   refused on battery. `shutdown.stop` is the exception the name hides — it *cancels* a
   shutdown, so it is `safe`, and the test says so in words.
   (c) `[dashboards]` in `/etc/ups/upsd.users` is `instcmds = all` at the operator's request,
   so this app may send every advertised command on both UPSes, power-cutting ones included.
   `.env.dev` (mode `0600`, gitignored) carries the credentials.
   What remains to know:
   - **Proven live, and reversibly.** `beeper.enable` returned `OK` and `ups.beeper.status`
     read `enabled` on the APC afterwards; `driver.killpower` without the word `DANGER` still
     answers 422 and never reaches the wire. A battery test has deliberately not been run —
     it puts real load on a battery things are plugged into, which stays the operator's call.
   - **`enabled: true` cannot mean "this account may send it".** `available_commands()`
     filters on `LIST CMD`, which upsd serves **unfiltered**, and NUT has no call for a
     user's privileges — so the wire cannot tell the page what the account is granted. With
     `instcmds = all` the two agree by configuration rather than by check, which is precisely
     why the risk tiers and the typed `DANGER` are load-bearing and not decoration.
   - **Pre-existing upsd log noise:** every upsd start logs `Unknown upsmon type =` (20
     occurrences over 12 starts, oldest before this work). Harmless — `upsmon` logs into both
     UPSes right after it — but unexplained.
   Editing `upsd.users` at all: never grant `actions = -`, and never write it blind — the
   running `upsmon` authenticates as its own account from that same file for both UPSes, so
   dropping it means no orderly power-down. Copy the untouched sections from the live bytes
   and `cmp` them before writing; `tee` truncates in place, so owner and mode survive.
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
   (`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
   fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
   all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the
   field map is written.
3. **UPS history depth.** Events and the numeric series are stored; long-window battery
   health (capacity fade over months) has the rollups but no dedicated view.

## Locked decisions (and why)

- **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10s resolution, one file the
  container owns. Rollups keep a year of charts at a few hundred MB.
- **Our own stdlib host agent, not node_exporter.** One JSON snapshot with exactly the
  fields the page needs (GPU model strings, per-mount usage, temperature labels) versus
  ~1500 series and a re-labeling layer; no dependency to install, `ProtectSystem=strict`
  friendly, and read-only by construction. `scripts/capture-node-exporter.sh` remains to
  compare ground truth if that is ever revisited.
- **Our own NUT client, not `pynut`/`upsc`.** The protocol is ~10 line commands, the
  third-party libs are subprocess-based or stalled, and we need instant commands with
  explicit gating.
- **In-house Prometheus parser.** Only two upstreams use the format; keeps `prometheus_client`
  out of the dependency set.
- **Port 8090**, inherited from the `llm_monitor` prototype this started from: 8000 is vLLM
  and 8731 is gufo in this fleet.
- **`.env` is a seed, never an override** — container/systemd env must win.

## Definition of done (per change)

The first three are exactly what `bash .agents/validate.sh` (the phase gate) checks, so no
phase may be called done on a subset of them.

`uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated`
empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted
in a browser against **live** data · fixtures re-captured if an upstream contract changed ·
this file and `README.md` updated when a rule, command, gap, or fleet fact changed.