# AGENTS.md — homelab dashboards

The plan, the rules, and the environment facts an agent needs to work on this repo without
re-deriving them. `README.md` is the operator-facing how-to; this file is the why, the
invariants, and the state of the fleet.

## Working protocol

This project runs the `.agents/phases/` phased-execution pipeline.

1. Always read `.agents/PLAN.md` first to understand the project context and goals.
2. Follow the phased execution protocol in `.agents/phases/`.
3. Never modify `.agents/PLAN.md` or any files in `.agents/phases/complete/`.
4. If you need to update any file in `.agents/phases/todo/`, you must ask the user for permission first.
5. Strictly adhere to the **LOCKED DECISIONS** listed in `.agents/PLAN.md`.

Everything below this section is the project's own content: what the app is, the rules that
keep its numbers honest, and the state of the fleet it watches.

## What this is

Three dashboards for one homelab, behind one domain, in one process, backed by one SQLite
file — plus an overview page that says which of the three you need:

| Route     | Question it answers                                                            | Source                                            |
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
| `/`       | Is anything wrong, and where do I look first?                                  | all three, in one request                         |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history   | `agent/host_agent.py`, our own stdlib agent/host  |
| `/llm`    | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles      | gufo (llama.cpp family) + vLLM `/metrics`         |
| `/ups`    | Battery health, runtime left, power events, battery-test buttons                | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |

History is a first-class feature, not a sparkline afterthought: every number worth graphing
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.

## Non-negotiable rules

These are the things that make the numbers trustworthy. Breaking one is a bug even if the
page still renders.

1. **Read-only by construction, with exactly one exception.** Nothing in this app writes to
   an upstream except `POST /api/ups/command`. That endpoint is gated four times
   (`NUT_ALLOW_ACTIONS`, not `DASH_READ_ONLY`, the upsd.users account actually having the
   command, and a UI confirmation — `DANGER` for anything that can cut power) and the server
   re-checks every gate on every request. Hiding a button is not a safety control.
   Commands that would cut output are refused outright while the UPS is already on battery.
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
   as `0`. Every derived value carries the `source` string it came from and `derived: true`
   so the UI can print *how* it was obtained (`computed:Δtokens/Δt`, `Δ10.0s`). Aggregating
   two different series into one rate is the failure mode this rule exists to catch — the
   rate estimator is keyed by the *resolved* upstream series name for exactly that reason.
3. **Unconfigured is not broken.** A source with no env config answers
   `404 {"configured": false}` **without dialling**, and the page says "not configured".
   A red panel for a server the operator never installed destroys the signal from the
   panels that matter.
4. **One dead source must not blink another — and nothing may stop polling in silence.** Each
   source is its own asyncio task with its own timeout and health record (`poller.py`). A
   failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it —
   on `/` too, where a host that has never answered still gets a fleet row
   (`unreachable: true`, dimmed, every number `n/a`) because dropping the row would turn a dead
   agent into "never configured". Transitions write events; repeated failure writes one line,
   not 500. **The task is the source's only poller, so no exception may end it**: a store that
   refuses an otherwise good poll is a *reported* failure (`ok: false`, the real reason in
   `error`, `last_ok` frozen so the page ages the numbers, one `store-error` event then one
   `recording` event on recovery), and anything that escapes the poll body is logged and the
   loop continues. Four dev-server sources once froze behind an unretrieved SQLite exception
   while `/api/health` went on calling them `ok` — a frozen source that looks healthy is the
   one failure mode worse than an outage.
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
   charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
   no outbound access — a CDN reference is a request that never comes back.
6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets`
   moves it into an `Authorization` header, so `Target.url` ends up clean). Nothing that leaves
   the process may carry one: `/api/config` serves `Target.display_url` and
   `config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message
   is redacted too — those strings go into the health record *and* the `event` table, which
   `/api/events` serves to the browser and `page-system.js` prints. No token, password, or
   personal hostname goes into tracked files, fixtures, or the browser.
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
   `sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
   2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
   `scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
   exporters) rather than trusting a README or a memory of an upstream.
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
   Check after touching dependencies: `uv pip list --outdated` must come back empty.
   Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
   `python3.14`) — do not lower these to make something else work.
9. **SQLite owns the history and must stay bounded.** Writes go through `store.py`; new
   metrics need a rollup path and a retention story, or a year of charts becomes a disk
   emergency. Put the DB on a volume in deployment.
10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`,
    http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot
    before they are called done. `data/logs/*.png` is where those land.

## Architecture

```
agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        ▼                                  ▼                        ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
        │  parse + Δcounter/Δt                                                     │
        ▼                                                                           ▼
sources/host.py · sources/llm.py · sources/nut.py                    poller.py (task per source)
        │        Result(payload, samples, events)                                    │
        ▼                                                                             ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/
```

Responsibilities, one line each:

- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
  `statvfs`; serves `GET /snapshot` + `GET /info` + `GET /health`. Refuses to bind
  off-loopback without a token (exit 2). Counters stay counters: the dashboard computes
  rates so the agent never remembers who polled it last.
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
  env var always wins. Defaults are safe except the sources, which default to unset.
- `poller.py` — job loop, health records, stale/last-good semantics, transition events. The
  success path records (payload → samples → events) and only *then* moves `last_ok`, so a
  refused SQLite write ages the source and alerts instead of leaving it current with charts
  that have silently stopped drawing.
  Events are stored under the **dashboard** name (`system`/`llm`/`ups`), not the sample
  source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
  (found on `/system` during phase 03 browser verification).
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping. Failure
  messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one
  that was dialled, because they become health records and event rows.
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
  (`gpu.busy_pct@<index>`).
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
  `VLLM_HISTOGRAMS`), measured Δ rates, prefill speed as
  `mean(prompt tokens)/mean(prefill time)`, session list from gufo `/slots`.
- `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not
  a mystery score), `COMMANDS` catalog with risk tiers, `COMMAND_ALIASES` + `resolve_command()`
  (drivers answer two names for the same action — `beeper.enable`/`beeper.on`,
  `beeper.disable`/`beeper.off` — so one entry is offered per action and any of its names hits
  that entry's verdict, never a gate the rendered name missed), `available_commands()` filtering,
  kelvin-vs-celsius normalisation, and `charger_status` normalised to the `CHARGER_STATUS`
  keys (the same lowercase token `states()` and `on_battery` already compare), and `states()`
  publishes one chip per fact — a driver exporting both `battery.charger.status` and the
  legacy flag that says the same word gets "charging, online", not "charging, online,
  charging". `battery_runtime`
  comes from `battery.runtime` **only** — `battery.runtime.low` is a shutdown threshold, not
  time left, so it lives at `battery_runtime_low` and never becomes the hero's "runtime left".
- `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a
  battery. It must never be the default, and every payload it produces says `simulated`.
  Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that
  ages), so a fresh simulator reads healthy and amber keeps meaning something. Its status
  follows the real drivers: a below-float pack on mains reports `OL CHRG`/`chg` (the glyph's
  bolt state is keyed on that token, and the model has to be able to produce it) and a battery
  being drawn from is `OB`/`dchg`, never `ol`. Every command it advertises is implemented and
  visible in a variable a real driver publishes — the beeper ones set `ups.beeper.status`
  (`enabled`/`muted`/`disabled`), and a test window that ends writes `ups.test.result: PASSED`
  while an aborted one writes nothing; the load/shutdown commands answer `OK` without modelling
  an outlet, because a faked power cut in the history is indistinguishable from a real one.
  `from_env()` is the single place the `NUT_SIMULATE` truthy rule lives — `app.py` builds the
  monitor through it, so the assembly and the model cannot disagree about what is enabled.
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
- `rates.py` — Δ/window estimator, keyed by resolved series name.
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
  `classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
  beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
  an unknown. The verdicts travel **additively** in payloads — `cpu.severity`,
  `memory.severity`, `disks.severity`, `gpus[].severity`, each UPS card's `severity`
  (charge/load/runtime), `severity` inside rated LLM cells (KV pressure, classified on a
  percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
  row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
- `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload,
  recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts`
  (`{id, severity, source, message, href, at}`, fail first then warn, then by source). A
  pure projection: no I/O, no store, no thresholds of its own — it reads job states for
  reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions.
  Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but
  older than 3× its own poll interval is `warn` while a source that stopped answering is
  `fail`; a dead source gets exactly one line (its stale payload adds nothing) that names it
  once — the poller's `"UpstreamError: "` wrapper and the duplicated host name are stripped,
  because a message that says `gateway` twice reads as two problems; `unknown` and missing
  numbers are silence; one alert per (source, metric) and at most 12, the last being a marker
  that says how many it hid. This is why the overview can be two panels.
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
  `/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
  the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `llm.engines`,
  `ups.ups`, the `states` maps and `recent_events` stay for programmatic consumers (live
  tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events` are unchanged
  and `alerts` is additive — `tests/test_api.py` pins every one of those key paths, with and
  without targets configured, so retiring a panel can never shrink the payload. A configured
  job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
  `system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
  counts the machines there are and `hosts_live` the ones answering.
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
  with an in-memory DB and no targets.
- `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the
  colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
  `ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
  `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
  `.hero` big number (plus an optional `.bar`) from that class — it never compares a number
  to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
  values the API does not rate, and **no page script calls it any more** — `/` was converted
  in phase 02, `/system` in phase 03, `/llm` in phase 04 and `/ups` in phase 05, so every
  colour on all four pages comes from a `severity` verdict the API sent
  (`tests/test_api.py` fails if a page script names it again). A value the API should rate
  gets a row in `severity.py`, never a bound in JS.

Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
series — `/api/metrics` answers with exactly these names.
That scheme is the history API's key — treat it as a public interface.

## The fleet (real environment, verified 2026-10)

| Host              | ssh alias          | What it gives                                                        |
| ----------------- | ------------------ | -------------------------------------------------------------------- |
| `driveripper`     | `driveripper-ops`  | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards |
| `deskwork`        | `deskwork-ops`     | Strix Halo (Radeon 8060S iGPU), gufo `:8731`                          |
| `gateway`         | `gateway-ops`      | router, no GPU                                                       |

- Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123`
  by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev.
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
  on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
  *Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
  | nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are
  exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`:
  `apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
  `ups.description`, the operator's label — the driver's own `identity.model` answers
  `GX1500U`, so the description is what identifies the box).
  Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date
  (`battery.date` is often an unset placeholder — `2001/09/25`, slashes and all, on the APC),
  CyberPower reports `battery.mfr.date: CPS` (not a date → `unknown`, never a fake age),
  **neither driver reports a battery temperature or the NUT 2.8 `battery.charger.status`**
  (so the battery glyph's state has to come from `on_battery`, and `charger_status` must stay
  optional), and `ups.test.result: No test initiated` is the factory state on both.
  Both UPSes are recorded verbatim (`LIST UPS`/`LIST VAR`/`LIST CMD`, serials scrubbed) as
  `tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, and the wire quotes a
  field only when it must — `VAR apc ups.status "OL"`, not three quoted fields.
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
  set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
  work that is still blocked).

## Commands

```bash
uv sync --python 3.14                     # deps + editable install
cp .env.example .env                      # production-shaped config
cp .env.dev.example .env.dev              # dev-only: ssh tunnels + live targets

./scripts/dev-tunnels.sh start            # agents (9121-9123) + DEV_FORWARDS (upsd 3494)
./scripts/dev-server.sh restart           # keep http://127.0.0.1:8090 up
./scripts/dev-server.sh status            # per-source ok/DOWN
./scripts/dev-server.sh logs

uv run pytest                             # hermetic: 543 passed + 8 live-skipped, no network
uv run pytest --cov                       # coverage gate (97.9%; floor 94%)
uv run pytest -m live --live              # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh             # re-record ground truth (agents + UPS captures)

bash .agents/validate.sh                  # the phase gate: the four checks above (pytest
                                          # --cov --cov-fail-under=94, ruff, pyright, plus a
                                          # node --check of web/static) + repo hygiene.
                                          # Always `uv run` — a bare `python3`/`ruff` on PATH
                                          # tests an interpreter nobody deploys.
```

Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
`--status`/`--uninstall`; it stores the token in a root-owned 0600 file and hands the agent
`--token-file`, never argv, and runs under `ProtectSystem=strict`).
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
service here.

## Testing policy

**Real captures for shape, fakes for behaviour, `live` for drift.**

- Anything that must match an upstream — `/proc` layouts, vLLM series names, NUT framing —
  is asserted against a verbatim capture in `tests/fixtures/`. If the schema drifts, a
  fixture test fails before a dashboard does.
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
  rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
- **Page payloads that a rewrite must not break are pinned as a table.**
  `tests/card_contract.py` holds the dotted `/api/system/snapshot` paths the `/system` hero
  cards read (`CARD_METRICS`, `GPU_METRICS`, `HOST_SHAPES`); `tests/test_system_contract.py`
  asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and
  `test_every_host_serves_the_hero_card_contract` asserts the same table against the live
  agents — so a rename fails a test, and a card that shows `n/a` cannot be "fixed" by
  inventing a field. `network` carries no `severity` on purpose (throughput has no good/bad),
  and the contract pins its absence. `tests/ups_contract.py` is the same device for the `/ups`
  hero row and battery glyph (`HERO_FIELDS`, `UPS_SHAPES`, `assert_ups_hero_contract`):
  `tests/test_ups_contract.py` drives it from `tests/fixtures/nut_<ups>.vars` — verbatim
  `LIST UPS`/`LIST VAR`/`LIST CMD` replies from the two real UPSes — and
  `test_ups_hero_fields_are_present_on_the_real_batteries` drives it against live upsd. The
  captures are what make the driver disagreements testable: one of them answers
  `battery.mfr.date: CPS`, neither exports `battery.temperature` or
  `battery.charger.status`.
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
  fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
  unset, so partial fleet availability is normal. **Live tests must not write** — a battery
  test on hardware something is plugged into is not a test.
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
  so it stays in the default run and is *not* marked `live`.
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
  94% (`--cov-fail-under`), and ~98% is where we are; keep it there. New/modified Python
  still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
  never a real target, and never a real upstream: `/api/ups/command`'s send is proven against
  a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live
  suite is read-only — that fake is the only place the write reaches a "driver".

## UI conventions

Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
`LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
silence, and the legend is a sibling that labels nothing) — `/system` does this for all three
slots, pinned by `tests/test_api.py`. Window chips (5m…30d on `/system` and `/llm`, 1h…90d on
`/ups`; the API accepts 5m…90d). A panel that cannot get data says
why in the panel — the operator should never have to open devtools to learn that a source is
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.

The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per
`/api/summary` alert: verdict class, source linked to its dashboard via the alert's own
`href`, message, age) and **Fleet**. A non-empty list floats the alerts panel above the
fleet table (`#overview-panels[data-state]` swaps them); an empty one is an explicit
"all clear" state below it, because an empty list is a message, not a blank box. A host that
is not answering keeps its row, dimmed with the shared `.stale` class and reading `n/a`, so
the table never implies last good numbers are live and never silently loses a machine.
No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the
two-panel shape against the served HTML.

Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an
`id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for`
attribute references — the orphan a deleted renderer leaves behind, caught on served bytes
because there is no JS test runner (a locked anchor). Same rule for `dash.css`: delete the
rules a removed panel made dead (`.grid.wide` went with the overview's fleet+events row);
component vocabulary the pages are still being converted to (`.hero-bar`, the grid and bar
variants) stays even while only one page uses it.

A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's
`severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*`
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare.

A cell-scale trend is `DashCharts.sparkline()` — the `llm_monitor` `drawSpark` idiom, so a
percent metric is plotted against `fixedMax` (**never** autoscaled: an 8 % KV window that once
held a 92 % spike must not read flat, and the axis gets top/bottom scale marks so "a third of
the way up" says a third of what). `align: 'right'` pins the newest sample right and grows the
line leftwards inside a real `span` in seconds — x stays time, not sample index. Stroke colour
comes from the payload `severity` via the `--ok`/`--warn`/`--fail` tokens only; an unrated
series draws a faint `--text-faint` stroke with no fill, because a shape is not a verdict.
Below `minPoints` (2) it prints `no history` rather than drawing a confident line through one
point, `labels: {current, max}` renders the `now / max` foot through the caller's `format`, and
there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the
other half of the same honesty for count metrics: history arrives as rollups, so eight seconds
of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for
a queue nobody was in — `/llm` passes `minSpan: 1` on sessions and queued so its axis is never
finer than one request.

A battery is `DashCharts.battery(container, {charge, severity, state, label})` — one inline SVG
whose **fill *is* the charge**: 1 viewBox unit per percent, clamped to 0–100 and never rounded up
(a 2 % cell must look nearly empty, because a reassuring battery above a flat one is a lie),
clipped to the plate so 0 % leaves an empty shell rather than a stray pixel. Colour comes only from
the payload `severity` token; a measured charge with no verdict draws faint grey at half strength
(`sparkline`'s rule — a shape is not a verdict) and never an empty shell ("0 %" and "not rated" are
different sentences). Every state has a **shape** as well as a colour, so the glyph survives
monochrome: solid shell `online`, bolt `charging`, dashed outline `onbattery`, no fill with a dash
pattern *through* the body `unknown`; a state word it does not know degrades to `unknown`, never to
a confident "on line". Accessible **by default** — `role="img"` + `<title>` + `aria-label` built
from the inputs ("apc: Charge 100 percent, on line"), because the glyph carries information the
markup does not say in words — and a page that prints that sentence next to it passes
`decorative: true` to make the `<svg>` `aria-hidden` rather than say it twice. No animation and no
`ResizeObserver`: one `viewBox` + `preserveAspectRatio` scaled by `.battery` in `dash.css`, so the
hero row gets a big cell on a desktop and a small one on a phone.

**The `/ups` card is led by the glyph and three heroes** (phase 05): the battery on the left, then
**Charge %**, **Load %** and **Runtime left** on `.heroes` at the shared 2.4rem, coloured from
`severity.charge_pct` / `severity.load_pct` / `severity.battery_runtime` and nothing else. Charge
carries **no** `.bar` — the glyph beside it is that value drawn as a shape, and one value drawn
twice in one row is the pattern this app avoids (Load keeps its bar; nothing else draws it). Its
`.sub` lines are the context each number needs and nothing more: `24 V nominal · 27.3 V` for the
pack, `128 W of 900 W` when the driver publishes both `realpower` figures (a wattage inferred from
a percent against another model's rating would be an invented number), `at the current load` for
runtime. Runtime prose is `runtimeText()`'s (`1h 32m`, `10m 21s`) and an estimate the driver never
made goes into `Dash.hero()` as `null`, so it prints faint and uncoloured rather than as `0m`. The
readings the hero row displaced — line in, output, battery volts and amps, battery °C — stay on the
card, demoted to `.ups-electrical`: muted, `.95rem`, never folded away, because they are what you
reach for *while* the UPS is on battery, and battery °C reads `n/a` on both real drivers here.
`state` is derived in the page from `on_battery` → `charger_status` (`chg` charges; `dchg`/`rec` are
the server's own on-battery tokens; `full` is **on line, not charging** — a rested floating battery
reports it) with one fallback: a state chip the API rated `ok`, because `nut.states()` reserves
`ok` for on-line/fully-charged and neither real driver here publishes `battery.charger.status` at
all. `ups.status` itself is never re-parsed in JS (`tests/test_ups_contract.py`). `/ups` passes
`decorative: true`, because the charge hero and the state chips already say the sentence. The glyph
repaints only when charge / verdict / state change (`drawGlyph()` keys a `data-drawn` triple): the
charge float jitters every `NUT_INTERVAL_SECONDS`, and rebuilding the `<svg>` on that re-announces
an accessible name for a difference no eye can see.

An event row names its subject **once**. Every feed (`/system`, `/llm`, `/ups`) renders the
message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a
*leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py`
messages start with their own label, and each row already prints that label as a chip, so
"gufo gufo is answering again" reads as two problems. A name in the middle of a message stays:
the UPS rows used to `replace()` it anywhere and mangled "Quick battery test sent to apc: OK".
This is presentation only — the stored event and `/api/events` keep every word, and the raw
error string still surfaces in `/api/health`.

The `/system` host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the
`.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script).
Colour comes from the payload's `severity` dicts and nowhere else; a missing number or
verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**. NET is
one slot with two lines (`.hero-lines`, `↓` in / `↑` out) and is deliberately colourless:
throughput has no good/bad, so `network` carries no severity. The GPU slot holds one hero
per card (model, VRAM used/total and the honest APU `pool` view in the `.sub`), and
everything not glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU
power/temp, filesystems, processes — folds into a collapsed "More detail" `<details>`: a
simpler surface, not a less complete page.

A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
those (a range when the hosts disagree) instead of the poll interval, which only coincides with
them at the defaults. `/llm`'s hero `.sub` prints the rate cell's own `window` through the same
`provenance()` tag, and the hero's `title` carries the exact `source` string — "78 tok/s" must
always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_monitor`'s
`setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification).

**The `/llm` page reads per engine** (phase 04, `NOTES-chart-set.md`): a row of two
`Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**, because 0 tok/s
on an idle server is healthy and `severity.py` has no throughput row), then a compact `.cells`
list — label left, tabular value right, a `sparkline()` fixed at 160px so rows line up, and a
mono `p95 …` foot under every latency mean (no hover; p95 needs no stored series). Rows are
engine-scoped by the `only` flag: gufo never grows a TTFT mystery, vLLM never a spec-accept
hole. `preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded
threshold); lifetime token counters are a quiet footnote line. **One `/api/llm/history` request
per refresh** carries every chart *and* spark series (`HISTORY_SERIES`; the endpoint answers
unknown names as empty series on purpose), and the two series that addition required
(`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) are stored additively in
`sources/llm.py` HISTORY_METRICS. The engine-totals card kept only engines-live/active/queued
and **hides below two engines** — a cross-engine sum of one engine duplicates that engine's own
panel, and the summed-KV bar went because `fleet_totals` carries no `severity` to render. The
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
sit at their own height (`#engines` carries `.items-start`): a 5-row gufo panel next to a 7-row
vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data".

**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
panel and the host/metric detail picker are deleted, along with the `#chart-*` / `#detail-*` ids and
the `select` / `label.field` styling only those pickers used. Each panel stacks its own
`.host-charts` — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where
the GPU slot is created and destroyed with `host.gpus`, so gateway shows neither a phantom card
nor a hole. **One `/api/system/history` request per host per refresh**
(`hosts=<that host>&metrics=<that panel's series>&range=…`, pinned by
`test_every_system_chart_is_drawn_inside_its_own_host_panel`): unknown metrics come back as
empty series on purpose, so a panel asks for every series it could draw and asks once. A legend
here names metrics (or one GPU's model), never hosts — the panel heading already says who this
is, and a legend that grew a second host is the bug this replaced. A host that stops reporting
loses its panel *and* its charts (`pruneHosts`), never a chart of nothing. Per-core visuals are
gone from `/system` and `renderCores` is **deleted** from `charts.js` with its last caller.
`drawHostChart()` shades a chart only when one series has points — two or three filled areas
stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy
memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryable from
`/api/system/history` — removing the plot is not removing the data.

## Known gaps / next work

1. **UPS commands work, and the account can now cut power — the app's gates are the control.**
   Three defects were cleared on 2026-10-17, each verified against real hardware. (a) `NutClient.login()` sent `USER`/`PASS`, which the deployed
   upsd 2.8.5 does not implement — each answered `ERR UNKNOWN-COMMAND`, so `_read_line()`
   raised and `POST /api/ups/command` could only 502 before `instcmd()`, credentials or not.
   It now sends `USERNAME`/`PASSWORD`, the pair the server's own `HELP` advertises
   (`HELP VER PROTVER GET LIST SET INSTCMD LOGIN LOGOUT USERNAME PASSWORD STARTTLS`), pinned
   by `test_login_uses_the_verbs_upsd_2_8_5_answers_to` — the fake upsd has no `USER`/`PASS`
   branch, so a regression answers `ERR UNKNOWN-COMMAND` exactly like the real thing.
   (b) 12 of the 23 commands the two real UPSes advertise had **no `COMMANDS` entry**, and
   `available_commands()` drops anything the catalog misses — the page silently offered less
   than the hardware does. Every advertised command is catalogued now, each with a risk tier:
   `test_every_command_the_hardware_advertises_is_catalogued` fails if a capture gains a
   command the catalog does not know, and
   `test_every_command_that_can_cut_power_is_in_the_dangerous_tier` keeps `load.off`,
   `driver.killpower` and the `shutdown.*` sequence in the tier that types `DANGER` and is
   refused on battery. `shutdown.stop` is the exception the name hides — it *cancels* a
   shutdown, so it is `safe`, and the test says so in words.
   (c) `[dashboards]` in `/etc/ups/upsd.users` is `instcmds = all` at the operator's request,
   so this app may send every advertised command on both UPSes, power-cutting ones included.
   `.env.dev` (mode `0600`, gitignored) carries the credentials.
   What remains to know:
   - **Proven live, and reversibly.** `beeper.enable` returned `OK` and `ups.beeper.status`
     read `enabled` on the APC afterwards; `driver.killpower` without the word `DANGER` still
     answers 422 and never reaches the wire. A battery test has deliberately not been run —
     it puts real load on a battery things are plugged into, which stays the operator's call.
   - **`enabled: true` cannot mean "this account may send it".** `available_commands()`
     filters on `LIST CMD`, which upsd serves **unfiltered**, and NUT has no call for a
     user's privileges — so the wire cannot tell the page what the account is granted. With
     `instcmds = all` the two agree by configuration rather than by check, which is precisely
     why the risk tiers and the typed `DANGER` are load-bearing and not decoration.
   - **Pre-existing upsd log noise:** every upsd start logs `Unknown upsmon type =` (20
     occurrences over 12 starts, oldest before this work). Harmless — `upsmon` logs into both
     UPSes right after it — but unexplained.
   Editing `upsd.users` at all: never grant `actions = -`, and never write it blind — the
   running `upsmon` authenticates as its own account from that same file for both UPSes, so
   dropping it means no orderly power-down. Copy the untouched sections from the live bytes
   and `cmp` them before writing; `tee` truncates in place, so owner and mode survive.
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
   (`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
   fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
   all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the
   field map is written.
3. **No remote yet.** No `origin` is configured (`git remote -v` is empty) — the user is
   providing it, so every phase so far is local-only history.
4. **UPS history depth.** Events and the numeric series are stored; long-window battery
   health (capacity fade over months) has the rollups but no dedicated view.

## Locked decisions (and why)

- **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10s resolution, one file the
  container owns. Rollups keep a year of charts at a few hundred MB.
- **Our own stdlib host agent, not node_exporter.** One JSON snapshot with exactly the
  fields the page needs (GPU model strings, per-mount usage, temperature labels) versus
  ~1500 series and a re-labeling layer; no dependency to install, `ProtectSystem=strict`
  friendly, and read-only by construction. `scripts/capture-node-exporter.sh` remains to
  compare ground truth if that is ever revisited.
- **Our own NUT client, not `pynut`/`upsc`.** The protocol is ~10 line commands, the
  third-party libs are subprocess-based or stalled, and we need instant commands with
  explicit gating.
- **In-house Prometheus parser.** Only two upstreams use the format; keeps `prometheus_client`
  out of the dependency set.
- **Port 8090**, inherited from the `llm_monitor` prototype this started from: 8000 is vLLM
  and 8731 is gufo in this fleet.
- **`.env` is a seed, never an override** — container/systemd env must win.

## Definition of done (per change)

The first three are exactly what `bash .agents/validate.sh` (the phase gate) checks, so no
phase may be called done on a subset of them.

`uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated`
empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted
in a browser against **live** data · fixtures re-captured if an upstream contract changed ·
this file and `README.md` updated when a rule, command, gap, or fleet fact changed.