ducoterra
fefbeb8227
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**
- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
864 lines
68 KiB
Markdown
864 lines
68 KiB
Markdown
# AGENTS.md — homelab dashboards
|
||
|
||
The plan, the rules, and the environment facts an agent needs to work on this repo without
|
||
re-deriving them. `README.md` is the operator-facing how-to; this file is the why, the
|
||
invariants, and the state of the fleet.
|
||
|
||
## Working protocol
|
||
|
||
This project runs the `.agents/phases/` phased-execution pipeline.
|
||
|
||
1. Always read `.agents/PLAN.md` first to understand the project context and goals.
|
||
2. Follow the phased execution protocol in `.agents/phases/`.
|
||
3. Never modify `.agents/PLAN.md` or any files in `.agents/phases/complete/`.
|
||
4. If you need to update any file in `.agents/phases/todo/`, you must ask the user for permission first.
|
||
5. Strictly adhere to the **LOCKED DECISIONS** listed in `.agents/PLAN.md`.
|
||
|
||
Everything below this section is the project's own content: what the app is, the rules that
|
||
keep its numbers honest, and the state of the fleet it watches.
|
||
|
||
## What this is
|
||
|
||
Four dashboards for one homelab, behind one domain, in one process, backed by one SQLite
|
||
file — plus an overview page that says which of the four you need:
|
||
|
||
| Route | Question it answers | Source |
|
||
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
|
||
| `/` | Is anything wrong, and where do I look first? | all four, in one request |
|
||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host |
|
||
| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) |
|
||
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
|
||
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
|
||
|
||
History is a first-class feature, not a sparkline afterthought: every number worth graphing
|
||
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
|
||
|
||
## Non-negotiable rules
|
||
|
||
These are the things that make the numbers trustworthy. Breaking one is a bug even if the
|
||
page still renders.
|
||
|
||
1. **Read-only by construction, with exactly one exception.** Nothing in this app writes to
|
||
an upstream except `POST /api/ups/command`. That endpoint is gated four times
|
||
(`NUT_ALLOW_ACTIONS`, not `DASH_READ_ONLY`, the upsd.users account actually having the
|
||
command, and a UI confirmation — `DANGER` for anything that can cut power) and the server
|
||
re-checks every gate on every request. Hiding a button is not a safety control.
|
||
Commands that would cut output are refused outright while the UPS is already on battery.
|
||
2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never
|
||
as `0`. Every derived value carries the `source` string it came from and `derived: true`
|
||
so the UI can print *how* it was obtained (`computed:Δtokens/Δt`, `Δ10.0s`). Aggregating
|
||
two different series into one rate is the failure mode this rule exists to catch — the
|
||
rate estimator is keyed by the *resolved* upstream series name for exactly that reason.
|
||
3. **Unconfigured is not broken.** A source with no env config answers
|
||
`404 {"configured": false}` **without dialling**, and the page says "not configured".
|
||
A red panel for a server the operator never installed destroys the signal from the
|
||
panels that matter.
|
||
4. **One dead source must not blink another — and nothing may stop polling in silence.** Each
|
||
source is its own asyncio task with its own timeout and health record (`poller.py`). A
|
||
failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it —
|
||
on `/` too, where a host that has never answered still gets a fleet row
|
||
(`unreachable: true`, dimmed, every number `n/a`) because dropping the row would turn a dead
|
||
agent into "never configured". Transitions write events; repeated failure writes one line,
|
||
not 500. **The task is the source's only poller, so no exception may end it**: a store that
|
||
refuses an otherwise good poll is a *reported* failure (`ok: false`, the real reason in
|
||
`error`, `last_ok` frozen so the page ages the numbers, one `store-error` event then one
|
||
`recording` event on recovery), and anything that escapes the poll body is logged and the
|
||
loop continues. Four dev-server sources once froze behind an unretrieved SQLite exception
|
||
while `/api/health` went on calling them `ok` — a frozen source that looks healthy is the
|
||
one failure mode worse than an outage.
|
||
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
|
||
charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
|
||
no outbound access — a CDN reference is a request that never comes back.
|
||
6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets`
|
||
moves it into an `Authorization` header, so `Target.url` ends up clean). Nothing that leaves
|
||
the process may carry one: `/api/config` serves `Target.display_url` and
|
||
`config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message
|
||
is redacted too — those strings go into the health record *and* the `event` table, which
|
||
`/api/events` serves to the browser and `page-system.js` prints. No token, password, or
|
||
personal hostname goes into tracked files, fixtures, or the browser.
|
||
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
|
||
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
|
||
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
|
||
`scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
|
||
exporters) rather than trusting a README or a memory of an upstream.
|
||
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
|
||
Check after touching dependencies: `uv pip list --outdated` must come back empty.
|
||
Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
|
||
`python3.14`) — do not lower these to make something else work.
|
||
9. **SQLite owns the history and must stay bounded.** Writes go through `store.py`; new
|
||
metrics need a rollup path and a retention story, or a year of charts becomes a disk
|
||
emergency. Put the DB on a volume in deployment.
|
||
10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`,
|
||
http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot
|
||
before they are called done. `data/logs/*.png` is where those land.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
|
||
│ │ + /slots (gufo) │
|
||
▼ ▼ ▼
|
||
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
|
||
│ parse + Δcounter/Δt │
|
||
▼ ▼
|
||
sources/host.py · disks.py · llm.py · nut.py poller.py (task per source)
|
||
│ Result(payload, samples, events) │
|
||
▼ ▼
|
||
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/
|
||
```
|
||
|
||
Responsibilities, one line each:
|
||
|
||
- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
|
||
`statvfs`; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an
|
||
HTML status page at `GET /`. Refuses to bind off-loopback without a token (exit 2).
|
||
Counters stay counters: the dashboard computes
|
||
rates so the agent never remembers who polled it last.
|
||
SMART (phase 09): `smart()` runs `smartctl -i -A -H` per whole `/sys/block` disk
|
||
(`parse_smart_output` covers ATA + NVMe; NVMe media errors map to
|
||
`offline_uncorrectable`, hex-packed vendor raws are never decoded, and an attribute row is
|
||
read from its **RAW_VALUE column** rather than the end of the line because this binary
|
||
annotates that column — `35 (Min/Max 34/44)`, `43894h+43m+32.519s` — so its last token is
|
||
not the value), 5 s per device /
|
||
20 s per pass / 240 s cache, attached as `disk_io.devices[].smart`. Missing binary →
|
||
**one** `errors` line for the pass (the word `not found` in it is what
|
||
`sources/disks.py` reads to answer "is smartctl installed", and
|
||
`tests/test_agent.py` pins that the line still says it); one denied/wedged device → that
|
||
disk's `smart` is `null` with its own line; `--no-smart` opts out silently. Never fails
|
||
the snapshot. **An expired cache refreshes on a background thread while the request is
|
||
served the previous result** — only the first pass after startup runs inline — because 20 s
|
||
of wedged disk on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the
|
||
*system* source red every four minutes over an add-on that page never draws. The refresher
|
||
is single-flight and keeps the old cache on failure, so a bad pass retries on the next
|
||
expiry instead of hammering the disk.
|
||
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
|
||
env var always wins. Defaults are safe except the sources, which default to unset.
|
||
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
|
||
`/disks` is a **source of its own** (`disks:<host>`, `DISKS_INTERVAL_SECONDS` 300 s,
|
||
`DISKS_TIMEOUT_SECONDS` 30 s, events under the dashboard name `disks`) reading the same
|
||
agents the `system` jobs read: the separate task is what keeps a SMART poll out of the
|
||
10-second utilisation loop and lets `/disks` age on its own instead of blinking `/system`.
|
||
The job list is built once (`app.build_hub()`) because it is wired twice — production and
|
||
the injected-store test path — and two hand-written copies drifting is how a source ends up
|
||
polling in tests and not in the container. The
|
||
success path records (payload → samples → events) and only *then* moves `last_ok`, so a
|
||
refused SQLite write ages the source and alerts instead of leaving it current with charts
|
||
that have silently stopped drawing.
|
||
Events are stored under the **dashboard** name (`system`/`disks`/`llm`/`ups`), not the sample
|
||
source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
|
||
(found on `/system` during phase 03 browser verification).
|
||
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping. Failure
|
||
messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one
|
||
that was dialled, because they become health records and event rows. `fetch_snapshot()` is
|
||
the one agent `/snapshot` read that `system` and `disks` share, so a dead host says the
|
||
same sentence on both pages and only the `timeout` differs (the disks poll has to outlast
|
||
the agent's 20 s smartctl pass, and `None` is translated into httpx's sentinel because a
|
||
literal `None` means *no timeout at all*). Agent `errors`
|
||
become the payload's `degraded` **except** the `smart:` add-on lines — a host without
|
||
smartmontools or device read access is unconfigured, not broken (invariant 3); they stay
|
||
in `agent.errors` for the `/disks` page to explain.
|
||
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
|
||
(`gpu.busy_pct@<index>`).
|
||
- `sources/disks.py` — the same snapshot projected onto **physical** drives for `/disks`
|
||
(phase 09): one row per whole-`/sys/block` disk (never a partition, never a mount), the
|
||
agent's `smart` block carried through verbatim, `health` the drive's own word normalised to
|
||
`ok`/`failed`/`None`, and `smartctl_available` three-way on purpose — `True` (a disk
|
||
answered), `False` (the agent said the binary is not installed, so the page can say *install
|
||
smartmontools*), `None` (an agent predating the pass, or `--no-smart`: nobody knows, so
|
||
`n/a` plus the agent's own `smart_errors` line and never a guessed verdict). **Nothing here
|
||
is a rate**: power-on hours and the sector counters are stored counters-as-reported, so the
|
||
source takes no `RateEstimator` and its payload carries no `window_seconds` (rule 2 — a rate
|
||
with no estimator, no `Δ<window>` label and no `source` string is an invented number).
|
||
`disk.health_ok@dev` is not a series either: a verdict is not a number, so the drive's word
|
||
travels in the payload only. Each row does carry a `severity` dict (`SEVERITY_FIELDS` = the
|
||
two rated numbers, `temp_c`/`percentage_used`, plus `health` through `HEALTH_SEVERITY`,
|
||
which translates the drive's own word instead of grading it), and `UNGRADED_FIELDS` names
|
||
the numbers that must never be graded. `fleet_disk_summary()` sums the per-host `summary`
|
||
counts for the page header and drops a host with no payload from the sums while still
|
||
counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does
|
||
arithmetic and a dead agent never shrinks a fleet total.
|
||
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
|
||
`VLLM_HISTOGRAMS`), measured Δ rates, prefill speed as
|
||
`mean(prompt tokens)/mean(prefill time)`, session list from gufo `/slots`.
|
||
- `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not
|
||
a mystery score), `COMMANDS` catalog with risk tiers, `COMMAND_ALIASES` + `resolve_command()`
|
||
(drivers answer two names for the same action — `beeper.enable`/`beeper.on`,
|
||
`beeper.disable`/`beeper.off` — so one entry is offered per action and any of its names hits
|
||
that entry's verdict, never a gate the rendered name missed), `available_commands()` filtering,
|
||
kelvin-vs-celsius normalisation, and `charger_status` normalised to the `CHARGER_STATUS`
|
||
keys (the same lowercase token `states()` and `on_battery` already compare), and `states()`
|
||
publishes one chip per fact — a driver exporting both `battery.charger.status` and the
|
||
legacy flag that says the same word gets "charging, online", not "charging, online,
|
||
charging". `battery_runtime`
|
||
comes from `battery.runtime` **only** — `battery.runtime.low` is a shutdown threshold, not
|
||
time left, so it lives at `battery_runtime_low` and never becomes the hero's "runtime left".
|
||
- `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a
|
||
battery. It must never be the default, and every payload it produces says `simulated`.
|
||
Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that
|
||
ages), so a fresh simulator reads healthy and amber keeps meaning something. Its status
|
||
follows the real drivers: a below-float pack on mains reports `OL CHRG`/`chg` (the glyph's
|
||
bolt state is keyed on that token, and the model has to be able to produce it) and a battery
|
||
being drawn from is `OB`/`dchg`, never `ol`. Every command it advertises is implemented and
|
||
visible in a variable a real driver publishes — the beeper ones set `ups.beeper.status`
|
||
(`enabled`/`muted`/`disabled`), and a test window that ends writes `ups.test.result: PASSED`
|
||
while an aborted one writes nothing; the load/shutdown commands answer `OK` without modelling
|
||
an outlet, because a faked power cut in the history is indistinguishable from a real one.
|
||
`from_env()` is the single place the `NUT_SIMULATE` truthy rule lives — `app.py` builds the
|
||
monitor through it, so the assembly and the model cannot disagree about what is enabled.
|
||
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
|
||
- `rates.py` — Δ/window estimator, keyed by resolved series name.
|
||
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
|
||
`classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
|
||
beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
|
||
an unknown. The verdicts travel **additively** in payloads — `cpu.severity`,
|
||
`memory.severity`, `disks.severity`, each UPS card's `severity` (charge/load/runtime),
|
||
`severity` inside rated LLM cells (KV pressure, classified on a
|
||
percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
|
||
row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
|
||
There is **no `gpu.*` row and no `gpus[].severity`** (phase 06): these are LLM servers,
|
||
a busy GPU is the job at every level, so the GPU card and the fleet GPU column render
|
||
uncoloured — like `net.*`, whose colourlessness predates it. Removing the verdict is not
|
||
removing the data: `gpu.busy_pct@N` / `gpu.mem_used_pct@N` are still stored and charted.
|
||
The `/disks` rows (phase 09) are `disk.temp_c` (50/70) and `disk.percentage_used` (80/100)
|
||
— an SSD at 100 % of its rated wear is a replacement, not a warning — attached per row by
|
||
`sources/disks.py`; the sector counters and `disk.power_on_hours` deliberately have **no
|
||
row**, because a non-zero counter is an alarm chip on presence and zero is not an
|
||
achievement (the `/llm` preemption-chip rule), so `classify("disk.reallocated", 1)` is
|
||
`None` and the page shows a chip instead of a colour.
|
||
- `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload,
|
||
recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts`
|
||
(`{id, severity, source, message, href, at}`, fail first then warn, then by source). A
|
||
pure projection: no I/O, no store, no thresholds of its own — it reads job states for
|
||
reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions.
|
||
Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but
|
||
older than 3× its own poll interval is `warn` while a source that stopped answering is
|
||
`fail` (the interval is the job's own dashboard's, so `disks` goes amber after 15 quiet
|
||
minutes rather than 30 seconds); a dead source gets exactly one line (its stale payload
|
||
adds nothing) that names it once — the poller's `"UpstreamError: "` wrapper and the
|
||
duplicated host name are stripped, because a message that says `gateway` twice reads as two
|
||
problems, and the same rule silences the **`disks` echo** of a dead agent (the disk job
|
||
dials the same `/snapshot`, so its failure carries no new information — while a `disks` job
|
||
failing *while its host answers*, SMART wedged inside the handler, is the only voice that
|
||
fault has); `unknown` and missing numbers are silence; one alert per (source, metric) and at most 12, the last being a marker
|
||
that says how many it hid. This is why the overview can be two panels. There is **no
|
||
disk-health rule and none is wanted**: a drive the hardware itself called `FAILED` is a red
|
||
chip on its `/disks` card and a line in that page's event feed, not an overview alert —
|
||
`severity.py` rates no sector counter and this module may not have a threshold of its own, so
|
||
grading a drive here would be the bug, not the fix (phase 09). Its `HREF` table does carry
|
||
`/disks`, so the one alert a disk job *can* raise — SMART failing while its host answers —
|
||
links to the page that names the drive.
|
||
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
|
||
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
|
||
`/api/disks/*` (phase 09) is the drive-health pair: `/snapshot` answers one entry per
|
||
configured host (disk rows + `smart` + `smartctl_available` + per-row `severity` + a
|
||
`fleet` block summed by the API, with a configured-but-never-answered host as an
|
||
`unreachable` stub whose counts are `None`, not `0`), and `/history` serves the
|
||
`disk.*@dev` series with unknown names answering as empty series like the other two
|
||
history endpoints. The configured gate runs **before** anything that could reach the
|
||
network — `HOST_TARGETS` empty is `404 {"configured": false}` with no client, no fetch and
|
||
no `disks` job in the hub at all (invariant 3), and `tests/test_api.py` proves it with
|
||
spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot`. The new surface grows no
|
||
write of its own — a fourth dashboard must not quietly become a second one, so the app's
|
||
only POST stays `/api/ups/command` (invariant 1, pinned by enumerating every served route's
|
||
verbs).
|
||
`/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
|
||
the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `disks.hosts`,
|
||
`llm.engines`, `ups.ups`, the `states` maps and `recent_events` stay for programmatic
|
||
consumers (live tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events`
|
||
are unchanged and `alerts` (phase 02) and `disks` (phase 09) are additive — the `disks`
|
||
block is `fleet_disk_summary` verbatim plus one row per host (`disks`/`with_smart`/
|
||
`health_failed`/`alarm_disks`/`smartctl_available`/`failed_devices`), so a notifier learns
|
||
*which* drive the hardware called `FAILED` without a second call, and
|
||
`tests/test_api.py` pins every one of those key paths, with and without targets configured,
|
||
so retiring a panel can never shrink the payload. A configured
|
||
job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
|
||
`system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
|
||
counts the machines there are and `hosts_live` the ones answering.
|
||
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
|
||
with an in-memory DB and no targets.
|
||
- `web/` — five static pages (`/`, `/system`, `/disks`, `/llm`, `/ups`), one stylesheet,
|
||
`charts.js`. No framework. `app.js`'s masthead status dots are one per dashboard job —
|
||
including `disks`, because the same agent can answer the 10-second utilisation poll and
|
||
have a wedged SMART pass, and one dot for both would be right about neither. `app.js` owns the
|
||
colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
|
||
`ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
|
||
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
|
||
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
|
||
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
|
||
values the API does not rate, and **no page script calls it any more** — `/` was converted
|
||
in phase 02, `/system` in phase 03, `/llm` in phase 04, `/ups` in phase 05 and `/disks`
|
||
shipped that way from its first commit, so every colour on all five pages comes from a
|
||
`severity` verdict the API sent
|
||
(`tests/test_api.py` fails if a page script names it again). A value the API should rate
|
||
gets a row in `severity.py`, never a bound in JS.
|
||
|
||
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
|
||
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
|
||
series — `/api/metrics` answers with exactly these names. The `sample.source` column is the
|
||
dashboard family — `host`, `llm`, `ups`, and `disks` for the SMART series
|
||
(`disk.temp_c@nvme0n1`, `disk.power_on_hours@…`, `disk.reallocated@…`, `disk.pending@…`,
|
||
`disk.uncorrectable@…`, `disk.percentage_used@…`). Those keep their own family instead of
|
||
sitting beside `host`'s `disk.used_pct@/mount` because the cadences differ thirty-fold and
|
||
`GET /api/metrics?source=disks` has to answer "what does *this page* have history for".
|
||
That scheme is the history API's key — treat it as a public interface.
|
||
|
||
## The fleet (real environment, verified 2026-10)
|
||
|
||
| Host | ssh alias | What it gives |
|
||
| ----------------- | ------------------ | -------------------------------------------------------------------- |
|
||
| `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards |
|
||
| `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` |
|
||
| `gateway` | `gateway-ops` | router, no GPU |
|
||
|
||
- Agents run from `~/lab-agent/host_agent.py` on each host, forwarded to `127.0.0.1:9121/9122/9123`
|
||
by `scripts/dev-tunnels.sh` (`DEV_TUNNELS`). Nothing listens on the hosts in dev.
|
||
- Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`.
|
||
- NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is
|
||
on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports
|
||
*Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
|
||
| nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are
|
||
exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`:
|
||
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
|
||
`ups.description`, the operator's label — the driver's own `identity.model` answers
|
||
`GX1500U`, so the description is what identifies the box).
|
||
Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date
|
||
(`battery.date` is often an unset placeholder — `2001/09/25`, slashes and all, on the APC),
|
||
CyberPower reports `battery.mfr.date: CPS` (not a date → `unknown`, never a fake age),
|
||
**neither driver reports a battery temperature or the NUT 2.8 `battery.charger.status`**
|
||
(so the battery glyph's state has to come from `on_battery`, and `charger_status` must stay
|
||
optional), and `ups.test.result: No test initiated` is the factory state on both.
|
||
Both UPSes are recorded verbatim (`LIST UPS`/`LIST VAR`/`LIST CMD`, serials scrubbed) as
|
||
`tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, and the wire quotes a
|
||
field only when it must — `VAR apc ups.status "OL"`, not three quoted fields.
|
||
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
|
||
set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
|
||
work that is still blocked).
|
||
- **SMART is captured from real hardware; the live agents still cannot read it.** `smartctl`
|
||
7.5 is installed on deskwork and driveripper. The `ops` account (which runs the dev agents)
|
||
is in no `disk` group; driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and
|
||
deskwork grants nothing, so the agent's own unprivileged `smartctl -i -A -H` — it shells the
|
||
binary and never calls sudo — answers *Permission denied* (exit 2) on every host and `smart`
|
||
stays `null` end to end: the designed degradation, not a bug. The three deployed agents also
|
||
predate the SMART pass, so the `disks` source today yields one row per physical disk (16 on
|
||
driveripper, 1 on deskwork, `vda` on gateway) with `smart: null`, `smart_errors: []` and
|
||
`smartctl_available: null` — "nobody told us either way", which is why that flag is
|
||
three-way and not a boolean.
|
||
**Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
|
||
drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
|
||
every one — which is what found the parser's one real bug: this binary annotates the
|
||
attribute table's RAW_VALUE column (`35 (Min/Max 34/44)` on every Seagate, and one Western
|
||
Digital writes its hours as `43894h+43m+32.519s`), so reading the *end* of the row lost the
|
||
temperature of all eight HDDs and one power-on count. The captures are also the first
|
||
honest look at this fleet's drives: `nvme5n1` reports 13 media-and-data-integrity errors and
|
||
`nvme1n1` is at 58 % of its rated wear, so the alarm chip and the amber wear bar on `/disks`
|
||
are real features with real data behind them, not demo furniture.
|
||
**Lighting up the live page is two operator actions that travel together:** give the agent
|
||
user device read access — the `disk` group (`sudo usermod -aG disk <agent-user>`, which is
|
||
what `deploy/install-agent.sh` does on install), *not* a sudo rule, because the agent must
|
||
read the device itself — and redeploy `agent/host_agent.py` (never redeploy without the
|
||
grant: 16 permission-denied lines would age nothing but are pure noise, while with the grant
|
||
they are 16 drives' worth of temperature, hours and sector counters). Until then `/disks`
|
||
renders the designed degradation: one dimmed card per physical drive, every number `n/a`, and
|
||
the one-line why — which is also what gateway shows permanently, being a VM with no physical
|
||
disk to read and no smartmontools at all.
|
||
`tests/fixtures/smartctl_reference_*.txt` stay as what they are: **reference shapes** (labels
|
||
verbatim from the deployed binary, values synthetic), kept beside the real captures so the
|
||
synthetic edge cases stay pinned. Re-record the real ones with
|
||
`SMART_HOSTS="driveripper-ops" ./scripts/capture-fixtures.sh`: it tries plain `smartctl`
|
||
first (so a fixture can never read healthier than the agent does), falls back to `sudo -n`,
|
||
refuses to overwrite good fixtures with unreadable output, and scrubs
|
||
Serial/WWN/EUI-64 (`test_smartctl_captures_carry_no_unscrubbed_serials` fails the day it
|
||
stops).
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
uv sync --python 3.14 # deps + editable install
|
||
cp .env.example .env # production-shaped config
|
||
cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
|
||
|
||
./scripts/dev-tunnels.sh start # agents (9121-9123) + DEV_FORWARDS (upsd 3494)
|
||
./scripts/dev-server.sh restart # keep http://127.0.0.1:8090 up
|
||
./scripts/dev-server.sh status # per-source ok/DOWN
|
||
./scripts/dev-server.sh logs
|
||
|
||
uv run pytest # hermetic: 769 passed + 11 live-skipped, no network
|
||
uv run pytest --cov # coverage gate (98.33%; floor 94%)
|
||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||
uv run ruff check src tests agent && uv run pyright
|
||
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
|
||
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
|
||
# ... plus smartctl per physical disk (needs device read
|
||
# access: plain smartctl, else `sudo -n`; scrubs serials)
|
||
|
||
bash .agents/validate.sh # the phase gate: the four checks above (pytest
|
||
# --cov --cov-fail-under=94, ruff, pyright, plus a
|
||
# node --check of web/static) + repo hygiene.
|
||
# Always `uv run` — a bare `python3`/`ruff` on PATH
|
||
# tests an interpreter nobody deploys.
|
||
```
|
||
|
||
Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
|
||
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
|
||
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
|
||
`--status`/`--uninstall`; it stores the token in a root-owned 0600 file and hands the agent
|
||
`--token-file`, never argv, and runs under `ProtectSystem=strict`).
|
||
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
|
||
service here. The source itself is `git@git.reeseapps.com:services/dashboards.git` (`origin`,
|
||
branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only gate.
|
||
|
||
## Testing policy
|
||
|
||
**Real captures for shape, fakes for behaviour, `live` for drift.**
|
||
|
||
- Anything that must match an upstream — `/proc` layouts, vLLM series names, NUT framing —
|
||
is asserted against a verbatim capture in `tests/fixtures/`. If the schema drifts, a
|
||
fixture test fails before a dashboard does. `smartctl_driveripper_<dev>.txt` is the whole
|
||
16-drive fleet read through a `smartctl` sudo rule (Serial/WWN/EUI-64 scrubbed), and
|
||
`test_the_real_drives_answer_every_number_the_disks_page_prints` asserts every number
|
||
`/disks` prints comes out of real drive output — that test is what caught the parser
|
||
reading the end of an attribute row instead of its annotated RAW_VALUE column.
|
||
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
|
||
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
|
||
- **Page payloads that a rewrite must not break are pinned as a table.**
|
||
`tests/card_contract.py` holds the dotted `/api/system/snapshot` paths the `/system` hero
|
||
cards read (`CARD_METRICS`, `GPU_METRICS`, `HOST_SHAPES`); `tests/test_system_contract.py`
|
||
asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and
|
||
`test_every_host_serves_the_hero_card_contract` asserts the same table against the live
|
||
agents — so a rename fails a test, and a card that shows `n/a` cannot be "fixed" by
|
||
inventing a field. `network` carries no `severity` on purpose (throughput has no good/bad)
|
||
and neither does a GPU row (phase 06: a busy GPU is the job on an LLM server) — the
|
||
contract pins both absences. `tests/ups_contract.py` is the same device for the `/ups`
|
||
hero row and battery glyph (`HERO_FIELDS`, `UPS_SHAPES`, `assert_ups_hero_contract`):
|
||
`tests/test_ups_contract.py` drives it from `tests/fixtures/nut_<ups>.vars` — verbatim
|
||
`LIST UPS`/`LIST VAR`/`LIST CMD` replies from the two real UPSes — and
|
||
`test_ups_hero_fields_are_present_on_the_real_batteries` drives it against live upsd. The
|
||
captures are what make the driver disagreements testable: one of them answers
|
||
`battery.mfr.date: CPS`, neither exports `battery.temperature` or
|
||
`battery.charger.status`. `tests/slots_contract.py` is the same device for gufo's `/slots`
|
||
rows, which the `/llm` slot grid (phase 07) reads (`assert_slot_shape`, required/optional
|
||
field tables, duplicate slot ids, the `next_token[0]` entries): hermetically
|
||
from `tests/fixtures/slots_gufo.json` — the verbatim `/slots` reply, idle slots and all,
|
||
captured whenever `GUFO_BASE_URL` is set — and live via
|
||
`test_gufo_slots_answer_the_pinned_shape`. That fixture exists because its absence is how
|
||
the deleted slot table rendered dashes forever: its field names (`slot.state`,
|
||
`slot.n_prompt`, `slot.decode_per_second`) were a memory of an upstream, not a capture.
|
||
**The contract names one canonical key per fact and the grid renders that one**: the capture
|
||
answers `id_task` *and* `task_id` for the same request, so `CANONICAL_TASK_ID_FIELD` decides
|
||
and the alias is captured but rendered by nothing — a rename fails a test instead of becoming
|
||
a silent `a ?? b` chain in the page. `tests/test_api.py` reads the served `SLOT_FIELDS` table
|
||
and resolves every key it names against the capture, in both directions.
|
||
A busy slot's `prompt` is the user's in-flight request text, so the capture script blanks
|
||
it and the page never reads it.
|
||
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
|
||
fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
|
||
unset, so partial fleet availability is normal. **Live tests must not write** — a battery
|
||
test on hardware something is plugged into is not a test.
|
||
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
|
||
so it stays in the default run and is *not* marked `live`.
|
||
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
|
||
94% (`--cov-fail-under`), and ~98% is where we are; keep it there. New/modified Python
|
||
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
|
||
never a real target, and never a real upstream: `/api/ups/command`'s send is proven against
|
||
a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live
|
||
suite is read-only — that fake is the only place the write reaches a "driver".
|
||
|
||
## UI conventions
|
||
|
||
Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
|
||
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
|
||
dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
|
||
`LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
|
||
silence, and the legend is a sibling that labels nothing). Every `LineChart` on every page has
|
||
one — `/system`'s three per-host slots, `/disks`'s two per-host plots, `/llm`'s four trend
|
||
plots, `/ups`'s four battery plots — each pinned by `tests/test_api.py`, so a chart that ships
|
||
anonymous fails a test.
|
||
Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups` and `/disks`; the API accepts
|
||
5m…90d — SMART lands every 300 s, so a 5-minute window on `/disks` is one dot). A
|
||
panel that cannot get data says why in the panel — the operator should never have to open
|
||
devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
||
|
||
**Flavour text is not helpful information.** A page does not narrate its own data pipeline or
|
||
its own design; copy exists only where the operator needs it to act — a why-this-panel-is-empty
|
||
sentence, a provenance tag, a measured `Δ<window>` label, a chart's unit, a `SIMULATED` badge,
|
||
an error that names what failed. The audit is one question per string: *would the operator act
|
||
differently without it?* No → delete it, container and all. Deleted in phase 08 (the cited
|
||
examples, from `TODO.md:7`): "every number is scraped from the box it belongs to · `null`
|
||
renders as `n/a`" (`/`'s footer), "agent-reported values; rates are measured over the Δ window
|
||
in the footer, and each panel's charts cover only that host" (`/system`'s *Per host* sub-line),
|
||
and the same genre — `/system`'s "numbers are measured, not inferred", `/llm`'s "each throughput
|
||
number carries its provenance", `/ups`'s "commands are the only write path in this app", each
|
||
event feed's content caption (the empty-state sentence already says it), and the design
|
||
clauses inside empty-state prose. What survived the audit did so because it is a measurement or
|
||
a why: every `Δ<window>` footnote, `provenance()` on each rate, `fleet.seams`, *sums only
|
||
engines answering now*, the queued-request note under the slot grid, the SIMULATED banners, and
|
||
the setup instructions in every empty state. `tests/test_api.py` pins the deleted sentences
|
||
**absent** from every served page and script (a sentence that migrates to another page is the
|
||
same slop), the kept functional strings **present**, and that no removed sub-line left an empty
|
||
`.sub`/`.faint` container behind.
|
||
|
||
The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per
|
||
`/api/summary` alert: verdict class, source linked to its dashboard via the alert's own
|
||
`href`, message, age) and **Fleet**. A non-empty list floats the alerts panel above the
|
||
fleet table (`#overview-panels[data-state]` swaps them); an empty one is an explicit
|
||
"all clear" state below it, because an empty list is a message, not a blank box. A host that
|
||
is not answering keeps its row, dimmed with the shared `.stale` class and reading `n/a`, so
|
||
the table never implies last good numbers are live and never silently loses a machine.
|
||
No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the
|
||
two-panel shape against the served HTML.
|
||
|
||
Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an
|
||
`id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for`
|
||
attribute references — the orphan a deleted renderer leaves behind, caught on served bytes
|
||
because there is no JS test runner (a locked anchor). Same rule for `dash.css`: delete the
|
||
rules a removed panel made dead (`.grid.wide` went with the overview's fleet+events row);
|
||
component vocabulary the pages are still being converted to (`.hero-bar`, the grid and bar
|
||
variants) stays even while only one page uses it.
|
||
|
||
A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
|
||
muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's
|
||
`severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*`
|
||
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
|
||
rule reaches for a token that `:root` does not declare.
|
||
|
||
**The `/disks` card is one panel per *physical* drive, led by identity and verdict, not by a
|
||
hero** (phase 09): a drive has no one-number story the way a UPS has charge %, so the card opens
|
||
with the device path (`/dev/nvme0n1`) and the drive's own word (`OK` / `FAILED` / `unknown`, a
|
||
`.chip` coloured only by `severity.health` — an unreadable drive gets grey `unknown`, never a
|
||
green pass), then the identity line (`model · size · SSD|HDD`, nothing invented where a field is
|
||
absent) and three equal `.metric` cells: **Temp**, **Power-on**, **Wear**. Colour comes from
|
||
`severity.temp_c` / `severity.percentage_used` and nowhere else; `power_on_hours` is an odometer,
|
||
written with a `null` verdict, so it cannot read green at 100 h or red at 60 000 h. The sector
|
||
counters are **`.chip.fail` alarm chips that exist only while non-zero** (the `/llm` preemption
|
||
rule, and why `severity.py` rates no counter): a drive at zero draws no chip row at all, because a
|
||
permanent "0 pending sectors" line trains the eye to ignore it and a green one would call a quiet
|
||
drive an achievement. Per host the three-way `smartctl_available` decides one `.banner.info`
|
||
why-line and nothing else — `false` says *install smartmontools and grant the disk group* (the
|
||
agent said so, so the page may), `null` prints the agent's own `smart_errors` line or names both
|
||
fixes without claiming to know which half is missing, and an agent that is not answering says
|
||
that instead — and in every case the drive grid takes the shared `.stale` dim while the host card
|
||
stays uncoloured (invariant 3: a missing binary is unconfigured, not broken, and a red panel here
|
||
would train the operator to ignore red on the drives that do report). Nothing on the page is per
|
||
partition or per mount: the rows are the agent's whole-`/sys/block` list, and a filesystem field
|
||
naming its way into `page-disks.js` fails a test. The fleet header renders
|
||
`fleet_disk_summary()` — hidden below two hosts (one box's sums duplicate its own cards) and
|
||
coloured nowhere, because a count of drives is not a graded number — and the per-host `Charts`
|
||
disclosure follows `/system` exactly: collapsed, lazy-built, **one** `/api/disks/history` request
|
||
per host per refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive (the
|
||
endpoint answers unknown names as empty series on purpose), the wear plot created only once some
|
||
drive publishes the field, legends naming drives because the card heading already says who, and
|
||
`pruneHosts` taking the cards, the toggle and the plots off the page together.
|
||
|
||
A cell-scale trend is `DashCharts.sparkline()` — the `llm_monitor` `drawSpark` idiom, so a
|
||
percent metric is plotted against `fixedMax` (**never** autoscaled: an 8 % KV window that once
|
||
held a 92 % spike must not read flat, and the axis gets top/bottom scale marks so "a third of
|
||
the way up" says a third of what). `align: 'right'` pins the newest sample right and grows the
|
||
line leftwards inside a real `span` in seconds — x stays time, not sample index. Stroke colour
|
||
comes from the payload `severity` via the `--ok`/`--warn`/`--fail` tokens only; an unrated
|
||
series draws a faint `--text-faint` stroke with no fill, because a shape is not a verdict.
|
||
Below `minPoints` (2) it prints `no history` rather than drawing a confident line through one
|
||
point, `labels: {current, max}` renders the `now / max` foot through the caller's `format`, and
|
||
there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the
|
||
other half of the same honesty for count metrics: history arrives as rollups, so eight seconds
|
||
of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for
|
||
a queue nobody was in. `/llm`'s count rows moved to the phase-07 stats strip (numbers, not
|
||
sparks), so no page passes `minSpan` today and `drawSparks()` still plumbs it through — a count
|
||
sparkline that comes back must set it, and the component's own contract test keeps the option
|
||
honest (`test_sparkline_ships_the_fixed_scale_and_verdict_options`).
|
||
|
||
A battery is `DashCharts.battery(container, {charge, severity, state, label})` — one inline SVG
|
||
whose **fill *is* the charge**: 1 viewBox unit per percent, clamped to 0–100 and never rounded up
|
||
(a 2 % cell must look nearly empty, because a reassuring battery above a flat one is a lie),
|
||
clipped to the plate so 0 % leaves an empty shell rather than a stray pixel. Colour comes only from
|
||
the payload `severity` token; a measured charge with no verdict draws faint grey at half strength
|
||
(`sparkline`'s rule — a shape is not a verdict) and never an empty shell ("0 %" and "not rated" are
|
||
different sentences). Every state has a **shape** as well as a colour, so the glyph survives
|
||
monochrome: solid shell `online`, bolt `charging`, dashed outline `onbattery`, no fill with a dash
|
||
pattern *through* the body `unknown`; a state word it does not know degrades to `unknown`, never to
|
||
a confident "on line". Accessible **by default** — `role="img"` + `<title>` + `aria-label` built
|
||
from the inputs ("apc: Charge 100 percent, on line"), because the glyph carries information the
|
||
markup does not say in words — and a page that prints that sentence next to it passes
|
||
`decorative: true` to make the `<svg>` `aria-hidden` rather than say it twice. No animation and no
|
||
`ResizeObserver`: one `viewBox` + `preserveAspectRatio` scaled by `.battery` in `dash.css`, so the
|
||
hero row gets a big cell on a desktop and a small one on a phone.
|
||
|
||
**The `/ups` card is led by the glyph and three heroes** (phase 05): the battery on the left, then
|
||
**Charge %**, **Load %** and **Runtime left** on `.heroes` at the shared 2.4rem, coloured from
|
||
`severity.charge_pct` / `severity.load_pct` / `severity.battery_runtime` and nothing else. Charge
|
||
carries **no** `.bar` — the glyph beside it is that value drawn as a shape, and one value drawn
|
||
twice in one row is the pattern this app avoids (Load keeps its bar; nothing else draws it). Its
|
||
`.sub` lines are the context each number needs and nothing more: `24 V nominal · 27.3 V` for the
|
||
pack, `128 W of 900 W` when the driver publishes both `realpower` figures (a wattage inferred from
|
||
a percent against another model's rating would be an invented number), `at the current load` for
|
||
runtime. Runtime prose is `runtimeText()`'s (`1h 32m`, `10m 21s`) and an estimate the driver never
|
||
made goes into `Dash.hero()` as `null`, so it prints faint and uncoloured rather than as `0m`. The
|
||
readings the hero row displaced — line in, output, battery volts and amps, battery °C — stay on the
|
||
card, demoted to `.ups-electrical`: muted, `.95rem`, never folded away, because they are what you
|
||
reach for *while* the UPS is on battery, and battery °C reads `n/a` on both real drivers here.
|
||
`state` is derived in the page from `on_battery` → `charger_status` (`chg` charges; `dchg`/`rec` are
|
||
the server's own on-battery tokens; `full` is **on line, not charging** — a rested floating battery
|
||
reports it) with one fallback: a state chip the API rated `ok`, because `nut.states()` reserves
|
||
`ok` for on-line/fully-charged and neither real driver here publishes `battery.charger.status` at
|
||
all. `ups.status` itself is never re-parsed in JS (`tests/test_ups_contract.py`). `/ups` passes
|
||
`decorative: true`, because the charge hero and the state chips already say the sentence. The glyph
|
||
repaints only when charge / verdict / state change (`drawGlyph()` keys a `data-drawn` triple): the
|
||
charge float jitters every `NUT_INTERVAL_SECONDS`, and rebuilding the `<svg>` on that re-announces
|
||
an accessible name for a difference no eye can see.
|
||
|
||
An event row names its subject **once**. Every feed (`/system`, `/disks`, `/llm`, `/ups`) renders the
|
||
message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a
|
||
*leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py`
|
||
messages start with their own label, and each row already prints that label as a chip, so
|
||
"gufo gufo is answering again" reads as two problems. A name in the middle of a message stays:
|
||
the UPS rows used to `replace()` it anywhere and mangled "Quick battery test sent to apc: OK".
|
||
This is presentation only — the stored event and `/api/events` keep every word, and the raw
|
||
error string still surfaces in `/api/health`.
|
||
|
||
An event feed is a **window**, not a printout (`Dash.eventsPager(el, events, opts)` in
|
||
`app.js`, one helper with four call sites (`/system`, `/disks`, `/llm`, `/ups`) — a pager that
|
||
works on one page is the bug this
|
||
shape prevents). `ul.events` is a bounded 320px scroll container (`overscroll-behavior:
|
||
contain`) and the pager puts `EVENT_PAGE_SIZE` = **50** events in it, with a `.events-pager`
|
||
strip under the list: `1–50 of 200` between `‹ prev` and `next ›`, both `aria-label`led for
|
||
direction (the feed is newest-first, so `prev` walks towards now) and `disabled` at the ends.
|
||
The strip is `hidden` when the feed fits on one page — every row on screen *is* the whole
|
||
story, and a disabled arrow says "more history" when there is none. Each page keeps its own
|
||
`row(event)` (so `/ups` keeps its longer timestamp and runtime footnote, and escaping stays
|
||
the page's job — the pager has no default renderer and says so in a row when a caller forgets
|
||
one) and its own `empty` sentence, so the `loading…` and empty states read exactly as they
|
||
did before paging existed. Two numbers are load-bearing and pinned together: the fetch
|
||
(`EVENT_FEED_LIMIT` = 200, `/api/events?limit=…`) must exceed one page, because a limit at or
|
||
below `EVENT_PAGE_SIZE` is a pager that silently never appears; `/api/events` itself is
|
||
**unchanged** — no `page`/`limit` vocabulary grew for one page's layout. Paging state lives in
|
||
a `WeakMap` keyed by the `<ul>`: it survives a poll (so page 3 is still there two seconds
|
||
later — resetting every refresh would make page 2 unreadable) and resets to page 1 when the
|
||
event *set* changes (an operator reading page 3 of yesterday's history must not land on page 3
|
||
of a list that just changed). Rows are re-written only when their markup changed, which is
|
||
what keeps the reader's scroll position inside the container.
|
||
|
||
The `/system` host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the
|
||
`.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script).
|
||
Colour comes from the payload's `severity` dicts and nowhere else; a missing number or
|
||
verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**. NET is
|
||
one slot with two lines (`.hero-lines`, `↓` in / `↑` out) and is deliberately colourless:
|
||
throughput has no good/bad, so `network` carries no severity. The GPU slot holds one hero
|
||
per card whose `.sub` carries model, VRAM used/total, the honest APU `pool` view and the
|
||
card's temperature (`gpus[].temp_c` — the agent's max over that card's hwmon labels, the
|
||
same value stored as `gpu.temp_c@N`, so the hero and the history cannot disagree), and it is
|
||
colourless for the same reason (phase 06): a 95 %-busy card on an LLM server is the job, so
|
||
the hero carries no `severity` and no utilisation bar — a bar under a verdictless number
|
||
would be a verdict by shape. A card whose driver exposes no temperature input prints **no**
|
||
temperature in that footnote rather than `n/a` or `0 °C`: "where you can" is the operator's
|
||
wording, and `n/a` is reserved for numbers the card is supposed to have. Everything not
|
||
glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU power and the
|
||
per-label temp breakdown, filesystems, processes — folds into a collapsed "More detail"
|
||
`<details>`: a simpler surface, not a less complete page (the table keeps its Temp column;
|
||
this was a promotion, not a de-duplication).
|
||
|
||
A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
|
||
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
|
||
those (a range when the hosts disagree) instead of the poll interval, which only coincides with
|
||
them at the defaults. `/llm`'s hero `.sub` prints the rate cell's own `window` through the same
|
||
`provenance()` tag, and the hero's `title` carries the exact `source` string — "78 tok/s" must
|
||
always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_monitor`'s
|
||
`setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification).
|
||
|
||
**The `/llm` page reads per engine** (phase 04, decided in
|
||
`.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`, laid out by phase
|
||
07): a row of two `Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**,
|
||
because 0 tok/s on an idle server is healthy and `severity.py` has no throughput row) and the
|
||
latency rows, each a mean with a mono `p95 …` foot (no hover; p95 needs no stored series) and a
|
||
`sparkline()` fixed at 160px so rows line up. Rows are engine-scoped by the `only` flag on both
|
||
components: gufo never grows a TTFT mystery, vLLM never a spec-accept hole.
|
||
|
||
**The stats strip (`#strip`, phase 07 task 03) is the summary; the panels are the detail, and no
|
||
row is printed in both** — the UPS hero row's "one value drawn twice" rule, widened to two
|
||
components. One `.card.stat` per *configured* engine between the fleet block and the panels: the
|
||
reachability dot, the engine name and the age stamp in the head (same distinction every card
|
||
draws — the dot speaks reachability, the numbers never do), then the **KV-cache bar** — the one
|
||
cell on the page with a verdict, so the fill and a breach's text take their colour from the
|
||
payload's `severity` through `.bar.ok/.warn/.fail` and the ratio→percent conversion the panel's
|
||
cell used to make, with `ok` staying quiet ink and a missing gauge an empty bar plus `n/a` — then
|
||
`prompt tok/s` / `decode tok/s` with their `Δ<window>` tags, `total tokens` (lifetime, so its
|
||
source is the tooltip), the sessions this engine reports, `prefix cache hit`, `spec accept`
|
||
(gufo) and `queue time` with its `p95 …` foot (vLLM). `STRIP_ROWS` is the whole table and its
|
||
`key` **is** the `cellValue()` key, so every number on it is a cell of `/api/llm/snapshot`: no
|
||
field was invented in `sources/llm.py` for this card, and there is deliberately **no peak-ctx
|
||
row** (phase assumption). The session row is engine-shaped — vLLM publishes `running`/`waiting`
|
||
counts, gufo's is `slots busy`, counted from the `sessions.list` rows `/slots` answered
|
||
(`is_processing`, the shape pinned by `tests/slots_contract.py`); when `/slots` did not answer
|
||
there are no rows to count, so it falls back to the published `sessions.active` carrying the
|
||
API's own `sessions.detail` sentence rather than reporting 0 busy slots from no data (rule 2).
|
||
`preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
|
||
container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded
|
||
threshold) and they live on this card, not in a permanent cell; the lifetime token counter left
|
||
the panel's footnote line for the strip's row. **One `/api/llm/history` request per refresh**
|
||
carries every chart *and* spark series (`HISTORY_SERIES` = the four chart series +
|
||
`SPARK_CELLS`; the endpoint answers unknown names as empty series on purpose), and the two series
|
||
phase 04 added (`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) stay stored
|
||
additively in `sources/llm.py` HISTORY_METRICS. Above the strip sits **the fleet block**
|
||
(`#fleet`, phase 07: "I'm missing the aggregate throughput at the top") — `ALL ENGINES`, a
|
||
reachability dot, an
|
||
`N of M engines` tag, the note *sums only engines answering now*, then running / waiting /
|
||
prompt tok/s / decode tok/s / total tokens. `FLEET_FIELDS` in `page-llm.js` is a list of
|
||
`fleet_totals()` keys and nothing else: the page sums no number of its own, a dead engine is
|
||
dropped from every total presented as current (its own stale-labelled panel still shows its
|
||
last good values), and the tag counts the answering engines, names the missing one in its
|
||
tooltip and takes the shared `.stale` dim while the sums are partial. It **hides below two
|
||
engines** — a cross-engine sum of one engine duplicates that engine's own panel — and it
|
||
carries **no severity colour anywhere**: `fleet_totals` has no `severity` to render (the
|
||
summed-KV bar is gone for that reason) and `severity.py` rates no throughput or queue depth,
|
||
so the numbers stay ink and only the dot speaks reachability. `fleet.seams` — the API's own
|
||
note on how the two engines count prompt tokens differently — is the prompt sum's tooltip
|
||
rather than another line of ink. The
|
||
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
|
||
sit at their own height (`#engines` carries `.items-start`): a two-hero gufo card next to a
|
||
latency-row vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". An
|
||
engine that leaves the payload leaves both components (`pruneEngines`), never a stale card in
|
||
either grid.
|
||
|
||
**The slot grid (`llm_monitor`'s, phase 07 task 04) belongs to the gufo panel, under its
|
||
`sessions (GET /slots)` label.** One card per row of `sessions.list`, **keyed by the row's own
|
||
`id`** in `state.slots`: built once by `slotCard()`, patched in place afterwards, and reordered
|
||
by `appendChild` of the existing node so DOM order stays slot order — rebuilding a busy slot's
|
||
card every 2 s poll re-announces the whole card to assistive tech for a difference of a few
|
||
tokens. `SLOT_FIELDS` is the whole table and each cell names **exactly one** `/slots` key
|
||
(`path: 'next_token'` reads `next_token[0]`, where the capture puts `n_remain`); `slotValue()`
|
||
is the only place a row is indexed, so `tests/test_api.py` can resolve every printed field
|
||
against `tests/fixtures/slots_gufo.json` and a rename fails there rather than printing dashes
|
||
or a plausible wrong number. No per-slot speed cell: the capture publishes no per-slot rate key
|
||
and a page-side Δn_decoded/Δt would be a measurement with no estimator, no `Δ<window>` label and
|
||
no `source` string (rule 2) — the engine's rate lives on its strip card and panel. llama.cpp's
|
||
`-1` sentinels print a word or nothing: `id_task: -1` is **`idle`**, `n_remain: -1` is **`n/a`**,
|
||
never a negative countdown and never the `0` that reads as one token from done. The busy marker
|
||
is a **border, a glyph (`○`/`●`) and the word `processing`** in the brand `--accent`, never an
|
||
`ok|warn|fail` token — a slot generating tokens is the job (phase 06's GPU reasoning), so
|
||
`severity.py` rates no slot and the stylesheet section contains no verdict token. The empty
|
||
state is a sentence, never a card of nothing, and it distinguishes the two cases: no `/slots`
|
||
answer prints the API's own `sessions.detail`, an answer with no rows prints *engine reports 0
|
||
slots*; `sessions.source` is the gate. The note under the grid stays because a **queued
|
||
(deferred) request is not in `/slots`** — the waiting row counts those. vLLM's panel gets one
|
||
line (*exposes no per-slot JSON*) with `sessions.detail` as its tooltip and **no grid**: a
|
||
component of permanent `n/a`s for a server that cannot answer is the dead-cell anti-pattern.
|
||
The gate is the engine name (`SLOT_ENGINES`), not row count, so an engine that has not answered
|
||
yet still shows its label and its honest empty sentence.
|
||
|
||
**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
|
||
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
|
||
panel and the host/metric detail picker are deleted, along with the `#chart-*` / `#detail-*` ids and
|
||
the `select` / `label.field` styling only those pickers used. Each panel stacks its own
|
||
`.host-charts` — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where
|
||
the GPU slot is created and destroyed with `host.gpus`, so gateway shows neither a phantom card
|
||
nor a hole. **That stack is collapsed behind a per-host `Charts` `<details>`** (phase 06: "only
|
||
visible if expanded… there will be lots of hosts"): the default `/system` view is the hero row,
|
||
the summary carries the window label the stack will draw (the chips themselves stay in the page
|
||
head, one control per page), and `drawHostCharts()` is the single place that decides whether to
|
||
paint — so nothing is built or drawn while the stack is shut (`LineChart` inside a
|
||
`display:none` container measures 0 px, and rebuilding three charts per host every poll for
|
||
plots nobody sees is the waste this removes). The **history request is deliberately unchanged**:
|
||
one per host per refresh, naming every series the panel could draw, and its response is cached
|
||
per host so the first expand paints from data in hand rather than waiting a poll; a cached
|
||
response from another window is re-fetched instead of being drawn under the wrong label.
|
||
Expanding one panel must not stretch its collapsed siblings into blank cards, so `#hosts`
|
||
carries the existing `.grid.items-start`. **One `/api/system/history` request per host per refresh**
|
||
(`hosts=<that host>&metrics=<that panel's series>&range=…`, pinned by
|
||
`test_every_system_chart_is_drawn_inside_its_own_host_panel` and the collapse by
|
||
`test_system_history_charts_are_collapsed_behind_a_toggle`): unknown metrics come back as
|
||
empty series on purpose, so a panel asks for every series it could draw and asks once. A legend
|
||
here names metrics (or one GPU's model), never hosts — the panel heading already says who this
|
||
is, and a legend that grew a second host is the bug this replaced. A host that stops reporting
|
||
loses its panel, its `Charts` toggle *and* its charts (`pruneHosts` deletes the card plus the
|
||
payload/chart/history maps), never a chart of nothing or an orphan control in the grid. Per-core
|
||
visuals are gone from `/system` and `renderCores` is **deleted** from `charts.js` with its last
|
||
caller.
|
||
`drawHostChart()` shades a chart only when one series has points — two or three filled areas
|
||
stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy
|
||
memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryable from
|
||
`/api/system/history` — removing the plot is not removing the data.
|
||
|
||
## Known gaps / next work
|
||
|
||
1. **UPS commands work, and the account can now cut power — the app's gates are the control.**
|
||
Three defects were cleared on 2026-10-17, each verified against real hardware. (a) `NutClient.login()` sent `USER`/`PASS`, which the deployed
|
||
upsd 2.8.5 does not implement — each answered `ERR UNKNOWN-COMMAND`, so `_read_line()`
|
||
raised and `POST /api/ups/command` could only 502 before `instcmd()`, credentials or not.
|
||
It now sends `USERNAME`/`PASSWORD`, the pair the server's own `HELP` advertises
|
||
(`HELP VER PROTVER GET LIST SET INSTCMD LOGIN LOGOUT USERNAME PASSWORD STARTTLS`), pinned
|
||
by `test_login_uses_the_verbs_upsd_2_8_5_answers_to` — the fake upsd has no `USER`/`PASS`
|
||
branch, so a regression answers `ERR UNKNOWN-COMMAND` exactly like the real thing.
|
||
(b) 12 of the 23 commands the two real UPSes advertise had **no `COMMANDS` entry**, and
|
||
`available_commands()` drops anything the catalog misses — the page silently offered less
|
||
than the hardware does. Every advertised command is catalogued now, each with a risk tier:
|
||
`test_every_command_the_hardware_advertises_is_catalogued` fails if a capture gains a
|
||
command the catalog does not know, and
|
||
`test_every_command_that_can_cut_power_is_in_the_dangerous_tier` keeps `load.off`,
|
||
`driver.killpower` and the `shutdown.*` sequence in the tier that types `DANGER` and is
|
||
refused on battery. `shutdown.stop` is the exception the name hides — it *cancels* a
|
||
shutdown, so it is `safe`, and the test says so in words.
|
||
(c) `[dashboards]` in `/etc/ups/upsd.users` is `instcmds = all` at the operator's request,
|
||
so this app may send every advertised command on both UPSes, power-cutting ones included.
|
||
`.env.dev` (mode `0600`, gitignored) carries the credentials.
|
||
What remains to know:
|
||
- **Proven live, and reversibly.** `beeper.enable` returned `OK` and `ups.beeper.status`
|
||
read `enabled` on the APC afterwards; `driver.killpower` without the word `DANGER` still
|
||
answers 422 and never reaches the wire. A battery test has deliberately not been run —
|
||
it puts real load on a battery things are plugged into, which stays the operator's call.
|
||
- **`enabled: true` cannot mean "this account may send it".** `available_commands()`
|
||
filters on `LIST CMD`, which upsd serves **unfiltered**, and NUT has no call for a
|
||
user's privileges — so the wire cannot tell the page what the account is granted. With
|
||
`instcmds = all` the two agree by configuration rather than by check, which is precisely
|
||
why the risk tiers and the typed `DANGER` are load-bearing and not decoration.
|
||
- **Pre-existing upsd log noise:** every upsd start logs `Unknown upsmon type =` (20
|
||
occurrences over 12 starts, oldest before this work). Harmless — `upsmon` logs into both
|
||
UPSes right after it — but unexplained.
|
||
Editing `upsd.users` at all: never grant `actions = -`, and never write it blind — the
|
||
running `upsmon` authenticates as its own account from that same file for both UPSes, so
|
||
dropping it means no orderly power-down. Copy the untouched sections from the live bytes
|
||
and `cmp` them before writing; `tee` truncates in place, so owner and mode survive.
|
||
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
|
||
(`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
|
||
fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
|
||
all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the
|
||
field map is written.
|
||
3. **UPS history depth.** Events and the numeric series are stored; long-window battery
|
||
health (capacity fade over months) has the rollups but no dedicated view.
|
||
4. **The dev DB is big enough to refuse a write now and then.** The dev store (~590 MB, WAL
|
||
grown to ~70 MB) logs `store-error: OperationalError: database is locked` a few times an
|
||
hour across `system`/`llm`/`ups`; the poller handles it exactly as invariant 4 requires —
|
||
the source goes red with the real reason, ages, and writes `recording` when the next poll
|
||
lands — so nothing freezes, but those polls' samples are lost. The one cause this repo can
|
||
prove is two servers on one file, which is why `scripts/dev-server.sh stop` waits for the
|
||
old pid to die (its comment records the incident). A controlled 25 s external read
|
||
transaction during this pass produced none, so checkpoint starvation is not yet shown.
|
||
Retention/`wal_checkpoint` tuning for a long-running store is its own phase, not a
|
||
phase-09 follow-up.
|
||
|
||
## Locked decisions (and why)
|
||
|
||
- **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10s resolution, one file the
|
||
container owns. Rollups keep a year of charts at a few hundred MB.
|
||
- **Our own stdlib host agent, not node_exporter.** One JSON snapshot with exactly the
|
||
fields the page needs (GPU model strings, per-mount usage, temperature labels) versus
|
||
~1500 series and a re-labeling layer; no dependency to install, `ProtectSystem=strict`
|
||
friendly, and read-only by construction. `scripts/capture-node-exporter.sh` remains to
|
||
compare ground truth if that is ever revisited.
|
||
- **Our own NUT client, not `pynut`/`upsc`.** The protocol is ~10 line commands, the
|
||
third-party libs are subprocess-based or stalled, and we need instant commands with
|
||
explicit gating.
|
||
- **In-house Prometheus parser.** Only two upstreams use the format; keeps `prometheus_client`
|
||
out of the dependency set.
|
||
- **Port 8090**, inherited from the `llm_monitor` prototype this started from: 8000 is vLLM
|
||
and 8731 is gufo in this fleet.
|
||
- **`.env` is a seed, never an override** — container/systemd env must win.
|
||
|
||
## Definition of done (per change)
|
||
|
||
The first three are exactly what `bash .agents/validate.sh` (the phase gate) checks, so no
|
||
phase may be called done on a subset of them.
|
||
|
||
`uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated`
|
||
empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted
|
||
in a browser against **live** data · fixtures re-captured if an upstream contract changed ·
|
||
this file and `README.md` updated when a rule, command, gap, or fleet fact changed.
|