dashboards/AGENTS.md

AGENTS.md — homelab dashboards

The plan, the rules, and the environment facts an agent needs to work on this repo without re-deriving them. README.md is the operator-facing how-to; this file is the why, the invariants, and the state of the fleet.

Working protocol

This project runs the .agents/phases/ phased-execution pipeline.

1. Always read .agents/PLAN.md first to understand the project context and goals.
2. Follow the phased execution protocol in .agents/phases/.
3. Never modify .agents/PLAN.md or any files in .agents/phases/complete/.
4. If you need to update any file in .agents/phases/todo/, you must ask the user for permission first.
5. Strictly adhere to the LOCKED DECISIONS listed in .agents/PLAN.md.

Everything below this section is the project's own content: what the app is, the rules that keep its numbers honest, and the state of the fleet it watches.

What this is

Four dashboards for one homelab, behind one domain, in one process, backed by one SQLite file — plus an overview page that says which of the four you need:

Route	Question it answers	Source
/	Is anything wrong, and where do I look first?	all four, in one request
/system	What are my machines doing — CPU, memory, disk, GPU, network, and the history	agent/host_agent.py, our own stdlib agent per host
/disks	Are any of my physical disks failing, and which one — health, temp, hours, wear	the same agents, smartctl -i -A -H per whole drive (never a partition)
/llm	Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles	gufo /metrics + /slots, vLLM /metrics
/ups	Battery health, runtime left, power events, battery-test buttons	NUT upsd (our own wire-protocol client; 3051 here, not 3493)

History is a first-class feature, not a sparkline afterthought: every number worth graphing is stored, rolled up (raw → 1m → 1h), and served from /api/*/history.

Non-negotiable rules

These are the things that make the numbers trustworthy. Breaking one is a bug even if the page still renders.

1. Read-only by construction, with exactly one exception. Nothing in this app writes to an upstream except POST /api/ups/command. That endpoint is gated four times (NUT_ALLOW_ACTIONS, not DASH_READ_ONLY, the upsd.users account actually having the command, and a UI confirmation — DANGER for anything that can cut power) and the server re-checks every gate on every request. Hiding a button is not a safety control. Commands that would cut output are refused outright while the UPS is already on battery.
2. Never invent a number. A value the upstream does not publish renders as n/a, never as 0. Every derived value carries the source string it came from and derived: true so the UI can print how it was obtained (computed:Δtokens/Δt, Δ10.0s). Aggregating two different series into one rate is the failure mode this rule exists to catch — the rate estimator is keyed by the resolved upstream series name for exactly that reason.
3. Unconfigured is not broken. A source with no env config answers 404 {"configured": false} without dialling, and the page says "not configured". A red panel for a server the operator never installed destroys the signal from the panels that matter.
4. One dead source must not blink another — and nothing may stop polling in silence. Each source is its own asyncio task with its own timeout and health record (poller.py). A failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it — on / too, where a host that has never answered still gets a fleet row (unreachable: true, dimmed, every number n/a) because dropping the row would turn a dead agent into "never configured". Transitions write events; repeated failure writes one line, not 500. The task is the source's only poller, so no exception may end it: a store that refuses an otherwise good poll is a reported failure (ok: false, the real reason in error, last_ok frozen so the page ages the numbers, one store-error event then one recording event on recovery), and anything that escapes the poll body is logged and the loop continues. Four dev-server sources once froze behind an unretrieved SQLite exception while /api/health went on calling them ok — a frozen source that looks healthy is the one failure mode worse than an outage.
5. No CDN, no build step, no internet at runtime. Every asset is served by this app; charts are hand-rolled SVG (web/static/charts.js). The engines live on a network with no outbound access — a CDN reference is a request that never comes back.
6. Never leak a credential. Target URLs may carry a userinfo token (parse_targets moves it into an Authorization header, so Target.url ends up clean). Nothing that leaves the process may carry one: /api/config serves Target.display_url and config.strip_userinfo-redacted engine URLs, and every URL inside an UpstreamError message is redacted too — those strings go into the health record and the event table, which /api/events serves to the browser and page-system.js prints. No token, password, or personal hostname goes into tracked files, fixtures, or the browser.
7. Verify against the deployed upstream; never guess a metric name. Field maps in sources/llm.py and sources/nut.py were captured from the running engines and NUT 2.8.5, and are pinned by fixtures in tests/fixtures/. Re-capture with scripts/capture-fixtures.sh (and scripts/capture-node-exporter.sh when comparing exporters) rather than trusting a README or a memory of an upstream.
8. Keep dependencies current on purpose. LLM-written version floors are usually stale. Check after touching dependencies: uv pip list --outdated must come back empty. Python is 3.14 (requires-python >=3.14, ruff py314, pyright 3.14, Containerfile python3.14) — do not lower these to make something else work.
9. SQLite owns the history and must stay bounded. Writes go through store.py; new metrics need a rollup path and a retention story, or a year of charts becomes a disk emergency. Put the DB on a volume in deployment.
10. The dev server stays up while work happens (./scripts/dev-server.sh restart, http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot before they are called done. data/logs/*.png is where those land.

Architecture

agent/host_agent.py (each host)      gufo / vLLM            upsd (NUT)
        │ GET /snapshot                    │ GET /metrics           │ LIST UPS / LIST VAR / LIST CMD
        │                                  │ + /slots (gufo)        │
        ▼                                  ▼                        ▼
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
        │  parse + Δcounter/Δt                                                     │
        ▼                                                                           ▼
sources/host.py · disks.py · llm.py · nut.py                         poller.py (task per source)
        │        Result(payload, samples, events)                                    │
        ▼                                                                             ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/

Responsibilities, one line each:

• agent/host_agent.py — stdlib-only (3.10+), reads /proc, /sys/block, /sys/class/drm, statvfs; serves GET /snapshot + GET /info + GET /schema + GET /health, plus an HTML status page at GET /. Refuses to bind off-loopback without a token (exit 2). Counters stay counters: the dashboard computes rates so the agent never remembers who polled it last. SMART (phase 09): smart() runs smartctl -i -A -H per whole /sys/block disk (parse_smart_output covers ATA + NVMe; NVMe media errors map to offline_uncorrectable, hex-packed vendor raws are never decoded, and an attribute row is read from its RAW_VALUE column rather than the end of the line because this binary annotates that column — 35 (Min/Max 34/44), 43894h+43m+32.519s — so its last token is not the value), 5 s per device / 20 s per pass / 240 s cache, attached as disk_io.devices[].smart. Missing binary → one errors line for the pass (the word not found in it is what sources/disks.py reads to answer "is smartctl installed", and tests/test_agent.py pins that the line still says it); one denied/wedged device → that disk's smart is null with its own line; --no-smart opts out silently. Never fails the snapshot. An expired cache refreshes on a background thread while the request is served the previous result — only the first pass after startup runs inline — because 20 s of wedged disk on the request path blows HOST_TIMEOUT_SECONDS and would turn the system source red every four minutes over an add-on that page never draws. The refresher is single-flight and keeps the old cache on failure, so a bad pass retries on the next expiry instead of hammering the disk.
• config.py — env only, no config file format, no CLI flags. .env is a seed; a real env var always wins. Defaults are safe except the sources, which default to unset.
• poller.py — job loop, health records, stale/last-good semantics, transition events. /disks is a source of its own (disks:<host>, DISKS_INTERVAL_SECONDS 300 s, DISKS_TIMEOUT_SECONDS 30 s, events under the dashboard name disks) reading the same agents the system jobs read: the separate task is what keeps a SMART poll out of the 10-second utilisation loop and lets /disks age on its own instead of blinking /system. The job list is built once (app.build_hub()) because it is wired twice — production and the injected-store test path — and two hand-written copies drifting is how a source ends up polling in tests and not in the container. The success path records (payload → samples → events) and only then moves last_ok, so a refused SQLite write ages the source and alerts instead of leaving it current with charts that have silently stopped drawing. Events are stored under the dashboard name (system/disks/llm/ups), not the sample source (host) — a page filtering /api/events by host gets an empty feed forever (found on /system during phase 03 browser verification).
• collectors.py — the fetch half. NutMonitor owns per-UPS transition bookkeeping. Failure messages name the redacted URL (strip_userinfo / Target.display_url), never the one that was dialled, because they become health records and event rows. fetch_snapshot() is the one agent /snapshot read that system and disks share, so a dead host says the same sentence on both pages and only the timeout differs (the disks poll has to outlast the agent's 20 s smartctl pass, and None is translated into httpx's sentinel because a literal None means no timeout at all). Agent errors become the payload's degraded except the smart: add-on lines — a host without smartmontools or device read access is unconfigured, not broken (invariant 3); they stay in agent.errors for the /disks page to explain.
• sources/host.py — snapshot → payload + samples; reboot detection; GPU series per card (gpu.busy_pct@<index>).
• sources/disks.py — the same snapshot projected onto physical drives for /disks (phase 09): one row per whole-/sys/block disk (never a partition, never a mount), the agent's smart block carried through verbatim, health the drive's own word normalised to ok/failed/None, and smartctl_available three-way on purpose — True (a disk answered), False (the agent said the binary is not installed, so the page can say install smartmontools), None (an agent predating the pass, or --no-smart: nobody knows, so n/a plus the agent's own smart_errors line and never a guessed verdict). Nothing here is a rate: power-on hours and the sector counters are stored counters-as-reported, so the source takes no RateEstimator and its payload carries no window_seconds (rule 2 — a rate with no estimator, no Δ<window> label and no source string is an invented number). disk.health_ok@dev is not a series either: a verdict is not a number, so the drive's word travels in the payload only. Each row does carry a severity dict (SEVERITY_FIELDS = the two rated numbers, temp_c/percentage_used, plus health through HEALTH_SEVERITY, which translates the drive's own word instead of grading it), and UNGRADED_FIELDS names the numbers that must never be graded. fleet_disk_summary() sums the per-host summary counts for the page header and drops a host with no payload from the sums while still counting it in hosts_total — the /llm fleet-block rule, so the page never does arithmetic and a dead agent never shrinks a fleet total.
• sources/llm.py — gufo + vLLM field maps (GUFO_SERIES, VLLM_SERIES, VLLM_HISTOGRAMS), measured Δ rates, prefill speed as mean(prompt tokens)/mean(prefill time), session list from gufo /slots.
• sources/nut.py — upsd client, UpsReading, VAR_MAP, assess() (explicit checks, not a mystery score), COMMANDS catalog with risk tiers, COMMAND_ALIASES + resolve_command() (drivers answer two names for the same action — beeper.enable/beeper.on, beeper.disable/beeper.off — so one entry is offered per action and any of its names hits that entry's verdict, never a gate the rendered name missed), available_commands() filtering, kelvin-vs-celsius normalisation, and charger_status normalised to the CHARGER_STATUS keys (the same lowercase token states() and on_battery already compare), and states() publishes one chip per fact — a driver exporting both battery.charger.status and the legacy flag that says the same word gets "charging, online", not "charging, online, charging". battery_runtime comes from battery.runtime only — battery.runtime.low is a shutdown threshold, not time left, so it lives at battery_runtime_low and never becomes the hero's "runtime left".
• sources/nut_sim.py — a labelled simulator so the UPS page is developable without a battery. It must never be the default, and every payload it produces says simulated. Its modelled battery.mfr.date is computed relative to today (not a fixed string that ages), so a fresh simulator reads healthy and amber keeps meaning something. Its status follows the real drivers: a below-float pack on mains reports OL CHRG/chg (the glyph's bolt state is keyed on that token, and the model has to be able to produce it) and a battery being drawn from is OB/dchg, never ol. Every command it advertises is implemented and visible in a variable a real driver publishes — the beeper ones set ups.beeper.status (enabled/muted/disabled), and a test window that ends writes ups.test.result: PASSED while an aborted one writes nothing; the load/shutdown commands answer OK without modelling an outlet, because a faked power cut in the history is indistinguishable from a real one. from_env() is the single place the NUT_SIMULATE truthy rule lives — app.py builds the monitor through it, so the assembly and the model cannot disagree about what is enabled.
• prom.py — Prometheus text-exposition parser (in-house; only for engines we don't own).
• rates.py — Δ/window estimator, keyed by resolved series name.
• severity.py — the only place thresholds live: THRESHOLDS/INVERTED per metric, classify() → ok/warn/fail/None, plus attach()/with_severity() to put the verdict beside a value. No measurement, no verdict: None in → None out, never a green pass for an unknown. The verdicts travel additively in payloads — cpu.severity, memory.severity, disks.severity, each UPS card's severity (charge/load/runtime), severity inside rated LLM cells (KV pressure, classified on a percent scale because the engines publish a 0-1 ratio), and severity per host/fleet/UPS row in /api/summary. Pages render the class; they never compare a number to a threshold. There is no gpu.* row and no gpus[].severity (phase 06): these are LLM servers, a busy GPU is the job at every level, so the GPU card and the fleet GPU column render uncoloured — like net.*, whose colourlessness predates it. Removing the verdict is not removing the data: gpu.busy_pct@N / gpu.mem_used_pct@N are still stored and charted. The /disks rows (phase 09) are disk.temp_c (50/70) and disk.percentage_used (80/100) — an SSD at 100 % of its rated wear is a replacement, not a warning — attached per row by sources/disks.py; the sector counters and disk.power_on_hours deliberately have no row, because a non-zero counter is an alarm chip on presence and zero is not an achievement (the /llm preemption-chip rule), so classify("disk.reallocated", 1) is None and the page shows a chip instead of a colour.
• alerts.py — build(settings, job_states, host_payloads, engine_payloads, ups_payload, recent_events) → the "what is wrong right now" list /api/summary serves as alerts ({id, severity, source, message, href, at}, fail first then warn, then by source). A pure projection: no I/O, no store, no thresholds of its own — it reads job states for reachability, the severity dicts for breaches, health.checks[] for UPS conditions. Rules: an unconfigured source never alerts (invariant 3); a source that is ok but older than 3× its own poll interval is warn while a source that stopped answering is fail (the interval is the job's own dashboard's, so disks goes amber after 15 quiet minutes rather than 30 seconds); a dead source gets exactly one line (its stale payload adds nothing) that names it once — the poller's "UpstreamError: " wrapper and the duplicated host name are stripped, because a message that says gateway twice reads as two problems, and the same rule silences the disks echo of a dead agent (the disk job dials the same /snapshot, so its failure carries no new information — while a disks job failing while its host answers, SMART wedged inside the handler, is the only voice that fault has); unknown and missing numbers are silence; one alert per (source, metric) and at most 12, the last being a marker that says how many it hid. This is why the overview can be two panels. There is no disk-health rule and none is wanted: a drive the hardware itself called FAILED is a red chip on its /disks card and a line in that page's event feed, not an overview alert — severity.py rates no sector counter and this module may not have a threshold of its own, so grading a drive here would be the bug, not the fix (phase 09). Its HREF table does carry /disks, so the one alert a disk job can raise — SMART failing while its host answers — links to the page that names the drive.
• store.py — sample(source,host,metric,res,ts) + event + meta, rollups, pruning.
• api.py — GETs everywhere plus the one POST. /api/config is redaction boundary. /api/disks/* (phase 09) is the drive-health pair: /snapshot answers one entry per configured host (disk rows + smart + smartctl_available + per-row severity + a fleet block summed by the API, with a configured-but-never-answered host as an unreachable stub whose counts are None, not 0), and /history serves the disk.*@dev series with unknown names answering as empty series like the other two history endpoints. The configured gate runs before anything that could reach the network — HOST_TARGETS empty is 404 {"configured": false} with no client, no fetch and no disks job in the hub at all (invariant 3), and tests/test_api.py proves it with spies on httpx.AsyncClient.get and collectors.fetch_snapshot. The new surface grows no write of its own — a fourth dashboard must not quietly become a second one, so the app's only POST stays /api/ups/command (invariant 1, pinned by enumerating every served route's verbs). /api/summary is the overview's single call and the cheapest whole-fleet probe there is: the page renders only alerts plus system.fleet/system.hosts, while disks.hosts, llm.engines, ups.ups, the states maps and recent_events stay for programmatic consumers (live tests, an external check, a notifier). system/llm/ups/recent_events are unchanged and alerts (phase 02) and disks (phase 09) are additive — the disks block is fleet_disk_summary verbatim plus one row per host (disks/with_smart/ health_failed/alarm_disks/smartctl_available/failed_devices), so a notifier learns which drive the hardware called FAILED without a second call, and tests/test_api.py pins every one of those key paths, with and without targets configured, so retiring a panel can never shrink the payload. A configured job with no payload yet is listed as a stub (unreachable: true, every value null) in system.hosts/llm.engines exactly as /api/system/snapshot does, so hosts_total counts the machines there are and hosts_live the ones answering.
• app.py — assembly; create_app() takes injectable settings/store so tests build an app with an in-memory DB and no targets.
• web/ — five static pages (/, /system, /disks, /llm, /ups), one stylesheet, charts.js. No framework. app.js's masthead status dots are one per dashboard job — including disks, because the same agent can answer the 10-second utilisation poll and have a wedged SMART pass, and one dot for both would be right about neither. app.js owns the colour vocabulary: Dash.severityFrom(cellOrSeverity) turns a payload verdict into an ok|warn|fail class ('' for anything unrated, so unknown never looks green) and Dash.hero(el, {label, value, unit, digits, severity, sub, bar}) renders/patches a .hero big number (plus an optional .bar) from that class — it never compares a number to a threshold. Dash.severityFor(value, warnAt, failAt) is the documented fallback for values the API does not rate, and no page script calls it any more — / was converted in phase 02, /system in phase 03, /llm in phase 04, /ups in phase 05 and /disks shipped that way from its first commit, so every colour on all five pages comes from a severity verdict the API sent (tests/test_api.py fails if a page script names it again). A value the API should rate gets a row in severity.py, never a bound in JS.

Metric naming: dotted (cpu.usage_pct, mem.used_pct, disk.used_pct@/srv, gpu.busy_pct@0, llm.decode_tokens_per_second, ups.charge_pct), @stream for per-device series — /api/metrics answers with exactly these names. The sample.source column is the dashboard family — host, llm, ups, and disks for the SMART series (disk.temp_c@nvme0n1, disk.power_on_hours@…, disk.reallocated@…, disk.pending@…, disk.uncorrectable@…, disk.percentage_used@…). Those keep their own family instead of sitting beside host's disk.used_pct@/mount because the cadences differ thirty-fold and GET /api/metrics?source=disks has to answer "what does this page have history for". That scheme is the history API's key — treat it as a public interface.

The fleet (real environment, verified 2026-10)

Host	ssh alias	What it gives
driveripper	driveripper-ops	Threadripper 24c/48t + Radeon AI PRO R9700, vLLM :8000, upsd, the dashboards
deskwork	deskwork-ops	Strix Halo (Radeon 8060S iGPU), gufo :8731
gateway	gateway-ops	router, no GPU

• Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05). The designed shape is one dashboards-agent systemd service per host, installed by ops through the deploy/agent-sudoers grant with deploy/install-agent.sh, bound to the address the dashboard dials, its bearer token carried in the HOST_TARGETS URL userinfo; driveripper's upsd serves the LAN on 3051 with the host and laptop ranges allowed by the firewall, never by a upsd.conf line. The old ssh-tunnel script is deleted and neither .env.dev nor .env.dev.example declares a forwarded port — dev dials the same addresses the deployed dashboard does, and scripts/capture-fixtures.sh reads HOST_TARGETS from .env.dev instead of naming hosts of its own (tests/test_deploy_scripts.py fails the day any tunnel name or port returns). Current state: gateway runs its installed service, bound to the address gateway.reeselink.com resolves to, and answers /health + /snapshot on the host (re-verified 2026-10-18, and the token in .env.dev matches /etc/dashboards-agent.token), but the laptop cannot reach its :9120 — and that is now proven to be nobody's firewall here: opening 9120/tcp in that host's own firewalld (a runtime rule, added and reverted in the phase-10 final pass; runtime and permanent port lists are back to their original set) changed nothing, because the SYN never arrives — ss -tan sport = :9120 on the host shows only the LISTEN — while the same laptop reaches :22 and :80 on the same address in milliseconds. The drop is the port-selective inter-VLAN filter on the lab gateway 10.2.0.1, the same one that eats 3051/3493. On driveripper and deskwork the ad-hoc ~/lab-agent/host_agent.py agents are gone (stopped with the old tunnel sessions, nothing replaced them, :9120 is empty on both) because their deploy/agent-sudoers grant is not installed yet, so the installer cannot run there. Every system/disks source is therefore DOWN with its real reason and renders as an unreachable stub row — the designed degradation, recorded rather than hidden — which is also why 6 of the live tests fail on transport while both engines pass. The per-host record, the exact root commands and the one firewall line each host still needs are in data/logs/NOTES-agent-install.md, how far the NUT wire reaches from each network is in data/logs/NOTES-upsd-acl.md, and the end-to-end direct-dial verification (health table, no-tunnel-port proof, live-suite tally, screenshots, and the probes that put the block on the lab gateway's port allow-list rather than in this app) is in data/logs/NOTES-verify.md.
• Engines are reached directly: gufo deskwork.reeselink.com:8731, vLLM driveripper.reeselink.com:8000.
• NUT 2.8.5 on driveripper: configs live in /etc/ups/, not /etc/nut/, and upsd is on port 3051, not 3493. Nothing listens on 3493 at all, so a default client reports Connection refused and it looks like a dead service — ss -ltnH plus printf 'LIST UPS\n' | nc 127.0.0.1 3051 (answers BEGIN LIST UPS) is how that was pinned down. NUT 2.8 has no ACL directive (this Fedora build is --without-wrap; upsd logs ACL in upsd.conf is no longer supported - switch to LISTEN once per line and then ignores it), so per-source restriction is a firewall rule and never an upsd.conf line — and 2.8 does not refuse non-local clients the way older docs suggest: LISTEN 0.0.0.0 3051 + LISTEN :: 3051 with 3051/tcp in the FedoraServer zone answers any client routed to it (reads are unauthenticated; SET/INSTCMD still need an upsd.users account). The host range 10.1.0.0/21 does reach it today (verified from deskwork); the laptop's 10.2.0.0/22 does not — the lab gateway 10.2.0.1 forwards 22/1234/8000-8004 and silently drops 3051/3493 in both families, which no sudo on a fleet host can change, so the laptop cannot yet dial NUT_PORT=3051 (probe evidence and the operator's one rule: data/logs/NOTES-upsd-acl.md). Two UPSes are exported over USB and dialled directly as NUT_HOST + NUT_PORT=3051 (from the laptop that is still the blocked path above; from the host itself, and from the deployed dashboard, it reads live): apc = APC Back-UPS RS 1500MS, cyberpower = CyberPower PR1500LCDRT2U (that is ups.description, the operator's label — the driver's own identity.model answers GX1500U, so the description is what identifies the box). Real-hardware quirks the parsers already handle: battery.mfr.date is the useful date (battery.date is often an unset placeholder — 2001/09/25, slashes and all, on the APC), CyberPower reports battery.mfr.date: CPS (not a date → unknown, never a fake age), neither driver reports a battery temperature or the NUT 2.8 battery.charger.status (so the battery glyph's state has to come from on_battery, and charger_status must stay optional), and ups.test.result: No test initiated is the factory state on both. Both UPSes are recorded verbatim (LIST UPS/LIST VAR/LIST CMD, serials scrubbed) as tests/fixtures/nut_<ups>.vars by ./scripts/capture-fixtures.sh, and the wire quotes a field only when it must — VAR apc ups.status "OL", not three quoted fields.
• Host access is via ssh keys as ops, and passwordless sudo is different on every host — that is what decides how much of the fleet an install can finish unattended. gateway grants (ALL) NOPASSWD: ALL, which is why its agent installed in phase 10 with no deploy/agent-sudoers line present; driveripper grants a command-scoped set (smartctl, upsc, udevadm, the /etc/ups reads, and the tee /etc/ups/upsd.users|upsd.conf + systemctl restart|reload nut-server pair documented in deploy/nut-sudoers); deskwork grants an unrelated set (btrfs scrub, snapper, setfacl /srv/*, osbuild-auto.sh). Neither driveripper nor deskwork can run the installer or visudo, so those two installs — and the firewall-cmd line each still needs — stay with the operator.
• SMART is captured from real hardware; the live agents still cannot read it. smartctl 7.5 is installed on deskwork and driveripper. The ops account (which runs the dev agents) is in no disk group; driveripper grants it (root) NOPASSWD: /usr/sbin/smartctl* and deskwork grants nothing, so the agent's own unprivileged smartctl -i -A -H — it shells the binary and never calls sudo — answers Permission denied (exit 2) on every host and smart stays null end to end: the designed degradation, not a bug. The agents deployed on driveripper and deskwork also predate the SMART pass, so the disks source today yields one row per physical disk (16 on driveripper, 1 on deskwork, vda on gateway) with smart: null, smart_errors: [] and smartctl_available: null — "nobody told us either way", which is why that flag is three-way and not a boolean. gateway's agent is the one exception as of phase 10: it is the installed systemd service, which does know about the SMART pass and answers smartctl_available: false there because smartmontools is not installed — the install it half of the three-way flag, from a host the dashboard cannot yet reach (its firewall). Real data is in the repo now: tests/fixtures/smartctl_driveripper_<dev>.txt is all 16 drives (8 NVMe, 8 spinning) read through that sudo rule, and tests/test_agent.py parses every one — which is what found the parser's one real bug: this binary annotates the attribute table's RAW_VALUE column (35 (Min/Max 34/44) on every Seagate, and one Western Digital writes its hours as 43894h+43m+32.519s), so reading the end of the row lost the temperature of all eight HDDs and one power-on count. The captures are also the first honest look at this fleet's drives: nvme5n1 reports 13 media-and-data-integrity errors and nvme1n1 is at 58 % of its rated wear, so the alarm chip and the amber wear bar on /disks are real features with real data behind them, not demo furniture. Lighting up the live page is two operator actions that travel together: give the agent user device read access — the disk group (sudo usermod -aG disk <agent-user>, which is what deploy/install-agent.sh does on install), not a sudo rule, because the agent must read the device itself — and redeploy agent/host_agent.py (never redeploy without the grant: 16 permission-denied lines would age nothing but are pure noise, while with the grant they are 16 drives' worth of temperature, hours and sector counters). Until then /disks renders the designed degradation: one dimmed card per physical drive, every number n/a, and the one-line why — which is also what gateway shows permanently, being a VM with no physical disk to read and no smartmontools at all. tests/fixtures/smartctl_reference_*.txt stay as what they are: reference shapes (labels verbatim from the deployed binary, values synthetic), kept beside the real captures so the synthetic edge cases stay pinned. Re-record the real ones with SMART_HOSTS="driveripper-ops" ./scripts/capture-fixtures.sh: it tries plain smartctl first (so a fixture can never read healthier than the agent does), falls back to sudo -n, refuses to overwrite good fixtures with unreadable output, and scrubs Serial/WWN/EUI-64 (test_smartctl_captures_carry_no_unscrubbed_serials fails the day it stops).

Commands

uv sync --python 3.14                     # deps + editable install
cp .env.example .env                      # production-shaped config
cp .env.dev.example .env.dev              # dev-only: the real hosts, tokens in URL userinfo

set -a; . ./.env.dev; set +a              # dev dials the fleet the prod way, no tunnels
./scripts/dev-server.sh restart           # keep http://127.0.0.1:8090 up
./scripts/dev-server.sh status            # per-source ok/DOWN
./scripts/dev-server.sh logs

uv run pytest                             # hermetic: 793 passed + 11 live-skipped, no network
uv run pytest --cov                       # coverage gate (98.33%; floor 94%)
uv run pytest -m live --live              # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh             # re-record ground truth (agents + UPS + gufo /slots)
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
                                          # ... plus smartctl per physical disk (needs device read
                                          # access: plain smartctl, else `sudo -n`; scrubs serials)

bash .agents/validate.sh                  # the phase gate: the four checks above (pytest
                                          # --cov --cov-fail-under=94, ruff, pyright, plus a
                                          # node --check of web/static) + repo hygiene.
                                          # Always `uv run` — a bare `python3`/`ruff` on PATH
                                          # tests an interpreter nobody deploys.

Deployment: Containerfile (two-stage, uv sync --frozen, runs as uid 10001 on 0.0.0.0:8090, SQLite under /var/lib/dashboards — put a volume there) and deploy/install-agent.sh per monitored host (systemd unit, own user, --bind/--port/--token/--rotate-token/ --status/--uninstall; it stores the token 0640 root:dashboards-agent — the unit hands the agent the path and the agent opens it as its own user, so that user must be able to read it and no other account may — never puts a token on argv, and runs under ProtectSystem=strict with DevicePolicy=strict reopening only the disk nodes plus the /dev/null family, because strict without /dev/null kills the unit at spawn (status=208/STDIN). No root shell? The one sudoers line that lets ops run the installer is deploy/agent-sudoers; run the script without root and it prints that line with its own resolved path.) The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other service here. The source itself is git@git.reeseapps.com:services/dashboards.git (origin, branch main); there is no CI on it, so bash .agents/validate.sh is the only gate.

Testing policy

Real captures for shape, fakes for behaviour, live for drift.

• Anything that must match an upstream — /proc layouts, vLLM series names, NUT framing — is asserted against a verbatim capture in tests/fixtures/. If the schema drifts, a fixture test fails before a dashboard does. smartctl_driveripper_<dev>.txt is the whole 16-drive fleet read through a smartctl sudo rule (Serial/WWN/EUI-64 scrubbed), and test_the_real_drives_answer_every_number_the_disks_page_prints asserts every number /disks prints comes out of real drive output — that test is what caught the parser reading the end of an attribute row instead of its annotated RAW_VALUE column.
• Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host rebooting mid-poll, a counter reset — is a fake in tests/conftest.py.
• Page payloads that a rewrite must not break are pinned as a table. tests/card_contract.py holds the dotted /api/system/snapshot paths the /system hero cards read (CARD_METRICS, GPU_METRICS, HOST_SHAPES); tests/test_system_contract.py asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and test_every_host_serves_the_hero_card_contract asserts the same table against the live agents — so a rename fails a test, and a card that shows n/a cannot be "fixed" by inventing a field. network carries no severity on purpose (throughput has no good/bad) and neither does a GPU row (phase 06: a busy GPU is the job on an LLM server) — the contract pins both absences. tests/ups_contract.py is the same device for the /ups hero row and battery glyph (HERO_FIELDS, UPS_SHAPES, assert_ups_hero_contract): tests/test_ups_contract.py drives it from tests/fixtures/nut_<ups>.vars — verbatim LIST UPS/LIST VAR/LIST CMD replies from the two real UPSes — and test_ups_hero_fields_are_present_on_the_real_batteries drives it against live upsd. The captures are what make the driver disagreements testable: one of them answers battery.mfr.date: CPS, neither exports battery.temperature or battery.charger.status. tests/slots_contract.py is the same device for gufo's /slots rows, which the /llm slot grid (phase 07) reads (assert_slot_shape, required/optional field tables, duplicate slot ids, the next_token[0] entries): hermetically from tests/fixtures/slots_gufo.json — the verbatim /slots reply, idle slots and all, captured whenever GUFO_BASE_URL is set — and live via test_gufo_slots_answer_the_pinned_shape. That fixture exists because its absence is how the deleted slot table rendered dashes forever: its field names (slot.state, slot.n_prompt, slot.decode_per_second) were a memory of an upstream, not a capture. The contract names one canonical key per fact and the grid renders that one: the capture answers id_task and task_id for the same request, so CANONICAL_TASK_ID_FIELD decides and the alias is captured but rendered by nothing — a rename fails a test instead of becoming a silent a ?? b chain in the page. tests/test_api.py reads the served SLOT_FIELDS table and resolves every key it names against the capture, in both directions. A busy slot's prompt is the user's in-flight request text, so the capture script blanks it and the page never reads it.
• live (tests/test_live_fleet.py) re-runs the production code paths against the real fleet, engines, and upsd. Skipped unless --live; each test skips when its source is unset, so partial fleet availability is normal. Live tests must not write — a battery test on hardware something is plugged into is not a test.
• The agent's real-/proc test (tests/test_agent.py) touches only the box running pytest, so it stays in the default run and is not marked live.
• Coverage gate: 80% is the repo floor, the phase gate in .agents/validate.sh enforces 94% (--cov-fail-under), and ~98% is where we are; keep it there. New/modified Python still has to clear >90% on its own. API tests use create_app() with an in-memory DB, never a real target, and never a real upstream: /api/ups/command's send is proven against a scripted nut.NutClient (ScriptedNutClient in tests/test_api.py) because the live suite is read-only — that fake is the only place the write reaches a "driver".

UI conventions

Hand-rolled SVG (charts.js), one stylesheet, dark/light toggle, Δ<window> labels on measured rates, n/a for the unknown, SIMULATED badges on modelled data, per-source status dots in the header. Every chart names itself for assistive tech: pages pass ariaLabel to LineChart, which puts role="img" + aria-label on the <svg> (an unlabelled plot is silence, and the legend is a sibling that labels nothing). Every LineChart on every page has one — /system's three per-host slots, /disks's two per-host plots, /llm's four trend plots, /ups's four battery plots — each pinned by tests/test_api.py, so a chart that ships anonymous fails a test. Window chips (5m…30d on /system and /llm, 1h…90d on /ups and /disks; the API accepts 5m…90d — SMART lands every 300 s, so a 5-minute window on /disks is one dot). A panel that cannot get data says why in the panel — the operator should never have to open devtools to learn that a source is unconfigured. Escape anything upstream-provided through DashCharts.escapeHtml.

Flavour text is not helpful information. A page does not narrate its own data pipeline or its own design; copy exists only where the operator needs it to act — a why-this-panel-is-empty sentence, a provenance tag, a measured Δ<window> label, a chart's unit, a SIMULATED badge, an error that names what failed. The audit is one question per string: would the operator act differently without it? No → delete it, container and all. Deleted in phase 08 (the cited examples, from TODO.md:7): "every number is scraped from the box it belongs to · null renders as n/a" (/'s footer), "agent-reported values; rates are measured over the Δ window in the footer, and each panel's charts cover only that host" (/system's Per host sub-line), and the same genre — /system's "numbers are measured, not inferred", /llm's "each throughput number carries its provenance", /ups's "commands are the only write path in this app", each event feed's content caption (the empty-state sentence already says it), and the design clauses inside empty-state prose. What survived the audit did so because it is a measurement or a why: every Δ<window> footnote, provenance() on each rate, fleet.seams, sums only engines answering now, the queued-request note under the slot grid, the SIMULATED banners, and the setup instructions in every empty state. tests/test_api.py pins the deleted sentences absent from every served page and script (a sentence that migrates to another page is the same slop), the kept functional strings present, and that no removed sub-line left an empty .sub/.faint container behind.

The overview is exactly two panels — Warnings & errors (ul.alerts, one row per /api/summary alert: verdict class, source linked to its dashboard via the alert's own href, message, age) and Fleet. A non-empty list floats the alerts panel above the fleet table (#overview-panels[data-state] swaps them); an empty one is an explicit "all clear" state below it, because an empty list is a message, not a blank box. A host that is not answering keeps its row, dimmed with the shared .stale class and reading n/a, so the table never implies last good numbers are live and never silently loses a machine. No summary cards and no raw event feed survive on / — tests/test_api.py pins the two-panel shape against the served HTML.

Nothing waits for a writer that never came: tests/test_api.py fails if a page keeps an id that no page script, no shared script, no stylesheet rule and no aria-*/for attribute references — the orphan a deleted renderer leaves behind, caught on served bytes because there is no JS test runner (a locked anchor). Same rule for dash.css: delete the rules a removed panel made dead (.grid.wide went with the overview's fleet+events row); component vocabulary the pages are still being converted to (.hero-bar, the grid and bar variants) stays even while only one page uses it.

A big number is Dash.hero() on .hero / .hero-bar (dash.css): 2.4rem tabular numerals, muted unit, mono .sub footnote (Δ10.0s, "at current load"), coloured by the payload's severity through the existing .ok / .warn / .fail classes — there is no .sev-* family and no second palette, and tests/test_api.py fails if either appears or if a CSS rule reaches for a token that :root does not declare.

The /disks card is one panel per physical drive, led by identity and verdict, not by a hero (phase 09): a drive has no one-number story the way a UPS has charge %, so the card opens with the device path (/dev/nvme0n1) and the drive's own word (OK / FAILED / unknown, a .chip coloured only by severity.health — an unreadable drive gets grey unknown, never a green pass), then the identity line (model · size · SSD|HDD, nothing invented where a field is absent) and three equal .metric cells: Temp, Power-on, Wear. Colour comes from severity.temp_c / severity.percentage_used and nowhere else; power_on_hours is an odometer, written with a null verdict, so it cannot read green at 100 h or red at 60 000 h. The sector counters are .chip.fail alarm chips that exist only while non-zero (the /llm preemption rule, and why severity.py rates no counter): a drive at zero draws no chip row at all, because a permanent "0 pending sectors" line trains the eye to ignore it and a green one would call a quiet drive an achievement. Per host the three-way smartctl_available decides one .banner.info why-line and nothing else — false says install smartmontools and grant the disk group (the agent said so, so the page may), null prints the agent's own smart_errors line or names both fixes without claiming to know which half is missing, and an agent that is not answering says that instead — and in every case the drive grid takes the shared .stale dim while the host card stays uncoloured (invariant 3: a missing binary is unconfigured, not broken, and a red panel here would train the operator to ignore red on the drives that do report). Nothing on the page is per partition or per mount: the rows are the agent's whole-/sys/block list, and a filesystem field naming its way into page-disks.js fails a test. The fleet header renders fleet_disk_summary() — hidden below two hosts (one box's sums duplicate its own cards) and coloured nowhere, because a count of drives is not a graded number — and the per-host Charts disclosure follows /system exactly: collapsed, lazy-built, one /api/disks/history request per host per refresh naming disk.temp_c@dev and disk.percentage_used@dev for every drive (the endpoint answers unknown names as empty series on purpose), the wear plot created only once some drive publishes the field, legends naming drives because the card heading already says who, and pruneHosts taking the cards, the toggle and the plots off the page together.

A cell-scale trend is DashCharts.sparkline() — the llm_monitor drawSpark idiom, so a percent metric is plotted against fixedMax (never autoscaled: an 8 % KV window that once held a 92 % spike must not read flat, and the axis gets top/bottom scale marks so "a third of the way up" says a third of what). align: 'right' pins the newest sample right and grows the line leftwards inside a real span in seconds — x stays time, not sample index. Stroke colour comes from the payload severity via the --ok/--warn/--fail tokens only; an unrated series draws a faint --text-faint stroke with no fill, because a shape is not a verdict. Below minPoints (2) it prints no history rather than drawing a confident line through one point, labels: {current, max} renders the now / max foot through the caller's format, and there is no animation at all (prefers-reduced-motion holds by construction). minSpan is the other half of the same honesty for count metrics: history arrives as rollups, so eight seconds of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for a queue nobody was in. /llm's count rows moved to the phase-07 stats strip (numbers, not sparks), so no page passes minSpan today and drawSparks() still plumbs it through — a count sparkline that comes back must set it, and the component's own contract test keeps the option honest (test_sparkline_ships_the_fixed_scale_and_verdict_options).

A battery is DashCharts.battery(container, {charge, severity, state, label}) — one inline SVG whose fill is the charge: 1 viewBox unit per percent, clamped to 0–100 and never rounded up (a 2 % cell must look nearly empty, because a reassuring battery above a flat one is a lie), clipped to the plate so 0 % leaves an empty shell rather than a stray pixel. Colour comes only from the payload severity token; a measured charge with no verdict draws faint grey at half strength (sparkline's rule — a shape is not a verdict) and never an empty shell ("0 %" and "not rated" are different sentences). Every state has a shape as well as a colour, so the glyph survives monochrome: solid shell online, bolt charging, dashed outline onbattery, no fill with a dash pattern through the body unknown; a state word it does not know degrades to unknown, never to a confident "on line". Accessible by default — role="img" + <title> + aria-label built from the inputs ("apc: Charge 100 percent, on line"), because the glyph carries information the markup does not say in words — and a page that prints that sentence next to it passes decorative: true to make the <svg> aria-hidden rather than say it twice. No animation and no ResizeObserver: one viewBox + preserveAspectRatio scaled by .battery in dash.css, so the hero row gets a big cell on a desktop and a small one on a phone.

The /ups card is led by the glyph and three heroes (phase 05): the battery on the left, then Charge %, Load % and Runtime left on .heroes at the shared 2.4rem, coloured from severity.charge_pct / severity.load_pct / severity.battery_runtime and nothing else. Charge carries no .bar — the glyph beside it is that value drawn as a shape, and one value drawn twice in one row is the pattern this app avoids (Load keeps its bar; nothing else draws it). Its .sub lines are the context each number needs and nothing more: 24 V nominal · 27.3 V for the pack, 128 W of 900 W when the driver publishes both realpower figures (a wattage inferred from a percent against another model's rating would be an invented number), at the current load for runtime. Runtime prose is runtimeText()'s (1h 32m, 10m 21s) and an estimate the driver never made goes into Dash.hero() as null, so it prints faint and uncoloured rather than as 0m. The readings the hero row displaced — line in, output, battery volts and amps, battery °C — stay on the card, demoted to .ups-electrical: muted, .95rem, never folded away, because they are what you reach for while the UPS is on battery, and battery °C reads n/a on both real drivers here. state is derived in the page from on_battery → charger_status (chg charges; dchg/rec are the server's own on-battery tokens; full is on line, not charging — a rested floating battery reports it) with one fallback: a state chip the API rated ok, because nut.states() reserves ok for on-line/fully-charged and neither real driver here publishes battery.charger.status at all. ups.status itself is never re-parsed in JS (tests/test_ups_contract.py). /ups passes decorative: true, because the charge hero and the state chips already say the sentence. The glyph repaints only when charge / verdict / state change (drawGlyph() keys a data-drawn triple): the charge float jitters every NUT_INTERVAL_SECONDS, and rebuilding the <svg> on that re-announces an accessible name for a difference no eye can see.

An event row names its subject once. Every feed (/system, /disks, /llm, /ups) renders the message through Dash.eventText(event), which drops the UpstreamError: wrapper and a leading duplicated subject the way alerts._clean_error() does for the overview — poller.py messages start with their own label, and each row already prints that label as a chip, so "gufo gufo is answering again" reads as two problems. A name in the middle of a message stays: the UPS rows used to replace() it anywhere and mangled "Quick battery test sent to apc: OK". This is presentation only — the stored event and /api/events keep every word, and the raw error string still surfaces in /api/health.

An event feed is a window, not a printout (Dash.eventsPager(el, events, opts) in app.js, one helper with four call sites (/system, /disks, /llm, /ups) — a pager that works on one page is the bug this shape prevents). ul.events is a bounded 320px scroll container (overscroll-behavior: contain) and the pager puts EVENT_PAGE_SIZE = 50 events in it, with a .events-pager strip under the list: 1–50 of 200 between ‹ prev and next ›, both aria-labelled for direction (the feed is newest-first, so prev walks towards now) and disabled at the ends. The strip is hidden when the feed fits on one page — every row on screen is the whole story, and a disabled arrow says "more history" when there is none. Each page keeps its own row(event) (so /ups keeps its longer timestamp and runtime footnote, and escaping stays the page's job — the pager has no default renderer and says so in a row when a caller forgets one) and its own empty sentence, so the loading… and empty states read exactly as they did before paging existed. Two numbers are load-bearing and pinned together: the fetch (EVENT_FEED_LIMIT = 200, /api/events?limit=…) must exceed one page, because a limit at or below EVENT_PAGE_SIZE is a pager that silently never appears; /api/events itself is unchanged — no page/limit vocabulary grew for one page's layout. Paging state lives in a WeakMap keyed by the <ul>: it survives a poll (so page 3 is still there two seconds later — resetting every refresh would make page 2 unreadable) and resets to page 1 when the event set changes (an operator reading page 3 of yesterday's history must not land on page 3 of a list that just changed). Rows are re-written only when their markup changed, which is what keeps the reader's scroll position inside the container.

The /system host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the .heroes grid; tests/test_api.py pins the slots and their order on the served script). Colour comes from the payload's severity dicts and nowhere else; a missing number or verdict — gateway's CPU temp, any host's absent GPU — prints n/a uncoloured. NET is one slot with two lines (.hero-lines, ↓ in / ↑ out) and is deliberately colourless: throughput has no good/bad, so network carries no severity. The GPU slot holds one hero per card whose .sub carries model, VRAM used/total, the honest APU pool view and the card's temperature (gpus[].temp_c — the agent's max over that card's hwmon labels, the same value stored as gpu.temp_c@N, so the hero and the history cannot disagree), and it is colourless for the same reason (phase 06): a 95 %-busy card on an LLM server is the job, so the hero carries no severity and no utilisation bar — a bar under a verdictless number would be a verdict by shape. A card whose driver exposes no temperature input prints no temperature in that footnote rather than n/a or 0 °C: "where you can" is the operator's wording, and n/a is reserved for numbers the card is supposed to have. Everything not glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU power and the per-label temp breakdown, filesystems, processes — folds into a collapsed "More detail" <details>: a simpler surface, not a less complete page (the table keeps its Temp column; this was a promotion, not a de-duplication).

A Δ<window> label is a measurement, never a config value: sources/host.py sends the window the estimator actually used as the payload's window_seconds, and /system's footnote prints those (a range when the hosts disagree) instead of the poll interval, which only coincides with them at the defaults. /llm's hero .sub prints the rate cell's own window through the same provenance() tag, and the hero's title carries the exact source string — "78 tok/s" must always say whether it was scraped, measured Δcounter/Δt, or estimated (llm_monitor's setThroughput idiom; dropping the tag to look cleaner is a regression, not a simplification).

The /llm page reads per engine (phase 04, decided in .agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md, laid out by phase 07): a row of two Dash.hero() throughput numbers (decode, prompt — deliberately uncoloured, because 0 tok/s on an idle server is healthy and severity.py has no throughput row) and the latency rows, each a mean with a mono p95 … foot (no hover; p95 needs no stored series) and a sparkline() fixed at 160px so rows line up. Rows are engine-scoped by the only flag on both components: gufo never grows a TTFT mystery, vLLM never a spec-accept hole.

The stats strip (#strip, phase 07 task 03) is the summary; the panels are the detail, and no row is printed in both — the UPS hero row's "one value drawn twice" rule, widened to two components. One .card.stat per configured engine between the fleet block and the panels: the reachability dot, the engine name and the age stamp in the head (same distinction every card draws — the dot speaks reachability, the numbers never do), then the KV-cache bar — the one cell on the page with a verdict, so the fill and a breach's text take their colour from the payload's severity through .bar.ok/.warn/.fail and the ratio→percent conversion the panel's cell used to make, with ok staying quiet ink and a missing gauge an empty bar plus n/a — then prompt tok/s / decode tok/s with their Δ<window> tags, total tokens (lifetime, so its source is the tooltip), the sessions this engine reports, prefix cache hit, spec accept (gufo) and queue time with its p95 … foot (vLLM). STRIP_ROWS is the whole table and its key is the cellValue() key, so every number on it is a cell of /api/llm/snapshot: no field was invented in sources/llm.py for this card, and there is deliberately no peak-ctx row (phase assumption). The session row is engine-shaped — vLLM publishes running/waiting counts, gufo's is slots busy, counted from the sessions.list rows /slots answered (is_processing, the shape pinned by tests/slots_contract.py); when /slots did not answer there are no rows to count, so it falls back to the published sessions.active carrying the API's own sessions.detail sentence rather than reporting 0 busy slots from no data (rule 2). preemptions / device_lost are alarm chips that exist only while non-zero (.chip.fail, container emptied and hidden at zero — the rule is the presence of an incident, never a graded threshold) and they live on this card, not in a permanent cell; the lifetime token counter left the panel's footnote line for the strip's row. One /api/llm/history request per refresh carries every chart and spark series (HISTORY_SERIES = the four chart series + SPARK_CELLS; the endpoint answers unknown names as empty series on purpose), and the two series phase 04 added (llm.tpot_seconds vLLM-only, llm.spec_accept_rate gufo-only) stay stored additively in sources/llm.py HISTORY_METRICS. Above the strip sits the fleet block (#fleet, phase 07: "I'm missing the aggregate throughput at the top") — ALL ENGINES, a reachability dot, an N of M engines tag, the note sums only engines answering now, then running / waiting / prompt tok/s / decode tok/s / total tokens. FLEET_FIELDS in page-llm.js is a list of fleet_totals() keys and nothing else: the page sums no number of its own, a dead engine is dropped from every total presented as current (its own stale-labelled panel still shows its last good values), and the tag counts the answering engines, names the missing one in its tooltip and takes the shared .stale dim while the sums are partial. It hides below two engines — a cross-engine sum of one engine duplicates that engine's own panel — and it carries no severity colour anywhere: fleet_totals has no severity to render (the summed-KV bar is gone for that reason) and severity.py rates no throughput or queue depth, so the numbers stay ink and only the dot speaks reachability. fleet.seams — the API's own note on how the two engines count prompt tokens differently — is the prompt sum's tooltip rather than another line of ink. The four trend charts stay but at height 110 below the panels, each with an ariaLabel. Panels sit at their own height (#engines carries .items-start): a two-hero gufo card next to a latency-row vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". An engine that leaves the payload leaves both components (pruneEngines), never a stale card in either grid.

The slot grid (llm_monitor's, phase 07 task 04) belongs to the gufo panel, under its sessions (GET /slots) label. One card per row of sessions.list, keyed by the row's own id in state.slots: built once by slotCard(), patched in place afterwards, and reordered by appendChild of the existing node so DOM order stays slot order — rebuilding a busy slot's card every 2 s poll re-announces the whole card to assistive tech for a difference of a few tokens. SLOT_FIELDS is the whole table and each cell names exactly one /slots key (path: 'next_token' reads next_token[0], where the capture puts n_remain); slotValue() is the only place a row is indexed, so tests/test_api.py can resolve every printed field against tests/fixtures/slots_gufo.json and a rename fails there rather than printing dashes or a plausible wrong number. No per-slot speed cell: the capture publishes no per-slot rate key and a page-side Δn_decoded/Δt would be a measurement with no estimator, no Δ<window> label and no source string (rule 2) — the engine's rate lives on its strip card and panel. llama.cpp's -1 sentinels print a word or nothing: id_task: -1 is idle, n_remain: -1 is n/a, never a negative countdown and never the 0 that reads as one token from done. The busy marker is a border, a glyph (○/●) and the word processing in the brand --accent, never an ok|warn|fail token — a slot generating tokens is the job (phase 06's GPU reasoning), so severity.py rates no slot and the stylesheet section contains no verdict token. The empty state is a sentence, never a card of nothing, and it distinguishes the two cases: no /slots answer prints the API's own sessions.detail, an answer with no rows prints engine reports 0 slots; sessions.source is the gate. The note under the grid stays because a queued (deferred) request is not in /slots — the waiting row counts those. vLLM's panel gets one line (exposes no per-slot JSON) with sessions.detail as its tooltip and no grid: a component of permanent n/as for a server that cannot answer is the dead-cell anti-pattern. The gate is the engine name (SLOT_ENGINES), not row count, so an engine that has not answered yet still shows its label and its honest empty sentence.

Charts belong to the host they chart. /system has two sections — the host panels and the transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals panel and the host/metric detail picker are deleted, along with the #chart-* / #detail-* ids and the select / label.field styling only those pickers used. Each panel stacks its own .host-charts — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where the GPU slot is created and destroyed with host.gpus, so gateway shows neither a phantom card nor a hole. That stack is collapsed behind a per-host Charts <details> (phase 06: "only visible if expanded… there will be lots of hosts"): the default /system view is the hero row, the summary carries the window label the stack will draw (the chips themselves stay in the page head, one control per page), and drawHostCharts() is the single place that decides whether to paint — so nothing is built or drawn while the stack is shut (LineChart inside a display:none container measures 0 px, and rebuilding three charts per host every poll for plots nobody sees is the waste this removes). The history request is deliberately unchanged: one per host per refresh, naming every series the panel could draw, and its response is cached per host so the first expand paints from data in hand rather than waiting a poll; a cached response from another window is re-fetched instead of being drawn under the wrong label. Expanding one panel must not stretch its collapsed siblings into blank cards, so #hosts carries the existing .grid.items-start. One /api/system/history request per host per refresh (hosts=<that host>&metrics=<that panel's series>&range=…, pinned by test_every_system_chart_is_drawn_inside_its_own_host_panel and the collapse by test_system_history_charts_are_collapsed_behind_a_toggle): unknown metrics come back as empty series on purpose, so a panel asks for every series it could draw and asks once. A legend here names metrics (or one GPU's model), never hosts — the panel heading already says who this is, and a legend that grew a second host is the bug this replaced. A host that stops reporting loses its panel, its Charts toggle and its charts (pruneHosts deletes the card plus the payload/chart/history maps), never a chart of nothing or an orphan control in the grid. Per-core visuals are gone from /system and renderCores is deleted from charts.js with its last caller. drawHostChart() shades a chart only when one series has points — two or three filled areas stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy memory plot; the cpu.usage_pct@coreN series is still stored and still queryable from /api/system/history — removing the plot is not removing the data.

Known gaps / next work

1. UPS commands work, and the account can now cut power — the app's gates are the control. Three defects were cleared on 2026-10-17, each verified against real hardware. (a) NutClient.login() sent USER/PASS, which the deployed upsd 2.8.5 does not implement — each answered ERR UNKNOWN-COMMAND, so _read_line() raised and POST /api/ups/command could only 502 before instcmd(), credentials or not. It now sends USERNAME/PASSWORD, the pair the server's own HELP advertises (HELP VER PROTVER GET LIST SET INSTCMD LOGIN LOGOUT USERNAME PASSWORD STARTTLS), pinned by test_login_uses_the_verbs_upsd_2_8_5_answers_to — the fake upsd has no USER/PASS branch, so a regression answers ERR UNKNOWN-COMMAND exactly like the real thing. (b) 12 of the 23 commands the two real UPSes advertise had no COMMANDS entry, and available_commands() drops anything the catalog misses — the page silently offered less than the hardware does. Every advertised command is catalogued now, each with a risk tier: test_every_command_the_hardware_advertises_is_catalogued fails if a capture gains a command the catalog does not know, and test_every_command_that_can_cut_power_is_in_the_dangerous_tier keeps load.off, driver.killpower and the shutdown.* sequence in the tier that types DANGER and is refused on battery. shutdown.stop is the exception the name hides — it cancels a shutdown, so it is safe, and the test says so in words. (c) [dashboards] in /etc/ups/upsd.users is instcmds = all at the operator's request, so this app may send every advertised command on both UPSes, power-cutting ones included. .env.dev (mode 0600, gitignored) carries the credentials. What remains to know:
   • Proven live, and reversibly. beeper.enable returned OK and ups.beeper.status read enabled on the APC afterwards; driver.killpower without the word DANGER still answers 422 and never reaches the wire. A battery test has deliberately not been run — it puts real load on a battery things are plugged into, which stays the operator's call.
   • enabled: true cannot mean "this account may send it". available_commands() filters on LIST CMD, which upsd serves unfiltered, and NUT has no call for a user's privileges — so the wire cannot tell the page what the account is granted. With instcmds = all the two agree by configuration rather than by check, which is precisely why the risk tiers and the typed DANGER are load-bearing and not decoration.
   • Pre-existing upsd log noise: every upsd start logs Unknown upsmon type = (20 occurrences over 12 starts, oldest before this work). Harmless — upsmon logs into both UPSes right after it — but unexplained. Editing upsd.users at all: never grant actions = -, and never write it blind — the running upsmon authenticates as its own account from that same file for both UPSes, so dropping it means no orderly power-down. Copy the untouched sections from the live bytes and cmp them before writing; tee truncates in place, so owner and mode survive.
2. GPU coverage is AMD only. host_agent.gpus() reads amdgpu DRM sysfs (gpu_busy_percent, mem_info_vram_*, hwmon temps, clocks). That covers the current fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at all — needs a nvidia-smi/NVML or DCGM path, and a fixture from a real card before the field map is written.
3. UPS history depth. Events and the numeric series are stored; long-window battery health (capacity fade over months) has the rollups but no dedicated view.
4. The dev DB is big enough to refuse a write now and then. The dev store (~610 MB, WAL grown past 140 MB) logs store-error: OperationalError: database is locked a few times an hour across system/llm/ups; the poller handles it exactly as invariant 4 requires — the source goes red with the real reason, ages, and writes recording when the next poll lands — so nothing freezes, but those polls' samples are lost. The one cause this repo can prove is two servers on one file, which is why scripts/dev-server.sh stop waits for the old pid to die (its comment records the incident). A controlled 25 s external read transaction during this pass produced none, so checkpoint starvation is not yet shown. Retention/wal_checkpoint tuning for a long-running store is its own phase, not a phase-09 follow-up.

Locked decisions (and why)

• SQLite, not Prometheus/Grafana. Handful of boxes, 1–10s resolution, one file the container owns. Rollups keep a year of charts at a few hundred MB.
• Our own stdlib host agent, not node_exporter. One JSON snapshot with exactly the fields the page needs (GPU model strings, per-mount usage, temperature labels) versus ~1500 series and a re-labeling layer; no dependency to install, ProtectSystem=strict friendly, and read-only by construction. scripts/capture-node-exporter.sh remains to compare ground truth if that is ever revisited.
• Our own NUT client, not pynut/upsc. The protocol is ~10 line commands, the third-party libs are subprocess-based or stalled, and we need instant commands with explicit gating.
• In-house Prometheus parser. Only two upstreams use the format; keeps prometheus_client out of the dependency set.
• Port 8090, inherited from the llm_monitor prototype this started from: 8000 is vLLM and 8731 is gufo in this fleet.
• .env is a seed, never an override — container/systemd env must win.

Definition of done (per change)

The first three are exactly what bash .agents/validate.sh (the phase gate) checks, so no phase may be called done on a subset of them.

uv run pytest green · coverage not down · ruff + pyright clean · uv pip list --outdated empty if deps moved · dev server restarted and /api/health sane · UI change screenshotted in a browser against live data · fixtures re-captured if an upstream contract changed · this file and README.md updated when a rule, command, gap, or fleet fact changed.