# AGENTS.md — homelab dashboards The plan, the rules, and the environment facts an agent needs to work on this repo without re-deriving them. `README.md` is the operator-facing how-to; this file is the why, the invariants, and the state of the fleet. ## Working protocol This project runs the `.agents/phases/` phased-execution pipeline. 1. Always read `.agents/PLAN.md` first to understand the project context and goals. 2. Follow the phased execution protocol in `.agents/phases/`. 3. Never modify `.agents/PLAN.md` or any files in `.agents/phases/complete/`. 4. If you need to update any file in `.agents/phases/todo/`, you must ask the user for permission first. 5. Strictly adhere to the **LOCKED DECISIONS** listed in `.agents/PLAN.md`. Everything below this section is the project's own content: what the app is, the rules that keep its numbers honest, and the state of the fleet it watches. ## What this is Four dashboards for one homelab, behind one domain, in one process, backed by one SQLite file — plus an overview page that says which of the four you need: | Route | Question it answers | Source | | --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- | | `/` | Is anything wrong, and where do I look first? | all four, in one request | | `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host | | `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) | | `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` | | `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) | History is a first-class feature, not a sparkline afterthought: every number worth graphing is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`. None of it is public. One admin signs in at `/login` (the only page an anonymous visitor reaches), and `/settings` — `⚙ Account` in the masthead — is where that session is read, ended, and re-credentialled. Every page above, `/settings`, and every `/api` route but the four named in invariant 12 require the session. ## Non-negotiable rules These are the things that make the numbers trustworthy. Breaking one is a bug even if the page still renders. 1. **Read-only by construction, with exactly one exception.** Nothing in this app writes to an upstream except `POST /api/ups/command`. That endpoint is gated four times (`NUT_ALLOW_ACTIONS`, not `DASH_READ_ONLY`, the upsd.users account actually having the command, and a UI confirmation — `DANGER` for anything that can cut power) and the server re-checks every gate on every request. Hiding a button is not a safety control. Commands that would cut output are refused outright while the UPS is already on battery. Since phase 11 there are two more in front of it — a session, and this session's CSRF token for a cookie write (invariant 12) — and they are additive: `tests/test_auth_api.py` proves the `DANGER` requirement and the wire send still behave exactly as they did without them. 2. **Never invent a number.** A value the upstream does not publish renders as `n/a`, never as `0`. Every derived value carries the `source` string it came from and `derived: true` so the UI can print *how* it was obtained (`computed:Δtokens/Δt`, `Δ10.0s`). Aggregating two different series into one rate is the failure mode this rule exists to catch — the rate estimator is keyed by the *resolved* upstream series name for exactly that reason. 3. **Unconfigured is not broken.** A source with no env config answers `404 {"configured": false}` **without dialling**, and the page says "not configured". A red panel for a server the operator never installed destroys the signal from the panels that matter. 4. **One dead source must not blink another — and nothing may stop polling in silence.** Each source is its own asyncio task with its own timeout and health record (`poller.py`). A failed poll keeps the last good payload and marks it stale in the UI instead of zeroing it — on `/` too, where a host that has never answered still gets a fleet row (`unreachable: true`, dimmed, every number `n/a`) because dropping the row would turn a dead agent into "never configured". Transitions write events; repeated failure writes one line, not 500. **The task is the source's only poller, so no exception may end it**: a store that refuses an otherwise good poll is a *reported* failure (`ok: false`, the real reason in `error`, `last_ok` frozen so the page ages the numbers, one `store-error` event then one `recording` event on recovery), and anything that escapes the poll body is logged and the loop continues. Four dev-server sources once froze behind an unretrieved SQLite exception while `/api/health` went on calling them `ok` — a frozen source that looks healthy is the one failure mode worse than an outage. 5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app; charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with no outbound access — a CDN reference is a request that never comes back. 6. **Never leak a credential.** Target URLs may carry a userinfo token (`parse_targets` moves it into an `Authorization` header, so `Target.url` ends up clean). Nothing that leaves the process may carry one: `/api/config` serves `Target.display_url` and `config.strip_userinfo`-redacted engine URLs, and every URL inside an `UpstreamError` message is redacted too — those strings go into the health record *and* the `event` table, which `/api/events` serves to the browser and `page-system.js` prints. No token, password, or personal hostname goes into tracked files, fixtures, or the browser. **The admin's own credential is the same case** (phase 11): `auth.py` records nothing and answers `False`/`None` to every rejection, because the *reason* a login was refused is a verifier, a salt, or a session token — and this app turns reasons into health records and `event` rows that a browser reads. Its one `logging` call and its two `raise` sites (the fail-closed startup and the empty-new-password refusal) carry **fixed** operator text with nothing interpolated, not even a prefix of a hash; `tests/test_auth.py` pins that structurally, so the day someone adds `logger.warning(f"bad verifier {v}")` a test fails instead of a log line. 7. **Verify against the deployed upstream; never guess a metric name.** Field maps in `sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT 2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with `scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing exporters) rather than trusting a README or a memory of an upstream. 8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale. Check after touching dependencies: `uv pip list --outdated` must come back empty. Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile `python3.14`) — do not lower these to make something else work. 9. **SQLite owns the history and must stay bounded.** Writes go through `store.py`; new metrics need a rollup path and a retention story, or a year of charts becomes a disk emergency. Put the DB on a volume in deployment. 10. **The dev server stays up while work happens** (`./scripts/dev-server.sh restart`, http://127.0.0.1:8090), and UI changes are verified in a real browser with a screenshot before they are called done. `data/logs/*.png` is where those land. 11. **The dashboard never starts open, and the stored password beats the environment** (phase 11). The admin verifier and the session-signing key are durable rows in the SQLite `meta` table (`auth.verifier`, `auth.session_secret`) and are the only place either secret lives; `DASH_ADMIN_PASSWORD` is a **seed** that hashes into the row on the first boot of a database and is ignored thereafter (one warning line, value never quoted), so a password is changeable without a redeploy and no container env can roll it back. A server with neither a stored verifier nor the seed **refuses to start** — a `RuntimeError` out of the lifespan, a non-zero exit, and a message naming the variable, the meta key, and the `sqlite3 "DELETE FROM meta WHERE key IN ('auth.verifier','auth.session_secret')"` reset command. "Start open with a warning" is not an option: the domain is public and the fleet's hostnames and UPS controls are behind that login. `testpass` is the local-testing password (dev config and the hermetic suite) and never a production fallback. Invariant 12 is the request-time half; this one is why there is always a credential to check against. 12. **The gate is default-deny and its public surface is four entries** (phase 11). `app.py`'s `PUBLIC_PATHS` (`/login`, `/api/auth/login`, `/api/health`) plus `PUBLIC_PREFIXES` (`/static/`) are the only things an anonymous caller can reach — every page, every `/api` data route, and `/api/auth/status`/`logout` require a session read from the `dash_session` cookie or an `Authorization: Bearer` token. `/api/health` is the one data route that stays open on purpose (the Containerfile `HEALTHCHECK` and `./scripts/dev-server.sh status` poll it unsigned, and it leaks nothing); its shape is now a contract with deployment, which means **nothing may be added to it without re-deciding that** — a new field there is a field the whole internet reads. Adding a path to that list is a security decision, and `tests/test_auth_api.py::test_every_route_that_is_not_public_requires_a_session` walks the **served** route table so a route that forgets the gate fails a test instead of shipping — the middleware guards by default, which is why no new route has to remember to. Rules the gate keeps and each has a test: a page `GET` gets `303` to `/login?next=…` (query string included) and everything else the one JSON refusal, because a redirect a script cannot render is not an answer; every answer carries `no-store`; `?next=` passes `auth.safe_next` and is never followed raw (an open redirect on a login page is a phishing kit with a homelab theme); a write additionally needs a matching `Origin`/`Sec-Fetch-Site` and — when it came in on the **cookie** — this session's `X-CSRF-Token`, because a bearer caller is something no other site can forge. A process whose lifespan never produced `ctx.auth` answers `503` for **everything, public paths included**: "healthy" from a server that cannot say who its admin is is the invariant-4 failure mode, and a login form that cannot verify a password is a worse promise than no form. `POST /api/ups/command` keeps all four of its own gates behind this one (the `DANGER` word is still what cuts power), and `POST /api/auth/logout` is gated like any other write because a sign-you-out link is a CSRF. `POST /api/auth/password` (task 04) is the third gated write and the app's only revocation: guesses at the current password spend their own `password:
` bucket, and a right one replaces `auth.verifier` **and** `auth.session_secret` so every browser that is not in front of the form dies on its next request — which is why the response is minted from the new state and never the one being replaced. ## Architecture ``` agent/host_agent.py (each host) gufo / vLLM upsd (NUT) │ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD │ │ + /slots (gufo) │ ▼ ▼ ▼ collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐ │ parse + Δcounter/Δt │ ▼ ▼ sources/host.py · disks.py · llm.py · nut.py poller.py (task per source) │ Result(payload, samples, events) │ ▼ ▼ store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/ ``` Responsibilities, one line each: - `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`, `statvfs`; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an HTML status page at `GET /`. Refuses to bind off-loopback without a token (exit 2). Counters stay counters: the dashboard computes rates so the agent never remembers who polled it last. SMART (phase 09): `smart()` runs `smartctl -i -A -H` per whole `/sys/block` disk (`parse_smart_output` covers ATA + NVMe; NVMe media errors map to `offline_uncorrectable`, hex-packed vendor raws are never decoded, and an attribute row is read from its **RAW_VALUE column** rather than the end of the line because this binary annotates that column — `35 (Min/Max 34/44)`, `43894h+43m+32.519s` — so its last token is not the value), 5 s per device / 20 s per pass / 240 s cache, attached as `disk_io.devices[].smart`. Missing binary → **one** `errors` line for the pass (the word `not found` in it is what `sources/disks.py` reads to answer "is smartctl installed", and `tests/test_agent.py` pins that the line still says it); one denied/wedged device → that disk's `smart` is `null` with its own line; `--no-smart` opts out silently. Never fails the snapshot. **An expired cache refreshes on a background thread while the request is served the previous result** — only the first pass after startup runs inline — because 20 s of wedged disk on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the *system* source red every four minutes over an add-on that page never draws. The refresher is single-flight and keeps the old cache on failure, so a bad pass retries on the next expiry instead of hammering the disk. - `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real env var always wins. Defaults are safe except the sources, which default to unset. The auth block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means *unset* rather than "the password is empty"), `DASH_SESSION_HOURS` (720 = 30 days: a dashboard that logs the operator out mid-outage gets bypassed), `DASH_COOKIE_SECURE` (off, because dev is `http://127.0.0.1`; task 03 also treats `X-Forwarded-Proto: https` as secure), and `DASH_PBKDF2_ITERATIONS` (default `auth.PBKDF2_ITERATIONS`, imported so the primitive and the setting cannot drift) — that knob exists so the hermetic suite hashes in microseconds instead of 0.3 s per client and is not a production tuning parameter. - `poller.py` — job loop, health records, stale/last-good semantics, transition events. `/disks` is a **source of its own** (`disks:`, `DISKS_INTERVAL_SECONDS` 300 s, `DISKS_TIMEOUT_SECONDS` 30 s, events under the dashboard name `disks`) reading the same agents the `system` jobs read: the separate task is what keeps a SMART poll out of the 10-second utilisation loop and lets `/disks` age on its own instead of blinking `/system`. The job list is built once (`app.build_hub()`) because it is wired twice — production and the injected-store test path — and two hand-written copies drifting is how a source ends up polling in tests and not in the container. The success path records (payload → samples → events) and only *then* moves `last_ok`, so a refused SQLite write ages the source and alerts instead of leaving it current with charts that have silently stopped drawing. Events are stored under the **dashboard** name (`system`/`disks`/`llm`/`ups`), not the sample source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever (found on `/system` during phase 03 browser verification). - `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping (its self-test event takes its severity from `nut.self_test_outcome()` and not its own word list, so a passed test is never an amber line in the feed while the card beside it reads green). Its per-UPS tolerance for a failed `LIST CMD`/`LIST RW` is for a *complete* error reply only: one battery whose command list is denied loses its buttons and the poll goes on, while a **stalled** reply fails the poll, because `NutClient` marks a socket it could not read to its `END` as unusable (a reply still arriving would otherwise be taken for the *next* command's answer — the real cause of the `expected 'BEGIN LIST VAR', got 'RW apc …'` lines this fleet's event table used to fill with, and the same hazard on `INSTCMD`, where a late `OK` read as a different command's success would be a gate that lies). Failure messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one that was dialled, because they become health records and event rows. `fetch_snapshot()` is the one agent `/snapshot` read that `system` and `disks` share, so a dead host says the same sentence on both pages and only the `timeout` differs (the disks poll has to outlast the agent's 20 s smartctl pass, and `None` is translated into httpx's sentinel because a literal `None` means *no timeout at all*). Agent `errors` become the payload's `degraded` **except** the `smart:` add-on lines — a host without smartmontools or device read access is unconfigured, not broken (invariant 3); they stay in `agent.errors` for the `/disks` page to explain. - `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card (`gpu.busy_pct@`). - `sources/disks.py` — the same snapshot projected onto **physical** drives for `/disks` (phase 09): one row per whole-`/sys/block` disk (never a partition, never a mount), the agent's `smart` block carried through verbatim, `health` the drive's own word normalised to `ok`/`failed`/`None`, and `smartctl_available` three-way on purpose — `True` (a disk answered), `False` (the agent said the binary is not installed, so the page can say *install smartmontools*), `None` (an agent predating the pass, or `--no-smart`: nobody knows, so `n/a` plus the agent's own `smart_errors` line and never a guessed verdict). **Nothing here is a rate**: power-on hours and the sector counters are stored counters-as-reported, so the source takes no `RateEstimator` and its payload carries no `window_seconds` (rule 2 — a rate with no estimator, no `Δ` label and no `source` string is an invented number). `disk.health_ok@dev` is not a series either: a verdict is not a number, so the drive's word travels in the payload only. Each row does carry a `severity` dict (`SEVERITY_FIELDS` = the two rated numbers, `temp_c`/`percentage_used`, plus `health` through `HEALTH_SEVERITY`, which translates the drive's own word instead of grading it), and `UNGRADED_FIELDS` names the numbers that must never be graded. `fleet_disk_summary()` sums the per-host `summary` counts for the page header and drops a host with no payload from the sums while still counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does arithmetic and a dead agent never shrinks a fleet total. - `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`, `VLLM_HISTOGRAMS`), measured Δ rates, prefill speed as `mean(prompt tokens)/mean(prefill time)`, session list from gufo `/slots`. - `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not a mystery score), `SELF_TEST_OUTCOMES` + `self_test_outcome()` (the `Last self-test` verdict follows the *outcome* of the driver's `*.test.result` word, not its phrasing: pass words → `ok` (including `Done and passed`, what the CyberPower here answers), fail words → `fail`, `No test initiated` / empty / no variable → `unknown` printing `NEVER_TESTED_DETAIL` so both "never tested" states say one sentence, and an unrecognised word → `warn`, because a driver inventing a verdict is drift worth noticing and the check prints its own word to be added), `COMMANDS` catalog with risk tiers, `COMMAND_ALIASES` + `resolve_command()` (drivers answer two names for the same action — `beeper.enable`/`beeper.on`, `beeper.disable`/`beeper.off` — so one entry is offered per action and any of its names hits that entry's verdict, never a gate the rendered name missed), `available_commands()` filtering, kelvin-vs-celsius normalisation, and `charger_status` normalised to the `CHARGER_STATUS` keys (the same lowercase token `states()` and `on_battery` already compare), and `states()` publishes one chip per fact — a driver exporting both `battery.charger.status` and the legacy flag that says the same word gets "charging, online", not "charging, online, charging". `battery_runtime` comes from `battery.runtime` **only** — `battery.runtime.low` is a shutdown threshold, not time left, so it lives at `battery_runtime_low` and never becomes the hero's "runtime left". - `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a battery. It must never be the default, and every payload it produces says `simulated`. Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that ages), so a fresh simulator reads healthy and amber keeps meaning something. Its status follows the real drivers: a below-float pack on mains reports `OL CHRG`/`chg` (the glyph's bolt state is keyed on that token, and the model has to be able to produce it) and a battery being drawn from is `OB`/`dchg`, never `ol`. Every command it advertises is implemented and visible in a variable a real driver publishes — the beeper ones set `ups.beeper.status` (`enabled`/`muted`/`disabled`), and a test window that ends writes `ups.test.result: PASSED` while an aborted one writes nothing; the load/shutdown commands answer `OK` without modelling an outlet, because a faked power cut in the history is indistinguishable from a real one. `from_env()` is the single place the `NUT_SIMULATE` truthy rule lives — `app.py` builds the monitor through it, so the assembly and the model cannot disagree about what is enabled. - `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own). - `rates.py` — Δ/window estimator, keyed by resolved series name. - `auth.py` — every authentication decision as a pure function, stdlib only (phase 11): `hash_password`/`verify_password`/`needs_rehash` over a self-describing `pbkdf2_sha256$$$` verifier (so the cost can rise without a migration), `mint_session`/`read_session` for an HMAC-signed stateless cookie (no session table and nothing to prune; rotating the stored secret signs every cookie ever issued out, which is how changing the password evicts every other browser), `mint_csrf`/`verify_csrf` bound to the session token instead of stored, `LoginThrottle` (in-process exponential backoff), and `origin_matches`/`sec_fetch_site_allowed` as string functions so the gate is testable without a `Request`. The primitives **never raise and record nothing**: a malformed verifier, a tampered cookie, and a rotated secret all answer `False`/`None`, because the reason an input was rejected is the credential (invariant 6). No HTTP, no SQLite driver, no new dependency — `app.py`'s middleware and `api.py`'s `/api/auth/*` are the callers, so a decision made here is made once and is unit-testable without a `Request`. It also carries the two strings the gate and the pages share (`LOGIN_PATH`, `LOGIN_NEXT`) and `safe_next()`, the only judge of a `?next=` value: kept verbatim (query string and all, because the range chips live there) when it stays on this app, dropped otherwise — scheme-prefixed, scheme-relative, the percent-encoded spellings of both, anything with a control character (not a path but a second `Location` header), and anything without a leading `/`. The **verifier bootstrap** (task 02) is the one deliberate exception to "never raises": `load_auth_state(store, admin_password=…, iterations=…, session_hours=…, db_path=…)` reads the `meta` rows `auth.verifier` / `auth.session_secret` through a three-method `MetaStore` `Protocol` (so this module still imports nothing in the package, and its tests use a dict) and returns a frozen `AuthState` — the environment **seeds** a password once and the stored verifier wins thereafter (`env_ignored` and one fixed-line warning when both are present, so `/settings` is the real answer and no redeploy can roll a changed password back); no row and no seed raises a `RuntimeError` naming `DASH_ADMIN_PASSWORD`, the meta key, and the `sqlite3 … DELETE FROM meta …` reset command, because "start open with a warning" is the failure this phase exists to prevent; a stored secret that is not 256 bits of hex is regenerated and persisted (the effect is one login prompt, never a lockout). Nothing here reads a clock, so the state carries no expiry — a session's is inside its signed payload. `update_verifier(store, state, new_password)` writes the new verifier **and** a fresh secret (secret first, so a crash cannot leave sessions outliving the password they were issued under) and refuses an empty/whitespace password; it is what `POST /api/auth/password` calls, because rotating the secret **is** this app's revocation list — a stateless cookie has no row to delete, so without it a stolen session would outlive the password that "revoked" it for another `DASH_SESSION_HOURS`. Its two raises, like the loader's, carry fixed text — `tests/test_auth.py` pins the file's raise sites and its one logging call structurally, so no future edit can interpolate a credential into either. - `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric, `classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for an unknown. The verdicts travel **additively** in payloads — `cpu.severity`, `memory.severity`, `disks.severity`, each UPS card's `severity` (charge/load/runtime), `severity` inside rated LLM cells (KV pressure, classified on a percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS row in `/api/summary`. Pages render the class; they never compare a number to a threshold. There is **no `gpu.*` row and no `gpus[].severity`** (phase 06): these are LLM servers, a busy GPU is the job at every level, so the GPU card and the fleet GPU column render uncoloured — like `net.*`, whose colourlessness predates it. Removing the verdict is not removing the data: `gpu.busy_pct@N` / `gpu.mem_used_pct@N` are still stored and charted. The `/disks` rows (phase 09) are `disk.temp_c` (50/70) and `disk.percentage_used` (80/100) — an SSD at 100 % of its rated wear is a replacement, not a warning — attached per row by `sources/disks.py`; the sector counters and `disk.power_on_hours` deliberately have **no row**, because a non-zero counter is an alarm chip on presence and zero is not an achievement (the `/llm` preemption-chip rule), so `classify("disk.reallocated", 1)` is `None` and the page shows a chip instead of a colour. - `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload, recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts` (`{id, severity, source, message, href, at}`, fail first then warn, then by source). A pure projection: no I/O, no store, no thresholds of its own — it reads job states for reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions. Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but older than 3× its own poll interval is `warn` while a source that stopped answering is `fail` (the interval is the job's own dashboard's, so `disks` goes amber after 15 quiet minutes rather than 30 seconds); a dead source gets exactly one line (its stale payload adds nothing) that names it once — the poller's `"UpstreamError: "` wrapper and the duplicated host name are stripped, because a message that says `gateway` twice reads as two problems, and the same rule silences the **`disks` echo** of a dead agent (the disk job dials the same `/snapshot`, so its failure carries no new information — while a `disks` job failing *while its host answers*, SMART wedged inside the handler, is the only voice that fault has); `unknown` and missing numbers are silence; one alert per (source, metric) and at most 12, the last being a marker that says how many it hid. This is why the overview can be two panels. There is **no disk-health rule and none is wanted**: a drive the hardware itself called `FAILED` is a red chip on its `/disks` card and a line in that page's event feed, not an overview alert — `severity.py` rates no sector counter and this module may not have a threshold of its own, so grading a drive here would be the bug, not the fix (phase 09). Its `HREF` table does carry `/disks`, so the one alert a disk job *can* raise — SMART failing while its host answers — links to the page that names the drive. - `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning. The `meta` table is the only durable key/value state in the app and is reached through three named helpers — `get_meta`/`set_meta`/`delete_meta` (the last returns its rowcount, because the password-reset procedure has to be able to say whether a row was there, and one key per call, because a reset that deletes a row it did not name deletes the operator's credential). `stats()` reads `last_maintain` through `get_meta()` too — outside its own lock, which `get_meta` takes itself and `threading.Lock` is not reentrant — while `maintain()`'s write stays inline inside that method's transaction. - `api.py` — GETs everywhere plus the one POST that dials an upstream, and the `auth_router()` that creates, describes, **re-credentials**, and ends a session (`POST /api/auth/login` public; `GET /api/auth/status`, `POST /api/auth/logout` and `POST /api/auth/password` protected). Those three are the app's only other writes and they touch this process and its `meta` table and dial nothing — invariant 1 is about the fleet, and `tests/test_api.py` pins the two lists separately rather than widening the one that names `/api/ups/command`. The login route is where a *sign-in* password is compared: it throttles before it hashes, refuses an absent/empty/non-object body with `422` so `""` never reaches `verify_password`, answers every failure with the one sentence `wrong password`, and answers `429` + `Retry-After` while the per-address backoff runs (the only extra fact it adds, because a wait is something the caller has to act on). After a good sign-in it also upgrades a verifier minted at a lower `DASH_PBKDF2_ITERATIONS` (`auth.needs_rehash`), keeping the signing secret — that knob has to actually buy something, and a login is the one moment the plaintext is in hand. A refused upgrade write is swallowed with one fixed-text `logging` line (this module's only one, carrying no credential: invariant 6) because `database is locked` on a bookkeeping row must not turn a correct password into no login at all. `POST /api/auth/password` is task 04: it validates `new` first (`MIN_PASSWORD_LENGTH = 8`, non-empty once stripped, different from `current` — length is the whole policy, because composition mandates produce sticky notes), throttles the *current* password on its own key `password:
`, refuses a wrong one with `403` + a `password-denied` event, and on the right one calls `auth.update_verifier` and answers from the **new** state with a cookie and CSRF token minted from the rotated secret. Building that response from the old `ctx.auth` is the bug the order exists to prevent: the operator would be signed out by their own form. `AppContext` carries `auth` and the one `LoginThrottle` per process — a throttle built per request is a throttle that never trips. A refused sign-in writes a `login-denied` **event** and nothing else, and so do the two password outcomes (`password-denied`, `password-changed`): `alerts.py` turns only `kind: "unreachable"` into an overview row and every feed filters by `source`, so `/api/events?source=auth` is the audit trail and the overview stays about outages (do not "fix" that by hiding the event). Neither fact appears on `/api/health` — which is public — or `/api/config`. `/api/config` is a redaction boundary. `/api/disks/*` (phase 09) is the drive-health pair: `/snapshot` answers one entry per configured host (disk rows + `smart` + `smartctl_available` + per-row `severity` + a `fleet` block summed by the API, with a configured-but-never-answered host as an `unreachable` stub whose counts are `None`, not `0`), and `/history` serves the `disk.*@dev` series with unknown names answering as empty series like the other two history endpoints. The configured gate runs **before** anything that could reach the network — `HOST_TARGETS` empty is `404 {"configured": false}` with no client, no fetch and no `disks` job in the hub at all (invariant 3), and `tests/test_api.py` proves it with spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot`. The new surface grows no write of its own — a fourth dashboard must not quietly become a second one, so the app's only POST stays `/api/ups/command` (invariant 1, pinned by enumerating every served route's verbs). `/api/summary` is the overview's single call and the cheapest whole-fleet probe there is: the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `disks.hosts`, `llm.engines`, `ups.ups`, the `states` maps and `recent_events` stay for programmatic consumers (live tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events` are unchanged and `alerts` (phase 02) and `disks` (phase 09) are additive — the `disks` block is `fleet_disk_summary` verbatim plus one row per host (`disks`/`with_smart`/ `health_failed`/`alarm_disks`/`smartctl_available`/`failed_devices`), so a notifier learns *which* drive the hardware called `FAILED` without a second call, and `tests/test_api.py` pins every one of those key paths, with and without targets configured, so retiring a panel can never shrink the payload. A configured job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in `system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total` counts the machines there are and `hosts_live` the ones answering. - `app.py` — assembly, and the **session gate** (`session_gate` middleware, invariant 12) whose `PUBLIC_PATHS`/`PUBLIC_PREFIXES` are the whole anonymous surface. It answers in this order and the order is the design: no `AuthState` → `503` for everything; no session and no such route → the router's `404` (a typo is not a sign-in problem, and "sign in required" would otherwise be the only answer anyone outside ever gets for a wrong URL); public → through, except that the origin check still applies to a public write (a cross-site login post spends the throttle of whoever is typing) and `GET /login` for someone already signed in goes to the dashboard, not back to the form; no session → `303` to `/login?next=…` for a page and the JSON refusal for everything else; a write → origin, then this session's CSRF token when it arrived on the cookie. `_own_origin` reads `X-Forwarded-Proto` for the same reason the cookie's `Secure` flag does, so behind nginx a real same-origin write is not a cross-site refusal — the bug that only appears in production, which is why it has a test. `create_app()` takes injectable settings/store so tests build an app with an in-memory DB and no targets. The lifespan builds `ctx.auth` (`auth.load_auth_state`) after the context exists and **before** the first poll, for the owned-store and injected-store branches alike — one call site, because two copies drift — and a `RuntimeError` from it leaves the process non-zero with the operator's fix in the output. That failure is never caught here: a dashboard that cannot say who its admin is does not start the loops either (`__main__.py` passes `SETTINGS` to `create_app()` explicitly so the served app is configured by the same object the bind address came from). - `web/` — five dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`) plus the login page (`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`. No framework. `/login` renders the masthead's **brand link and theme toggle** and nothing else from it: no tabs (every tab bounces straight back here, and a nav of links that all redirect is a lie about what is available), no status dots and no clock (there is nothing signed in for them to say). It posts through `Dash.fetchJSON` like every other script, which is only safe because `bounceToLogin()` does nothing when the browser is already on `LOGIN_PATH` — without that guard a `401` would reload the form instead of printing `Wrong password.`, and the page that exists to end a bounce loop would be one. So the form branches on `err.status`: `401` is the inline sentence plus `field.select()` (fixing a typo without retyping is the whole UX here), `429` shows the server's `detail` and **counts down its `Retry-After`** with the button disabled until the wait is over — `fetchJSON` therefore carries `response.headers` on the `Error` it throws, because a wait this page made up would be an invented number (rule 2) — and anything else prints the server's `detail` verbatim. Every refusal goes out through `textContent`, never `innerHTML`: this is the one page a stranger reaches first. Its own copy of `auth.safe_next` guards the one redirect followed on client strength (already-signed-in `?next=`), and `Dash.ensureAuth()` up front stops a flash of the form for a browser that is already in. The login page is audited for orphan `id`s and CDN references too (`tests/test_auth_api.py`), fetched unsigned because a signed-in `GET /login` is a `303` and would audit `/`'s markup under the wrong name. `/settings` (`pages/settings.html` + `static/page-settings.js`) is a **page and not a sixth dashboard**: same `` and masthead, the five tabs with **no** `aria-current` (it is not one of them), no status dots, and it is reached from the `⚙ Account` link in `.masthead-right` of every dashboard — outside `.tabs`, so the pinned tab structure is untouched (`tests/test_api.py::test_every_dashboard_page_links_to_settings`). It prints `/api/auth/status` (name, expiry as a local timestamp *and* as time left), signs out through `POST /api/auth/logout`, and changes the password through `POST /api/auth/password` after checking locally what it can judge itself (non-empty current, ≥ 8 new characters — `tests/test_api.py` pins that number against `api.MIN_PASSWORD_LENGTH` so the two cannot drift — new matching confirm), printing `403`/`422`/`429` through `textContent` as the server words them, then re-reads the status so the panel cannot show a session the rotation ended. Its static sub-line is the precedence rule (env seed → stored verifier wins → reset by deleting `auth.verifier`), because that sentence is what stops the "I edited the env file and nothing happened" call. So `app.js` exports **`resetAuth()`**: it clears the memoised `authStatus` and its in-flight promise, and must be called after a password change (the secret rotated, so the cached `csrf` is now invalid — every later write, the UPS buttons included, 403s until a reload) and on any `401` bounce. `app.js`'s masthead status dots are one per dashboard job — including `disks`, because the same agent can answer the 10-second utilisation poll and have a wedged SMART pass, and one dot for both would be right about neither. `app.js` owns the colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an `ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a `.hero` big number (plus an optional `.bar`) from that class — it never compares a number to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for values the API does not rate, and **no page script calls it any more** — `/` was converted in phase 02, `/system` in phase 03, `/llm` in phase 04, `/ups` in phase 05 and `/disks` shipped that way from its first commit, so every colour on all five pages comes from a `severity` verdict the API sent (`tests/test_api.py` fails if a page script names it again). A value the API should rate gets a row in `severity.py`, never a bound in JS. Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`, `gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device series — `/api/metrics` answers with exactly these names. The `sample.source` column is the dashboard family — `host`, `llm`, `ups`, and `disks` for the SMART series (`disk.temp_c@nvme0n1`, `disk.power_on_hours@…`, `disk.reallocated@…`, `disk.pending@…`, `disk.uncorrectable@…`, `disk.percentage_used@…`). Those keep their own family instead of sitting beside `host`'s `disk.used_pct@/mount` because the cadences differ thirty-fold and `GET /api/metrics?source=disks` has to answer "what does *this page* have history for". That scheme is the history API's key — treat it as a public interface. ## The fleet (real environment, verified 2026-10) | Host | ssh alias | What it gives | | ----------------- | ------------------ | -------------------------------------------------------------------- | | `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards | | `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` | | `gateway` | `gateway-ops` | router, no GPU | - **Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05).** The designed shape is one `dashboards-agent` **systemd service per host**, installed by `ops` through the `deploy/agent-sudoers` grant with `deploy/install-agent.sh`, bound to the address the dashboard dials, its bearer token carried in the `HOST_TARGETS` URL userinfo; driveripper's upsd serves the LAN on 3051 with the host and laptop ranges allowed by the **firewall**, never by a `upsd.conf` line. The old ssh-tunnel script is deleted and neither `.env.dev` nor `.env.dev.example` declares a forwarded port — dev dials the same addresses the deployed dashboard does, and `scripts/capture-fixtures.sh` reads `HOST_TARGETS` from `.env.dev` instead of naming hosts of its own (`tests/test_deploy_scripts.py` fails the day any tunnel name or port returns). **Fleet transport, re-verified from the laptop on 2026-10-19.** The port-selective inter-VLAN drop phase 10 recorded on the lab gateway `10.2.0.1` is **gone**: `gateway` (`10.1.1.27:9120`) and `deskwork` (`10.1.0.51:9120`) both take a plain TCP connect and answer `/snapshot`, and `driveripper`'s upsd answers `LIST UPS` on `10.1.0.126:3051` from this same laptop. Two hosts run the **installed** service — `systemctl is-active dashboards-agent` is `active` on both, with `/usr/local/lib/dashboards-agent/host_agent.py` and a `--token-file` unit bound to the address the dashboard dials. `driveripper` is the one host with no agent at all (unit `inactive`, no installed file, nothing listening on `:9120` → `ConnectionRefused`), and it renders as an `unreachable` stub row: the designed degradation, recorded rather than hidden. `/api/health` today reads `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok`, `nut ok`, `llm:gufo ok`, `llm:vllm ok` with the two `driveripper` jobs `DOWN`, and the `live` suite is 8 passed / 3 failed with every failure that one transport. Phase 10's proof that the old drop was not this repo's firewall stands as history in `data/logs/NOTES-verify.md` — do not re-add a tunnel on the strength of it, and re-run the probe rather than trusting either note. The per-host record and the exact root commands are in `data/logs/NOTES-agent-install.md` (its firewall lines are the phase-10 state, not today's), how far the NUT wire reaches from each network is in `data/logs/NOTES-upsd-acl.md`, and the end-to-end direct-dial verification (health table, no-tunnel-port proof, live-suite tally, screenshots, and the probes that put the block on the lab gateway's port allow-list rather than in this app) is in `data/logs/NOTES-verify.md`, and the phase-11 login verification (the `curl` refusal codes, the six screenshots, the two `meta` keys the database actually holds, and the operator's deployment-repo actions) is in `data/logs/NOTES-auth.md`. Those notes are local evidence, not tracked files: `data/` is gitignored (invariant 10's `data/logs/*.png` included), so a fresh clone has the conclusions in the prose above and the captures live on the box that ran them — re-run the probe rather than trusting either. - Engines are reached directly: gufo `deskwork.reeselink.com:8731`, vLLM `driveripper.reeselink.com:8000`. - NUT 2.8.5 on driveripper: **configs live in `/etc/ups/`, not `/etc/nut/`**, and **upsd is on port 3051, not 3493**. Nothing listens on 3493 at all, so a default client reports *Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n' | nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. **NUT 2.8 has no `ACL` directive** (this Fedora build is `--without-wrap`; upsd logs `ACL in upsd.conf is no longer supported - switch to LISTEN` once per line and then ignores it), so per-source restriction is a **firewall** rule and never an `upsd.conf` line — and 2.8 does not refuse non-local clients the way older docs suggest: `LISTEN 0.0.0.0 3051` + `LISTEN :: 3051` with `3051/tcp` in the `FedoraServer` zone answers any client routed to it (reads are unauthenticated; `SET`/`INSTCMD` still need an `upsd.users` account). The host range `10.1.0.0/21` reaches it, and **since 2026-10-19 so does the laptop**: a raw `LIST UPS` to `10.1.0.126:3051` from the dev box answers `BEGIN LIST UPS`, which is why `nut` reads `ok` in `./scripts/dev-server.sh status` with two real UPSes and no `NUT_SIMULATE`. Phase 10 recorded `10.2.0.1` dropping 3051/3493 for this laptop; that is history now, and the probes behind both statements are in `data/logs/NOTES-upsd-acl.md`. Two UPSes are exported over USB and dialled directly as `NUT_HOST` + `NUT_PORT=3051`, and that reads live from every vantage the dashboard has ever used — the host itself, the deployed dashboard, and since 2026-10-19 the laptop (`/ups` shows both boxes at 100 % charge, `OL`, with no `SIMULATED` badge anywhere): `apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is `ups.description`, the operator's label — the driver's own `identity.model` answers `GX1500U`, so the description is what identifies the box). Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date (`battery.date` is often an unset placeholder — `2001/09/25`, slashes and all, on the APC), CyberPower reports `battery.mfr.date: CPS` (not a date → `unknown`, never a fake age), **neither driver reports a battery temperature or the NUT 2.8 `battery.charger.status`** (so the battery glyph's state has to come from `on_battery`, and `charger_status` must stay optional), and **`ups.test.result` is where the two boxes disagree**: the APC still answers the factory `No test initiated` while the CyberPower answers `Done and passed` (a test has run on it since the phase-05 capture; both words re-read from the wire on 2026-10-19 and pinned as `LIVE_SELF_TEST_WORDS` in `tests/test_nut.py`, which fails if the fixture and the capture ever drift). Both UPSes are recorded verbatim (`LIST UPS`/`LIST VAR`/`LIST CMD`, serials scrubbed) as `tests/fixtures/nut_.vars` by `./scripts/capture-fixtures.sh`, and the wire quotes a field only when it must — `VAR apc ups.status "OL"`, not three quoted fields. - Host access is via ssh keys as `ops`, and **passwordless sudo is different on every host** — that is what decides how much of the fleet an install can finish unattended. `gateway` grants `(ALL) NOPASSWD: ALL`, which is why its agent installed in phase 10 with no `deploy/agent-sudoers` line present; `driveripper` grants a command-scoped set (`smartctl`, `upsc`, `udevadm`, the `/etc/ups` reads, and the `tee /etc/ups/upsd.users|upsd.conf` + `systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork` grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`). Neither host grants `ops` the installer or `visudo` (re-checked 2026-10-19), so every install that has happened on those two went through root by the operator — including the one still outstanding, `driveripper`'s. - **SMART: real captures, and one live host finally reads them.** `smartctl` 7.5 is installed on deskwork and driveripper. The `ops` account (which is who runs a hand-started dev agent) is in no `disk` group, driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and deskwork grants nothing, and `host_agent.py` shells the binary **as itself** — it never calls sudo — so an agent run as `ops` still answers *Permission denied* (exit 2) and reports `smart: null`. What changed by 2026-10-19 is that `deskwork`'s **installed** service runs as `dashboards-agent`, which is in the `disk` group (`getent group disk` → `disk:dashboards-agent`) on an agent installed after the SMART pass: `/disks` shows real numbers there — `smartctl_available: true`, `nvme0n1` healthy at 52 °C, 19 160 power-on hours (an odometer, so that figure is a capture date, not a fact about the drive), 11 % of rated wear — so the wear bar and the alarm chips are drawn from live hardware, not only fixtures. `gateway` answers `smartctl_available: false` (no smartmontools in its VM: the *install it* half of the three-way flag), and `driveripper` answers nothing at all until its agent exists, which is where `smartctl_available: null` — "nobody told us either way" — still comes from. **Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_.txt` is all 16 drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses every one — which is what found the parser's one real bug: this binary annotates the attribute table's RAW_VALUE column (`35 (Min/Max 34/44)` on every Seagate, and one Western Digital writes its hours as `43894h+43m+32.519s`), so reading the *end* of the row lost the temperature of all eight HDDs and one power-on count. The captures are also the first honest look at this fleet's drives: `nvme5n1` reports 13 media-and-data-integrity errors and `nvme1n1` is at 58 % of its rated wear, so the alarm chip and the amber wear bar on `/disks` are real features with real data behind them, not demo furniture. **Lighting up the rest of it is the same two operator actions that travel together:** install the agent on `driveripper`, and give its agent user device read access — the `disk` group (`sudo usermod -aG disk `, which is what `deploy/install-agent.sh` does on install), *not* a sudo rule, because the agent must read the device itself — on an `agent/host_agent.py` from phase 09 or later (never redeploy without the grant: 16 permission-denied lines would age nothing but are pure noise, while with the grant they are 16 drives' worth of temperature, hours and sector counters). `gateway` needs smartmontools first and will still read no SMART after that, being a VM with no physical disk behind `vda`. A host that cannot read its drives renders the designed degradation: one dimmed card per physical drive, every number `n/a`, and the one-line why. `tests/fixtures/smartctl_reference_*.txt` stay as what they are: **reference shapes** (labels verbatim from the deployed binary, values synthetic), kept beside the real captures so the synthetic edge cases stay pinned. Re-record the real ones with `SMART_HOSTS="driveripper-ops" ./scripts/capture-fixtures.sh`: it tries plain `smartctl` first (so a fixture can never read healthier than the agent does), falls back to `sudo -n`, refuses to overwrite good fixtures with unreadable output, and scrubs Serial/WWN/EUI-64 (`test_smartctl_captures_carry_no_unscrubbed_serials` fails the day it stops). ## Commands ```bash uv sync --python 3.14 # deps + editable install cp .env.example .env # production-shaped config (DASH_ADMIN_PASSWORD seeds # the login on the first boot of a database — with no # seed and no stored verifier the server exits non-zero) cp .env.dev.example .env.dev # dev-only: the real hosts, tokens in URL userinfo set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no tunnels ./scripts/dev-server.sh restart # keep http://127.0.0.1:8090 up; sign in with the # DASH_ADMIN_PASSWORD from .env.dev (`testpass`) ./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health ./scripts/dev-server.sh logs uv run pytest # hermetic: 1172 passed + 11 live-skipped, no network uv run pytest --cov # coverage gate (98.6%; floor 94%) uv run pytest -m live --live # opt-in: real fleet / engines / upsd uv run ruff check src tests agent && uv run pyright ./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots) SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh # ... plus smartctl per physical disk (needs device read # access: plain smartctl, else `sudo -n`; scrubs serials) bash .agents/validate.sh # the phase gate: the four checks above (pytest # --cov --cov-fail-under=94, ruff, pyright, plus a # node --check of web/static) + repo hygiene. # Always `uv run` — a bare `python3`/`ruff` on PATH # tests an interpreter nobody deploys. ``` Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`, SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh` per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/ `--status`/`--uninstall`; it stores the token `0640 root:dashboards-agent` — the unit hands the agent the *path* and the agent opens it as its own user, so that user must be able to read it and no other account may — never puts a token on argv, and runs under `ProtectSystem=strict` with `DevicePolicy=strict` reopening only the disk nodes plus the `/dev/null` family, because strict without `/dev/null` kills the unit at spawn (`status=208/STDIN`). No root shell? The one sudoers line that lets `ops` run the installer is `deploy/agent-sudoers`; run the script without root and it prints that line with its own resolved path.) The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other service here. The source itself is `git@git.reeseapps.com:services/dashboards.git` (`origin`, branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only gate. ## Testing policy **Real captures for shape, fakes for behaviour, `live` for drift.** - Anything that must match an upstream — `/proc` layouts, vLLM series names, NUT framing — is asserted against a verbatim capture in `tests/fixtures/`. If the schema drifts, a fixture test fails before a dashboard does. `smartctl_driveripper_.txt` is the whole 16-drive fleet read through a `smartctl` sudo rule (Serial/WWN/EUI-64 scrubbed), and `test_the_real_drives_answer_every_number_the_disks_page_prints` asserts every number `/disks` prints comes out of real drive output — that test is what caught the parser reading the end of an attribute row instead of its annotated RAW_VALUE column. - Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`. - **A verdict the operator sees is pinned on the surface that shows it, in both directions.** `tests/test_nut.py` grades the word; `tests/test_api.py`'s `self_test_app` harness holds the `nut` job's own payload (its collector is swapped, so a poll cannot overwrite what the test mutated) and rewrites the `Last self-test` check in the object `/api/ups/snapshot` serves: a pass (`Done and passed`) and a never-tested pack produce **no** `/api/summary` alert row, a fail produces exactly one `fail` row per UPS linking to `/ups`. The two directions ship together on purpose — an absence-only assertion is satisfied by deleting the check, which is not the fix the report asked for (phase 12). The same file pins that the `/ups` check list needs **no** markup for a new verdict: `
  • ` plus `ul.checks li. .mark` in `dash.css` is the whole render path, and no page asset or stylesheet rule names one check (a second opinion about the driver's word is the bug). - **Page payloads that a rewrite must not break are pinned as a table.** `tests/card_contract.py` holds the dotted `/api/system/snapshot` paths the `/system` hero cards read (`CARD_METRICS`, `GPU_METRICS`, `HOST_SHAPES`); `tests/test_system_contract.py` asserts it against the three captures (discrete GPU, APU, no GPU) and over HTTP, and `test_every_host_serves_the_hero_card_contract` asserts the same table against the live agents — so a rename fails a test, and a card that shows `n/a` cannot be "fixed" by inventing a field. `network` carries no `severity` on purpose (throughput has no good/bad) and neither does a GPU row (phase 06: a busy GPU is the job on an LLM server) — the contract pins both absences. `tests/ups_contract.py` is the same device for the `/ups` hero row and battery glyph (`HERO_FIELDS`, `UPS_SHAPES`, `assert_ups_hero_contract`): `tests/test_ups_contract.py` drives it from `tests/fixtures/nut_.vars` — verbatim `LIST UPS`/`LIST VAR`/`LIST CMD` replies from the two real UPSes — and `test_ups_hero_fields_are_present_on_the_real_batteries` drives it against live upsd. The captures are what make the driver disagreements testable: one of them answers `battery.mfr.date: CPS`, neither exports `battery.temperature` or `battery.charger.status`. `tests/slots_contract.py` is the same device for gufo's `/slots` rows, which the `/llm` slot grid (phase 07) reads (`assert_slot_shape`, required/optional field tables, duplicate slot ids, the `next_token[0]` entries): hermetically from `tests/fixtures/slots_gufo.json` — the verbatim `/slots` reply, idle slots and all, captured whenever `GUFO_BASE_URL` is set — and live via `test_gufo_slots_answer_the_pinned_shape`. That fixture exists because its absence is how the deleted slot table rendered dashes forever: its field names (`slot.state`, `slot.n_prompt`, `slot.decode_per_second`) were a memory of an upstream, not a capture. **The contract names one canonical key per fact and the grid renders that one**: the capture answers `id_task` *and* `task_id` for the same request, so `CANONICAL_TASK_ID_FIELD` decides and the alias is captured but rendered by nothing — a rename fails a test instead of becoming a silent `a ?? b` chain in the page. `tests/test_api.py` reads the served `SLOT_FIELDS` table and resolves every key it names against the capture, in both directions. A busy slot's `prompt` is the user's in-flight request text, so the capture script blanks it and the page never reads it. - `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is unset, so partial fleet availability is normal. **Live tests must not write** — a battery test on hardware something is plugged into is not a test. - The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest, so it stays in the default run and is *not* marked `live`. - `tests/auth_contract.py` is why the HTTP suite can start a server at all: every app in a test is seeded with `admin_password="testpass"` (the local-testing password — a server with no stored verifier and no seed refuses to start, so an app under test needs a credential exactly like a deployment does) and hashed at `pbkdf2_iterations=1000`, because the real 600 000 costs ~0.3 s per client and the suite starts a hundred-odd of them. The real figure stays pinned in `test_auth.py` and `test_config.py`, so the cheap cost here cannot become the default there. - **The perimeter is its own file, and it is paranoid in two specific ways.** `tests/auth_client.py::sign_in(client)` is the one way a test gets a session: it posts `testpass`, asserts the sign-in worked (so a suite cannot silently start from a 401 and fail three lines later on something that looks like an unrelated bug), and puts the returned CSRF token on the client's default headers — which is what let the ~200 pre-existing HTTP assertions in `tests/test_api.py` stay untouched, asserting UPS behaviour rather than the new gate. `tests/test_auth_api.py` then runs the gate through a real ASGI request with `follow_redirects=False` on every anonymous call: `TestClient` follows redirects by default, so an unsigned `GET /api/config` would fetch the login page and report `200` — the way a gated suite ends up passing while testing nothing. Two of its tests are the ones to keep if you keep none: `test_every_route_that_is_not_public_requires_a_session` walks the **served** route table (`walk_routes` flattens pages, `include_router` wrappers and the `/static` mount, and raises on anything it does not recognise, so the audit cannot silently shrink) and demands `401`/`303` for every path outside `PUBLIC_PATHS`, GET and POST — the test that protects the next phase rather than this one; and the UPS pair prove the CSRF refusal on **the wire** (a scripted `nut.NutClient` records nothing for a tokenless write and does record it once the header is there, and `driver.killpower` still needs `DANGER` behind the session), because 403 after `instcmd` would be a power cut with an error message attached. - **The setup surface has its own pins, because prose rots quietly.** `tests/test_auth_docs.py` reads the artefacts an operator actually holds when a dashboard will not come up: both tracked env templates (every `DASH_*` auth setting declared; every secret-shaped *assignment* a placeholder — the same allow-list `.agents/validate.sh` 2b uses, so `uv run pytest` catches a committed password on the working tree and not only in the gate; the local-testing word allowed in prose and forbidden as a value), `scripts/dev-server.sh` (its probe and its `status` ride `/api/health`, no `dash_session`/`/api/auth/`/`Authorization:` anywhere, the password named but never assigned), and `README.md`/`AGENTS.md` (the *Signing in* section before *Configuration*, the four public entries, the reset `DELETE FROM meta …` command verbatim, the four config rows, the `DASH_COOKIE_SECURE=1` deployment note, the three auth gaps). No server starts and no request is made: a doc that stops naming the reset command is a locked-out admin, and that is worth a test even though it asserts nothing about Python. - Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces 94% (`--cov-fail-under`), and ~98% is where we are; keep it there. New/modified Python still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB, never a real target, and never a real upstream: `/api/ups/command`'s send is proven against a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live suite is read-only — that fake is the only place the write reaches a "driver". ## UI conventions Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ` labels on measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to `LineChart`, which puts `role="img"` + `aria-label` on the `` (an unlabelled plot is silence, and the legend is a sibling that labels nothing). Every `LineChart` on every page has one — `/system`'s three per-host slots, `/disks`'s two per-host plots, `/llm`'s four trend plots, `/ups`'s four battery plots — each pinned by `tests/test_api.py`, so a chart that ships anonymous fails a test. Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups` and `/disks`; the API accepts 5m…90d — SMART lands every 300 s, so a 5-minute window on `/disks` is one dot). A panel that cannot get data says why in the panel — the operator should never have to open devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`. **Flavour text is not helpful information.** A page does not narrate its own data pipeline or its own design; copy exists only where the operator needs it to act — a why-this-panel-is-empty sentence, a provenance tag, a measured `Δ` label, a chart's unit, a `SIMULATED` badge, an error that names what failed. The audit is one question per string: *would the operator act differently without it?* No → delete it, container and all. Deleted in phase 08 (the cited examples, from `TODO.md:7`): "every number is scraped from the box it belongs to · `null` renders as `n/a`" (`/`'s footer), "agent-reported values; rates are measured over the Δ window in the footer, and each panel's charts cover only that host" (`/system`'s *Per host* sub-line), and the same genre — `/system`'s "numbers are measured, not inferred", `/llm`'s "each throughput number carries its provenance", `/ups`'s "commands are the only write path in this app", each event feed's content caption (the empty-state sentence already says it), and the design clauses inside empty-state prose. What survived the audit did so because it is a measurement or a why: every `Δ` footnote, `provenance()` on each rate, `fleet.seams`, *sums only engines answering now*, the queued-request note under the slot grid, the SIMULATED banners, and the setup instructions in every empty state. `tests/test_api.py` pins the deleted sentences **absent** from every served page and script (a sentence that migrates to another page is the same slop), the kept functional strings **present**, and that no removed sub-line left an empty `.sub`/`.faint` container behind. The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per `/api/summary` alert: verdict class, source linked to its dashboard via the alert's own `href`, message, age) and **Fleet**. A non-empty list floats the alerts panel above the fleet table (`#overview-panels[data-state]` swaps them); an empty one is an explicit "all clear" state below it, because an empty list is a message, not a blank box. A host that is not answering keeps its row, dimmed with the shared `.stale` class and reading `n/a`, so the table never implies last good numbers are live and never silently loses a machine. No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the two-panel shape against the served HTML. Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an `id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for` attribute references — the orphan a deleted renderer leaves behind, caught on served bytes because there is no JS test runner (a locked anchor). Same rule for `dash.css`: delete the rules a removed panel made dead (`.grid.wide` went with the overview's fleet+events row); component vocabulary the pages are still being converted to (`.hero-bar`, the grid and bar variants) stays even while only one page uses it. A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals, muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's `severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*` family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS rule reaches for a token that `:root` does not declare. **The `/disks` card is one panel per *physical* drive, led by identity and verdict, not by a hero** (phase 09): a drive has no one-number story the way a UPS has charge %, so the card opens with the device path (`/dev/nvme0n1`) and the drive's own word (`OK` / `FAILED` / `unknown`, a `.chip` coloured only by `severity.health` — an unreadable drive gets grey `unknown`, never a green pass), then the identity line (`model · size · SSD|HDD`, nothing invented where a field is absent) and three equal `.metric` cells: **Temp**, **Power-on**, **Wear**. Colour comes from `severity.temp_c` / `severity.percentage_used` and nowhere else; `power_on_hours` is an odometer, written with a `null` verdict, so it cannot read green at 100 h or red at 60 000 h. The sector counters are **`.chip.fail` alarm chips that exist only while non-zero** (the `/llm` preemption rule, and why `severity.py` rates no counter): a drive at zero draws no chip row at all, because a permanent "0 pending sectors" line trains the eye to ignore it and a green one would call a quiet drive an achievement. Per host the three-way `smartctl_available` decides one `.banner.info` why-line and nothing else — `false` says *install smartmontools and grant the disk group* (the agent said so, so the page may), `null` prints the agent's own `smart_errors` line or names both fixes without claiming to know which half is missing, and an agent that is not answering says that instead — and in every case the drive grid takes the shared `.stale` dim while the host card stays uncoloured (invariant 3: a missing binary is unconfigured, not broken, and a red panel here would train the operator to ignore red on the drives that do report). Nothing on the page is per partition or per mount: the rows are the agent's whole-`/sys/block` list, and a filesystem field naming its way into `page-disks.js` fails a test. The fleet header renders `fleet_disk_summary()` — hidden below two hosts (one box's sums duplicate its own cards) and coloured nowhere, because a count of drives is not a graded number — and the per-host `Charts` disclosure follows `/system` exactly: collapsed, lazy-built, **one** `/api/disks/history` request per host per refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive (the endpoint answers unknown names as empty series on purpose), the wear plot created only once some drive publishes the field, legends naming drives because the card heading already says who, and `pruneHosts` taking the cards, the toggle and the plots off the page together. A cell-scale trend is `DashCharts.sparkline()` — the `llm_monitor` `drawSpark` idiom, so a percent metric is plotted against `fixedMax` (**never** autoscaled: an 8 % KV window that once held a 92 % spike must not read flat, and the axis gets top/bottom scale marks so "a third of the way up" says a third of what). `align: 'right'` pins the newest sample right and grows the line leftwards inside a real `span` in seconds — x stays time, not sample index. Stroke colour comes from the payload `severity` via the `--ok`/`--warn`/`--fail` tokens only; an unrated series draws a faint `--text-faint` stroke with no fill, because a shape is not a verdict. Below `minPoints` (2) it prints `no history` rather than drawing a confident line through one point, `labels: {current, max}` renders the `now / max` foot through the caller's `format`, and there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the other half of the same honesty for count metrics: history arrives as rollups, so eight seconds of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for a queue nobody was in. `/llm`'s count rows moved to the phase-07 stats strip (numbers, not sparks), so no page passes `minSpan` today and `drawSparks()` still plumbs it through — a count sparkline that comes back must set it, and the component's own contract test keeps the option honest (`test_sparkline_ships_the_fixed_scale_and_verdict_options`). A battery is `DashCharts.battery(container, {charge, severity, state, label})` — one inline SVG whose **fill *is* the charge**: 1 viewBox unit per percent, clamped to 0–100 and never rounded up (a 2 % cell must look nearly empty, because a reassuring battery above a flat one is a lie), clipped to the plate so 0 % leaves an empty shell rather than a stray pixel. Colour comes only from the payload `severity` token; a measured charge with no verdict draws faint grey at half strength (`sparkline`'s rule — a shape is not a verdict) and never an empty shell ("0 %" and "not rated" are different sentences). Every state has a **shape** as well as a colour, so the glyph survives monochrome: solid shell `online`, bolt `charging`, dashed outline `onbattery`, no fill with a dash pattern *through* the body `unknown`; a state word it does not know degrades to `unknown`, never to a confident "on line". Accessible **by default** — `role="img"` + `` + `aria-label` built from the inputs ("apc: Charge 100 percent, on line"), because the glyph carries information the markup does not say in words — and a page that prints that sentence next to it passes `decorative: true` to make the `` `aria-hidden` rather than say it twice. No animation and no `ResizeObserver`: one `viewBox` + `preserveAspectRatio` scaled by `.battery` in `dash.css`, so the hero row gets a big cell on a desktop and a small one on a phone. **The `/ups` card is led by the glyph and three heroes** (phase 05): the battery on the left, then **Charge %**, **Load %** and **Runtime left** on `.heroes` at the shared 2.4rem, coloured from `severity.charge_pct` / `severity.load_pct` / `severity.battery_runtime` and nothing else. Charge carries **no** `.bar` — the glyph beside it is that value drawn as a shape, and one value drawn twice in one row is the pattern this app avoids (Load keeps its bar; nothing else draws it). Its `.sub` lines are the context each number needs and nothing more: `24 V nominal · 27.3 V` for the pack, `128 W of 900 W` when the driver publishes both `realpower` figures (a wattage inferred from a percent against another model's rating would be an invented number), `at the current load` for runtime. Runtime prose is `runtimeText()`'s (`1h 32m`, `10m 21s`) and an estimate the driver never made goes into `Dash.hero()` as `null`, so it prints faint and uncoloured rather than as `0m`. The readings the hero row displaced — line in, output, battery volts and amps, battery °C — stay on the card, demoted to `.ups-electrical`: muted, `.95rem`, never folded away, because they are what you reach for *while* the UPS is on battery, and battery °C reads `n/a` on both real drivers here. `state` is derived in the page from `on_battery` → `charger_status` (`chg` charges; `dchg`/`rec` are the server's own on-battery tokens; `full` is **on line, not charging** — a rested floating battery reports it) with one fallback: a state chip the API rated `ok`, because `nut.states()` reserves `ok` for on-line/fully-charged and neither real driver here publishes `battery.charger.status` at all. `ups.status` itself is never re-parsed in JS (`tests/test_ups_contract.py`). `/ups` passes `decorative: true`, because the charge hero and the state chips already say the sentence. The glyph repaints only when charge / verdict / state change (`drawGlyph()` keys a `data-drawn` triple): the charge float jitters every `NUT_INTERVAL_SECONDS`, and rebuilding the `` on that re-announces an accessible name for a difference no eye can see. An event row names its subject **once**. Every feed (`/system`, `/disks`, `/llm`, `/ups`) renders the message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a *leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py` messages start with their own label, and each row already prints that label as a chip, so "gufo gufo is answering again" reads as two problems. A name in the middle of a message stays: the UPS rows used to `replace()` it anywhere and mangled "Quick battery test sent to apc: OK". This is presentation only — the stored event and `/api/events` keep every word, and the raw error string still surfaces in `/api/health`. An event feed is a **window**, not a printout (`Dash.eventsPager(el, events, opts)` in `app.js`, one helper with four call sites (`/system`, `/disks`, `/llm`, `/ups`) — a pager that works on one page is the bug this shape prevents). `ul.events` is a bounded 320px scroll container (`overscroll-behavior: contain`) and the pager puts `EVENT_PAGE_SIZE` = **50** events in it, with a `.events-pager` strip under the list: `1–50 of 200` between `‹ prev` and `next ›`, both `aria-label`led for direction (the feed is newest-first, so `prev` walks towards now) and `disabled` at the ends. The strip is `hidden` when the feed fits on one page — every row on screen *is* the whole story, and a disabled arrow says "more history" when there is none. Each page keeps its own `row(event)` (so `/ups` keeps its longer timestamp and runtime footnote, and escaping stays the page's job — the pager has no default renderer and says so in a row when a caller forgets one) and its own `empty` sentence, so the `loading…` and empty states read exactly as they did before paging existed. Two numbers are load-bearing and pinned together: the fetch (`EVENT_FEED_LIMIT` = 200, `/api/events?limit=…`) must exceed one page, because a limit at or below `EVENT_PAGE_SIZE` is a pager that silently never appears; `/api/events` itself is **unchanged** — no `page`/`limit` vocabulary grew for one page's layout. Paging state lives in a `WeakMap` keyed by the `
      `: it survives a poll (so page 3 is still there two seconds later — resetting every refresh would make page 2 unreadable) and resets to page 1 when the event *set* changes (an operator reading page 3 of yesterday's history must not land on page 3 of a list that just changed). Rows are re-written only when their markup changed, which is what keeps the reader's scroll position inside the container. The `/system` host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the `.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script). Colour comes from the payload's `severity` dicts and nowhere else; a missing number or verdict — gateway's CPU temp, any host's absent GPU — prints `n/a` **uncoloured**. NET is one slot with two lines (`.hero-lines`, `↓` in / `↑` out) and is deliberately colourless: throughput has no good/bad, so `network` carries no severity. The GPU slot holds one hero per card whose `.sub` carries model, VRAM used/total, the honest APU `pool` view and the card's temperature (`gpus[].temp_c` — the agent's max over that card's hwmon labels, the same value stored as `gpu.temp_c@N`, so the hero and the history cannot disagree), and it is colourless for the same reason (phase 06): a 95 %-busy card on an LLM server is the job, so the hero carries no `severity` and no utilisation bar — a bar under a verdictless number would be a verdict by shape. A card whose driver exposes no temperature input prints **no** temperature in that footnote rather than `n/a` or `0 °C`: "where you can" is the operator's wording, and `n/a` is reserved for numbers the card is supposed to have. Everything not glance-worthy — load averages, clock, disk read/write/busy, uptime, GPU power and the per-label temp breakdown, filesystems, processes — folds into a collapsed "More detail" `
      `: a simpler surface, not a less complete page (the table keeps its Temp column; this was a promotion, not a de-duplication). A `Δ` label is a measurement, never a config value: `sources/host.py` sends the window the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints those (a range when the hosts disagree) instead of the poll interval, which only coincides with them at the defaults. `/llm`'s hero `.sub` prints the rate cell's own `window` through the same `provenance()` tag, and the hero's `title` carries the exact `source` string — "78 tok/s" must always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_monitor`'s `setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification). **The `/llm` page reads per engine** (phase 04, decided in `.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`, laid out by phase 07): a row of two `Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**, because 0 tok/s on an idle server is healthy and `severity.py` has no throughput row) and the latency rows, each a mean with a mono `p95 …` foot (no hover; p95 needs no stored series) and a `sparkline()` fixed at 160px so rows line up. Rows are engine-scoped by the `only` flag on both components: gufo never grows a TTFT mystery, vLLM never a spec-accept hole. **The stats strip (`#strip`, phase 07 task 03) is the summary; the panels are the detail, and no row is printed in both** — the UPS hero row's "one value drawn twice" rule, widened to two components. One `.card.stat` per *configured* engine between the fleet block and the panels: the reachability dot, the engine name and the age stamp in the head (same distinction every card draws — the dot speaks reachability, the numbers never do), then the **KV-cache bar** — the one cell on the page with a verdict, so the fill and a breach's text take their colour from the payload's `severity` through `.bar.ok/.warn/.fail` and the ratio→percent conversion the panel's cell used to make, with `ok` staying quiet ink and a missing gauge an empty bar plus `n/a` — then `prompt tok/s` / `decode tok/s` with their `Δ` tags, `total tokens` (lifetime, so its source is the tooltip), the sessions this engine reports, `prefix cache hit`, `spec accept` (gufo) and `queue time` with its `p95 …` foot (vLLM). `STRIP_ROWS` is the whole table and its `key` **is** the `cellValue()` key, so every number on it is a cell of `/api/llm/snapshot`: no field was invented in `sources/llm.py` for this card, and there is deliberately **no peak-ctx row** (phase assumption). The session row is engine-shaped — vLLM publishes `running`/`waiting` counts, gufo's is `slots busy`, counted from the `sessions.list` rows `/slots` answered (`is_processing`, the shape pinned by `tests/slots_contract.py`); when `/slots` did not answer there are no rows to count, so it falls back to the published `sessions.active` carrying the API's own `sessions.detail` sentence rather than reporting 0 busy slots from no data (rule 2). `preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`, container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded threshold) and they live on this card, not in a permanent cell; the lifetime token counter left the panel's footnote line for the strip's row. **One `/api/llm/history` request per refresh** carries every chart *and* spark series (`HISTORY_SERIES` = the four chart series + `SPARK_CELLS`; the endpoint answers unknown names as empty series on purpose), and the two series phase 04 added (`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) stay stored additively in `sources/llm.py` HISTORY_METRICS. Above the strip sits **the fleet block** (`#fleet`, phase 07: "I'm missing the aggregate throughput at the top") — `ALL ENGINES`, a reachability dot, an `N of M engines` tag, the note *sums only engines answering now*, then running / waiting / prompt tok/s / decode tok/s / total tokens. `FLEET_FIELDS` in `page-llm.js` is a list of `fleet_totals()` keys and nothing else: the page sums no number of its own, a dead engine is dropped from every total presented as current (its own stale-labelled panel still shows its last good values), and the tag counts the answering engines, names the missing one in its tooltip and takes the shared `.stale` dim while the sums are partial. It **hides below two engines** — a cross-engine sum of one engine duplicates that engine's own panel — and it carries **no severity colour anywhere**: `fleet_totals` has no `severity` to render (the summed-KV bar is gone for that reason) and `severity.py` rates no throughput or queue depth, so the numbers stay ink and only the dot speaks reachability. `fleet.seams` — the API's own note on how the two engines count prompt tokens differently — is the prompt sum's tooltip rather than another line of ink. The four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels sit at their own height (`#engines` carries `.items-start`): a two-hero gufo card next to a latency-row vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". An engine that leaves the payload leaves both components (`pruneEngines`), never a stale card in either grid. **The slot grid (`llm_monitor`'s, phase 07 task 04) belongs to the gufo panel, under its `sessions (GET /slots)` label.** One card per row of `sessions.list`, **keyed by the row's own `id`** in `state.slots`: built once by `slotCard()`, patched in place afterwards, and reordered by `appendChild` of the existing node so DOM order stays slot order — rebuilding a busy slot's card every 2 s poll re-announces the whole card to assistive tech for a difference of a few tokens. `SLOT_FIELDS` is the whole table and each cell names **exactly one** `/slots` key (`path: 'next_token'` reads `next_token[0]`, where the capture puts `n_remain`); `slotValue()` is the only place a row is indexed, so `tests/test_api.py` can resolve every printed field against `tests/fixtures/slots_gufo.json` and a rename fails there rather than printing dashes or a plausible wrong number. No per-slot speed cell: the capture publishes no per-slot rate key and a page-side Δn_decoded/Δt would be a measurement with no estimator, no `Δ` label and no `source` string (rule 2) — the engine's rate lives on its strip card and panel. llama.cpp's `-1` sentinels print a word or nothing: `id_task: -1` is **`idle`**, `n_remain: -1` is **`n/a`**, never a negative countdown and never the `0` that reads as one token from done. The busy marker is a **border, a glyph (`○`/`●`) and the word `processing`** in the brand `--accent`, never an `ok|warn|fail` token — a slot generating tokens is the job (phase 06's GPU reasoning), so `severity.py` rates no slot and the stylesheet section contains no verdict token. The empty state is a sentence, never a card of nothing, and it distinguishes the two cases: no `/slots` answer prints the API's own `sessions.detail`, an answer with no rows prints *engine reports 0 slots*; `sessions.source` is the gate. The note under the grid stays because a **queued (deferred) request is not in `/slots`** — the waiting row counts those. vLLM's panel gets one line (*exposes no per-slot JSON*) with `sessions.detail` as its tooltip and **no grid**: a component of permanent `n/a`s for a server that cannot answer is the dead-cell anti-pattern. The gate is the engine name (`SLOT_ENGINES`), not row count, so an engine that has not answered yet still shows its label and its honest empty sentence. **Charts belong to the host they chart.** `/system` has two sections — the host panels and the transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals panel and the host/metric detail picker are deleted, along with the `#chart-*` / `#detail-*` ids and the `select` / `label.field` styling only those pickers used. Each panel stacks its own `.host-charts` — utilisation (CPU, memory, worst disk), network (in/out), then GPU busy — where the GPU slot is created and destroyed with `host.gpus`, so gateway shows neither a phantom card nor a hole. **That stack is collapsed behind a per-host `Charts` `
      `** (phase 06: "only visible if expanded… there will be lots of hosts"): the default `/system` view is the hero row, the summary carries the window label the stack will draw (the chips themselves stay in the page head, one control per page), and `drawHostCharts()` is the single place that decides whether to paint — so nothing is built or drawn while the stack is shut (`LineChart` inside a `display:none` container measures 0 px, and rebuilding three charts per host every poll for plots nobody sees is the waste this removes). The **history request is deliberately unchanged**: one per host per refresh, naming every series the panel could draw, and its response is cached per host so the first expand paints from data in hand rather than waiting a poll; a cached response from another window is re-fetched instead of being drawn under the wrong label. Expanding one panel must not stretch its collapsed siblings into blank cards, so `#hosts` carries the existing `.grid.items-start`. **One `/api/system/history` request per host per refresh** (`hosts=&metrics=&range=…`, pinned by `test_every_system_chart_is_drawn_inside_its_own_host_panel` and the collapse by `test_system_history_charts_are_collapsed_behind_a_toggle`): unknown metrics come back as empty series on purpose, so a panel asks for every series it could draw and asks once. A legend here names metrics (or one GPU's model), never hosts — the panel heading already says who this is, and a legend that grew a second host is the bug this replaced. A host that stops reporting loses its panel, its `Charts` toggle *and* its charts (`pruneHosts` deletes the card plus the payload/chart/history maps), never a chart of nothing or an orphan control in the grid. Per-core visuals are gone from `/system` and `renderCores` is **deleted** from `charts.js` with its last caller. `drawHostChart()` shades a chart only when one series has points — two or three filled areas stack into one block and bury the quieter line, which is how a idle CPU would hide behind a busy memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryable from `/api/system/history` — removing the plot is not removing the data. ## Known gaps / next work 1. **UPS commands work, and the account can now cut power — the app's gates are the control.** Three defects were cleared on 2026-10-17, each verified against real hardware. (a) `NutClient.login()` sent `USER`/`PASS`, which the deployed upsd 2.8.5 does not implement — each answered `ERR UNKNOWN-COMMAND`, so `_read_line()` raised and `POST /api/ups/command` could only 502 before `instcmd()`, credentials or not. It now sends `USERNAME`/`PASSWORD`, the pair the server's own `HELP` advertises (`HELP VER PROTVER GET LIST SET INSTCMD LOGIN LOGOUT USERNAME PASSWORD STARTTLS`), pinned by `test_login_uses_the_verbs_upsd_2_8_5_answers_to` — the fake upsd has no `USER`/`PASS` branch, so a regression answers `ERR UNKNOWN-COMMAND` exactly like the real thing. (b) 12 of the 23 commands the two real UPSes advertise had **no `COMMANDS` entry**, and `available_commands()` drops anything the catalog misses — the page silently offered less than the hardware does. Every advertised command is catalogued now, each with a risk tier: `test_every_command_the_hardware_advertises_is_catalogued` fails if a capture gains a command the catalog does not know, and `test_every_command_that_can_cut_power_is_in_the_dangerous_tier` keeps `load.off`, `driver.killpower` and the `shutdown.*` sequence in the tier that types `DANGER` and is refused on battery. `shutdown.stop` is the exception the name hides — it *cancels* a shutdown, so it is `safe`, and the test says so in words. (c) `[dashboards]` in `/etc/ups/upsd.users` is `instcmds = all` at the operator's request, so this app may send every advertised command on both UPSes, power-cutting ones included. `.env.dev` (mode `0600`, gitignored) carries the credentials. What remains to know: - **Proven live, and reversibly.** `beeper.enable` returned `OK` and `ups.beeper.status` read `enabled` on the APC afterwards; `driver.killpower` without the word `DANGER` still answers 422 and never reaches the wire. A battery test has deliberately not been run — it puts real load on a battery things are plugged into, which stays the operator's call. - **`enabled: true` cannot mean "this account may send it".** `available_commands()` filters on `LIST CMD`, which upsd serves **unfiltered**, and NUT has no call for a user's privileges — so the wire cannot tell the page what the account is granted. With `instcmds = all` the two agree by configuration rather than by check, which is precisely why the risk tiers and the typed `DANGER` are load-bearing and not decoration. - **Pre-existing upsd log noise:** every upsd start logs `Unknown upsmon type =` (20 occurrences over 12 starts, oldest before this work). Harmless — `upsmon` logs into both UPSes right after it — but unexplained. - **One connection every few minutes answers nothing for >4 s.** Live upsd here stalls a single poll past `NUT_TIMEOUT_SECONDS` roughly every 2–4 minutes and then answers normally again; `nut` reads `ok` throughout, because the poller's job is one amber line plus the recovery, not a frozen page (invariant 4). What phase 12's verification pass fixed on our side was the **cascade** that used to follow it — the event table filling with `expected 'BEGIN LIST VAR', got 'RW apc battery.charge.low "10"'`, which blamed the wire protocol for a stall two commands earlier (see the `NutClient` bullet above). The stall itself is an upsd/driver-side question with no repo bug in it, and a battery test is not how to chase it. Editing `upsd.users` at all: never grant `actions = -`, and never write it blind — the running `upsmon` authenticates as its own account from that same file for both UPSes, so dropping it means no orderly power-down. Copy the untouched sections from the live bytes and `cmp` them before writing; `tee` truncates in place, so owner and mode survive. 2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs (`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the field map is written. 3. **UPS history depth.** Events and the numeric series are stored; long-window battery health (capacity fade over months) has the rollups but no dedicated view. 4. **The dev DB is big enough to refuse a write now and then.** The dev store (~640 MB on 2026-10-19, its WAL grown past 140 MB at its worst) logged `store-error: OperationalError: database is locked` a few times an hour across `system`/`llm`/`ups` through 2026-10-18; the poller handles it exactly as invariant 4 requires — the source goes red with the real reason, ages, and writes `recording` when the next poll lands — so nothing freezes, but those polls' samples are lost. The one cause this repo can prove is two servers on one file, which is why `scripts/dev-server.sh stop` waits for the old pid to die (its comment records the incident). A controlled 25 s external read transaction during the phase-09 pass produced none, and phase 12's re-check on 2026-10-19 found the 190 stored `store-error` rows all older than 2026-10-18 06:33 with none in the previous 24 h of the running dev store — the absence of a sample, not the absence of the pressure, since the file only keeps growing. Retention/`wal_checkpoint` tuning for a long-running store is its own phase, not a phase-09 follow-up. 5. **The login throttle keys on the socket peer address** (phase 11), and the one `LoginThrottle` instance now holds two keys per address: the bare one for `POST /api/auth/login` and `password:
      ` for `POST /api/auth/password`. Behind nginx every login shares one bucket, so anyone who can reach the login form can lock the operator out of the *dashboard* — not the UPS, not the data, not a host — for at most `LoginThrottle.max_seconds` (1 h by default) at a time, and a process restart clears the counters. The two keys are separate on purpose, so neither form can be hammered at the other's rate and a stolen cookie plus the change form is not an unthrottled oracle for the old password; the cost of that choice is that one address can be blocked twice over. Accepted for a single-admin box rather than becoming an `X-Forwarded-For` knob: that alternative is spoofable by anyone the moment somebody forgets to configure which proxy to trust, which would hand the attacker the throttle key instead of leaving it on the network layer. PBKDF2-HMAC-SHA256 at `PBKDF2_ITERATIONS` (600 000) stays the real cost of every guess; the throttle only makes hammering the form slower than useless. 6. **One admin, one factor — no second factor and no per-user accounts, by design** (phase 11). The operator asked for a single admin, and that is what the model is: one verifier row, one username in every signed session, one name in every `password-changed` event, no read-only role and no per-viewer view. The consequence is worth stating out loud, because it is the whole security budget of a public domain: the PBKDF2 cost plus the backoff is the only thing between a stranger and the fleet's hostnames and the UPS command buttons, so there is no defence in depth behind it — no TOTP, no WebAuthn, no per-IP allow-list, no "this login is from somewhere new" notice. Adding a second human means more than a second row: the gate has no notion of scope, so a colleague would be able to cut power unless somebody designs a read-only session first. That is a phase, not an edit. 7. **The throttle is per-process, so a second uvicorn worker multiplies it** (phase 11). `LoginThrottle` is a dict in one process's memory: `uvicorn --workers 4` hands anyone who reaches the form four independent buckets and four times the attempts before a `429`, and a restart clears all of them (the gap-5 note above). Nothing else in the auth design has that problem — the session is an HMAC-signed cookie and the CSRF token derives from it, so both are worker-safe by construction with no store to share and nothing to prune. The reason this stays unfixed is that the app is single-process by nature anyway: every worker would run its own poller loops against the same SQLite file (AGENTS.md invariant 4, and the `database is locked` incident in gap 4 is what two processes on one store already looks like). If this is ever revisited, the fix is a shared counter (the `meta` table would do, at login rates), not a bigger number in one process. 8. **A logout ends this browser's cookie, not a copy of it** (phase 11). The session is an HMAC-signed cookie with no server-side record — that is the design invariant 12 and gap 7 rest on — so `POST /api/auth/logout` can only tell *this* browser to drop the cookie (verified against a running server: a replayed token from a logged-out jar still authenticates until it expires). The same is true of an `Authorization: Bearer` token, which was never in a cookie to begin with. The only revocation this app has is rotating `auth.session_secret`, which `POST /api/auth/password` does on purpose, and the reset `DELETE FROM meta …` does as the lockout escape hatch. Accepted for a single-admin box on a domain nobody else signs in to — the operator's own machine is the threat model here, not a shared kiosk — and the reason a lost laptop is a password change and not a sign-out. A real revocation list would be a row per session with a prune rule, which is the thing this design traded away. ## Locked decisions (and why) - **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10s resolution, one file the container owns. Rollups keep a year of charts at a few hundred MB. - **Our own stdlib host agent, not node_exporter.** One JSON snapshot with exactly the fields the page needs (GPU model strings, per-mount usage, temperature labels) versus ~1500 series and a re-labeling layer; no dependency to install, `ProtectSystem=strict` friendly, and read-only by construction. `scripts/capture-node-exporter.sh` remains to compare ground truth if that is ever revisited. - **Our own NUT client, not `pynut`/`upsc`.** The protocol is ~10 line commands, the third-party libs are subprocess-based or stalled, and we need instant commands with explicit gating. - **In-house Prometheus parser.** Only two upstreams use the format; keeps `prometheus_client` out of the dependency set. - **Port 8090**, inherited from the `llm_monitor` prototype this started from: 8000 is vLLM and 8731 is gufo in this fleet. - **`.env` is a seed, never an override** — container/systemd env must win. ## Definition of done (per change) The first three are exactly what `bash .agents/validate.sh` (the phase gate) checks, so no phase may be called done on a subset of them. `uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated` empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted in a browser against **live** data · fixtures re-captured if an upstream contract changed · this file and `README.md` updated when a rule, command, gap, or fleet fact changed.