phase: 05_ups_hero_battery_glyph

**Phase 05 final pass — all criteria met; whole project 100 % green.**

Verified (live upsd + isolated simulator):
- Real `apc`/`cyberpower`: Charge/Load/Runtime are the card's three largest elements (38.4 px vs 15.2 px next), coloured only by API `severity` — live right now 100 % ok / 79.0 % **warn** / 5m **fail**; glyph fill = charge (100/100 viewBox units, `--ok` token), `<title>` "apc: Charge 100 percent, on line"; SIMULATED hidden; battery °C `n/a` uncoloured; every button disabled with its reason; `confirm_text: DANGER` on power-cutting commands; **0 non-GET requests** → `data/logs/ups-verify-live-{card,full}.png`.
- Simulator on `:8091` (own `/tmp` DB, killed after): caution dialog → Cancel, dangerous dialog → Run disabled until `DANGER` → Cancel, **0 POSTs**; one deliberate quick test *to the sim only* flipped the card to `OB` (glyph `stroke-dasharray="7 5"`, "on battery", health "discharging, on battery") then `OL CHRG` (bolt) then `OL` → `ups-verify-sim-{dialog-caution,dialog-danger,onbattery-button,onbattery-page,charging-button}.png`; glyph matrix `ups-glyph-*.png` (100/mid/low/unknown/empty/unrated, dark/light/200 %) present; `grep -c "function battery"` = 1.

Defects found in earlier work and fixed:
1. **`poller.py` could stop polling in silence** — 4 of 6 dev-server sources had frozen `attempts` (ages 12–38 min) while `/api/health` still said `ok: true`: the store-write half of `run_once` was outside the `try`, and a dead task's exception is never retrieved. Now `_commit()` moves `last_ok` only after the write lands, `_store_refused()` reports it (`ok:false`, real reason, one `store-error` event + one `recording` on recovery, payload kept), and `_loop()` catches anything that escapes and keeps polling — 4 new tests in `tests/test_poller.py`.
2. **`nut_sim.py` (modified by this phase) sat at 83 % coverage**; its whole command path was untested. Added tests (every advertised command implemented, unknown → `ValueError`, alarms → card + `assess()` fail, `NUT_SIMULATE` truthy rule) and fixed two model lies: `beeper.*` answered OK while changing nothing (now publishes `ups.beeper.status` enabled/muted/disabled) and a finished self-test never wrote `ups.test.result` (aborted writes none). → 100 %.
3. `nut_sim.from_env()` was dead code duplicating `app._truthy`; `app.py` now builds the simulator through it, so the enable gate lives once.

Tests / lint (exact): `bash .agents/validate.sh` → **EXIT=0** — `uv run pytest --cov --cov-fail-under=94` **543 passed, 8 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` clean; `uv run pyright` 0 errors; `node --check` clean; repo hygiene clean. `uv run pytest -m live --live` **with `.env.dev` sourced → 8 passed** (one earlier run: 2 transient `9121/snapshot timed out` against the 4 s `HOST_TIMEOUT`; three later runs 8/8). `uv pip list --outdated` empty. Dev server restarted on the final code: `health ok`, 6/6 sources fresh.

Docs: AGENTS.md — invariant 4 extended ("nothing may stop polling in silence" + store-refusal semantics), `poller.py`/`nut_sim.py` responsibility lines, 533→543 and 96.9 %→97.9 %, "~97 %"→"~98 %". README.md — poller bullet gained the same guarantee, Testing gained "source `.env.dev` before `-m live --live`" (bare runs skip 7 of 8 live tests, which reads like a pass), Battery-test section gained the `deploy/nut-sudoers` install line and the read-before-write warning.

Notes/deviations: no live instant command was ever sent to real hardware (the four gates leave them disabled — AGENTS gap 1 unchanged); the on-battery proof was driven through the real button against `NUT_SIMULATE=1`, which is stronger than the NOTES hand-driven states. `NOTES-verify.md` under `todo/` left untouched per instructions. Next phase: **none** — 05 is the last in the PLAN.md roadmap.
This commit is contained in:
2026-10-17 10:27:23 -04:00
parent f44a617ef9
commit 34854d32af
44 changed files with 2374 additions and 105 deletions
+36 -8
View File
@@ -132,7 +132,9 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
* `poller.py` — every source gets its own asyncio task, its own health record, and its own
error handling. A dead host keeps its last good values and is labelled stale; it never
blanks or blocks another source.
blanks or blocks another source. And it never quietly *stops*: the task is that source's only
poller, so a SQLite write that gets refused is reported (`ok: false`, the real reason, the
numbers aging, one line in the event log) instead of freezing the source under a green dot.
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
pruned, so a year of history is a few hundred MB rather than a disk emergency.
* `severity.py` — the single green/amber/red table (`THRESHOLDS`, plus `INVERTED` for the
@@ -162,7 +164,13 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
A big colour-coded number is `Dash.hero(el, {...})` on the
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
with no verdict renders `n/a` with no colour at all. Each `/system` host is one panel led
with no verdict renders `n/a` with no colour at all. A battery is
`DashCharts.battery(el, {charge, severity, state, label})`: one inline SVG whose fill *is* the
charge — clamped to 0–100, never rounded up, clipped to the plate so 0 % is an empty shell —
coloured only by the payload verdict, with a shape per state (solid shell on line, bolt
charging, dashed outline on battery, a dash through the body when there is no reading) and a
`<title>` / `aria-label` that names it ("apc: Charge 100 percent, on line") unless the number
beside it already does. Each `/system` host is one panel led
by exactly six of them — CPU %, MEM %, TEMP, DISK % (mount underneath), GPU busy (VRAM and
the APU pool in the footnote), and NET on two lines (↓ in / ↑ out, never coloured: a byte
rate has no good/bad). Under the numbers, **each panel charts only its own host**: utilisation
@@ -183,16 +191,23 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
while non-zero; the token counter is a footnote. The four trend charts sit below the panels
at half height, and one `/api/llm/history` request per refresh feeds all of it. With a
single engine configured the totals card hides and the page is one clean half — never a red
panel for a server you do not run.
panel for a server you do not run. Each UPS card (`/ups`) opens with its battery glyph and
three big numbers — **charge %**, **load %**, **runtime left** — coloured from the card's
`severity` dict, with the electrical readings (line in, output, battery volts and amps,
battery °C) kept below them in smaller muted type because they are what you read *while* the
UPS is on battery. Health checks, the battery-test buttons (with their `DANGER`
confirmation) and the NUT variable dump are unchanged, just second.
## Testing
```bash
uv run pytest # unit + API tests, fully mocked, no network
uv run pytest --cov # coverage gate
set -a; . ./.env.dev; set +a # the live suite reads the same env as the app
uv run pytest -m live --live # optional: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright # lint + types
./scripts/capture-fixtures.sh # re-record ground-truth captures
./scripts/capture-fixtures.sh # re-record ground truth: host agents + UPS replies
./scripts/capture-fixtures.sh --with-llm # ... and the engines' /metrics while they are up
```
Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must
@@ -200,15 +215,23 @@ match an upstream (`/proc` layouts, vLLM metric names, NUT protocol framing) is
against a verbatim capture in `tests/fixtures/`; behaviour that needs a failure — upsd
refusing a command, an engine returning 500, a host rebooting — is a fake. The `live` tests
(`tests/test_live_fleet.py`) are the bridge: they SKIP unless you pass `--live`, each one
also skips when its source is unset, they drive the production code paths (`collectors.*`,
`NutMonitor.collect`) against whatever hardware is reachable, and they never write — a
battery test on live hardware is not a unit test.
also skips when its source is unset (**source `.env.dev` first** — run bare and they all skip
quietly, which reads exactly like a pass), they drive the production code paths
(`collectors.*`, `NutMonitor.collect`) against whatever hardware is reachable, and they never
write — a battery test on live hardware is not a unit test.
One more file is worth knowing about: `tests/card_contract.py` is the table of
`/api/system/snapshot` paths the system page's per-host cards are allowed to read. It is
checked against all three captured host shapes (discrete GPU, APU, no GPU), over HTTP, and
again against the live fleet, so a card can never be "fixed" by inventing a number the agent
does not report.
does not report. `tests/ups_contract.py` does the same job for the UPS page's hero row and
battery glyph (`charge %`, `load %`, `runtime left`, `battery_voltage`, the glyph's
`charger_status` + `on_battery`, the card header), against the two real UPSes — recorded
verbatim as `tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`, with the
device serials scrubbed — and again against live upsd. The captures earn their place: the
CyberPower answers `battery.mfr.date: CPS`, the APC's `battery.date` is a factory
placeholder, and **neither** publishes `battery.temperature` or `battery.charger.status`, so
a hand-written variable table would have tested nobody's memory.
## Battery-test buttons
@@ -229,6 +252,11 @@ not `actions = -`:
instcmds = test.battery.start.quick test.battery.start.deep test.battery.stop
```
`deploy/nut-sudoers` is the ruleset that lets the `ops` account read (and scoped-write) those
files: `sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut`.
Read `upsd.users` before you write it — `upsmon` authenticates from the same file, so
overwriting it blind can take the box down at the wrong moment.
Bind upsd to loopback (`LISTEN 127.0.0.1` in `upsd.conf`) and reach it with an ssh forward
(`DEV_FORWARDS`) or a tailnet address. `NUT_SIMULATE=1` stands in for upsd during
development; everything it produces is badged `SIMULATED`, and it is never the default.