diff --git a/.agents/phases/todo/03_system_per_host_panels/NOTES-verify.md b/.agents/phases/todo/03_system_per_host_panels/NOTES-verify.md deleted file mode 100644 index e6c4862..0000000 --- a/.agents/phases/todo/03_system_per_host_panels/NOTES-verify.md +++ /dev/null @@ -1,66 +0,0 @@ -# Verification notes — `/system` on every host shape - -**Date:** 2026-10-17 (dev loop: tunnels 9121/9122/9123 + upsd 3494 all OK; dev-server -`/api/health` all six sources `ok` before and after verification) - -## What was checked, on live data - -Verified against `http://127.0.0.1:8090/system` with all three real agents answering, in a -real browser (headless Chromium/Playwright), DOM assertions plus screenshots. - -| Host (shape) | Result | Screenshot | -| --- | --- | --- | -| `driveripper` (discrete Radeon AI PRO R9700) | GPU hero: `3 %` ok, `.sub` = "Radeon AI PRO R9700 · VRAM 30.5 GiB / 31.9 GiB · pool 33 %"; util/net/GPU charts all present inside the panel | `system-driveripper.png` (dark), `system-driveripper-light.png` | -| `deskwork` (Strix Halo APU, two memory pools) | GPU hero busy went 82 % ok → 85 % warn → 99 % fail across the session, so threshold colour visibly moves; `.sub` = "Radeon 8060S Graphics · VRAM 172.7 MiB / 512.0 MiB · pool 78 %" — small dedicated VRAM and high shared pool read as two facts, not a contradiction; MEM 91 % and TEMP ~83 °C amber | `system-deskwork.png` | -| `gateway` (no GPU, no CPU temp sensor) | CPU TEMP `n/a` with **no** severity class; GPU slot = `n/a` + "No GPU reported by this agent", uncoloured; **no** GPU chart and no hole where one would be | `system-gateway.png` (dark), `system-gateway-light.png` | - -Full-page shapes: `system-shapes-dark.png`, `system-shapes-light.png`. - -Failure path: stopped only the gateway tunnel (same commands `dev-tunnels.sh stop` runs, -scoped to the 9123 entry; remote agent stopped too). After 15 poll failures the panel kept -its last-good heroes (MEM 32 %, DISK 61 %, NET rates — nothing zeroed, nothing `n/a`), its -dot turned red, the header read `stale 2.1m`, and the card dimmed via `.stale`. The other -two panels stayed green ("one dead source must not blink another"). Store recorded exactly -one `unreachable` and, after `./scripts/dev-tunnels.sh start`, one `reachable` event. -Screenshot: `system-stale-gateway.png` (+ `system-stale-overview.png` full page). Restored; -`dev-tunnels.sh status` and `/api/health` back to all-OK. - -Accessibility pass: - -- hero values are real selectable text (`div.value > span.num/span.unit`, no SVG paths) — - asserted in-page for every hero on all three panels; -- colour is never the only signal: each hero prints the number, the NET slot is - deliberately colourless, staleness reads as text (`updated Ns` / `stale 2.1m`); -- charts — **defect found**: the SVGs were anonymous (`aria-label` absent on container and - svg). Fixed: `LineChart` gained an `ariaLabel` option that sets `role="img"` + - `aria-label` on the ``, and every `/system` chart now names its host and metrics - (e.g. "driveripper — utilisation over the selected window: CPU, memory and worst disk in - per cent"). Pinned by `tests/test_api.py::test_every_system_chart_names_itself_for_assistive_tech`. - The `/llm` and `/ups` charts can adopt the same option in phases 04/05. - -Both themes checked via the header toggle (severity colours come from theme tokens; the -light-theme deskwork fail-red and warn-amber read clearly — see -`system-shapes-light.png`, `system-gateway-light.png`). Note for the next shooter: set -`prefers-color-scheme` or seed `localStorage['dash-theme']`; Playwright defaults to light, -which silently mislabels a "dark" screenshot. - -`curl "localhost:8090/api/system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h"` -→ 354 points: dropping the per-core plot did not drop the data. - -## Defects found and fixed during verification - -1. **Unlabelled charts** (above) — fixed in `charts.js` + `page-system.js`, test-pinned, - re-screenshotted. -2. **`/system` Events panel was permanently empty** — the poller stores transition events - under the *dashboard* name (`system`) while the page queried `/api/events?source=host`, - so the feed said "No host transitions recorded yet" even with the deliberate gateway - outage in the store. Fixed to `source=system` (with a comment on the naming trap); - `tests/test_api.py::test_system_event_feed_queries_the_source_transitions_are_stored_under` - pins it; the re-shot full-page screenshots show the feed populated with the outage and - recovery. Pre-existing since the first commit, not introduced by phase 03. - -## After the fixes - -`bash .agents/validate.sh` → exit 0: 460 passed + 7 skipped, coverage 96.86 % (≥ 94), -ruff + pyright clean, `node --check` clean. `uv run pytest -m live --live` → 7 passed. -Console stayed error-free throughout. diff --git a/.agents/phases/todo/04_llm_chart_readability/NOTES-chart-set.md b/.agents/phases/todo/04_llm_chart_readability/NOTES-chart-set.md deleted file mode 100644 index 7afb58e..0000000 --- a/.agents/phases/todo/04_llm_chart_readability/NOTES-chart-set.md +++ /dev/null @@ -1,206 +0,0 @@ -# NOTES-chart-set — what `/llm` is for, and what it shows - -**Phase 04, task 01 deliverable (no code changes).** Decides the cell set, the colour rules, -and the contracts tasks 02–03 implement. Both implementations were read side by side on -2026-10-17; every line number below was verified against the file it cites. - -## The two sources - -**Reference — `~/Projects/Experimental/llm_monitor/static/index.html`** (separate app on -`:8095`; design reference only, never a dependency — ASSUMPTION kept: it is not ours to own, -and importing it would break the no-build single-app structure): - -| What | Where | -|---|---| -| sparkline CSS vocabulary (`.sparks`, `.spark`, `.spark-now`, `.spark-axis`, `.spark-line`, `.spark-area`) | L398–495 | -| `UI.historyLen = 60` (client ring buffer; `pushHist` L1542–1545; fed per engine L1570–1571, L1582–1583) | L1003 | -| `drawSpark(name, values, format, fixedMax)` | L1092 (`SPARK_W = 100`/`SPARK_H = 30` L1090–1091) | -| right-aligned growth — `const step = SPARK_W / (UI.historyLen - 1)`, `x = SPARK_W - (values.length - 1 - i) * step` | L1158–1163 | -| empty state: `< 2` values → `empty` class, blank paths (never a zero line) | L1112–1118 | -| flat-window honesty: pad keeps the line mid-plot, axis reports the one real value, not the pad (`const flat` L1128) | L1126–1133 | -| top/middle/bottom scale marks (`axis()` helper, `levels`, `fmtAxis` with enough decimals that the three labels stay distinguishable; axis writes L1155–1157) | L1101–1104, L1135, L1138–1154 | -| `fixedMax` ratio axes: KV sparks called with `fixedMax = 1` → 0–100% always | L1356 (gufo), L1420 (vLLM) | -| per-engine throughput `dd`s: `#gufo-ptps` / `#gufo-dtps`, `#vllm-ptps` / `#vllm-dtps` | L757 / L761, L802 / L806 | -| `setThroughput(el, cell)` — value + honest tag (`Δ` for measured, `lifetime avg`, `from TPOT`, `derived`; `title` = exact `source`) | L1303–1320 | -| percentile foot lines: `#vllm-ttft-p95`, `#vllm-tpot-p95`, `#vllm-queue-p95` under their mean cards | L950, L955, L960 | -| token counters as cells whose foot line names the upstream series (`#vllm-ptok`, `#vllm-gtok`) | L964, L969 | -| slot cards: `SLOT_FIELDS` L1170–1177, `buildSlotCard` L1179 (built once, patched in place), `renderSlots` L1209 (DOM order == slot order, vanished slots removed) | — | -| the only thresholds in that app: KV bar warn ≥ 0.75, crit ≥ 0.9 | L1085–1086 | -| single-engine case: fleet block hidden when one engine — "a cross-engine sum of one engine just duplicates that engine's own card" | L1500–1504 | -| dead engine holds last good data + `stale` class + error banner, never zeroes | L1514–1535 | - -**Current page** — `src/dashboards/web/pages/llm.html` + `src/dashboards/web/static/page-llm.js`: - -| What | Where | -|---|---| -| engine-totals card (`#f-decode` L45, `#f-prompt` L47, KV bar L54–56, tokens lifetime L52–53) | `llm.html` L39–60 | -| four full-height `LineChart`s `#chart-decode/prompt/kv/queue` | `llm.html` L62–81 (chart divs L66, L71, L75, L79) | -| per-engine panels grid `#engines` + honest "No engines configured" empty state | `llm.html` L83–91 | -| gufo slots table `#slots-card` L94–102; events feed L103–106; provenance footer L109–113 | `llm.html` | -| `CELLS` — the 14-equal-size grid this phase dissolves | `page-llm.js` L16–33 | -| `cellValue()` — per-key payload mapping | L35–93 | -| `provenance()` — `Δ10.0s` / formula tag from `source`, nothing for scraped gauges | L96–105 | -| `enginePanel()` / `renderPanel()` | L112–136 / L138–214 | -| **JS thresholds to remove**: `severityFor(value.value, 75, 90)` on the KV bar | L194–195 | -| device_lost non-zero special case (the alarm-chip seed) | L196–200 | -| `makeCharts()` — `height: 165` for all four | L241–249 (165 at L242) | -| `refreshCharts()` — three `/api/llm/history` calls; the third already carries `requests_running,requests_waiting,kv_cache_usage` | L256–292 (L264–265) | -| **JS thresholds to remove**: `severityFor(kv, 75, 90)` on fleet KV | L307 | -| window chips `Dash.rangeControl` (affects charts today, sparklines after this phase) | L374 | - -Shared helpers this builds on: `charts.js` `LineChart` L43, `renderLegend` L329 (with -`showPeak`), `sparkline` L348–373 (the function task 02 upgrades); `app.js` `severityFor` -L118 (documented FALLBACK ONLY — phases 04–05 exist to retire its remaining call sites), -`severityFrom` L139, `hero()` L187. - -## Decision - -### KEEP (unchanged, do not regress) - -- **Provenance tags on every throughput number** — `provenance()` (`page-llm.js` L96–105) - and the footer legend (`llm.html` L109–113). AGENTS.md invariant 2; the `llm_monitor` - equivalent is `setThroughput` L1303–1320. Ours is strictly better (it prints the measured - Δ window); it ports as-is into the hero cells. -- **History window chips** (`page-llm.js` L374, `Dash.rangeControl`). Divergence from the - reference on purpose: `llm_monitor` sparks use a fixed 60-sample client buffer (L1003); - our sparklines read `/api/llm/history` with the *selected* window, so chips and sparks - always agree. Do not copy `historyLen`. -- **Per-engine panels** (`llm.html` L84) with per-source health, stale/last-good dimming, - and the upstream chip. Unconfigured engines render no panel at all (L85–90) — invariant 3. -- **gufo slot list** (`llm.html` L94–102, `renderSlots` L216–239). Adopt from - `llm_monitor` only the *in-place patch* idiom (`buildSlotCard` L1179) if rows churn; the - table itself stays. -- **Events card** (`llm.html` L103–106). Phase 02 deleted the raw feed from `/` only; - per-engine up/down transitions stay here. - -### HERO — the two largest numbers, per engine panel - -- `llm.decode_tokens_per_second` and `llm.prompt_tokens_per_second` (payload cells built in - `sources/llm.py`: gufo measured Δcounters, vLLM native gauge → measured → lifetime - estimate, each labelled with its `source`). -- Rendered with `Dash.hero()` (`app.js` L187) with the `Δ` provenance in `.sub`. -- **Throughput has NO severity.** `severity.py` `THRESHOLDS` (L46–57) has no throughput row - and gets none: 0 tok/s on an idle server is not red, and a page-invented threshold is - exactly what invariant "a threshold exists nowhere else" forbids. The number renders in - neutral ink; colour in the hero row comes from the engine's reachability (status dot + - `.stale` dimming, already implemented in `renderPanel`), mirroring the `/system` NET slot - precedent ("throughput has no good/bad", AGENTS.md UI conventions). *Note: the phase - checklist says the heroes are "colour-coded" — the honest reading is the page's colour - language (neutral numbers, verdict colour on rated cells, reachability colour on the - panel). Task 04's side-by-side screenshots should show this; if the operator wants the - digits themselves tinted, that needs a ratified `severity.py` row first, not a JS guess.* - -### SPARKLINE CELLS — compact rows, one per metric, each with its window's history - -| Cell | Payload source (`cellValue`, L35–93) | History series for the spark | Colour | -|---|---|---|---| -| Sessions active | `sessions.active` (gufo `/slots`, vLLM count) | `llm.requests_running` ✔ stored | none | -| Queued | `requests_deferred` / `num_requests_waiting` | `llm.requests_waiting` ✔ stored | none | -| KV cache % | `kv_cache_usage_ratio` / `gpu_cache_usage` (×100) | `llm.kv_cache_usage` ✔ stored | **payload `severity`** | -| TTFT | `time_to_first_token_seconds__mean`, foot: `__p95` | `llm.ttft_seconds` ✔ stored | none | -| TPOT | `time_per_output_token_seconds__mean`, foot: `__p95` | `llm.tpot_seconds` — **new, see Python delta** | none | -| Queue mean | `request_queue_time_seconds__mean` | `llm.queue_seconds` ✔ stored | none | -| Prefix cache hit % | `prefix_cache_hit_ratio` / `prompt_cache_hit_ratio` | `llm.prefix_cache_hit_ratio` ✔ stored | fixed max 100 | -| Spec accept % | `spec_accept_rate` (gufo only) | `llm.spec_accept_rate` — **new, see Python delta** | fixed max 100 | - -- Percentiles print as **foot lines, no hover** (reference: L950/L955/L960), from the live - snapshot cells — p95 needs no stored series. -- TTFT/TPOT/queue are vLLM-only; spec accept is gufo-only. The other engine's row renders - `n/a` uncoloured (rule 2), or the row is engine-scoped — task 03's choice, but a - gufo panel must never show a red/empty latency mystery. -- `/api/llm/history` (`api.py` L353–365) answers *any* metric name; unknown ones come back - as empty series — so the page can request the full cell list in one call. `refreshCharts` - (L256–292) already makes three calls; extend the third (L264–265) with the cell series - instead of adding requests, and feed both charts and sparks from the same responses. - -### CONDITIONAL / DEMOTED - -- **`tokens_total` (lifetime counter)** → quiet footnote line in each engine panel (the - existing `.sessions-note`/`.debug` strip, `page-llm.js` L203–213), like `llm_monitor`'s - counter cards whose foot line just names the series (L964, L969). A lifetime sum is not a - glance metric. -- **`preemptions` and `device_lost`** → **alarm chips rendered only when non-zero, absent - otherwise** (seed: the `device_lost` special case L196–200). A permanent cell for - something that is always 0 trains the eye to ignore the row. Chip class is `.chip.fail` - (existing vocabulary, `dash.css` L336); the rule is *existence of an incident*, not a - graded threshold — `sources/llm.py` already treats any non-zero `gufo_device_lost_total` - as an incident (comment at `GUFO_SERIES`), and zero/null simply draws no chip. -- **Engine-totals card (L39–60)**: keep engines-live, decode/prompt sums, active, queued — - demote tokens to footnote, and **remove the fleet KV bar** (L54–56 + `severityFor` at - L307): `fleet_totals` (`sources/llm.py` L325–368, `kv_cache_usage_max` L360–363) carries - no `severity`, and the rated per-engine KV cells plus the KV chart say it better. Hide the - totals card when fewer than two engines are configured — copy `llm_monitor` L1500–1504 - ("a cross-engine sum of one engine just duplicates that engine's own card"); this also - satisfies the single-engine completion criterion. - -### CHARTS - -- All four `LineChart`s stay: decode, prompt, KV, queue — one engine per series (queue chart: - running+waiting per engine), existing legend `renderLegend` (`charts.js` L329) with - `showPeak` on the bursty throughput plots, units-bearing y-format (already passed). -- **Drop `height: 165` → ~110** (`page-llm.js` L242) **and move the chart section below the - engine panels** so the cells read first (the task says half-height "so the cells read - first"; keeping full-height charts on top would contradict that). -- Task 03 must pass `ariaLabel` to all four charts (AGENTS.md: every chart names itself; - `/system` did this in phase 03, `/llm` never adopted it). - -### COLOUR RULES (from `severity.py` and nowhere else) - -- **KV pressure**: rated. `THRESHOLDS["llm.kv_cache_usage_pct"] = (75, 90)` - (`severity.py` L53), attached per engine cell by `SEVERITY_CELLS` (`sources/llm.py` - L99–103) with `scale=100` because both engines publish a 0–1 ratio. The page reads it via - `Dash.severityFrom` (`app.js` L139) and **both `severityFor(..., 75, 90)` call sites - (L194–195, L307) are deleted** — the `app.js` L112–117 fallback comment stays true only - if phase 04 does this. -- **Queue depth**: **not rated** — `THRESHOLDS` has no queue row (L46–57). `severityFrom` - returns `''` → no colour. The queue chart and sparkline carry the signal as *shape*. If - the operator wants a verdict, that is a future `severity.py` row (ratified numbers, - tests), never a JS threshold. -- **Throughput, sessions, latency (TTFT/TPOT/queue mean)**: not rated → colourless numbers. - Explicit statement per the task: **0 tok/s on an idle server is not red.** -- **Alarm chips**: `.chip.fail` when the counter is non-zero; absent otherwise (above). - -### SPARKLINE CONTRACT (input for task 02, mirroring `drawSpark` L1092–1176) - -`DashCharts.sparkline()` (currently L348–373: full-width autoscale, no labels) becomes: - -1. **Right-aligned growth**: newest sample pinned right, x mapped over the *selected time - window* (`[now − range, now]`, like `LineChart._xExtent`) rather than over sample index — - a partly-filled window grows in from the right exactly like `drawSpark`'s fixed-step - buffer (L1158–1163) but stays honest about *when* samples happened. -2. **Fixed max** for ratio metrics (KV, prefix hit, spec accept: 0–100) so a squiggle means - the same thing on every screen — reference `fixedMax` calls L1356/L1420; autoscale - otherwise. -3. **Current/max labels**: current value in the cell header (reference `.spark-now`, L419) - and window max as the label; where the reference prints three axis marks (L1135, - L1155–1157) our compact rows print `now` + `max` — adopt its formatting discipline: - enough decimals that labels stay distinguishable (L1137–1152), a flat window reports the - one real value, never the pad artifact (L1126–1133). -4. **Severity colour**: stroke/area from the payload verdict (`ok`/`warn`/`fail` tokens from - `severityFrom`, unrated → `--series-1`), not `llm_monitor`'s fixed per-engine hue. -5. **Empty**: < 2 finite points → "no history", never a zero line (reference L1112–1118; - current code already does this). - -## Python delta (the phase's only one, flagged for task 03) - -`HISTORY_METRICS` (`sources/llm.py` L113–123; sampled at L292–297) does **not** store TPOT -or spec-accept, so those two cells would show permanent "no history" slots — the dead-cell -anti-pattern this note exists to prevent. Decision: extend `HISTORY_METRICS` additively — - -- `tpot_seconds` ← `("time_per_output_token_seconds__mean",)` (vLLM only) -- `spec_accept_rate` ← `("spec_accept_rate",)` (gufo only) - -New *names* in the public metric namespace (allowed: "phases may stop plotting a series, -they may not rename or stop storing it" — adding is the safe direction), consumed by the -generic `Sample("llm", engine, f"llm.{metric}", …)` loop (L292–297) and the store's generic -rollups; retention needs nothing new. The phase overview says "Python: none *expected*" — -this is a small, additive, tested exception; `/api/llm/*` fields keep their names/meanings -(additive-only anchor holds), and task 03 adds a unit test per new series plus an -`/api/llm/history` round-trip. Everything else in this phase is HTML/JS/CSS. - -## Non-goals - -- No changes to `/api/llm/snapshot` consumers outside this page, no `/ups`-style hero glyph - work (phase 05), no per-slot sparklines, no touching `llm_monitor` itself. -- The stored-metric set is never pruned by this phase: cells leave the *page*, series stay - in SQLite (PLAN.md, dropped-visuals anchor). diff --git a/.agents/phases/todo/05_ups_hero_battery_glyph/NOTES-verify.md b/.agents/phases/todo/05_ups_hero_battery_glyph/NOTES-verify.md deleted file mode 100644 index 199fd18..0000000 --- a/.agents/phases/todo/05_ups_hero_battery_glyph/NOTES-verify.md +++ /dev/null @@ -1,109 +0,0 @@ -# Verification notes — the hero row on real batteries, and the states only the sim can give - -**Date:** 2026-10-17. Real pass: dev tunnels 9121/9122/9123 + upsd forward `3494 → driveripper:3051` -all OK, `dev-server.sh status` six sources `ok`, `/api/ups/snapshot` `simulated: false`. -Simulator pass: the production app (`create_app()` with the real poller, payload builder and -severity table) run in-process on `127.0.0.1:8091` with `NUT_SIMULATE=1` — states driven through -`SimUps.tick()/step()` and `NutMonitor.read()` exactly as the 5-second poll loop drives them. -All screenshots are headless Chromium (the accepted verification browser for phases 02–04), dark -theme, poll paused before each shot; every shot was opened and read, not just written. - -## 1. Real fleet — `data/logs/ups-real-apc.png`, `ups-real-cyberpower.png` - -| | `apc` (Back-UPS RS 1500MS) | `cyberpower` (GX1500U / PR1500LCDRT2U label) | -| --- | --- | --- | -| glyph | full green cell, solid shell, no bolt | same | -| Charge | **100 %** green, `24 V nominal · 27.3 V` | **100 %** green, `24 V nominal · 27.2 V` | -| Load | **48.0 %** green + bar (no `realpower` from this driver → no watt sub) | **14.0 %** green + bar, `127 W of 900 W` | -| Runtime left | **10m** green, `at the current load` | **56m** green, `at the current load` | -| Battery age check | **warn** — "5.3 years — past the 4-year replacement window" | **unknown** — "no battery.mfr.date / battery.date from this driver" (`battery.mfr.date: CPS` did not become a fake age) | -| Battery °C | `n/a`, uncoloured | `n/a`, uncoloured | -| health chip | `degraded` (from the age warn) | `degraded` (from "No test initiated") | -| SIMULATED badge | **absent** (`hidden`, and no banner) — real upsd read | **absent** | - -The two driver disagreements the task calls the point are visibly different on the page: the APC -gets an amber age warning, the CyberPower gets a `?` "unknown" — the page is not rendering a -template. Zero POSTs on the real page. - -## 2. States only `nut_sim.py` can produce — five shots, badge visible in every one - -`NUT_SIMULATE=1`, `NUT_ALLOW_ACTIONS=0`; apc card driven to each state (cyberpower left at 100 % -on line as the contrast). The `SIMULATED` chip is visible on the card header in every shot, and -the full-page shot (`ups-sim-12-low-page.png`) shows the "SIMULATED DATA. NUT_SIMULATE=1" banner -plus the "Command buttons are disabled" info banner. - -| Shot | State proven (API-confirmed before the shot) | -| --- | --- | -| `ups-sim-100-online.png` | 100 % / `ok` green, glyph solid shell **no bolt**, chips `fully charged, online, OL`, health healthy | -| `ups-sim-60-charging.png` | 59 % / `ok` green, glyph fill ~59 % **with bolt**, chips `charging, online, OL CHRG` | -| `ups-sim-35-onbattery.png` | 34 % / **`warn` amber**, glyph dashed shell (on-battery shape), chips `discharging, on battery, OB`, battery V 21.8 V | -| `ups-sim-12-low.png` (+ `ups-sim-12-low-page.png`) | 12 % / **`fail` red**, `OB LB` → chip `low battery[fail]`, glyph red dashed shell nearly empty, runtime `9m` amber, health `at risk`, events `apc OL → OL CHRG → OB → OB LB` | -| `ups-sim-nacharge.png` | `battery.charge`/`battery.runtime` not exported → Charge **`n/a`** and Runtime **`n/a`**, no severity class on either, glyph = grey outline, empty body, no colour; checks say "battery.charge is not exported" / "battery.runtime is not exported (the driver cannot estimate it)"; health chip `unknown` | - -Runtime on the low-charge shots is the model's own estimate (`step()` drain under load), not a -typed-in number; the 99.9 %→`full` and <20 %→`LB` boundaries fired on their own as charge fell. - -## 3. Defects found and fixed while doing this - -1. **The simulator could not produce the "charging" state.** `SimUps.vars()` answered `ol`/`OL` - while the pack was recharging and `ol` even while `OB` near full, so `charger_status == "chg"` - — the only route to the glyph's bolt — was unreachable in dev, and flag and token - contradicted each other. Fixed (`sources/nut_sim.py`): on battery → `OB`(+`LB`)/`dchg`; - on line below float → `OL CHRG`/`chg` (exactly what a real APC prints); ≥99.9 % → `OL`/`full`. - Pinned by `test_a_recharging_simulated_ups_reports_charging_like_a_real_driver` and - `test_a_discharging_simulated_ups_is_never_online`. The `ups-sim-60-charging.png` above was - shot after the fix; that state is now reproducible for anyone (`drain on battery`, come back - on line). -2. **The chip row said "charging" twice** on drivers exporting both `battery.charger.status` and - the legacy flag (made visible by 1). `UpsReading.states()` now emits one chip per fact — the - 2.8 token wins, the subsumed flag is dropped — pinned by - `test_states_never_say_the_same_word_twice` (incl. the `dchg`+`OB DISCHRG` pair and the - flag-only route keeping both facts). - -Both were pre-existing (the old code duplicated `online` the same way for any token+flag pair); -neither changes a stored metric, a severity, or a gate. - -## 4. Dead source — `data/logs/ups-stale-lastgood.png` - -Stopped **only the 3494 forward** (the generated supervisor + its `ssh -N`, the scoped variant of -`dev-tunnels.sh stop`; remote `nut-server` untouched). After 75 s: red banner -`upsd: NutError: cannot reach upsd at 127.0.0.1:3494`, both cards keep their last-good heroes -(100 % / 44.0 % / 12m, 100 % / 14.0 % / 59m — nothing zeroed, nothing `n/a`), dots red, header -`stale 75s`. Restored with `./scripts/dev-tunnels.sh start`; status and `/api/ups/snapshot` back -to answering real upsd (`simulated: false`). - -## 5. The command gate — exercised, cancelled, nothing sent - -On the live page every command button is disabled server-side ("NUT_USER / NUT_PASSWORD must name -a user with `actions` in upsd.users" — AGENTS gap 1, unchanged), so the dialog was exercised on -the simulator run **with `NUT_ALLOW_ACTIONS=1`**, where the buttons are genuinely enabled — no -DOM hacks — and Cancel guarantees nothing is sent: - -- **Quick battery test** (caution) → dialog `risk: caution`, "this UPS is simulated" in the - target line, no DANGER input for a caution command → **Cancel**, dialog closed - (`ups-dialog-caution.png`). -- **Deep battery test** (dangerous) → dialog with `dangerous` fail chip, Run **disabled**, types - `DANGER` → Run enables (`action dangerous`) → **Cancel** - (`ups-dialog-danger-typed.png`). -- Fetch spy recorded **0 POSTs** across the session, and the sim's `ups.test.result` was still - `PASSED (2026-06-14)` afterwards — the command never reached a "driver". - -No live instant command was sent; the operator did not ask for one. (`ups.test.result` on both -real UPSes remains `No test initiated` — still a warn row each.) - -## 6. Note on the event feed during the outage - -The event log in the stale screenshot carries failure/recovery flapping from ~08:33 onward that -predates this task (another process was restarting the forward and probing it — see the -`expected 'BEGIN LIST VAR', got 'CMD apc beeper:enable'` rows at 08:24, which is a raw-socket -probe sharing the port, not the dashboard's client; our poller speaks one request per -connection). Each row is a genuine transition, and the invariant-4 rule (one line per repeated -failure) held during the clean 09:10 outage: exactly one fail line and one recovery line. - -## After the fixes - -`bash .agents/validate.sh` → **EXIT=0**: 533 passed + 8 skipped, coverage **96.98 %** (floor 94), -ruff + pyright clean, `node --check` clean. `uv run pytest -m live --live` → **8 passed** (both -real UPSes through the hero contract, agents, engines). Docs: `AGENTS.md` — sim status semantics, -one-chip-per-fact rule, test count 530→533; no README change needed (`NUT_SIMULATE` wording was -already right).