phase: 02_overview_fleet_and_alerts

**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**

- Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
This commit is contained in:
2026-10-17 00:59:09 -04:00
parent ad993c7b03
commit 8b048599f3
34 changed files with 1955 additions and 225 deletions
+57 -9
View File
@@ -54,8 +54,10 @@ page still renders.
panels that matter.
4. **One dead source must not blink another.** Each source is its own asyncio task with its
own timeout and health record (`poller.py`). A failed poll keeps the last good payload and
marks it stale in the UI instead of zeroing it. Transitions write events; repeated failure
writes one line, not 500.
marks it stale in the UI instead of zeroing it — on `/` too, where a host that has never
answered still gets a fleet row (`unreachable: true`, dimmed, every number `n/a`) because
dropping the row would turn a dead agent into "never configured". Transitions write
events; repeated failure writes one line, not 500.
5. **No CDN, no build step, no internet at runtime.** Every asset is served by this app;
charts are hand-rolled SVG (`web/static/charts.js`). The engines live on a network with
no outbound access — a CDN reference is a request that never comes back.
@@ -113,6 +115,8 @@ Responsibilities, one line each:
kelvin-vs-celsius normalisation.
- `sources/nut_sim.py` — a **labelled** simulator so the UPS page is developable without a
battery. It must never be the default, and every payload it produces says `simulated`.
Its modelled `battery.mfr.date` is computed relative to today (not a fixed string that
ages), so a fresh simulator reads healthy and amber keeps meaning something.
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
- `rates.py` — Δ/window estimator, keyed by resolved series name.
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
@@ -123,8 +127,29 @@ Responsibilities, one line each:
(charge/load/runtime), `severity` inside rated LLM cells (KV pressure, classified on a
percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
- `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload,
recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts`
(`{id, severity, source, message, href, at}`, fail first then warn, then by source). A
pure projection: no I/O, no store, no thresholds of its own — it reads job states for
reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions.
Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but
older than 3× its own poll interval is `warn` while a source that stopped answering is
`fail`; a dead source gets exactly one line (its stale payload adds nothing) that names it
once — the poller's `"UpstreamError: "` wrapper and the duplicated host name are stripped,
because a message that says `gateway` twice reads as two problems; `unknown` and missing
numbers are silence; one alert per (source, metric) and at most 12, the last being a marker
that says how many it hid. This is why the overview can be two panels.
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
`/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `llm.engines`,
`ups.ups`, the `states` maps and `recent_events` stay for programmatic consumers (live
tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events` are unchanged
and `alerts` is additive — `tests/test_api.py` pins every one of those key paths, with and
without targets configured, so retiring a panel can never shrink the payload. A configured
job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
`system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
counts the machines there are and `hosts_live` the ones answering.
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
with an in-memory DB and no targets.
- `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the
@@ -133,10 +158,12 @@ Responsibilities, one line each:
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
values the API does not rate; converting its remaining call sites is phases 02–05's job.
values the API does not rate; converting its remaining call sites is phases 03–05's job
(`/` was converted in phase 02: the overview now colours only from `/api/summary` verdicts).
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tps`, `ups.charge_pct`), `@stream` for per-device series.
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
series — `/api/metrics` answers with exactly these names.
That scheme is the history API's key — treat it as a public interface.
## The fleet (real environment, verified 2026-10)
@@ -178,8 +205,8 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh logs
uv run pytest # hermetic: 356 tests, no network
uv run pytest --cov # coverage gate (94.75%; floor 94%)
uv run pytest # hermetic: 439 passed + 6 live-skipped, no network
uv run pytest --cov # coverage gate (96.8%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth
@@ -215,18 +242,39 @@ service here.
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
so it stays in the default run and is *not* marked `live`.
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
94% (`--cov-fail-under`), and ~95% is where we are; keep it there. New/modified Python
94% (`--cov-fail-under`), and ~97% is where we are; keep it there. New/modified Python
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
never a real target.
never a real target, and never a real upstream: `/api/ups/command`'s send is proven against
a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live
suite is read-only — that fake is the only place the write reaches a "driver".
## UI conventions
Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` labels on
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
dots in the header, window chips (1h…90d) on history pages. A panel that cannot get data says
dots in the header, window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API
accepts 5m…90d). A panel that cannot get data says
why in the panel — the operator should never have to open devtools to learn that a source is
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per
`/api/summary` alert: verdict class, source linked to its dashboard via the alert's own
`href`, message, age) and **Fleet**. A non-empty list floats the alerts panel above the
fleet table (`#overview-panels[data-state]` swaps them); an empty one is an explicit
"all clear" state below it, because an empty list is a message, not a blank box. A host that
is not answering keeps its row, dimmed with the shared `.stale` class and reading `n/a`, so
the table never implies last good numbers are live and never silently loses a machine.
No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the
two-panel shape against the served HTML.
Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an
`id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for`
attribute references — the orphan a deleted renderer leaves behind, caught on served bytes
because there is no JS test runner (a locked anchor). Same rule for `dash.css`: delete the
rules a removed panel made dead (`.grid.wide` went with the overview's fleet+events row);
component vocabulary the pages are still being converted to (`.hero-bar`, the grid and bar
variants) stays even while only one page uses it.
A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's
`severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*`