phase: 02_overview_fleet_and_alerts

**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**

- Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
This commit is contained in:
2026-10-17 00:59:09 -04:00
parent ad993c7b03
commit 8b048599f3
34 changed files with 1955 additions and 225 deletions
+17 -2
View File
@@ -4,7 +4,7 @@ Three dashboards, one domain, one process, one SQLite file:
| Page | What it answers | Source |
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
| `/` | Is anything wrong, and which dashboard do I need? | all of the below, one request |
| `/` | Is anything wrong, and which dashboard do I need? | fleet + alerts in one request (`/api/summary`) |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
@@ -136,8 +136,23 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
colours the same way and a value nobody reported renders `n/a` instead of green. A metric
published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays
what the upstream published.
* `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states,
host_payloads, engine_payloads, ups_payload, recent_events)` is a pure projection of the
state the poller already keeps plus the verdicts `severity.py` and the UPS health checks
already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as
`alerts`, worst first: `{id, severity, source, message, href, at}`. A configured source
that stopped answering is `fail`; one that is stale but still serving its last good value
is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than
one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a
marker that says how many it hid.
* `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No
framework, no CDN, no bundler. A big colour-coded number is `Dash.hero(el, {...})` on the
framework, no CDN, no bundler. The overview (`/`) is exactly two panels — a
**Warnings & errors** list built from `/api/summary`'s `alerts` (each row links to the
dashboard that explains it) and the **Fleet** table — and nothing else: an empty alert
list reads as an explicit "all clear", and a live problem moves the panel above the
table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the
fleet never quietly gets smaller and last good values are never mistaken for live ones.
A big colour-coded number is `Dash.hero(el, {...})` on the
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
with no verdict renders `n/a` with no colour at all.