phase: 02_overview_fleet_and_alerts
**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**
- Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
This commit is contained in:
@@ -0,0 +1,47 @@
|
||||
# Phase 02 — Overview is fleet and alerts
|
||||
|
||||
**Source:** `TODO.md` L3 — "The overview page should only show Fleet and warnings/errors. Should be very simple."
|
||||
**Story:** n/a
|
||||
**Context:** `.agents/PLAN.md` → *Data Model* (`/api/summary`), `01_severity_language` (alerts reuse the verdicts), `src/dashboards/api.py:153` (`/api/summary`), `src/dashboards/web/pages/index.html`, `src/dashboards/web/static/page-index.js`
|
||||
|
||||
## Objective
|
||||
|
||||
Turn the landing page into one question answered instantly: *is anything wrong, and which
|
||||
page do I need?* Fleet table plus a warnings/errors panel — the three summary cards and the
|
||||
raw event feed go away.
|
||||
|
||||
The event list is what makes the current overview busy. Its useful content is not "vLLM is
|
||||
answering again" (that is history) but "something is wrong right now", which is a derived
|
||||
state and belongs server-side.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `01_severity_language` (todo) — alerts are built from the `severity` verdicts and the UPS
|
||||
`health.checks[]` results, not from new thresholds invented in the browser.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_alerts_builder.md` — `src/dashboards/alerts.py`: pure function from hub states + payloads to an alert list; `/api/summary` returns it.
|
||||
2. `02_rewrite_overview_page.md` — `index.html` + `page-index.js` reduced to fleet table + alerts panel.
|
||||
3. `03_retire_unused_overview_code.md` — delete the dead cards/events code and confirm the API contract other pages depend on is intact.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/test_alerts.py` — a source down, a source stale-but-serving, a UPS `fail`
|
||||
check, a threshold breach, an agent reporting `degraded`, an unconfigured source (must
|
||||
**not** alert — unconfigured is not broken, `AGENTS.md` invariant 3), a simulated UPS
|
||||
(warn, labelled), and the all-clear case producing `[]`.
|
||||
- API: `/api/summary` gains `alerts` and keeps every existing key.
|
||||
- Coverage: **>90%** on `alerts.py` and the changed part of `api.py`.
|
||||
- Browser: screenshot of `/` with a real problem present (stop the driveripper agent tunnel
|
||||
and re-shoot) — an overview that has never been seen in its alarm state has not been
|
||||
tested.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `/` shows exactly two panels: Fleet and Warnings/Errors
|
||||
- [ ] `curl -s localhost:8090/api/summary | python3 -c "import json,sys; print(json.load(sys.stdin)['alerts'])"` is `[]` when everything is healthy
|
||||
- [ ] Killing one host tunnel produces exactly one alert naming that host, with a link to `/system`
|
||||
- [ ] `uv run pytest` green (plus `-m live --live` unchanged), coverage ≥94%, `ruff` + `pyright` clean
|
||||
- [ ] no behaviour change on `/system`, `/llm`, `/ups`
|
||||
- [ ] no behavior change in completed phases
|
||||
@@ -0,0 +1,56 @@
|
||||
# Task 01 — Alerts builder
|
||||
|
||||
**Phase:** `02_overview_fleet_and_alerts` · **Source:** `TODO.md:3` — "The overview page should only show Fleet and warnings/errors."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Derive "what is wrong right now" on the server, so the overview is a list of verdicts and
|
||||
every other consumer (future mobile view, notification later) gets the same answer.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/alerts.py` (new) — `build(settings, hub_states, host_payloads, engine_payloads, ups_payload, recent_events) -> list[dict]`,
|
||||
each alert `{"id", "severity": "ok|warn|fail", "source", "message", "href", "at"}`:
|
||||
- **source down** — a configured job whose `Job.state.ok` is false → `fail`, message from
|
||||
the recorded error, `href` to its dashboard. Stale-but-serving (`age_seconds` beyond
|
||||
~3× its poll interval) is `warn`, not `fail`: the last good value is still on screen and
|
||||
saying "dead" would be a lie.
|
||||
- **host threshold breach** — from the `severity` sub-dicts added by phase 01: any
|
||||
`fail` → an alert naming host + metric + value; `warn` → `warn`.
|
||||
- **agent degraded** — `payload["degraded"]` non-empty (a `/proc` read failed on the box)
|
||||
→ `warn`, naming the sections.
|
||||
- **UPS** — any `health.checks[]` entry with `status: "fail"` → `fail`, `"warn"` → `warn`,
|
||||
carrying the check name and its detail (`"Battery age: 5.3 years — past the 4-year
|
||||
replacement window"` is exactly the kind of thing the landing page should say);
|
||||
`simulated: true` → one `warn` alert saying the numbers are modelled, not batteries.
|
||||
- **bad target config** — `settings.host_target_problems` → `warn` per problem.
|
||||
- **never** for an unset/unconfigured source. `AGENTS.md` invariant 3: a source nobody
|
||||
installed must not raise a red flag.
|
||||
- **dedupe and bound** — one alert per (source, metric); cap the list (say 12) with the
|
||||
count of suppressed ones so a 40-disk fleet cannot flood the page.
|
||||
- sort fail → warn, then by source name.
|
||||
2. `src/dashboards/api.py` — `/api/summary` calls it and adds `"alerts": [...]`. Keep
|
||||
`system`, `llm`, `ups`, `recent_events` exactly as they are (additive-only anchor),
|
||||
because `/api/summary` is also the cheapest whole-fleet probe anyone has.
|
||||
3. `tests/test_alerts.py` (new) — every bullet above, using the existing fixtures
|
||||
(`tests/fixtures/agent_*.json`) and the fakes in `tests/conftest.py`; the all-clear case
|
||||
asserts `[]`, and the unconfigured-source case asserts no alert is produced even though
|
||||
the state map has nothing for it.
|
||||
- ASSUMPTION: `recent_events` is passed in only so a *currently failing* source can show
|
||||
when it started failing; the event list itself is not re-rendered as alerts.
|
||||
- ASSUMPTION: staleness threshold is 3× the source's own poll interval rather than a
|
||||
single global number, because system polls at 10s and UPS at 5s.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit tests for all rules; no I/O in `alerts.py` (it is a pure projection, which is what
|
||||
makes it worth its own module).
|
||||
- Coverage: **>90%**.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_alerts.py -v` green
|
||||
- [ ] `curl -s localhost:8090/api/summary | python3 -m json.tool | head -40` shows `alerts` alongside the untouched `system`/`llm`/`ups`
|
||||
- [ ] with a healthy fleet the list is `[]`; with `NUT_SIMULATE=1` it contains exactly the simulated-data warning
|
||||
- [ ] `uv run pytest` green; `ruff` + `pyright` clean
|
||||
@@ -0,0 +1,59 @@
|
||||
# Task 02 — Rewrite the overview as fleet + alerts
|
||||
|
||||
**Phase:** `02_overview_fleet_and_alerts` · **Source:** `TODO.md:3` — "The overview page should only show Fleet and warnings/errors. Should be very simple."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The landing page becomes two things: the fleet table, and a list of what needs attention.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/pages/index.html` —
|
||||
- delete the three dashboard summary cards: the whole
|
||||
`<section class="grid auto" aria-label="Dashboards">` block, i.e. `#sys-hosts-live`,
|
||||
`#sys-cpu`, `#sys-mem`, `#sys-mem-unit`, `#sys-disk`, `#sys-gpus`, `#llm-live`,
|
||||
`#llm-decode`, `#llm-prompt`, `#llm-active`, `#llm-waiting`, `#llm-kv`, `#ups-list`
|
||||
and the `#dot-system` / `#dot-llm` / `#dot-ups` dots they own;
|
||||
- delete the events panel (`#events` and its card) — alerts replace it;
|
||||
- keep the masthead, `#nav-status` (the header source dots already live in `app.js`
|
||||
chrome), `#clock`, `#theme-toggle`, `#config-chip`, `#banners`, `#fleet-table`,
|
||||
`#fleet-empty`, `#foot-versions`;
|
||||
- add one panel **Warnings & errors** (`#alerts`, `aria-live="polite"`) with an empty
|
||||
state that says the fleet is clear — an empty list is a message, not a blank box.
|
||||
- keep `#fleet-table` columns as they are (host, CPU, load 1, mem, worst disk, GPU busy,
|
||||
temp, up). It is already the glanceable fleet view; simplifying it further loses the
|
||||
comparison that makes it worth having.
|
||||
2. `src/dashboards/web/static/page-index.js` —
|
||||
- delete `renderEvents()` and the `#sys-*` / `#llm-*` / `#ups-list` population code in
|
||||
`renderSummary()`;
|
||||
- add `renderAlerts(alerts)`: one row per alert with the severity class from phase 01
|
||||
(`.ok`/`.warn`/`.fail`), the source name as a link to its dashboard (`href` from the
|
||||
API), the message, and the age via `Dash.age()`; empty → the "all clear" state;
|
||||
- `refresh()` still calls `/api/summary` once and updates `#summary-line`
|
||||
("3 hosts · 2 engines · 2 UPS · 1 warning") — that one line is the only summary left;
|
||||
- keep `Dash.bindPause`, `Dash.rangeControl` (if the page used it) and the poll loop.
|
||||
3. Layout: alerts panel sits above the fleet table when something is wrong and below it when
|
||||
the list is empty, so a clean morning looks calm and a bad one puts the problem first.
|
||||
- ASSUMPTION: the UPS/LLM summary content the cards showed is *available* on their own
|
||||
pages, so removing it here loses nothing that is not one click away. The header status
|
||||
dots stay, because "which of the three is down" is the overview's job.
|
||||
- ASSUMPTION: `#banners` (config/setup hints, simulated-data notice) stays — it is a
|
||||
warning surface, which is exactly what the TODO keeps.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Update `tests/test_api.py` only if it asserts page structure; the page is not unit-tested
|
||||
(locked anchor: no JS tests).
|
||||
- Browser check with live data, both states:
|
||||
healthy (`alerts: []` → "all clear") and broken (stop one host agent:
|
||||
`ssh <host> "pkill -f '[h]ost_agent.py'"` or stop its tunnel) → one red row naming it.
|
||||
Screenshots to `data/logs/overview-clear.png` and `data/logs/overview-alert.png`.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `/` renders exactly two panels; `document.querySelectorAll('section').length` is 2 (check via the browser console or an `evaluate`)
|
||||
- [ ] `#events`, `#sys-cpu`, `#llm-kv`, `#ups-list` no longer exist in the served HTML (`curl -s localhost:8090/ | grep -c 'id="sys-cpu"'` → 0)
|
||||
- [ ] alert rows link to `/system`, `/llm`, `/ups` correctly
|
||||
- [ ] both screenshots exist in `data/logs/` and show what they claim
|
||||
- [ ] `uv run pytest` green; `ruff` + `pyright` clean
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
# Task 03 — Retire the dead overview code, keep the API honest
|
||||
|
||||
**Phase:** `02_overview_fleet_and_alerts` · **Source:** `TODO.md:3` — "Should be very simple." (simplicity includes not carrying code nothing renders)
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Leave no orphaned code behind, and prove the `/api/summary` fields the *other* consumers
|
||||
rely on were not quietly dropped along with the cards.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/static/page-index.js` — remove helpers that only the deleted cards
|
||||
used; run a grep for each symbol before deleting so nothing still referenced goes:
|
||||
`grep -n "sys-cpu\|llm-kv\|ups-list\|renderEvents" src/dashboards/web/static/*`.
|
||||
2. `src/dashboards/web/static/dash.css` — drop rules that only the deleted cards matched
|
||||
(check with `grep -n "metric\b" dash.css` against the remaining pages before removing;
|
||||
`.metric` is used by `/system` and `/llm` too, so most of it stays).
|
||||
3. `src/dashboards/api.py` — **do not remove** the `system`/`llm`/`ups` sections from
|
||||
`/api/summary` even though the page no longer renders them: it is the single cheapest
|
||||
whole-fleet probe, the live tests and future alerting want it, and the API anchor is
|
||||
additive-only. Add a short comment on `/api/summary` saying the page renders
|
||||
`alerts` + `system.fleet`/`hosts` and that the rest is for programmatic use.
|
||||
4. `tests/test_api.py` — add an explicit regression test: `/api/summary` contains the keys
|
||||
`system.enabled`, `system.fleet`, `system.hosts`, `llm.engines`, `ups.ups`,
|
||||
`recent_events`, `alerts`, so a later cleanup cannot silently shrink the contract.
|
||||
5. `README.md` — the overview row in the top table still says "one request, all of it";
|
||||
update the wording to "fleet + alerts in one request".
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Full suite green, including the new contract test.
|
||||
- Coverage: >90% on changed Python (`api.py` only).
|
||||
- `grep` evidence that no dangling selector survives (paste the empty grep result in the
|
||||
commit body or task notes).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -rn "sys-cpu\|llm-kv\|ups-list\|renderEvents" src/dashboards/web/` returns nothing
|
||||
- [ ] `curl -s localhost:8090/api/summary` still returns every key listed in step 4
|
||||
- [ ] `uv run pytest` green (contract test included), coverage ≥94%, `ruff` + `pyright` clean
|
||||
- [ ] `uv run pytest -m live --live` still 6 passed
|
||||
- [ ] no behavior change in completed phases (01's severity fields still present)
|
||||
Reference in New Issue
Block a user