chore(agent): phase roadmap from TODO.md, 5 phases

The five UI items in TODO.md become an executable pipeline. Planning files
only — no application code changed.

- 01_severity_language: one Python threshold table + `severity` on the
  values the pages colour, plus the .hero style. Today the thresholds are
  magic numbers at six JS call sites (75/90 here, 80/92 there), which is
  exactly how "same colour, four meanings" happens.
- 02_overview_fleet_and_alerts: /api/summary gains a derived `alerts`
  list; the page drops to fleet + warnings. Unconfigured sources must not
  alert — that is invariant 3, tested.
- 03_system_per_host_panels: per-host hero cards, no per-core, shared
  multi-host charts and the detail picker deleted. The six card metrics
  already exist in the payload, so the first task pins them with tests
  instead of inventing fields.
- 04_llm_chart_readability: charts.js already exports sparkline() and no
  page calls it — that is why the page reads flat. Port llm_monitor's
  drawSpark behaviour (fixed max, right growth, p95 foots) and keep the
  provenance tags.
- 05_ups_hero_battery_glyph: charge/load/runtime hero + an SVG battery
  whose fill is the charge, verified against both real UPSes and the
  low-charge/on-battery states only the simulator can produce.

PLAN.md records the ratified anchors and the deviation worth knowing:
severity vocabulary is ok|warn|fail, matching dash.css and nut.py, not the
"danger" wording in the proposal — one synonym for one colour.

TODO.md cleared to a bare heading; the items are now phases. .gitignore
stops ignoring all of .agents/ (the roadmap is project history) while
keeping phase-sessions/ and pipeline.log out. deploy/nut-sudoers rewritten:
its header still blamed a port mystery that is solved; the remaining need
is reading upsd.users so adding a scoped dashboards user does not delete
the accounts upsmon authenticates as.

LLM-Generated: true
This commit is contained in:
2026-10-16 20:10:23 -04:00
parent 1e454a8f81
commit 4b4962d7fb
29 changed files with 1429 additions and 46 deletions
@@ -0,0 +1,51 @@
# Phase 04 — LLM page is readable again
**Source:** `TODO.md` L6 — "LLM usage drifed from initial request to copy Experimental/llm_monitor. The charts got less useful. Need simplified and more readable."
**Story:** n/a
**Context:** `.agents/PLAN.md` → *Anchors* (hand-rolled SVG only), `~/Projects/Experimental/llm_monitor/static/index.html` (the reference implementation), `src/dashboards/web/static/page-llm.js` (`CELLS`, `makeCharts`), `src/dashboards/web/static/charts.js` (`sparkline`, `LineChart`), `src/dashboards/sources/llm.py`
## Objective
The LLM page drifted into a 14-cell grid where everything is the same size and nothing shows
recent shape. Bring back what made `llm_monitor` legible — big throughput numbers, a sparkline
in each metric cell, percentiles as foot lines — without losing what this project added since:
per-engine provenance tags, history windows, and severity colour.
`llm_monitor` is the design reference, not a dependency. It is a separate app on `:8095` and
stays that way; nothing here imports from it.
## Dependencies
- `01_severity_language` (todo) — KV-cache pressure and queue depth get coloured from the
API verdict, not a per-page threshold.
- `03_system_per_host_panels` (todo) — touches `charts.js` first; this phase builds on the
same shared helpers and must not fight it for them.
## Tasks
1. `01_chart_set_decision.md` — read both implementations side by side and write the decision note (which cells live, which get sparklines, what is demoted).
2. `02_sparkline_upgrade.md` — bring `DashCharts.sparkline()` up to `llm_monitor`'s `drawSpark` behaviour: right-aligned growth, fixed max, current/max labels, severity colour.
3. `03_llm_layout_simplified.md` — restructure `llm.html` + `page-llm.js`: hero throughput, compact sparkline cells, percentile foot lines, conditional alarm cells.
4. `04_side_by_side_verification.md` — verify against `llm_monitor` on `:8095` with the same live engines.
## Testing & Quality
- Python: none expected beyond the phase-01 severity fields; `/api/llm/*` stays as-is
(additive-only anchor).
- `uv run pytest -m live --live` must stay green — `test_engine_still_publishes_the_series_we_parse`
is the assertion that the numbers on screen are backed by resolved upstream series.
- Browser: both engines live (gufo on `deskwork:8731`, vLLM on `driveripper:8000`), plus the
single-engine case: with one `*_BASE_URL` unset the page must show one clean half rather
than a red panel.
- Coverage: ≥94% overall maintained.
## Completion Criteria
- [ ] decode and prompt throughput are the two largest numbers per engine, colour-coded
- [ ] every remaining metric cell carries a sparkline showing the selected window
- [ ] latency cells show mean **and** p95 without needing a hover
- [ ] a source-provenance tag is still visible on every throughput number (`AGENTS.md`: never invent a number)
- [ ] `/llm` with only one engine configured still renders cleanly
- [ ] screenshots of this page and `llm_monitor` side by side in `data/logs/`, plus `NOTES-chart-set.md`
- [ ] `uv run pytest` green, coverage ≥94%, `ruff` + `pyright` clean
- [ ] no behavior change in completed phases
@@ -0,0 +1,52 @@
# Task 01 — Chart-set decision note
**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "LLM usage drifed from initial request to copy Experimental/llm_monitor. The charts got less useful. Need simplified and more readable."
**Story:** n/a
## Objective
Decide, in writing, what "simplified and more readable" means for this page before touching
it — the drift happened precisely because nobody wrote down what the page was for.
## Work
1. Read the reference implementation:
`~/Projects/Experimental/llm_monitor/static/index.html` — specifically
`drawSpark(name, values, format, fixedMax)` (~L1092, with the right-aligned growth and the
top/middle/bottom scale marks ~L1134), `setThroughput(el, cell)` (~L1303),
`UI.historyLen = 60` (~L1003), the per-engine cell markup (`#gufo-ptps`, `#gufo-dtps`,
`#vllm-ptps`, `#vllm-dtps`), the percentile foot lines (`#vllm-ttft-p95`,
`#vllm-tpot-p95`, `#vllm-queue-p95`), and `buildSlotCard`/`renderSlots` (~L1179/L1209).
2. Read the current page: `src/dashboards/web/pages/llm.html` (engine-totals card L39–60,
four charts L62–81, `#engines` grid L83–91) and `page-llm.js` (`CELLS`, 14 entries;
`cellValue()`; `severityFor(value, 75, 90)` calls at L195/L307).
3. Write `.agents/phases/todo/04_llm_chart_readability/NOTES-chart-set.md`:
- **keep**: provenance tags per throughput cell, history window chips, per-engine panels,
the gufo slot/session list;
- **hero**: `decode_tokens_per_second`, `prompt_tokens_per_second` per engine;
- **sparkline cells** (compact, one per row): sessions active, queued, KV cache %,
TTFT (mean + p95 as foot), TPOT (mean + p95 as foot), queue mean, prefix-cache hit %,
spec-accept %;
- **conditional / demoted**: tokens-lifetime counter (quiet footnote), `preemptions` and
`device_lost` — shown as alarm chips **only when non-zero**, otherwise absent. A permanent
cell for something that is always 0 trains the eye to ignore the row.
- **charts**: keep decode/prompt/KV/queue `LineChart`s, one engine per series, with the
existing legend and a units-bearing y-axis; the four stay but drop to half-height so the
cells read first.
- the colour rules for KV pressure and queue depth (from `severity.py`), and the explicit
statement that throughput has *no* severity — 0 tok/s on an idle server is not red.
4. No code changes.
- ASSUMPTION: the cell set above is the judgment call; if the operator disagrees it is
cheaper to change this note now than the page later.
- ASSUMPTION: `llm_monitor` stays a separate app. Importing its files would break the
no-build, single-app structure and it is not ours to own.
## Testing & Quality
- The note is the deliverable; it must name real line numbers in both codebases so the next
task does not re-derive them.
## Completion Criteria
- [ ] `NOTES-chart-set.md` exists in this phase directory, with keep/hero/demote lists and the two source files cited by path and line
- [ ] no application file changed in this task (`git status` clean except the note)
@@ -0,0 +1,51 @@
# Task 02 — Sparkline upgrade
**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "The charts got less useful."
**Story:** n/a
## Objective
Make the shared `sparkline()` as informative as `llm_monitor`'s `drawSpark`, because every
page (LLM cells now, per-host cards later) draws on it.
`charts.js:352` already has `sparkline(container, points, opts)` and it is exported at
`charts.js:377` — **no page uses it today**, which is exactly why the LLM page reads flat.
## Work
1. `src/dashboards/web/static/charts.js` — extend `sparkline()` with these opts, defaulting
to today's behaviour so no existing caller breaks:
- `fixedMax` — plot against a known ceiling (percent metrics must not rescale to their own
data: 8 % KV cache next to a 92 % spike is the lie this fixes);
- `align: 'right'` — grow from the left, newest pinned right, as in `drawSpark`;
- `labels: {current, max}` — the small `now / max` foot `llm_monitor` prints, using the
`format` callback;
- `severity` — stroke colour from the theme token for `ok|warn|fail` (`cssVar('--ok')` etc.),
no colour and a faint grey stroke when severity is `null`;
- `minPoints` (default 2) — keep the existing "no history" text for a series that has no
shape yet; never draw a confident line through one point.
2. Keep it dependency-free SVG (`<svg>` + `<path>`/`<polyline>` + optional baseline), and keep
the output small enough to sit in a 160×34 box — that is the whole point of a sparkline.
3. Add scale marks (top/bottom values only) when `fixedMax` is set, so a reader can tell that
"the line is at a third" means a third *of what*.
4. If `renderCores` survived phase 03's cleanup, leave it alone — do not widen this task.
- ASSUMPTION: sparkline stays in `charts.js` rather than a new file; one shared chart
module is the current architecture.
- ASSUMPTION: no animation, but respect `prefers-reduced-motion` by not adding any
transition at all.
## Testing & Quality
- `tests/test_api.py` already asserts static assets are served; extend it to assert the new
option names are present in `/static/charts.js` (cheap, catches a bad merge).
- Browser: a page that uses it (any LLM cell after task 03) plus a synthetic check in devtools
— draw into an empty container (0, 1, 2, 60 points) and confirm each renders the right thing
rather than throwing.
- Coverage: ≥94% overall maintained; no new Python expected.
## Completion Criteria
- [ ] `curl -s localhost:8090/static/charts.js | grep -c "fixedMax\|align\|severity"` ≥ 3
- [ ] a sparkline with `fixedMax: 100` does not rescale when its data ranges 5–10 (verified in devtools, screenshot)
- [ ] 0- and 1-point series render "no history", not a spike
- [ ] `uv run pytest` green; `ruff` + `pyright` clean; `LineChart` untouched (`git diff` shows no changes to its code)
@@ -0,0 +1,62 @@
# Task 03 — Simplify the LLM page layout
**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "LLM usage drifed from initial request to copy Experimental/llm_monitor. The charts got less useful. Need simplified and more readable."
**Story:** n/a
## Objective
Per engine: two big throughput numbers you can read across the room, then a compact list of
metric cells that each carry their own sparkline and percentile foot line.
## Work
1. `src/dashboards/web/pages/llm.html` —
- keep the engine-totals card (L39–60) but shrink it to what the overview no longer shows:
engines answering, active sessions, queued. Drop the summed decode/prompt/KV numbers —
per-engine is the honest view and the overview owns the aggregate;
- reduce the four `LineChart` cards (L62–81) to half height, and drop `#chart-queue` if
`NOTES-chart-set.md` says queue reads better as a cell;
- per-engine panel (`#engines` grid, `page-llm.js:140+`): hero row (decode, prompt) using
the phase-01 `.hero` markup, then the cell list from `NOTES-chart-set.md`, then the gufo
slot list (unchanged behaviour).
2. `src/dashboards/web/static/page-llm.js` —
- `CELLS`: mark each entry with `hero: true` / `spark: true` / `alarm: true` and drive the
markup from those flags instead of 14 identical cells;
- render each `spark: true` cell through `DashCharts.sparkline()` with data from
`/api/llm/history` (same endpoint, one request per engine per window — batch the metrics
the endpoint supports rather than one XHR per cell);
- percentile foot lines: TTFT/TPOT/queue cells print `mean` big and `p95 …` beneath, as
`llm_monitor` does (`#vllm-ttft-p95` pattern);
- `alarm: true` cells (`preemptions`, `device_lost`) render only when non-zero, as a
`.chip.fail`; the row must not exist when they are 0;
- replace `severityFor(value, 75, 90)` (L195, L307) with the payload `severity` via
`Dash.severityFrom()`;
- **keep the provenance tag** on every throughput cell — the file's header comment explains
why (`78 tok/s` means different things when scraped, measured Δ/Δt, or estimated from
TPOT). Removing it to look cleaner is a regression, not a simplification.
3. `src/dashboards/web/static/dash.css` — cell-list styling: label left, value right-aligned
tabular, sparkline on the third column at a fixed width so rows line up; the alarm chip
reuses `.chip.fail`.
- ASSUMPTION: throughput gets no severity colour. A server doing 0 tok/s while idle is
healthy; a red zero every night destroys the signal.
- ASSUMPTION: cells are a vertical list per engine, not a 4×4 grid — grids make 14 numbers
look equally important, which is the complaint.
## Testing & Quality
- `uv run pytest -m live --live` against the real gufo and vLLM: `test_engine_still_publishes_the_series_we_parse`
guards the metric names the new layout renders.
- Browser, both engines live, and with one engine disabled (`VLLM_BASE_URL=` unset, restart)
to prove the page degrades to one clean half instead of a dead panel
(`AGENTS.md` invariant 3).
- Idle vs loaded engine if possible (`llm_monitor` on `:8095` is watching the same upstreams,
so compare like-for-like).
## Completion Criteria
- [ ] decode + prompt are the largest elements per engine, and the rest of the cells are visibly secondary
- [ ] every cell has a sparkline for the selected window; TTFT/TPOT show mean + p95
- [ ] `preemptions` / `device_lost` are absent at zero and appear as red chips when non-zero (verified in devtools with a mocked payload)
- [ ] provenance tags still present on throughput cells (`grep -n "source" page-llm.js` shows the tag path intact)
- [ ] `curl -s localhost:8090/llm` renders with one engine configured and shows no dead panel (screenshot)
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
@@ -0,0 +1,44 @@
# Task 04 — Verify side by side with llm_monitor
**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "LLM usage drifed from initial request to copy Experimental/llm_monitor."
**Story:** n/a
## Objective
Prove the new page is at least as readable as the app it drifted from, using the same live
upstreams.
## Work
1. Both apps watch the same engines, so compare honestly:
- `llm_monitor` on `http://127.0.0.1:8095` (it is already running in this fleet's dev
loop; if not, `cd ~/Projects/Experimental/llm_monitor && uv run uvicorn app:app --port 8095`);
- this app on `http://127.0.0.1:8090/llm`.
2. Screenshot both, same moment, same window length. Read the two images and answer in
writing: which one tells you *right now* whether the engines are busy, whether anyone is
queued, and whether the KV cache is about to bite? The answer must be "ours" or the phase
is not done.
3. Exercise the states, not just the happy path:
- one engine down (stop gufo or point `GUFO_BASE_URL` at a dead port, restart the dev
server) → one clean half, no red panel for the missing engine;
- both engines idle → throughput `0 tok/s` with **no** severity colour, and the sparklines
still show the shape of the last window rather than a flat lie;
- a real request in flight if the operator is willing (`curl` a chat completion against
gufo) so decode/prompt are visibly non-zero and the provenance tag is readable.
4. Confirm the history window chips (1h…90d) still drive the four `LineChart`s and that the
sparklines use the same window — a cell that silently ignores the selected range is worse
than no sparkline.
5. Append the comparison and the answers to `NOTES-chart-set.md`.
- ASSUMPTION: `llm_monitor` is treated as the design reference and left running; this phase
neither modifies nor deletes it.
## Testing & Quality
- Evidence task. Any defect found is fixed here, then re-screenshotted.
## Completion Criteria
- [ ] `data/logs/llm-before-monitor.png` (the reference) and `data/logs/llm-after.png` (ours) exist and were both read
- [ ] the single-engine and idle states are screenshotted too
- [ ] `NOTES-chart-set.md` has a short written verdict + what was fixed after looking
- [ ] `uv run pytest` and `uv run pytest -m live --live` green