phase: 04_llm_chart_readability
**Phase 04 final verification pass — all tasks already in `complete/`; verified + fixed 3 defects found in verification.**
- Verified `/llm` live (both engines, gufo-only instance on :8099, `llm_monitor` on :8095) in a real browser; dev server left running/healthy.
- Completed task-04's truncated layout fix: `#engines` now carries `.items-start` (the CSS rule existed but nothing used it) — gufo 544 px vs vLLM 715 px, no blank card; pinned by `test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling`.
- Fixed: event rows said the subject twice ("gufo gufo is answering again"), and `/ups` mangled mid-message names ("sent to apc: OK" → "sent to OK") → shared `Dash.eventText()` (strips `UpstreamError: ` + a *leading* name only, mirroring `alerts._clean_error`) used by `/system`, `/llm`, `/ups`; new test.
- Fixed: count sparklines autoscaled rollup averages (queue max 0.1379) into full-height spikes → `sparkline({minSpan})`, `minSpan: 1` on sessions/queued (spike 30 px → 4 px); assertions added.
- Tests: `bash .agents/validate.sh` → **EXIT=0** (`uv run pytest --cov --cov-fail-under=94`: **485 passed, 7 skipped**, coverage **96.87%**; `ruff check src tests agent`: All checks passed; `pyright`: 0 errors; `node --check` all static JS clean; repo hygiene clean; `uv pip list --outdated` empty). `uv run pytest -m live --live` → **7 passed**.
- Completion criteria: heroes largest per engine ✔ (38.4 px vs 16.8 px row values); sparkline per cell + selected window ✔ (5/5 gufo, 7/7 vLLM; 6 h chip moved KV max 53 %→58 %); mean+p95 without hover ✔ (243 / 48.8 / 285 ms feet); provenance tag ✔ (`Δ10.0s`/`Δ8.0s` + `title`=exact source); single-engine ✔ (1 panel, totals hidden, zero red); screenshots ✔ `data/logs/llm-verify-{both-engines,charts-and-slots,single-engine,monitor-reference}.png` + `NOTES-chart-set.md`; pytest/cov/ruff/pyright ✔; earlier phases unchanged-behaviour ✔ (all 01–03 tests pass).
- Deviation (documented, not code): throughput heroes stay **neutral ink** — `severity.py` has no throughput row and 0 tok/s idle is healthy; colour comes from rated cells (KV) + reachability. Kept per `NOTES-chart-set`.
- Deviation: could not append the task-04 written verdict to `NOTES-chart-set.md` (it lives in `todo/`, which I must not edit). Verdict: ours reads better than `:8095` — bigger throughput numbers, per-cell shape over the selected window, percentiles inline; `llm_monitor` only sparklines decode+KV and has no window control.
- Docs: AGENTS.md updated (485 test count, `sparkline` `minSpan` rule, `Dash.eventText` event-row rule, `/llm` panel `items-start`); README reviewed in full — commands, config table (all `config.py` vars present), architecture bullets, page descriptions verified accurate; no other inconsistencies found.
- Note for the harness: `.agents/phases/todo/03…/NOTES-verify.md` and `todo/04…/NOTES-chart-set.md` are untracked notes left behind by their phases (task files moved, notes did not).
- Next pending phase: `05_ups_hero_battery_glyph`.
This commit is contained in:
@@ -0,0 +1,51 @@
|
||||
# Phase 04 — LLM page is readable again
|
||||
|
||||
**Source:** `TODO.md` L6 — "LLM usage drifed from initial request to copy Experimental/llm_monitor. The charts got less useful. Need simplified and more readable."
|
||||
**Story:** n/a
|
||||
**Context:** `.agents/PLAN.md` → *Anchors* (hand-rolled SVG only), `~/Projects/Experimental/llm_monitor/static/index.html` (the reference implementation), `src/dashboards/web/static/page-llm.js` (`CELLS`, `makeCharts`), `src/dashboards/web/static/charts.js` (`sparkline`, `LineChart`), `src/dashboards/sources/llm.py`
|
||||
|
||||
## Objective
|
||||
|
||||
The LLM page drifted into a 14-cell grid where everything is the same size and nothing shows
|
||||
recent shape. Bring back what made `llm_monitor` legible — big throughput numbers, a sparkline
|
||||
in each metric cell, percentiles as foot lines — without losing what this project added since:
|
||||
per-engine provenance tags, history windows, and severity colour.
|
||||
|
||||
`llm_monitor` is the design reference, not a dependency. It is a separate app on `:8095` and
|
||||
stays that way; nothing here imports from it.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `01_severity_language` (todo) — KV-cache pressure and queue depth get coloured from the
|
||||
API verdict, not a per-page threshold.
|
||||
- `03_system_per_host_panels` (todo) — touches `charts.js` first; this phase builds on the
|
||||
same shared helpers and must not fight it for them.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_chart_set_decision.md` — read both implementations side by side and write the decision note (which cells live, which get sparklines, what is demoted).
|
||||
2. `02_sparkline_upgrade.md` — bring `DashCharts.sparkline()` up to `llm_monitor`'s `drawSpark` behaviour: right-aligned growth, fixed max, current/max labels, severity colour.
|
||||
3. `03_llm_layout_simplified.md` — restructure `llm.html` + `page-llm.js`: hero throughput, compact sparkline cells, percentile foot lines, conditional alarm cells.
|
||||
4. `04_side_by_side_verification.md` — verify against `llm_monitor` on `:8095` with the same live engines.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Python: none expected beyond the phase-01 severity fields; `/api/llm/*` stays as-is
|
||||
(additive-only anchor).
|
||||
- `uv run pytest -m live --live` must stay green — `test_engine_still_publishes_the_series_we_parse`
|
||||
is the assertion that the numbers on screen are backed by resolved upstream series.
|
||||
- Browser: both engines live (gufo on `deskwork:8731`, vLLM on `driveripper:8000`), plus the
|
||||
single-engine case: with one `*_BASE_URL` unset the page must show one clean half rather
|
||||
than a red panel.
|
||||
- Coverage: ≥94% overall maintained.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] decode and prompt throughput are the two largest numbers per engine, colour-coded
|
||||
- [ ] every remaining metric cell carries a sparkline showing the selected window
|
||||
- [ ] latency cells show mean **and** p95 without needing a hover
|
||||
- [ ] a source-provenance tag is still visible on every throughput number (`AGENTS.md`: never invent a number)
|
||||
- [ ] `/llm` with only one engine configured still renders cleanly
|
||||
- [ ] screenshots of this page and `llm_monitor` side by side in `data/logs/`, plus `NOTES-chart-set.md`
|
||||
- [ ] `uv run pytest` green, coverage ≥94%, `ruff` + `pyright` clean
|
||||
- [ ] no behavior change in completed phases
|
||||
@@ -0,0 +1,52 @@
|
||||
# Task 01 — Chart-set decision note
|
||||
|
||||
**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "LLM usage drifed from initial request to copy Experimental/llm_monitor. The charts got less useful. Need simplified and more readable."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Decide, in writing, what "simplified and more readable" means for this page before touching
|
||||
it — the drift happened precisely because nobody wrote down what the page was for.
|
||||
|
||||
## Work
|
||||
|
||||
1. Read the reference implementation:
|
||||
`~/Projects/Experimental/llm_monitor/static/index.html` — specifically
|
||||
`drawSpark(name, values, format, fixedMax)` (~L1092, with the right-aligned growth and the
|
||||
top/middle/bottom scale marks ~L1134), `setThroughput(el, cell)` (~L1303),
|
||||
`UI.historyLen = 60` (~L1003), the per-engine cell markup (`#gufo-ptps`, `#gufo-dtps`,
|
||||
`#vllm-ptps`, `#vllm-dtps`), the percentile foot lines (`#vllm-ttft-p95`,
|
||||
`#vllm-tpot-p95`, `#vllm-queue-p95`), and `buildSlotCard`/`renderSlots` (~L1179/L1209).
|
||||
2. Read the current page: `src/dashboards/web/pages/llm.html` (engine-totals card L39–60,
|
||||
four charts L62–81, `#engines` grid L83–91) and `page-llm.js` (`CELLS`, 14 entries;
|
||||
`cellValue()`; `severityFor(value, 75, 90)` calls at L195/L307).
|
||||
3. Write `.agents/phases/todo/04_llm_chart_readability/NOTES-chart-set.md`:
|
||||
- **keep**: provenance tags per throughput cell, history window chips, per-engine panels,
|
||||
the gufo slot/session list;
|
||||
- **hero**: `decode_tokens_per_second`, `prompt_tokens_per_second` per engine;
|
||||
- **sparkline cells** (compact, one per row): sessions active, queued, KV cache %,
|
||||
TTFT (mean + p95 as foot), TPOT (mean + p95 as foot), queue mean, prefix-cache hit %,
|
||||
spec-accept %;
|
||||
- **conditional / demoted**: tokens-lifetime counter (quiet footnote), `preemptions` and
|
||||
`device_lost` — shown as alarm chips **only when non-zero**, otherwise absent. A permanent
|
||||
cell for something that is always 0 trains the eye to ignore the row.
|
||||
- **charts**: keep decode/prompt/KV/queue `LineChart`s, one engine per series, with the
|
||||
existing legend and a units-bearing y-axis; the four stay but drop to half-height so the
|
||||
cells read first.
|
||||
- the colour rules for KV pressure and queue depth (from `severity.py`), and the explicit
|
||||
statement that throughput has *no* severity — 0 tok/s on an idle server is not red.
|
||||
4. No code changes.
|
||||
- ASSUMPTION: the cell set above is the judgment call; if the operator disagrees it is
|
||||
cheaper to change this note now than the page later.
|
||||
- ASSUMPTION: `llm_monitor` stays a separate app. Importing its files would break the
|
||||
no-build, single-app structure and it is not ours to own.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- The note is the deliverable; it must name real line numbers in both codebases so the next
|
||||
task does not re-derive them.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `NOTES-chart-set.md` exists in this phase directory, with keep/hero/demote lists and the two source files cited by path and line
|
||||
- [ ] no application file changed in this task (`git status` clean except the note)
|
||||
@@ -0,0 +1,51 @@
|
||||
# Task 02 — Sparkline upgrade
|
||||
|
||||
**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "The charts got less useful."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Make the shared `sparkline()` as informative as `llm_monitor`'s `drawSpark`, because every
|
||||
page (LLM cells now, per-host cards later) draws on it.
|
||||
|
||||
`charts.js:352` already has `sparkline(container, points, opts)` and it is exported at
|
||||
`charts.js:377` — **no page uses it today**, which is exactly why the LLM page reads flat.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/static/charts.js` — extend `sparkline()` with these opts, defaulting
|
||||
to today's behaviour so no existing caller breaks:
|
||||
- `fixedMax` — plot against a known ceiling (percent metrics must not rescale to their own
|
||||
data: 8 % KV cache next to a 92 % spike is the lie this fixes);
|
||||
- `align: 'right'` — grow from the left, newest pinned right, as in `drawSpark`;
|
||||
- `labels: {current, max}` — the small `now / max` foot `llm_monitor` prints, using the
|
||||
`format` callback;
|
||||
- `severity` — stroke colour from the theme token for `ok|warn|fail` (`cssVar('--ok')` etc.),
|
||||
no colour and a faint grey stroke when severity is `null`;
|
||||
- `minPoints` (default 2) — keep the existing "no history" text for a series that has no
|
||||
shape yet; never draw a confident line through one point.
|
||||
2. Keep it dependency-free SVG (`<svg>` + `<path>`/`<polyline>` + optional baseline), and keep
|
||||
the output small enough to sit in a 160×34 box — that is the whole point of a sparkline.
|
||||
3. Add scale marks (top/bottom values only) when `fixedMax` is set, so a reader can tell that
|
||||
"the line is at a third" means a third *of what*.
|
||||
4. If `renderCores` survived phase 03's cleanup, leave it alone — do not widen this task.
|
||||
- ASSUMPTION: sparkline stays in `charts.js` rather than a new file; one shared chart
|
||||
module is the current architecture.
|
||||
- ASSUMPTION: no animation, but respect `prefers-reduced-motion` by not adding any
|
||||
transition at all.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `tests/test_api.py` already asserts static assets are served; extend it to assert the new
|
||||
option names are present in `/static/charts.js` (cheap, catches a bad merge).
|
||||
- Browser: a page that uses it (any LLM cell after task 03) plus a synthetic check in devtools
|
||||
— draw into an empty container (0, 1, 2, 60 points) and confirm each renders the right thing
|
||||
rather than throwing.
|
||||
- Coverage: ≥94% overall maintained; no new Python expected.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `curl -s localhost:8090/static/charts.js | grep -c "fixedMax\|align\|severity"` ≥ 3
|
||||
- [ ] a sparkline with `fixedMax: 100` does not rescale when its data ranges 5–10 (verified in devtools, screenshot)
|
||||
- [ ] 0- and 1-point series render "no history", not a spike
|
||||
- [ ] `uv run pytest` green; `ruff` + `pyright` clean; `LineChart` untouched (`git diff` shows no changes to its code)
|
||||
@@ -0,0 +1,62 @@
|
||||
# Task 03 — Simplify the LLM page layout
|
||||
|
||||
**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "LLM usage drifed from initial request to copy Experimental/llm_monitor. The charts got less useful. Need simplified and more readable."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Per engine: two big throughput numbers you can read across the room, then a compact list of
|
||||
metric cells that each carry their own sparkline and percentile foot line.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/pages/llm.html` —
|
||||
- keep the engine-totals card (L39–60) but shrink it to what the overview no longer shows:
|
||||
engines answering, active sessions, queued. Drop the summed decode/prompt/KV numbers —
|
||||
per-engine is the honest view and the overview owns the aggregate;
|
||||
- reduce the four `LineChart` cards (L62–81) to half height, and drop `#chart-queue` if
|
||||
`NOTES-chart-set.md` says queue reads better as a cell;
|
||||
- per-engine panel (`#engines` grid, `page-llm.js:140+`): hero row (decode, prompt) using
|
||||
the phase-01 `.hero` markup, then the cell list from `NOTES-chart-set.md`, then the gufo
|
||||
slot list (unchanged behaviour).
|
||||
2. `src/dashboards/web/static/page-llm.js` —
|
||||
- `CELLS`: mark each entry with `hero: true` / `spark: true` / `alarm: true` and drive the
|
||||
markup from those flags instead of 14 identical cells;
|
||||
- render each `spark: true` cell through `DashCharts.sparkline()` with data from
|
||||
`/api/llm/history` (same endpoint, one request per engine per window — batch the metrics
|
||||
the endpoint supports rather than one XHR per cell);
|
||||
- percentile foot lines: TTFT/TPOT/queue cells print `mean` big and `p95 …` beneath, as
|
||||
`llm_monitor` does (`#vllm-ttft-p95` pattern);
|
||||
- `alarm: true` cells (`preemptions`, `device_lost`) render only when non-zero, as a
|
||||
`.chip.fail`; the row must not exist when they are 0;
|
||||
- replace `severityFor(value, 75, 90)` (L195, L307) with the payload `severity` via
|
||||
`Dash.severityFrom()`;
|
||||
- **keep the provenance tag** on every throughput cell — the file's header comment explains
|
||||
why (`78 tok/s` means different things when scraped, measured Δ/Δt, or estimated from
|
||||
TPOT). Removing it to look cleaner is a regression, not a simplification.
|
||||
3. `src/dashboards/web/static/dash.css` — cell-list styling: label left, value right-aligned
|
||||
tabular, sparkline on the third column at a fixed width so rows line up; the alarm chip
|
||||
reuses `.chip.fail`.
|
||||
- ASSUMPTION: throughput gets no severity colour. A server doing 0 tok/s while idle is
|
||||
healthy; a red zero every night destroys the signal.
|
||||
- ASSUMPTION: cells are a vertical list per engine, not a 4×4 grid — grids make 14 numbers
|
||||
look equally important, which is the complaint.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `uv run pytest -m live --live` against the real gufo and vLLM: `test_engine_still_publishes_the_series_we_parse`
|
||||
guards the metric names the new layout renders.
|
||||
- Browser, both engines live, and with one engine disabled (`VLLM_BASE_URL=` unset, restart)
|
||||
to prove the page degrades to one clean half instead of a dead panel
|
||||
(`AGENTS.md` invariant 3).
|
||||
- Idle vs loaded engine if possible (`llm_monitor` on `:8095` is watching the same upstreams,
|
||||
so compare like-for-like).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] decode + prompt are the largest elements per engine, and the rest of the cells are visibly secondary
|
||||
- [ ] every cell has a sparkline for the selected window; TTFT/TPOT show mean + p95
|
||||
- [ ] `preemptions` / `device_lost` are absent at zero and appear as red chips when non-zero (verified in devtools with a mocked payload)
|
||||
- [ ] provenance tags still present on throughput cells (`grep -n "source" page-llm.js` shows the tag path intact)
|
||||
- [ ] `curl -s localhost:8090/llm` renders with one engine configured and shows no dead panel (screenshot)
|
||||
- [ ] `uv run pytest` green, `ruff` + `pyright` clean
|
||||
@@ -0,0 +1,44 @@
|
||||
# Task 04 — Verify side by side with llm_monitor
|
||||
|
||||
**Phase:** `04_llm_chart_readability` · **Source:** `TODO.md:6` — "LLM usage drifed from initial request to copy Experimental/llm_monitor."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Prove the new page is at least as readable as the app it drifted from, using the same live
|
||||
upstreams.
|
||||
|
||||
## Work
|
||||
|
||||
1. Both apps watch the same engines, so compare honestly:
|
||||
- `llm_monitor` on `http://127.0.0.1:8095` (it is already running in this fleet's dev
|
||||
loop; if not, `cd ~/Projects/Experimental/llm_monitor && uv run uvicorn app:app --port 8095`);
|
||||
- this app on `http://127.0.0.1:8090/llm`.
|
||||
2. Screenshot both, same moment, same window length. Read the two images and answer in
|
||||
writing: which one tells you *right now* whether the engines are busy, whether anyone is
|
||||
queued, and whether the KV cache is about to bite? The answer must be "ours" or the phase
|
||||
is not done.
|
||||
3. Exercise the states, not just the happy path:
|
||||
- one engine down (stop gufo or point `GUFO_BASE_URL` at a dead port, restart the dev
|
||||
server) → one clean half, no red panel for the missing engine;
|
||||
- both engines idle → throughput `0 tok/s` with **no** severity colour, and the sparklines
|
||||
still show the shape of the last window rather than a flat lie;
|
||||
- a real request in flight if the operator is willing (`curl` a chat completion against
|
||||
gufo) so decode/prompt are visibly non-zero and the provenance tag is readable.
|
||||
4. Confirm the history window chips (1h…90d) still drive the four `LineChart`s and that the
|
||||
sparklines use the same window — a cell that silently ignores the selected range is worse
|
||||
than no sparkline.
|
||||
5. Append the comparison and the answers to `NOTES-chart-set.md`.
|
||||
- ASSUMPTION: `llm_monitor` is treated as the design reference and left running; this phase
|
||||
neither modifies nor deletes it.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Evidence task. Any defect found is fixed here, then re-screenshotted.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `data/logs/llm-before-monitor.png` (the reference) and `data/logs/llm-after.png` (ours) exist and were both read
|
||||
- [ ] the single-engine and idle states are screenshotted too
|
||||
- [ ] `NOTES-chart-set.md` has a short written verdict + what was fixed after looking
|
||||
- [ ] `uv run pytest` and `uv run pytest -m live --live` green
|
||||
Reference in New Issue
Block a user