phase: 04_llm_chart_readability

**Phase 04 final verification pass — all tasks already in `complete/`; verified + fixed 3 defects found in verification.**

- Verified `/llm` live (both engines, gufo-only instance on :8099, `llm_monitor` on :8095) in a real browser; dev server left running/healthy.
- Completed task-04's truncated layout fix: `#engines` now carries `.items-start` (the CSS rule existed but nothing used it) — gufo 544 px vs vLLM 715 px, no blank card; pinned by `test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling`.
- Fixed: event rows said the subject twice ("gufo gufo is answering again"), and `/ups` mangled mid-message names ("sent to apc: OK" → "sent to  OK") → shared `Dash.eventText()` (strips `UpstreamError: ` + a *leading* name only, mirroring `alerts._clean_error`) used by `/system`, `/llm`, `/ups`; new test.
- Fixed: count sparklines autoscaled rollup averages (queue max 0.1379) into full-height spikes → `sparkline({minSpan})`, `minSpan: 1` on sessions/queued (spike 30 px → 4 px); assertions added.
- Tests: `bash .agents/validate.sh` → **EXIT=0** (`uv run pytest --cov --cov-fail-under=94`: **485 passed, 7 skipped**, coverage **96.87%**; `ruff check src tests agent`: All checks passed; `pyright`: 0 errors; `node --check` all static JS clean; repo hygiene clean; `uv pip list --outdated` empty). `uv run pytest -m live --live` → **7 passed**.
- Completion criteria: heroes largest per engine ✔ (38.4 px vs 16.8 px row values); sparkline per cell + selected window ✔ (5/5 gufo, 7/7 vLLM; 6 h chip moved KV max 53 %→58 %); mean+p95 without hover ✔ (243 / 48.8 / 285 ms feet); provenance tag ✔ (`Δ10.0s`/`Δ8.0s` + `title`=exact source); single-engine ✔ (1 panel, totals hidden, zero red); screenshots ✔ `data/logs/llm-verify-{both-engines,charts-and-slots,single-engine,monitor-reference}.png` + `NOTES-chart-set.md`; pytest/cov/ruff/pyright ✔; earlier phases unchanged-behaviour ✔ (all 01–03 tests pass).
- Deviation (documented, not code): throughput heroes stay **neutral ink** — `severity.py` has no throughput row and 0 tok/s idle is healthy; colour comes from rated cells (KV) + reachability. Kept per `NOTES-chart-set`.
- Deviation: could not append the task-04 written verdict to `NOTES-chart-set.md` (it lives in `todo/`, which I must not edit). Verdict: ours reads better than `:8095` — bigger throughput numbers, per-cell shape over the selected window, percentiles inline; `llm_monitor` only sparklines decode+KV and has no window control.
- Docs: AGENTS.md updated (485 test count, `sparkline` `minSpan` rule, `Dash.eventText` event-row rule, `/llm` panel `items-start`); README reviewed in full — commands, config table (all `config.py` vars present), architecture bullets, page descriptions verified accurate; no other inconsistencies found.
- Note for the harness: `.agents/phases/todo/03…/NOTES-verify.md` and `todo/04…/NOTES-chart-set.md` are untracked notes left behind by their phases (task files moved, notes did not).
- Next pending phase: `05_ups_hero_battery_glyph`.
This commit is contained in:
2026-10-17 07:03:58 -04:00
parent 9757e3c947
commit f44a617ef9
42 changed files with 1428 additions and 194 deletions
@@ -0,0 +1,206 @@
# NOTES-chart-set — what `/llm` is for, and what it shows
**Phase 04, task 01 deliverable (no code changes).** Decides the cell set, the colour rules,
and the contracts tasks 02–03 implement. Both implementations were read side by side on
2026-10-17; every line number below was verified against the file it cites.
## The two sources
**Reference — `~/Projects/Experimental/llm_monitor/static/index.html`** (separate app on
`:8095`; design reference only, never a dependency — ASSUMPTION kept: it is not ours to own,
and importing it would break the no-build single-app structure):
| What | Where |
|---|---|
| sparkline CSS vocabulary (`.sparks`, `.spark`, `.spark-now`, `.spark-axis`, `.spark-line`, `.spark-area`) | L398–495 |
| `UI.historyLen = 60` (client ring buffer; `pushHist` L1542–1545; fed per engine L1570–1571, L1582–1583) | L1003 |
| `drawSpark(name, values, format, fixedMax)` | L1092 (`SPARK_W = 100`/`SPARK_H = 30` L1090–1091) |
| right-aligned growth — `const step = SPARK_W / (UI.historyLen - 1)`, `x = SPARK_W - (values.length - 1 - i) * step` | L1158–1163 |
| empty state: `< 2` values → `empty` class, blank paths (never a zero line) | L1112–1118 |
| flat-window honesty: pad keeps the line mid-plot, axis reports the one real value, not the pad (`const flat` L1128) | L1126–1133 |
| top/middle/bottom scale marks (`axis()` helper, `levels`, `fmtAxis` with enough decimals that the three labels stay distinguishable; axis writes L1155–1157) | L1101–1104, L1135, L1138–1154 |
| `fixedMax` ratio axes: KV sparks called with `fixedMax = 1` → 0–100% always | L1356 (gufo), L1420 (vLLM) |
| per-engine throughput `dd`s: `#gufo-ptps` / `#gufo-dtps`, `#vllm-ptps` / `#vllm-dtps` | L757 / L761, L802 / L806 |
| `setThroughput(el, cell)` — value + honest tag (`Δ<w>` for measured, `lifetime avg`, `from TPOT`, `derived`; `title` = exact `source`) | L1303–1320 |
| percentile foot lines: `#vllm-ttft-p95`, `#vllm-tpot-p95`, `#vllm-queue-p95` under their mean cards | L950, L955, L960 |
| token counters as cells whose foot line names the upstream series (`#vllm-ptok`, `#vllm-gtok`) | L964, L969 |
| slot cards: `SLOT_FIELDS` L1170–1177, `buildSlotCard` L1179 (built once, patched in place), `renderSlots` L1209 (DOM order == slot order, vanished slots removed) | — |
| the only thresholds in that app: KV bar warn ≥ 0.75, crit ≥ 0.9 | L1085–1086 |
| single-engine case: fleet block hidden when one engine — "a cross-engine sum of one engine just duplicates that engine's own card" | L1500–1504 |
| dead engine holds last good data + `stale` class + error banner, never zeroes | L1514–1535 |
**Current page** — `src/dashboards/web/pages/llm.html` + `src/dashboards/web/static/page-llm.js`:
| What | Where |
|---|---|
| engine-totals card (`#f-decode` L45, `#f-prompt` L47, KV bar L54–56, tokens lifetime L52–53) | `llm.html` L39–60 |
| four full-height `LineChart`s `#chart-decode/prompt/kv/queue` | `llm.html` L62–81 (chart divs L66, L71, L75, L79) |
| per-engine panels grid `#engines` + honest "No engines configured" empty state | `llm.html` L83–91 |
| gufo slots table `#slots-card` L94–102; events feed L103–106; provenance footer L109–113 | `llm.html` |
| `CELLS` — the 14-equal-size grid this phase dissolves | `page-llm.js` L16–33 |
| `cellValue()` — per-key payload mapping | L35–93 |
| `provenance()` — `Δ10.0s` / formula tag from `source`, nothing for scraped gauges | L96–105 |
| `enginePanel()` / `renderPanel()` | L112–136 / L138–214 |
| **JS thresholds to remove**: `severityFor(value.value, 75, 90)` on the KV bar | L194–195 |
| device_lost non-zero special case (the alarm-chip seed) | L196–200 |
| `makeCharts()` — `height: 165` for all four | L241–249 (165 at L242) |
| `refreshCharts()` — three `/api/llm/history` calls; the third already carries `requests_running,requests_waiting,kv_cache_usage` | L256–292 (L264–265) |
| **JS thresholds to remove**: `severityFor(kv, 75, 90)` on fleet KV | L307 |
| window chips `Dash.rangeControl` (affects charts today, sparklines after this phase) | L374 |
Shared helpers this builds on: `charts.js` `LineChart` L43, `renderLegend` L329 (with
`showPeak`), `sparkline` L348–373 (the function task 02 upgrades); `app.js` `severityFor`
L118 (documented FALLBACK ONLY — phases 04–05 exist to retire its remaining call sites),
`severityFrom` L139, `hero()` L187.
## Decision
### KEEP (unchanged, do not regress)
- **Provenance tags on every throughput number** — `provenance()` (`page-llm.js` L96–105)
and the footer legend (`llm.html` L109–113). AGENTS.md invariant 2; the `llm_monitor`
equivalent is `setThroughput` L1303–1320. Ours is strictly better (it prints the measured
Δ window); it ports as-is into the hero cells.
- **History window chips** (`page-llm.js` L374, `Dash.rangeControl`). Divergence from the
reference on purpose: `llm_monitor` sparks use a fixed 60-sample client buffer (L1003);
our sparklines read `/api/llm/history` with the *selected* window, so chips and sparks
always agree. Do not copy `historyLen`.
- **Per-engine panels** (`llm.html` L84) with per-source health, stale/last-good dimming,
and the upstream chip. Unconfigured engines render no panel at all (L85–90) — invariant 3.
- **gufo slot list** (`llm.html` L94–102, `renderSlots` L216–239). Adopt from
`llm_monitor` only the *in-place patch* idiom (`buildSlotCard` L1179) if rows churn; the
table itself stays.
- **Events card** (`llm.html` L103–106). Phase 02 deleted the raw feed from `/` only;
per-engine up/down transitions stay here.
### HERO — the two largest numbers, per engine panel
- `llm.decode_tokens_per_second` and `llm.prompt_tokens_per_second` (payload cells built in
`sources/llm.py`: gufo measured Δcounters, vLLM native gauge → measured → lifetime
estimate, each labelled with its `source`).
- Rendered with `Dash.hero()` (`app.js` L187) with the `Δ<window>` provenance in `.sub`.
- **Throughput has NO severity.** `severity.py` `THRESHOLDS` (L46–57) has no throughput row
and gets none: 0 tok/s on an idle server is not red, and a page-invented threshold is
exactly what invariant "a threshold exists nowhere else" forbids. The number renders in
neutral ink; colour in the hero row comes from the engine's reachability (status dot +
`.stale` dimming, already implemented in `renderPanel`), mirroring the `/system` NET slot
precedent ("throughput has no good/bad", AGENTS.md UI conventions). *Note: the phase
checklist says the heroes are "colour-coded" — the honest reading is the page's colour
language (neutral numbers, verdict colour on rated cells, reachability colour on the
panel). Task 04's side-by-side screenshots should show this; if the operator wants the
digits themselves tinted, that needs a ratified `severity.py` row first, not a JS guess.*
### SPARKLINE CELLS — compact rows, one per metric, each with its window's history
| Cell | Payload source (`cellValue`, L35–93) | History series for the spark | Colour |
|---|---|---|---|
| Sessions active | `sessions.active` (gufo `/slots`, vLLM count) | `llm.requests_running` ✔ stored | none |
| Queued | `requests_deferred` / `num_requests_waiting` | `llm.requests_waiting` ✔ stored | none |
| KV cache % | `kv_cache_usage_ratio` / `gpu_cache_usage` (×100) | `llm.kv_cache_usage` ✔ stored | **payload `severity`** |
| TTFT | `time_to_first_token_seconds__mean`, foot: `__p95` | `llm.ttft_seconds` ✔ stored | none |
| TPOT | `time_per_output_token_seconds__mean`, foot: `__p95` | `llm.tpot_seconds` — **new, see Python delta** | none |
| Queue mean | `request_queue_time_seconds__mean` | `llm.queue_seconds` ✔ stored | none |
| Prefix cache hit % | `prefix_cache_hit_ratio` / `prompt_cache_hit_ratio` | `llm.prefix_cache_hit_ratio` ✔ stored | fixed max 100 |
| Spec accept % | `spec_accept_rate` (gufo only) | `llm.spec_accept_rate` — **new, see Python delta** | fixed max 100 |
- Percentiles print as **foot lines, no hover** (reference: L950/L955/L960), from the live
snapshot cells — p95 needs no stored series.
- TTFT/TPOT/queue are vLLM-only; spec accept is gufo-only. The other engine's row renders
`n/a` uncoloured (rule 2), or the row is engine-scoped — task 03's choice, but a
gufo panel must never show a red/empty latency mystery.
- `/api/llm/history` (`api.py` L353–365) answers *any* metric name; unknown ones come back
as empty series — so the page can request the full cell list in one call. `refreshCharts`
(L256–292) already makes three calls; extend the third (L264–265) with the cell series
instead of adding requests, and feed both charts and sparks from the same responses.
### CONDITIONAL / DEMOTED
- **`tokens_total` (lifetime counter)** → quiet footnote line in each engine panel (the
existing `.sessions-note`/`.debug` strip, `page-llm.js` L203–213), like `llm_monitor`'s
counter cards whose foot line just names the series (L964, L969). A lifetime sum is not a
glance metric.
- **`preemptions` and `device_lost`** → **alarm chips rendered only when non-zero, absent
otherwise** (seed: the `device_lost` special case L196–200). A permanent cell for
something that is always 0 trains the eye to ignore the row. Chip class is `.chip.fail`
(existing vocabulary, `dash.css` L336); the rule is *existence of an incident*, not a
graded threshold — `sources/llm.py` already treats any non-zero `gufo_device_lost_total`
as an incident (comment at `GUFO_SERIES`), and zero/null simply draws no chip.
- **Engine-totals card (L39–60)**: keep engines-live, decode/prompt sums, active, queued —
demote tokens to footnote, and **remove the fleet KV bar** (L54–56 + `severityFor` at
L307): `fleet_totals` (`sources/llm.py` L325–368, `kv_cache_usage_max` L360–363) carries
no `severity`, and the rated per-engine KV cells plus the KV chart say it better. Hide the
totals card when fewer than two engines are configured — copy `llm_monitor` L1500–1504
("a cross-engine sum of one engine just duplicates that engine's own card"); this also
satisfies the single-engine completion criterion.
### CHARTS
- All four `LineChart`s stay: decode, prompt, KV, queue — one engine per series (queue chart:
running+waiting per engine), existing legend `renderLegend` (`charts.js` L329) with
`showPeak` on the bursty throughput plots, units-bearing y-format (already passed).
- **Drop `height: 165` → ~110** (`page-llm.js` L242) **and move the chart section below the
engine panels** so the cells read first (the task says half-height "so the cells read
first"; keeping full-height charts on top would contradict that).
- Task 03 must pass `ariaLabel` to all four charts (AGENTS.md: every chart names itself;
`/system` did this in phase 03, `/llm` never adopted it).
### COLOUR RULES (from `severity.py` and nowhere else)
- **KV pressure**: rated. `THRESHOLDS["llm.kv_cache_usage_pct"] = (75, 90)`
(`severity.py` L53), attached per engine cell by `SEVERITY_CELLS` (`sources/llm.py`
L99–103) with `scale=100` because both engines publish a 0–1 ratio. The page reads it via
`Dash.severityFrom` (`app.js` L139) and **both `severityFor(..., 75, 90)` call sites
(L194–195, L307) are deleted** — the `app.js` L112–117 fallback comment stays true only
if phase 04 does this.
- **Queue depth**: **not rated** — `THRESHOLDS` has no queue row (L46–57). `severityFrom`
returns `''` → no colour. The queue chart and sparkline carry the signal as *shape*. If
the operator wants a verdict, that is a future `severity.py` row (ratified numbers,
tests), never a JS threshold.
- **Throughput, sessions, latency (TTFT/TPOT/queue mean)**: not rated → colourless numbers.
Explicit statement per the task: **0 tok/s on an idle server is not red.**
- **Alarm chips**: `.chip.fail` when the counter is non-zero; absent otherwise (above).
### SPARKLINE CONTRACT (input for task 02, mirroring `drawSpark` L1092–1176)
`DashCharts.sparkline()` (currently L348–373: full-width autoscale, no labels) becomes:
1. **Right-aligned growth**: newest sample pinned right, x mapped over the *selected time
window* (`[now − range, now]`, like `LineChart._xExtent`) rather than over sample index —
a partly-filled window grows in from the right exactly like `drawSpark`'s fixed-step
buffer (L1158–1163) but stays honest about *when* samples happened.
2. **Fixed max** for ratio metrics (KV, prefix hit, spec accept: 0–100) so a squiggle means
the same thing on every screen — reference `fixedMax` calls L1356/L1420; autoscale
otherwise.
3. **Current/max labels**: current value in the cell header (reference `.spark-now`, L419)
and window max as the label; where the reference prints three axis marks (L1135,
L1155–1157) our compact rows print `now` + `max` — adopt its formatting discipline:
enough decimals that labels stay distinguishable (L1137–1152), a flat window reports the
one real value, never the pad artifact (L1126–1133).
4. **Severity colour**: stroke/area from the payload verdict (`ok`/`warn`/`fail` tokens from
`severityFrom`, unrated → `--series-1`), not `llm_monitor`'s fixed per-engine hue.
5. **Empty**: < 2 finite points → "no history", never a zero line (reference L1112–1118;
current code already does this).
## Python delta (the phase's only one, flagged for task 03)
`HISTORY_METRICS` (`sources/llm.py` L113–123; sampled at L292–297) does **not** store TPOT
or spec-accept, so those two cells would show permanent "no history" slots — the dead-cell
anti-pattern this note exists to prevent. Decision: extend `HISTORY_METRICS` additively —
- `tpot_seconds` ← `("time_per_output_token_seconds__mean",)` (vLLM only)
- `spec_accept_rate` ← `("spec_accept_rate",)` (gufo only)
New *names* in the public metric namespace (allowed: "phases may stop plotting a series,
they may not rename or stop storing it" — adding is the safe direction), consumed by the
generic `Sample("llm", engine, f"llm.{metric}", …)` loop (L292–297) and the store's generic
rollups; retention needs nothing new. The phase overview says "Python: none *expected*" —
this is a small, additive, tested exception; `/api/llm/*` fields keep their names/meanings
(additive-only anchor holds), and task 03 adds a unit test per new series plus an
`/api/llm/history` round-trip. Everything else in this phase is HTML/JS/CSS.
## Non-goals
- No changes to `/api/llm/snapshot` consumers outside this page, no `/ups`-style hero glyph
work (phase 05), no per-slot sparklines, no touching `llm_monitor` itself.
- The stored-metric set is never pruned by this phase: cells leave the *page*, series stay
in SQLite (PLAN.md, dropped-visuals anchor).
@@ -0,0 +1,13 @@
**Phase 04 final verification pass — all tasks already in `complete/`; verified + fixed 3 defects found in verification.**
- Verified `/llm` live (both engines, gufo-only instance on :8099, `llm_monitor` on :8095) in a real browser; dev server left running/healthy.
- Completed task-04's truncated layout fix: `#engines` now carries `.items-start` (the CSS rule existed but nothing used it) — gufo 544 px vs vLLM 715 px, no blank card; pinned by `test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling`.
- Fixed: event rows said the subject twice ("gufo gufo is answering again"), and `/ups` mangled mid-message names ("sent to apc: OK" → "sent to OK") → shared `Dash.eventText()` (strips `UpstreamError: ` + a *leading* name only, mirroring `alerts._clean_error`) used by `/system`, `/llm`, `/ups`; new test.
- Fixed: count sparklines autoscaled rollup averages (queue max 0.1379) into full-height spikes → `sparkline({minSpan})`, `minSpan: 1` on sessions/queued (spike 30 px → 4 px); assertions added.
- Tests: `bash .agents/validate.sh` → **EXIT=0** (`uv run pytest --cov --cov-fail-under=94`: **485 passed, 7 skipped**, coverage **96.87%**; `ruff check src tests agent`: All checks passed; `pyright`: 0 errors; `node --check` all static JS clean; repo hygiene clean; `uv pip list --outdated` empty). `uv run pytest -m live --live` → **7 passed**.
- Completion criteria: heroes largest per engine ✔ (38.4 px vs 16.8 px row values); sparkline per cell + selected window ✔ (5/5 gufo, 7/7 vLLM; 6 h chip moved KV max 53 %→58 %); mean+p95 without hover ✔ (243 / 48.8 / 285 ms feet); provenance tag ✔ (`Δ10.0s`/`Δ8.0s` + `title`=exact source); single-engine ✔ (1 panel, totals hidden, zero red); screenshots ✔ `data/logs/llm-verify-{both-engines,charts-and-slots,single-engine,monitor-reference}.png` + `NOTES-chart-set.md`; pytest/cov/ruff/pyright ✔; earlier phases unchanged-behaviour ✔ (all 01–03 tests pass).
- Deviation (documented, not code): throughput heroes stay **neutral ink** — `severity.py` has no throughput row and 0 tok/s idle is healthy; colour comes from rated cells (KV) + reachability. Kept per `NOTES-chart-set`.
- Deviation: could not append the task-04 written verdict to `NOTES-chart-set.md` (it lives in `todo/`, which I must not edit). Verdict: ours reads better than `:8095` — bigger throughput numbers, per-cell shape over the selected window, percentiles inline; `llm_monitor` only sparklines decode+KV and has no window control.
- Docs: AGENTS.md updated (485 test count, `sparkline` `minSpan` rule, `Dash.eventText` event-row rule, `/llm` panel `items-start`); README reviewed in full — commands, config table (all `config.py` vars present), architecture bullets, page descriptions verified accurate; no other inconsistencies found.
- Note for the harness: `.agents/phases/todo/03…/NOTES-verify.md` and `todo/04…/NOTES-chart-set.md` are untracked notes left behind by their phases (task files moved, notes did not).
- Next pending phase: `05_ups_hero_battery_glyph`.
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 14%]
........................................................................ [ 29%]
........................................................................ [ 43%]
......................................sssssss........................... [ 58%]
........................................................................ [ 73%]
........................................................................ [ 87%]
............................................................ [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
485 passed, 7 skipped, 1 warning in 44.38s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1 @@
Now I have everything verified. Writing the decision note:
@@ -0,0 +1 @@
error: assistant error: litellm.APIConnectionError: APIConnectionError: OpenAIException - Response payload is not completed: <TransferEncodingError: 400, message='Not…
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 30%]
........................................................................ [ 45%]
...........................sssssss...................................... [ 60%]
........................................................................ [ 75%]
........................................................................ [ 90%]
............................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
472 passed, 7 skipped, 1 warning in 43.29s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1,7 @@
**Task 01 complete — chart-set decision note written.**
- **Implemented:** `.agents/phases/todo/04_llm_chart_readability/NOTES-chart-set.md` — keep/hero/sparkline-cell/conditional-demote/charts decisions with every claim cited by path+line from both `llm_monitor/static/index.html` (drawSpark L1092, right-aligned growth L1158–1163, fixedMax L1356/L1420, setThroughput L1303, historyLen L1003, percentile feet L950/955/960, slot cards L1179/L1209, single-engine hide L1500–1504) and this repo (`page-llm.js` CELLS L16–33 + `severityFor` sites L194–195/L307, `charts.js` sparkline L348, `llm.py` HISTORY_METRICS L113/SEVERITY_CELLS L99, `severity.py` L46–57); colour rules = KV rated via payload severity (75/90), queue depth & throughput explicitly unrated → colourless; sparkline contract for task 02; flagged one additive Python delta (store `llm.tpot_seconds`, `llm.spec_accept_rate` so no cell is a permanent "no history" slot).
- **Tests/lint:** `uv run pytest` exit 0 (all passed, 7 live-skipped) · `--cov --cov-fail-under=94` → TOTAL 97% · `uv run ruff check src tests agent` clean · `uv run pyright` 0 errors · `node --check` all `web/static/*.js` clean.
- **Docs:** no doc impact — note-only task, no user-visible change yet; `git status` shows only the new note (plus pre-existing untracked harness files).
- **Notable decisions:** throughput heroes stay neutral-ink (phase checklist's "colour-coded" reconciled with the no-invented-threshold rule via the `/system` NET precedent; noted for task 04); fleet-totals KV bar removed and the card hidden below two engines; `HISTORY_METRICS` extension marked as the phase's single Python exception for task 03 to implement with tests.
- **Next pending task:** `02_sparkline_upgrade.md`.
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 30%]
........................................................................ [ 45%]
...........................sssssss...................................... [ 60%]
........................................................................ [ 75%]
........................................................................ [ 90%]
............................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
472 passed, 7 skipped, 1 warning in 43.32s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1,10 @@
All green. Task complete.
**Report**
- **Implemented** — `DashCharts.sparkline()` upgraded to the `drawSpark` contract (task 02) in `src/dashboards/web/static/charts.js`: new opts `fixedMax` (data clamps to a fixed `[0, ceiling]` axis + top/bottom scale marks, never axis-to-data rescale), `align: 'right'` + `span` (newest pinned right, growth leftwards over real time), `labels: {current, max}` foot via caller's `format`, `severity` (stroke from `--ok/--warn/--fail` tokens; null → faint `--text-faint` stroke, no fill), `minPoints` (default 2 → "no history"), `ariaLabel`; dependency-free SVG, no animation/transition; `LineChart` untouched (single diff hunk in the sparkline region).
- **Tests** — added `test_sparkline_ships_the_fixed_scale_and_verdict_options` to `tests/test_api.py` (served-bytes pin: option semantics, clamp expression, `x1 - o.span`, token colours, `minPoints: 2`, no `.sev-*`, no transitions).
- **Verification** — `uv run pytest`: **473 passed, 7 skipped**; `--cov-fail-under=94`: **96.87%**; `ruff` + `pyright` clean; `node --check` clean; `bash .agents/validate.sh` passes. Live suite `uv run pytest -m live --live`: **7 passed**. Browser synthetic check on live `/llm`: 10/10 PASS incl. `fixedMax: 100` keeps data 5–10 at y≈29/34 (autoscaled control hits y≈2), 0/1-point → "no history", align-right first x=160.8/190 — screenshot at `data/logs/sparkline-upgrade-task02.png`. Criterion grep on served file: 14 (≥3).
- **Docs** — AGENTS.md UI conventions gained the shared sparkline contract paragraph; no README impact (no setup/command/API change).
- **Decisions** — unrated spark defaults to faint grey (task-file rule) over NOTES-chart-set's `--series-1`; callers wanting orange pass `color`/`severity`. `align:'right'` anchors at the newest sample's ts (task's "newest pinned right"), with `span` setting the window. No new Python, no CSS/HTML changes.
- **Next pending task**: `03_llm_layout_simplified.md`.
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 30%]
........................................................................ [ 45%]
............................sssssss..................................... [ 60%]
........................................................................ [ 75%]
........................................................................ [ 90%]
................................................ [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
473 passed, 7 skipped, 1 warning in 43.62s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1,18 @@
All green. Final state verified; dev server stays up on http://127.0.0.1:8090 with the new page live.
## Report — Task 03 `llm_layout_simplified` (phase 04)
**Implemented**
- `web/pages/llm.html`: totals card shrunk to engines-answering/active/queued and hidden below 2 engines; summed decode/prompt/KV/tokens + seam note dropped; four charts moved *below* the engine panels at half height.
- `page-llm.js` rewritten: `CELLS` flags `hero/spark/alarm` drive markup — two `Dash.hero()` throughput heroes per engine (neutral ink, `Δ<window>` provenance in `.sub`, exact `source` in `title`); compact `.cells` rows with right-aligned tabular values, per-cell `sparkline()` (align:'right'+span, fixedMax 100 on ratio cells, `max` foot, KV coloured from payload `severity` via `severityFrom`); TTFT/TPOT/queue-time show mean + `p95 …` foot line; `preemptions`/`device_lost` render only as `.chip.fail` when non-zero (absent at zero); engine-scoped rows (`only:`) so gufo has no latency mystery; tokens lifetime demoted to footnote; **one** batched `/api/llm/history` request feeds charts + all sparks; both `severityFor(…,75,90)` call sites deleted.
- `sources/llm.py`: additive `HISTORY_METRICS` `tpot_seconds` (vLLM) + `spec_accept_rate` (gufo) per NOTES-chart-set.
- `dash.css`: `.cells/.cell` (label | tabular value | 160px spark column, p95 foot, KV verdict classes, alarm chip row, narrow-screen fallback).
- Fixed two defects found only against live data: hero `cellValue` key mapping and `drawChart` legend-sibling null crash.
**Tests/lint** — `bash .agents/validate.sh` exit 0: **483 passed, 7 skipped**, coverage 96.87 % (≥94), ruff + pyright clean, `node --check` clean. New pins: LLM payload-colour, hero row, spark/p95 contract, one-request + `HISTORY_METRICS` round-trip, alarm-chip rule, totals-card shape, chart height/aria; `test_severity_for_is_kept_as_the_documented_fallback` narrowed to `/ups`. Live: `uv run pytest -m live --live` → 7 passed. Browser (both engines live): heroes/sparks/p95 verified; mocked payload in devtools → red chips at non-zero, absent at zero; single-engine instance (`VLLM_BASE_URL=` unset, :8091, throwaway DB) → one clean panel, totals hidden, no dead panel. Screenshots: `data/logs/llm-layout-both-engines.png`, `llm-alarm-chips-mock.png`, `llm-single-engine.png`. One tiny vLLM chat request (16 tok) was sent to make latency rows non-empty — read-only class of traffic, same kind llm_monitor watches.
**Docs** — AGENTS.md (`/llm` layout convention + `severityFor` now only `/ups` remains) and README (`web/` section `/llm` description) updated.
**Deviations** — Totals card also hides below 2 engines (llm_monitor rule cited by NOTES); heroes neutral per the phase-01 colour language (documented in NOTES + AGENTS.md).
**Next pending task:** `04_side_by_side_verification.md` (same phase).
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 14%]
........................................................................ [ 29%]
........................................................................ [ 44%]
....................................sssssss............................. [ 58%]
........................................................................ [ 73%]
........................................................................ [ 88%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
483 passed, 7 skipped, 1 warning in 47.36s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
@@ -0,0 +1 @@
Now let me fix the layout void I measured (the shorter engine card stretched to the taller one, leaving 172 px of empty card):
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 14%]
........................................................................ [ 29%]
........................................................................ [ 44%]
....................................sssssss............................. [ 58%]
........................................................................ [ 73%]
........................................................................ [ 88%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
483 passed, 7 skipped, 1 warning in 45.25s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
+52 -5
View File
@@ -167,9 +167,10 @@ Responsibilities, one line each:
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number `.hero` big number (plus an optional `.bar`) from that class — it never compares a number
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
values the API does not rate; converting its remaining call sites is phases 04–05's job values the API does not rate; converting its remaining call sites is phase 05's job
(`/` was converted in phase 02, `/system` in phase 03 — those pages colour only from the (`/` was converted in phase 02, `/system` in phase 03 and `/llm` in phase 04 — those pages
`severity` verdicts the API sends, and no `severityFor(` call survives in their scripts). colour only from the `severity` verdicts the API sends, and no `severityFor(` call survives
in their scripts).
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`, Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device `gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
@@ -215,7 +216,7 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
./scripts/dev-server.sh status # per-source ok/DOWN ./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh logs ./scripts/dev-server.sh logs
uv run pytest # hermetic: 472 passed + 7 live-skipped, no network uv run pytest # hermetic: 485 passed + 7 live-skipped, no network
uv run pytest --cov # coverage gate (96.9%; floor 94%) uv run pytest --cov # coverage gate (96.9%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright uv run ruff check src tests agent && uv run pyright
@@ -302,6 +303,30 @@ muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare. rule reaches for a token that `:root` does not declare.
A cell-scale trend is `DashCharts.sparkline()` — the `llm_monitor` `drawSpark` idiom, so a
percent metric is plotted against `fixedMax` (**never** autoscaled: an 8 % KV window that once
held a 92 % spike must not read flat, and the axis gets top/bottom scale marks so "a third of
the way up" says a third of what). `align: 'right'` pins the newest sample right and grows the
line leftwards inside a real `span` in seconds — x stays time, not sample index. Stroke colour
comes from the payload `severity` via the `--ok`/`--warn`/`--fail` tokens only; an unrated
series draws a faint `--text-faint` stroke with no fill, because a shape is not a verdict.
Below `minPoints` (2) it prints `no history` rather than drawing a confident line through one
point, `labels: {current, max}` renders the `now / max` foot through the caller's `format`, and
there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the
other half of the same honesty for count metrics: history arrives as rollups, so eight seconds
of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for
a queue nobody was in — `/llm` passes `minSpan: 1` on sessions and queued so its axis is never
finer than one request.
An event row names its subject **once**. Every feed (`/system`, `/llm`, `/ups`) renders the
message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a
*leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py`
messages start with their own label, and each row already prints that label as a chip, so
"gufo gufo is answering again" reads as two problems. A name in the middle of a message stays:
the UPS rows used to `replace()` it anywhere and mangled "Quick battery test sent to apc: OK".
This is presentation only — the stored event and `/api/events` keep every word, and the raw
error string still surfaces in `/api/health`.
The `/system` host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the The `/system` host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the
`.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script). `.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script).
Colour comes from the payload's `severity` dicts and nowhere else; a missing number or Colour comes from the payload's `severity` dicts and nowhere else; a missing number or
@@ -316,7 +341,29 @@ simpler surface, not a less complete page.
A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
those (a range when the hosts disagree) instead of the poll interval, which only coincides with those (a range when the hosts disagree) instead of the poll interval, which only coincides with
them at the defaults. them at the defaults. `/llm`'s hero `.sub` prints the rate cell's own `window` through the same
`provenance()` tag, and the hero's `title` carries the exact `source` string — "78 tok/s" must
always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_monitor`'s
`setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification).
**The `/llm` page reads per engine** (phase 04, `NOTES-chart-set.md`): a row of two
`Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**, because 0 tok/s
on an idle server is healthy and `severity.py` has no throughput row), then a compact `.cells`
list — label left, tabular value right, a `sparkline()` fixed at 160px so rows line up, and a
mono `p95 …` foot under every latency mean (no hover; p95 needs no stored series). Rows are
engine-scoped by the `only` flag: gufo never grows a TTFT mystery, vLLM never a spec-accept
hole. `preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded
threshold); lifetime token counters are a quiet footnote line. **One `/api/llm/history` request
per refresh** carries every chart *and* spark series (`HISTORY_SERIES`; the endpoint answers
unknown names as empty series on purpose), and the two series that addition required
(`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) are stored additively in
`sources/llm.py` HISTORY_METRICS. The engine-totals card kept only engines-live/active/queued
and **hides below two engines** — a cross-engine sum of one engine duplicates that engine's own
panel, and the summed-KV bar went because `fleet_totals` carries no `severity` to render. The
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
sit at their own height (`#engines` carries `.items-start`): a 5-row gufo panel next to a 7-row
vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data".
**Charts belong to the host they chart.** `/system` has two sections — the host panels and the **Charts belong to the host they chart.** `/system` has two sections — the host panels and the
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
+10 -1
View File
@@ -174,7 +174,16 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
load, clock, disk I/O, uptime, GPU power/temp, filesystems, processes — folds into a collapsed load, clock, disk I/O, uptime, GPU power/temp, filesystems, processes — folds into a collapsed
"More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request "More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request
(`hosts=<that host>&metrics=…&range=…`), and a host that stops reporting takes its panel and (`hosts=<that host>&metrics=…&range=…`), and a host that stops reporting takes its panel and
its charts off the page. its charts off the page. The LLM page (`/llm`) is per engine too: two big
throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is
healthy) with their provenance tag (`Δ10.0s`, scraped, estimated), then a compact list of
metric rows — sessions, queue, KV pressure (the only rated cell), prefix/spec percentages,
and TTFT/TPOT/queue-time means with a `p95 …` foot line — each carrying its own sparkline
over the selected window. Preemptions and GPU-context losses render only as red alarm chips
while non-zero; the token counter is a footnote. The four trend charts sit below the panels
at half height, and one `/api/llm/history` request per refresh feeds all of it. With a
single engine configured the totals card hides and the page is one clean half — never a red
panel for a server you do not run.
## Testing ## Testing
+2
View File
@@ -118,8 +118,10 @@ HISTORY_METRICS: dict[str, tuple[str, ...]] = {
"requests_waiting": ("requests_deferred", "num_requests_waiting"), "requests_waiting": ("requests_deferred", "num_requests_waiting"),
"ttft_seconds": ("time_to_first_token_seconds__mean",), "ttft_seconds": ("time_to_first_token_seconds__mean",),
"ttft_p95_seconds": ("time_to_first_token_seconds__p95",), "ttft_p95_seconds": ("time_to_first_token_seconds__p95",),
"tpot_seconds": ("time_per_output_token_seconds__mean",), # vLLM only
"queue_seconds": ("request_queue_time_seconds__mean",), "queue_seconds": ("request_queue_time_seconds__mean",),
"prefix_cache_hit_ratio": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"), "prefix_cache_hit_ratio": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"),
"spec_accept_rate": ("spec_accept_rate",), # gufo only
} }
+33 -35
View File
@@ -36,52 +36,29 @@
<div id="banners" class="stack" style="margin-bottom:.85rem"></div> <div id="banners" class="stack" style="margin-bottom:.85rem"></div>
<section class="card flat" aria-label="Engine totals"> <!-- Engine totals, hidden until a second engine exists: a cross-engine sum of one
engine only duplicates that engine's own panel. What stays here is what no panel
says — how many engines are answering, and the session/queue counts across them.
Summed decode/prompt/KV left with phase 04: per-engine is the honest view, and
averaging a Strix Halo iGPU with a Threadripper is not one number. -->
<section class="card flat" id="totals" aria-label="Engine totals" hidden>
<div class="body"> <div class="body">
<div class="metrics"> <div class="metrics">
<div class="metric"><div class="label">Engines answering</div> <div class="metric"><div class="label">Engines answering</div>
<div class="value"><span id="f-live">n/a</span><span class="unit" id="f-total"></span></div></div> <div class="value"><span id="f-live">n/a</span><span class="unit" id="f-total"></span></div></div>
<div class="metric"><div class="label">Decode (sum)</div>
<div class="value"><span id="f-decode">n/a</span><span class="unit">tok/s</span></div></div>
<div class="metric"><div class="label">Prompt (sum)</div>
<div class="value"><span id="f-prompt">n/a</span><span class="unit">tok/s</span></div></div>
<div class="metric"><div class="label">Active sessions</div> <div class="metric"><div class="label">Active sessions</div>
<div class="value" id="f-active">n/a</div></div> <div class="value" id="f-active">n/a</div></div>
<div class="metric"><div class="label">Queued</div> <div class="metric"><div class="label">Queued</div>
<div class="value" id="f-waiting">n/a</div></div> <div class="value" id="f-waiting">n/a</div></div>
<div class="metric"><div class="label">Tokens lifetime</div>
<div class="value"><span id="f-tokens">n/a</span><span class="unit"></span></div></div>
<div class="metric"><div class="label" title="highest KV-cache use across engines right now, not a window peak">KV cache now</div>
<div class="value"><span id="f-kv">n/a</span><span class="unit">%</span></div>
<div class="bar" style="margin-top:.3rem"><i></i></div></div>
</div> </div>
<div class="faint" id="seam-note" style="margin-top:.5rem;font-size:11.5px"></div>
</div> </div>
</section> </section>
<section class="grid cols-2" style="margin-top:.85rem" aria-label="Throughput charts"> <!-- The page leads with the engine panels: two throughput heroes per engine, a compact
<article class="card"> sparkline cell list, and the gufo slot table below. The four trend charts follow,
<header><h3>Decode throughput</h3><span class="spacer"></span> half-height — supporting evidence, not the main event. -->
<span class="faint">tok/s · every engine, summed slots</span></header>
<div class="body"><div class="chart" id="chart-decode"></div><div class="legend" id="legend-decode"></div></div>
</article>
<article class="card">
<header><h3>Prompt / prefill throughput</h3><span class="spacer"></span>
<span class="faint">tok/s · newly computed tokens</span></header>
<div class="body"><div class="chart" id="chart-prompt"></div><div class="legend" id="legend-prompt"></div></div>
</article>
<article class="card">
<header><h3>KV / prefix cache pressure</h3><span class="spacer"></span><span class="faint">% of cache in use</span></header>
<div class="body"><div class="chart" id="chart-kv"></div><div class="legend" id="legend-kv"></div></div>
</article>
<article class="card">
<header><h3>Sessions &amp; queue</h3><span class="spacer"></span><span class="faint">running vs waiting</span></header>
<div class="body"><div class="chart" id="chart-queue"></div><div class="legend" id="legend-queue"></div></div>
</article>
</section>
<section style="margin-top:.85rem"> <section style="margin-top:.85rem">
<div class="grid cols-2" id="engines"></div> <div class="grid cols-2 items-start" id="engines"></div>
<div class="empty-state" id="engines-empty" hidden> <div class="empty-state" id="engines-empty" hidden>
<h3>No engines configured</h3> <h3>No engines configured</h3>
<p>Set <code>GUFO_BASE_URL</code> (llama.cpp / gufo) and/or <code>VLLM_BASE_URL</code>. <p>Set <code>GUFO_BASE_URL</code> (llama.cpp / gufo) and/or <code>VLLM_BASE_URL</code>.
@@ -90,6 +67,27 @@
</div> </div>
</section> </section>
<section class="grid cols-2" style="margin-top:.85rem" aria-label="Trend charts">
<article class="card">
<header><h3>Decode throughput</h3><span class="spacer"></span>
<span class="faint">tok/s</span></header>
<div class="body"><div class="chart" id="chart-decode"></div><div class="legend" id="legend-decode"></div></div>
</article>
<article class="card">
<header><h3>Prompt / prefill throughput</h3><span class="spacer"></span>
<span class="faint">tok/s</span></header>
<div class="body"><div class="chart" id="chart-prompt"></div><div class="legend" id="legend-prompt"></div></div>
</article>
<article class="card">
<header><h3>KV / prefix cache pressure</h3><span class="spacer"></span><span class="faint">% in use</span></header>
<div class="body"><div class="chart" id="chart-kv"></div><div class="legend" id="legend-kv"></div></div>
</article>
<article class="card">
<header><h3>Sessions &amp; queue</h3><span class="spacer"></span><span class="faint">running vs waiting</span></header>
<div class="body"><div class="chart" id="chart-queue"></div><div class="legend" id="legend-queue"></div></div>
</article>
</section>
<section class="grid cols-2" style="margin-top:.85rem"> <section class="grid cols-2" style="margin-top:.85rem">
<article class="card" id="slots-card"> <article class="card" id="slots-card">
<header><h3>gufo slots</h3><span class="spacer"></span> <header><h3>gufo slots</h3><span class="spacer"></span>
@@ -107,8 +105,8 @@
</section> </section>
<footer class="pagefoot"> <footer class="pagefoot">
<span>each cell carries its provenance: scraped gauge, <span class="mono">Δ10s</span> measured <span>each throughput number carries its provenance: scraped gauge, <span class="mono">Δ10s</span>
counter, or lifetime estimate</span> measured counter, or lifetime estimate</span>
<span class="spacer" style="margin-inline-start:auto"></span> <span class="spacer" style="margin-inline-start:auto"></span>
<span class="mono" id="foot-note"></span> <span class="mono" id="foot-note"></span>
</footer> </footer>
+27 -4
View File
@@ -70,6 +70,28 @@ const Dash = (() => {
const naText = 'n/a'; const naText = 'n/a';
/**
* The text of an event-feed row: the subject said **once**.
*
* `poller.py` writes messages that name their own subject ("gufo is answering again
* after 1 failure"), and every feed prints the host as its own chip beside the message,
* so printing the stored words verbatim reads as two problems — the same reasoning
* `alerts._clean_error()` applies to the overview's alert list, including the drop of
* `UpstreamError: ` (this app has one upstream error type, so its name is never a fact).
* A leading `host:` goes too, but only at the front: the UPS rows used to `replace()`
* the name anywhere, which mangled "Quick battery test sent to apc: OK". Presentation
* only — the stored event and `/api/events` keep every word.
*/
function eventText(event) {
const text = String((event && event.message) || '');
const host = String((event && event.host) || '');
const out = text.replace(/^UpstreamError:\s+/, '');
if (!host) return out;
const esc = host.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
const stripped = out.replace(new RegExp('^' + esc + '\\s*[:\\-]?\\s*'), '');
return stripped.trim() ? stripped : out;
}
// ---- DOM helpers -------------------------------------------------------- // ---- DOM helpers --------------------------------------------------------
/** Write text only when it changed: avoids killing transitions and focus. */ /** Write text only when it changed: avoids killing transitions and focus. */
function setTxt(el, value) { function setTxt(el, value) {
@@ -110,9 +132,9 @@ const Dash = (() => {
* *
* Thresholds live in `src/dashboards/severity.py` and travel in the payload as * Thresholds live in `src/dashboards/severity.py` and travel in the payload as
* `severity`; `severityFrom()` below is how a page reads them. This function survives * `severity`; `severityFrom()` below is how a page reads them. This function survives
* only because `page-llm.js` and `page-ups.js` still call it with their own numbers * only because `page-ups.js` still calls it with its own numbers (phase 05 converts it,
* (phases 04–05 convert them page by page, so each page's change is reviewable on its * so each page's change is reviewable on its own — `/` was converted in phase 02,
* own — `/` was converted in phase 02, `/system` in phase 03). Do not add a new call * `/system` in phase 03, `/llm` in phase 04). Do not add a new call
* site: a threshold written in JS is a threshold that can disagree with the other pages. * site: a threshold written in JS is a threshold that can disagree with the other pages.
*/ */
function severityFor(value, warnAt, failAt) { function severityFor(value, warnAt, failAt) {
@@ -444,7 +466,8 @@ const Dash = (() => {
return { $, $$, fetchJSON, poll, fmtBytes, bytes, rate, pct, num, duration, age, setTxt, return { $, $$, fetchJSON, poll, fmtBytes, bytes, rate, pct, num, duration, age, setTxt,
setValue, setBar, setDot, chip, initChrome, rangeControl, toast, bindPause, setValue, setBar, setDot, chip, initChrome, rangeControl, toast, bindPause,
staleness, markStale, severityFor, severityFrom, hero, ageSeconds, naText }; staleness, markStale, severityFor, severityFrom, hero, ageSeconds, naText,
eventText };
})(); })();
document.addEventListener('DOMContentLoaded', () => Dash.initChrome()); document.addEventListener('DOMContentLoaded', () => Dash.initChrome());
+116 -15
View File
@@ -344,30 +344,131 @@ function renderLegend(container, series, format, showPeak) {
}).join(''); }).join('');
} }
/** Small non-interactive sparkline for cards where a full chart would shout. */ /**
* Small non-interactive sparkline for cards where a full chart would shout.
*
* Same honesty rules as `LineChart`, at 160×34: fewer than `minPoints` points say
* "no history" (never a flat zero line, never a confident line through one point), and
* labels quote the data, not the padded axis — a flat window reports the one real value.
*
* opts (all optional; omitting them keeps the autoscaled, full-width line this function
* has always drawn):
* - height: plot height in px (default 34; the box is the point of a sparkline)
* - color: explicit stroke/fill colour; wins over `severity`
* - severity: payload verdict ('ok'|'warn'|'fail', or a `{severity}` cell) → stroke from
* that theme token (`cssVar('--ok')` etc.). Anything else — null, '', an
* unknown word — draws **no colour**: a faint grey stroke and no fill,
* because an unrated series is a shape, not a verdict (an unknown number
* must never look green)
* - min / max: explicit y bounds (today's behaviour)
* - minSpan: smallest y span an autoscaled axis may have, in the series' own units.
* For counts (sessions, queued) pass 1: a 1-minute rollup averages 8 seconds
* of one waiting request down to 0.14, and an axis that shrinks to fit turns
* that into a full-height spike — the twin of the rescale fixedMax fixes. An
* all-zero window still pads to the mid-line, so "nothing queued" reads flat
* - fixedMax: known ceiling for percent-like metrics — plot against [min ?? 0, fixedMax]
* instead of rescaling to the data, so an 8 % KV window that once held a
* 92 % spike reads the same as one that did not. Adds top/bottom scale marks
* so "the line is at a third" says a third *of what*; values clamp to the
* axis rather than drawing off-box
* - align: 'right' pins the newest sample to the right edge and grows the line
* leftwards as history fills in (llm_monitor's drawSpark). With `span` a
* partly-filled window occupies only the right of the plot — and x stays
* time, not sample index, so the shape keeps meaning *when*. Default 'left'
* spans the data domain (today's behaviour)
* - span: x window in seconds for align:'right' (default: the oldest sample)
* - labels: `{current, max}` → the small `now / max` foot drawSpark prints, formatted
* with `format`; `max` is the window maximum of the data, never the fixed
* axis
* - format: value → string for the labels and scale marks (default DEFAULT_FORMAT)
* - minPoints: finite points required before a line is drawn (default 2)
* - ariaLabel: names the spark for assistive tech; the generated fallback quotes values
*
* No animation and no CSS transition at all — `prefers-reduced-motion` is honoured by
* construction, and a moving squiggle competes with the number it annotates.
*/
function sparkline(container, points, opts) { function sparkline(container, points, opts) {
const o = Object.assign({ height: 34, color: cssVar('--series-1'), min: null, max: null }, opts || {}); const o = Object.assign({
const pts = (points || []).filter((p) => p && Number.isFinite(p.value)); height: 34, color: null, severity: null, min: null, max: null, fixedMax: null,
minSpan: null, align: 'left', span: null, labels: null, format: DEFAULT_FORMAT,
minPoints: 2, ariaLabel: null,
}, opts || {});
const fmt = typeof o.format === 'function' ? o.format : DEFAULT_FORMAT;
const pts = (points || [])
.map((p, i) => (p && Number.isFinite(p.value)
? { ts: Number.isFinite(p.ts) ? p.ts : i, value: p.value }
: null))
.filter(Boolean)
.sort((a, b) => a.ts - b.ts);
const w = Math.max(60, container.clientWidth || 160); const w = Math.max(60, container.clientWidth || 160);
const h = o.height; const h = o.height;
container.innerHTML = ''; container.innerHTML = '';
if (pts.length < 2) { if (pts.length < Math.max(1, o.minPoints)) {
container.innerHTML = '<div class="faint mono" style="font-size:11px">no history</div>'; container.innerHTML = '<div class="faint mono" style="font-size:11px">no history</div>';
return; return;
} }
const xs = pts.map((p) => p.ts), ys = pts.map((p) => p.value); const ys = pts.map((p) => p.value);
const x0 = Math.min(...xs), x1 = Math.max(...xs); const dataMax = Math.max(...ys);
let lo = o.min !== null ? o.min : Math.min(...ys); let lo, hi;
let hi = o.max !== null ? o.max : Math.max(...ys); if (o.fixedMax != null) {
lo = o.min != null ? o.min : 0;
hi = Math.max(o.fixedMax, lo); // never an inverted axis
} else {
lo = o.min !== null ? o.min : Math.min(...ys);
hi = o.max !== null ? o.max : Math.max(...ys);
if (lo === hi) { const pad = Math.abs(lo) * 0.1 || 1; lo -= pad; hi += pad; } if (lo === hi) { const pad = Math.abs(lo) * 0.1 || 1; lo -= pad; hi += pad; }
const sx = (ts) => ((ts - x0) / (x1 - x0 || 1)) * (w - 2) + 1; // the axis never gets finer than one thing (see `minSpan` above)
const sy = (v) => h - 2 - ((v - lo) / (hi - lo)) * (h - 4); if (o.minSpan > 0 && hi - lo < o.minSpan) hi = lo + o.minSpan;
}
const first = pts[0];
const last = pts[pts.length - 1];
let x0 = first.ts;
const x1 = last.ts;
if (o.align === 'right' && o.span > 0) x0 = x1 - o.span;
// Marks live in a left gutter so they never sit on the line; a 160px box has the pixels.
const marksOn = o.fixedMax != null;
const padL = marksOn ? 24 : 1;
const inner = { l: padL, r: 1, t: 2, b: 2 };
const plotW = Math.max(10, w - inner.l - inner.r);
const plotH = Math.max(6, h - inner.t - inner.b);
const sx = (ts) => {
const span = (x1 - x0) || 1;
const clamped = Math.min(Math.max(ts, x0), x1);
return inner.l + ((clamped - x0) / span) * plotW;
};
const sy = (v) => {
const c = o.fixedMax != null ? Math.min(Math.max(v, lo), hi) : v;
return inner.t + plotH - ((c - lo) / ((hi - lo) || 1)) * plotH;
};
let sev = o.severity;
if (sev && typeof sev === 'object') sev = sev.severity;
const sevColor = sev === 'ok' ? cssVar('--ok')
: sev === 'warn' ? cssVar('--warn')
: sev === 'fail' ? cssVar('--fail') : null;
const stroke = o.color || sevColor || cssVar('--text-faint');
const filled = Boolean(o.color || sevColor);
const d = pts.map((p, i) => `${i ? 'L' : 'M'}${sx(p.ts).toFixed(1)} ${sy(p.value).toFixed(1)}`).join(' '); const d = pts.map((p, i) => `${i ? 'L' : 'M'}${sx(p.ts).toFixed(1)} ${sy(p.value).toFixed(1)}`).join(' ');
const svg = `<svg viewBox="0 0 ${w} ${h}" width="100%" height="${h}" preserveAspectRatio="none" const baseY = (inner.t + plotH).toFixed(1);
role="img" aria-label="trend from ${lo.toFixed(2)} to ${hi.toFixed(2)}"> const area = `<path d="${d} L${sx(last.ts).toFixed(1)} ${baseY} L${sx(first.ts).toFixed(1)} ${baseY} Z" fill="${stroke}" opacity=".14"/>`;
<path d="${d} L${sx(x1).toFixed(1)} ${h} L${sx(x0).toFixed(1)} ${h} Z" fill="${o.color}" opacity=".14"/> const baseline = `<line x1="${inner.l}" x2="${(inner.l + plotW).toFixed(1)}" y1="${baseY}" y2="${baseY}" stroke="${stroke}" opacity=".22"/>`;
<path d="${d}" fill="none" stroke="${o.color}" stroke-width="1.4"/></svg>`; let marks = '';
container.innerHTML = svg; if (marksOn) {
// top/bottom of the *axis* only — enough to read the line as a fraction of the ceiling
const mk = (v, y) => `<text x="2" y="${y}" font-size="8" fill="${cssVar('--text-faint')}" dominant-baseline="${y < h / 2 ? 'hanging' : 'alphabetic'}">${escapeHtml(fmt(v))}</text>`;
marks = mk(hi, inner.t + 7) + mk(lo, h - inner.b);
}
const aria = o.ariaLabel ||
`trend from ${fmt(first.value)} to ${fmt(last.value)}, max ${fmt(dataMax)}`;
let foot = '';
if (o.labels && (o.labels.current || o.labels.max)) {
const bits = [];
if (o.labels.current) bits.push(`now ${fmt(last.value)}`);
if (o.labels.max) bits.push(`max ${fmt(dataMax)}`);
foot = `<div class="faint mono" style="font-size:10px;line-height:1.25">${escapeHtml(bits.join(' / '))}</div>`;
}
container.innerHTML = `<svg viewBox="0 0 ${w} ${h}" width="100%" height="${h}" preserveAspectRatio="none"
role="img" aria-label="${escapeHtml(aria)}">${filled ? area : ''}${baseline}
<path d="${d}" fill="none" stroke="${stroke}" stroke-width="1.4"/>${marks}</svg>${foot}`;
} }
window.DashCharts = { LineChart, niceTicks, renderLegend, sparkline, cssVar, window.DashCharts = { LineChart, niceTicks, renderLegend, sparkline, cssVar,
+44
View File
@@ -179,6 +179,11 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
.grid.auto { grid-template-columns: repeat(auto-fit, minmax(min(360px, 100%), 1fr)); } .grid.auto { grid-template-columns: repeat(auto-fit, minmax(min(360px, 100%), 1fr)); }
.grid.cols-2 { grid-template-columns: repeat(auto-fit, minmax(min(480px, 100%), 1fr)); } .grid.cols-2 { grid-template-columns: repeat(auto-fit, minmax(min(480px, 100%), 1fr)); }
.grid.cols-3 { grid-template-columns: repeat(auto-fit, minmax(min(300px, 100%), 1fr)); } .grid.cols-3 { grid-template-columns: repeat(auto-fit, minmax(min(300px, 100%), 1fr)); }
/* Cards that carry different amounts of content sit at their own height instead of being
stretched to the tallest sibling: a 5-row engine panel next to a 6-row one otherwise
grows a blank bottom, and a blank panel reads as "no data" (verified on `/llm`, phase
04 — 172px of empty card under the shorter panel). */
.grid.items-start { align-items: start; }
.stack { display: grid; gap: .85rem; align-content: start; } .stack { display: grid; gap: .85rem; align-content: start; }
/* ---- cards -------------------------------------------------------------- */ /* ---- cards -------------------------------------------------------------- */
@@ -234,6 +239,45 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
.na { color: var(--text-faint); } .na { color: var(--text-faint); }
/* ---- engine cell list -----------------------------------------------------
* `/llm`'s secondary rows: label left, value right-aligned in tabular numerals, sparkline
* in a fixed-width third column so rows line up and growth stays right-aligned the way
* `llm_monitor`'s drawSpark did. Percentiles ride under the value as a mono foot line —
* visible without a hover, because a mean latency without its p95 is half the truth.
* The heroes above are the loudest thing here on purpose; nothing in this block competes
* with them, and only payload verdicts colour it (KV through `.cell.warn/.cell.fail`).
*/
.cells { display: grid; gap: .45rem; }
.cell {
display: grid;
grid-template-columns: minmax(0, 1fr) max-content 160px;
gap: .1rem .7rem;
align-items: center;
}
.cell .cell-label {
font-size: 10.5px; text-transform: uppercase; letter-spacing: .06em; color: var(--text-faint);
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
}
.cell .cell-value {
font-family: var(--mono); font-size: 1.05rem; font-weight: 600; line-height: 1.3;
font-variant-numeric: tabular-nums;
text-align: right; white-space: nowrap;
}
.cell .cell-value .unit { font-size: .62em; color: var(--text-muted); font-weight: 500; margin-inline-start: .18rem; }
.cell .cell-value.is-null { color: var(--text-faint); }
.cell .cell-foot { grid-column: 2; text-align: right; font-family: var(--mono); font-size: 10.5px; color: var(--text-faint); }
.cell .cell-spark { display: block; width: 160px; }
/* the KV verdict, taken from the API's `severity` exactly as the hero cards take theirs */
.cell.warn .cell-value { color: var(--warn); }
.cell.fail .cell-value { color: var(--fail); }
/* alarm chips reuse the one fail chip vocabulary — the row itself only exists while the
counter is non-zero, so this is the presence of an incident, never a graded threshold */
.alarms { display: flex; flex-wrap: wrap; gap: .4rem; }
@media (max-width: 640px) {
.cell { grid-template-columns: minmax(0, 1fr) max-content; }
.cell .cell-spark { grid-column: 1 / -1; width: 100%; }
}
/* ---- hero numbers -------------------------------------------------------- /* ---- hero numbers --------------------------------------------------------
* The big glanceable number: label, value + unit, and optional bar and footnote. * The big glanceable number: label, value + unit, and optional bar and footnote.
* Colour comes from the *same* `ok|warn|fail` classes the rest of the stylesheet uses * Colour comes from the *same* `ok|warn|fail` classes the rest of the stylesheet uses
+295 -122
View File
@@ -1,40 +1,116 @@
/* LLM dashboard. /* LLM dashboard.
*
* Layout (phase 04, per NOTES-chart-set): per engine, two big throughput numbers you can
* read across the room, then a compact list of metric cells that each carry their own
* sparkline over the *selected* window, percentile foot lines under the latency means, and
* alarm chips that exist only while there is an incident. The four trend charts stay, but
* half-height and below the panels: they are supporting evidence, not the main event.
* *
* Every throughput cell shows where its number came from, because "78 tok/s" means * Every throughput cell shows where its number came from, because "78 tok/s" means
* different things depending on whether it was scraped from an aggregate gauge, measured as * different things depending on whether it was scraped from an aggregate gauge, measured as
* Δtokens/Δt across all slots, or estimated from TPOT over the process lifetime. The tag is * Δtokens/Δt across all slots, or estimated from TPOT over the process lifetime. The tag is
* generated from the cell's `source`, never hardcoded in markup, so a parser change cannot * generated from the cell's `source`, never hardcoded in markup, so a parser change cannot
* leave a stale label behind. * leave a stale label behind.
*
* Colour comes from payload verdicts only (`Dash.severityFrom` / `Dash.hero`). Throughput,
* sessions, queue depth and latency carry **no** severity — 0 tok/s on an idle server is
* not red, and a page-invented threshold is what `severity.py` exists to prevent.
*/ */
'use strict'; 'use strict';
const { $, fetchJSON, setTxt, setValue, setBar, setDot, num, pct, duration, poll, const { $, fetchJSON, setTxt, setDot, num, duration, poll, markStale, severityFrom, hero,
severityFor } = Dash; eventText } = Dash;
const state = { range: 3600, panels: new Map(), charts: {}, poller: null, pollMs: 2000 }; const state = {
range: 3600,
panels: new Map(), // engine -> card element
charts: {},
poller: null,
pollMs: 2000,
};
const fmtCount = (v) => num(v, 0);
const fmtPct = (v) => `${num(v, 0)}%`;
const fmtMs = (v) => `${num(v, 0)} ms`;
/**
* What each engine panel shows, driven by flags instead of fourteen equal-size cells:
* `hero` is the read-across-the-room row, `spark` is one compact row with a fixed-width
* sparkline, `alarm` renders only as a red chip while the counter is non-zero, and
* `footnote` is the quiet lifetime strip.
*
* `minSpan: 1` on the count rows keeps their axis at least one request tall: history is
* stored as rollups, so eight seconds of a single queued request averages to 0.14 and an
* axis fitted to that draws a full-height spike for a queue nobody was in.
*
* `spark` is the `/api/llm/history` series (without the `llm.` prefix); `scale` converts
* stored units to printed ones (a 0-1 ratio to %, seconds to ms) for the plot as well as
* the number, so the line and the digits agree. `fixedMax` pins percent-like metrics to a
* 0-100 axis — an 8 % window that once held a 92 % spike must not read flat. `only` scopes
* a row to the engines that publish it, so a gufo panel never shows a latency mystery and a
* vLLM panel never shows a speculative-decoding row it cannot fill.
*/
const CELLS = [ const CELLS = [
{ key: 'decode_tokens_per_second', label: 'Decode', unit: 'tok/s', digits: 1, { key: 'decode', label: 'Decode', unit: 'tok/s', digits: 1, hero: true,
tip: 'Aggregate generation speed across every busy slot.' }, tip: 'Aggregate generation speed across every busy slot.' },
{ key: 'prompt_tokens_per_second', label: 'Prompt', unit: 'tok/s', digits: 1, { key: 'prompt', label: 'Prompt', unit: 'tok/s', digits: 1, hero: true,
tip: 'Prefill speed for newly computed tokens.' }, tip: 'Prefill speed for newly computed tokens.' },
{ key: 'sessions', label: 'Sessions', unit: 'active', digits: 0 }, { key: 'sessions', label: 'Sessions', unit: 'active', digits: 0,
{ key: 'queued', label: 'Queued', unit: 'waiting', digits: 0 }, spark: 'requests_running', format: fmtCount, minSpan: 1 },
{ key: 'cache', label: 'KV cache', unit: '%', digits: 0, bar: true }, { key: 'queued', label: 'Queued', unit: 'waiting', digits: 0,
{ key: 'ttft_mean', label: 'TTFT mean', unit: 'ms', digits: 0 }, spark: 'requests_waiting', format: fmtCount, minSpan: 1 },
{ key: 'ttft_p95', label: 'TTFT p95', unit: 'ms', digits: 0 }, { key: 'cache', label: 'KV cache', unit: '%', digits: 0, fixedMax: 100,
{ key: 'tpot_mean', label: 'TPOT mean', unit: 'ms', digits: 1 }, spark: 'kv_cache_usage', scale: 100, format: fmtPct,
{ key: 'queue_mean', label: 'Queue mean', unit: 'ms', digits: 0 }, tip: 'Share of the KV cache in use — the only engine-pressure verdict on this page, '
{ key: 'cache_hit', label: 'Prefix cache hit', unit: '%', digits: 0 }, + 'and it comes from the API.' },
{ key: 'spec_accept', label: 'Spec accept', unit: '%', digits: 0 }, { key: 'ttft', label: 'TTFT', unit: 'ms', digits: 0, p95: true, only: ['vllm'],
{ key: 'tokens_total', label: 'Tokens lifetime', unit: '', digits: 0, human: true }, spark: 'ttft_seconds', scale: 1000, format: fmtMs },
{ key: 'preemptions', label: 'Preemptions', unit: '', digits: 0 }, { key: 'tpot', label: 'TPOT', unit: 'ms', digits: 1, p95: true, only: ['vllm'],
{ key: 'device_lost', label: 'GPU context lost', unit: '', digits: 0 }, spark: 'tpot_seconds', scale: 1000, format: fmtMs },
{ key: 'queue_mean', label: 'Queue time', unit: 'ms', digits: 0, p95: true, only: ['vllm'],
spark: 'queue_seconds', scale: 1000, format: fmtMs },
{ key: 'cache_hit', label: 'Prefix cache hit', unit: '%', digits: 0, fixedMax: 100,
spark: 'prefix_cache_hit_ratio', scale: 100, format: fmtPct },
{ key: 'spec_accept', label: 'Spec accept', unit: '%', digits: 0, fixedMax: 100,
only: ['gufo'], spark: 'spec_accept_rate', scale: 100, format: fmtPct },
{ key: 'preemptions', label: 'Preemptions', alarm: true,
tip: 'Running requests kicked back to the queue — an incident, not a stat.' },
{ key: 'device_lost', label: 'GPU context lost', alarm: true,
tip: 'The engine lost its GPU context — an incident, not a stat.' },
{ key: 'tokens_total', footnote: true },
]; ];
const HERO_CELLS = CELLS.filter((cell) => cell.hero);
const SPARK_CELLS = CELLS.filter((cell) => cell.spark);
const ALARM_CELLS = CELLS.filter((cell) => cell.alarm);
/** One batched history request feeds every chart *and* every cell sparkline. */
const HISTORY_SERIES = Array.from(new Set([
'decode_tokens_per_second', 'prompt_tokens_per_second',
...SPARK_CELLS.map((cell) => cell.spark),
]));
const WINDOW_NAMES = { 300: '5 minutes', 900: '15 minutes', 3600: '1 hour',
21600: '6 hours', 86400: '24 hours', 604800: '7 days',
2592000: '30 days' };
function windowName(seconds) {
return WINDOW_NAMES[seconds] || duration(seconds);
}
/** The payload cell behind a UI key. A missing number stays `null` — never a 0. */
function cellValue(engine, metrics, sessions, key) { function cellValue(engine, metrics, sessions, key) {
const get = (k) => (metrics[k] && metrics[k].value != null ? metrics[k] : null); const get = (k) => (metrics[k] && metrics[k].value != null ? metrics[k] : null);
const scaled = (cell, factor) => ({ value: cell.value * factor, source: cell.source,
derived: cell.derived });
switch (key) { switch (key) {
case 'decode':
case 'prompt': {
// the hero keys are short for the flags; the payload keeps the public metric name
const cell = get(`${key}_tokens_per_second`);
return cell ? { value: cell.value, source: cell.source, derived: cell.derived,
window: cell.window } : { value: null };
}
case 'sessions': { case 'sessions': {
const value = sessions && sessions.active != null ? sessions.active : null; const value = sessions && sessions.active != null ? sessions.active : null;
return { value, source: 'session count', derived: false }; return { value, source: 'session count', derived: false };
@@ -44,32 +120,33 @@ function cellValue(engine, metrics, sessions, key) {
return cell ? { value: cell.value, source: cell.source, derived: false } : { value: null }; return cell ? { value: cell.value, source: cell.source, derived: false } : { value: null };
} }
case 'cache': { case 'cache': {
const cell = get('kv_cache_usage_ratio') || get('gpu_cache_usage'); // the ratio cell carries the API verdict (`sources/llm.py` SEVERITY_CELLS ->
return cell ? { value: cell.value * 100, source: cell.source, derived: false } : { value: null }; // severity.py); it travels with the converted value so the row and its sparkline
// paint exactly what the overview's alerts paint
const cell = metrics.kv_cache_usage_ratio || metrics.gpu_cache_usage;
if (!cell || cell.value == null) return { value: null };
return { value: cell.value * 100, source: cell.source, derived: cell.derived,
severity: cell.severity };
} }
case 'ttft_mean': { case 'ttft': {
const cell = get('time_to_first_token_seconds__mean'); const cell = get('time_to_first_token_seconds__mean');
return cell ? { value: cell.value * 1000, source: cell.source, derived: false } : { value: null }; return cell ? scaled(cell, 1000) : { value: null };
} }
case 'ttft_p95': { case 'tpot': {
const cell = get('time_to_first_token_seconds__p95');
return cell ? { value: cell.value * 1000, source: cell.source, derived: false } : { value: null };
}
case 'tpot_mean': {
const cell = get('time_per_output_token_seconds__mean'); const cell = get('time_per_output_token_seconds__mean');
return cell ? { value: cell.value * 1000, source: cell.source, derived: false } : { value: null }; return cell ? scaled(cell, 1000) : { value: null };
} }
case 'queue_mean': { case 'queue_mean': {
const cell = get('request_queue_time_seconds__mean'); const cell = get('request_queue_time_seconds__mean');
return cell ? { value: cell.value * 1000, source: cell.source, derived: false } : { value: null }; return cell ? scaled(cell, 1000) : { value: null };
} }
case 'cache_hit': { case 'cache_hit': {
const cell = get('prefix_cache_hit_ratio') || get('prompt_cache_hit_ratio'); const cell = get('prefix_cache_hit_ratio') || get('prompt_cache_hit_ratio');
return cell ? { value: cell.value * 100, source: cell.source, derived: true } : { value: null }; return cell ? scaled(cell, 100) : { value: null };
} }
case 'spec_accept': { case 'spec_accept': {
const cell = get('spec_accept_rate'); const cell = get('spec_accept_rate');
return cell ? { value: cell.value * 100, source: cell.source, derived: true } : { value: null }; return cell ? scaled(cell, 100) : { value: null };
} }
case 'tokens_total': { case 'tokens_total': {
const cell = get('tokens_total'); const cell = get('tokens_total');
@@ -91,6 +168,15 @@ function cellValue(engine, metrics, sessions, key) {
} }
} }
/** The histogram p95 beside a latency mean — printed as a foot line, never behind a hover. */
function p95Value(metrics, key) {
const source = { ttft: 'time_to_first_token_seconds__p95',
tpot: 'time_per_output_token_seconds__p95',
queue_mean: 'request_queue_time_seconds__p95' }[key];
const cell = source && metrics[source];
return cell && cell.value != null ? cell.value * 1000 : null;
}
/** The provenance tag: Δ10.0s for measured rates, the formula for estimates, nothing for a /** The provenance tag: Δ10.0s for measured rates, the formula for estimates, nothing for a
* plain scraped gauge (which would only add noise). */ * plain scraped gauge (which would only add noise). */
function provenance(cell) { function provenance(cell) {
@@ -109,10 +195,25 @@ function humanNumber(value) {
return { text: value.toFixed(0), unit: 'tok' }; return { text: value.toFixed(0), unit: 'tok' };
} }
/** The rows this engine publishes. A row it can never fill is not shown at all — a
* permanent `n/a` is the dead-cell anti-pattern NOTES-chart-set exists to prevent. */
function sparkCellsFor(engine) {
return SPARK_CELLS.filter((cell) => !cell.only || cell.only.indexOf(engine) >= 0);
}
function enginePanel(engine) { function enginePanel(engine) {
const card = document.createElement('article'); const card = document.createElement('article');
card.className = 'card'; card.className = 'card';
card.dataset.engine = engine; card.dataset.engine = engine;
const heroes = HERO_CELLS.map((cell) =>
`<div class="hero" data-hero="${cell.key}"></div>`).join('');
const rows = sparkCellsFor(engine).map((cell) => `
<div class="cell" data-cell="${cell.key}"${cell.tip ? ` title="${cell.tip}"` : ''}>
<span class="cell-label">${cell.label}</span>
<span class="cell-value"><span data-v>n/a</span><span class="unit">${cell.unit}</span></span>
<span class="cell-spark" data-spark data-metric="${cell.spark}"></span>
${cell.p95 ? '<span class="cell-foot" data-foot></span>' : ''}
</div>`).join('');
card.innerHTML = ` card.innerHTML = `
<header> <header>
<span class="dot idle"></span> <span class="dot idle"></span>
@@ -123,13 +224,11 @@ function enginePanel(engine) {
</header> </header>
<div class="body stack"> <div class="body stack">
<div class="banner err" hidden></div> <div class="banner err" hidden></div>
<div class="metrics">${CELLS.map((cell) => ` <div class="heroes">${heroes}</div>
<div class="metric" data-cell="${cell.key}" ${cell.tip ? `title="${cell.tip}"` : ''}> <div class="alarms" data-k="alarms" hidden></div>
<div class="label">${cell.label}</div> <div class="cells">${rows}</div>
<div class="value${cell.digits === 0 ? '' : ''}"><span data-v>n/a</span><span class="unit">${cell.unit}</span></div>
${cell.bar ? '<div class="bar" style="margin-top:.25rem"><i></i></div>' : ''}
</div>`).join('')}</div>
<div class="faint mono sessions-note" style="font-size:11.5px"></div> <div class="faint mono sessions-note" style="font-size:11.5px"></div>
<div class="faint mono token-note" style="font-size:11px"></div>
<div class="faint mono debug" style="font-size:11px"></div> <div class="faint mono debug" style="font-size:11px"></div>
</div>`; </div>`;
return card; return card;
@@ -146,7 +245,7 @@ function renderPanel(card, enginePayload, stateRow) {
? (stateRow.error || 'never reached') ? (stateRow.error || 'never reached')
: (Dash.ageSeconds(stateRow.last_ok) > 15 ? `stale ${duration(Dash.ageSeconds(stateRow.last_ok))}` : (Dash.ageSeconds(stateRow.last_ok) > 15 ? `stale ${duration(Dash.ageSeconds(stateRow.last_ok))}`
: `updated ${duration(Dash.ageSeconds(stateRow.last_ok))}`)) : '—'); : `updated ${duration(Dash.ageSeconds(stateRow.last_ok))}`)) : '—');
Dash.markStale(card, stateRow && (!stateRow.ok || Dash.ageSeconds(stateRow.last_ok) > 15)); markStale(card, stateRow && (!stateRow.ok || Dash.ageSeconds(stateRow.last_ok) > 15));
const banner = q('.banner.err'); const banner = q('.banner.err');
if (stateRow && !stateRow.ok) { if (stateRow && !stateRow.ok) {
@@ -157,51 +256,57 @@ function renderPanel(card, enginePayload, stateRow) {
banner.hidden = true; banner.hidden = true;
} }
for (const cell of CELLS) { // The two heroes. No severity, by design: an idle server at 0 tok/s is healthy, so the
const wrapper = q(`[data-cell="${cell.key}"]`); // digits stay neutral ink and reachability colour lives on the dot and the stale dimming
if (!wrapper) continue; // (the `/system` NET precedent). `Dash.hero()` still owns the markup and the n/a rule.
for (const cell of HERO_CELLS) {
const value = cellValue(enginePayload.engine, metrics, sessions, cell.key); const value = cellValue(enginePayload.engine, metrics, sessions, cell.key);
const valueEl = wrapper.querySelector('[data-v]'); const node = hero(q(`[data-hero="${cell.key}"]`), {
const unitEl = wrapper.querySelector('.unit'); label: cell.label, value: value.value, unit: cell.unit, digits: cell.digits,
sub: provenance(value),
});
if (node) node.title = value.source || ''; // the exact source, as llm_monitor's tag does
}
// Compact value rows; their sparklines are drawn once per window per refresh from the
// one batched history response (`refreshHistory`).
for (const cell of sparkCellsFor(enginePayload.engine)) {
const row = q(`[data-cell="${cell.key}"]`);
if (!row) continue;
const value = cellValue(enginePayload.engine, metrics, sessions, cell.key);
const valueEl = row.querySelector('[data-v]');
const unitEl = row.querySelector('.unit');
// the row carries the payload verdict (KV cache is the only rated cell) so the
// sparkline, drawn from the history response, paints the same verdict the number does.
// Like the fleet table, only a breach paints the *text* — a healthy row stays quiet
// ink; the spark still strokes green through the same verdict.
const sev = severityFrom(value.severity);
row.dataset.sev = sev;
['ok', 'warn', 'fail'].forEach((name) => row.classList.toggle(name, name === sev));
if (value.value == null) { if (value.value == null) {
setTxt(valueEl, 'n/a'); setTxt(valueEl, 'n/a');
setTxt(unitEl, ''); // "n/a %" reads like a measurement of nothing setTxt(unitEl, ''); // "n/a %" reads like a measurement of nothing
wrapper.classList.add('is-null');
} else {
wrapper.classList.remove('is-null');
if (cell.human) {
const parts = humanNumber(value.value);
setTxt(valueEl, parts ? parts.text : 'n/a');
setTxt(unitEl, parts ? parts.unit : '');
} else { } else {
setTxt(valueEl, num(value.value, cell.digits)); setTxt(valueEl, num(value.value, cell.digits));
setTxt(unitEl, cell.unit); setTxt(unitEl, cell.unit);
} }
} if (cell.p95) {
let tag = wrapper.querySelector('.prov'); const p95 = p95Value(metrics, cell.key);
const label = provenance(value); setTxt(row.querySelector('[data-foot]'),
if (label) { p95 == null ? 'p95 n/a' : `p95 ${num(p95, cell.digits)} ${cell.unit}`);
if (!tag) {
tag = document.createElement('span');
tag.className = 'prov';
wrapper.querySelector('.value').appendChild(tag);
}
setTxt(tag, label);
tag.title = `${value.source || ''}${value.window ? ` · measured over ${value.window.toFixed(1)}s` : ''}`;
} else if (tag) {
tag.remove();
}
if (cell.bar) setBar(wrapper.querySelector('.bar'), value.value,
severityFor(value.value, 75, 90));
if (cell.key === 'device_lost' && value.value != null && value.value > 0) {
wrapper.querySelector('.value').classList.add('is-null');
wrapper.classList.add('fail');
wrapper.title = `GPU context has been lost ${value.value} times — an incident, not a stat`;
} }
} }
renderAlarms(card, enginePayload);
const note = q('.sessions-note'); const note = q('.sessions-note');
setTxt(note, sessions.detail || `${sessions.active == null ? 'n/a' : sessions.active} active sessions`); setTxt(note, sessions.detail || `${sessions.active == null ? 'n/a' : sessions.active} active sessions`);
// a lifetime counter is a footnote, not a glance metric (NOTES-chart-set, demoted)
const tokens = cellValue(enginePayload.engine, metrics, sessions, 'tokens_total');
const parts = tokens.value == null ? null : humanNumber(tokens.value);
setTxt(q('.token-note'), parts
? `lifetime ${parts.text} ${parts.unit} · ${tokens.source || 'unknown source'}`
: 'lifetime tokens n/a');
const upstream = q('.chip.upstream'); const upstream = q('.chip.upstream');
if (enginePayload.upstream) { if (enginePayload.upstream) {
upstream.hidden = false; upstream.hidden = false;
@@ -213,6 +318,39 @@ function renderPanel(card, enginePayload, stateRow) {
: ''); : '');
} }
/**
* Alarm chips exist only while there is an incident.
*
* A permanent cell for a counter that is always 0 trains the eye to ignore the row, so at
* zero the row does not exist: the container is emptied and hidden, and only a non-zero —
* an *incident*, not a graded threshold — puts a `.chip.fail` back. `sources/llm.py`
* already treats any non-zero gufo_device_lost_total this way.
*/
function renderAlarms(card, enginePayload) {
const host = card.querySelector('[data-k="alarms"]');
if (!host) return;
const chips = [];
for (const cell of ALARM_CELLS) {
const value = cellValue(enginePayload.engine, enginePayload.metrics || {},
enginePayload.sessions || {}, cell.key);
if (value.value != null && value.value > 0) {
chips.push(`<span class="chip fail" data-alarm="${cell.key}">${cell.label} ${num(value.value, 0)}</span>`);
}
}
const signature = chips.join('');
if (host.dataset.sig !== signature) {
host.innerHTML = signature;
host.dataset.sig = signature;
}
for (const chip of host.querySelectorAll('[data-alarm]')) {
const cell = ALARM_CELLS.find((c) => c.key === chip.dataset.alarm);
const value = cellValue(enginePayload.engine, enginePayload.metrics || {},
enginePayload.sessions || {}, cell.key);
chip.title = `${cell.tip || cell.label} · ${value.source || 'source unknown'}`;
}
host.hidden = chips.length === 0;
}
function renderSlots(payloads) { function renderSlots(payloads) {
const gufo = payloads.find((p) => p.engine === 'gufo'); const gufo = payloads.find((p) => p.engine === 'gufo');
const rows = gufo && gufo.sessions ? gufo.sessions.list || [] : []; const rows = gufo && gufo.sessions ? gufo.sessions.list || [] : [];
@@ -239,13 +377,19 @@ function renderSlots(payloads) {
} }
function makeCharts() { function makeCharts() {
const common = { height: 165, xSpan: state.range }; // Half of the old 165px: the cells read first, the charts back them up. Every plot
state.charts.decode = new DashCharts.LineChart('#chart-decode', Object.assign({}, common, { yMin: 0 })); // names itself for assistive tech — the legend is a sibling that labels nothing.
state.charts.prompt = new DashCharts.LineChart('#chart-prompt', Object.assign({}, common, { yMin: 0 })); const common = { height: 110, xSpan: state.range };
state.charts.kv = new DashCharts.LineChart('#chart-kv', Object.assign({}, common, state.charts.decode = new DashCharts.LineChart('#chart-decode', Object.assign({}, common, {
{ yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) })); yMin: 0, ariaLabel: 'Decode throughput in tokens per second over the selected window, one line per engine' }));
state.charts.queue = new DashCharts.LineChart('#chart-queue', Object.assign({}, common, state.charts.prompt = new DashCharts.LineChart('#chart-prompt', Object.assign({}, common, {
{ yMin: 0, yFormat: (v) => num(v, 0) })); yMin: 0, ariaLabel: 'Prompt / prefill throughput in tokens per second over the selected window, one line per engine' }));
state.charts.kv = new DashCharts.LineChart('#chart-kv', Object.assign({}, common, {
yMin: 0, yMax: 100, yFormat: (v) => num(v, 0),
ariaLabel: 'KV / prefix cache pressure in per cent over the selected window, one line per engine' }));
state.charts.queue = new DashCharts.LineChart('#chart-queue', Object.assign({}, common, {
yMin: 0, yFormat: (v) => num(v, 0),
ariaLabel: 'Requests running and requests waiting over the selected window, one pair of lines per engine' }));
} }
function pointsFor(response, engine, metric) { function pointsFor(response, engine, metric) {
@@ -253,59 +397,89 @@ function pointsFor(response, engine, metric) {
return found ? found.points.map((p) => ({ ts: p.ts, value: p.value })) : []; return found ? found.points.map((p) => ({ ts: p.ts, value: p.value })) : [];
} }
async function refreshCharts(engineNames) { function drawChart(chartEl, legendEl, chart, series, format, showPeak) {
chart.opts.xSpan = state.range;
chart.setData(series);
DashCharts.renderLegend(legendEl, series, format, showPeak);
}
/**
* Every chart and every cell sparkline, from ONE `/api/llm/history` request.
*
* The endpoint answers a metric list for every engine, and unknown names come back as
* empty series on purpose — so the page asks for the full cell list once per refresh
* instead of one XHR per cell, and a partly-filled window stays partly filled rather than
* rescaling to whatever samples exist.
*/
async function refreshHistory(engineNames) {
if (!engineNames.length) return; if (!engineNames.length) return;
const list = engineNames.join(','); const response = await fetchJSON(
const [decode, prompt, queue] = await Promise.all([ `/api/llm/history?engines=${engineNames.join(',')}` +
fetchJSON(`/api/llm/history?engines=${list}&metrics=decode_tokens_per_second&range=${state.range}`) `&metrics=${HISTORY_SERIES.join(',')}&range=${state.range}`).catch(() => null);
.catch(() => null), if (!response) return;
fetchJSON(`/api/llm/history?engines=${list}&metrics=prompt_tokens_per_second&range=${state.range}`) drawChart($('#chart-decode'), $('#legend-decode'), state.charts.decode,
.catch(() => null), engineNames.map((engine) => ({ name: engine,
fetchJSON(`/api/llm/history?engines=${list}&metrics=requests_running,requests_waiting,` + points: pointsFor(response, engine, 'decode_tokens_per_second') })),
`kv_cache_usage&range=${state.range}`).catch(() => null), (v) => num(v, 1) + ' tok/s', true);
]); drawChart($('#chart-prompt'), $('#legend-prompt'), state.charts.prompt,
if (decode) { engineNames.map((engine) => ({ name: engine,
const series = engineNames.map((engine) => ({ name: engine, points: pointsFor(response, engine, 'prompt_tokens_per_second') })),
points: pointsFor(decode, engine, 'decode_tokens_per_second') })); (v) => num(v, 0) + ' tok/s', true);
state.charts.decode.setData(series); drawChart($('#chart-kv'), $('#legend-kv'), state.charts.kv,
DashCharts.renderLegend($('#legend-decode'), series, (v) => num(v, 1) + ' tok/s', true); engineNames.map((engine) => ({ name: engine,
} points: pointsFor(response, engine, 'kv_cache_usage').map((p) => ({ ts: p.ts, value: p.value * 100 })) })),
if (prompt) { (v) => num(v, 1) + '%', true);
const series = engineNames.map((engine) => ({ name: engine, const sessionsSeries = [];
points: pointsFor(prompt, engine, 'prompt_tokens_per_second') }));
state.charts.prompt.setData(series);
DashCharts.renderLegend($('#legend-prompt'), series, (v) => num(v, 0) + ' tok/s', true);
}
if (queue) {
const series = [];
for (const engine of engineNames) { for (const engine of engineNames) {
series.push({ name: `${engine} running`, points: pointsFor(queue, engine, 'requests_running') }); sessionsSeries.push({ name: `${engine} running`, points: pointsFor(response, engine, 'requests_running') });
series.push({ name: `${engine} waiting`, points: pointsFor(queue, engine, 'requests_waiting') }); sessionsSeries.push({ name: `${engine} waiting`, points: pointsFor(response, engine, 'requests_waiting') });
}
drawChart($('#chart-queue'), $('#legend-queue'), state.charts.queue, sessionsSeries,
(v) => num(v, 0), false);
drawSparks(response);
}
/** One sparkline per cell, over the *selected* window, x mapped to real time. */
function drawSparks(response) {
for (const [engine, card] of state.panels) {
for (const cell of sparkCellsFor(engine)) {
const row = card.querySelector(`[data-cell="${cell.key}"]`);
const slot = row && row.querySelector('[data-spark]');
if (!slot) continue;
const points = pointsFor(response, engine, cell.spark)
.map((p) => ({ ts: p.ts, value: p.value * (cell.scale || 1) }));
DashCharts.sparkline(slot, points, {
align: 'right', // newest pinned right; x stays time, not sample index
span: state.range,
fixedMax: cell.fixedMax != null ? cell.fixedMax : null,
minSpan: cell.minSpan != null ? cell.minSpan : null,
// the only rated spark is KV, and its colour is the API's verdict; everything else
// draws the faint unrated stroke, because a shape is not a verdict
severity: row.dataset.sev || '',
labels: { max: true },
format: cell.format,
ariaLabel: `${engine} ${cell.label} over the last ${windowName(state.range)}`,
});
} }
state.charts.queue.setData(series);
DashCharts.renderLegend($('#legend-queue'), series, (v) => num(v, 0));
const kv = engineNames.map((engine) => ({ name: engine,
points: pointsFor(queue, engine, 'kv_cache_usage').map((p) => ({ ts: p.ts, value: p.value * 100 })) }));
state.charts.kv.setData(kv);
DashCharts.renderLegend($('#legend-kv'), kv, (v) => num(v, 1) + '%', true);
} }
} }
/**
* The engine-totals card: only what no single panel can say.
*
* Summed decode/prompt/KV left with phase 04 (per-engine is the honest view), and the card
* hides entirely below two engines — a fleet of one does not need a fleet row.
*/
function renderFleet(fleet) { function renderFleet(fleet) {
if (!fleet) return; const card = $('#totals');
if (!card) return;
const total = fleet && fleet.engines_total != null ? fleet.engines_total : 0;
card.hidden = total < 2;
if (!total) return;
setTxt($('#f-live'), fleet.engines_live == null ? 'n/a' : String(fleet.engines_live)); setTxt($('#f-live'), fleet.engines_live == null ? 'n/a' : String(fleet.engines_live));
setTxt($('#f-total'), fleet.engines_total == null ? '' : `of ${fleet.engines_total}`); setTxt($('#f-total'), fleet.engines_total == null ? '' : `of ${fleet.engines_total}`);
setValue($('#f-decode'), fleet.decode_tokens_per_second, 1);
setValue($('#f-prompt'), fleet.prompt_tokens_per_second, 0);
setTxt($('#f-active'), fleet.requests_active == null ? 'n/a' : num(fleet.requests_active, 0)); setTxt($('#f-active'), fleet.requests_active == null ? 'n/a' : num(fleet.requests_active, 0));
setTxt($('#f-waiting'), fleet.requests_waiting == null ? 'n/a' : num(fleet.requests_waiting, 0)); setTxt($('#f-waiting'), fleet.requests_waiting == null ? 'n/a' : num(fleet.requests_waiting, 0));
const tokens = humanNumber(fleet.tokens_total);
setTxt($('#f-tokens'), tokens ? tokens.text : 'n/a');
$('#f-tokens').nextElementSibling && setTxt($('#f-tokens').nextElementSibling, tokens ? tokens.unit : '');
const kv = fleet.kv_cache_usage_max == null ? null : fleet.kv_cache_usage_max * 100;
setValue($('#f-kv'), kv, 0);
setBar($('#f-kv').closest('.metric').querySelector('.bar'), kv, severityFor(kv, 75, 90));
$('#seam-note').textContent = fleet.seams || '';
} }
async function refresh() { async function refresh() {
@@ -339,7 +513,7 @@ async function refresh() {
`window ${duration(state.range)}`; `window ${duration(state.range)}`;
$('#foot-note').textContent = `poll ${Math.round(state.pollMs / 1000)}s · rates over the ` + $('#foot-note').textContent = `poll ${Math.round(state.pollMs / 1000)}s · rates over the ` +
`Δ window in each tag`; `Δ window in each tag`;
await refreshCharts(names); await refreshHistory(names);
} }
async function refreshEvents() { async function refreshEvents() {
@@ -353,7 +527,7 @@ async function refreshEvents() {
list.innerHTML = data.events.map((event) => `<li><time>${ list.innerHTML = data.events.map((event) => `<li><time>${
new Date(event.ts * 1000).toLocaleTimeString(undefined, { hour: '2-digit', minute: '2-digit', hour12: false }) new Date(event.ts * 1000).toLocaleTimeString(undefined, { hour: '2-digit', minute: '2-digit', hour12: false })
}</time><span><span class="sev ${event.severity}">${DashCharts.escapeHtml(event.host)}</span> ` + }</time><span><span class="sev ${event.severity}">${DashCharts.escapeHtml(event.host)}</span> ` +
`${DashCharts.escapeHtml(event.message)}</span></li>`).join(''); `${DashCharts.escapeHtml(eventText(event))}</span></li>`).join('');
} catch (err) { /* next poll */ } } catch (err) { /* next poll */ }
} }
@@ -373,8 +547,7 @@ document.addEventListener('DOMContentLoaded', async () => {
Dash.rangeControl($('#range'), state.range, (seconds) => { Dash.rangeControl($('#range'), state.range, (seconds) => {
state.range = seconds; state.range = seconds;
Object.values(state.charts).forEach((chart) => { chart.opts.xSpan = seconds; }); refresh(); // charts and sparklines both re-read the selected window
refresh();
}); });
state.poller = poll(async () => { await refresh(); await refreshEvents(); }, state.pollMs); state.poller = poll(async () => { await refresh(); await refreshEvents(); }, state.pollMs);
+1 -1
View File
@@ -472,7 +472,7 @@ async function refreshEvents() {
list.innerHTML = data.events.map((event) => `<li><time>${ list.innerHTML = data.events.map((event) => `<li><time>${
new Date(event.ts * 1000).toLocaleTimeString(undefined, { hour: '2-digit', minute: '2-digit', hour12: false }) new Date(event.ts * 1000).toLocaleTimeString(undefined, { hour: '2-digit', minute: '2-digit', hour12: false })
}</time><span><span class="sev ${event.severity}">${DashCharts.escapeHtml(event.host)}</span> ` + }</time><span><span class="sev ${event.severity}">${DashCharts.escapeHtml(event.host)}</span> ` +
`${DashCharts.escapeHtml(event.message)}</span></li>`).join(''); `${DashCharts.escapeHtml(Dash.eventText(event))}</span></li>`).join('');
} catch (err) { /* the next poll will try again */ } } catch (err) { /* the next poll will try again */ }
} }
+3 -2
View File
@@ -10,7 +10,8 @@
*/ */
'use strict'; 'use strict';
const { $, fetchJSON, setTxt, setValue, setBar, setDot, num, duration, poll, severityFor } = Dash; const { $, fetchJSON, setTxt, setValue, setBar, setDot, num, duration, poll, severityFor,
eventText } = Dash;
const state = { range: 86400, cards: new Map(), charts: {}, poller: null, pollMs: 5000, const state = { range: 86400, cards: new Map(), charts: {}, poller: null, pollMs: 5000,
simulated: false, pending: null }; simulated: false, pending: null };
@@ -327,7 +328,7 @@ async function refreshEvents() {
new Date(event.ts * 1000).toLocaleString(undefined, new Date(event.ts * 1000).toLocaleString(undefined,
{ month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit', hour12: false }) { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit', hour12: false })
}</time><span><span class="sev ${event.severity}">${DashCharts.escapeHtml(event.host)}</span> ` + }</time><span><span class="sev ${event.severity}">${DashCharts.escapeHtml(event.host)}</span> ` +
`${DashCharts.escapeHtml(event.message.replace(event.host + ':', '').trim())}` + `${DashCharts.escapeHtml(eventText(event))}` +
`${event.data && event.data.runtime_seconds ? ` <span class="faint mono">(${ `${event.data && event.data.runtime_seconds ? ` <span class="faint mono">(${
duration(event.data.runtime_seconds)} runtime)</span>` : ''}</span></li>`).join(''); duration(event.data.runtime_seconds)} runtime)</span>` : ''}</span></li>`).join('');
} catch (err) { /* next poll */ } } catch (err) { /* next poll */ }
+277 -6
View File
@@ -24,6 +24,7 @@ from dashboards.app import create_app
from dashboards.config import Settings, Target from dashboards.config import Settings, Target
from dashboards.severity import classify from dashboards.severity import classify
from dashboards.sources import nut from dashboards.sources import nut
from dashboards.sources.llm import HISTORY_METRICS
from dashboards.store import Event, Sample, SampleStore from dashboards.store import Event, Sample, SampleStore
# refused immediately, and never dials out of the test environment # refused immediately, and never dials out of the test environment
@@ -576,15 +577,15 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None: def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
"""The old call sites still work until the remaining phases convert each page. """The old call sites still work until the remaining phases convert each page.
`/` was converted in phase 02 and `/system` in phase 03: both colour themselves from `/` was converted in phase 02, `/system` in phase 03 and `/llm` in phase 04: each
the `severity` verdicts the API already sends, so they carry no threshold of their colours itself from the `severity` verdicts the API already sends, so it carries no
own. The two pages still to come must keep compiling against the fallback until they threshold of its own. `/ups` is phase 05's job and must keep compiling against the
are converted — deleting it early would break a page mid-migration. fallback until it is converted — deleting it early would break that page.
""" """
js = client.get("/static/app.js").text js = client.get("/static/app.js").text
assert "FALLBACK ONLY" in js assert "FALLBACK ONLY" in js
for page in ("/static/page-llm.js", "/static/page-ups.js"): assert "severityFor" in client.get("/static/page-ups.js").text, \
assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site" "/ups still uses the fallback until phase 05 converts it"
def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> None: def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> None:
@@ -786,6 +787,222 @@ def test_no_per_core_visual_library_survives_the_page(client) -> None:
assert name in client.get("/static/charts.js").text assert name in client.get("/static/charts.js").text
def test_sparkline_ships_the_fixed_scale_and_verdict_options(client) -> None:
"""Phase 04 turns `/llm` cells into sparkline rows, so the shared `sparkline()` must
behave like `llm_monitor`'s `drawSpark` (the task-02 contract, NOTES-chart-set).
Pinned on the served bytes because there is no JS test runner (a locked anchor). What
matters is not the option names but the semantics behind each one: a percent metric
that silently rescaled to its own data would make an 8 % KV cache and a 92 % spike the
same squiggle, and a stroke coloured from a page-invented threshold would break the
one-colour-language rule the same way.
"""
charts_js = client.get("/static/charts.js").text
body = charts_js[charts_js.index("function sparkline("):charts_js.index("window.DashCharts")]
# the phase grep criterion, as a test: the options are really in the shipped file
hits = [ln for ln in charts_js.splitlines()
if re.search(r"fixedMax|align|severity", ln)]
assert len(hits) >= 3, "the sparkline upgrade is not in the served charts.js"
for opt in ("fixedMax", "align", "labels", "severity", "minPoints", "span", "format",
"minSpan"):
assert opt in body, f"sparkline() lost its `{opt}` option"
# fixedMax: the data clamps to the axis, never the axis to the data
assert "Math.min(Math.max(v, lo), hi)" in body, \
"a fixedMax sparkline rescales to its own data again"
# align:'right' grows within a real time window (span seconds), not a sample count
assert "x1 - o.span" in body, "align:'right' forgot the window it grows inside"
# verdict colour comes from the theme tokens only; unrated is faint grey, never green
for token in ("cssVar('--ok')", "cssVar('--warn')", "cssVar('--fail')",
"cssVar('--text-faint')"):
assert token in body, f"sparkline() stopped taking its colour from {token}"
assert ".sev-" not in body, "sparkline invented a parallel severity vocabulary"
# the `now / max` foot llm_monitor prints, via the caller's format callback
assert "now ${fmt(last.value)}" in body and "max ${fmt(dataMax)}" in body
# never a confident line through one point, and the foot quotes data, not the padded axis
assert "minPoints: 2" in body and "no history" in body
# ASSUMPTION (task 02): no animation at all, so prefers-reduced-motion holds by design
assert "transition:" not in body and "animation:" not in body
# ---- the `/llm` layout (phase 04 task 03) ---------------------------------
#
# The drift this phase fixes was a 14-cell grid where everything was the same size and
# nothing showed recent shape. These tests pin the *shape* the page committed to on the
# served bytes (a locked anchor: no JS test runner), so a regression to "all cells equal"
# fails a test rather than a screenshot.
def test_llm_page_colours_from_the_payload_not_from_js_thresholds(client) -> None:
"""`page-llm.js` paints the KV verdict `severity.py` produced; it owns no threshold.
Both `severityFor(value, 75, 90)` call sites are gone (the task's explicit work item):
the KV row takes `severity` from the rated ratio cell, the fleet KV bar was deleted
with the summed card, and every unrated number — throughput above all, because 0 tok/s
on an idle server is healthy — draws no colour at all.
"""
js = client.get("/static/page-llm.js").text
assert "severityFor(" not in js, "the LLM page re-invented a threshold"
assert "severityFrom" in js
# the KV verdict comes from the payload cells `sources/llm.py` rates
kv = js[js.index("case 'cache':"):js.index("case 'ttft':")]
assert "kv_cache_usage_ratio" in kv and "gpu_cache_usage" in kv and "severity" in kv
def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
"""Decode and prompt are the largest elements per engine; the rest is visibly secondary.
The hero row is `Dash.hero()` markup (2.4rem via `.hero`), built from the `hero: true`
flags in `CELLS`, and throughput carries **no** severity — `severity.py` has no
throughput row and the page invents none (the neutral-ink decision in NOTES-chart-set,
mirroring the `/system` NET precedent). The provenance tag stays in the hero's `.sub`
with the exact `source` as the tooltip: removing it to look cleaner is a regression.
"""
js = client.get("/static/page-llm.js").text
panel = js[js.index("function enginePanel("):js.index("function renderPanel(")]
assert 'data-hero="${cell.key}"' in panel, "the hero row is not flag-driven markup"
assert '<div class="heroes">' in panel, "the hero row left the engine panel"
heroes = re.findall(r"{ key: '(\w+)'[^}]*?hero: true", js)
assert heroes == ["decode", "prompt"], "the hero row is no longer the two throughputs"
# driven by flags, not fourteen identical cells
assert "SPARK_CELLS" in js and "ALARM_CELLS" in js
assert js.count("spark: '") >= 6, "the compact sparkline list shrank back to a grid"
# the hero call must not pass a severity — unrated by design
hero_call = js[js.index("const node = hero("):js.index("if (node) node.title")]
assert "severity" not in hero_call, "throughput invented a verdict"
# provenance intact: the tag is generated from the cell's `source`, never hardcoded
assert "provenance" in js and "Δ" in js
assert "node.title = value.source" in js
def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client) -> None:
"""Each compact cell draws its own sparkline over the *selected* window; latency cells
print mean **and** p95 without a hover.
The window is honest time (`align: 'right'` + `span`), percent-like metrics keep a
fixed 0-100 axis, and the p95 comes from the snapshot's histogram cells — p95 needs no
stored series. The `only` scopes are what keep a gufo panel from showing a latency
mystery and a vLLM panel from showing a spec-decode row it cannot fill.
"""
js = client.get("/static/page-llm.js").text
sparks = js[js.index("function drawSparks("):js.index("function renderFleet(")]
assert "DashCharts.sparkline(" in sparks
for opt in ("align: 'right'", "span: state.range", "fixedMax", "labels: { max: true }"):
assert opt in sparks, f"cell sparklines lost `{opt}`"
# percent cells pin the axis; ratio series are scaled to the printed percent
assert js.count("fixedMax: 100") >= 3, "KV / prefix-hit / spec-accept lost their 0-100 axis"
assert "scale: 1000" in js and "scale: 100" in js
# every spark cell declares its history series in the markup it is built from
assert 'data-spark data-metric="${cell.spark}"' in js
# percentile foot lines: mean big in the row, p95 beneath, n/a when nothing measured
assert "function p95Value(" in js
for name in ("time_to_first_token_seconds__p95", "time_per_output_token_seconds__p95",
"request_queue_time_seconds__p95"):
assert name in js, f"{name} no longer feeds a foot line"
assert "p95 n/a" in js
assert 'data-foot' in js
# engine-scoped rows: gufo never grows a TTFT row, vLLM never a spec-accept row
assert "only: ['vllm']" in js and "only: ['gufo']" in js
scoped = js[js.index("function sparkCellsFor("):js.index("function enginePanel(")]
assert "cell.only" in scoped
# a count row never gets an axis finer than one request: history arrives as rollups,
# so 8 s of a single queued request averages to 0.14 and a fitted axis draws a spike for
# a queue nobody was in — the twin of the rescale `fixedMax` fixes
assert _code(js).count("minSpan: 1") == 2, "sessions/queued lost their one-request axis floor"
assert "minSpan: cell.minSpan" in sparks, "cell minSpan never reaches the sparkline"
def test_llm_sparklines_and_charts_cost_one_batched_history_request(client) -> None:
"""One `/api/llm/history` call per refresh feeds charts *and* every sparkline row.
The endpoint answers a metric list for every engine, and unknown names come back as
empty series on purpose — so the page asks for the full cell set once instead of one
XHR per cell, and the two series phase 04 added (`tpot_seconds`, `spec_accept_rate`)
must be in that list or those rows are permanent "no history" slots.
"""
js = client.get("/static/page-llm.js").text
# comments aside, exactly one fetch hits the endpoint: one request, every metric
calls = re.findall(r"fetchJSON\(\s*`/api/llm/history", _code(js))
assert len(calls) == 1, "the page makes more than one history request again"
assert "HISTORY_SERIES.join" in js and "SPARK_CELLS.map((cell) => cell.spark)" in js
request = js[js.index("async function refreshHistory("):js.index("function drawSparks(")]
assert "metrics=${HISTORY_SERIES.join" in request
assert "range=${state.range}" in request
# and the endpoint really answers every name the page asks for
for name in ("tpot_seconds", "spec_accept_rate", "ttft_seconds", "queue_seconds",
"kv_cache_usage", "requests_running", "requests_waiting",
"prefix_cache_hit_ratio"):
assert name in HISTORY_METRICS, f"llm.{name} is not stored, so its spark is dead"
def test_llm_alarm_cells_exist_only_while_there_is_an_incident(client) -> None:
"""`preemptions` / `device_lost` are absent at zero and red chips above zero.
The rule is the *existence of an incident*, not a graded threshold — `sources/llm.py`
already treats any non-zero `gufo_device_lost_total` as one. The chip reuses the shared
`.chip.fail` vocabulary; nothing new is painted, and a container emptied at zero means
the row does not exist rather than showing a permanent ignored 0.
"""
js = client.get("/static/page-llm.js").text
alarms = js[js.index("function renderAlarms("):js.index("function renderSlots(")]
assert "class=\"chip fail\"" in alarms, "the alarm chip left the shared fail vocabulary"
assert "value.value > 0" in alarms
assert "host.hidden = chips.length === 0" in alarms
assert 'data-alarm="${cell.key}"' in alarms
assert js.count("alarm: true") == 2
def test_llm_totals_card_says_only_what_no_engine_panel_says(client) -> None:
"""Engines answering, active, queued — and hidden below two engines.
The summed decode/prompt and the un-rated fleet KV bar are gone: per-engine is the
honest view (the fleet KV bar had no `severity` to render, which is invariant-grade
trouble), and a cross-engine sum of one engine just duplicates that engine's own card
(the `llm_monitor` rule). `/api/llm/snapshot` keeps every field — this is a page
decision, not an API one.
"""
html = _code(client.get("/llm").text)
for gone in ("f-decode", "f-prompt", "f-kv", "f-tokens", "seam-note"):
assert gone not in html, f"/llm still totals {gone}"
for kept in ("f-live", "f-active", "f-waiting", 'id="totals"'):
assert kept in html, f"/llm lost {kept}"
js = client.get("/static/page-llm.js").text
assert "card.hidden = total < 2" in js, "the totals card no longer hides for one engine"
def test_llm_charts_are_secondary_and_named(client) -> None:
"""The four trend charts stay, at half height, *below* the engine panels, each named.
Half of the old 165px and moved after `#engines` is the "cells read first" decision;
every `LineChart` gets an `ariaLabel` because the legend is a sibling that labels
nothing (`/system` did this in phase 03, `/llm` never had).
"""
html = _code(client.get("/llm").text)
assert html.index('id="engines"') < html.index('id="chart-decode"'), \
"the charts are back on top of the engine panels"
for chart in ("chart-decode", "chart-prompt", "chart-kv", "chart-queue"):
assert f'id="{chart}"' in html, f"/llm lost the {chart} chart"
js = client.get("/static/page-llm.js").text
assert "height: 110" in js and "height: 165" not in js
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
assert charts.count("ariaLabel:") == 4, "an /llm chart lost its accessible name"
def test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling(client) -> None:
"""A 5-row gufo panel beside a 7-row vLLM panel keeps its own height.
Found while comparing against `llm_monitor`: the shorter engine card grew 172 px of
blank card, and a blank panel reads as "no data" — the exact misreading the unconfigured
rules exist to prevent. Pinned on the served markup so the `.grid.items-start` rule in
`dash.css` cannot silently become dead CSS (and the void cannot silently come back).
"""
html = _code(client.get("/llm").text)
engines_div = html[html.index('<div class="grid'):html.index('id="engines"') + 40]
assert "items-start" in engines_div, "the engine grid stretches its cards again"
css = client.get("/static/dash.css").text
assert ".grid.items-start" in css, "the rule the engine grid relies on is gone"
def test_overview_is_exactly_two_panels(client) -> None: def test_overview_is_exactly_two_panels(client) -> None:
"""The landing page answers one question: what is wrong, and what is the fleet doing. """The landing page answers one question: what is wrong, and what is the fleet doing.
@@ -1019,6 +1236,38 @@ def test_llm_history_charts_both_engines_by_default(
("vllm", "llm.kv_cache_usage_ratio")} ("vllm", "llm.kv_cache_usage_ratio")}
def test_one_batched_llm_history_request_answers_every_cell_series(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The exact query `/llm` issues per refresh returns every chart *and* sparkline series.
Phase 04 sends one request carrying the chart trio and the whole spark cell list —
including the two series the layout required (`tpot_seconds` for the vLLM TPOT row,
`spec_accept_rate` for the gufo spec row). A name missing here is a cell that reads
"no history" forever in the browser, which is the dead-cell anti-pattern the phase
wrote NOTES-chart-set to prevent; a rename in `sources/llm.py` fails here first.
"""
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
now = time.time()
metrics = ["decode_tokens_per_second", "prompt_tokens_per_second",
"kv_cache_usage", "requests_running", "requests_waiting",
"ttft_seconds", "tpot_seconds", "queue_seconds",
"prefix_cache_hit_ratio", "spec_accept_rate"]
for metric in metrics:
store.add("llm", "gufo", f"llm.{metric}", 1.5, ts=now - 60)
store.add("llm", "vllm", f"llm.{metric}", 2.5, ts=now - 30)
body = test_client.get("/api/llm/history", params={
"engines": "gufo,vllm", "metrics": ",".join(metrics), "range": "1h"}).json()
assert body["range_seconds"] == 3_600.0
by_key = {(item["host"], item["metric"]) for item in body["series"]}
assert by_key == {(engine, f"llm.{metric}")
for engine in ("gufo", "vllm") for metric in metrics}
points = {(item["host"], item["metric"]): item["points"] for item in body["series"]}
assert points[("gufo", "llm.spec_accept_rate")][0]["value"] == pytest.approx(1.5)
assert points[("vllm", "llm.tpot_seconds")][0]["value"] == pytest.approx(2.5)
def test_metrics_endpoint_lists_what_actually_has_history(client) -> None: def test_metrics_endpoint_lists_what_actually_has_history(client) -> None:
body = client.get("/api/metrics").json() body = client.get("/api/metrics").json()
assert body["metrics"] == [] assert body["metrics"] == []
@@ -1039,6 +1288,28 @@ def test_events_are_returned_in_json(tmp_path: Path, monkeypatch: pytest.MonkeyP
store.close() store.close()
def test_event_rows_say_their_subject_once(client) -> None:
"""`poller.py` messages name their own subject, and the row prints a host chip too.
"gufo gufo is answering again after 1 failure" reads as two problems — the same defect
`alerts._clean_error()` already fixes for the overview's alert list. `/ups` had the
mirror-image bug: it `replace()`d `host + ':'` *anywhere*, so "Quick battery test sent
to apc: OK" printed as "Quick battery test sent to OK". All three feeds now share
`Dash.eventText()`, which strips the wrapper and a **leading** name only.
"""
app_js = client.get("/static/app.js").text
helper = app_js[app_js.index("function eventText("):app_js.index("// ---- DOM helpers")]
assert "eventText" in app_js[app_js.index("return {"):], "eventText is not exported"
assert "^UpstreamError:\\s+" in helper, "the wrapper `alerts.py` drops is back"
assert "new RegExp" in helper and "stripped.trim() ? stripped : out" in helper, \
"the host strip is no longer anchored to the front, or can empty a message"
for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"):
js = client.get(page).text
assert "eventText(event)" in js, f"{page} prints the raw message again"
assert "replace(event.host" not in client.get("/static/page-ups.js").text, \
"/ups is mangling the host name mid-message again"
# ---- the one write endpoint ------------------------------------------------ # # ---- the one write endpoint ------------------------------------------------ #
def test_command_endpoint_is_post_only(ups_client) -> None: def test_command_endpoint_is_post_only(ups_client) -> None:
+27
View File
@@ -231,6 +231,33 @@ def test_engine_result_charts_a_measured_rate_after_the_second_poll(metrics_text
assert any(s.metric == "llm.decode_tokens_per_second" for s in result.samples) assert any(s.metric == "llm.decode_tokens_per_second" for s in result.samples)
def test_vllm_stores_the_tpot_series_the_cell_sparkline_draws(metrics_text) -> None:
"""Phase 04 added `llm.tpot_seconds` so the vLLM TPOT row has a shape, not a dead slot.
Hermetic twin of the live check: the captured exposition must resolve the histogram the
page charts, or `/llm` shows a permanent "no history" next to a real mean.
"""
result = engine_result("vllm", "http://v:1", metrics_text("vllm"), RateEstimator(), now=1.0)
mean = result.payload["metrics"]["time_per_output_token_seconds__mean"]["value"]
assert mean is not None, "the vLLM capture lost its TPOT histogram — re-capture fixtures"
stored = {s.metric: s.value for s in result.samples}
assert stored["llm.tpot_seconds"] == pytest.approx(mean)
# gufo publishes no output-token histogram, so it stores no TPOT series (never a 0)
gufo = engine_result("gufo", "http://g:1", metrics_text("gufo"), RateEstimator(), now=1.0)
assert "llm.tpot_seconds" not in {s.metric for s in gufo.samples}
def test_gufo_stores_the_spec_accept_series_the_cell_sparkline_draws(metrics_text) -> None:
"""`llm.spec_accept_rate` is the gufo-only half of the phase-04 history addition."""
result = engine_result("gufo", "http://g:1", metrics_text("gufo"), RateEstimator(), now=1.0)
ratio = result.payload["metrics"]["spec_accept_rate"]["value"]
assert ratio is not None and 0.0 < ratio <= 1.0
stored = {s.metric: s.value for s in result.samples}
assert stored["llm.spec_accept_rate"] == pytest.approx(ratio)
vllm = engine_result("vllm", "http://v:1", metrics_text("vllm"), RateEstimator(), now=1.0)
assert "llm.spec_accept_rate" not in {s.metric for s in vllm.samples}
def test_sessions_pass_through_slots_for_gufo() -> None: def test_sessions_pass_through_slots_for_gufo() -> None:
slots = [{"id": 0, "idle": False, "n_prompt": 12, "n_predict": 40}] slots = [{"id": 0, "idle": False, "n_prompt": 12, "n_predict": 40}]
result = engine_result("gufo", "http://g:1", "", RateEstimator(), slots=slots, now=1.0) result = engine_result("gufo", "http://g:1", "", RateEstimator(), slots=slots, now=1.0)