services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 04_llm_chart_readability

Browse Source 
**Phase 04 final verification pass — all tasks already in `complete/`; verified + fixed 3 defects found in verification.**

- Verified `/llm` live (both engines, gufo-only instance on :8099, `llm_monitor` on :8095) in a real browser; dev server left running/healthy.
- Completed task-04's truncated layout fix: `#engines` now carries `.items-start` (the CSS rule existed but nothing used it) — gufo 544 px vs vLLM 715 px, no blank card; pinned by `test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling`.
- Fixed: event rows said the subject twice ("gufo gufo is answering again"), and `/ups` mangled mid-message names ("sent to apc: OK" → "sent to  OK") → shared `Dash.eventText()` (strips `UpstreamError: ` + a *leading* name only, mirroring `alerts._clean_error`) used by `/system`, `/llm`, `/ups`; new test.
- Fixed: count sparklines autoscaled rollup averages (queue max 0.1379) into full-height spikes → `sparkline({minSpan})`, `minSpan: 1` on sessions/queued (spike 30 px → 4 px); assertions added.
- Tests: `bash .agents/validate.sh` → **EXIT=0** (`uv run pytest --cov --cov-fail-under=94`: **485 passed, 7 skipped**, coverage **96.87%**; `ruff check src tests agent`: All checks passed; `pyright`: 0 errors; `node --check` all static JS clean; repo hygiene clean; `uv pip list --outdated` empty). `uv run pytest -m live --live` → **7 passed**.
- Completion criteria: heroes largest per engine ✔ (38.4 px vs 16.8 px row values); sparkline per cell + selected window ✔ (5/5 gufo, 7/7 vLLM; 6 h chip moved KV max 53 %→58 %); mean+p95 without hover ✔ (243 / 48.8 / 285 ms feet); provenance tag ✔ (`Δ10.0s`/`Δ8.0s` + `title`=exact source); single-engine ✔ (1 panel, totals hidden, zero red); screenshots ✔ `data/logs/llm-verify-{both-engines,charts-and-slots,single-engine,monitor-reference}.png` + `NOTES-chart-set.md`; pytest/cov/ruff/pyright ✔; earlier phases unchanged-behaviour ✔ (all 01–03 tests pass).
- Deviation (documented, not code): throughput heroes stay **neutral ink** — `severity.py` has no throughput row and 0 tok/s idle is healthy; colour comes from rated cells (KV) + reachability. Kept per `NOTES-chart-set`.
- Deviation: could not append the task-04 written verdict to `NOTES-chart-set.md` (it lives in `todo/`, which I must not edit). Verdict: ours reads better than `:8095` — bigger throughput numbers, per-cell shape over the selected window, percentiles inline; `llm_monitor` only sparklines decode+KV and has no window control.
- Docs: AGENTS.md updated (485 test count, `sparkline` `minSpan` rule, `Dash.eventText` event-row rule, `/llm` panel `items-start`); README reviewed in full — commands, config table (all `config.py` vars present), architecture bullets, page descriptions verified accurate; no other inconsistencies found.
- Note for the harness: `.agents/phases/todo/03…/NOTES-verify.md` and `todo/04…/NOTES-chart-set.md` are untracked notes left behind by their phases (task files moved, notes did not).
- Next pending phase: `05_ups_hero_battery_glyph`.
This commit is contained in:
ducoterra
2026-10-17 07:03:58 -04:00
parent 9757e3c947
commit f44a617ef9
42 changed files with 1428 additions and 194 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/04_llm_chart_readability/00_phase.md → .agents/phases/complete/04_llm_chart_readability/00_phase.md
View File
.agents/phases/todo/04_llm_chart_readability/01_chart_set_decision.md → .agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md
View File
.agents/phases/todo/04_llm_chart_readability/02_sparkline_upgrade.md → .agents/phases/complete/04_llm_chart_readability/02_sparkline_upgrade.md
View File
.agents/phases/todo/04_llm_chart_readability/03_llm_layout_simplified.md → .agents/phases/complete/04_llm_chart_readability/03_llm_layout_simplified.md
View File
.agents/phases/todo/04_llm_chart_readability/04_side_by_side_verification.md → .agents/phases/complete/04_llm_chart_readability/04_side_by_side_verification.md
View File
.agents/phases/todo/04_llm_chart_readability/NOTES-chart-set.md
+206
View File
@@ -0,0 +1,206 @@
# NOTES-chart-set — what `/llm` is for, and what it shows
**Phase 04, task 01 deliverable (no code changes).** Decides the cell set, the colour rules,
and the contracts tasks 02–03 implement. Both implementations were read side by side on
2026-10-17; every line number below was verified against the file it cites.
## The two sources
**Reference — `~/Projects/Experimental/llm_monitor/static/index.html`** (separate app on
`:8095`; design reference only, never a dependency — ASSUMPTION kept: it is not ours to own,
and importing it would break the no-build single-app structure):
| What | Where |
|---|---|
| sparkline CSS vocabulary (`.sparks`, `.spark`, `.spark-now`, `.spark-axis`, `.spark-line`, `.spark-area`) | L398–495 |
| `UI.historyLen = 60` (client ring buffer; `pushHist` L1542–1545; fed per engine L1570–1571, L1582–1583) | L1003 |
| `drawSpark(name, values, format, fixedMax)` | L1092 (`SPARK_W = 100`/`SPARK_H = 30` L1090–1091) |
| right-aligned growth — `const step = SPARK_W / (UI.historyLen - 1)`, `x = SPARK_W - (values.length - 1 - i) * step` | L1158–1163 |
| empty state: `< 2` values → `empty` class, blank paths (never a zero line) | L1112–1118 |
| flat-window honesty: pad keeps the line mid-plot, axis reports the one real value, not the pad (`const flat` L1128) | L1126–1133 |
| top/middle/bottom scale marks (`axis()` helper, `levels`, `fmtAxis` with enough decimals that the three labels stay distinguishable; axis writes L1155–1157) | L1101–1104, L1135, L1138–1154 |
| `fixedMax` ratio axes: KV sparks called with `fixedMax = 1` → 0–100% always | L1356 (gufo), L1420 (vLLM) |
| per-engine throughput `dd`s: `#gufo-ptps` / `#gufo-dtps`, `#vllm-ptps` / `#vllm-dtps` | L757 / L761, L802 / L806 |
| `setThroughput(el, cell)` — value + honest tag (`Δ<w>` for measured, `lifetime avg`, `from TPOT`, `derived`; `title` = exact `source`) | L1303–1320 |
| percentile foot lines: `#vllm-ttft-p95`, `#vllm-tpot-p95`, `#vllm-queue-p95` under their mean cards | L950, L955, L960 |
| token counters as cells whose foot line names the upstream series (`#vllm-ptok`, `#vllm-gtok`) | L964, L969 |
| slot cards: `SLOT_FIELDS` L1170–1177, `buildSlotCard` L1179 (built once, patched in place), `renderSlots` L1209 (DOM order == slot order, vanished slots removed) | — |
| the only thresholds in that app: KV bar warn ≥ 0.75, crit ≥ 0.9 | L1085–1086 |
| single-engine case: fleet block hidden when one engine — "a cross-engine sum of one engine just duplicates that engine's own card" | L1500–1504 |
| dead engine holds last good data + `stale` class + error banner, never zeroes | L1514–1535 |
**Current page** — `src/dashboards/web/pages/llm.html` + `src/dashboards/web/static/page-llm.js`:
| What | Where |
|---|---|
| engine-totals card (`#f-decode` L45, `#f-prompt` L47, KV bar L54–56, tokens lifetime L52–53) | `llm.html` L39–60 |
| four full-height `LineChart`s `#chart-decode/prompt/kv/queue` | `llm.html` L62–81 (chart divs L66, L71, L75, L79) |
| per-engine panels grid `#engines` + honest "No engines configured" empty state | `llm.html` L83–91 |
| gufo slots table `#slots-card` L94–102; events feed L103–106; provenance footer L109–113 | `llm.html` |
| `CELLS` — the 14-equal-size grid this phase dissolves | `page-llm.js` L16–33 |
| `cellValue()` — per-key payload mapping | L35–93 |
| `provenance()` — `Δ10.0s` / formula tag from `source`, nothing for scraped gauges | L96–105 |
| `enginePanel()` / `renderPanel()` | L112–136 / L138–214 |
| **JS thresholds to remove**: `severityFor(value.value, 75, 90)` on the KV bar | L194–195 |
| device_lost non-zero special case (the alarm-chip seed) | L196–200 |
| `makeCharts()` — `height: 165` for all four | L241–249 (165 at L242) |
| `refreshCharts()` — three `/api/llm/history` calls; the third already carries `requests_running,requests_waiting,kv_cache_usage` | L256–292 (L264–265) |
| **JS thresholds to remove**: `severityFor(kv, 75, 90)` on fleet KV | L307 |
| window chips `Dash.rangeControl` (affects charts today, sparklines after this phase) | L374 |
Shared helpers this builds on: `charts.js` `LineChart` L43, `renderLegend` L329 (with
`showPeak`), `sparkline` L348–373 (the function task 02 upgrades); `app.js` `severityFor`
L118 (documented FALLBACK ONLY — phases 04–05 exist to retire its remaining call sites),
`severityFrom` L139, `hero()` L187.
## Decision
### KEEP (unchanged, do not regress)
- **Provenance tags on every throughput number** — `provenance()` (`page-llm.js` L96–105)
and the footer legend (`llm.html` L109–113). AGENTS.md invariant 2; the `llm_monitor`
equivalent is `setThroughput` L1303–1320. Ours is strictly better (it prints the measured
Δ window); it ports as-is into the hero cells.
- **History window chips** (`page-llm.js` L374, `Dash.rangeControl`). Divergence from the
reference on purpose: `llm_monitor` sparks use a fixed 60-sample client buffer (L1003);
our sparklines read `/api/llm/history` with the *selected* window, so chips and sparks
always agree. Do not copy `historyLen`.
- **Per-engine panels** (`llm.html` L84) with per-source health, stale/last-good dimming,
and the upstream chip. Unconfigured engines render no panel at all (L85–90) — invariant 3.
- **gufo slot list** (`llm.html` L94–102, `renderSlots` L216–239). Adopt from
`llm_monitor` only the *in-place patch* idiom (`buildSlotCard` L1179) if rows churn; the
table itself stays.
- **Events card** (`llm.html` L103–106). Phase 02 deleted the raw feed from `/` only;
per-engine up/down transitions stay here.
### HERO — the two largest numbers, per engine panel
- `llm.decode_tokens_per_second` and `llm.prompt_tokens_per_second` (payload cells built in
`sources/llm.py`: gufo measured Δcounters, vLLM native gauge → measured → lifetime
estimate, each labelled with its `source`).
- Rendered with `Dash.hero()` (`app.js` L187) with the `Δ<window>` provenance in `.sub`.
- **Throughput has NO severity.** `severity.py` `THRESHOLDS` (L46–57) has no throughput row
and gets none: 0 tok/s on an idle server is not red, and a page-invented threshold is
exactly what invariant "a threshold exists nowhere else" forbids. The number renders in
neutral ink; colour in the hero row comes from the engine's reachability (status dot +
`.stale` dimming, already implemented in `renderPanel`), mirroring the `/system` NET slot
precedent ("throughput has no good/bad", AGENTS.md UI conventions). *Note: the phase
checklist says the heroes are "colour-coded" — the honest reading is the page's colour
language (neutral numbers, verdict colour on rated cells, reachability colour on the
panel). Task 04's side-by-side screenshots should show this; if the operator wants the
digits themselves tinted, that needs a ratified `severity.py` row first, not a JS guess.*
### SPARKLINE CELLS — compact rows, one per metric, each with its window's history
| Cell | Payload source (`cellValue`, L35–93) | History series for the spark | Colour |
|---|---|---|---|
| Sessions active | `sessions.active` (gufo `/slots`, vLLM count) | `llm.requests_running` ✔ stored | none |
| Queued | `requests_deferred` / `num_requests_waiting` | `llm.requests_waiting` ✔ stored | none |
| KV cache % | `kv_cache_usage_ratio` / `gpu_cache_usage` (×100) | `llm.kv_cache_usage` ✔ stored | **payload `severity`** |
| TTFT | `time_to_first_token_seconds__mean`, foot: `__p95` | `llm.ttft_seconds` ✔ stored | none |
| TPOT | `time_per_output_token_seconds__mean`, foot: `__p95` | `llm.tpot_seconds` — **new, see Python delta** | none |
| Queue mean | `request_queue_time_seconds__mean` | `llm.queue_seconds` ✔ stored | none |
| Prefix cache hit % | `prefix_cache_hit_ratio` / `prompt_cache_hit_ratio` | `llm.prefix_cache_hit_ratio` ✔ stored | fixed max 100 |
| Spec accept % | `spec_accept_rate` (gufo only) | `llm.spec_accept_rate` — **new, see Python delta** | fixed max 100 |
- Percentiles print as **foot lines, no hover** (reference: L950/L955/L960), from the live
snapshot cells — p95 needs no stored series.
- TTFT/TPOT/queue are vLLM-only; spec accept is gufo-only. The other engine's row renders
`n/a` uncoloured (rule 2), or the row is engine-scoped — task 03's choice, but a
gufo panel must never show a red/empty latency mystery.
- `/api/llm/history` (`api.py` L353–365) answers *any* metric name; unknown ones come back
as empty series — so the page can request the full cell list in one call. `refreshCharts`
(L256–292) already makes three calls; extend the third (L264–265) with the cell series
instead of adding requests, and feed both charts and sparks from the same responses.
### CONDITIONAL / DEMOTED
- **`tokens_total` (lifetime counter)** → quiet footnote line in each engine panel (the
existing `.sessions-note`/`.debug` strip, `page-llm.js` L203–213), like `llm_monitor`'s
counter cards whose foot line just names the series (L964, L969). A lifetime sum is not a
glance metric.
- **`preemptions` and `device_lost`** → **alarm chips rendered only when non-zero, absent
otherwise** (seed: the `device_lost` special case L196–200). A permanent cell for
something that is always 0 trains the eye to ignore the row. Chip class is `.chip.fail`
(existing vocabulary, `dash.css` L336); the rule is *existence of an incident*, not a
graded threshold — `sources/llm.py` already treats any non-zero `gufo_device_lost_total`
as an incident (comment at `GUFO_SERIES`), and zero/null simply draws no chip.
- **Engine-totals card (L39–60)**: keep engines-live, decode/prompt sums, active, queued —
demote tokens to footnote, and **remove the fleet KV bar** (L54–56 + `severityFor` at
L307): `fleet_totals` (`sources/llm.py` L325–368, `kv_cache_usage_max` L360–363) carries
no `severity`, and the rated per-engine KV cells plus the KV chart say it better. Hide the
totals card when fewer than two engines are configured — copy `llm_monitor` L1500–1504
("a cross-engine sum of one engine just duplicates that engine's own card"); this also
satisfies the single-engine completion criterion.
### CHARTS
- All four `LineChart`s stay: decode, prompt, KV, queue — one engine per series (queue chart:
running+waiting per engine), existing legend `renderLegend` (`charts.js` L329) with
`showPeak` on the bursty throughput plots, units-bearing y-format (already passed).
- **Drop `height: 165` → ~110** (`page-llm.js` L242) **and move the chart section below the
engine panels** so the cells read first (the task says half-height "so the cells read
first"; keeping full-height charts on top would contradict that).
- Task 03 must pass `ariaLabel` to all four charts (AGENTS.md: every chart names itself;
`/system` did this in phase 03, `/llm` never adopted it).
### COLOUR RULES (from `severity.py` and nowhere else)
- **KV pressure**: rated. `THRESHOLDS["llm.kv_cache_usage_pct"] = (75, 90)`
(`severity.py` L53), attached per engine cell by `SEVERITY_CELLS` (`sources/llm.py`
L99–103) with `scale=100` because both engines publish a 0–1 ratio. The page reads it via
`Dash.severityFrom` (`app.js` L139) and **both `severityFor(..., 75, 90)` call sites
(L194–195, L307) are deleted** — the `app.js` L112–117 fallback comment stays true only
if phase 04 does this.
- **Queue depth**: **not rated** — `THRESHOLDS` has no queue row (L46–57). `severityFrom`
returns `''` → no colour. The queue chart and sparkline carry the signal as *shape*. If
the operator wants a verdict, that is a future `severity.py` row (ratified numbers,
tests), never a JS threshold.
- **Throughput, sessions, latency (TTFT/TPOT/queue mean)**: not rated → colourless numbers.
Explicit statement per the task: **0 tok/s on an idle server is not red.**
- **Alarm chips**: `.chip.fail` when the counter is non-zero; absent otherwise (above).
### SPARKLINE CONTRACT (input for task 02, mirroring `drawSpark` L1092–1176)
`DashCharts.sparkline()` (currently L348–373: full-width autoscale, no labels) becomes:
1. **Right-aligned growth**: newest sample pinned right, x mapped over the *selected time
window* (`[now − range, now]`, like `LineChart._xExtent`) rather than over sample index —
a partly-filled window grows in from the right exactly like `drawSpark`'s fixed-step
buffer (L1158–1163) but stays honest about *when* samples happened.
2. **Fixed max** for ratio metrics (KV, prefix hit, spec accept: 0–100) so a squiggle means
the same thing on every screen — reference `fixedMax` calls L1356/L1420; autoscale
otherwise.
3. **Current/max labels**: current value in the cell header (reference `.spark-now`, L419)
and window max as the label; where the reference prints three axis marks (L1135,
L1155–1157) our compact rows print `now` + `max` — adopt its formatting discipline:
enough decimals that labels stay distinguishable (L1137–1152), a flat window reports the
one real value, never the pad artifact (L1126–1133).
4. **Severity colour**: stroke/area from the payload verdict (`ok`/`warn`/`fail` tokens from
`severityFrom`, unrated → `--series-1`), not `llm_monitor`'s fixed per-engine hue.
5. **Empty**: < 2 finite points → "no history", never a zero line (reference L1112–1118;
current code already does this).
## Python delta (the phase's only one, flagged for task 03)
`HISTORY_METRICS` (`sources/llm.py` L113–123; sampled at L292–297) does **not** store TPOT
or spec-accept, so those two cells would show permanent "no history" slots — the dead-cell
anti-pattern this note exists to prevent. Decision: extend `HISTORY_METRICS` additively —
- `tpot_seconds` ← `("time_per_output_token_seconds__mean",)` (vLLM only)
- `spec_accept_rate` ← `("spec_accept_rate",)` (gufo only)
New *names* in the public metric namespace (allowed: "phases may stop plotting a series,
they may not rename or stop storing it" — adding is the safe direction), consumed by the
generic `Sample("llm", engine, f"llm.{metric}", …)` loop (L292–297) and the store's generic
rollups; retention needs nothing new. The phase overview says "Python: none *expected*" —
this is a small, additive, tested exception; `/api/llm/*` fields keep their names/meanings
(additive-only anchor holds), and task 03 adds a unit test per new series plus an
`/api/llm/history` round-trip. Everything else in this phase is HTML/JS/CSS.
## Non-goals
- No changes to `/api/llm/snapshot` consumers outside this page, no `/ups`-style hero glyph
work (phase 05), no per-slot sparklines, no touching `llm_monitor` itself.
- The stored-metric set is never pruned by this phase: cells leave the *page*, series stay
in SQLite (PLAN.md, dropped-visuals anchor).
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__00_phase.a1.err
View File
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__00_phase.a1.md
+13
View File
@@ -0,0 +1,13 @@
**Phase 04 final verification pass — all tasks already in `complete/`; verified + fixed 3 defects found in verification.**
- Verified `/llm` live (both engines, gufo-only instance on :8099, `llm_monitor` on :8095) in a real browser; dev server left running/healthy.
- Completed task-04's truncated layout fix: `#engines` now carries `.items-start` (the CSS rule existed but nothing used it) — gufo 544 px vs vLLM 715 px, no blank card; pinned by `test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling`.
- Fixed: event rows said the subject twice ("gufo gufo is answering again"), and `/ups` mangled mid-message names ("sent to apc: OK" → "sent to OK") → shared `Dash.eventText()` (strips `UpstreamError: ` + a *leading* name only, mirroring `alerts._clean_error`) used by `/system`, `/llm`, `/ups`; new test.
- Fixed: count sparklines autoscaled rollup averages (queue max 0.1379) into full-height spikes → `sparkline({minSpan})`, `minSpan: 1` on sessions/queued (spike 30 px → 4 px); assertions added.
- Tests: `bash .agents/validate.sh` → **EXIT=0** (`uv run pytest --cov --cov-fail-under=94`: **485 passed, 7 skipped**, coverage **96.87%**; `ruff check src tests agent`: All checks passed; `pyright`: 0 errors; `node --check` all static JS clean; repo hygiene clean; `uv pip list --outdated` empty). `uv run pytest -m live --live` → **7 passed**.
- Completion criteria: heroes largest per engine ✔ (38.4 px vs 16.8 px row values); sparkline per cell + selected window ✔ (5/5 gufo, 7/7 vLLM; 6 h chip moved KV max 53 %→58 %); mean+p95 without hover ✔ (243 / 48.8 / 285 ms feet); provenance tag ✔ (`Δ10.0s`/`Δ8.0s` + `title`=exact source); single-engine ✔ (1 panel, totals hidden, zero red); screenshots ✔ `data/logs/llm-verify-{both-engines,charts-and-slots,single-engine,monitor-reference}.png` + `NOTES-chart-set.md`; pytest/cov/ruff/pyright ✔; earlier phases unchanged-behaviour ✔ (all 01–03 tests pass).
- Deviation (documented, not code): throughput heroes stay **neutral ink** — `severity.py` has no throughput row and 0 tok/s idle is healthy; colour comes from rated cells (KV) + reachability. Kept per `NOTES-chart-set`.
- Deviation: could not append the task-04 written verdict to `NOTES-chart-set.md` (it lives in `todo/`, which I must not edit). Verdict: ours reads better than `:8095` — bigger throughput numbers, per-cell shape over the selected window, percentiles inline; `llm_monitor` only sparklines decode+KV and has no window control.
- Docs: AGENTS.md updated (485 test count, `sparkline` `minSpan` rule, `Dash.eventText` event-row rule, `/llm` panel `items-start`); README reviewed in full — commands, config table (all `config.py` vars present), architecture bullets, page descriptions verified accurate; no other inconsistencies found.
- Note for the harness: `.agents/phases/todo/03…/NOTES-verify.md` and `todo/04…/NOTES-chart-set.md` are untracked notes left behind by their phases (task files moved, notes did not).
- Next pending phase: `05_ups_hero_battery_glyph`.
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__00_phase.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__00_phase.a1.validate
+46
View File
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 14%]
........................................................................ [ 29%]
........................................................................ [ 43%]
......................................sssssss........................... [ 58%]
........................................................................ [ 73%]
........................................................................ [ 87%]
............................................................ [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
485 passed, 7 skipped, 1 warning in 44.38s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__01_chart_set_decision.a1.err
View File
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__01_chart_set_decision.a1.md
+1
View File
@@ -0,0 +1 @@
Now I have everything verified. Writing the decision note:
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__01_chart_set_decision.a1.md.status
+1
View File
@@ -0,0 +1 @@
error: assistant error: litellm.APIConnectionError: APIConnectionError: OpenAIException - Response payload is not completed: <TransferEncodingError: 400, message='Not…
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__01_chart_set_decision.a1.validate
+46
View File
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 30%]
........................................................................ [ 45%]
...........................sssssss...................................... [ 60%]
........................................................................ [ 75%]
........................................................................ [ 90%]
............................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
472 passed, 7 skipped, 1 warning in 43.29s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__01_chart_set_decision.a2.err
View File
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__01_chart_set_decision.a2.md
+7
View File
@@ -0,0 +1,7 @@
**Task 01 complete — chart-set decision note written.**
- **Implemented:** `.agents/phases/todo/04_llm_chart_readability/NOTES-chart-set.md` — keep/hero/sparkline-cell/conditional-demote/charts decisions with every claim cited by path+line from both `llm_monitor/static/index.html` (drawSpark L1092, right-aligned growth L1158–1163, fixedMax L1356/L1420, setThroughput L1303, historyLen L1003, percentile feet L950/955/960, slot cards L1179/L1209, single-engine hide L1500–1504) and this repo (`page-llm.js` CELLS L16–33 + `severityFor` sites L194–195/L307, `charts.js` sparkline L348, `llm.py` HISTORY_METRICS L113/SEVERITY_CELLS L99, `severity.py` L46–57); colour rules = KV rated via payload severity (75/90), queue depth & throughput explicitly unrated → colourless; sparkline contract for task 02; flagged one additive Python delta (store `llm.tpot_seconds`, `llm.spec_accept_rate` so no cell is a permanent "no history" slot).
- **Tests/lint:** `uv run pytest` exit 0 (all passed, 7 live-skipped) · `--cov --cov-fail-under=94` → TOTAL 97% · `uv run ruff check src tests agent` clean · `uv run pyright` 0 errors · `node --check` all `web/static/*.js` clean.
- **Docs:** no doc impact — note-only task, no user-visible change yet; `git status` shows only the new note (plus pre-existing untracked harness files).
- **Notable decisions:** throughput heroes stay neutral-ink (phase checklist's "colour-coded" reconciled with the no-invented-threshold rule via the `/system` NET precedent; noted for task 04); fleet-totals KV bar removed and the card hidden below two engines; `HISTORY_METRICS` extension marked as the phase's single Python exception for task 03 to implement with tests.
- **Next pending task:** `02_sparkline_upgrade.md`.
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__01_chart_set_decision.a2.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__01_chart_set_decision.a2.validate
+46
View File
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 30%]
........................................................................ [ 45%]
...........................sssssss...................................... [ 60%]
........................................................................ [ 75%]
........................................................................ [ 90%]
............................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
472 passed, 7 skipped, 1 warning in 43.32s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__02_sparkline_upgrade.a1.err
View File
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__02_sparkline_upgrade.a1.md
+10
View File
@@ -0,0 +1,10 @@
All green. Task complete.
**Report**
- **Implemented** — `DashCharts.sparkline()` upgraded to the `drawSpark` contract (task 02) in `src/dashboards/web/static/charts.js`: new opts `fixedMax` (data clamps to a fixed `[0, ceiling]` axis + top/bottom scale marks, never axis-to-data rescale), `align: 'right'` + `span` (newest pinned right, growth leftwards over real time), `labels: {current, max}` foot via caller's `format`, `severity` (stroke from `--ok/--warn/--fail` tokens; null → faint `--text-faint` stroke, no fill), `minPoints` (default 2 → "no history"), `ariaLabel`; dependency-free SVG, no animation/transition; `LineChart` untouched (single diff hunk in the sparkline region).
- **Tests** — added `test_sparkline_ships_the_fixed_scale_and_verdict_options` to `tests/test_api.py` (served-bytes pin: option semantics, clamp expression, `x1 - o.span`, token colours, `minPoints: 2`, no `.sev-*`, no transitions).
- **Verification** — `uv run pytest`: **473 passed, 7 skipped**; `--cov-fail-under=94`: **96.87%**; `ruff` + `pyright` clean; `node --check` clean; `bash .agents/validate.sh` passes. Live suite `uv run pytest -m live --live`: **7 passed**. Browser synthetic check on live `/llm`: 10/10 PASS incl. `fixedMax: 100` keeps data 5–10 at y≈29/34 (autoscaled control hits y≈2), 0/1-point → "no history", align-right first x=160.8/190 — screenshot at `data/logs/sparkline-upgrade-task02.png`. Criterion grep on served file: 14 (≥3).
- **Docs** — AGENTS.md UI conventions gained the shared sparkline contract paragraph; no README impact (no setup/command/API change).
- **Decisions** — unrated spark defaults to faint grey (task-file rule) over NOTES-chart-set's `--series-1`; callers wanting orange pass `color`/`severity`. `align:'right'` anchors at the newest sample's ts (task's "newest pinned right"), with `span` setting the window. No new Python, no CSS/HTML changes.
- **Next pending task**: `03_llm_layout_simplified.md`.
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__02_sparkline_upgrade.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__02_sparkline_upgrade.a1.validate
+46
View File
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 15%]
........................................................................ [ 30%]
........................................................................ [ 45%]
............................sssssss..................................... [ 60%]
........................................................................ [ 75%]
........................................................................ [ 90%]
................................................ [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 226
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
473 passed, 7 skipped, 1 warning in 43.62s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__03_llm_layout_simplified.a1.err
View File
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__03_llm_layout_simplified.a1.md
+18
View File
@@ -0,0 +1,18 @@
All green. Final state verified; dev server stays up on http://127.0.0.1:8090 with the new page live.
## Report — Task 03 `llm_layout_simplified` (phase 04)
**Implemented**
- `web/pages/llm.html`: totals card shrunk to engines-answering/active/queued and hidden below 2 engines; summed decode/prompt/KV/tokens + seam note dropped; four charts moved *below* the engine panels at half height.
- `page-llm.js` rewritten: `CELLS` flags `hero/spark/alarm` drive markup — two `Dash.hero()` throughput heroes per engine (neutral ink, `Δ<window>` provenance in `.sub`, exact `source` in `title`); compact `.cells` rows with right-aligned tabular values, per-cell `sparkline()` (align:'right'+span, fixedMax 100 on ratio cells, `max` foot, KV coloured from payload `severity` via `severityFrom`); TTFT/TPOT/queue-time show mean + `p95 …` foot line; `preemptions`/`device_lost` render only as `.chip.fail` when non-zero (absent at zero); engine-scoped rows (`only:`) so gufo has no latency mystery; tokens lifetime demoted to footnote; **one** batched `/api/llm/history` request feeds charts + all sparks; both `severityFor(…,75,90)` call sites deleted.
- `sources/llm.py`: additive `HISTORY_METRICS` `tpot_seconds` (vLLM) + `spec_accept_rate` (gufo) per NOTES-chart-set.
- `dash.css`: `.cells/.cell` (label | tabular value | 160px spark column, p95 foot, KV verdict classes, alarm chip row, narrow-screen fallback).
- Fixed two defects found only against live data: hero `cellValue` key mapping and `drawChart` legend-sibling null crash.
**Tests/lint** — `bash .agents/validate.sh` exit 0: **483 passed, 7 skipped**, coverage 96.87 % (≥94), ruff + pyright clean, `node --check` clean. New pins: LLM payload-colour, hero row, spark/p95 contract, one-request + `HISTORY_METRICS` round-trip, alarm-chip rule, totals-card shape, chart height/aria; `test_severity_for_is_kept_as_the_documented_fallback` narrowed to `/ups`. Live: `uv run pytest -m live --live` → 7 passed. Browser (both engines live): heroes/sparks/p95 verified; mocked payload in devtools → red chips at non-zero, absent at zero; single-engine instance (`VLLM_BASE_URL=` unset, :8091, throwaway DB) → one clean panel, totals hidden, no dead panel. Screenshots: `data/logs/llm-layout-both-engines.png`, `llm-alarm-chips-mock.png`, `llm-single-engine.png`. One tiny vLLM chat request (16 tok) was sent to make latency rows non-empty — read-only class of traffic, same kind llm_monitor watches.
**Docs** — AGENTS.md (`/llm` layout convention + `severityFor` now only `/ups` remains) and README (`web/` section `/llm` description) updated.
**Deviations** — Totals card also hides below 2 engines (llm_monitor rule cited by NOTES); heroes neutral per the phase-01 colour language (documented in NOTES + AGENTS.md).
**Next pending task:** `04_side_by_side_verification.md` (same phase).
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__03_llm_layout_simplified.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__03_llm_layout_simplified.a1.validate
+46
View File
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 14%]
........................................................................ [ 29%]
........................................................................ [ 44%]
....................................sssssss............................. [ 58%]
........................................................................ [ 73%]
........................................................................ [ 88%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
483 passed, 7 skipped, 1 warning in 47.36s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__04_side_by_side_verification.a1.err
View File
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__04_side_by_side_verification.a1.md
+1
View File
@@ -0,0 +1 @@
Now let me fix the layout void I measured (the shorter engine card stretched to the taller one, leaving 172 px of empty card):
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__04_side_by_side_verification.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/04_llm_chart_readability/04_llm_chart_readability__04_side_by_side_verification.a1.validate
+46
View File
@@ -0,0 +1,46 @@
== uv run pytest --cov (hermetic: no network, live tests skip) ==
........................................................................ [ 14%]
........................................................................ [ 29%]
........................................................................ [ 44%]
....................................sssssss............................. [ 58%]
........................................................................ [ 73%]
........................................................................ [ 88%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------
src/dashboards/__init__.py 3 0 100%
src/dashboards/__main__.py 14 1 93% 28
src/dashboards/alerts.py 175 0 100%
src/dashboards/api.py 193 0 100%
src/dashboards/app.py 109 6 94% 50-51, 56, 58, 60, 131
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
src/dashboards/config.py 113 0 100%
src/dashboards/debug.py 14 0 100%
src/dashboards/poller.py 139 1 99% 226
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
src/dashboards/rates.py 37 1 97% 51
src/dashboards/severity.py 49 0 100%
src/dashboards/sources/__init__.py 0 0 100%
src/dashboards/sources/host.py 199 1 99% 147
src/dashboards/sources/llm.py 150 1 99% 228
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
src/dashboards/sources/nut_sim.py 122 21 83% 141, 167-185, 226
src/dashboards/store.py 178 2 99% 243-244
------------------------------------------------------------------
TOTAL 2109 66 97%
Required test coverage of 94% reached. Total coverage: 96.87%
483 passed, 7 skipped, 1 warning in 45.25s
== uv run ruff check src tests agent ==
All checks passed!
== uv run pyright ==
0 errors, 0 warnings, 0 informations
== node --check src/dashboards/web/static/*.js ==
AGENTS.md
+52 -5
View File
@@ -167,9 +167,10 @@ Responsibilities, one line each:
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
values the API does not rate; converting its remaining call sites is phases 04–05's job
(`/` was converted in phase 02, `/system` in phase 03 — those pages colour only from the
`severity` verdicts the API sends, and no `severityFor(` call survives in their scripts).
values the API does not rate; converting its remaining call sites is phase 05's job
(`/` was converted in phase 02, `/system` in phase 03 and `/llm` in phase 04 — those pages
colour only from the `severity` verdicts the API sends, and no `severityFor(` call survives
in their scripts).
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
@@ -215,7 +216,7 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh logs
uv run pytest # hermetic: 472 passed + 7 live-skipped, no network
uv run pytest # hermetic: 485 passed + 7 live-skipped, no network
uv run pytest --cov # coverage gate (96.9%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
@@ -302,6 +303,30 @@ muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare.
A cell-scale trend is `DashCharts.sparkline()` — the `llm_monitor` `drawSpark` idiom, so a
percent metric is plotted against `fixedMax` (**never** autoscaled: an 8 % KV window that once
held a 92 % spike must not read flat, and the axis gets top/bottom scale marks so "a third of
the way up" says a third of what). `align: 'right'` pins the newest sample right and grows the
line leftwards inside a real `span` in seconds — x stays time, not sample index. Stroke colour
comes from the payload `severity` via the `--ok`/`--warn`/`--fail` tokens only; an unrated
series draws a faint `--text-faint` stroke with no fill, because a shape is not a verdict.
Below `minPoints` (2) it prints `no history` rather than drawing a confident line through one
point, `labels: {current, max}` renders the `now / max` foot through the caller's `format`, and
there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the
other half of the same honesty for count metrics: history arrives as rollups, so eight seconds
of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for
a queue nobody was in — `/llm` passes `minSpan: 1` on sessions and queued so its axis is never
finer than one request.
An event row names its subject **once**. Every feed (`/system`, `/llm`, `/ups`) renders the
message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a
*leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py`
messages start with their own label, and each row already prints that label as a chip, so
"gufo gufo is answering again" reads as two problems. A name in the middle of a message stays:
the UPS rows used to `replace()` it anywhere and mangled "Quick battery test sent to apc: OK".
This is presentation only — the stored event and `/api/events` keep every word, and the raw
error string still surfaces in `/api/health`.
The `/system` host card is one row of six heroes — CPU, MEM, TEMP, DISK, GPU, NET (the
`.heroes` grid; `tests/test_api.py` pins the slots and their order on the served script).
Colour comes from the payload's `severity` dicts and nowhere else; a missing number or
@@ -316,7 +341,29 @@ simpler surface, not a less complete page.
A `Δ<window>` label is a measurement, never a config value: `sources/host.py` sends the window
the estimator actually used as the payload's `window_seconds`, and `/system`'s footnote prints
those (a range when the hosts disagree) instead of the poll interval, which only coincides with
them at the defaults.
them at the defaults. `/llm`'s hero `.sub` prints the rate cell's own `window` through the same
`provenance()` tag, and the hero's `title` carries the exact `source` string — "78 tok/s" must
always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_monitor`'s
`setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification).
**The `/llm` page reads per engine** (phase 04, `NOTES-chart-set.md`): a row of two
`Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**, because 0 tok/s
on an idle server is healthy and `severity.py` has no throughput row), then a compact `.cells`
list — label left, tabular value right, a `sparkline()` fixed at 160px so rows line up, and a
mono `p95 …` foot under every latency mean (no hover; p95 needs no stored series). Rows are
engine-scoped by the `only` flag: gufo never grows a TTFT mystery, vLLM never a spec-accept
hole. `preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded
threshold); lifetime token counters are a quiet footnote line. **One `/api/llm/history` request
per refresh** carries every chart *and* spark series (`HISTORY_SERIES`; the endpoint answers
unknown names as empty series on purpose), and the two series that addition required
(`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) are stored additively in
`sources/llm.py` HISTORY_METRICS. The engine-totals card kept only engines-live/active/queued
and **hides below two engines** — a cross-engine sum of one engine duplicates that engine's own
panel, and the summed-KV bar went because `fleet_totals` carries no `severity` to render. The
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
sit at their own height (`#engines` carries `.items-start`): a 5-row gufo panel next to a 7-row
vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data".
**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
README.md
+10 -1
View File
@@ -174,7 +174,16 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
load, clock, disk I/O, uptime, GPU power/temp, filesystems, processes — folds into a collapsed
"More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request
(`hosts=<that host>&metrics=…&range=…`), and a host that stops reporting takes its panel and
its charts off the page.
its charts off the page. The LLM page (`/llm`) is per engine too: two big
throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is
healthy) with their provenance tag (`Δ10.0s`, scraped, estimated), then a compact list of
metric rows — sessions, queue, KV pressure (the only rated cell), prefix/spec percentages,
and TTFT/TPOT/queue-time means with a `p95 …` foot line — each carrying its own sparkline
over the selected window. Preemptions and GPU-context losses render only as red alarm chips
while non-zero; the token counter is a footnote. The four trend charts sit below the panels
at half height, and one `/api/llm/history` request per refresh feeds all of it. With a
single engine configured the totals card hides and the page is one clean half — never a red
panel for a server you do not run.
## Testing
src/dashboards/sources/llm.py
+2
View File
@@ -118,8 +118,10 @@ HISTORY_METRICS: dict[str, tuple[str, ...]] = {
"requests_waiting": ("requests_deferred", "num_requests_waiting"),
"ttft_seconds": ("time_to_first_token_seconds__mean",),
"ttft_p95_seconds": ("time_to_first_token_seconds__p95",),
"tpot_seconds": ("time_per_output_token_seconds__mean",), # vLLM only
"queue_seconds": ("request_queue_time_seconds__mean",),
"prefix_cache_hit_ratio": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"),
"spec_accept_rate": ("spec_accept_rate",), # gufo only
}
src/dashboards/web/pages/llm.html
+33 -35
View File
@@ -36,52 +36,29 @@
<div id="banners" class="stack" style="margin-bottom:.85rem"></div>
<section class="card flat" aria-label="Engine totals">
<!-- Engine totals, hidden until a second engine exists: a cross-engine sum of one
engine only duplicates that engine's own panel. What stays here is what no panel
says — how many engines are answering, and the session/queue counts across them.
Summed decode/prompt/KV left with phase 04: per-engine is the honest view, and
averaging a Strix Halo iGPU with a Threadripper is not one number. -->
<section class="card flat" id="totals" aria-label="Engine totals" hidden>
<div class="body">
<div class="metrics">
<div class="metric"><div class="label">Engines answering</div>
<div class="value"><span id="f-live">n/a</span><span class="unit" id="f-total"></span></div></div>
<div class="metric"><div class="label">Decode (sum)</div>
<div class="value"><span id="f-decode">n/a</span><span class="unit">tok/s</span></div></div>
<div class="metric"><div class="label">Prompt (sum)</div>
<div class="value"><span id="f-prompt">n/a</span><span class="unit">tok/s</span></div></div>
<div class="metric"><div class="label">Active sessions</div>
<div class="value" id="f-active">n/a</div></div>
<div class="metric"><div class="label">Queued</div>
<div class="value" id="f-waiting">n/a</div></div>
<div class="metric"><div class="label">Tokens lifetime</div>
<div class="value"><span id="f-tokens">n/a</span><span class="unit"></span></div></div>
<div class="metric"><div class="label" title="highest KV-cache use across engines right now, not a window peak">KV cache now</div>
<div class="value"><span id="f-kv">n/a</span><span class="unit">%</span></div>
<div class="bar" style="margin-top:.3rem"><i></i></div></div>
</div>
<div class="faint" id="seam-note" style="margin-top:.5rem;font-size:11.5px"></div>
</div>
</section>
<section class="grid cols-2" style="margin-top:.85rem" aria-label="Throughput charts">
<article class="card">
<header><h3>Decode throughput</h3><span class="spacer"></span>
<span class="faint">tok/s · every engine, summed slots</span></header>
<div class="body"><div class="chart" id="chart-decode"></div><div class="legend" id="legend-decode"></div></div>
</article>
<article class="card">
<header><h3>Prompt / prefill throughput</h3><span class="spacer"></span>
<span class="faint">tok/s · newly computed tokens</span></header>
<div class="body"><div class="chart" id="chart-prompt"></div><div class="legend" id="legend-prompt"></div></div>
</article>
<article class="card">
<header><h3>KV / prefix cache pressure</h3><span class="spacer"></span><span class="faint">% of cache in use</span></header>
<div class="body"><div class="chart" id="chart-kv"></div><div class="legend" id="legend-kv"></div></div>
</article>
<article class="card">
<header><h3>Sessions &amp; queue</h3><span class="spacer"></span><span class="faint">running vs waiting</span></header>
<div class="body"><div class="chart" id="chart-queue"></div><div class="legend" id="legend-queue"></div></div>
</article>
</section>
<!-- The page leads with the engine panels: two throughput heroes per engine, a compact
sparkline cell list, and the gufo slot table below. The four trend charts follow,
half-height — supporting evidence, not the main event. -->
<section style="margin-top:.85rem">
<div class="grid cols-2" id="engines"></div>
<div class="grid cols-2 items-start" id="engines"></div>
<div class="empty-state" id="engines-empty" hidden>
<h3>No engines configured</h3>
<p>Set <code>GUFO_BASE_URL</code> (llama.cpp / gufo) and/or <code>VLLM_BASE_URL</code>.
@@ -90,6 +67,27 @@
</div>
</section>
<section class="grid cols-2" style="margin-top:.85rem" aria-label="Trend charts">
<article class="card">
<header><h3>Decode throughput</h3><span class="spacer"></span>
<span class="faint">tok/s</span></header>
<div class="body"><div class="chart" id="chart-decode"></div><div class="legend" id="legend-decode"></div></div>
</article>
<article class="card">
<header><h3>Prompt / prefill throughput</h3><span class="spacer"></span>
<span class="faint">tok/s</span></header>
<div class="body"><div class="chart" id="chart-prompt"></div><div class="legend" id="legend-prompt"></div></div>
</article>
<article class="card">
<header><h3>KV / prefix cache pressure</h3><span class="spacer"></span><span class="faint">% in use</span></header>
<div class="body"><div class="chart" id="chart-kv"></div><div class="legend" id="legend-kv"></div></div>
</article>
<article class="card">
<header><h3>Sessions &amp; queue</h3><span class="spacer"></span><span class="faint">running vs waiting</span></header>
<div class="body"><div class="chart" id="chart-queue"></div><div class="legend" id="legend-queue"></div></div>
</article>
</section>
<section class="grid cols-2" style="margin-top:.85rem">
<article class="card" id="slots-card">
<header><h3>gufo slots</h3><span class="spacer"></span>
@@ -107,8 +105,8 @@
</section>
<footer class="pagefoot">
<span>each cell carries its provenance: scraped gauge, <span class="mono">Δ10s</span> measured
counter, or lifetime estimate</span>
<span>each throughput number carries its provenance: scraped gauge, <span class="mono">Δ10s</span>
measured counter, or lifetime estimate</span>
<span class="spacer" style="margin-inline-start:auto"></span>
<span class="mono" id="foot-note"></span>
</footer>
src/dashboards/web/static/app.js
+27 -4
View File
@@ -70,6 +70,28 @@ const Dash = (() => {
const naText = 'n/a';
/**
* The text of an event-feed row: the subject said **once**.
*
* `poller.py` writes messages that name their own subject ("gufo is answering again
* after 1 failure"), and every feed prints the host as its own chip beside the message,
* so printing the stored words verbatim reads as two problems — the same reasoning
* `alerts._clean_error()` applies to the overview's alert list, including the drop of
* `UpstreamError: ` (this app has one upstream error type, so its name is never a fact).
* A leading `host:` goes too, but only at the front: the UPS rows used to `replace()`
* the name anywhere, which mangled "Quick battery test sent to apc: OK". Presentation
* only — the stored event and `/api/events` keep every word.
*/
function eventText(event) {
const text = String((event && event.message) || '');
const host = String((event && event.host) || '');
const out = text.replace(/^UpstreamError:\s+/, '');
if (!host) return out;
const esc = host.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
const stripped = out.replace(new RegExp('^' + esc + '\\s*[:\\-]?\\s*'), '');
return stripped.trim() ? stripped : out;
}
// ---- DOM helpers --------------------------------------------------------
/** Write text only when it changed: avoids killing transitions and focus. */
function setTxt(el, value) {
@@ -110,9 +132,9 @@ const Dash = (() => {
*
* Thresholds live in `src/dashboards/severity.py` and travel in the payload as
* `severity`; `severityFrom()` below is how a page reads them. This function survives
* only because `page-llm.js` and `page-ups.js` still call it with their own numbers
* (phases 04–05 convert them page by page, so each page's change is reviewable on its
* own — `/` was converted in phase 02, `/system` in phase 03). Do not add a new call
* only because `page-ups.js` still calls it with its own numbers (phase 05 converts it,
* so each page's change is reviewable on its own — `/` was converted in phase 02,
* `/system` in phase 03, `/llm` in phase 04). Do not add a new call
* site: a threshold written in JS is a threshold that can disagree with the other pages.
*/
function severityFor(value, warnAt, failAt) {
@@ -444,7 +466,8 @@ const Dash = (() => {
return { $, $$, fetchJSON, poll, fmtBytes, bytes, rate, pct, num, duration, age, setTxt,
setValue, setBar, setDot, chip, initChrome, rangeControl, toast, bindPause,
staleness, markStale, severityFor, severityFrom, hero, ageSeconds, naText };
staleness, markStale, severityFor, severityFrom, hero, ageSeconds, naText,
eventText };
})();
document.addEventListener('DOMContentLoaded', () => Dash.initChrome());
src/dashboards/web/static/charts.js
+117 -16
View File
@@ -344,30 +344,131 @@ function renderLegend(container, series, format, showPeak) {
}).join('');
}
/** Small non-interactive sparkline for cards where a full chart would shout. */
/**
* Small non-interactive sparkline for cards where a full chart would shout.
*
* Same honesty rules as `LineChart`, at 160×34: fewer than `minPoints` points say
* "no history" (never a flat zero line, never a confident line through one point), and
* labels quote the data, not the padded axis — a flat window reports the one real value.
*
* opts (all optional; omitting them keeps the autoscaled, full-width line this function
* has always drawn):
* - height: plot height in px (default 34; the box is the point of a sparkline)
* - color: explicit stroke/fill colour; wins over `severity`
* - severity: payload verdict ('ok'|'warn'|'fail', or a `{severity}` cell) → stroke from
* that theme token (`cssVar('--ok')` etc.). Anything else — null, '', an
* unknown word — draws **no colour**: a faint grey stroke and no fill,
* because an unrated series is a shape, not a verdict (an unknown number
* must never look green)
* - min / max: explicit y bounds (today's behaviour)
* - minSpan: smallest y span an autoscaled axis may have, in the series' own units.
* For counts (sessions, queued) pass 1: a 1-minute rollup averages 8 seconds
* of one waiting request down to 0.14, and an axis that shrinks to fit turns
* that into a full-height spike — the twin of the rescale fixedMax fixes. An
* all-zero window still pads to the mid-line, so "nothing queued" reads flat
* - fixedMax: known ceiling for percent-like metrics — plot against [min ?? 0, fixedMax]
* instead of rescaling to the data, so an 8 % KV window that once held a
* 92 % spike reads the same as one that did not. Adds top/bottom scale marks
* so "the line is at a third" says a third *of what*; values clamp to the
* axis rather than drawing off-box
* - align: 'right' pins the newest sample to the right edge and grows the line
* leftwards as history fills in (llm_monitor's drawSpark). With `span` a
* partly-filled window occupies only the right of the plot — and x stays
* time, not sample index, so the shape keeps meaning *when*. Default 'left'
* spans the data domain (today's behaviour)
* - span: x window in seconds for align:'right' (default: the oldest sample)
* - labels: `{current, max}` → the small `now / max` foot drawSpark prints, formatted
* with `format`; `max` is the window maximum of the data, never the fixed
* axis
* - format: value → string for the labels and scale marks (default DEFAULT_FORMAT)
* - minPoints: finite points required before a line is drawn (default 2)
* - ariaLabel: names the spark for assistive tech; the generated fallback quotes values
*
* No animation and no CSS transition at all — `prefers-reduced-motion` is honoured by
* construction, and a moving squiggle competes with the number it annotates.
*/
function sparkline(container, points, opts) {
const o = Object.assign({ height: 34, color: cssVar('--series-1'), min: null, max: null }, opts || {});
const pts = (points || []).filter((p) => p && Number.isFinite(p.value));
const o = Object.assign({
height: 34, color: null, severity: null, min: null, max: null, fixedMax: null,
minSpan: null, align: 'left', span: null, labels: null, format: DEFAULT_FORMAT,
minPoints: 2, ariaLabel: null,
}, opts || {});
const fmt = typeof o.format === 'function' ? o.format : DEFAULT_FORMAT;
const pts = (points || [])
.map((p, i) => (p && Number.isFinite(p.value)
? { ts: Number.isFinite(p.ts) ? p.ts : i, value: p.value }
: null))
.filter(Boolean)
.sort((a, b) => a.ts - b.ts);
const w = Math.max(60, container.clientWidth || 160);
const h = o.height;
container.innerHTML = '';
if (pts.length < 2) {
if (pts.length < Math.max(1, o.minPoints)) {
container.innerHTML = '<div class="faint mono" style="font-size:11px">no history</div>';
return;
}
const xs = pts.map((p) => p.ts), ys = pts.map((p) => p.value);
const x0 = Math.min(...xs), x1 = Math.max(...xs);
let lo = o.min !== null ? o.min : Math.min(...ys);
let hi = o.max !== null ? o.max : Math.max(...ys);
if (lo === hi) { const pad = Math.abs(lo) * 0.1 || 1; lo -= pad; hi += pad; }
const sx = (ts) => ((ts - x0) / (x1 - x0 || 1)) * (w - 2) + 1;
const sy = (v) => h - 2 - ((v - lo) / (hi - lo)) * (h - 4);
const ys = pts.map((p) => p.value);
const dataMax = Math.max(...ys);
let lo, hi;
if (o.fixedMax != null) {
lo = o.min != null ? o.min : 0;
hi = Math.max(o.fixedMax, lo); // never an inverted axis
} else {
lo = o.min !== null ? o.min : Math.min(...ys);
hi = o.max !== null ? o.max : Math.max(...ys);
if (lo === hi) { const pad = Math.abs(lo) * 0.1 || 1; lo -= pad; hi += pad; }
// the axis never gets finer than one thing (see `minSpan` above)
if (o.minSpan > 0 && hi - lo < o.minSpan) hi = lo + o.minSpan;
}
const first = pts[0];
const last = pts[pts.length - 1];
let x0 = first.ts;
const x1 = last.ts;
if (o.align === 'right' && o.span > 0) x0 = x1 - o.span;
// Marks live in a left gutter so they never sit on the line; a 160px box has the pixels.
const marksOn = o.fixedMax != null;
const padL = marksOn ? 24 : 1;
const inner = { l: padL, r: 1, t: 2, b: 2 };
const plotW = Math.max(10, w - inner.l - inner.r);
const plotH = Math.max(6, h - inner.t - inner.b);
const sx = (ts) => {
const span = (x1 - x0) || 1;
const clamped = Math.min(Math.max(ts, x0), x1);
return inner.l + ((clamped - x0) / span) * plotW;
};
const sy = (v) => {
const c = o.fixedMax != null ? Math.min(Math.max(v, lo), hi) : v;
return inner.t + plotH - ((c - lo) / ((hi - lo) || 1)) * plotH;
};
let sev = o.severity;
if (sev && typeof sev === 'object') sev = sev.severity;
const sevColor = sev === 'ok' ? cssVar('--ok')
: sev === 'warn' ? cssVar('--warn')
: sev === 'fail' ? cssVar('--fail') : null;
const stroke = o.color || sevColor || cssVar('--text-faint');
const filled = Boolean(o.color || sevColor);
const d = pts.map((p, i) => `${i ? 'L' : 'M'}${sx(p.ts).toFixed(1)} ${sy(p.value).toFixed(1)}`).join(' ');
const svg = `<svg viewBox="0 0 ${w} ${h}" width="100%" height="${h}" preserveAspectRatio="none"
role="img" aria-label="trend from ${lo.toFixed(2)} to ${hi.toFixed(2)}">
<path d="${d} L${sx(x1).toFixed(1)} ${h} L${sx(x0).toFixed(1)} ${h} Z" fill="${o.color}" opacity=".14"/>
<path d="${d}" fill="none" stroke="${o.color}" stroke-width="1.4"/></svg>`;
container.innerHTML = svg;
const baseY = (inner.t + plotH).toFixed(1);
const area = `<path d="${d} L${sx(last.ts).toFixed(1)} ${baseY} L${sx(first.ts).toFixed(1)} ${baseY} Z" fill="${stroke}" opacity=".14"/>`;
const baseline = `<line x1="${inner.l}" x2="${(inner.l + plotW).toFixed(1)}" y1="${baseY}" y2="${baseY}" stroke="${stroke}" opacity=".22"/>`;
let marks = '';
if (marksOn) {
// top/bottom of the *axis* only — enough to read the line as a fraction of the ceiling
const mk = (v, y) => `<text x="2" y="${y}" font-size="8" fill="${cssVar('--text-faint')}" dominant-baseline="${y < h / 2 ? 'hanging' : 'alphabetic'}">${escapeHtml(fmt(v))}</text>`;
marks = mk(hi, inner.t + 7) + mk(lo, h - inner.b);
}
const aria = o.ariaLabel ||
`trend from ${fmt(first.value)} to ${fmt(last.value)}, max ${fmt(dataMax)}`;
let foot = '';
if (o.labels && (o.labels.current || o.labels.max)) {
const bits = [];
if (o.labels.current) bits.push(`now ${fmt(last.value)}`);
if (o.labels.max) bits.push(`max ${fmt(dataMax)}`);
foot = `<div class="faint mono" style="font-size:10px;line-height:1.25">${escapeHtml(bits.join(' / '))}</div>`;
}
container.innerHTML = `<svg viewBox="0 0 ${w} ${h}" width="100%" height="${h}" preserveAspectRatio="none"
role="img" aria-label="${escapeHtml(aria)}">${filled ? area : ''}${baseline}
<path d="${d}" fill="none" stroke="${stroke}" stroke-width="1.4"/>${marks}</svg>${foot}`;
}
window.DashCharts = { LineChart, niceTicks, renderLegend, sparkline, cssVar,
src/dashboards/web/static/dash.css
+44
View File
@@ -179,6 +179,11 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
.grid.auto { grid-template-columns: repeat(auto-fit, minmax(min(360px, 100%), 1fr)); }
.grid.cols-2 { grid-template-columns: repeat(auto-fit, minmax(min(480px, 100%), 1fr)); }
.grid.cols-3 { grid-template-columns: repeat(auto-fit, minmax(min(300px, 100%), 1fr)); }
/* Cards that carry different amounts of content sit at their own height instead of being
stretched to the tallest sibling: a 5-row engine panel next to a 6-row one otherwise
grows a blank bottom, and a blank panel reads as "no data" (verified on `/llm`, phase
04 — 172px of empty card under the shorter panel). */
.grid.items-start { align-items: start; }
.stack { display: grid; gap: .85rem; align-content: start; }
/* ---- cards -------------------------------------------------------------- */
@@ -234,6 +239,45 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
.na { color: var(--text-faint); }
/* ---- engine cell list -----------------------------------------------------
* `/llm`'s secondary rows: label left, value right-aligned in tabular numerals, sparkline
* in a fixed-width third column so rows line up and growth stays right-aligned the way
* `llm_monitor`'s drawSpark did. Percentiles ride under the value as a mono foot line —
* visible without a hover, because a mean latency without its p95 is half the truth.
* The heroes above are the loudest thing here on purpose; nothing in this block competes
* with them, and only payload verdicts colour it (KV through `.cell.warn/.cell.fail`).
*/
.cells { display: grid; gap: .45rem; }
.cell {
display: grid;
grid-template-columns: minmax(0, 1fr) max-content 160px;
gap: .1rem .7rem;
align-items: center;
}
.cell .cell-label {
font-size: 10.5px; text-transform: uppercase; letter-spacing: .06em; color: var(--text-faint);
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
}
.cell .cell-value {
font-family: var(--mono); font-size: 1.05rem; font-weight: 600; line-height: 1.3;
font-variant-numeric: tabular-nums;
text-align: right; white-space: nowrap;
}
.cell .cell-value .unit { font-size: .62em; color: var(--text-muted); font-weight: 500; margin-inline-start: .18rem; }
.cell .cell-value.is-null { color: var(--text-faint); }
.cell .cell-foot { grid-column: 2; text-align: right; font-family: var(--mono); font-size: 10.5px; color: var(--text-faint); }
.cell .cell-spark { display: block; width: 160px; }
/* the KV verdict, taken from the API's `severity` exactly as the hero cards take theirs */
.cell.warn .cell-value { color: var(--warn); }
.cell.fail .cell-value { color: var(--fail); }
/* alarm chips reuse the one fail chip vocabulary — the row itself only exists while the
counter is non-zero, so this is the presence of an incident, never a graded threshold */
.alarms { display: flex; flex-wrap: wrap; gap: .4rem; }
@media (max-width: 640px) {
.cell { grid-template-columns: minmax(0, 1fr) max-content; }
.cell .cell-spark { grid-column: 1 / -1; width: 100%; }
}
/* ---- hero numbers --------------------------------------------------------
* The big glanceable number: label, value + unit, and optional bar and footnote.
* Colour comes from the *same* `ok|warn|fail` classes the rest of the stylesheet uses
src/dashboards/web/static/page-llm.js
+297 -124
View File
@@ -1,40 +1,116 @@
/* LLM dashboard.
*
* Layout (phase 04, per NOTES-chart-set): per engine, two big throughput numbers you can
* read across the room, then a compact list of metric cells that each carry their own
* sparkline over the *selected* window, percentile foot lines under the latency means, and
* alarm chips that exist only while there is an incident. The four trend charts stay, but
* half-height and below the panels: they are supporting evidence, not the main event.
*
* Every throughput cell shows where its number came from, because "78 tok/s" means
* different things depending on whether it was scraped from an aggregate gauge, measured as
* Δtokens/Δt across all slots, or estimated from TPOT over the process lifetime. The tag is
* generated from the cell's `source`, never hardcoded in markup, so a parser change cannot
* leave a stale label behind.
*
* Colour comes from payload verdicts only (`Dash.severityFrom` / `Dash.hero`). Throughput,
* sessions, queue depth and latency carry **no** severity — 0 tok/s on an idle server is
* not red, and a page-invented threshold is what `severity.py` exists to prevent.
*/
'use strict';
const { $, fetchJSON, setTxt, setValue, setBar, setDot, num, pct, duration, poll,
severityFor } = Dash;
const { $, fetchJSON, setTxt, setDot, num, duration, poll, markStale, severityFrom, hero,
eventText } = Dash;
const state = { range: 3600, panels: new Map(), charts: {}, poller: null, pollMs: 2000 };
const state = {
range: 3600,
panels: new Map(), // engine -> card element
charts: {},
poller: null,
pollMs: 2000,
};
const fmtCount = (v) => num(v, 0);
const fmtPct = (v) => `${num(v, 0)}%`;
const fmtMs = (v) => `${num(v, 0)} ms`;
/**
* What each engine panel shows, driven by flags instead of fourteen equal-size cells:
* `hero` is the read-across-the-room row, `spark` is one compact row with a fixed-width
* sparkline, `alarm` renders only as a red chip while the counter is non-zero, and
* `footnote` is the quiet lifetime strip.
*
* `minSpan: 1` on the count rows keeps their axis at least one request tall: history is
* stored as rollups, so eight seconds of a single queued request averages to 0.14 and an
* axis fitted to that draws a full-height spike for a queue nobody was in.
*
* `spark` is the `/api/llm/history` series (without the `llm.` prefix); `scale` converts
* stored units to printed ones (a 0-1 ratio to %, seconds to ms) for the plot as well as
* the number, so the line and the digits agree. `fixedMax` pins percent-like metrics to a
* 0-100 axis — an 8 % window that once held a 92 % spike must not read flat. `only` scopes
* a row to the engines that publish it, so a gufo panel never shows a latency mystery and a
* vLLM panel never shows a speculative-decoding row it cannot fill.
*/
const CELLS = [
{ key: 'decode_tokens_per_second', label: 'Decode', unit: 'tok/s', digits: 1,
{ key: 'decode', label: 'Decode', unit: 'tok/s', digits: 1, hero: true,
tip: 'Aggregate generation speed across every busy slot.' },
{ key: 'prompt_tokens_per_second', label: 'Prompt', unit: 'tok/s', digits: 1,
{ key: 'prompt', label: 'Prompt', unit: 'tok/s', digits: 1, hero: true,
tip: 'Prefill speed for newly computed tokens.' },
{ key: 'sessions', label: 'Sessions', unit: 'active', digits: 0 },
{ key: 'queued', label: 'Queued', unit: 'waiting', digits: 0 },
{ key: 'cache', label: 'KV cache', unit: '%', digits: 0, bar: true },
{ key: 'ttft_mean', label: 'TTFT mean', unit: 'ms', digits: 0 },
{ key: 'ttft_p95', label: 'TTFT p95', unit: 'ms', digits: 0 },
{ key: 'tpot_mean', label: 'TPOT mean', unit: 'ms', digits: 1 },
{ key: 'queue_mean', label: 'Queue mean', unit: 'ms', digits: 0 },
{ key: 'cache_hit', label: 'Prefix cache hit', unit: '%', digits: 0 },
{ key: 'spec_accept', label: 'Spec accept', unit: '%', digits: 0 },
{ key: 'tokens_total', label: 'Tokens lifetime', unit: '', digits: 0, human: true },
{ key: 'preemptions', label: 'Preemptions', unit: '', digits: 0 },
{ key: 'device_lost', label: 'GPU context lost', unit: '', digits: 0 },
{ key: 'sessions', label: 'Sessions', unit: 'active', digits: 0,
spark: 'requests_running', format: fmtCount, minSpan: 1 },
{ key: 'queued', label: 'Queued', unit: 'waiting', digits: 0,
spark: 'requests_waiting', format: fmtCount, minSpan: 1 },
{ key: 'cache', label: 'KV cache', unit: '%', digits: 0, fixedMax: 100,
spark: 'kv_cache_usage', scale: 100, format: fmtPct,
tip: 'Share of the KV cache in use — the only engine-pressure verdict on this page, '
+ 'and it comes from the API.' },
{ key: 'ttft', label: 'TTFT', unit: 'ms', digits: 0, p95: true, only: ['vllm'],
spark: 'ttft_seconds', scale: 1000, format: fmtMs },
{ key: 'tpot', label: 'TPOT', unit: 'ms', digits: 1, p95: true, only: ['vllm'],
spark: 'tpot_seconds', scale: 1000, format: fmtMs },
{ key: 'queue_mean', label: 'Queue time', unit: 'ms', digits: 0, p95: true, only: ['vllm'],
spark: 'queue_seconds', scale: 1000, format: fmtMs },
{ key: 'cache_hit', label: 'Prefix cache hit', unit: '%', digits: 0, fixedMax: 100,
spark: 'prefix_cache_hit_ratio', scale: 100, format: fmtPct },
{ key: 'spec_accept', label: 'Spec accept', unit: '%', digits: 0, fixedMax: 100,
only: ['gufo'], spark: 'spec_accept_rate', scale: 100, format: fmtPct },
{ key: 'preemptions', label: 'Preemptions', alarm: true,
tip: 'Running requests kicked back to the queue — an incident, not a stat.' },
{ key: 'device_lost', label: 'GPU context lost', alarm: true,
tip: 'The engine lost its GPU context — an incident, not a stat.' },
{ key: 'tokens_total', footnote: true },
];
const HERO_CELLS = CELLS.filter((cell) => cell.hero);
const SPARK_CELLS = CELLS.filter((cell) => cell.spark);
const ALARM_CELLS = CELLS.filter((cell) => cell.alarm);
/** One batched history request feeds every chart *and* every cell sparkline. */
const HISTORY_SERIES = Array.from(new Set([
'decode_tokens_per_second', 'prompt_tokens_per_second',
...SPARK_CELLS.map((cell) => cell.spark),
]));
const WINDOW_NAMES = { 300: '5 minutes', 900: '15 minutes', 3600: '1 hour',
21600: '6 hours', 86400: '24 hours', 604800: '7 days',
2592000: '30 days' };
function windowName(seconds) {
return WINDOW_NAMES[seconds] || duration(seconds);
}
/** The payload cell behind a UI key. A missing number stays `null` — never a 0. */
function cellValue(engine, metrics, sessions, key) {
const get = (k) => (metrics[k] && metrics[k].value != null ? metrics[k] : null);
const scaled = (cell, factor) => ({ value: cell.value * factor, source: cell.source,
derived: cell.derived });
switch (key) {
case 'decode':
case 'prompt': {
// the hero keys are short for the flags; the payload keeps the public metric name
const cell = get(`${key}_tokens_per_second`);
return cell ? { value: cell.value, source: cell.source, derived: cell.derived,
window: cell.window } : { value: null };
}
case 'sessions': {
const value = sessions && sessions.active != null ? sessions.active : null;
return { value, source: 'session count', derived: false };
@@ -44,32 +120,33 @@ function cellValue(engine, metrics, sessions, key) {
return cell ? { value: cell.value, source: cell.source, derived: false } : { value: null };
}
case 'cache': {
const cell = get('kv_cache_usage_ratio') || get('gpu_cache_usage');
return cell ? { value: cell.value * 100, source: cell.source, derived: false } : { value: null };
// the ratio cell carries the API verdict (`sources/llm.py` SEVERITY_CELLS ->
// severity.py); it travels with the converted value so the row and its sparkline
// paint exactly what the overview's alerts paint
const cell = metrics.kv_cache_usage_ratio || metrics.gpu_cache_usage;
if (!cell || cell.value == null) return { value: null };
return { value: cell.value * 100, source: cell.source, derived: cell.derived,
severity: cell.severity };
}
case 'ttft_mean': {
case 'ttft': {
const cell = get('time_to_first_token_seconds__mean');
return cell ? { value: cell.value * 1000, source: cell.source, derived: false } : { value: null };
return cell ? scaled(cell, 1000) : { value: null };
}
case 'ttft_p95': {
const cell = get('time_to_first_token_seconds__p95');
return cell ? { value: cell.value * 1000, source: cell.source, derived: false } : { value: null };
}
case 'tpot_mean': {
case 'tpot': {
const cell = get('time_per_output_token_seconds__mean');
return cell ? { value: cell.value * 1000, source: cell.source, derived: false } : { value: null };
return cell ? scaled(cell, 1000) : { value: null };
}
case 'queue_mean': {
const cell = get('request_queue_time_seconds__mean');
return cell ? { value: cell.value * 1000, source: cell.source, derived: false } : { value: null };
return cell ? scaled(cell, 1000) : { value: null };
}
case 'cache_hit': {
const cell = get('prefix_cache_hit_ratio') || get('prompt_cache_hit_ratio');
return cell ? { value: cell.value * 100, source: cell.source, derived: true } : { value: null };
return cell ? scaled(cell, 100) : { value: null };
}
case 'spec_accept': {
const cell = get('spec_accept_rate');
return cell ? { value: cell.value * 100, source: cell.source, derived: true } : { value: null };
return cell ? scaled(cell, 100) : { value: null };
}
case 'tokens_total': {
const cell = get('tokens_total');
@@ -91,6 +168,15 @@ function cellValue(engine, metrics, sessions, key) {
}
}
/** The histogram p95 beside a latency mean — printed as a foot line, never behind a hover. */
function p95Value(metrics, key) {
const source = { ttft: 'time_to_first_token_seconds__p95',
tpot: 'time_per_output_token_seconds__p95',
queue_mean: 'request_queue_time_seconds__p95' }[key];
const cell = source && metrics[source];
return cell && cell.value != null ? cell.value * 1000 : null;
}
/** The provenance tag: Δ10.0s for measured rates, the formula for estimates, nothing for a
* plain scraped gauge (which would only add noise). */
function provenance(cell) {
@@ -109,10 +195,25 @@ function humanNumber(value) {
return { text: value.toFixed(0), unit: 'tok' };
}
/** The rows this engine publishes. A row it can never fill is not shown at all — a
* permanent `n/a` is the dead-cell anti-pattern NOTES-chart-set exists to prevent. */
function sparkCellsFor(engine) {
return SPARK_CELLS.filter((cell) => !cell.only || cell.only.indexOf(engine) >= 0);
}
function enginePanel(engine) {
const card = document.createElement('article');
card.className = 'card';
card.dataset.engine = engine;
const heroes = HERO_CELLS.map((cell) =>
`<div class="hero" data-hero="${cell.key}"></div>`).join('');
const rows = sparkCellsFor(engine).map((cell) => `
<div class="cell" data-cell="${cell.key}"${cell.tip ? ` title="${cell.tip}"` : ''}>
<span class="cell-label">${cell.label}</span>
<span class="cell-value"><span data-v>n/a</span><span class="unit">${cell.unit}</span></span>
<span class="cell-spark" data-spark data-metric="${cell.spark}"></span>
${cell.p95 ? '<span class="cell-foot" data-foot></span>' : ''}
</div>`).join('');
card.innerHTML = `
<header>
<span class="dot idle"></span>
@@ -123,13 +224,11 @@ function enginePanel(engine) {
</header>
<div class="body stack">
<div class="banner err" hidden></div>
<div class="metrics">${CELLS.map((cell) => `
<div class="metric" data-cell="${cell.key}" ${cell.tip ? `title="${cell.tip}"` : ''}>
<div class="label">${cell.label}</div>
<div class="value${cell.digits === 0 ? '' : ''}"><span data-v>n/a</span><span class="unit">${cell.unit}</span></div>
${cell.bar ? '<div class="bar" style="margin-top:.25rem"><i></i></div>' : ''}
</div>`).join('')}</div>
<div class="heroes">${heroes}</div>
<div class="alarms" data-k="alarms" hidden></div>
<div class="cells">${rows}</div>
<div class="faint mono sessions-note" style="font-size:11.5px"></div>
<div class="faint mono token-note" style="font-size:11px"></div>
<div class="faint mono debug" style="font-size:11px"></div>
</div>`;
return card;
@@ -146,7 +245,7 @@ function renderPanel(card, enginePayload, stateRow) {
? (stateRow.error || 'never reached')
: (Dash.ageSeconds(stateRow.last_ok) > 15 ? `stale ${duration(Dash.ageSeconds(stateRow.last_ok))}`
: `updated ${duration(Dash.ageSeconds(stateRow.last_ok))}`)) : '—');
Dash.markStale(card, stateRow && (!stateRow.ok || Dash.ageSeconds(stateRow.last_ok) > 15));
markStale(card, stateRow && (!stateRow.ok || Dash.ageSeconds(stateRow.last_ok) > 15));
const banner = q('.banner.err');
if (stateRow && !stateRow.ok) {
@@ -157,51 +256,57 @@ function renderPanel(card, enginePayload, stateRow) {
banner.hidden = true;
}
for (const cell of CELLS) {
const wrapper = q(`[data-cell="${cell.key}"]`);
if (!wrapper) continue;
// The two heroes. No severity, by design: an idle server at 0 tok/s is healthy, so the
// digits stay neutral ink and reachability colour lives on the dot and the stale dimming
// (the `/system` NET precedent). `Dash.hero()` still owns the markup and the n/a rule.
for (const cell of HERO_CELLS) {
const value = cellValue(enginePayload.engine, metrics, sessions, cell.key);
const valueEl = wrapper.querySelector('[data-v]');
const unitEl = wrapper.querySelector('.unit');
const node = hero(q(`[data-hero="${cell.key}"]`), {
label: cell.label, value: value.value, unit: cell.unit, digits: cell.digits,
sub: provenance(value),
});
if (node) node.title = value.source || ''; // the exact source, as llm_monitor's tag does
}
// Compact value rows; their sparklines are drawn once per window per refresh from the
// one batched history response (`refreshHistory`).
for (const cell of sparkCellsFor(enginePayload.engine)) {
const row = q(`[data-cell="${cell.key}"]`);
if (!row) continue;
const value = cellValue(enginePayload.engine, metrics, sessions, cell.key);
const valueEl = row.querySelector('[data-v]');
const unitEl = row.querySelector('.unit');
// the row carries the payload verdict (KV cache is the only rated cell) so the
// sparkline, drawn from the history response, paints the same verdict the number does.
// Like the fleet table, only a breach paints the *text* — a healthy row stays quiet
// ink; the spark still strokes green through the same verdict.
const sev = severityFrom(value.severity);
row.dataset.sev = sev;
['ok', 'warn', 'fail'].forEach((name) => row.classList.toggle(name, name === sev));
if (value.value == null) {
setTxt(valueEl, 'n/a');
setTxt(unitEl, ''); // "n/a %" reads like a measurement of nothing
wrapper.classList.add('is-null');
} else {
wrapper.classList.remove('is-null');
if (cell.human) {
const parts = humanNumber(value.value);
setTxt(valueEl, parts ? parts.text : 'n/a');
setTxt(unitEl, parts ? parts.unit : '');
} else {
setTxt(valueEl, num(value.value, cell.digits));
setTxt(unitEl, cell.unit);
}
setTxt(valueEl, num(value.value, cell.digits));
setTxt(unitEl, cell.unit);
}
let tag = wrapper.querySelector('.prov');
const label = provenance(value);
if (label) {
if (!tag) {
tag = document.createElement('span');
tag.className = 'prov';
wrapper.querySelector('.value').appendChild(tag);
}
setTxt(tag, label);
tag.title = `${value.source || ''}${value.window ? ` · measured over ${value.window.toFixed(1)}s` : ''}`;
} else if (tag) {
tag.remove();
}
if (cell.bar) setBar(wrapper.querySelector('.bar'), value.value,
severityFor(value.value, 75, 90));
if (cell.key === 'device_lost' && value.value != null && value.value > 0) {
wrapper.querySelector('.value').classList.add('is-null');
wrapper.classList.add('fail');
wrapper.title = `GPU context has been lost ${value.value} times — an incident, not a stat`;
if (cell.p95) {
const p95 = p95Value(metrics, cell.key);
setTxt(row.querySelector('[data-foot]'),
p95 == null ? 'p95 n/a' : `p95 ${num(p95, cell.digits)} ${cell.unit}`);
}
}
renderAlarms(card, enginePayload);
const note = q('.sessions-note');
setTxt(note, sessions.detail || `${sessions.active == null ? 'n/a' : sessions.active} active sessions`);
// a lifetime counter is a footnote, not a glance metric (NOTES-chart-set, demoted)
const tokens = cellValue(enginePayload.engine, metrics, sessions, 'tokens_total');
const parts = tokens.value == null ? null : humanNumber(tokens.value);
setTxt(q('.token-note'), parts
? `lifetime ${parts.text} ${parts.unit} · ${tokens.source || 'unknown source'}`
: 'lifetime tokens n/a');
const upstream = q('.chip.upstream');
if (enginePayload.upstream) {
upstream.hidden = false;
@@ -213,6 +318,39 @@ function renderPanel(card, enginePayload, stateRow) {
: '');
}
/**
* Alarm chips exist only while there is an incident.
*
* A permanent cell for a counter that is always 0 trains the eye to ignore the row, so at
* zero the row does not exist: the container is emptied and hidden, and only a non-zero —
* an *incident*, not a graded threshold — puts a `.chip.fail` back. `sources/llm.py`
* already treats any non-zero gufo_device_lost_total this way.
*/
function renderAlarms(card, enginePayload) {
const host = card.querySelector('[data-k="alarms"]');
if (!host) return;
const chips = [];
for (const cell of ALARM_CELLS) {
const value = cellValue(enginePayload.engine, enginePayload.metrics || {},
enginePayload.sessions || {}, cell.key);
if (value.value != null && value.value > 0) {
chips.push(`<span class="chip fail" data-alarm="${cell.key}">${cell.label} ${num(value.value, 0)}</span>`);
}
}
const signature = chips.join('');
if (host.dataset.sig !== signature) {
host.innerHTML = signature;
host.dataset.sig = signature;
}
for (const chip of host.querySelectorAll('[data-alarm]')) {
const cell = ALARM_CELLS.find((c) => c.key === chip.dataset.alarm);
const value = cellValue(enginePayload.engine, enginePayload.metrics || {},
enginePayload.sessions || {}, cell.key);
chip.title = `${cell.tip || cell.label} · ${value.source || 'source unknown'}`;
}
host.hidden = chips.length === 0;
}
function renderSlots(payloads) {
const gufo = payloads.find((p) => p.engine === 'gufo');
const rows = gufo && gufo.sessions ? gufo.sessions.list || [] : [];
@@ -239,13 +377,19 @@ function renderSlots(payloads) {
}
function makeCharts() {
const common = { height: 165, xSpan: state.range };
state.charts.decode = new DashCharts.LineChart('#chart-decode', Object.assign({}, common, { yMin: 0 }));
state.charts.prompt = new DashCharts.LineChart('#chart-prompt', Object.assign({}, common, { yMin: 0 }));
state.charts.kv = new DashCharts.LineChart('#chart-kv', Object.assign({}, common,
{ yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) }));
state.charts.queue = new DashCharts.LineChart('#chart-queue', Object.assign({}, common,
{ yMin: 0, yFormat: (v) => num(v, 0) }));
// Half of the old 165px: the cells read first, the charts back them up. Every plot
// names itself for assistive tech — the legend is a sibling that labels nothing.
const common = { height: 110, xSpan: state.range };
state.charts.decode = new DashCharts.LineChart('#chart-decode', Object.assign({}, common, {
yMin: 0, ariaLabel: 'Decode throughput in tokens per second over the selected window, one line per engine' }));
state.charts.prompt = new DashCharts.LineChart('#chart-prompt', Object.assign({}, common, {
yMin: 0, ariaLabel: 'Prompt / prefill throughput in tokens per second over the selected window, one line per engine' }));
state.charts.kv = new DashCharts.LineChart('#chart-kv', Object.assign({}, common, {
yMin: 0, yMax: 100, yFormat: (v) => num(v, 0),
ariaLabel: 'KV / prefix cache pressure in per cent over the selected window, one line per engine' }));
state.charts.queue = new DashCharts.LineChart('#chart-queue', Object.assign({}, common, {
yMin: 0, yFormat: (v) => num(v, 0),
ariaLabel: 'Requests running and requests waiting over the selected window, one pair of lines per engine' }));
}
function pointsFor(response, engine, metric) {
@@ -253,59 +397,89 @@ function pointsFor(response, engine, metric) {
return found ? found.points.map((p) => ({ ts: p.ts, value: p.value })) : [];
}
async function refreshCharts(engineNames) {
function drawChart(chartEl, legendEl, chart, series, format, showPeak) {
chart.opts.xSpan = state.range;
chart.setData(series);
DashCharts.renderLegend(legendEl, series, format, showPeak);
}
/**
* Every chart and every cell sparkline, from ONE `/api/llm/history` request.
*
* The endpoint answers a metric list for every engine, and unknown names come back as
* empty series on purpose — so the page asks for the full cell list once per refresh
* instead of one XHR per cell, and a partly-filled window stays partly filled rather than
* rescaling to whatever samples exist.
*/
async function refreshHistory(engineNames) {
if (!engineNames.length) return;
const list = engineNames.join(',');
const [decode, prompt, queue] = await Promise.all([
fetchJSON(`/api/llm/history?engines=${list}&metrics=decode_tokens_per_second&range=${state.range}`)
.catch(() => null),
fetchJSON(`/api/llm/history?engines=${list}&metrics=prompt_tokens_per_second&range=${state.range}`)
.catch(() => null),
fetchJSON(`/api/llm/history?engines=${list}&metrics=requests_running,requests_waiting,` +
`kv_cache_usage&range=${state.range}`).catch(() => null),
]);
if (decode) {
const series = engineNames.map((engine) => ({ name: engine,
points: pointsFor(decode, engine, 'decode_tokens_per_second') }));
state.charts.decode.setData(series);
DashCharts.renderLegend($('#legend-decode'), series, (v) => num(v, 1) + ' tok/s', true);
const response = await fetchJSON(
`/api/llm/history?engines=${engineNames.join(',')}` +
`&metrics=${HISTORY_SERIES.join(',')}&range=${state.range}`).catch(() => null);
if (!response) return;
drawChart($('#chart-decode'), $('#legend-decode'), state.charts.decode,
engineNames.map((engine) => ({ name: engine,
points: pointsFor(response, engine, 'decode_tokens_per_second') })),
(v) => num(v, 1) + ' tok/s', true);
drawChart($('#chart-prompt'), $('#legend-prompt'), state.charts.prompt,
engineNames.map((engine) => ({ name: engine,
points: pointsFor(response, engine, 'prompt_tokens_per_second') })),
(v) => num(v, 0) + ' tok/s', true);
drawChart($('#chart-kv'), $('#legend-kv'), state.charts.kv,
engineNames.map((engine) => ({ name: engine,
points: pointsFor(response, engine, 'kv_cache_usage').map((p) => ({ ts: p.ts, value: p.value * 100 })) })),
(v) => num(v, 1) + '%', true);
const sessionsSeries = [];
for (const engine of engineNames) {
sessionsSeries.push({ name: `${engine} running`, points: pointsFor(response, engine, 'requests_running') });
sessionsSeries.push({ name: `${engine} waiting`, points: pointsFor(response, engine, 'requests_waiting') });
}
if (prompt) {
const series = engineNames.map((engine) => ({ name: engine,
points: pointsFor(prompt, engine, 'prompt_tokens_per_second') }));
state.charts.prompt.setData(series);
DashCharts.renderLegend($('#legend-prompt'), series, (v) => num(v, 0) + ' tok/s', true);
}
if (queue) {
const series = [];
for (const engine of engineNames) {
series.push({ name: `${engine} running`, points: pointsFor(queue, engine, 'requests_running') });
series.push({ name: `${engine} waiting`, points: pointsFor(queue, engine, 'requests_waiting') });
drawChart($('#chart-queue'), $('#legend-queue'), state.charts.queue, sessionsSeries,
(v) => num(v, 0), false);
drawSparks(response);
}
/** One sparkline per cell, over the *selected* window, x mapped to real time. */
function drawSparks(response) {
for (const [engine, card] of state.panels) {
for (const cell of sparkCellsFor(engine)) {
const row = card.querySelector(`[data-cell="${cell.key}"]`);
const slot = row && row.querySelector('[data-spark]');
if (!slot) continue;
const points = pointsFor(response, engine, cell.spark)
.map((p) => ({ ts: p.ts, value: p.value * (cell.scale || 1) }));
DashCharts.sparkline(slot, points, {
align: 'right', // newest pinned right; x stays time, not sample index
span: state.range,
fixedMax: cell.fixedMax != null ? cell.fixedMax : null,
minSpan: cell.minSpan != null ? cell.minSpan : null,
// the only rated spark is KV, and its colour is the API's verdict; everything else
// draws the faint unrated stroke, because a shape is not a verdict
severity: row.dataset.sev || '',
labels: { max: true },
format: cell.format,
ariaLabel: `${engine} ${cell.label} over the last ${windowName(state.range)}`,
});
}
state.charts.queue.setData(series);
DashCharts.renderLegend($('#legend-queue'), series, (v) => num(v, 0));
const kv = engineNames.map((engine) => ({ name: engine,
points: pointsFor(queue, engine, 'kv_cache_usage').map((p) => ({ ts: p.ts, value: p.value * 100 })) }));
state.charts.kv.setData(kv);
DashCharts.renderLegend($('#legend-kv'), kv, (v) => num(v, 1) + '%', true);
}
}
/**
* The engine-totals card: only what no single panel can say.
*
* Summed decode/prompt/KV left with phase 04 (per-engine is the honest view), and the card
* hides entirely below two engines — a fleet of one does not need a fleet row.
*/
function renderFleet(fleet) {
if (!fleet) return;
const card = $('#totals');
if (!card) return;
const total = fleet && fleet.engines_total != null ? fleet.engines_total : 0;
card.hidden = total < 2;
if (!total) return;
setTxt($('#f-live'), fleet.engines_live == null ? 'n/a' : String(fleet.engines_live));
setTxt($('#f-total'), fleet.engines_total == null ? '' : `of ${fleet.engines_total}`);
setValue($('#f-decode'), fleet.decode_tokens_per_second, 1);
setValue($('#f-prompt'), fleet.prompt_tokens_per_second, 0);
setTxt($('#f-active'), fleet.requests_active == null ? 'n/a' : num(fleet.requests_active, 0));
setTxt($('#f-waiting'), fleet.requests_waiting == null ? 'n/a' : num(fleet.requests_waiting, 0));
const tokens = humanNumber(fleet.tokens_total);
setTxt($('#f-tokens'), tokens ? tokens.text : 'n/a');
$('#f-tokens').nextElementSibling && setTxt($('#f-tokens').nextElementSibling, tokens ? tokens.unit : '');
const kv = fleet.kv_cache_usage_max == null ? null : fleet.kv_cache_usage_max * 100;
setValue($('#f-kv'), kv, 0);
setBar($('#f-kv').closest('.metric').querySelector('.bar'), kv, severityFor(kv, 75, 90));
$('#seam-note').textContent = fleet.seams || '';
}
async function refresh() {
@@ -339,7 +513,7 @@ async function refresh() {
`window ${duration(state.range)}`;
$('#foot-note').textContent = `poll ${Math.round(state.pollMs / 1000)}s · rates over the ` +
`Δ window in each tag`;
await refreshCharts(names);
await refreshHistory(names);
}
async function refreshEvents() {
@@ -353,7 +527,7 @@ async function refreshEvents() {
list.innerHTML = data.events.map((event) => `<li><time>${
new Date(event.ts * 1000).toLocaleTimeString(undefined, { hour: '2-digit', minute: '2-digit', hour12: false })
}</time><span><span class="sev ${event.severity}">${DashCharts.escapeHtml(event.host)}</span> ` +
`${DashCharts.escapeHtml(event.message)}</span></li>`).join('');
`${DashCharts.escapeHtml(eventText(event))}</span></li>`).join('');
} catch (err) { /* next poll */ }
}
@@ -373,8 +547,7 @@ document.addEventListener('DOMContentLoaded', async () => {
Dash.rangeControl($('#range'), state.range, (seconds) => {
state.range = seconds;
Object.values(state.charts).forEach((chart) => { chart.opts.xSpan = seconds; });
refresh();
refresh(); // charts and sparklines both re-read the selected window
});
state.poller = poll(async () => { await refresh(); await refreshEvents(); }, state.pollMs);
src/dashboards/web/static/page-system.js
+1 -1
View File
@@ -472,7 +472,7 @@ async function refreshEvents() {
list.innerHTML = data.events.map((event) => `<li><time>${
new Date(event.ts * 1000).toLocaleTimeString(undefined, { hour: '2-digit', minute: '2-digit', hour12: false })
}</time><span><span class="sev ${event.severity}">${DashCharts.escapeHtml(event.host)}</span> ` +
`${DashCharts.escapeHtml(event.message)}</span></li>`).join('');
`${DashCharts.escapeHtml(Dash.eventText(event))}</span></li>`).join('');
} catch (err) { /* the next poll will try again */ }
}
src/dashboards/web/static/page-ups.js
+3 -2
View File
@@ -10,7 +10,8 @@
*/
'use strict';
const { $, fetchJSON, setTxt, setValue, setBar, setDot, num, duration, poll, severityFor } = Dash;
const { $, fetchJSON, setTxt, setValue, setBar, setDot, num, duration, poll, severityFor,
eventText } = Dash;
const state = { range: 86400, cards: new Map(), charts: {}, poller: null, pollMs: 5000,
simulated: false, pending: null };
@@ -327,7 +328,7 @@ async function refreshEvents() {
new Date(event.ts * 1000).toLocaleString(undefined,
{ month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit', hour12: false })
}</time><span><span class="sev ${event.severity}">${DashCharts.escapeHtml(event.host)}</span> ` +
`${DashCharts.escapeHtml(event.message.replace(event.host + ':', '').trim())}` +
`${DashCharts.escapeHtml(eventText(event))}` +
`${event.data && event.data.runtime_seconds ? ` <span class="faint mono">(${
duration(event.data.runtime_seconds)} runtime)</span>` : ''}</span></li>`).join('');
} catch (err) { /* next poll */ }
tests/test_api.py
+277 -6
View File
@@ -24,6 +24,7 @@ from dashboards.app import create_app
from dashboards.config import Settings, Target
from dashboards.severity import classify
from dashboards.sources import nut
from dashboards.sources.llm import HISTORY_METRICS
from dashboards.store import Event, Sample, SampleStore
# refused immediately, and never dials out of the test environment
@@ -576,15 +577,15 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
"""The old call sites still work until the remaining phases convert each page.
`/` was converted in phase 02 and `/system` in phase 03: both colour themselves from
the `severity` verdicts the API already sends, so they carry no threshold of their
own. The two pages still to come must keep compiling against the fallback until they
are converted — deleting it early would break a page mid-migration.
`/` was converted in phase 02, `/system` in phase 03 and `/llm` in phase 04: each
colours itself from the `severity` verdicts the API already sends, so it carries no
threshold of its own. `/ups` is phase 05's job and must keep compiling against the
fallback until it is converted — deleting it early would break that page.
"""
js = client.get("/static/app.js").text
assert "FALLBACK ONLY" in js
for page in ("/static/page-llm.js", "/static/page-ups.js"):
assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site"
assert "severityFor" in client.get("/static/page-ups.js").text, \
"/ups still uses the fallback until phase 05 converts it"
def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> None:
@@ -786,6 +787,222 @@ def test_no_per_core_visual_library_survives_the_page(client) -> None:
assert name in client.get("/static/charts.js").text
def test_sparkline_ships_the_fixed_scale_and_verdict_options(client) -> None:
"""Phase 04 turns `/llm` cells into sparkline rows, so the shared `sparkline()` must
behave like `llm_monitor`'s `drawSpark` (the task-02 contract, NOTES-chart-set).
Pinned on the served bytes because there is no JS test runner (a locked anchor). What
matters is not the option names but the semantics behind each one: a percent metric
that silently rescaled to its own data would make an 8 % KV cache and a 92 % spike the
same squiggle, and a stroke coloured from a page-invented threshold would break the
one-colour-language rule the same way.
"""
charts_js = client.get("/static/charts.js").text
body = charts_js[charts_js.index("function sparkline("):charts_js.index("window.DashCharts")]
# the phase grep criterion, as a test: the options are really in the shipped file
hits = [ln for ln in charts_js.splitlines()
if re.search(r"fixedMax|align|severity", ln)]
assert len(hits) >= 3, "the sparkline upgrade is not in the served charts.js"
for opt in ("fixedMax", "align", "labels", "severity", "minPoints", "span", "format",
"minSpan"):
assert opt in body, f"sparkline() lost its `{opt}` option"
# fixedMax: the data clamps to the axis, never the axis to the data
assert "Math.min(Math.max(v, lo), hi)" in body, \
"a fixedMax sparkline rescales to its own data again"
# align:'right' grows within a real time window (span seconds), not a sample count
assert "x1 - o.span" in body, "align:'right' forgot the window it grows inside"
# verdict colour comes from the theme tokens only; unrated is faint grey, never green
for token in ("cssVar('--ok')", "cssVar('--warn')", "cssVar('--fail')",
"cssVar('--text-faint')"):
assert token in body, f"sparkline() stopped taking its colour from {token}"
assert ".sev-" not in body, "sparkline invented a parallel severity vocabulary"
# the `now / max` foot llm_monitor prints, via the caller's format callback
assert "now ${fmt(last.value)}" in body and "max ${fmt(dataMax)}" in body
# never a confident line through one point, and the foot quotes data, not the padded axis
assert "minPoints: 2" in body and "no history" in body
# ASSUMPTION (task 02): no animation at all, so prefers-reduced-motion holds by design
assert "transition:" not in body and "animation:" not in body
# ---- the `/llm` layout (phase 04 task 03) ---------------------------------
#
# The drift this phase fixes was a 14-cell grid where everything was the same size and
# nothing showed recent shape. These tests pin the *shape* the page committed to on the
# served bytes (a locked anchor: no JS test runner), so a regression to "all cells equal"
# fails a test rather than a screenshot.
def test_llm_page_colours_from_the_payload_not_from_js_thresholds(client) -> None:
"""`page-llm.js` paints the KV verdict `severity.py` produced; it owns no threshold.
Both `severityFor(value, 75, 90)` call sites are gone (the task's explicit work item):
the KV row takes `severity` from the rated ratio cell, the fleet KV bar was deleted
with the summed card, and every unrated number — throughput above all, because 0 tok/s
on an idle server is healthy — draws no colour at all.
"""
js = client.get("/static/page-llm.js").text
assert "severityFor(" not in js, "the LLM page re-invented a threshold"
assert "severityFrom" in js
# the KV verdict comes from the payload cells `sources/llm.py` rates
kv = js[js.index("case 'cache':"):js.index("case 'ttft':")]
assert "kv_cache_usage_ratio" in kv and "gpu_cache_usage" in kv and "severity" in kv
def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
"""Decode and prompt are the largest elements per engine; the rest is visibly secondary.
The hero row is `Dash.hero()` markup (2.4rem via `.hero`), built from the `hero: true`
flags in `CELLS`, and throughput carries **no** severity — `severity.py` has no
throughput row and the page invents none (the neutral-ink decision in NOTES-chart-set,
mirroring the `/system` NET precedent). The provenance tag stays in the hero's `.sub`
with the exact `source` as the tooltip: removing it to look cleaner is a regression.
"""
js = client.get("/static/page-llm.js").text
panel = js[js.index("function enginePanel("):js.index("function renderPanel(")]
assert 'data-hero="${cell.key}"' in panel, "the hero row is not flag-driven markup"
assert '<div class="heroes">' in panel, "the hero row left the engine panel"
heroes = re.findall(r"{ key: '(\w+)'[^}]*?hero: true", js)
assert heroes == ["decode", "prompt"], "the hero row is no longer the two throughputs"
# driven by flags, not fourteen identical cells
assert "SPARK_CELLS" in js and "ALARM_CELLS" in js
assert js.count("spark: '") >= 6, "the compact sparkline list shrank back to a grid"
# the hero call must not pass a severity — unrated by design
hero_call = js[js.index("const node = hero("):js.index("if (node) node.title")]
assert "severity" not in hero_call, "throughput invented a verdict"
# provenance intact: the tag is generated from the cell's `source`, never hardcoded
assert "provenance" in js and "Δ" in js
assert "node.title = value.source" in js
def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client) -> None:
"""Each compact cell draws its own sparkline over the *selected* window; latency cells
print mean **and** p95 without a hover.
The window is honest time (`align: 'right'` + `span`), percent-like metrics keep a
fixed 0-100 axis, and the p95 comes from the snapshot's histogram cells — p95 needs no
stored series. The `only` scopes are what keep a gufo panel from showing a latency
mystery and a vLLM panel from showing a spec-decode row it cannot fill.
"""
js = client.get("/static/page-llm.js").text
sparks = js[js.index("function drawSparks("):js.index("function renderFleet(")]
assert "DashCharts.sparkline(" in sparks
for opt in ("align: 'right'", "span: state.range", "fixedMax", "labels: { max: true }"):
assert opt in sparks, f"cell sparklines lost `{opt}`"
# percent cells pin the axis; ratio series are scaled to the printed percent
assert js.count("fixedMax: 100") >= 3, "KV / prefix-hit / spec-accept lost their 0-100 axis"
assert "scale: 1000" in js and "scale: 100" in js
# every spark cell declares its history series in the markup it is built from
assert 'data-spark data-metric="${cell.spark}"' in js
# percentile foot lines: mean big in the row, p95 beneath, n/a when nothing measured
assert "function p95Value(" in js
for name in ("time_to_first_token_seconds__p95", "time_per_output_token_seconds__p95",
"request_queue_time_seconds__p95"):
assert name in js, f"{name} no longer feeds a foot line"
assert "p95 n/a" in js
assert 'data-foot' in js
# engine-scoped rows: gufo never grows a TTFT row, vLLM never a spec-accept row
assert "only: ['vllm']" in js and "only: ['gufo']" in js
scoped = js[js.index("function sparkCellsFor("):js.index("function enginePanel(")]
assert "cell.only" in scoped
# a count row never gets an axis finer than one request: history arrives as rollups,
# so 8 s of a single queued request averages to 0.14 and a fitted axis draws a spike for
# a queue nobody was in — the twin of the rescale `fixedMax` fixes
assert _code(js).count("minSpan: 1") == 2, "sessions/queued lost their one-request axis floor"
assert "minSpan: cell.minSpan" in sparks, "cell minSpan never reaches the sparkline"
def test_llm_sparklines_and_charts_cost_one_batched_history_request(client) -> None:
"""One `/api/llm/history` call per refresh feeds charts *and* every sparkline row.
The endpoint answers a metric list for every engine, and unknown names come back as
empty series on purpose — so the page asks for the full cell set once instead of one
XHR per cell, and the two series phase 04 added (`tpot_seconds`, `spec_accept_rate`)
must be in that list or those rows are permanent "no history" slots.
"""
js = client.get("/static/page-llm.js").text
# comments aside, exactly one fetch hits the endpoint: one request, every metric
calls = re.findall(r"fetchJSON\(\s*`/api/llm/history", _code(js))
assert len(calls) == 1, "the page makes more than one history request again"
assert "HISTORY_SERIES.join" in js and "SPARK_CELLS.map((cell) => cell.spark)" in js
request = js[js.index("async function refreshHistory("):js.index("function drawSparks(")]
assert "metrics=${HISTORY_SERIES.join" in request
assert "range=${state.range}" in request
# and the endpoint really answers every name the page asks for
for name in ("tpot_seconds", "spec_accept_rate", "ttft_seconds", "queue_seconds",
"kv_cache_usage", "requests_running", "requests_waiting",
"prefix_cache_hit_ratio"):
assert name in HISTORY_METRICS, f"llm.{name} is not stored, so its spark is dead"
def test_llm_alarm_cells_exist_only_while_there_is_an_incident(client) -> None:
"""`preemptions` / `device_lost` are absent at zero and red chips above zero.
The rule is the *existence of an incident*, not a graded threshold — `sources/llm.py`
already treats any non-zero `gufo_device_lost_total` as one. The chip reuses the shared
`.chip.fail` vocabulary; nothing new is painted, and a container emptied at zero means
the row does not exist rather than showing a permanent ignored 0.
"""
js = client.get("/static/page-llm.js").text
alarms = js[js.index("function renderAlarms("):js.index("function renderSlots(")]
assert "class=\"chip fail\"" in alarms, "the alarm chip left the shared fail vocabulary"
assert "value.value > 0" in alarms
assert "host.hidden = chips.length === 0" in alarms
assert 'data-alarm="${cell.key}"' in alarms
assert js.count("alarm: true") == 2
def test_llm_totals_card_says_only_what_no_engine_panel_says(client) -> None:
"""Engines answering, active, queued — and hidden below two engines.
The summed decode/prompt and the un-rated fleet KV bar are gone: per-engine is the
honest view (the fleet KV bar had no `severity` to render, which is invariant-grade
trouble), and a cross-engine sum of one engine just duplicates that engine's own card
(the `llm_monitor` rule). `/api/llm/snapshot` keeps every field — this is a page
decision, not an API one.
"""
html = _code(client.get("/llm").text)
for gone in ("f-decode", "f-prompt", "f-kv", "f-tokens", "seam-note"):
assert gone not in html, f"/llm still totals {gone}"
for kept in ("f-live", "f-active", "f-waiting", 'id="totals"'):
assert kept in html, f"/llm lost {kept}"
js = client.get("/static/page-llm.js").text
assert "card.hidden = total < 2" in js, "the totals card no longer hides for one engine"
def test_llm_charts_are_secondary_and_named(client) -> None:
"""The four trend charts stay, at half height, *below* the engine panels, each named.
Half of the old 165px and moved after `#engines` is the "cells read first" decision;
every `LineChart` gets an `ariaLabel` because the legend is a sibling that labels
nothing (`/system` did this in phase 03, `/llm` never had).
"""
html = _code(client.get("/llm").text)
assert html.index('id="engines"') < html.index('id="chart-decode"'), \
"the charts are back on top of the engine panels"
for chart in ("chart-decode", "chart-prompt", "chart-kv", "chart-queue"):
assert f'id="{chart}"' in html, f"/llm lost the {chart} chart"
js = client.get("/static/page-llm.js").text
assert "height: 110" in js and "height: 165" not in js
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
assert charts.count("ariaLabel:") == 4, "an /llm chart lost its accessible name"
def test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling(client) -> None:
"""A 5-row gufo panel beside a 7-row vLLM panel keeps its own height.
Found while comparing against `llm_monitor`: the shorter engine card grew 172 px of
blank card, and a blank panel reads as "no data" — the exact misreading the unconfigured
rules exist to prevent. Pinned on the served markup so the `.grid.items-start` rule in
`dash.css` cannot silently become dead CSS (and the void cannot silently come back).
"""
html = _code(client.get("/llm").text)
engines_div = html[html.index('<div class="grid'):html.index('id="engines"') + 40]
assert "items-start" in engines_div, "the engine grid stretches its cards again"
css = client.get("/static/dash.css").text
assert ".grid.items-start" in css, "the rule the engine grid relies on is gone"
def test_overview_is_exactly_two_panels(client) -> None:
"""The landing page answers one question: what is wrong, and what is the fleet doing.
@@ -1019,6 +1236,38 @@ def test_llm_history_charts_both_engines_by_default(
("vllm", "llm.kv_cache_usage_ratio")}
def test_one_batched_llm_history_request_answers_every_cell_series(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The exact query `/llm` issues per refresh returns every chart *and* sparkline series.
Phase 04 sends one request carrying the chart trio and the whole spark cell list —
including the two series the layout required (`tpot_seconds` for the vLLM TPOT row,
`spec_accept_rate` for the gufo spec row). A name missing here is a cell that reads
"no history" forever in the browser, which is the dead-cell anti-pattern the phase
wrote NOTES-chart-set to prevent; a rename in `sources/llm.py` fails here first.
"""
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
now = time.time()
metrics = ["decode_tokens_per_second", "prompt_tokens_per_second",
"kv_cache_usage", "requests_running", "requests_waiting",
"ttft_seconds", "tpot_seconds", "queue_seconds",
"prefix_cache_hit_ratio", "spec_accept_rate"]
for metric in metrics:
store.add("llm", "gufo", f"llm.{metric}", 1.5, ts=now - 60)
store.add("llm", "vllm", f"llm.{metric}", 2.5, ts=now - 30)
body = test_client.get("/api/llm/history", params={
"engines": "gufo,vllm", "metrics": ",".join(metrics), "range": "1h"}).json()
assert body["range_seconds"] == 3_600.0
by_key = {(item["host"], item["metric"]) for item in body["series"]}
assert by_key == {(engine, f"llm.{metric}")
for engine in ("gufo", "vllm") for metric in metrics}
points = {(item["host"], item["metric"]): item["points"] for item in body["series"]}
assert points[("gufo", "llm.spec_accept_rate")][0]["value"] == pytest.approx(1.5)
assert points[("vllm", "llm.tpot_seconds")][0]["value"] == pytest.approx(2.5)
def test_metrics_endpoint_lists_what_actually_has_history(client) -> None:
body = client.get("/api/metrics").json()
assert body["metrics"] == []
@@ -1039,6 +1288,28 @@ def test_events_are_returned_in_json(tmp_path: Path, monkeypatch: pytest.MonkeyP
store.close()
def test_event_rows_say_their_subject_once(client) -> None:
"""`poller.py` messages name their own subject, and the row prints a host chip too.
"gufo gufo is answering again after 1 failure" reads as two problems — the same defect
`alerts._clean_error()` already fixes for the overview's alert list. `/ups` had the
mirror-image bug: it `replace()`d `host + ':'` *anywhere*, so "Quick battery test sent
to apc: OK" printed as "Quick battery test sent to OK". All three feeds now share
`Dash.eventText()`, which strips the wrapper and a **leading** name only.
"""
app_js = client.get("/static/app.js").text
helper = app_js[app_js.index("function eventText("):app_js.index("// ---- DOM helpers")]
assert "eventText" in app_js[app_js.index("return {"):], "eventText is not exported"
assert "^UpstreamError:\\s+" in helper, "the wrapper `alerts.py` drops is back"
assert "new RegExp" in helper and "stripped.trim() ? stripped : out" in helper, \
"the host strip is no longer anchored to the front, or can empty a message"
for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"):
js = client.get(page).text
assert "eventText(event)" in js, f"{page} prints the raw message again"
assert "replace(event.host" not in client.get("/static/page-ups.js").text, \
"/ups is mangling the host name mid-message again"
# ---- the one write endpoint ------------------------------------------------ #
def test_command_endpoint_is_post_only(ups_client) -> None:
tests/test_llm_source.py
+27
View File
@@ -231,6 +231,33 @@ def test_engine_result_charts_a_measured_rate_after_the_second_poll(metrics_text
assert any(s.metric == "llm.decode_tokens_per_second" for s in result.samples)
def test_vllm_stores_the_tpot_series_the_cell_sparkline_draws(metrics_text) -> None:
"""Phase 04 added `llm.tpot_seconds` so the vLLM TPOT row has a shape, not a dead slot.
Hermetic twin of the live check: the captured exposition must resolve the histogram the
page charts, or `/llm` shows a permanent "no history" next to a real mean.
"""
result = engine_result("vllm", "http://v:1", metrics_text("vllm"), RateEstimator(), now=1.0)
mean = result.payload["metrics"]["time_per_output_token_seconds__mean"]["value"]
assert mean is not None, "the vLLM capture lost its TPOT histogram — re-capture fixtures"
stored = {s.metric: s.value for s in result.samples}
assert stored["llm.tpot_seconds"] == pytest.approx(mean)
# gufo publishes no output-token histogram, so it stores no TPOT series (never a 0)
gufo = engine_result("gufo", "http://g:1", metrics_text("gufo"), RateEstimator(), now=1.0)
assert "llm.tpot_seconds" not in {s.metric for s in gufo.samples}
def test_gufo_stores_the_spec_accept_series_the_cell_sparkline_draws(metrics_text) -> None:
"""`llm.spec_accept_rate` is the gufo-only half of the phase-04 history addition."""
result = engine_result("gufo", "http://g:1", metrics_text("gufo"), RateEstimator(), now=1.0)
ratio = result.payload["metrics"]["spec_accept_rate"]["value"]
assert ratio is not None and 0.0 < ratio <= 1.0
stored = {s.metric: s.value for s in result.samples}
assert stored["llm.spec_accept_rate"] == pytest.approx(ratio)
vllm = engine_result("vllm", "http://v:1", metrics_text("vllm"), RateEstimator(), now=1.0)
assert "llm.spec_accept_rate" not in {s.metric for s in vllm.samples}
def test_sessions_pass_through_slots_for_gufo() -> None:
slots = [{"id": 0, "idle": False, "n_prompt": 12, "n_predict": 40}]
result = engine_result("gufo", "http://g:1", "", RateEstimator(), slots=slots, now=1.0)