phase: 04_llm_chart_readability
**Phase 04 final verification pass — all tasks already in `complete/`; verified + fixed 3 defects found in verification.**
- Verified `/llm` live (both engines, gufo-only instance on :8099, `llm_monitor` on :8095) in a real browser; dev server left running/healthy.
- Completed task-04's truncated layout fix: `#engines` now carries `.items-start` (the CSS rule existed but nothing used it) — gufo 544 px vs vLLM 715 px, no blank card; pinned by `test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling`.
- Fixed: event rows said the subject twice ("gufo gufo is answering again"), and `/ups` mangled mid-message names ("sent to apc: OK" → "sent to OK") → shared `Dash.eventText()` (strips `UpstreamError: ` + a *leading* name only, mirroring `alerts._clean_error`) used by `/system`, `/llm`, `/ups`; new test.
- Fixed: count sparklines autoscaled rollup averages (queue max 0.1379) into full-height spikes → `sparkline({minSpan})`, `minSpan: 1` on sessions/queued (spike 30 px → 4 px); assertions added.
- Tests: `bash .agents/validate.sh` → **EXIT=0** (`uv run pytest --cov --cov-fail-under=94`: **485 passed, 7 skipped**, coverage **96.87%**; `ruff check src tests agent`: All checks passed; `pyright`: 0 errors; `node --check` all static JS clean; repo hygiene clean; `uv pip list --outdated` empty). `uv run pytest -m live --live` → **7 passed**.
- Completion criteria: heroes largest per engine ✔ (38.4 px vs 16.8 px row values); sparkline per cell + selected window ✔ (5/5 gufo, 7/7 vLLM; 6 h chip moved KV max 53 %→58 %); mean+p95 without hover ✔ (243 / 48.8 / 285 ms feet); provenance tag ✔ (`Δ10.0s`/`Δ8.0s` + `title`=exact source); single-engine ✔ (1 panel, totals hidden, zero red); screenshots ✔ `data/logs/llm-verify-{both-engines,charts-and-slots,single-engine,monitor-reference}.png` + `NOTES-chart-set.md`; pytest/cov/ruff/pyright ✔; earlier phases unchanged-behaviour ✔ (all 01–03 tests pass).
- Deviation (documented, not code): throughput heroes stay **neutral ink** — `severity.py` has no throughput row and 0 tok/s idle is healthy; colour comes from rated cells (KV) + reachability. Kept per `NOTES-chart-set`.
- Deviation: could not append the task-04 written verdict to `NOTES-chart-set.md` (it lives in `todo/`, which I must not edit). Verdict: ours reads better than `:8095` — bigger throughput numbers, per-cell shape over the selected window, percentiles inline; `llm_monitor` only sparklines decode+KV and has no window control.
- Docs: AGENTS.md updated (485 test count, `sparkline` `minSpan` rule, `Dash.eventText` event-row rule, `/llm` panel `items-start`); README reviewed in full — commands, config table (all `config.py` vars present), architecture bullets, page descriptions verified accurate; no other inconsistencies found.
- Note for the harness: `.agents/phases/todo/03…/NOTES-verify.md` and `todo/04…/NOTES-chart-set.md` are untracked notes left behind by their phases (task files moved, notes did not).
- Next pending phase: `05_ups_hero_battery_glyph`.
This commit is contained in:
+277
-6
@@ -24,6 +24,7 @@ from dashboards.app import create_app
|
||||
from dashboards.config import Settings, Target
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources import nut
|
||||
from dashboards.sources.llm import HISTORY_METRICS
|
||||
from dashboards.store import Event, Sample, SampleStore
|
||||
|
||||
# refused immediately, and never dials out of the test environment
|
||||
@@ -576,15 +577,15 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
|
||||
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
|
||||
"""The old call sites still work until the remaining phases convert each page.
|
||||
|
||||
`/` was converted in phase 02 and `/system` in phase 03: both colour themselves from
|
||||
the `severity` verdicts the API already sends, so they carry no threshold of their
|
||||
own. The two pages still to come must keep compiling against the fallback until they
|
||||
are converted — deleting it early would break a page mid-migration.
|
||||
`/` was converted in phase 02, `/system` in phase 03 and `/llm` in phase 04: each
|
||||
colours itself from the `severity` verdicts the API already sends, so it carries no
|
||||
threshold of its own. `/ups` is phase 05's job and must keep compiling against the
|
||||
fallback until it is converted — deleting it early would break that page.
|
||||
"""
|
||||
js = client.get("/static/app.js").text
|
||||
assert "FALLBACK ONLY" in js
|
||||
for page in ("/static/page-llm.js", "/static/page-ups.js"):
|
||||
assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site"
|
||||
assert "severityFor" in client.get("/static/page-ups.js").text, \
|
||||
"/ups still uses the fallback until phase 05 converts it"
|
||||
|
||||
|
||||
def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> None:
|
||||
@@ -786,6 +787,222 @@ def test_no_per_core_visual_library_survives_the_page(client) -> None:
|
||||
assert name in client.get("/static/charts.js").text
|
||||
|
||||
|
||||
def test_sparkline_ships_the_fixed_scale_and_verdict_options(client) -> None:
|
||||
"""Phase 04 turns `/llm` cells into sparkline rows, so the shared `sparkline()` must
|
||||
behave like `llm_monitor`'s `drawSpark` (the task-02 contract, NOTES-chart-set).
|
||||
|
||||
Pinned on the served bytes because there is no JS test runner (a locked anchor). What
|
||||
matters is not the option names but the semantics behind each one: a percent metric
|
||||
that silently rescaled to its own data would make an 8 % KV cache and a 92 % spike the
|
||||
same squiggle, and a stroke coloured from a page-invented threshold would break the
|
||||
one-colour-language rule the same way.
|
||||
"""
|
||||
charts_js = client.get("/static/charts.js").text
|
||||
body = charts_js[charts_js.index("function sparkline("):charts_js.index("window.DashCharts")]
|
||||
|
||||
# the phase grep criterion, as a test: the options are really in the shipped file
|
||||
hits = [ln for ln in charts_js.splitlines()
|
||||
if re.search(r"fixedMax|align|severity", ln)]
|
||||
assert len(hits) >= 3, "the sparkline upgrade is not in the served charts.js"
|
||||
for opt in ("fixedMax", "align", "labels", "severity", "minPoints", "span", "format",
|
||||
"minSpan"):
|
||||
assert opt in body, f"sparkline() lost its `{opt}` option"
|
||||
|
||||
# fixedMax: the data clamps to the axis, never the axis to the data
|
||||
assert "Math.min(Math.max(v, lo), hi)" in body, \
|
||||
"a fixedMax sparkline rescales to its own data again"
|
||||
# align:'right' grows within a real time window (span seconds), not a sample count
|
||||
assert "x1 - o.span" in body, "align:'right' forgot the window it grows inside"
|
||||
# verdict colour comes from the theme tokens only; unrated is faint grey, never green
|
||||
for token in ("cssVar('--ok')", "cssVar('--warn')", "cssVar('--fail')",
|
||||
"cssVar('--text-faint')"):
|
||||
assert token in body, f"sparkline() stopped taking its colour from {token}"
|
||||
assert ".sev-" not in body, "sparkline invented a parallel severity vocabulary"
|
||||
# the `now / max` foot llm_monitor prints, via the caller's format callback
|
||||
assert "now ${fmt(last.value)}" in body and "max ${fmt(dataMax)}" in body
|
||||
# never a confident line through one point, and the foot quotes data, not the padded axis
|
||||
assert "minPoints: 2" in body and "no history" in body
|
||||
# ASSUMPTION (task 02): no animation at all, so prefers-reduced-motion holds by design
|
||||
assert "transition:" not in body and "animation:" not in body
|
||||
|
||||
|
||||
# ---- the `/llm` layout (phase 04 task 03) ---------------------------------
|
||||
#
|
||||
# The drift this phase fixes was a 14-cell grid where everything was the same size and
|
||||
# nothing showed recent shape. These tests pin the *shape* the page committed to on the
|
||||
# served bytes (a locked anchor: no JS test runner), so a regression to "all cells equal"
|
||||
# fails a test rather than a screenshot.
|
||||
|
||||
def test_llm_page_colours_from_the_payload_not_from_js_thresholds(client) -> None:
|
||||
"""`page-llm.js` paints the KV verdict `severity.py` produced; it owns no threshold.
|
||||
|
||||
Both `severityFor(value, 75, 90)` call sites are gone (the task's explicit work item):
|
||||
the KV row takes `severity` from the rated ratio cell, the fleet KV bar was deleted
|
||||
with the summed card, and every unrated number — throughput above all, because 0 tok/s
|
||||
on an idle server is healthy — draws no colour at all.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
assert "severityFor(" not in js, "the LLM page re-invented a threshold"
|
||||
assert "severityFrom" in js
|
||||
# the KV verdict comes from the payload cells `sources/llm.py` rates
|
||||
kv = js[js.index("case 'cache':"):js.index("case 'ttft':")]
|
||||
assert "kv_cache_usage_ratio" in kv and "gpu_cache_usage" in kv and "severity" in kv
|
||||
|
||||
|
||||
def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
|
||||
"""Decode and prompt are the largest elements per engine; the rest is visibly secondary.
|
||||
|
||||
The hero row is `Dash.hero()` markup (2.4rem via `.hero`), built from the `hero: true`
|
||||
flags in `CELLS`, and throughput carries **no** severity — `severity.py` has no
|
||||
throughput row and the page invents none (the neutral-ink decision in NOTES-chart-set,
|
||||
mirroring the `/system` NET precedent). The provenance tag stays in the hero's `.sub`
|
||||
with the exact `source` as the tooltip: removing it to look cleaner is a regression.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
panel = js[js.index("function enginePanel("):js.index("function renderPanel(")]
|
||||
assert 'data-hero="${cell.key}"' in panel, "the hero row is not flag-driven markup"
|
||||
assert '<div class="heroes">' in panel, "the hero row left the engine panel"
|
||||
heroes = re.findall(r"{ key: '(\w+)'[^}]*?hero: true", js)
|
||||
assert heroes == ["decode", "prompt"], "the hero row is no longer the two throughputs"
|
||||
# driven by flags, not fourteen identical cells
|
||||
assert "SPARK_CELLS" in js and "ALARM_CELLS" in js
|
||||
assert js.count("spark: '") >= 6, "the compact sparkline list shrank back to a grid"
|
||||
# the hero call must not pass a severity — unrated by design
|
||||
hero_call = js[js.index("const node = hero("):js.index("if (node) node.title")]
|
||||
assert "severity" not in hero_call, "throughput invented a verdict"
|
||||
# provenance intact: the tag is generated from the cell's `source`, never hardcoded
|
||||
assert "provenance" in js and "Δ" in js
|
||||
assert "node.title = value.source" in js
|
||||
|
||||
|
||||
def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client) -> None:
|
||||
"""Each compact cell draws its own sparkline over the *selected* window; latency cells
|
||||
print mean **and** p95 without a hover.
|
||||
|
||||
The window is honest time (`align: 'right'` + `span`), percent-like metrics keep a
|
||||
fixed 0-100 axis, and the p95 comes from the snapshot's histogram cells — p95 needs no
|
||||
stored series. The `only` scopes are what keep a gufo panel from showing a latency
|
||||
mystery and a vLLM panel from showing a spec-decode row it cannot fill.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
sparks = js[js.index("function drawSparks("):js.index("function renderFleet(")]
|
||||
assert "DashCharts.sparkline(" in sparks
|
||||
for opt in ("align: 'right'", "span: state.range", "fixedMax", "labels: { max: true }"):
|
||||
assert opt in sparks, f"cell sparklines lost `{opt}`"
|
||||
# percent cells pin the axis; ratio series are scaled to the printed percent
|
||||
assert js.count("fixedMax: 100") >= 3, "KV / prefix-hit / spec-accept lost their 0-100 axis"
|
||||
assert "scale: 1000" in js and "scale: 100" in js
|
||||
# every spark cell declares its history series in the markup it is built from
|
||||
assert 'data-spark data-metric="${cell.spark}"' in js
|
||||
# percentile foot lines: mean big in the row, p95 beneath, n/a when nothing measured
|
||||
assert "function p95Value(" in js
|
||||
for name in ("time_to_first_token_seconds__p95", "time_per_output_token_seconds__p95",
|
||||
"request_queue_time_seconds__p95"):
|
||||
assert name in js, f"{name} no longer feeds a foot line"
|
||||
assert "p95 n/a" in js
|
||||
assert 'data-foot' in js
|
||||
# engine-scoped rows: gufo never grows a TTFT row, vLLM never a spec-accept row
|
||||
assert "only: ['vllm']" in js and "only: ['gufo']" in js
|
||||
scoped = js[js.index("function sparkCellsFor("):js.index("function enginePanel(")]
|
||||
assert "cell.only" in scoped
|
||||
# a count row never gets an axis finer than one request: history arrives as rollups,
|
||||
# so 8 s of a single queued request averages to 0.14 and a fitted axis draws a spike for
|
||||
# a queue nobody was in — the twin of the rescale `fixedMax` fixes
|
||||
assert _code(js).count("minSpan: 1") == 2, "sessions/queued lost their one-request axis floor"
|
||||
assert "minSpan: cell.minSpan" in sparks, "cell minSpan never reaches the sparkline"
|
||||
|
||||
|
||||
def test_llm_sparklines_and_charts_cost_one_batched_history_request(client) -> None:
|
||||
"""One `/api/llm/history` call per refresh feeds charts *and* every sparkline row.
|
||||
|
||||
The endpoint answers a metric list for every engine, and unknown names come back as
|
||||
empty series on purpose — so the page asks for the full cell set once instead of one
|
||||
XHR per cell, and the two series phase 04 added (`tpot_seconds`, `spec_accept_rate`)
|
||||
must be in that list or those rows are permanent "no history" slots.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
# comments aside, exactly one fetch hits the endpoint: one request, every metric
|
||||
calls = re.findall(r"fetchJSON\(\s*`/api/llm/history", _code(js))
|
||||
assert len(calls) == 1, "the page makes more than one history request again"
|
||||
assert "HISTORY_SERIES.join" in js and "SPARK_CELLS.map((cell) => cell.spark)" in js
|
||||
request = js[js.index("async function refreshHistory("):js.index("function drawSparks(")]
|
||||
assert "metrics=${HISTORY_SERIES.join" in request
|
||||
assert "range=${state.range}" in request
|
||||
# and the endpoint really answers every name the page asks for
|
||||
for name in ("tpot_seconds", "spec_accept_rate", "ttft_seconds", "queue_seconds",
|
||||
"kv_cache_usage", "requests_running", "requests_waiting",
|
||||
"prefix_cache_hit_ratio"):
|
||||
assert name in HISTORY_METRICS, f"llm.{name} is not stored, so its spark is dead"
|
||||
|
||||
|
||||
def test_llm_alarm_cells_exist_only_while_there_is_an_incident(client) -> None:
|
||||
"""`preemptions` / `device_lost` are absent at zero and red chips above zero.
|
||||
|
||||
The rule is the *existence of an incident*, not a graded threshold — `sources/llm.py`
|
||||
already treats any non-zero `gufo_device_lost_total` as one. The chip reuses the shared
|
||||
`.chip.fail` vocabulary; nothing new is painted, and a container emptied at zero means
|
||||
the row does not exist rather than showing a permanent ignored 0.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
alarms = js[js.index("function renderAlarms("):js.index("function renderSlots(")]
|
||||
assert "class=\"chip fail\"" in alarms, "the alarm chip left the shared fail vocabulary"
|
||||
assert "value.value > 0" in alarms
|
||||
assert "host.hidden = chips.length === 0" in alarms
|
||||
assert 'data-alarm="${cell.key}"' in alarms
|
||||
assert js.count("alarm: true") == 2
|
||||
|
||||
|
||||
def test_llm_totals_card_says_only_what_no_engine_panel_says(client) -> None:
|
||||
"""Engines answering, active, queued — and hidden below two engines.
|
||||
|
||||
The summed decode/prompt and the un-rated fleet KV bar are gone: per-engine is the
|
||||
honest view (the fleet KV bar had no `severity` to render, which is invariant-grade
|
||||
trouble), and a cross-engine sum of one engine just duplicates that engine's own card
|
||||
(the `llm_monitor` rule). `/api/llm/snapshot` keeps every field — this is a page
|
||||
decision, not an API one.
|
||||
"""
|
||||
html = _code(client.get("/llm").text)
|
||||
for gone in ("f-decode", "f-prompt", "f-kv", "f-tokens", "seam-note"):
|
||||
assert gone not in html, f"/llm still totals {gone}"
|
||||
for kept in ("f-live", "f-active", "f-waiting", 'id="totals"'):
|
||||
assert kept in html, f"/llm lost {kept}"
|
||||
js = client.get("/static/page-llm.js").text
|
||||
assert "card.hidden = total < 2" in js, "the totals card no longer hides for one engine"
|
||||
|
||||
|
||||
def test_llm_charts_are_secondary_and_named(client) -> None:
|
||||
"""The four trend charts stay, at half height, *below* the engine panels, each named.
|
||||
|
||||
Half of the old 165px and moved after `#engines` is the "cells read first" decision;
|
||||
every `LineChart` gets an `ariaLabel` because the legend is a sibling that labels
|
||||
nothing (`/system` did this in phase 03, `/llm` never had).
|
||||
"""
|
||||
html = _code(client.get("/llm").text)
|
||||
assert html.index('id="engines"') < html.index('id="chart-decode"'), \
|
||||
"the charts are back on top of the engine panels"
|
||||
for chart in ("chart-decode", "chart-prompt", "chart-kv", "chart-queue"):
|
||||
assert f'id="{chart}"' in html, f"/llm lost the {chart} chart"
|
||||
js = client.get("/static/page-llm.js").text
|
||||
assert "height: 110" in js and "height: 165" not in js
|
||||
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
|
||||
assert charts.count("ariaLabel:") == 4, "an /llm chart lost its accessible name"
|
||||
|
||||
|
||||
def test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling(client) -> None:
|
||||
"""A 5-row gufo panel beside a 7-row vLLM panel keeps its own height.
|
||||
|
||||
Found while comparing against `llm_monitor`: the shorter engine card grew 172 px of
|
||||
blank card, and a blank panel reads as "no data" — the exact misreading the unconfigured
|
||||
rules exist to prevent. Pinned on the served markup so the `.grid.items-start` rule in
|
||||
`dash.css` cannot silently become dead CSS (and the void cannot silently come back).
|
||||
"""
|
||||
html = _code(client.get("/llm").text)
|
||||
engines_div = html[html.index('<div class="grid'):html.index('id="engines"') + 40]
|
||||
assert "items-start" in engines_div, "the engine grid stretches its cards again"
|
||||
css = client.get("/static/dash.css").text
|
||||
assert ".grid.items-start" in css, "the rule the engine grid relies on is gone"
|
||||
|
||||
|
||||
def test_overview_is_exactly_two_panels(client) -> None:
|
||||
"""The landing page answers one question: what is wrong, and what is the fleet doing.
|
||||
|
||||
@@ -1019,6 +1236,38 @@ def test_llm_history_charts_both_engines_by_default(
|
||||
("vllm", "llm.kv_cache_usage_ratio")}
|
||||
|
||||
|
||||
def test_one_batched_llm_history_request_answers_every_cell_series(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The exact query `/llm` issues per refresh returns every chart *and* sparkline series.
|
||||
|
||||
Phase 04 sends one request carrying the chart trio and the whole spark cell list —
|
||||
including the two series the layout required (`tpot_seconds` for the vLLM TPOT row,
|
||||
`spec_accept_rate` for the gufo spec row). A name missing here is a cell that reads
|
||||
"no history" forever in the browser, which is the dead-cell anti-pattern the phase
|
||||
wrote NOTES-chart-set to prevent; a rename in `sources/llm.py` fails here first.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
|
||||
vllm_url=DEAD_URL) as test_client:
|
||||
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
|
||||
now = time.time()
|
||||
metrics = ["decode_tokens_per_second", "prompt_tokens_per_second",
|
||||
"kv_cache_usage", "requests_running", "requests_waiting",
|
||||
"ttft_seconds", "tpot_seconds", "queue_seconds",
|
||||
"prefix_cache_hit_ratio", "spec_accept_rate"]
|
||||
for metric in metrics:
|
||||
store.add("llm", "gufo", f"llm.{metric}", 1.5, ts=now - 60)
|
||||
store.add("llm", "vllm", f"llm.{metric}", 2.5, ts=now - 30)
|
||||
body = test_client.get("/api/llm/history", params={
|
||||
"engines": "gufo,vllm", "metrics": ",".join(metrics), "range": "1h"}).json()
|
||||
assert body["range_seconds"] == 3_600.0
|
||||
by_key = {(item["host"], item["metric"]) for item in body["series"]}
|
||||
assert by_key == {(engine, f"llm.{metric}")
|
||||
for engine in ("gufo", "vllm") for metric in metrics}
|
||||
points = {(item["host"], item["metric"]): item["points"] for item in body["series"]}
|
||||
assert points[("gufo", "llm.spec_accept_rate")][0]["value"] == pytest.approx(1.5)
|
||||
assert points[("vllm", "llm.tpot_seconds")][0]["value"] == pytest.approx(2.5)
|
||||
|
||||
|
||||
def test_metrics_endpoint_lists_what_actually_has_history(client) -> None:
|
||||
body = client.get("/api/metrics").json()
|
||||
assert body["metrics"] == []
|
||||
@@ -1039,6 +1288,28 @@ def test_events_are_returned_in_json(tmp_path: Path, monkeypatch: pytest.MonkeyP
|
||||
store.close()
|
||||
|
||||
|
||||
def test_event_rows_say_their_subject_once(client) -> None:
|
||||
"""`poller.py` messages name their own subject, and the row prints a host chip too.
|
||||
|
||||
"gufo gufo is answering again after 1 failure" reads as two problems — the same defect
|
||||
`alerts._clean_error()` already fixes for the overview's alert list. `/ups` had the
|
||||
mirror-image bug: it `replace()`d `host + ':'` *anywhere*, so "Quick battery test sent
|
||||
to apc: OK" printed as "Quick battery test sent to OK". All three feeds now share
|
||||
`Dash.eventText()`, which strips the wrapper and a **leading** name only.
|
||||
"""
|
||||
app_js = client.get("/static/app.js").text
|
||||
helper = app_js[app_js.index("function eventText("):app_js.index("// ---- DOM helpers")]
|
||||
assert "eventText" in app_js[app_js.index("return {"):], "eventText is not exported"
|
||||
assert "^UpstreamError:\\s+" in helper, "the wrapper `alerts.py` drops is back"
|
||||
assert "new RegExp" in helper and "stripped.trim() ? stripped : out" in helper, \
|
||||
"the host strip is no longer anchored to the front, or can empty a message"
|
||||
for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"):
|
||||
js = client.get(page).text
|
||||
assert "eventText(event)" in js, f"{page} prints the raw message again"
|
||||
assert "replace(event.host" not in client.get("/static/page-ups.js").text, \
|
||||
"/ups is mangling the host name mid-message again"
|
||||
|
||||
|
||||
# ---- the one write endpoint ------------------------------------------------ #
|
||||
|
||||
def test_command_endpoint_is_post_only(ups_client) -> None:
|
||||
|
||||
@@ -231,6 +231,33 @@ def test_engine_result_charts_a_measured_rate_after_the_second_poll(metrics_text
|
||||
assert any(s.metric == "llm.decode_tokens_per_second" for s in result.samples)
|
||||
|
||||
|
||||
def test_vllm_stores_the_tpot_series_the_cell_sparkline_draws(metrics_text) -> None:
|
||||
"""Phase 04 added `llm.tpot_seconds` so the vLLM TPOT row has a shape, not a dead slot.
|
||||
|
||||
Hermetic twin of the live check: the captured exposition must resolve the histogram the
|
||||
page charts, or `/llm` shows a permanent "no history" next to a real mean.
|
||||
"""
|
||||
result = engine_result("vllm", "http://v:1", metrics_text("vllm"), RateEstimator(), now=1.0)
|
||||
mean = result.payload["metrics"]["time_per_output_token_seconds__mean"]["value"]
|
||||
assert mean is not None, "the vLLM capture lost its TPOT histogram — re-capture fixtures"
|
||||
stored = {s.metric: s.value for s in result.samples}
|
||||
assert stored["llm.tpot_seconds"] == pytest.approx(mean)
|
||||
# gufo publishes no output-token histogram, so it stores no TPOT series (never a 0)
|
||||
gufo = engine_result("gufo", "http://g:1", metrics_text("gufo"), RateEstimator(), now=1.0)
|
||||
assert "llm.tpot_seconds" not in {s.metric for s in gufo.samples}
|
||||
|
||||
|
||||
def test_gufo_stores_the_spec_accept_series_the_cell_sparkline_draws(metrics_text) -> None:
|
||||
"""`llm.spec_accept_rate` is the gufo-only half of the phase-04 history addition."""
|
||||
result = engine_result("gufo", "http://g:1", metrics_text("gufo"), RateEstimator(), now=1.0)
|
||||
ratio = result.payload["metrics"]["spec_accept_rate"]["value"]
|
||||
assert ratio is not None and 0.0 < ratio <= 1.0
|
||||
stored = {s.metric: s.value for s in result.samples}
|
||||
assert stored["llm.spec_accept_rate"] == pytest.approx(ratio)
|
||||
vllm = engine_result("vllm", "http://v:1", metrics_text("vllm"), RateEstimator(), now=1.0)
|
||||
assert "llm.spec_accept_rate" not in {s.metric for s in vllm.samples}
|
||||
|
||||
|
||||
def test_sessions_pass_through_slots_for_gufo() -> None:
|
||||
slots = [{"id": 0, "idle": False, "n_prompt": 12, "n_predict": 40}]
|
||||
result = engine_result("gufo", "http://g:1", "", RateEstimator(), slots=slots, now=1.0)
|
||||
|
||||
Reference in New Issue
Block a user