services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 07_llm_monitor_style_pages

Browse Source 
**Phase 07 final verification pass — all tasks already in `complete/`; every criterion re-verified live.**

- **Old slot table gone**: no `#slots-card`/`#slots-table`/`renderSlots` in HTML/JS/CSS (only the new grid's `.slots-empty` *class*); pinned by `test_the_broken_slot_table_is_gone_from_html_js_and_css`.
- **`tests/fixtures/slots_gufo.json`**: present; live `GET ${GUFO_BASE_URL}/slots` this pass matches it key-for-key (row + `next_token[0]`), so the capture is reproducible; drift proven to fail (mutation tests + live `test_gufo_slots_answer_the_pinned_shape`, 9 live tests pass).
- **Aggregate block**: live page shows `2 of 2 engines`, running/waiting/prompt/decode/total-tokens straight from `fleet_totals()`, dot `ok`, no verdict colour; on a throwaway single-engine instance (:8091, private temp DB) `#fleet` is `hidden`, one full-width strip card + one panel, zero banners → "one clean half".
- **Strip + slot grid on live data**: 2 `.card.stat` (KV bars `bar ok` 13 %/0 %), gufo grid rendered real cards incl. a busy slot (`#2189`, 67849 prompt tok, 99 % cache share, `processing`), vLLM note-only with no grid; no `NaN`/`undefined` anywhere. Screenshots: `data/logs/llm-phase07-final-live-{two-engines,single-engine}.png`, `ups-charts-aria-labels.png`.
- **Defect found in earlier work and fixed**: `/ups`' four `LineChart`s shipped no `ariaLabel`, violating AGENTS.md's "every chart names itself for assistive tech" (`/system`+`/llm` were pinned, `/ups` wasn't) → added labels to `page-ups.js` and `test_every_ups_chart_names_itself_for_assistive_tech`; browser-verified `role="img"` + label on all four.
- **Gate**: `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **604 passed, 9 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → **0 errors**; `node --check` every page script ok; `uv run pytest -m live --live` → **9 passed**; `uv pip list --outdated` empty.
- **Docs**: README + AGENTS.md reviewed end-to-end; fixed test count (582→604), `/llm` source column + both architecture diagrams now name gufo `/slots`, AGENTS.md agent line gained `GET /schema` + status page, ariaLabel convention now covers all three chart pages, README Testing block now lists `bash .agents/validate.sh`. Env-var table, agent/install flags, Containerfile facts, `/api/*` shapes, range chips, remote URL all verified accurate — no other changes needed.
- **Live-infra**: no fleet/host/config change; the temp :8091 instance, its temp DB and the fake processes are removed; dev server restarted and `/api/health` `ok` (all 6 sources ok). One thing I deliberately did **not** touch: the gitignored local `.env.dev` still carries a comment claiming the NUT `[dashboards]` account has only battery-test `instcmds`, whereas AGENTS.md/README record `instcmds = all` (verified on hardware 2026-10-17) — operator-owned file, flag for the operator to reword.
- **Next pending phase**: `08_events_paging_and_copy`.
This commit is contained in:
ducoterra
2026-10-17 17:50:19 -04:00
parent de325caa49
commit 3d5ccd7a39
43 changed files with 2544 additions and 210 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_api.py
+754 -30
View File
@@ -24,8 +24,22 @@ from dashboards.app import create_app
from dashboards.config import Settings, Target
from dashboards.severity import classify
from dashboards.sources import nut
from dashboards.sources.llm import HISTORY_METRICS
from dashboards.sources.llm import (
HISTORY_METRICS,
parse_gufo_metrics,
parse_vllm_metrics,
)
from dashboards.store import Event, Sample, SampleStore
from slots_contract import (
CANONICAL_TASK_ID_FIELD,
OPTIONAL_NEXT_TOKEN_FIELDS,
OPTIONAL_ROW_FIELDS,
REQUIRED_NEXT_TOKEN_FIELDS,
REQUIRED_ROW_FIELDS,
TASK_ID_ALIASES,
USER_CONTENT_FIELDS,
load_slots_capture,
)
# refused immediately, and never dials out of the test environment
DEAD_URL = "http://127.0.0.1:1"
@@ -547,6 +561,42 @@ def _code(body: str) -> str:
return re.sub(r"^\s*//.*$", " ", body, flags=re.M)
# The `fleet` object paths `/llm`'s aggregate block reads. `engines_total` drives the
# "hidden below two engines" rule and the tag; `engines_live` is what makes a partial sum
# visible; the rest are the row itself.
FLEET_BLOCK_PATHS = frozenset({
"engines_total", "engines_live", "requests_active", "requests_waiting",
"prompt_tokens_per_second", "decode_tokens_per_second", "tokens_total",
})
# The summed half of that list — the ones that must be null, not 0, with nothing answering.
FLEET_SUM_PATHS = ("requests_active", "requests_waiting", "prompt_tokens_per_second",
"decode_tokens_per_second", "tokens_total")
def _fleet_tag(html: str) -> str:
"""The aggregate block's opening tag — the one place its `hidden` default lives."""
start = html.index('<section class="card flat fleet"')
return html[start:html.index(">", start)]
def _fleet_render_path(js: str) -> str:
"""Just the body of `renderFleet()` — the aggregate block's whole code path, comments out.
Everything the block writes goes through here, so it is the slice that can be asserted
verdict-free without tripping over the engine panels' rated KV sparkline next door.
"""
js = _code(js)
return js[js.index("function renderFleet("):js.index("async function refresh(")]
def _fleet_field_metrics(js: str) -> list[str]:
"""The `fleet` keys the block's rows are bound to, in the order the block prints them."""
js = _code(js)
start = js.index("const FLEET_FIELDS = [")
table = js[start:js.index("];", start) + 2]
return re.findall(r"metric: '([a-z_]+)'", table)
def test_no_parallel_severity_vocabulary_exists_anywhere(client) -> None:
"""One word per colour: `ok|warn|fail`, as in severity.py and the existing components."""
for asset in (*ASSETS, "/", "/system", "/llm", "/ups"):
@@ -1219,6 +1269,22 @@ def test_ups_hero_row_makes_room_for_the_glyph(client) -> None:
assert ".ups-hero" in block and "minmax(0, 1fr)" in block, "the row never stacks"
def test_every_ups_chart_names_itself_for_assistive_tech(client) -> None:
"""The four trend plots on `/ups` carry an accessible name, like every other chart here.
`AGENTS.md` states the convention as *every* chart names itself: `LineChart` puts
`role="img"` + `aria-label` on the `<svg>` when the caller passes `ariaLabel`, because
the legend is a sibling that labels nothing and an unlabelled plot is silence. `/system`
(phase 03) and `/llm` (phase 04) were pinned; these four were the stragglers, so the
rule is now pinned on every page that draws one.
"""
js = client.get("/static/page-ups.js").text
charts = js[js.index("function makeCharts("):js.index("function points(")]
assert charts.count("ariaLabel:") == 4, "a `/ups` chart lost its accessible name"
for phrase in ("Battery charge", "Battery runtime", "Battery and line voltage", "UPS load"):
assert phrase in charts, f"the {phrase!r} chart names nothing readable"
# ---- the `/llm` layout (phase 04 task 03) ---------------------------------
#
# The drift this phase fixes was a 14-cell grid where everything was the same size and
@@ -1257,9 +1323,12 @@ def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
assert '<div class="heroes">' in panel, "the hero row left the engine panel"
heroes = re.findall(r"{ key: '(\w+)'[^}]*?hero: true", js)
assert heroes == ["decode", "prompt"], "the hero row is no longer the two throughputs"
# driven by flags, not fourteen identical cells
# driven by flags, not fourteen identical cells. Phase 07 task 03 lifted the summary rows
# out of the panel into the stats strip above it, so what is left here is the pair plus
# the latency rows the strip does not print — never a fourteen-cell grid again.
assert "SPARK_CELLS" in js and "ALARM_CELLS" in js
assert js.count("spark: '") >= 6, "the compact sparkline list shrank back to a grid"
assert js.count("spark: '") >= 2, "the panel lost the latency sparkline rows it kept"
assert js.count("hero: true") == 2, "the hero row is no longer exactly two numbers"
# the hero call must not pass a severity — unrated by design
hero_call = js[js.index("const node = hero("):js.index("if (node) node.title")]
assert "severity" not in hero_call, "throughput invented a verdict"
@@ -1269,22 +1338,32 @@ def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client) -> None:
"""Each compact cell draws its own sparkline over the *selected* window; latency cells
print mean **and** p95 without a hover.
"""Each panel row draws its own sparkline over the *selected* window and prints mean
**and** p95 without a hover.
The window is honest time (`align: 'right'` + `span`), percent-like metrics keep a
fixed 0-100 axis, and the p95 comes from the snapshot's histogram cells — p95 needs no
stored series. The `only` scopes are what keep a gufo panel from showing a latency
mystery and a vLLM panel from showing a spec-decode row it cannot fill.
The window is honest time (`align: 'right'` + `span`) and the p95 comes from the
snapshot's histogram cells — p95 needs no stored series. The `only` scopes are what keep
a gufo panel from showing a latency mystery and a vLLM strip card from showing a
spec-decode row it cannot fill.
Phase 07 task 03 moved the percent and count rows to the stats strip, so the plotted
percent series left the cell list and the axis guarantee moved with it: the KV trend
chart pins `yMax: 100` (an 8 % window that once held a 92 % spike must not read flat),
and the strip converts its 0-1 ratios to per cent in `cellValue()` — a bar cannot
autoscale, so it cannot lie that way either. The `fixedMax`/`minSpan` options stay
plumbed through `drawSparks()` and are pinned on the component itself by
`test_sparkline_ships_the_fixed_scale_and_verdict_options`: a count sparkline that comes
back must not get an axis finer than one request.
"""
js = client.get("/static/page-llm.js").text
sparks = js[js.index("function drawSparks("):js.index("function renderFleet(")]
assert "DashCharts.sparkline(" in sparks
for opt in ("align: 'right'", "span: state.range", "fixedMax", "labels: { max: true }"):
assert opt in sparks, f"cell sparklines lost `{opt}`"
# percent cells pin the axis; ratio series are scaled to the printed percent
assert js.count("fixedMax: 100") >= 3, "KV / prefix-hit / spec-accept lost their 0-100 axis"
assert "scale: 1000" in js and "scale: 100" in js
# the one percent series still plotted owns a fixed 0-100 axis; seconds stay ms
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
assert "yMax: 100" in charts, "the KV trend chart no longer pins its 0-100 axis"
assert "scale: 1000" in js and "cell.scale || 1" in sparks
# every spark cell declares its history series in the markup it is built from
assert 'data-spark data-metric="${cell.spark}"' in js
# percentile foot lines: mean big in the row, p95 beneath, n/a when nothing measured
@@ -1298,10 +1377,6 @@ def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client)
assert "only: ['vllm']" in js and "only: ['gufo']" in js
scoped = js[js.index("function sparkCellsFor("):js.index("function enginePanel(")]
assert "cell.only" in scoped
# a count row never gets an axis finer than one request: history arrives as rollups,
# so 8 s of a single queued request averages to 0.14 and a fitted axis draws a spike for
# a queue nobody was in — the twin of the rescale `fixedMax` fixes
assert _code(js).count("minSpan: 1") == 2, "sessions/queued lost their one-request axis floor"
assert "minSpan: cell.minSpan" in sparks, "cell minSpan never reaches the sparkline"
@@ -1335,32 +1410,681 @@ def test_llm_alarm_cells_exist_only_while_there_is_an_incident(client) -> None:
already treats any non-zero `gufo_device_lost_total` as one. The chip reuses the shared
`.chip.fail` vocabulary; nothing new is painted, and a container emptied at zero means
the row does not exist rather than showing a permanent ignored 0.
Phase 07 task 03 renders them on the engine's strip card (the summary you read first)
instead of inside the panel, so the two `alarm: true` rows live in `STRIP_ROWS` and
`ALARM_CELLS` filters them out of it — counted with the comments stripped, because the
docstring on that table explains the flag by name.
"""
js = client.get("/static/page-llm.js").text
alarms = js[js.index("function renderAlarms("):js.index("function renderSlots(")]
alarms = js[js.index("function renderAlarms("):js.index("function makeCharts(")]
assert "class=\"chip fail\"" in alarms, "the alarm chip left the shared fail vocabulary"
assert "value.value > 0" in alarms
assert "host.hidden = chips.length === 0" in alarms
assert 'data-alarm="${cell.key}"' in alarms
assert js.count("alarm: true") == 2
assert _code(js).count("alarm: true") == 2
assert "ALARM_CELLS = STRIP_ROWS.filter" in js, "the alarm rows are not strip rows"
# the panel no longer carries a second container for the same incident
panel = js[js.index("function enginePanel("):js.index("function renderPanel(")]
assert 'data-k="alarms"' not in panel, "the alarm chips are drawn twice"
strip = js[js.index("function stripCard("):js.index("function renderStripCard(")]
assert 'data-k="alarms"' in strip, "the alarm chips left the strip card"
def test_llm_totals_card_says_only_what_no_engine_panel_says(client) -> None:
"""Engines answering, active, queued — and hidden below two engines.
# ---- the `/llm` stats strip (phase 07 task 03) ---------------------------
#
# `llm_monitor`'s shared strip (`index.html` ~L743), ported: one card per configured engine
# between the fleet block and the panels. Two things make it a summary instead of a second
# set of panels, and both are pinned below: every row is a cell `/api/llm/snapshot` really
# publishes, and no row is printed in the strip *and* in the panel under it.
The summed decode/prompt and the un-rated fleet KV bar are gone: per-engine is the
honest view (the fleet KV bar had no `severity` to render, which is invariant-grade
trouble), and a cross-engine sum of one engine just duplicates that engine's own card
(the `llm_monitor` rule). `/api/llm/snapshot` keeps every field — this is a page
decision, not an API one.
#: row key -> the cells `cellValue()` resolves it from (`sessions.*` is the gufo slot count,
#: which is not a metrics cell). The served `STRIP_ROWS` table must match these keys exactly.
STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
"prompt": ("prompt_tokens_per_second",),
"decode": ("decode_tokens_per_second",),
"tokens_total": ("tokens_total",),
"slots_busy": ("sessions.list", "sessions.active"),
"running": ("requests_processing", "num_requests_running"),
"waiting": ("requests_deferred", "num_requests_waiting"),
"cache_hit": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"),
"spec_accept": ("spec_accept_rate",),
"queue_mean": ("request_queue_time_seconds__mean",),
"preemptions": ("num_preemptions_total",),
"device_lost": ("device_lost_total",),
}
#: rows that are incidents, not stats: absent where the engine does not count them.
ALARM_ROW_KEYS = ("preemptions", "device_lost")
def _strip_rows(js: str) -> dict[str, str | None]:
"""The served `STRIP_ROWS` table: row key -> its `only` scope (`None` = every engine)."""
body = _code(js)
start = body.index("const STRIP_ROWS = [")
table = body[start:body.index("\n];", start)]
rows: dict[str, str | None] = {}
for chunk in table.split("{ key: '")[1:]:
scoped = re.search(r"only: \[([^\]]*)\]", chunk)
rows[chunk.split("'")[0]] = scoped.group(1) if scoped else None
return rows
def _strip_render_path(js: str) -> str:
"""Just `renderStripCard()` — the whole code path that paints one strip card."""
js = _code(js)
return js[js.index("function renderStripCard("):js.index("function pruneEngines(")]
def test_llm_stats_strip_sits_between_the_fleet_block_and_the_panels(client) -> None:
"""The strip is there, ships hidden, names itself, and appears only with engines.
An unconfigured engine gets no card and no strip (invariant 3) — the section is empty in
the markup and `page-llm.js` unhides it only when the payload has an engine to draw.
"""
html = _code(client.get("/llm").text)
for gone in ("f-decode", "f-prompt", "f-kv", "f-tokens", "seam-note"):
assert gone not in html, f"/llm still totals {gone}"
for kept in ("f-live", "f-active", "f-waiting", 'id="totals"'):
assert kept in html, f"/llm lost {kept}"
assert 'id="strip"' in html, "/llm lost the per-engine stats strip"
assert html.index('id="fleet"') < html.index('id="strip"') < html.index('id="engines"'), \
"the strip is no longer between the fleet block and the engine panels"
start = html.index('<section class="strip"')
tag = html[start:html.index(">", start)]
assert "hidden" in tag, "the strip no longer ships hidden"
assert 'aria-label="Engine summary"' in tag, "the strip is a region that names nothing"
js = client.get("/static/page-llm.js").text
assert "card.hidden = total < 2" in js, "the totals card no longer hides for one engine"
assert "$('#strip').hidden = payloads.length === 0" in js
def test_llm_strip_card_is_one_card_per_engine_of_shared_vocabulary(client) -> None:
"""`stripCard()` = `.card` + `.kv` + `.bar` + `.chip.fail`: a layout, not a new component.
Dot, name and age in the head (reachability, the same distinction the panels and the
fleet block draw), the KV bar, the rows keyed by payload field, the alarm container.
"""
js = client.get("/static/page-llm.js").text
card = js[js.index("function stripCard("):js.index("function renderStripCard(")]
assert "card.className = 'card stat'" in card
assert "card.dataset.engine = engine" in card, "the card stopped naming its engine"
assert '<span class="dot idle"></span>' in card
assert '<span class="faint mono updated"></span>' in card, "the card lost its timestamp"
assert '<span class="bar" data-kv-bar><i></i></span>' in card, "the KV bar left `.bar`"
assert '<dl class="kv">' in card, "the strip rows left the shared `.kv` vocabulary"
assert 'data-row="${row.key}"' in card
assert 'data-k="alarms"' in card
# the wrapper is why the shared `.kv` grid has to move *inside* it: applied to the
# wrappers, `max-content 1fr` lays two rows side by side and the card reads as a
# column of orphaned labels (found in the browser, not in a test failure)
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- engine stats strip"):css.index("/* ---- fleet totals")]
assert ".strip .kv { margin: 0; grid-template-columns: 1fr; }" in section
assert ".strip .kv > div {" in section
assert "grid-template-columns: max-content minmax(0, 1fr)" in section
refresh = js[js.index("async function refresh("):js.index("async function refreshEvents(")]
assert "state.strips.get(payload.engine)" in refresh, "the strip rebuilds its cards per poll"
assert "$('#strip').appendChild(strip)" in refresh
assert "renderStripCard(strip, payload, stateRow)" in refresh
prune = js[js.index("function pruneEngines("):js.index("function makeCharts(")]
assert "state.strips" in prune and "state.panels" in prune, \
"an engine that leaves the payload leaves a card behind"
def test_llm_strip_scopes_its_rows_to_the_engines_that_publish_them(client) -> None:
"""gufo gets slots and speculative decoding; vLLM gets running and queue time."""
rows = _strip_rows(client.get("/static/page-llm.js").text)
assert rows["slots_busy"] == "'gufo'" and rows["spec_accept"] == "'gufo'"
assert rows["running"] == "'vllm'" and rows["queue_mean"] == "'vllm'"
assert rows["waiting"] is None and rows["cache_hit"] is None and rows["tokens_total"] is None
js = client.get("/static/page-llm.js").text
scoped = js[js.index("function stripRowsFor("):js.index("function stripCard(")]
assert "row.only" in scoped and "row.only.indexOf(engine)" in scoped
def test_llm_strip_rows_name_cells_the_engines_publish(client, metrics_text) -> None:
"""The strip renders what the payload sends — no row whose cell no engine produces.
`STRIP_ROW_PATHS` mirrors the JS table; every metric cell is checked against the
capture-parsed metrics of each engine that would render that row, so a rename in
`sources/llm.py` fails here instead of printing a permanent `n/a` in the browser, and a
row scoped to one engine cannot leak into the other's card.
"""
rows = _strip_rows(client.get("/static/page-llm.js").text)
assert set(rows) == set(STRIP_ROW_PATHS), \
f"the served strip rows drifted: {set(rows) ^ set(STRIP_ROW_PATHS)}"
parsed = {"gufo": parse_gufo_metrics(metrics_text("gufo")),
"vllm": parse_vllm_metrics(metrics_text("vllm"))}
for key, only in rows.items():
paths = STRIP_ROW_PATHS[key]
if paths[0].startswith("sessions."):
continue # its own payload object; pinned on the JS below
shown = {engine: metrics for engine, metrics in parsed.items()
if not only or engine in only}
present = [engine for engine in shown if any(path in shown[engine] for path in paths)]
if key in ALARM_ROW_KEYS:
# an alarm the engine does not count simply is not there: the chip renders only
# while non-zero, so absence *is* the zero state rather than a dead row
assert present, f"no engine publishes {'/'.join(paths)} for the {key} alarm"
continue
assert len(present) == len(shown), (
f"the {sorted(set(shown) - set(present))} card would print `{key}` as n/a forever "
f"({'/'.join(paths)} is not published)")
def test_llm_strip_slot_count_counts_published_slots_and_never_invents_zero(client) -> None:
"""`slots busy` is a count of `/slots` rows; when /slots did not answer it is `n/a`.
Counting rows the server published is a count of published facts. An empty list from a
server that never answered is not a measurement of an idle engine, so the row falls back
to the published `sessions.active` carrying the API's own `detail` string (rule 2).
"""
js = _code(client.get("/static/page-llm.js").text)
slots = js[js.index("case 'slots_busy':"):js.index("case 'cache':")]
assert "sessions.source" in slots and "sessions.list" in slots
assert "is_processing" in slots, "the count ignores the authoritative busy verdict"
assert "sessions.detail" in slots, "the fallback lost the API's own provenance"
def test_llm_strip_kv_bar_is_coloured_by_the_payload_verdict_only(client) -> None:
"""The strip's KV bar is the page's one rated cell: `severity` in, class out.
`sources/llm.py` rates `llm.kv_cache_usage_pct`, the ratio→percent conversion the panel
cell used to make rides with it, and `Dash.severityFrom` is the only thing that decides a
colour — no threshold in the page, and the stylesheet adds no palette (`ok` stays quiet
ink, exactly like the `.cell` rows and the fleet table).
"""
body = _strip_render_path(client.get("/static/page-llm.js").text)
assert "severityFrom(kv.severity)" in body
assert "cellValue(engine, metrics, sessions, 'cache')" in body
assert "Math.max(0, Math.min(100, kv.value))" in body, "the fill stopped clamping"
assert body.count("severityFrom") == 1, "a second strip row invented a verdict"
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- engine stats strip"):css.index("/* ---- fleet totals")]
assert ".stat-bar.warn .value { color: var(--warn); }" in section
assert ".stat-bar.fail .value { color: var(--fail); }" in section
assert ".stat-bar.ok" not in section, "a healthy KV reading must stay quiet ink"
assert ".prov:empty" in section, "an empty provenance tag draws a pill of nothing"
def test_llm_strip_throughput_carries_provenance_without_colour(client) -> None:
"""`prompt tok/s` and `decode tok/s` keep the `Δ<window>` tag and stay ink.
The tag is generated from the cell's `source` (never hardcoded), the exact source rides
in the row's tooltip, and `severity.py` rates no throughput — so nothing here may paint
it (0 tok/s on an idle engine is healthy).
"""
js = client.get("/static/page-llm.js").text
body = _strip_render_path(js)
assert "row.rate ? provenance(value) : ''" in body
assert "notes.join" in body and "value.source" in body
assert "severityFor" not in js
def test_llm_strip_and_panel_print_no_value_twice(client) -> None:
"""One value, one home — the UPS hero row's rule, applied across two components.
Task 03 lifted sessions, queue depth, KV pressure, the cache-hit and spec-accept percents,
the lifetime token counter and the alarm chips out of the panel cell list rather than
drawing them twice. The two throughput heroes are the exemption the task names: the panel
keeps its hero row (so does `llm_monitor`, which carries the pair in both places), and
everything else appears once. Queue time moved with its p95 foot, so the move lost nothing.
"""
js = _code(client.get("/static/page-llm.js").text)
start = js.index("const CELLS = [")
panel_keys = set(re.findall(r"\{ key: '(\w+)'", js[start:js.index("\n];", start)]))
assert panel_keys == {"decode", "prompt", "ttft", "tpot"}, \
f"the panel cell list changed shape: {sorted(panel_keys)}"
shared = set(_strip_rows(js)) & panel_keys
assert shared == {"decode", "prompt"}, f"printed in the strip and the panel: {sorted(shared)}"
body = _strip_render_path(js)
assert "p95Value(metrics, row.key)" in body, "queue time lost its p95 in the move"
def test_llm_strip_prints_no_peak_context_number(client) -> None:
"""The phase's assumption, pinned: `llm_monitor`'s peak-ctx row does not come over.
Its cell is `llamacpp:n_tokens_max`, a since-start maximum rather than a glance number,
and the task says this strip gets no peak-context cell. The series stays stored and
queryable from `/api/llm/history` — not plotting it is not removing it.
"""
js = _code(client.get("/static/page-llm.js").text)
assert "n_tokens_max" not in js, "/llm reads the peak-context gauge again"
assert not [key for key in _strip_rows(js) if "peak" in key]
def test_llm_fleet_block_leads_the_page_with_the_engine_sums(client) -> None:
"""The page opens with llm_monitor's aggregate block: the sums no panel can say.
Phase 04 retired these numbers to counts; the operator asked for the aggregate
throughput back (`TODO.md:5`), and phase 07 restores it in `llm_monitor`'s shape —
status dot, "all engines", an `N of M engines` tag, the note that the sums cover only
the engines answering now, then running / waiting / prompt / decode / total tokens.
The counts-only card it replaces (`#totals`, `f-live`, `f-active`) is gone, ids and all:
an element nothing writes is the orphan this repo pins against.
"""
html = _code(client.get("/llm").text)
assert 'id="fleet"' in html, "/llm lost the aggregate block"
assert html.index('id="fleet"') < html.index('id="engines"'), \
"the fleet block is no longer above the engine panels"
for part in ("fleet-dot", "fleet-count", "f-running", "f-waiting", "f-prompt",
"f-decode", "f-tokens"):
assert f'id="{part}"' in html, f"/llm lost the fleet block's {part}"
for words in ("all engines", "sums only engines answering now"):
assert words in html, f"/llm lost the fleet block's `{words}` line"
for gone in ('id="totals"', "f-live", "f-total", "f-active", "f-kv", "seam-note"):
assert gone not in html, f"/llm still carries {gone} from the retired counts card"
js = client.get("/static/page-llm.js").text
body = _fleet_render_path(js)
assert "block.hidden = total < 2" in body, "the fleet block no longer hides for one engine"
assert "hidden" in _fleet_tag(html), \
"the block no longer ships hidden, so a single engine would show a fleet row"
def test_llm_fleet_block_reads_only_paths_the_api_publishes(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Every number the block prints is a `fleet` key of `/api/llm/snapshot`.
The page sums nothing itself — `fleet_totals()` does — so the field table in
`page-llm.js` is a list of API paths, and a rename in `sources/llm.py` must fail here
rather than print a permanent `n/a` in the browser. Pinned with both engines configured
(the keys exist, empty-handed because nothing answered) and with neither (404
`configured: false` without dialling, invariant 3 — and the shipped block stays hidden).
"""
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
fleet = test_client.get("/api/llm/snapshot").json()["fleet"]
assert set(fleet) >= FLEET_BLOCK_PATHS, FLEET_BLOCK_PATHS - set(fleet)
# the served field table is exactly the sum rows, in llm_monitor's order
js = test_client.get("/static/page-llm.js").text
table = _fleet_field_metrics(js)
assert table == ["requests_active", "requests_waiting",
"prompt_tokens_per_second", "decode_tokens_per_second",
"tokens_total"], "the fleet row changed shape without the API"
assert set(FLEET_SUM_PATHS) == set(table), "the block totals something the API does not"
# and the tag's two numbers are API paths too, not arithmetic the page invented
body = _fleet_render_path(js)
assert "fleet.engines_total" in body and "fleet.engines_live" in body
with build_client(tmp_path, monkeypatch) as bare:
response = bare.get("/api/llm/snapshot")
assert response.status_code == 404
assert response.json()["configured"] is False
html = _code(bare.get("/llm").text)
assert 'id="fleet"' in html and "hidden" in _fleet_tag(html), \
"an unconfigured engine must not leave a fleet row on the page"
def test_llm_fleet_totals_drop_an_engine_that_stopped_answering(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A total presented as current is current: a dead engine contributes nothing.
`fleet_totals()`'s docstring is the contract and the block is where it is read. The
injected payload survives the dead-port polls (a failed poll keeps the last good one), so
these are the numbers a browser would get with gufo answering and vLLM down — the live
engine's sums verbatim, `engines_live` one of two, and not one zero where a number the
dead engine never published would sit (rule 2).
"""
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
hub.jobs["llm:gufo"].state.payload = {
"engine": "gufo",
"metrics": {"decode_tokens_per_second": {"value": 41.0},
"prompt_tokens_per_second": {"value": 17.5},
"requests_processing": {"value": 2.0},
"requests_deferred": {"value": 1.0},
"tokens_total": {"value": 512.0}},
"sessions": {"active": 2, "list": []}}
fleet = test_client.get("/api/llm/snapshot").json()["fleet"]
assert fleet["engines_total"] == 2 and fleet["engines_live"] == 1
assert fleet["decode_tokens_per_second"] == pytest.approx(41.0)
assert fleet["prompt_tokens_per_second"] == pytest.approx(17.5)
assert fleet["requests_active"] == pytest.approx(2.0)
assert fleet["requests_waiting"] == pytest.approx(1.0)
assert fleet["tokens_total"] == pytest.approx(512.0)
# and with neither engine answering, every sum is null, never a confident 0
hub.jobs["llm:gufo"].state.payload = {"engine": "gufo", "metrics": {},
"sessions": {"active": None, "list": []}}
empty = test_client.get("/api/llm/snapshot").json()["fleet"]
assert empty["engines_live"] == 0
assert [empty[key] for key in FLEET_SUM_PATHS] == [None] * len(FLEET_SUM_PATHS)
def test_llm_fleet_block_proves_where_its_sums_come_from(client) -> None:
"""The seam tooltip, the excluded-engine tooltip, and an `n/a` with no unit.
Two provenance rules the rest of the app holds to, on the one block that adds engines
together: the prompt sum carries `fleet.seams` — the API's own words on why gufo's and
vLLM's prompt counters are not the same quantity — and the tag's tooltip names the
engines `fleet_totals()` left out, so a partial total says so in words and not only by
being dimmed. And a sum the API did not publish prints `n/a` with its unit cleared: "n/a
tok/s" would read as a measurement of nothing.
"""
js = _code(client.get("/static/page-llm.js").text)
body = _fleet_render_path(js)
assert "fleet.seams" in body, "the prompt sum lost the API's seam note"
assert "not answering — excluded from every sum below" in body, \
"the tag no longer names the engines left out of the totals"
assert "value == null ? ''" in body, "an n/a sum can still print a unit"
assert "value == null ? ''" in body, "an n/a sum can still print a unit"
def test_llm_fleet_block_colours_no_number(client) -> None:
"""No verdict on the aggregate row — and no dimming of anything but the tag.
`severity.py` rates no throughput and no queue depth (0 tok/s on an idle fleet is
healthy), so the block renders ink: the render path touches no `severity`, calls no
`Dash.hero()`, and the stylesheet section for the block reaches for no verdict token.
The one coloured thing on it is the shared **reachability** dot — the same distinction
the engine panels draw between their numbers and their header dot — and the partial-fleet
signal is the shared `.stale` dim on the count tag, which is opacity, not a verdict.
"""
js = _code(client.get("/static/page-llm.js").text)
body = _fleet_render_path(js)
assert "severity" not in body, "the fleet block invented a verdict"
assert "hero(" not in body, "the fleet block drew a hero number it cannot rate"
assert "setDot(" in body and "markStale(tag" in body, \
"a dead engine no longer dims the tag or darkens the block's dot"
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- fleet totals"):css.index("/* ---- hero numbers")]
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
assert token not in section, f"the fleet block coloured a number with {token}"
assert ".fleet-head" in section and ".fleet-note" in section, \
"the fleet block's head row lost its styling"
# ---- the `/llm` gufo slot grid (phase 07 task 04) -------------------------
#
# `llm_monitor`'s slot cards (`index.html` ~L1179), rebuilt against the capture instead of a
# memory: one keyed card per gufo slot, every cell naming **one** key of
# `tests/fixtures/slots_gufo.json`. Two things are pinned below, because they are the two ways
# this component failed before: a card that reads a key the server does not publish (dashes
# forever), and a card that is rebuilt every poll so a busy slot flickers in front of you.
#: The keys a slot card reads, and where they live: `field` (and its optional `sub` /
#: `ratioAgainst` partners) index the row, or its first `next_token` entry when `path` says so.
SLOT_ROW_PATHS = ("field", "sub", "ratioAgainst")
def _slot_table(js: str) -> list[dict[str, str | None]]:
"""The served `SLOT_FIELDS` table, one dict per printed cell."""
body = _code(js)
start = body.index("const SLOT_FIELDS = [")
table = body[start:body.index("\n];", start)]
cells: list[dict[str, str | None]] = []
for chunk in table.split("{ key: '")[1:]:
cell: dict[str, str | None] = {"key": chunk.split("'")[0]}
for option in (*SLOT_ROW_PATHS, "path"):
found = re.search(rf"{option}: '([a-z_0-9]+)'", chunk)
cell[option] = found.group(1) if found else None
cells.append(cell)
return cells
def _slot_js(js: str) -> str:
"""The slot grid's whole code path, comments out: `slotValue()` to the end of the block."""
body = _code(js)
return body[body.index("function slotValue("):body.index("function makeCharts(")]
def _slot_markup(js: str) -> str:
"""Just `slotSection()` — the markup the grid, its label and its empty states are built from."""
body = _code(js)
return body[body.index("function slotSection("):body.index("function renderPanel(")]
def test_llm_gufo_panel_holds_the_slot_grid_under_its_label(client) -> None:
"""`sessions (GET /slots)`, then the keyed grid, then the note about what it cannot contain.
The label is `llm_monitor`'s, the grid ships `hidden` (the empty state is a sentence, never
a card of nothing), and the note stays because a queued (deferred) request is not in
`/slots` at all — without that line an idle grid reads as "nothing waiting".
"""
js = client.get("/static/page-llm.js").text
markup = _slot_markup(js)
assert "sessions (GET /slots)" in markup, "/llm lost the slot grid's label"
# label, grid, empty state, note — in that order, and the grid and the empty state both
# ship hidden so nothing paints before the payload says which of them is true
order = ['class="section-label" data-slots-label', "sessions (GET /slots)",
'<div class="slots" data-slots role="list" hidden>',
'class="slots-empty" data-slots-empty hidden',
'class="slots-note" data-slots-note hidden']
position = -1
for needle in order:
found = markup.index(needle)
assert found > position, f"`{needle}` is out of order in the slot section"
position = found
assert "queued (deferred) requests are not in /slots" in js, "the queued-request note is gone"
assert "renderSlotGrid(card, enginePayload)" in js, "nothing draws the grid any more"
# the label is a heading and the grid a list: a screen reader gets the block's name and
# its item count, which a stack of divs does not say
assert '<h4 class="section-label" data-slots-label>' in markup, \
"the slot section's label stopped being a heading"
assert 'data-slots role="list"' in markup, "the grid lost its list semantics"
# the grid's own markup: a keyed card, the state pill, and one dd per printed cell
card = js[js.index("function slotCard("):js.index("function slotSub(")]
assert "setAttribute('role', 'listitem')" in card, "the slot cards left the list"
assert 'data-slot-cell="${cell.key}"' in card, "the card cells are not table-driven"
assert 'data-pill' in card and 'data-mark' in card, "the card lost its state pill or glyph"
def test_llm_slot_grid_style_is_a_marker_and_not_a_verdict(client) -> None:
"""A busy slot is a border, a glyph and a word — `severity.py` rates no slot.
Phase 06 settled this argument for GPU utilisation on an LLM server: a slot generating
tokens is the job, so the accent colour says *this is the live one* and no `--ok/--warn/
--fail` token appears anywhere in the section. The glyph (`○`/`●`) and the pill's word mean
the marker survives monochrome, which is the battery glyph's rule applied here.
"""
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- engine slot grid"):css.index("/* ---- fleet totals")]
for selector in (".slots {", ".slot {", ".slot.busy", ".slot-top", ".slot-id",
".slot-model", ".slot dt", ".slot dd", ".slots-empty", ".slots-note",
".section-label"):
assert selector in section, f"dash.css lost the {selector} rule"
assert "var(--accent)" in section, "the busy marker stopped using the non-verdict accent"
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
assert token not in section, f"the slot grid coloured a state with {token}"
js = client.get("/static/page-llm.js").text
assert "'\\u25cf'" in js or "'●'" in js, "the busy glyph left the card"
assert "processing" in _slot_js(js), "the pill stopped saying the state in a word"
def test_llm_slot_cells_each_name_one_key_the_capture_publishes(client) -> None:
"""Every printed cell traces to a key of `tests/fixtures/slots_gufo.json`.
One `field` per cell and no `a ?? b`: the deleted table's `slot.n_prompt ??
slot.prompt_tokens` guessing is what made it print dashes and then plausible noise. A key
the capture does not have fails here, and the alias the capture answers beside `id_task`
must not appear in the page at all — the contract names the canonical one.
"""
# comments out: this test bans names the prose is allowed to explain
js = _code(client.get("/static/page-llm.js").text)
cells = _slot_table(js)
assert cells, "the slot grid has no cell table to read"
assert [cell["key"] for cell in cells] == ["req", "prompt", "cache", "generated",
"remain", "ctx"], \
"the slot card's cell set changed without the capture"
for cell in cells:
assert cell["field"], f"the {cell['key']} cell names no row key"
rows = load_slots_capture()
assert rows, "the capture is empty, so this test would prove nothing"
for cell in cells:
for option in SLOT_ROW_PATHS:
key = cell[option]
if key is None:
continue
in_next = cell["path"] == "next_token"
required = REQUIRED_NEXT_TOKEN_FIELDS if in_next else REQUIRED_ROW_FIELDS
optional = OPTIONAL_NEXT_TOKEN_FIELDS if in_next else OPTIONAL_ROW_FIELDS
assert key in required or key in optional, (
f"the {cell['key']} cell reads {key!r}, which no contract table declares — the "
"page would be reading a field nobody pinned")
for row in rows:
source = row["next_token"][0] if in_next else row
if key in required:
assert key in source, (
f"the {cell['key']} cell reads {key!r}, which "
f"{'next_token[0]' if in_next else 'the row'} does not publish — this is "
"how the deleted table rendered dashes")
elif key in source: # optional: `prob` is on no captured row, and that is legal
assert isinstance(source[key], (int, float)), (
f"{key!r} is {source[key]!r}, which the cell cannot format")
assert cells[0]["field"] == CANONICAL_TASK_ID_FIELD, \
"the request cell stopped reading the canonical task id"
for alias in TASK_ID_ALIASES:
assert alias not in js, f"page-llm.js renders the {alias} alias the contract does not name"
# the old table's invented names stay invented, and so does its per-slot speed
for gone in ("n_predict", "decode_per_second", "cache_usable", "n_ctx_used"):
assert gone not in js, f"page-llm.js reads the invented {gone} again"
def test_llm_slot_card_prints_no_speed_it_cannot_measure(client) -> None:
"""No per-slot speed cell: the capture publishes no per-slot rate key.
`llm_monitor`'s grid has one, and so did the deleted table (`decode_per_second`) — which is
why it was dashes. A page-side Δn_decoded/Δt would be a measurement with no estimator, no
`Δ<window>` label and no `source` string, i.e. rule 2 with a confident number printed on it,
so the number stays on the engine's strip card and panel, where `rates.py` owns it.
"""
js = _code(client.get("/static/page-llm.js").text)
assert not [cell for cell in _slot_table(js)
if cell["key"] in {"speed", "tok_s", "decode"}], "a slot cell grew a rate"
assert "decode_per_second" not in js
def test_llm_slot_sentinels_print_a_word_or_na_never_a_negative_number(client) -> None:
"""`-1` is llama.cpp's "nothing in this slot", and it never becomes a countdown.
An idle slot answers `id_task: -1` and `next_token[0].n_remain: -1` (both in the capture).
The request cell says `idle` in a word; the remaining cell prints `n/a`. Neither prints a
negative number, and neither prints the `0` that would read as "one token from done".
"""
body = _slot_js(client.get("/static/page-llm.js").text)
assert "value < 0 ? 'idle'" in body, "the request cell prints the -1 sentinel again"
assert "value >= 0 ? value : null" in body, "the remaining cell prints the -1 sentinel"
def test_llm_slot_card_header_reads_only_fields_the_contract_declares(client) -> None:
"""The card top prints `model`, `speculative`, `id` and the state pair — and nothing else.
`id` keys the card (escaped once at build time), `is_processing` is the authoritative pill,
`state` labels a number the boolean cannot express, and `model` + `speculative` are the
footnote beside them. Each is declared in a contract table, so a rename fails there. The
user-content field a busy row carries (`prompt`) is the one key this page must never read.
"""
js = client.get("/static/page-llm.js").text
header = js[js.index("function renderSlotCard("):js.index("function renderSlotGrid(")]
read = {"is_processing", "state", "model", "speculative"}
for key in read:
assert f"row.{key}" in header, f"the slot card header stopped reading {key}"
declared = set(REQUIRED_ROW_FIELDS) | set(OPTIONAL_ROW_FIELDS)
assert read <= declared, "the header reads a field no contract table declares"
for field in USER_CONTENT_FIELDS:
assert field not in header, f"the slot card prints the user's {field}"
assert "escapeHtml(id)" in js, "the slot id reached the markup unescaped"
def test_llm_slot_cards_are_keyed_created_once_and_patched_in_place(client) -> None:
"""A busy slot does not get rebuilt every poll — and a slot that leaves loses its card.
`appendChild` of the existing node keeps DOM order == slot order; everything else is a text
patch, so a poll cannot flicker a card or re-announce it to assistive tech for a difference
of a few tokens. An engine that leaves the payload loses its card map with its panel, or the
refs of removed nodes would be re-appended when it came back.
"""
js = client.get("/static/page-llm.js").text
body = _slot_js(js)
assert "state.slots.get(enginePayload.engine)" in body, "the grid stopped keying its cards"
assert "if (!refs) {" in body and "refs = slotCard(id)" in body
assert "grid.appendChild(refs.root)" in body, "slot order is no longer kept by moving nodes"
assert "byId.delete(id)" in body, "a slot that left keeps its card forever"
assert "byId.forEach((refs) => refs.root.remove())" in body, \
"the empty state left cards on the page"
assert "setTxt(box.value" in body, "the cell values stopped being patched in place"
assert "innerHTML" not in body.split("function renderSlotCard(")[1], \
"a card is rebuilt from markup on a poll"
prune = js[js.index("function pruneEngines("):js.index("/* ---- the gufo slot grid")]
assert "state.slots.delete(engine)" in prune, "a pruned engine kept its slot cards"
def test_llm_slot_grid_says_why_there_are_no_rows(client) -> None:
"""Two honest empty states, and neither of them is a card or a 0-count claim.
`/slots` did not answer → the API's own `sessions.detail` sentence (the same words the
strip's session row carries). It answered with no rows → "engine reports 0 slots", which is
a fact about that answer. `sessions.source` is the gate that tells the two apart.
"""
body = _slot_js(client.get("/static/page-llm.js").text)
assert "Boolean(sessions.source)" in body, "the grid cannot tell no-answer from no-rows"
assert "engine reports 0 slots" in body
assert "sessions.detail ||" in body, "the no-answer state lost the API's own sentence"
assert "no /slots answer from this engine" in body, "the no-answer state can print undefined"
def test_llm_vllm_panel_says_there_is_no_per_slot_json(client) -> None:
"""vLLM gets one line and no grid — no fake rows for a server that cannot answer.
`sources/llm.py`'s `_sessions()` publishes counts for vLLM and says so in `detail`; the
panel turns that into the `llm_monitor` line, with the API's sentence as the tooltip. The
gate is the engine's name (`SLOT_ENGINES`), not `sessions.list.length`, so an engine that
has not answered yet still shows its label and the honest empty sentence.
"""
js = client.get("/static/page-llm.js").text
markup = _slot_markup(js)
branch = markup.split("if (SLOT_ENGINES.indexOf(engine) < 0) {")[1].split("}")[0]
assert 'class="slots-note" data-slots-note' in branch, "the vLLM line lost its element"
assert "NO_SLOT_JSON" in branch and "vLLM exposes no per-slot JSON" in js
assert '<div class="slots"' not in branch, "the vLLM panel grew a grid it cannot fill"
body = _slot_js(js)
assert body.startswith("function slotValue("), "the grid's code path moved"
assert "if (!grid) {" in body, "renderSlotGrid no longer leaves a panel with no grid alone"
assert "note.title = enginePayload.sessions.detail" in body, \
"the vLLM line stopped carrying the API's own words"
def test_the_captured_slot_rows_fill_the_cells_the_grid_prints(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The capture, through the API, into the keys the served cell table reads.
The fixture rows are injected as a live job payload (the `/slots` passthrough verbatim), so
this is the render path's input as the browser would receive it: `sessions.source` set (the
grid's gate), every row present, and every key `SLOT_FIELDS` names resolvable in it. A
rename in `sources/llm.py` or in gufo fails here rather than leaving a card of `n/a`s.
"""
rows = load_slots_capture()
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
hub.jobs["llm:gufo"].state.payload = {
"engine": "gufo", "metrics": {"requests_processing": {"value": 1.0}},
"sessions": {"active": 1, "list": rows, "source": "gufo /slots", "detail": None},
}
engines = test_client.get("/api/llm/snapshot").json()["engines"]
gufo = next(row for row in engines if row["engine"] == "gufo")
sessions = gufo["sessions"]
assert sessions["source"] == "gufo /slots", "the grid's gate is gone from the payload"
assert sessions["list"] == rows, "the rows are no longer the passthrough the grid reads"
cells = _slot_table(test_client.get("/static/page-llm.js").text)
for cell in cells:
for option in SLOT_ROW_PATHS:
key = cell[option]
if key is None:
continue
in_next = cell["path"] == "next_token"
required = REQUIRED_NEXT_TOKEN_FIELDS if in_next else REQUIRED_ROW_FIELDS
for row in sessions["list"]:
source = row["next_token"][0] if in_next else row
if key not in required and key not in source:
continue # an optional sub-line this build does not publish
value = source[key]
assert isinstance(value, (int, float)) and not isinstance(value, bool), (
f"the {cell['key']} cell reads {key!r} = {value!r}, which the number "
"formatter cannot print")
def test_llm_charts_are_secondary_and_named(client) -> None: