phase: 07_llm_monitor_style_pages
**Phase 07 final verification pass — all tasks already in `complete/`; every criterion re-verified live.**
- **Old slot table gone**: no `#slots-card`/`#slots-table`/`renderSlots` in HTML/JS/CSS (only the new grid's `.slots-empty` *class*); pinned by `test_the_broken_slot_table_is_gone_from_html_js_and_css`.
- **`tests/fixtures/slots_gufo.json`**: present; live `GET ${GUFO_BASE_URL}/slots` this pass matches it key-for-key (row + `next_token[0]`), so the capture is reproducible; drift proven to fail (mutation tests + live `test_gufo_slots_answer_the_pinned_shape`, 9 live tests pass).
- **Aggregate block**: live page shows `2 of 2 engines`, running/waiting/prompt/decode/total-tokens straight from `fleet_totals()`, dot `ok`, no verdict colour; on a throwaway single-engine instance (:8091, private temp DB) `#fleet` is `hidden`, one full-width strip card + one panel, zero banners → "one clean half".
- **Strip + slot grid on live data**: 2 `.card.stat` (KV bars `bar ok` 13 %/0 %), gufo grid rendered real cards incl. a busy slot (`#2189`, 67849 prompt tok, 99 % cache share, `processing`), vLLM note-only with no grid; no `NaN`/`undefined` anywhere. Screenshots: `data/logs/llm-phase07-final-live-{two-engines,single-engine}.png`, `ups-charts-aria-labels.png`.
- **Defect found in earlier work and fixed**: `/ups`' four `LineChart`s shipped no `ariaLabel`, violating AGENTS.md's "every chart names itself for assistive tech" (`/system`+`/llm` were pinned, `/ups` wasn't) → added labels to `page-ups.js` and `test_every_ups_chart_names_itself_for_assistive_tech`; browser-verified `role="img"` + label on all four.
- **Gate**: `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **604 passed, 9 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → **0 errors**; `node --check` every page script ok; `uv run pytest -m live --live` → **9 passed**; `uv pip list --outdated` empty.
- **Docs**: README + AGENTS.md reviewed end-to-end; fixed test count (582→604), `/llm` source column + both architecture diagrams now name gufo `/slots`, AGENTS.md agent line gained `GET /schema` + status page, ariaLabel convention now covers all three chart pages, README Testing block now lists `bash .agents/validate.sh`. Env-var table, agent/install flags, Containerfile facts, `/api/*` shapes, range chips, remote URL all verified accurate — no other changes needed.
- **Live-infra**: no fleet/host/config change; the temp :8091 instance, its temp DB and the fake processes are removed; dev server restarted and `/api/health` `ok` (all 6 sources ok). One thing I deliberately did **not** touch: the gitignored local `.env.dev` still carries a comment claiming the NUT `[dashboards]` account has only battery-test `instcmds`, whereas AGENTS.md/README record `instcmds = all` (verified on hardware 2026-10-17) — operator-owned file, flag for the operator to reword.
- **Next pending phase**: `08_events_paging_and_copy`.
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
**Phase 07 final verification pass — all tasks already in `complete/`; every criterion re-verified live.**
|
||||
|
||||
- **Old slot table gone**: no `#slots-card`/`#slots-table`/`renderSlots` in HTML/JS/CSS (only the new grid's `.slots-empty` *class*); pinned by `test_the_broken_slot_table_is_gone_from_html_js_and_css`.
|
||||
- **`tests/fixtures/slots_gufo.json`**: present; live `GET ${GUFO_BASE_URL}/slots` this pass matches it key-for-key (row + `next_token[0]`), so the capture is reproducible; drift proven to fail (mutation tests + live `test_gufo_slots_answer_the_pinned_shape`, 9 live tests pass).
|
||||
- **Aggregate block**: live page shows `2 of 2 engines`, running/waiting/prompt/decode/total-tokens straight from `fleet_totals()`, dot `ok`, no verdict colour; on a throwaway single-engine instance (:8091, private temp DB) `#fleet` is `hidden`, one full-width strip card + one panel, zero banners → "one clean half".
|
||||
- **Strip + slot grid on live data**: 2 `.card.stat` (KV bars `bar ok` 13 %/0 %), gufo grid rendered real cards incl. a busy slot (`#2189`, 67849 prompt tok, 99 % cache share, `processing`), vLLM note-only with no grid; no `NaN`/`undefined` anywhere. Screenshots: `data/logs/llm-phase07-final-live-{two-engines,single-engine}.png`, `ups-charts-aria-labels.png`.
|
||||
- **Defect found in earlier work and fixed**: `/ups`' four `LineChart`s shipped no `ariaLabel`, violating AGENTS.md's "every chart names itself for assistive tech" (`/system`+`/llm` were pinned, `/ups` wasn't) → added labels to `page-ups.js` and `test_every_ups_chart_names_itself_for_assistive_tech`; browser-verified `role="img"` + label on all four.
|
||||
- **Gate**: `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **604 passed, 9 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → **0 errors**; `node --check` every page script ok; `uv run pytest -m live --live` → **9 passed**; `uv pip list --outdated` empty.
|
||||
- **Docs**: README + AGENTS.md reviewed end-to-end; fixed test count (582→604), `/llm` source column + both architecture diagrams now name gufo `/slots`, AGENTS.md agent line gained `GET /schema` + status page, ariaLabel convention now covers all three chart pages, README Testing block now lists `bash .agents/validate.sh`. Env-var table, agent/install flags, Containerfile facts, `/api/*` shapes, range chips, remote URL all verified accurate — no other changes needed.
|
||||
- **Live-infra**: no fleet/host/config change; the temp :8091 instance, its temp DB and the fake processes are removed; dev server restarted and `/api/health` `ok` (all 6 sources ok). One thing I deliberately did **not** touch: the gitignored local `.env.dev` still carries a comment claiming the NUT `[dashboards]` account has only battery-test `instcmds`, whereas AGENTS.md/README record `instcmds = all` (verified on hardware 2026-10-17) — operator-owned file, flag for the operator to reword.
|
||||
- **Next pending phase**: `08_events_paging_and_copy`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 46%]
|
||||
.......sssssssss........................................................ [ 58%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 93%]
|
||||
..................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 166 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 107 6 94% 50-51, 56, 58, 60, 127
|
||||
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
|
||||
src/dashboards/config.py 113 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 228
|
||||
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2158 45 98%
|
||||
Required test coverage of 94% reached. Total coverage: 97.91%
|
||||
604 passed, 9 skipped, 1 warning in 52.69s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
**Task 01 complete — all green.**
|
||||
|
||||
**Implemented**
|
||||
- `scripts/capture-fixtures.sh`: gufo `/slots` capture (→ `tests/fixtures/slots_gufo.json`), `/v1/slots` fallback in the collector's order, atomic tmp+mv, clean skip when `GUFO_BASE_URL` unset; scrubber now blanks busy-slot `prompt` (user request text) in `slots_*.json`.
|
||||
- Deleted the broken slot table end-to-end: `#slots-card`/`#slots-table`/`#slots-empty` markup (`llm.html`), `renderSlots()` + call (`page-llm.js`), dead `.chip.info` rule (`dash.css`).
|
||||
- New `tests/slots_contract.py` (`assert_slot_shape`, required/optional row fields, `id_task`/`task_id` alias, `next_token[]` sub-fields, id uniqueness, `is_processing`/`state` consistency) + `tests/test_llm_slots_contract.py` (fixture-driven, mutation tests proving drift fails, deletion pins on served bytes); `test_llm_source.py` passthrough now uses the captured rows; live twin `test_gufo_slots_answer_the_pinned_shape` added to `test_live_fleet.py`.
|
||||
- Fixture recorded from live gufo (idle 2-slot shape); full re-capture also refreshed agent/NUT fixtures (volatile-value drift only, suite green). Verified `/llm` in a real browser against live data: 2 engine cards, zero JS errors (screenshot in `data/logs/`).
|
||||
|
||||
**Gate** — `bash .agents/validate.sh` exit 0: `uv run pytest --cov` **578 passed, 9 skipped, coverage 97.91%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; `uv pip list --outdated` empty. `uv run pytest -m live --live` **9 passed** (incl. the new live `/slots` contract).
|
||||
|
||||
**Docs** — AGENTS.md (test-count 578/9, capture-script line, testing-policy entry for the slots contract + prompt privacy) and README.md (capture commands, contract paragraph).
|
||||
|
||||
**Decisions**: `/slots` capture runs without `--with-llm` (the task names only the GUFO_BASE_URL skip and reproducibility via the bare script; it's one tiny request the hermetic test needs). `prompt` treated as user content: captured blanked, never rendered (pinned).
|
||||
|
||||
**Next pending task**: `07_llm_monitor_style_pages/02_fleet_totals_block.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 36%]
|
||||
.......................................................sssssssss........ [ 49%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 98%]
|
||||
........... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 166 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 107 6 94% 50-51, 56, 58, 60, 127
|
||||
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
|
||||
src/dashboards/config.py 113 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 228
|
||||
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2158 45 98%
|
||||
Required test coverage of 94% reached. Total coverage: 97.91%
|
||||
578 passed, 9 skipped, 1 warning in 51.48s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
**Implemented**
|
||||
- `/llm` now opens with the `llm_monitor` aggregate block (`#fleet`): reachability dot, `all engines`, an `N of M engines` tag, the note *"sums only engines answering now"*, then **running / waiting / prompt tok/s / decode tok/s / total tokens** — every number straight from `fleet_totals()` (`/api/llm/snapshot` → `fleet`); the counts-only `#totals` card (`f-live`/`f-total`/`f-active`/`f-waiting`) is deleted, ids and all.
|
||||
- `FLEET_FIELDS` is a list of API paths only (page sums nothing); hidden below two engines; a dead engine is dropped from every sum, dims the tag (shared `.stale`) and is named in its tooltip; `fleet.seams` rides on the prompt sum as a tooltip; an `n/a` sum clears its unit; no severity/`hero()` anywhere on the block.
|
||||
- `dash.css`: new `fleet totals` section built only from existing vocabulary (`.card.flat`, `.metrics`, `.dot`, `.chip`, `.stale`) — no new palette, no verdict tokens.
|
||||
- `tests/test_api.py`: replaced the phase‑04 totals-card pin with 5 fleet-block tests (markup/order + old ids gone; `fleet` key paths with both engines configured and with neither → 404 `configured:false`; injected-payload test proving a dead engine contributes nothing and null sums never print 0; no-verdict colouring; provenance) + `_fleet_tag`/`_fleet_render_path`/`_fleet_field_metrics`/`FLEET_BLOCK_PATHS` helpers.
|
||||
|
||||
**Gate** — `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` **582 passed, 9 skipped, coverage 97.91%**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` on all page scripts clean. `uv run pytest -m live --live` **9 passed**. `uv pip list --outdated` empty (no dependency change).
|
||||
|
||||
**Docs** — AGENTS.md (`/llm` UI-conventions paragraph now describes the fleet block + her test count 582) and README.md (same, plus "single engine → one clean half").
|
||||
|
||||
**Decisions / deviations**
|
||||
- The task asked for a status dot while also requiring "no severity colour anywhere on the block": the dot is the shared *reachability* dot (as the nav and every card header), and the tests pin that no `severity`/`hero()` reaches the block and its CSS uses no `--ok/--warn/--fail`.
|
||||
- Browser-verified live (screenshots in `data/logs/`): `llm-fleet-block-two-engines.png` (sums hand-checked: decode 10.8 = gufo 10.8 + vLLM 0.0, total tokens 177.55 M = 172.56 M + 4.98 M), `llm-fleet-block-single-engine.png` (block absent, one clean half), `llm-fleet-block-partial.png` (vLLM down → tag "1 of 2 engines" dimmed, dot amber, sums = gufo only). Those used two throwaway local instances on :8091/:8092 with private temp DBs, killed afterwards; no fleet/config change, dev server restarted and all sources `ok`.
|
||||
|
||||
**Next pending task**: `.agents/phases/todo/07_llm_monitor_style_pages/03_engine_stats_strip.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 36%]
|
||||
...........................................................sssssssss.... [ 48%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 97%]
|
||||
............... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 166 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 107 6 94% 50-51, 56, 58, 60, 127
|
||||
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
|
||||
src/dashboards/config.py 113 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 228
|
||||
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2158 45 98%
|
||||
Required test coverage of 94% reached. Total coverage: 97.91%
|
||||
582 passed, 9 skipped, 1 warning in 49.72s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
**Implemented (task 03 — per-engine stats strip)**
|
||||
- `llm.html`: new `#strip` section (`aria-label="Engine summary"`, ships empty+hidden) between `#fleet` and `#engines`.
|
||||
- `page-llm.js`: new `STRIP_ROWS` table (key **is** the `cellValue()` key), `stripCard()`/`renderStripCard()`/`stripRowsFor()`/`pruneEngines()`; card = dot + name + age stamp, KV-cache bar, prompt/decode tok/s with `Δ<window>` tags, total tokens, `slots busy` (gufo, counted from `sessions.list` `is_processing`, falls back to `sessions.active` + API `detail` — never a 0 counted from no rows), `running`/`waiting`, `prefix cache hit`, `spec accept` (gufo), `queue time` + `p95` foot (vLLM), alarm chips.
|
||||
- Panel cell list demoted to heroes + TTFT/TPOT (`spark:` list shrank 8→2): sessions, queued, KV, cache-hit, spec-accept, queue-mean, token footnote and the alarm container all left the panel — one value, one home. `HISTORY_SERIES` = explicit `CHART_SERIES` + `SPARK_CELLS`.
|
||||
- `dash.css`: new "engine stats strip" section built only from existing vocabulary (`.card`, `.kv`, `.bar`, `.chip.fail`, `.prov`; `.prov:empty` collapses; only `warn`/`fail` paint text), including the fix that moves the `.kv` grid *inside* the row wrapper.
|
||||
- `tests/test_api.py`: 9 new strip tests (placement/hidden/aria, shared-vocabulary markup + the `.kv` layout rule, `only` scoping, every row a cell the capture-parsed engines publish, slot-count rule-2 rule, KV colour from `severity` only, provenance-without-colour, strip/panel disjointness, no peak-ctx row) + 3 phase-04 pins restated for the new split.
|
||||
- Browser-verified live: `data/logs/llm-stats-strip-two-engines.png`, `-single-engine.png` (one full-width card, fleet block hidden), `-alarm-chip.png` (`.chip.fail` "Preemptions 3" on the incident card only) — the last two used throwaway local instances on :8091/:8092 plus a local fixture-backed fake engine, all killed afterwards; no fleet/config change, dev server restarted and `status: ok`.
|
||||
|
||||
**Gates:** `uv run pytest` → 591 passed, 9 skipped · `bash .agents/validate.sh` exit 0 (coverage 97.91%, floor 94) · `ruff check src tests agent` All checks passed · `pyright` 0 errors · `node --check` every script ok · `uv run pytest -m live --live` 9 passed · `uv pip list --outdated` empty (deps untouched).
|
||||
|
||||
**Docs:** README page tour rewritten for strip-then-panels; `AGENTS.md` gained the strip conventions paragraph (payload-only rows, KV colour, session-row shaping, alarms on the strip, `HISTORY_SERIES` composition, `pruneEngines`) and the `minSpan` sentence reworded.
|
||||
|
||||
**Decisions/deviations**
|
||||
- Task item 2 (strip gets vLLM "queue time") and item 6 (panel keeps "queue mean") conflict; satisfied the hard criterion "no value printed twice" by giving queue time — mean *and* p95 — to the strip, panel keeps TTFT/TPOT.
|
||||
- "No value twice" applied to the cell lists; the two throughput heroes stay in both components because items 1 and 6 both demand it (llm_monitor does the same) — pinned as the explicit exemption.
|
||||
- Alarm chips moved to the strip card per item 3 (phase overview's older wording kept them in the panel); panel alarm container deleted, asserted not to come back.
|
||||
- Observation: `llamacpp:n_tokens_max` (llm_monitor's peak-ctx cell) *is* in our gufo payload, contrary to the phase assumption's wording; followed the task (no peak-ctx cell) and pinned that the page never reads it.
|
||||
- `minSpan` now has no page caller (count rows are numbers, not sparks); kept in `charts.js` where the component contract test still pins it.
|
||||
|
||||
**Next pending task:** `07_llm_monitor_style_pages/04_slot_grid.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 36%]
|
||||
....................................................................ssss [ 48%]
|
||||
sssss................................................................... [ 60%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 96%]
|
||||
........................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 166 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 107 6 94% 50-51, 56, 58, 60, 127
|
||||
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
|
||||
src/dashboards/config.py 113 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 228
|
||||
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2158 45 98%
|
||||
Required test coverage of 94% reached. Total coverage: 97.91%
|
||||
591 passed, 9 skipped, 1 warning in 50.69s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
**Implemented (task 07/04 — gufo slot grid)**
|
||||
- `page-llm.js`: `SLOT_FIELDS` table (one fixture key per cell, `path:'next_token'` → `next_token[0]`), `slotValue()` as the only row accessor, `slotCard()`/`renderSlotCard()`/`renderSlotGrid()` keyed by slot `id` in `state.slots` (built once, text-patched, `appendChild` keeps slot order, gone slots + pruned engines cleaned up); `slotSection()` puts `sessions (GET /slots)` (`<h4>`) + `role=list` grid + the queued/deferred note in the gufo panel, and vLLM only the one-line note (with `sessions.detail` as tooltip) — no grid.
|
||||
- Cells: request (`id_task`, `-1`→`idle`), prompt tok (+`processed …`), cache hit (+`% of prompt`), generated (+`p=` when published), remaining (`-1`→`n/a`), context, model `+spec`. No per-slot speed cell (capture has no rate key; a JS Δ would be an unlabelled measurement). Busy marker = accent border + `○/●` + the word `processing`, no verdict token.
|
||||
- Empty states: no `/slots` answer → the API's own `sessions.detail`; answer with no rows → “engine reports 0 slots”; grid ships `hidden`.
|
||||
- `dash.css`: new “engine slot grid” section from existing tokens only (no `--ok/--warn/--fail`).
|
||||
- `tests/slots_contract.py`: canonicalised the request id (`CANONICAL_TASK_ID_FIELD="id_task"` required; `task_id` captured, never rendered) and made `next_token[0].n_remain` the single remaining path — replacing `llm_monitor`’s `a ?? b` chains.
|
||||
- `tests/test_api.py`: 9 new tests (markup order + hidden defaults, style-is-a-marker, every printed key resolves against the capture in both directions, no invented speed/sentinel leaks, keyed+patched-in-place+pruning, both empty states, vLLM note-only, capture→`/api/llm/snapshot`→cell-keys end-to-end); `tests/test_llm_slots_contract.py` updated for the canonical/alias rules.
|
||||
|
||||
**Gates:** `bash .agents/validate.sh` exit 0 → `603 passed, 9 skipped`, coverage **97.91 %** (floor 94) · `uv run ruff check src tests agent` All checks passed · `uv run pyright` 0 errors · `node --check` every script ok · `uv run pytest -m live --live` **9 passed** · `uv pip list --outdated` empty (deps untouched).
|
||||
|
||||
**Browser (live, `data/logs/`):** `llm-slots-idle.png`, `llm-slots-one-busy.png`, `llm-slots-two-busy.png` (real ids/tokens/cache share), `llm-slots-no-slots-answer.png`, `llm-slots-vllm-only.png`; DOM-node tagging proved cards survive polls (`survived: true`, numbers changing), console clean, no `NaN`/`undefined`. Dev server restarted (`status: ok`); throwaway :8091/:8092 instances, the fake engine and their temp DBs removed; no fleet/config change (two short gufo completions only, to photograph busy slots).
|
||||
|
||||
**Docs:** `AGENTS.md` — new slot-grid UI-conventions paragraph + testing-policy rewrite of the `/slots` contract (canonical key per fact); `README.md` — `/llm` page tour now covers the grid, sentinels, empty state and the vLLM line, plus the canonical-key note.
|
||||
|
||||
**Deviations:** (1) no per-slot speed cell, despite the task listing “per-slot speed” — the pinned capture publishes no rate key, so rendering it would be a permanent dead cell or an unlabelled page-side Δ (rule 2); decision and reason are in the code comment, AGENTS.md and a test. (2) `tests/slots_contract.py` gained the canonical task-id rule that task 04’s ASSUMPTION requires, so one task-01 test (`neither task_id alias answered`) was replaced by canonical-must-exist + alias-is-optional.
|
||||
|
||||
**Next pending task:** none — this was the last task in `07_llm_monitor_style_pages` (harness may complete the phase).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 47%]
|
||||
......sssssssss......................................................... [ 58%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 94%]
|
||||
.................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/alerts.py 166 0 100%
|
||||
src/dashboards/api.py 193 0 100%
|
||||
src/dashboards/app.py 107 6 94% 50-51, 56, 58, 60, 127
|
||||
src/dashboards/collectors.py 142 5 96% 97-98, 102-103, 120
|
||||
src/dashboards/config.py 113 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 228
|
||||
src/dashboards/sources/nut.py 389 20 95% 128, 142, 148, 160, 166, 178, 186, 199, 270-271, 273, 489, 501, 536, 569, 573, 630-631, 636, 648
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 178 2 99% 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2158 45 98%
|
||||
Required test coverage of 94% reached. Total coverage: 97.91%
|
||||
603 passed, 9 skipped, 1 warning in 51.25s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
@@ -26,7 +26,7 @@ file — plus an overview page that says which of the three you need:
|
||||
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
|
||||
| `/` | Is anything wrong, and where do I look first? | all three, in one request |
|
||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo (llama.cpp family) + vLLM `/metrics` |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
|
||||
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
|
||||
|
||||
History is a first-class feature, not a sparkline afterthought: every number worth graphing
|
||||
@@ -96,6 +96,7 @@ page still renders.
|
||||
```
|
||||
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
||||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
|
||||
│ │ + /slots (gufo) │
|
||||
▼ ▼ ▼
|
||||
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
|
||||
│ parse + Δcounter/Δt │
|
||||
@@ -109,8 +110,9 @@ store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ─
|
||||
Responsibilities, one line each:
|
||||
|
||||
- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
|
||||
`statvfs`; serves `GET /snapshot` + `GET /info` + `GET /health`. Refuses to bind
|
||||
off-loopback without a token (exit 2). Counters stay counters: the dashboard computes
|
||||
`statvfs`; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an
|
||||
HTML status page at `GET /`. Refuses to bind off-loopback without a token (exit 2).
|
||||
Counters stay counters: the dashboard computes
|
||||
rates so the agent never remembers who polled it last.
|
||||
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
|
||||
env var always wins. Defaults are safe except the sources, which default to unset.
|
||||
@@ -254,11 +256,11 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
|
||||
./scripts/dev-server.sh status # per-source ok/DOWN
|
||||
./scripts/dev-server.sh logs
|
||||
|
||||
uv run pytest # hermetic: 550 passed + 8 live-skipped, no network
|
||||
uv run pytest # hermetic: 604 passed + 9 live-skipped, no network
|
||||
uv run pytest --cov # coverage gate (97.9%; floor 94%)
|
||||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||||
uv run ruff check src tests agent && uv run pyright
|
||||
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS captures)
|
||||
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
|
||||
|
||||
bash .agents/validate.sh # the phase gate: the four checks above (pytest
|
||||
# --cov --cov-fail-under=94, ruff, pyright, plus a
|
||||
@@ -300,7 +302,21 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
|
||||
`test_ups_hero_fields_are_present_on_the_real_batteries` drives it against live upsd. The
|
||||
captures are what make the driver disagreements testable: one of them answers
|
||||
`battery.mfr.date: CPS`, neither exports `battery.temperature` or
|
||||
`battery.charger.status`.
|
||||
`battery.charger.status`. `tests/slots_contract.py` is the same device for gufo's `/slots`
|
||||
rows, which the `/llm` slot grid (phase 07) reads (`assert_slot_shape`, required/optional
|
||||
field tables, duplicate slot ids, the `next_token[0]` entries): hermetically
|
||||
from `tests/fixtures/slots_gufo.json` — the verbatim `/slots` reply, idle slots and all,
|
||||
captured whenever `GUFO_BASE_URL` is set — and live via
|
||||
`test_gufo_slots_answer_the_pinned_shape`. That fixture exists because its absence is how
|
||||
the deleted slot table rendered dashes forever: its field names (`slot.state`,
|
||||
`slot.n_prompt`, `slot.decode_per_second`) were a memory of an upstream, not a capture.
|
||||
**The contract names one canonical key per fact and the grid renders that one**: the capture
|
||||
answers `id_task` *and* `task_id` for the same request, so `CANONICAL_TASK_ID_FIELD` decides
|
||||
and the alias is captured but rendered by nothing — a rename fails a test instead of becoming
|
||||
a silent `a ?? b` chain in the page. `tests/test_api.py` reads the served `SLOT_FIELDS` table
|
||||
and resolves every key it names against the capture, in both directions.
|
||||
A busy slot's `prompt` is the user's in-flight request text, so the capture script blanks
|
||||
it and the page never reads it.
|
||||
- `live` (`tests/test_live_fleet.py`) re-runs the production code paths against the real
|
||||
fleet, engines, and upsd. Skipped unless `--live`; each test skips when its source is
|
||||
unset, so partial fleet availability is normal. **Live tests must not write** — a battery
|
||||
@@ -320,11 +336,12 @@ Hand-rolled SVG (`charts.js`), one stylesheet, dark/light toggle, `Δ<window>` l
|
||||
measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-source status
|
||||
dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
|
||||
`LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
|
||||
silence, and the legend is a sibling that labels nothing) — `/system` does this for all three
|
||||
slots, pinned by `tests/test_api.py`. Window chips (5m…30d on `/system` and `/llm`, 1h…90d on
|
||||
`/ups`; the API accepts 5m…90d). A panel that cannot get data says
|
||||
why in the panel — the operator should never have to open devtools to learn that a source is
|
||||
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
||||
silence, and the legend is a sibling that labels nothing). Every `LineChart` on every page has
|
||||
one — `/system`'s three per-host slots, `/llm`'s four trend plots, `/ups`'s four battery
|
||||
plots — each pinned by `tests/test_api.py`, so a chart that ships anonymous fails a test.
|
||||
Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API accepts 5m…90d). A
|
||||
panel that cannot get data says why in the panel — the operator should never have to open
|
||||
devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
||||
|
||||
The overview is exactly two panels — **Warnings & errors** (`ul.alerts`, one row per
|
||||
`/api/summary` alert: verdict class, source linked to its dashboard via the alert's own
|
||||
@@ -362,8 +379,10 @@ point, `labels: {current, max}` renders the `now / max` foot through the caller'
|
||||
there is no animation at all (`prefers-reduced-motion` holds by construction). `minSpan` is the
|
||||
other half of the same honesty for count metrics: history arrives as rollups, so eight seconds
|
||||
of one queued request averages to 0.14 and an axis fitted to that draws a full-height spike for
|
||||
a queue nobody was in — `/llm` passes `minSpan: 1` on sessions and queued so its axis is never
|
||||
finer than one request.
|
||||
a queue nobody was in. `/llm`'s count rows moved to the phase-07 stats strip (numbers, not
|
||||
sparks), so no page passes `minSpan` today and `drawSparks()` still plumbs it through — a count
|
||||
sparkline that comes back must set it, and the component's own contract test keeps the option
|
||||
honest (`test_sparkline_ships_the_fixed_scale_and_verdict_options`).
|
||||
|
||||
A battery is `DashCharts.battery(container, {charge, severity, state, label})` — one inline SVG
|
||||
whose **fill *is* the charge**: 1 viewBox unit per percent, clamped to 0–100 and never rounded up
|
||||
@@ -441,24 +460,84 @@ always say whether it was scraped, measured `Δcounter/Δt`, or estimated (`llm_
|
||||
`setThroughput` idiom; dropping the tag to look cleaner is a regression, not a simplification).
|
||||
|
||||
**The `/llm` page reads per engine** (phase 04, decided in
|
||||
`.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`): a row of two
|
||||
`Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**, because 0 tok/s
|
||||
on an idle server is healthy and `severity.py` has no throughput row), then a compact `.cells`
|
||||
list — label left, tabular value right, a `sparkline()` fixed at 160px so rows line up, and a
|
||||
mono `p95 …` foot under every latency mean (no hover; p95 needs no stored series). Rows are
|
||||
engine-scoped by the `only` flag: gufo never grows a TTFT mystery, vLLM never a spec-accept
|
||||
hole. `preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
|
||||
`.agents/phases/complete/04_llm_chart_readability/01_chart_set_decision.md`, laid out by phase
|
||||
07): a row of two `Dash.hero()` throughput numbers (decode, prompt — deliberately **uncoloured**,
|
||||
because 0 tok/s on an idle server is healthy and `severity.py` has no throughput row) and the
|
||||
latency rows, each a mean with a mono `p95 …` foot (no hover; p95 needs no stored series) and a
|
||||
`sparkline()` fixed at 160px so rows line up. Rows are engine-scoped by the `only` flag on both
|
||||
components: gufo never grows a TTFT mystery, vLLM never a spec-accept hole.
|
||||
|
||||
**The stats strip (`#strip`, phase 07 task 03) is the summary; the panels are the detail, and no
|
||||
row is printed in both** — the UPS hero row's "one value drawn twice" rule, widened to two
|
||||
components. One `.card.stat` per *configured* engine between the fleet block and the panels: the
|
||||
reachability dot, the engine name and the age stamp in the head (same distinction every card
|
||||
draws — the dot speaks reachability, the numbers never do), then the **KV-cache bar** — the one
|
||||
cell on the page with a verdict, so the fill and a breach's text take their colour from the
|
||||
payload's `severity` through `.bar.ok/.warn/.fail` and the ratio→percent conversion the panel's
|
||||
cell used to make, with `ok` staying quiet ink and a missing gauge an empty bar plus `n/a` — then
|
||||
`prompt tok/s` / `decode tok/s` with their `Δ<window>` tags, `total tokens` (lifetime, so its
|
||||
source is the tooltip), the sessions this engine reports, `prefix cache hit`, `spec accept`
|
||||
(gufo) and `queue time` with its `p95 …` foot (vLLM). `STRIP_ROWS` is the whole table and its
|
||||
`key` **is** the `cellValue()` key, so every number on it is a cell of `/api/llm/snapshot`: no
|
||||
field was invented in `sources/llm.py` for this card, and there is deliberately **no peak-ctx
|
||||
row** (phase assumption). The session row is engine-shaped — vLLM publishes `running`/`waiting`
|
||||
counts, gufo's is `slots busy`, counted from the `sessions.list` rows `/slots` answered
|
||||
(`is_processing`, the shape pinned by `tests/slots_contract.py`); when `/slots` did not answer
|
||||
there are no rows to count, so it falls back to the published `sessions.active` carrying the
|
||||
API's own `sessions.detail` sentence rather than reporting 0 busy slots from no data (rule 2).
|
||||
`preemptions` / `device_lost` are **alarm chips that exist only while non-zero** (`.chip.fail`,
|
||||
container emptied and `hidden` at zero — the rule is the presence of an incident, never a graded
|
||||
threshold); lifetime token counters are a quiet footnote line. **One `/api/llm/history` request
|
||||
per refresh** carries every chart *and* spark series (`HISTORY_SERIES`; the endpoint answers
|
||||
unknown names as empty series on purpose), and the two series that addition required
|
||||
(`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) are stored additively in
|
||||
`sources/llm.py` HISTORY_METRICS. The engine-totals card kept only engines-live/active/queued
|
||||
and **hides below two engines** — a cross-engine sum of one engine duplicates that engine's own
|
||||
panel, and the summed-KV bar went because `fleet_totals` carries no `severity` to render. The
|
||||
threshold) and they live on this card, not in a permanent cell; the lifetime token counter left
|
||||
the panel's footnote line for the strip's row. **One `/api/llm/history` request per refresh**
|
||||
carries every chart *and* spark series (`HISTORY_SERIES` = the four chart series +
|
||||
`SPARK_CELLS`; the endpoint answers unknown names as empty series on purpose), and the two series
|
||||
phase 04 added (`llm.tpot_seconds` vLLM-only, `llm.spec_accept_rate` gufo-only) stay stored
|
||||
additively in `sources/llm.py` HISTORY_METRICS. Above the strip sits **the fleet block**
|
||||
(`#fleet`, phase 07: "I'm missing the aggregate throughput at the top") — `ALL ENGINES`, a
|
||||
reachability dot, an
|
||||
`N of M engines` tag, the note *sums only engines answering now*, then running / waiting /
|
||||
prompt tok/s / decode tok/s / total tokens. `FLEET_FIELDS` in `page-llm.js` is a list of
|
||||
`fleet_totals()` keys and nothing else: the page sums no number of its own, a dead engine is
|
||||
dropped from every total presented as current (its own stale-labelled panel still shows its
|
||||
last good values), and the tag counts the answering engines, names the missing one in its
|
||||
tooltip and takes the shared `.stale` dim while the sums are partial. It **hides below two
|
||||
engines** — a cross-engine sum of one engine duplicates that engine's own panel — and it
|
||||
carries **no severity colour anywhere**: `fleet_totals` has no `severity` to render (the
|
||||
summed-KV bar is gone for that reason) and `severity.py` rates no throughput or queue depth,
|
||||
so the numbers stay ink and only the dot speaks reachability. `fleet.seams` — the API's own
|
||||
note on how the two engines count prompt tokens differently — is the prompt sum's tooltip
|
||||
rather than another line of ink. The
|
||||
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
|
||||
sit at their own height (`#engines` carries `.items-start`): a 5-row gufo panel next to a 7-row
|
||||
vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data".
|
||||
sit at their own height (`#engines` carries `.items-start`): a two-hero gufo card next to a
|
||||
latency-row vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". An
|
||||
engine that leaves the payload leaves both components (`pruneEngines`), never a stale card in
|
||||
either grid.
|
||||
|
||||
**The slot grid (`llm_monitor`'s, phase 07 task 04) belongs to the gufo panel, under its
|
||||
`sessions (GET /slots)` label.** One card per row of `sessions.list`, **keyed by the row's own
|
||||
`id`** in `state.slots`: built once by `slotCard()`, patched in place afterwards, and reordered
|
||||
by `appendChild` of the existing node so DOM order stays slot order — rebuilding a busy slot's
|
||||
card every 2 s poll re-announces the whole card to assistive tech for a difference of a few
|
||||
tokens. `SLOT_FIELDS` is the whole table and each cell names **exactly one** `/slots` key
|
||||
(`path: 'next_token'` reads `next_token[0]`, where the capture puts `n_remain`); `slotValue()`
|
||||
is the only place a row is indexed, so `tests/test_api.py` can resolve every printed field
|
||||
against `tests/fixtures/slots_gufo.json` and a rename fails there rather than printing dashes
|
||||
or a plausible wrong number. No per-slot speed cell: the capture publishes no per-slot rate key
|
||||
and a page-side Δn_decoded/Δt would be a measurement with no estimator, no `Δ<window>` label and
|
||||
no `source` string (rule 2) — the engine's rate lives on its strip card and panel. llama.cpp's
|
||||
`-1` sentinels print a word or nothing: `id_task: -1` is **`idle`**, `n_remain: -1` is **`n/a`**,
|
||||
never a negative countdown and never the `0` that reads as one token from done. The busy marker
|
||||
is a **border, a glyph (`○`/`●`) and the word `processing`** in the brand `--accent`, never an
|
||||
`ok|warn|fail` token — a slot generating tokens is the job (phase 06's GPU reasoning), so
|
||||
`severity.py` rates no slot and the stylesheet section contains no verdict token. The empty
|
||||
state is a sentence, never a card of nothing, and it distinguishes the two cases: no `/slots`
|
||||
answer prints the API's own `sessions.detail`, an answer with no rows prints *engine reports 0
|
||||
slots*; `sessions.source` is the gate. The note under the grid stays because a **queued
|
||||
(deferred) request is not in `/slots`** — the waiting row counts those. vLLM's panel gets one
|
||||
line (*exposes no per-slot JSON*) with `sessions.detail` as its tooltip and **no grid**: a
|
||||
component of permanent `n/a`s for a server that cannot answer is the dead-cell anti-pattern.
|
||||
The gate is the engine name (`SLOT_ENGINES`), not row count, so an engine that has not answered
|
||||
yet still shows its label and its honest empty sentence.
|
||||
|
||||
**Charts belong to the host they chart.** `/system` has two sections — the host panels and the
|
||||
transition events — and nothing else: the fleet-wide chart stack, the nine-figure fleet-totals
|
||||
|
||||
@@ -6,7 +6,7 @@ Three dashboards, one domain, one process, one SQLite file:
|
||||
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
||||
| `/` | Is anything wrong, and which dashboard do I need? | fleet + alerts in one request (`/api/summary`) |
|
||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
||||
|
||||
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
||||
@@ -120,6 +120,7 @@ unreachable host writes its failure message into the event table and `/api/event
|
||||
```
|
||||
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
||||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
|
||||
│ │ + /slots (gufo) │
|
||||
▼ ▼ ▼
|
||||
src/dashboards/collectors.py ────────────────────────────────────────────┐
|
||||
│ parse + Δcounter/Δt (rates.py) │
|
||||
@@ -193,16 +194,45 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
|
||||
processes — folds into a collapsed
|
||||
"More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request
|
||||
(`hosts=<that host>&metrics=…&range=…`) every refresh, stack open or not, and a host that stops
|
||||
reporting takes its panel, its Charts toggle and its charts off the page. The LLM page (`/llm`) is per engine too: two big
|
||||
reporting takes its panel, its Charts toggle and its charts off the page. `/llm` opens with
|
||||
the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, the note
|
||||
*sums only engines answering now*, then **running / waiting / prompt tok/s / decode tok/s /
|
||||
total tokens** — and every number on it is `fleet_totals()`, so the page sums nothing itself.
|
||||
A dead engine keeps its last-good numbers in its own (stale-labelled) panel but is dropped
|
||||
from a total presented as current: the tag counts what answered, names the missing engine in
|
||||
its tooltip and dims while the sums are partial. Nothing on the block is coloured —
|
||||
`severity.py` rates no throughput and no queue depth, because 0 tok/s on an idle fleet is
|
||||
healthy — the dot is reachability, and `fleet.seams` (how the two engines count prompt
|
||||
tokens differently) is a tooltip on the prompt sum rather than another line of ink. Below
|
||||
two configured engines the block is hidden: a cross-engine sum of one engine duplicates that
|
||||
engine's panel. Below the fleet block sits the **stats strip**: one card per configured
|
||||
engine — reachability dot, name, age stamp, the **KV-cache bar** (the one rated cell: the
|
||||
fill and a breach's text take their colour from the payload's `severity` and nothing else),
|
||||
then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ10.0s`, scraped,
|
||||
estimated), `total tokens`, the sessions this engine reports (`slots busy` counted from
|
||||
gufo's `/slots`, or `running`/`waiting` for vLLM, plus `waiting`), `prefix cache hit`,
|
||||
`spec accept` (gufo), `queue time` with its `p95 …` foot (vLLM) — and red alarm chips for
|
||||
preemptions / GPU-context losses that exist only while non-zero. Then the engine panels
|
||||
hold the detail the strip does not print: two big
|
||||
throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is
|
||||
healthy) with their provenance tag (`Δ10.0s`, scraped, estimated), then a compact list of
|
||||
metric rows — sessions, queue, KV pressure (the only rated cell), prefix/spec percentages,
|
||||
and TTFT/TPOT/queue-time means with a `p95 …` foot line — each carrying its own sparkline
|
||||
over the selected window. Preemptions and GPU-context losses render only as red alarm chips
|
||||
while non-zero; the token counter is a footnote. The four trend charts sit below the panels
|
||||
healthy) with their provenance tag, and the latency rows (TTFT, TPOT) each with a `p95 …`
|
||||
foot line and its own sparkline over the selected window. Under the gufo panel's
|
||||
**`sessions (GET /slots)`** label sits the **slot grid**: one card per parallel slot, keyed by
|
||||
the slot's own id, created once and patched in place so a busy slot never flickers — request
|
||||
id, prompt tokens (with `processed …` while prefill runs), cache hit and its share of the
|
||||
prompt, generated tokens, remaining, context, and the model beside the slot number. A busy
|
||||
card is a border, a glyph and the word `processing`, never a green/amber/red verdict (a slot
|
||||
generating tokens is the job). Every cell reads exactly one key of the captured `/slots`
|
||||
shape — no `a ?? b` fallback chains — so a field rename fails a contract test instead of
|
||||
printing dashes, and `-1` sentinels print `idle`/`n/a`, never a negative countdown. When
|
||||
`/slots` does not answer, the panel says so with the API's own sentence rather than rendering
|
||||
a card of nothing, and vLLM's panel carries one line (*exposes no per-slot JSON*) and no grid.
|
||||
No row appears in both places —
|
||||
one value, one home — and there is no peak-context cell, because this page prints a number
|
||||
the upstream published or `n/a`. The four trend charts sit below the panels
|
||||
at half height, and one `/api/llm/history` request per refresh feeds all of it. With a
|
||||
single engine configured the totals card hides and the page is one clean half — never a red
|
||||
panel for a server you do not run. Each UPS card (`/ups`) opens with its battery glyph and
|
||||
single engine configured the page is one clean card in the strip and one clean panel —
|
||||
never a red half for a server you do not run. Each UPS card (`/ups`) opens with its battery glyph and
|
||||
three big numbers — **charge %**, **load %**, **runtime left** — coloured from the card's
|
||||
`severity` dict, with the electrical readings (line in, output, battery volts and amps,
|
||||
battery °C) kept below them in smaller muted type because they are what you read *while* the
|
||||
@@ -217,8 +247,10 @@ uv run pytest --cov # coverage gate
|
||||
set -a; . ./.env.dev; set +a # the live suite reads the same env as the app
|
||||
uv run pytest -m live --live # optional: real fleet / engines / upsd
|
||||
uv run ruff check src tests agent && uv run pyright # lint + types
|
||||
./scripts/capture-fixtures.sh # re-record ground truth: host agents + UPS replies
|
||||
./scripts/capture-fixtures.sh # re-record ground truth: host agents, UPS replies, gufo /slots
|
||||
./scripts/capture-fixtures.sh --with-llm # ... and the engines' /metrics while they are up
|
||||
bash .agents/validate.sh # all of the above + `node --check` on every page
|
||||
# script + repo hygiene, with the 94% coverage floor
|
||||
```
|
||||
|
||||
Test philosophy: **real captures as fixtures, mocks for behaviour.** Anything that must
|
||||
@@ -242,7 +274,17 @@ verbatim as `tests/fixtures/nut_<ups>.vars` by `./scripts/capture-fixtures.sh`,
|
||||
device serials scrubbed — and again against live upsd. The captures earn their place: the
|
||||
CyberPower answers `battery.mfr.date: CPS`, the APC's `battery.date` is a factory
|
||||
placeholder, and **neither** publishes `battery.temperature` or `battery.charger.status`, so
|
||||
a hand-written variable table would have tested nobody's memory.
|
||||
a hand-written variable table would have tested nobody's memory. `tests/slots_contract.py`
|
||||
pins gufo's `/slots` rows the same way — hermetically from
|
||||
`tests/fixtures/slots_gufo.json` (the verbatim `/slots` reply, captured whenever
|
||||
`GUFO_BASE_URL` is set; the idle-slot rows are the shape the grid must handle) and again
|
||||
against the running engine. It exists because that table was once guessed: the old slot
|
||||
table read field names the server never published and rendered dashes forever. It also picks
|
||||
the **canonical name of each fact** — the live server answers `id_task` *and* `task_id` for
|
||||
one request — and the page renders only the canonical key, so a rename fails a test instead of
|
||||
quietly becoming a fallback chain. A busy
|
||||
slot's `prompt` is the user's in-flight request text, so the capture script blanks it and
|
||||
the page never reads it.
|
||||
|
||||
## Battery-test buttons
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
#
|
||||
# ./scripts/capture-fixtures.sh # needs the tunnels up: scripts/dev-tunnels.sh start
|
||||
# ./scripts/capture-fixtures.sh --with-llm # also scrape the live engines' /metrics
|
||||
# # (gufo /slots is captured without the flag)
|
||||
#
|
||||
# Fixtures are verbatim, not hand-written: a fixture you typed is a test of your memory.
|
||||
# When a host's kernel or an engine's metric names change, re-run this and read the diff.
|
||||
@@ -105,6 +106,30 @@ else
|
||||
echo " (skipped UPS captures — NUT_HOST unset; see .env.dev.example)"
|
||||
fi
|
||||
|
||||
# --- gufo /slots (per-slot rows) ---------------------------------------------
|
||||
# `tests/slots_contract.py` pins the shape of the `/llm` slot grid (task 04) against
|
||||
# `tests/fixtures/slots_gufo.json`, so this capture is NOT behind `--with-llm`: it is one
|
||||
# small JSON document and the hermetic contract test needs it on disk. `/v1/slots` is the
|
||||
# documented alias, tried second — the same order `collectors.fetch_engine` and
|
||||
# `llm_monitor`'s passthrough use, so a server that only exposes the alias is still
|
||||
# capturable. Skipped when GUFO_BASE_URL is unset, like the UPS captures. An idle engine
|
||||
# still answers one row per parallel slot; that idle shape is what the grid must handle.
|
||||
if [ -n "${GUFO_BASE_URL:-}" ]; then
|
||||
gufo_base="${GUFO_BASE_URL%/}"
|
||||
slots_tmp="$FIX/.slots_gufo.json.tmp"
|
||||
if curl -fsS --max-time 8 "${gufo_base}/slots" -o "$slots_tmp" ||
|
||||
curl -fsS --max-time 8 "${gufo_base}/v1/slots" -o "$slots_tmp"; then
|
||||
mv "$slots_tmp" "$FIX/slots_gufo.json"
|
||||
echo " ✓ slots_gufo.json ($(wc -c <"$FIX/slots_gufo.json") bytes)"
|
||||
else
|
||||
rm -f "$slots_tmp" # never overwrite a good fixture with a dead engine's empty file
|
||||
echo " ✗ slots_gufo.json — gufo answered neither /slots nor /v1/slots at $gufo_base"
|
||||
failed=1
|
||||
fi
|
||||
else
|
||||
echo " (skipped gufo /slots capture — GUFO_BASE_URL unset; see .env.dev.example)"
|
||||
fi
|
||||
|
||||
if [ "${1:-}" = "--with-llm" ]; then
|
||||
if [ ${#ENGINES[@]} -eq 0 ]; then
|
||||
echo " ! no GUFO_BASE_URL / VLLM_BASE_URL set (see .env.dev) — nothing to capture"
|
||||
@@ -135,7 +160,10 @@ fi
|
||||
# FQDN a host thinks it has, and a UPS's `ups.serial`/`device.serial` (which identifies the
|
||||
# physical unit, and is echoed back in the card's variable dump). They are replaced
|
||||
# deterministically so re-capturing does not churn the fixture, and the tests keep the same
|
||||
# shapes to assert against.
|
||||
# shapes to assert against. A fourth belongs only to the /slots capture: a busy slot's
|
||||
# `prompt` field is *the user's in-flight request text* — personal data the slot grid never
|
||||
# renders — so any non-empty prompt value is blanked, keeping the key (the shape) without
|
||||
# the content.
|
||||
python3 - "$FIX" <<'PY'
|
||||
import pathlib
|
||||
import re
|
||||
@@ -167,6 +195,9 @@ for path in sorted(fix.iterdir()):
|
||||
# "hostname": "gw.lab.example.com" -> "gw": the dashboard's host key is the target name
|
||||
# from HOST_TARGETS, never whatever the box calls itself
|
||||
text = re.sub(r'("hostname": "[^"]*?)\.[^"]*(")', r"\1\2", text)
|
||||
if path.name.startswith("slots_"):
|
||||
# only non-empty values are replaced, so an idle capture does not churn
|
||||
text = re.sub(r'("prompt"\s*:\s*)"(?:[^"\\]|\\.)+"', r'\1""', text)
|
||||
|
||||
def scrub_serial(match: re.Match[str]) -> str:
|
||||
value = match.group(2)
|
||||
|
||||
@@ -36,27 +36,71 @@
|
||||
|
||||
<div id="banners" class="stack" style="margin-bottom:.85rem"></div>
|
||||
|
||||
<!-- Engine totals, hidden until a second engine exists: a cross-engine sum of one
|
||||
engine only duplicates that engine's own panel. What stays here is what no panel
|
||||
says — how many engines are answering, and the session/queue counts across them.
|
||||
Summed decode/prompt/KV left with phase 04: per-engine is the honest view, and
|
||||
averaging a Strix Halo iGPU with a Threadripper is not one number. -->
|
||||
<section class="card flat" id="totals" aria-label="Engine totals" hidden>
|
||||
<!-- The aggregate block, in llm_monitor's shape: the sums no single engine panel can
|
||||
say, above the panels. Hidden until a second engine exists — a cross-engine sum of
|
||||
one engine only duplicates that engine's own panel.
|
||||
|
||||
Every number here is `fleet_totals()` over the engines that answered *this poll*: a
|
||||
dead engine keeps its last-good values inside its own card (correct there, because
|
||||
that card is labelled stale) but is dropped from a total presented as current, and
|
||||
the tag above says how many engines the sum covers. No number on this block carries
|
||||
a verdict — `severity.py` has no throughput or queue-depth row, because 0 tok/s on
|
||||
an idle fleet is healthy — so nothing here is coloured, and the KV bar that used to
|
||||
sit beside these sums left with phase 04 for exactly that reason. -->
|
||||
<section class="card flat fleet" id="fleet" aria-label="Fleet totals" hidden>
|
||||
<div class="body">
|
||||
<div class="fleet-head">
|
||||
<span class="dot idle" id="fleet-dot"></span>
|
||||
<span class="fleet-name">all engines</span>
|
||||
<span class="chip" id="fleet-count">no engines yet</span>
|
||||
<span class="fleet-note">sums only engines answering now</span>
|
||||
</div>
|
||||
<div class="metrics">
|
||||
<div class="metric"><div class="label">Engines answering</div>
|
||||
<div class="value"><span id="f-live">n/a</span><span class="unit" id="f-total"></span></div></div>
|
||||
<div class="metric"><div class="label">Active sessions</div>
|
||||
<div class="value" id="f-active">n/a</div></div>
|
||||
<div class="metric"><div class="label">Queued</div>
|
||||
<div class="value" id="f-waiting">n/a</div></div>
|
||||
<div class="metric" title="Requests generating tokens across every answering engine.">
|
||||
<div class="label">running</div>
|
||||
<div class="value"><span id="f-running">n/a</span><span class="unit"></span></div>
|
||||
</div>
|
||||
<div class="metric" title="Requests queued behind them across every answering engine.">
|
||||
<div class="label">waiting</div>
|
||||
<div class="value"><span id="f-waiting">n/a</span><span class="unit"></span></div>
|
||||
</div>
|
||||
<div class="metric">
|
||||
<div class="label">prompt tok/s</div>
|
||||
<div class="value"><span id="f-prompt">n/a</span><span class="unit">tok/s</span></div>
|
||||
</div>
|
||||
<div class="metric" title="Every busy slot on every answering engine: tokens generated per second.">
|
||||
<div class="label">decode tok/s</div>
|
||||
<div class="value"><span id="f-decode">n/a</span><span class="unit">tok/s</span></div>
|
||||
</div>
|
||||
<div class="metric" title="Tokens counted since each answering engine started.">
|
||||
<div class="label">total tokens</div>
|
||||
<div class="value"><span id="f-tokens">n/a</span><span class="unit"></span></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- The page leads with the engine panels: two throughput heroes per engine, a compact
|
||||
sparkline cell list, and the gufo slot table below. The four trend charts follow,
|
||||
half-height — supporting evidence, not the main event. -->
|
||||
<!-- The per-engine stats strip (phase 07 task 03, `llm_monitor`'s shared strip): one card
|
||||
per **configured** engine, filled by `page-llm.js` from `/api/llm/snapshot` — the
|
||||
KV-cache bar (the one cell on this page with a verdict, coloured from the payload), the
|
||||
two throughputs with their provenance tags, lifetime tokens, the sessions this engine
|
||||
reports, its own extra rows, and alarm chips that exist only while non-zero.
|
||||
|
||||
It ships empty and `hidden`: an engine that is not configured gets no card, and a
|
||||
single-engine deployment gets one card across the page instead of a red half (the
|
||||
fleet block above simply stays hidden below two engines). No value is printed here and
|
||||
in the panels below — the rows that moved in were deleted from the panel cell list in
|
||||
the same change. -->
|
||||
<section class="strip" id="strip" aria-label="Engine summary" hidden></section>
|
||||
|
||||
<!-- Then the engine panels: two throughput heroes per engine and the detail the strip
|
||||
does not print (the latency means with their p95 feet and window sparklines). The four
|
||||
trend charts follow, half-height — supporting evidence, not the main event. The gufo
|
||||
slot table that used to sit beside the events panel was deleted in phase 07: it read
|
||||
field names (`slot.state`, `slot.n_prompt_tokens`, `slot.decode_per_second`) that
|
||||
gufo's live `/slots` never publishes, so it could only ever render dashes. Its
|
||||
replacement is built from the shape pinned in `tests/fixtures/slots_gufo.json`
|
||||
(`tests/slots_contract.py`). -->
|
||||
<section style="margin-top:.85rem">
|
||||
<div class="grid cols-2 items-start" id="engines"></div>
|
||||
<div class="empty-state" id="engines-empty" hidden>
|
||||
@@ -88,16 +132,7 @@
|
||||
</article>
|
||||
</section>
|
||||
|
||||
<section class="grid cols-2" style="margin-top:.85rem">
|
||||
<article class="card" id="slots-card">
|
||||
<header><h3>gufo slots</h3><span class="spacer"></span>
|
||||
<span class="faint">one row per parallel slot</span></header>
|
||||
<div class="body tight"><table class="data" id="slots-table">
|
||||
<thead><tr><th>Slot</th><th>State</th><th>Tokens</th><th>N prompt</th><th>N gen</th>
|
||||
<th>Cache</th><th>Speed</th></tr></thead><tbody></tbody></table>
|
||||
<div class="empty-state" id="slots-empty" hidden><p>No <code>/slots</code> rows.</p></div>
|
||||
</div>
|
||||
</article>
|
||||
<section style="margin-top:.85rem">
|
||||
<article class="card">
|
||||
<header><h3>Events</h3><span class="spacer"></span><span class="faint">engine up / down</span></header>
|
||||
<div class="body"><ul class="events" id="events"><li class="muted">loading…</li></ul></div>
|
||||
|
||||
@@ -278,6 +278,134 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
|
||||
.cell .cell-spark { grid-column: 1 / -1; width: 100%; }
|
||||
}
|
||||
|
||||
/* ---- engine stats strip ---------------------------------------------------
|
||||
* `/llm`'s per-engine summary card (`llm_monitor`'s shared strip): the KV-cache bar and a
|
||||
* `dl` of the numbers you read across the room, built entirely from vocabulary this
|
||||
* stylesheet already owned — `.card` for the panel, `.kv` for label/value rows, `.bar` for
|
||||
* the fill, `.chip.fail` for an incident, `.prov` for the rate tag. Nothing new is painted
|
||||
* and no `.sev-*` family exists.
|
||||
*
|
||||
* Two rules worth naming, because they are the difference between a summary and a lie:
|
||||
* - the bar's fill takes the **payload** verdict through the shared `.bar.ok/.warn/.fail`
|
||||
* tokens, and only a breach paints the *text* (the `.cell` rows' rule): an unrated KV
|
||||
* reading stays an unfilled shell in ink, and a missing one prints `n/a` with no fill at
|
||||
* all rather than a confident 0 %;
|
||||
* - `.prov:empty` collapses. An empty tag is a 1 px pill of nothing, which reads as a
|
||||
* label for a number that is not there.
|
||||
*
|
||||
* `auto-fit` is the single-engine rule: one engine gets one card across the page, never a
|
||||
* red half of a two-column row.
|
||||
*/
|
||||
.strip {
|
||||
display: grid; gap: .85rem; align-items: start;
|
||||
grid-template-columns: repeat(auto-fit, minmax(min(340px, 100%), 1fr));
|
||||
}
|
||||
.strip > .card > .body { display: grid; gap: .6rem; }
|
||||
/* `.kv` lays dt/dd pairs out side by side; a strip row wraps its pair in a `div` so the row
|
||||
can carry the payload key it reads, so the two-column grid moves *inside* the wrapper:
|
||||
one row per line, label left, value right — otherwise every wrapper becomes a grid item
|
||||
and the card reads as two columns of orphaned labels. */
|
||||
.strip .kv { margin: 0; grid-template-columns: 1fr; }
|
||||
.strip .kv > div {
|
||||
display: grid; grid-template-columns: max-content minmax(0, 1fr);
|
||||
gap: .18rem .8rem; align-items: baseline;
|
||||
}
|
||||
.strip .kv dd .unit { font-size: .78em; color: var(--text-muted); margin-inline-start: .18rem; }
|
||||
.strip .kv dd .foot { display: block; font-size: 10.5px; color: var(--text-faint); }
|
||||
.prov:empty { display: none; }
|
||||
.stat-bar {
|
||||
display: grid; grid-template-columns: max-content minmax(0, 1fr) max-content;
|
||||
gap: .55rem; align-items: center;
|
||||
}
|
||||
.stat-bar .label {
|
||||
font-size: 10.5px; text-transform: uppercase; letter-spacing: .06em; color: var(--text-faint);
|
||||
white-space: nowrap;
|
||||
}
|
||||
.stat-bar .value {
|
||||
font-family: var(--mono); font-size: .95rem; font-weight: 600;
|
||||
font-variant-numeric: tabular-nums; text-align: right;
|
||||
}
|
||||
/* the verdict, when the payload sent one — `ok` leaves the digits alone (quiet ink) */
|
||||
.stat-bar.warn .value { color: var(--warn); }
|
||||
.stat-bar.fail .value { color: var(--fail); }
|
||||
|
||||
/* ---- engine slot grid ------------------------------------------------------
|
||||
* `/llm`'s gufo slot cards — `llm_monitor`'s `.slots`/`.slot` (~L499) ported to this
|
||||
* vocabulary: a bordered mini-panel per parallel slot, `.chip` for the state pill, `.dt`/`.dd`
|
||||
* for the numbers. `section-label` heads a block *inside* a card the way `.metric .label`
|
||||
* heads a number, and is the only new word here.
|
||||
*
|
||||
* The busy marker is a **border, a glyph and a word** — never a verdict. A slot generating
|
||||
* tokens is the job on an LLM server (phase 06's reasoning about GPU utilisation), so
|
||||
* `severity.py` rates no slot and nothing in this section reaches for `--ok/--warn/--fail`;
|
||||
* `--accent` is the app's *this is the live one* colour, not a good/bad colour, and the pill
|
||||
* says "processing" so the marker survives monochrome and screen readers alike.
|
||||
*
|
||||
* `.slots` ships `hidden` and `.slots-empty` is a sentence: an engine whose `/slots` did not
|
||||
* answer says so, because a grid of `n/a` cards is the failure this page already had once.
|
||||
*/
|
||||
/* an `<h4>` so the block it heads is navigable, styled as the micro-label the strip's rows use */
|
||||
.section-label {
|
||||
margin: 0; font-size: 10.5px; font-weight: 600; text-transform: uppercase;
|
||||
letter-spacing: .07em; color: var(--text-faint);
|
||||
}
|
||||
.slots {
|
||||
display: grid; gap: .5rem;
|
||||
grid-template-columns: repeat(auto-fill, minmax(min(190px, 100%), 1fr));
|
||||
}
|
||||
.slot {
|
||||
border: 1px solid var(--border); border-radius: var(--radius-sm);
|
||||
padding: .5rem .6rem; background: var(--bg-sunken);
|
||||
}
|
||||
.slot.busy {
|
||||
border-color: color-mix(in srgb, var(--accent) 45%, var(--border));
|
||||
background: linear-gradient(180deg, var(--accent-soft), transparent 60%), var(--bg-sunken);
|
||||
}
|
||||
.slot-top { display: flex; align-items: center; gap: .45rem; margin-bottom: .4rem; }
|
||||
.slot-id {
|
||||
font-family: var(--mono); font-size: 11.5px; color: var(--text-muted); white-space: nowrap;
|
||||
}
|
||||
.slot-mark { color: var(--text-faint); margin-inline-end: .25rem; }
|
||||
.slot.busy .slot-mark { color: var(--accent); }
|
||||
.slot-model {
|
||||
flex: 1; min-width: 0; font-size: 10.5px; color: var(--text-faint);
|
||||
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
|
||||
}
|
||||
.slot-top .chip { margin-inline-start: auto; }
|
||||
.slot dl { display: grid; grid-template-columns: max-content minmax(0, 1fr); gap: .1rem .5rem; margin: 0; }
|
||||
.slot dt {
|
||||
font-size: 10px; text-transform: uppercase; letter-spacing: .06em; color: var(--text-faint);
|
||||
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
|
||||
}
|
||||
.slot dd {
|
||||
margin: 0; font-family: var(--mono); font-size: 12.5px; text-align: right;
|
||||
font-variant-numeric: tabular-nums; overflow-wrap: anywhere;
|
||||
}
|
||||
.slot dd small { display: block; font-size: 10px; color: var(--text-faint); }
|
||||
.slots-empty, .slots-note { margin: 0; font-size: 11.5px; color: var(--text-muted); }
|
||||
|
||||
/* ---- fleet totals ----------------------------------------------------------
|
||||
* The `/llm` aggregate block — `llm_monitor`'s `#fleet`, assembled from vocabulary this
|
||||
* stylesheet already owns: `.card.flat` for the panel, `.metrics`/`.metric` for the sums,
|
||||
* `.dot` and `.chip` for the head. Nothing here colours a number, because nothing here has
|
||||
* a verdict to render (`severity.py` rates no throughput and no queue depth: 0 tok/s on an
|
||||
* idle fleet is healthy), and `llm_monitor`'s coloured left border — a state colour on the
|
||||
* panel that holds the numbers — is precisely the reading this block must not invite. The
|
||||
* one state it does express is *how many engines went into the sum*: the shared dot for
|
||||
* reachability, and the shared `.stale` dim on the count tag when an engine is missing.
|
||||
*/
|
||||
.fleet > .body { padding: .6rem .8rem .7rem; }
|
||||
.fleet-head {
|
||||
display: flex; align-items: center; gap: .45rem; flex-wrap: wrap;
|
||||
font-size: 10.5px; text-transform: uppercase; letter-spacing: .06em; color: var(--text-muted);
|
||||
}
|
||||
.fleet-name { font-weight: 650; }
|
||||
.fleet-note {
|
||||
margin-inline-start: auto; font-size: 10.5px; color: var(--text-faint);
|
||||
text-transform: none; letter-spacing: 0;
|
||||
}
|
||||
.fleet .metrics { margin-top: .55rem; }
|
||||
|
||||
/* ---- hero numbers --------------------------------------------------------
|
||||
* The big glanceable number: label, value + unit, and optional bar and footnote.
|
||||
* Colour comes from the *same* `ok|warn|fail` classes the rest of the stylesheet uses
|
||||
@@ -437,7 +565,6 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
|
||||
.chip.ok { color: var(--ok); background: var(--ok-soft); border-color: color-mix(in srgb, var(--ok) 35%, transparent); }
|
||||
.chip.warn { color: var(--warn); background: var(--warn-soft); border-color: color-mix(in srgb, var(--warn) 35%, transparent); }
|
||||
.chip.fail { color: var(--fail); background: var(--fail-soft); border-color: color-mix(in srgb, var(--fail) 35%, transparent); }
|
||||
.chip.info { color: var(--info); background: var(--info-soft); border-color: color-mix(in srgb, var(--info) 35%, transparent); }
|
||||
.chip.accent { color: var(--accent); background: var(--accent-soft); border-color: color-mix(in srgb, var(--accent) 35%, transparent); }
|
||||
|
||||
.banner {
|
||||
|
||||
@@ -1,10 +1,18 @@
|
||||
/* LLM dashboard.
|
||||
*
|
||||
* Layout (phase 04, per NOTES-chart-set): per engine, two big throughput numbers you can
|
||||
* read across the room, then a compact list of metric cells that each carry their own
|
||||
* sparkline over the *selected* window, percentile foot lines under the latency means, and
|
||||
* alarm chips that exist only while there is an incident. The four trend charts stay, but
|
||||
* half-height and below the panels: they are supporting evidence, not the main event.
|
||||
* Layout (phase 04 per NOTES-chart-set, re-shaped by phase 07): the fleet sums at the top,
|
||||
* then **one stats card per engine** — the KV-cache bar, the two throughputs with their
|
||||
* provenance, lifetime tokens, the sessions this engine reports, its own extra rows, and
|
||||
* alarm chips that exist only while there is an incident — then the engine panels with the
|
||||
* detail the strip does not print (the big throughput pair, the latency means with their p95
|
||||
* feet and their sparklines over the *selected* window, and gufo's **slot grid**), then the
|
||||
* four trend charts, half-height and below: supporting evidence, not the main event.
|
||||
*
|
||||
* Strip and panels hold disjoint rows on purpose (phase 07 task 03): the strip is the
|
||||
* read-across-the-room summary of every engine side by side, the panel is one engine's
|
||||
* detail, and a value printed in both is the pattern this app avoids. The slot grid is not a
|
||||
* second copy of the strip's `slots busy` row: the row counts slots, the grid is what is
|
||||
* happening inside each of them (phase 07 task 04).
|
||||
*
|
||||
* Every throughput cell shows where its number came from, because "78 tok/s" means
|
||||
* different things depending on whether it was scraped from an aggregate gauge, measured as
|
||||
@@ -18,75 +26,149 @@
|
||||
*/
|
||||
'use strict';
|
||||
|
||||
const { $, fetchJSON, setTxt, setDot, num, duration, poll, markStale, severityFrom, hero,
|
||||
eventText } = Dash;
|
||||
const { $, fetchJSON, setTxt, setValue, setDot, num, duration, poll, markStale, severityFrom,
|
||||
hero, eventText } = Dash;
|
||||
|
||||
const state = {
|
||||
range: 3600,
|
||||
panels: new Map(), // engine -> card element
|
||||
panels: new Map(), // engine -> detail card element
|
||||
strips: new Map(), // engine -> stats strip card element
|
||||
slots: new Map(), // engine -> Map(slot id -> { root, pill, model, mark, cells })
|
||||
charts: {},
|
||||
poller: null,
|
||||
pollMs: 2000,
|
||||
};
|
||||
|
||||
const fmtCount = (v) => num(v, 0);
|
||||
const fmtPct = (v) => `${num(v, 0)}%`;
|
||||
const fmtMs = (v) => `${num(v, 0)} ms`;
|
||||
|
||||
/**
|
||||
* What each engine panel shows, driven by flags instead of fourteen equal-size cells:
|
||||
* `hero` is the read-across-the-room row, `spark` is one compact row with a fixed-width
|
||||
* sparkline, `alarm` renders only as a red chip while the counter is non-zero, and
|
||||
* `footnote` is the quiet lifetime strip.
|
||||
* What each engine *panel* keeps after phase 07: the two throughput heroes (the biggest
|
||||
* numbers on the page, deliberately uncoloured — `severity.py` rates no throughput, because
|
||||
* 0 tok/s on an idle server is healthy) and the latency rows, each a mean with its p95
|
||||
* beside it and its own sparkline over the *selected* window.
|
||||
*
|
||||
* `minSpan: 1` on the count rows keeps their axis at least one request tall: history is
|
||||
* stored as rollups, so eight seconds of a single queued request averages to 0.14 and an
|
||||
* axis fitted to that draws a full-height spike for a queue nobody was in.
|
||||
* Everything else the panel used to hold — sessions, queue depth, KV pressure, the cache-hit
|
||||
* and speculative-decoding percents, the lifetime token counter, the alarm chips — moved to
|
||||
* the stats strip above (task 03) rather than being drawn in both places.
|
||||
*
|
||||
* `spark` is the `/api/llm/history` series (without the `llm.` prefix); `scale` converts
|
||||
* stored units to printed ones (a 0-1 ratio to %, seconds to ms) for the plot as well as
|
||||
* the number, so the line and the digits agree. `fixedMax` pins percent-like metrics to a
|
||||
* 0-100 axis — an 8 % window that once held a 92 % spike must not read flat. `only` scopes
|
||||
* a row to the engines that publish it, so a gufo panel never shows a latency mystery and a
|
||||
* vLLM panel never shows a speculative-decoding row it cannot fill.
|
||||
* stored units to printed ones (seconds to ms) for the plot as well as the number, so the
|
||||
* line and the digits agree. `only` scopes a row to the engines that publish it: a gufo
|
||||
* panel never grows a TTFT mystery.
|
||||
*/
|
||||
const CELLS = [
|
||||
{ key: 'decode', label: 'Decode', unit: 'tok/s', digits: 1, hero: true,
|
||||
tip: 'Aggregate generation speed across every busy slot.' },
|
||||
{ key: 'prompt', label: 'Prompt', unit: 'tok/s', digits: 1, hero: true,
|
||||
tip: 'Prefill speed for newly computed tokens.' },
|
||||
{ key: 'sessions', label: 'Sessions', unit: 'active', digits: 0,
|
||||
spark: 'requests_running', format: fmtCount, minSpan: 1 },
|
||||
{ key: 'queued', label: 'Queued', unit: 'waiting', digits: 0,
|
||||
spark: 'requests_waiting', format: fmtCount, minSpan: 1 },
|
||||
{ key: 'cache', label: 'KV cache', unit: '%', digits: 0, fixedMax: 100,
|
||||
spark: 'kv_cache_usage', scale: 100, format: fmtPct,
|
||||
tip: 'Share of the KV cache in use — the only engine-pressure verdict on this page, '
|
||||
+ 'and it comes from the API.' },
|
||||
{ key: 'ttft', label: 'TTFT', unit: 'ms', digits: 0, p95: true, only: ['vllm'],
|
||||
spark: 'ttft_seconds', scale: 1000, format: fmtMs },
|
||||
{ key: 'tpot', label: 'TPOT', unit: 'ms', digits: 1, p95: true, only: ['vllm'],
|
||||
spark: 'tpot_seconds', scale: 1000, format: fmtMs },
|
||||
{ key: 'queue_mean', label: 'Queue time', unit: 'ms', digits: 0, p95: true, only: ['vllm'],
|
||||
spark: 'queue_seconds', scale: 1000, format: fmtMs },
|
||||
{ key: 'cache_hit', label: 'Prefix cache hit', unit: '%', digits: 0, fixedMax: 100,
|
||||
spark: 'prefix_cache_hit_ratio', scale: 100, format: fmtPct },
|
||||
{ key: 'spec_accept', label: 'Spec accept', unit: '%', digits: 0, fixedMax: 100,
|
||||
only: ['gufo'], spark: 'spec_accept_rate', scale: 100, format: fmtPct },
|
||||
];
|
||||
|
||||
/**
|
||||
* The stats strip: one card per configured engine between the fleet block and the panels —
|
||||
* `llm_monitor`'s shared strip (`index.html` ~L743) drawn with vocabulary this stylesheet
|
||||
* already owns (`.card`, `.kv`, `.bar`, `.chip.fail`).
|
||||
*
|
||||
* `key` **is** the `cellValue()` key, so every row here names cells of
|
||||
* `/api/llm/snapshot` (or a documented conversion of one: the 0-1 ratios the engines publish
|
||||
* as fractions, the histogram mean behind `queue_mean`). Nothing was invented in
|
||||
* `sources/llm.py` for this card. `llm_monitor`'s peak-ctx row does not come over — the
|
||||
* phase's assumption is that this strip prints no peak-context cell, and a since-start
|
||||
* maximum is not one of the numbers this layout is for.
|
||||
*
|
||||
* `only` scopes a row to the engines that can fill it, exactly like the panels' `CELLS`: a
|
||||
* gufo card never grows a vLLM queue-time row, a vLLM card never grows the
|
||||
* speculative-decoding row. `rate: true` rows carry their provenance tag and **no colour**
|
||||
* (throughput has no verdict), and `alarm: true` rows are incidents rather than stats — they
|
||||
* render as `.chip.fail`, and only while the counter is non-zero.
|
||||
*/
|
||||
const STRIP_ROWS = [
|
||||
{ key: 'prompt', label: 'prompt tok/s', digits: 1, rate: true,
|
||||
tip: 'Newly computed prompt tokens per second, all slots together.' },
|
||||
{ key: 'decode', label: 'decode tok/s', digits: 1, rate: true,
|
||||
tip: 'Tokens generated per second, every busy slot counted.' },
|
||||
{ key: 'tokens_total', label: 'total tokens', human: true,
|
||||
tip: 'Tokens counted since this engine started.' },
|
||||
{ key: 'slots_busy', label: 'slots busy', digits: 0, only: ['gufo'],
|
||||
tip: 'Parallel slots generating right now, counted from gufo /slots.' },
|
||||
{ key: 'running', label: 'running', digits: 0, only: ['vllm'],
|
||||
tip: 'Requests generating tokens right now.' },
|
||||
{ key: 'waiting', label: 'waiting', digits: 0,
|
||||
tip: 'Requests queued behind them.' },
|
||||
{ key: 'cache_hit', label: 'prefix cache hit', digits: 0, unit: '%',
|
||||
tip: 'Share of prompt tokens served from the cache instead of recomputed.' },
|
||||
{ key: 'spec_accept', label: 'spec accept', digits: 0, unit: '%', only: ['gufo'],
|
||||
tip: 'Speculative-decoding draft tokens the model accepted.' },
|
||||
{ key: 'queue_mean', label: 'queue time', digits: 0, unit: 'ms', p95: true, only: ['vllm'],
|
||||
tip: 'Time a request spends waiting for KV space — mean, with the p95 under it.' },
|
||||
{ key: 'preemptions', label: 'Preemptions', alarm: true,
|
||||
tip: 'Running requests kicked back to the queue — an incident, not a stat.' },
|
||||
{ key: 'device_lost', label: 'GPU context lost', alarm: true,
|
||||
tip: 'The engine lost its GPU context — an incident, not a stat.' },
|
||||
{ key: 'tokens_total', footnote: true },
|
||||
];
|
||||
|
||||
/**
|
||||
* The gufo slot grid's cell table — `llm_monitor`'s `SLOT_FIELDS` (~L1170), rebuilt against
|
||||
* a capture instead of a memory.
|
||||
*
|
||||
* One card per parallel slot; one **fixture key** per cell. `path: 'next_token'` reads the
|
||||
* first entry of that array (`next_token[0]`), which is where the live capture actually puts
|
||||
* `n_remain`. There is no `a ?? b` chain anywhere in here: the deleted table's
|
||||
* `slot.n_prompt ?? slot.prompt_tokens` guessing is exactly why it printed dashes forever
|
||||
* and why nothing it printed could be trusted. The keys are pinned against
|
||||
* `tests/fixtures/slots_gufo.json` by `tests/slots_contract.py`, and `tests/test_api.py`
|
||||
* fails if a cell here names a key the capture does not have.
|
||||
*
|
||||
* Two cells `llm_monitor` has are deliberately **absent**:
|
||||
* - **per-slot speed** — the capture publishes no per-slot rate key (`decode_per_second`
|
||||
* was the old table's invention). A page-side Δn_decoded/Δt would be a measurement with
|
||||
* no estimator, no window label and no `source` string: rule 2 with a confident number on
|
||||
* it. The engine's own decode rate is on its strip card and its panel;
|
||||
* - **the prompt text** — a busy slot's `prompt` is the operator's in-flight request. The
|
||||
* capture script blanks it and this page never reads the key.
|
||||
*
|
||||
* No cell carries a `severity`: a slot generating tokens is the job on an LLM server (the
|
||||
* reasoning phase 06 applied to GPU utilisation), so the busy marker is a border and a glyph
|
||||
* in the brand accent, never a verdict token.
|
||||
*/
|
||||
const SLOT_FIELDS = [
|
||||
{ key: 'req', label: 'request', field: 'id_task' },
|
||||
{ key: 'prompt', label: 'prompt tok', field: 'n_prompt_tokens',
|
||||
sub: 'n_prompt_tokens_processed' },
|
||||
{ key: 'cache', label: 'cache hit', field: 'n_prompt_tokens_cache',
|
||||
ratioAgainst: 'n_prompt_tokens' },
|
||||
{ key: 'generated', label: 'generated', field: 'n_decoded', path: 'next_token',
|
||||
sub: 'prob' },
|
||||
{ key: 'remain', label: 'remaining', field: 'n_remain', path: 'next_token' },
|
||||
{ key: 'ctx', label: 'context', field: 'n_ctx' },
|
||||
];
|
||||
|
||||
/** The engines whose payload carries per-slot JSON — the same fact `_sessions()` decides in
|
||||
* the API: gufo answers `/slots`, vLLM publishes counts and says so in `sessions.detail`. */
|
||||
const SLOT_ENGINES = ['gufo'];
|
||||
|
||||
/** The vLLM panel's one line about why it has no grid (llm_monitor keeps the same line). */
|
||||
const NO_SLOT_JSON = 'vLLM exposes no per-slot JSON — its metrics are the equivalent of '
|
||||
+ 'the session grid.';
|
||||
|
||||
/** What the note under the grid has to say, because an empty grid is not "nothing waiting". */
|
||||
const SLOT_QUEUE_NOTE = 'queued (deferred) requests are not in /slots — this engine’s '
|
||||
+ 'waiting row counts them';
|
||||
|
||||
const HERO_CELLS = CELLS.filter((cell) => cell.hero);
|
||||
const SPARK_CELLS = CELLS.filter((cell) => cell.spark);
|
||||
const ALARM_CELLS = CELLS.filter((cell) => cell.alarm);
|
||||
const ALARM_CELLS = STRIP_ROWS.filter((row) => row.alarm);
|
||||
|
||||
/** One batched history request feeds every chart *and* every cell sparkline. */
|
||||
/** The four trend charts' series — no longer cells, so they are named where they are drawn. */
|
||||
const CHART_SERIES = ['decode_tokens_per_second', 'prompt_tokens_per_second',
|
||||
'kv_cache_usage', 'requests_running', 'requests_waiting'];
|
||||
|
||||
/** One batched history request feeds every chart *and* every panel sparkline. */
|
||||
const HISTORY_SERIES = Array.from(new Set([
|
||||
'decode_tokens_per_second', 'prompt_tokens_per_second',
|
||||
...CHART_SERIES,
|
||||
...SPARK_CELLS.map((cell) => cell.spark),
|
||||
]));
|
||||
|
||||
@@ -111,14 +193,33 @@ function cellValue(engine, metrics, sessions, key) {
|
||||
return cell ? { value: cell.value, source: cell.source, derived: cell.derived,
|
||||
window: cell.window } : { value: null };
|
||||
}
|
||||
case 'sessions': {
|
||||
const value = sessions && sessions.active != null ? sessions.active : null;
|
||||
return { value, source: 'session count', derived: false };
|
||||
case 'running': {
|
||||
const cell = get('requests_processing') || get('num_requests_running');
|
||||
return cell ? { value: cell.value, source: cell.source, derived: false } : { value: null };
|
||||
}
|
||||
case 'queued': {
|
||||
case 'waiting': {
|
||||
const cell = get('requests_deferred') || get('num_requests_waiting');
|
||||
return cell ? { value: cell.value, source: cell.source, derived: false } : { value: null };
|
||||
}
|
||||
case 'slots_busy': {
|
||||
// gufo's /slots answers one row per parallel slot (`sessions.list` — the shape pinned
|
||||
// in `tests/fixtures/slots_gufo.json`), `is_processing` is the authoritative verdict
|
||||
// and the legacy numeric `state` the fallback. Counting rows the server published is
|
||||
// a count of published facts; but when /slots did not answer (`sessions.source` null)
|
||||
// there are no rows to count, so the row falls back to the published
|
||||
// `sessions.active` and says so through the API's own `detail` string. It never
|
||||
// prints a 0 derived from an empty list — that is rule 2, not tidiness.
|
||||
if (sessions && sessions.source && Array.isArray(sessions.list)) {
|
||||
const busy = sessions.list.filter((slot) => slot && (
|
||||
slot.is_processing != null
|
||||
? slot.is_processing === true
|
||||
: Number(slot.state) !== 0)).length;
|
||||
return { value: busy, source: sessions.source, derived: true,
|
||||
total: sessions.list.length };
|
||||
}
|
||||
const value = sessions && sessions.active != null ? sessions.active : null;
|
||||
return { value, source: sessions ? sessions.detail : null, derived: false };
|
||||
}
|
||||
case 'cache': {
|
||||
// the ratio cell carries the API verdict (`sources/llm.py` SEVERITY_CELLS ->
|
||||
// severity.py); it travels with the converted value so the row and its sparkline
|
||||
@@ -201,6 +302,7 @@ function sparkCellsFor(engine) {
|
||||
return SPARK_CELLS.filter((cell) => !cell.only || cell.only.indexOf(engine) >= 0);
|
||||
}
|
||||
|
||||
/** One engine's detail card: the two heroes plus the latency rows it can fill. */
|
||||
function enginePanel(engine) {
|
||||
const card = document.createElement('article');
|
||||
card.className = 'card';
|
||||
@@ -214,6 +316,9 @@ function enginePanel(engine) {
|
||||
<span class="cell-spark" data-spark data-metric="${cell.spark}"></span>
|
||||
${cell.p95 ? '<span class="cell-foot" data-foot></span>' : ''}
|
||||
</div>`).join('');
|
||||
// an engine with no latency rows gets no `.cells` box at all — an empty grid in a
|
||||
// `.stack` is padding around nothing, and a blank panel reads as "no data"
|
||||
const cellList = rows ? `<div class="cells">${rows}</div>` : '';
|
||||
card.innerHTML = `
|
||||
<header>
|
||||
<span class="dot idle"></span>
|
||||
@@ -225,15 +330,39 @@ function enginePanel(engine) {
|
||||
<div class="body stack">
|
||||
<div class="banner err" hidden></div>
|
||||
<div class="heroes">${heroes}</div>
|
||||
<div class="alarms" data-k="alarms" hidden></div>
|
||||
<div class="cells">${rows}</div>
|
||||
<div class="faint mono sessions-note" style="font-size:11.5px"></div>
|
||||
<div class="faint mono token-note" style="font-size:11px"></div>
|
||||
<div class="faint mono debug" style="font-size:11px"></div>
|
||||
${cellList}
|
||||
${slotSection(engine)}
|
||||
<div class="faint mono debug" style="font-size:11px" hidden></div>
|
||||
</div>`;
|
||||
return card;
|
||||
}
|
||||
|
||||
/**
|
||||
* The per-slot half of an engine card, chosen by the engine's **name**, not by what arrived.
|
||||
*
|
||||
* gufo's panel gets `llm_monitor`'s `sessions (GET /slots)` label with the keyed card grid
|
||||
* under it, and the note that a queued (deferred) request is not in `/slots` at all — the
|
||||
* waiting row counts those, and without that line an idle grid reads as "nothing waiting".
|
||||
*
|
||||
* vLLM's panel gets one sentence and **no grid**: `_sessions()` publishes counts for it, so a
|
||||
* grid of `n/a`s would be a component of permanent dead cells for a server that cannot answer
|
||||
* it. Both grid and empty state ship `hidden` — the panel never renders a card of nothing.
|
||||
*
|
||||
* `SLOT_ENGINES` is the gate rather than `sessions.list.length` on purpose: an engine that
|
||||
* has not answered yet must still show the label and the honest empty sentence, which is how
|
||||
* an operator learns the difference between "idle" and "nobody asked".
|
||||
*/
|
||||
function slotSection(engine) {
|
||||
if (SLOT_ENGINES.indexOf(engine) < 0) {
|
||||
return `<p class="slots-note" data-slots-note>${NO_SLOT_JSON}</p>`;
|
||||
}
|
||||
return `
|
||||
<h4 class="section-label" data-slots-label>sessions (GET /slots)</h4>
|
||||
<div class="slots" data-slots role="list" hidden></div>
|
||||
<p class="slots-empty" data-slots-empty hidden></p>
|
||||
<p class="slots-note" data-slots-note hidden>${SLOT_QUEUE_NOTE}</p>`;
|
||||
}
|
||||
|
||||
function renderPanel(card, enginePayload, stateRow) {
|
||||
const metrics = enginePayload.metrics || {};
|
||||
const sessions = enginePayload.sessions || {};
|
||||
@@ -297,25 +426,24 @@ function renderPanel(card, enginePayload, stateRow) {
|
||||
}
|
||||
}
|
||||
|
||||
renderAlarms(card, enginePayload);
|
||||
// The per-slot grid: keyed cards, patched in place, or the honest empty sentence.
|
||||
renderSlotGrid(card, enginePayload);
|
||||
|
||||
const note = q('.sessions-note');
|
||||
setTxt(note, sessions.detail || `${sessions.active == null ? 'n/a' : sessions.active} active sessions`);
|
||||
// a lifetime counter is a footnote, not a glance metric (NOTES-chart-set, demoted)
|
||||
const tokens = cellValue(enginePayload.engine, metrics, sessions, 'tokens_total');
|
||||
const parts = tokens.value == null ? null : humanNumber(tokens.value);
|
||||
setTxt(q('.token-note'), parts
|
||||
? `lifetime ${parts.text} ${parts.unit} · ${tokens.source || 'unknown source'}`
|
||||
: 'lifetime tokens n/a');
|
||||
// sessions, lifetime tokens and the alarm chips live on this engine's strip card now
|
||||
// (phase 07 task 03) — one value, one home — so the panel says nothing about them here.
|
||||
const upstream = q('.chip.upstream');
|
||||
if (enginePayload.upstream) {
|
||||
upstream.hidden = false;
|
||||
upstream.textContent = enginePayload.upstream.replace(/^https?:\/\//, '');
|
||||
upstream.title = enginePayload.upstream;
|
||||
}
|
||||
setTxt(q('.debug'), enginePayload.unparsed != null
|
||||
? `${enginePayload.series_count} series · ${enginePayload.unparsed} unparsed`
|
||||
: '');
|
||||
// how much of this engine's /metrics the parser could not name — hidden while there is
|
||||
// nothing to admit, so the card ends at its numbers
|
||||
const debugText = enginePayload.unparsed != null
|
||||
? `${enginePayload.series_count} series · ${enginePayload.unparsed} unparsed` : '';
|
||||
const debugEl = q('.debug');
|
||||
debugEl.hidden = debugText === '';
|
||||
setTxt(debugEl, debugText);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -325,6 +453,9 @@ function renderPanel(card, enginePayload, stateRow) {
|
||||
* zero the row does not exist: the container is emptied and hidden, and only a non-zero —
|
||||
* an *incident*, not a graded threshold — puts a `.chip.fail` back. `sources/llm.py`
|
||||
* already treats any non-zero gufo_device_lost_total this way.
|
||||
*
|
||||
* Phase 07 task 03 renders them on the engine's **strip card**: an incident belongs with the
|
||||
* summary you read first, and the row itself still only exists while it is true.
|
||||
*/
|
||||
function renderAlarms(card, enginePayload) {
|
||||
const host = card.querySelector('[data-k="alarms"]');
|
||||
@@ -351,29 +482,322 @@ function renderAlarms(card, enginePayload) {
|
||||
host.hidden = chips.length === 0;
|
||||
}
|
||||
|
||||
function renderSlots(payloads) {
|
||||
const gufo = payloads.find((p) => p.engine === 'gufo');
|
||||
const rows = gufo && gufo.sessions ? gufo.sessions.list || [] : [];
|
||||
const tbody = $('#slots-table tbody');
|
||||
$('#slots-empty').hidden = rows.length > 0;
|
||||
$('#slots-card').hidden = !gufo;
|
||||
tbody.innerHTML = rows.map((slot) => {
|
||||
const id = slot.id != null ? slot.id : slot.slot != null ? slot.slot : '—';
|
||||
const stateText = slot.state || (slot.idle === true ? 'idle' : slot.idle === false ? 'busy' : '—');
|
||||
const withText = (slot.with_messages && slot.with_messages.length) || slot.with_message;
|
||||
const nPrompt = slot.n_prompt != null ? slot.n_prompt : (slot.prompt_tokens != null ? slot.prompt_tokens : null);
|
||||
const nPredict = slot.n_predict != null ? slot.n_predict : (slot.generated_tokens != null ? slot.generated_tokens : null);
|
||||
const cache = slot.cache != null ? slot.cache : slot.cache_usable;
|
||||
const speed = slot.decode_per_second || slot.tokens_per_second || null;
|
||||
const cell = (value, digits) => `<td class="num">${value == null ? '<span class="mute">—</span>' : num(value, digits == null ? 0 : digits)}</td>`;
|
||||
return `<tr>
|
||||
<td class="name mono" style="font-size:11.5px">${DashCharts.escapeHtml(id)}</td>
|
||||
<td>${stateText === 'busy' || stateText === 'processing' ? '<span class="chip ok">busy</span>'
|
||||
: stateText === 'idle' ? '<span class="chip">idle</span>'
|
||||
: `<span class="chip warn">${DashCharts.escapeHtml(stateText)}</span>`}</td>
|
||||
<td>${withText ? '<span class="chip info">has msg</span>' : '<span class="mute">—</span>'}</td>
|
||||
${cell(nPrompt)}${cell(nPredict)}${cell(cache)}${cell(speed, 1)}</tr>`;
|
||||
}).join('');
|
||||
/** The strip rows this engine publishes — the same `only` scoping as the panels' cells. */
|
||||
function stripRowsFor(engine) {
|
||||
return STRIP_ROWS.filter((row) => !row.only || row.only.indexOf(engine) >= 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* One engine's stats card: head (reachability dot, name, age), the KV-cache bar, the rows,
|
||||
* and the alarm chips. Built once per engine, then patched in place — like every other
|
||||
* component here, so a poll cannot restart a CSS transition or steal focus.
|
||||
*/
|
||||
function stripCard(engine) {
|
||||
const card = document.createElement('article');
|
||||
card.className = 'card stat';
|
||||
card.dataset.engine = engine;
|
||||
const rows = stripRowsFor(engine).filter((row) => !row.alarm).map((row) => `
|
||||
<div data-row="${row.key}">
|
||||
<dt>${row.label}</dt>
|
||||
<dd><span data-v>n/a</span><span class="unit" data-unit></span>`
|
||||
+ `<span class="prov" data-prov></span>`
|
||||
+ `${row.p95 ? '<span class="foot" data-foot></span>' : ''}</dd>
|
||||
</div>`).join('');
|
||||
card.innerHTML = `
|
||||
<header>
|
||||
<span class="dot idle"></span>
|
||||
<h3>${DashCharts.escapeHtml(engine)}</h3>
|
||||
<span class="spacer"></span>
|
||||
<span class="faint mono updated"></span>
|
||||
</header>
|
||||
<div class="body stack">
|
||||
<div class="stat-bar" data-row="kv">
|
||||
<span class="label">KV cache</span>
|
||||
<span class="bar" data-kv-bar><i></i></span>
|
||||
<span class="value" data-kv-value>n/a</span>
|
||||
</div>
|
||||
<dl class="kv">${rows}</dl>
|
||||
<div class="alarms" data-k="alarms" hidden></div>
|
||||
</div>`;
|
||||
return card;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fill one strip card from the engine's payload — nothing else.
|
||||
*
|
||||
* The KV bar is the only cell on this page with a verdict, and the colour comes from the
|
||||
* payload (`sources/llm.py` SEVERITY_CELLS → `severity.py`), the same ratio→percent
|
||||
* conversion the panel's cell used to make: the bar says *how full*, the verdict says
|
||||
* *what colour*, and an engine that publishes no KV gauge gets an empty bar and a faint
|
||||
* `n/a` rather than a 0 % claim. Everything else on the card is ink: throughput, queue
|
||||
* depth and cache-hit percents have no row in `severity.py`, because none of them is good
|
||||
* or bad at a number the operator can read off a bar.
|
||||
*/
|
||||
function renderStripCard(card, enginePayload, stateRow) {
|
||||
const engine = enginePayload.engine;
|
||||
const metrics = enginePayload.metrics || {};
|
||||
const sessions = enginePayload.sessions || {};
|
||||
const q = (sel) => card.querySelector(sel);
|
||||
|
||||
// reachability, not a verdict: the dot and the shared `.stale` dim, exactly as the panels
|
||||
// and the fleet block draw them
|
||||
const age = stateRow ? Dash.ageSeconds(stateRow.last_ok) : Infinity;
|
||||
setDot(q('.dot'), !stateRow ? 'idle' : stateRow.ok ? (age > 15 ? 'warn' : 'ok') : 'fail');
|
||||
setTxt(q('.updated'), stateRow ? (stateRow.last_ok == null
|
||||
? (stateRow.error || 'never reached')
|
||||
: (age > 15 ? `stale ${duration(age)}` : `updated ${duration(age)}`)) : '—');
|
||||
markStale(card, stateRow && (!stateRow.ok || age > 15));
|
||||
|
||||
const kv = cellValue(engine, metrics, sessions, 'cache');
|
||||
const barRow = q('[data-row="kv"]');
|
||||
const bar = q('[data-kv-bar]');
|
||||
const sev = severityFrom(kv.severity);
|
||||
[barRow, bar].forEach((el) => {
|
||||
['ok', 'warn', 'fail'].forEach((name) => el.classList.toggle(name, name === sev));
|
||||
});
|
||||
const fill = bar.querySelector('i');
|
||||
if (fill) fill.style.width = `${kv.value == null ? 0 : Math.max(0, Math.min(100, kv.value))}%`;
|
||||
setTxt(q('[data-kv-value]'), kv.value == null ? null : `${num(kv.value, 0)}%`);
|
||||
barRow.title = kv.source
|
||||
? `${kv.source} · the share of the KV cache in use`
|
||||
: 'this engine publishes no KV-cache gauge';
|
||||
|
||||
for (const row of stripRowsFor(engine)) {
|
||||
if (row.alarm) continue; // chips below, never a row
|
||||
const el = q(`[data-row="${row.key}"]`);
|
||||
if (!el) continue;
|
||||
const value = cellValue(engine, metrics, sessions, row.key);
|
||||
const parts = row.human && value.value != null ? humanNumber(value.value) : null;
|
||||
setTxt(el.querySelector('[data-v]'),
|
||||
value.value == null ? null : (parts ? parts.text : num(value.value, row.digits)));
|
||||
// "n/a %" would read as a measurement of nothing, so the unit leaves with the number
|
||||
setTxt(el.querySelector('[data-unit]'),
|
||||
value.value == null ? '' : (parts ? parts.unit : (row.unit || '')));
|
||||
setTxt(el.querySelector('[data-prov]'), row.rate ? provenance(value) : '');
|
||||
if (row.p95) {
|
||||
const p95 = p95Value(metrics, row.key);
|
||||
setTxt(el.querySelector('[data-foot]'),
|
||||
p95 == null ? 'p95 n/a' : `p95 ${num(p95, row.digits)} ${row.unit || ''}`);
|
||||
}
|
||||
const notes = [];
|
||||
if (row.tip) notes.push(row.tip);
|
||||
if (value.total != null) notes.push(`${value.value} of ${value.total} slots busy`);
|
||||
if (value.source) notes.push(value.source);
|
||||
// the API's own sentence about how *this* engine counts sessions rides on the rows it
|
||||
// explains (gufo's says so when /slots did not answer; vLLM's says there are no sessions)
|
||||
if (sessions.detail && (row.key === 'running' || row.key === 'waiting'
|
||||
|| row.key === 'slots_busy')) notes.push(sessions.detail);
|
||||
el.title = notes.join(' · ');
|
||||
}
|
||||
|
||||
renderAlarms(card, enginePayload);
|
||||
}
|
||||
|
||||
/** An engine that leaves the payload leaves the strip and the panels — never a stale card. */
|
||||
function pruneEngines(names) {
|
||||
[state.strips, state.panels].forEach((map) => {
|
||||
for (const engine of Array.from(map.keys())) {
|
||||
if (names.indexOf(engine) < 0) {
|
||||
const card = map.get(engine);
|
||||
if (card) card.remove();
|
||||
map.delete(engine);
|
||||
}
|
||||
}
|
||||
});
|
||||
// and its slot cards go with the panel: keeping their refs would re-append DOM nodes that
|
||||
// were removed two polls ago the moment the engine came back
|
||||
for (const engine of Array.from(state.slots.keys())) {
|
||||
if (names.indexOf(engine) < 0) state.slots.delete(engine);
|
||||
}
|
||||
}
|
||||
|
||||
/* ---- the gufo slot grid (phase 07 task 04) ------------------------------- */
|
||||
|
||||
/** A number a slot row actually published, or `null`. `-1` sentinels are handled per cell. */
|
||||
function slotNumber(value) {
|
||||
return typeof value === 'number' && isFinite(value) ? value : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The **only** place a `/slots` row is indexed.
|
||||
*
|
||||
* One key, named by `SLOT_FIELDS`, and no second key when it is missing: a cell whose key is
|
||||
* absent prints `n/a`. That single-entry-point shape is what makes the rule checkable —
|
||||
* `tests/test_api.py` reads the table and asserts every named key exists in
|
||||
* `tests/fixtures/slots_gufo.json`, so a renamed field fails a test instead of printing a
|
||||
* number nobody recognised.
|
||||
*/
|
||||
function slotValue(row, field, path) {
|
||||
const source = path === 'next_token'
|
||||
? (Array.isArray(row.next_token) ? row.next_token[0] : null)
|
||||
: row;
|
||||
if (!source || typeof source !== 'object') return null;
|
||||
return slotNumber(source[field]);
|
||||
}
|
||||
|
||||
/** The number and the sub-line for one cell of one slot row; `null` prints `n/a`. */
|
||||
function slotCell(row, cell) {
|
||||
const value = slotValue(row, cell.field, cell.path);
|
||||
switch (cell.key) {
|
||||
case 'req':
|
||||
// `-1` is llama.cpp's "nothing in this slot": the slot is idle, it has no request id,
|
||||
// so it says so in a word rather than printing a negative number or an invented 0.
|
||||
return { value: value == null ? null : (value < 0 ? 'idle' : `#${value}`) };
|
||||
case 'prompt': {
|
||||
// `processed` is a second fact (how much of the prompt has been read), not a fallback
|
||||
const processed = slotValue(row, cell.sub);
|
||||
return { value,
|
||||
sub: processed != null && processed !== value
|
||||
? `processed ${num(processed, 0)}` : null };
|
||||
}
|
||||
case 'cache': {
|
||||
// the driver publishes cached *tokens*; the share is those tokens over the prompt it is
|
||||
// a share of. No verdict: a cache hit is neither good nor bad at a number.
|
||||
const prompt = slotValue(row, cell.ratioAgainst);
|
||||
return { value,
|
||||
sub: value != null && prompt > 0
|
||||
? `${num(Math.min(100, (value / prompt) * 100), 0)}% of prompt` : null };
|
||||
}
|
||||
case 'generated': {
|
||||
// `prob` is optional in the capture — this gufo publishes it on no row, so the sub-line
|
||||
// stays empty rather than printing `p=NaN` or `p=n/a`
|
||||
const prob = slotValue(row, cell.sub, cell.path);
|
||||
return { value, sub: prob == null ? null : `p=${prob.toFixed(2)}` };
|
||||
}
|
||||
case 'remain':
|
||||
// `-1` is "nothing pending", not a countdown: `n/a`, never `-1` and never the 0 that
|
||||
// would read as "one token from done"
|
||||
return { value: value != null && value >= 0 ? value : null };
|
||||
default:
|
||||
return { value };
|
||||
}
|
||||
}
|
||||
|
||||
/** Build one slot card. Called the first time a slot id is seen and never again. */
|
||||
function slotCard(id) {
|
||||
const root = document.createElement('div');
|
||||
root.className = 'slot';
|
||||
root.setAttribute('role', 'listitem'); // a list of slots under the section's heading
|
||||
const cells = SLOT_FIELDS.map((cell) => `
|
||||
<dt>${cell.label}</dt>
|
||||
<dd data-slot-cell="${cell.key}"><span data-v>n/a</span><small data-sub></small></dd>`).join('');
|
||||
root.innerHTML = `
|
||||
<div class="slot-top">
|
||||
<span class="slot-id"><span class="slot-mark" data-mark aria-hidden="true">○</span>slot ${DashCharts.escapeHtml(id)}</span>
|
||||
<span class="slot-model" data-model></span>
|
||||
<span class="chip" data-pill>idle</span>
|
||||
</div>
|
||||
<dl>${cells}</dl>`;
|
||||
const cells_ = {};
|
||||
for (const cell of SLOT_FIELDS) {
|
||||
const box = root.querySelector(`[data-slot-cell="${cell.key}"]`);
|
||||
cells_[cell.key] = { value: box.querySelector('[data-v]'), sub: box.querySelector('[data-sub]') };
|
||||
}
|
||||
return { root, cells: cells_, pill: root.querySelector('[data-pill]'),
|
||||
model: root.querySelector('[data-model]'), mark: root.querySelector('[data-mark]') };
|
||||
}
|
||||
|
||||
/** Text that says nothing when there is nothing: a sub-line prints blank, never `n/a`. */
|
||||
function slotSub(el, text) {
|
||||
const value = text == null ? '' : String(text);
|
||||
if (el.textContent !== value) el.textContent = value;
|
||||
}
|
||||
|
||||
/** Patch one card from one row — the numbers, the state pill, the glyph, the model. */
|
||||
function renderSlotCard(refs, row, source) {
|
||||
// `is_processing` is the authoritative verdict the contract pins; `state` only labels a
|
||||
// number the boolean cannot express, and an unknown number is labelled as what it is
|
||||
// rather than being called idle.
|
||||
const processing = row.is_processing === true;
|
||||
const other = !processing && Number.isFinite(Number(row.state)) && Number(row.state) !== 0;
|
||||
refs.root.classList.toggle('busy', processing);
|
||||
slotSub(refs.mark, processing ? '●' : '○');
|
||||
const pill = processing ? 'processing' : other ? `state ${row.state}` : 'idle';
|
||||
slotSub(refs.pill, pill);
|
||||
refs.pill.title = `${source || 'slot state'} · is_processing=${String(row.is_processing)}`
|
||||
+ ` · state=${String(row.state)}`;
|
||||
const model = typeof row.model === 'string' && row.model ? row.model : null;
|
||||
setTxt(refs.model, model ? (row.speculative === true ? `${model} +spec` : model) : null);
|
||||
for (const cell of SLOT_FIELDS) {
|
||||
const out = slotCell(row, cell);
|
||||
const box = refs.cells[cell.key];
|
||||
setTxt(box.value, out.value == null ? null
|
||||
: (typeof out.value === 'string' ? out.value : num(out.value, 0)));
|
||||
slotSub(box.sub, out.sub);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* One keyed card per `/slots` row, created once and patched in place afterwards.
|
||||
*
|
||||
* Keying is the slot's own `id` (`tests/slots_contract.py` fails if a capture ever answers
|
||||
* duplicate ids, because keyed cards would collide). Rebuilding a busy slot's card on every
|
||||
* 2 s poll would re-announce the whole card to assistive tech for a difference of a few
|
||||
* tokens, so `appendChild` of an existing node is used to keep DOM order == slot order and
|
||||
* everything else is a text patch.
|
||||
*
|
||||
* The second half of this function is the honest empty state, which is the part that decides
|
||||
* whether the grid can be trusted:
|
||||
* - `/slots` did not answer (`sessions.source` is null) → `sessions.detail`, the API's own
|
||||
* sentence about why there are no rows — not a card of nothing, not a 0-slot claim;
|
||||
* - it answered with no rows → "engine reports 0 slots", which is a fact about that answer;
|
||||
* - rows arrived → cards, and the note about queued requests the list cannot contain.
|
||||
*/
|
||||
function renderSlotGrid(card, enginePayload) {
|
||||
const grid = card.querySelector('[data-slots]');
|
||||
if (!grid) {
|
||||
// vLLM: the one-line note only, carrying the API's own words on its session counts
|
||||
const note = card.querySelector('[data-slots-note]');
|
||||
if (note && enginePayload.sessions && enginePayload.sessions.detail) {
|
||||
note.title = enginePayload.sessions.detail;
|
||||
}
|
||||
return;
|
||||
}
|
||||
const sessions = enginePayload.sessions || {};
|
||||
let byId = state.slots.get(enginePayload.engine);
|
||||
if (!byId) {
|
||||
byId = new Map();
|
||||
state.slots.set(enginePayload.engine, byId);
|
||||
}
|
||||
const empty = card.querySelector('[data-slots-empty]');
|
||||
const note = card.querySelector('[data-slots-note]');
|
||||
const rows = Array.isArray(sessions.list) ? sessions.list : [];
|
||||
const answered = Boolean(sessions.source);
|
||||
|
||||
if (!answered || !rows.length) {
|
||||
byId.forEach((refs) => refs.root.remove());
|
||||
byId.clear();
|
||||
grid.hidden = true;
|
||||
setTxt(empty, answered ? 'engine reports 0 slots'
|
||||
: (sessions.detail || 'no /slots answer from this engine'));
|
||||
empty.hidden = false;
|
||||
note.hidden = true;
|
||||
return;
|
||||
}
|
||||
|
||||
empty.hidden = true;
|
||||
grid.hidden = false;
|
||||
grid.title = sessions.source; // which upstream these rows came from, on hover
|
||||
const seen = new Set();
|
||||
rows.forEach((row, index) => {
|
||||
if (!row || typeof row !== 'object') return;
|
||||
const id = slotNumber(row.id) == null ? index : slotNumber(row.id);
|
||||
seen.add(id);
|
||||
let refs = byId.get(id);
|
||||
if (!refs) {
|
||||
refs = slotCard(id);
|
||||
byId.set(id, refs);
|
||||
}
|
||||
grid.appendChild(refs.root); // moves an existing node: order stays slot order
|
||||
renderSlotCard(refs, row, sessions.source);
|
||||
});
|
||||
for (const [id, refs] of Array.from(byId)) {
|
||||
if (!seen.has(id)) {
|
||||
refs.root.remove();
|
||||
byId.delete(id);
|
||||
}
|
||||
}
|
||||
note.hidden = false;
|
||||
}
|
||||
|
||||
function makeCharts() {
|
||||
@@ -453,8 +877,9 @@ function drawSparks(response) {
|
||||
span: state.range,
|
||||
fixedMax: cell.fixedMax != null ? cell.fixedMax : null,
|
||||
minSpan: cell.minSpan != null ? cell.minSpan : null,
|
||||
// the only rated spark is KV, and its colour is the API's verdict; everything else
|
||||
// draws the faint unrated stroke, because a shape is not a verdict
|
||||
// the verdict travels from the row, and since phase 07 no panel row has one (KV
|
||||
// moved to the strip's bar), so these sparks draw the faint unrated stroke: a shape
|
||||
// is not a verdict
|
||||
severity: row.dataset.sev || '',
|
||||
labels: { max: true },
|
||||
format: cell.format,
|
||||
@@ -465,21 +890,68 @@ function drawSparks(response) {
|
||||
}
|
||||
|
||||
/**
|
||||
* The engine-totals card: only what no single panel can say.
|
||||
* The aggregate block: one sum per metric, above the panels — `llm_monitor`'s `#fleet`.
|
||||
*
|
||||
* Summed decode/prompt/KV left with phase 04 (per-engine is the honest view), and the card
|
||||
* hides entirely below two engines — a fleet of one does not need a fleet row.
|
||||
* Every number is `fleet_totals()`'s, computed in the API over the engines whose poll
|
||||
* answered. A dead engine keeps its last-good payload in its **own** panel (correct there,
|
||||
* because that panel is labelled stale) but is dropped from a total presented as current, so
|
||||
* the tag counts what is answering, names who is missing in its tooltip, and dims while the
|
||||
* sums below are partial. Hidden under two engines: a cross-engine sum of one engine just
|
||||
* duplicates that engine's panel.
|
||||
*
|
||||
* Nothing on the block is coloured, and no `severity` reaches it: `severity.py` rates no
|
||||
* throughput and no queue depth, because 0 tok/s on an idle fleet is healthy. The dot is the
|
||||
* shared *reachability* dot — the same distinction the engine panels draw between their
|
||||
* numbers and their dots — and the seam the two engines disagree about (`fleet.seams`, the
|
||||
* API's own words) rides on the prompt sum as a tooltip rather than as ink.
|
||||
*
|
||||
* `metric` is the key inside the snapshot's `fleet` object; `tests/test_api.py` fails if one
|
||||
* of them stops existing, which is what keeps a rename from becoming a permanent `n/a`.
|
||||
*/
|
||||
function renderFleet(fleet) {
|
||||
const card = $('#totals');
|
||||
if (!card) return;
|
||||
const FLEET_FIELDS = [
|
||||
{ id: 'f-running', metric: 'requests_active', digits: 0 },
|
||||
{ id: 'f-waiting', metric: 'requests_waiting', digits: 0 },
|
||||
{ id: 'f-prompt', metric: 'prompt_tokens_per_second', digits: 1, unit: 'tok/s' },
|
||||
{ id: 'f-decode', metric: 'decode_tokens_per_second', digits: 1, unit: 'tok/s' },
|
||||
{ id: 'f-tokens', metric: 'tokens_total', human: true },
|
||||
];
|
||||
|
||||
function renderFleet(fleet, engines) {
|
||||
const block = $('#fleet');
|
||||
if (!block) return;
|
||||
const total = fleet && fleet.engines_total != null ? fleet.engines_total : 0;
|
||||
card.hidden = total < 2;
|
||||
const live = fleet && fleet.engines_live != null ? fleet.engines_live : 0;
|
||||
block.hidden = total < 2; // one engine: the sum would duplicate that engine's own panel
|
||||
if (!total) return;
|
||||
setTxt($('#f-live'), fleet.engines_live == null ? 'n/a' : String(fleet.engines_live));
|
||||
setTxt($('#f-total'), fleet.engines_total == null ? '' : `of ${fleet.engines_total}`);
|
||||
setTxt($('#f-active'), fleet.requests_active == null ? 'n/a' : num(fleet.requests_active, 0));
|
||||
setTxt($('#f-waiting'), fleet.requests_waiting == null ? 'n/a' : num(fleet.requests_waiting, 0));
|
||||
|
||||
// the same liveness rule `fleet_totals()` uses — a payload with no metrics answered nothing
|
||||
const down = (engines || []).filter((p) => !p.metrics || !Object.keys(p.metrics).length);
|
||||
setDot($('#fleet-dot'), live === 0 ? 'fail' : live < total ? 'warn' : 'ok');
|
||||
const tag = $('#fleet-count');
|
||||
setTxt(tag, live ? `${live} of ${total} engines` : `${total} engines · no data yet`);
|
||||
markStale(tag, down.length > 0); // dim it: the totals underneath are not the whole fleet
|
||||
tag.title = down.length
|
||||
? `${down.map((p) => p.engine).join(', ')} not answering — excluded from every sum below`
|
||||
: live ? 'every engine answered this poll' : 'no engine has answered yet';
|
||||
|
||||
for (const field of FLEET_FIELDS) {
|
||||
const el = $(`#${field.id}`);
|
||||
if (!el) continue;
|
||||
const raw = fleet[field.metric];
|
||||
const parts = field.human ? humanNumber(raw) : null;
|
||||
const value = field.human ? (parts ? parts.text : null) : raw;
|
||||
setValue(el, value, field.digits);
|
||||
const unitEl = el.nextElementSibling;
|
||||
if (unitEl && unitEl.classList.contains('unit')) {
|
||||
// "n/a tok/s" would read as a measurement of nothing — the unit goes with the number
|
||||
setTxt(unitEl, value == null ? '' : (parts ? parts.unit : (field.unit || '')));
|
||||
}
|
||||
}
|
||||
// the API's own words on what this sum cannot be; no assignment when it has none, so the
|
||||
// tooltip never becomes the string "undefined"
|
||||
const prompt = $('#f-prompt');
|
||||
const promptMetric = fleet.seams && prompt && prompt.closest ? prompt.closest('.metric') : null;
|
||||
if (promptMetric) promptMetric.title = fleet.seams;
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
@@ -494,19 +966,29 @@ async function refresh() {
|
||||
(snapshot.states || []).forEach((row) => { rows[row.label.toLowerCase()] = row; });
|
||||
const payloads = snapshot.engines || [];
|
||||
const names = payloads.map((p) => p.engine);
|
||||
pruneEngines(names);
|
||||
payloads.forEach((payload) => {
|
||||
const stateRow = rows[payload.engine]
|
||||
|| (snapshot.states || []).find((row) => row.key.endsWith(payload.engine));
|
||||
let strip = state.strips.get(payload.engine);
|
||||
if (!strip) {
|
||||
strip = stripCard(payload.engine);
|
||||
state.strips.set(payload.engine, strip);
|
||||
$('#strip').appendChild(strip);
|
||||
}
|
||||
renderStripCard(strip, payload, stateRow);
|
||||
let card = state.panels.get(payload.engine);
|
||||
if (!card) {
|
||||
card = enginePanel(payload.engine);
|
||||
state.panels.set(payload.engine, card);
|
||||
$('#engines').appendChild(card);
|
||||
}
|
||||
renderPanel(card, payload, rows[payload.engine] ||
|
||||
(snapshot.states || []).find((row) => row.key.endsWith(payload.engine)));
|
||||
renderPanel(card, payload, stateRow);
|
||||
});
|
||||
// the strip is the summary of the engines that *exist*: no engines, no cards, no strip
|
||||
$('#strip').hidden = payloads.length === 0;
|
||||
$('#engines-empty').hidden = payloads.length > 0;
|
||||
renderFleet(snapshot.fleet);
|
||||
renderSlots(payloads);
|
||||
renderFleet(snapshot.fleet, payloads);
|
||||
const fleet = snapshot.fleet || {};
|
||||
$('#sub').textContent = `${fleet.engines_live || 0}/${fleet.engines_total || 0} engines · ` +
|
||||
`${fleet.requests_active == null ? 'n/a' : num(fleet.requests_active, 0)} active · ` +
|
||||
|
||||
@@ -325,14 +325,20 @@ async function runPending() {
|
||||
// ---- charts ---------------------------------------------------------------
|
||||
function makeCharts() {
|
||||
const common = { height: 160, xSpan: state.range };
|
||||
// Every plot names itself: the legend below is a sibling that labels nothing, so an
|
||||
// unlabelled `<svg>` is silence for assistive tech (`/system` phase 03, `/llm` phase 04).
|
||||
state.charts.charge = new DashCharts.LineChart('#chart-charge', Object.assign({}, common,
|
||||
{ yMin: 0, yMax: 100, yFormat: (v) => num(v, 0) }));
|
||||
{ yMin: 0, yMax: 100, yFormat: (v) => num(v, 0),
|
||||
ariaLabel: 'Battery charge in per cent over the selected window, one line per UPS' }));
|
||||
state.charts.runtime = new DashCharts.LineChart('#chart-runtime', Object.assign({}, common,
|
||||
{ yMin: 0, yFormat: (v) => duration(v) }));
|
||||
{ yMin: 0, yFormat: (v) => duration(v),
|
||||
ariaLabel: 'Battery runtime remaining over the selected window, one line per UPS' }));
|
||||
state.charts.voltage = new DashCharts.LineChart('#chart-voltage', Object.assign({}, common,
|
||||
{ yFormat: (v) => num(v, 1) }));
|
||||
{ yFormat: (v) => num(v, 1),
|
||||
ariaLabel: 'Battery and line voltage over the selected window, one pair of lines per UPS' }));
|
||||
state.charts.load = new DashCharts.LineChart('#chart-load', Object.assign({}, common,
|
||||
{ yMin: 0, yFormat: (v) => num(v, 0) }));
|
||||
{ yMin: 0, yFormat: (v) => num(v, 0),
|
||||
ariaLabel: 'UPS load in per cent over the selected window, one line per UPS' }));
|
||||
}
|
||||
|
||||
function points(response, upsName, metric) {
|
||||
|
||||
Vendored
+1
-1
File diff suppressed because one or more lines are too long
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+1
-1
@@ -1 +1 @@
|
||||
{"ok": true, "schema": 1, "agent_version": "1.0.0", "ts": 1792178443.0656452, "duration_ms": 10.22, "host": {"hostname": "gateway", "os_pretty": "AlmaLinux 10.2 (Lavender Lion)", "os_id": "almalinux", "kernel": "6.12.0-211.65.1.el10_2.x86_64_v2", "arch": "x86_64", "python": "3.12.15", "cpu_model": "AMD Ryzen Threadripper 3960X 24-Core Processor", "cpu_cores_logical": 2, "cpu_cores_physical": 2, "memory_total_bytes": 2051493888.0, "boot_ts": 1791967457.1014743, "uptime_seconds": 210985.95, "container": false, "agent_version": "1.0.0"}, "cpu": {"modes": {"user": 561498.0, "nice": 2756.0, "system": 122750.0, "idle": 41118362.0, "iowait": 290460.0, "irq": 24373.0, "softirq": 46665.0, "steal": 4918.0, "guest": 0.0, "guest_nice": 0.0, "total": 42171782.0}, "cores": [{"core": 0, "modes": {"user": 285993.0, "nice": 1771.0, "system": 60082.0, "idle": 20538975.0, "iowait": 143245.0, "irq": 14642.0, "softirq": 38622.0, "steal": 3640.0, "guest": 0.0, "guest_nice": 0.0, "total": 21086970.0}}, {"core": 1, "modes": {"user": 275504.0, "nice": 985.0, "system": 62667.0, "idle": 20579386.0, "iowait": 147214.0, "irq": 9730.0, "softirq": 8043.0, "steal": 1277.0, "guest": 0.0, "guest_nice": 0.0, "total": 21084806.0}}], "clock_ticks_per_second": 100, "load1": 0.01, "load5": 0.07, "load15": 0.08, "procs_running": 1.0, "procs_blocked": 2.0, "ctxt_total": 64391428.0, "processes_total": 343402.0, "freq_mhz": null, "freq_max_mhz": null}, "memory": {"total_bytes": 2051493888.0, "free_bytes": 172544000.0, "available_bytes": 1400086528.0, "used_bytes": 651407360.0, "buffers_bytes": 106156032.0, "cached_bytes": 1315049472.0, "dirty_bytes": 2158592.0, "swap_total_bytes": 0.0, "swap_used_bytes": 0.0, "committed_bytes": 720392192.0}, "filesystems": [{"device": "/dev/vda4", "mountpoint": "/", "fstype": "ext4", "total_bytes": 9197531136, "used_bytes": 5252460544, "free_bytes": 3455754240, "used_pct": 60.316157493618384, "read_only": false}, {"device": "/dev/vda3", "mountpoint": "/boot", "fstype": "ext4", "total_bytes": 1020702720, "used_bytes": 272543744, "free_bytes": 677695488, "used_pct": 28.68159246870582, "read_only": false}, {"device": "/dev/vda2", "mountpoint": "/boot/efi", "fstype": "vfat", "total_bytes": 209489920, "used_bytes": 9560064, "free_bytes": 199929856, "used_pct": 4.56349594290742, "read_only": false}], "disk_io": {"devices": [{"name": "vda", "model": null, "rotational": true, "size_bytes": 10737418240.0, "reads_total": 14464.0, "writes_total": 1989589.0, "read_bytes_total": 645659136.0, "write_bytes_total": 22551089664.0, "in_flight": 0.0, "io_time_ms_total": 2882080.0, "weighted_io_ms_total": 6688324.0, "discard_bytes_total": 0.0}], "device_names": ["vda"]}, "network": {"interfaces": [{"name": "eth0", "rx_bytes_total": 14580699012.0, "rx_packets_total": 22626423.0, "rx_errors_total": 0.0, "rx_dropped_total": 287.0, "tx_bytes_total": 10807075678.0, "tx_packets_total": 10084825.0, "tx_errors_total": 0.0, "tx_dropped_total": 0.0, "speed_mbps": null}]}, "gpus": [], "temperatures": [], "cpu_temperature_celsius": null, "processes": {"total": 142, "top_rss": [{"pid": 592, "name": "systemd-journal", "rss_bytes": 105857024.0}, {"pid": 1182, "name": "haproxy", "rss_bytes": 99045376.0}, {"pid": 723, "name": "firewalld", "rss_bytes": 45318144.0}, {"pid": 1091, "name": "polkitd", "rss_bytes": 41594880.0}, {"pid": 1, "name": "systemd", "rss_bytes": 37564416.0}], "top_cpu": [{"pid": 342784, "name": "python3", "cpu_pct": 2.680722502695548}, {"pid": 1, "name": "systemd", "cpu_pct": 0.0}, {"pid": 2, "name": "kthreadd", "cpu_pct": 0.0}, {"pid": 3, "name": "pool_workqueue_release", "cpu_pct": 0.0}, {"pid": 4, "name": "kworker/R-rcu_gp", "cpu_pct": 0.0}]}, "errors": []}
|
||||
{"ok": true, "schema": 1, "agent_version": "1.0.0", "ts": 1792261953.8432987, "duration_ms": 8.87, "host": {"hostname": "gateway", "os_pretty": "AlmaLinux 10.2 (Lavender Lion)", "os_id": "almalinux", "kernel": "6.12.0-211.65.1.el10_2.x86_64_v2", "arch": "x86_64", "python": "3.12.15", "cpu_model": "AMD Ryzen Threadripper 3960X 24-Core Processor", "cpu_cores_logical": 2, "cpu_cores_physical": 2, "memory_total_bytes": 2051493888.0, "boot_ts": 1792225629.17656, "uptime_seconds": 36324.66, "container": false, "agent_version": "1.0.0"}, "cpu": {"modes": {"user": 99306.0, "nice": 125.0, "system": 22821.0, "idle": 7044209.0, "iowait": 79362.0, "irq": 4688.0, "softirq": 8817.0, "steal": 1044.0, "guest": 0.0, "guest_nice": 0.0, "total": 7260372.0}, "cores": [{"core": 0, "modes": {"user": 48970.0, "nice": 52.0, "system": 11522.0, "idle": 3520221.0, "iowait": 45624.0, "irq": 1835.0, "softirq": 1491.0, "steal": 365.0, "guest": 0.0, "guest_nice": 0.0, "total": 3630080.0}}, {"core": 1, "modes": {"user": 50335.0, "nice": 72.0, "system": 11298.0, "idle": 3523988.0, "iowait": 33738.0, "irq": 2853.0, "softirq": 7326.0, "steal": 679.0, "guest": 0.0, "guest_nice": 0.0, "total": 3630289.0}}], "clock_ticks_per_second": 100, "load1": 0.09, "load5": 0.1, "load15": 0.08, "procs_running": 1.0, "procs_blocked": 0.0, "ctxt_total": 12398220.0, "processes_total": 62850.0, "freq_mhz": null, "freq_max_mhz": null}, "memory": {"total_bytes": 2051493888.0, "free_bytes": 673202176.0, "available_bytes": 1482231808.0, "used_bytes": 569262080.0, "buffers_bytes": 89178112.0, "cached_bytes": 889634816.0, "dirty_bytes": 331776.0, "swap_total_bytes": 0.0, "swap_used_bytes": 0.0, "committed_bytes": 600129536.0}, "filesystems": [{"device": "/dev/vda4", "mountpoint": "/", "fstype": "ext4", "total_bytes": 9197531136, "used_bytes": 5252186112, "free_bytes": 3456028672, "used_pct": 60.313006078468355, "read_only": false}, {"device": "/dev/vda3", "mountpoint": "/boot", "fstype": "ext4", "total_bytes": 1020702720, "used_bytes": 272543744, "free_bytes": 677695488, "used_pct": 28.68159246870582, "read_only": false}, {"device": "/dev/vda2", "mountpoint": "/boot/efi", "fstype": "vfat", "total_bytes": 209489920, "used_bytes": 9560064, "free_bytes": 199929856, "used_pct": 4.56349594290742, "read_only": false}], "disk_io": {"devices": [{"name": "vda", "model": null, "rotational": true, "size_bytes": 10737418240.0, "reads_total": 9597.0, "writes_total": 346444.0, "read_bytes_total": 786491904.0, "write_bytes_total": 3761259008.0, "in_flight": 0.0, "io_time_ms_total": 654478.0, "weighted_io_ms_total": 3369847.0, "discard_bytes_total": 0.0}], "device_names": ["vda"]}, "network": {"interfaces": [{"name": "eth0", "rx_bytes_total": 2597455186.0, "rx_packets_total": 4555586.0, "rx_errors_total": 0.0, "rx_dropped_total": 29.0, "tx_bytes_total": 1960356455.0, "tx_packets_total": 2173933.0, "tx_errors_total": 0.0, "tx_dropped_total": 0.0, "speed_mbps": null}]}, "gpus": [], "temperatures": [], "cpu_temperature_celsius": null, "processes": {"total": 138, "top_rss": [{"pid": 593, "name": "systemd-journal", "rss_bytes": 105168896.0}, {"pid": 1182, "name": "haproxy", "rss_bytes": 98664448.0}, {"pid": 733, "name": "firewalld", "rss_bytes": 45543424.0}, {"pid": 1, "name": "systemd", "rss_bytes": 37601280.0}, {"pid": 767, "name": "tuned", "rss_bytes": 28323840.0}], "top_cpu": [{"pid": 1441, "name": "python3", "cpu_pct": 0.20127003994925524}, {"pid": 1, "name": "systemd", "cpu_pct": 0.0}, {"pid": 2, "name": "kthreadd", "cpu_pct": 0.0}, {"pid": 3, "name": "pool_workqueue_release", "cpu_pct": 0.0}, {"pid": 4, "name": "kworker/R-rcu_gp", "cpu_pct": 0.0}]}, "errors": []}
|
||||
Vendored
+2
-2
@@ -9,7 +9,7 @@ VAR apc battery.charge.low "10"
|
||||
VAR apc battery.charge.warning "50"
|
||||
VAR apc battery.date "2001/09/25"
|
||||
VAR apc battery.mfr.date "2021/06/24"
|
||||
VAR apc battery.runtime "648"
|
||||
VAR apc battery.runtime "633"
|
||||
VAR apc battery.runtime.low "120"
|
||||
VAR apc battery.type "PbAc"
|
||||
VAR apc battery.voltage "27.3"
|
||||
@@ -42,7 +42,7 @@ VAR apc ups.beeper.status "disabled"
|
||||
VAR apc ups.delay.shutdown "20"
|
||||
VAR apc ups.firmware "952.e5 .D"
|
||||
VAR apc ups.firmware.aux "e5 "
|
||||
VAR apc ups.load "48"
|
||||
VAR apc ups.load "50"
|
||||
VAR apc ups.mfr "American Power Conversion"
|
||||
VAR apc ups.mfr.date "2021/01/08"
|
||||
VAR apc ups.model "Back-UPS RS 1500MS"
|
||||
|
||||
Vendored
+2
-2
@@ -46,10 +46,10 @@ VAR cyberpower ups.firmware "CR01906AC911"
|
||||
VAR cyberpower ups.load "14"
|
||||
VAR cyberpower ups.mfr "CPS"
|
||||
VAR cyberpower ups.model "GX1500U"
|
||||
VAR cyberpower ups.power "131"
|
||||
VAR cyberpower ups.power "132"
|
||||
VAR cyberpower ups.power.nominal "1500"
|
||||
VAR cyberpower ups.productid "0601"
|
||||
VAR cyberpower ups.realpower "127"
|
||||
VAR cyberpower ups.realpower "129"
|
||||
VAR cyberpower ups.realpower.nominal "900"
|
||||
VAR cyberpower ups.serial "SCRUBBEDSERIAL1"
|
||||
VAR cyberpower ups.status "OL"
|
||||
|
||||
Vendored
+1
@@ -0,0 +1 @@
|
||||
[{"id":0,"n_ctx":262144,"speculative":true,"is_processing":false,"id_task":-1,"task_id":-1,"state":0,"n_prompt_tokens":0,"n_prompt_tokens_cache":0,"n_prompt_tokens_processed":0,"prompt":"","next_token":[{"has_next_token":false,"has_new_line":false,"n_remain":-1,"n_decoded":0}],"model":"turbo"},{"id":1,"n_ctx":262144,"speculative":true,"is_processing":false,"id_task":-1,"task_id":-1,"state":0,"n_prompt_tokens":0,"n_prompt_tokens_cache":0,"n_prompt_tokens_processed":0,"prompt":"","next_token":[{"has_next_token":false,"has_new_line":false,"n_remain":-1,"n_decoded":0}],"model":"turbo"}]
|
||||
@@ -0,0 +1,204 @@
|
||||
"""The `/slots` row-shape contract behind the `/llm` slot grid (phase 07, task 01).
|
||||
|
||||
The old gufo slot table was deleted unread: it looked for `slot.state` as a *string*,
|
||||
`slot.n_prompt`, `slot.n_predict`, `slot.cache` and `slot.decode_per_second`, and the live
|
||||
server publishes none of those names — every row rendered em-dashes forever. That is
|
||||
`AGENTS.md` rule 7 failing in slow motion ("trust a memory of an upstream"), so the
|
||||
replacement grid (task 04) is written against a *recorded* ground truth instead:
|
||||
`tests/fixtures/slots_gufo.json`, the verbatim `/slots` reply from the running gufo,
|
||||
re-captured by `./scripts/capture-fixtures.sh`.
|
||||
|
||||
The same device as `tests/card_contract.py`: the tables live here so the hermetic test
|
||||
(`tests/test_llm_slots_contract.py`, driven from the fixture) and the live test
|
||||
(`tests/test_live_fleet.py::test_gufo_slots_answer_the_pinned_shape`) assert one list and
|
||||
cannot drift apart.
|
||||
|
||||
====================== ==========================================================
|
||||
Slot card cell Path in a `/slots` row
|
||||
====================== ==========================================================
|
||||
card key ``id`` (unique per row — the grid keys one card per slot)
|
||||
state pill ``is_processing`` (authoritative bool) / ``state`` (legacy 0/1)
|
||||
request ``id_task`` — canonical; the ``task_id`` alias is captured but
|
||||
**never rendered**
|
||||
prompt tok ``n_prompt_tokens`` (+ ``n_prompt_tokens_processed`` sub-line)
|
||||
cache hit ``n_prompt_tokens_cache`` (+ ratio against ``n_prompt_tokens``)
|
||||
generated ``next_token[0].n_decoded`` (+ optional ``prob`` sub-line)
|
||||
remaining ``next_token[0].n_remain`` — where the capture puts it, never a
|
||||
row-level ``n_remain`` fallback
|
||||
context ``n_ctx``
|
||||
model name ``model``
|
||||
====================== ==========================================================
|
||||
|
||||
The field names and types are the live capture's. `llm_monitor` reads the same set with
|
||||
``a ?? b`` fallbacks (the task-id alias pair, ``n_remain`` at either level); the grid drops
|
||||
them. Task 04's rule is that **this contract names the canonical key of each fact and the
|
||||
page renders that one**, so a rename fails a test here instead of quietly reading a different
|
||||
field. `llm_monitor` is the design reference, not a dependency.
|
||||
|
||||
``prompt`` is deliberately **not** a card cell: on a busy slot it is the user's in-flight
|
||||
request text. `./scripts/capture-fixtures.sh` blanks it in the fixture, and the page must
|
||||
never read it.
|
||||
|
||||
The capture is an **idle** engine — gufo answers one row per parallel slot even when
|
||||
nothing is generating, so "every slot idle, prompt blank" is the shape the grid must
|
||||
handle; that assumption is pinned in the hermetic test, and the mutation tests below keep
|
||||
the contract honest for busy rows too.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
FIXTURES = Path(__file__).resolve().parent / "fixtures"
|
||||
|
||||
#: The one name the grid prints for "which request is in this slot". The live capture answers
|
||||
#: **both** `id_task` and `task_id` for the same fact (and `llm_monitor` reads `id_task`
|
||||
#: first), so a name has to be made canonical rather than chained: `slot.id_task ??
|
||||
#: slot.task_id` is exactly the guessing that made the deleted table impossible to trust. If a
|
||||
#: re-capture ever answers only the alias, this constant changes and the grid follows — the
|
||||
#: fallback never moves into the page (`page-llm.js`'s `SLOT_FIELDS` names this key once).
|
||||
CANONICAL_TASK_ID_FIELD: str = "id_task"
|
||||
|
||||
#: Other names a row may answer for that one fact: type-checked when present, rendered by
|
||||
#: nothing.
|
||||
TASK_ID_ALIASES: tuple[str, ...] = ("task_id",)
|
||||
|
||||
#: Every name known for the request id, canonical first (for messages and cross-checks).
|
||||
TASK_ID_FIELDS: tuple[str, ...] = (CANONICAL_TASK_ID_FIELD, *TASK_ID_ALIASES)
|
||||
|
||||
#: Doc table: slot-grid cell -> the row path(s) that cell reads. The authoritative
|
||||
#: presence/type rules are the three tables below it.
|
||||
SLOT_CELLS: dict[str, tuple[str, ...]] = {
|
||||
"card key": ("id",),
|
||||
"state pill": ("is_processing", "state"),
|
||||
"request": (CANONICAL_TASK_ID_FIELD,),
|
||||
"prompt tok": ("n_prompt_tokens", "n_prompt_tokens_processed"),
|
||||
"cache hit": ("n_prompt_tokens_cache",),
|
||||
"generated": ("next_token.n_decoded", "next_token.prob"),
|
||||
"remaining": ("next_token.n_remain",),
|
||||
"context": ("n_ctx",),
|
||||
"model name": ("model",),
|
||||
}
|
||||
|
||||
#: Field -> (accepted types, why the grid needs it). Every row must carry every one of
|
||||
#: these; a missing key is the drift that made the old table render dashes.
|
||||
REQUIRED_ROW_FIELDS: dict[str, tuple[tuple[type, ...], str]] = {
|
||||
"id": ((int,), "the grid keys one card per slot — a duplicate or absent id is two "
|
||||
"cards for one slot or a slot nobody updates"),
|
||||
"is_processing": (((bool,), "the authoritative busy/idle verdict behind the state pill")),
|
||||
"state": ((int,), "the legacy numeric state (0 idle / 1 processing); the pill prefers "
|
||||
"is_processing but labels anything else it cannot name"),
|
||||
CANONICAL_TASK_ID_FIELD: ((int,), "the request cell — the only name the grid reads for "
|
||||
"it, so its absence is drift rather than a blank cell"),
|
||||
"n_ctx": ((int,), "the context cell — the slot's window size"),
|
||||
"n_prompt_tokens": ((int, float), "the prompt cell"),
|
||||
"n_prompt_tokens_processed": ((int, float), "the `processed …` sub-line while prefill runs"),
|
||||
"n_prompt_tokens_cache": ((int, float), "the cache-hit cell (ratio against n_prompt_tokens)"),
|
||||
"next_token": ((list,), "the generated cell reads the first entry's n_decoded"),
|
||||
}
|
||||
|
||||
#: Fields the grid reads *when present* — absence degrades to `n/a`, presence must keep the
|
||||
#: type. `model` names the slot header; `prob` is a sub-line llm_monitor prints only if
|
||||
#: the server publishes it (this gufo does not, on the captured rows).
|
||||
OPTIONAL_ROW_FIELDS: dict[str, tuple[tuple[type, ...], str]] = {
|
||||
"model": ((str,), "the model name beside the slot id"),
|
||||
"speculative": ((bool,), "whether speculative decoding is on for this slot"),
|
||||
"n_remain": ((int, float), "some builds answer it at the row level; the grid reads "
|
||||
"next_token[0].n_remain, which every captured row carries"),
|
||||
"task_id": ((int,), f"the alias of {CANONICAL_TASK_ID_FIELD} — captured, never rendered"),
|
||||
}
|
||||
|
||||
#: Every entry of `next_token` that is present must carry these; an empty list is legal
|
||||
# (the grid prints `n/a` for generated/remaining, never a fabricated 0).
|
||||
REQUIRED_NEXT_TOKEN_FIELDS: dict[str, tuple[tuple[type, ...], str]] = {
|
||||
"n_decoded": ((int, float), "the generated cell"),
|
||||
"n_remain": ((int, float), "the remaining cell — the captured rows carry it here, "
|
||||
"not at the row's top level"),
|
||||
}
|
||||
|
||||
OPTIONAL_NEXT_TOKEN_FIELDS: dict[str, tuple[tuple[type, ...], str]] = {
|
||||
"prob": ((int, float), "the `p=…` sub-line; this capture omits it and the grid copes"),
|
||||
}
|
||||
|
||||
#: Row fields that are user content: captured blanked, never rendered by the page.
|
||||
USER_CONTENT_FIELDS: tuple[str, ...] = ("prompt",)
|
||||
|
||||
|
||||
def load_slots_capture(name: str = "gufo") -> list[dict[str, Any]]:
|
||||
"""The verbatim `/slots` capture, as the live server answered it."""
|
||||
path = FIXTURES / f"slots_{name}.json"
|
||||
if not path.exists():
|
||||
raise AssertionError(
|
||||
f"{path} is missing — run ./scripts/capture-fixtures.sh (it captures gufo's "
|
||||
"/slots whenever GUFO_BASE_URL is set)"
|
||||
)
|
||||
rows = json.loads(path.read_text())
|
||||
assert isinstance(rows, list), f"{path.name}: /slots answered {type(rows).__name__}, not a list"
|
||||
return rows
|
||||
|
||||
|
||||
def _matches(value: Any, types: tuple[type, ...]) -> bool:
|
||||
"""`bool` is a subclass of `int`; never let `true` answer for a number or vice versa."""
|
||||
if types == (bool,):
|
||||
return isinstance(value, bool)
|
||||
if any(t in (int, float) for t in types):
|
||||
return isinstance(value, (int, float)) and not isinstance(value, bool)
|
||||
return isinstance(value, types)
|
||||
|
||||
|
||||
def _check_fields(row: dict[str, Any], table: dict[str, tuple[tuple[type, ...], str]],
|
||||
*, require: bool, where: str) -> None:
|
||||
for field, (types, why) in table.items():
|
||||
if require:
|
||||
assert field in row, (
|
||||
f"{where}: no key {field!r} — {why}. The deleted slot table rendered dashes "
|
||||
"for exactly this kind of drift; fail here instead"
|
||||
)
|
||||
elif field not in row:
|
||||
continue
|
||||
assert _matches(row[field], types), (
|
||||
f"{where}.{field} is {type(row[field]).__name__}, expected "
|
||||
f"{'/'.join(t.__name__ for t in types)} — {why}"
|
||||
)
|
||||
|
||||
|
||||
def assert_slot_shape(rows: list[Any]) -> None:
|
||||
"""Every `/slots` row has the shape the slot grid reads. Empty list is legal.
|
||||
|
||||
Drift in either direction fails: a renamed/missing key (including the canonical request
|
||||
id — the alias does not excuse its absence), a type the grid cannot format, or a
|
||||
duplicated slot id (keyed cards would collide).
|
||||
"""
|
||||
assert isinstance(rows, list), f"/slots answered {type(rows).__name__}, not a list of rows"
|
||||
seen_ids: list[int] = []
|
||||
for index, row in enumerate(rows):
|
||||
where = f"row[{index}]"
|
||||
assert isinstance(row, dict), f"{where} is {type(row).__name__}, not an object"
|
||||
_check_fields(row, REQUIRED_ROW_FIELDS, require=True, where=where)
|
||||
_check_fields(row, OPTIONAL_ROW_FIELDS, require=False, where=where)
|
||||
|
||||
# The canonical request id is required above. An alias may ride along (every
|
||||
# captured row answers both) and the grid ignores it.
|
||||
tokens = row["next_token"]
|
||||
for entry in tokens:
|
||||
assert isinstance(entry, dict), f"{where}.next_token entry is not an object"
|
||||
_check_fields(entry, REQUIRED_NEXT_TOKEN_FIELDS, require=True,
|
||||
where=f"{where}.next_token")
|
||||
_check_fields(entry, OPTIONAL_NEXT_TOKEN_FIELDS, require=False,
|
||||
where=f"{where}.next_token")
|
||||
|
||||
# The pill: `is_processing` is authoritative, and `state == 0` must not contradict
|
||||
# it — llm_monitor's exact reading, so a busy row and an idle row stay tellable.
|
||||
if row["is_processing"]:
|
||||
assert row["state"] != 0, (
|
||||
f"{where}: is_processing=True with state=0 — the pill cannot tell the operator "
|
||||
"which field to believe"
|
||||
)
|
||||
|
||||
seen_ids.append(row["id"])
|
||||
duplicates = sorted({i for i in seen_ids if seen_ids.count(i) > 1})
|
||||
assert not duplicates, (
|
||||
f"duplicate slot ids {duplicates} — the grid keys one card per slot and a dupe "
|
||||
"renders one card twice (or drops the other)"
|
||||
)
|
||||
+754
-30
@@ -24,8 +24,22 @@ from dashboards.app import create_app
|
||||
from dashboards.config import Settings, Target
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources import nut
|
||||
from dashboards.sources.llm import HISTORY_METRICS
|
||||
from dashboards.sources.llm import (
|
||||
HISTORY_METRICS,
|
||||
parse_gufo_metrics,
|
||||
parse_vllm_metrics,
|
||||
)
|
||||
from dashboards.store import Event, Sample, SampleStore
|
||||
from slots_contract import (
|
||||
CANONICAL_TASK_ID_FIELD,
|
||||
OPTIONAL_NEXT_TOKEN_FIELDS,
|
||||
OPTIONAL_ROW_FIELDS,
|
||||
REQUIRED_NEXT_TOKEN_FIELDS,
|
||||
REQUIRED_ROW_FIELDS,
|
||||
TASK_ID_ALIASES,
|
||||
USER_CONTENT_FIELDS,
|
||||
load_slots_capture,
|
||||
)
|
||||
|
||||
# refused immediately, and never dials out of the test environment
|
||||
DEAD_URL = "http://127.0.0.1:1"
|
||||
@@ -547,6 +561,42 @@ def _code(body: str) -> str:
|
||||
return re.sub(r"^\s*//.*$", " ", body, flags=re.M)
|
||||
|
||||
|
||||
# The `fleet` object paths `/llm`'s aggregate block reads. `engines_total` drives the
|
||||
# "hidden below two engines" rule and the tag; `engines_live` is what makes a partial sum
|
||||
# visible; the rest are the row itself.
|
||||
FLEET_BLOCK_PATHS = frozenset({
|
||||
"engines_total", "engines_live", "requests_active", "requests_waiting",
|
||||
"prompt_tokens_per_second", "decode_tokens_per_second", "tokens_total",
|
||||
})
|
||||
# The summed half of that list — the ones that must be null, not 0, with nothing answering.
|
||||
FLEET_SUM_PATHS = ("requests_active", "requests_waiting", "prompt_tokens_per_second",
|
||||
"decode_tokens_per_second", "tokens_total")
|
||||
|
||||
|
||||
def _fleet_tag(html: str) -> str:
|
||||
"""The aggregate block's opening tag — the one place its `hidden` default lives."""
|
||||
start = html.index('<section class="card flat fleet"')
|
||||
return html[start:html.index(">", start)]
|
||||
|
||||
|
||||
def _fleet_render_path(js: str) -> str:
|
||||
"""Just the body of `renderFleet()` — the aggregate block's whole code path, comments out.
|
||||
|
||||
Everything the block writes goes through here, so it is the slice that can be asserted
|
||||
verdict-free without tripping over the engine panels' rated KV sparkline next door.
|
||||
"""
|
||||
js = _code(js)
|
||||
return js[js.index("function renderFleet("):js.index("async function refresh(")]
|
||||
|
||||
|
||||
def _fleet_field_metrics(js: str) -> list[str]:
|
||||
"""The `fleet` keys the block's rows are bound to, in the order the block prints them."""
|
||||
js = _code(js)
|
||||
start = js.index("const FLEET_FIELDS = [")
|
||||
table = js[start:js.index("];", start) + 2]
|
||||
return re.findall(r"metric: '([a-z_]+)'", table)
|
||||
|
||||
|
||||
def test_no_parallel_severity_vocabulary_exists_anywhere(client) -> None:
|
||||
"""One word per colour: `ok|warn|fail`, as in severity.py and the existing components."""
|
||||
for asset in (*ASSETS, "/", "/system", "/llm", "/ups"):
|
||||
@@ -1219,6 +1269,22 @@ def test_ups_hero_row_makes_room_for_the_glyph(client) -> None:
|
||||
assert ".ups-hero" in block and "minmax(0, 1fr)" in block, "the row never stacks"
|
||||
|
||||
|
||||
def test_every_ups_chart_names_itself_for_assistive_tech(client) -> None:
|
||||
"""The four trend plots on `/ups` carry an accessible name, like every other chart here.
|
||||
|
||||
`AGENTS.md` states the convention as *every* chart names itself: `LineChart` puts
|
||||
`role="img"` + `aria-label` on the `<svg>` when the caller passes `ariaLabel`, because
|
||||
the legend is a sibling that labels nothing and an unlabelled plot is silence. `/system`
|
||||
(phase 03) and `/llm` (phase 04) were pinned; these four were the stragglers, so the
|
||||
rule is now pinned on every page that draws one.
|
||||
"""
|
||||
js = client.get("/static/page-ups.js").text
|
||||
charts = js[js.index("function makeCharts("):js.index("function points(")]
|
||||
assert charts.count("ariaLabel:") == 4, "a `/ups` chart lost its accessible name"
|
||||
for phrase in ("Battery charge", "Battery runtime", "Battery and line voltage", "UPS load"):
|
||||
assert phrase in charts, f"the {phrase!r} chart names nothing readable"
|
||||
|
||||
|
||||
# ---- the `/llm` layout (phase 04 task 03) ---------------------------------
|
||||
#
|
||||
# The drift this phase fixes was a 14-cell grid where everything was the same size and
|
||||
@@ -1257,9 +1323,12 @@ def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
|
||||
assert '<div class="heroes">' in panel, "the hero row left the engine panel"
|
||||
heroes = re.findall(r"{ key: '(\w+)'[^}]*?hero: true", js)
|
||||
assert heroes == ["decode", "prompt"], "the hero row is no longer the two throughputs"
|
||||
# driven by flags, not fourteen identical cells
|
||||
# driven by flags, not fourteen identical cells. Phase 07 task 03 lifted the summary rows
|
||||
# out of the panel into the stats strip above it, so what is left here is the pair plus
|
||||
# the latency rows the strip does not print — never a fourteen-cell grid again.
|
||||
assert "SPARK_CELLS" in js and "ALARM_CELLS" in js
|
||||
assert js.count("spark: '") >= 6, "the compact sparkline list shrank back to a grid"
|
||||
assert js.count("spark: '") >= 2, "the panel lost the latency sparkline rows it kept"
|
||||
assert js.count("hero: true") == 2, "the hero row is no longer exactly two numbers"
|
||||
# the hero call must not pass a severity — unrated by design
|
||||
hero_call = js[js.index("const node = hero("):js.index("if (node) node.title")]
|
||||
assert "severity" not in hero_call, "throughput invented a verdict"
|
||||
@@ -1269,22 +1338,32 @@ def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
|
||||
|
||||
|
||||
def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client) -> None:
|
||||
"""Each compact cell draws its own sparkline over the *selected* window; latency cells
|
||||
print mean **and** p95 without a hover.
|
||||
"""Each panel row draws its own sparkline over the *selected* window and prints mean
|
||||
**and** p95 without a hover.
|
||||
|
||||
The window is honest time (`align: 'right'` + `span`), percent-like metrics keep a
|
||||
fixed 0-100 axis, and the p95 comes from the snapshot's histogram cells — p95 needs no
|
||||
stored series. The `only` scopes are what keep a gufo panel from showing a latency
|
||||
mystery and a vLLM panel from showing a spec-decode row it cannot fill.
|
||||
The window is honest time (`align: 'right'` + `span`) and the p95 comes from the
|
||||
snapshot's histogram cells — p95 needs no stored series. The `only` scopes are what keep
|
||||
a gufo panel from showing a latency mystery and a vLLM strip card from showing a
|
||||
spec-decode row it cannot fill.
|
||||
|
||||
Phase 07 task 03 moved the percent and count rows to the stats strip, so the plotted
|
||||
percent series left the cell list and the axis guarantee moved with it: the KV trend
|
||||
chart pins `yMax: 100` (an 8 % window that once held a 92 % spike must not read flat),
|
||||
and the strip converts its 0-1 ratios to per cent in `cellValue()` — a bar cannot
|
||||
autoscale, so it cannot lie that way either. The `fixedMax`/`minSpan` options stay
|
||||
plumbed through `drawSparks()` and are pinned on the component itself by
|
||||
`test_sparkline_ships_the_fixed_scale_and_verdict_options`: a count sparkline that comes
|
||||
back must not get an axis finer than one request.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
sparks = js[js.index("function drawSparks("):js.index("function renderFleet(")]
|
||||
assert "DashCharts.sparkline(" in sparks
|
||||
for opt in ("align: 'right'", "span: state.range", "fixedMax", "labels: { max: true }"):
|
||||
assert opt in sparks, f"cell sparklines lost `{opt}`"
|
||||
# percent cells pin the axis; ratio series are scaled to the printed percent
|
||||
assert js.count("fixedMax: 100") >= 3, "KV / prefix-hit / spec-accept lost their 0-100 axis"
|
||||
assert "scale: 1000" in js and "scale: 100" in js
|
||||
# the one percent series still plotted owns a fixed 0-100 axis; seconds stay ms
|
||||
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
|
||||
assert "yMax: 100" in charts, "the KV trend chart no longer pins its 0-100 axis"
|
||||
assert "scale: 1000" in js and "cell.scale || 1" in sparks
|
||||
# every spark cell declares its history series in the markup it is built from
|
||||
assert 'data-spark data-metric="${cell.spark}"' in js
|
||||
# percentile foot lines: mean big in the row, p95 beneath, n/a when nothing measured
|
||||
@@ -1298,10 +1377,6 @@ def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client)
|
||||
assert "only: ['vllm']" in js and "only: ['gufo']" in js
|
||||
scoped = js[js.index("function sparkCellsFor("):js.index("function enginePanel(")]
|
||||
assert "cell.only" in scoped
|
||||
# a count row never gets an axis finer than one request: history arrives as rollups,
|
||||
# so 8 s of a single queued request averages to 0.14 and a fitted axis draws a spike for
|
||||
# a queue nobody was in — the twin of the rescale `fixedMax` fixes
|
||||
assert _code(js).count("minSpan: 1") == 2, "sessions/queued lost their one-request axis floor"
|
||||
assert "minSpan: cell.minSpan" in sparks, "cell minSpan never reaches the sparkline"
|
||||
|
||||
|
||||
@@ -1335,32 +1410,681 @@ def test_llm_alarm_cells_exist_only_while_there_is_an_incident(client) -> None:
|
||||
already treats any non-zero `gufo_device_lost_total` as one. The chip reuses the shared
|
||||
`.chip.fail` vocabulary; nothing new is painted, and a container emptied at zero means
|
||||
the row does not exist rather than showing a permanent ignored 0.
|
||||
|
||||
Phase 07 task 03 renders them on the engine's strip card (the summary you read first)
|
||||
instead of inside the panel, so the two `alarm: true` rows live in `STRIP_ROWS` and
|
||||
`ALARM_CELLS` filters them out of it — counted with the comments stripped, because the
|
||||
docstring on that table explains the flag by name.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
alarms = js[js.index("function renderAlarms("):js.index("function renderSlots(")]
|
||||
alarms = js[js.index("function renderAlarms("):js.index("function makeCharts(")]
|
||||
assert "class=\"chip fail\"" in alarms, "the alarm chip left the shared fail vocabulary"
|
||||
assert "value.value > 0" in alarms
|
||||
assert "host.hidden = chips.length === 0" in alarms
|
||||
assert 'data-alarm="${cell.key}"' in alarms
|
||||
assert js.count("alarm: true") == 2
|
||||
assert _code(js).count("alarm: true") == 2
|
||||
assert "ALARM_CELLS = STRIP_ROWS.filter" in js, "the alarm rows are not strip rows"
|
||||
# the panel no longer carries a second container for the same incident
|
||||
panel = js[js.index("function enginePanel("):js.index("function renderPanel(")]
|
||||
assert 'data-k="alarms"' not in panel, "the alarm chips are drawn twice"
|
||||
strip = js[js.index("function stripCard("):js.index("function renderStripCard(")]
|
||||
assert 'data-k="alarms"' in strip, "the alarm chips left the strip card"
|
||||
|
||||
|
||||
def test_llm_totals_card_says_only_what_no_engine_panel_says(client) -> None:
|
||||
"""Engines answering, active, queued — and hidden below two engines.
|
||||
# ---- the `/llm` stats strip (phase 07 task 03) ---------------------------
|
||||
#
|
||||
# `llm_monitor`'s shared strip (`index.html` ~L743), ported: one card per configured engine
|
||||
# between the fleet block and the panels. Two things make it a summary instead of a second
|
||||
# set of panels, and both are pinned below: every row is a cell `/api/llm/snapshot` really
|
||||
# publishes, and no row is printed in the strip *and* in the panel under it.
|
||||
|
||||
The summed decode/prompt and the un-rated fleet KV bar are gone: per-engine is the
|
||||
honest view (the fleet KV bar had no `severity` to render, which is invariant-grade
|
||||
trouble), and a cross-engine sum of one engine just duplicates that engine's own card
|
||||
(the `llm_monitor` rule). `/api/llm/snapshot` keeps every field — this is a page
|
||||
decision, not an API one.
|
||||
#: row key -> the cells `cellValue()` resolves it from (`sessions.*` is the gufo slot count,
|
||||
#: which is not a metrics cell). The served `STRIP_ROWS` table must match these keys exactly.
|
||||
STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
|
||||
"prompt": ("prompt_tokens_per_second",),
|
||||
"decode": ("decode_tokens_per_second",),
|
||||
"tokens_total": ("tokens_total",),
|
||||
"slots_busy": ("sessions.list", "sessions.active"),
|
||||
"running": ("requests_processing", "num_requests_running"),
|
||||
"waiting": ("requests_deferred", "num_requests_waiting"),
|
||||
"cache_hit": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"),
|
||||
"spec_accept": ("spec_accept_rate",),
|
||||
"queue_mean": ("request_queue_time_seconds__mean",),
|
||||
"preemptions": ("num_preemptions_total",),
|
||||
"device_lost": ("device_lost_total",),
|
||||
}
|
||||
|
||||
#: rows that are incidents, not stats: absent where the engine does not count them.
|
||||
ALARM_ROW_KEYS = ("preemptions", "device_lost")
|
||||
|
||||
|
||||
def _strip_rows(js: str) -> dict[str, str | None]:
|
||||
"""The served `STRIP_ROWS` table: row key -> its `only` scope (`None` = every engine)."""
|
||||
body = _code(js)
|
||||
start = body.index("const STRIP_ROWS = [")
|
||||
table = body[start:body.index("\n];", start)]
|
||||
rows: dict[str, str | None] = {}
|
||||
for chunk in table.split("{ key: '")[1:]:
|
||||
scoped = re.search(r"only: \[([^\]]*)\]", chunk)
|
||||
rows[chunk.split("'")[0]] = scoped.group(1) if scoped else None
|
||||
return rows
|
||||
|
||||
|
||||
def _strip_render_path(js: str) -> str:
|
||||
"""Just `renderStripCard()` — the whole code path that paints one strip card."""
|
||||
js = _code(js)
|
||||
return js[js.index("function renderStripCard("):js.index("function pruneEngines(")]
|
||||
|
||||
|
||||
def test_llm_stats_strip_sits_between_the_fleet_block_and_the_panels(client) -> None:
|
||||
"""The strip is there, ships hidden, names itself, and appears only with engines.
|
||||
|
||||
An unconfigured engine gets no card and no strip (invariant 3) — the section is empty in
|
||||
the markup and `page-llm.js` unhides it only when the payload has an engine to draw.
|
||||
"""
|
||||
html = _code(client.get("/llm").text)
|
||||
for gone in ("f-decode", "f-prompt", "f-kv", "f-tokens", "seam-note"):
|
||||
assert gone not in html, f"/llm still totals {gone}"
|
||||
for kept in ("f-live", "f-active", "f-waiting", 'id="totals"'):
|
||||
assert kept in html, f"/llm lost {kept}"
|
||||
assert 'id="strip"' in html, "/llm lost the per-engine stats strip"
|
||||
assert html.index('id="fleet"') < html.index('id="strip"') < html.index('id="engines"'), \
|
||||
"the strip is no longer between the fleet block and the engine panels"
|
||||
start = html.index('<section class="strip"')
|
||||
tag = html[start:html.index(">", start)]
|
||||
assert "hidden" in tag, "the strip no longer ships hidden"
|
||||
assert 'aria-label="Engine summary"' in tag, "the strip is a region that names nothing"
|
||||
js = client.get("/static/page-llm.js").text
|
||||
assert "card.hidden = total < 2" in js, "the totals card no longer hides for one engine"
|
||||
assert "$('#strip').hidden = payloads.length === 0" in js
|
||||
|
||||
|
||||
def test_llm_strip_card_is_one_card_per_engine_of_shared_vocabulary(client) -> None:
|
||||
"""`stripCard()` = `.card` + `.kv` + `.bar` + `.chip.fail`: a layout, not a new component.
|
||||
|
||||
Dot, name and age in the head (reachability, the same distinction the panels and the
|
||||
fleet block draw), the KV bar, the rows keyed by payload field, the alarm container.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
card = js[js.index("function stripCard("):js.index("function renderStripCard(")]
|
||||
assert "card.className = 'card stat'" in card
|
||||
assert "card.dataset.engine = engine" in card, "the card stopped naming its engine"
|
||||
assert '<span class="dot idle"></span>' in card
|
||||
assert '<span class="faint mono updated"></span>' in card, "the card lost its timestamp"
|
||||
assert '<span class="bar" data-kv-bar><i></i></span>' in card, "the KV bar left `.bar`"
|
||||
assert '<dl class="kv">' in card, "the strip rows left the shared `.kv` vocabulary"
|
||||
assert 'data-row="${row.key}"' in card
|
||||
assert 'data-k="alarms"' in card
|
||||
# the wrapper is why the shared `.kv` grid has to move *inside* it: applied to the
|
||||
# wrappers, `max-content 1fr` lays two rows side by side and the card reads as a
|
||||
# column of orphaned labels (found in the browser, not in a test failure)
|
||||
css = client.get("/static/dash.css").text
|
||||
section = css[css.index("/* ---- engine stats strip"):css.index("/* ---- fleet totals")]
|
||||
assert ".strip .kv { margin: 0; grid-template-columns: 1fr; }" in section
|
||||
assert ".strip .kv > div {" in section
|
||||
assert "grid-template-columns: max-content minmax(0, 1fr)" in section
|
||||
refresh = js[js.index("async function refresh("):js.index("async function refreshEvents(")]
|
||||
assert "state.strips.get(payload.engine)" in refresh, "the strip rebuilds its cards per poll"
|
||||
assert "$('#strip').appendChild(strip)" in refresh
|
||||
assert "renderStripCard(strip, payload, stateRow)" in refresh
|
||||
prune = js[js.index("function pruneEngines("):js.index("function makeCharts(")]
|
||||
assert "state.strips" in prune and "state.panels" in prune, \
|
||||
"an engine that leaves the payload leaves a card behind"
|
||||
|
||||
|
||||
def test_llm_strip_scopes_its_rows_to_the_engines_that_publish_them(client) -> None:
|
||||
"""gufo gets slots and speculative decoding; vLLM gets running and queue time."""
|
||||
rows = _strip_rows(client.get("/static/page-llm.js").text)
|
||||
assert rows["slots_busy"] == "'gufo'" and rows["spec_accept"] == "'gufo'"
|
||||
assert rows["running"] == "'vllm'" and rows["queue_mean"] == "'vllm'"
|
||||
assert rows["waiting"] is None and rows["cache_hit"] is None and rows["tokens_total"] is None
|
||||
js = client.get("/static/page-llm.js").text
|
||||
scoped = js[js.index("function stripRowsFor("):js.index("function stripCard(")]
|
||||
assert "row.only" in scoped and "row.only.indexOf(engine)" in scoped
|
||||
|
||||
|
||||
def test_llm_strip_rows_name_cells_the_engines_publish(client, metrics_text) -> None:
|
||||
"""The strip renders what the payload sends — no row whose cell no engine produces.
|
||||
|
||||
`STRIP_ROW_PATHS` mirrors the JS table; every metric cell is checked against the
|
||||
capture-parsed metrics of each engine that would render that row, so a rename in
|
||||
`sources/llm.py` fails here instead of printing a permanent `n/a` in the browser, and a
|
||||
row scoped to one engine cannot leak into the other's card.
|
||||
"""
|
||||
rows = _strip_rows(client.get("/static/page-llm.js").text)
|
||||
assert set(rows) == set(STRIP_ROW_PATHS), \
|
||||
f"the served strip rows drifted: {set(rows) ^ set(STRIP_ROW_PATHS)}"
|
||||
parsed = {"gufo": parse_gufo_metrics(metrics_text("gufo")),
|
||||
"vllm": parse_vllm_metrics(metrics_text("vllm"))}
|
||||
for key, only in rows.items():
|
||||
paths = STRIP_ROW_PATHS[key]
|
||||
if paths[0].startswith("sessions."):
|
||||
continue # its own payload object; pinned on the JS below
|
||||
shown = {engine: metrics for engine, metrics in parsed.items()
|
||||
if not only or engine in only}
|
||||
present = [engine for engine in shown if any(path in shown[engine] for path in paths)]
|
||||
if key in ALARM_ROW_KEYS:
|
||||
# an alarm the engine does not count simply is not there: the chip renders only
|
||||
# while non-zero, so absence *is* the zero state rather than a dead row
|
||||
assert present, f"no engine publishes {'/'.join(paths)} for the {key} alarm"
|
||||
continue
|
||||
assert len(present) == len(shown), (
|
||||
f"the {sorted(set(shown) - set(present))} card would print `{key}` as n/a forever "
|
||||
f"({'/'.join(paths)} is not published)")
|
||||
|
||||
|
||||
def test_llm_strip_slot_count_counts_published_slots_and_never_invents_zero(client) -> None:
|
||||
"""`slots busy` is a count of `/slots` rows; when /slots did not answer it is `n/a`.
|
||||
|
||||
Counting rows the server published is a count of published facts. An empty list from a
|
||||
server that never answered is not a measurement of an idle engine, so the row falls back
|
||||
to the published `sessions.active` carrying the API's own `detail` string (rule 2).
|
||||
"""
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
slots = js[js.index("case 'slots_busy':"):js.index("case 'cache':")]
|
||||
assert "sessions.source" in slots and "sessions.list" in slots
|
||||
assert "is_processing" in slots, "the count ignores the authoritative busy verdict"
|
||||
assert "sessions.detail" in slots, "the fallback lost the API's own provenance"
|
||||
|
||||
|
||||
def test_llm_strip_kv_bar_is_coloured_by_the_payload_verdict_only(client) -> None:
|
||||
"""The strip's KV bar is the page's one rated cell: `severity` in, class out.
|
||||
|
||||
`sources/llm.py` rates `llm.kv_cache_usage_pct`, the ratio→percent conversion the panel
|
||||
cell used to make rides with it, and `Dash.severityFrom` is the only thing that decides a
|
||||
colour — no threshold in the page, and the stylesheet adds no palette (`ok` stays quiet
|
||||
ink, exactly like the `.cell` rows and the fleet table).
|
||||
"""
|
||||
body = _strip_render_path(client.get("/static/page-llm.js").text)
|
||||
assert "severityFrom(kv.severity)" in body
|
||||
assert "cellValue(engine, metrics, sessions, 'cache')" in body
|
||||
assert "Math.max(0, Math.min(100, kv.value))" in body, "the fill stopped clamping"
|
||||
assert body.count("severityFrom") == 1, "a second strip row invented a verdict"
|
||||
css = client.get("/static/dash.css").text
|
||||
section = css[css.index("/* ---- engine stats strip"):css.index("/* ---- fleet totals")]
|
||||
assert ".stat-bar.warn .value { color: var(--warn); }" in section
|
||||
assert ".stat-bar.fail .value { color: var(--fail); }" in section
|
||||
assert ".stat-bar.ok" not in section, "a healthy KV reading must stay quiet ink"
|
||||
assert ".prov:empty" in section, "an empty provenance tag draws a pill of nothing"
|
||||
|
||||
|
||||
def test_llm_strip_throughput_carries_provenance_without_colour(client) -> None:
|
||||
"""`prompt tok/s` and `decode tok/s` keep the `Δ<window>` tag and stay ink.
|
||||
|
||||
The tag is generated from the cell's `source` (never hardcoded), the exact source rides
|
||||
in the row's tooltip, and `severity.py` rates no throughput — so nothing here may paint
|
||||
it (0 tok/s on an idle engine is healthy).
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
body = _strip_render_path(js)
|
||||
assert "row.rate ? provenance(value) : ''" in body
|
||||
assert "notes.join" in body and "value.source" in body
|
||||
assert "severityFor" not in js
|
||||
|
||||
|
||||
def test_llm_strip_and_panel_print_no_value_twice(client) -> None:
|
||||
"""One value, one home — the UPS hero row's rule, applied across two components.
|
||||
|
||||
Task 03 lifted sessions, queue depth, KV pressure, the cache-hit and spec-accept percents,
|
||||
the lifetime token counter and the alarm chips out of the panel cell list rather than
|
||||
drawing them twice. The two throughput heroes are the exemption the task names: the panel
|
||||
keeps its hero row (so does `llm_monitor`, which carries the pair in both places), and
|
||||
everything else appears once. Queue time moved with its p95 foot, so the move lost nothing.
|
||||
"""
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
start = js.index("const CELLS = [")
|
||||
panel_keys = set(re.findall(r"\{ key: '(\w+)'", js[start:js.index("\n];", start)]))
|
||||
assert panel_keys == {"decode", "prompt", "ttft", "tpot"}, \
|
||||
f"the panel cell list changed shape: {sorted(panel_keys)}"
|
||||
shared = set(_strip_rows(js)) & panel_keys
|
||||
assert shared == {"decode", "prompt"}, f"printed in the strip and the panel: {sorted(shared)}"
|
||||
body = _strip_render_path(js)
|
||||
assert "p95Value(metrics, row.key)" in body, "queue time lost its p95 in the move"
|
||||
|
||||
|
||||
def test_llm_strip_prints_no_peak_context_number(client) -> None:
|
||||
"""The phase's assumption, pinned: `llm_monitor`'s peak-ctx row does not come over.
|
||||
|
||||
Its cell is `llamacpp:n_tokens_max`, a since-start maximum rather than a glance number,
|
||||
and the task says this strip gets no peak-context cell. The series stays stored and
|
||||
queryable from `/api/llm/history` — not plotting it is not removing it.
|
||||
"""
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
assert "n_tokens_max" not in js, "/llm reads the peak-context gauge again"
|
||||
assert not [key for key in _strip_rows(js) if "peak" in key]
|
||||
|
||||
|
||||
def test_llm_fleet_block_leads_the_page_with_the_engine_sums(client) -> None:
|
||||
"""The page opens with llm_monitor's aggregate block: the sums no panel can say.
|
||||
|
||||
Phase 04 retired these numbers to counts; the operator asked for the aggregate
|
||||
throughput back (`TODO.md:5`), and phase 07 restores it in `llm_monitor`'s shape —
|
||||
status dot, "all engines", an `N of M engines` tag, the note that the sums cover only
|
||||
the engines answering now, then running / waiting / prompt / decode / total tokens.
|
||||
The counts-only card it replaces (`#totals`, `f-live`, `f-active`) is gone, ids and all:
|
||||
an element nothing writes is the orphan this repo pins against.
|
||||
"""
|
||||
html = _code(client.get("/llm").text)
|
||||
assert 'id="fleet"' in html, "/llm lost the aggregate block"
|
||||
assert html.index('id="fleet"') < html.index('id="engines"'), \
|
||||
"the fleet block is no longer above the engine panels"
|
||||
for part in ("fleet-dot", "fleet-count", "f-running", "f-waiting", "f-prompt",
|
||||
"f-decode", "f-tokens"):
|
||||
assert f'id="{part}"' in html, f"/llm lost the fleet block's {part}"
|
||||
for words in ("all engines", "sums only engines answering now"):
|
||||
assert words in html, f"/llm lost the fleet block's `{words}` line"
|
||||
for gone in ('id="totals"', "f-live", "f-total", "f-active", "f-kv", "seam-note"):
|
||||
assert gone not in html, f"/llm still carries {gone} from the retired counts card"
|
||||
|
||||
js = client.get("/static/page-llm.js").text
|
||||
body = _fleet_render_path(js)
|
||||
assert "block.hidden = total < 2" in body, "the fleet block no longer hides for one engine"
|
||||
assert "hidden" in _fleet_tag(html), \
|
||||
"the block no longer ships hidden, so a single engine would show a fleet row"
|
||||
|
||||
|
||||
def test_llm_fleet_block_reads_only_paths_the_api_publishes(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Every number the block prints is a `fleet` key of `/api/llm/snapshot`.
|
||||
|
||||
The page sums nothing itself — `fleet_totals()` does — so the field table in
|
||||
`page-llm.js` is a list of API paths, and a rename in `sources/llm.py` must fail here
|
||||
rather than print a permanent `n/a` in the browser. Pinned with both engines configured
|
||||
(the keys exist, empty-handed because nothing answered) and with neither (404
|
||||
`configured: false` without dialling, invariant 3 — and the shipped block stays hidden).
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
|
||||
vllm_url=DEAD_URL) as test_client:
|
||||
fleet = test_client.get("/api/llm/snapshot").json()["fleet"]
|
||||
assert set(fleet) >= FLEET_BLOCK_PATHS, FLEET_BLOCK_PATHS - set(fleet)
|
||||
# the served field table is exactly the sum rows, in llm_monitor's order
|
||||
js = test_client.get("/static/page-llm.js").text
|
||||
table = _fleet_field_metrics(js)
|
||||
assert table == ["requests_active", "requests_waiting",
|
||||
"prompt_tokens_per_second", "decode_tokens_per_second",
|
||||
"tokens_total"], "the fleet row changed shape without the API"
|
||||
assert set(FLEET_SUM_PATHS) == set(table), "the block totals something the API does not"
|
||||
# and the tag's two numbers are API paths too, not arithmetic the page invented
|
||||
body = _fleet_render_path(js)
|
||||
assert "fleet.engines_total" in body and "fleet.engines_live" in body
|
||||
|
||||
with build_client(tmp_path, monkeypatch) as bare:
|
||||
response = bare.get("/api/llm/snapshot")
|
||||
assert response.status_code == 404
|
||||
assert response.json()["configured"] is False
|
||||
html = _code(bare.get("/llm").text)
|
||||
assert 'id="fleet"' in html and "hidden" in _fleet_tag(html), \
|
||||
"an unconfigured engine must not leave a fleet row on the page"
|
||||
|
||||
|
||||
def test_llm_fleet_totals_drop_an_engine_that_stopped_answering(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A total presented as current is current: a dead engine contributes nothing.
|
||||
|
||||
`fleet_totals()`'s docstring is the contract and the block is where it is read. The
|
||||
injected payload survives the dead-port polls (a failed poll keeps the last good one), so
|
||||
these are the numbers a browser would get with gufo answering and vLLM down — the live
|
||||
engine's sums verbatim, `engines_live` one of two, and not one zero where a number the
|
||||
dead engine never published would sit (rule 2).
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
|
||||
vllm_url=DEAD_URL) as test_client:
|
||||
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
|
||||
hub.jobs["llm:gufo"].state.payload = {
|
||||
"engine": "gufo",
|
||||
"metrics": {"decode_tokens_per_second": {"value": 41.0},
|
||||
"prompt_tokens_per_second": {"value": 17.5},
|
||||
"requests_processing": {"value": 2.0},
|
||||
"requests_deferred": {"value": 1.0},
|
||||
"tokens_total": {"value": 512.0}},
|
||||
"sessions": {"active": 2, "list": []}}
|
||||
fleet = test_client.get("/api/llm/snapshot").json()["fleet"]
|
||||
assert fleet["engines_total"] == 2 and fleet["engines_live"] == 1
|
||||
assert fleet["decode_tokens_per_second"] == pytest.approx(41.0)
|
||||
assert fleet["prompt_tokens_per_second"] == pytest.approx(17.5)
|
||||
assert fleet["requests_active"] == pytest.approx(2.0)
|
||||
assert fleet["requests_waiting"] == pytest.approx(1.0)
|
||||
assert fleet["tokens_total"] == pytest.approx(512.0)
|
||||
|
||||
# and with neither engine answering, every sum is null, never a confident 0
|
||||
hub.jobs["llm:gufo"].state.payload = {"engine": "gufo", "metrics": {},
|
||||
"sessions": {"active": None, "list": []}}
|
||||
empty = test_client.get("/api/llm/snapshot").json()["fleet"]
|
||||
assert empty["engines_live"] == 0
|
||||
assert [empty[key] for key in FLEET_SUM_PATHS] == [None] * len(FLEET_SUM_PATHS)
|
||||
|
||||
|
||||
def test_llm_fleet_block_proves_where_its_sums_come_from(client) -> None:
|
||||
"""The seam tooltip, the excluded-engine tooltip, and an `n/a` with no unit.
|
||||
|
||||
Two provenance rules the rest of the app holds to, on the one block that adds engines
|
||||
together: the prompt sum carries `fleet.seams` — the API's own words on why gufo's and
|
||||
vLLM's prompt counters are not the same quantity — and the tag's tooltip names the
|
||||
engines `fleet_totals()` left out, so a partial total says so in words and not only by
|
||||
being dimmed. And a sum the API did not publish prints `n/a` with its unit cleared: "n/a
|
||||
tok/s" would read as a measurement of nothing.
|
||||
"""
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
body = _fleet_render_path(js)
|
||||
assert "fleet.seams" in body, "the prompt sum lost the API's seam note"
|
||||
assert "not answering — excluded from every sum below" in body, \
|
||||
"the tag no longer names the engines left out of the totals"
|
||||
assert "value == null ? ''" in body, "an n/a sum can still print a unit"
|
||||
|
||||
assert "value == null ? ''" in body, "an n/a sum can still print a unit"
|
||||
|
||||
|
||||
def test_llm_fleet_block_colours_no_number(client) -> None:
|
||||
"""No verdict on the aggregate row — and no dimming of anything but the tag.
|
||||
|
||||
`severity.py` rates no throughput and no queue depth (0 tok/s on an idle fleet is
|
||||
healthy), so the block renders ink: the render path touches no `severity`, calls no
|
||||
`Dash.hero()`, and the stylesheet section for the block reaches for no verdict token.
|
||||
The one coloured thing on it is the shared **reachability** dot — the same distinction
|
||||
the engine panels draw between their numbers and their header dot — and the partial-fleet
|
||||
signal is the shared `.stale` dim on the count tag, which is opacity, not a verdict.
|
||||
"""
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
body = _fleet_render_path(js)
|
||||
assert "severity" not in body, "the fleet block invented a verdict"
|
||||
assert "hero(" not in body, "the fleet block drew a hero number it cannot rate"
|
||||
assert "setDot(" in body and "markStale(tag" in body, \
|
||||
"a dead engine no longer dims the tag or darkens the block's dot"
|
||||
|
||||
css = client.get("/static/dash.css").text
|
||||
section = css[css.index("/* ---- fleet totals"):css.index("/* ---- hero numbers")]
|
||||
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
|
||||
assert token not in section, f"the fleet block coloured a number with {token}"
|
||||
assert ".fleet-head" in section and ".fleet-note" in section, \
|
||||
"the fleet block's head row lost its styling"
|
||||
|
||||
|
||||
# ---- the `/llm` gufo slot grid (phase 07 task 04) -------------------------
|
||||
#
|
||||
# `llm_monitor`'s slot cards (`index.html` ~L1179), rebuilt against the capture instead of a
|
||||
# memory: one keyed card per gufo slot, every cell naming **one** key of
|
||||
# `tests/fixtures/slots_gufo.json`. Two things are pinned below, because they are the two ways
|
||||
# this component failed before: a card that reads a key the server does not publish (dashes
|
||||
# forever), and a card that is rebuilt every poll so a busy slot flickers in front of you.
|
||||
|
||||
#: The keys a slot card reads, and where they live: `field` (and its optional `sub` /
|
||||
#: `ratioAgainst` partners) index the row, or its first `next_token` entry when `path` says so.
|
||||
SLOT_ROW_PATHS = ("field", "sub", "ratioAgainst")
|
||||
|
||||
|
||||
def _slot_table(js: str) -> list[dict[str, str | None]]:
|
||||
"""The served `SLOT_FIELDS` table, one dict per printed cell."""
|
||||
body = _code(js)
|
||||
start = body.index("const SLOT_FIELDS = [")
|
||||
table = body[start:body.index("\n];", start)]
|
||||
cells: list[dict[str, str | None]] = []
|
||||
for chunk in table.split("{ key: '")[1:]:
|
||||
cell: dict[str, str | None] = {"key": chunk.split("'")[0]}
|
||||
for option in (*SLOT_ROW_PATHS, "path"):
|
||||
found = re.search(rf"{option}: '([a-z_0-9]+)'", chunk)
|
||||
cell[option] = found.group(1) if found else None
|
||||
cells.append(cell)
|
||||
return cells
|
||||
|
||||
|
||||
def _slot_js(js: str) -> str:
|
||||
"""The slot grid's whole code path, comments out: `slotValue()` to the end of the block."""
|
||||
body = _code(js)
|
||||
return body[body.index("function slotValue("):body.index("function makeCharts(")]
|
||||
|
||||
|
||||
def _slot_markup(js: str) -> str:
|
||||
"""Just `slotSection()` — the markup the grid, its label and its empty states are built from."""
|
||||
body = _code(js)
|
||||
return body[body.index("function slotSection("):body.index("function renderPanel(")]
|
||||
|
||||
|
||||
def test_llm_gufo_panel_holds_the_slot_grid_under_its_label(client) -> None:
|
||||
"""`sessions (GET /slots)`, then the keyed grid, then the note about what it cannot contain.
|
||||
|
||||
The label is `llm_monitor`'s, the grid ships `hidden` (the empty state is a sentence, never
|
||||
a card of nothing), and the note stays because a queued (deferred) request is not in
|
||||
`/slots` at all — without that line an idle grid reads as "nothing waiting".
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
markup = _slot_markup(js)
|
||||
assert "sessions (GET /slots)" in markup, "/llm lost the slot grid's label"
|
||||
# label, grid, empty state, note — in that order, and the grid and the empty state both
|
||||
# ship hidden so nothing paints before the payload says which of them is true
|
||||
order = ['class="section-label" data-slots-label', "sessions (GET /slots)",
|
||||
'<div class="slots" data-slots role="list" hidden>',
|
||||
'class="slots-empty" data-slots-empty hidden',
|
||||
'class="slots-note" data-slots-note hidden']
|
||||
position = -1
|
||||
for needle in order:
|
||||
found = markup.index(needle)
|
||||
assert found > position, f"`{needle}` is out of order in the slot section"
|
||||
position = found
|
||||
assert "queued (deferred) requests are not in /slots" in js, "the queued-request note is gone"
|
||||
assert "renderSlotGrid(card, enginePayload)" in js, "nothing draws the grid any more"
|
||||
# the label is a heading and the grid a list: a screen reader gets the block's name and
|
||||
# its item count, which a stack of divs does not say
|
||||
assert '<h4 class="section-label" data-slots-label>' in markup, \
|
||||
"the slot section's label stopped being a heading"
|
||||
assert 'data-slots role="list"' in markup, "the grid lost its list semantics"
|
||||
# the grid's own markup: a keyed card, the state pill, and one dd per printed cell
|
||||
card = js[js.index("function slotCard("):js.index("function slotSub(")]
|
||||
assert "setAttribute('role', 'listitem')" in card, "the slot cards left the list"
|
||||
assert 'data-slot-cell="${cell.key}"' in card, "the card cells are not table-driven"
|
||||
assert 'data-pill' in card and 'data-mark' in card, "the card lost its state pill or glyph"
|
||||
|
||||
|
||||
def test_llm_slot_grid_style_is_a_marker_and_not_a_verdict(client) -> None:
|
||||
"""A busy slot is a border, a glyph and a word — `severity.py` rates no slot.
|
||||
|
||||
Phase 06 settled this argument for GPU utilisation on an LLM server: a slot generating
|
||||
tokens is the job, so the accent colour says *this is the live one* and no `--ok/--warn/
|
||||
--fail` token appears anywhere in the section. The glyph (`○`/`●`) and the pill's word mean
|
||||
the marker survives monochrome, which is the battery glyph's rule applied here.
|
||||
"""
|
||||
css = client.get("/static/dash.css").text
|
||||
section = css[css.index("/* ---- engine slot grid"):css.index("/* ---- fleet totals")]
|
||||
for selector in (".slots {", ".slot {", ".slot.busy", ".slot-top", ".slot-id",
|
||||
".slot-model", ".slot dt", ".slot dd", ".slots-empty", ".slots-note",
|
||||
".section-label"):
|
||||
assert selector in section, f"dash.css lost the {selector} rule"
|
||||
assert "var(--accent)" in section, "the busy marker stopped using the non-verdict accent"
|
||||
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
|
||||
assert token not in section, f"the slot grid coloured a state with {token}"
|
||||
js = client.get("/static/page-llm.js").text
|
||||
assert "'\\u25cf'" in js or "'●'" in js, "the busy glyph left the card"
|
||||
assert "processing" in _slot_js(js), "the pill stopped saying the state in a word"
|
||||
|
||||
|
||||
def test_llm_slot_cells_each_name_one_key_the_capture_publishes(client) -> None:
|
||||
"""Every printed cell traces to a key of `tests/fixtures/slots_gufo.json`.
|
||||
|
||||
One `field` per cell and no `a ?? b`: the deleted table's `slot.n_prompt ??
|
||||
slot.prompt_tokens` guessing is what made it print dashes and then plausible noise. A key
|
||||
the capture does not have fails here, and the alias the capture answers beside `id_task`
|
||||
must not appear in the page at all — the contract names the canonical one.
|
||||
"""
|
||||
# comments out: this test bans names the prose is allowed to explain
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
cells = _slot_table(js)
|
||||
assert cells, "the slot grid has no cell table to read"
|
||||
assert [cell["key"] for cell in cells] == ["req", "prompt", "cache", "generated",
|
||||
"remain", "ctx"], \
|
||||
"the slot card's cell set changed without the capture"
|
||||
for cell in cells:
|
||||
assert cell["field"], f"the {cell['key']} cell names no row key"
|
||||
|
||||
rows = load_slots_capture()
|
||||
assert rows, "the capture is empty, so this test would prove nothing"
|
||||
for cell in cells:
|
||||
for option in SLOT_ROW_PATHS:
|
||||
key = cell[option]
|
||||
if key is None:
|
||||
continue
|
||||
in_next = cell["path"] == "next_token"
|
||||
required = REQUIRED_NEXT_TOKEN_FIELDS if in_next else REQUIRED_ROW_FIELDS
|
||||
optional = OPTIONAL_NEXT_TOKEN_FIELDS if in_next else OPTIONAL_ROW_FIELDS
|
||||
assert key in required or key in optional, (
|
||||
f"the {cell['key']} cell reads {key!r}, which no contract table declares — the "
|
||||
"page would be reading a field nobody pinned")
|
||||
for row in rows:
|
||||
source = row["next_token"][0] if in_next else row
|
||||
if key in required:
|
||||
assert key in source, (
|
||||
f"the {cell['key']} cell reads {key!r}, which "
|
||||
f"{'next_token[0]' if in_next else 'the row'} does not publish — this is "
|
||||
"how the deleted table rendered dashes")
|
||||
elif key in source: # optional: `prob` is on no captured row, and that is legal
|
||||
assert isinstance(source[key], (int, float)), (
|
||||
f"{key!r} is {source[key]!r}, which the cell cannot format")
|
||||
|
||||
assert cells[0]["field"] == CANONICAL_TASK_ID_FIELD, \
|
||||
"the request cell stopped reading the canonical task id"
|
||||
for alias in TASK_ID_ALIASES:
|
||||
assert alias not in js, f"page-llm.js renders the {alias} alias the contract does not name"
|
||||
# the old table's invented names stay invented, and so does its per-slot speed
|
||||
for gone in ("n_predict", "decode_per_second", "cache_usable", "n_ctx_used"):
|
||||
assert gone not in js, f"page-llm.js reads the invented {gone} again"
|
||||
|
||||
|
||||
def test_llm_slot_card_prints_no_speed_it_cannot_measure(client) -> None:
|
||||
"""No per-slot speed cell: the capture publishes no per-slot rate key.
|
||||
|
||||
`llm_monitor`'s grid has one, and so did the deleted table (`decode_per_second`) — which is
|
||||
why it was dashes. A page-side Δn_decoded/Δt would be a measurement with no estimator, no
|
||||
`Δ<window>` label and no `source` string, i.e. rule 2 with a confident number printed on it,
|
||||
so the number stays on the engine's strip card and panel, where `rates.py` owns it.
|
||||
"""
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
assert not [cell for cell in _slot_table(js)
|
||||
if cell["key"] in {"speed", "tok_s", "decode"}], "a slot cell grew a rate"
|
||||
assert "decode_per_second" not in js
|
||||
|
||||
|
||||
def test_llm_slot_sentinels_print_a_word_or_na_never_a_negative_number(client) -> None:
|
||||
"""`-1` is llama.cpp's "nothing in this slot", and it never becomes a countdown.
|
||||
|
||||
An idle slot answers `id_task: -1` and `next_token[0].n_remain: -1` (both in the capture).
|
||||
The request cell says `idle` in a word; the remaining cell prints `n/a`. Neither prints a
|
||||
negative number, and neither prints the `0` that would read as "one token from done".
|
||||
"""
|
||||
body = _slot_js(client.get("/static/page-llm.js").text)
|
||||
assert "value < 0 ? 'idle'" in body, "the request cell prints the -1 sentinel again"
|
||||
assert "value >= 0 ? value : null" in body, "the remaining cell prints the -1 sentinel"
|
||||
|
||||
|
||||
def test_llm_slot_card_header_reads_only_fields_the_contract_declares(client) -> None:
|
||||
"""The card top prints `model`, `speculative`, `id` and the state pair — and nothing else.
|
||||
|
||||
`id` keys the card (escaped once at build time), `is_processing` is the authoritative pill,
|
||||
`state` labels a number the boolean cannot express, and `model` + `speculative` are the
|
||||
footnote beside them. Each is declared in a contract table, so a rename fails there. The
|
||||
user-content field a busy row carries (`prompt`) is the one key this page must never read.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
header = js[js.index("function renderSlotCard("):js.index("function renderSlotGrid(")]
|
||||
read = {"is_processing", "state", "model", "speculative"}
|
||||
for key in read:
|
||||
assert f"row.{key}" in header, f"the slot card header stopped reading {key}"
|
||||
declared = set(REQUIRED_ROW_FIELDS) | set(OPTIONAL_ROW_FIELDS)
|
||||
assert read <= declared, "the header reads a field no contract table declares"
|
||||
for field in USER_CONTENT_FIELDS:
|
||||
assert field not in header, f"the slot card prints the user's {field}"
|
||||
assert "escapeHtml(id)" in js, "the slot id reached the markup unescaped"
|
||||
|
||||
|
||||
def test_llm_slot_cards_are_keyed_created_once_and_patched_in_place(client) -> None:
|
||||
"""A busy slot does not get rebuilt every poll — and a slot that leaves loses its card.
|
||||
|
||||
`appendChild` of the existing node keeps DOM order == slot order; everything else is a text
|
||||
patch, so a poll cannot flicker a card or re-announce it to assistive tech for a difference
|
||||
of a few tokens. An engine that leaves the payload loses its card map with its panel, or the
|
||||
refs of removed nodes would be re-appended when it came back.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
body = _slot_js(js)
|
||||
assert "state.slots.get(enginePayload.engine)" in body, "the grid stopped keying its cards"
|
||||
assert "if (!refs) {" in body and "refs = slotCard(id)" in body
|
||||
assert "grid.appendChild(refs.root)" in body, "slot order is no longer kept by moving nodes"
|
||||
assert "byId.delete(id)" in body, "a slot that left keeps its card forever"
|
||||
assert "byId.forEach((refs) => refs.root.remove())" in body, \
|
||||
"the empty state left cards on the page"
|
||||
assert "setTxt(box.value" in body, "the cell values stopped being patched in place"
|
||||
assert "innerHTML" not in body.split("function renderSlotCard(")[1], \
|
||||
"a card is rebuilt from markup on a poll"
|
||||
prune = js[js.index("function pruneEngines("):js.index("/* ---- the gufo slot grid")]
|
||||
assert "state.slots.delete(engine)" in prune, "a pruned engine kept its slot cards"
|
||||
|
||||
|
||||
def test_llm_slot_grid_says_why_there_are_no_rows(client) -> None:
|
||||
"""Two honest empty states, and neither of them is a card or a 0-count claim.
|
||||
|
||||
`/slots` did not answer → the API's own `sessions.detail` sentence (the same words the
|
||||
strip's session row carries). It answered with no rows → "engine reports 0 slots", which is
|
||||
a fact about that answer. `sessions.source` is the gate that tells the two apart.
|
||||
"""
|
||||
body = _slot_js(client.get("/static/page-llm.js").text)
|
||||
assert "Boolean(sessions.source)" in body, "the grid cannot tell no-answer from no-rows"
|
||||
assert "engine reports 0 slots" in body
|
||||
assert "sessions.detail ||" in body, "the no-answer state lost the API's own sentence"
|
||||
assert "no /slots answer from this engine" in body, "the no-answer state can print undefined"
|
||||
|
||||
|
||||
def test_llm_vllm_panel_says_there_is_no_per_slot_json(client) -> None:
|
||||
"""vLLM gets one line and no grid — no fake rows for a server that cannot answer.
|
||||
|
||||
`sources/llm.py`'s `_sessions()` publishes counts for vLLM and says so in `detail`; the
|
||||
panel turns that into the `llm_monitor` line, with the API's sentence as the tooltip. The
|
||||
gate is the engine's name (`SLOT_ENGINES`), not `sessions.list.length`, so an engine that
|
||||
has not answered yet still shows its label and the honest empty sentence.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
markup = _slot_markup(js)
|
||||
branch = markup.split("if (SLOT_ENGINES.indexOf(engine) < 0) {")[1].split("}")[0]
|
||||
assert 'class="slots-note" data-slots-note' in branch, "the vLLM line lost its element"
|
||||
assert "NO_SLOT_JSON" in branch and "vLLM exposes no per-slot JSON" in js
|
||||
assert '<div class="slots"' not in branch, "the vLLM panel grew a grid it cannot fill"
|
||||
body = _slot_js(js)
|
||||
assert body.startswith("function slotValue("), "the grid's code path moved"
|
||||
assert "if (!grid) {" in body, "renderSlotGrid no longer leaves a panel with no grid alone"
|
||||
assert "note.title = enginePayload.sessions.detail" in body, \
|
||||
"the vLLM line stopped carrying the API's own words"
|
||||
|
||||
|
||||
def test_the_captured_slot_rows_fill_the_cells_the_grid_prints(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The capture, through the API, into the keys the served cell table reads.
|
||||
|
||||
The fixture rows are injected as a live job payload (the `/slots` passthrough verbatim), so
|
||||
this is the render path's input as the browser would receive it: `sessions.source` set (the
|
||||
grid's gate), every row present, and every key `SLOT_FIELDS` names resolvable in it. A
|
||||
rename in `sources/llm.py` or in gufo fails here rather than leaving a card of `n/a`s.
|
||||
"""
|
||||
rows = load_slots_capture()
|
||||
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
|
||||
vllm_url=DEAD_URL) as test_client:
|
||||
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
|
||||
hub.jobs["llm:gufo"].state.payload = {
|
||||
"engine": "gufo", "metrics": {"requests_processing": {"value": 1.0}},
|
||||
"sessions": {"active": 1, "list": rows, "source": "gufo /slots", "detail": None},
|
||||
}
|
||||
engines = test_client.get("/api/llm/snapshot").json()["engines"]
|
||||
gufo = next(row for row in engines if row["engine"] == "gufo")
|
||||
sessions = gufo["sessions"]
|
||||
assert sessions["source"] == "gufo /slots", "the grid's gate is gone from the payload"
|
||||
assert sessions["list"] == rows, "the rows are no longer the passthrough the grid reads"
|
||||
|
||||
cells = _slot_table(test_client.get("/static/page-llm.js").text)
|
||||
for cell in cells:
|
||||
for option in SLOT_ROW_PATHS:
|
||||
key = cell[option]
|
||||
if key is None:
|
||||
continue
|
||||
in_next = cell["path"] == "next_token"
|
||||
required = REQUIRED_NEXT_TOKEN_FIELDS if in_next else REQUIRED_ROW_FIELDS
|
||||
for row in sessions["list"]:
|
||||
source = row["next_token"][0] if in_next else row
|
||||
if key not in required and key not in source:
|
||||
continue # an optional sub-line this build does not publish
|
||||
value = source[key]
|
||||
assert isinstance(value, (int, float)) and not isinstance(value, bool), (
|
||||
f"the {cell['key']} cell reads {key!r} = {value!r}, which the number "
|
||||
"formatter cannot print")
|
||||
|
||||
|
||||
def test_llm_charts_are_secondary_and_named(client) -> None:
|
||||
|
||||
@@ -33,6 +33,7 @@ from dashboards.rates import RateEstimator
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources import nut
|
||||
from dashboards.sources.host import HostCollector
|
||||
from slots_contract import assert_slot_shape
|
||||
from ups_contract import assert_ups_hero_contract
|
||||
|
||||
pytestmark = pytest.mark.live
|
||||
@@ -127,6 +128,30 @@ async def test_engine_still_publishes_the_series_we_parse(engine: str,
|
||||
assert result.samples, f"{engine}: nothing to store, so the charts would stay empty"
|
||||
|
||||
|
||||
async def test_gufo_slots_answer_the_pinned_shape(settings: Settings) -> None:
|
||||
"""The `/slots` rows behind the phase-07 slot grid, asserted against the running gufo.
|
||||
|
||||
Hermetic twin: `tests/test_llm_slots_contract.py`, driven from
|
||||
`tests/fixtures/slots_gufo.json`. This is the half that catches gufo renaming a slot
|
||||
field — the old table rendered dashes because nobody had ever pinned the shape, and the
|
||||
assertion here goes through the production path (`collectors.fetch_engine`, which owns
|
||||
the `/slots` → `/v1/slots` fallback), not a hand-rolled request.
|
||||
"""
|
||||
if not settings.gufo_url:
|
||||
pytest.skip("GUFO_BASE_URL is unset — no gufo to ask for /slots")
|
||||
rates = RateEstimator(span=settings.rate_window)
|
||||
async with httpx.AsyncClient(timeout=settings.llm_timeout) as client:
|
||||
result = await collectors.fetch_engine(client, "gufo", settings.gufo_url,
|
||||
settings.gufo_api_key, rates, with_slots=True)
|
||||
rows = result.payload["sessions"]["list"]
|
||||
assert rows, (
|
||||
"gufo answered /slots with no rows — even an idle engine reports one row per "
|
||||
"parallel slot (the idle shape pinned in tests/fixtures/slots_gufo.json); "
|
||||
f"detail: {result.payload['sessions']['detail']!r}"
|
||||
)
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
def require_upsd(settings: Settings) -> None:
|
||||
"""Skip, not fail, when there is no real upsd to talk to."""
|
||||
if not settings.nut_enabled:
|
||||
|
||||
@@ -0,0 +1,246 @@
|
||||
"""The `/slots` shape contract, driven from the live capture (phase 07, task 01).
|
||||
|
||||
The old gufo slot table read field names the server never published and rendered dashes
|
||||
forever; this file is the fix for the *process*, not just the markup: the shape the slot
|
||||
grid (task 04) will read is pinned against `tests/fixtures/slots_gufo.json` — the verbatim
|
||||
`/slots` reply recorded by `./scripts/capture-fixtures.sh` — and against the running gufo
|
||||
in `tests/test_live_fleet.py::test_gufo_slots_answer_the_pinned_shape`. If the fixture (or
|
||||
the live server) drifts, a test fails here, not a dashboard at 2am (`AGENTS.md` rule 7).
|
||||
|
||||
The table and the assertion live in `tests/slots_contract.py` so both suites read one
|
||||
list, exactly like `tests/card_contract.py` does for `/system`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from dashboards.app import create_app
|
||||
from dashboards.config import Settings
|
||||
from dashboards.store import SampleStore
|
||||
from slots_contract import (
|
||||
CANONICAL_TASK_ID_FIELD,
|
||||
REQUIRED_NEXT_TOKEN_FIELDS,
|
||||
REQUIRED_ROW_FIELDS,
|
||||
SLOT_CELLS,
|
||||
TASK_ID_ALIASES,
|
||||
assert_slot_shape,
|
||||
load_slots_capture,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def rows() -> list[dict[str, Any]]:
|
||||
"""A fresh mutable copy of the capture — mutations in one test cannot leak."""
|
||||
return copy.deepcopy(load_slots_capture())
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(tmp_path: Path) -> Iterator[TestClient]:
|
||||
"""A running app, so the deletion assertions run on the bytes a browser would get."""
|
||||
db = str(tmp_path / "slots-contract.sqlite3")
|
||||
store = SampleStore(db)
|
||||
app = create_app(Settings(db_path=db), store)
|
||||
test_client = TestClient(app)
|
||||
test_client.__enter__()
|
||||
try:
|
||||
yield test_client
|
||||
finally:
|
||||
test_client.__exit__(None, None, None)
|
||||
store.close()
|
||||
|
||||
|
||||
def _code(body: str) -> str:
|
||||
"""An asset with its comments stripped — prose may *name* a dead thing the code must not be."""
|
||||
body = re.sub(r"/\*.*?\*/", " ", body, flags=re.S)
|
||||
body = re.sub(r"<!--.*?-->", " ", body, flags=re.S)
|
||||
return re.sub(r"^\s*//.*$", " ", body, flags=re.M)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The capture is real, and the contract passes on it
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_the_gufo_slots_capture_is_on_disk_and_is_a_real_answer(
|
||||
rows: list[dict[str, Any]]) -> None:
|
||||
"""Every row carries a dozen fields — the whole point of capturing the wire.
|
||||
|
||||
Without this, a two-key hand-written stub would satisfy "the key exists" and the
|
||||
contract would prove nothing (`test_ups_contract.py` holds the same line for the UPS
|
||||
captures).
|
||||
"""
|
||||
assert rows, (
|
||||
"gufo answered /slots with no rows — even an idle engine reports one row per "
|
||||
"parallel slot, and that idle shape is exactly what the grid must handle"
|
||||
)
|
||||
for row in rows:
|
||||
assert len(row) >= 10, f"{len(row)} keys — this is not a capture, it is a guess"
|
||||
|
||||
|
||||
def test_the_slot_grid_fields_hold_on_every_captured_row(
|
||||
rows: list[dict[str, Any]]) -> None:
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
def test_the_capture_records_the_idle_shape_the_grid_must_handle(
|
||||
rows: list[dict[str, Any]]) -> None:
|
||||
"""The assumption written into the phase: a fresh capture is an idle engine.
|
||||
|
||||
Every row is idle with a blank prompt — the grid must render that as N idle cards, not
|
||||
as "no data". If a re-capture lands mid-request this test names the drift loudly
|
||||
instead of letting the contract quietly widen; busy rows are covered below.
|
||||
"""
|
||||
for row in rows:
|
||||
assert row["is_processing"] is False, "the capture caught a busy slot; re-run it when idle"
|
||||
assert row["state"] == 0
|
||||
assert row["n_prompt_tokens"] == 0
|
||||
|
||||
|
||||
def test_a_busy_row_of_the_same_shape_still_satisfies_the_contract(
|
||||
rows: list[dict[str, Any]]) -> None:
|
||||
"""The contract is the row shape, not the idle numbers — a busy slot must pass too."""
|
||||
busy = rows[0]
|
||||
busy.update({"is_processing": True, "state": 1, "id_task": 42,
|
||||
"n_prompt_tokens": 128, "n_prompt_tokens_processed": 64,
|
||||
"n_prompt_tokens_cache": 16})
|
||||
busy["next_token"][0]["n_decoded"] = 7
|
||||
busy["next_token"][0]["prob"] = 0.93 # llm_monitor prints it only when present
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
def test_an_empty_row_list_passes_the_contract() -> None:
|
||||
"""Zero slots is a state the grid renders (an explicit empty message), not a crash."""
|
||||
assert_slot_shape([])
|
||||
|
||||
|
||||
def test_the_contract_names_every_cell_the_grid_reads() -> None:
|
||||
"""The doc table covers the llm_monitor cell set; a rename here is a rename there."""
|
||||
assert set(SLOT_CELLS) == {
|
||||
"card key", "state pill", "request", "prompt tok", "cache hit",
|
||||
"generated", "remaining", "context", "model name",
|
||||
}
|
||||
for label, paths in SLOT_CELLS.items():
|
||||
assert paths, f"{label} lists no row path"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Drift fails: the contract is only real if mutations are caught
|
||||
# --------------------------------------------------------------------------- #
|
||||
@pytest.mark.parametrize("field", sorted(REQUIRED_ROW_FIELDS))
|
||||
def test_the_contract_fails_when_a_row_drops_a_field_the_grid_reads(
|
||||
rows: list[dict[str, Any]], field: str) -> None:
|
||||
del rows[0][field]
|
||||
with pytest.raises(AssertionError, match=field):
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("field", "value"), [
|
||||
("id", "0"), # a string id would key two cards "0" and 0 apart
|
||||
("is_processing", 0), # the legacy numeric state is not the authoritative bool
|
||||
("state", "OL"), # this is not NUT
|
||||
("n_ctx", None), # None would reach the number formatter
|
||||
("n_prompt_tokens", True), # bool sneaks past a bare isinstance(x, int)
|
||||
("next_token", {"n_decoded": 1}),
|
||||
])
|
||||
def test_the_contract_fails_on_a_type_the_grid_cannot_render(
|
||||
rows: list[dict[str, Any]], field: str, value: Any) -> None:
|
||||
rows[0][field] = value
|
||||
with pytest.raises(AssertionError, match=field):
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field", sorted(REQUIRED_NEXT_TOKEN_FIELDS))
|
||||
def test_the_contract_fails_when_next_token_loses_a_field(
|
||||
rows: list[dict[str, Any]], field: str) -> None:
|
||||
del rows[0]["next_token"][0][field]
|
||||
with pytest.raises(AssertionError, match=field):
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
def test_the_contract_fails_when_the_canonical_request_id_is_missing(
|
||||
rows: list[dict[str, Any]]) -> None:
|
||||
"""`id_task` is the key the grid prints, so its absence is drift, not a blank cell.
|
||||
|
||||
The alias exists in the same row, and `llm_monitor` would have fallen back to it. This
|
||||
contract does not: a renamed fact fails here rather than rendering a plausible-looking
|
||||
card out of a different field (`page-llm.js`'s `SLOT_FIELDS` names one key per cell).
|
||||
"""
|
||||
del rows[0][CANONICAL_TASK_ID_FIELD]
|
||||
with pytest.raises(AssertionError, match=CANONICAL_TASK_ID_FIELD):
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
def test_the_request_id_alias_is_recorded_and_never_needed(rows: list[dict[str, Any]]) -> None:
|
||||
"""Dropping the alias must not fail the contract — proof it is not a second source."""
|
||||
assert set(TASK_ID_ALIASES) == {"task_id"}
|
||||
for row in rows:
|
||||
del row["task_id"]
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
def test_the_contract_fails_on_duplicate_slot_ids(rows: list[dict[str, Any]]) -> None:
|
||||
assert len(rows) > 1, "the capture needs two slots for a dupe to be possible"
|
||||
rows[1]["id"] = rows[0]["id"]
|
||||
with pytest.raises(AssertionError, match="duplicate slot ids"):
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
def test_the_contract_fails_when_the_two_state_fields_disagree(
|
||||
rows: list[dict[str, Any]]) -> None:
|
||||
rows[0]["is_processing"] = True # state stays 0 — which field is the pill to believe?
|
||||
with pytest.raises(AssertionError, match="state"):
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
def test_the_contract_fails_when_a_row_is_not_an_object(rows: list[dict[str, Any]]) -> None:
|
||||
broken: list[Any] = list(rows)
|
||||
broken[0] = "idle"
|
||||
with pytest.raises(AssertionError, match="not an object"):
|
||||
assert_slot_shape(broken)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The deleted table stays deleted (completion criterion, on served bytes)
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_the_broken_slot_table_is_gone_from_html_js_and_css(client: TestClient) -> None:
|
||||
"""No `#slots-card` / `#slots-table` / `#slots-empty` / `renderSlots` survives.
|
||||
|
||||
Asserted on served bytes because there is no JS test runner (a locked anchor). The
|
||||
guessed field names go with it — those are the memory of an upstream the capture
|
||||
replaced. The `.chip.info` rule only ever painted that table's "has msg" cell, so it
|
||||
goes too (orphan-rule policy, `AGENTS.md`).
|
||||
"""
|
||||
html = _code(client.get("/llm").text)
|
||||
for gone in ("slots-card", "slots-table", "slots-empty"):
|
||||
assert f'id="{gone}"' not in html, f"/llm still renders the {gone}"
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
for gone in ("renderSlots", "with_message", "n_predict", "decode_per_second",
|
||||
"cache_usable"):
|
||||
assert gone not in js, f"page-llm.js still reads {gone}"
|
||||
css = client.get("/static/dash.css").text
|
||||
assert ".chip.info" not in css, "dash.css still styles a class nothing renders"
|
||||
|
||||
|
||||
def test_the_prompt_field_is_user_content_that_never_reaches_the_page(
|
||||
client: TestClient, rows: list[dict[str, Any]]) -> None:
|
||||
"""A busy slot's `prompt` is the operator's in-flight request text.
|
||||
|
||||
The capture script blanks it (fixture stays shape-true, content-free), and the page
|
||||
never reads the key — the grid renders tokens, not somebody's prompt.
|
||||
"""
|
||||
for row in rows:
|
||||
assert row.get("prompt", "") == "", "the capture script must blank prompt content"
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
# A slot-row read is `slot.prompt` / `row.prompt` / `s.prompt` / `["prompt"]`. A bare
|
||||
# `.prompt` ban is impossible — `state.charts.prompt` is a chart object — so the name
|
||||
# before the dot must be a whole word (the lookbehind is what stops `charts.prompt`
|
||||
# from matching through its trailing "s").
|
||||
assert not re.search(r"(?<![\w.])(?:slot|row|s|item)\.prompt\b", js), \
|
||||
"page-llm.js reads a slot row's user-content field"
|
||||
assert '["prompt"]' not in js and "['prompt']" not in js, \
|
||||
"page-llm.js indexes a row's prompt by name"
|
||||
@@ -17,6 +17,7 @@ from dashboards.sources.llm import (
|
||||
parse_vllm_metrics,
|
||||
series_names,
|
||||
)
|
||||
from slots_contract import load_slots_capture
|
||||
|
||||
|
||||
def bump_metric(text: str, name: str, delta: float) -> str:
|
||||
@@ -259,10 +260,18 @@ def test_gufo_stores_the_spec_accept_series_the_cell_sparkline_draws(metrics_tex
|
||||
|
||||
|
||||
def test_sessions_pass_through_slots_for_gufo() -> None:
|
||||
slots = [{"id": 0, "idle": False, "n_prompt": 12, "n_predict": 40}]
|
||||
"""`sessions.list` is the raw gufo rows — the grid, not the source, reads slot fields.
|
||||
|
||||
The rows are phase 07's live capture (`tests/fixtures/slots_gufo.json`), not a guessed
|
||||
shape: the old table's `idle`/`n_prompt`/`n_predict` names were invented and the server
|
||||
never answered them. `_sessions()` must pass the rows through untouched — the shape is
|
||||
contracted separately in `tests/slots_contract.py`.
|
||||
"""
|
||||
slots = load_slots_capture()
|
||||
result = engine_result("gufo", "http://g:1", "", RateEstimator(), slots=slots, now=1.0)
|
||||
sessions = result.payload["sessions"]
|
||||
assert sessions["list"] == slots
|
||||
assert sessions["list"][0]["is_processing"] is False, "the passthrough re-shaped the rows"
|
||||
assert sessions["source"] == "gufo /slots"
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user