services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 07_llm_monitor_style_pages

Browse Source 
**Phase 07 final verification pass — all tasks already in `complete/`; every criterion re-verified live.**

- **Old slot table gone**: no `#slots-card`/`#slots-table`/`renderSlots` in HTML/JS/CSS (only the new grid's `.slots-empty` *class*); pinned by `test_the_broken_slot_table_is_gone_from_html_js_and_css`.
- **`tests/fixtures/slots_gufo.json`**: present; live `GET ${GUFO_BASE_URL}/slots` this pass matches it key-for-key (row + `next_token[0]`), so the capture is reproducible; drift proven to fail (mutation tests + live `test_gufo_slots_answer_the_pinned_shape`, 9 live tests pass).
- **Aggregate block**: live page shows `2 of 2 engines`, running/waiting/prompt/decode/total-tokens straight from `fleet_totals()`, dot `ok`, no verdict colour; on a throwaway single-engine instance (:8091, private temp DB) `#fleet` is `hidden`, one full-width strip card + one panel, zero banners → "one clean half".
- **Strip + slot grid on live data**: 2 `.card.stat` (KV bars `bar ok` 13 %/0 %), gufo grid rendered real cards incl. a busy slot (`#2189`, 67849 prompt tok, 99 % cache share, `processing`), vLLM note-only with no grid; no `NaN`/`undefined` anywhere. Screenshots: `data/logs/llm-phase07-final-live-{two-engines,single-engine}.png`, `ups-charts-aria-labels.png`.
- **Defect found in earlier work and fixed**: `/ups`' four `LineChart`s shipped no `ariaLabel`, violating AGENTS.md's "every chart names itself for assistive tech" (`/system`+`/llm` were pinned, `/ups` wasn't) → added labels to `page-ups.js` and `test_every_ups_chart_names_itself_for_assistive_tech`; browser-verified `role="img"` + label on all four.
- **Gate**: `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **604 passed, 9 skipped, coverage 97.91 %**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → **0 errors**; `node --check` every page script ok; `uv run pytest -m live --live` → **9 passed**; `uv pip list --outdated` empty.
- **Docs**: README + AGENTS.md reviewed end-to-end; fixed test count (582→604), `/llm` source column + both architecture diagrams now name gufo `/slots`, AGENTS.md agent line gained `GET /schema` + status page, ariaLabel convention now covers all three chart pages, README Testing block now lists `bash .agents/validate.sh`. Env-var table, agent/install flags, Containerfile facts, `/api/*` shapes, range chips, remote URL all verified accurate — no other changes needed.
- **Live-infra**: no fleet/host/config change; the temp :8091 instance, its temp DB and the fake processes are removed; dev server restarted and `/api/health` `ok` (all 6 sources ok). One thing I deliberately did **not** touch: the gitignored local `.env.dev` still carries a comment claiming the NUT `[dashboards]` account has only battery-test `instcmds`, whereas AGENTS.md/README record `instcmds = all` (verified on hardware 2026-10-17) — operator-owned file, flag for the operator to reword.
- **Next pending phase**: `08_events_paging_and_copy`.
This commit is contained in:
ducoterra
2026-10-17 17:50:19 -04:00
parent de325caa49
commit 3d5ccd7a39
43 changed files with 2544 additions and 210 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/fixtures/agent_deskwork.json
Vendored
+1 -1
View File
File diff suppressed because one or more lines are too long
tests/fixtures/agent_driveripper.json
Vendored
+1 -1
View File
File diff suppressed because one or more lines are too long
tests/fixtures/agent_gateway.json
Vendored
+1 -1
View File
@@ -1 +1 @@
{"ok": true, "schema": 1, "agent_version": "1.0.0", "ts": 1792178443.0656452, "duration_ms": 10.22, "host": {"hostname": "gateway", "os_pretty": "AlmaLinux 10.2 (Lavender Lion)", "os_id": "almalinux", "kernel": "6.12.0-211.65.1.el10_2.x86_64_v2", "arch": "x86_64", "python": "3.12.15", "cpu_model": "AMD Ryzen Threadripper 3960X 24-Core Processor", "cpu_cores_logical": 2, "cpu_cores_physical": 2, "memory_total_bytes": 2051493888.0, "boot_ts": 1791967457.1014743, "uptime_seconds": 210985.95, "container": false, "agent_version": "1.0.0"}, "cpu": {"modes": {"user": 561498.0, "nice": 2756.0, "system": 122750.0, "idle": 41118362.0, "iowait": 290460.0, "irq": 24373.0, "softirq": 46665.0, "steal": 4918.0, "guest": 0.0, "guest_nice": 0.0, "total": 42171782.0}, "cores": [{"core": 0, "modes": {"user": 285993.0, "nice": 1771.0, "system": 60082.0, "idle": 20538975.0, "iowait": 143245.0, "irq": 14642.0, "softirq": 38622.0, "steal": 3640.0, "guest": 0.0, "guest_nice": 0.0, "total": 21086970.0}}, {"core": 1, "modes": {"user": 275504.0, "nice": 985.0, "system": 62667.0, "idle": 20579386.0, "iowait": 147214.0, "irq": 9730.0, "softirq": 8043.0, "steal": 1277.0, "guest": 0.0, "guest_nice": 0.0, "total": 21084806.0}}], "clock_ticks_per_second": 100, "load1": 0.01, "load5": 0.07, "load15": 0.08, "procs_running": 1.0, "procs_blocked": 2.0, "ctxt_total": 64391428.0, "processes_total": 343402.0, "freq_mhz": null, "freq_max_mhz": null}, "memory": {"total_bytes": 2051493888.0, "free_bytes": 172544000.0, "available_bytes": 1400086528.0, "used_bytes": 651407360.0, "buffers_bytes": 106156032.0, "cached_bytes": 1315049472.0, "dirty_bytes": 2158592.0, "swap_total_bytes": 0.0, "swap_used_bytes": 0.0, "committed_bytes": 720392192.0}, "filesystems": [{"device": "/dev/vda4", "mountpoint": "/", "fstype": "ext4", "total_bytes": 9197531136, "used_bytes": 5252460544, "free_bytes": 3455754240, "used_pct": 60.316157493618384, "read_only": false}, {"device": "/dev/vda3", "mountpoint": "/boot", "fstype": "ext4", "total_bytes": 1020702720, "used_bytes": 272543744, "free_bytes": 677695488, "used_pct": 28.68159246870582, "read_only": false}, {"device": "/dev/vda2", "mountpoint": "/boot/efi", "fstype": "vfat", "total_bytes": 209489920, "used_bytes": 9560064, "free_bytes": 199929856, "used_pct": 4.56349594290742, "read_only": false}], "disk_io": {"devices": [{"name": "vda", "model": null, "rotational": true, "size_bytes": 10737418240.0, "reads_total": 14464.0, "writes_total": 1989589.0, "read_bytes_total": 645659136.0, "write_bytes_total": 22551089664.0, "in_flight": 0.0, "io_time_ms_total": 2882080.0, "weighted_io_ms_total": 6688324.0, "discard_bytes_total": 0.0}], "device_names": ["vda"]}, "network": {"interfaces": [{"name": "eth0", "rx_bytes_total": 14580699012.0, "rx_packets_total": 22626423.0, "rx_errors_total": 0.0, "rx_dropped_total": 287.0, "tx_bytes_total": 10807075678.0, "tx_packets_total": 10084825.0, "tx_errors_total": 0.0, "tx_dropped_total": 0.0, "speed_mbps": null}]}, "gpus": [], "temperatures": [], "cpu_temperature_celsius": null, "processes": {"total": 142, "top_rss": [{"pid": 592, "name": "systemd-journal", "rss_bytes": 105857024.0}, {"pid": 1182, "name": "haproxy", "rss_bytes": 99045376.0}, {"pid": 723, "name": "firewalld", "rss_bytes": 45318144.0}, {"pid": 1091, "name": "polkitd", "rss_bytes": 41594880.0}, {"pid": 1, "name": "systemd", "rss_bytes": 37564416.0}], "top_cpu": [{"pid": 342784, "name": "python3", "cpu_pct": 2.680722502695548}, {"pid": 1, "name": "systemd", "cpu_pct": 0.0}, {"pid": 2, "name": "kthreadd", "cpu_pct": 0.0}, {"pid": 3, "name": "pool_workqueue_release", "cpu_pct": 0.0}, {"pid": 4, "name": "kworker/R-rcu_gp", "cpu_pct": 0.0}]}, "errors": []}
{"ok": true, "schema": 1, "agent_version": "1.0.0", "ts": 1792261953.8432987, "duration_ms": 8.87, "host": {"hostname": "gateway", "os_pretty": "AlmaLinux 10.2 (Lavender Lion)", "os_id": "almalinux", "kernel": "6.12.0-211.65.1.el10_2.x86_64_v2", "arch": "x86_64", "python": "3.12.15", "cpu_model": "AMD Ryzen Threadripper 3960X 24-Core Processor", "cpu_cores_logical": 2, "cpu_cores_physical": 2, "memory_total_bytes": 2051493888.0, "boot_ts": 1792225629.17656, "uptime_seconds": 36324.66, "container": false, "agent_version": "1.0.0"}, "cpu": {"modes": {"user": 99306.0, "nice": 125.0, "system": 22821.0, "idle": 7044209.0, "iowait": 79362.0, "irq": 4688.0, "softirq": 8817.0, "steal": 1044.0, "guest": 0.0, "guest_nice": 0.0, "total": 7260372.0}, "cores": [{"core": 0, "modes": {"user": 48970.0, "nice": 52.0, "system": 11522.0, "idle": 3520221.0, "iowait": 45624.0, "irq": 1835.0, "softirq": 1491.0, "steal": 365.0, "guest": 0.0, "guest_nice": 0.0, "total": 3630080.0}}, {"core": 1, "modes": {"user": 50335.0, "nice": 72.0, "system": 11298.0, "idle": 3523988.0, "iowait": 33738.0, "irq": 2853.0, "softirq": 7326.0, "steal": 679.0, "guest": 0.0, "guest_nice": 0.0, "total": 3630289.0}}], "clock_ticks_per_second": 100, "load1": 0.09, "load5": 0.1, "load15": 0.08, "procs_running": 1.0, "procs_blocked": 0.0, "ctxt_total": 12398220.0, "processes_total": 62850.0, "freq_mhz": null, "freq_max_mhz": null}, "memory": {"total_bytes": 2051493888.0, "free_bytes": 673202176.0, "available_bytes": 1482231808.0, "used_bytes": 569262080.0, "buffers_bytes": 89178112.0, "cached_bytes": 889634816.0, "dirty_bytes": 331776.0, "swap_total_bytes": 0.0, "swap_used_bytes": 0.0, "committed_bytes": 600129536.0}, "filesystems": [{"device": "/dev/vda4", "mountpoint": "/", "fstype": "ext4", "total_bytes": 9197531136, "used_bytes": 5252186112, "free_bytes": 3456028672, "used_pct": 60.313006078468355, "read_only": false}, {"device": "/dev/vda3", "mountpoint": "/boot", "fstype": "ext4", "total_bytes": 1020702720, "used_bytes": 272543744, "free_bytes": 677695488, "used_pct": 28.68159246870582, "read_only": false}, {"device": "/dev/vda2", "mountpoint": "/boot/efi", "fstype": "vfat", "total_bytes": 209489920, "used_bytes": 9560064, "free_bytes": 199929856, "used_pct": 4.56349594290742, "read_only": false}], "disk_io": {"devices": [{"name": "vda", "model": null, "rotational": true, "size_bytes": 10737418240.0, "reads_total": 9597.0, "writes_total": 346444.0, "read_bytes_total": 786491904.0, "write_bytes_total": 3761259008.0, "in_flight": 0.0, "io_time_ms_total": 654478.0, "weighted_io_ms_total": 3369847.0, "discard_bytes_total": 0.0}], "device_names": ["vda"]}, "network": {"interfaces": [{"name": "eth0", "rx_bytes_total": 2597455186.0, "rx_packets_total": 4555586.0, "rx_errors_total": 0.0, "rx_dropped_total": 29.0, "tx_bytes_total": 1960356455.0, "tx_packets_total": 2173933.0, "tx_errors_total": 0.0, "tx_dropped_total": 0.0, "speed_mbps": null}]}, "gpus": [], "temperatures": [], "cpu_temperature_celsius": null, "processes": {"total": 138, "top_rss": [{"pid": 593, "name": "systemd-journal", "rss_bytes": 105168896.0}, {"pid": 1182, "name": "haproxy", "rss_bytes": 98664448.0}, {"pid": 733, "name": "firewalld", "rss_bytes": 45543424.0}, {"pid": 1, "name": "systemd", "rss_bytes": 37601280.0}, {"pid": 767, "name": "tuned", "rss_bytes": 28323840.0}], "top_cpu": [{"pid": 1441, "name": "python3", "cpu_pct": 0.20127003994925524}, {"pid": 1, "name": "systemd", "cpu_pct": 0.0}, {"pid": 2, "name": "kthreadd", "cpu_pct": 0.0}, {"pid": 3, "name": "pool_workqueue_release", "cpu_pct": 0.0}, {"pid": 4, "name": "kworker/R-rcu_gp", "cpu_pct": 0.0}]}, "errors": []}
tests/fixtures/nut_apc.vars
Vendored
+2 -2
View File
@@ -9,7 +9,7 @@ VAR apc battery.charge.low "10"
VAR apc battery.charge.warning "50"
VAR apc battery.date "2001/09/25"
VAR apc battery.mfr.date "2021/06/24"
VAR apc battery.runtime "648"
VAR apc battery.runtime "633"
VAR apc battery.runtime.low "120"
VAR apc battery.type "PbAc"
VAR apc battery.voltage "27.3"
@@ -42,7 +42,7 @@ VAR apc ups.beeper.status "disabled"
VAR apc ups.delay.shutdown "20"
VAR apc ups.firmware "952.e5 .D"
VAR apc ups.firmware.aux "e5 "
VAR apc ups.load "48"
VAR apc ups.load "50"
VAR apc ups.mfr "American Power Conversion"
VAR apc ups.mfr.date "2021/01/08"
VAR apc ups.model "Back-UPS RS 1500MS"
tests/fixtures/nut_cyberpower.vars
Vendored
+2 -2
View File
@@ -46,10 +46,10 @@ VAR cyberpower ups.firmware "CR01906AC911"
VAR cyberpower ups.load "14"
VAR cyberpower ups.mfr "CPS"
VAR cyberpower ups.model "GX1500U"
VAR cyberpower ups.power "131"
VAR cyberpower ups.power "132"
VAR cyberpower ups.power.nominal "1500"
VAR cyberpower ups.productid "0601"
VAR cyberpower ups.realpower "127"
VAR cyberpower ups.realpower "129"
VAR cyberpower ups.realpower.nominal "900"
VAR cyberpower ups.serial "SCRUBBEDSERIAL1"
VAR cyberpower ups.status "OL"
tests/fixtures/slots_gufo.json
Vendored
+1
View File
@@ -0,0 +1 @@
[{"id":0,"n_ctx":262144,"speculative":true,"is_processing":false,"id_task":-1,"task_id":-1,"state":0,"n_prompt_tokens":0,"n_prompt_tokens_cache":0,"n_prompt_tokens_processed":0,"prompt":"","next_token":[{"has_next_token":false,"has_new_line":false,"n_remain":-1,"n_decoded":0}],"model":"turbo"},{"id":1,"n_ctx":262144,"speculative":true,"is_processing":false,"id_task":-1,"task_id":-1,"state":0,"n_prompt_tokens":0,"n_prompt_tokens_cache":0,"n_prompt_tokens_processed":0,"prompt":"","next_token":[{"has_next_token":false,"has_new_line":false,"n_remain":-1,"n_decoded":0}],"model":"turbo"}]
tests/slots_contract.py
+204
View File
@@ -0,0 +1,204 @@
"""The `/slots` row-shape contract behind the `/llm` slot grid (phase 07, task 01).
The old gufo slot table was deleted unread: it looked for `slot.state` as a *string*,
`slot.n_prompt`, `slot.n_predict`, `slot.cache` and `slot.decode_per_second`, and the live
server publishes none of those names — every row rendered em-dashes forever. That is
`AGENTS.md` rule 7 failing in slow motion ("trust a memory of an upstream"), so the
replacement grid (task 04) is written against a *recorded* ground truth instead:
`tests/fixtures/slots_gufo.json`, the verbatim `/slots` reply from the running gufo,
re-captured by `./scripts/capture-fixtures.sh`.
The same device as `tests/card_contract.py`: the tables live here so the hermetic test
(`tests/test_llm_slots_contract.py`, driven from the fixture) and the live test
(`tests/test_live_fleet.py::test_gufo_slots_answer_the_pinned_shape`) assert one list and
cannot drift apart.
====================== ==========================================================
Slot card cell Path in a `/slots` row
====================== ==========================================================
card key ``id`` (unique per row — the grid keys one card per slot)
state pill ``is_processing`` (authoritative bool) / ``state`` (legacy 0/1)
request ``id_task`` — canonical; the ``task_id`` alias is captured but
**never rendered**
prompt tok ``n_prompt_tokens`` (+ ``n_prompt_tokens_processed`` sub-line)
cache hit ``n_prompt_tokens_cache`` (+ ratio against ``n_prompt_tokens``)
generated ``next_token[0].n_decoded`` (+ optional ``prob`` sub-line)
remaining ``next_token[0].n_remain`` — where the capture puts it, never a
row-level ``n_remain`` fallback
context ``n_ctx``
model name ``model``
====================== ==========================================================
The field names and types are the live capture's. `llm_monitor` reads the same set with
``a ?? b`` fallbacks (the task-id alias pair, ``n_remain`` at either level); the grid drops
them. Task 04's rule is that **this contract names the canonical key of each fact and the
page renders that one**, so a rename fails a test here instead of quietly reading a different
field. `llm_monitor` is the design reference, not a dependency.
``prompt`` is deliberately **not** a card cell: on a busy slot it is the user's in-flight
request text. `./scripts/capture-fixtures.sh` blanks it in the fixture, and the page must
never read it.
The capture is an **idle** engine — gufo answers one row per parallel slot even when
nothing is generating, so "every slot idle, prompt blank" is the shape the grid must
handle; that assumption is pinned in the hermetic test, and the mutation tests below keep
the contract honest for busy rows too.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
FIXTURES = Path(__file__).resolve().parent / "fixtures"
#: The one name the grid prints for "which request is in this slot". The live capture answers
#: **both** `id_task` and `task_id` for the same fact (and `llm_monitor` reads `id_task`
#: first), so a name has to be made canonical rather than chained: `slot.id_task ??
#: slot.task_id` is exactly the guessing that made the deleted table impossible to trust. If a
#: re-capture ever answers only the alias, this constant changes and the grid follows — the
#: fallback never moves into the page (`page-llm.js`'s `SLOT_FIELDS` names this key once).
CANONICAL_TASK_ID_FIELD: str = "id_task"
#: Other names a row may answer for that one fact: type-checked when present, rendered by
#: nothing.
TASK_ID_ALIASES: tuple[str, ...] = ("task_id",)
#: Every name known for the request id, canonical first (for messages and cross-checks).
TASK_ID_FIELDS: tuple[str, ...] = (CANONICAL_TASK_ID_FIELD, *TASK_ID_ALIASES)
#: Doc table: slot-grid cell -> the row path(s) that cell reads. The authoritative
#: presence/type rules are the three tables below it.
SLOT_CELLS: dict[str, tuple[str, ...]] = {
"card key": ("id",),
"state pill": ("is_processing", "state"),
"request": (CANONICAL_TASK_ID_FIELD,),
"prompt tok": ("n_prompt_tokens", "n_prompt_tokens_processed"),
"cache hit": ("n_prompt_tokens_cache",),
"generated": ("next_token.n_decoded", "next_token.prob"),
"remaining": ("next_token.n_remain",),
"context": ("n_ctx",),
"model name": ("model",),
}
#: Field -> (accepted types, why the grid needs it). Every row must carry every one of
#: these; a missing key is the drift that made the old table render dashes.
REQUIRED_ROW_FIELDS: dict[str, tuple[tuple[type, ...], str]] = {
"id": ((int,), "the grid keys one card per slot — a duplicate or absent id is two "
"cards for one slot or a slot nobody updates"),
"is_processing": (((bool,), "the authoritative busy/idle verdict behind the state pill")),
"state": ((int,), "the legacy numeric state (0 idle / 1 processing); the pill prefers "
"is_processing but labels anything else it cannot name"),
CANONICAL_TASK_ID_FIELD: ((int,), "the request cell — the only name the grid reads for "
"it, so its absence is drift rather than a blank cell"),
"n_ctx": ((int,), "the context cell — the slot's window size"),
"n_prompt_tokens": ((int, float), "the prompt cell"),
"n_prompt_tokens_processed": ((int, float), "the `processed …` sub-line while prefill runs"),
"n_prompt_tokens_cache": ((int, float), "the cache-hit cell (ratio against n_prompt_tokens)"),
"next_token": ((list,), "the generated cell reads the first entry's n_decoded"),
}
#: Fields the grid reads *when present* — absence degrades to `n/a`, presence must keep the
#: type. `model` names the slot header; `prob` is a sub-line llm_monitor prints only if
#: the server publishes it (this gufo does not, on the captured rows).
OPTIONAL_ROW_FIELDS: dict[str, tuple[tuple[type, ...], str]] = {
"model": ((str,), "the model name beside the slot id"),
"speculative": ((bool,), "whether speculative decoding is on for this slot"),
"n_remain": ((int, float), "some builds answer it at the row level; the grid reads "
"next_token[0].n_remain, which every captured row carries"),
"task_id": ((int,), f"the alias of {CANONICAL_TASK_ID_FIELD} — captured, never rendered"),
}
#: Every entry of `next_token` that is present must carry these; an empty list is legal
# (the grid prints `n/a` for generated/remaining, never a fabricated 0).
REQUIRED_NEXT_TOKEN_FIELDS: dict[str, tuple[tuple[type, ...], str]] = {
"n_decoded": ((int, float), "the generated cell"),
"n_remain": ((int, float), "the remaining cell — the captured rows carry it here, "
"not at the row's top level"),
}
OPTIONAL_NEXT_TOKEN_FIELDS: dict[str, tuple[tuple[type, ...], str]] = {
"prob": ((int, float), "the `p=…` sub-line; this capture omits it and the grid copes"),
}
#: Row fields that are user content: captured blanked, never rendered by the page.
USER_CONTENT_FIELDS: tuple[str, ...] = ("prompt",)
def load_slots_capture(name: str = "gufo") -> list[dict[str, Any]]:
"""The verbatim `/slots` capture, as the live server answered it."""
path = FIXTURES / f"slots_{name}.json"
if not path.exists():
raise AssertionError(
f"{path} is missing — run ./scripts/capture-fixtures.sh (it captures gufo's "
"/slots whenever GUFO_BASE_URL is set)"
)
rows = json.loads(path.read_text())
assert isinstance(rows, list), f"{path.name}: /slots answered {type(rows).__name__}, not a list"
return rows
def _matches(value: Any, types: tuple[type, ...]) -> bool:
"""`bool` is a subclass of `int`; never let `true` answer for a number or vice versa."""
if types == (bool,):
return isinstance(value, bool)
if any(t in (int, float) for t in types):
return isinstance(value, (int, float)) and not isinstance(value, bool)
return isinstance(value, types)
def _check_fields(row: dict[str, Any], table: dict[str, tuple[tuple[type, ...], str]],
*, require: bool, where: str) -> None:
for field, (types, why) in table.items():
if require:
assert field in row, (
f"{where}: no key {field!r} — {why}. The deleted slot table rendered dashes "
"for exactly this kind of drift; fail here instead"
)
elif field not in row:
continue
assert _matches(row[field], types), (
f"{where}.{field} is {type(row[field]).__name__}, expected "
f"{'/'.join(t.__name__ for t in types)} — {why}"
)
def assert_slot_shape(rows: list[Any]) -> None:
"""Every `/slots` row has the shape the slot grid reads. Empty list is legal.
Drift in either direction fails: a renamed/missing key (including the canonical request
id — the alias does not excuse its absence), a type the grid cannot format, or a
duplicated slot id (keyed cards would collide).
"""
assert isinstance(rows, list), f"/slots answered {type(rows).__name__}, not a list of rows"
seen_ids: list[int] = []
for index, row in enumerate(rows):
where = f"row[{index}]"
assert isinstance(row, dict), f"{where} is {type(row).__name__}, not an object"
_check_fields(row, REQUIRED_ROW_FIELDS, require=True, where=where)
_check_fields(row, OPTIONAL_ROW_FIELDS, require=False, where=where)
# The canonical request id is required above. An alias may ride along (every
# captured row answers both) and the grid ignores it.
tokens = row["next_token"]
for entry in tokens:
assert isinstance(entry, dict), f"{where}.next_token entry is not an object"
_check_fields(entry, REQUIRED_NEXT_TOKEN_FIELDS, require=True,
where=f"{where}.next_token")
_check_fields(entry, OPTIONAL_NEXT_TOKEN_FIELDS, require=False,
where=f"{where}.next_token")
# The pill: `is_processing` is authoritative, and `state == 0` must not contradict
# it — llm_monitor's exact reading, so a busy row and an idle row stay tellable.
if row["is_processing"]:
assert row["state"] != 0, (
f"{where}: is_processing=True with state=0 — the pill cannot tell the operator "
"which field to believe"
)
seen_ids.append(row["id"])
duplicates = sorted({i for i in seen_ids if seen_ids.count(i) > 1})
assert not duplicates, (
f"duplicate slot ids {duplicates} — the grid keys one card per slot and a dupe "
"renders one card twice (or drops the other)"
)
tests/test_api.py
+754 -30
View File
@@ -24,8 +24,22 @@ from dashboards.app import create_app
from dashboards.config import Settings, Target
from dashboards.severity import classify
from dashboards.sources import nut
from dashboards.sources.llm import HISTORY_METRICS
from dashboards.sources.llm import (
HISTORY_METRICS,
parse_gufo_metrics,
parse_vllm_metrics,
)
from dashboards.store import Event, Sample, SampleStore
from slots_contract import (
CANONICAL_TASK_ID_FIELD,
OPTIONAL_NEXT_TOKEN_FIELDS,
OPTIONAL_ROW_FIELDS,
REQUIRED_NEXT_TOKEN_FIELDS,
REQUIRED_ROW_FIELDS,
TASK_ID_ALIASES,
USER_CONTENT_FIELDS,
load_slots_capture,
)
# refused immediately, and never dials out of the test environment
DEAD_URL = "http://127.0.0.1:1"
@@ -547,6 +561,42 @@ def _code(body: str) -> str:
return re.sub(r"^\s*//.*$", " ", body, flags=re.M)
# The `fleet` object paths `/llm`'s aggregate block reads. `engines_total` drives the
# "hidden below two engines" rule and the tag; `engines_live` is what makes a partial sum
# visible; the rest are the row itself.
FLEET_BLOCK_PATHS = frozenset({
"engines_total", "engines_live", "requests_active", "requests_waiting",
"prompt_tokens_per_second", "decode_tokens_per_second", "tokens_total",
})
# The summed half of that list — the ones that must be null, not 0, with nothing answering.
FLEET_SUM_PATHS = ("requests_active", "requests_waiting", "prompt_tokens_per_second",
"decode_tokens_per_second", "tokens_total")
def _fleet_tag(html: str) -> str:
"""The aggregate block's opening tag — the one place its `hidden` default lives."""
start = html.index('<section class="card flat fleet"')
return html[start:html.index(">", start)]
def _fleet_render_path(js: str) -> str:
"""Just the body of `renderFleet()` — the aggregate block's whole code path, comments out.
Everything the block writes goes through here, so it is the slice that can be asserted
verdict-free without tripping over the engine panels' rated KV sparkline next door.
"""
js = _code(js)
return js[js.index("function renderFleet("):js.index("async function refresh(")]
def _fleet_field_metrics(js: str) -> list[str]:
"""The `fleet` keys the block's rows are bound to, in the order the block prints them."""
js = _code(js)
start = js.index("const FLEET_FIELDS = [")
table = js[start:js.index("];", start) + 2]
return re.findall(r"metric: '([a-z_]+)'", table)
def test_no_parallel_severity_vocabulary_exists_anywhere(client) -> None:
"""One word per colour: `ok|warn|fail`, as in severity.py and the existing components."""
for asset in (*ASSETS, "/", "/system", "/llm", "/ups"):
@@ -1219,6 +1269,22 @@ def test_ups_hero_row_makes_room_for_the_glyph(client) -> None:
assert ".ups-hero" in block and "minmax(0, 1fr)" in block, "the row never stacks"
def test_every_ups_chart_names_itself_for_assistive_tech(client) -> None:
"""The four trend plots on `/ups` carry an accessible name, like every other chart here.
`AGENTS.md` states the convention as *every* chart names itself: `LineChart` puts
`role="img"` + `aria-label` on the `<svg>` when the caller passes `ariaLabel`, because
the legend is a sibling that labels nothing and an unlabelled plot is silence. `/system`
(phase 03) and `/llm` (phase 04) were pinned; these four were the stragglers, so the
rule is now pinned on every page that draws one.
"""
js = client.get("/static/page-ups.js").text
charts = js[js.index("function makeCharts("):js.index("function points(")]
assert charts.count("ariaLabel:") == 4, "a `/ups` chart lost its accessible name"
for phrase in ("Battery charge", "Battery runtime", "Battery and line voltage", "UPS load"):
assert phrase in charts, f"the {phrase!r} chart names nothing readable"
# ---- the `/llm` layout (phase 04 task 03) ---------------------------------
#
# The drift this phase fixes was a 14-cell grid where everything was the same size and
@@ -1257,9 +1323,12 @@ def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
assert '<div class="heroes">' in panel, "the hero row left the engine panel"
heroes = re.findall(r"{ key: '(\w+)'[^}]*?hero: true", js)
assert heroes == ["decode", "prompt"], "the hero row is no longer the two throughputs"
# driven by flags, not fourteen identical cells
# driven by flags, not fourteen identical cells. Phase 07 task 03 lifted the summary rows
# out of the panel into the stats strip above it, so what is left here is the pair plus
# the latency rows the strip does not print — never a fourteen-cell grid again.
assert "SPARK_CELLS" in js and "ALARM_CELLS" in js
assert js.count("spark: '") >= 6, "the compact sparkline list shrank back to a grid"
assert js.count("spark: '") >= 2, "the panel lost the latency sparkline rows it kept"
assert js.count("hero: true") == 2, "the hero row is no longer exactly two numbers"
# the hero call must not pass a severity — unrated by design
hero_call = js[js.index("const node = hero("):js.index("if (node) node.title")]
assert "severity" not in hero_call, "throughput invented a verdict"
@@ -1269,22 +1338,32 @@ def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client) -> None:
"""Each compact cell draws its own sparkline over the *selected* window; latency cells
print mean **and** p95 without a hover.
"""Each panel row draws its own sparkline over the *selected* window and prints mean
**and** p95 without a hover.
The window is honest time (`align: 'right'` + `span`), percent-like metrics keep a
fixed 0-100 axis, and the p95 comes from the snapshot's histogram cells — p95 needs no
stored series. The `only` scopes are what keep a gufo panel from showing a latency
mystery and a vLLM panel from showing a spec-decode row it cannot fill.
The window is honest time (`align: 'right'` + `span`) and the p95 comes from the
snapshot's histogram cells — p95 needs no stored series. The `only` scopes are what keep
a gufo panel from showing a latency mystery and a vLLM strip card from showing a
spec-decode row it cannot fill.
Phase 07 task 03 moved the percent and count rows to the stats strip, so the plotted
percent series left the cell list and the axis guarantee moved with it: the KV trend
chart pins `yMax: 100` (an 8 % window that once held a 92 % spike must not read flat),
and the strip converts its 0-1 ratios to per cent in `cellValue()` — a bar cannot
autoscale, so it cannot lie that way either. The `fixedMax`/`minSpan` options stay
plumbed through `drawSparks()` and are pinned on the component itself by
`test_sparkline_ships_the_fixed_scale_and_verdict_options`: a count sparkline that comes
back must not get an axis finer than one request.
"""
js = client.get("/static/page-llm.js").text
sparks = js[js.index("function drawSparks("):js.index("function renderFleet(")]
assert "DashCharts.sparkline(" in sparks
for opt in ("align: 'right'", "span: state.range", "fixedMax", "labels: { max: true }"):
assert opt in sparks, f"cell sparklines lost `{opt}`"
# percent cells pin the axis; ratio series are scaled to the printed percent
assert js.count("fixedMax: 100") >= 3, "KV / prefix-hit / spec-accept lost their 0-100 axis"
assert "scale: 1000" in js and "scale: 100" in js
# the one percent series still plotted owns a fixed 0-100 axis; seconds stay ms
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
assert "yMax: 100" in charts, "the KV trend chart no longer pins its 0-100 axis"
assert "scale: 1000" in js and "cell.scale || 1" in sparks
# every spark cell declares its history series in the markup it is built from
assert 'data-spark data-metric="${cell.spark}"' in js
# percentile foot lines: mean big in the row, p95 beneath, n/a when nothing measured
@@ -1298,10 +1377,6 @@ def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client)
assert "only: ['vllm']" in js and "only: ['gufo']" in js
scoped = js[js.index("function sparkCellsFor("):js.index("function enginePanel(")]
assert "cell.only" in scoped
# a count row never gets an axis finer than one request: history arrives as rollups,
# so 8 s of a single queued request averages to 0.14 and a fitted axis draws a spike for
# a queue nobody was in — the twin of the rescale `fixedMax` fixes
assert _code(js).count("minSpan: 1") == 2, "sessions/queued lost their one-request axis floor"
assert "minSpan: cell.minSpan" in sparks, "cell minSpan never reaches the sparkline"
@@ -1335,32 +1410,681 @@ def test_llm_alarm_cells_exist_only_while_there_is_an_incident(client) -> None:
already treats any non-zero `gufo_device_lost_total` as one. The chip reuses the shared
`.chip.fail` vocabulary; nothing new is painted, and a container emptied at zero means
the row does not exist rather than showing a permanent ignored 0.
Phase 07 task 03 renders them on the engine's strip card (the summary you read first)
instead of inside the panel, so the two `alarm: true` rows live in `STRIP_ROWS` and
`ALARM_CELLS` filters them out of it — counted with the comments stripped, because the
docstring on that table explains the flag by name.
"""
js = client.get("/static/page-llm.js").text
alarms = js[js.index("function renderAlarms("):js.index("function renderSlots(")]
alarms = js[js.index("function renderAlarms("):js.index("function makeCharts(")]
assert "class=\"chip fail\"" in alarms, "the alarm chip left the shared fail vocabulary"
assert "value.value > 0" in alarms
assert "host.hidden = chips.length === 0" in alarms
assert 'data-alarm="${cell.key}"' in alarms
assert js.count("alarm: true") == 2
assert _code(js).count("alarm: true") == 2
assert "ALARM_CELLS = STRIP_ROWS.filter" in js, "the alarm rows are not strip rows"
# the panel no longer carries a second container for the same incident
panel = js[js.index("function enginePanel("):js.index("function renderPanel(")]
assert 'data-k="alarms"' not in panel, "the alarm chips are drawn twice"
strip = js[js.index("function stripCard("):js.index("function renderStripCard(")]
assert 'data-k="alarms"' in strip, "the alarm chips left the strip card"
def test_llm_totals_card_says_only_what_no_engine_panel_says(client) -> None:
"""Engines answering, active, queued — and hidden below two engines.
# ---- the `/llm` stats strip (phase 07 task 03) ---------------------------
#
# `llm_monitor`'s shared strip (`index.html` ~L743), ported: one card per configured engine
# between the fleet block and the panels. Two things make it a summary instead of a second
# set of panels, and both are pinned below: every row is a cell `/api/llm/snapshot` really
# publishes, and no row is printed in the strip *and* in the panel under it.
The summed decode/prompt and the un-rated fleet KV bar are gone: per-engine is the
honest view (the fleet KV bar had no `severity` to render, which is invariant-grade
trouble), and a cross-engine sum of one engine just duplicates that engine's own card
(the `llm_monitor` rule). `/api/llm/snapshot` keeps every field — this is a page
decision, not an API one.
#: row key -> the cells `cellValue()` resolves it from (`sessions.*` is the gufo slot count,
#: which is not a metrics cell). The served `STRIP_ROWS` table must match these keys exactly.
STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
"prompt": ("prompt_tokens_per_second",),
"decode": ("decode_tokens_per_second",),
"tokens_total": ("tokens_total",),
"slots_busy": ("sessions.list", "sessions.active"),
"running": ("requests_processing", "num_requests_running"),
"waiting": ("requests_deferred", "num_requests_waiting"),
"cache_hit": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"),
"spec_accept": ("spec_accept_rate",),
"queue_mean": ("request_queue_time_seconds__mean",),
"preemptions": ("num_preemptions_total",),
"device_lost": ("device_lost_total",),
}
#: rows that are incidents, not stats: absent where the engine does not count them.
ALARM_ROW_KEYS = ("preemptions", "device_lost")
def _strip_rows(js: str) -> dict[str, str | None]:
"""The served `STRIP_ROWS` table: row key -> its `only` scope (`None` = every engine)."""
body = _code(js)
start = body.index("const STRIP_ROWS = [")
table = body[start:body.index("\n];", start)]
rows: dict[str, str | None] = {}
for chunk in table.split("{ key: '")[1:]:
scoped = re.search(r"only: \[([^\]]*)\]", chunk)
rows[chunk.split("'")[0]] = scoped.group(1) if scoped else None
return rows
def _strip_render_path(js: str) -> str:
"""Just `renderStripCard()` — the whole code path that paints one strip card."""
js = _code(js)
return js[js.index("function renderStripCard("):js.index("function pruneEngines(")]
def test_llm_stats_strip_sits_between_the_fleet_block_and_the_panels(client) -> None:
"""The strip is there, ships hidden, names itself, and appears only with engines.
An unconfigured engine gets no card and no strip (invariant 3) — the section is empty in
the markup and `page-llm.js` unhides it only when the payload has an engine to draw.
"""
html = _code(client.get("/llm").text)
for gone in ("f-decode", "f-prompt", "f-kv", "f-tokens", "seam-note"):
assert gone not in html, f"/llm still totals {gone}"
for kept in ("f-live", "f-active", "f-waiting", 'id="totals"'):
assert kept in html, f"/llm lost {kept}"
assert 'id="strip"' in html, "/llm lost the per-engine stats strip"
assert html.index('id="fleet"') < html.index('id="strip"') < html.index('id="engines"'), \
"the strip is no longer between the fleet block and the engine panels"
start = html.index('<section class="strip"')
tag = html[start:html.index(">", start)]
assert "hidden" in tag, "the strip no longer ships hidden"
assert 'aria-label="Engine summary"' in tag, "the strip is a region that names nothing"
js = client.get("/static/page-llm.js").text
assert "card.hidden = total < 2" in js, "the totals card no longer hides for one engine"
assert "$('#strip').hidden = payloads.length === 0" in js
def test_llm_strip_card_is_one_card_per_engine_of_shared_vocabulary(client) -> None:
"""`stripCard()` = `.card` + `.kv` + `.bar` + `.chip.fail`: a layout, not a new component.
Dot, name and age in the head (reachability, the same distinction the panels and the
fleet block draw), the KV bar, the rows keyed by payload field, the alarm container.
"""
js = client.get("/static/page-llm.js").text
card = js[js.index("function stripCard("):js.index("function renderStripCard(")]
assert "card.className = 'card stat'" in card
assert "card.dataset.engine = engine" in card, "the card stopped naming its engine"
assert '<span class="dot idle"></span>' in card
assert '<span class="faint mono updated"></span>' in card, "the card lost its timestamp"
assert '<span class="bar" data-kv-bar><i></i></span>' in card, "the KV bar left `.bar`"
assert '<dl class="kv">' in card, "the strip rows left the shared `.kv` vocabulary"
assert 'data-row="${row.key}"' in card
assert 'data-k="alarms"' in card
# the wrapper is why the shared `.kv` grid has to move *inside* it: applied to the
# wrappers, `max-content 1fr` lays two rows side by side and the card reads as a
# column of orphaned labels (found in the browser, not in a test failure)
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- engine stats strip"):css.index("/* ---- fleet totals")]
assert ".strip .kv { margin: 0; grid-template-columns: 1fr; }" in section
assert ".strip .kv > div {" in section
assert "grid-template-columns: max-content minmax(0, 1fr)" in section
refresh = js[js.index("async function refresh("):js.index("async function refreshEvents(")]
assert "state.strips.get(payload.engine)" in refresh, "the strip rebuilds its cards per poll"
assert "$('#strip').appendChild(strip)" in refresh
assert "renderStripCard(strip, payload, stateRow)" in refresh
prune = js[js.index("function pruneEngines("):js.index("function makeCharts(")]
assert "state.strips" in prune and "state.panels" in prune, \
"an engine that leaves the payload leaves a card behind"
def test_llm_strip_scopes_its_rows_to_the_engines_that_publish_them(client) -> None:
"""gufo gets slots and speculative decoding; vLLM gets running and queue time."""
rows = _strip_rows(client.get("/static/page-llm.js").text)
assert rows["slots_busy"] == "'gufo'" and rows["spec_accept"] == "'gufo'"
assert rows["running"] == "'vllm'" and rows["queue_mean"] == "'vllm'"
assert rows["waiting"] is None and rows["cache_hit"] is None and rows["tokens_total"] is None
js = client.get("/static/page-llm.js").text
scoped = js[js.index("function stripRowsFor("):js.index("function stripCard(")]
assert "row.only" in scoped and "row.only.indexOf(engine)" in scoped
def test_llm_strip_rows_name_cells_the_engines_publish(client, metrics_text) -> None:
"""The strip renders what the payload sends — no row whose cell no engine produces.
`STRIP_ROW_PATHS` mirrors the JS table; every metric cell is checked against the
capture-parsed metrics of each engine that would render that row, so a rename in
`sources/llm.py` fails here instead of printing a permanent `n/a` in the browser, and a
row scoped to one engine cannot leak into the other's card.
"""
rows = _strip_rows(client.get("/static/page-llm.js").text)
assert set(rows) == set(STRIP_ROW_PATHS), \
f"the served strip rows drifted: {set(rows) ^ set(STRIP_ROW_PATHS)}"
parsed = {"gufo": parse_gufo_metrics(metrics_text("gufo")),
"vllm": parse_vllm_metrics(metrics_text("vllm"))}
for key, only in rows.items():
paths = STRIP_ROW_PATHS[key]
if paths[0].startswith("sessions."):
continue # its own payload object; pinned on the JS below
shown = {engine: metrics for engine, metrics in parsed.items()
if not only or engine in only}
present = [engine for engine in shown if any(path in shown[engine] for path in paths)]
if key in ALARM_ROW_KEYS:
# an alarm the engine does not count simply is not there: the chip renders only
# while non-zero, so absence *is* the zero state rather than a dead row
assert present, f"no engine publishes {'/'.join(paths)} for the {key} alarm"
continue
assert len(present) == len(shown), (
f"the {sorted(set(shown) - set(present))} card would print `{key}` as n/a forever "
f"({'/'.join(paths)} is not published)")
def test_llm_strip_slot_count_counts_published_slots_and_never_invents_zero(client) -> None:
"""`slots busy` is a count of `/slots` rows; when /slots did not answer it is `n/a`.
Counting rows the server published is a count of published facts. An empty list from a
server that never answered is not a measurement of an idle engine, so the row falls back
to the published `sessions.active` carrying the API's own `detail` string (rule 2).
"""
js = _code(client.get("/static/page-llm.js").text)
slots = js[js.index("case 'slots_busy':"):js.index("case 'cache':")]
assert "sessions.source" in slots and "sessions.list" in slots
assert "is_processing" in slots, "the count ignores the authoritative busy verdict"
assert "sessions.detail" in slots, "the fallback lost the API's own provenance"
def test_llm_strip_kv_bar_is_coloured_by_the_payload_verdict_only(client) -> None:
"""The strip's KV bar is the page's one rated cell: `severity` in, class out.
`sources/llm.py` rates `llm.kv_cache_usage_pct`, the ratio→percent conversion the panel
cell used to make rides with it, and `Dash.severityFrom` is the only thing that decides a
colour — no threshold in the page, and the stylesheet adds no palette (`ok` stays quiet
ink, exactly like the `.cell` rows and the fleet table).
"""
body = _strip_render_path(client.get("/static/page-llm.js").text)
assert "severityFrom(kv.severity)" in body
assert "cellValue(engine, metrics, sessions, 'cache')" in body
assert "Math.max(0, Math.min(100, kv.value))" in body, "the fill stopped clamping"
assert body.count("severityFrom") == 1, "a second strip row invented a verdict"
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- engine stats strip"):css.index("/* ---- fleet totals")]
assert ".stat-bar.warn .value { color: var(--warn); }" in section
assert ".stat-bar.fail .value { color: var(--fail); }" in section
assert ".stat-bar.ok" not in section, "a healthy KV reading must stay quiet ink"
assert ".prov:empty" in section, "an empty provenance tag draws a pill of nothing"
def test_llm_strip_throughput_carries_provenance_without_colour(client) -> None:
"""`prompt tok/s` and `decode tok/s` keep the `Δ<window>` tag and stay ink.
The tag is generated from the cell's `source` (never hardcoded), the exact source rides
in the row's tooltip, and `severity.py` rates no throughput — so nothing here may paint
it (0 tok/s on an idle engine is healthy).
"""
js = client.get("/static/page-llm.js").text
body = _strip_render_path(js)
assert "row.rate ? provenance(value) : ''" in body
assert "notes.join" in body and "value.source" in body
assert "severityFor" not in js
def test_llm_strip_and_panel_print_no_value_twice(client) -> None:
"""One value, one home — the UPS hero row's rule, applied across two components.
Task 03 lifted sessions, queue depth, KV pressure, the cache-hit and spec-accept percents,
the lifetime token counter and the alarm chips out of the panel cell list rather than
drawing them twice. The two throughput heroes are the exemption the task names: the panel
keeps its hero row (so does `llm_monitor`, which carries the pair in both places), and
everything else appears once. Queue time moved with its p95 foot, so the move lost nothing.
"""
js = _code(client.get("/static/page-llm.js").text)
start = js.index("const CELLS = [")
panel_keys = set(re.findall(r"\{ key: '(\w+)'", js[start:js.index("\n];", start)]))
assert panel_keys == {"decode", "prompt", "ttft", "tpot"}, \
f"the panel cell list changed shape: {sorted(panel_keys)}"
shared = set(_strip_rows(js)) & panel_keys
assert shared == {"decode", "prompt"}, f"printed in the strip and the panel: {sorted(shared)}"
body = _strip_render_path(js)
assert "p95Value(metrics, row.key)" in body, "queue time lost its p95 in the move"
def test_llm_strip_prints_no_peak_context_number(client) -> None:
"""The phase's assumption, pinned: `llm_monitor`'s peak-ctx row does not come over.
Its cell is `llamacpp:n_tokens_max`, a since-start maximum rather than a glance number,
and the task says this strip gets no peak-context cell. The series stays stored and
queryable from `/api/llm/history` — not plotting it is not removing it.
"""
js = _code(client.get("/static/page-llm.js").text)
assert "n_tokens_max" not in js, "/llm reads the peak-context gauge again"
assert not [key for key in _strip_rows(js) if "peak" in key]
def test_llm_fleet_block_leads_the_page_with_the_engine_sums(client) -> None:
"""The page opens with llm_monitor's aggregate block: the sums no panel can say.
Phase 04 retired these numbers to counts; the operator asked for the aggregate
throughput back (`TODO.md:5`), and phase 07 restores it in `llm_monitor`'s shape —
status dot, "all engines", an `N of M engines` tag, the note that the sums cover only
the engines answering now, then running / waiting / prompt / decode / total tokens.
The counts-only card it replaces (`#totals`, `f-live`, `f-active`) is gone, ids and all:
an element nothing writes is the orphan this repo pins against.
"""
html = _code(client.get("/llm").text)
assert 'id="fleet"' in html, "/llm lost the aggregate block"
assert html.index('id="fleet"') < html.index('id="engines"'), \
"the fleet block is no longer above the engine panels"
for part in ("fleet-dot", "fleet-count", "f-running", "f-waiting", "f-prompt",
"f-decode", "f-tokens"):
assert f'id="{part}"' in html, f"/llm lost the fleet block's {part}"
for words in ("all engines", "sums only engines answering now"):
assert words in html, f"/llm lost the fleet block's `{words}` line"
for gone in ('id="totals"', "f-live", "f-total", "f-active", "f-kv", "seam-note"):
assert gone not in html, f"/llm still carries {gone} from the retired counts card"
js = client.get("/static/page-llm.js").text
body = _fleet_render_path(js)
assert "block.hidden = total < 2" in body, "the fleet block no longer hides for one engine"
assert "hidden" in _fleet_tag(html), \
"the block no longer ships hidden, so a single engine would show a fleet row"
def test_llm_fleet_block_reads_only_paths_the_api_publishes(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Every number the block prints is a `fleet` key of `/api/llm/snapshot`.
The page sums nothing itself — `fleet_totals()` does — so the field table in
`page-llm.js` is a list of API paths, and a rename in `sources/llm.py` must fail here
rather than print a permanent `n/a` in the browser. Pinned with both engines configured
(the keys exist, empty-handed because nothing answered) and with neither (404
`configured: false` without dialling, invariant 3 — and the shipped block stays hidden).
"""
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
fleet = test_client.get("/api/llm/snapshot").json()["fleet"]
assert set(fleet) >= FLEET_BLOCK_PATHS, FLEET_BLOCK_PATHS - set(fleet)
# the served field table is exactly the sum rows, in llm_monitor's order
js = test_client.get("/static/page-llm.js").text
table = _fleet_field_metrics(js)
assert table == ["requests_active", "requests_waiting",
"prompt_tokens_per_second", "decode_tokens_per_second",
"tokens_total"], "the fleet row changed shape without the API"
assert set(FLEET_SUM_PATHS) == set(table), "the block totals something the API does not"
# and the tag's two numbers are API paths too, not arithmetic the page invented
body = _fleet_render_path(js)
assert "fleet.engines_total" in body and "fleet.engines_live" in body
with build_client(tmp_path, monkeypatch) as bare:
response = bare.get("/api/llm/snapshot")
assert response.status_code == 404
assert response.json()["configured"] is False
html = _code(bare.get("/llm").text)
assert 'id="fleet"' in html and "hidden" in _fleet_tag(html), \
"an unconfigured engine must not leave a fleet row on the page"
def test_llm_fleet_totals_drop_an_engine_that_stopped_answering(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A total presented as current is current: a dead engine contributes nothing.
`fleet_totals()`'s docstring is the contract and the block is where it is read. The
injected payload survives the dead-port polls (a failed poll keeps the last good one), so
these are the numbers a browser would get with gufo answering and vLLM down — the live
engine's sums verbatim, `engines_live` one of two, and not one zero where a number the
dead engine never published would sit (rule 2).
"""
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
hub.jobs["llm:gufo"].state.payload = {
"engine": "gufo",
"metrics": {"decode_tokens_per_second": {"value": 41.0},
"prompt_tokens_per_second": {"value": 17.5},
"requests_processing": {"value": 2.0},
"requests_deferred": {"value": 1.0},
"tokens_total": {"value": 512.0}},
"sessions": {"active": 2, "list": []}}
fleet = test_client.get("/api/llm/snapshot").json()["fleet"]
assert fleet["engines_total"] == 2 and fleet["engines_live"] == 1
assert fleet["decode_tokens_per_second"] == pytest.approx(41.0)
assert fleet["prompt_tokens_per_second"] == pytest.approx(17.5)
assert fleet["requests_active"] == pytest.approx(2.0)
assert fleet["requests_waiting"] == pytest.approx(1.0)
assert fleet["tokens_total"] == pytest.approx(512.0)
# and with neither engine answering, every sum is null, never a confident 0
hub.jobs["llm:gufo"].state.payload = {"engine": "gufo", "metrics": {},
"sessions": {"active": None, "list": []}}
empty = test_client.get("/api/llm/snapshot").json()["fleet"]
assert empty["engines_live"] == 0
assert [empty[key] for key in FLEET_SUM_PATHS] == [None] * len(FLEET_SUM_PATHS)
def test_llm_fleet_block_proves_where_its_sums_come_from(client) -> None:
"""The seam tooltip, the excluded-engine tooltip, and an `n/a` with no unit.
Two provenance rules the rest of the app holds to, on the one block that adds engines
together: the prompt sum carries `fleet.seams` — the API's own words on why gufo's and
vLLM's prompt counters are not the same quantity — and the tag's tooltip names the
engines `fleet_totals()` left out, so a partial total says so in words and not only by
being dimmed. And a sum the API did not publish prints `n/a` with its unit cleared: "n/a
tok/s" would read as a measurement of nothing.
"""
js = _code(client.get("/static/page-llm.js").text)
body = _fleet_render_path(js)
assert "fleet.seams" in body, "the prompt sum lost the API's seam note"
assert "not answering — excluded from every sum below" in body, \
"the tag no longer names the engines left out of the totals"
assert "value == null ? ''" in body, "an n/a sum can still print a unit"
assert "value == null ? ''" in body, "an n/a sum can still print a unit"
def test_llm_fleet_block_colours_no_number(client) -> None:
"""No verdict on the aggregate row — and no dimming of anything but the tag.
`severity.py` rates no throughput and no queue depth (0 tok/s on an idle fleet is
healthy), so the block renders ink: the render path touches no `severity`, calls no
`Dash.hero()`, and the stylesheet section for the block reaches for no verdict token.
The one coloured thing on it is the shared **reachability** dot — the same distinction
the engine panels draw between their numbers and their header dot — and the partial-fleet
signal is the shared `.stale` dim on the count tag, which is opacity, not a verdict.
"""
js = _code(client.get("/static/page-llm.js").text)
body = _fleet_render_path(js)
assert "severity" not in body, "the fleet block invented a verdict"
assert "hero(" not in body, "the fleet block drew a hero number it cannot rate"
assert "setDot(" in body and "markStale(tag" in body, \
"a dead engine no longer dims the tag or darkens the block's dot"
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- fleet totals"):css.index("/* ---- hero numbers")]
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
assert token not in section, f"the fleet block coloured a number with {token}"
assert ".fleet-head" in section and ".fleet-note" in section, \
"the fleet block's head row lost its styling"
# ---- the `/llm` gufo slot grid (phase 07 task 04) -------------------------
#
# `llm_monitor`'s slot cards (`index.html` ~L1179), rebuilt against the capture instead of a
# memory: one keyed card per gufo slot, every cell naming **one** key of
# `tests/fixtures/slots_gufo.json`. Two things are pinned below, because they are the two ways
# this component failed before: a card that reads a key the server does not publish (dashes
# forever), and a card that is rebuilt every poll so a busy slot flickers in front of you.
#: The keys a slot card reads, and where they live: `field` (and its optional `sub` /
#: `ratioAgainst` partners) index the row, or its first `next_token` entry when `path` says so.
SLOT_ROW_PATHS = ("field", "sub", "ratioAgainst")
def _slot_table(js: str) -> list[dict[str, str | None]]:
"""The served `SLOT_FIELDS` table, one dict per printed cell."""
body = _code(js)
start = body.index("const SLOT_FIELDS = [")
table = body[start:body.index("\n];", start)]
cells: list[dict[str, str | None]] = []
for chunk in table.split("{ key: '")[1:]:
cell: dict[str, str | None] = {"key": chunk.split("'")[0]}
for option in (*SLOT_ROW_PATHS, "path"):
found = re.search(rf"{option}: '([a-z_0-9]+)'", chunk)
cell[option] = found.group(1) if found else None
cells.append(cell)
return cells
def _slot_js(js: str) -> str:
"""The slot grid's whole code path, comments out: `slotValue()` to the end of the block."""
body = _code(js)
return body[body.index("function slotValue("):body.index("function makeCharts(")]
def _slot_markup(js: str) -> str:
"""Just `slotSection()` — the markup the grid, its label and its empty states are built from."""
body = _code(js)
return body[body.index("function slotSection("):body.index("function renderPanel(")]
def test_llm_gufo_panel_holds_the_slot_grid_under_its_label(client) -> None:
"""`sessions (GET /slots)`, then the keyed grid, then the note about what it cannot contain.
The label is `llm_monitor`'s, the grid ships `hidden` (the empty state is a sentence, never
a card of nothing), and the note stays because a queued (deferred) request is not in
`/slots` at all — without that line an idle grid reads as "nothing waiting".
"""
js = client.get("/static/page-llm.js").text
markup = _slot_markup(js)
assert "sessions (GET /slots)" in markup, "/llm lost the slot grid's label"
# label, grid, empty state, note — in that order, and the grid and the empty state both
# ship hidden so nothing paints before the payload says which of them is true
order = ['class="section-label" data-slots-label', "sessions (GET /slots)",
'<div class="slots" data-slots role="list" hidden>',
'class="slots-empty" data-slots-empty hidden',
'class="slots-note" data-slots-note hidden']
position = -1
for needle in order:
found = markup.index(needle)
assert found > position, f"`{needle}` is out of order in the slot section"
position = found
assert "queued (deferred) requests are not in /slots" in js, "the queued-request note is gone"
assert "renderSlotGrid(card, enginePayload)" in js, "nothing draws the grid any more"
# the label is a heading and the grid a list: a screen reader gets the block's name and
# its item count, which a stack of divs does not say
assert '<h4 class="section-label" data-slots-label>' in markup, \
"the slot section's label stopped being a heading"
assert 'data-slots role="list"' in markup, "the grid lost its list semantics"
# the grid's own markup: a keyed card, the state pill, and one dd per printed cell
card = js[js.index("function slotCard("):js.index("function slotSub(")]
assert "setAttribute('role', 'listitem')" in card, "the slot cards left the list"
assert 'data-slot-cell="${cell.key}"' in card, "the card cells are not table-driven"
assert 'data-pill' in card and 'data-mark' in card, "the card lost its state pill or glyph"
def test_llm_slot_grid_style_is_a_marker_and_not_a_verdict(client) -> None:
"""A busy slot is a border, a glyph and a word — `severity.py` rates no slot.
Phase 06 settled this argument for GPU utilisation on an LLM server: a slot generating
tokens is the job, so the accent colour says *this is the live one* and no `--ok/--warn/
--fail` token appears anywhere in the section. The glyph (`○`/`●`) and the pill's word mean
the marker survives monochrome, which is the battery glyph's rule applied here.
"""
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- engine slot grid"):css.index("/* ---- fleet totals")]
for selector in (".slots {", ".slot {", ".slot.busy", ".slot-top", ".slot-id",
".slot-model", ".slot dt", ".slot dd", ".slots-empty", ".slots-note",
".section-label"):
assert selector in section, f"dash.css lost the {selector} rule"
assert "var(--accent)" in section, "the busy marker stopped using the non-verdict accent"
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
assert token not in section, f"the slot grid coloured a state with {token}"
js = client.get("/static/page-llm.js").text
assert "'\\u25cf'" in js or "'●'" in js, "the busy glyph left the card"
assert "processing" in _slot_js(js), "the pill stopped saying the state in a word"
def test_llm_slot_cells_each_name_one_key_the_capture_publishes(client) -> None:
"""Every printed cell traces to a key of `tests/fixtures/slots_gufo.json`.
One `field` per cell and no `a ?? b`: the deleted table's `slot.n_prompt ??
slot.prompt_tokens` guessing is what made it print dashes and then plausible noise. A key
the capture does not have fails here, and the alias the capture answers beside `id_task`
must not appear in the page at all — the contract names the canonical one.
"""
# comments out: this test bans names the prose is allowed to explain
js = _code(client.get("/static/page-llm.js").text)
cells = _slot_table(js)
assert cells, "the slot grid has no cell table to read"
assert [cell["key"] for cell in cells] == ["req", "prompt", "cache", "generated",
"remain", "ctx"], \
"the slot card's cell set changed without the capture"
for cell in cells:
assert cell["field"], f"the {cell['key']} cell names no row key"
rows = load_slots_capture()
assert rows, "the capture is empty, so this test would prove nothing"
for cell in cells:
for option in SLOT_ROW_PATHS:
key = cell[option]
if key is None:
continue
in_next = cell["path"] == "next_token"
required = REQUIRED_NEXT_TOKEN_FIELDS if in_next else REQUIRED_ROW_FIELDS
optional = OPTIONAL_NEXT_TOKEN_FIELDS if in_next else OPTIONAL_ROW_FIELDS
assert key in required or key in optional, (
f"the {cell['key']} cell reads {key!r}, which no contract table declares — the "
"page would be reading a field nobody pinned")
for row in rows:
source = row["next_token"][0] if in_next else row
if key in required:
assert key in source, (
f"the {cell['key']} cell reads {key!r}, which "
f"{'next_token[0]' if in_next else 'the row'} does not publish — this is "
"how the deleted table rendered dashes")
elif key in source: # optional: `prob` is on no captured row, and that is legal
assert isinstance(source[key], (int, float)), (
f"{key!r} is {source[key]!r}, which the cell cannot format")
assert cells[0]["field"] == CANONICAL_TASK_ID_FIELD, \
"the request cell stopped reading the canonical task id"
for alias in TASK_ID_ALIASES:
assert alias not in js, f"page-llm.js renders the {alias} alias the contract does not name"
# the old table's invented names stay invented, and so does its per-slot speed
for gone in ("n_predict", "decode_per_second", "cache_usable", "n_ctx_used"):
assert gone not in js, f"page-llm.js reads the invented {gone} again"
def test_llm_slot_card_prints_no_speed_it_cannot_measure(client) -> None:
"""No per-slot speed cell: the capture publishes no per-slot rate key.
`llm_monitor`'s grid has one, and so did the deleted table (`decode_per_second`) — which is
why it was dashes. A page-side Δn_decoded/Δt would be a measurement with no estimator, no
`Δ<window>` label and no `source` string, i.e. rule 2 with a confident number printed on it,
so the number stays on the engine's strip card and panel, where `rates.py` owns it.
"""
js = _code(client.get("/static/page-llm.js").text)
assert not [cell for cell in _slot_table(js)
if cell["key"] in {"speed", "tok_s", "decode"}], "a slot cell grew a rate"
assert "decode_per_second" not in js
def test_llm_slot_sentinels_print_a_word_or_na_never_a_negative_number(client) -> None:
"""`-1` is llama.cpp's "nothing in this slot", and it never becomes a countdown.
An idle slot answers `id_task: -1` and `next_token[0].n_remain: -1` (both in the capture).
The request cell says `idle` in a word; the remaining cell prints `n/a`. Neither prints a
negative number, and neither prints the `0` that would read as "one token from done".
"""
body = _slot_js(client.get("/static/page-llm.js").text)
assert "value < 0 ? 'idle'" in body, "the request cell prints the -1 sentinel again"
assert "value >= 0 ? value : null" in body, "the remaining cell prints the -1 sentinel"
def test_llm_slot_card_header_reads_only_fields_the_contract_declares(client) -> None:
"""The card top prints `model`, `speculative`, `id` and the state pair — and nothing else.
`id` keys the card (escaped once at build time), `is_processing` is the authoritative pill,
`state` labels a number the boolean cannot express, and `model` + `speculative` are the
footnote beside them. Each is declared in a contract table, so a rename fails there. The
user-content field a busy row carries (`prompt`) is the one key this page must never read.
"""
js = client.get("/static/page-llm.js").text
header = js[js.index("function renderSlotCard("):js.index("function renderSlotGrid(")]
read = {"is_processing", "state", "model", "speculative"}
for key in read:
assert f"row.{key}" in header, f"the slot card header stopped reading {key}"
declared = set(REQUIRED_ROW_FIELDS) | set(OPTIONAL_ROW_FIELDS)
assert read <= declared, "the header reads a field no contract table declares"
for field in USER_CONTENT_FIELDS:
assert field not in header, f"the slot card prints the user's {field}"
assert "escapeHtml(id)" in js, "the slot id reached the markup unescaped"
def test_llm_slot_cards_are_keyed_created_once_and_patched_in_place(client) -> None:
"""A busy slot does not get rebuilt every poll — and a slot that leaves loses its card.
`appendChild` of the existing node keeps DOM order == slot order; everything else is a text
patch, so a poll cannot flicker a card or re-announce it to assistive tech for a difference
of a few tokens. An engine that leaves the payload loses its card map with its panel, or the
refs of removed nodes would be re-appended when it came back.
"""
js = client.get("/static/page-llm.js").text
body = _slot_js(js)
assert "state.slots.get(enginePayload.engine)" in body, "the grid stopped keying its cards"
assert "if (!refs) {" in body and "refs = slotCard(id)" in body
assert "grid.appendChild(refs.root)" in body, "slot order is no longer kept by moving nodes"
assert "byId.delete(id)" in body, "a slot that left keeps its card forever"
assert "byId.forEach((refs) => refs.root.remove())" in body, \
"the empty state left cards on the page"
assert "setTxt(box.value" in body, "the cell values stopped being patched in place"
assert "innerHTML" not in body.split("function renderSlotCard(")[1], \
"a card is rebuilt from markup on a poll"
prune = js[js.index("function pruneEngines("):js.index("/* ---- the gufo slot grid")]
assert "state.slots.delete(engine)" in prune, "a pruned engine kept its slot cards"
def test_llm_slot_grid_says_why_there_are_no_rows(client) -> None:
"""Two honest empty states, and neither of them is a card or a 0-count claim.
`/slots` did not answer → the API's own `sessions.detail` sentence (the same words the
strip's session row carries). It answered with no rows → "engine reports 0 slots", which is
a fact about that answer. `sessions.source` is the gate that tells the two apart.
"""
body = _slot_js(client.get("/static/page-llm.js").text)
assert "Boolean(sessions.source)" in body, "the grid cannot tell no-answer from no-rows"
assert "engine reports 0 slots" in body
assert "sessions.detail ||" in body, "the no-answer state lost the API's own sentence"
assert "no /slots answer from this engine" in body, "the no-answer state can print undefined"
def test_llm_vllm_panel_says_there_is_no_per_slot_json(client) -> None:
"""vLLM gets one line and no grid — no fake rows for a server that cannot answer.
`sources/llm.py`'s `_sessions()` publishes counts for vLLM and says so in `detail`; the
panel turns that into the `llm_monitor` line, with the API's sentence as the tooltip. The
gate is the engine's name (`SLOT_ENGINES`), not `sessions.list.length`, so an engine that
has not answered yet still shows its label and the honest empty sentence.
"""
js = client.get("/static/page-llm.js").text
markup = _slot_markup(js)
branch = markup.split("if (SLOT_ENGINES.indexOf(engine) < 0) {")[1].split("}")[0]
assert 'class="slots-note" data-slots-note' in branch, "the vLLM line lost its element"
assert "NO_SLOT_JSON" in branch and "vLLM exposes no per-slot JSON" in js
assert '<div class="slots"' not in branch, "the vLLM panel grew a grid it cannot fill"
body = _slot_js(js)
assert body.startswith("function slotValue("), "the grid's code path moved"
assert "if (!grid) {" in body, "renderSlotGrid no longer leaves a panel with no grid alone"
assert "note.title = enginePayload.sessions.detail" in body, \
"the vLLM line stopped carrying the API's own words"
def test_the_captured_slot_rows_fill_the_cells_the_grid_prints(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The capture, through the API, into the keys the served cell table reads.
The fixture rows are injected as a live job payload (the `/slots` passthrough verbatim), so
this is the render path's input as the browser would receive it: `sessions.source` set (the
grid's gate), every row present, and every key `SLOT_FIELDS` names resolvable in it. A
rename in `sources/llm.py` or in gufo fails here rather than leaving a card of `n/a`s.
"""
rows = load_slots_capture()
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
hub.jobs["llm:gufo"].state.payload = {
"engine": "gufo", "metrics": {"requests_processing": {"value": 1.0}},
"sessions": {"active": 1, "list": rows, "source": "gufo /slots", "detail": None},
}
engines = test_client.get("/api/llm/snapshot").json()["engines"]
gufo = next(row for row in engines if row["engine"] == "gufo")
sessions = gufo["sessions"]
assert sessions["source"] == "gufo /slots", "the grid's gate is gone from the payload"
assert sessions["list"] == rows, "the rows are no longer the passthrough the grid reads"
cells = _slot_table(test_client.get("/static/page-llm.js").text)
for cell in cells:
for option in SLOT_ROW_PATHS:
key = cell[option]
if key is None:
continue
in_next = cell["path"] == "next_token"
required = REQUIRED_NEXT_TOKEN_FIELDS if in_next else REQUIRED_ROW_FIELDS
for row in sessions["list"]:
source = row["next_token"][0] if in_next else row
if key not in required and key not in source:
continue # an optional sub-line this build does not publish
value = source[key]
assert isinstance(value, (int, float)) and not isinstance(value, bool), (
f"the {cell['key']} cell reads {key!r} = {value!r}, which the number "
"formatter cannot print")
def test_llm_charts_are_secondary_and_named(client) -> None:
tests/test_live_fleet.py
+25
View File
@@ -33,6 +33,7 @@ from dashboards.rates import RateEstimator
from dashboards.severity import classify
from dashboards.sources import nut
from dashboards.sources.host import HostCollector
from slots_contract import assert_slot_shape
from ups_contract import assert_ups_hero_contract
pytestmark = pytest.mark.live
@@ -127,6 +128,30 @@ async def test_engine_still_publishes_the_series_we_parse(engine: str,
assert result.samples, f"{engine}: nothing to store, so the charts would stay empty"
async def test_gufo_slots_answer_the_pinned_shape(settings: Settings) -> None:
"""The `/slots` rows behind the phase-07 slot grid, asserted against the running gufo.
Hermetic twin: `tests/test_llm_slots_contract.py`, driven from
`tests/fixtures/slots_gufo.json`. This is the half that catches gufo renaming a slot
field — the old table rendered dashes because nobody had ever pinned the shape, and the
assertion here goes through the production path (`collectors.fetch_engine`, which owns
the `/slots` → `/v1/slots` fallback), not a hand-rolled request.
"""
if not settings.gufo_url:
pytest.skip("GUFO_BASE_URL is unset — no gufo to ask for /slots")
rates = RateEstimator(span=settings.rate_window)
async with httpx.AsyncClient(timeout=settings.llm_timeout) as client:
result = await collectors.fetch_engine(client, "gufo", settings.gufo_url,
settings.gufo_api_key, rates, with_slots=True)
rows = result.payload["sessions"]["list"]
assert rows, (
"gufo answered /slots with no rows — even an idle engine reports one row per "
"parallel slot (the idle shape pinned in tests/fixtures/slots_gufo.json); "
f"detail: {result.payload['sessions']['detail']!r}"
)
assert_slot_shape(rows)
def require_upsd(settings: Settings) -> None:
"""Skip, not fail, when there is no real upsd to talk to."""
if not settings.nut_enabled:
tests/test_llm_slots_contract.py
+246
View File
@@ -0,0 +1,246 @@
"""The `/slots` shape contract, driven from the live capture (phase 07, task 01).
The old gufo slot table read field names the server never published and rendered dashes
forever; this file is the fix for the *process*, not just the markup: the shape the slot
grid (task 04) will read is pinned against `tests/fixtures/slots_gufo.json` — the verbatim
`/slots` reply recorded by `./scripts/capture-fixtures.sh` — and against the running gufo
in `tests/test_live_fleet.py::test_gufo_slots_answer_the_pinned_shape`. If the fixture (or
the live server) drifts, a test fails here, not a dashboard at 2am (`AGENTS.md` rule 7).
The table and the assertion live in `tests/slots_contract.py` so both suites read one
list, exactly like `tests/card_contract.py` does for `/system`.
"""
from __future__ import annotations
import copy
import re
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from fastapi.testclient import TestClient
from dashboards.app import create_app
from dashboards.config import Settings
from dashboards.store import SampleStore
from slots_contract import (
CANONICAL_TASK_ID_FIELD,
REQUIRED_NEXT_TOKEN_FIELDS,
REQUIRED_ROW_FIELDS,
SLOT_CELLS,
TASK_ID_ALIASES,
assert_slot_shape,
load_slots_capture,
)
@pytest.fixture
def rows() -> list[dict[str, Any]]:
"""A fresh mutable copy of the capture — mutations in one test cannot leak."""
return copy.deepcopy(load_slots_capture())
@pytest.fixture
def client(tmp_path: Path) -> Iterator[TestClient]:
"""A running app, so the deletion assertions run on the bytes a browser would get."""
db = str(tmp_path / "slots-contract.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db), store)
test_client = TestClient(app)
test_client.__enter__()
try:
yield test_client
finally:
test_client.__exit__(None, None, None)
store.close()
def _code(body: str) -> str:
"""An asset with its comments stripped — prose may *name* a dead thing the code must not be."""
body = re.sub(r"/\*.*?\*/", " ", body, flags=re.S)
body = re.sub(r"<!--.*?-->", " ", body, flags=re.S)
return re.sub(r"^\s*//.*$", " ", body, flags=re.M)
# --------------------------------------------------------------------------- #
# The capture is real, and the contract passes on it
# --------------------------------------------------------------------------- #
def test_the_gufo_slots_capture_is_on_disk_and_is_a_real_answer(
rows: list[dict[str, Any]]) -> None:
"""Every row carries a dozen fields — the whole point of capturing the wire.
Without this, a two-key hand-written stub would satisfy "the key exists" and the
contract would prove nothing (`test_ups_contract.py` holds the same line for the UPS
captures).
"""
assert rows, (
"gufo answered /slots with no rows — even an idle engine reports one row per "
"parallel slot, and that idle shape is exactly what the grid must handle"
)
for row in rows:
assert len(row) >= 10, f"{len(row)} keys — this is not a capture, it is a guess"
def test_the_slot_grid_fields_hold_on_every_captured_row(
rows: list[dict[str, Any]]) -> None:
assert_slot_shape(rows)
def test_the_capture_records_the_idle_shape_the_grid_must_handle(
rows: list[dict[str, Any]]) -> None:
"""The assumption written into the phase: a fresh capture is an idle engine.
Every row is idle with a blank prompt — the grid must render that as N idle cards, not
as "no data". If a re-capture lands mid-request this test names the drift loudly
instead of letting the contract quietly widen; busy rows are covered below.
"""
for row in rows:
assert row["is_processing"] is False, "the capture caught a busy slot; re-run it when idle"
assert row["state"] == 0
assert row["n_prompt_tokens"] == 0
def test_a_busy_row_of_the_same_shape_still_satisfies_the_contract(
rows: list[dict[str, Any]]) -> None:
"""The contract is the row shape, not the idle numbers — a busy slot must pass too."""
busy = rows[0]
busy.update({"is_processing": True, "state": 1, "id_task": 42,
"n_prompt_tokens": 128, "n_prompt_tokens_processed": 64,
"n_prompt_tokens_cache": 16})
busy["next_token"][0]["n_decoded"] = 7
busy["next_token"][0]["prob"] = 0.93 # llm_monitor prints it only when present
assert_slot_shape(rows)
def test_an_empty_row_list_passes_the_contract() -> None:
"""Zero slots is a state the grid renders (an explicit empty message), not a crash."""
assert_slot_shape([])
def test_the_contract_names_every_cell_the_grid_reads() -> None:
"""The doc table covers the llm_monitor cell set; a rename here is a rename there."""
assert set(SLOT_CELLS) == {
"card key", "state pill", "request", "prompt tok", "cache hit",
"generated", "remaining", "context", "model name",
}
for label, paths in SLOT_CELLS.items():
assert paths, f"{label} lists no row path"
# --------------------------------------------------------------------------- #
# Drift fails: the contract is only real if mutations are caught
# --------------------------------------------------------------------------- #
@pytest.mark.parametrize("field", sorted(REQUIRED_ROW_FIELDS))
def test_the_contract_fails_when_a_row_drops_a_field_the_grid_reads(
rows: list[dict[str, Any]], field: str) -> None:
del rows[0][field]
with pytest.raises(AssertionError, match=field):
assert_slot_shape(rows)
@pytest.mark.parametrize(("field", "value"), [
("id", "0"), # a string id would key two cards "0" and 0 apart
("is_processing", 0), # the legacy numeric state is not the authoritative bool
("state", "OL"), # this is not NUT
("n_ctx", None), # None would reach the number formatter
("n_prompt_tokens", True), # bool sneaks past a bare isinstance(x, int)
("next_token", {"n_decoded": 1}),
])
def test_the_contract_fails_on_a_type_the_grid_cannot_render(
rows: list[dict[str, Any]], field: str, value: Any) -> None:
rows[0][field] = value
with pytest.raises(AssertionError, match=field):
assert_slot_shape(rows)
@pytest.mark.parametrize("field", sorted(REQUIRED_NEXT_TOKEN_FIELDS))
def test_the_contract_fails_when_next_token_loses_a_field(
rows: list[dict[str, Any]], field: str) -> None:
del rows[0]["next_token"][0][field]
with pytest.raises(AssertionError, match=field):
assert_slot_shape(rows)
def test_the_contract_fails_when_the_canonical_request_id_is_missing(
rows: list[dict[str, Any]]) -> None:
"""`id_task` is the key the grid prints, so its absence is drift, not a blank cell.
The alias exists in the same row, and `llm_monitor` would have fallen back to it. This
contract does not: a renamed fact fails here rather than rendering a plausible-looking
card out of a different field (`page-llm.js`'s `SLOT_FIELDS` names one key per cell).
"""
del rows[0][CANONICAL_TASK_ID_FIELD]
with pytest.raises(AssertionError, match=CANONICAL_TASK_ID_FIELD):
assert_slot_shape(rows)
def test_the_request_id_alias_is_recorded_and_never_needed(rows: list[dict[str, Any]]) -> None:
"""Dropping the alias must not fail the contract — proof it is not a second source."""
assert set(TASK_ID_ALIASES) == {"task_id"}
for row in rows:
del row["task_id"]
assert_slot_shape(rows)
def test_the_contract_fails_on_duplicate_slot_ids(rows: list[dict[str, Any]]) -> None:
assert len(rows) > 1, "the capture needs two slots for a dupe to be possible"
rows[1]["id"] = rows[0]["id"]
with pytest.raises(AssertionError, match="duplicate slot ids"):
assert_slot_shape(rows)
def test_the_contract_fails_when_the_two_state_fields_disagree(
rows: list[dict[str, Any]]) -> None:
rows[0]["is_processing"] = True # state stays 0 — which field is the pill to believe?
with pytest.raises(AssertionError, match="state"):
assert_slot_shape(rows)
def test_the_contract_fails_when_a_row_is_not_an_object(rows: list[dict[str, Any]]) -> None:
broken: list[Any] = list(rows)
broken[0] = "idle"
with pytest.raises(AssertionError, match="not an object"):
assert_slot_shape(broken)
# --------------------------------------------------------------------------- #
# The deleted table stays deleted (completion criterion, on served bytes)
# --------------------------------------------------------------------------- #
def test_the_broken_slot_table_is_gone_from_html_js_and_css(client: TestClient) -> None:
"""No `#slots-card` / `#slots-table` / `#slots-empty` / `renderSlots` survives.
Asserted on served bytes because there is no JS test runner (a locked anchor). The
guessed field names go with it — those are the memory of an upstream the capture
replaced. The `.chip.info` rule only ever painted that table's "has msg" cell, so it
goes too (orphan-rule policy, `AGENTS.md`).
"""
html = _code(client.get("/llm").text)
for gone in ("slots-card", "slots-table", "slots-empty"):
assert f'id="{gone}"' not in html, f"/llm still renders the {gone}"
js = _code(client.get("/static/page-llm.js").text)
for gone in ("renderSlots", "with_message", "n_predict", "decode_per_second",
"cache_usable"):
assert gone not in js, f"page-llm.js still reads {gone}"
css = client.get("/static/dash.css").text
assert ".chip.info" not in css, "dash.css still styles a class nothing renders"
def test_the_prompt_field_is_user_content_that_never_reaches_the_page(
client: TestClient, rows: list[dict[str, Any]]) -> None:
"""A busy slot's `prompt` is the operator's in-flight request text.
The capture script blanks it (fixture stays shape-true, content-free), and the page
never reads the key — the grid renders tokens, not somebody's prompt.
"""
for row in rows:
assert row.get("prompt", "") == "", "the capture script must blank prompt content"
js = _code(client.get("/static/page-llm.js").text)
# A slot-row read is `slot.prompt` / `row.prompt` / `s.prompt` / `["prompt"]`. A bare
# `.prompt` ban is impossible — `state.charts.prompt` is a chart object — so the name
# before the dot must be a whole word (the lookbehind is what stops `charts.prompt`
# from matching through its trailing "s").
assert not re.search(r"(?<![\w.])(?:slot|row|s|item)\.prompt\b", js), \
"page-llm.js reads a slot row's user-content field"
assert '["prompt"]' not in js and "['prompt']" not in js, \
"page-llm.js indexes a row's prompt by name"
tests/test_llm_source.py
+10 -1
View File
@@ -17,6 +17,7 @@ from dashboards.sources.llm import (
parse_vllm_metrics,
series_names,
)
from slots_contract import load_slots_capture
def bump_metric(text: str, name: str, delta: float) -> str:
@@ -259,10 +260,18 @@ def test_gufo_stores_the_spec_accept_series_the_cell_sparkline_draws(metrics_tex
def test_sessions_pass_through_slots_for_gufo() -> None:
slots = [{"id": 0, "idle": False, "n_prompt": 12, "n_predict": 40}]
"""`sessions.list` is the raw gufo rows — the grid, not the source, reads slot fields.
The rows are phase 07's live capture (`tests/fixtures/slots_gufo.json`), not a guessed
shape: the old table's `idle`/`n_prompt`/`n_predict` names were invented and the server
never answered them. `_sessions()` must pass the rows through untouched — the shape is
contracted separately in `tests/slots_contract.py`.
"""
slots = load_slots_capture()
result = engine_result("gufo", "http://g:1", "", RateEstimator(), slots=slots, now=1.0)
sessions = result.payload["sessions"]
assert sessions["list"] == slots
assert sessions["list"][0]["is_processing"] is False, "the passthrough re-shaped the rows"
assert sessions["source"] == "gufo /slots"