services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 34_radiance_llm_engine

Browse Source 
Everything is verified and green. Final report:

**Phase 34 final verification pass — complete**

- Verified all 6 completed tasks' work against the live fleet and hermetic suite: `llm:radiance` job registered, `llm_engines()` registry drives 404 gate / history defaults / `/api/config` (additive, byte-identical gufo/vllm keys), radiance field map + `/stats` sidecar + 7-position spec curve, `/llm` strip/curve/rows.
- **Defect found & fixed (pre-existing, task 05):** the vLLM-only sentence "vLLM exposes no per-slot JSON…" rendered on the **radiance** panel (`slotSection`'s non-gufo branch). Fix: vLLM line gated to `engine === 'vllm'`; radiance's panel slot section renders nothing (its session home is the strip) — `page-llm.js` + the served-byte pin in `tests/test_api.py` (now asserts the radiance branch is empty and the sentence can't leak). Browser-verified; re-shot `data/logs/34-llm-radiance-rest.png` (the 4.5-min wait found no in-flight request, so the busy shot remains the earlier pass's — its strip rows still stand; only the panel text predates the fix).
- Tests: `uv run pytest` → **2031 passed, 19 skipped** (100%); `uv run pytest --cov` → **98.73%** (floor 94%; config 100%, llm 99%, collectors 98%, api 99%); `uv run ruff check src tests agent && uv run pyright` → clean; `node --check` OK; `uv pip list --outdated` → empty.
- `uv run pytest -m live --live` → **13 passed, 2 skipped** (vLLM unset — correct; live-write opt-in — correct); the read-only radiance test passed.
- Completion criteria: (1) engines `gufo`+`radiance`, `fleet.engines_live` 2, `config.llm.radiance.enabled` true — **pass (live curl)**; (2) unset → 404 no-dial + no job — **pass (spy-pinned, green)**; (3) fixtures present, both families (79 `radiance:` / 198 `vllm:` lines), drift pin — **pass**; (4) panel/curve/rows — **pass (re-shot rest screenshot)**; (5) live radiance test — **pass**; (6) suite/coverage/lint/`bash .agents/validate.sh` (exit 0) — **pass**; (7) docs — below; (8) gufo/vLLM behaviour unchanged — **pass (existing pins untouched)**.
- Documentation: **README.md** full review — env table, LLM-engines section, architecture diagram, capture-script and `/llm` section all name radiance correctly; no inconsistencies found, no changes needed. **AGENTS.md** — the three task-06 fixes (fleet line, route row, module bullets) are in place and all other statements match reality; no changes needed.
- Dev server healthy (`./scripts/dev-server.sh status`: all sources ok, `llm:radiance` answering); all changes left in the working tree for the harness.
- Next pending phase: none — 34 was the last phase in `todo/`.
This commit is contained in:
ducoterra
2026-11-01 18:53:46 -05:00
parent 86c8bf4992
commit 219d7a7c3b
77 changed files with 3517 additions and 149 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/fixtures/metrics_gufo.txt
Vendored
+17 -11
View File
@@ -3,37 +3,43 @@
gufo_device_lost_total 0
# HELP llamacpp:prompt_tokens_total Total prompt tokens processed, excluding cache hits
# TYPE llamacpp:prompt_tokens_total counter
llamacpp:prompt_tokens_total 1350666
llamacpp:prompt_tokens_total 213784
# HELP llamacpp:prompt_tokens_cached_total Total prompt tokens reused from cache
# TYPE llamacpp:prompt_tokens_cached_total counter
llamacpp:prompt_tokens_cached_total 124529105
llamacpp:prompt_tokens_cached_total 38070188
# HELP llamacpp:prompt_seconds_total Prompt process time
# TYPE llamacpp:prompt_seconds_total counter
llamacpp:prompt_seconds_total 1478.1443332270005
llamacpp:prompt_seconds_total 215.41327585600013
# HELP llamacpp:tokens_predicted_total Total tokens generated
# TYPE llamacpp:tokens_predicted_total counter
llamacpp:tokens_predicted_total 877336
llamacpp:tokens_predicted_total 268387
# HELP llamacpp:tokens_predicted_seconds_total Predict process time
# TYPE llamacpp:tokens_predicted_seconds_total counter
llamacpp:tokens_predicted_seconds_total 26565.247833160014
llamacpp:tokens_predicted_seconds_total 6520.4400505340027
# HELP llamacpp:n_tokens_max Largest observed n_tokens.
# TYPE llamacpp:n_tokens_max counter
llamacpp:n_tokens_max 252622
llamacpp:n_tokens_max 222926
# HELP llamacpp:spec_decode_num_drafts_total Total speculative verification rounds
# TYPE llamacpp:spec_decode_num_drafts_total counter
llamacpp:spec_decode_num_drafts_total 291738
llamacpp:spec_decode_num_drafts_total 83278
# HELP llamacpp:spec_decode_num_draft_tokens_total Total draft tokens proposed
# TYPE llamacpp:spec_decode_num_draft_tokens_total counter
llamacpp:spec_decode_num_draft_tokens_total 789028
llamacpp:spec_decode_num_draft_tokens_total 234468
# HELP llamacpp:spec_decode_num_accepted_tokens_total Total draft tokens accepted
# TYPE llamacpp:spec_decode_num_accepted_tokens_total counter
llamacpp:spec_decode_num_accepted_tokens_total 578514
llamacpp:spec_decode_num_accepted_tokens_total 180837
# HELP gufo_spec_lookup_tokens_total Total prompt-lookup copies proposed
# TYPE gufo_spec_lookup_tokens_total counter
gufo_spec_lookup_tokens_total 17416
# HELP gufo_spec_lookup_accepted_tokens_total Total prompt-lookup copies accepted
# TYPE gufo_spec_lookup_accepted_tokens_total counter
gufo_spec_lookup_accepted_tokens_total 13985
# HELP llamacpp:prompt_tokens_seconds Prompt processing speed in tokens per second
# TYPE llamacpp:prompt_tokens_seconds gauge
llamacpp:prompt_tokens_seconds 160.47362399350274
llamacpp:prompt_tokens_seconds 594.36656560014194
# HELP llamacpp:predicted_tokens_seconds Generation speed in tokens per second
# TYPE llamacpp:predicted_tokens_seconds gauge
llamacpp:predicted_tokens_seconds 42.621031668072597
llamacpp:predicted_tokens_seconds 53.07443936596944
# HELP llamacpp:requests_processing Number of admitted requests, including cache preparation
# TYPE llamacpp:requests_processing gauge
llamacpp:requests_processing 0
tests/fixtures/metrics_radiance.txt
Vendored
+428
View File
@@ -0,0 +1,428 @@
# Verbatim /metrics reply from the radiance engine, recorded by
# ./scripts/capture-fixtures.sh — the engine's own Prometheus text
# format: the radiance: native series + the vllm: compatibility family.
# Volatile values (counters, gauges) move every poll, so a diff here
# on re-capture is expected; the contract asserts the series-name set
# against RADIANCE_SERIES/RADIANCE_HISTOGRAMS/RADIANCE_OPTIONAL in
# sources/llm.py — never the numbers (the nut_*.vars precedent).
# HELP vllm:num_requests_running Number of requests currently running on GPU.
# TYPE vllm:num_requests_running gauge
vllm:num_requests_running{model_name="juggernaut",engine="0"} 0
# HELP vllm:num_requests_waiting Number of requests waiting to be processed.
# TYPE vllm:num_requests_waiting gauge
vllm:num_requests_waiting{model_name="juggernaut",engine="0"} 0
# HELP vllm:gpu_cache_usage_perc GPU KV-cache usage. 1 means 100 percent usage.
# TYPE vllm:gpu_cache_usage_perc gauge
vllm:gpu_cache_usage_perc{model_name="juggernaut",engine="0"} 0.74272018671035767
# HELP vllm:kv_cache_usage_perc KV-cache usage. 1 means 100 percent usage.
# TYPE vllm:kv_cache_usage_perc gauge
vllm:kv_cache_usage_perc{model_name="juggernaut",engine="0"} 0.74272018671035767
# HELP radiance:kv_cache_max_tokens Tokens of context the KV cache holds with every block it lends the experts taken back.
# TYPE radiance:kv_cache_max_tokens gauge
radiance:kv_cache_max_tokens{model_name="juggernaut",engine="0"} 443968
# HELP radiance:kv_cache_max_bytes VRAM the KV cache holds with every block it lends the experts taken back, one card.
# TYPE radiance:kv_cache_max_bytes gauge
radiance:kv_cache_max_bytes{model_name="juggernaut",engine="0"} 15901392896
# HELP radiance:kv_resident_blocks Blocks of every streamed KV group the device ring holds; 0 when no group streams.
# TYPE radiance:kv_resident_blocks gauge
radiance:kv_resident_blocks{model_name="juggernaut",engine="0"} 0
# HELP radiance:kv_stream_misses_total Blocks copied into a streamed KV group's ring from the host mirror.
# TYPE radiance:kv_stream_misses_total counter
radiance:kv_stream_misses_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:kv_stream_bytes_total Bytes a streamed KV group's resolve moved, by direction.
# TYPE radiance:kv_stream_bytes_total counter
radiance:kv_stream_bytes_total{model_name="juggernaut",engine="0",dir="in"} 0
# HELP vllm:gpu_prefix_cache_hit_rate GPU prefix cache block hit rate.
# TYPE vllm:gpu_prefix_cache_hit_rate gauge
vllm:gpu_prefix_cache_hit_rate{model_name="juggernaut",engine="0"} 0.97886091470718384
# HELP vllm:prompt_tokens_total Number of prefill tokens processed.
# TYPE vllm:prompt_tokens_total counter
vllm:prompt_tokens_total{model_name="juggernaut",engine="0"} 56553179
# HELP vllm:generation_tokens_total Number of generation tokens processed.
# TYPE vllm:generation_tokens_total counter
vllm:generation_tokens_total{model_name="juggernaut",engine="0"} 423779
# HELP vllm:num_preemptions_total Cumulative number of preemptions from the engine.
# TYPE vllm:num_preemptions_total counter
vllm:num_preemptions_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:preempt_forced_total Preempted prefill waits that ended by reclaiming cache blocks and forcing a lowest-priority running victim (--preempt-wait-ms).
# TYPE radiance:preempt_forced_total counter
radiance:preempt_forced_total{model_name="juggernaut",engine="0"} 0
# HELP vllm:request_success_total Count of successfully processed requests.
# TYPE vllm:request_success_total counter
vllm:request_success_total{model_name="juggernaut",engine="0",finished_reason="stop"} 7
vllm:request_success_total{model_name="juggernaut",engine="0",finished_reason="length"} 0
vllm:request_success_total{model_name="juggernaut",engine="0",finished_reason="tool_calls"} 594
vllm:request_success_total{model_name="juggernaut",engine="0",finished_reason="abort"} 2
vllm:request_success_total{model_name="juggernaut",engine="0",finished_reason="error"} 0
# HELP vllm:time_to_first_token_seconds Histogram of time to first token in seconds.
# TYPE vllm:time_to_first_token_seconds histogram
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.001"} 0
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.0050000000000000001"} 0
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.01"} 0
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.02"} 0
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.040000000000000001"} 0
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.059999999999999998"} 0
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.080000000000000002"} 0
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.10000000000000001"} 0
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.25"} 0
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.5"} 0
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.75"} 1
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="1"} 9
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="2.5"} 407
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="5"} 584
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="7.5"} 595
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="10"} 599
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="20"} 603
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="40"} 603
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="80"} 603
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="160"} 603
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="640"} 603
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="2560"} 603
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
vllm:time_to_first_token_seconds_sum{model_name="juggernaut",engine="0"} 1448.9084620109995
vllm:time_to_first_token_seconds_count{model_name="juggernaut",engine="0"} 603
# HELP vllm:time_per_output_token_seconds Histogram of time per output token in seconds.
# TYPE vllm:time_per_output_token_seconds histogram
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.01"} 237
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.025000000000000001"} 588
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.050000000000000003"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.074999999999999997"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.10000000000000001"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.14999999999999999"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.20000000000000001"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.29999999999999999"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.40000000000000002"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.5"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.75"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="1"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="2.5"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="5"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="7.5"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="10"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="20"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="40"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="80"} 603
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
vllm:time_per_output_token_seconds_sum{model_name="juggernaut",engine="0"} 6.9956323654383148
vllm:time_per_output_token_seconds_count{model_name="juggernaut",engine="0"} 603
# HELP vllm:inter_token_latency_seconds Histogram of inter-token latency in seconds.
# TYPE vllm:inter_token_latency_seconds histogram
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.01"} 237
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.025000000000000001"} 588
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.050000000000000003"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.074999999999999997"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.10000000000000001"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.14999999999999999"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.20000000000000001"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.29999999999999999"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.40000000000000002"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.5"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.75"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="1"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="2.5"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="5"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="7.5"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="10"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="20"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="40"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="80"} 603
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
vllm:inter_token_latency_seconds_sum{model_name="juggernaut",engine="0"} 6.9956323654383148
vllm:inter_token_latency_seconds_count{model_name="juggernaut",engine="0"} 603
# HELP vllm:e2e_request_latency_seconds Histogram of end to end request latency in seconds.
# TYPE vllm:e2e_request_latency_seconds histogram
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.29999999999999999"} 0
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.5"} 0
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.80000000000000004"} 0
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="1"} 0
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="1.5"} 0
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="2"} 10
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="2.5"} 38
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="5"} 271
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="10"} 450
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="15"} 514
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="20"} 538
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="30"} 566
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="40"} 575
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="50"} 585
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="60"} 593
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="120"} 600
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="240"} 602
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="480"} 603
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="960"} 603
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="1920"} 603
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="7680"} 603
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
vllm:e2e_request_latency_seconds_sum{model_name="juggernaut",engine="0"} 6416.8390601850087
vllm:e2e_request_latency_seconds_count{model_name="juggernaut",engine="0"} 603
# HELP vllm:request_decode_time_seconds Histogram of time spent in the decode phase, in seconds.
# TYPE vllm:request_decode_time_seconds histogram
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="0.29999999999999999"} 0
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="0.5"} 2
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="0.80000000000000004"} 44
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="1"} 85
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="1.5"} 155
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="2"} 219
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="2.5"} 274
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="5"} 397
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="10"} 493
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="15"} 531
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="20"} 544
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="30"} 570
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="40"} 579
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="50"} 587
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="60"} 593
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="120"} 600
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="240"} 602
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="480"} 603
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="960"} 603
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="1920"} 603
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="7680"} 603
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
vllm:request_decode_time_seconds_sum{model_name="juggernaut",engine="0"} 4966.4525284600077
vllm:request_decode_time_seconds_count{model_name="juggernaut",engine="0"} 603
# HELP vllm:request_prompt_tokens Number of prefill tokens processed.
# TYPE vllm:request_prompt_tokens histogram
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="1"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="2"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="5"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="10"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="20"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="50"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="100"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="200"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="500"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="1000"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="2000"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="5000"} 0
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="10000"} 8
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="20000"} 25
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="50000"} 111
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="100000"} 353
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="200000"} 603
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
vllm:request_prompt_tokens_sum{model_name="juggernaut",engine="0"} 56553179
vllm:request_prompt_tokens_count{model_name="juggernaut",engine="0"} 603
# HELP vllm:request_generation_tokens Number of generation tokens processed.
# TYPE vllm:request_generation_tokens histogram
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="1"} 0
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="2"} 0
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="5"} 0
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="10"} 0
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="20"} 0
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="50"} 0
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="100"} 52
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="200"} 244
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="500"} 437
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="1000"} 516
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="2000"} 564
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="5000"} 594
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="10000"} 599
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="20000"} 602
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="50000"} 603
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="100000"} 603
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="200000"} 603
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
vllm:request_generation_tokens_sum{model_name="juggernaut",engine="0"} 423779
vllm:request_generation_tokens_count{model_name="juggernaut",engine="0"} 603
# HELP vllm:request_params_n Histogram of the n request parameter.
# TYPE vllm:request_params_n histogram
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="1"} 603
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="2"} 603
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="5"} 603
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="10"} 603
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="20"} 603
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
vllm:request_params_n_sum{model_name="juggernaut",engine="0"} 603
vllm:request_params_n_count{model_name="juggernaut",engine="0"} 603
# HELP vllm:request_params_max_tokens Histogram of the max_tokens request parameter.
# TYPE vllm:request_params_max_tokens histogram
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="1"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="2"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="5"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="10"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="20"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="50"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="100"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="200"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="500"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="1000"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="2000"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="5000"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="10000"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="20000"} 0
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="50000"} 30
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="100000"} 603
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="200000"} 603
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
vllm:request_params_max_tokens_sum{model_name="juggernaut",engine="0"} 38535168
vllm:request_params_max_tokens_count{model_name="juggernaut",engine="0"} 603
# HELP radiance:prefill_tokens_per_second Prefill throughput over the scheduler's recent window.
# TYPE radiance:prefill_tokens_per_second gauge
radiance:prefill_tokens_per_second{model_name="juggernaut",engine="0"} 54.816162109375
# HELP radiance:decode_tokens_per_second Decode throughput over the scheduler's recent window.
# TYPE radiance:decode_tokens_per_second gauge
radiance:decode_tokens_per_second{model_name="juggernaut",engine="0"} 50.574676513671875
# HELP radiance:decode_step_seconds_best Fastest single-sequence decode window this process has run, in seconds per step: the mean of 32 consecutive single-stream steps. With speculation a step emits 1 + accepted tokens. Resets on restart.
# TYPE radiance:decode_step_seconds_best gauge
radiance:decode_step_seconds_best{model_name="juggernaut",engine="0"} 0.032004689000000003
# HELP radiance:decode_step_seconds_recent Most recent single-sequence decode window, in seconds per step (see _best). Holds its value while no single-stream window completes.
# TYPE radiance:decode_step_seconds_recent gauge
radiance:decode_step_seconds_recent{model_name="juggernaut",engine="0"} 0.041546212999999999
# HELP radiance:engine_steps_total Scheduler steps executed. One forward pass of the model, plus whatever draft rounds followed it; with speculation a step emits more than one token.
# TYPE radiance:engine_steps_total counter
radiance:engine_steps_total{model_name="juggernaut",engine="0"} 113985
# HELP radiance:engine_steps_prefill_total Scheduler steps carrying only prefill chunks.
# TYPE radiance:engine_steps_prefill_total counter
radiance:engine_steps_prefill_total{model_name="juggernaut",engine="0"} 1174
# HELP radiance:engine_steps_mixed_total Scheduler steps carrying both a prefill chunk and a decode row.
# TYPE radiance:engine_steps_mixed_total counter
radiance:engine_steps_mixed_total{model_name="juggernaut",engine="0"} 388
# HELP radiance:engine_steps_decode_total Scheduler steps carrying only decode rows.
# TYPE radiance:engine_steps_decode_total counter
radiance:engine_steps_decode_total{model_name="juggernaut",engine="0"} 112423
# HELP radiance:engine_steps_ahead_total Steps whose forward pass was issued behind the step before, ahead of its commit.
# TYPE radiance:engine_steps_ahead_total counter
radiance:engine_steps_ahead_total{model_name="juggernaut",engine="0"} 112279
# HELP radiance:engine_prefill_chunks_capped_total Prefill chunks held back so decode rows and waiting requests got into the step (--busy-prefill-chunk).
# TYPE radiance:engine_prefill_chunks_capped_total counter
radiance:engine_prefill_chunks_capped_total{model_name="juggernaut",engine="0"} 2
# HELP radiance:engine_steps_held_total Decode-only steps run after a prefill chunk while other sequences decode (--prefill-interleave).
# TYPE radiance:engine_steps_held_total counter
radiance:engine_steps_held_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:decode_rows_unspeculated_total Decode rows that took no draft, for any reason.
# TYPE radiance:decode_rows_unspeculated_total counter
radiance:decode_rows_unspeculated_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:engine_step_seconds_total Seconds spent inside a scheduler step, all shapes.
# TYPE radiance:engine_step_seconds_total counter
radiance:engine_step_seconds_total{model_name="juggernaut",engine="0"} 5702.6847473990001
# HELP radiance:engine_step_seconds_decode_total Seconds spent inside a scheduler step carrying only decode rows.
# TYPE radiance:engine_step_seconds_decode_total counter
radiance:engine_step_seconds_decode_total{model_name="juggernaut",engine="0"} 4295.7781233209998
# HELP radiance:prefill_tokens_total Prompt tokens the engine has computed, exact.
# TYPE radiance:prefill_tokens_total counter
radiance:prefill_tokens_total{model_name="juggernaut",engine="0"} 2320091
# HELP radiance:decode_tokens_total Tokens the engine has generated, exact, counted when the step commits rather than when the request finishes.
# TYPE radiance:decode_tokens_total counter
radiance:decode_tokens_total{model_name="juggernaut",engine="0"} 423176
# HELP radiance:draft_tokens_total Speculative tokens proposed.
# TYPE radiance:draft_tokens_total counter
radiance:draft_tokens_total{model_name="juggernaut",engine="0"} 850906
# HELP radiance:draft_accepted_total Speculative tokens accepted; the pair the acceptance ratio is computed from.
# TYPE radiance:draft_accepted_total counter
radiance:draft_accepted_total{model_name="juggernaut",engine="0"} 301618
# HELP radiance:draft_refills_total Prefix hits ended a drafter window early so a windowed block drafter has context.
# TYPE radiance:draft_refills_total counter
radiance:draft_refills_total{model_name="juggernaut",engine="0"} 323
# HELP radiance:draft_refill_tokens_total Cached prompt tokens those hits gave back to be recomputed.
# TYPE radiance:draft_refill_tokens_total counter
radiance:draft_refill_tokens_total{model_name="juggernaut",engine="0"} 661504
# HELP radiance:lookup_proposals_total Prompt-lookup proposals made, with prompt lookup the only drafter.
# TYPE radiance:lookup_proposals_total counter
radiance:lookup_proposals_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:lookup_held_total Prompt-lookup proposals held back unverified: not expected to pay for their verify rows at the measured step cost.
# TYPE radiance:lookup_held_total counter
radiance:lookup_held_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:lookup_held_drafts_total Drafts in the held-back proposals.
# TYPE radiance:lookup_held_drafts_total counter
radiance:lookup_held_drafts_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:lookup_held_hits_total Of the held-back proposals scored against the text that followed, the drafts that came true: what verifying them would have accepted.
# TYPE radiance:lookup_held_hits_total counter
radiance:lookup_held_hits_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:linear_prefix_cache_hit_rate Fraction of prompt tokens served from a linear-state checkpoint (spec 7.3).
# TYPE radiance:linear_prefix_cache_hit_rate gauge
radiance:linear_prefix_cache_hit_rate{model_name="juggernaut",engine="0"} 0.95897507667541504
# HELP radiance:attn_cached_tokens_total Prompt tokens served from the paged attention prefix cache.
# TYPE radiance:attn_cached_tokens_total counter
radiance:attn_cached_tokens_total{model_name="juggernaut",engine="0"} 55357696
# HELP radiance:linear_cached_tokens_total Prompt tokens served from a linear-state checkpoint.
# TYPE radiance:linear_cached_tokens_total counter
radiance:linear_cached_tokens_total{model_name="juggernaut",engine="0"} 54233088
# HELP radiance:linear_checkpoints_written_total Linear-state checkpoints written.
# TYPE radiance:linear_checkpoints_written_total counter
radiance:linear_checkpoints_written_total{model_name="juggernaut",engine="0"} 825
# HELP radiance:prefix_cache_evictions_total Prefix-cache entries evicted (attention blocks).
# TYPE radiance:prefix_cache_evictions_total counter
radiance:prefix_cache_evictions_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:prefix_cache_ckpt_evictions_total Prefix-cache linear-state checkpoints evicted.
# TYPE radiance:prefix_cache_ckpt_evictions_total counter
radiance:prefix_cache_ckpt_evictions_total{model_name="juggernaut",engine="0"} 437
# HELP radiance:prefix_cache_expired_total Prefix-cache entries no request used for --prefix-cache-ttl, taken out.
# TYPE radiance:prefix_cache_expired_total counter
radiance:prefix_cache_expired_total{model_name="juggernaut",engine="0"} 35942
# HELP radiance:prefix_cache_ckpt_expired_total Prefix-cache linear-state checkpoints taken out by --prefix-cache-ttl.
# TYPE radiance:prefix_cache_ckpt_expired_total counter
radiance:prefix_cache_ckpt_expired_total{model_name="juggernaut",engine="0"} 33
# HELP radiance:prefix_cache_ckpt_evictions_by_reason_total Linear-state checkpoints evicted, by reason: released by --checkpoint-policy, or the least recently used of a store retention left full.
# TYPE radiance:prefix_cache_ckpt_evictions_by_reason_total counter
radiance:prefix_cache_ckpt_evictions_by_reason_total{model_name="juggernaut",engine="0",reason="retention"} 437
radiance:prefix_cache_ckpt_evictions_by_reason_total{model_name="juggernaut",engine="0",reason="store_full"} 0
# HELP radiance:prefix_cache_released_total Prefix-cache entries and linear-state snapshots a client released on purpose (DELETE /sessions).
# TYPE radiance:prefix_cache_released_total counter
radiance:prefix_cache_released_total{model_name="juggernaut",engine="0",what="entries"} 0
radiance:prefix_cache_released_total{model_name="juggernaut",engine="0",what="snapshots"} 0
# HELP radiance:prefix_cache_ckpt_refused_total Checkpoints not written because every snapshot slot was in use by a restore in the same step.
# TYPE radiance:prefix_cache_ckpt_refused_total counter
radiance:prefix_cache_ckpt_refused_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:prefix_hit_cuts_total Admissions whose prefix hit was cut, by reason: reused nothing for want of a snapshot at or below the hit, for a windowed group no window snapshot covers, or for blocks that could not be adopted; or resumed at a snapshot more than one checkpoint interval below the hit.
# TYPE radiance:prefix_hit_cuts_total counter
radiance:prefix_hit_cuts_total{model_name="juggernaut",engine="0",reason="no_snapshot"} 0
radiance:prefix_hit_cuts_total{model_name="juggernaut",engine="0",reason="windowed_group"} 0
radiance:prefix_hit_cuts_total{model_name="juggernaut",engine="0",reason="adopt_failed"} 0
radiance:prefix_hit_cuts_total{model_name="juggernaut",engine="0",reason="resumed_below"} 136
# HELP radiance:prefix_restore_cuts_total Restores from the host tier the pool could not hold whole, even after giving up the VRAM copies of other stored conversations; each came back only in part, on a model with recurrent state up to the deepest snapshot the room reaches.
# TYPE radiance:prefix_restore_cuts_total counter
radiance:prefix_restore_cuts_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:prefix_hit_cut_tokens_total Prompt tokens the attention cache held and admission computed again, because the hit resumed at a snapshot below it or reused nothing.
# TYPE radiance:prefix_hit_cut_tokens_total counter
radiance:prefix_hit_cut_tokens_total{model_name="juggernaut",engine="0"} 1124608
# HELP radiance:draft_acceptance_ratio Fraction of speculative draft tokens accepted.
# TYPE radiance:draft_acceptance_ratio gauge
radiance:draft_acceptance_ratio{model_name="juggernaut",engine="0"} 0.35446688532829285
# HELP radiance:draft_position_acceptance_ratio Conditional acceptance at each draft position, given every earlier one was taken.
# TYPE radiance:draft_position_acceptance_ratio gauge
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="0"} 0.73114069004096804
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="1"} 0.72421126063279173
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="2"} 0.73612988425386472
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="3"} 0.74895000105527532
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="4"} 0.76621766330383811
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="5"} 0.7862449429937477
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="6"} 0.79249695949106558
# HELP radiance:draft_position_reached_total Verifies that evaluated each draft position.
# TYPE radiance:draft_position_reached_total counter
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="0"} 121558
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="1"} 88876
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="2"} 64365
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="3"} 47381
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="4"} 35486
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="5"} 27190
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="6"} 21378
# HELP radiance:draft_position_accepted_total Verifies that accepted each draft position.
# TYPE radiance:draft_position_accepted_total counter
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="0"} 88876
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="1"} 64365
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="2"} 47381
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="3"} 35486
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="4"} 27190
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="5"} 21378
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="6"} 16942
# HELP radiance:num_requests_preempted Requests currently preempted and awaiting recompute.
# TYPE radiance:num_requests_preempted gauge
radiance:num_requests_preempted{model_name="juggernaut",engine="0"} 0
# HELP radiance:http_queue_depth Requests admitted by the HTTP layer and not yet finished.
# TYPE radiance:http_queue_depth gauge
radiance:http_queue_depth{model_name="juggernaut",engine="0"} 0
# HELP radiance:http_admission_rejected_total Requests refused with 429 because the admission queue was full.
# TYPE radiance:http_admission_rejected_total counter
radiance:http_admission_rejected_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:client_disconnect_total Streams cancelled because the client went away.
# TYPE radiance:client_disconnect_total counter
radiance:client_disconnect_total{model_name="juggernaut",engine="0"} 2
# HELP radiance:stream_overflow_total Requests failed because the HTTP reader could not drain the sink.
# TYPE radiance:stream_overflow_total counter
radiance:stream_overflow_total{model_name="juggernaut",engine="0"} 0
# HELP radiance:http_responses_total HTTP responses by status class.
# TYPE radiance:http_responses_total counter
radiance:http_responses_total{model_name="juggernaut",engine="0",class="2xx"} 655
radiance:http_responses_total{model_name="juggernaut",engine="0",class="4xx"} 0
radiance:http_responses_total{model_name="juggernaut",engine="0",class="5xx"} 0
tests/fixtures/models_radiance.json
Vendored
+5
View File
@@ -0,0 +1,5 @@
# Verbatim /v1/models reply from the radiance engine, recorded by
# ./scripts/capture-fixtures.sh — one JSON document. The model id and
# max_model_len are stable; created is an epoch timestamp that moves.
# The contract asserts the object/data shape — never the numbers.
{"object":"list","data":[{"id":"juggernaut","object":"model","created":1793555024,"owned_by":"radiance","max_model_len":262144,"capabilities":{"chat":true,"tools":true,"reasoning":true,"grammar":true,"embedding":false,"image":true,"video":true,"audio":false,"logprobs":true,"image_generation":false,"image_edit":false,"audio_generation":false,"video_generation":false,"decision":null,"reasoning_efforts":["none","minimal","low","medium","xhigh"]}}]}
tests/fixtures/nut_apc.vars
Vendored
+4 -3
View File
@@ -9,7 +9,7 @@ VAR apc battery.charge.low "10"
VAR apc battery.charge.warning "50"
VAR apc battery.date "2001/09/25"
VAR apc battery.mfr.date "2021/06/24"
VAR apc battery.runtime "621"
VAR apc battery.runtime "264"
VAR apc battery.runtime.low "120"
VAR apc battery.type "PbAc"
VAR apc battery.voltage "27.3"
@@ -36,13 +36,14 @@ VAR apc driver.version.usb "libusb-1.0.30 (API: 0x0100010C)"
VAR apc input.sensitivity "medium"
VAR apc input.transfer.high "144"
VAR apc input.transfer.low "88"
VAR apc input.voltage "124.0"
VAR apc input.transfer.reason "input voltage out of range"
VAR apc input.voltage "115.0"
VAR apc input.voltage.nominal "120"
VAR apc ups.beeper.status "enabled"
VAR apc ups.delay.shutdown "20"
VAR apc ups.firmware "952.e5 .D"
VAR apc ups.firmware.aux "e5 "
VAR apc ups.load "46"
VAR apc ups.load "81"
VAR apc ups.mfr "American Power Conversion"
VAR apc ups.mfr.date "2021/01/08"
VAR apc ups.model "Back-UPS RS 1500MS"
tests/fixtures/nut_cyberpower.vars
Vendored
+6 -6
View File
@@ -8,7 +8,7 @@ VAR cyberpower battery.charge "100"
VAR cyberpower battery.charge.low "10"
VAR cyberpower battery.charge.warning "20"
VAR cyberpower battery.mfr.date "CPS"
VAR cyberpower battery.runtime "3350"
VAR cyberpower battery.runtime "3525"
VAR cyberpower battery.runtime.low "300"
VAR cyberpower battery.status "100%"
VAR cyberpower battery.type "PbAcid"
@@ -36,20 +36,20 @@ VAR cyberpower driver.version.usb "libusb-1.0.30 (API: 0x0100010C)"
VAR cyberpower input.sensitivity "normal"
VAR cyberpower input.transfer.high "139"
VAR cyberpower input.transfer.low "100"
VAR cyberpower input.voltage "122.0"
VAR cyberpower input.voltage "115.0"
VAR cyberpower input.voltage.nominal "120"
VAR cyberpower output.voltage "122.0"
VAR cyberpower output.voltage "115.0"
VAR cyberpower ups.beeper.status "disabled"
VAR cyberpower ups.delay.shutdown "60"
VAR cyberpower ups.delay.start "120"
VAR cyberpower ups.firmware "CR01906AC911"
VAR cyberpower ups.load "14"
VAR cyberpower ups.load "13"
VAR cyberpower ups.mfr "CPS"
VAR cyberpower ups.model "GX1500U"
VAR cyberpower ups.power "132"
VAR cyberpower ups.power "130"
VAR cyberpower ups.power.nominal "1500"
VAR cyberpower ups.productid "0601"
VAR cyberpower ups.realpower "126"
VAR cyberpower ups.realpower "121"
VAR cyberpower ups.realpower.nominal "900"
VAR cyberpower ups.serial "SCRUBBEDSERIAL1"
VAR cyberpower ups.status "OL"
tests/fixtures/smartctl_driveripper_nvme0n1.txt
Vendored
+1 -1
View File
@@ -20,7 +20,7 @@ Namespace 1 Size/Capacity: 2,000,398,934,016 [2.00 TB]
Namespace 1 Utilization: 1,616,151,379,968 [1.61 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
Local Time is: Sat Oct 17 23:55:46 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
tests/fixtures/smartctl_driveripper_nvme1n1.txt
Vendored
+1 -1
View File
@@ -20,7 +20,7 @@ Namespace 1 Size/Capacity: 250,059,350,016 [250 GB]
Namespace 1 Utilization: 30,404,874,240 [30.4 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
Local Time is: Sat Oct 17 23:55:47 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
tests/fixtures/smartctl_driveripper_nvme2n1.txt
Vendored
+1 -1
View File
@@ -19,7 +19,7 @@ Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
Local Time is: Sat Oct 17 23:55:47 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
tests/fixtures/smartctl_driveripper_nvme3n1.txt
Vendored
+1 -1
View File
@@ -19,7 +19,7 @@ Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
Local Time is: Sat Oct 17 23:55:48 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
tests/fixtures/smartctl_driveripper_nvme4n1.txt
Vendored
+1 -1
View File
@@ -19,7 +19,7 @@ Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
Local Time is: Sat Oct 17 23:55:48 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
tests/fixtures/smartctl_driveripper_nvme5n1.txt
Vendored
+1 -1
View File
@@ -20,7 +20,7 @@ Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
Namespace 1 Utilization: 699,798,638,592 [699 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
Local Time is: Sat Oct 17 23:55:49 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
tests/fixtures/smartctl_driveripper_nvme6n1.txt
Vendored
+1 -1
View File
@@ -19,7 +19,7 @@ Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
Local Time is: Sat Oct 17 23:55:49 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
tests/fixtures/smartctl_driveripper_nvme7n1.txt
Vendored
+1 -1
View File
@@ -20,7 +20,7 @@ Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
Namespace 1 Utilization: 694,159,712,256 [694 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
Local Time is: Sat Oct 17 23:55:50 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
tests/fixtures/smartctl_driveripper_sda.txt
Vendored
+1 -1
View File
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
Device is: In smartctl database 7.5/6204
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Oct 17 23:55:50 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
tests/fixtures/smartctl_driveripper_sdb.txt
Vendored
+1 -1
View File
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
Device is: In smartctl database 7.5/6204
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Oct 17 23:55:51 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
tests/fixtures/smartctl_driveripper_sdc.txt
Vendored
+1 -1
View File
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
Device is: In smartctl database 7.5/6204
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Oct 17 23:55:52 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
tests/fixtures/smartctl_driveripper_sdd.txt
Vendored
+1 -1
View File
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
Device is: In smartctl database 7.5/6204
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Oct 17 23:55:52 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
tests/fixtures/smartctl_driveripper_sde.txt
Vendored
+1 -1
View File
@@ -17,7 +17,7 @@ Form Factor: 3.5 inches
Device is: Not in smartctl database 7.5/6204
ATA Version is: ACS-4 published, ANSI INCITS 529-2018
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Oct 17 23:55:53 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
tests/fixtures/smartctl_driveripper_sdf.txt
Vendored
+1 -1
View File
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
Device is: In smartctl database 7.5/6204
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Oct 17 23:55:53 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
tests/fixtures/smartctl_driveripper_sdg.txt
Vendored
+1 -1
View File
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
Device is: In smartctl database 7.5/6204
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Oct 17 23:55:54 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
tests/fixtures/smartctl_driveripper_sdh.txt
Vendored
+1 -1
View File
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
Device is: In smartctl database 7.5/6204
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Oct 17 23:55:54 2026 EDT
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
tests/fixtures/stats_radiance.json
Vendored
+9
View File
@@ -0,0 +1,9 @@
# Verbatim /stats reply from the radiance engine, recorded by
# ./scripts/capture-fixtures.sh — one JSON document, the engine's own
# shape. Volatile values (uptime_s, now_ms, the counters, kv_*) move
# every poll, so a diff here on re-capture is expected; the contract
# asserts the top-level key set and the cards[]/requests[]/
# draft_positions[] shape — never those numbers (the nut_*.vars
# precedent).
# in_flight was 0 at capture (requests[] has 0 row(s)).
{"model":"juggernaut","version":"1.15.1","uptime_s":15215,"started_at":1793555024,"startup_s":48,"in_flight":0,"queue_cap":32,"now_ms":1793570239000,"running":0,"waiting":0,"preempted":0,"kv_util":0.7427201867103577,"kv_tokens_used":329744,"kv_tokens_total":443968,"kv_tokens_cached":329744,"kv_bytes_used":10805051392.0,"kv_bytes_total":15901392896.0,"kv_bytes_carved":15901392896.0,"kv_tokens_max":443968,"prefix_hit_rate":0.9788609147071838,"prefix_evictions":0.0,"linear_hit_rate":0.958975076675415,"linear_cached_tokens":54233088.0,"checkpoints_written":825.0,"ckpt_evictions":{"retention":437,"store_full":0},"ckpt_refused":0,"released":{"entries":0,"snapshots":0},"hit_cuts":{"no_snapshot":0,"windowed_group":0,"adopt_failed":0,"resumed_below":136},"hit_cut_tokens":1124608.0,"prefill_tps":50.586456298828125,"decode_tps":46.6722526550293,"draft_accept":0.35446688532829285,"preemptions":0,"total_requests":603,"steps":113985,"steps_prefill":1174,"steps_mixed":388,"prefill_chunks_capped":2,"steps_decode":112423,"steps_held":0,"step_ns":5702684747399,"step_ns_decode":4295778123321,"step_ns_decode_best":32004689,"step_ns_decode_recent":41546213,"rows_unspec":0,"draft_positions":[{"reached":121558,"accepted":88876,"rate":0.731140690040968},{"reached":88876,"accepted":64365,"rate":0.7242112606327917},{"reached":64365,"accepted":47381,"rate":0.7361298842538647},{"reached":47381,"accepted":35486,"rate":0.7489500010552753},{"reached":35486,"accepted":27190,"rate":0.7662176633038381},{"reached":27190,"accepted":21378,"rate":0.7862449429937477},{"reached":21378,"accepted":16942,"rate":0.7924969594910656}],"requests":[],"cards":[{"index":0,"name":"AMD Radeon AI PRO R9700","vram_used":34159771648,"vram_total":34208743424}],"link":{"world":1,"ar_calls":0,"ar_bytes":0,"h2d_bytes":0,"d2h_bytes":0,"ssd_reads":0,"ssd_bytes":0,"ssd_staging_bytes":0,"ssd_promoted_bytes":0,"ssd_deposits":0,"ssd_deposit_bytes":0,"pool_evictions":0,"prefetched":0,"prefetch_bytes":0,"staged_bytes":0,"stream_bytes":0,"kv_stream_bytes":0,"kv_streaming":false,"kv_stream_active":false},"passes":{"played":335897,"recorded":2354,"audited":1382,"issued_unkeyed":29,"issued_refused":0,"issued_first":1546,"dispatched":85311704,"unordered":7386027}}
tests/test_api.py
+303 -12
View File
@@ -45,7 +45,9 @@ from dashboards.sources.disks import SMART_SERIES, DisksCollector, parse_disks_s
from dashboards.sources.host import HostCollector, parse_host_snapshot
from dashboards.sources.llm import (
HISTORY_METRICS,
engine_result,
parse_gufo_metrics,
parse_radiance_metrics,
parse_vllm_metrics,
)
from dashboards.store import RES_RAW, Event, Sample, SampleStore
@@ -937,6 +939,152 @@ def test_config_reports_the_window_the_llm_rates_are_actually_measured_over(
assert llm["rate_window_seconds"] == llm["interval_seconds"] == 2.0
# ---- the llm engine registry (phase 34 task 02) --------------------------- #
def test_config_serves_radiance_additively_beside_byte_identical_gufo_and_vllm(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The additive pin: `radiance` joins the llm block as a third key, and nothing about
`gufo`/`vllm` moves — same keys, same shape, same order — while an unset radiance is
present as `enabled: false` (a key that appeared only when configured would be the
two-engine world leaking back into the API). The redaction is the one rule:
`RADIANCE_API_KEY` is the documented way to authenticate, and a token written into
the URL instead is stripped before it is served (invariant 6), like the other two.
"""
with build_client(tmp_path, monkeypatch) as plain:
before = plain.get("/api/config").json()["llm"]
with build_client(tmp_path, monkeypatch,
radiance_url="http://SEKRIT-RADIANCE-TOKEN@driveripper:8000",
radiance_api_key="radiance-key") as with_radiance:
after = with_radiance.get("/api/config").json()["llm"]
assert before["gufo"] == after["gufo"]
assert before["vllm"] == after["vllm"]
assert before["radiance"] == {"enabled": False, "base_url": "", "auth": False}
assert after["radiance"] == {"enabled": True, "base_url": "http://driveripper:8000",
"auth": True}
assert "SEKRIT-RADIANCE-TOKEN" not in json.dumps(after), "the URL token must not be served"
assert list(after) == ["gufo", "vllm", "radiance", "interval_seconds", "poll_ms",
"rate_window_seconds"], "the registry's stable order is the byte order"
def test_the_llm_404_gate_answers_from_the_registry_in_all_configurations(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""One engine configured — whichever of the three — is a configured source, and the
404 detail names all three variable names so the not-configured arm says the fix
for the whole world, not just the two it used to know."""
single: list[dict[str, Any]] = [{"gufo_url": "http://deskwork:8731"},
{"vllm_url": "http://vllm-box:8000"},
{"radiance_url": "http://driveripper:8000"}]
for overrides in single:
with build_client(tmp_path, monkeypatch, **overrides) as test_client:
response = test_client.get("/api/llm/snapshot")
assert response.status_code == 200, f"only {next(iter(overrides))} configured must not 404"
with build_client(tmp_path, monkeypatch) as test_client:
response = test_client.get("/api/llm/snapshot")
assert response.status_code == 404
body = response.json()
assert body["configured"] is False
for name in ("GUFO_BASE_URL", "VLLM_BASE_URL", "RADIANCE_BASE_URL"):
assert name in body["detail"], f"the not-configured arm must name {name}"
def test_llm_history_defaults_list_the_configured_engines_from_the_registry(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The defaults are the registry's configured `key`s in registry order — `key` is the
history DB's host key samples are written under, not the display name — and an
unconfigured engine is not among them, so it is never requested and never dialled.
"""
with build_client(tmp_path, monkeypatch,
gufo_url="http://deskwork:8731",
radiance_url="http://driveripper:8000") as test_client:
body = test_client.get("/api/llm/history").json()
assert body["engines"] == ["gufo", "radiance"], "the vLLM entry is unconfigured and absent"
with build_client(tmp_path, monkeypatch) as test_client:
body = test_client.get("/api/llm/history").json()
assert body["engines"] == []
def test_summary_llm_enabled_follows_the_registry(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
with build_client(tmp_path, monkeypatch,
radiance_url="http://driveripper:8000") as test_client:
assert test_client.get("/api/summary").json()["llm"]["enabled"] is True
def test_an_unconfigured_radiance_never_becomes_a_job_that_could_look_broken(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Invariant 3's shape, the `disks` precedent: no configured URL, no `llm:radiance`
job at all — so nothing dials a hostname the operator never installed, and no health
record exists to turn red. The job appears exactly when the URL does.
"""
with build_client(tmp_path, monkeypatch) as test_client:
keys = {row["key"] for row in test_client.get("/api/health").json()["sources"]}
assert not any(key.startswith("llm:") for key in keys)
with build_client(tmp_path, monkeypatch,
radiance_url="http://driveripper:8000") as test_client:
keys = {row["key"] for row in test_client.get("/api/health").json()["sources"]}
assert "llm:radiance" in keys
def _radiance_stats_doc() -> dict[str, Any]:
"""The verbatim /stats capture, its `#` header comment lines stripped."""
path = Path(__file__).resolve().parents[1] / "tests" / "fixtures" / "stats_radiance.json"
if not path.exists():
pytest.skip(f"missing fixture {path.name} — run scripts/capture-fixtures.sh --with-llm")
body = "\n".join(line for line in path.read_text().splitlines() if not line.startswith("#"))
return json.loads(body)
def test_radiance_payload_reaches_the_page_rows_and_rest(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, metrics_text) -> None:
"""The render path's input as the browser would receive it: the fixture /stats, through
the real `engine_result`, into the `llm:radiance` job — and `/api/llm/snapshot` hands
it to the page. At rest (the capture's own empty `requests[]`): the counts-otherwise
sentence, the whitelisted stats block, and `queue_cap` from the sidecar. A request in
flight: the rows verbatim under `sessions.list`. A dead sidecar: the metrics gauges
with their provenance said, and the cap `null`, never `0` (rule 2)."""
stats = _radiance_stats_doc()
rows = [{"id": 42, "model": "juggernaut", "state": "running",
"prompt_tokens": 1_200, "completed_tokens": 250}]
with build_client(tmp_path, monkeypatch, radiance_url=DEAD_URL) as test_client:
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
job = hub.jobs["llm:radiance"]
job.state.payload = engine_result("radiance", "http://r:8000",
metrics_text("radiance"), RateEstimator(),
stats=stats, now=1.0).payload
rest = next(p for p in test_client.get("/api/llm/snapshot").json()["engines"]
if p["engine"] == "radiance")
assert rest["sessions"]["list"] == stats["requests"]
if not stats["requests"]:
assert "counts, not sessions" in rest["sessions"]["detail"]
assert rest["stats"]["version"] == "1.15.1"
assert isinstance(rest["stats"]["uptime_s"], int) and rest["stats"]["uptime_s"] >= 0, \
"uptime is the engine's own seconds — volatile between captures"
assert rest["stats"]["cards"][0]["vram_used"] == stats["cards"][0]["vram_used"]
assert rest["metrics"]["queue_cap"]["value"] == stats["queue_cap"]
assert len(rest["metrics"]["spec_positions"]) == 7
busy = {**stats, "requests": rows, "running": 1}
job.state.payload = engine_result("radiance", "http://r:8000",
metrics_text("radiance"), RateEstimator(),
stats=busy, now=1.0).payload
again = next(p for p in test_client.get("/api/llm/snapshot").json()["engines"]
if p["engine"] == "radiance")
assert again["sessions"]["list"] == rows, "the rows are not the verbatim passthrough"
assert again["sessions"]["source"] == "radiance /stats"
assert again["sessions"]["detail"] is None
job.state.payload = engine_result("radiance", "http://r:8000",
metrics_text("radiance"), RateEstimator(),
now=1.0).payload # stats=None: dead sidecar
dead = next(p for p in test_client.get("/api/llm/snapshot").json()["engines"]
if p["engine"] == "radiance")
assert dead["sessions"]["queue_cap"] is None, "null, not 0 — rule 2"
assert "gauges" in dead["sessions"]["detail"]
assert "stats" not in dead, "no stats block when the sidecar never answered"
assert dead["metrics"]["queue_cap"]["value"] is None
def wait_for_state(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
deadline = time.time() + timeout
while time.time() < deadline:
@@ -2995,8 +3143,11 @@ def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client)
assert name in js, f"{name} no longer feeds a foot line"
assert "p95 n/a" in js
assert 'data-foot' in js
# engine-scoped rows: gufo never grows a TTFT row, vLLM never a spec-accept row
assert "only: ['vllm']" in js and "only: ['gufo']" in js
# engine-scoped rows: gufo never grows a TTFT row, radiance joins the vLLM latency
# scope (the vLLM-shaped histograms this build carries, task 03) and spec acceptance
# (the gauge reuses gufo's key, the same 0-1 scale)
assert "only: ['vllm', 'radiance']" in js and "only: ['gufo', 'radiance']" in js
assert "only: ['gufo']" in js, "slots busy left its gufo scope"
scoped = js[js.index("function sparkCellsFor("):js.index("function enginePanel(")]
assert "cell.only" in scoped
assert "minSpan: cell.minSpan" in sparks, "cell minSpan never reaches the sparkline"
@@ -3061,7 +3212,8 @@ def test_llm_alarm_cells_exist_only_while_there_is_an_incident(client) -> None:
# publishes, and no row is printed in the strip *and* in the panel under it.
#: row key -> the cells `cellValue()` resolves it from (`sessions.*` is the gufo slot count,
#: which is not a metrics cell). The served `STRIP_ROWS` table must match these keys exactly.
#: `stats.*` radiance's /stats subset — neither is a metrics cell). The served `STRIP_ROWS`
#: table must match these keys exactly.
STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
"prompt": ("prompt_tokens_per_second",),
"decode": ("decode_tokens_per_second",),
@@ -3072,6 +3224,10 @@ STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
"cache_hit": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"),
"spec_accept": ("spec_accept_rate",),
"queue_mean": ("request_queue_time_seconds__mean",),
"cap": ("queue_cap",),
"build": ("stats.version",),
"uptime": ("stats.uptime_s",),
"vram": ("stats.cards",),
"preemptions": ("num_preemptions_total",),
"device_lost": ("device_lost_total",),
}
@@ -3079,6 +3235,13 @@ STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
#: rows that are incidents, not stats: absent where the engine does not count them.
ALARM_ROW_KEYS = ("preemptions", "device_lost")
#: (row key, engine) pairs the scope keeps even though **this capture** does not publish
#: the cell: the series is standard for the engine family, so another build fills it.
#: `queue_mean` on radiance — this build publishes no `request_queue_time_seconds` (the
#: vLLM-compat family is pinned in RADIANCE_OPTIONAL), so the row prints `n/a` by
#: absence, never back-filled. A row the engine could *never* fill is not shown at all.
DOCUMENTED_ABSENCE = {("queue_mean", "radiance")}
def _strip_rows(js: str) -> dict[str, str | None]:
"""The served `STRIP_ROWS` table: row key -> its `only` scope (`None` = every engine)."""
@@ -3150,10 +3313,13 @@ def test_llm_strip_card_is_one_card_per_engine_of_shared_vocabulary(client) -> N
def test_llm_strip_scopes_its_rows_to_the_engines_that_publish_them(client) -> None:
"""gufo gets slots and speculative decoding; vLLM gets running and queue time."""
"""gufo gets slots; vLLM and radiance get running and queue time; the radiance
strip's four own cells (cap / build / uptime / VRAM) are radiance's alone."""
rows = _strip_rows(client.get("/static/page-llm.js").text)
assert rows["slots_busy"] == "'gufo'" and rows["spec_accept"] == "'gufo'"
assert rows["running"] == "'vllm'" and rows["queue_mean"] == "'vllm'"
assert rows["slots_busy"] == "'gufo'" and rows["spec_accept"] == "'gufo', 'radiance'"
assert rows["running"] == "'vllm', 'radiance'" and rows["queue_mean"] == "'vllm', 'radiance'"
for key in ("cap", "build", "uptime", "vram"):
assert rows[key] == "'radiance'", f"the {key} row left its radiance scope"
assert rows["waiting"] is None and rows["cache_hit"] is None and rows["tokens_total"] is None
js = client.get("/static/page-llm.js").text
scoped = js[js.index("function stripRowsFor("):js.index("function stripCard(")]
@@ -3166,19 +3332,24 @@ def test_llm_strip_rows_name_cells_the_engines_publish(client, metrics_text) ->
`STRIP_ROW_PATHS` mirrors the JS table; every metric cell is checked against the
capture-parsed metrics of each engine that would render that row, so a rename in
`sources/llm.py` fails here instead of printing a permanent `n/a` in the browser, and a
row scoped to one engine cannot leak into the other's card.
row scoped to one engine cannot leak into the other's card. `DOCUMENTED_ABSENCE` is the
one exemption: a series standard for the engine family that this build's capture does
not publish (the row prints `n/a` by absence, never back-filled — the scope stays for
the builds that do).
"""
rows = _strip_rows(client.get("/static/page-llm.js").text)
assert set(rows) == set(STRIP_ROW_PATHS), \
f"the served strip rows drifted: {set(rows) ^ set(STRIP_ROW_PATHS)}"
parsed = {"gufo": parse_gufo_metrics(metrics_text("gufo")),
"vllm": parse_vllm_metrics(metrics_text("vllm"))}
"vllm": parse_vllm_metrics(metrics_text("vllm")),
"radiance": parse_radiance_metrics(metrics_text("radiance"))}
for key, only in rows.items():
paths = STRIP_ROW_PATHS[key]
if paths[0].startswith("sessions."):
if paths[0].startswith("sessions.") or paths[0].startswith("stats."):
continue # its own payload object; pinned on the JS below
shown = {engine: metrics for engine, metrics in parsed.items()
if not only or engine in only}
if (not only or engine in only)
and (key, engine) not in DOCUMENTED_ABSENCE}
present = [engine for engine in shown if any(path in shown[engine] for path in paths)]
if key in ALARM_ROW_KEYS:
# an alarm the engine does not count simply is not there: the chip renders only
@@ -3743,14 +3914,19 @@ def test_llm_vllm_panel_says_there_is_no_per_slot_json(client) -> None:
`sources/llm.py`'s `_sessions()` publishes counts for vLLM and says so in `detail`; the
panel turns that into the `llm_monitor` line, with the API's sentence as the tooltip. The
gate is the engine's name (`SLOT_ENGINES`), not `sessions.list.length`, so an engine that
has not answered yet still shows its label and the honest empty sentence.
has not answered yet still shows its label and the honest empty sentence. And the line
stays vLLM's: radiance's panel slot section is empty (its per-request rows live in the
strip), so a sentence naming vLLM on the radiance card is a line the engine never said.
"""
js = client.get("/static/page-llm.js").text
markup = _slot_markup(js)
branch = markup.split("if (SLOT_ENGINES.indexOf(engine) < 0) {")[1].split("}")[0]
branch = markup.split("if (engine === 'vllm') {")[1].split("}")[0]
assert 'class="slots-note" data-slots-note' in branch, "the vLLM line lost its element"
assert "NO_SLOT_JSON" in branch and "vLLM exposes no per-slot JSON" in js
assert '<div class="slots"' not in branch, "the vLLM panel grew a grid it cannot fill"
radiance_branch = markup.split("if (SLOT_ENGINES.indexOf(engine) < 0) {")[1].split("}")[0]
assert "return ''" in radiance_branch, "the radiance panel re-grew a slot section"
assert "NO_SLOT_JSON" not in radiance_branch, "the vLLM sentence leaked onto the radiance panel"
body = _slot_js(js)
assert body.startswith("function slotValue("), "the grid's code path moved"
assert "if (!grid) {" in body, "renderSlotGrid no longer leaves a panel with no grid alone"
@@ -3815,6 +3991,80 @@ def test_llm_charts_are_secondary_and_named(client) -> None:
assert "height: 110" in js and "height: 165" not in js
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
assert charts.count("ariaLabel:") == 4, "an /llm chart lost its accessible name"
# the radiance strip's 7-bar spec curve (phase 34) joins the named list: `role="img"`
# and an `aria-label` computed from the payload's `spec_positions` — a label that
# cannot say which ratios it plots is not an accessible name, and an absent label
# fails here
curve = _code(client.get("/static/page-llm.js").text)
curve = curve[curve.index("function renderSpecCurve("):curve.index("function reqCellText(")]
assert 'role="img"' in curve, "the spec curve is an unlabelled plot"
assert 'aria-label="' in curve and "escapeHtml(label)" in curve, \
"the spec curve's label is not the computed one"
def test_the_radiance_spec_curve_is_neutral_and_built_from_the_payload(client) -> None:
"""Decision 4, pinned both ways: the 7-bar curve draws in the page's neutral ink,
reaches for no verdict token, and its label is computed from the payload — the
connector word turns on the first/last published ratio's order, and an absent
position stays absent rather than becoming a zero bar (rule 2)."""
js = _code(client.get("/static/page-llm.js").text)
curve = js[js.index("function renderSpecCurve("):js.index("function reqCellText(")]
assert "spec_positions" in curve, "the curve stopped reading the payload's positions"
assert "cssVar('--text-faint')" in curve, "the curve left the neutral ink"
assert "Math.min(1, row.ratio)" in curve, "a ratio above 1 would stretch a bar"
for token in ("--ok", "--warn", "--fail", "severity"):
assert token not in curve, f"the spec curve grew a verdict: {token}"
assert "falling to" in curve and "rising to" in curve and "holding at" in curve, \
"the connector word is not computed from the payload's order"
assert "Number.isFinite(row.ratio)" in curve, \
"a missing position must stay absent, not zeroed"
def test_radiance_strip_cells_print_na_never_zero_when_stats_never_answered(client) -> None:
"""A radiance payload without the /stats block: cap / build / uptime / vram print
`n/a`, never `0` — the null guards on the served path (a missing sidecar is an
absent number, rule 2), and the number formatter never sees a string value."""
js = _code(client.get("/static/page-llm.js").text)
value = js[js.index("function cellValue("):js.index("function p95Value(")]
assert "case 'cap':" in value and "get('queue_cap')" in value, \
"cap stopped reading the /stats passthrough the server degrades to null"
assert "stats && typeof stats.version === 'string'" in value, \
"build must be the engine's own string or null"
assert "Number.isFinite(stats.uptime_s)" in value and "duration(stats.uptime_s)" in value
assert "cards.find((c) => c && c.index === 0)" in value, "vram reads the wrong card"
assert "(used / GIB).toFixed(1)" in value and "2 ** 30" in js, \
"the one conversion is GiB = 2³⁰, labelled in the tip"
body = _strip_render_path(js)
assert "cellValue(engine, metrics, sessions, row.key, stats)" in body, \
"the radiance cells lost the payload's stats block"
assert "typeof value.value === 'string' ? value.value" in body, \
"a string value hits the number formatter"
def test_radiance_request_rows_render_as_is_and_the_rest_state_says_so(client) -> None:
"""The rows are the engine's own fields, as-is: the columns are the union of the
keys the rows publish (no invented key), a number prints only when it is a number,
anything else its string verbatim — and an empty list hides the table and puts the
API's own count-otherwise sentence in the note slot."""
js = _code(client.get("/static/page-llm.js").text)
card = js[js.index("function stripCard("):js.index("function renderStripCard(")]
assert 'data-spec-curve' in card and 'data-reqs' in card and 'data-reqs-note' in card, \
"the radiance card lost its curve / rows / note homes"
cell = js[js.index("function reqCellText("):js.index("function renderReqs(")]
assert "typeof value === 'number' && isFinite(value)" in cell, \
"a number prints only when it is a number"
assert "typeof value === 'string') return value" in cell, \
"anything else prints its string verbatim"
reqs = js[js.index("function renderReqs("):js.index("function makeCharts(")]
assert "sessions.list" in reqs and "Object.keys(row)" in reqs, \
"the columns are a hardcoded list again"
assert "indexOf(key) < 0" in reqs, "first-seen order must keep the rows as published"
assert "note.hidden = !sessions.detail" in reqs, \
"the rest state lost the API's own sentence"
assert "note.hidden = true" in reqs, "rows visible: the sentence must leave"
assert "table.title = sessions.source" in reqs, "the table's source left its tooltip"
# the gufo gate is untouched: the slot grid is gufo's, the radiance rows live in the strip
assert "SLOT_ENGINES = ['gufo']" in js
def test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling(client) -> None:
@@ -4006,6 +4256,9 @@ def test_the_copy_the_operator_acts_on_survives_the_audit(client) -> None:
assert "smartmontools" in pages["/disks"] + scripts["page-disks.js"]
assert "disk group" in scripts["page-disks.js"]
assert "No engines configured" in pages["/llm"]
# the not-configured arm is actionable only if it names the variable for every engine
for name in ("GUFO_BASE_URL", "VLLM_BASE_URL", "RADIANCE_BASE_URL"):
assert name in pages["/llm"], f"/llm's empty state stopped naming {name}"
assert "No UPS reporting" in pages["/ups"]
assert "instcmds" in pages["/ups"], "the UPS setup note stopped naming the grant that matters"
assert "3493" in pages["/ups"], "the page stopped warning that upsd's port is not always 3493"
@@ -10087,6 +10340,44 @@ def test_a_card_disclosure_scrolls_what_it_reveals_at_every_width(client) -> Non
"at 1280 px, where it would then do nothing")
def test_the_radiance_reqs_table_scrolls_instead_of_stretching_its_card(client) -> None:
"""The radiance strip's request table must not grow the card to its min-content.
Measured on the live fleet (1280 px, two strip cards across, each 613 px): with a
request in flight the `.reqs` table — the union of the row's ten published keys —
sat at ~950 px of min-content, and because the strip body is a one-column grid
(`.strip > .card > .body`) with an implicit auto column, a grid item's automatic
minimum size is its content's min-content width, so the *column* grew to ~946 px.
Every row of the card laid out at 946 px inside a 613 px card: the value column of
each `dl.kv` row resolved to ~857 px, the right-aligned value span painted at
x≈1582 against a card edge at 1260, and `.card { overflow: hidden }` ate it — the
strip printed its labels with no values while the request row and the spec curve
(their own boxes) painted fine. `min-width: 0` is the first half: without it the
table's min-content still sets the column and nothing below can overflow. `overflow-x:
auto` is the second: the table reaches its last column by scrolling inside the card,
only where it is wider than the card (a table that fits paints no scrollbar —
horizontal only, no height cap, no vertical scrollport).
"""
reqs = _declarations(_css_section(_css(client), "engine slot grid"), ".reqs")
assert reqs.get("min-width") == "0", (
"the `.reqs` item may not be narrower than its table: without this the ten-column "
"request table's min-content grows the strip body's grid column and every value "
"cell of the card paints past the card, clipped by `.card { overflow: hidden }`")
assert reqs.get("overflow-x") == "auto", (
"the request table lost its sideways scroll inside the card — the columns beyond "
"the card edge are unreachable again")
for forbidden in ("max-height", "overflow-y", "overflow"):
assert forbidden not in reqs, (
f"the `.reqs` rule declares `{forbidden}`: this is a sideways scroll only — a "
"height cap or vertical scrollport here would hide rows the operator has not "
"chosen to hide")
# the strip body is the grid the item sits in: a one-column body is what the
# min-content growth acts on, so the fix is inert if the body stops being a grid
body = _declarations(_css_section(_css(client), "engine stats strip"), ".strip > .card > .body")
assert body.get("display") == "grid", (
"the strip body is not a one-column grid anymore — re-measure before touching `.reqs`")
def test_every_table_on_every_page_carries_the_class_the_rule_targets(client) -> None:
"""The mobile rule is class-level, so a `<table>` without `data` would silently not scroll.
tests/test_collectors.py
+226 -1
View File
@@ -20,6 +20,7 @@ import httpx
import pytest
from dashboards import certs
from dashboards.app import build_hub
from dashboards.collectors import (
AgentTls,
NutMonitor,
@@ -33,10 +34,12 @@ from dashboards.collectors import (
fetch_snapshot,
gateway_job,
host_jobs,
llm_jobs,
post_route,
)
from dashboards.config import Settings, Target
from dashboards.poller import Result
from dashboards.prom import Prom
from dashboards.rates import RateEstimator
from dashboards.sources import nut
from dashboards.sources.disks import DisksCollector
@@ -1121,7 +1124,10 @@ async def test_first_gufo_poll_has_no_measured_rate_yet() -> None:
decode = result.payload["metrics"]["decode_tokens_per_second"]
assert decode["value"] is None
assert decode["source"] is None
assert result.payload["sessions"]["active"] == 0.0
# `active` is the /metrics gauge, verbatim the fixture's own number — the capture
# may have been taken with a request in flight (volatile between re-captures)
assert result.payload["sessions"]["active"] == \
Prom(GUFO).scalar(["llamacpp:requests_processing"])
assert "no /slots answer" in result.payload["sessions"]["detail"]
@@ -1204,6 +1210,225 @@ async def test_engine_timeout_is_504() -> None:
assert excinfo.value.status == 504
# --------------------------------------------------------------------------- #
# radiance: the /stats sidecar (phase 34, task 04)
# --------------------------------------------------------------------------- #
RADIANCE = (FIXTURES / "metrics_radiance.txt").read_text()
def _radiance_stats() -> dict[str, Any]:
"""The capture's /stats document (the `#` header lines are the script's, not JSON)."""
body = "\n".join(line for line in
(FIXTURES / "stats_radiance.json").read_text().splitlines()
if not line.startswith("#"))
return json.loads(body)
def fake_engine(metrics: httpx.Response | BaseException,
stats: httpx.Response | BaseException | None = None
) -> tuple[httpx.AsyncClient, list[str]]:
"""One fake engine: a scripted `/metrics` answer, and a scripted `/stats` sidecar answer
if the job dials for it; every requested path is kept, so a retry or an alias shows up.
"""
seen: list[str] = []
def handler(request: httpx.Request) -> httpx.Response:
seen.append(request.url.path)
if request.url.path == "/metrics":
if isinstance(metrics, BaseException):
raise metrics
return metrics
if request.url.path == "/stats":
assert stats is not None, "the job dialled /stats without one scripted"
if isinstance(stats, BaseException):
raise stats
return stats
return httpx.Response(404)
return httpx.AsyncClient(transport=httpx.MockTransport(handler)), seen
async def test_a_configured_radiance_alone_builds_exactly_one_llm_job() -> None:
"""Task 02's registry shape, extended with the job's own fields: a configured
`RADIANCE_BASE_URL` (gufo and vLLM unset) is exactly one job, with the LLM dashboard,
the registry's display name and the LLM cadence — the same three facts `llm:gufo` and
`llm:vllm` already carried."""
settings = Settings(radiance_url="http://radiance.test:8000")
async with httpx.AsyncClient() as client:
jobs = llm_jobs(settings, client, RateEstimator())
assert len(jobs) == 1
job = jobs[0]
assert job.key == "llm:radiance"
assert job.dashboard == "llm"
assert job.label == "radiance"
assert job.interval == settings.llm_interval, "the LLM cadence, not the host's"
async def test_the_hub_wires_the_radiance_job_with_the_llm_cadence_and_estimator(
store: Any, monkeypatch: pytest.MonkeyPatch) -> None:
"""`app.py` needed no change — this is the check, asserted by running the job through
its own `fn` rather than by comment: the `llm:radiance` job flows into the hub from
`llm_jobs` with the LLM interval and the LLM-span estimator (one estimator per
measuring cadence, phase 23), and the sidecar flag reaches `fetch_engine` from the
job's call alone."""
seen: dict[str, Any] = {}
async def spy_fetch(client: Any, engine: str, base_url: str, api_key: str,
rates: RateEstimator, **kwargs: Any) -> Result:
seen["engine"] = engine
seen["rates"] = rates
seen["kwargs"] = kwargs
return Result(payload={"engine": engine})
monkeypatch.setattr("dashboards.collectors.fetch_engine", spy_fetch)
settings = Settings(radiance_url="http://radiance.test:8000")
async with httpx.AsyncClient() as client:
hub, _ = build_hub(settings, store, client)
assert "llm:radiance" in hub.jobs
job = hub.jobs["llm:radiance"]
assert job.interval == settings.llm_interval
assert await job.run_once(store) is True
assert seen["engine"] == "radiance"
assert seen["rates"].span == settings.llm_interval, "the LLM-span estimator"
assert seen["kwargs"] == {"with_slots": False, "with_stats": True}
@pytest.mark.parametrize("settings, expected", [
(Settings(gufo_url="http://gufo.test:8731"), ["/metrics", "/slots"]),
(Settings(vllm_url="http://vllm.test:8000"), ["/metrics"]),
(Settings(radiance_url="http://radiance.test:8000"), ["/metrics", "/stats"]),
])
async def test_each_engine_job_dials_only_its_own_sidecars(
settings: Settings, expected: list[str]) -> None:
"""The grep criterion, proven by behaviour: `llm_jobs`' flag is what makes
`fetch_engine` dial `/stats` — the gufo job's `/slots` pass and the vLLM job's bare
`/metrics` read behave exactly as before (gufo's alias aside, a 200 `/slots` answers
before `/v1/slots` is tried), and radiance is the only engine that dials the sidecar,
exactly once and on its unaliased path."""
text_by_host = {"gufo.test": GUFO, "vllm.test": VLLM, "radiance.test": RADIANCE}
paths: list[str] = []
def handler(request: httpx.Request) -> httpx.Response:
paths.append(request.url.path)
if request.url.path == "/metrics":
return httpx.Response(200, text=text_by_host[request.url.host])
if request.url.path == "/slots":
return httpx.Response(200, json=[{"index": 0}])
if request.url.path == "/stats":
return httpx.Response(200, json=_radiance_stats())
return httpx.Response(404)
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
jobs = llm_jobs(settings, client, RateEstimator())
assert len(jobs) == 1
result = await jobs[0].fn()
assert isinstance(result, Result)
assert paths == expected, f"{jobs[0].key}: the job dials only its own sidecars"
@pytest.mark.parametrize("stats_failure", ["http-500", "non-json", "json-list", "connection-error"])
async def test_a_dead_or_partial_stats_sidecar_keeps_the_metrics_poll_healthy(
stats_failure: str) -> None:
"""Invariant 4, the sidecar's own arm: a `/stats` that is dead, 500, non-JSON, or a
JSON that is not a dict degrades the session arm (task 03's missing-stats fallback —
the counts from the /metrics gauges, the cap n/a, never 0) and never takes the
`/metrics` poll down: one best-effort read, no retry, no second dial, no alias, no
`UpstreamError`."""
if stats_failure == "http-500":
stats: Any = httpx.Response(500, text="sidecar down")
elif stats_failure == "non-json":
stats = httpx.Response(200, text="not json at all")
elif stats_failure == "json-list":
stats = httpx.Response(200, json=[1, 2])
else:
stats = httpx.ConnectError("sidecar down")
client, seen = fake_engine(httpx.Response(200, text=RADIANCE), stats)
async with client:
result = await fetch_engine(client, "radiance", "http://radiance.test:8000",
"", RateEstimator(), with_stats=True)
assert result.payload["engine"] == "radiance"
assert result.payload["metrics"]["num_requests_running"]["value"] == \
Prom(RADIANCE).scalar(["vllm:num_requests_running"]), \
"the /metrics answer is the poll's one truth, verbatim the fixture's gauge"
assert "stats" not in result.payload, "an absent or non-dict sidecar carries nothing"
detail = result.payload["sessions"]["detail"]
assert "did not answer" in detail, "the session arm names the missing sidecar"
assert "gauges" in detail, "…and the fallback counts' different provenance"
assert result.payload["sessions"]["queue_cap"] is None, "n/a, never 0 (rule 2)"
assert [path for path in seen if path == "/stats"] == ["/stats"], (
"one best-effort read, no retry, no second dial")
async def test_a_healthy_stats_sidecar_travels_into_the_result() -> None:
client, seen = fake_engine(httpx.Response(200, text=RADIANCE),
httpx.Response(200, json=_radiance_stats()))
async with client:
result = await fetch_engine(client, "radiance", "http://radiance.test:8000",
"", RateEstimator(), with_stats=True)
block = result.payload["stats"]
assert block["model"] == "juggernaut"
for key in ("version", "uptime_s", "startup_s", "total_requests", "cards"):
assert key in block, "the whitelisted subset the page reads (task 03)"
assert block["cards"][0]["name"] == "AMD Radeon AI PRO R9700"
assert "vram_used_pct" in block["cards"][0], "the one computed number in the block"
sessions = result.payload["sessions"]
assert sessions["source"] == "radiance /stats"
assert sessions["queue_cap"] == _radiance_stats()["queue_cap"]
if not _radiance_stats()["requests"]:
assert "counts, not sessions" in sessions["detail"], "the capture is at rest"
assert seen == ["/metrics", "/stats"], "one unaliased read, nothing else"
async def test_a_dead_metrics_is_the_upstream_error_and_the_sidecar_never_masks_it() -> None:
"""A 500 `/metrics` beside a perfect `/stats`: the failure is the metrics one, and
the sidecar is never even consulted — consulted first it could mask the real failure,
consulted after it would only be wasted work on a poll that is already failed."""
client, seen = fake_engine(httpx.Response(500, text="boom"),
httpx.Response(200, json=_radiance_stats()))
async with client:
with pytest.raises(UpstreamError) as excinfo:
await fetch_engine(client, "radiance", "http://radiance.test:8000",
"", RateEstimator(), with_stats=True)
assert "HTTP 500" in excinfo.value.message
assert "/metrics" in excinfo.value.message
assert "/stats" not in seen, "a failed metrics dial never reaches for the sidecar"
async def test_a_failing_radiance_never_echoes_credentials_in_its_url() -> None:
"""Invariant 6 for the radiance dial, the engine-fetch precedent extended: a
`RADIANCE_BASE_URL` written with a userinfo token fails with the redacted URL, and
the key-free form leaks nothing either — there is nothing to leak."""
client, _ = fake_engine(httpx.ConnectError("connection refused"))
async with client:
with pytest.raises(UpstreamError, match="unreachable") as excinfo:
await fetch_engine(client, "radiance",
"http://SEKRIT-RADIANCE-TOKEN@radiance.test:8000", "",
RateEstimator(), with_stats=True)
assert "SEKRIT-RADIANCE-TOKEN" not in excinfo.value.message
assert "http://radiance.test:8000/metrics" in excinfo.value.message
async def test_the_job_list_never_dials_an_unconfigured_engine(
monkeypatch: pytest.MonkeyPatch) -> None:
"""The registry's gate, spy-pinned at the dial layer (the existing spy pattern): with
every engine unset, `llm_jobs` builds nothing, and nothing that was built dials
anything — an unconfigured engine has no job to fail and no dial to hang (invariant 3)."""
calls: list[str] = []
async def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
calls.append(str(url))
raise AssertionError(f"an unconfigured engine was dialled at {url}")
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
settings = Settings()
async with httpx.AsyncClient() as client:
jobs = llm_jobs(settings, client, RateEstimator())
for job in jobs:
await job.fn()
assert jobs == []
assert calls == []
# --------------------------------------------------------------------------- #
# NUT monitor
# --------------------------------------------------------------------------- #
tests/test_config.py
+92
View File
@@ -6,11 +6,14 @@ import os
from dataclasses import FrozenInstanceError, fields
from typing import Any
import httpx
import pytest
from dashboards import config
from dashboards.auth import PBKDF2_ITERATIONS
from dashboards.collectors import llm_jobs
from dashboards.config import Settings, parse_targets
from dashboards.rates import RateEstimator
def test_parses_whitespace_and_comma_separated_targets() -> None:
@@ -397,3 +400,92 @@ def test_the_pbkdf2_knob_is_for_tests_and_not_a_performance_tune(
assert config._int("DASH_PBKDF2_ITERATIONS", 600_000) == 600_000
monkeypatch.setenv("DASH_PBKDF2_ITERATIONS", "1000")
assert config._int("DASH_PBKDF2_ITERATIONS", 600_000) == 1000
# ---- the llm engine registry (phase 34 task 02) --------------------------- #
def test_radiance_defaults_to_unset_the_way_the_other_engines_do() -> None:
"""Unset means "this deployment does not run it" (invariant 3), like `gufo_url`.
It must be invisible to `any_source_enabled` too: a deployment that runs no source at
all is `empty`, not broken, and an unset engine is the half of the page that
disappears, never a red panel.
"""
settings = Settings()
assert settings.radiance_url == ""
assert settings.radiance_api_key == ""
assert settings.any_source_enabled is False
def test_radiance_settings_come_from_the_environment(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("RADIANCE_BASE_URL", " http://driveripper.example.net:8000/ ")
monkeypatch.setenv("RADIANCE_API_KEY", " engine-key ")
reloaded = importlib.reload(config)
try:
assert reloaded.SETTINGS.radiance_url == "http://driveripper.example.net:8000", "`_base_url` strips whitespace and the trailing slash"
assert reloaded.SETTINGS.radiance_api_key == "engine-key", "_env() strips"
assert reloaded.SETTINGS.any_source_enabled is True
finally:
for name in ("RADIANCE_BASE_URL", "RADIANCE_API_KEY"):
monkeypatch.delenv(name)
importlib.reload(config)
def test_llm_engines_is_the_one_registry_in_its_stable_three_engine_order() -> None:
"""The order `gufo`, `vllm`, `radiance` is the contract: the UI's card order and the
history defaults follow it, so a reordered registry is a page change, not a list edit.
`key` is the history DB's host key and the job id; `display` is what a health record
and a page print; the api key travels verbatim into the dial's `Authorization` header.
"""
settings = Settings(gufo_url="http://gufo.example.net:8731", gufo_api_key="gufo-key",
vllm_url="http://vllm.example.net:8000", vllm_api_key="vllm-key",
radiance_url="http://driveripper.example.net:8000",
radiance_api_key="radiance-key")
assert settings.llm_engines() == [
("gufo", "gufo", "http://gufo.example.net:8731", "gufo-key"),
("vllm", "vLLM", "http://vllm.example.net:8000", "vllm-key"),
("radiance", "radiance", "http://driveripper.example.net:8000", "radiance-key"),
]
def test_llm_engines_lists_an_unset_engine_with_an_empty_url() -> None:
"""The registry is the full fleet the way `parse_targets` keeps the host list: an
unset engine is *listed*, with an empty `url` — configured-ness is the call site's
`bool(url)` test (the `nut_enabled` shape), never a missing entry, so a site that
iterates the list cannot silently read a two-engine world.
"""
entries = Settings().llm_engines()
assert [key for key, _display, _url, _api_key in entries] == ["gufo", "vllm", "radiance"]
assert [display for _key, display, _url, _api_key in entries] == ["gufo", "vLLM", "radiance"]
assert [url for _key, _display, url, _api_key in entries] == ["", "", ""]
def _llm_job_keys(settings: Settings) -> list[str]:
"""The job ids `llm_jobs` would build, from the registry, for one settings object.
The client is never touched by the construction (the jobs close over it for their
own poll loops), so a bare, never-entered `AsyncClient` is the right fake — and an
unentered one warns nothing and needs no closing.
"""
client = httpx.AsyncClient()
return [job.key for job in llm_jobs(settings, client, RateEstimator())]
def test_radiance_unset_gets_no_job_and_changes_nothing_about_the_source() -> None:
"""No job, and `any_source_enabled` is unchanged by it: an unconfigured engine is never
dialled and never shows a red record (invariant 3)."""
settings = Settings()
assert settings.any_source_enabled is False
assert _llm_job_keys(settings) == []
def test_radiance_set_gets_its_job_and_enables_the_source() -> None:
"""One `llm:radiance` job, in the registry's position, and the same job set a three-engine
deployment already had before the third engine existed (gufo/vllm byte-identical)."""
settings = Settings(radiance_url="http://driveripper.example.net:8000")
assert settings.any_source_enabled is True
assert _llm_job_keys(settings) == ["llm:radiance"]
settings = Settings(gufo_url="http://gufo.example.net:8731",
vllm_url="http://vllm.example.net:8000",
radiance_url="http://driveripper.example.net:8000")
assert _llm_job_keys(settings) == ["llm:gufo", "llm:vllm", "llm:radiance"]
tests/test_deploy_scripts.py
+210
View File
@@ -3362,3 +3362,213 @@ def test_the_install_checks_the_agent_can_reach_the_socket(tmp_path: Path) -> No
assert result.returncode == 1, "an unreachable socket is a failed install, not a silent success"
assert "cannot connect" in result.stderr
assert "sudo -u dashboards-agent python3" in call_log.read_text(), "the check ran as the agent uid"
# --- scripts/capture-fixtures.sh: the radiance pass (phase 34, task 01) --------------
#
# The script learns radiance through the same `--with-llm` gate as the other engines and
# carries no deployment detail of its own: RADIANCE_BASE_URL is the dial, `/metrics`
# joins the per-engine loop, and `/stats` + `/v1/models` are recorded in the same pass
# (radiance is a vLLM derivative; the vLLM engine and env var stay for other deployments).
# A stub `curl` answers the three endpoints from canned bodies, so the script's own
# branches run — verbatim bodies, the script's header, the at-rest hint, the
# never-overwrite-a-good-fixture rule — with no engine in sight.
RADIANCE_CAPTURE = SCRIPTS / "capture-fixtures.sh"
#: The stub `curl`: it drops the options the script passes (`-fsS`, `--max-time N`, `-o`),
#: records every call, and answers the engine's three read-only endpoints from `$CANNED_*`.
#: Nothing here can reach a network.
RADIANCE_CURL_STUB = """#!/bin/sh
printf '%s\\n' "$*" >> "$STUB_LOG"
if [ "${FAIL_CURL:-0}" = "1" ]; then
echo "curl: (7) Failed to connect" >&2
exit 7
fi
out=""
url=""
while [ $# -gt 0 ]; do
case "$1" in
--max-time) shift 2 ;;
-o) out="$2"; shift 2 ;;
-*) shift ;;
*) url="$1"; shift ;;
esac
done
[ -n "$out" ] || { echo "stub curl: no -o target" >&2; exit 2; }
case "$url" in
*/metrics) printf '%s' "$CANNED_METRICS" > "$out" ;;
*/stats) printf '%s' "$CANNED_STATS" > "$out" ;;
*/v1/models) printf '%s' "$CANNED_MODELS" > "$out" ;;
*) echo "stub curl: unexpected URL $url" >&2; exit 2 ;;
esac
"""
#: The seven per-position spec-acceptance lines: the phase pins position="0".."6".
RADIANCE_CANNED_METRICS = (
"# HELP vllm:num_requests_running Number of requests currently running on GPU.\n"
"# TYPE vllm:num_requests_running gauge\n"
'vllm:num_requests_running{model_name="juggernaut",engine="0"} 0\n'
'radiance:decode_tokens_per_second{model_name="juggernaut",engine="0"} 110.9\n'
+ "".join(
f'radiance:draft_position_acceptance_ratio{{model_name="juggernaut",engine="0",'
f'position="{i}"}} 0.7\n'
for i in range(7)
)
)
RADIANCE_CANNED_STATS = (
'{"model":"juggernaut","version":"1.15.1","uptime_s":67490,"queue_cap":24,'
'"in_flight":0,"kv_util":0.7241322994232178,"prefix_hit_rate":0.9572382569313049,'
'"draft_positions":[{"reached":701741,"accepted":516252,"rate":0.7356731329650114}],'
'"requests":[],"cards":[{"index":0,"name":"AMD Radeon AI PRO R9700",'
'"vram_used":32119517184,"vram_total":34208743424}]}'
)
RADIANCE_CANNED_STATS_BUSY = (
'{"model":"juggernaut","version":"1.15.1","uptime_s":67500,"queue_cap":24,'
'"in_flight":1,"kv_util":0.72,"prefix_hit_rate":0.95,'
'"draft_positions":[{"reached":701741,"accepted":516252,"rate":0.735}],'
'"requests":[{"id":"req-1","queue_pos":1}],'
'"cards":[{"index":0,"name":"AMD Radeon AI PRO R9700",'
'"vram_used":32119517184,"vram_total":34208743424}]}'
)
RADIANCE_CANNED_MODELS = (
'{"object":"list","data":[{"id":"juggernaut","object":"model","created":1793484879,'
'"owned_by":"radiance","max_model_len":131072,"capabilities":{"chat":true}}]}'
)
RADIANCE_BASE = "http://radiance.test:8000"
def _radiance_capture_sandbox(
tmp_path: Path,
*,
recorded: dict[str, str] | None = None,
fail_curl: bool = False,
busy: bool = False,
no_engine: bool = False,
) -> tuple[Path, Path, Path, dict[str, str]]:
"""A copy of the script, a canned radiance, and a PATH whose `curl` reaches only the canned.
Returns (script, fixtures dir, stub call log, env). The script is copied into
`<tmp>/scripts/` because it derives `tests/fixtures/` from its own location, which is
what keeps the repository's fixtures out of reach of a test.
"""
script = tmp_path / "scripts" / RADIANCE_CAPTURE.name
script.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(RADIANCE_CAPTURE, script)
fixtures = tmp_path / "tests" / "fixtures"
fixtures.mkdir(parents=True, exist_ok=True)
for name, body in (recorded or {}).items():
(fixtures / name).write_text(body, encoding="utf-8")
stubs = tmp_path / "stubs"
stubs.mkdir()
stub = stubs / "curl"
stub.write_text(RADIANCE_CURL_STUB, encoding="utf-8")
stub.chmod(0o755)
log = tmp_path / "curl.log"
env = {
"PATH": f"{stubs}{os.pathsep}{os.environ['PATH']}",
"STUB_LOG": str(log),
"CANNED_METRICS": RADIANCE_CANNED_METRICS,
"CANNED_STATS": RADIANCE_CANNED_STATS_BUSY if busy else RADIANCE_CANNED_STATS,
"CANNED_MODELS": RADIANCE_CANNED_MODELS,
}
if not no_engine:
env["RADIANCE_BASE_URL"] = RADIANCE_BASE
if fail_curl:
env["FAIL_CURL"] = "1"
return script, fixtures, log, env
def _radiance_run(tmp_path: Path, **kwargs: object) -> subprocess.CompletedProcess[str]:
script, _fixtures, _log, env = _radiance_capture_sandbox(tmp_path, **kwargs) # type: ignore[arg-type]
return subprocess.run([BASH, str(script), "--with-llm"], capture_output=True, text=True,
env=env, check=False)
def _dialled_urls(log: Path) -> list[str]:
urls: list[str] = []
for line in log.read_text().splitlines():
for token in line.split():
if token.startswith("http"):
urls.append(token)
return urls
def _assert_verbatim(path: Path, canned: str) -> None:
"""The engine's reply inside a fixture, claimed verbatim.
The script's header is comment text; the engine's reply is the file's complete tail —
so the reply is verbatim iff the file ends with it byte for byte and everything before
it is comment lines (the metrics body's own `# HELP` / `# TYPE` lines are part of the
reply and stay in the fixture).
"""
text = path.read_text()
assert text.endswith(canned), f"{path.name} lost the tail of the engine's reply"
prefix = text[: -len(canned)]
non_comment = [line for line in prefix.splitlines() if line and not line.startswith("#")]
assert not non_comment, f"{path.name} has non-comment text before the reply: {non_comment}"
assert prefix, f"{path.name} lost its verbatim header"
def test_the_radiance_capture_records_all_three_endpoints_verbatim(tmp_path: Path) -> None:
"""The happy path, run: three fixtures, each with the script's verbatim header, the body
byte-for-byte what the engine answered, and a dial to exactly the three read-only
endpoints — nothing else, in the documented order."""
result = _radiance_run(tmp_path)
assert result.returncode == 0, result.stdout + result.stderr
fixtures = tmp_path / "tests" / "fixtures"
assert {p.name for p in fixtures.iterdir()} == {
"metrics_radiance.txt", "stats_radiance.json", "models_radiance.json"}
for name in ("metrics_radiance.txt", "stats_radiance.json", "models_radiance.json"):
text = (fixtures / name).read_text()
assert text.startswith("# Verbatim"), f"{name} lost its verbatim header"
assert "./scripts/capture-fixtures.sh" in text, f"{name} names its recorder"
_assert_verbatim(fixtures / "metrics_radiance.txt", RADIANCE_CANNED_METRICS)
_assert_verbatim(fixtures / "stats_radiance.json", RADIANCE_CANNED_STATS)
_assert_verbatim(fixtures / "models_radiance.json", RADIANCE_CANNED_MODELS)
assert _dialled_urls(fixtures.parent.parent / "curl.log") == [
f"{RADIANCE_BASE}/metrics", f"{RADIANCE_BASE}/stats", f"{RADIANCE_BASE}/v1/models"]
# the capture at rest records `requests: []` — the script says so, so the operator
# re-runs with a request in flight to record its row shape
assert "requests[] was empty — re-run with a request in flight to record its shape" in result.stdout
def test_the_radiance_capture_says_nothing_to_re_run_when_a_request_is_in_flight(tmp_path: Path) -> None:
"""With a request in flight the hint is gone (it would send the operator to re-run a
capture that already has the row shape), and the header still records what it saw."""
result = _radiance_run(tmp_path, busy=True)
assert result.returncode == 0, result.stdout + result.stderr
assert "re-run with a request in flight" not in result.stdout
stats = (tmp_path / "tests" / "fixtures" / "stats_radiance.json").read_text()
assert 'requests":[{"id":"req-1","queue_pos":1}]' in stats
assert "in_flight was 1 at capture" in stats
def test_the_radiance_capture_without_an_engine_says_which_env_var_it_wants(tmp_path: Path) -> None:
"""No engine env var is not a silent no-op: the gate names the var, like the others."""
script, _fixtures, _log, env = _radiance_capture_sandbox(tmp_path, no_engine=True)
result = subprocess.run([BASH, str(script), "--with-llm"], capture_output=True, text=True,
env=env, check=False)
assert result.returncode == 1
assert "no GUFO_BASE_URL / VLLM_BASE_URL / RADIANCE_BASE_URL set" in result.stdout
def test_a_dead_radiance_engine_never_replaces_a_recorded_capture(tmp_path: Path) -> None:
"""The `slots_gufo.json` rule: a capture script must never turn three good fixtures into
three empty files, because then the shape tests pass against nothing (or fail for the
wrong reason) and the ground truth is gone."""
good = {"metrics_radiance.txt": "# old\nradiance:kept 1\n",
"stats_radiance.json": "# old\n{\"cards\":[]}\n",
"models_radiance.json": "# old\n{\"object\":\"list\"}\n"}
script, fixtures, _log, env = _radiance_capture_sandbox(tmp_path, recorded=good, fail_curl=True)
result = subprocess.run([BASH, str(script), "--with-llm"], capture_output=True, text=True,
env=env, check=False)
assert result.returncode == 1
for name, body in good.items():
assert (fixtures / name).read_text() == body, f"{name} was replaced by a dead engine"
tests/test_live_fleet.py
+81
View File
@@ -25,6 +25,7 @@ from __future__ import annotations
import asyncio
import ipaddress
import json
import math
import os
import sqlite3
from collections.abc import Iterator
@@ -479,6 +480,86 @@ async def test_gufo_slots_answer_the_pinned_shape(settings: Settings) -> None:
assert_slot_shape(rows)
async def test_radiance_answers_the_pinned_shape(settings: Settings) -> None:
"""Phase 34: the radiance `/metrics` + `/stats` poll over the **running** engine.
The production path — `collectors.fetch_engine` with the LLM cadence's estimator,
`with_stats` the way `llm_jobs` runs it — so what fails here is a rename between the
engine and the map, not a typo in a fixture. Read-only: radiance exposes no write
endpoint, so "read-only" is a list of GETs. Skips when `RADIANCE_BASE_URL` is unset
(invariant 3's live twin: unset is not deployed here, not broken). The assertions
are shape, never numbers: the `stats` block's whitelist, the ordered 7-position spec
curve (when the build publishes it), one of the two specced session arms, and every
cell the engine answered is a finite number or `None` — an absent gauge is `None`,
never a faked `0` (rule 2's live twin). `unparsed` must be empty: a parse error on
the live wire is drift, not a tolerance.
"""
if not settings.radiance_url:
pytest.skip("RADIANCE_BASE_URL is unset — that engine is not deployed here")
# The span the LLM jobs really get: phase 23 spans the LLM estimator by the poll
# interval, not by the host's `RATE_WINDOW_SECONDS` (the other tests' idiom).
rates = RateEstimator(span=settings.llm_interval)
async with httpx.AsyncClient(timeout=settings.llm_timeout) as client:
result = await collectors.fetch_engine(client, "radiance", settings.radiance_url,
settings.radiance_api_key, rates,
with_stats=True)
payload = result.payload
assert payload["engine"] == "radiance"
assert payload["unparsed"] == 0, (
f"a parse error on the live wire is drift, not a tolerance: {payload['unparsed']}")
metrics = payload["metrics"]
# The /stats block, task 03's whitelist: present only when the sidecar answered, and
# then only those keys (the engine internals stay out — decision 3).
stats = payload.get("stats")
if stats is not None:
allowed = {"model", "version", "uptime_s", "startup_s", "total_requests", "cards"}
extra = set(stats) - allowed
assert not extra, f"the /stats subset carried keys the whitelist does not name: {sorted(extra)}"
for card in stats.get("cards", []):
assert set(card) <= {"index", "name", "vram_used", "vram_total", "vram_used_pct"}, card
# The spec-acceptance curve: the ordered seven, when the build publishes it.
positions = metrics["spec_positions"]
assert isinstance(positions, list)
if positions:
assert len(positions) == 7, (
f"the spec curve has {len(positions)} positions — the pin is 0..6, "
f"not {[p['position'] for p in positions]}")
assert [p["position"] for p in positions] == list(range(7)), ("not 0..6 in order")
for pos in positions:
ratio = pos["ratio"]
assert ratio is None or (isinstance(ratio, int | float)
and not isinstance(ratio, bool)
and math.isfinite(ratio) and 0.0 <= ratio <= 1.0), (
f"position {pos['position']}'s ratio {ratio!r} is not a 0-1 gauge")
# Every cell the engine answered is a finite number or None — never a faked 0.
for key, cell in metrics.items():
if key == "spec_positions":
continue # the ordered list, checked above
if not isinstance(cell, dict) or "value" not in cell:
continue
value = cell["value"]
assert value is None or (isinstance(value, int | float)
and not isinstance(value, bool)
and math.isfinite(value)), (
f"{key}: {value!r} is neither a number nor None")
# The session arm is one of the two specced arms: /stats answered (the engine's own
# summary is the source) or it did not (the /metrics gauges, and it says so).
sessions = payload["sessions"]
if stats is not None:
assert sessions["source"] == "radiance /stats"
assert "running" in sessions and "waiting" in sessions and "queue_cap" in sessions
else:
assert sessions["source"] is None
assert sessions.get("queue_cap") is None, ("an absent /stats leaves the cap n/a, never 0")
assert "/stats" in (sessions.get("detail") or ""), (
f"the fallback arm must name the missing sidecar: {sessions.get('detail')!r}")
assert sessions["active"] is None or isinstance(sessions["active"], int | float)
def require_upsd(settings: Settings) -> None:
"""Skip, not fail, when there is no real upsd to talk to."""
if not settings.nut_enabled:
tests/test_llm_source.py
+646 -13
View File
@@ -5,17 +5,25 @@ the deployed servers (tests/fixtures/metrics_*.txt), not to upstream documentati
"""
from __future__ import annotations
import json
import re
from collections.abc import Callable, Mapping, Sequence
from pathlib import Path
from typing import Any
import pytest
from dashboards.prom import Prom
from dashboards.rates import MIN_WINDOW_SECONDS, RateEstimator
from dashboards.severity import classify
from dashboards.sources.llm import (
RADIANCE_HISTOGRAMS,
RADIANCE_OPTIONAL,
RADIANCE_SERIES,
engine_result,
fleet_totals,
parse_gufo_metrics,
parse_radiance_metrics,
parse_vllm_metrics,
rate_key,
series_names,
@@ -62,19 +70,31 @@ def poll_engine(metrics_text: Any, engine: str, span: float, stamps: Sequence[fl
# ---- gufo / llama.cpp ----------------------------------------------------- #
def test_gufo_counters_come_straight_from_the_capture(metrics_text) -> None:
metrics = parse_gufo_metrics(metrics_text("gufo"))
assert metrics["tokens_generated_total"]["value"] == 877_336.0
assert metrics["tokens_prompt_total"]["value"] == 1_350_666.0
assert metrics["tokens_prompt_cached_total"]["value"] == 124_529_105.0
assert metrics["requests_processing"]["value"] == 0.0
assert metrics["device_lost_total"]["value"] == 0.0
assert metrics["kv_cache_usage_ratio"]["value"] == 0.0
"""The cell's value is the number the exposition declares, verbatim: no
conversion, no scaling, and the series' own name is the source. Compared against
the raw `Prom` parse of the same capture — never a value frozen at capture time
(the counters reset on every engine restart)."""
text = metrics_text("gufo")
prom = Prom(text)
metrics = parse_gufo_metrics(text)
for key, name in (
("tokens_generated_total", "llamacpp:tokens_predicted_total"),
("tokens_prompt_total", "llamacpp:prompt_tokens_total"),
("tokens_prompt_cached_total", "llamacpp:prompt_tokens_cached_total"),
("requests_processing", "llamacpp:requests_processing"),
("device_lost_total", "gufo_device_lost_total"),
("kv_cache_usage_ratio", "llamacpp:kv_cache_usage_ratio"),
):
assert metrics[key]["value"] == prom.scalar([name]), key
assert metrics[key]["source"] == name, "the counter's own name is the source"
def test_gufo_speculative_accept_rate_is_accepted_over_draft(metrics_text) -> None:
metrics = parse_gufo_metrics(metrics_text("gufo"))
accept = metrics["spec_accept_rate"]["value"]
assert accept == pytest.approx(578_514 / 789_028)
accepted = metrics["spec_accepted_tokens_total"]["value"]
draft_tokens = metrics["spec_draft_tokens_total"]["value"]
assert accept == pytest.approx(accepted / draft_tokens)
assert str(metrics["spec_accept_rate"]["source"]).startswith("computed")
@@ -82,7 +102,10 @@ def test_gufo_lifetime_tokens_include_cache_hits(metrics_text) -> None:
"""gufo's prompt counter excludes cache hits; the lifetime total must not."""
metrics = parse_gufo_metrics(metrics_text("gufo"))
total = metrics["tokens_total"]["value"]
assert total == 1_350_666 + 124_529_105 + 877_336
prompt = metrics["tokens_prompt_total"]["value"]
cached = metrics["tokens_prompt_cached_total"]["value"] or 0.0
generated = metrics["tokens_generated_total"]["value"]
assert total == pytest.approx(prompt + cached + generated)
def test_gufo_throughput_prefers_measured_delta_over_the_native_gauge(metrics_text) -> None:
@@ -99,8 +122,11 @@ def test_gufo_throughput_prefers_measured_delta_over_the_native_gauge(metrics_te
assert decode["value"] == pytest.approx(40.0)
assert decode["window"] == pytest.approx(10.0)
assert "Δ" in decode["source"] and "tokens_predicted_total" in decode["source"]
# the native gauge is still there, labelled, because it answers a different question
assert metrics["decode_tokens_per_second_native"]["value"] == pytest.approx(42.621031668072597)
# the native gauge is still there, labelled, because it answers a different question —
# verbatim the exposition's own number (a volatile gauge: no frozen value)
assert metrics["decode_tokens_per_second_native"]["value"] == \
Prom(text).scalar(["llamacpp:predicted_tokens_seconds"])
assert metrics["decode_tokens_per_second_native"]["source"] == "llamacpp:predicted_tokens_seconds"
def test_gufo_the_window_the_llm_jobs_are_spanned_by_is_one_that_fills(metrics_text) -> None:
@@ -323,10 +349,12 @@ def test_absent_gufo_metrics_are_none_with_no_source() -> None:
def test_gufo_kv_cache_cell_keeps_provenance_and_gains_severity(metrics_text) -> None:
metrics = parse_gufo_metrics(metrics_text("gufo"))
cell = metrics["kv_cache_usage_ratio"]
assert cell["value"] == 0.0
assert cell["value"] == Prom(metrics_text("gufo")).scalar(["llamacpp:kv_cache_usage_ratio"]), \
"verbatim the exposition's own ratio — the cell stays what it was"
assert cell["source"] == "llamacpp:kv_cache_usage_ratio"
assert cell["derived"] is False, "the verdict is additive; the cell still says what it was"
assert cell["severity"] == "ok"
assert cell["severity"] == classify("llm.kv_cache_usage_pct", cell["value"] * 100.0), \
"the verdict is the table's for this value (percent scale)"
@pytest.mark.parametrize("ratio,expected", [
@@ -667,3 +695,608 @@ def test_vllm_a_measured_zero_beats_the_running_estimate(metrics_text) -> None:
decode = metrics["decode_tokens_per_second"]
assert decode["value"] == 0.0
assert "running/tpot" not in str(decode["source"])
# ---- radiance: the ground-truth fixtures (phase 34, task 01) ---------------- #
#
# Radiance is a vLLM derivative: its /metrics carries the `radiance:` native series plus
# a `vllm:` compatibility family, and the capture records /stats and /v1/models in the
# same pass. These tests pin the *shape and the name set* of those captures, never the
# numbers (the nut_*.vars precedent): a metric rename or a new build's series fails here
# before a dashboard does.
def _load_radiance_json(name: str) -> Any:
"""Load a radiance JSON fixture, skipping with the capture command like the others."""
path = Path(__file__).resolve().parent / "fixtures" / name
if not path.exists():
pytest.skip(f"missing fixture {path.name} — run scripts/capture-fixtures.sh --with-llm")
# the capture script writes its verbatim header as `#` comment lines; the document
# is what follows them
body = "\n".join(line for line in path.read_text().splitlines() if not line.startswith("#"))
return json.loads(body)
def _radiance_map_names() -> set[str]:
return ({name for names, _ in RADIANCE_SERIES.values() for name in names}
| {name for names in RADIANCE_HISTOGRAMS.values() for name in names})
def test_radiance_fixture_name_set_is_pinned_by_the_map_and_optionals(metrics_text) -> None:
"""The drift pin: every name in the capture's radiance:/vllm: families is either a
payload cell (the map) or a declared non-cell (RADIANCE_OPTIONAL), and vice versa —
so a rename, a dropped series, or a typo in a map name fails the suite before a
dashboard does."""
names = set()
for line in metrics_text("radiance").splitlines():
if line and not line.startswith("#"):
names.add(line.split("{")[0].split()[0])
contract = {name for name in names if name.startswith(("radiance:", "vllm:"))}
assert contract, "the capture lost its series — re-run the capture with the engine up"
map_names = _radiance_map_names()
unexplained = contract - map_names - RADIANCE_OPTIONAL
assert not unexplained, (
f"names in the capture that are neither a payload cell nor declared optional: "
f"{sorted(unexplained)}")
absent_everywhere = map_names - contract - RADIANCE_OPTIONAL
assert not absent_everywhere, (
f"map names in neither the capture nor RADIANCE_OPTIONAL (a rename or a typo): "
f"{sorted(absent_everywhere)}")
stale = RADIANCE_OPTIONAL - contract - map_names
assert not stale, f"RADIANCE_OPTIONAL entries that are neither real names: {sorted(stale)}"
def test_radiance_fixture_carries_both_families(metrics_text) -> None:
"""The vLLM-derived shape is part of the contract: the capture must hold both the
native series and the compatibility family, or the name set has drifted."""
text = metrics_text("radiance")
native = sum(1 for line in text.splitlines() if line.startswith("radiance:"))
compat = sum(1 for line in text.splitlines() if line.startswith("vllm:"))
assert native > 0, "the capture lost its radiance: native series"
assert compat > 0, "the capture lost its vllm: compatibility family"
def test_radiance_per_position_series_has_the_seven_positions_the_phase_pins(
metrics_text,
) -> None:
"""The per-position spec-acceptance series is the 7-bar curve: its label set is
position="0".."6", each position a series of its own."""
positions = set()
for line in metrics_text("radiance").splitlines():
if line.startswith("radiance:draft_position_acceptance_ratio{"):
m = re.search(r'position="(\d+)"', line)
assert m, f"a per-position line without a position label: {line}"
positions.add(m.group(1))
assert positions == {str(i) for i in range(7)}, (
f"the phase pins seven draft positions; the capture has {sorted(positions)}")
STATS_TOP_LEVEL_KEYS = (
"cards", "checkpoints_written", "ckpt_evictions", "ckpt_refused", "decode_tps",
"draft_accept", "draft_positions", "hit_cut_tokens", "hit_cuts", "in_flight",
"kv_bytes_carved", "kv_bytes_total", "kv_bytes_used", "kv_tokens_cached",
"kv_tokens_max", "kv_tokens_total", "kv_tokens_used", "kv_util",
"linear_cached_tokens", "linear_hit_rate", "link", "model", "now_ms", "passes",
"preempted", "preemptions", "prefill_chunks_capped", "prefill_tps",
"prefix_evictions", "prefix_hit_rate", "queue_cap", "released", "requests",
"rows_unspec", "running", "started_at", "startup_s", "step_ns", "step_ns_decode",
"step_ns_decode_best", "step_ns_decode_recent", "steps", "steps_decode", "steps_held",
"steps_mixed", "steps_prefill", "total_requests", "uptime_s", "version", "waiting",
)
def test_radiance_stats_fixture_shape() -> None:
"""The /stats document: the top-level key set, the per-card VRAM rows, the requests
list and the per-position spec-acceptance list — shape, not numbers. A capture at
rest records `requests: []` (the header comment says so); a row shape is recorded
by re-running the capture with a request in flight."""
stats = _load_radiance_json("stats_radiance.json")
assert isinstance(stats, dict)
assert tuple(sorted(stats.keys())) == STATS_TOP_LEVEL_KEYS, (
"the /stats top-level key set changed — the shape the page reads drifted; "
"re-capture and re-decide the payload set")
cards = stats["cards"]
assert isinstance(cards, list) and cards, "cards[] is the per-card VRAM panel's source"
for card in cards:
assert isinstance(card, dict)
assert isinstance(card["index"], int)
assert isinstance(card["name"], str) and card["name"]
assert isinstance(card["vram_used"], int)
assert isinstance(card["vram_total"], int)
assert 0 < card["vram_used"] <= card["vram_total"]
assert isinstance(stats["requests"], list)
assert isinstance(stats["queue_cap"], int)
assert isinstance(stats["in_flight"], int)
positions = stats["draft_positions"]
assert isinstance(positions, list)
assert len(positions) == 7, (
"the phase pins seven draft positions; the capture has "
f"{len(positions)} — the 7-bar curve needs re-deciding")
for pos in positions:
assert isinstance(pos, dict)
assert isinstance(pos["reached"], int)
assert isinstance(pos["accepted"], int)
assert isinstance(pos["rate"], float)
def test_radiance_models_fixture_shape() -> None:
"""The /v1/models document: one model, and the capabilities / max_model_len it
publishes — the only place the engine states what it can do."""
models = _load_radiance_json("models_radiance.json")
assert isinstance(models, dict)
assert models["object"] == "list"
data = models["data"]
assert isinstance(data, list) and data
model = data[0]
assert model["owned_by"] == "radiance", "the engine's own word for its family"
assert isinstance(model["id"], str) and model["id"]
assert isinstance(model["max_model_len"], int)
assert isinstance(model["capabilities"], dict)
# ---- radiance: the field map (phase 34, task 03) ----------------------- #
#
# The map is built from the task-01 fixture (rule 7); the tests below pin the three
# throughput arms (gauge wins / measured fills / n/a on a refused window), the severity
# pass-through on the existing KV row, the /stats session arms, the whitelisted stats
# block and the additive history series. As always: shape and name set, never the
# numbers of a live engine.
def _radiance_stats() -> dict:
return _load_radiance_json("stats_radiance.json")
# ---- the fixture parse -------------------------------------------------- #
def test_radiance_fixture_parse_every_mapped_key_resolves(metrics_text) -> None:
"""Every mapped key resolves on the verbatim capture, with its pinned source string."""
metrics = parse_radiance_metrics(metrics_text("radiance"))
expected_source = {
"decode_tokens_per_second_native": "radiance:decode_tokens_per_second",
"prompt_tokens_per_second_native": "radiance:prefill_tokens_per_second",
"num_requests_running": "vllm:num_requests_running",
"num_requests_waiting": "vllm:num_requests_waiting",
"gpu_cache_usage": "vllm:kv_cache_usage_perc",
"kv_cache_max_tokens": "radiance:kv_cache_max_tokens",
"kv_cache_max_bytes": "radiance:kv_cache_max_bytes",
"queue_depth": "radiance:http_queue_depth",
"admission_rejected_total": "radiance:http_admission_rejected_total",
"num_preemptions_total": "vllm:num_preemptions_total",
"prompt_tokens_total": "vllm:prompt_tokens_total",
"generation_tokens_total": "vllm:generation_tokens_total",
"gpu_prefix_cache_hit_rate": "vllm:gpu_prefix_cache_hit_rate",
"spec_accept_rate": "radiance:draft_acceptance_ratio",
"decode_step_seconds_recent": "radiance:decode_step_seconds_recent",
"decode_step_seconds_best": "radiance:decode_step_seconds_best",
}
for key, source in expected_source.items():
cell = metrics[key]
assert set(cell) >= {"value", "source", "derived"}, f"{key} lost its cell shape"
assert cell["source"] == source, f"{key} resolved to {cell['source']!r}, not {source!r}"
assert cell["value"] is not None, f"{key} should resolve on the capture"
# a map name this build does not carry: n/a, never 0
assert metrics["num_requests_swapped"]["value"] is None
assert metrics["num_requests_swapped"]["source"] is None
def test_radiance_a_series_removed_from_the_capture_is_n_a_never_zero(metrics_text) -> None:
"""The completion-criteria arm: a line removed from a fixture copy answers
`{\"value\": null}` — 0 would claim \"the engine published zero\", which it did not."""
by_key = {
"radiance:http_queue_depth": "queue_depth",
"radiance:kv_cache_max_bytes": "kv_cache_max_bytes",
"radiance:decode_step_seconds_best": "decode_step_seconds_best",
"vllm:num_preemptions_total": "num_preemptions_total",
"vllm:gpu_prefix_cache_hit_rate": "gpu_prefix_cache_hit_rate",
"radiance:draft_acceptance_ratio": "spec_accept_rate",
}
for name, key in by_key.items():
text = "\n".join(line for line in metrics_text("radiance").splitlines()
if not line.startswith(name))
cell = parse_radiance_metrics(text)[key]
assert cell["value"] is None, f"{key} answered a value for a removed series: {cell}"
assert cell["source"] is None
def test_radiance_spec_positions_are_the_ordered_seven(metrics_text) -> None:
metrics = parse_radiance_metrics(metrics_text("radiance"))
positions = metrics["spec_positions"]
assert [p["position"] for p in positions] == list(range(7)), \
"the phase pins seven draft positions, in the order the engine publishes them"
for p in positions:
assert 0.0 <= p["ratio"] <= 1.0, "a per-position ratio is a 0-1 gauge"
# the ratios are a running-window aggregate: volatile between captures, so the
# pin is the shape (seven, ordered, in-range) — the name set is pinned above
prom = Prom(metrics_text("radiance"))
by_position = prom.by_label(["radiance:draft_position_acceptance_ratio"], "position")
for p in positions:
assert p["ratio"] == by_position[str(p["position"])], \
f"position {p['position']} is not the exposition's own number"
def test_radiance_an_absent_position_stays_absent_not_zeroed(metrics_text) -> None:
text = "\n".join(
line for line in metrics_text("radiance").splitlines()
if not ('radiance:draft_position_acceptance_ratio{' in line
and 'position="6"' in line)
)
positions = parse_radiance_metrics(text)["spec_positions"]
assert [p["position"] for p in positions] == [0, 1, 2, 3, 4, 5], \
"position 6 is absent from this capture: the list holds what the engine published"
assert all(p["ratio"] is not None and p["ratio"] > 0.0 for p in positions), \
"an absent position must not be filled with a 0"
def test_radiance_tokens_total_is_the_two_vllm_counters(metrics_text) -> None:
metrics = parse_radiance_metrics(metrics_text("radiance"))
prompt = metrics["prompt_tokens_total"]["value"]
generated = metrics["generation_tokens_total"]["value"]
assert metrics["tokens_total"]["value"] == pytest.approx(prompt + generated)
assert metrics["tokens_total"]["source"] == "computed:prompt+generated"
# both counters removed: n/a, never 0
text = metrics_text("radiance")
for name in ("vllm:prompt_tokens_total", "vllm:generation_tokens_total"):
text = "\n".join(line for line in text.splitlines()
if not line.startswith(name))
assert parse_radiance_metrics(text)["tokens_total"]["value"] is None
def test_radiance_prefix_hit_ratio_is_the_gauge_source_named_not_derived(metrics_text) -> None:
"""This build publishes no prefix counters, so a hits/queries ratio would be an
invented number — the gauge is the answer, under the shared UI key."""
text = metrics_text("radiance")
metrics = parse_radiance_metrics(text)
gauge = Prom(text).scalar(["vllm:gpu_prefix_cache_hit_rate"])
cell = metrics["prefix_cache_hit_ratio"]
assert cell["value"] == pytest.approx(gauge)
assert cell["source"] == "vllm:gpu_prefix_cache_hit_rate"
assert cell["derived"] is False, "the gauge is not a computed ratio"
# the gauge cell itself is the payload cell too (the map entry)
assert metrics["gpu_prefix_cache_hit_rate"]["value"] == pytest.approx(gauge)
def test_radiance_latency_histograms_are_the_vllm_shaped_family(metrics_text) -> None:
metrics = parse_radiance_metrics(metrics_text("radiance"))
ttft = metrics["time_to_first_token_seconds"]
assert ttft and ttft["count"] > 0 and ttft["mean"] > 0
assert metrics["time_to_first_token_seconds__p95"]["value"] >= ttft["mean"]
assert metrics["time_per_output_token_seconds"]["mean"] > 0
assert metrics["e2e_request_latency_seconds"]["count"] > 0
# no request_queue_time on this build: the cell is absent, never back-filled
assert "request_queue_time_seconds" not in metrics
assert "request_queue_time_seconds__mean" not in metrics
# ---- rule 2, the three throughput arms ----------------------------------- #
def test_radiance_throughput_arm_gauge_wins(metrics_text) -> None:
"""The native gauge is the scheduler's recent-window aggregate, so it wins."""
text = metrics_text("radiance")
metrics = parse_radiance_metrics(text, RateEstimator(span=10), 1_000.0)
decode = metrics["decode_tokens_per_second"]
# the gauge's value is the exposition's own number (volatile — the pin is the
# passthrough, not a frozen value), and it wins the cell
assert decode["value"] == metrics["decode_tokens_per_second_native"]["value"]
assert decode["value"] == Prom(text).scalar(["radiance:decode_tokens_per_second"], agg="avg")
assert decode["source"] == "radiance:decode_tokens_per_second"
assert decode["derived"] is False, "a scraped gauge is not a computed Δ"
assert "window" not in decode, "a scraped gauge names no Δ window"
prompt = metrics["prompt_tokens_per_second"]
assert prompt["value"] == metrics["prompt_tokens_per_second_native"]["value"]
assert prompt["source"] == "radiance:prefill_tokens_per_second"
def test_radiance_throughput_arm_measured_delta_when_the_gauge_is_absent(metrics_text) -> None:
"""A gauge this build does not publish falls back to the measured Δ, per label set."""
text = metrics_text("radiance")
text = "\n".join(line for line in text.splitlines()
if not line.startswith("radiance:decode_tokens_per_second"))
rates = RateEstimator(span=10)
parse_radiance_metrics(text, rates, 1_000.0)
text = bump_metric(text, "vllm:generation_tokens_total", 250.0)
metrics = parse_radiance_metrics(text, rates, 1_010.0)
decode = metrics["decode_tokens_per_second"]
assert decode["value"] == pytest.approx(25.0)
assert decode["derived"] is True
assert decode["source"] == "computed:Δvllm:generation_tokens_total/Δt"
assert decode["window"] == pytest.approx(10.0), \
"the label must name the window the value used"
# the arms are independent: the prompt gauge still wins its own cell
prompt = metrics["prompt_tokens_per_second"]
assert prompt["value"] == metrics["prompt_tokens_per_second_native"]["value"]
assert prompt["derived"] is False
def test_radiance_throughput_arm_a_refused_window_is_n_a_with_its_label(metrics_text) -> None:
"""A refused window is still an answer: n/a beside the honest Δ<window> it refused."""
text = metrics_text("radiance")
text = "\n".join(line for line in text.splitlines()
if not line.startswith("radiance:decode_tokens_per_second"))
rates = RateEstimator(span=10)
parse_radiance_metrics(text, rates, 1_000.0)
text = bump_metric(text, "vllm:generation_tokens_total", 50.0)
parse_radiance_metrics(text, rates, 1_010.0)
text = bump_metric(text, "vllm:generation_tokens_total", 60.0)
decode = parse_radiance_metrics(text, rates, 1_010.3)["decode_tokens_per_second"]
assert decode["value"] is None, "0.3 s is the loop's catch-up gap, not a rate"
assert decode["window"] == pytest.approx(0.3)
assert decode["window"] < MIN_WINDOW_SECONDS
assert decode["source"] == "computed:Δvllm:generation_tokens_total/Δt"
def test_radiance_throughput_arm_both_absent_is_n_a_never_zero(metrics_text) -> None:
text = metrics_text("radiance")
for name in ("radiance:decode_tokens_per_second", "radiance:prefill_tokens_per_second",
"vllm:generation_tokens_total", "vllm:prompt_tokens_total"):
text = "\n".join(line for line in text.splitlines() if not line.startswith(name))
metrics = parse_radiance_metrics(text) # no estimator: no measured arm either
for key in ("decode_tokens_per_second", "prompt_tokens_per_second"):
cell = metrics[key]
assert cell["value"] is None, f"{key} printed a value with no source: {cell}"
assert cell["source"] is None
# ---- severity: the KV cell is rated, the spec cell is not --------------- #
def test_radiance_kv_cell_carries_the_severity_verdict(metrics_text) -> None:
"""Radiance publishes the same 0-1 ratio the vLLM map feeds the table: the
existing `llm.kv_cache_usage_pct` row rates it, no new threshold anywhere."""
text = metrics_text("radiance")
base = Prom(text).scalar(["vllm:kv_cache_usage_perc"], agg="max")
assert base is not None
cell = parse_radiance_metrics(text)["gpu_cache_usage"]
assert cell["value"] == pytest.approx(base)
assert cell["source"] == "vllm:kv_cache_usage_perc"
assert cell["derived"] is False, "the verdict is additive; the cell still says what it was"
assert cell["severity"] == classify("llm.kv_cache_usage_pct", base * 100.0), \
"the table's verdict for this value (percent scale)"
# push it above the table's fail bound (90 %): the verdict follows the value
target = 0.99
pressed = bump_metric(text, "vllm:kv_cache_usage_perc", target - base)
cell = parse_radiance_metrics(pressed)["gpu_cache_usage"]
assert cell["value"] == pytest.approx(target)
assert cell["severity"] == "fail"
def test_radiance_spec_accept_carries_no_severity(metrics_text) -> None:
"""Decision 4 pinned both ways: the KV cell is rated, the spec cell is not — an
efficiency number is not a verdict."""
text = metrics_text("radiance")
metrics = parse_radiance_metrics(text)
assert metrics["spec_accept_rate"]["value"] == \
Prom(text).scalar(["radiance:draft_acceptance_ratio"], agg="avg")
assert "severity" not in metrics["spec_accept_rate"]
assert metrics["gpu_cache_usage"]["severity"] is not None, \
"the other direction: the KV cell is rated"
# ---- /stats as the session source ---------------------------------------- #
def test_radiance_sessions_stats_arm_at_rest_counts_and_cap(metrics_text) -> None:
"""The at-rest capture: requests[] empty → the counts-otherwise sentence, the cap
from /stats, `active` from the /metrics gauge (one definition in every arm).
The numbers come from the capture itself — volatile between re-captures."""
stats = _radiance_stats()
result = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
RateEstimator(), stats=stats, now=1.0)
sessions = result.payload["sessions"]
active_expected = Prom(metrics_text("radiance")).scalar(["vllm:num_requests_running"])
assert sessions["active"] == active_expected, "active stays the /metrics gauge in every arm"
assert sessions["running"] == stats["running"]
assert sessions["waiting"] == stats["waiting"]
assert sessions["queue_cap"] == stats["queue_cap"]
assert sessions["list"] == stats["requests"]
assert sessions["source"] == "radiance /stats"
if not stats["requests"]:
assert sessions["detail"] == (
f"radiance reports counts, not sessions: {stats['running']:g} running"
f" of cap {stats['queue_cap']:g}. Per-request detail appears here while "
f"requests are in flight.")
else:
assert sessions["detail"] is None, "rows in flight: no at-rest sentence"
def test_radiance_sessions_stats_arm_non_empty_rows_verbatim(metrics_text) -> None:
"""A request in flight: the rows are the engine's own shape, verbatim."""
stats = _radiance_stats()
rows = [{"id": 42, "model": "juggernaut", "state": "running",
"prompt_tokens": 1_200, "completed_tokens": 250}]
stats = {**stats, "requests": rows, "running": 1, "waiting": 2}
result = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
RateEstimator(), stats=stats, now=1.0)
sessions = result.payload["sessions"]
assert sessions["list"] == rows, "the rows pass through untouched"
assert sessions["source"] == "radiance /stats"
assert sessions["detail"] is None
assert sessions["running"] == 1
assert sessions["waiting"] == 2
assert sessions["queue_cap"] == stats["queue_cap"] # the capture's cap, untouched
def test_radiance_sessions_missing_stats_fall_back_to_the_gauges(metrics_text) -> None:
"""The sidecar failed: the poll is still healthy, the counts come from the
/metrics gauges (a different provenance, said so), and the cap is null — never 0."""
result = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
RateEstimator(), now=1.0) # stats=None: /stats did not answer
sessions = result.payload["sessions"]
assert sessions["running"] == 0.0, "the /metrics gauge's value"
assert sessions["waiting"] == 0.0
assert sessions["queue_cap"] is None, "null, not 0 — rule 2"
assert sessions["list"] == []
assert "did not answer" in sessions["detail"]
assert "gauges" in sessions["detail"]
assert "stats" not in result.payload, "no stats block: the sidecar never answered"
# the metrics passthroughs degrade with it
assert result.payload["metrics"]["queue_cap"]["value"] is None
assert result.payload["metrics"]["vram_used_pct"]["value"] is None
# ---- the whitelisted stats block ------------------------------------------ #
def test_radiance_payload_stats_block_is_whitelisted(metrics_text) -> None:
stats = _radiance_stats()
result = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
RateEstimator(), stats=stats, now=1.0)
block = result.payload["stats"]
assert set(block) == {"model", "version", "uptime_s", "startup_s",
"total_requests", "cards"}, \
"the engine internals (link / passes / steps* / …) stay out of the payload"
assert block["model"] == "juggernaut"
assert block["version"] == "1.15.1"
assert isinstance(block["uptime_s"], int) and block["uptime_s"] >= 0, \
"uptime is the engine's own seconds — volatile between captures"
card = block["cards"][0]
src = stats["cards"][0]
assert set(card) == {"index", "name", "vram_used", "vram_total", "vram_used_pct"}
assert card["vram_used"] == src["vram_used"], \
"the raw bytes travel as the engine published them"
assert card["vram_total"] == src["vram_total"]
pct = card["vram_used_pct"]
assert pct["derived"] is True, "the one computed number in the block"
assert pct["source"] == "computed:vram_used/vram_total*100"
assert pct["value"] == pytest.approx(src["vram_used"] / src["vram_total"] * 100.0)
def test_radiance_a_renamed_stats_key_degrades_to_absent(metrics_text) -> None:
"""A radiance upgrade that renames a /stats key degrades it to absent, not a
crash — the poll still returns its /metrics payload."""
stats = _radiance_stats()
stats["engine_version"] = stats.pop("version")
result = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
RateEstimator(), stats=stats, now=1.0)
block = result.payload["stats"]
assert "version" not in block, "renamed away: absent, never a guessed value"
assert "engine_version" not in block, "and not carried under the new name either"
assert block["model"] == "juggernaut", "the rest of the block is intact"
assert result.payload["metrics"]["spec_accept_rate"]["value"] is not None, \
"the poll still returns"
# and a renamed card key degrades the computed pct, not the whole block
stats2 = _radiance_stats()
card = {**stats2["cards"][0], "vram_in_use": stats2["cards"][0]["vram_used"]}
del card["vram_used"]
stats2["cards"] = [card]
result2 = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
RateEstimator(), stats=stats2, now=1.0)
row = result2.payload["stats"]["cards"][0]
assert "vram_used" not in row and "vram_in_use" not in row
assert row["vram_used_pct"]["value"] is None, "the computed number degrades to n/a"
# ---- the /stats passthroughs in the metrics dict ------------------------- #
def test_radiance_queue_cap_is_the_stats_passthrough(metrics_text) -> None:
stats = _radiance_stats()
metrics = parse_radiance_metrics(metrics_text("radiance"), stats=stats)
cell = metrics["queue_cap"]
assert cell["value"] == stats["queue_cap"], "the /stats number, verbatim"
assert cell["source"] == "radiance /stats"
assert cell["derived"] is False
absent = parse_radiance_metrics(metrics_text("radiance"))
assert absent["queue_cap"]["value"] is None, "null, not 0 — rule 2"
assert absent["queue_cap"]["source"] is None
# ---- the history samples ------------------------------------------------- #
def test_radiance_history_samples_include_the_new_series(metrics_text) -> None:
text = metrics_text("radiance")
stats = _radiance_stats()
result = engine_result("radiance", "http://r:8000", text,
RateEstimator(), stats=stats, now=1.0)
names = {s.metric for s in result.samples}
for expected in ("llm.decode_tokens_per_second", "llm.prompt_tokens_per_second",
"llm.kv_cache_usage", "llm.requests_running", "llm.requests_waiting",
"llm.spec_accept_rate",
"llm.prefix_cache_hit_ratio", "llm.tpot_seconds",
"llm.ttft_seconds", "llm.ttft_p95_seconds",
"llm.queue_depth", "llm.queue_cap",
"llm.decode_step_seconds_recent", "llm.vram_used_pct"):
assert expected in names, f"{expected} missing from the radiance samples"
# `tokens_total` is a payload key, not a history series (it is not an existing
# HISTORY_METRICS key, and the task's additions are the four above): the fleet
# block reads it from the payload, the chart never drew it
assert "llm.tokens_total" not in names
# the stored values are the capture's own numbers (volatile — pinned as passthrough,
# never as frozen values)
prom = Prom(text)
stored = {s.metric: s.value for s in result.samples}
assert stored["llm.decode_tokens_per_second"] == \
prom.scalar(["radiance:decode_tokens_per_second"], agg="avg")
assert stored["llm.prompt_tokens_per_second"] == \
prom.scalar(["radiance:prefill_tokens_per_second"], agg="avg")
assert stored["llm.queue_cap"] == stats["queue_cap"]
assert stored["llm.queue_depth"] == prom.scalar(["radiance:http_queue_depth"], agg="max")
assert stored["llm.decode_step_seconds_recent"] == \
prom.scalar(["radiance:decode_step_seconds_recent"], agg="max")
assert stored["llm.kv_cache_usage"] == \
prom.scalar(["vllm:kv_cache_usage_perc"], agg="max")
# the per-position ratios are snapshot-only (decision 2): no history series
assert not any("position" in name for name in names)
# no request_queue_time on this build: queue_seconds is n/a by absence
assert "llm.queue_seconds" not in names
def test_radiance_history_does_not_touch_gufo_or_vllm_samples(metrics_text) -> None:
"""The existing keys and their candidate order are untouched: a gufo or vLLM
payload never grows a radiance series."""
for engine in ("gufo", "vllm"):
result = engine_result(engine, "http://x:1", metrics_text(engine),
RateEstimator(), now=1.0)
names = {s.metric for s in result.samples}
for foreign in ("llm.queue_depth", "llm.queue_cap",
"llm.decode_step_seconds_recent", "llm.vram_used_pct"):
assert foreign not in names, f"{engine} grew a radiance series: {foreign}"
def test_series_names_radiance_is_its_own_map() -> None:
"""`series_names` is the parse catalog — the map's keys, per engine."""
names = series_names("radiance")
assert "spec_accept_rate" in names
assert "gpu_cache_usage" in names
assert "queue_depth" in names
assert "tokens_generated_total" not in names, "that is the gufo key"
assert "spec_accept_rate" not in series_names("vllm")
# ---- the fleet totals ---------------------------------------------------- #
def test_fleet_totals_with_a_gufo_and_radiance_payload(metrics_text) -> None:
"""The code needed no change: the aliases and the KV max already cover a radiance
payload; the seams sentence says the known asymmetries in its own words.
The numbers are re-derived from the two captures (volatile between re-captures).
"""
gufo_text = metrics_text("gufo")
rad_text = metrics_text("radiance")
gufo = engine_result("gufo", "http://g:1", gufo_text, RateEstimator(), now=1.0)
rad = engine_result("radiance", "http://r:8000", rad_text,
RateEstimator(), stats=_radiance_stats(), now=1.0)
totals = fleet_totals([gufo.payload, rad.payload])
assert totals["engines_total"] == 2
assert totals["engines_live"] == 2
# running/waiting: the sum of each capture's own counters (gufo's aliases + the gauges)
gp = Prom(gufo_text)
rp = Prom(rad_text)
assert totals["requests_active"] == pytest.approx(
(gp.scalar(["llamacpp:requests_processing"]) or 0.0)
+ (rp.scalar(["vllm:num_requests_running"]) or 0.0))
assert totals["requests_waiting"] == pytest.approx(
(gp.scalar(["llamacpp:requests_deferred"]) or 0.0)
+ (rp.scalar(["vllm:num_requests_waiting"]) or 0.0))
gufo_kv = gp.scalar(["llamacpp:kv_cache_usage_ratio"], agg="max") or 0.0
rad_kv = rp.scalar(["vllm:kv_cache_usage_perc"], agg="max") or 0.0
assert totals["kv_cache_usage_max"] == pytest.approx(max(gufo_kv, rad_kv)), \
"the true max: the lower engine's value must not hide the other's pressure"
assert totals["tokens_total"] == pytest.approx(
gufo.payload["metrics"]["tokens_total"]["value"]
+ rad.payload["metrics"]["tokens_total"]["value"])
# a fresh estimator: gufo's decode is n/a on its first poll, radiance's gauge wins
assert totals["decode_tokens_per_second"] == pytest.approx(
rad.payload["metrics"]["decode_tokens_per_second"]["value"])
assert totals["prompt_tokens_per_second"] == pytest.approx(
rad.payload["metrics"]["prompt_tokens_per_second"]["value"])
seams = totals["seams"]
assert "cache-hit" in seams, "gufo's prompt-counter asymmetry stays named"
assert "provenance" in seams, "the seams name the provenance labelling"
assert "vllm:prompt_tokens_total" in seams, \
"and the thing the dashboard deliberately does not claim"
tests/test_prom.py
+6 -2
View File
@@ -14,8 +14,12 @@ from dashboards.prom import Prom
def test_parses_counters_and_gauges_from_a_real_capture(metrics_text) -> None:
prom = Prom(metrics_text("gufo"))
assert prom.scalar(["llamacpp:tokens_predicted_total"]) == 877_336.0
assert prom.scalar(["llamacpp:kv_cache_usage_ratio"]) == 0.0
# the capture's counters and gauges are live numbers (volatile between
# re-captures): the pin is that they parse at all, in a sane range
generated = prom.scalar(["llamacpp:tokens_predicted_total"])
assert generated is not None and generated >= 0
usage = prom.scalar(["llamacpp:kv_cache_usage_ratio"])
assert usage is not None and 0.0 <= usage <= 1.0
assert prom.parse_errors == 0