phase: 34_radiance_llm_engine
Everything is verified and green. Final report: **Phase 34 final verification pass — complete** - Verified all 6 completed tasks' work against the live fleet and hermetic suite: `llm:radiance` job registered, `llm_engines()` registry drives 404 gate / history defaults / `/api/config` (additive, byte-identical gufo/vllm keys), radiance field map + `/stats` sidecar + 7-position spec curve, `/llm` strip/curve/rows. - **Defect found & fixed (pre-existing, task 05):** the vLLM-only sentence "vLLM exposes no per-slot JSON…" rendered on the **radiance** panel (`slotSection`'s non-gufo branch). Fix: vLLM line gated to `engine === 'vllm'`; radiance's panel slot section renders nothing (its session home is the strip) — `page-llm.js` + the served-byte pin in `tests/test_api.py` (now asserts the radiance branch is empty and the sentence can't leak). Browser-verified; re-shot `data/logs/34-llm-radiance-rest.png` (the 4.5-min wait found no in-flight request, so the busy shot remains the earlier pass's — its strip rows still stand; only the panel text predates the fix). - Tests: `uv run pytest` → **2031 passed, 19 skipped** (100%); `uv run pytest --cov` → **98.73%** (floor 94%; config 100%, llm 99%, collectors 98%, api 99%); `uv run ruff check src tests agent && uv run pyright` → clean; `node --check` OK; `uv pip list --outdated` → empty. - `uv run pytest -m live --live` → **13 passed, 2 skipped** (vLLM unset — correct; live-write opt-in — correct); the read-only radiance test passed. - Completion criteria: (1) engines `gufo`+`radiance`, `fleet.engines_live` 2, `config.llm.radiance.enabled` true — **pass (live curl)**; (2) unset → 404 no-dial + no job — **pass (spy-pinned, green)**; (3) fixtures present, both families (79 `radiance:` / 198 `vllm:` lines), drift pin — **pass**; (4) panel/curve/rows — **pass (re-shot rest screenshot)**; (5) live radiance test — **pass**; (6) suite/coverage/lint/`bash .agents/validate.sh` (exit 0) — **pass**; (7) docs — below; (8) gufo/vLLM behaviour unchanged — **pass (existing pins untouched)**. - Documentation: **README.md** full review — env table, LLM-engines section, architecture diagram, capture-script and `/llm` section all name radiance correctly; no inconsistencies found, no changes needed. **AGENTS.md** — the three task-06 fixes (fleet line, route row, module bullets) are in place and all other statements match reality; no changes needed. - Dev server healthy (`./scripts/dev-server.sh status`: all sources ok, `llm:radiance` answering); all changes left in the working tree for the harness. - Next pending phase: none — 34 was the last phase in `todo/`.
This commit is contained in:
@@ -0,0 +1,12 @@
|
||||
Everything is verified and green. Final report:
|
||||
|
||||
**Phase 34 final verification pass — complete**
|
||||
|
||||
- Verified all 6 completed tasks' work against the live fleet and hermetic suite: `llm:radiance` job registered, `llm_engines()` registry drives 404 gate / history defaults / `/api/config` (additive, byte-identical gufo/vllm keys), radiance field map + `/stats` sidecar + 7-position spec curve, `/llm` strip/curve/rows.
|
||||
- **Defect found & fixed (pre-existing, task 05):** the vLLM-only sentence "vLLM exposes no per-slot JSON…" rendered on the **radiance** panel (`slotSection`'s non-gufo branch). Fix: vLLM line gated to `engine === 'vllm'`; radiance's panel slot section renders nothing (its session home is the strip) — `page-llm.js` + the served-byte pin in `tests/test_api.py` (now asserts the radiance branch is empty and the sentence can't leak). Browser-verified; re-shot `data/logs/34-llm-radiance-rest.png` (the 4.5-min wait found no in-flight request, so the busy shot remains the earlier pass's — its strip rows still stand; only the panel text predates the fix).
|
||||
- Tests: `uv run pytest` → **2031 passed, 19 skipped** (100%); `uv run pytest --cov` → **98.73%** (floor 94%; config 100%, llm 99%, collectors 98%, api 99%); `uv run ruff check src tests agent && uv run pyright` → clean; `node --check` OK; `uv pip list --outdated` → empty.
|
||||
- `uv run pytest -m live --live` → **13 passed, 2 skipped** (vLLM unset — correct; live-write opt-in — correct); the read-only radiance test passed.
|
||||
- Completion criteria: (1) engines `gufo`+`radiance`, `fleet.engines_live` 2, `config.llm.radiance.enabled` true — **pass (live curl)**; (2) unset → 404 no-dial + no job — **pass (spy-pinned, green)**; (3) fixtures present, both families (79 `radiance:` / 198 `vllm:` lines), drift pin — **pass**; (4) panel/curve/rows — **pass (re-shot rest screenshot)**; (5) live radiance test — **pass**; (6) suite/coverage/lint/`bash .agents/validate.sh` (exit 0) — **pass**; (7) docs — below; (8) gufo/vLLM behaviour unchanged — **pass (existing pins untouched)**.
|
||||
- Documentation: **README.md** full review — env table, LLM-engines section, architecture diagram, capture-script and `/llm` section all name radiance correctly; no inconsistencies found, no changes needed. **AGENTS.md** — the three task-06 fixes (fleet line, route row, module bullets) are in place and all other statements match reality; no changes needed.
|
||||
- Dev server healthy (`./scripts/dev-server.sh status`: all sources ok, `llm:radiance` answering); all changes left in the working tree for the harness.
|
||||
- Next pending phase: none — 34 was the last phase in `todo/`.
|
||||
@@ -0,0 +1 @@
|
||||
ok
|
||||
@@ -0,0 +1,72 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 31%]
|
||||
...................................s.................................... [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 73%]
|
||||
............sssssssssssssss............................................. [ 77%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 94%]
|
||||
................................................sss..................... [ 98%]
|
||||
.................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 625 9 99% 229, 1278-1280, 1315-1318, 1387, 1395, 1451, 1454
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 256 4 98% 405, 409-410, 445
|
||||
src/dashboards/config.py 132 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 4 96% 126, 163, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 282 4 99% 467, 517, 554, 607
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4489 57 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.73%
|
||||
2031 passed, 19 skipped, 1 warning in 221.40s (0:03:41)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
Every completion criterion is verified. Task complete — here's the report:
|
||||
|
||||
## Task 01 — Radiance ground truth: complete
|
||||
|
||||
**Live capture (read-only, the documented procedure).** `set -a; . ./.env.dev; set +a && ./scripts/capture-fixtures.sh --with-llm` against `driveripper:8000` recorded all three endpoints, and the script printed the at-rest hint: `(requests[] was empty — re-run with a request in flight to record its shape)`.
|
||||
|
||||
### What changed
|
||||
- **`tests/fixtures/` (3 new files, verbatim with the script's header)**
|
||||
- `metrics_radiance.txt` — 425 lines: 79 `radiance:` native series + 195 `vllm:` compatibility lines (the vLLM-derived shape is part of the contract)
|
||||
- `stats_radiance.json` — header records `in_flight was 0 at capture (requests[] has 0 row(s))`
|
||||
- `models_radiance.json` — `juggernaut`, `owned_by: radiance`, `max_model_len 131072`, full capability set
|
||||
- **`scripts/capture-fixtures.sh`** — engine entries restructured to `<name> <url>` pairs; a `radiance` entry behind the same `--with-llm` gate, learned only from `RADIANCE_BASE_URL` (no hard-coded URLs); the loop captures all three radiance endpoints in one pass with the verbatim header, the dynamic at-rest hint, and the same never-overwrite-a-good-fixture temp-file rule. The gufo/vllm metrics and agent/UPS/smartctl passes are byte-for-byte unchanged.
|
||||
- **`src/dashboards/sources/llm.py`** — `RADIANCE_SERIES` (28 names: exactly the decision-3 payload set — native gauges, measured counters with vLLM-compat aliases, KV usage+max, queue depth/cap/admission, preemptions, prefix hit rate, overall + 7-label-set per-position spec acceptance, decode step gauges), `RADIANCE_HISTOGRAMS` (deliberately empty — this build's latency-histogram family is engine bookkeeping, documented in the comment), `RADIANCE_OPTIONAL` (66 names: 64 in-fixture non-cells mapped to the exclusion list + 2 cross-build aliases). Parse behaviour, `engine_result`, `fleet_totals`, `HISTORY_METRICS`, `SEVERITY_CELLS` untouched (task 03).
|
||||
- **`tests/test_llm_source.py`** — 5 hermetic tests on the real capture (conftest skip pattern + a `#`-header-stripping JSON loader): the two-way name-set drift pin (fixture↔map↔OPTIONAL, incl. stale-entry and typo detection), both-families-present, the `position="0".."6"` label set, the `/stats` shape (48-key top-level set pinned, `cards[]`, `requests[]`, 7-entry `draft_positions[]`), the `/v1/models` shape.
|
||||
- **`tests/test_deploy_scripts.py`** — 4 sandbox tests with a stub `curl` (canned engine, call log, no network): all three endpoints recorded verbatim with the header and exact dial order; the busy-run suppresses the hint; the no-engine gate names `RADIANCE_BASE_URL`; a dead engine never replaces a recorded capture.
|
||||
- **`.env.dev`** — `VLLM_BASE_URL` line → `RADIANCE_BASE_URL` with the decision comment (vLLM code/env var stay for other deployments).
|
||||
- **`README.md`** — the `--with-llm` line notes radiance's `/stats` + `/v1/models` pass. **`AGENTS.md`** — fleet fact only: driveripper provides `radiance :8000`. (`.env.dev.example` deliberately untouched — the env var only becomes real in task 02's config.)
|
||||
|
||||
### Notes
|
||||
- The live run also re-captured gufo/NUT/smartctl; the gufo engine had been rebooted (counters reset), which broke value-pinned completed tests, so I restored those non-radiance fixtures to their committed state — the working-tree diff is exactly the task scope. (The script's *agent* pass failed on self-signed-certificate verification: pre-existing drift since the fleet went mTLS — the script dials plain `curl` while the app's `AgentTls` carries the bundle; untouched per the no-behavior-change rule, agent fixtures intact.)
|
||||
- **Gate**: `1979 passed, 18 skipped` (the standard hermetic set), 0 failed; coverage **98.72%** (floor 94%); `ruff` + `pyright` clean; `uv pip list --outdated` empty; dev server restarted and healthy (hosts/disks/gateway/nut/gufo `ok`; `llm:vllm` gone by design until task 02 registers `llm:radiance`). No UI changes, so no screenshot was required.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 32%]
|
||||
.........................s.............................................. [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 72%]
|
||||
.......................................................ssssssssssssss... [ 75%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 93%]
|
||||
...................................................................sss.. [ 97%]
|
||||
..................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 625 9 99% 229, 1273-1275, 1310-1313, 1382, 1390, 1446, 1449
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 244 4 98% 387, 391-392, 409
|
||||
src/dashboards/config.py 128 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 181 1 99% 432
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4372 55 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.74%
|
||||
1979 passed, 18 skipped, 1 warning in 193.75s (0:03:13)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
All verification complete. The working tree is clean of artifacts, no phase/plan files touched.
|
||||
|
||||
## Report
|
||||
|
||||
**Implemented** (task 02 — registry + config for radiance; core code was already staged by a prior pass, this run completed the last enumeration site + all tests):
|
||||
- `config.py`: `radiance_url`/`radiance_api_key` (unset = "not configured"), `llm_engines()` registry — ordered `(key, display, url, api_key)` for all three engines, unset listed with empty url; `any_source_enabled` reads the registry
|
||||
- `collectors.py`: `llm_jobs` rewritten over the registry — one `llm:<key>` job per configured engine, `with_slots` for gufo, `with_stats` for radiance (no-op flag for task 04)
|
||||
- `api.py`: `/api/config`'s llm block now built from the registry (additive — gufo/vllm bytes untouched); the 404 gate, `/api/llm/history` defaults, and the summary's `llm.enabled` all read the registry; the 404 detail names all three variable names
|
||||
- `.env.dev.example`: `RADIANCE_BASE_URL` line added, vLLM line kept; `README.md` env-var table rows gained radiance in the same sentence shape
|
||||
- Tests: 7 new in `tests/test_config.py` (registry order/display/key-passthrough pins, unset-listed-with-empty-url, env surface, job + `any_source_enabled` both arms) and 5 new in `tests/test_api.py` (additive diff-pin with redaction, 404 gate in all four configurations, history defaults, summary enabled, unconfigured radiance never a job — no dial)
|
||||
|
||||
**Verification** — all green:
|
||||
- `uv run pytest`: 1990 passed, 18 skipped · `uv run pytest --cov`: 98.74% (config.py 100%, api.py 99%, collectors.py 98% — all >90%)
|
||||
- `uv run ruff check src tests agent` + `uv run pyright`: clean · `bash .agents/validate.sh`: exit 0
|
||||
- `uv run pytest -m live --live` (`.env.dev` sourced): 12 passed, 2 skipped (read-only)
|
||||
- Dev server restarted: `llm:radiance ok`; live `/api/llm/snapshot` lists `gufo` + `radiance` with `engines_live: 2`; `/api/config` serves the three keys in order; history defaults `['gufo', 'radiance']`
|
||||
- `python -c` criterion: set → one configured entry; unset → present with empty url; grep criterion: no `gufo_url`/`vllm_url` in collectors.py/api.py
|
||||
|
||||
**Docs**: README env-var rows updated (user-visible); AGENTS.md's `vLLM :8000` fleet fact was already fixed in the working tree by an earlier pass of this phase — nothing this task made stale, so it was left untouched.
|
||||
|
||||
**Decisions**: the `/api/config` block is the one site a prior pass had left hand-written; it now iterates the registry, keeping byte-identical output (key order `gufo`, `vllm`, `radiance`).
|
||||
|
||||
**Next pending task**: `03_the_radiance_field_map.md`
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 32%]
|
||||
..............................s......................................... [ 35%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 71%]
|
||||
..................................................................ssssss [ 75%]
|
||||
ssssssss................................................................ [ 78%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 96%]
|
||||
......sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 625 9 99% 229, 1278-1280, 1315-1318, 1387, 1395, 1451, 1454
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 244 4 98% 392, 396-397, 414
|
||||
src/dashboards/config.py 132 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 181 1 99% 432
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4376 55 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.74%
|
||||
1990 passed, 18 skipped, 1 warning in 196.84s (0:03:16)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
Everything is green. Final state:
|
||||
|
||||
**Implemented** (`src/dashboards/sources/llm.py` + `tests/test_llm_source.py`):
|
||||
- `RADIANCE_SERIES` reshaped to the exact decision-3 payload set (native throughput gauges, `vllm:`-shaped counters as the measured-Δ source, `gpu_cache_usage` resolving to the fixture name, `spec_accept_rate` reusing gufo's key, prefix gauge `gpu_prefix_cache_hit_rate`, per-position series); `RADIANCE_HISTOGRAMS` now the 3 vLLM-shaped latency families; `RADIANCE_OPTIONAL` recomputed as the exact symmetric difference (all 3 drift-pin conditions verified)
|
||||
- `parse_radiance_metrics()`: rule-2 vLLM-side precedence (gauge wins → per-label-set measured Δ with the refused-window `n/a`+`Δ` arm → `n/a`, never `0`, **no third estimate**); ordered `spec_positions` (absent positions stay absent); `tokens_total` from the two `vllm:` counters; `prefix_cache_hit_ratio` as the source-named, non-derived gauge; `/stats` passthrough `queue_cap` + computed card-0 `vram_used_pct`; severity through the **existing** `gpu_cache_usage` row, spec cell carries no verdict (pinned both ways)
|
||||
- `engine_result` 3-way dispatch + whitelisted `payload["stats"]` subset (renamed key → absent, never crash); `_sessions()` third arm (stats present: counts/cap/verbatim rows + at-rest sentence; stats missing: `/metrics` gauge fallback, cap `null`, honest detail); `HISTORY_METRICS` +4 additive candidates (per-position ratios snapshot-only); `fleet_totals` code unchanged, seams rewritten to name provenance labelling and the deliberate non-claim; `series_names` 3-way
|
||||
- 23 new tests, all fixture-driven
|
||||
|
||||
**Verification:** `uv run pytest` → **2013 passed, 18 skipped, 0 failed** · `uv run pytest --cov` → **98.70% total (floor 94%), `sources/llm.py` 99% (>90%)** · `ruff` clean, `pyright` 0 errors · `bash .agents/validate.sh` exit 0 · `uv run pytest -m live --live` → **12 passed, 2 skipped** · dev server restarted, `/api/health` ok, live radiance payload verified read-only against the running engine (gauge win, KV rated `ok`, spec unrated, 7 ordered positions, `engines_live: 2`, missing-stats fallback arm with a real in-flight request)
|
||||
|
||||
**Docs:** no README/AGENTS.md changes — no user-visible surface this task (payload is task 05's); the stale `vLLM :8000` fleet fact and the README `/llm` section are assigned to the phase final pass (task 06).
|
||||
|
||||
**Notable:** `tokens_total` is *not* an existing `HISTORY_METRICS` key (the task's "existing keys" prose was imprecise) — the spec's additions list is exactly the four candidates, so `tokens_total` stays a payload/fleet key only, pinned in the test. `vllm:gpu_cache_usage_perc` kept as an alias (vLLM-map idiom); the position series stays in the map for the drift pin but is never aggregated into a scalar.
|
||||
|
||||
**Next pending task:** `.agents/phases/todo/34_radiance_llm_engine/04_the_radiance_job.md`
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+74
@@ -0,0 +1,74 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 31%]
|
||||
..............................s......................................... [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 70%]
|
||||
..................................................................ssssss [ 74%]
|
||||
ssssssss................................................................ [ 77%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 95%]
|
||||
.............................sss........................................ [ 99%]
|
||||
............... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 625 9 99% 229, 1278-1280, 1315-1318, 1387, 1395, 1451, 1454
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 244 4 98% 392, 396-397, 414
|
||||
src/dashboards/config.py 132 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 282 4 99% 467, 517, 554, 607
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4477 58 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.70%
|
||||
2013 passed, 18 skipped, 1 warning in 198.88s (0:03:18)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
pytest-asyncio 1.4.1 1.4.1.post1 wheel
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
All done. Final report:
|
||||
|
||||
**Task 04 complete — the `llm:radiance` job with a best-effort `/stats` sidecar**
|
||||
|
||||
- **Implemented**
|
||||
- `collectors.fetch_engine`: real `with_stats` path beside the `/slots` loop — one unaliased `GET {base}/stats`; any `httpx.HTTPError`, status ≥ 400, non-JSON body, or non-dict JSON leaves `stats=None` → task 03's missing-stats session arm; no `UpstreamError` from the sidecar, no retry, no second dial; metrics-dial redaction (`strip_userinfo`) untouched
|
||||
- `llm_jobs` already fed `with_stats=key == "radiance"` (task 02); `grep` criterion holds — the flag reaches `fetch_engine` from `llm_jobs` only
|
||||
- `app.py` unchanged; wiring asserted in tests: `build_hub` hands `llm:radiance` the LLM interval, the LLM-span estimator, and `with_stats=True`
|
||||
- **Tests added** (`tests/test_collectors.py`, 13): job shape; hub wiring spy; per-engine sidecar pin (gufo → `/metrics`,`/slots`; vLLM → `/metrics` only; radiance → `/metrics`,`/stats` once); 4 degraded-sidecar arms (500 / non-JSON / JSON-list / connection error) keep the poll healthy; healthy `/stats` travels into `payload["stats"]`; dead `/metrics` is the `UpstreamError` and the sidecar is never consulted; tokenised-URL redaction pin; unconfigured engine never dialled (spy)
|
||||
- **Results**: `uv run pytest` → 2026 passed, 18 skipped · `uv run pytest --cov` → 98.71% (gate 94%; collectors.py 98%, new lines fully covered) · `ruff` + `pyright` clean · `uv pip list --outdated` empty (upgraded `pytest-asyncio` 1.4.1→1.4.1.post1, the gate's only advisory) · `bash .agents/validate.sh` exit 0 · `uv run pytest -m live --live` → 12 passed, 2 skipped
|
||||
- **Live check**: dev-server restarted; `status` shows `llm:radiance ok`; authenticated `/api/llm/snapshot` carries the radiance payload with the `stats` block (`source: radiance /stats`, cap 32, KV severity, native-gauge decode)
|
||||
- **Docs**: no doc impact — README already documents `RADIANCE_BASE_URL` (task 02); no AGENTS.md statement went stale
|
||||
- **Next pending task**: `05_the_llm_page_learns_radiance.md`
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+72
@@ -0,0 +1,72 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 31%]
|
||||
..............................s......................................... [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 73%]
|
||||
.......ssssssssssssss................................................... [ 77%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 95%]
|
||||
..........................................sss........................... [ 98%]
|
||||
............................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 625 9 99% 229, 1278-1280, 1315-1318, 1387, 1395, 1451, 1454
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 256 4 98% 405, 409-410, 445
|
||||
src/dashboards/config.py 132 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 282 4 99% 467, 517, 554, 607
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4489 58 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.71%
|
||||
2026 passed, 18 skipped, 1 warning in 196.20s (0:03:16)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
All green. Final state verified:
|
||||
|
||||
**Report — phase 34, task 05 (`/llm` learns radiance)**
|
||||
|
||||
**Implemented**
|
||||
- `page-llm.js`: `only:` scopes extended (`ttft`/`tpot`/`queue_mean`/`running` → `['vllm','radiance']`, `spec_accept` → `['gufo','radiance']`); 4 radiance-only strip cells (`queue cap`, `build`, `uptime`, `VRAM (card 0)` — GiB = 2³⁰, null-guards → `n/a` never `0`); 7-bar spec-acceptance curve (hand-rolled SVG, neutral `--text-faint` ink — no verdict tokens, `role="img"` + payload-computed `aria-label` with order-based connector word, absent positions stay absent); `requests[]` rows beside the counts (columns = union of published keys, numbers only-when-numbers, strings verbatim, no invented keys) with the API's own count-otherwise sentence in the rest state; `SLOT_ENGINES` stays `['gufo']`; config banner now three-engine; fleet note payload-driven from `fleet.seams`
|
||||
- `llm.html`: empty state names all three `*_BASE_URL` variables
|
||||
- `dash.css`: minimal rules for the curve/table/note (existing tokens only)
|
||||
- `test_api.py`: 4 new tests (curve neutral+named, cells never-0, rows as-is + rest state, fixture-driven payload through real `engine_result` — rest/in-flight/dead-sidecar arms) + extended scope, row-path (with `DOCUMENTED_ABSENCE` for this build's missing queue-time histogram), chart-naming, and empty-state pins
|
||||
|
||||
**Results**
|
||||
- `uv run pytest -p no:warnings`: **2030 passed, 18 skipped** (baseline 2026 + 4 new; skips are opt-in live/store_perf, unchanged) — 100% green
|
||||
- `bash .agents/validate.sh`: **exit 0** — coverage 98.71% (gate 94%), ruff clean, pyright 0 errors, node syntax clean
|
||||
- Browser (real Chromium, live dev server, both engines answering): `data/logs/34-llm-radiance-rest.png` — fleet block "2 of 2 engines", radiance strip (KV 74% the only verdict colour, native Δ2.0s throughputs, cap 32 / build 1.15.1 / uptime 3.1h / VRAM 31.8/31.9 GiB), 7-bar curve labelled "74% at position 0 rising to 79% at position 6", count-otherwise sentence at rest, TTFT/TPOT rows on the radiance panel; no in-flight request during the pass, so no second screenshot
|
||||
|
||||
**Docs**: README route table + `/llm` page section updated for radiance; AGENTS.md one stale statement fixed (`/llm` source cell now names radiance `/metrics` + `/stats`) — no other AGENTS.md changes (final pass re-checks)
|
||||
|
||||
**Decisions**: `queue_mean` kept in radiance scope per the task (this build publishes no queue-time histogram → `n/a` by absence, never back-filled; documented as an explicit test exemption, since the series is vLLM-standard for other builds). Browser verification caught a real bug the pins couldn't (undeclared `stats` in `renderStripCard`), fixed.
|
||||
|
||||
**Next pending task**: `06_docs_and_live_verify` (live read-only test, fixture re-capture, README/AGENTS.md pass, live-data screenshots).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+72
@@ -0,0 +1,72 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 31%]
|
||||
..................................s..................................... [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 73%]
|
||||
...........ssssssssssssss............................................... [ 77%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 94%]
|
||||
..............................................sss....................... [ 98%]
|
||||
................................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 625 9 99% 229, 1278-1280, 1315-1318, 1387, 1395, 1451, 1454
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 256 4 98% 405, 409-410, 445
|
||||
src/dashboards/config.py 132 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 282 4 99% 467, 517, 554, 607
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4489 58 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.71%
|
||||
2030 passed, 18 skipped, 1 warning in 202.00s (0:03:22)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
All working-tree state accounted for (the non-mine changes belong to this phase's earlier tasks / prior phases; the two `TODO.md` items are operator notes for future phases). The task is complete — final state is green across the board.
|
||||
|
||||
**Report — phase 34, task 06 (docs & live verify)**
|
||||
|
||||
- **Live test**: `tests/test_live_fleet.py::test_radiance_answers_the_pinned_shape` — skips when `RADIANCE_BASE_URL` unset (proven: `env -u` run skips), dials through production `collectors.fetch_engine` (LLM-cadence estimator, `with_stats`), asserts shape only: `engine == "radiance"`, `stats` whitelist, ordered 7-position spec curve, one of the two session arms, every cell finite-number-or-`None`, `unparsed == 0`
|
||||
- **Fixture re-capture**: `capture-fixtures.sh --with-llm` re-run ×2 (second pass caught gufo idle per the slots test's own instruction); diff read — volatile values only; 2 new `gufo_*` lookup series (gufo name set is not pinned; radiance name-set pin passes unchanged); `requests[]` still empty at capture (engine idle) — in-flight row shape covered by the hermetic arm test + live busy screenshot
|
||||
- **Docs**: README radiance engine paragraph + architecture diagram; `.env.example` gained the radiance vars (was the one gap); AGENTS.md — the four prescribed stale spots now fixed (config.py paragraph + `sources/llm.py` bullet by me; fleet line + route line by task 02), nothing added beyond them
|
||||
- **Bug found & fixed** (browser verification): the radiance `.reqs` table's min-content (~950 px) stretched the strip body's grid column — every value cell painted past the card and was clipped; `dash.css` `.reqs { min-width: 0; overflow-x: auto }`, pinned by `test_the_radiance_reqs_table_scrolls_instead_of_stretching_its_card`, re-verified in Chromium with a request in flight
|
||||
- **Results**: `uv run pytest` 2031 passed / 19 skipped · `--cov` 98.71% (floor 94%) · ruff + pyright clean · `uv pip list --outdated` empty · `bash .agents/validate.sh` exit 0 · live suite 13 passed / 2 expected skips · dev server restarted, `/api/health` + status `llm:radiance ok`
|
||||
- **Screenshots** (`data/logs/`): `34-llm-radiance-rest.png` (task 05), `34-llm-radiance-busy.png` (in-flight, fixed layout, KV 93% fail), `34-overview-radiance-alert.png` (radiance alert row "KV cache 76% (warn ≥75)"); measurement scripts kept as `34-task06-*.mjs`
|
||||
- **Deviations**: the port-8090 locked-decision line ("8000 is vLLM") left untouched — the task's completion criterion enumerates exactly the four AGENTS.md spots and says nothing beyond them
|
||||
|
||||
Next: none — task 06 is phase 34's final task; the phase's completion gate is met.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+72
@@ -0,0 +1,72 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 31%]
|
||||
...................................s.................................... [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 73%]
|
||||
............sssssssssssssss............................................. [ 77%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 94%]
|
||||
................................................sss..................... [ 98%]
|
||||
.................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 625 9 99% 229, 1278-1280, 1315-1318, 1387, 1395, 1451, 1454
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 256 4 98% 405, 409-410, 445
|
||||
src/dashboards/config.py 132 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 613 10 98% 339, 588, 601-602, 652, 785, 1089, 1140, 1207, 1569
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 282 4 99% 467, 517, 554, 607
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4489 58 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.71%
|
||||
2031 passed, 19 skipped, 1 warning in 198.19s (0:03:18)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
@@ -59,7 +59,11 @@ export HOST_TARGETS="driveripper=https://driveripper.example.net:9120 deskwork=h
|
||||
|
||||
# The engines are dialled directly on their hostnames, like the agents.
|
||||
export GUFO_BASE_URL=http://deskwork.example.net:8731
|
||||
# vLLM stays a supported engine for other deployments; on this dev fleet the :8000 box is
|
||||
# radiance now (it replaced vLLM there) — so the two lines point at one box on purpose,
|
||||
# and a deployment running a real vLLM simply drops the radiance line.
|
||||
export VLLM_BASE_URL=http://driveripper.example.net:8000
|
||||
export RADIANCE_BASE_URL=http://driveripper.example.net:8000
|
||||
|
||||
# The UPS source: the real upsd, dialled directly. Do not assume NUT's stock 3493 —
|
||||
# nonstandard ports are common and a refused connection then reads as "upsd is broken"
|
||||
|
||||
+6
-1
@@ -96,13 +96,18 @@ GATEWAY_TIMEOUT_SECONDS=10
|
||||
# GATEWAY_WRITE_TIMEOUT_SECONDS=35
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# LLM dashboard: either engine is optional — unset means "not deployed here"
|
||||
# LLM dashboard: each engine is optional — unset means "not deployed here"
|
||||
# and its half of the page disappears instead of showing a dead panel.
|
||||
# --------------------------------------------------------------------------- #
|
||||
GUFO_BASE_URL=
|
||||
GUFO_API_KEY=
|
||||
VLLM_BASE_URL=
|
||||
VLLM_API_KEY=
|
||||
# Radiance: a vLLM-derived engine. Its /stats sidecar (the session source, the
|
||||
# queue cap) is fetched best-effort by the poll — a dead sidecar never fails the
|
||||
# /metrics poll, and the queue cap reads n/a, never 0, while it is absent.
|
||||
RADIANCE_BASE_URL=
|
||||
RADIANCE_API_KEY=
|
||||
# The LLM Δ window IS this interval: the llm jobs get their own estimator spanned by
|
||||
# LLM_INTERVAL_SECONDS (phase 23), so a slot start reads steady within two polls instead of
|
||||
# ramping over RATE_WINDOW_SECONDS. Slowing this slows the window with it.
|
||||
|
||||
@@ -26,7 +26,7 @@ file — plus an overview page:
|
||||
| `/` | Is anything wrong, and where do I look first? | all five, in one request |
|
||||
| `/system` | CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host |
|
||||
| `/disks` | Are any physical disks failing, and which — health, temp, wear | the same agents, `smartctl -i -A -H` per whole drive |
|
||||
| `/llm` | Tokens/s, active sessions, queue depth, KV pressure, latency | gufo `/metrics` + `/slots`, vLLM `/metrics` |
|
||||
| `/llm` | Tokens/s, active sessions, queue depth, KV pressure, latency | gufo `/metrics` + `/slots`, vLLM `/metrics`, radiance `/metrics` + `/stats` |
|
||||
| `/ups` | Battery health, runtime left, power events, test buttons | NUT `upsd` (our own wire client; port **3051**, not 3493) |
|
||||
| `/gateway` | Which domains on which port, and where each goes | the same agent's `haproxy` + `nginx` sections joined to `/proc/net/tcp{,6}` on the one `GATEWAY_HOST` host (the page needs agent **≥ 1.2.0**; the route switch's `#off` marker read is **1.4.0**) |
|
||||
|
||||
@@ -164,7 +164,9 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th
|
||||
agent's: `ROUTE_MARKER` / `split_marker` / `config_line_index` — `disabled` on
|
||||
`server`/`proxy_pass`/`map` nodes; a marked line stays in the document).
|
||||
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
|
||||
env var always wins. Safe defaults except the sources, which default to unset. The
|
||||
env var always wins. Safe defaults except the sources, which default to unset
|
||||
(`HOST_TARGETS` and the engine URLs `GUFO_BASE_URL` / `VLLM_BASE_URL` /
|
||||
`RADIANCE_BASE_URL`, each with its optional `*_API_KEY`). The
|
||||
gateway source is enabled only when `GATEWAY_HOST` names a `HOST_TARGETS` entry
|
||||
(default `gateway`). The write's two settings are beside it: `GATEWAY_ALLOW_WRITES`
|
||||
(default **off**, the NUT flag's truthy rule) and `GATEWAY_WRITE_TIMEOUT_SECONDS`
|
||||
@@ -214,8 +216,9 @@ One line each — the load-bearing behaviour is in the bullet, the rest is in th
|
||||
same shape from a hand-deleted line still warns; the switch-off tally is published
|
||||
as `/api/summary`'s `disabled` field. No `severity` anywhere in the payload (a busy
|
||||
proxy is the job).
|
||||
- `sources/llm.py` — gufo + vLLM field maps, measured Δ rates **diffed per label set**,
|
||||
prefill speed, the session list from gufo `/slots`. A refused estimator window is still
|
||||
- `sources/llm.py` — the gufo / vLLM / radiance field maps, measured Δ rates **diffed per label set**,
|
||||
prefill speed, the session list from gufo `/slots` (radiance's from its `/stats`
|
||||
sidecar). A refused estimator window is still
|
||||
an answer: the cell keeps its `source` and an honest `Δ<window>` beside a `null`.
|
||||
- `sources/nut.py` — the NUT wire client, `assess()` (explicit checks, not a mystery
|
||||
score), `SELF_TEST_OUTCOMES`, the `COMMANDS` catalog with risk tiers and
|
||||
@@ -296,7 +299,7 @@ Treat the scheme as a public interface of the history API.
|
||||
|
||||
| Host | ssh | What it gives |
|
||||
| ------------- | ------------- | ------------------------------------------------------------------------- |
|
||||
| `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards. 16 whole disks; the agent unit runs `User=root` with `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's unprivileged-passthrough gate; an empty bounding set is capability-less, `CapEff 0`) |
|
||||
| `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, radiance `:8000` (replaced vLLM there, 2026-10-24; the vLLM code and env var stay for other deployments), **upsd**, the dashboards. 16 whole disks; the agent unit runs `User=root` with `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's unprivileged-passthrough gate; an empty bounding set is capability-less, `CapEff 0`) |
|
||||
| `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` |
|
||||
| `gateway` | `gateway-ops` | the router, the fleet's ingress: haproxy (`/etc/haproxy/haproxy.cfg`, six frontends → six backends) + nginx (`/etc/nginx/nginx.conf` plus `conf.d/` vhosts and `stream.d/`'s `map` on `:443`); runs agent **1.4.0** with the `dashboards-routectl` helper units |
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ file:
|
||||
| `/` | Is anything wrong, and which dashboard do I need? | all five, in one request (`/api/summary`) |
|
||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
||||
| `/disks` | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount); a host whose agent has no `smartctl` is not a host on this page |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics`, radiance `/metrics` plus its `/stats` as a best-effort sidecar (its answer to `/slots`) — read-only |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
||||
| `/gateway` | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, plus **1.3.0**'s `resolver` section to put a number on a route whose target the config writes as a *name* (without it those lines are drawn and dashed), plus **1.4.0**'s `#off` marker read to show a route the switch has turned off **or back on** (without it an off route is drawn on, because the marker is a comment a 1.3.0 agent does not read) — an off route stays drawn and counted, its row simply reads `enabled: false` (its id gains a `#off` suffix), and the config-integrity warnings the marker *caused* (a backend emptied of its last `server`, a map entry pointed at the switch's sink) are read as a chosen state while the same shape from a hand-deleted line is still a warning — and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
|
||||
|
||||
@@ -770,7 +770,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
||||
| Variable | Meaning |
|
||||
| ------------------------------------- | -------------------------------------------------------------- |
|
||||
| `HOST_TARGETS` | `name=url` per host running the agent. The shape a new host gets is `https://host:9120` with **no userinfo** — a certificate-authenticated agent, verified against this app's own root CA and presented with its client certificate, with a revoked host refused before the dial (`certs.py`, `collectors.AgentTls`). The compatibility shape is a token-mode agent's URL userinfo (`http://TOKEN@host:9120`), moved into an `Authorization` header by `parse_targets` so it never reaches `/api/config` or an error message |
|
||||
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
||||
| `GUFO_BASE_URL` / `VLLM_BASE_URL` / `RADIANCE_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
||||
| `RATE_WINDOW_SECONDS` | the **host's** Δ window (default 10) — bytes/sec and jiffies→CPU %, measured over what a 10 s poll can fill; each value carries the window it actually got, and `/system`'s footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one `n/a` instead of a spike, and a rate 8× above the last one is held until a window measured from a **different** sample confirms it (phase 13). The LLM rates do **not** use it: they are measured over `LLM_INTERVAL_SECONDS` (phase 23) |
|
||||
| `LLM_INTERVAL_SECONDS` | the `/llm` poll cadence (default 2 s) **and** the Δ window behind every token rate — one estimator per measuring cadence, so a slot that starts reads steady within two polls instead of ramping over five. Slowing the poll slows the window with it; there is no separate LLM window knob, and a window narrower than the interval would answer `n/a` every poll |
|
||||
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
|
||||
@@ -779,7 +779,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
||||
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
|
||||
| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and a wedged SMART pass must not read as a dead host |
|
||||
| `GATEWAY_HOST` | which `HOST_TARGETS` name is the ingress the `/gateway` page watches (default `gateway`). It has no URL of its own — the source reads the same agent `/snapshot` the system page reads, projected onto agent 1.2.0's `haproxy`/`nginx`/`sockets` sections — so a name that is not in `HOST_TARGETS` means "no ingress to watch", and the source is unconfigured rather than red. The page refetches itself every 15 s in the browser and has no `POLL_MS_*` knob, because the payload only moves when that one job re-reads the ingress |
|
||||
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
||||
| `GUFO_API_KEY` / `VLLM_API_KEY` / `RADIANCE_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
||||
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
|
||||
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
||||
| `NUT_ALLOW_ACTIONS` | `1` enables the UPS write endpoint that dials **anything** — the `/api/auth/*` and `/api/certs/*` writes touch only this process and its `meta` table |
|
||||
@@ -804,10 +804,10 @@ installed.
|
||||
## Architecture
|
||||
|
||||
```
|
||||
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
||||
agent/host_agent.py (each host) gufo / vLLM / radiance upsd (NUT)
|
||||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
|
||||
│ (three readers: system 10s, │ + /slots (gufo) │
|
||||
│ disks 5m, gateway 30s) │ │
|
||||
│ disks 5m, gateway 30s) │ + /stats (radiance) │
|
||||
▼ ▼ ▼
|
||||
src/dashboards/collectors.py ────────────────────────────────────────────┐
|
||||
│ parse + Δcounter/Δt (rates.py) │
|
||||
@@ -1132,16 +1132,28 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
||||
duplicate its own cards), and each host's **Charts** disclosure behaves exactly like `/system`'s:
|
||||
collapsed, built only while open, and fed by one `/api/disks/history` request per host per
|
||||
refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive it has.
|
||||
`/llm` opens with
|
||||
the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, the note
|
||||
*sums only engines answering now*, then **running / waiting / prompt tok/s / decode tok/s /
|
||||
total tokens** — and every number on it is `fleet_totals()`, so the page sums nothing itself.
|
||||
`/llm` watches up to three engines, each configured by its own `*_BASE_URL` (unset means
|
||||
*not deployed here*, and its card and panel disappear — never a dead half): **gufo**
|
||||
(a llama.cpp-family server — `/metrics` plus the per-slot `/slots` JSON), **vLLM**
|
||||
(`/metrics` only, which is why its panel says *exposes no per-slot JSON*), and
|
||||
**radiance** — a vLLM-derived engine (this fleet's `driveripper:8000`) that answers
|
||||
`/metrics` (its `radiance:` native windowed gauges **win** the throughput cells when
|
||||
present, falling back to the measured `Δ<window>` of its vLLM-shaped counters when
|
||||
absent), `/stats` — its answer to `/slots`: fetched best-effort beside the `/metrics`
|
||||
poll, so a dead sidecar degrades the session arm and the queue cap to `n/a`, never `0`,
|
||||
and never fails the poll — and `/v1/models`. `/llm` opens with
|
||||
the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, a note
|
||||
that is the API's own `fleet.seams` sentence (the shipped line, *sums only engines
|
||||
answering now*, is the fallback for a payload that predates it), then **running / waiting /
|
||||
prompt tok/s / decode tok/s / total tokens** — and every number on it is `fleet_totals()`,
|
||||
so the page sums nothing itself.
|
||||
A dead engine keeps its last-good numbers in its own (stale-labelled) panel but is dropped
|
||||
from a total presented as current: the tag counts what answered, names the missing engine in
|
||||
its tooltip and dims while the sums are partial. Nothing on the block is coloured —
|
||||
`severity.py` rates no throughput and no queue depth, because 0 tok/s on an idle fleet is
|
||||
healthy — the dot is reachability, and `fleet.seams` (how the two engines count prompt
|
||||
tokens differently) is a tooltip on the prompt sum rather than another line of ink. Below
|
||||
healthy — the dot is reachability, and `fleet.seams` (how the engines count prompt tokens
|
||||
and where their throughputs come from) is the note under the count and a tooltip on the
|
||||
prompt sum. Below
|
||||
two configured engines the block is hidden: a cross-engine sum of one engine duplicates that
|
||||
engine's panel. Below the fleet block — with the page's shared `.85rem` gap, as everywhere else
|
||||
in the section stack — sits the **stats strip**: one card per configured
|
||||
@@ -1149,12 +1161,23 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
||||
fill and a breach's text take their colour from the payload's `severity` and nothing else),
|
||||
then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ2.0s` at the default cadence —
|
||||
the LLM window is one poll interval, phase 23 — scraped, estimated), `total tokens`, the sessions this engine reports (`slots busy` counted from
|
||||
gufo's `/slots`, or `running`/`waiting` for vLLM, plus `waiting`), `prefix cache hit`,
|
||||
`spec accept` (gufo), `queue time` with its `p95 …` foot (vLLM) — and red alarm chips for
|
||||
preemptions / GPU-context losses that exist only while non-zero. Then the engine panels
|
||||
gufo's `/slots`, or `running`/`waiting` for vLLM and radiance, plus `waiting`), `prefix cache hit`,
|
||||
`spec accept` (gufo and radiance — radiance's gauge reuses gufo's key, the same 0-1
|
||||
scale), `queue time` with its `p95 …` foot (vLLM; the dev radiance build publishes no
|
||||
queue-time histogram, so its cell is `n/a` by absence, never back-filled) — and red alarm chips for
|
||||
preemptions / GPU-context losses that exist only while non-zero. The radiance card adds
|
||||
its own four cells — `queue cap` (the admission ceiling, `n/a` while `/stats` is not
|
||||
answering, never `0`), `build` (the engine's own version string), `uptime`, and `VRAM
|
||||
(card 0)` (the engine's own view of the card in GiB = 2³⁰; `/system`'s GPU row is the
|
||||
agent's view of the same card) — and two more things: a **7-bar spec-acceptance curve**
|
||||
(one bar per position the engine published, in order, height ∝ the 0-1 ratio, drawn in
|
||||
neutral ink because an efficiency number is not a verdict) and its **per-request rows**
|
||||
beside the counts (the engine's own fields, as-is — at rest the table is hidden and the
|
||||
API's own count-otherwise sentence takes their place). Then the engine panels
|
||||
hold the detail the strip does not print: two big
|
||||
throughput numbers (decode, prompt — uncoloured, because 0 tok/s on an idle server is
|
||||
healthy) with their provenance tag, and the latency rows (TTFT, TPOT) each with a `p95 …`
|
||||
healthy) with their provenance tag, and the latency rows (TTFT, TPOT — vLLM and radiance;
|
||||
gufo's build does not publish them) each with a `p95 …`
|
||||
foot line and its own sparkline over the selected window. Under the gufo panel's
|
||||
**`sessions (GET /slots)`** label sits the **slot grid**: one card per parallel slot, keyed by
|
||||
the slot's own id, created once and patched in place so a busy slot never flickers — request
|
||||
@@ -1312,6 +1335,7 @@ DASH_LLM_WATCH_REPLAY=data/logs/llm-window-watch.tsv \
|
||||
uv run ruff check src tests agent && uv run pyright # lint + types
|
||||
./scripts/capture-fixtures.sh # re-record ground truth: host agents, UPS replies, gufo /slots
|
||||
./scripts/capture-fixtures.sh --with-llm # ... and the engines' /metrics while they are up
|
||||
# (radiance also records /stats + /v1/models in the same pass)
|
||||
SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh
|
||||
# ... and smartctl per physical disk: plain smartctl,
|
||||
# `sudo -n` if that is denied; scrubs serials/WWNs
|
||||
|
||||
@@ -33,9 +33,13 @@ for entry in ${targets//,/ }; do # space or comma separated, like config.parse
|
||||
done
|
||||
# Engine URLs come from the same env too, so this script carries no deployment
|
||||
# detail of its own — and never records a tokenised URL into a fixture by accident.
|
||||
# Engine entries are `<name> <url>` pairs; the name is the fixture suffix and the env
|
||||
# var that dials the engine (gufo / vllm / radiance — a vLLM derivative that also
|
||||
# answers /stats and /v1/models, captured in the same pass).
|
||||
ENGINES=()
|
||||
[ -n "${GUFO_BASE_URL:-}" ] && ENGINES+=("gufo:${GUFO_BASE_URL%/}/metrics")
|
||||
[ -n "${VLLM_BASE_URL:-}" ] && ENGINES+=("vllm:${VLLM_BASE_URL%/}/metrics")
|
||||
[ -n "${GUFO_BASE_URL:-}" ] && ENGINES+=("gufo ${GUFO_BASE_URL%/}/metrics")
|
||||
[ -n "${VLLM_BASE_URL:-}" ] && ENGINES+=("vllm ${VLLM_BASE_URL%/}/metrics")
|
||||
[ -n "${RADIANCE_BASE_URL:-}" ] && ENGINES+=("radiance ${RADIANCE_BASE_URL%/}/metrics")
|
||||
|
||||
failed=0
|
||||
|
||||
@@ -211,17 +215,78 @@ fi
|
||||
|
||||
if [ "${1:-}" = "--with-llm" ]; then
|
||||
if [ ${#ENGINES[@]} -eq 0 ]; then
|
||||
echo " ! no GUFO_BASE_URL / VLLM_BASE_URL set (see .env.dev) — nothing to capture"
|
||||
echo " ! no GUFO_BASE_URL / VLLM_BASE_URL / RADIANCE_BASE_URL set (see .env.dev) — nothing to capture"
|
||||
failed=1
|
||||
fi
|
||||
for entry in "${ENGINES[@]}"; do
|
||||
name=${entry%%:*}; url=${entry#*:}
|
||||
name=${entry%% *}; url=${entry#* }
|
||||
if curl -fsS --max-time 20 "$url" -o "$FIX/metrics_${name}.txt"; then
|
||||
# Radiance's capture carries the script's verbatim header, like the other fixtures —
|
||||
# gufo's and vLLM's metrics files stay raw, as before.
|
||||
if [ "$name" = "radiance" ]; then
|
||||
{ echo "# Verbatim /metrics reply from the radiance engine, recorded by"
|
||||
echo "# ./scripts/capture-fixtures.sh — the engine's own Prometheus text"
|
||||
echo "# format: the radiance: native series + the vllm: compatibility family."
|
||||
echo "# Volatile values (counters, gauges) move every poll, so a diff here"
|
||||
echo "# on re-capture is expected; the contract asserts the series-name set"
|
||||
echo "# against RADIANCE_SERIES/RADIANCE_HISTOGRAMS/RADIANCE_OPTIONAL in"
|
||||
echo "# sources/llm.py — never the numbers (the nut_*.vars precedent)."
|
||||
cat "$FIX/metrics_${name}.txt"; } > "$FIX/.metrics_radiance.txt.tmp" &&
|
||||
mv "$FIX/.metrics_radiance.txt.tmp" "$FIX/metrics_${name}.txt"
|
||||
fi
|
||||
echo " ✓ metrics_${name}.txt ($(wc -l <"$FIX/metrics_${name}.txt") lines)"
|
||||
else
|
||||
echo " ✗ metrics_${name}.txt — $url unreachable (VPN? engine down?)"
|
||||
failed=1
|
||||
fi
|
||||
# radiance is a vLLM derivative and answers two more read-only endpoints: /stats
|
||||
# (the engine's own JSON snapshot: cards[], requests[], draft_positions[], ...)
|
||||
# and /v1/models (the model's capabilities — the only place the engine states what
|
||||
# it can do). Both are recorded verbatim in the same pass the gufo /slots JSON
|
||||
# document is, with the same verbatim header; the hermetic shape tests in
|
||||
# tests/test_llm_source.py need them on disk.
|
||||
if [ "$name" = "radiance" ]; then
|
||||
base=${url%/metrics}
|
||||
stats_tmp="$FIX/.stats_radiance.json.tmp"
|
||||
models_tmp="$FIX/.models_radiance.json.tmp"
|
||||
if curl -fsS --max-time 10 "${base}/stats" -o "$stats_tmp"; then
|
||||
info=$(python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print(d.get('in_flight', 0), len(d.get('requests', [])))" "$stats_tmp" 2>/dev/null || echo '? 0')
|
||||
inflight=${info%% *}; rows=${info##* }
|
||||
{ echo "# Verbatim /stats reply from the radiance engine, recorded by"
|
||||
echo "# ./scripts/capture-fixtures.sh — one JSON document, the engine's own"
|
||||
echo "# shape. Volatile values (uptime_s, now_ms, the counters, kv_*) move"
|
||||
echo "# every poll, so a diff here on re-capture is expected; the contract"
|
||||
echo "# asserts the top-level key set and the cards[]/requests[]/"
|
||||
echo "# draft_positions[] shape — never those numbers (the nut_*.vars"
|
||||
echo "# precedent)."
|
||||
echo "# in_flight was ${inflight} at capture (requests[] has ${rows} row(s))."
|
||||
cat "$stats_tmp"; } > "$FIX/.stats_radiance.json.tmp2" &&
|
||||
mv "$FIX/.stats_radiance.json.tmp2" "$FIX/stats_radiance.json"
|
||||
rm -f "$stats_tmp"
|
||||
if [ "$inflight" = "0" ]; then
|
||||
echo " (requests[] was empty — re-run with a request in flight to record its shape)"
|
||||
fi
|
||||
echo " ✓ stats_radiance.json ($(wc -c <"$FIX/stats_radiance.json") bytes)"
|
||||
else
|
||||
rm -f "$stats_tmp" # never overwrite a good fixture with a dead engine's empty file
|
||||
echo " ✗ stats_radiance.json — ${base}/stats unreachable"
|
||||
failed=1
|
||||
fi
|
||||
if curl -fsS --max-time 10 "${base}/v1/models" -o "$models_tmp"; then
|
||||
{ echo "# Verbatim /v1/models reply from the radiance engine, recorded by"
|
||||
echo "# ./scripts/capture-fixtures.sh — one JSON document. The model id and"
|
||||
echo "# max_model_len are stable; created is an epoch timestamp that moves."
|
||||
echo "# The contract asserts the object/data shape — never the numbers."
|
||||
cat "$models_tmp"; } > "$FIX/.models_radiance.json.tmp2" &&
|
||||
mv "$FIX/.models_radiance.json.tmp2" "$FIX/models_radiance.json"
|
||||
rm -f "$models_tmp"
|
||||
echo " ✓ models_radiance.json ($(wc -c <"$FIX/models_radiance.json") bytes)"
|
||||
else
|
||||
rm -f "$models_tmp" # never overwrite a good fixture with a dead engine's empty file
|
||||
echo " ✗ models_radiance.json — ${base}/v1/models unreachable"
|
||||
failed=1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
else
|
||||
echo " (skipped engine /metrics captures — pass --with-llm when the engines are reachable)"
|
||||
|
||||
+22
-13
@@ -691,15 +691,19 @@ def common_router() -> APIRouter:
|
||||
"target_problems": list(settings.host_target_problems),
|
||||
},
|
||||
"llm": {
|
||||
# `strip_userinfo` on the engine URLs too: `GUFO_API_KEY` is the documented
|
||||
# way to authenticate, but a URL that carries credentials must never be echoed
|
||||
# to a browser whatever the config says (invariant 6).
|
||||
"gufo": {"enabled": bool(settings.gufo_url),
|
||||
"base_url": strip_userinfo(settings.gufo_url),
|
||||
"auth": bool(settings.gufo_api_key)},
|
||||
"vllm": {"enabled": bool(settings.vllm_url),
|
||||
"base_url": strip_userinfo(settings.vllm_url),
|
||||
"auth": bool(settings.vllm_api_key)},
|
||||
# The registry is the one place the engine list is decided (phase 34 task 02):
|
||||
# every key this block serves — `gufo`, `vllm` and the additive `radiance` —
|
||||
# is one entry in `settings.llm_engines()`, so a deployment that runs two of
|
||||
# the three gets exactly the two old keys' bytes plus `radiance` at
|
||||
# `enabled: false`, and the next engine is a one-line addition there. Each key
|
||||
# keeps the additive-only shape `{enabled, base_url, auth}` in the registry's
|
||||
# stable order. `strip_userinfo` on every URL: the `*_API_KEY` variables are
|
||||
# the documented way to authenticate, but a URL that carries credentials must
|
||||
# never be echoed to a browser whatever the config says (invariant 6).
|
||||
**{key: {"enabled": bool(url),
|
||||
"base_url": strip_userinfo(url),
|
||||
"auth": bool(api_key)}
|
||||
for key, _display, url, api_key in settings.llm_engines()},
|
||||
"interval_seconds": settings.llm_interval,
|
||||
"poll_ms": settings.poll_ms_llm,
|
||||
# The Δ window these engines' rates are actually measured over. Phase 23 gave the
|
||||
@@ -871,7 +875,8 @@ def common_router() -> APIRouter:
|
||||
"severity": _host_severity(h)}
|
||||
for h in hosts],
|
||||
"states": ctx.hub.states("system")},
|
||||
"llm": {"enabled": bool(ctx.settings.gufo_url or ctx.settings.vllm_url),
|
||||
"llm": {"enabled": any(url for _key, _display, url, _api_key
|
||||
in ctx.settings.llm_engines()),
|
||||
"fleet": fleet_totals(engines) if engines else None,
|
||||
"engines": [{"engine": e["engine"],
|
||||
"decode_tokens_per_second": (e.get("metrics") or {}).get(
|
||||
@@ -1460,10 +1465,11 @@ def llm_router() -> APIRouter:
|
||||
|
||||
@router.get("/snapshot")
|
||||
async def snapshot(ctx: AppContext = Depends(get_context)) -> dict[str, Any]:
|
||||
if not (ctx.settings.gufo_url or ctx.settings.vllm_url):
|
||||
if not any(url for _key, _display, url, _api_key in ctx.settings.llm_engines()):
|
||||
raise HTTPException(404, detail={"configured": False,
|
||||
"detail": "no engines configured — set "
|
||||
"GUFO_BASE_URL / VLLM_BASE_URL"})
|
||||
"GUFO_BASE_URL / VLLM_BASE_URL / "
|
||||
"RADIANCE_BASE_URL"})
|
||||
engines = []
|
||||
for job in ctx.hub.jobs.values():
|
||||
if job.dashboard != "llm":
|
||||
@@ -1482,7 +1488,10 @@ def llm_router() -> APIRouter:
|
||||
span = parse_range(range)
|
||||
wanted_metrics = [item.strip() for item in metrics.split(",") if item.strip()]
|
||||
wanted_engines = [item.strip() for item in engines.split(",") if item.strip()] or [
|
||||
name for name, url in (("gufo", ctx.settings.gufo_url), ("vllm", ctx.settings.vllm_url)) if url]
|
||||
# the registry's configured `key`s, in its stable order — `key` is the history
|
||||
# DB's host key (`Sample("llm", engine, …)` is written under it), not the
|
||||
# display name a health record prints (phase 34 task 02)
|
||||
key for key, _display, url, _api_key in ctx.settings.llm_engines() if url]
|
||||
return {"engines": wanted_engines, "range_seconds": span,
|
||||
"series": _history_batch(ctx.store, "llm", wanted_engines,
|
||||
[f"llm.{metric}" for metric in wanted_metrics],
|
||||
|
||||
@@ -361,7 +361,25 @@ async def fetch_gateway_snapshot(client: httpx.AsyncClient, target: Target,
|
||||
# --------------------------------------------------------------------------- #
|
||||
async def fetch_engine(client: httpx.AsyncClient, engine: str, base_url: str,
|
||||
api_key: str, rates: RateEstimator, *,
|
||||
with_slots: bool = False) -> Result:
|
||||
with_slots: bool = False, with_stats: bool = False) -> Result:
|
||||
"""One engine's `/metrics` (the poll's one truth) plus the sidecars its sessions need.
|
||||
|
||||
`with_slots` is gufo's `/slots` pass; `with_stats` is radiance's `/stats` one (task 04).
|
||||
Both are best-effort by construction: a dead or partial sidecar degrades that engine's
|
||||
session arm (task 03's missing-stats fallback) and never takes the `/metrics` poll down —
|
||||
invariant 4's shape, where the failed sidecar keeps the last good payload and marks
|
||||
nothing stale while a dead `/metrics` is the real `UpstreamError` below. One read per
|
||||
sidecar, no retry, no second dial: `/slots` tries its documented alias because a server
|
||||
may expose only that, while `/stats` has no alias — radiance's own page uses exactly
|
||||
that path — and a second attempt at a wedged sidecar would just hold the poll open.
|
||||
The `/stats` answer must be a JSON **object** (the engine's summary document): a
|
||||
connection error, a status of 400 or above, a non-JSON body, or a JSON that is a list
|
||||
leaves `stats` `None`, and the function returns the metrics result with the missing-stats
|
||||
session arm. Nothing in the sidecar may raise an `UpstreamError` — a sidecar failure is
|
||||
a degraded arm, never a failed source — and its failure sentence would leak nothing
|
||||
anyway, because it is never written: the metrics dial's `UpstreamError` messages carry
|
||||
the redacted URL (invariant 6) exactly as before.
|
||||
"""
|
||||
headers = {"Authorization": f"Bearer {api_key}"} if api_key else {}
|
||||
url = f"{base_url}/metrics"
|
||||
shown = strip_userinfo(url) # same rule as the host fetch above: no credential in a message
|
||||
@@ -392,7 +410,25 @@ async def fetch_engine(client: httpx.AsyncClient, engine: str, base_url: str,
|
||||
continue
|
||||
slots = data if isinstance(data, list) else data.get("slots", [])
|
||||
break
|
||||
return llm_source.engine_result(engine, base_url, text, rates, slots=slots)
|
||||
|
||||
stats: dict[str, Any] | None = None
|
||||
if with_stats:
|
||||
# radiance's session source, one best-effort read: /stats has no alias, and the
|
||||
# failure arms (dead, >= 400, non-JSON, JSON that is not a dict) all leave
|
||||
# `stats` None — the session arm then falls back to the /metrics gauges and the
|
||||
# queue cap stays n/a, never 0 (rule 2). No error escapes this block (the docstring).
|
||||
try:
|
||||
stats_response = await client.get(f"{base_url}/stats", headers=headers)
|
||||
except httpx.HTTPError:
|
||||
stats = None
|
||||
else:
|
||||
if stats_response.status_code < 400:
|
||||
try:
|
||||
data = stats_response.json()
|
||||
except ValueError:
|
||||
data = None
|
||||
stats = data if isinstance(data, dict) else None
|
||||
return llm_source.engine_result(engine, base_url, text, rates, slots=slots, stats=stats)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -604,15 +640,26 @@ def gateway_job(settings: Settings, client: httpx.AsyncClient, *,
|
||||
|
||||
|
||||
def llm_jobs(settings: Any, client: httpx.AsyncClient, rates: RateEstimator) -> list[Job]:
|
||||
"""One `llm:<key>` job per **configured** engine, from the one registry.
|
||||
|
||||
`settings.llm_engines()` is the single place an engine's presence is decided (phase 34
|
||||
task 02): an entry with an empty `url` is unconfigured and gets no job at all, so an
|
||||
unconfigured engine is never dialled and never shows a red record (invariant 3). The
|
||||
job's `label` is the registry's `display` (what a health record and the page print),
|
||||
while `fetch_engine` takes the registry's `key` — the history DB's host key — exactly
|
||||
as the two hand-written blocks did. The per-engine fetch flags follow the key:
|
||||
`with_slots` is gufo's `/slots` pass, `with_stats` is radiance's `/stats` one (task 04);
|
||||
vLLM gets neither, as before.
|
||||
"""
|
||||
jobs: list[Job] = []
|
||||
if settings.gufo_url:
|
||||
jobs.append(Job("llm:gufo", "llm", "gufo", settings.llm_interval,
|
||||
lambda: fetch_engine(client, "gufo", settings.gufo_url,
|
||||
settings.gufo_api_key, rates, with_slots=True)))
|
||||
if settings.vllm_url:
|
||||
jobs.append(Job("llm:vllm", "llm", "vLLM", settings.llm_interval,
|
||||
lambda: fetch_engine(client, "vllm", settings.vllm_url,
|
||||
settings.vllm_api_key, rates)))
|
||||
for key, display, url, api_key in settings.llm_engines():
|
||||
if not url:
|
||||
continue
|
||||
jobs.append(Job(f"llm:{key}", "llm", display, settings.llm_interval,
|
||||
lambda key=key, display=display, url=url, api_key=api_key:
|
||||
fetch_engine(client, key, url, api_key, rates,
|
||||
with_slots=key == "gufo",
|
||||
with_stats=key == "radiance")))
|
||||
return jobs
|
||||
|
||||
|
||||
|
||||
@@ -89,9 +89,10 @@ def strip_userinfo(url: str) -> str:
|
||||
rather than belt-and-braces: a URL does not travel only in the fetch — an `UpstreamError`
|
||||
message repeats the URL it was dialling, and that string lands in the job's health record
|
||||
(`/api/health`) and in the `event` table (`/api/events`), both of which a browser reads —
|
||||
and the engine URLs (`GUFO_BASE_URL`, `VLLM_BASE_URL`) are used verbatim, with no
|
||||
`parse_targets`-style move of userinfo into a header, so a credential written into one of
|
||||
them would otherwise be served by `/api/config` and echoed by every failed poll.
|
||||
and the engine URLs (`GUFO_BASE_URL`, `VLLM_BASE_URL`, `RADIANCE_BASE_URL`) are used
|
||||
verbatim, with no `parse_targets`-style move of userinfo into a header, so a credential
|
||||
written into one of them would otherwise be served by `/api/config` and echoed by every
|
||||
failed poll.
|
||||
`HOST_TARGETS` strips its token into a header, so this is the backstop for a `Target` that
|
||||
still carries one. One rule, stated once: AGENTS.md invariant 6, no credential in anything
|
||||
a browser can read.
|
||||
@@ -224,6 +225,11 @@ class Settings:
|
||||
gufo_api_key: str = _env("GUFO_API_KEY")
|
||||
vllm_url: str = _base_url("VLLM_BASE_URL")
|
||||
vllm_api_key: str = _env("VLLM_API_KEY")
|
||||
# Radiance (phase 34): a vLLM-derived engine, configured the way the other two are —
|
||||
# a base URL, an optional API key, unset by default. Unset means "this deployment does
|
||||
# not run it", which is "not configured", never a red panel (invariant 3).
|
||||
radiance_url: str = _base_url("RADIANCE_BASE_URL")
|
||||
radiance_api_key: str = _env("RADIANCE_API_KEY")
|
||||
# The LLM window *is* `llm_interval`: `app.build_hub` gives the `llm` jobs a
|
||||
# `RateEstimator(span=llm_interval)`, so the cadence knob is also the Δ-window knob and
|
||||
# there is no second number to keep in step with it (phase 23).
|
||||
@@ -281,9 +287,32 @@ class Settings:
|
||||
|
||||
@property
|
||||
def any_source_enabled(self) -> bool:
|
||||
return (bool(self.host_targets) or bool(self.gufo_url or self.vllm_url)
|
||||
return (bool(self.host_targets)
|
||||
or any(url for _key, _display, url, _api_key in self.llm_engines())
|
||||
or self.nut_enabled or self.gateway_enabled)
|
||||
|
||||
def llm_engines(self) -> list[tuple[str, str, str, str]]:
|
||||
"""The LLM engines, in the stable display order `gufo`, `vllm`, `radiance`.
|
||||
|
||||
One entry per engine, `(key, display, url, api_key)`, **all of them, configured or
|
||||
not** — the registry is the full fleet the way `parse_targets` keeps the host list:
|
||||
an unset engine is an entry with an empty `url`, and configured-ness is the call
|
||||
site's `bool(url)` test (the `nut_enabled` shape), never a field on the entry.
|
||||
**This list is the single place an engine's presence is decided.** The five places
|
||||
that used to enumerate the gufo/vllm pair by hand — `collectors.llm_jobs`, the
|
||||
`llm_router` 404 gate, the `/api/llm/history` defaults, `/api/config`'s llm block,
|
||||
and the summary's `llm.enabled` — all read this, so the next engine is a one-line
|
||||
addition here and no enumeration site can silently keep a two-engine world. The
|
||||
order is part of the contract: the UI's card order and the history defaults follow
|
||||
it (pinned in `tests/test_config.py`). The `key` is the history DB's host key and
|
||||
the 404/health job id; the `display` is what a health record and a page print.
|
||||
"""
|
||||
return [
|
||||
("gufo", "gufo", self.gufo_url, self.gufo_api_key),
|
||||
("vllm", "vLLM", self.vllm_url, self.vllm_api_key),
|
||||
("radiance", "radiance", self.radiance_url, self.radiance_api_key),
|
||||
]
|
||||
|
||||
|
||||
def load() -> Settings:
|
||||
"""Settings from the environment as it stands now.
|
||||
|
||||
+397
-15
@@ -6,8 +6,10 @@ expect. Do not "fix" a name here without scraping the real thing:
|
||||
|
||||
curl -s $GUFO_BASE_URL/metrics | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
|
||||
curl -s $VLLM_BASE_URL/metrics | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
|
||||
curl -s $RADIANCE_BASE_URL/metrics | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
|
||||
|
||||
and drop the capture into tests/fixtures/.
|
||||
and drop the capture into tests/fixtures/ (radiance's /stats and /v1/models sidecars are
|
||||
recorded by `scripts/capture-fixtures.sh --with-llm`).
|
||||
|
||||
Two rules carried over verbatim, because they are the difference between a truthful and a
|
||||
plausible dashboard:
|
||||
@@ -92,13 +94,184 @@ VLLM_HISTOGRAMS: dict[str, tuple[str, ...]] = {
|
||||
"e2e_request_latency_seconds": ("vllm:e2e_request_latency_seconds",),
|
||||
}
|
||||
|
||||
#: Cells that have a row in `severity.THRESHOLDS`, mapped to (metric, scale). Both engines
|
||||
# Radiance (phase 34): a vLLM-derivative engine — the dev fleet's :8000 engine replaced
|
||||
# vLLM at that address (operator decision, 2026-10-24), while VLLM_BASE_URL still
|
||||
# configures a vLLM engine for other deployments. Its /metrics publishes the `radiance:`
|
||||
# native series alongside a `vllm:` compatibility family, and its ground truth is the
|
||||
# verbatim capture in tests/fixtures/metrics_radiance.txt (plus stats_radiance.json /
|
||||
# models_radiance.json), recorded by scripts/capture-fixtures.sh --with-llm. One `Prom`
|
||||
# parse serves both families because they share one file.
|
||||
#
|
||||
# RADIANCE_SERIES holds exactly the decision-3 payload set: native + measured throughput,
|
||||
# KV usage + max tokens/bytes, queue depth/cap/admission rejections, preemptions, the
|
||||
# prefix hit-rate gauge, overall + per-position spec acceptance, decode step gauges and
|
||||
# lifetime token counters. `spec_accept_rate` reuses gufo's UI key, so the history name,
|
||||
# the fleet block and the UI row all work without a new vocabulary. The per-position
|
||||
# series carries seven label sets (position="0".."6"); the parser reads those label sets
|
||||
# as `spec_positions` (an ordered list) rather than collapsing them through the map's
|
||||
# aggregation — a mean of the seven positions would be a number the engine never
|
||||
# published. Engine internals the operator cannot act on stay out of the map (decision 3):
|
||||
# the exclusion list — link / passes / steps* / ckpt* / hit_cuts / released / kv_stream* /
|
||||
# lookup* / linear* / attn_cached / client_disconnect / stream_overflow — plus the
|
||||
# vLLM-compat latency-histogram family and the radiance-internal counters, are pinned
|
||||
# below in RADIANCE_OPTIONAL so a rename or a new build's series still fails the drift
|
||||
# test in tests/test_llm_source.py before a dashboard does.
|
||||
RADIANCE_SERIES: dict[str, tuple[tuple[str, ...], str]] = {
|
||||
# native gauges: this build's throughput cells — the scheduler's recent-window
|
||||
# aggregate, so (like the vLLM map) the gauge wins over a measured Δ
|
||||
"decode_tokens_per_second_native": (("radiance:decode_tokens_per_second",), "avg"),
|
||||
"prompt_tokens_per_second_native": (("radiance:prefill_tokens_per_second",), "avg"),
|
||||
# measured-Δ source: the vLLM-compat token counters this build carries. The
|
||||
# radiance: exact counters measure something else (tokens the engine *computed*,
|
||||
# per the engine's own HELP) and stay out of the map, in RADIANCE_OPTIONAL.
|
||||
"prompt_tokens_total": (("vllm:prompt_tokens_total",), "sum"),
|
||||
"generation_tokens_total": (("vllm:generation_tokens_total",), "sum"),
|
||||
# queue depth / admission rejections (queue_cap itself comes from /stats)
|
||||
"num_requests_running": (("vllm:num_requests_running",), "sum"),
|
||||
"num_requests_waiting": (("vllm:num_requests_waiting",), "sum"),
|
||||
"queue_depth": (("radiance:http_queue_depth",), "max"),
|
||||
"admission_rejected_total": (("radiance:http_admission_rejected_total",), "sum"),
|
||||
# preemptions: a running request kicked back to the queue is the real KV-pressure signal
|
||||
"num_preemptions_total": (("vllm:num_preemptions_total",), "sum"),
|
||||
"num_requests_swapped": (("vllm:num_requests_swapped",), "sum"),
|
||||
# KV usage + max (decision 4): the same 0-1 ratio the vLLM map feeds the
|
||||
# llm.kv_cache_usage_pct verdict, so attach_cell_severity rates it with no new row;
|
||||
# first is the name this build resolves to, the third is the vLLM-legacy name
|
||||
"gpu_cache_usage": (("vllm:kv_cache_usage_perc", "vllm:gpu_cache_usage_perc",
|
||||
"vllm:gpu_cache_usage"), "max"),
|
||||
"kv_cache_max_tokens": (("radiance:kv_cache_max_tokens",), "max"),
|
||||
"kv_cache_max_bytes": (("radiance:kv_cache_max_bytes",), "max"),
|
||||
# prefix hit rate: this build publishes no prefix counters, so a hits/queries ratio
|
||||
# would be an invented number — the gauge is the answer, source-named, not derived
|
||||
# (the documented asymmetry, the gufo *_native note's sibling)
|
||||
"gpu_prefix_cache_hit_rate": (("vllm:gpu_prefix_cache_hit_rate",), "avg"),
|
||||
# spec acceptance: the overall 0-1 gauge reuses gufo's UI key (the per-position
|
||||
# series is read by the parser as spec_positions, not through this aggregation)
|
||||
"spec_accept_rate": (("radiance:draft_acceptance_ratio",), "avg"),
|
||||
# the seven label sets (position="0".."6"); the parser reads them as spec_positions
|
||||
"draft_position_acceptance_ratio": (("radiance:draft_position_acceptance_ratio",), "avg"),
|
||||
# decode step gauges (the cumulative engine_step_seconds* counters stay out)
|
||||
"decode_step_seconds_recent": (("radiance:decode_step_seconds_recent",), "max"),
|
||||
"decode_step_seconds_best": (("radiance:decode_step_seconds_best",), "max"),
|
||||
}
|
||||
|
||||
#: Radiance histogram series: the vLLM-shaped latency family this build carries
|
||||
#: (ttft / tpot / e2e, with inter-token latency as the tpot alias). This build has
|
||||
#: **no** `request_queue_time_seconds`, so the page's `queue_seconds` cell is `n/a`
|
||||
#: for radiance by absence — never back-filled from another series.
|
||||
RADIANCE_HISTOGRAMS: dict[str, tuple[str, ...]] = {
|
||||
"time_to_first_token_seconds": ("vllm:time_to_first_token_seconds",),
|
||||
"time_per_output_token_seconds": ("vllm:time_per_output_token_seconds",
|
||||
"vllm:inter_token_latency_seconds"),
|
||||
"e2e_request_latency_seconds": ("vllm:e2e_request_latency_seconds",),
|
||||
}
|
||||
|
||||
#: Radiance series names that are in RADIANCE_SERIES/RADIANCE_HISTOGRAMS or in this
|
||||
#: fixture's engine but are not a payload cell — a name in neither the map nor this set
|
||||
#: fails the drift test in tests/test_llm_source.py, and a map name absent from the
|
||||
#: fixture must appear here (present on other radiance builds), so "absent here" and
|
||||
#: "absent everywhere" are different things in that test. A histogram *base* name is a
|
||||
#: map name too, but a base is never a standalone exposition line — its `_bucket` /
|
||||
#: `_count` / `_sum` forms are the lines, which is why each base sits beside its three
|
||||
#: suffix forms below.
|
||||
RADIANCE_OPTIONAL: frozenset[str] = frozenset({
|
||||
# in the map, absent from this capture (other builds): the vLLM-legacy KV name and
|
||||
# the vLLM-compat swap counter
|
||||
"vllm:gpu_cache_usage",
|
||||
"vllm:num_requests_swapped",
|
||||
# in the map, absent as standalone lines: the histogram base names (see above)
|
||||
"vllm:time_to_first_token_seconds",
|
||||
"vllm:time_per_output_token_seconds",
|
||||
"vllm:inter_token_latency_seconds",
|
||||
"vllm:e2e_request_latency_seconds",
|
||||
# the radiance-internal counters the decision-3 payload list does not carry
|
||||
"radiance:decode_rows_unspeculated_total",
|
||||
"radiance:decode_tokens_total",
|
||||
"radiance:draft_accepted_total",
|
||||
"radiance:draft_position_accepted_total",
|
||||
"radiance:draft_position_reached_total",
|
||||
"radiance:draft_refill_tokens_total",
|
||||
"radiance:draft_refills_total",
|
||||
"radiance:draft_tokens_total",
|
||||
"radiance:engine_prefill_chunks_capped_total",
|
||||
"radiance:engine_step_seconds_decode_total",
|
||||
"radiance:engine_step_seconds_total",
|
||||
"radiance:engine_steps_ahead_total",
|
||||
"radiance:engine_steps_decode_total",
|
||||
"radiance:engine_steps_held_total",
|
||||
"radiance:engine_steps_mixed_total",
|
||||
"radiance:engine_steps_prefill_total",
|
||||
"radiance:engine_steps_total",
|
||||
"radiance:http_responses_total",
|
||||
"radiance:kv_resident_blocks",
|
||||
"radiance:kv_stream_bytes_total",
|
||||
"radiance:kv_stream_misses_total",
|
||||
"radiance:num_requests_preempted",
|
||||
"radiance:preempt_forced_total",
|
||||
"radiance:prefill_tokens_total",
|
||||
# engine internals — the decision-3 exclusion list (link / passes / steps* / ckpt* /
|
||||
# hit_cuts / released / kv_stream* / lookup* / linear* / attn_cached /
|
||||
# client_disconnect / stream_overflow)
|
||||
"radiance:attn_cached_tokens_total",
|
||||
"radiance:client_disconnect_total",
|
||||
"radiance:linear_cached_tokens_total",
|
||||
"radiance:linear_checkpoints_written_total",
|
||||
"radiance:linear_prefix_cache_hit_rate",
|
||||
"radiance:lookup_held_drafts_total",
|
||||
"radiance:lookup_held_hits_total",
|
||||
"radiance:lookup_held_total",
|
||||
"radiance:lookup_proposals_total",
|
||||
"radiance:prefix_cache_ckpt_evictions_by_reason_total",
|
||||
"radiance:prefix_cache_ckpt_evictions_total",
|
||||
"radiance:prefix_cache_ckpt_expired_total",
|
||||
"radiance:prefix_cache_ckpt_refused_total",
|
||||
"radiance:prefix_cache_evictions_total",
|
||||
"radiance:prefix_cache_expired_total",
|
||||
"radiance:prefix_cache_released_total",
|
||||
"radiance:prefix_hit_cut_tokens_total",
|
||||
"radiance:prefix_hit_cuts_total",
|
||||
"radiance:prefix_restore_cuts_total",
|
||||
"radiance:stream_overflow_total",
|
||||
# the vLLM-compat latency-histogram family (nine series x bucket/count/sum, the
|
||||
# request_decode_time and request-param distributions included) and the
|
||||
# request-success counter — engine bookkeeping, not a decision-3 payload cell
|
||||
"vllm:e2e_request_latency_seconds_bucket",
|
||||
"vllm:e2e_request_latency_seconds_count",
|
||||
"vllm:e2e_request_latency_seconds_sum",
|
||||
"vllm:inter_token_latency_seconds_bucket",
|
||||
"vllm:inter_token_latency_seconds_count",
|
||||
"vllm:inter_token_latency_seconds_sum",
|
||||
"vllm:request_decode_time_seconds_bucket",
|
||||
"vllm:request_decode_time_seconds_count",
|
||||
"vllm:request_decode_time_seconds_sum",
|
||||
"vllm:request_generation_tokens_bucket",
|
||||
"vllm:request_generation_tokens_count",
|
||||
"vllm:request_generation_tokens_sum",
|
||||
"vllm:request_params_max_tokens_bucket",
|
||||
"vllm:request_params_max_tokens_count",
|
||||
"vllm:request_params_max_tokens_sum",
|
||||
"vllm:request_params_n_bucket",
|
||||
"vllm:request_params_n_count",
|
||||
"vllm:request_params_n_sum",
|
||||
"vllm:request_prompt_tokens_bucket",
|
||||
"vllm:request_prompt_tokens_count",
|
||||
"vllm:request_prompt_tokens_sum",
|
||||
"vllm:request_success_total",
|
||||
"vllm:time_per_output_token_seconds_bucket",
|
||||
"vllm:time_per_output_token_seconds_count",
|
||||
"vllm:time_per_output_token_seconds_sum",
|
||||
"vllm:time_to_first_token_seconds_bucket",
|
||||
"vllm:time_to_first_token_seconds_count",
|
||||
"vllm:time_to_first_token_seconds_sum",
|
||||
})
|
||||
|
||||
#: Cells that have a row in `severity.THRESHOLDS`, mapped to (metric, scale). All engines
|
||||
#: publish KV pressure as a 0-1 ratio while the threshold table is written in percent, so
|
||||
#: the scale converts for the *verdict only* — `value` stays what the engine published
|
||||
#: (rule 2: no invented number), and 0.85 is `warn` rather than a permanent `ok`.
|
||||
SEVERITY_CELLS: dict[str, tuple[str, float]] = {
|
||||
"kv_cache_usage_ratio": ("llm.kv_cache_usage_pct", 100.0), # gufo / llama.cpp
|
||||
"gpu_cache_usage": ("llm.kv_cache_usage_pct", 100.0), # vLLM
|
||||
"gpu_cache_usage": ("llm.kv_cache_usage_pct", 100.0), # vLLM and radiance
|
||||
}
|
||||
|
||||
|
||||
@@ -121,7 +294,16 @@ HISTORY_METRICS: dict[str, tuple[str, ...]] = {
|
||||
"tpot_seconds": ("time_per_output_token_seconds__mean",), # vLLM only
|
||||
"queue_seconds": ("request_queue_time_seconds__mean",),
|
||||
"prefix_cache_hit_ratio": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"),
|
||||
"spec_accept_rate": ("spec_accept_rate",), # gufo only
|
||||
"spec_accept_rate": ("spec_accept_rate",), # gufo and radiance (the gauge reuses gufo's key)
|
||||
# radiance additions (phase 34): additive candidates, the existing keys and their
|
||||
# candidate order untouched. `queue_cap` is the /stats passthrough in the metrics
|
||||
# dict; `vram_used_pct` is card 0, the engine's own view of the same card /system
|
||||
# reads through the agent — different source, different host key, both honest.
|
||||
# The seven per-position spec-acceptance ratios stay snapshot-only (decision 2).
|
||||
"queue_depth": ("queue_depth",),
|
||||
"queue_cap": ("queue_cap",),
|
||||
"decode_step_seconds_recent": ("decode_step_seconds_recent",),
|
||||
"vram_used_pct": ("vram_used_pct",),
|
||||
}
|
||||
|
||||
|
||||
@@ -328,19 +510,172 @@ def parse_vllm_metrics(text: str, rates: RateEstimator | None = None,
|
||||
return out
|
||||
|
||||
|
||||
def _stats_number(stats: dict[str, Any] | None, key: str) -> float | None:
|
||||
"""One whitelisted /stats value, or `None` — never a 0 for an absent key."""
|
||||
value = stats.get(key) if isinstance(stats, dict) else None
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||
return None
|
||||
return value
|
||||
|
||||
|
||||
def _vram_used_pct(stats: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""Card 0's `vram_used / vram_total * 100` — the one computed number in the block."""
|
||||
cards = stats.get("cards") if isinstance(stats, dict) else None
|
||||
if isinstance(cards, list):
|
||||
for card in cards:
|
||||
if isinstance(card, dict) and card.get("index") == 0:
|
||||
used, total = card.get("vram_used"), card.get("vram_total")
|
||||
if (isinstance(used, (int, float)) and isinstance(total, (int, float))
|
||||
and total):
|
||||
return field(used / total * 100.0,
|
||||
"computed:vram_used/vram_total*100", derived=True)
|
||||
break
|
||||
return field(None, None)
|
||||
|
||||
|
||||
def _stats_subset(stats: dict[str, Any]) -> dict[str, Any]:
|
||||
"""The small additive /stats subset `payload["stats"]` carries.
|
||||
|
||||
Whitelisted from the fixture's shape (tests/fixtures/stats_radiance.json), so a
|
||||
radiance upgrade that renames a /stats key degrades that key to **absent**, not a
|
||||
crash, and the poll still returns (invariant 4's poll-body rule). The engine
|
||||
internals stay out (decision 3); the raw card bytes travel as the engine published
|
||||
them, with each card's computed `vram_used_pct` beside them.
|
||||
"""
|
||||
subset: dict[str, Any] = {}
|
||||
for key in ("model", "version", "uptime_s", "startup_s", "total_requests"):
|
||||
if key in stats:
|
||||
subset[key] = stats[key]
|
||||
cards = stats.get("cards")
|
||||
if isinstance(cards, list):
|
||||
rows: list[dict[str, Any]] = []
|
||||
for card in cards:
|
||||
if not isinstance(card, dict):
|
||||
continue
|
||||
row = {key: card[key] for key in ("index", "name", "vram_used", "vram_total")
|
||||
if key in card}
|
||||
used, total = card.get("vram_used"), card.get("vram_total")
|
||||
pct = (used / total * 100.0
|
||||
if isinstance(used, (int, float)) and isinstance(total, (int, float)) and total
|
||||
else None)
|
||||
row["vram_used_pct"] = field(pct, "computed:vram_used/vram_total*100", derived=True)
|
||||
rows.append(row)
|
||||
subset["cards"] = rows
|
||||
return subset
|
||||
|
||||
|
||||
def parse_radiance_metrics(text: str, rates: RateEstimator | None = None,
|
||||
now: float | None = None,
|
||||
*, stats: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""One radiance `/metrics` poll (and its best-effort `/stats` document) parsed.
|
||||
|
||||
Rule 2's **vLLM-side** throughput precedence: the native gauges are the scheduler's
|
||||
recent-window aggregate, so they win; a gauge this build does not publish falls back
|
||||
to the measured Δ of the vLLM-compat token counters, per label set (a refused window
|
||||
is still an answer — `n/a` beside its honest `Δ<window>`); both absent is `n/a`,
|
||||
never `0`. There is deliberately **no third estimate** (no `running * 1/tpot`): a
|
||||
gauge + a counter + a guess stacked on one cell is exactly the aggregation rule 2
|
||||
exists to catch, and the engine's own page shows only the first two.
|
||||
"""
|
||||
parsed = Prom(text)
|
||||
out: dict[str, Any] = {}
|
||||
for key, (names, agg) in RADIANCE_SERIES.items():
|
||||
if key == "draft_position_acceptance_ratio":
|
||||
continue # read as spec_positions below, never an aggregated scalar
|
||||
out[key] = field(parsed.scalar(names, agg), parsed.resolve(names))
|
||||
|
||||
# Per-position spec acceptance: the label sets in the order the engine publishes
|
||||
# them; an absent position stays absent, not zeroed, and a line without a numeric
|
||||
# position label drops out rather than crashing the poll (invariant 4).
|
||||
by_position = parsed.by_label(("radiance:draft_position_acceptance_ratio",), "position")
|
||||
out["spec_positions"] = [
|
||||
{"position": int(position), "ratio": ratio}
|
||||
for position, ratio in sorted(
|
||||
((position, ratio)
|
||||
for position, ratio in by_position.items() if position.isdigit()),
|
||||
key=lambda item: int(item[0]))
|
||||
]
|
||||
|
||||
def measured(total_key: str) -> dict[str, Any] | None:
|
||||
"""Δcounter/Δt per label set, keyed by the resolved name plus its labels —
|
||||
radiance's counters are labelled the same way vLLM's are, so a summed diff over
|
||||
a changing label set is exactly the carried-counter bug phase 13 pinned."""
|
||||
if rates is None:
|
||||
return None
|
||||
source = out[total_key]["source"]
|
||||
if source is None:
|
||||
return None
|
||||
rate, window = _series_delta(parsed, rates, source, now)
|
||||
if rate is None and window is None:
|
||||
return None
|
||||
return field(rate, f"computed:Δ{source}/Δt", derived=True, window=window)
|
||||
|
||||
for ui_key, native_key, total_key in (
|
||||
("decode_tokens_per_second", "decode_tokens_per_second_native",
|
||||
"generation_tokens_total"),
|
||||
("prompt_tokens_per_second", "prompt_tokens_per_second_native",
|
||||
"prompt_tokens_total"),
|
||||
):
|
||||
native_value = out[native_key]["value"]
|
||||
if native_value is not None:
|
||||
# the gauge is the aggregate: it wins, and its name is the source
|
||||
out[ui_key] = field(native_value, out[native_key]["source"])
|
||||
else:
|
||||
cell = measured(total_key)
|
||||
out[ui_key] = cell if cell is not None else field(None, None)
|
||||
|
||||
# the vLLM-shaped latency family this build carries — the vLLM-path idiom: the
|
||||
# cell plus its __mean / __p95 companions (no request_queue_time on this build,
|
||||
# so queue_seconds stays n/a by absence, never back-filled)
|
||||
for ui_key, names in RADIANCE_HISTOGRAMS.items():
|
||||
out[ui_key] = parsed.histogram(names)
|
||||
out[f"{ui_key}__mean"] = field(out[ui_key]["mean"] if out[ui_key] else None,
|
||||
out[ui_key]["source"] if out[ui_key] else None)
|
||||
out[f"{ui_key}__p95"] = field(out[ui_key]["p95"] if out[ui_key] else None,
|
||||
out[ui_key]["source"] if out[ui_key] else None)
|
||||
|
||||
# lifetime tokens: the two vLLM-compat counters, as the vLLM path does
|
||||
parts = [v for v in (out["prompt_tokens_total"]["value"],
|
||||
out["generation_tokens_total"]["value"]) if v is not None]
|
||||
out["tokens_total"] = (field(sum(parts), "computed:prompt+generated")
|
||||
if parts else field(None, None))
|
||||
|
||||
# prefix hit rate under the shared UI key: the gauge itself (source-named, not
|
||||
# derived — the documented asymmetry, the gufo *_native note's sibling)
|
||||
out["prefix_cache_hit_ratio"] = dict(out["gpu_prefix_cache_hit_rate"])
|
||||
|
||||
# /stats passthroughs: absent /stats degrades them to n/a, never 0 (rule 2)
|
||||
out["queue_cap"] = (field(_stats_number(stats, "queue_cap"), "radiance /stats")
|
||||
if stats is not None and _stats_number(stats, "queue_cap") is not None
|
||||
else field(None, None))
|
||||
out["vram_used_pct"] = _vram_used_pct(stats)
|
||||
|
||||
attach_cell_severity(out)
|
||||
return out
|
||||
|
||||
|
||||
def engine_result(engine: str, base_url: str, text: str, rates: RateEstimator,
|
||||
*, slots: list[dict[str, Any]] | None = None, now: float | None = None) -> Result:
|
||||
"""Parse one engine's /metrics into a poll Result (payload + history samples)."""
|
||||
*, slots: list[dict[str, Any]] | None = None,
|
||||
stats: dict[str, Any] | None = None, now: float | None = None) -> Result:
|
||||
"""Parse one engine's /metrics into a poll Result (payload + history samples).
|
||||
|
||||
`stats` is radiance's best-effort /stats document (task 04's sidecar fetch); a
|
||||
missing sidecar degrades the session arm and the queue-cap cell, it never fails
|
||||
the poll (invariant 4).
|
||||
"""
|
||||
now = time.time() if now is None else now
|
||||
if engine == "gufo":
|
||||
metrics = parse_gufo_metrics(text, rates, now)
|
||||
active = metrics["requests_processing"]["value"]
|
||||
elif engine == "radiance":
|
||||
metrics = parse_radiance_metrics(text, rates, now, stats=stats)
|
||||
active = metrics["num_requests_running"]["value"]
|
||||
else:
|
||||
metrics = parse_vllm_metrics(text, rates, now)
|
||||
active = metrics["num_requests_running"]["value"]
|
||||
|
||||
parsed = Prom(text)
|
||||
sessions = _sessions(engine, slots, active, metrics)
|
||||
sessions = _sessions(engine, slots, active, metrics, stats=stats)
|
||||
payload = {
|
||||
"engine": engine,
|
||||
"upstream": base_url,
|
||||
@@ -350,6 +685,8 @@ def engine_result(engine: str, base_url: str, text: str, rates: RateEstimator,
|
||||
"series_count": len(parsed.series),
|
||||
"unparsed": parsed.parse_errors,
|
||||
}
|
||||
if engine == "radiance" and isinstance(stats, dict):
|
||||
payload["stats"] = _stats_subset(stats)
|
||||
samples: list[Sample] = []
|
||||
for metric, candidates in HISTORY_METRICS.items():
|
||||
value = next((metrics[key]["value"] for key in candidates
|
||||
@@ -360,12 +697,17 @@ def engine_result(engine: str, base_url: str, text: str, rates: RateEstimator,
|
||||
|
||||
|
||||
def _sessions(engine: str, slots: list[dict[str, Any]] | None, active: float | None,
|
||||
metrics: dict[str, Any]) -> dict[str, Any]:
|
||||
metrics: dict[str, Any],
|
||||
stats: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""Active sessions, however this engine expresses them.
|
||||
|
||||
gufo exposes /slots (one row per parallel slot, with per-slot tokens and speed), so it
|
||||
gets a real list. vLLM publishes no per-session endpoint in its /metrics, so it gets
|
||||
counts and says the list is not available rather than inventing rows.
|
||||
counts and says the list is not available rather than inventing rows. radiance's
|
||||
/stats is its answer to /slots — the engine's own running/waiting/queue-cap summary
|
||||
and its per-request rows; a missing /stats (the sidecar is best-effort, the poll is
|
||||
still healthy) falls back to the /metrics gauges for the counts and says so. `active`
|
||||
stays the /metrics gauge in every arm: one definition.
|
||||
"""
|
||||
out: dict[str, Any] = {"active": active, "list": [], "source": None, "detail": None}
|
||||
if engine == "gufo":
|
||||
@@ -375,6 +717,38 @@ def _sessions(engine: str, slots: list[dict[str, Any]] | None, active: float | N
|
||||
else:
|
||||
out["detail"] = ("no /slots answer — active count still comes from "
|
||||
"llamacpp:requests_processing")
|
||||
elif engine == "radiance":
|
||||
if isinstance(stats, dict):
|
||||
running = _stats_number(stats, "running")
|
||||
waiting = _stats_number(stats, "waiting")
|
||||
cap = _stats_number(stats, "queue_cap")
|
||||
rows = stats.get("requests")
|
||||
out["running"] = running
|
||||
out["waiting"] = waiting
|
||||
out["queue_cap"] = cap
|
||||
out["list"] = rows if isinstance(rows, list) else []
|
||||
out["source"] = "radiance /stats"
|
||||
if not out["list"]:
|
||||
run_text = f"{running:g}" if running is not None else "n/a"
|
||||
cap_text = f" of cap {cap:g}" if cap is not None else ""
|
||||
out["detail"] = (f"radiance reports counts, not sessions: {run_text} running"
|
||||
f"{cap_text}. Per-request detail appears here while "
|
||||
f"requests are in flight.")
|
||||
else:
|
||||
# /stats did not answer this poll: the counts fall back to the /metrics
|
||||
# gauges — real numbers, different provenance — and the cap stays n/a,
|
||||
# never 0 (rule 2).
|
||||
running = metrics.get("num_requests_running", {}).get("value")
|
||||
waiting = metrics.get("num_requests_waiting", {}).get("value")
|
||||
out["running"] = running
|
||||
out["waiting"] = waiting
|
||||
out["queue_cap"] = None
|
||||
run_text = f"{running:g}" if isinstance(running, (int, float)) else "n/a"
|
||||
wait_text = (f", {waiting:g} queued"
|
||||
if isinstance(waiting, (int, float)) and waiting else "")
|
||||
out["detail"] = (f"/stats did not answer this poll: {run_text} running"
|
||||
f"{wait_text} from the /metrics gauges; the queue cap is "
|
||||
f"n/a until it does.")
|
||||
else:
|
||||
waiting = metrics.get("num_requests_waiting", {}).get("value")
|
||||
queued = f", {waiting:g} queued" if waiting else ""
|
||||
@@ -416,18 +790,26 @@ def fleet_totals(engines: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"requests_active": sum_pairs(("requests_processing", "num_requests_running")),
|
||||
"requests_waiting": sum_pairs(("requests_deferred", "num_requests_waiting")),
|
||||
"tokens_total": sum_metric("tokens_total"),
|
||||
# a real max across both engines' aliases: `a or b` returned gufo's 25% while vLLM
|
||||
# was sitting at 75%, which is the opposite of a pressure signal
|
||||
# a real max across the engines' aliases: `a or b` returned gufo's 25% while
|
||||
# vLLM was sitting at 75%, which is the opposite of a pressure signal
|
||||
"kv_cache_usage_max": max(
|
||||
(v for v in (max_metric("kv_cache_usage_ratio"), max_metric("gpu_cache_usage"))
|
||||
if v is not None),
|
||||
default=None,
|
||||
),
|
||||
"seams": ("gufo's prompt counter excludes cache-hit tokens while vLLM's includes "
|
||||
"cached prefix tokens, so the summed prompt rate leans slightly low for "
|
||||
"gufo. Both are counted honestly per engine."),
|
||||
"seams": ("gufo's prompt counter excludes cache-hit tokens, and the summed "
|
||||
"throughput is provenance-labelled per engine — a native scheduler "
|
||||
"window (radiance's gauges) or a measured Δ where the gauge is "
|
||||
"absent; neither the sum nor the dashboard claims what a counter "
|
||||
"includes or excludes that its engine does not say (radiance's "
|
||||
"vllm:prompt_tokens_total and cached tokens among them). Both are "
|
||||
"counted honestly per engine."),
|
||||
}
|
||||
|
||||
|
||||
def series_names(engine: str) -> Sequence[str]:
|
||||
return tuple(GUFO_SERIES if engine == "gufo" else VLLM_SERIES)
|
||||
if engine == "gufo":
|
||||
return tuple(GUFO_SERIES)
|
||||
if engine == "radiance":
|
||||
return tuple(RADIANCE_SERIES)
|
||||
return tuple(VLLM_SERIES)
|
||||
|
||||
@@ -110,7 +110,7 @@
|
||||
<div class="grid cols-2 items-start" id="engines"></div>
|
||||
<div class="empty-state" id="engines-empty" hidden>
|
||||
<h3>No engines configured</h3>
|
||||
<p>Set <code>GUFO_BASE_URL</code> (llama.cpp / gufo) and/or <code>VLLM_BASE_URL</code>.</p>
|
||||
<p>Set <code>GUFO_BASE_URL</code> (llama.cpp / gufo), <code>VLLM_BASE_URL</code> and/or <code>RADIANCE_BASE_URL</code>.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
||||
@@ -708,6 +708,24 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
|
||||
.slot dd small { display: block; font-size: 10px; color: var(--text-faint); }
|
||||
.slots-empty, .slots-note { margin: 0; font-size: 11.5px; color: var(--text-muted); }
|
||||
|
||||
/* the radiance strip's own block (phase 34): the 7-bar spec-acceptance curve, the
|
||||
per-request rows beside the counts, and the count-otherwise sentence the API writes
|
||||
into the note slot while the table is hidden. Neutral ink only — the curve is an
|
||||
efficiency number (never a verdict token) and the rows are the engine's own fields
|
||||
in `table.data` at the strip's scale. The `.reqs` box must not be narrower than the
|
||||
card is: the strip body is a one-column grid, and a grid item's automatic minimum
|
||||
is its content's min-content — the ten-column request table's min-content (~950 px)
|
||||
grew the column and painted every value cell of the card past the card's edge,
|
||||
where `.card { overflow: hidden }` ate it. `min-width: 0` lets the box stay the
|
||||
card's width; `overflow-x: auto` gives the table a sideways scroll inside the card
|
||||
only where it is wider than the card. */
|
||||
.spec-curve { margin: 0; }
|
||||
.spec-curve svg { display: block; }
|
||||
.reqs { margin: 0; min-width: 0; overflow-x: auto; }
|
||||
.reqs table.data { font-size: 12px; }
|
||||
.reqs table.data th, .reqs table.data td { padding: .25rem .5rem; }
|
||||
.reqs-note { margin: 0; font-size: 11.5px; color: var(--text-muted); }
|
||||
|
||||
/* ---- fleet totals ----------------------------------------------------------
|
||||
* The `/llm` aggregate block — `llm_monitor`'s `#fleet`, assembled from vocabulary this
|
||||
* stylesheet already owns: `.card.flat` for the panel, `.metrics`/`.metric` for the sums,
|
||||
|
||||
@@ -41,6 +41,9 @@ const state = {
|
||||
|
||||
const fmtMs = (v) => `${num(v, 0)} ms`;
|
||||
|
||||
/** GiB = 2³⁰ — the page's one conversion (the radiance strip's VRAM cell says so). */
|
||||
const GIB = 2 ** 30;
|
||||
|
||||
/**
|
||||
* What each engine *panel* keeps after phase 07: the two throughput heroes (the biggest
|
||||
* numbers on the page, deliberately uncoloured — `severity.py` rates no throughput, because
|
||||
@@ -61,9 +64,9 @@ const CELLS = [
|
||||
tip: 'Aggregate generation speed across every busy slot.' },
|
||||
{ key: 'prompt', label: 'Prompt', unit: 'tok/s', digits: 1, hero: true,
|
||||
tip: 'Prefill speed for newly computed tokens.' },
|
||||
{ key: 'ttft', label: 'TTFT', unit: 'ms', digits: 0, p95: true, only: ['vllm'],
|
||||
{ key: 'ttft', label: 'TTFT', unit: 'ms', digits: 0, p95: true, only: ['vllm', 'radiance'],
|
||||
spark: 'ttft_seconds', scale: 1000, format: fmtMs },
|
||||
{ key: 'tpot', label: 'TPOT', unit: 'ms', digits: 1, p95: true, only: ['vllm'],
|
||||
{ key: 'tpot', label: 'TPOT', unit: 'ms', digits: 1, p95: true, only: ['vllm', 'radiance'],
|
||||
spark: 'tpot_seconds', scale: 1000, format: fmtMs },
|
||||
];
|
||||
|
||||
@@ -94,16 +97,24 @@ const STRIP_ROWS = [
|
||||
tip: 'Tokens counted since this engine started.' },
|
||||
{ key: 'slots_busy', label: 'slots busy', digits: 0, only: ['gufo'],
|
||||
tip: 'Parallel slots generating right now, counted from gufo /slots.' },
|
||||
{ key: 'running', label: 'running', digits: 0, only: ['vllm'],
|
||||
{ key: 'running', label: 'running', digits: 0, only: ['vllm', 'radiance'],
|
||||
tip: 'Requests generating tokens right now.' },
|
||||
{ key: 'waiting', label: 'waiting', digits: 0,
|
||||
tip: 'Requests queued behind them.' },
|
||||
{ key: 'cache_hit', label: 'prefix cache hit', digits: 0, unit: '%',
|
||||
tip: 'Share of prompt tokens served from the cache instead of recomputed.' },
|
||||
{ key: 'spec_accept', label: 'spec accept', digits: 0, unit: '%', only: ['gufo'],
|
||||
{ key: 'spec_accept', label: 'spec accept', digits: 0, unit: '%', only: ['gufo', 'radiance'],
|
||||
tip: 'Speculative-decoding draft tokens the model accepted.' },
|
||||
{ key: 'queue_mean', label: 'queue time', digits: 0, unit: 'ms', p95: true, only: ['vllm'],
|
||||
{ key: 'queue_mean', label: 'queue time', digits: 0, unit: 'ms', p95: true, only: ['vllm', 'radiance'],
|
||||
tip: 'Time a request spends waiting for KV space.' },
|
||||
{ key: 'cap', label: 'queue cap', digits: 0, only: ['radiance'],
|
||||
tip: 'The engine’s admission ceiling: requests beyond this queue depth are rejected, not queued. n/a while /stats is not answering.' },
|
||||
{ key: 'build', label: 'build', only: ['radiance'],
|
||||
tip: 'The engine’s own version string — a radiance upgrade changes it, the panel does not' },
|
||||
{ key: 'uptime', label: 'uptime', only: ['radiance'],
|
||||
tip: 'How long this engine process has been up — a restart zeroes it and the lifetime counters.' },
|
||||
{ key: 'vram', label: 'VRAM (card 0)', only: ['radiance'],
|
||||
tip: 'the engine’s own view of the card (GiB = 2³⁰); /system’s GPU row is the agent’s view of the same card' },
|
||||
{ key: 'preemptions', label: 'Preemptions', alarm: true,
|
||||
tip: 'Running requests kicked back to the queue.' },
|
||||
{ key: 'device_lost', label: 'GPU context lost', alarm: true,
|
||||
@@ -150,7 +161,8 @@ const SLOT_FIELDS = [
|
||||
* the API: gufo answers `/slots`, vLLM publishes counts and says so in `sessions.detail`. */
|
||||
const SLOT_ENGINES = ['gufo'];
|
||||
|
||||
/** The vLLM panel's one line about why it has no grid (llm_monitor keeps the same line). */
|
||||
/** The vLLM panel's one line about why it has no grid (llm_monitor keeps the same line).
|
||||
* vLLM's alone: radiance's per-request rows live in the strip, not the panel. */
|
||||
const NO_SLOT_JSON = 'vLLM exposes no per-slot JSON — its metrics are the equivalent of '
|
||||
+ 'the session grid.';
|
||||
|
||||
@@ -180,8 +192,12 @@ function windowName(seconds) {
|
||||
return WINDOW_NAMES[seconds] || duration(seconds);
|
||||
}
|
||||
|
||||
/** The payload cell behind a UI key. A missing number stays `null` — never a 0. */
|
||||
function cellValue(engine, metrics, sessions, key) {
|
||||
/** The payload cell behind a UI key. A missing number stays `null` — never a 0.
|
||||
* `stats` is radiance's whitelisted /stats subset (task 03); the radiance strip's
|
||||
* `build` / `uptime` / `vram` cells read it, and a missing sidecar leaves them null.
|
||||
* `cap` resolves through the default branch to `metrics.queue_cap`, which the server
|
||||
* degrades to null when /stats did not answer (never 0 — rule 2). */
|
||||
function cellValue(engine, metrics, sessions, key, stats) {
|
||||
const get = (k) => (metrics[k] && metrics[k].value != null ? metrics[k] : null);
|
||||
const scaled = (cell, factor) => ({ value: cell.value * factor, source: cell.source,
|
||||
derived: cell.derived });
|
||||
@@ -249,6 +265,38 @@ function cellValue(engine, metrics, sessions, key) {
|
||||
const cell = get('spec_accept_rate');
|
||||
return cell ? scaled(cell, 100) : { value: null };
|
||||
}
|
||||
case 'cap': {
|
||||
// the UI key is short; the metric is the /stats passthrough the server degrades
|
||||
// to null when the sidecar did not answer (never 0 — rule 2)
|
||||
const cell = get('queue_cap');
|
||||
return cell ? { value: cell.value, source: cell.source, derived: cell.derived }
|
||||
: { value: null };
|
||||
}
|
||||
case 'build': {
|
||||
// the engine's own version string, verbatim: a radiance upgrade changes it, the
|
||||
// panel does not (a cell that tracks the engine's version would be a second
|
||||
// source of truth for a fact the engine owns)
|
||||
const value = (stats && typeof stats.version === 'string' && stats.version)
|
||||
? stats.version : null;
|
||||
return { value, source: value ? 'radiance /stats' : null, derived: false };
|
||||
}
|
||||
case 'uptime': {
|
||||
const value = (stats && Number.isFinite(stats.uptime_s))
|
||||
? duration(stats.uptime_s) : null;
|
||||
return { value, source: value ? 'radiance /stats' : null, derived: false };
|
||||
}
|
||||
case 'vram': {
|
||||
// card 0's bytes → GiB (2³⁰, the page's one conversion, labelled in the tip):
|
||||
// the engine's own view of the card — /system's GPU row is the agent's view of
|
||||
// the same card. `n/a` when no card answered, never a 0 GiB claim.
|
||||
const cards = (stats && Array.isArray(stats.cards)) ? stats.cards : [];
|
||||
const card = cards.find((c) => c && c.index === 0) || cards.find((c) => c);
|
||||
const used = card && Number.isFinite(card.vram_used) ? card.vram_used : null;
|
||||
const total = card && Number.isFinite(card.vram_total) ? card.vram_total : null;
|
||||
const value = (used != null && total != null)
|
||||
? `${(used / GIB).toFixed(1)} / ${(total / GIB).toFixed(1)} GiB` : null;
|
||||
return { value, source: value ? 'radiance /stats' : null, derived: false };
|
||||
}
|
||||
case 'tokens_total': {
|
||||
const cell = get('tokens_total');
|
||||
return cell ? { value: cell.value, source: cell.source, derived: true } : { value: null };
|
||||
@@ -349,14 +397,21 @@ function enginePanel(engine) {
|
||||
* grid of `n/a`s would be a component of permanent dead cells for a server that cannot answer
|
||||
* it. Both grid and empty state ship `hidden` — the panel never renders a card of nothing.
|
||||
*
|
||||
* radiance's panel gets **nothing** here: its per-request rows and the count-otherwise
|
||||
* sentence live in the strip, so the panel has no grid whose absence needs explaining — and
|
||||
* a sentence naming vLLM on the radiance card is a line the engine never said.
|
||||
*
|
||||
* `SLOT_ENGINES` is the gate rather than `sessions.list.length` on purpose: an engine that
|
||||
* has not answered yet must still show the label and the honest empty sentence, which is how
|
||||
* an operator learns the difference between "idle" and "nobody asked".
|
||||
*/
|
||||
function slotSection(engine) {
|
||||
if (SLOT_ENGINES.indexOf(engine) < 0) {
|
||||
if (engine === 'vllm') {
|
||||
return `<p class="slots-note" data-slots-note>${NO_SLOT_JSON}</p>`;
|
||||
}
|
||||
if (SLOT_ENGINES.indexOf(engine) < 0) {
|
||||
return ''; // radiance: the session home is the strip's reqs block, not the panel
|
||||
}
|
||||
return `
|
||||
<h4 class="section-label" data-slots-label>sessions (GET /slots)</h4>
|
||||
<div class="slots" data-slots role="list" hidden></div>
|
||||
@@ -518,6 +573,10 @@ function stripCard(engine) {
|
||||
<span class="value" data-kv-value>n/a</span>
|
||||
</div>
|
||||
<dl class="kv">${rows}</dl>
|
||||
${engine === 'radiance' ? `
|
||||
<div class="spec-curve" data-spec-curve hidden></div>
|
||||
<div class="reqs" data-reqs hidden></div>
|
||||
<p class="reqs-note" data-reqs-note hidden></p>` : ''}
|
||||
<div class="alarms" data-k="alarms" hidden></div>
|
||||
</div>`;
|
||||
return card;
|
||||
@@ -538,6 +597,7 @@ function renderStripCard(card, enginePayload, stateRow) {
|
||||
const engine = enginePayload.engine;
|
||||
const metrics = enginePayload.metrics || {};
|
||||
const sessions = enginePayload.sessions || {};
|
||||
const stats = enginePayload.stats || null; // radiance's /stats subset; absent for the others
|
||||
const q = (sel) => card.querySelector(sel);
|
||||
|
||||
// reachability, not a verdict: the dot and the shared `.stale` dim, exactly as the panels
|
||||
@@ -567,10 +627,12 @@ function renderStripCard(card, enginePayload, stateRow) {
|
||||
if (row.alarm) continue; // chips below, never a row
|
||||
const el = q(`[data-row="${row.key}"]`);
|
||||
if (!el) continue;
|
||||
const value = cellValue(engine, metrics, sessions, row.key);
|
||||
const value = cellValue(engine, metrics, sessions, row.key, stats);
|
||||
const parts = row.human && value.value != null ? humanNumber(value.value) : null;
|
||||
setTxt(el.querySelector('[data-v]'),
|
||||
value.value == null ? null : (parts ? parts.text : num(value.value, row.digits)));
|
||||
value.value == null ? null
|
||||
: (parts ? parts.text
|
||||
: (typeof value.value === 'string' ? value.value : num(value.value, row.digits))));
|
||||
// "n/a %" would read as a measurement of nothing, so the unit leaves with the number
|
||||
setTxt(el.querySelector('[data-unit]'),
|
||||
value.value == null ? '' : (parts ? parts.unit : (row.unit || '')));
|
||||
@@ -592,6 +654,106 @@ function renderStripCard(card, enginePayload, stateRow) {
|
||||
}
|
||||
|
||||
renderAlarms(card, enginePayload);
|
||||
// the radiance strip's own block: the 7-bar spec curve (never a verdict colour,
|
||||
// decision 4) and the per-request rows beside the counts — the rest state is the
|
||||
// API's own count-otherwise sentence in the note slot
|
||||
if (engine === 'radiance') {
|
||||
renderSpecCurve(card, metrics);
|
||||
renderReqs(card, enginePayload);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The 7-bar spec-acceptance curve (phase 34 decision 2): one bar per position the
|
||||
* engine published, in the order it published them, height ∝ the 0-1 ratio.
|
||||
*
|
||||
* An absent position stays absent — a zero bar would claim the engine measured 0 %
|
||||
* there, which it did not (rule 2). The fill is the page's neutral ink
|
||||
* (`--text-faint`, the sparkline's unrated stroke): spec acceptance is an efficiency
|
||||
* number, not a verdict — `severity.py` has no row for it and nothing here may reach
|
||||
* for a verdict token (decision 4). The label is computed from the payload: the first
|
||||
* and last published position with its ratio, the connector word chosen by their
|
||||
* order — a label that cannot say which ratios it plots is not an accessible name.
|
||||
*/
|
||||
function renderSpecCurve(card, metrics) {
|
||||
const el = card.querySelector('[data-spec-curve]');
|
||||
if (!el) return;
|
||||
const rows = (metrics.spec_positions || [])
|
||||
.filter((row) => row && Number.isFinite(row.ratio))
|
||||
.sort((a, b) => a.position - b.position);
|
||||
if (!rows.length) { el.hidden = true; return; }
|
||||
el.hidden = false;
|
||||
const barW = 16, gap = 8, H = 46;
|
||||
const W = rows.length * (barW + gap);
|
||||
const ink = DashCharts.cssVar('--text-faint');
|
||||
const bar = (row, i) => {
|
||||
const h = Math.max(2, Math.min(1, row.ratio) * (H - 16));
|
||||
const x = i * (barW + gap);
|
||||
return `<rect x="${x}" y="${(H - 8 - h).toFixed(1)}" width="${barW}" height="${h.toFixed(1)}" rx="2" fill="${ink}"/>`
|
||||
+ `<text x="${x + barW / 2}" y="${H}" font-size="8" fill="${ink}" text-anchor="middle">${row.position}</text>`;
|
||||
};
|
||||
const first = rows[0], last = rows[rows.length - 1];
|
||||
const word = rows.length < 2 ? ''
|
||||
: (last.ratio < first.ratio ? ' falling to'
|
||||
: last.ratio > first.ratio ? ' rising to' : ' holding at');
|
||||
const pct = (ratio) => `${Math.round(ratio * 100)}%`;
|
||||
const label = `Spec-decode draft acceptance by position: ${pct(first.ratio)} at position ${first.position}`
|
||||
+ (rows.length > 1 ? `${word} ${pct(last.ratio)} at position ${last.position}` : '');
|
||||
el.innerHTML = `<svg viewBox="0 0 ${W} ${H}" width="100%" height="${H}" preserveAspectRatio="xMinYMax meet"
|
||||
role="img" aria-label="${DashCharts.escapeHtml(label)}">${rows.map(bar).join('')}</svg>`;
|
||||
}
|
||||
|
||||
/** A row value the engine published, or `null` — a number prints only when it is a
|
||||
* number, anything else its string verbatim (no conversion, no rounding of a string). */
|
||||
function reqCellText(value) {
|
||||
if (typeof value === 'number' && isFinite(value)) return num(value, 2);
|
||||
if (typeof value === 'string') return value;
|
||||
if (typeof value === 'boolean') return String(value);
|
||||
return null; // null / object: nothing to print, no conversion
|
||||
}
|
||||
|
||||
/**
|
||||
* Radiance's per-request rows, beside its counts on the strip (phase 34 decision 3).
|
||||
*
|
||||
* `sessions.list` is the engine's own rows, verbatim — the fixture's shape is the
|
||||
* contract, and a row's fields render **as-is**: the columns are the union of the
|
||||
* keys the rows publish, in first-seen order (no invented key, no re-ranking), a
|
||||
* number prints only when it is a number and anything else its string verbatim.
|
||||
* gufo's keyed slot grid does not apply: `SLOT_ENGINES` stays gufo's, these rows
|
||||
* live here. An empty list is the count-otherwise state: the table is hidden and
|
||||
* the API's own sentence (`sessions.detail`) takes the note slot — where a request
|
||||
* in flight would put the rows.
|
||||
*/
|
||||
function renderReqs(card, enginePayload) {
|
||||
const table = card.querySelector('[data-reqs]');
|
||||
const note = card.querySelector('[data-reqs-note]');
|
||||
if (!table) return; // a non-radiance card has no home for these rows
|
||||
const sessions = enginePayload.sessions || {};
|
||||
const rows = (Array.isArray(sessions.list) ? sessions.list : [])
|
||||
.filter((row) => row && typeof row === 'object' && !Array.isArray(row));
|
||||
if (!rows.length) {
|
||||
table.hidden = true;
|
||||
table.innerHTML = '';
|
||||
if (note) { note.hidden = !sessions.detail; setTxt(note, sessions.detail || null); }
|
||||
return;
|
||||
}
|
||||
if (note) note.hidden = true;
|
||||
const columns = [];
|
||||
for (const row of rows) {
|
||||
for (const key of Object.keys(row)) {
|
||||
if (columns.indexOf(key) < 0) columns.push(key);
|
||||
}
|
||||
}
|
||||
const head = `<thead><tr>${columns.map((key) =>
|
||||
`<th>${DashCharts.escapeHtml(key)}</th>`).join('')}</tr></thead>`;
|
||||
const body = `<tbody>${rows.map((row) => `<tr>${columns.map((key) => {
|
||||
const text = reqCellText(row[key]);
|
||||
return text == null ? '<td class="mute">n/a</td>'
|
||||
: `<td>${DashCharts.escapeHtml(text)}</td>`;
|
||||
}).join('')}</tr>`).join('')}</tbody>`;
|
||||
table.innerHTML = `<table class="data reqs-table">${head}${body}</table>`;
|
||||
table.title = sessions.source || '';
|
||||
table.hidden = false;
|
||||
}
|
||||
|
||||
/** An engine that leaves the payload leaves the strip and the panels — never a stale card. */
|
||||
@@ -747,7 +909,8 @@ function renderSlotCard(refs, row, source) {
|
||||
function renderSlotGrid(card, enginePayload) {
|
||||
const grid = card.querySelector('[data-slots]');
|
||||
if (!grid) {
|
||||
// vLLM: the one-line note only, carrying the API's own words on its session counts
|
||||
// vLLM: the one-line note, carrying the API's own words on its session counts — and
|
||||
// radiance: no grid and no note at all (its session home is the strip), so nothing to set
|
||||
const note = card.querySelector('[data-slots-note]');
|
||||
if (note && enginePayload.sessions && enginePayload.sessions.detail) {
|
||||
note.title = enginePayload.sessions.detail;
|
||||
@@ -953,6 +1116,11 @@ function renderFleet(fleet, engines) {
|
||||
const prompt = $('#f-prompt');
|
||||
const promptMetric = fleet.seams && prompt && prompt.closest ? prompt.closest('.metric') : null;
|
||||
if (promptMetric) promptMetric.title = fleet.seams;
|
||||
// the note under the count is the API's own seams sentence (task 03's rewrite) —
|
||||
// payload-driven, no new hardcoded copy; the shipped line is the fallback for a
|
||||
// payload that predates it
|
||||
const noteEl = $('.fleet-note');
|
||||
if (noteEl) setTxt(noteEl, fleet.seams || 'sums only engines answering now');
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
@@ -1020,8 +1188,9 @@ document.addEventListener('DOMContentLoaded', async () => {
|
||||
const config = await fetchJSON('/api/config');
|
||||
state.pollMs = config.llm.poll_ms || 2000;
|
||||
const banners = [];
|
||||
if (!config.llm.gufo.enabled && !config.llm.vllm.enabled) {
|
||||
banners.push(['warn', 'Neither engine is configured.']);
|
||||
if (!config.llm.gufo.enabled && !config.llm.vllm.enabled
|
||||
&& !(config.llm.radiance && config.llm.radiance.enabled)) {
|
||||
banners.push(['warn', 'No engines are configured.']);
|
||||
}
|
||||
if (config.ups.simulated) banners.push(['warn', 'UPS numbers on other pages are simulated.']);
|
||||
$('#banners').innerHTML = banners.map(([kind, html]) => `<div class="banner ${kind}">${html}</div>`)
|
||||
|
||||
Vendored
+17
-11
@@ -3,37 +3,43 @@
|
||||
gufo_device_lost_total 0
|
||||
# HELP llamacpp:prompt_tokens_total Total prompt tokens processed, excluding cache hits
|
||||
# TYPE llamacpp:prompt_tokens_total counter
|
||||
llamacpp:prompt_tokens_total 1350666
|
||||
llamacpp:prompt_tokens_total 213784
|
||||
# HELP llamacpp:prompt_tokens_cached_total Total prompt tokens reused from cache
|
||||
# TYPE llamacpp:prompt_tokens_cached_total counter
|
||||
llamacpp:prompt_tokens_cached_total 124529105
|
||||
llamacpp:prompt_tokens_cached_total 38070188
|
||||
# HELP llamacpp:prompt_seconds_total Prompt process time
|
||||
# TYPE llamacpp:prompt_seconds_total counter
|
||||
llamacpp:prompt_seconds_total 1478.1443332270005
|
||||
llamacpp:prompt_seconds_total 215.41327585600013
|
||||
# HELP llamacpp:tokens_predicted_total Total tokens generated
|
||||
# TYPE llamacpp:tokens_predicted_total counter
|
||||
llamacpp:tokens_predicted_total 877336
|
||||
llamacpp:tokens_predicted_total 268387
|
||||
# HELP llamacpp:tokens_predicted_seconds_total Predict process time
|
||||
# TYPE llamacpp:tokens_predicted_seconds_total counter
|
||||
llamacpp:tokens_predicted_seconds_total 26565.247833160014
|
||||
llamacpp:tokens_predicted_seconds_total 6520.4400505340027
|
||||
# HELP llamacpp:n_tokens_max Largest observed n_tokens.
|
||||
# TYPE llamacpp:n_tokens_max counter
|
||||
llamacpp:n_tokens_max 252622
|
||||
llamacpp:n_tokens_max 222926
|
||||
# HELP llamacpp:spec_decode_num_drafts_total Total speculative verification rounds
|
||||
# TYPE llamacpp:spec_decode_num_drafts_total counter
|
||||
llamacpp:spec_decode_num_drafts_total 291738
|
||||
llamacpp:spec_decode_num_drafts_total 83278
|
||||
# HELP llamacpp:spec_decode_num_draft_tokens_total Total draft tokens proposed
|
||||
# TYPE llamacpp:spec_decode_num_draft_tokens_total counter
|
||||
llamacpp:spec_decode_num_draft_tokens_total 789028
|
||||
llamacpp:spec_decode_num_draft_tokens_total 234468
|
||||
# HELP llamacpp:spec_decode_num_accepted_tokens_total Total draft tokens accepted
|
||||
# TYPE llamacpp:spec_decode_num_accepted_tokens_total counter
|
||||
llamacpp:spec_decode_num_accepted_tokens_total 578514
|
||||
llamacpp:spec_decode_num_accepted_tokens_total 180837
|
||||
# HELP gufo_spec_lookup_tokens_total Total prompt-lookup copies proposed
|
||||
# TYPE gufo_spec_lookup_tokens_total counter
|
||||
gufo_spec_lookup_tokens_total 17416
|
||||
# HELP gufo_spec_lookup_accepted_tokens_total Total prompt-lookup copies accepted
|
||||
# TYPE gufo_spec_lookup_accepted_tokens_total counter
|
||||
gufo_spec_lookup_accepted_tokens_total 13985
|
||||
# HELP llamacpp:prompt_tokens_seconds Prompt processing speed in tokens per second
|
||||
# TYPE llamacpp:prompt_tokens_seconds gauge
|
||||
llamacpp:prompt_tokens_seconds 160.47362399350274
|
||||
llamacpp:prompt_tokens_seconds 594.36656560014194
|
||||
# HELP llamacpp:predicted_tokens_seconds Generation speed in tokens per second
|
||||
# TYPE llamacpp:predicted_tokens_seconds gauge
|
||||
llamacpp:predicted_tokens_seconds 42.621031668072597
|
||||
llamacpp:predicted_tokens_seconds 53.07443936596944
|
||||
# HELP llamacpp:requests_processing Number of admitted requests, including cache preparation
|
||||
# TYPE llamacpp:requests_processing gauge
|
||||
llamacpp:requests_processing 0
|
||||
|
||||
Vendored
+428
@@ -0,0 +1,428 @@
|
||||
# Verbatim /metrics reply from the radiance engine, recorded by
|
||||
# ./scripts/capture-fixtures.sh — the engine's own Prometheus text
|
||||
# format: the radiance: native series + the vllm: compatibility family.
|
||||
# Volatile values (counters, gauges) move every poll, so a diff here
|
||||
# on re-capture is expected; the contract asserts the series-name set
|
||||
# against RADIANCE_SERIES/RADIANCE_HISTOGRAMS/RADIANCE_OPTIONAL in
|
||||
# sources/llm.py — never the numbers (the nut_*.vars precedent).
|
||||
# HELP vllm:num_requests_running Number of requests currently running on GPU.
|
||||
# TYPE vllm:num_requests_running gauge
|
||||
vllm:num_requests_running{model_name="juggernaut",engine="0"} 0
|
||||
# HELP vllm:num_requests_waiting Number of requests waiting to be processed.
|
||||
# TYPE vllm:num_requests_waiting gauge
|
||||
vllm:num_requests_waiting{model_name="juggernaut",engine="0"} 0
|
||||
# HELP vllm:gpu_cache_usage_perc GPU KV-cache usage. 1 means 100 percent usage.
|
||||
# TYPE vllm:gpu_cache_usage_perc gauge
|
||||
vllm:gpu_cache_usage_perc{model_name="juggernaut",engine="0"} 0.74272018671035767
|
||||
# HELP vllm:kv_cache_usage_perc KV-cache usage. 1 means 100 percent usage.
|
||||
# TYPE vllm:kv_cache_usage_perc gauge
|
||||
vllm:kv_cache_usage_perc{model_name="juggernaut",engine="0"} 0.74272018671035767
|
||||
# HELP radiance:kv_cache_max_tokens Tokens of context the KV cache holds with every block it lends the experts taken back.
|
||||
# TYPE radiance:kv_cache_max_tokens gauge
|
||||
radiance:kv_cache_max_tokens{model_name="juggernaut",engine="0"} 443968
|
||||
# HELP radiance:kv_cache_max_bytes VRAM the KV cache holds with every block it lends the experts taken back, one card.
|
||||
# TYPE radiance:kv_cache_max_bytes gauge
|
||||
radiance:kv_cache_max_bytes{model_name="juggernaut",engine="0"} 15901392896
|
||||
# HELP radiance:kv_resident_blocks Blocks of every streamed KV group the device ring holds; 0 when no group streams.
|
||||
# TYPE radiance:kv_resident_blocks gauge
|
||||
radiance:kv_resident_blocks{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:kv_stream_misses_total Blocks copied into a streamed KV group's ring from the host mirror.
|
||||
# TYPE radiance:kv_stream_misses_total counter
|
||||
radiance:kv_stream_misses_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:kv_stream_bytes_total Bytes a streamed KV group's resolve moved, by direction.
|
||||
# TYPE radiance:kv_stream_bytes_total counter
|
||||
radiance:kv_stream_bytes_total{model_name="juggernaut",engine="0",dir="in"} 0
|
||||
# HELP vllm:gpu_prefix_cache_hit_rate GPU prefix cache block hit rate.
|
||||
# TYPE vllm:gpu_prefix_cache_hit_rate gauge
|
||||
vllm:gpu_prefix_cache_hit_rate{model_name="juggernaut",engine="0"} 0.97886091470718384
|
||||
# HELP vllm:prompt_tokens_total Number of prefill tokens processed.
|
||||
# TYPE vllm:prompt_tokens_total counter
|
||||
vllm:prompt_tokens_total{model_name="juggernaut",engine="0"} 56553179
|
||||
# HELP vllm:generation_tokens_total Number of generation tokens processed.
|
||||
# TYPE vllm:generation_tokens_total counter
|
||||
vllm:generation_tokens_total{model_name="juggernaut",engine="0"} 423779
|
||||
# HELP vllm:num_preemptions_total Cumulative number of preemptions from the engine.
|
||||
# TYPE vllm:num_preemptions_total counter
|
||||
vllm:num_preemptions_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:preempt_forced_total Preempted prefill waits that ended by reclaiming cache blocks and forcing a lowest-priority running victim (--preempt-wait-ms).
|
||||
# TYPE radiance:preempt_forced_total counter
|
||||
radiance:preempt_forced_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP vllm:request_success_total Count of successfully processed requests.
|
||||
# TYPE vllm:request_success_total counter
|
||||
vllm:request_success_total{model_name="juggernaut",engine="0",finished_reason="stop"} 7
|
||||
vllm:request_success_total{model_name="juggernaut",engine="0",finished_reason="length"} 0
|
||||
vllm:request_success_total{model_name="juggernaut",engine="0",finished_reason="tool_calls"} 594
|
||||
vllm:request_success_total{model_name="juggernaut",engine="0",finished_reason="abort"} 2
|
||||
vllm:request_success_total{model_name="juggernaut",engine="0",finished_reason="error"} 0
|
||||
# HELP vllm:time_to_first_token_seconds Histogram of time to first token in seconds.
|
||||
# TYPE vllm:time_to_first_token_seconds histogram
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.001"} 0
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.0050000000000000001"} 0
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.01"} 0
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.02"} 0
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.040000000000000001"} 0
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.059999999999999998"} 0
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.080000000000000002"} 0
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.10000000000000001"} 0
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.25"} 0
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.5"} 0
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.75"} 1
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="1"} 9
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="2.5"} 407
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="5"} 584
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="7.5"} 595
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="10"} 599
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="20"} 603
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="40"} 603
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="80"} 603
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="160"} 603
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="640"} 603
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="2560"} 603
|
||||
vllm:time_to_first_token_seconds_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
|
||||
vllm:time_to_first_token_seconds_sum{model_name="juggernaut",engine="0"} 1448.9084620109995
|
||||
vllm:time_to_first_token_seconds_count{model_name="juggernaut",engine="0"} 603
|
||||
# HELP vllm:time_per_output_token_seconds Histogram of time per output token in seconds.
|
||||
# TYPE vllm:time_per_output_token_seconds histogram
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.01"} 237
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.025000000000000001"} 588
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.050000000000000003"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.074999999999999997"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.10000000000000001"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.14999999999999999"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.20000000000000001"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.29999999999999999"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.40000000000000002"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.5"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="0.75"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="1"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="2.5"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="5"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="7.5"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="10"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="20"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="40"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="80"} 603
|
||||
vllm:time_per_output_token_seconds_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
|
||||
vllm:time_per_output_token_seconds_sum{model_name="juggernaut",engine="0"} 6.9956323654383148
|
||||
vllm:time_per_output_token_seconds_count{model_name="juggernaut",engine="0"} 603
|
||||
# HELP vllm:inter_token_latency_seconds Histogram of inter-token latency in seconds.
|
||||
# TYPE vllm:inter_token_latency_seconds histogram
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.01"} 237
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.025000000000000001"} 588
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.050000000000000003"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.074999999999999997"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.10000000000000001"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.14999999999999999"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.20000000000000001"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.29999999999999999"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.40000000000000002"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.5"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.75"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="1"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="2.5"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="5"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="7.5"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="10"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="20"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="40"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="80"} 603
|
||||
vllm:inter_token_latency_seconds_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
|
||||
vllm:inter_token_latency_seconds_sum{model_name="juggernaut",engine="0"} 6.9956323654383148
|
||||
vllm:inter_token_latency_seconds_count{model_name="juggernaut",engine="0"} 603
|
||||
# HELP vllm:e2e_request_latency_seconds Histogram of end to end request latency in seconds.
|
||||
# TYPE vllm:e2e_request_latency_seconds histogram
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.29999999999999999"} 0
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.5"} 0
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="0.80000000000000004"} 0
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="1"} 0
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="1.5"} 0
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="2"} 10
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="2.5"} 38
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="5"} 271
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="10"} 450
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="15"} 514
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="20"} 538
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="30"} 566
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="40"} 575
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="50"} 585
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="60"} 593
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="120"} 600
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="240"} 602
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="480"} 603
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="960"} 603
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="1920"} 603
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="7680"} 603
|
||||
vllm:e2e_request_latency_seconds_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
|
||||
vllm:e2e_request_latency_seconds_sum{model_name="juggernaut",engine="0"} 6416.8390601850087
|
||||
vllm:e2e_request_latency_seconds_count{model_name="juggernaut",engine="0"} 603
|
||||
# HELP vllm:request_decode_time_seconds Histogram of time spent in the decode phase, in seconds.
|
||||
# TYPE vllm:request_decode_time_seconds histogram
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="0.29999999999999999"} 0
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="0.5"} 2
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="0.80000000000000004"} 44
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="1"} 85
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="1.5"} 155
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="2"} 219
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="2.5"} 274
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="5"} 397
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="10"} 493
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="15"} 531
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="20"} 544
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="30"} 570
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="40"} 579
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="50"} 587
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="60"} 593
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="120"} 600
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="240"} 602
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="480"} 603
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="960"} 603
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="1920"} 603
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="7680"} 603
|
||||
vllm:request_decode_time_seconds_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
|
||||
vllm:request_decode_time_seconds_sum{model_name="juggernaut",engine="0"} 4966.4525284600077
|
||||
vllm:request_decode_time_seconds_count{model_name="juggernaut",engine="0"} 603
|
||||
# HELP vllm:request_prompt_tokens Number of prefill tokens processed.
|
||||
# TYPE vllm:request_prompt_tokens histogram
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="1"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="2"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="5"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="10"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="20"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="50"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="100"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="200"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="500"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="1000"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="2000"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="5000"} 0
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="10000"} 8
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="20000"} 25
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="50000"} 111
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="100000"} 353
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="200000"} 603
|
||||
vllm:request_prompt_tokens_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
|
||||
vllm:request_prompt_tokens_sum{model_name="juggernaut",engine="0"} 56553179
|
||||
vllm:request_prompt_tokens_count{model_name="juggernaut",engine="0"} 603
|
||||
# HELP vllm:request_generation_tokens Number of generation tokens processed.
|
||||
# TYPE vllm:request_generation_tokens histogram
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="1"} 0
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="2"} 0
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="5"} 0
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="10"} 0
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="20"} 0
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="50"} 0
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="100"} 52
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="200"} 244
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="500"} 437
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="1000"} 516
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="2000"} 564
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="5000"} 594
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="10000"} 599
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="20000"} 602
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="50000"} 603
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="100000"} 603
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="200000"} 603
|
||||
vllm:request_generation_tokens_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
|
||||
vllm:request_generation_tokens_sum{model_name="juggernaut",engine="0"} 423779
|
||||
vllm:request_generation_tokens_count{model_name="juggernaut",engine="0"} 603
|
||||
# HELP vllm:request_params_n Histogram of the n request parameter.
|
||||
# TYPE vllm:request_params_n histogram
|
||||
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="1"} 603
|
||||
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="2"} 603
|
||||
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="5"} 603
|
||||
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="10"} 603
|
||||
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="20"} 603
|
||||
vllm:request_params_n_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
|
||||
vllm:request_params_n_sum{model_name="juggernaut",engine="0"} 603
|
||||
vllm:request_params_n_count{model_name="juggernaut",engine="0"} 603
|
||||
# HELP vllm:request_params_max_tokens Histogram of the max_tokens request parameter.
|
||||
# TYPE vllm:request_params_max_tokens histogram
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="1"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="2"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="5"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="10"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="20"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="50"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="100"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="200"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="500"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="1000"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="2000"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="5000"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="10000"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="20000"} 0
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="50000"} 30
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="100000"} 603
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="200000"} 603
|
||||
vllm:request_params_max_tokens_bucket{model_name="juggernaut",engine="0",le="+Inf"} 603
|
||||
vllm:request_params_max_tokens_sum{model_name="juggernaut",engine="0"} 38535168
|
||||
vllm:request_params_max_tokens_count{model_name="juggernaut",engine="0"} 603
|
||||
# HELP radiance:prefill_tokens_per_second Prefill throughput over the scheduler's recent window.
|
||||
# TYPE radiance:prefill_tokens_per_second gauge
|
||||
radiance:prefill_tokens_per_second{model_name="juggernaut",engine="0"} 54.816162109375
|
||||
# HELP radiance:decode_tokens_per_second Decode throughput over the scheduler's recent window.
|
||||
# TYPE radiance:decode_tokens_per_second gauge
|
||||
radiance:decode_tokens_per_second{model_name="juggernaut",engine="0"} 50.574676513671875
|
||||
# HELP radiance:decode_step_seconds_best Fastest single-sequence decode window this process has run, in seconds per step: the mean of 32 consecutive single-stream steps. With speculation a step emits 1 + accepted tokens. Resets on restart.
|
||||
# TYPE radiance:decode_step_seconds_best gauge
|
||||
radiance:decode_step_seconds_best{model_name="juggernaut",engine="0"} 0.032004689000000003
|
||||
# HELP radiance:decode_step_seconds_recent Most recent single-sequence decode window, in seconds per step (see _best). Holds its value while no single-stream window completes.
|
||||
# TYPE radiance:decode_step_seconds_recent gauge
|
||||
radiance:decode_step_seconds_recent{model_name="juggernaut",engine="0"} 0.041546212999999999
|
||||
# HELP radiance:engine_steps_total Scheduler steps executed. One forward pass of the model, plus whatever draft rounds followed it; with speculation a step emits more than one token.
|
||||
# TYPE radiance:engine_steps_total counter
|
||||
radiance:engine_steps_total{model_name="juggernaut",engine="0"} 113985
|
||||
# HELP radiance:engine_steps_prefill_total Scheduler steps carrying only prefill chunks.
|
||||
# TYPE radiance:engine_steps_prefill_total counter
|
||||
radiance:engine_steps_prefill_total{model_name="juggernaut",engine="0"} 1174
|
||||
# HELP radiance:engine_steps_mixed_total Scheduler steps carrying both a prefill chunk and a decode row.
|
||||
# TYPE radiance:engine_steps_mixed_total counter
|
||||
radiance:engine_steps_mixed_total{model_name="juggernaut",engine="0"} 388
|
||||
# HELP radiance:engine_steps_decode_total Scheduler steps carrying only decode rows.
|
||||
# TYPE radiance:engine_steps_decode_total counter
|
||||
radiance:engine_steps_decode_total{model_name="juggernaut",engine="0"} 112423
|
||||
# HELP radiance:engine_steps_ahead_total Steps whose forward pass was issued behind the step before, ahead of its commit.
|
||||
# TYPE radiance:engine_steps_ahead_total counter
|
||||
radiance:engine_steps_ahead_total{model_name="juggernaut",engine="0"} 112279
|
||||
# HELP radiance:engine_prefill_chunks_capped_total Prefill chunks held back so decode rows and waiting requests got into the step (--busy-prefill-chunk).
|
||||
# TYPE radiance:engine_prefill_chunks_capped_total counter
|
||||
radiance:engine_prefill_chunks_capped_total{model_name="juggernaut",engine="0"} 2
|
||||
# HELP radiance:engine_steps_held_total Decode-only steps run after a prefill chunk while other sequences decode (--prefill-interleave).
|
||||
# TYPE radiance:engine_steps_held_total counter
|
||||
radiance:engine_steps_held_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:decode_rows_unspeculated_total Decode rows that took no draft, for any reason.
|
||||
# TYPE radiance:decode_rows_unspeculated_total counter
|
||||
radiance:decode_rows_unspeculated_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:engine_step_seconds_total Seconds spent inside a scheduler step, all shapes.
|
||||
# TYPE radiance:engine_step_seconds_total counter
|
||||
radiance:engine_step_seconds_total{model_name="juggernaut",engine="0"} 5702.6847473990001
|
||||
# HELP radiance:engine_step_seconds_decode_total Seconds spent inside a scheduler step carrying only decode rows.
|
||||
# TYPE radiance:engine_step_seconds_decode_total counter
|
||||
radiance:engine_step_seconds_decode_total{model_name="juggernaut",engine="0"} 4295.7781233209998
|
||||
# HELP radiance:prefill_tokens_total Prompt tokens the engine has computed, exact.
|
||||
# TYPE radiance:prefill_tokens_total counter
|
||||
radiance:prefill_tokens_total{model_name="juggernaut",engine="0"} 2320091
|
||||
# HELP radiance:decode_tokens_total Tokens the engine has generated, exact, counted when the step commits rather than when the request finishes.
|
||||
# TYPE radiance:decode_tokens_total counter
|
||||
radiance:decode_tokens_total{model_name="juggernaut",engine="0"} 423176
|
||||
# HELP radiance:draft_tokens_total Speculative tokens proposed.
|
||||
# TYPE radiance:draft_tokens_total counter
|
||||
radiance:draft_tokens_total{model_name="juggernaut",engine="0"} 850906
|
||||
# HELP radiance:draft_accepted_total Speculative tokens accepted; the pair the acceptance ratio is computed from.
|
||||
# TYPE radiance:draft_accepted_total counter
|
||||
radiance:draft_accepted_total{model_name="juggernaut",engine="0"} 301618
|
||||
# HELP radiance:draft_refills_total Prefix hits ended a drafter window early so a windowed block drafter has context.
|
||||
# TYPE radiance:draft_refills_total counter
|
||||
radiance:draft_refills_total{model_name="juggernaut",engine="0"} 323
|
||||
# HELP radiance:draft_refill_tokens_total Cached prompt tokens those hits gave back to be recomputed.
|
||||
# TYPE radiance:draft_refill_tokens_total counter
|
||||
radiance:draft_refill_tokens_total{model_name="juggernaut",engine="0"} 661504
|
||||
# HELP radiance:lookup_proposals_total Prompt-lookup proposals made, with prompt lookup the only drafter.
|
||||
# TYPE radiance:lookup_proposals_total counter
|
||||
radiance:lookup_proposals_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:lookup_held_total Prompt-lookup proposals held back unverified: not expected to pay for their verify rows at the measured step cost.
|
||||
# TYPE radiance:lookup_held_total counter
|
||||
radiance:lookup_held_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:lookup_held_drafts_total Drafts in the held-back proposals.
|
||||
# TYPE radiance:lookup_held_drafts_total counter
|
||||
radiance:lookup_held_drafts_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:lookup_held_hits_total Of the held-back proposals scored against the text that followed, the drafts that came true: what verifying them would have accepted.
|
||||
# TYPE radiance:lookup_held_hits_total counter
|
||||
radiance:lookup_held_hits_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:linear_prefix_cache_hit_rate Fraction of prompt tokens served from a linear-state checkpoint (spec 7.3).
|
||||
# TYPE radiance:linear_prefix_cache_hit_rate gauge
|
||||
radiance:linear_prefix_cache_hit_rate{model_name="juggernaut",engine="0"} 0.95897507667541504
|
||||
# HELP radiance:attn_cached_tokens_total Prompt tokens served from the paged attention prefix cache.
|
||||
# TYPE radiance:attn_cached_tokens_total counter
|
||||
radiance:attn_cached_tokens_total{model_name="juggernaut",engine="0"} 55357696
|
||||
# HELP radiance:linear_cached_tokens_total Prompt tokens served from a linear-state checkpoint.
|
||||
# TYPE radiance:linear_cached_tokens_total counter
|
||||
radiance:linear_cached_tokens_total{model_name="juggernaut",engine="0"} 54233088
|
||||
# HELP radiance:linear_checkpoints_written_total Linear-state checkpoints written.
|
||||
# TYPE radiance:linear_checkpoints_written_total counter
|
||||
radiance:linear_checkpoints_written_total{model_name="juggernaut",engine="0"} 825
|
||||
# HELP radiance:prefix_cache_evictions_total Prefix-cache entries evicted (attention blocks).
|
||||
# TYPE radiance:prefix_cache_evictions_total counter
|
||||
radiance:prefix_cache_evictions_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:prefix_cache_ckpt_evictions_total Prefix-cache linear-state checkpoints evicted.
|
||||
# TYPE radiance:prefix_cache_ckpt_evictions_total counter
|
||||
radiance:prefix_cache_ckpt_evictions_total{model_name="juggernaut",engine="0"} 437
|
||||
# HELP radiance:prefix_cache_expired_total Prefix-cache entries no request used for --prefix-cache-ttl, taken out.
|
||||
# TYPE radiance:prefix_cache_expired_total counter
|
||||
radiance:prefix_cache_expired_total{model_name="juggernaut",engine="0"} 35942
|
||||
# HELP radiance:prefix_cache_ckpt_expired_total Prefix-cache linear-state checkpoints taken out by --prefix-cache-ttl.
|
||||
# TYPE radiance:prefix_cache_ckpt_expired_total counter
|
||||
radiance:prefix_cache_ckpt_expired_total{model_name="juggernaut",engine="0"} 33
|
||||
# HELP radiance:prefix_cache_ckpt_evictions_by_reason_total Linear-state checkpoints evicted, by reason: released by --checkpoint-policy, or the least recently used of a store retention left full.
|
||||
# TYPE radiance:prefix_cache_ckpt_evictions_by_reason_total counter
|
||||
radiance:prefix_cache_ckpt_evictions_by_reason_total{model_name="juggernaut",engine="0",reason="retention"} 437
|
||||
radiance:prefix_cache_ckpt_evictions_by_reason_total{model_name="juggernaut",engine="0",reason="store_full"} 0
|
||||
# HELP radiance:prefix_cache_released_total Prefix-cache entries and linear-state snapshots a client released on purpose (DELETE /sessions).
|
||||
# TYPE radiance:prefix_cache_released_total counter
|
||||
radiance:prefix_cache_released_total{model_name="juggernaut",engine="0",what="entries"} 0
|
||||
radiance:prefix_cache_released_total{model_name="juggernaut",engine="0",what="snapshots"} 0
|
||||
# HELP radiance:prefix_cache_ckpt_refused_total Checkpoints not written because every snapshot slot was in use by a restore in the same step.
|
||||
# TYPE radiance:prefix_cache_ckpt_refused_total counter
|
||||
radiance:prefix_cache_ckpt_refused_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:prefix_hit_cuts_total Admissions whose prefix hit was cut, by reason: reused nothing for want of a snapshot at or below the hit, for a windowed group no window snapshot covers, or for blocks that could not be adopted; or resumed at a snapshot more than one checkpoint interval below the hit.
|
||||
# TYPE radiance:prefix_hit_cuts_total counter
|
||||
radiance:prefix_hit_cuts_total{model_name="juggernaut",engine="0",reason="no_snapshot"} 0
|
||||
radiance:prefix_hit_cuts_total{model_name="juggernaut",engine="0",reason="windowed_group"} 0
|
||||
radiance:prefix_hit_cuts_total{model_name="juggernaut",engine="0",reason="adopt_failed"} 0
|
||||
radiance:prefix_hit_cuts_total{model_name="juggernaut",engine="0",reason="resumed_below"} 136
|
||||
# HELP radiance:prefix_restore_cuts_total Restores from the host tier the pool could not hold whole, even after giving up the VRAM copies of other stored conversations; each came back only in part, on a model with recurrent state up to the deepest snapshot the room reaches.
|
||||
# TYPE radiance:prefix_restore_cuts_total counter
|
||||
radiance:prefix_restore_cuts_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:prefix_hit_cut_tokens_total Prompt tokens the attention cache held and admission computed again, because the hit resumed at a snapshot below it or reused nothing.
|
||||
# TYPE radiance:prefix_hit_cut_tokens_total counter
|
||||
radiance:prefix_hit_cut_tokens_total{model_name="juggernaut",engine="0"} 1124608
|
||||
# HELP radiance:draft_acceptance_ratio Fraction of speculative draft tokens accepted.
|
||||
# TYPE radiance:draft_acceptance_ratio gauge
|
||||
radiance:draft_acceptance_ratio{model_name="juggernaut",engine="0"} 0.35446688532829285
|
||||
# HELP radiance:draft_position_acceptance_ratio Conditional acceptance at each draft position, given every earlier one was taken.
|
||||
# TYPE radiance:draft_position_acceptance_ratio gauge
|
||||
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="0"} 0.73114069004096804
|
||||
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="1"} 0.72421126063279173
|
||||
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="2"} 0.73612988425386472
|
||||
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="3"} 0.74895000105527532
|
||||
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="4"} 0.76621766330383811
|
||||
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="5"} 0.7862449429937477
|
||||
radiance:draft_position_acceptance_ratio{model_name="juggernaut",engine="0",position="6"} 0.79249695949106558
|
||||
# HELP radiance:draft_position_reached_total Verifies that evaluated each draft position.
|
||||
# TYPE radiance:draft_position_reached_total counter
|
||||
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="0"} 121558
|
||||
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="1"} 88876
|
||||
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="2"} 64365
|
||||
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="3"} 47381
|
||||
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="4"} 35486
|
||||
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="5"} 27190
|
||||
radiance:draft_position_reached_total{model_name="juggernaut",engine="0",position="6"} 21378
|
||||
# HELP radiance:draft_position_accepted_total Verifies that accepted each draft position.
|
||||
# TYPE radiance:draft_position_accepted_total counter
|
||||
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="0"} 88876
|
||||
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="1"} 64365
|
||||
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="2"} 47381
|
||||
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="3"} 35486
|
||||
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="4"} 27190
|
||||
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="5"} 21378
|
||||
radiance:draft_position_accepted_total{model_name="juggernaut",engine="0",position="6"} 16942
|
||||
# HELP radiance:num_requests_preempted Requests currently preempted and awaiting recompute.
|
||||
# TYPE radiance:num_requests_preempted gauge
|
||||
radiance:num_requests_preempted{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:http_queue_depth Requests admitted by the HTTP layer and not yet finished.
|
||||
# TYPE radiance:http_queue_depth gauge
|
||||
radiance:http_queue_depth{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:http_admission_rejected_total Requests refused with 429 because the admission queue was full.
|
||||
# TYPE radiance:http_admission_rejected_total counter
|
||||
radiance:http_admission_rejected_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:client_disconnect_total Streams cancelled because the client went away.
|
||||
# TYPE radiance:client_disconnect_total counter
|
||||
radiance:client_disconnect_total{model_name="juggernaut",engine="0"} 2
|
||||
# HELP radiance:stream_overflow_total Requests failed because the HTTP reader could not drain the sink.
|
||||
# TYPE radiance:stream_overflow_total counter
|
||||
radiance:stream_overflow_total{model_name="juggernaut",engine="0"} 0
|
||||
# HELP radiance:http_responses_total HTTP responses by status class.
|
||||
# TYPE radiance:http_responses_total counter
|
||||
radiance:http_responses_total{model_name="juggernaut",engine="0",class="2xx"} 655
|
||||
radiance:http_responses_total{model_name="juggernaut",engine="0",class="4xx"} 0
|
||||
radiance:http_responses_total{model_name="juggernaut",engine="0",class="5xx"} 0
|
||||
Vendored
+5
@@ -0,0 +1,5 @@
|
||||
# Verbatim /v1/models reply from the radiance engine, recorded by
|
||||
# ./scripts/capture-fixtures.sh — one JSON document. The model id and
|
||||
# max_model_len are stable; created is an epoch timestamp that moves.
|
||||
# The contract asserts the object/data shape — never the numbers.
|
||||
{"object":"list","data":[{"id":"juggernaut","object":"model","created":1793555024,"owned_by":"radiance","max_model_len":262144,"capabilities":{"chat":true,"tools":true,"reasoning":true,"grammar":true,"embedding":false,"image":true,"video":true,"audio":false,"logprobs":true,"image_generation":false,"image_edit":false,"audio_generation":false,"video_generation":false,"decision":null,"reasoning_efforts":["none","minimal","low","medium","xhigh"]}}]}
|
||||
Vendored
+4
-3
@@ -9,7 +9,7 @@ VAR apc battery.charge.low "10"
|
||||
VAR apc battery.charge.warning "50"
|
||||
VAR apc battery.date "2001/09/25"
|
||||
VAR apc battery.mfr.date "2021/06/24"
|
||||
VAR apc battery.runtime "621"
|
||||
VAR apc battery.runtime "264"
|
||||
VAR apc battery.runtime.low "120"
|
||||
VAR apc battery.type "PbAc"
|
||||
VAR apc battery.voltage "27.3"
|
||||
@@ -36,13 +36,14 @@ VAR apc driver.version.usb "libusb-1.0.30 (API: 0x0100010C)"
|
||||
VAR apc input.sensitivity "medium"
|
||||
VAR apc input.transfer.high "144"
|
||||
VAR apc input.transfer.low "88"
|
||||
VAR apc input.voltage "124.0"
|
||||
VAR apc input.transfer.reason "input voltage out of range"
|
||||
VAR apc input.voltage "115.0"
|
||||
VAR apc input.voltage.nominal "120"
|
||||
VAR apc ups.beeper.status "enabled"
|
||||
VAR apc ups.delay.shutdown "20"
|
||||
VAR apc ups.firmware "952.e5 .D"
|
||||
VAR apc ups.firmware.aux "e5 "
|
||||
VAR apc ups.load "46"
|
||||
VAR apc ups.load "81"
|
||||
VAR apc ups.mfr "American Power Conversion"
|
||||
VAR apc ups.mfr.date "2021/01/08"
|
||||
VAR apc ups.model "Back-UPS RS 1500MS"
|
||||
|
||||
Vendored
+6
-6
@@ -8,7 +8,7 @@ VAR cyberpower battery.charge "100"
|
||||
VAR cyberpower battery.charge.low "10"
|
||||
VAR cyberpower battery.charge.warning "20"
|
||||
VAR cyberpower battery.mfr.date "CPS"
|
||||
VAR cyberpower battery.runtime "3350"
|
||||
VAR cyberpower battery.runtime "3525"
|
||||
VAR cyberpower battery.runtime.low "300"
|
||||
VAR cyberpower battery.status "100%"
|
||||
VAR cyberpower battery.type "PbAcid"
|
||||
@@ -36,20 +36,20 @@ VAR cyberpower driver.version.usb "libusb-1.0.30 (API: 0x0100010C)"
|
||||
VAR cyberpower input.sensitivity "normal"
|
||||
VAR cyberpower input.transfer.high "139"
|
||||
VAR cyberpower input.transfer.low "100"
|
||||
VAR cyberpower input.voltage "122.0"
|
||||
VAR cyberpower input.voltage "115.0"
|
||||
VAR cyberpower input.voltage.nominal "120"
|
||||
VAR cyberpower output.voltage "122.0"
|
||||
VAR cyberpower output.voltage "115.0"
|
||||
VAR cyberpower ups.beeper.status "disabled"
|
||||
VAR cyberpower ups.delay.shutdown "60"
|
||||
VAR cyberpower ups.delay.start "120"
|
||||
VAR cyberpower ups.firmware "CR01906AC911"
|
||||
VAR cyberpower ups.load "14"
|
||||
VAR cyberpower ups.load "13"
|
||||
VAR cyberpower ups.mfr "CPS"
|
||||
VAR cyberpower ups.model "GX1500U"
|
||||
VAR cyberpower ups.power "132"
|
||||
VAR cyberpower ups.power "130"
|
||||
VAR cyberpower ups.power.nominal "1500"
|
||||
VAR cyberpower ups.productid "0601"
|
||||
VAR cyberpower ups.realpower "126"
|
||||
VAR cyberpower ups.realpower "121"
|
||||
VAR cyberpower ups.realpower.nominal "900"
|
||||
VAR cyberpower ups.serial "SCRUBBEDSERIAL1"
|
||||
VAR cyberpower ups.status "OL"
|
||||
|
||||
+1
-1
@@ -20,7 +20,7 @@ Namespace 1 Size/Capacity: 2,000,398,934,016 [2.00 TB]
|
||||
Namespace 1 Utilization: 1,616,151,379,968 [1.61 TB]
|
||||
Namespace 1 Formatted LBA Size: 512
|
||||
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
|
||||
Local Time is: Sat Oct 17 23:55:46 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
|
||||
=== START OF SMART DATA SECTION ===
|
||||
SMART overall-health self-assessment test result: PASSED
|
||||
|
||||
+1
-1
@@ -20,7 +20,7 @@ Namespace 1 Size/Capacity: 250,059,350,016 [250 GB]
|
||||
Namespace 1 Utilization: 30,404,874,240 [30.4 GB]
|
||||
Namespace 1 Formatted LBA Size: 512
|
||||
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
|
||||
Local Time is: Sat Oct 17 23:55:47 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
|
||||
=== START OF SMART DATA SECTION ===
|
||||
SMART overall-health self-assessment test result: PASSED
|
||||
|
||||
+1
-1
@@ -19,7 +19,7 @@ Number of Namespaces: 1
|
||||
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
|
||||
Namespace 1 Formatted LBA Size: 512
|
||||
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
|
||||
Local Time is: Sat Oct 17 23:55:47 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
|
||||
=== START OF SMART DATA SECTION ===
|
||||
SMART overall-health self-assessment test result: PASSED
|
||||
|
||||
+1
-1
@@ -19,7 +19,7 @@ Number of Namespaces: 1
|
||||
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
|
||||
Namespace 1 Formatted LBA Size: 512
|
||||
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
|
||||
Local Time is: Sat Oct 17 23:55:48 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
|
||||
=== START OF SMART DATA SECTION ===
|
||||
SMART overall-health self-assessment test result: PASSED
|
||||
|
||||
+1
-1
@@ -19,7 +19,7 @@ Number of Namespaces: 1
|
||||
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
|
||||
Namespace 1 Formatted LBA Size: 512
|
||||
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
|
||||
Local Time is: Sat Oct 17 23:55:48 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
|
||||
=== START OF SMART DATA SECTION ===
|
||||
SMART overall-health self-assessment test result: PASSED
|
||||
|
||||
+1
-1
@@ -20,7 +20,7 @@ Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
|
||||
Namespace 1 Utilization: 699,798,638,592 [699 GB]
|
||||
Namespace 1 Formatted LBA Size: 512
|
||||
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
|
||||
Local Time is: Sat Oct 17 23:55:49 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
|
||||
=== START OF SMART DATA SECTION ===
|
||||
SMART overall-health self-assessment test result: PASSED
|
||||
|
||||
+1
-1
@@ -19,7 +19,7 @@ Number of Namespaces: 1
|
||||
Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
|
||||
Namespace 1 Formatted LBA Size: 512
|
||||
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
|
||||
Local Time is: Sat Oct 17 23:55:49 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
|
||||
=== START OF SMART DATA SECTION ===
|
||||
SMART overall-health self-assessment test result: PASSED
|
||||
|
||||
+1
-1
@@ -20,7 +20,7 @@ Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB]
|
||||
Namespace 1 Utilization: 694,159,712,256 [694 GB]
|
||||
Namespace 1 Formatted LBA Size: 512
|
||||
Namespace 1 IEEE EUI-64: SCRUBBEDEUI64
|
||||
Local Time is: Sat Oct 17 23:55:50 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
|
||||
=== START OF SMART DATA SECTION ===
|
||||
SMART overall-health self-assessment test result: PASSED
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
|
||||
Device is: In smartctl database 7.5/6204
|
||||
ATA Version is: ACS-4 (minor revision not indicated)
|
||||
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
|
||||
Local Time is: Sat Oct 17 23:55:50 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
SMART support is: Available - device has SMART capability.
|
||||
SMART support is: Enabled
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
|
||||
Device is: In smartctl database 7.5/6204
|
||||
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
|
||||
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
|
||||
Local Time is: Sat Oct 17 23:55:51 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
SMART support is: Available - device has SMART capability.
|
||||
SMART support is: Enabled
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
|
||||
Device is: In smartctl database 7.5/6204
|
||||
ATA Version is: ACS-3 T13/2161-D revision 5
|
||||
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
|
||||
Local Time is: Sat Oct 17 23:55:52 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
SMART support is: Available - device has SMART capability.
|
||||
SMART support is: Enabled
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
|
||||
Device is: In smartctl database 7.5/6204
|
||||
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
|
||||
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
|
||||
Local Time is: Sat Oct 17 23:55:52 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
SMART support is: Available - device has SMART capability.
|
||||
SMART support is: Enabled
|
||||
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ Form Factor: 3.5 inches
|
||||
Device is: Not in smartctl database 7.5/6204
|
||||
ATA Version is: ACS-4 published, ANSI INCITS 529-2018
|
||||
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
|
||||
Local Time is: Sat Oct 17 23:55:53 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
SMART support is: Available - device has SMART capability.
|
||||
SMART support is: Enabled
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
|
||||
Device is: In smartctl database 7.5/6204
|
||||
ATA Version is: ACS-3 T13/2161-D revision 5
|
||||
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
|
||||
Local Time is: Sat Oct 17 23:55:53 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
SMART support is: Available - device has SMART capability.
|
||||
SMART support is: Enabled
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
|
||||
Device is: In smartctl database 7.5/6204
|
||||
ATA Version is: ACS-3 T13/2161-D revision 5
|
||||
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
|
||||
Local Time is: Sat Oct 17 23:55:54 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
SMART support is: Available - device has SMART capability.
|
||||
SMART support is: Enabled
|
||||
|
||||
|
||||
+1
-1
@@ -18,7 +18,7 @@ Form Factor: 3.5 inches
|
||||
Device is: In smartctl database 7.5/6204
|
||||
ATA Version is: ACS-3 T13/2161-D revision 5
|
||||
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
|
||||
Local Time is: Sat Oct 17 23:55:54 2026 EDT
|
||||
Local Time is: (time of capture removed by scripts/capture-fixtures.sh)
|
||||
SMART support is: Available - device has SMART capability.
|
||||
SMART support is: Enabled
|
||||
|
||||
|
||||
Vendored
+9
@@ -0,0 +1,9 @@
|
||||
# Verbatim /stats reply from the radiance engine, recorded by
|
||||
# ./scripts/capture-fixtures.sh — one JSON document, the engine's own
|
||||
# shape. Volatile values (uptime_s, now_ms, the counters, kv_*) move
|
||||
# every poll, so a diff here on re-capture is expected; the contract
|
||||
# asserts the top-level key set and the cards[]/requests[]/
|
||||
# draft_positions[] shape — never those numbers (the nut_*.vars
|
||||
# precedent).
|
||||
# in_flight was 0 at capture (requests[] has 0 row(s)).
|
||||
{"model":"juggernaut","version":"1.15.1","uptime_s":15215,"started_at":1793555024,"startup_s":48,"in_flight":0,"queue_cap":32,"now_ms":1793570239000,"running":0,"waiting":0,"preempted":0,"kv_util":0.7427201867103577,"kv_tokens_used":329744,"kv_tokens_total":443968,"kv_tokens_cached":329744,"kv_bytes_used":10805051392.0,"kv_bytes_total":15901392896.0,"kv_bytes_carved":15901392896.0,"kv_tokens_max":443968,"prefix_hit_rate":0.9788609147071838,"prefix_evictions":0.0,"linear_hit_rate":0.958975076675415,"linear_cached_tokens":54233088.0,"checkpoints_written":825.0,"ckpt_evictions":{"retention":437,"store_full":0},"ckpt_refused":0,"released":{"entries":0,"snapshots":0},"hit_cuts":{"no_snapshot":0,"windowed_group":0,"adopt_failed":0,"resumed_below":136},"hit_cut_tokens":1124608.0,"prefill_tps":50.586456298828125,"decode_tps":46.6722526550293,"draft_accept":0.35446688532829285,"preemptions":0,"total_requests":603,"steps":113985,"steps_prefill":1174,"steps_mixed":388,"prefill_chunks_capped":2,"steps_decode":112423,"steps_held":0,"step_ns":5702684747399,"step_ns_decode":4295778123321,"step_ns_decode_best":32004689,"step_ns_decode_recent":41546213,"rows_unspec":0,"draft_positions":[{"reached":121558,"accepted":88876,"rate":0.731140690040968},{"reached":88876,"accepted":64365,"rate":0.7242112606327917},{"reached":64365,"accepted":47381,"rate":0.7361298842538647},{"reached":47381,"accepted":35486,"rate":0.7489500010552753},{"reached":35486,"accepted":27190,"rate":0.7662176633038381},{"reached":27190,"accepted":21378,"rate":0.7862449429937477},{"reached":21378,"accepted":16942,"rate":0.7924969594910656}],"requests":[],"cards":[{"index":0,"name":"AMD Radeon AI PRO R9700","vram_used":34159771648,"vram_total":34208743424}],"link":{"world":1,"ar_calls":0,"ar_bytes":0,"h2d_bytes":0,"d2h_bytes":0,"ssd_reads":0,"ssd_bytes":0,"ssd_staging_bytes":0,"ssd_promoted_bytes":0,"ssd_deposits":0,"ssd_deposit_bytes":0,"pool_evictions":0,"prefetched":0,"prefetch_bytes":0,"staged_bytes":0,"stream_bytes":0,"kv_stream_bytes":0,"kv_streaming":false,"kv_stream_active":false},"passes":{"played":335897,"recorded":2354,"audited":1382,"issued_unkeyed":29,"issued_refused":0,"issued_first":1546,"dispatched":85311704,"unordered":7386027}}
|
||||
+303
-12
@@ -45,7 +45,9 @@ from dashboards.sources.disks import SMART_SERIES, DisksCollector, parse_disks_s
|
||||
from dashboards.sources.host import HostCollector, parse_host_snapshot
|
||||
from dashboards.sources.llm import (
|
||||
HISTORY_METRICS,
|
||||
engine_result,
|
||||
parse_gufo_metrics,
|
||||
parse_radiance_metrics,
|
||||
parse_vllm_metrics,
|
||||
)
|
||||
from dashboards.store import RES_RAW, Event, Sample, SampleStore
|
||||
@@ -937,6 +939,152 @@ def test_config_reports_the_window_the_llm_rates_are_actually_measured_over(
|
||||
assert llm["rate_window_seconds"] == llm["interval_seconds"] == 2.0
|
||||
|
||||
|
||||
# ---- the llm engine registry (phase 34 task 02) --------------------------- #
|
||||
|
||||
def test_config_serves_radiance_additively_beside_byte_identical_gufo_and_vllm(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The additive pin: `radiance` joins the llm block as a third key, and nothing about
|
||||
`gufo`/`vllm` moves — same keys, same shape, same order — while an unset radiance is
|
||||
present as `enabled: false` (a key that appeared only when configured would be the
|
||||
two-engine world leaking back into the API). The redaction is the one rule:
|
||||
`RADIANCE_API_KEY` is the documented way to authenticate, and a token written into
|
||||
the URL instead is stripped before it is served (invariant 6), like the other two.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch) as plain:
|
||||
before = plain.get("/api/config").json()["llm"]
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
radiance_url="http://SEKRIT-RADIANCE-TOKEN@driveripper:8000",
|
||||
radiance_api_key="radiance-key") as with_radiance:
|
||||
after = with_radiance.get("/api/config").json()["llm"]
|
||||
assert before["gufo"] == after["gufo"]
|
||||
assert before["vllm"] == after["vllm"]
|
||||
assert before["radiance"] == {"enabled": False, "base_url": "", "auth": False}
|
||||
assert after["radiance"] == {"enabled": True, "base_url": "http://driveripper:8000",
|
||||
"auth": True}
|
||||
assert "SEKRIT-RADIANCE-TOKEN" not in json.dumps(after), "the URL token must not be served"
|
||||
assert list(after) == ["gufo", "vllm", "radiance", "interval_seconds", "poll_ms",
|
||||
"rate_window_seconds"], "the registry's stable order is the byte order"
|
||||
|
||||
|
||||
def test_the_llm_404_gate_answers_from_the_registry_in_all_configurations(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""One engine configured — whichever of the three — is a configured source, and the
|
||||
404 detail names all three variable names so the not-configured arm says the fix
|
||||
for the whole world, not just the two it used to know."""
|
||||
single: list[dict[str, Any]] = [{"gufo_url": "http://deskwork:8731"},
|
||||
{"vllm_url": "http://vllm-box:8000"},
|
||||
{"radiance_url": "http://driveripper:8000"}]
|
||||
for overrides in single:
|
||||
with build_client(tmp_path, monkeypatch, **overrides) as test_client:
|
||||
response = test_client.get("/api/llm/snapshot")
|
||||
assert response.status_code == 200, f"only {next(iter(overrides))} configured must not 404"
|
||||
with build_client(tmp_path, monkeypatch) as test_client:
|
||||
response = test_client.get("/api/llm/snapshot")
|
||||
assert response.status_code == 404
|
||||
body = response.json()
|
||||
assert body["configured"] is False
|
||||
for name in ("GUFO_BASE_URL", "VLLM_BASE_URL", "RADIANCE_BASE_URL"):
|
||||
assert name in body["detail"], f"the not-configured arm must name {name}"
|
||||
|
||||
|
||||
def test_llm_history_defaults_list_the_configured_engines_from_the_registry(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The defaults are the registry's configured `key`s in registry order — `key` is the
|
||||
history DB's host key samples are written under, not the display name — and an
|
||||
unconfigured engine is not among them, so it is never requested and never dialled.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
gufo_url="http://deskwork:8731",
|
||||
radiance_url="http://driveripper:8000") as test_client:
|
||||
body = test_client.get("/api/llm/history").json()
|
||||
assert body["engines"] == ["gufo", "radiance"], "the vLLM entry is unconfigured and absent"
|
||||
with build_client(tmp_path, monkeypatch) as test_client:
|
||||
body = test_client.get("/api/llm/history").json()
|
||||
assert body["engines"] == []
|
||||
|
||||
|
||||
def test_summary_llm_enabled_follows_the_registry(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
radiance_url="http://driveripper:8000") as test_client:
|
||||
assert test_client.get("/api/summary").json()["llm"]["enabled"] is True
|
||||
|
||||
|
||||
def test_an_unconfigured_radiance_never_becomes_a_job_that_could_look_broken(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Invariant 3's shape, the `disks` precedent: no configured URL, no `llm:radiance`
|
||||
job at all — so nothing dials a hostname the operator never installed, and no health
|
||||
record exists to turn red. The job appears exactly when the URL does.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch) as test_client:
|
||||
keys = {row["key"] for row in test_client.get("/api/health").json()["sources"]}
|
||||
assert not any(key.startswith("llm:") for key in keys)
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
radiance_url="http://driveripper:8000") as test_client:
|
||||
keys = {row["key"] for row in test_client.get("/api/health").json()["sources"]}
|
||||
assert "llm:radiance" in keys
|
||||
|
||||
|
||||
def _radiance_stats_doc() -> dict[str, Any]:
|
||||
"""The verbatim /stats capture, its `#` header comment lines stripped."""
|
||||
path = Path(__file__).resolve().parents[1] / "tests" / "fixtures" / "stats_radiance.json"
|
||||
if not path.exists():
|
||||
pytest.skip(f"missing fixture {path.name} — run scripts/capture-fixtures.sh --with-llm")
|
||||
body = "\n".join(line for line in path.read_text().splitlines() if not line.startswith("#"))
|
||||
return json.loads(body)
|
||||
|
||||
|
||||
def test_radiance_payload_reaches_the_page_rows_and_rest(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, metrics_text) -> None:
|
||||
"""The render path's input as the browser would receive it: the fixture /stats, through
|
||||
the real `engine_result`, into the `llm:radiance` job — and `/api/llm/snapshot` hands
|
||||
it to the page. At rest (the capture's own empty `requests[]`): the counts-otherwise
|
||||
sentence, the whitelisted stats block, and `queue_cap` from the sidecar. A request in
|
||||
flight: the rows verbatim under `sessions.list`. A dead sidecar: the metrics gauges
|
||||
with their provenance said, and the cap `null`, never `0` (rule 2)."""
|
||||
stats = _radiance_stats_doc()
|
||||
rows = [{"id": 42, "model": "juggernaut", "state": "running",
|
||||
"prompt_tokens": 1_200, "completed_tokens": 250}]
|
||||
with build_client(tmp_path, monkeypatch, radiance_url=DEAD_URL) as test_client:
|
||||
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
|
||||
job = hub.jobs["llm:radiance"]
|
||||
|
||||
job.state.payload = engine_result("radiance", "http://r:8000",
|
||||
metrics_text("radiance"), RateEstimator(),
|
||||
stats=stats, now=1.0).payload
|
||||
rest = next(p for p in test_client.get("/api/llm/snapshot").json()["engines"]
|
||||
if p["engine"] == "radiance")
|
||||
assert rest["sessions"]["list"] == stats["requests"]
|
||||
if not stats["requests"]:
|
||||
assert "counts, not sessions" in rest["sessions"]["detail"]
|
||||
assert rest["stats"]["version"] == "1.15.1"
|
||||
assert isinstance(rest["stats"]["uptime_s"], int) and rest["stats"]["uptime_s"] >= 0, \
|
||||
"uptime is the engine's own seconds — volatile between captures"
|
||||
assert rest["stats"]["cards"][0]["vram_used"] == stats["cards"][0]["vram_used"]
|
||||
assert rest["metrics"]["queue_cap"]["value"] == stats["queue_cap"]
|
||||
assert len(rest["metrics"]["spec_positions"]) == 7
|
||||
|
||||
busy = {**stats, "requests": rows, "running": 1}
|
||||
job.state.payload = engine_result("radiance", "http://r:8000",
|
||||
metrics_text("radiance"), RateEstimator(),
|
||||
stats=busy, now=1.0).payload
|
||||
again = next(p for p in test_client.get("/api/llm/snapshot").json()["engines"]
|
||||
if p["engine"] == "radiance")
|
||||
assert again["sessions"]["list"] == rows, "the rows are not the verbatim passthrough"
|
||||
assert again["sessions"]["source"] == "radiance /stats"
|
||||
assert again["sessions"]["detail"] is None
|
||||
|
||||
job.state.payload = engine_result("radiance", "http://r:8000",
|
||||
metrics_text("radiance"), RateEstimator(),
|
||||
now=1.0).payload # stats=None: dead sidecar
|
||||
dead = next(p for p in test_client.get("/api/llm/snapshot").json()["engines"]
|
||||
if p["engine"] == "radiance")
|
||||
assert dead["sessions"]["queue_cap"] is None, "null, not 0 — rule 2"
|
||||
assert "gauges" in dead["sessions"]["detail"]
|
||||
assert "stats" not in dead, "no stats block when the sidecar never answered"
|
||||
assert dead["metrics"]["queue_cap"]["value"] is None
|
||||
|
||||
|
||||
def wait_for_state(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
@@ -2995,8 +3143,11 @@ def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client)
|
||||
assert name in js, f"{name} no longer feeds a foot line"
|
||||
assert "p95 n/a" in js
|
||||
assert 'data-foot' in js
|
||||
# engine-scoped rows: gufo never grows a TTFT row, vLLM never a spec-accept row
|
||||
assert "only: ['vllm']" in js and "only: ['gufo']" in js
|
||||
# engine-scoped rows: gufo never grows a TTFT row, radiance joins the vLLM latency
|
||||
# scope (the vLLM-shaped histograms this build carries, task 03) and spec acceptance
|
||||
# (the gauge reuses gufo's key, the same 0-1 scale)
|
||||
assert "only: ['vllm', 'radiance']" in js and "only: ['gufo', 'radiance']" in js
|
||||
assert "only: ['gufo']" in js, "slots busy left its gufo scope"
|
||||
scoped = js[js.index("function sparkCellsFor("):js.index("function enginePanel(")]
|
||||
assert "cell.only" in scoped
|
||||
assert "minSpan: cell.minSpan" in sparks, "cell minSpan never reaches the sparkline"
|
||||
@@ -3061,7 +3212,8 @@ def test_llm_alarm_cells_exist_only_while_there_is_an_incident(client) -> None:
|
||||
# publishes, and no row is printed in the strip *and* in the panel under it.
|
||||
|
||||
#: row key -> the cells `cellValue()` resolves it from (`sessions.*` is the gufo slot count,
|
||||
#: which is not a metrics cell). The served `STRIP_ROWS` table must match these keys exactly.
|
||||
#: `stats.*` radiance's /stats subset — neither is a metrics cell). The served `STRIP_ROWS`
|
||||
#: table must match these keys exactly.
|
||||
STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
|
||||
"prompt": ("prompt_tokens_per_second",),
|
||||
"decode": ("decode_tokens_per_second",),
|
||||
@@ -3072,6 +3224,10 @@ STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
|
||||
"cache_hit": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"),
|
||||
"spec_accept": ("spec_accept_rate",),
|
||||
"queue_mean": ("request_queue_time_seconds__mean",),
|
||||
"cap": ("queue_cap",),
|
||||
"build": ("stats.version",),
|
||||
"uptime": ("stats.uptime_s",),
|
||||
"vram": ("stats.cards",),
|
||||
"preemptions": ("num_preemptions_total",),
|
||||
"device_lost": ("device_lost_total",),
|
||||
}
|
||||
@@ -3079,6 +3235,13 @@ STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
|
||||
#: rows that are incidents, not stats: absent where the engine does not count them.
|
||||
ALARM_ROW_KEYS = ("preemptions", "device_lost")
|
||||
|
||||
#: (row key, engine) pairs the scope keeps even though **this capture** does not publish
|
||||
#: the cell: the series is standard for the engine family, so another build fills it.
|
||||
#: `queue_mean` on radiance — this build publishes no `request_queue_time_seconds` (the
|
||||
#: vLLM-compat family is pinned in RADIANCE_OPTIONAL), so the row prints `n/a` by
|
||||
#: absence, never back-filled. A row the engine could *never* fill is not shown at all.
|
||||
DOCUMENTED_ABSENCE = {("queue_mean", "radiance")}
|
||||
|
||||
|
||||
def _strip_rows(js: str) -> dict[str, str | None]:
|
||||
"""The served `STRIP_ROWS` table: row key -> its `only` scope (`None` = every engine)."""
|
||||
@@ -3150,10 +3313,13 @@ def test_llm_strip_card_is_one_card_per_engine_of_shared_vocabulary(client) -> N
|
||||
|
||||
|
||||
def test_llm_strip_scopes_its_rows_to_the_engines_that_publish_them(client) -> None:
|
||||
"""gufo gets slots and speculative decoding; vLLM gets running and queue time."""
|
||||
"""gufo gets slots; vLLM and radiance get running and queue time; the radiance
|
||||
strip's four own cells (cap / build / uptime / VRAM) are radiance's alone."""
|
||||
rows = _strip_rows(client.get("/static/page-llm.js").text)
|
||||
assert rows["slots_busy"] == "'gufo'" and rows["spec_accept"] == "'gufo'"
|
||||
assert rows["running"] == "'vllm'" and rows["queue_mean"] == "'vllm'"
|
||||
assert rows["slots_busy"] == "'gufo'" and rows["spec_accept"] == "'gufo', 'radiance'"
|
||||
assert rows["running"] == "'vllm', 'radiance'" and rows["queue_mean"] == "'vllm', 'radiance'"
|
||||
for key in ("cap", "build", "uptime", "vram"):
|
||||
assert rows[key] == "'radiance'", f"the {key} row left its radiance scope"
|
||||
assert rows["waiting"] is None and rows["cache_hit"] is None and rows["tokens_total"] is None
|
||||
js = client.get("/static/page-llm.js").text
|
||||
scoped = js[js.index("function stripRowsFor("):js.index("function stripCard(")]
|
||||
@@ -3166,19 +3332,24 @@ def test_llm_strip_rows_name_cells_the_engines_publish(client, metrics_text) ->
|
||||
`STRIP_ROW_PATHS` mirrors the JS table; every metric cell is checked against the
|
||||
capture-parsed metrics of each engine that would render that row, so a rename in
|
||||
`sources/llm.py` fails here instead of printing a permanent `n/a` in the browser, and a
|
||||
row scoped to one engine cannot leak into the other's card.
|
||||
row scoped to one engine cannot leak into the other's card. `DOCUMENTED_ABSENCE` is the
|
||||
one exemption: a series standard for the engine family that this build's capture does
|
||||
not publish (the row prints `n/a` by absence, never back-filled — the scope stays for
|
||||
the builds that do).
|
||||
"""
|
||||
rows = _strip_rows(client.get("/static/page-llm.js").text)
|
||||
assert set(rows) == set(STRIP_ROW_PATHS), \
|
||||
f"the served strip rows drifted: {set(rows) ^ set(STRIP_ROW_PATHS)}"
|
||||
parsed = {"gufo": parse_gufo_metrics(metrics_text("gufo")),
|
||||
"vllm": parse_vllm_metrics(metrics_text("vllm"))}
|
||||
"vllm": parse_vllm_metrics(metrics_text("vllm")),
|
||||
"radiance": parse_radiance_metrics(metrics_text("radiance"))}
|
||||
for key, only in rows.items():
|
||||
paths = STRIP_ROW_PATHS[key]
|
||||
if paths[0].startswith("sessions."):
|
||||
if paths[0].startswith("sessions.") or paths[0].startswith("stats."):
|
||||
continue # its own payload object; pinned on the JS below
|
||||
shown = {engine: metrics for engine, metrics in parsed.items()
|
||||
if not only or engine in only}
|
||||
if (not only or engine in only)
|
||||
and (key, engine) not in DOCUMENTED_ABSENCE}
|
||||
present = [engine for engine in shown if any(path in shown[engine] for path in paths)]
|
||||
if key in ALARM_ROW_KEYS:
|
||||
# an alarm the engine does not count simply is not there: the chip renders only
|
||||
@@ -3743,14 +3914,19 @@ def test_llm_vllm_panel_says_there_is_no_per_slot_json(client) -> None:
|
||||
`sources/llm.py`'s `_sessions()` publishes counts for vLLM and says so in `detail`; the
|
||||
panel turns that into the `llm_monitor` line, with the API's sentence as the tooltip. The
|
||||
gate is the engine's name (`SLOT_ENGINES`), not `sessions.list.length`, so an engine that
|
||||
has not answered yet still shows its label and the honest empty sentence.
|
||||
has not answered yet still shows its label and the honest empty sentence. And the line
|
||||
stays vLLM's: radiance's panel slot section is empty (its per-request rows live in the
|
||||
strip), so a sentence naming vLLM on the radiance card is a line the engine never said.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
markup = _slot_markup(js)
|
||||
branch = markup.split("if (SLOT_ENGINES.indexOf(engine) < 0) {")[1].split("}")[0]
|
||||
branch = markup.split("if (engine === 'vllm') {")[1].split("}")[0]
|
||||
assert 'class="slots-note" data-slots-note' in branch, "the vLLM line lost its element"
|
||||
assert "NO_SLOT_JSON" in branch and "vLLM exposes no per-slot JSON" in js
|
||||
assert '<div class="slots"' not in branch, "the vLLM panel grew a grid it cannot fill"
|
||||
radiance_branch = markup.split("if (SLOT_ENGINES.indexOf(engine) < 0) {")[1].split("}")[0]
|
||||
assert "return ''" in radiance_branch, "the radiance panel re-grew a slot section"
|
||||
assert "NO_SLOT_JSON" not in radiance_branch, "the vLLM sentence leaked onto the radiance panel"
|
||||
body = _slot_js(js)
|
||||
assert body.startswith("function slotValue("), "the grid's code path moved"
|
||||
assert "if (!grid) {" in body, "renderSlotGrid no longer leaves a panel with no grid alone"
|
||||
@@ -3815,6 +3991,80 @@ def test_llm_charts_are_secondary_and_named(client) -> None:
|
||||
assert "height: 110" in js and "height: 165" not in js
|
||||
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
|
||||
assert charts.count("ariaLabel:") == 4, "an /llm chart lost its accessible name"
|
||||
# the radiance strip's 7-bar spec curve (phase 34) joins the named list: `role="img"`
|
||||
# and an `aria-label` computed from the payload's `spec_positions` — a label that
|
||||
# cannot say which ratios it plots is not an accessible name, and an absent label
|
||||
# fails here
|
||||
curve = _code(client.get("/static/page-llm.js").text)
|
||||
curve = curve[curve.index("function renderSpecCurve("):curve.index("function reqCellText(")]
|
||||
assert 'role="img"' in curve, "the spec curve is an unlabelled plot"
|
||||
assert 'aria-label="' in curve and "escapeHtml(label)" in curve, \
|
||||
"the spec curve's label is not the computed one"
|
||||
|
||||
|
||||
def test_the_radiance_spec_curve_is_neutral_and_built_from_the_payload(client) -> None:
|
||||
"""Decision 4, pinned both ways: the 7-bar curve draws in the page's neutral ink,
|
||||
reaches for no verdict token, and its label is computed from the payload — the
|
||||
connector word turns on the first/last published ratio's order, and an absent
|
||||
position stays absent rather than becoming a zero bar (rule 2)."""
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
curve = js[js.index("function renderSpecCurve("):js.index("function reqCellText(")]
|
||||
assert "spec_positions" in curve, "the curve stopped reading the payload's positions"
|
||||
assert "cssVar('--text-faint')" in curve, "the curve left the neutral ink"
|
||||
assert "Math.min(1, row.ratio)" in curve, "a ratio above 1 would stretch a bar"
|
||||
for token in ("--ok", "--warn", "--fail", "severity"):
|
||||
assert token not in curve, f"the spec curve grew a verdict: {token}"
|
||||
assert "falling to" in curve and "rising to" in curve and "holding at" in curve, \
|
||||
"the connector word is not computed from the payload's order"
|
||||
assert "Number.isFinite(row.ratio)" in curve, \
|
||||
"a missing position must stay absent, not zeroed"
|
||||
|
||||
|
||||
def test_radiance_strip_cells_print_na_never_zero_when_stats_never_answered(client) -> None:
|
||||
"""A radiance payload without the /stats block: cap / build / uptime / vram print
|
||||
`n/a`, never `0` — the null guards on the served path (a missing sidecar is an
|
||||
absent number, rule 2), and the number formatter never sees a string value."""
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
value = js[js.index("function cellValue("):js.index("function p95Value(")]
|
||||
assert "case 'cap':" in value and "get('queue_cap')" in value, \
|
||||
"cap stopped reading the /stats passthrough the server degrades to null"
|
||||
assert "stats && typeof stats.version === 'string'" in value, \
|
||||
"build must be the engine's own string or null"
|
||||
assert "Number.isFinite(stats.uptime_s)" in value and "duration(stats.uptime_s)" in value
|
||||
assert "cards.find((c) => c && c.index === 0)" in value, "vram reads the wrong card"
|
||||
assert "(used / GIB).toFixed(1)" in value and "2 ** 30" in js, \
|
||||
"the one conversion is GiB = 2³⁰, labelled in the tip"
|
||||
body = _strip_render_path(js)
|
||||
assert "cellValue(engine, metrics, sessions, row.key, stats)" in body, \
|
||||
"the radiance cells lost the payload's stats block"
|
||||
assert "typeof value.value === 'string' ? value.value" in body, \
|
||||
"a string value hits the number formatter"
|
||||
|
||||
|
||||
def test_radiance_request_rows_render_as_is_and_the_rest_state_says_so(client) -> None:
|
||||
"""The rows are the engine's own fields, as-is: the columns are the union of the
|
||||
keys the rows publish (no invented key), a number prints only when it is a number,
|
||||
anything else its string verbatim — and an empty list hides the table and puts the
|
||||
API's own count-otherwise sentence in the note slot."""
|
||||
js = _code(client.get("/static/page-llm.js").text)
|
||||
card = js[js.index("function stripCard("):js.index("function renderStripCard(")]
|
||||
assert 'data-spec-curve' in card and 'data-reqs' in card and 'data-reqs-note' in card, \
|
||||
"the radiance card lost its curve / rows / note homes"
|
||||
cell = js[js.index("function reqCellText("):js.index("function renderReqs(")]
|
||||
assert "typeof value === 'number' && isFinite(value)" in cell, \
|
||||
"a number prints only when it is a number"
|
||||
assert "typeof value === 'string') return value" in cell, \
|
||||
"anything else prints its string verbatim"
|
||||
reqs = js[js.index("function renderReqs("):js.index("function makeCharts(")]
|
||||
assert "sessions.list" in reqs and "Object.keys(row)" in reqs, \
|
||||
"the columns are a hardcoded list again"
|
||||
assert "indexOf(key) < 0" in reqs, "first-seen order must keep the rows as published"
|
||||
assert "note.hidden = !sessions.detail" in reqs, \
|
||||
"the rest state lost the API's own sentence"
|
||||
assert "note.hidden = true" in reqs, "rows visible: the sentence must leave"
|
||||
assert "table.title = sessions.source" in reqs, "the table's source left its tooltip"
|
||||
# the gufo gate is untouched: the slot grid is gufo's, the radiance rows live in the strip
|
||||
assert "SLOT_ENGINES = ['gufo']" in js
|
||||
|
||||
|
||||
def test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling(client) -> None:
|
||||
@@ -4006,6 +4256,9 @@ def test_the_copy_the_operator_acts_on_survives_the_audit(client) -> None:
|
||||
assert "smartmontools" in pages["/disks"] + scripts["page-disks.js"]
|
||||
assert "disk group" in scripts["page-disks.js"]
|
||||
assert "No engines configured" in pages["/llm"]
|
||||
# the not-configured arm is actionable only if it names the variable for every engine
|
||||
for name in ("GUFO_BASE_URL", "VLLM_BASE_URL", "RADIANCE_BASE_URL"):
|
||||
assert name in pages["/llm"], f"/llm's empty state stopped naming {name}"
|
||||
assert "No UPS reporting" in pages["/ups"]
|
||||
assert "instcmds" in pages["/ups"], "the UPS setup note stopped naming the grant that matters"
|
||||
assert "3493" in pages["/ups"], "the page stopped warning that upsd's port is not always 3493"
|
||||
@@ -10087,6 +10340,44 @@ def test_a_card_disclosure_scrolls_what_it_reveals_at_every_width(client) -> Non
|
||||
"at 1280 px, where it would then do nothing")
|
||||
|
||||
|
||||
def test_the_radiance_reqs_table_scrolls_instead_of_stretching_its_card(client) -> None:
|
||||
"""The radiance strip's request table must not grow the card to its min-content.
|
||||
|
||||
Measured on the live fleet (1280 px, two strip cards across, each 613 px): with a
|
||||
request in flight the `.reqs` table — the union of the row's ten published keys —
|
||||
sat at ~950 px of min-content, and because the strip body is a one-column grid
|
||||
(`.strip > .card > .body`) with an implicit auto column, a grid item's automatic
|
||||
minimum size is its content's min-content width, so the *column* grew to ~946 px.
|
||||
Every row of the card laid out at 946 px inside a 613 px card: the value column of
|
||||
each `dl.kv` row resolved to ~857 px, the right-aligned value span painted at
|
||||
x≈1582 against a card edge at 1260, and `.card { overflow: hidden }` ate it — the
|
||||
strip printed its labels with no values while the request row and the spec curve
|
||||
(their own boxes) painted fine. `min-width: 0` is the first half: without it the
|
||||
table's min-content still sets the column and nothing below can overflow. `overflow-x:
|
||||
auto` is the second: the table reaches its last column by scrolling inside the card,
|
||||
only where it is wider than the card (a table that fits paints no scrollbar —
|
||||
horizontal only, no height cap, no vertical scrollport).
|
||||
"""
|
||||
reqs = _declarations(_css_section(_css(client), "engine slot grid"), ".reqs")
|
||||
assert reqs.get("min-width") == "0", (
|
||||
"the `.reqs` item may not be narrower than its table: without this the ten-column "
|
||||
"request table's min-content grows the strip body's grid column and every value "
|
||||
"cell of the card paints past the card, clipped by `.card { overflow: hidden }`")
|
||||
assert reqs.get("overflow-x") == "auto", (
|
||||
"the request table lost its sideways scroll inside the card — the columns beyond "
|
||||
"the card edge are unreachable again")
|
||||
for forbidden in ("max-height", "overflow-y", "overflow"):
|
||||
assert forbidden not in reqs, (
|
||||
f"the `.reqs` rule declares `{forbidden}`: this is a sideways scroll only — a "
|
||||
"height cap or vertical scrollport here would hide rows the operator has not "
|
||||
"chosen to hide")
|
||||
# the strip body is the grid the item sits in: a one-column body is what the
|
||||
# min-content growth acts on, so the fix is inert if the body stops being a grid
|
||||
body = _declarations(_css_section(_css(client), "engine stats strip"), ".strip > .card > .body")
|
||||
assert body.get("display") == "grid", (
|
||||
"the strip body is not a one-column grid anymore — re-measure before touching `.reqs`")
|
||||
|
||||
|
||||
def test_every_table_on_every_page_carries_the_class_the_rule_targets(client) -> None:
|
||||
"""The mobile rule is class-level, so a `<table>` without `data` would silently not scroll.
|
||||
|
||||
|
||||
+226
-1
@@ -20,6 +20,7 @@ import httpx
|
||||
import pytest
|
||||
|
||||
from dashboards import certs
|
||||
from dashboards.app import build_hub
|
||||
from dashboards.collectors import (
|
||||
AgentTls,
|
||||
NutMonitor,
|
||||
@@ -33,10 +34,12 @@ from dashboards.collectors import (
|
||||
fetch_snapshot,
|
||||
gateway_job,
|
||||
host_jobs,
|
||||
llm_jobs,
|
||||
post_route,
|
||||
)
|
||||
from dashboards.config import Settings, Target
|
||||
from dashboards.poller import Result
|
||||
from dashboards.prom import Prom
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.sources import nut
|
||||
from dashboards.sources.disks import DisksCollector
|
||||
@@ -1121,7 +1124,10 @@ async def test_first_gufo_poll_has_no_measured_rate_yet() -> None:
|
||||
decode = result.payload["metrics"]["decode_tokens_per_second"]
|
||||
assert decode["value"] is None
|
||||
assert decode["source"] is None
|
||||
assert result.payload["sessions"]["active"] == 0.0
|
||||
# `active` is the /metrics gauge, verbatim the fixture's own number — the capture
|
||||
# may have been taken with a request in flight (volatile between re-captures)
|
||||
assert result.payload["sessions"]["active"] == \
|
||||
Prom(GUFO).scalar(["llamacpp:requests_processing"])
|
||||
assert "no /slots answer" in result.payload["sessions"]["detail"]
|
||||
|
||||
|
||||
@@ -1204,6 +1210,225 @@ async def test_engine_timeout_is_504() -> None:
|
||||
assert excinfo.value.status == 504
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# radiance: the /stats sidecar (phase 34, task 04)
|
||||
# --------------------------------------------------------------------------- #
|
||||
RADIANCE = (FIXTURES / "metrics_radiance.txt").read_text()
|
||||
|
||||
|
||||
def _radiance_stats() -> dict[str, Any]:
|
||||
"""The capture's /stats document (the `#` header lines are the script's, not JSON)."""
|
||||
body = "\n".join(line for line in
|
||||
(FIXTURES / "stats_radiance.json").read_text().splitlines()
|
||||
if not line.startswith("#"))
|
||||
return json.loads(body)
|
||||
|
||||
|
||||
def fake_engine(metrics: httpx.Response | BaseException,
|
||||
stats: httpx.Response | BaseException | None = None
|
||||
) -> tuple[httpx.AsyncClient, list[str]]:
|
||||
"""One fake engine: a scripted `/metrics` answer, and a scripted `/stats` sidecar answer
|
||||
if the job dials for it; every requested path is kept, so a retry or an alias shows up.
|
||||
"""
|
||||
seen: list[str] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
seen.append(request.url.path)
|
||||
if request.url.path == "/metrics":
|
||||
if isinstance(metrics, BaseException):
|
||||
raise metrics
|
||||
return metrics
|
||||
if request.url.path == "/stats":
|
||||
assert stats is not None, "the job dialled /stats without one scripted"
|
||||
if isinstance(stats, BaseException):
|
||||
raise stats
|
||||
return stats
|
||||
return httpx.Response(404)
|
||||
|
||||
return httpx.AsyncClient(transport=httpx.MockTransport(handler)), seen
|
||||
|
||||
|
||||
async def test_a_configured_radiance_alone_builds_exactly_one_llm_job() -> None:
|
||||
"""Task 02's registry shape, extended with the job's own fields: a configured
|
||||
`RADIANCE_BASE_URL` (gufo and vLLM unset) is exactly one job, with the LLM dashboard,
|
||||
the registry's display name and the LLM cadence — the same three facts `llm:gufo` and
|
||||
`llm:vllm` already carried."""
|
||||
settings = Settings(radiance_url="http://radiance.test:8000")
|
||||
async with httpx.AsyncClient() as client:
|
||||
jobs = llm_jobs(settings, client, RateEstimator())
|
||||
assert len(jobs) == 1
|
||||
job = jobs[0]
|
||||
assert job.key == "llm:radiance"
|
||||
assert job.dashboard == "llm"
|
||||
assert job.label == "radiance"
|
||||
assert job.interval == settings.llm_interval, "the LLM cadence, not the host's"
|
||||
|
||||
|
||||
async def test_the_hub_wires_the_radiance_job_with_the_llm_cadence_and_estimator(
|
||||
store: Any, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""`app.py` needed no change — this is the check, asserted by running the job through
|
||||
its own `fn` rather than by comment: the `llm:radiance` job flows into the hub from
|
||||
`llm_jobs` with the LLM interval and the LLM-span estimator (one estimator per
|
||||
measuring cadence, phase 23), and the sidecar flag reaches `fetch_engine` from the
|
||||
job's call alone."""
|
||||
seen: dict[str, Any] = {}
|
||||
|
||||
async def spy_fetch(client: Any, engine: str, base_url: str, api_key: str,
|
||||
rates: RateEstimator, **kwargs: Any) -> Result:
|
||||
seen["engine"] = engine
|
||||
seen["rates"] = rates
|
||||
seen["kwargs"] = kwargs
|
||||
return Result(payload={"engine": engine})
|
||||
|
||||
monkeypatch.setattr("dashboards.collectors.fetch_engine", spy_fetch)
|
||||
settings = Settings(radiance_url="http://radiance.test:8000")
|
||||
async with httpx.AsyncClient() as client:
|
||||
hub, _ = build_hub(settings, store, client)
|
||||
assert "llm:radiance" in hub.jobs
|
||||
job = hub.jobs["llm:radiance"]
|
||||
assert job.interval == settings.llm_interval
|
||||
assert await job.run_once(store) is True
|
||||
assert seen["engine"] == "radiance"
|
||||
assert seen["rates"].span == settings.llm_interval, "the LLM-span estimator"
|
||||
assert seen["kwargs"] == {"with_slots": False, "with_stats": True}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("settings, expected", [
|
||||
(Settings(gufo_url="http://gufo.test:8731"), ["/metrics", "/slots"]),
|
||||
(Settings(vllm_url="http://vllm.test:8000"), ["/metrics"]),
|
||||
(Settings(radiance_url="http://radiance.test:8000"), ["/metrics", "/stats"]),
|
||||
])
|
||||
async def test_each_engine_job_dials_only_its_own_sidecars(
|
||||
settings: Settings, expected: list[str]) -> None:
|
||||
"""The grep criterion, proven by behaviour: `llm_jobs`' flag is what makes
|
||||
`fetch_engine` dial `/stats` — the gufo job's `/slots` pass and the vLLM job's bare
|
||||
`/metrics` read behave exactly as before (gufo's alias aside, a 200 `/slots` answers
|
||||
before `/v1/slots` is tried), and radiance is the only engine that dials the sidecar,
|
||||
exactly once and on its unaliased path."""
|
||||
text_by_host = {"gufo.test": GUFO, "vllm.test": VLLM, "radiance.test": RADIANCE}
|
||||
paths: list[str] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
paths.append(request.url.path)
|
||||
if request.url.path == "/metrics":
|
||||
return httpx.Response(200, text=text_by_host[request.url.host])
|
||||
if request.url.path == "/slots":
|
||||
return httpx.Response(200, json=[{"index": 0}])
|
||||
if request.url.path == "/stats":
|
||||
return httpx.Response(200, json=_radiance_stats())
|
||||
return httpx.Response(404)
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
jobs = llm_jobs(settings, client, RateEstimator())
|
||||
assert len(jobs) == 1
|
||||
result = await jobs[0].fn()
|
||||
assert isinstance(result, Result)
|
||||
assert paths == expected, f"{jobs[0].key}: the job dials only its own sidecars"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stats_failure", ["http-500", "non-json", "json-list", "connection-error"])
|
||||
async def test_a_dead_or_partial_stats_sidecar_keeps_the_metrics_poll_healthy(
|
||||
stats_failure: str) -> None:
|
||||
"""Invariant 4, the sidecar's own arm: a `/stats` that is dead, 500, non-JSON, or a
|
||||
JSON that is not a dict degrades the session arm (task 03's missing-stats fallback —
|
||||
the counts from the /metrics gauges, the cap n/a, never 0) and never takes the
|
||||
`/metrics` poll down: one best-effort read, no retry, no second dial, no alias, no
|
||||
`UpstreamError`."""
|
||||
if stats_failure == "http-500":
|
||||
stats: Any = httpx.Response(500, text="sidecar down")
|
||||
elif stats_failure == "non-json":
|
||||
stats = httpx.Response(200, text="not json at all")
|
||||
elif stats_failure == "json-list":
|
||||
stats = httpx.Response(200, json=[1, 2])
|
||||
else:
|
||||
stats = httpx.ConnectError("sidecar down")
|
||||
client, seen = fake_engine(httpx.Response(200, text=RADIANCE), stats)
|
||||
async with client:
|
||||
result = await fetch_engine(client, "radiance", "http://radiance.test:8000",
|
||||
"", RateEstimator(), with_stats=True)
|
||||
assert result.payload["engine"] == "radiance"
|
||||
assert result.payload["metrics"]["num_requests_running"]["value"] == \
|
||||
Prom(RADIANCE).scalar(["vllm:num_requests_running"]), \
|
||||
"the /metrics answer is the poll's one truth, verbatim the fixture's gauge"
|
||||
assert "stats" not in result.payload, "an absent or non-dict sidecar carries nothing"
|
||||
detail = result.payload["sessions"]["detail"]
|
||||
assert "did not answer" in detail, "the session arm names the missing sidecar"
|
||||
assert "gauges" in detail, "…and the fallback counts' different provenance"
|
||||
assert result.payload["sessions"]["queue_cap"] is None, "n/a, never 0 (rule 2)"
|
||||
assert [path for path in seen if path == "/stats"] == ["/stats"], (
|
||||
"one best-effort read, no retry, no second dial")
|
||||
|
||||
|
||||
async def test_a_healthy_stats_sidecar_travels_into_the_result() -> None:
|
||||
client, seen = fake_engine(httpx.Response(200, text=RADIANCE),
|
||||
httpx.Response(200, json=_radiance_stats()))
|
||||
async with client:
|
||||
result = await fetch_engine(client, "radiance", "http://radiance.test:8000",
|
||||
"", RateEstimator(), with_stats=True)
|
||||
block = result.payload["stats"]
|
||||
assert block["model"] == "juggernaut"
|
||||
for key in ("version", "uptime_s", "startup_s", "total_requests", "cards"):
|
||||
assert key in block, "the whitelisted subset the page reads (task 03)"
|
||||
assert block["cards"][0]["name"] == "AMD Radeon AI PRO R9700"
|
||||
assert "vram_used_pct" in block["cards"][0], "the one computed number in the block"
|
||||
sessions = result.payload["sessions"]
|
||||
assert sessions["source"] == "radiance /stats"
|
||||
assert sessions["queue_cap"] == _radiance_stats()["queue_cap"]
|
||||
if not _radiance_stats()["requests"]:
|
||||
assert "counts, not sessions" in sessions["detail"], "the capture is at rest"
|
||||
assert seen == ["/metrics", "/stats"], "one unaliased read, nothing else"
|
||||
|
||||
|
||||
async def test_a_dead_metrics_is_the_upstream_error_and_the_sidecar_never_masks_it() -> None:
|
||||
"""A 500 `/metrics` beside a perfect `/stats`: the failure is the metrics one, and
|
||||
the sidecar is never even consulted — consulted first it could mask the real failure,
|
||||
consulted after it would only be wasted work on a poll that is already failed."""
|
||||
client, seen = fake_engine(httpx.Response(500, text="boom"),
|
||||
httpx.Response(200, json=_radiance_stats()))
|
||||
async with client:
|
||||
with pytest.raises(UpstreamError) as excinfo:
|
||||
await fetch_engine(client, "radiance", "http://radiance.test:8000",
|
||||
"", RateEstimator(), with_stats=True)
|
||||
assert "HTTP 500" in excinfo.value.message
|
||||
assert "/metrics" in excinfo.value.message
|
||||
assert "/stats" not in seen, "a failed metrics dial never reaches for the sidecar"
|
||||
|
||||
|
||||
async def test_a_failing_radiance_never_echoes_credentials_in_its_url() -> None:
|
||||
"""Invariant 6 for the radiance dial, the engine-fetch precedent extended: a
|
||||
`RADIANCE_BASE_URL` written with a userinfo token fails with the redacted URL, and
|
||||
the key-free form leaks nothing either — there is nothing to leak."""
|
||||
client, _ = fake_engine(httpx.ConnectError("connection refused"))
|
||||
async with client:
|
||||
with pytest.raises(UpstreamError, match="unreachable") as excinfo:
|
||||
await fetch_engine(client, "radiance",
|
||||
"http://SEKRIT-RADIANCE-TOKEN@radiance.test:8000", "",
|
||||
RateEstimator(), with_stats=True)
|
||||
assert "SEKRIT-RADIANCE-TOKEN" not in excinfo.value.message
|
||||
assert "http://radiance.test:8000/metrics" in excinfo.value.message
|
||||
|
||||
|
||||
async def test_the_job_list_never_dials_an_unconfigured_engine(
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The registry's gate, spy-pinned at the dial layer (the existing spy pattern): with
|
||||
every engine unset, `llm_jobs` builds nothing, and nothing that was built dials
|
||||
anything — an unconfigured engine has no job to fail and no dial to hang (invariant 3)."""
|
||||
calls: list[str] = []
|
||||
|
||||
async def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
|
||||
calls.append(str(url))
|
||||
raise AssertionError(f"an unconfigured engine was dialled at {url}")
|
||||
|
||||
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
|
||||
settings = Settings()
|
||||
async with httpx.AsyncClient() as client:
|
||||
jobs = llm_jobs(settings, client, RateEstimator())
|
||||
for job in jobs:
|
||||
await job.fn()
|
||||
assert jobs == []
|
||||
assert calls == []
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# NUT monitor
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
@@ -6,11 +6,14 @@ import os
|
||||
from dataclasses import FrozenInstanceError, fields
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from dashboards import config
|
||||
from dashboards.auth import PBKDF2_ITERATIONS
|
||||
from dashboards.collectors import llm_jobs
|
||||
from dashboards.config import Settings, parse_targets
|
||||
from dashboards.rates import RateEstimator
|
||||
|
||||
|
||||
def test_parses_whitespace_and_comma_separated_targets() -> None:
|
||||
@@ -397,3 +400,92 @@ def test_the_pbkdf2_knob_is_for_tests_and_not_a_performance_tune(
|
||||
assert config._int("DASH_PBKDF2_ITERATIONS", 600_000) == 600_000
|
||||
monkeypatch.setenv("DASH_PBKDF2_ITERATIONS", "1000")
|
||||
assert config._int("DASH_PBKDF2_ITERATIONS", 600_000) == 1000
|
||||
|
||||
|
||||
# ---- the llm engine registry (phase 34 task 02) --------------------------- #
|
||||
|
||||
def test_radiance_defaults_to_unset_the_way_the_other_engines_do() -> None:
|
||||
"""Unset means "this deployment does not run it" (invariant 3), like `gufo_url`.
|
||||
|
||||
It must be invisible to `any_source_enabled` too: a deployment that runs no source at
|
||||
all is `empty`, not broken, and an unset engine is the half of the page that
|
||||
disappears, never a red panel.
|
||||
"""
|
||||
settings = Settings()
|
||||
assert settings.radiance_url == ""
|
||||
assert settings.radiance_api_key == ""
|
||||
assert settings.any_source_enabled is False
|
||||
|
||||
|
||||
def test_radiance_settings_come_from_the_environment(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("RADIANCE_BASE_URL", " http://driveripper.example.net:8000/ ")
|
||||
monkeypatch.setenv("RADIANCE_API_KEY", " engine-key ")
|
||||
reloaded = importlib.reload(config)
|
||||
try:
|
||||
assert reloaded.SETTINGS.radiance_url == "http://driveripper.example.net:8000", "`_base_url` strips whitespace and the trailing slash"
|
||||
assert reloaded.SETTINGS.radiance_api_key == "engine-key", "_env() strips"
|
||||
assert reloaded.SETTINGS.any_source_enabled is True
|
||||
finally:
|
||||
for name in ("RADIANCE_BASE_URL", "RADIANCE_API_KEY"):
|
||||
monkeypatch.delenv(name)
|
||||
importlib.reload(config)
|
||||
|
||||
|
||||
def test_llm_engines_is_the_one_registry_in_its_stable_three_engine_order() -> None:
|
||||
"""The order `gufo`, `vllm`, `radiance` is the contract: the UI's card order and the
|
||||
history defaults follow it, so a reordered registry is a page change, not a list edit.
|
||||
`key` is the history DB's host key and the job id; `display` is what a health record
|
||||
and a page print; the api key travels verbatim into the dial's `Authorization` header.
|
||||
"""
|
||||
settings = Settings(gufo_url="http://gufo.example.net:8731", gufo_api_key="gufo-key",
|
||||
vllm_url="http://vllm.example.net:8000", vllm_api_key="vllm-key",
|
||||
radiance_url="http://driveripper.example.net:8000",
|
||||
radiance_api_key="radiance-key")
|
||||
assert settings.llm_engines() == [
|
||||
("gufo", "gufo", "http://gufo.example.net:8731", "gufo-key"),
|
||||
("vllm", "vLLM", "http://vllm.example.net:8000", "vllm-key"),
|
||||
("radiance", "radiance", "http://driveripper.example.net:8000", "radiance-key"),
|
||||
]
|
||||
|
||||
|
||||
def test_llm_engines_lists_an_unset_engine_with_an_empty_url() -> None:
|
||||
"""The registry is the full fleet the way `parse_targets` keeps the host list: an
|
||||
unset engine is *listed*, with an empty `url` — configured-ness is the call site's
|
||||
`bool(url)` test (the `nut_enabled` shape), never a missing entry, so a site that
|
||||
iterates the list cannot silently read a two-engine world.
|
||||
"""
|
||||
entries = Settings().llm_engines()
|
||||
assert [key for key, _display, _url, _api_key in entries] == ["gufo", "vllm", "radiance"]
|
||||
assert [display for _key, display, _url, _api_key in entries] == ["gufo", "vLLM", "radiance"]
|
||||
assert [url for _key, _display, url, _api_key in entries] == ["", "", ""]
|
||||
|
||||
|
||||
def _llm_job_keys(settings: Settings) -> list[str]:
|
||||
"""The job ids `llm_jobs` would build, from the registry, for one settings object.
|
||||
|
||||
The client is never touched by the construction (the jobs close over it for their
|
||||
own poll loops), so a bare, never-entered `AsyncClient` is the right fake — and an
|
||||
unentered one warns nothing and needs no closing.
|
||||
"""
|
||||
client = httpx.AsyncClient()
|
||||
return [job.key for job in llm_jobs(settings, client, RateEstimator())]
|
||||
|
||||
|
||||
def test_radiance_unset_gets_no_job_and_changes_nothing_about_the_source() -> None:
|
||||
"""No job, and `any_source_enabled` is unchanged by it: an unconfigured engine is never
|
||||
dialled and never shows a red record (invariant 3)."""
|
||||
settings = Settings()
|
||||
assert settings.any_source_enabled is False
|
||||
assert _llm_job_keys(settings) == []
|
||||
|
||||
|
||||
def test_radiance_set_gets_its_job_and_enables_the_source() -> None:
|
||||
"""One `llm:radiance` job, in the registry's position, and the same job set a three-engine
|
||||
deployment already had before the third engine existed (gufo/vllm byte-identical)."""
|
||||
settings = Settings(radiance_url="http://driveripper.example.net:8000")
|
||||
assert settings.any_source_enabled is True
|
||||
assert _llm_job_keys(settings) == ["llm:radiance"]
|
||||
settings = Settings(gufo_url="http://gufo.example.net:8731",
|
||||
vllm_url="http://vllm.example.net:8000",
|
||||
radiance_url="http://driveripper.example.net:8000")
|
||||
assert _llm_job_keys(settings) == ["llm:gufo", "llm:vllm", "llm:radiance"]
|
||||
|
||||
@@ -3362,3 +3362,213 @@ def test_the_install_checks_the_agent_can_reach_the_socket(tmp_path: Path) -> No
|
||||
assert result.returncode == 1, "an unreachable socket is a failed install, not a silent success"
|
||||
assert "cannot connect" in result.stderr
|
||||
assert "sudo -u dashboards-agent python3" in call_log.read_text(), "the check ran as the agent uid"
|
||||
|
||||
|
||||
# --- scripts/capture-fixtures.sh: the radiance pass (phase 34, task 01) --------------
|
||||
#
|
||||
# The script learns radiance through the same `--with-llm` gate as the other engines and
|
||||
# carries no deployment detail of its own: RADIANCE_BASE_URL is the dial, `/metrics`
|
||||
# joins the per-engine loop, and `/stats` + `/v1/models` are recorded in the same pass
|
||||
# (radiance is a vLLM derivative; the vLLM engine and env var stay for other deployments).
|
||||
# A stub `curl` answers the three endpoints from canned bodies, so the script's own
|
||||
# branches run — verbatim bodies, the script's header, the at-rest hint, the
|
||||
# never-overwrite-a-good-fixture rule — with no engine in sight.
|
||||
|
||||
RADIANCE_CAPTURE = SCRIPTS / "capture-fixtures.sh"
|
||||
|
||||
#: The stub `curl`: it drops the options the script passes (`-fsS`, `--max-time N`, `-o`),
|
||||
#: records every call, and answers the engine's three read-only endpoints from `$CANNED_*`.
|
||||
#: Nothing here can reach a network.
|
||||
RADIANCE_CURL_STUB = """#!/bin/sh
|
||||
printf '%s\\n' "$*" >> "$STUB_LOG"
|
||||
if [ "${FAIL_CURL:-0}" = "1" ]; then
|
||||
echo "curl: (7) Failed to connect" >&2
|
||||
exit 7
|
||||
fi
|
||||
out=""
|
||||
url=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--max-time) shift 2 ;;
|
||||
-o) out="$2"; shift 2 ;;
|
||||
-*) shift ;;
|
||||
*) url="$1"; shift ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$out" ] || { echo "stub curl: no -o target" >&2; exit 2; }
|
||||
case "$url" in
|
||||
*/metrics) printf '%s' "$CANNED_METRICS" > "$out" ;;
|
||||
*/stats) printf '%s' "$CANNED_STATS" > "$out" ;;
|
||||
*/v1/models) printf '%s' "$CANNED_MODELS" > "$out" ;;
|
||||
*) echo "stub curl: unexpected URL $url" >&2; exit 2 ;;
|
||||
esac
|
||||
"""
|
||||
|
||||
#: The seven per-position spec-acceptance lines: the phase pins position="0".."6".
|
||||
RADIANCE_CANNED_METRICS = (
|
||||
"# HELP vllm:num_requests_running Number of requests currently running on GPU.\n"
|
||||
"# TYPE vllm:num_requests_running gauge\n"
|
||||
'vllm:num_requests_running{model_name="juggernaut",engine="0"} 0\n'
|
||||
'radiance:decode_tokens_per_second{model_name="juggernaut",engine="0"} 110.9\n'
|
||||
+ "".join(
|
||||
f'radiance:draft_position_acceptance_ratio{{model_name="juggernaut",engine="0",'
|
||||
f'position="{i}"}} 0.7\n'
|
||||
for i in range(7)
|
||||
)
|
||||
)
|
||||
|
||||
RADIANCE_CANNED_STATS = (
|
||||
'{"model":"juggernaut","version":"1.15.1","uptime_s":67490,"queue_cap":24,'
|
||||
'"in_flight":0,"kv_util":0.7241322994232178,"prefix_hit_rate":0.9572382569313049,'
|
||||
'"draft_positions":[{"reached":701741,"accepted":516252,"rate":0.7356731329650114}],'
|
||||
'"requests":[],"cards":[{"index":0,"name":"AMD Radeon AI PRO R9700",'
|
||||
'"vram_used":32119517184,"vram_total":34208743424}]}'
|
||||
)
|
||||
|
||||
RADIANCE_CANNED_STATS_BUSY = (
|
||||
'{"model":"juggernaut","version":"1.15.1","uptime_s":67500,"queue_cap":24,'
|
||||
'"in_flight":1,"kv_util":0.72,"prefix_hit_rate":0.95,'
|
||||
'"draft_positions":[{"reached":701741,"accepted":516252,"rate":0.735}],'
|
||||
'"requests":[{"id":"req-1","queue_pos":1}],'
|
||||
'"cards":[{"index":0,"name":"AMD Radeon AI PRO R9700",'
|
||||
'"vram_used":32119517184,"vram_total":34208743424}]}'
|
||||
)
|
||||
|
||||
RADIANCE_CANNED_MODELS = (
|
||||
'{"object":"list","data":[{"id":"juggernaut","object":"model","created":1793484879,'
|
||||
'"owned_by":"radiance","max_model_len":131072,"capabilities":{"chat":true}}]}'
|
||||
)
|
||||
|
||||
RADIANCE_BASE = "http://radiance.test:8000"
|
||||
|
||||
|
||||
def _radiance_capture_sandbox(
|
||||
tmp_path: Path,
|
||||
*,
|
||||
recorded: dict[str, str] | None = None,
|
||||
fail_curl: bool = False,
|
||||
busy: bool = False,
|
||||
no_engine: bool = False,
|
||||
) -> tuple[Path, Path, Path, dict[str, str]]:
|
||||
"""A copy of the script, a canned radiance, and a PATH whose `curl` reaches only the canned.
|
||||
|
||||
Returns (script, fixtures dir, stub call log, env). The script is copied into
|
||||
`<tmp>/scripts/` because it derives `tests/fixtures/` from its own location, which is
|
||||
what keeps the repository's fixtures out of reach of a test.
|
||||
"""
|
||||
script = tmp_path / "scripts" / RADIANCE_CAPTURE.name
|
||||
script.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(RADIANCE_CAPTURE, script)
|
||||
fixtures = tmp_path / "tests" / "fixtures"
|
||||
fixtures.mkdir(parents=True, exist_ok=True)
|
||||
for name, body in (recorded or {}).items():
|
||||
(fixtures / name).write_text(body, encoding="utf-8")
|
||||
|
||||
stubs = tmp_path / "stubs"
|
||||
stubs.mkdir()
|
||||
stub = stubs / "curl"
|
||||
stub.write_text(RADIANCE_CURL_STUB, encoding="utf-8")
|
||||
stub.chmod(0o755)
|
||||
log = tmp_path / "curl.log"
|
||||
|
||||
env = {
|
||||
"PATH": f"{stubs}{os.pathsep}{os.environ['PATH']}",
|
||||
"STUB_LOG": str(log),
|
||||
"CANNED_METRICS": RADIANCE_CANNED_METRICS,
|
||||
"CANNED_STATS": RADIANCE_CANNED_STATS_BUSY if busy else RADIANCE_CANNED_STATS,
|
||||
"CANNED_MODELS": RADIANCE_CANNED_MODELS,
|
||||
}
|
||||
if not no_engine:
|
||||
env["RADIANCE_BASE_URL"] = RADIANCE_BASE
|
||||
if fail_curl:
|
||||
env["FAIL_CURL"] = "1"
|
||||
return script, fixtures, log, env
|
||||
|
||||
|
||||
def _radiance_run(tmp_path: Path, **kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||
script, _fixtures, _log, env = _radiance_capture_sandbox(tmp_path, **kwargs) # type: ignore[arg-type]
|
||||
return subprocess.run([BASH, str(script), "--with-llm"], capture_output=True, text=True,
|
||||
env=env, check=False)
|
||||
|
||||
|
||||
def _dialled_urls(log: Path) -> list[str]:
|
||||
urls: list[str] = []
|
||||
for line in log.read_text().splitlines():
|
||||
for token in line.split():
|
||||
if token.startswith("http"):
|
||||
urls.append(token)
|
||||
return urls
|
||||
|
||||
|
||||
def _assert_verbatim(path: Path, canned: str) -> None:
|
||||
"""The engine's reply inside a fixture, claimed verbatim.
|
||||
|
||||
The script's header is comment text; the engine's reply is the file's complete tail —
|
||||
so the reply is verbatim iff the file ends with it byte for byte and everything before
|
||||
it is comment lines (the metrics body's own `# HELP` / `# TYPE` lines are part of the
|
||||
reply and stay in the fixture).
|
||||
"""
|
||||
text = path.read_text()
|
||||
assert text.endswith(canned), f"{path.name} lost the tail of the engine's reply"
|
||||
prefix = text[: -len(canned)]
|
||||
non_comment = [line for line in prefix.splitlines() if line and not line.startswith("#")]
|
||||
assert not non_comment, f"{path.name} has non-comment text before the reply: {non_comment}"
|
||||
assert prefix, f"{path.name} lost its verbatim header"
|
||||
|
||||
|
||||
def test_the_radiance_capture_records_all_three_endpoints_verbatim(tmp_path: Path) -> None:
|
||||
"""The happy path, run: three fixtures, each with the script's verbatim header, the body
|
||||
byte-for-byte what the engine answered, and a dial to exactly the three read-only
|
||||
endpoints — nothing else, in the documented order."""
|
||||
result = _radiance_run(tmp_path)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
fixtures = tmp_path / "tests" / "fixtures"
|
||||
assert {p.name for p in fixtures.iterdir()} == {
|
||||
"metrics_radiance.txt", "stats_radiance.json", "models_radiance.json"}
|
||||
for name in ("metrics_radiance.txt", "stats_radiance.json", "models_radiance.json"):
|
||||
text = (fixtures / name).read_text()
|
||||
assert text.startswith("# Verbatim"), f"{name} lost its verbatim header"
|
||||
assert "./scripts/capture-fixtures.sh" in text, f"{name} names its recorder"
|
||||
_assert_verbatim(fixtures / "metrics_radiance.txt", RADIANCE_CANNED_METRICS)
|
||||
_assert_verbatim(fixtures / "stats_radiance.json", RADIANCE_CANNED_STATS)
|
||||
_assert_verbatim(fixtures / "models_radiance.json", RADIANCE_CANNED_MODELS)
|
||||
assert _dialled_urls(fixtures.parent.parent / "curl.log") == [
|
||||
f"{RADIANCE_BASE}/metrics", f"{RADIANCE_BASE}/stats", f"{RADIANCE_BASE}/v1/models"]
|
||||
# the capture at rest records `requests: []` — the script says so, so the operator
|
||||
# re-runs with a request in flight to record its row shape
|
||||
assert "requests[] was empty — re-run with a request in flight to record its shape" in result.stdout
|
||||
|
||||
|
||||
def test_the_radiance_capture_says_nothing_to_re_run_when_a_request_is_in_flight(tmp_path: Path) -> None:
|
||||
"""With a request in flight the hint is gone (it would send the operator to re-run a
|
||||
capture that already has the row shape), and the header still records what it saw."""
|
||||
result = _radiance_run(tmp_path, busy=True)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
assert "re-run with a request in flight" not in result.stdout
|
||||
stats = (tmp_path / "tests" / "fixtures" / "stats_radiance.json").read_text()
|
||||
assert 'requests":[{"id":"req-1","queue_pos":1}]' in stats
|
||||
assert "in_flight was 1 at capture" in stats
|
||||
|
||||
|
||||
def test_the_radiance_capture_without_an_engine_says_which_env_var_it_wants(tmp_path: Path) -> None:
|
||||
"""No engine env var is not a silent no-op: the gate names the var, like the others."""
|
||||
script, _fixtures, _log, env = _radiance_capture_sandbox(tmp_path, no_engine=True)
|
||||
result = subprocess.run([BASH, str(script), "--with-llm"], capture_output=True, text=True,
|
||||
env=env, check=False)
|
||||
assert result.returncode == 1
|
||||
assert "no GUFO_BASE_URL / VLLM_BASE_URL / RADIANCE_BASE_URL set" in result.stdout
|
||||
|
||||
|
||||
def test_a_dead_radiance_engine_never_replaces_a_recorded_capture(tmp_path: Path) -> None:
|
||||
"""The `slots_gufo.json` rule: a capture script must never turn three good fixtures into
|
||||
three empty files, because then the shape tests pass against nothing (or fail for the
|
||||
wrong reason) and the ground truth is gone."""
|
||||
good = {"metrics_radiance.txt": "# old\nradiance:kept 1\n",
|
||||
"stats_radiance.json": "# old\n{\"cards\":[]}\n",
|
||||
"models_radiance.json": "# old\n{\"object\":\"list\"}\n"}
|
||||
script, fixtures, _log, env = _radiance_capture_sandbox(tmp_path, recorded=good, fail_curl=True)
|
||||
result = subprocess.run([BASH, str(script), "--with-llm"], capture_output=True, text=True,
|
||||
env=env, check=False)
|
||||
assert result.returncode == 1
|
||||
for name, body in good.items():
|
||||
assert (fixtures / name).read_text() == body, f"{name} was replaced by a dead engine"
|
||||
|
||||
@@ -25,6 +25,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sqlite3
|
||||
from collections.abc import Iterator
|
||||
@@ -479,6 +480,86 @@ async def test_gufo_slots_answer_the_pinned_shape(settings: Settings) -> None:
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
async def test_radiance_answers_the_pinned_shape(settings: Settings) -> None:
|
||||
"""Phase 34: the radiance `/metrics` + `/stats` poll over the **running** engine.
|
||||
|
||||
The production path — `collectors.fetch_engine` with the LLM cadence's estimator,
|
||||
`with_stats` the way `llm_jobs` runs it — so what fails here is a rename between the
|
||||
engine and the map, not a typo in a fixture. Read-only: radiance exposes no write
|
||||
endpoint, so "read-only" is a list of GETs. Skips when `RADIANCE_BASE_URL` is unset
|
||||
(invariant 3's live twin: unset is not deployed here, not broken). The assertions
|
||||
are shape, never numbers: the `stats` block's whitelist, the ordered 7-position spec
|
||||
curve (when the build publishes it), one of the two specced session arms, and every
|
||||
cell the engine answered is a finite number or `None` — an absent gauge is `None`,
|
||||
never a faked `0` (rule 2's live twin). `unparsed` must be empty: a parse error on
|
||||
the live wire is drift, not a tolerance.
|
||||
"""
|
||||
if not settings.radiance_url:
|
||||
pytest.skip("RADIANCE_BASE_URL is unset — that engine is not deployed here")
|
||||
# The span the LLM jobs really get: phase 23 spans the LLM estimator by the poll
|
||||
# interval, not by the host's `RATE_WINDOW_SECONDS` (the other tests' idiom).
|
||||
rates = RateEstimator(span=settings.llm_interval)
|
||||
async with httpx.AsyncClient(timeout=settings.llm_timeout) as client:
|
||||
result = await collectors.fetch_engine(client, "radiance", settings.radiance_url,
|
||||
settings.radiance_api_key, rates,
|
||||
with_stats=True)
|
||||
payload = result.payload
|
||||
assert payload["engine"] == "radiance"
|
||||
assert payload["unparsed"] == 0, (
|
||||
f"a parse error on the live wire is drift, not a tolerance: {payload['unparsed']}")
|
||||
metrics = payload["metrics"]
|
||||
|
||||
# The /stats block, task 03's whitelist: present only when the sidecar answered, and
|
||||
# then only those keys (the engine internals stay out — decision 3).
|
||||
stats = payload.get("stats")
|
||||
if stats is not None:
|
||||
allowed = {"model", "version", "uptime_s", "startup_s", "total_requests", "cards"}
|
||||
extra = set(stats) - allowed
|
||||
assert not extra, f"the /stats subset carried keys the whitelist does not name: {sorted(extra)}"
|
||||
for card in stats.get("cards", []):
|
||||
assert set(card) <= {"index", "name", "vram_used", "vram_total", "vram_used_pct"}, card
|
||||
|
||||
# The spec-acceptance curve: the ordered seven, when the build publishes it.
|
||||
positions = metrics["spec_positions"]
|
||||
assert isinstance(positions, list)
|
||||
if positions:
|
||||
assert len(positions) == 7, (
|
||||
f"the spec curve has {len(positions)} positions — the pin is 0..6, "
|
||||
f"not {[p['position'] for p in positions]}")
|
||||
assert [p["position"] for p in positions] == list(range(7)), ("not 0..6 in order")
|
||||
for pos in positions:
|
||||
ratio = pos["ratio"]
|
||||
assert ratio is None or (isinstance(ratio, int | float)
|
||||
and not isinstance(ratio, bool)
|
||||
and math.isfinite(ratio) and 0.0 <= ratio <= 1.0), (
|
||||
f"position {pos['position']}'s ratio {ratio!r} is not a 0-1 gauge")
|
||||
|
||||
# Every cell the engine answered is a finite number or None — never a faked 0.
|
||||
for key, cell in metrics.items():
|
||||
if key == "spec_positions":
|
||||
continue # the ordered list, checked above
|
||||
if not isinstance(cell, dict) or "value" not in cell:
|
||||
continue
|
||||
value = cell["value"]
|
||||
assert value is None or (isinstance(value, int | float)
|
||||
and not isinstance(value, bool)
|
||||
and math.isfinite(value)), (
|
||||
f"{key}: {value!r} is neither a number nor None")
|
||||
|
||||
# The session arm is one of the two specced arms: /stats answered (the engine's own
|
||||
# summary is the source) or it did not (the /metrics gauges, and it says so).
|
||||
sessions = payload["sessions"]
|
||||
if stats is not None:
|
||||
assert sessions["source"] == "radiance /stats"
|
||||
assert "running" in sessions and "waiting" in sessions and "queue_cap" in sessions
|
||||
else:
|
||||
assert sessions["source"] is None
|
||||
assert sessions.get("queue_cap") is None, ("an absent /stats leaves the cap n/a, never 0")
|
||||
assert "/stats" in (sessions.get("detail") or ""), (
|
||||
f"the fallback arm must name the missing sidecar: {sessions.get('detail')!r}")
|
||||
assert sessions["active"] is None or isinstance(sessions["active"], int | float)
|
||||
|
||||
|
||||
def require_upsd(settings: Settings) -> None:
|
||||
"""Skip, not fail, when there is no real upsd to talk to."""
|
||||
if not settings.nut_enabled:
|
||||
|
||||
+646
-13
@@ -5,17 +5,25 @@ the deployed servers (tests/fixtures/metrics_*.txt), not to upstream documentati
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from dashboards.prom import Prom
|
||||
from dashboards.rates import MIN_WINDOW_SECONDS, RateEstimator
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources.llm import (
|
||||
RADIANCE_HISTOGRAMS,
|
||||
RADIANCE_OPTIONAL,
|
||||
RADIANCE_SERIES,
|
||||
engine_result,
|
||||
fleet_totals,
|
||||
parse_gufo_metrics,
|
||||
parse_radiance_metrics,
|
||||
parse_vllm_metrics,
|
||||
rate_key,
|
||||
series_names,
|
||||
@@ -62,19 +70,31 @@ def poll_engine(metrics_text: Any, engine: str, span: float, stamps: Sequence[fl
|
||||
# ---- gufo / llama.cpp ----------------------------------------------------- #
|
||||
|
||||
def test_gufo_counters_come_straight_from_the_capture(metrics_text) -> None:
|
||||
metrics = parse_gufo_metrics(metrics_text("gufo"))
|
||||
assert metrics["tokens_generated_total"]["value"] == 877_336.0
|
||||
assert metrics["tokens_prompt_total"]["value"] == 1_350_666.0
|
||||
assert metrics["tokens_prompt_cached_total"]["value"] == 124_529_105.0
|
||||
assert metrics["requests_processing"]["value"] == 0.0
|
||||
assert metrics["device_lost_total"]["value"] == 0.0
|
||||
assert metrics["kv_cache_usage_ratio"]["value"] == 0.0
|
||||
"""The cell's value is the number the exposition declares, verbatim: no
|
||||
conversion, no scaling, and the series' own name is the source. Compared against
|
||||
the raw `Prom` parse of the same capture — never a value frozen at capture time
|
||||
(the counters reset on every engine restart)."""
|
||||
text = metrics_text("gufo")
|
||||
prom = Prom(text)
|
||||
metrics = parse_gufo_metrics(text)
|
||||
for key, name in (
|
||||
("tokens_generated_total", "llamacpp:tokens_predicted_total"),
|
||||
("tokens_prompt_total", "llamacpp:prompt_tokens_total"),
|
||||
("tokens_prompt_cached_total", "llamacpp:prompt_tokens_cached_total"),
|
||||
("requests_processing", "llamacpp:requests_processing"),
|
||||
("device_lost_total", "gufo_device_lost_total"),
|
||||
("kv_cache_usage_ratio", "llamacpp:kv_cache_usage_ratio"),
|
||||
):
|
||||
assert metrics[key]["value"] == prom.scalar([name]), key
|
||||
assert metrics[key]["source"] == name, "the counter's own name is the source"
|
||||
|
||||
|
||||
def test_gufo_speculative_accept_rate_is_accepted_over_draft(metrics_text) -> None:
|
||||
metrics = parse_gufo_metrics(metrics_text("gufo"))
|
||||
accept = metrics["spec_accept_rate"]["value"]
|
||||
assert accept == pytest.approx(578_514 / 789_028)
|
||||
accepted = metrics["spec_accepted_tokens_total"]["value"]
|
||||
draft_tokens = metrics["spec_draft_tokens_total"]["value"]
|
||||
assert accept == pytest.approx(accepted / draft_tokens)
|
||||
assert str(metrics["spec_accept_rate"]["source"]).startswith("computed")
|
||||
|
||||
|
||||
@@ -82,7 +102,10 @@ def test_gufo_lifetime_tokens_include_cache_hits(metrics_text) -> None:
|
||||
"""gufo's prompt counter excludes cache hits; the lifetime total must not."""
|
||||
metrics = parse_gufo_metrics(metrics_text("gufo"))
|
||||
total = metrics["tokens_total"]["value"]
|
||||
assert total == 1_350_666 + 124_529_105 + 877_336
|
||||
prompt = metrics["tokens_prompt_total"]["value"]
|
||||
cached = metrics["tokens_prompt_cached_total"]["value"] or 0.0
|
||||
generated = metrics["tokens_generated_total"]["value"]
|
||||
assert total == pytest.approx(prompt + cached + generated)
|
||||
|
||||
|
||||
def test_gufo_throughput_prefers_measured_delta_over_the_native_gauge(metrics_text) -> None:
|
||||
@@ -99,8 +122,11 @@ def test_gufo_throughput_prefers_measured_delta_over_the_native_gauge(metrics_te
|
||||
assert decode["value"] == pytest.approx(40.0)
|
||||
assert decode["window"] == pytest.approx(10.0)
|
||||
assert "Δ" in decode["source"] and "tokens_predicted_total" in decode["source"]
|
||||
# the native gauge is still there, labelled, because it answers a different question
|
||||
assert metrics["decode_tokens_per_second_native"]["value"] == pytest.approx(42.621031668072597)
|
||||
# the native gauge is still there, labelled, because it answers a different question —
|
||||
# verbatim the exposition's own number (a volatile gauge: no frozen value)
|
||||
assert metrics["decode_tokens_per_second_native"]["value"] == \
|
||||
Prom(text).scalar(["llamacpp:predicted_tokens_seconds"])
|
||||
assert metrics["decode_tokens_per_second_native"]["source"] == "llamacpp:predicted_tokens_seconds"
|
||||
|
||||
|
||||
def test_gufo_the_window_the_llm_jobs_are_spanned_by_is_one_that_fills(metrics_text) -> None:
|
||||
@@ -323,10 +349,12 @@ def test_absent_gufo_metrics_are_none_with_no_source() -> None:
|
||||
def test_gufo_kv_cache_cell_keeps_provenance_and_gains_severity(metrics_text) -> None:
|
||||
metrics = parse_gufo_metrics(metrics_text("gufo"))
|
||||
cell = metrics["kv_cache_usage_ratio"]
|
||||
assert cell["value"] == 0.0
|
||||
assert cell["value"] == Prom(metrics_text("gufo")).scalar(["llamacpp:kv_cache_usage_ratio"]), \
|
||||
"verbatim the exposition's own ratio — the cell stays what it was"
|
||||
assert cell["source"] == "llamacpp:kv_cache_usage_ratio"
|
||||
assert cell["derived"] is False, "the verdict is additive; the cell still says what it was"
|
||||
assert cell["severity"] == "ok"
|
||||
assert cell["severity"] == classify("llm.kv_cache_usage_pct", cell["value"] * 100.0), \
|
||||
"the verdict is the table's for this value (percent scale)"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ratio,expected", [
|
||||
@@ -667,3 +695,608 @@ def test_vllm_a_measured_zero_beats_the_running_estimate(metrics_text) -> None:
|
||||
decode = metrics["decode_tokens_per_second"]
|
||||
assert decode["value"] == 0.0
|
||||
assert "running/tpot" not in str(decode["source"])
|
||||
|
||||
|
||||
# ---- radiance: the ground-truth fixtures (phase 34, task 01) ---------------- #
|
||||
#
|
||||
# Radiance is a vLLM derivative: its /metrics carries the `radiance:` native series plus
|
||||
# a `vllm:` compatibility family, and the capture records /stats and /v1/models in the
|
||||
# same pass. These tests pin the *shape and the name set* of those captures, never the
|
||||
# numbers (the nut_*.vars precedent): a metric rename or a new build's series fails here
|
||||
# before a dashboard does.
|
||||
|
||||
|
||||
def _load_radiance_json(name: str) -> Any:
|
||||
"""Load a radiance JSON fixture, skipping with the capture command like the others."""
|
||||
path = Path(__file__).resolve().parent / "fixtures" / name
|
||||
if not path.exists():
|
||||
pytest.skip(f"missing fixture {path.name} — run scripts/capture-fixtures.sh --with-llm")
|
||||
# the capture script writes its verbatim header as `#` comment lines; the document
|
||||
# is what follows them
|
||||
body = "\n".join(line for line in path.read_text().splitlines() if not line.startswith("#"))
|
||||
return json.loads(body)
|
||||
|
||||
|
||||
def _radiance_map_names() -> set[str]:
|
||||
return ({name for names, _ in RADIANCE_SERIES.values() for name in names}
|
||||
| {name for names in RADIANCE_HISTOGRAMS.values() for name in names})
|
||||
|
||||
|
||||
def test_radiance_fixture_name_set_is_pinned_by_the_map_and_optionals(metrics_text) -> None:
|
||||
"""The drift pin: every name in the capture's radiance:/vllm: families is either a
|
||||
payload cell (the map) or a declared non-cell (RADIANCE_OPTIONAL), and vice versa —
|
||||
so a rename, a dropped series, or a typo in a map name fails the suite before a
|
||||
dashboard does."""
|
||||
names = set()
|
||||
for line in metrics_text("radiance").splitlines():
|
||||
if line and not line.startswith("#"):
|
||||
names.add(line.split("{")[0].split()[0])
|
||||
contract = {name for name in names if name.startswith(("radiance:", "vllm:"))}
|
||||
assert contract, "the capture lost its series — re-run the capture with the engine up"
|
||||
map_names = _radiance_map_names()
|
||||
unexplained = contract - map_names - RADIANCE_OPTIONAL
|
||||
assert not unexplained, (
|
||||
f"names in the capture that are neither a payload cell nor declared optional: "
|
||||
f"{sorted(unexplained)}")
|
||||
absent_everywhere = map_names - contract - RADIANCE_OPTIONAL
|
||||
assert not absent_everywhere, (
|
||||
f"map names in neither the capture nor RADIANCE_OPTIONAL (a rename or a typo): "
|
||||
f"{sorted(absent_everywhere)}")
|
||||
stale = RADIANCE_OPTIONAL - contract - map_names
|
||||
assert not stale, f"RADIANCE_OPTIONAL entries that are neither real names: {sorted(stale)}"
|
||||
|
||||
|
||||
def test_radiance_fixture_carries_both_families(metrics_text) -> None:
|
||||
"""The vLLM-derived shape is part of the contract: the capture must hold both the
|
||||
native series and the compatibility family, or the name set has drifted."""
|
||||
text = metrics_text("radiance")
|
||||
native = sum(1 for line in text.splitlines() if line.startswith("radiance:"))
|
||||
compat = sum(1 for line in text.splitlines() if line.startswith("vllm:"))
|
||||
assert native > 0, "the capture lost its radiance: native series"
|
||||
assert compat > 0, "the capture lost its vllm: compatibility family"
|
||||
|
||||
|
||||
def test_radiance_per_position_series_has_the_seven_positions_the_phase_pins(
|
||||
metrics_text,
|
||||
) -> None:
|
||||
"""The per-position spec-acceptance series is the 7-bar curve: its label set is
|
||||
position="0".."6", each position a series of its own."""
|
||||
positions = set()
|
||||
for line in metrics_text("radiance").splitlines():
|
||||
if line.startswith("radiance:draft_position_acceptance_ratio{"):
|
||||
m = re.search(r'position="(\d+)"', line)
|
||||
assert m, f"a per-position line without a position label: {line}"
|
||||
positions.add(m.group(1))
|
||||
assert positions == {str(i) for i in range(7)}, (
|
||||
f"the phase pins seven draft positions; the capture has {sorted(positions)}")
|
||||
|
||||
|
||||
STATS_TOP_LEVEL_KEYS = (
|
||||
"cards", "checkpoints_written", "ckpt_evictions", "ckpt_refused", "decode_tps",
|
||||
"draft_accept", "draft_positions", "hit_cut_tokens", "hit_cuts", "in_flight",
|
||||
"kv_bytes_carved", "kv_bytes_total", "kv_bytes_used", "kv_tokens_cached",
|
||||
"kv_tokens_max", "kv_tokens_total", "kv_tokens_used", "kv_util",
|
||||
"linear_cached_tokens", "linear_hit_rate", "link", "model", "now_ms", "passes",
|
||||
"preempted", "preemptions", "prefill_chunks_capped", "prefill_tps",
|
||||
"prefix_evictions", "prefix_hit_rate", "queue_cap", "released", "requests",
|
||||
"rows_unspec", "running", "started_at", "startup_s", "step_ns", "step_ns_decode",
|
||||
"step_ns_decode_best", "step_ns_decode_recent", "steps", "steps_decode", "steps_held",
|
||||
"steps_mixed", "steps_prefill", "total_requests", "uptime_s", "version", "waiting",
|
||||
)
|
||||
|
||||
|
||||
def test_radiance_stats_fixture_shape() -> None:
|
||||
"""The /stats document: the top-level key set, the per-card VRAM rows, the requests
|
||||
list and the per-position spec-acceptance list — shape, not numbers. A capture at
|
||||
rest records `requests: []` (the header comment says so); a row shape is recorded
|
||||
by re-running the capture with a request in flight."""
|
||||
stats = _load_radiance_json("stats_radiance.json")
|
||||
assert isinstance(stats, dict)
|
||||
assert tuple(sorted(stats.keys())) == STATS_TOP_LEVEL_KEYS, (
|
||||
"the /stats top-level key set changed — the shape the page reads drifted; "
|
||||
"re-capture and re-decide the payload set")
|
||||
cards = stats["cards"]
|
||||
assert isinstance(cards, list) and cards, "cards[] is the per-card VRAM panel's source"
|
||||
for card in cards:
|
||||
assert isinstance(card, dict)
|
||||
assert isinstance(card["index"], int)
|
||||
assert isinstance(card["name"], str) and card["name"]
|
||||
assert isinstance(card["vram_used"], int)
|
||||
assert isinstance(card["vram_total"], int)
|
||||
assert 0 < card["vram_used"] <= card["vram_total"]
|
||||
assert isinstance(stats["requests"], list)
|
||||
assert isinstance(stats["queue_cap"], int)
|
||||
assert isinstance(stats["in_flight"], int)
|
||||
positions = stats["draft_positions"]
|
||||
assert isinstance(positions, list)
|
||||
assert len(positions) == 7, (
|
||||
"the phase pins seven draft positions; the capture has "
|
||||
f"{len(positions)} — the 7-bar curve needs re-deciding")
|
||||
for pos in positions:
|
||||
assert isinstance(pos, dict)
|
||||
assert isinstance(pos["reached"], int)
|
||||
assert isinstance(pos["accepted"], int)
|
||||
assert isinstance(pos["rate"], float)
|
||||
|
||||
|
||||
def test_radiance_models_fixture_shape() -> None:
|
||||
"""The /v1/models document: one model, and the capabilities / max_model_len it
|
||||
publishes — the only place the engine states what it can do."""
|
||||
models = _load_radiance_json("models_radiance.json")
|
||||
assert isinstance(models, dict)
|
||||
assert models["object"] == "list"
|
||||
data = models["data"]
|
||||
assert isinstance(data, list) and data
|
||||
model = data[0]
|
||||
assert model["owned_by"] == "radiance", "the engine's own word for its family"
|
||||
assert isinstance(model["id"], str) and model["id"]
|
||||
assert isinstance(model["max_model_len"], int)
|
||||
assert isinstance(model["capabilities"], dict)
|
||||
|
||||
|
||||
# ---- radiance: the field map (phase 34, task 03) ----------------------- #
|
||||
#
|
||||
# The map is built from the task-01 fixture (rule 7); the tests below pin the three
|
||||
# throughput arms (gauge wins / measured fills / n/a on a refused window), the severity
|
||||
# pass-through on the existing KV row, the /stats session arms, the whitelisted stats
|
||||
# block and the additive history series. As always: shape and name set, never the
|
||||
# numbers of a live engine.
|
||||
|
||||
|
||||
def _radiance_stats() -> dict:
|
||||
return _load_radiance_json("stats_radiance.json")
|
||||
|
||||
|
||||
# ---- the fixture parse -------------------------------------------------- #
|
||||
|
||||
def test_radiance_fixture_parse_every_mapped_key_resolves(metrics_text) -> None:
|
||||
"""Every mapped key resolves on the verbatim capture, with its pinned source string."""
|
||||
metrics = parse_radiance_metrics(metrics_text("radiance"))
|
||||
expected_source = {
|
||||
"decode_tokens_per_second_native": "radiance:decode_tokens_per_second",
|
||||
"prompt_tokens_per_second_native": "radiance:prefill_tokens_per_second",
|
||||
"num_requests_running": "vllm:num_requests_running",
|
||||
"num_requests_waiting": "vllm:num_requests_waiting",
|
||||
"gpu_cache_usage": "vllm:kv_cache_usage_perc",
|
||||
"kv_cache_max_tokens": "radiance:kv_cache_max_tokens",
|
||||
"kv_cache_max_bytes": "radiance:kv_cache_max_bytes",
|
||||
"queue_depth": "radiance:http_queue_depth",
|
||||
"admission_rejected_total": "radiance:http_admission_rejected_total",
|
||||
"num_preemptions_total": "vllm:num_preemptions_total",
|
||||
"prompt_tokens_total": "vllm:prompt_tokens_total",
|
||||
"generation_tokens_total": "vllm:generation_tokens_total",
|
||||
"gpu_prefix_cache_hit_rate": "vllm:gpu_prefix_cache_hit_rate",
|
||||
"spec_accept_rate": "radiance:draft_acceptance_ratio",
|
||||
"decode_step_seconds_recent": "radiance:decode_step_seconds_recent",
|
||||
"decode_step_seconds_best": "radiance:decode_step_seconds_best",
|
||||
}
|
||||
for key, source in expected_source.items():
|
||||
cell = metrics[key]
|
||||
assert set(cell) >= {"value", "source", "derived"}, f"{key} lost its cell shape"
|
||||
assert cell["source"] == source, f"{key} resolved to {cell['source']!r}, not {source!r}"
|
||||
assert cell["value"] is not None, f"{key} should resolve on the capture"
|
||||
# a map name this build does not carry: n/a, never 0
|
||||
assert metrics["num_requests_swapped"]["value"] is None
|
||||
assert metrics["num_requests_swapped"]["source"] is None
|
||||
|
||||
|
||||
def test_radiance_a_series_removed_from_the_capture_is_n_a_never_zero(metrics_text) -> None:
|
||||
"""The completion-criteria arm: a line removed from a fixture copy answers
|
||||
`{\"value\": null}` — 0 would claim \"the engine published zero\", which it did not."""
|
||||
by_key = {
|
||||
"radiance:http_queue_depth": "queue_depth",
|
||||
"radiance:kv_cache_max_bytes": "kv_cache_max_bytes",
|
||||
"radiance:decode_step_seconds_best": "decode_step_seconds_best",
|
||||
"vllm:num_preemptions_total": "num_preemptions_total",
|
||||
"vllm:gpu_prefix_cache_hit_rate": "gpu_prefix_cache_hit_rate",
|
||||
"radiance:draft_acceptance_ratio": "spec_accept_rate",
|
||||
}
|
||||
for name, key in by_key.items():
|
||||
text = "\n".join(line for line in metrics_text("radiance").splitlines()
|
||||
if not line.startswith(name))
|
||||
cell = parse_radiance_metrics(text)[key]
|
||||
assert cell["value"] is None, f"{key} answered a value for a removed series: {cell}"
|
||||
assert cell["source"] is None
|
||||
|
||||
|
||||
def test_radiance_spec_positions_are_the_ordered_seven(metrics_text) -> None:
|
||||
metrics = parse_radiance_metrics(metrics_text("radiance"))
|
||||
positions = metrics["spec_positions"]
|
||||
assert [p["position"] for p in positions] == list(range(7)), \
|
||||
"the phase pins seven draft positions, in the order the engine publishes them"
|
||||
for p in positions:
|
||||
assert 0.0 <= p["ratio"] <= 1.0, "a per-position ratio is a 0-1 gauge"
|
||||
# the ratios are a running-window aggregate: volatile between captures, so the
|
||||
# pin is the shape (seven, ordered, in-range) — the name set is pinned above
|
||||
prom = Prom(metrics_text("radiance"))
|
||||
by_position = prom.by_label(["radiance:draft_position_acceptance_ratio"], "position")
|
||||
for p in positions:
|
||||
assert p["ratio"] == by_position[str(p["position"])], \
|
||||
f"position {p['position']} is not the exposition's own number"
|
||||
|
||||
|
||||
def test_radiance_an_absent_position_stays_absent_not_zeroed(metrics_text) -> None:
|
||||
text = "\n".join(
|
||||
line for line in metrics_text("radiance").splitlines()
|
||||
if not ('radiance:draft_position_acceptance_ratio{' in line
|
||||
and 'position="6"' in line)
|
||||
)
|
||||
positions = parse_radiance_metrics(text)["spec_positions"]
|
||||
assert [p["position"] for p in positions] == [0, 1, 2, 3, 4, 5], \
|
||||
"position 6 is absent from this capture: the list holds what the engine published"
|
||||
assert all(p["ratio"] is not None and p["ratio"] > 0.0 for p in positions), \
|
||||
"an absent position must not be filled with a 0"
|
||||
|
||||
|
||||
def test_radiance_tokens_total_is_the_two_vllm_counters(metrics_text) -> None:
|
||||
metrics = parse_radiance_metrics(metrics_text("radiance"))
|
||||
prompt = metrics["prompt_tokens_total"]["value"]
|
||||
generated = metrics["generation_tokens_total"]["value"]
|
||||
assert metrics["tokens_total"]["value"] == pytest.approx(prompt + generated)
|
||||
assert metrics["tokens_total"]["source"] == "computed:prompt+generated"
|
||||
# both counters removed: n/a, never 0
|
||||
text = metrics_text("radiance")
|
||||
for name in ("vllm:prompt_tokens_total", "vllm:generation_tokens_total"):
|
||||
text = "\n".join(line for line in text.splitlines()
|
||||
if not line.startswith(name))
|
||||
assert parse_radiance_metrics(text)["tokens_total"]["value"] is None
|
||||
|
||||
|
||||
def test_radiance_prefix_hit_ratio_is_the_gauge_source_named_not_derived(metrics_text) -> None:
|
||||
"""This build publishes no prefix counters, so a hits/queries ratio would be an
|
||||
invented number — the gauge is the answer, under the shared UI key."""
|
||||
text = metrics_text("radiance")
|
||||
metrics = parse_radiance_metrics(text)
|
||||
gauge = Prom(text).scalar(["vllm:gpu_prefix_cache_hit_rate"])
|
||||
cell = metrics["prefix_cache_hit_ratio"]
|
||||
assert cell["value"] == pytest.approx(gauge)
|
||||
assert cell["source"] == "vllm:gpu_prefix_cache_hit_rate"
|
||||
assert cell["derived"] is False, "the gauge is not a computed ratio"
|
||||
# the gauge cell itself is the payload cell too (the map entry)
|
||||
assert metrics["gpu_prefix_cache_hit_rate"]["value"] == pytest.approx(gauge)
|
||||
|
||||
|
||||
def test_radiance_latency_histograms_are_the_vllm_shaped_family(metrics_text) -> None:
|
||||
metrics = parse_radiance_metrics(metrics_text("radiance"))
|
||||
ttft = metrics["time_to_first_token_seconds"]
|
||||
assert ttft and ttft["count"] > 0 and ttft["mean"] > 0
|
||||
assert metrics["time_to_first_token_seconds__p95"]["value"] >= ttft["mean"]
|
||||
assert metrics["time_per_output_token_seconds"]["mean"] > 0
|
||||
assert metrics["e2e_request_latency_seconds"]["count"] > 0
|
||||
# no request_queue_time on this build: the cell is absent, never back-filled
|
||||
assert "request_queue_time_seconds" not in metrics
|
||||
assert "request_queue_time_seconds__mean" not in metrics
|
||||
|
||||
|
||||
# ---- rule 2, the three throughput arms ----------------------------------- #
|
||||
|
||||
def test_radiance_throughput_arm_gauge_wins(metrics_text) -> None:
|
||||
"""The native gauge is the scheduler's recent-window aggregate, so it wins."""
|
||||
text = metrics_text("radiance")
|
||||
metrics = parse_radiance_metrics(text, RateEstimator(span=10), 1_000.0)
|
||||
decode = metrics["decode_tokens_per_second"]
|
||||
# the gauge's value is the exposition's own number (volatile — the pin is the
|
||||
# passthrough, not a frozen value), and it wins the cell
|
||||
assert decode["value"] == metrics["decode_tokens_per_second_native"]["value"]
|
||||
assert decode["value"] == Prom(text).scalar(["radiance:decode_tokens_per_second"], agg="avg")
|
||||
assert decode["source"] == "radiance:decode_tokens_per_second"
|
||||
assert decode["derived"] is False, "a scraped gauge is not a computed Δ"
|
||||
assert "window" not in decode, "a scraped gauge names no Δ window"
|
||||
prompt = metrics["prompt_tokens_per_second"]
|
||||
assert prompt["value"] == metrics["prompt_tokens_per_second_native"]["value"]
|
||||
assert prompt["source"] == "radiance:prefill_tokens_per_second"
|
||||
|
||||
|
||||
def test_radiance_throughput_arm_measured_delta_when_the_gauge_is_absent(metrics_text) -> None:
|
||||
"""A gauge this build does not publish falls back to the measured Δ, per label set."""
|
||||
text = metrics_text("radiance")
|
||||
text = "\n".join(line for line in text.splitlines()
|
||||
if not line.startswith("radiance:decode_tokens_per_second"))
|
||||
rates = RateEstimator(span=10)
|
||||
parse_radiance_metrics(text, rates, 1_000.0)
|
||||
text = bump_metric(text, "vllm:generation_tokens_total", 250.0)
|
||||
metrics = parse_radiance_metrics(text, rates, 1_010.0)
|
||||
decode = metrics["decode_tokens_per_second"]
|
||||
assert decode["value"] == pytest.approx(25.0)
|
||||
assert decode["derived"] is True
|
||||
assert decode["source"] == "computed:Δvllm:generation_tokens_total/Δt"
|
||||
assert decode["window"] == pytest.approx(10.0), \
|
||||
"the label must name the window the value used"
|
||||
# the arms are independent: the prompt gauge still wins its own cell
|
||||
prompt = metrics["prompt_tokens_per_second"]
|
||||
assert prompt["value"] == metrics["prompt_tokens_per_second_native"]["value"]
|
||||
assert prompt["derived"] is False
|
||||
|
||||
|
||||
def test_radiance_throughput_arm_a_refused_window_is_n_a_with_its_label(metrics_text) -> None:
|
||||
"""A refused window is still an answer: n/a beside the honest Δ<window> it refused."""
|
||||
text = metrics_text("radiance")
|
||||
text = "\n".join(line for line in text.splitlines()
|
||||
if not line.startswith("radiance:decode_tokens_per_second"))
|
||||
rates = RateEstimator(span=10)
|
||||
parse_radiance_metrics(text, rates, 1_000.0)
|
||||
text = bump_metric(text, "vllm:generation_tokens_total", 50.0)
|
||||
parse_radiance_metrics(text, rates, 1_010.0)
|
||||
text = bump_metric(text, "vllm:generation_tokens_total", 60.0)
|
||||
decode = parse_radiance_metrics(text, rates, 1_010.3)["decode_tokens_per_second"]
|
||||
assert decode["value"] is None, "0.3 s is the loop's catch-up gap, not a rate"
|
||||
assert decode["window"] == pytest.approx(0.3)
|
||||
assert decode["window"] < MIN_WINDOW_SECONDS
|
||||
assert decode["source"] == "computed:Δvllm:generation_tokens_total/Δt"
|
||||
|
||||
|
||||
def test_radiance_throughput_arm_both_absent_is_n_a_never_zero(metrics_text) -> None:
|
||||
text = metrics_text("radiance")
|
||||
for name in ("radiance:decode_tokens_per_second", "radiance:prefill_tokens_per_second",
|
||||
"vllm:generation_tokens_total", "vllm:prompt_tokens_total"):
|
||||
text = "\n".join(line for line in text.splitlines() if not line.startswith(name))
|
||||
metrics = parse_radiance_metrics(text) # no estimator: no measured arm either
|
||||
for key in ("decode_tokens_per_second", "prompt_tokens_per_second"):
|
||||
cell = metrics[key]
|
||||
assert cell["value"] is None, f"{key} printed a value with no source: {cell}"
|
||||
assert cell["source"] is None
|
||||
|
||||
|
||||
# ---- severity: the KV cell is rated, the spec cell is not --------------- #
|
||||
|
||||
def test_radiance_kv_cell_carries_the_severity_verdict(metrics_text) -> None:
|
||||
"""Radiance publishes the same 0-1 ratio the vLLM map feeds the table: the
|
||||
existing `llm.kv_cache_usage_pct` row rates it, no new threshold anywhere."""
|
||||
text = metrics_text("radiance")
|
||||
base = Prom(text).scalar(["vllm:kv_cache_usage_perc"], agg="max")
|
||||
assert base is not None
|
||||
cell = parse_radiance_metrics(text)["gpu_cache_usage"]
|
||||
assert cell["value"] == pytest.approx(base)
|
||||
assert cell["source"] == "vllm:kv_cache_usage_perc"
|
||||
assert cell["derived"] is False, "the verdict is additive; the cell still says what it was"
|
||||
assert cell["severity"] == classify("llm.kv_cache_usage_pct", base * 100.0), \
|
||||
"the table's verdict for this value (percent scale)"
|
||||
# push it above the table's fail bound (90 %): the verdict follows the value
|
||||
target = 0.99
|
||||
pressed = bump_metric(text, "vllm:kv_cache_usage_perc", target - base)
|
||||
cell = parse_radiance_metrics(pressed)["gpu_cache_usage"]
|
||||
assert cell["value"] == pytest.approx(target)
|
||||
assert cell["severity"] == "fail"
|
||||
|
||||
|
||||
def test_radiance_spec_accept_carries_no_severity(metrics_text) -> None:
|
||||
"""Decision 4 pinned both ways: the KV cell is rated, the spec cell is not — an
|
||||
efficiency number is not a verdict."""
|
||||
text = metrics_text("radiance")
|
||||
metrics = parse_radiance_metrics(text)
|
||||
assert metrics["spec_accept_rate"]["value"] == \
|
||||
Prom(text).scalar(["radiance:draft_acceptance_ratio"], agg="avg")
|
||||
assert "severity" not in metrics["spec_accept_rate"]
|
||||
assert metrics["gpu_cache_usage"]["severity"] is not None, \
|
||||
"the other direction: the KV cell is rated"
|
||||
|
||||
|
||||
# ---- /stats as the session source ---------------------------------------- #
|
||||
|
||||
def test_radiance_sessions_stats_arm_at_rest_counts_and_cap(metrics_text) -> None:
|
||||
"""The at-rest capture: requests[] empty → the counts-otherwise sentence, the cap
|
||||
from /stats, `active` from the /metrics gauge (one definition in every arm).
|
||||
The numbers come from the capture itself — volatile between re-captures."""
|
||||
stats = _radiance_stats()
|
||||
result = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
|
||||
RateEstimator(), stats=stats, now=1.0)
|
||||
sessions = result.payload["sessions"]
|
||||
active_expected = Prom(metrics_text("radiance")).scalar(["vllm:num_requests_running"])
|
||||
assert sessions["active"] == active_expected, "active stays the /metrics gauge in every arm"
|
||||
assert sessions["running"] == stats["running"]
|
||||
assert sessions["waiting"] == stats["waiting"]
|
||||
assert sessions["queue_cap"] == stats["queue_cap"]
|
||||
assert sessions["list"] == stats["requests"]
|
||||
assert sessions["source"] == "radiance /stats"
|
||||
if not stats["requests"]:
|
||||
assert sessions["detail"] == (
|
||||
f"radiance reports counts, not sessions: {stats['running']:g} running"
|
||||
f" of cap {stats['queue_cap']:g}. Per-request detail appears here while "
|
||||
f"requests are in flight.")
|
||||
else:
|
||||
assert sessions["detail"] is None, "rows in flight: no at-rest sentence"
|
||||
|
||||
|
||||
def test_radiance_sessions_stats_arm_non_empty_rows_verbatim(metrics_text) -> None:
|
||||
"""A request in flight: the rows are the engine's own shape, verbatim."""
|
||||
stats = _radiance_stats()
|
||||
rows = [{"id": 42, "model": "juggernaut", "state": "running",
|
||||
"prompt_tokens": 1_200, "completed_tokens": 250}]
|
||||
stats = {**stats, "requests": rows, "running": 1, "waiting": 2}
|
||||
result = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
|
||||
RateEstimator(), stats=stats, now=1.0)
|
||||
sessions = result.payload["sessions"]
|
||||
assert sessions["list"] == rows, "the rows pass through untouched"
|
||||
assert sessions["source"] == "radiance /stats"
|
||||
assert sessions["detail"] is None
|
||||
assert sessions["running"] == 1
|
||||
assert sessions["waiting"] == 2
|
||||
assert sessions["queue_cap"] == stats["queue_cap"] # the capture's cap, untouched
|
||||
|
||||
|
||||
def test_radiance_sessions_missing_stats_fall_back_to_the_gauges(metrics_text) -> None:
|
||||
"""The sidecar failed: the poll is still healthy, the counts come from the
|
||||
/metrics gauges (a different provenance, said so), and the cap is null — never 0."""
|
||||
result = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
|
||||
RateEstimator(), now=1.0) # stats=None: /stats did not answer
|
||||
sessions = result.payload["sessions"]
|
||||
assert sessions["running"] == 0.0, "the /metrics gauge's value"
|
||||
assert sessions["waiting"] == 0.0
|
||||
assert sessions["queue_cap"] is None, "null, not 0 — rule 2"
|
||||
assert sessions["list"] == []
|
||||
assert "did not answer" in sessions["detail"]
|
||||
assert "gauges" in sessions["detail"]
|
||||
assert "stats" not in result.payload, "no stats block: the sidecar never answered"
|
||||
# the metrics passthroughs degrade with it
|
||||
assert result.payload["metrics"]["queue_cap"]["value"] is None
|
||||
assert result.payload["metrics"]["vram_used_pct"]["value"] is None
|
||||
|
||||
|
||||
# ---- the whitelisted stats block ------------------------------------------ #
|
||||
|
||||
def test_radiance_payload_stats_block_is_whitelisted(metrics_text) -> None:
|
||||
stats = _radiance_stats()
|
||||
result = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
|
||||
RateEstimator(), stats=stats, now=1.0)
|
||||
block = result.payload["stats"]
|
||||
assert set(block) == {"model", "version", "uptime_s", "startup_s",
|
||||
"total_requests", "cards"}, \
|
||||
"the engine internals (link / passes / steps* / …) stay out of the payload"
|
||||
assert block["model"] == "juggernaut"
|
||||
assert block["version"] == "1.15.1"
|
||||
assert isinstance(block["uptime_s"], int) and block["uptime_s"] >= 0, \
|
||||
"uptime is the engine's own seconds — volatile between captures"
|
||||
card = block["cards"][0]
|
||||
src = stats["cards"][0]
|
||||
assert set(card) == {"index", "name", "vram_used", "vram_total", "vram_used_pct"}
|
||||
assert card["vram_used"] == src["vram_used"], \
|
||||
"the raw bytes travel as the engine published them"
|
||||
assert card["vram_total"] == src["vram_total"]
|
||||
pct = card["vram_used_pct"]
|
||||
assert pct["derived"] is True, "the one computed number in the block"
|
||||
assert pct["source"] == "computed:vram_used/vram_total*100"
|
||||
assert pct["value"] == pytest.approx(src["vram_used"] / src["vram_total"] * 100.0)
|
||||
|
||||
|
||||
def test_radiance_a_renamed_stats_key_degrades_to_absent(metrics_text) -> None:
|
||||
"""A radiance upgrade that renames a /stats key degrades it to absent, not a
|
||||
crash — the poll still returns its /metrics payload."""
|
||||
stats = _radiance_stats()
|
||||
stats["engine_version"] = stats.pop("version")
|
||||
result = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
|
||||
RateEstimator(), stats=stats, now=1.0)
|
||||
block = result.payload["stats"]
|
||||
assert "version" not in block, "renamed away: absent, never a guessed value"
|
||||
assert "engine_version" not in block, "and not carried under the new name either"
|
||||
assert block["model"] == "juggernaut", "the rest of the block is intact"
|
||||
assert result.payload["metrics"]["spec_accept_rate"]["value"] is not None, \
|
||||
"the poll still returns"
|
||||
# and a renamed card key degrades the computed pct, not the whole block
|
||||
stats2 = _radiance_stats()
|
||||
card = {**stats2["cards"][0], "vram_in_use": stats2["cards"][0]["vram_used"]}
|
||||
del card["vram_used"]
|
||||
stats2["cards"] = [card]
|
||||
result2 = engine_result("radiance", "http://r:8000", metrics_text("radiance"),
|
||||
RateEstimator(), stats=stats2, now=1.0)
|
||||
row = result2.payload["stats"]["cards"][0]
|
||||
assert "vram_used" not in row and "vram_in_use" not in row
|
||||
assert row["vram_used_pct"]["value"] is None, "the computed number degrades to n/a"
|
||||
|
||||
|
||||
# ---- the /stats passthroughs in the metrics dict ------------------------- #
|
||||
|
||||
def test_radiance_queue_cap_is_the_stats_passthrough(metrics_text) -> None:
|
||||
stats = _radiance_stats()
|
||||
metrics = parse_radiance_metrics(metrics_text("radiance"), stats=stats)
|
||||
cell = metrics["queue_cap"]
|
||||
assert cell["value"] == stats["queue_cap"], "the /stats number, verbatim"
|
||||
assert cell["source"] == "radiance /stats"
|
||||
assert cell["derived"] is False
|
||||
absent = parse_radiance_metrics(metrics_text("radiance"))
|
||||
assert absent["queue_cap"]["value"] is None, "null, not 0 — rule 2"
|
||||
assert absent["queue_cap"]["source"] is None
|
||||
|
||||
|
||||
# ---- the history samples ------------------------------------------------- #
|
||||
|
||||
def test_radiance_history_samples_include_the_new_series(metrics_text) -> None:
|
||||
text = metrics_text("radiance")
|
||||
stats = _radiance_stats()
|
||||
result = engine_result("radiance", "http://r:8000", text,
|
||||
RateEstimator(), stats=stats, now=1.0)
|
||||
names = {s.metric for s in result.samples}
|
||||
for expected in ("llm.decode_tokens_per_second", "llm.prompt_tokens_per_second",
|
||||
"llm.kv_cache_usage", "llm.requests_running", "llm.requests_waiting",
|
||||
"llm.spec_accept_rate",
|
||||
"llm.prefix_cache_hit_ratio", "llm.tpot_seconds",
|
||||
"llm.ttft_seconds", "llm.ttft_p95_seconds",
|
||||
"llm.queue_depth", "llm.queue_cap",
|
||||
"llm.decode_step_seconds_recent", "llm.vram_used_pct"):
|
||||
assert expected in names, f"{expected} missing from the radiance samples"
|
||||
# `tokens_total` is a payload key, not a history series (it is not an existing
|
||||
# HISTORY_METRICS key, and the task's additions are the four above): the fleet
|
||||
# block reads it from the payload, the chart never drew it
|
||||
assert "llm.tokens_total" not in names
|
||||
# the stored values are the capture's own numbers (volatile — pinned as passthrough,
|
||||
# never as frozen values)
|
||||
prom = Prom(text)
|
||||
stored = {s.metric: s.value for s in result.samples}
|
||||
assert stored["llm.decode_tokens_per_second"] == \
|
||||
prom.scalar(["radiance:decode_tokens_per_second"], agg="avg")
|
||||
assert stored["llm.prompt_tokens_per_second"] == \
|
||||
prom.scalar(["radiance:prefill_tokens_per_second"], agg="avg")
|
||||
assert stored["llm.queue_cap"] == stats["queue_cap"]
|
||||
assert stored["llm.queue_depth"] == prom.scalar(["radiance:http_queue_depth"], agg="max")
|
||||
assert stored["llm.decode_step_seconds_recent"] == \
|
||||
prom.scalar(["radiance:decode_step_seconds_recent"], agg="max")
|
||||
assert stored["llm.kv_cache_usage"] == \
|
||||
prom.scalar(["vllm:kv_cache_usage_perc"], agg="max")
|
||||
# the per-position ratios are snapshot-only (decision 2): no history series
|
||||
assert not any("position" in name for name in names)
|
||||
# no request_queue_time on this build: queue_seconds is n/a by absence
|
||||
assert "llm.queue_seconds" not in names
|
||||
|
||||
|
||||
def test_radiance_history_does_not_touch_gufo_or_vllm_samples(metrics_text) -> None:
|
||||
"""The existing keys and their candidate order are untouched: a gufo or vLLM
|
||||
payload never grows a radiance series."""
|
||||
for engine in ("gufo", "vllm"):
|
||||
result = engine_result(engine, "http://x:1", metrics_text(engine),
|
||||
RateEstimator(), now=1.0)
|
||||
names = {s.metric for s in result.samples}
|
||||
for foreign in ("llm.queue_depth", "llm.queue_cap",
|
||||
"llm.decode_step_seconds_recent", "llm.vram_used_pct"):
|
||||
assert foreign not in names, f"{engine} grew a radiance series: {foreign}"
|
||||
|
||||
|
||||
def test_series_names_radiance_is_its_own_map() -> None:
|
||||
"""`series_names` is the parse catalog — the map's keys, per engine."""
|
||||
names = series_names("radiance")
|
||||
assert "spec_accept_rate" in names
|
||||
assert "gpu_cache_usage" in names
|
||||
assert "queue_depth" in names
|
||||
assert "tokens_generated_total" not in names, "that is the gufo key"
|
||||
assert "spec_accept_rate" not in series_names("vllm")
|
||||
|
||||
|
||||
# ---- the fleet totals ---------------------------------------------------- #
|
||||
|
||||
def test_fleet_totals_with_a_gufo_and_radiance_payload(metrics_text) -> None:
|
||||
"""The code needed no change: the aliases and the KV max already cover a radiance
|
||||
payload; the seams sentence says the known asymmetries in its own words.
|
||||
The numbers are re-derived from the two captures (volatile between re-captures).
|
||||
"""
|
||||
gufo_text = metrics_text("gufo")
|
||||
rad_text = metrics_text("radiance")
|
||||
gufo = engine_result("gufo", "http://g:1", gufo_text, RateEstimator(), now=1.0)
|
||||
rad = engine_result("radiance", "http://r:8000", rad_text,
|
||||
RateEstimator(), stats=_radiance_stats(), now=1.0)
|
||||
totals = fleet_totals([gufo.payload, rad.payload])
|
||||
assert totals["engines_total"] == 2
|
||||
assert totals["engines_live"] == 2
|
||||
# running/waiting: the sum of each capture's own counters (gufo's aliases + the gauges)
|
||||
gp = Prom(gufo_text)
|
||||
rp = Prom(rad_text)
|
||||
assert totals["requests_active"] == pytest.approx(
|
||||
(gp.scalar(["llamacpp:requests_processing"]) or 0.0)
|
||||
+ (rp.scalar(["vllm:num_requests_running"]) or 0.0))
|
||||
assert totals["requests_waiting"] == pytest.approx(
|
||||
(gp.scalar(["llamacpp:requests_deferred"]) or 0.0)
|
||||
+ (rp.scalar(["vllm:num_requests_waiting"]) or 0.0))
|
||||
gufo_kv = gp.scalar(["llamacpp:kv_cache_usage_ratio"], agg="max") or 0.0
|
||||
rad_kv = rp.scalar(["vllm:kv_cache_usage_perc"], agg="max") or 0.0
|
||||
assert totals["kv_cache_usage_max"] == pytest.approx(max(gufo_kv, rad_kv)), \
|
||||
"the true max: the lower engine's value must not hide the other's pressure"
|
||||
assert totals["tokens_total"] == pytest.approx(
|
||||
gufo.payload["metrics"]["tokens_total"]["value"]
|
||||
+ rad.payload["metrics"]["tokens_total"]["value"])
|
||||
# a fresh estimator: gufo's decode is n/a on its first poll, radiance's gauge wins
|
||||
assert totals["decode_tokens_per_second"] == pytest.approx(
|
||||
rad.payload["metrics"]["decode_tokens_per_second"]["value"])
|
||||
assert totals["prompt_tokens_per_second"] == pytest.approx(
|
||||
rad.payload["metrics"]["prompt_tokens_per_second"]["value"])
|
||||
seams = totals["seams"]
|
||||
assert "cache-hit" in seams, "gufo's prompt-counter asymmetry stays named"
|
||||
assert "provenance" in seams, "the seams name the provenance labelling"
|
||||
assert "vllm:prompt_tokens_total" in seams, \
|
||||
"and the thing the dashboard deliberately does not claim"
|
||||
|
||||
+6
-2
@@ -14,8 +14,12 @@ from dashboards.prom import Prom
|
||||
|
||||
def test_parses_counters_and_gauges_from_a_real_capture(metrics_text) -> None:
|
||||
prom = Prom(metrics_text("gufo"))
|
||||
assert prom.scalar(["llamacpp:tokens_predicted_total"]) == 877_336.0
|
||||
assert prom.scalar(["llamacpp:kv_cache_usage_ratio"]) == 0.0
|
||||
# the capture's counters and gauges are live numbers (volatile between
|
||||
# re-captures): the pin is that they parse at all, in a sane range
|
||||
generated = prom.scalar(["llamacpp:tokens_predicted_total"])
|
||||
assert generated is not None and generated >= 0
|
||||
usage = prom.scalar(["llamacpp:kv_cache_usage_ratio"])
|
||||
assert usage is not None and 0.0 <= usage <= 1.0
|
||||
assert prom.parse_errors == 0
|
||||
|
||||
|
||||
|
||||
@@ -626,14 +626,14 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "pytest-asyncio"
|
||||
version = "1.4.1"
|
||||
version = "1.4.1.post1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "pytest" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f8/9a/887535e477847f5159377ab35303923644840b13c0ca03c0a467f84d4680/pytest_asyncio-1.4.1.tar.gz", hash = "sha256:e8266eb094e5ee9ffdb2068c9836f33e209ea77a970eb846933d416b5602669b", size = 145237, upload-time = "2026-10-25T21:05:37.135Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/73/7f/c60a4338420b33453d4f814ac0c930abd399f162b9bdd21af4b9e363037f/pytest_asyncio-1.4.1.post1.tar.gz", hash = "sha256:f582d9ea4a1c082195abe8bd29a1fe106aeb1cd435e3dad2d3bc2749935c36a2", size = 145360, upload-time = "2026-11-01T18:26:03.732Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/46/d7/cd9e2708afdbf059658adaf08bdb0792ca52489871051d6ce7d498b51277/pytest_asyncio-1.4.1-py3-none-any.whl", hash = "sha256:507afa78ec77c6661164ef1a2368bc7820383fd1a14b2a7f813de8b5a92949cc", size = 17061, upload-time = "2026-10-25T21:05:35.473Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e3/de/adbc7578959cfab07e5e7c05cf6d54224a51e7667ea2c07458b6bc571e75/pytest_asyncio-1.4.1.post1-py3-none-any.whl", hash = "sha256:0755e488cc0a2b9bd2eab7292cc6390d4cbb8390f1020745647bae5a708e0165", size = 17140, upload-time = "2026-11-01T18:26:02.168Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
Reference in New Issue
Block a user