phase: 01_severity_language
**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail).
**Verification (all green)**
- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean
- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty
- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓
- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png`
**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them.
**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05.
**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`).
This commit is contained in:
@@ -0,0 +1,13 @@
|
||||
**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail).
|
||||
|
||||
**Verification (all green)**
|
||||
- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean
|
||||
- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty
|
||||
- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓
|
||||
- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png`
|
||||
|
||||
**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them.
|
||||
|
||||
**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05.
|
||||
|
||||
**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`).
|
||||
@@ -0,0 +1 @@
|
||||
ok
|
||||
@@ -0,0 +1,217 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 19%]
|
||||
.....................................................................sss [ 39%]
|
||||
sss..................................................................... [ 59%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 99%]
|
||||
../usr/lib64/python3.14/collections/__init__.py:453: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59ca0487c0>
|
||||
@classmethod
|
||||
ResourceWarning: Enable tracemalloc to get the object allocation traceback
|
||||
/usr/lib64/python3.14/collections/__init__.py:455: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c9817a60>
|
||||
result = tuple_new(cls, iterable)
|
||||
ResourceWarning: Enable tracemalloc to get the object allocation traceback
|
||||
[100%]/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/_pytest/unraisableexception.py:33: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c9817010>
|
||||
gc.collect()
|
||||
ResourceWarning: Enable tracemalloc to get the object allocation traceback
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/_pytest/unraisableexception.py:33: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59ca19db70>
|
||||
gc.collect()
|
||||
ResourceWarning: Enable tracemalloc to get the object allocation traceback
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/_pytest/unraisableexception.py:33: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59ca19cf40>
|
||||
gc.collect()
|
||||
ResourceWarning: Enable tracemalloc to get the object allocation traceback
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/_pytest/unraisableexception.py:33: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c9797f10>
|
||||
gc.collect()
|
||||
ResourceWarning: Enable tracemalloc to get the object allocation traceback
|
||||
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
tests/test_api.py::test_unconfigured_source_is_404_configured_false[/api/ups/snapshot]
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/telemetry/_runtime.py:213: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cbae7f10>
|
||||
async def wrapped_receive() -> Message:
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_api.py::test_unconfigured_source_is_404_configured_false[/api/ups/snapshot]
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/telemetry/_runtime.py:213: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cbaf89a0>
|
||||
async def wrapped_receive() -> Message:
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_api.py::test_unconfigured_source_is_404_configured_false[/api/ups/snapshot]
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/telemetry/_runtime.py:213: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cbc43a60>
|
||||
async def wrapped_receive() -> Message:
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_api.py::test_summary_still_answers_with_no_sources
|
||||
/usr/lib64/python3.14/urllib/request.py:1888: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59ca19e5c0>
|
||||
if len(name) > 5 and name[-6] == "_" and name[-5:].lower() == "proxy":
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_api.py::test_agent_tokens_never_reach_the_browser
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/utils.py:43: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59ca19e2f0>
|
||||
def get_path_param_names(path: str) -> set[str]:
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cb16a2f0>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cb1a7b50>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cb1a7d30>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cb0671f0>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cb0d3c40>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59caf37880>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cae1be20>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cae1bf10>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cae9bc40>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59caefe200>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cad53b50>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cadef2e0>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cac6bb50>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cac6b970>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59cab4b790>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c8d97790>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c8d976a0>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c848b4c0>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c832aa70>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c8397970>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c82535b0>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c82c62f0>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c81fb010>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c809bc40>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
tests/test_main.py::test_main_serves_the_assembled_app
|
||||
<string>:0: ResourceWarning: unclosed database in <sqlite3.Connection object at 0x7f59c8163790>
|
||||
Enable tracemalloc to get traceback where the object was allocated.
|
||||
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 28
|
||||
src/dashboards/api.py 190 28 85% 59-60, 62, 280-286, 303-311, 318-322, 406, 411-412, 418-429
|
||||
src/dashboards/app.py 109 7 94% 50-51, 56, 58, 60, 97, 131
|
||||
src/dashboards/collectors.py 140 9 94% 78-79, 92-93, 97-98, 115, 232, 236
|
||||
src/dashboards/config.py 110 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 139 1 99% 226
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/host.py 199 2 99% 122, 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 226
|
||||
src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617
|
||||
src/dashboards/sources/nut_sim.py 120 21 82% 129, 155-173, 214
|
||||
src/dashboards/store.py 178 3 98% 204, 243-244
|
||||
------------------------------------------------------------------
|
||||
TOTAL 1924 101 95%
|
||||
Required test coverage of 94% reached. Total coverage: 94.75%
|
||||
356 passed, 6 skipped, 31 warnings in 39.85s
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
@@ -0,0 +1,17 @@
|
||||
**Implemented**
|
||||
- `src/dashboards/severity.py` (new leaf module): `THRESHOLDS` (the 10 ratified rows), `INVERTED` (`ups.charge_pct`, `ups.battery_runtime`), `classify()` → `ok|warn|fail|None`, `attach()` for bare numbers, `with_severity()` for `{value, source, derived}` cells; docstring states "unknown is not ok" + "a threshold exists nowhere else" and that `_pct` metrics are 0-100.
|
||||
- `tests/test_severity.py` (new, 64 tests): every metric walked across both boundaries, warn/danger edges inclusive, both directions for inverted metrics, table-sanity test (bound ordering per direction, `INVERTED ⊆ THRESHOLDS`), pinned ratified table, unknown metric / `None` / NaN / ±inf → `None`, `@stream` inheritance (`disk.used_pct@/srv`, `gpu.busy_pct@0`), attach/with_severity shapes incl. no caller-mutation and JSON-safety.
|
||||
- No page, API, CSS, or payload code touched — severity is not consumed yet. No live infra touched.
|
||||
|
||||
**Verification**
|
||||
- `uv run pytest tests/test_severity.py -v` → 64 passed
|
||||
- `uv run pytest` → **322 passed, 6 skipped** (baseline 258+6, no regressions); `uv run pytest --cov` → `severity.py` **100%**, total **94%**
|
||||
- `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors
|
||||
- `uv run python -c "from dashboards.severity import classify as c; print(c('cpu.usage_pct',95), c('cpu.usage_pct',60), c('cpu.usage_pct',None), c('ups.charge_pct',15))"` → `fail ok None fail`
|
||||
- Deps unchanged → `uv pip list --outdated` not applicable.
|
||||
|
||||
**Docs**: `AGENTS.md` — one `severity.py` line added to the module-responsibility list. No README impact (no new setup/command/API surface yet).
|
||||
|
||||
**Decisions**: reaching a threshold trips it in both directions (`cpu.usage_pct` 75 → warn, 90 → fail; `ups.charge_pct` 50 → warn, 20 → fail); `metric@stream` classifies with its base metric's row; `attach()` normalises non-finite values to `None` so payloads stay valid JSON, `with_severity()` is strictly additive (a missing cell gains only `severity`). No screenshot taken — this task changes no UI; the phase's render-unchanged screenshot belongs with task 03.
|
||||
|
||||
**Next pending task**: `.agents/phases/todo/01_severity_language/02_severity_in_payloads.md`
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
........................................................................ [ 21%]
|
||||
.......................................................ssssss........... [ 43%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 87%]
|
||||
........................................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
../../../.local/lib/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/.local/lib/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
All checks passed!
|
||||
repo hygiene: TRACKED files look like log drops / key material (untrack with 'git rm --cached <file>' and add a .gitignore rule):
|
||||
.env.dev.example
|
||||
.env.example
|
||||
validation FAILED (see output above)
|
||||
@@ -0,0 +1,17 @@
|
||||
**Fixed (rejection cause)**
|
||||
- `validate.sh` gate #2 flagged the repo's tracked env templates `.env.example` / `.env.dev.example` — they are tracked **by design** (`.gitignore` ignores `.env.*` and un-ignores exactly those two so a clone has documented setup; `cp .env.example .env` is in README/AGENTS.md). Untracking them would break the documented workflow, so I adapted the gate instead (its header says to adapt it per project):
|
||||
- narrowly allow-list `(^|/)\.env[^/]*\.(example|template|sample)$` — real `.env`, `.env.dev`, `*.pem/key/crt`, `*.log`, `*.jsonl` still fail (verified against a synthetic file list).
|
||||
- added gate **2b** so the allow-list can't smuggle a secret: every tracked `*.example` template's `*_PASSWORD|*_TOKEN|*_SECRET|*_KEY` assignment must be empty or a placeholder word. Passes on both templates (`replace-me-on-the-host`, `SECRET`, empty), fails on a planted real-looking value.
|
||||
- No test/lint/code defects found in this or prior work; `severity.py` + tests unchanged from the accepted implementation.
|
||||
|
||||
**Verification**
|
||||
- `./.agents/validate.sh` → **rc=0** (pytest 322 passed / 6 skipped; `ruff check .` All checks passed; hygiene gates silent)
|
||||
- `uv run pytest` → **322 passed, 6 skipped**; `uv run pytest --cov` → `severity.py` **100%**, total **94%**
|
||||
- `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors
|
||||
- `classify` criteria one-liner → `fail ok None fail`
|
||||
|
||||
**Docs**: `AGENTS.md` — `severity.py` line in the module-responsibility list (from the prior pass, still accurate). Gate adaptation is documented inline in `.agents/validate.sh`; no README change needed (setup steps unchanged).
|
||||
|
||||
**Notable decisions**: kept the templates tracked rather than untracking, because the hygiene gate's intent (no key material / log drops) is preserved and strengthened by gate 2b; no `git rm --cached` used (not sanctioned by this task file).
|
||||
|
||||
**Next pending task**: `.agents/phases/todo/01_severity_language/02_severity_in_payloads.md`
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
........................................................................ [ 21%]
|
||||
.......................................................ssssss........... [ 43%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 87%]
|
||||
........................................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
../../../.local/lib/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/.local/lib/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
All checks passed!
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
**Task 02 — Severity in the payloads: done**
|
||||
|
||||
- `sources/host.py`: added `severity` sub-dicts — `cpu.severity{usage_pct,temp_c}`, `memory.severity{used_pct}`, `disks.severity{worst_used_pct}`, `gpus[].severity{busy_pct,mem_used_pct}`; all existing keys untouched.
|
||||
- `sources/nut.py`: each UPS card now has `severity{charge_pct,load_pct,battery_runtime}`; `health`/`checks[].status`/`numbers` unchanged.
|
||||
- `sources/llm.py`: `attach_cell_severity()` puts `severity` inside the KV cells (`kv_cache_usage_ratio`, `gpu_cache_usage`) via `severity.with_severity`; `value/source/derived` intact.
|
||||
- `api.py` `/api/summary`: `severity` per host row, per UPS row, and on the fleet aggregate (`/api/health`, `/api/config` untouched).
|
||||
- `severity.with_severity` gained an optional `scale` (ratio→percent, **verdict only**) so 0.9 KV pressure is `fail`, not a permanent green; task-01 tests unchanged.
|
||||
- Regression caught and fixed by the new tests: a local-name collision in `parse_host_snapshot` had overwritten `memory.used_pct` with the GPU pool %; test asserts `memory.used_pct == used/total`.
|
||||
- Tests: `uv run pytest --cov` → **349 passed, 6 skipped, TOTAL 95%** (severity.py 100%, api.py/host.py/nut.py/llm.py gaps all pre-existing route bodies); `ruff check src tests agent` clean; `pyright` 0 errors; `uv pip list --outdated` empty.
|
||||
- Live: `uv run pytest -m live --live` → **6 passed**; dev server restarted, curl confirms severity on `/api/system/snapshot` (gateway `temp_c: null`), `/api/ups/snapshot` (apc + cyberpower ok; `battery_temperature` null with health check still `unknown`), `/api/summary`, `/api/llm/snapshot`.
|
||||
- Docs: AGENTS.md severity bullet now states the additive payload contract; test count 258→349; README architecture gained a `severity.py` bullet.
|
||||
- Note: `deploy/nut-sudoers` was modified in the working tree by something outside this task (grant line, mtime mid-session) — I left it untouched for the harness to judge.
|
||||
- Next pending task: `01_severity_language/03_hero_number_style.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
........................................................................ [ 20%]
|
||||
..............................................................ssssss.... [ 40%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 81%]
|
||||
................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
../../../.local/lib/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/.local/lib/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
All checks passed!
|
||||
@@ -0,0 +1,10 @@
|
||||
**Task 03 — Hero-number style and the JS side of severity: complete**
|
||||
|
||||
- `dash.css`: new `hero numbers` block (`.hero`, `.hero .label/.value/.unit/.sub`, `.hero.ok|.warn|.fail .value`, `.hero-bar` + `.hero-bar.ok .bar > i`) built only from existing tokens; 2.4rem tabular-numeral value, faint `n/a` placed after the verdict rules so an unmeasured number is never coloured.
|
||||
- `app.js`: `Dash.hero(el,{label,value,unit,digits,severity,sub,bar})` renders/patches a hero in place (`n/a` + **no** severity class when the value is null/non-finite, colour class dropped on the transition); `Dash.severityFrom(cell|string|null)` → `ok|warn|fail|''`; `severityFor` kept, documented as the fallback with "do not add a call site"; both new helpers exported. Pages untouched (phases 02–05 convert them).
|
||||
- Tests (7 new, `tests/test_api.py`): served CSS has every hero rule + the shared verdict tokens; no `.sev-*`/synonym classes in any asset or page (comments stripped first); `hero()` body must call `severityFrom` and must not contain `severityFor`; no stylesheet rule uses an undeclared `var(--token)`; `severityFor` still exported for the four pages.
|
||||
- Live browser verification (real Chromium against `:8090`, live fleet/engines/upsd): `severityFrom` cell/bare/null/junk → `fail`/`warn`/`''`/`''`; hero classes `hero ok|hero warn|hero fail|hero`, fail computed colour `rgb(207,34,46)` at 38.4px (2.4rem), bar width+class from the verdict, null value = `n/a` with `is-null`; all four pages render unchanged. Screenshots: `data/logs/hero-number-preview.png`, `hero-task-{index,system,llm,ups}.png`.
|
||||
- Green: `uv run pytest` → **356 passed, 6 skipped**; `uv run pytest --cov` → TOTAL **95%**, `severity.py` 100%; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` empty; `curl -s localhost:8090/static/dash.css | grep -c "\.hero"` → 17.
|
||||
- Docs: AGENTS.md — `web/` responsibility names `Dash.hero`/`Dash.severityFrom`, UI conventions describe `.hero`/`.hero-bar` and the no-`.sev-*` rule, counts refreshed (356 tests, ~95%); README — `web/` bullet documents the hero helper and "no verdict → `n/a`, no colour".
|
||||
- Decision: `hero()`'s optional `bar` colours itself only from the payload verdict (never `severityFor`), and `.hero-bar.ok .bar > i` lets hand-written markup inherit the verdict from the hero, so no component can reintroduce a JS threshold.
|
||||
- Next pending task: `02_overview_fleet_and_alerts/01_alerts_builder.md` (phase 01 tasks all done).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
........................................................................ [ 19%]
|
||||
.....................................................................sss [ 39%]
|
||||
sss..................................................................... [ 59%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 99%]
|
||||
.. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
../../../.local/lib/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/.local/lib/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
All checks passed!
|
||||
+9
-3
@@ -10,7 +10,9 @@
|
||||
# scripts/dev-tunnels.sh forwards each one to 127.0.0.1:<local_port> so the dashboard on
|
||||
# the laptop reads live fleet data with nothing listening on the hosts.
|
||||
export DEV_TUNNELS="9121:driveripper-ops:9120 9122:deskwork-ops:9120 9123:gateway-ops:9120"
|
||||
export DEV_FORWARDS="3494:driveripper-ops:3493"
|
||||
# The remote port is whatever upsd actually listens on — 3493 is NUT's stock default, and
|
||||
# this fleet's driveripper answers on 3051, so check before copying the number.
|
||||
export DEV_FORWARDS="3494:driveripper-ops:3051"
|
||||
|
||||
# Do not assume upsd is on 3493 — nonstandard ports are common and a refused connection then
|
||||
# reads as "upsd is broken" while the service is fine. Ask the host what actually speaks NUT:
|
||||
@@ -20,8 +22,12 @@ export DEV_FORWARDS="3494:driveripper-ops:3493"
|
||||
|
||||
# Point the dashboard at those tunnels.
|
||||
export HOST_TARGETS="driveripper=http://127.0.0.1:9121 deskwork=http://127.0.0.1:9122 gateway=http://127.0.0.1:9123"
|
||||
export GUFO_BASE_URL=http://127.0.0.1:8731
|
||||
export VLLM_BASE_URL=http://127.0.0.1:8000
|
||||
# The engines are reached directly on their hostnames (nothing tunnels them here). If the
|
||||
# laptop cannot route to them, add them to DEV_FORWARDS above
|
||||
# (e.g. `8731:deskwork-ops:8731 8000:driveripper-ops:8000`) and point these at 127.0.0.1.
|
||||
# The host names below are placeholders (RFC 2606) — use your own.
|
||||
export GUFO_BASE_URL=http://deskwork.example.net:8731
|
||||
export VLLM_BASE_URL=http://driveripper.example.net:8000
|
||||
# The UPS source: the forwarded upsd above. NUT_PORT is the LOCAL port (3494), not 3493.
|
||||
# Leave NUT_SIMULATE=1 only while upsd is unreachable — it labels every panel SIMULATED.
|
||||
export NUT_HOST=127.0.0.1
|
||||
|
||||
@@ -27,7 +27,7 @@ file — plus an overview page that says which of the three you need:
|
||||
| `/` | Is anything wrong, and where do I look first? | all three, in one request |
|
||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo (llama.cpp family) + vLLM `/metrics` |
|
||||
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` on 3493 (our own wire-protocol client) |
|
||||
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
|
||||
|
||||
History is a first-class feature, not a sparkline afterthought: every number worth graphing
|
||||
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
|
||||
@@ -115,11 +115,25 @@ Responsibilities, one line each:
|
||||
battery. It must never be the default, and every payload it produces says `simulated`.
|
||||
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
|
||||
- `rates.py` — Δ/window estimator, keyed by resolved series name.
|
||||
- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric,
|
||||
`classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict
|
||||
beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for
|
||||
an unknown. The verdicts travel **additively** in payloads — `cpu.severity`,
|
||||
`memory.severity`, `disks.severity`, `gpus[].severity`, each UPS card's `severity`
|
||||
(charge/load/runtime), `severity` inside rated LLM cells (KV pressure, classified on a
|
||||
percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS
|
||||
row in `/api/summary`. Pages render the class; they never compare a number to a threshold.
|
||||
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
|
||||
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
|
||||
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
|
||||
with an in-memory DB and no targets.
|
||||
- `web/` — four static pages, one stylesheet, `charts.js`. No framework.
|
||||
- `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the
|
||||
colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
|
||||
`ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
|
||||
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
|
||||
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
|
||||
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
|
||||
values the API does not rate; converting its remaining call sites is phases 02–05's job.
|
||||
|
||||
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
|
||||
`gpu.busy_pct@0`, `llm.decode_tps`, `ups.charge_pct`), `@stream` for per-device series.
|
||||
@@ -141,11 +155,13 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
*Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n'
|
||||
| nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are
|
||||
exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`:
|
||||
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U.
|
||||
`apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is
|
||||
`ups.description`, the operator's label — the driver's own `identity.model` answers
|
||||
`GX1500U`, so the description is what identifies the box).
|
||||
Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date
|
||||
(`battery.date` is often an unset placeholder like `2001-09-25`), CyberPower reports
|
||||
`battery.mfr.date: CPS` (not a date → `unknown`, never a fake age), neither driver reports
|
||||
a battery temperature, and `ups.test.result: No test initiated` is the factory state.
|
||||
a battery temperature, and `ups.test.result: No test initiated` is the factory state on both.
|
||||
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
|
||||
set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
|
||||
work that is still blocked).
|
||||
@@ -162,16 +178,24 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
|
||||
./scripts/dev-server.sh status # per-source ok/DOWN
|
||||
./scripts/dev-server.sh logs
|
||||
|
||||
uv run pytest # hermetic: 258 tests, no network
|
||||
uv run pytest --cov # coverage gate (~94%)
|
||||
uv run pytest # hermetic: 356 tests, no network
|
||||
uv run pytest --cov # coverage gate (94.75%; floor 94%)
|
||||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||||
uv run ruff check src tests agent && uv run pyright
|
||||
./scripts/capture-fixtures.sh # re-record ground truth
|
||||
|
||||
bash .agents/validate.sh # the phase gate: the four checks above (pytest
|
||||
# --cov --cov-fail-under=94, ruff, pyright, plus a
|
||||
# node --check of web/static) + repo hygiene.
|
||||
# Always `uv run` — a bare `python3`/`ruff` on PATH
|
||||
# tests an interpreter nobody deploys.
|
||||
```
|
||||
|
||||
Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
|
||||
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
|
||||
per monitored host (systemd unit, own user, `--token-file`, `ProtectSystem=strict`).
|
||||
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
|
||||
`--status`/`--uninstall`; it stores the token in a root-owned 0600 file and hands the agent
|
||||
`--token-file`, never argv, and runs under `ProtectSystem=strict`).
|
||||
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
|
||||
service here.
|
||||
|
||||
@@ -190,8 +214,10 @@ service here.
|
||||
test on hardware something is plugged into is not a test.
|
||||
- The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest,
|
||||
so it stays in the default run and is *not* marked `live`.
|
||||
- Coverage gate: 80% is the floor, ~94% is where we are; keep it there. API tests use
|
||||
`create_app()` with an in-memory DB, never a real target.
|
||||
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
|
||||
94% (`--cov-fail-under`), and ~95% is where we are; keep it there. New/modified Python
|
||||
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
|
||||
never a real target.
|
||||
|
||||
## UI conventions
|
||||
|
||||
@@ -201,6 +227,12 @@ dots in the header, window chips (1h…90d) on history pages. A panel that canno
|
||||
why in the panel — the operator should never have to open devtools to learn that a source is
|
||||
unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
||||
|
||||
A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals,
|
||||
muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's
|
||||
`severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*`
|
||||
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
|
||||
rule reaches for a token that `:root` does not declare.
|
||||
|
||||
## Known gaps / next work
|
||||
|
||||
1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see
|
||||
@@ -208,15 +240,22 @@ unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
||||
`actions` in upsd.users" because no such user exists yet. Creating one safely needs a
|
||||
*read* of `/etc/ups/upsd.users` first — `sudo tee` can write the file but overwriting it
|
||||
blind would delete the users upsmon already authenticates as, which can take the box down
|
||||
at the wrong moment. `deploy/nut-sudoers` lists the read rules; once they are in, add
|
||||
`[dashboards]` with `instcmds` limited to the battery tests, put the credentials in
|
||||
`.env.dev`, and the buttons come alive. Never grant `actions = -`.
|
||||
at the wrong moment. `deploy/nut-sudoers` is the installable ruleset that makes that
|
||||
read (and only a scoped write + reload) possible — it carries the
|
||||
`ops ALL=(root) NOPASSWD: NUT_READ, NUT_WRITE` grant and is installed with
|
||||
`sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut`,
|
||||
but it is **repo-side only so far**: the live UPS cards still answer "NUT_USER /
|
||||
NUT_PASSWORD must name a user with `actions` in upsd.users" (verified through
|
||||
`/api/ups/snapshot`). Once it is installed: read `upsd.users`, add `[dashboards]` with
|
||||
`instcmds` limited to the battery tests, put the credentials in `.env.dev`, and the
|
||||
buttons come alive. Never grant `actions = -`, and never write the file blind.
|
||||
2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs
|
||||
(`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current
|
||||
fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at
|
||||
all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the
|
||||
field map is written.
|
||||
3. **No remote yet.** Two commits, no `origin` — the user is providing it.
|
||||
3. **No remote yet.** No `origin` is configured (`git remote -v` is empty) — the user is
|
||||
providing it, so every phase so far is local-only history.
|
||||
4. **UPS history depth.** Events and the numeric series are stored; long-window battery
|
||||
health (capacity fade over months) has the rollups but no dedicated view.
|
||||
|
||||
@@ -240,6 +279,9 @@ unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
||||
|
||||
## Definition of done (per change)
|
||||
|
||||
The first three are exactly what `bash .agents/validate.sh` (the phase gate) checks, so no
|
||||
phase may be called done on a subset of them.
|
||||
|
||||
`uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated`
|
||||
empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted
|
||||
in a browser against **live** data · fixtures re-captured if an upstream contract changed ·
|
||||
|
||||
@@ -7,7 +7,7 @@ Three dashboards, one domain, one process, one SQLite file:
|
||||
| `/` | Is anything wrong, and which dashboard do I need? | all of the below, one request |
|
||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd` on 3493 (its own wire protocol client) |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
||||
|
||||
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
||||
runtime; every asset is served by this app.
|
||||
@@ -39,6 +39,8 @@ for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"):
|
||||
PY
|
||||
./scripts/dev-tunnels.sh start # 9121/9122/9123 -> each host's agent,
|
||||
# + DEV_FORWARDS -> upsd on 127.0.0.1:3494
|
||||
# (the remote port is upsd's real one — 3051
|
||||
# on driveripper, not the 3493 default)
|
||||
set -a; . ./.env.dev; set +a
|
||||
uv run dashboards # foreground
|
||||
./scripts/dev-server.sh restart # or backgrounded, with a pidfile + logs
|
||||
@@ -51,7 +53,9 @@ that already listens there (upsd) — that is how the UPS page reads a real batt
|
||||
## The host agent
|
||||
|
||||
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
|
||||
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` as JSON.
|
||||
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
|
||||
JSON document), `GET /info` (slow-changing identity: OS, kernel, CPU model, GPU list), and
|
||||
`GET /health` (the one route that never needs the token) as JSON.
|
||||
|
||||
```bash
|
||||
python3 host_agent.py --once # print one snapshot, exit
|
||||
@@ -84,15 +88,20 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
||||
| Variable | Meaning |
|
||||
| ------------------------------------- | -------------------------------------------------------------- |
|
||||
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
|
||||
| `HOST_INTERVAL_SECONDS` | poll interval for hosts (default 10) |
|
||||
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
||||
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) |
|
||||
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
|
||||
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
|
||||
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
|
||||
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
||||
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
|
||||
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
||||
| `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app |
|
||||
| `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |
|
||||
| `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags |
|
||||
| `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) |
|
||||
| `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer |
|
||||
| `DEBUGPY` / `DEBUGPY_PORT` | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup |
|
||||
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |
|
||||
|
||||
A source that is not configured answers `404 {"configured": false}` **without dialling**,
|
||||
@@ -119,8 +128,19 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
|
||||
blanks or blocks another source.
|
||||
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
|
||||
pruned, so a year of history is a few hundred MB rather than a disk emergency.
|
||||
* `severity.py` — the single green/amber/red table (`THRESHOLDS`, plus `INVERTED` for the
|
||||
metrics where lower is worse). Rated values in the API payloads travel with a sibling
|
||||
`severity` (`cpu.severity.usage_pct`, `memory.severity`, `disks.severity`,
|
||||
`gpus[].severity`, a UPS card's `severity.charge_pct`, `severity` inside the rated LLM
|
||||
cells such as KV pressure, `/api/summary` rows), computed server-side, so every page
|
||||
colours the same way and a value nobody reported renders `n/a` instead of green. A metric
|
||||
published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays
|
||||
what the upstream published.
|
||||
* `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No
|
||||
framework, no CDN, no bundler.
|
||||
framework, no CDN, no bundler. A big colour-coded number is `Dash.hero(el, {...})` on the
|
||||
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
|
||||
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
|
||||
with no verdict renders `n/a` with no colour at all.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
@@ -45,6 +45,12 @@ Cmnd_Alias NUT_WRITE = \
|
||||
/usr/bin/systemctl restart nut-server, \
|
||||
/usr/bin/systemctl reload nut-server
|
||||
|
||||
# ---- grants --------------------------------------------------------------------- #
|
||||
# A Cmnd_Alias grants nothing on its own — this line is the point of the file.
|
||||
# Scope: read the NUT config/state, add/replace a scoped upsd.users entry, and
|
||||
# restart/reload the server so upsd re-reads it. Nothing else.
|
||||
ops ALL=(root) NOPASSWD: NUT_READ, NUT_WRITE
|
||||
|
||||
# ---- deliberately NOT granted -------------------------------------------------------- #
|
||||
# /usr/bin/upscmd, /usr/bin/upsrw — the dashboard's gated POST /api/ups/command is the
|
||||
# ONLY command path in this project, by design
|
||||
|
||||
+46
-3
@@ -21,6 +21,7 @@ from fastapi import APIRouter, Body, Depends, HTTPException, Query, Request
|
||||
from .collectors import NutMonitor
|
||||
from .config import Settings
|
||||
from .poller import Hub
|
||||
from .severity import classify
|
||||
from .sources import nut
|
||||
from .sources.host import fleet_summary
|
||||
from .sources.llm import fleet_totals
|
||||
@@ -74,6 +75,43 @@ def _history(store: SampleStore, source: str, host: str, metric: str, span: floa
|
||||
}
|
||||
|
||||
|
||||
def _host_severity(host: dict[str, Any]) -> dict[str, str | None]:
|
||||
"""Verdicts for the overview's fleet-table columns, from the one threshold table.
|
||||
|
||||
The row flattens `cpu.usage_pct` into `cpu_usage_pct`; the verdict is looked up under
|
||||
the row's own key so the page never has to know which section a number came from.
|
||||
"""
|
||||
cpu = host.get("cpu") or {}
|
||||
return {
|
||||
"cpu_usage_pct": classify("cpu.usage_pct", cpu.get("usage_pct")),
|
||||
"mem_used_pct": classify("mem.used_pct", (host.get("memory") or {}).get("used_pct")),
|
||||
"disk_worst_used_pct": classify("disk.used_pct",
|
||||
(host.get("disks") or {}).get("worst_used_pct")),
|
||||
"temp_c": classify("cpu.temp_c", cpu.get("temp_c")),
|
||||
}
|
||||
|
||||
|
||||
def _fleet_severity(fleet: dict[str, Any]) -> dict[str, str | None]:
|
||||
"""Fleet aggregates judged by the same rows as the per-host values — no fleet-only
|
||||
thresholds, so "the fleet is amber" means the same thing as one amber host."""
|
||||
return {
|
||||
"cpu_usage_pct_avg": classify("cpu.usage_pct", fleet.get("cpu_usage_pct_avg")),
|
||||
"memory_used_pct": classify("mem.used_pct", fleet.get("memory_used_pct")),
|
||||
"disk_worst_used_pct": classify("disk.used_pct", fleet.get("disk_worst_used_pct")),
|
||||
"gpu_busy_pct_avg": classify("gpu.busy_pct", fleet.get("gpu_busy_pct_avg")),
|
||||
}
|
||||
|
||||
|
||||
def _ups_severity(card: dict[str, Any]) -> dict[str, str | None]:
|
||||
"""The three hero battery numbers, keyed as the overview row names them."""
|
||||
numbers = card.get("numbers") or {}
|
||||
return {
|
||||
"charge_pct": classify("ups.charge_pct", numbers.get("charge_pct")),
|
||||
"load_pct": classify("ups.load_pct", numbers.get("load_pct")),
|
||||
"runtime_seconds": classify("ups.battery_runtime", numbers.get("battery_runtime")),
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Common routes
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -161,10 +199,13 @@ def common_router() -> APIRouter:
|
||||
nut_job = ctx.hub.jobs.get("nut")
|
||||
if nut_job and nut_job.state.payload:
|
||||
ups_payload = nut_job.state.payload
|
||||
fleet = fleet_summary(hosts) if hosts else None
|
||||
if fleet is not None:
|
||||
fleet["severity"] = _fleet_severity(fleet)
|
||||
return {
|
||||
"ts": time.time(),
|
||||
"system": {"enabled": bool(ctx.settings.host_targets),
|
||||
"fleet": fleet_summary(hosts) if hosts else None,
|
||||
"fleet": fleet,
|
||||
"hosts": [{"name": h["name"], "os": (h.get("info") or {}).get("os_pretty"),
|
||||
"cpu_usage_pct": h["cpu"]["usage_pct"],
|
||||
"load1": h["cpu"].get("load1"),
|
||||
@@ -173,7 +214,8 @@ def common_router() -> APIRouter:
|
||||
"disk_worst_used_pct": h["disks"]["worst_used_pct"],
|
||||
"gpu_busy_pct": [g.get("busy_pct") for g in h.get("gpus") or []],
|
||||
"temp_c": h["cpu"].get("temp_c"),
|
||||
"uptime_seconds": (h.get("info") or {}).get("uptime_seconds")}
|
||||
"uptime_seconds": (h.get("info") or {}).get("uptime_seconds"),
|
||||
"severity": _host_severity(h)}
|
||||
for h in hosts],
|
||||
"states": ctx.hub.states("system")},
|
||||
"llm": {"enabled": bool(ctx.settings.gufo_url or ctx.settings.vllm_url),
|
||||
@@ -190,7 +232,8 @@ def common_router() -> APIRouter:
|
||||
"status": card["status"],
|
||||
"charge_pct": card["numbers"].get("charge_pct"),
|
||||
"load_pct": card["numbers"].get("load_pct"),
|
||||
"runtime_seconds": card["numbers"].get("battery_runtime")}
|
||||
"runtime_seconds": card["numbers"].get("battery_runtime"),
|
||||
"severity": _ups_severity(card)}
|
||||
for card in (ups_payload or {}).get("ups", [])],
|
||||
"states": ctx.hub.states("ups")},
|
||||
"recent_events": [event.as_json() for event in ctx.store.recent_events(12)],
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
"""The one place that decides whether a number is fine, worrying, or bad.
|
||||
|
||||
Two rules, and the whole module exists to keep them true:
|
||||
|
||||
1. **Unknown is not ok.** A value the upstream does not publish — `None`, a NaN that
|
||||
leaked through a parser, a metric with no entry in :data:`THRESHOLDS` — classifies to
|
||||
``None``, never to ``ok``. Green means "measured and good", the same promise the
|
||||
`AGENTS.md` "never invent a number" rule makes about the value itself. The page renders
|
||||
`n/a` with no colour class.
|
||||
2. **A threshold exists nowhere else in the app.** Pages render the class the API sent and
|
||||
must not compare a number to a magic value; the JS helpers in ``web/static`` read the
|
||||
``severity`` field instead of re-deriving it. Four pages with four opinions about what
|
||||
red means is the bug this module prevents.
|
||||
|
||||
Vocabulary is ``ok`` / ``warn`` / ``fail`` — the words ``dash.css`` (``.bar.ok``,
|
||||
``.dot.warn``, ``.chip.fail``) and ``sources/nut.py``'s ``health.checks[].status`` already
|
||||
use. "Danger" in the spec means ``fail``; there is no synonym for the same colour.
|
||||
|
||||
Reaching a threshold trips it, in both directions: for ``cpu.usage_pct`` (warn 75, danger
|
||||
90) 75 is ``warn`` and 90 is ``fail``; for the inverted ``ups.charge_pct`` (warn 50,
|
||||
danger 20) 50 is ``warn`` and 20 is ``fail``, because there a *smaller* number is worse.
|
||||
:data:`INVERTED` is the only place that direction is recorded, so a caller cannot get it
|
||||
backwards by accident.
|
||||
|
||||
Every `_pct` metric is on a 0-100 scale, never 0-1: an upstream that publishes a ratio
|
||||
(`llamacpp:kv_cache_usage_ratio`) is converted to percent *before* it is classified, or it
|
||||
will read as "fine" forever.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from collections.abc import Mapping
|
||||
from typing import Any
|
||||
|
||||
OK = "ok"
|
||||
WARN = "warn"
|
||||
FAIL = "fail"
|
||||
|
||||
#: metric name -> (warn, danger). The numbers were ratified on 2026-10-16 and match what
|
||||
#: the old JS call sites passed, so turning the thresholds into data changes colour on the
|
||||
#: pages only where they had already drifted. The two UPS metrics are new: a battery at
|
||||
#: half charge or with ten minutes of runtime left is worth amber before it is worth red.
|
||||
#: Keys are the dotted metric names of the history namespace; ``@stream`` suffixes
|
||||
#: (``disk.used_pct@/srv``, ``gpu.busy_pct@0``) inherit their base metric's row.
|
||||
THRESHOLDS: dict[str, tuple[float, float]] = {
|
||||
"cpu.usage_pct": (75.0, 90.0),
|
||||
"mem.used_pct": (80.0, 92.0),
|
||||
"disk.used_pct": (80.0, 90.0),
|
||||
"cpu.temp_c": (75.0, 90.0),
|
||||
"gpu.busy_pct": (85.0, 95.0),
|
||||
"gpu.mem_used_pct": (85.0, 95.0),
|
||||
"llm.kv_cache_usage_pct": (75.0, 90.0),
|
||||
"ups.charge_pct": (50.0, 20.0),
|
||||
"ups.battery_runtime": (600.0, 300.0), # seconds of runtime left
|
||||
"ups.load_pct": (70.0, 90.0),
|
||||
}
|
||||
|
||||
#: Metrics where **lower is worse**, so ``classify`` owns the direction. Every entry must
|
||||
#: also be in :data:`THRESHOLDS` (a test enforces it, along with bound ordering).
|
||||
INVERTED: set[str] = {"ups.charge_pct", "ups.battery_runtime"}
|
||||
|
||||
|
||||
def _number(value: float | None) -> float | None:
|
||||
"""The value as a finite float, or ``None`` when there is no number to classify.
|
||||
|
||||
NaN and ±inf are not renderable and not JSON-serialisable; they become the same
|
||||
`n/a` the app uses for a missing upstream value rather than a colour nobody can justify.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
num = float(value)
|
||||
return num if math.isfinite(num) else None
|
||||
|
||||
|
||||
def _resolve(metric: str) -> tuple[float, float, bool] | None:
|
||||
"""(warn, danger, inverted) for a metric, or ``None`` if the table does not know it.
|
||||
|
||||
``metric@stream`` resolves to ``metric``: a per-core, per-mount, or per-card series is
|
||||
judged by the same rule as the aggregate it came from.
|
||||
"""
|
||||
base = metric.split("@", 1)[0]
|
||||
pair = THRESHOLDS.get(base)
|
||||
if pair is None:
|
||||
return None
|
||||
warn, danger = pair
|
||||
return warn, danger, base in INVERTED
|
||||
|
||||
|
||||
def classify(metric: str, value: float | None) -> str | None:
|
||||
"""`ok`, `warn`, `fail` — or ``None`` when nothing honest can be said.
|
||||
|
||||
``None`` in → ``None`` out, always: no measurement, no verdict. An unknown metric name
|
||||
is the same case (no threshold, no colour) and never raises.
|
||||
"""
|
||||
num = _number(value)
|
||||
if num is None:
|
||||
return None
|
||||
bounds = _resolve(metric)
|
||||
if bounds is None:
|
||||
return None
|
||||
warn, danger, inverted = bounds
|
||||
if inverted:
|
||||
if num <= danger:
|
||||
return FAIL
|
||||
if num <= warn:
|
||||
return WARN
|
||||
return OK
|
||||
if num >= danger:
|
||||
return FAIL
|
||||
if num >= warn:
|
||||
return WARN
|
||||
return OK
|
||||
|
||||
|
||||
def attach(metric: str, value: float | None) -> dict[str, Any]:
|
||||
"""`{"value": …, "severity": …}` for payloads where the metric is a bare number.
|
||||
|
||||
The only mutation of the caller's value is turning a non-finite float into ``None``,
|
||||
which keeps the payload valid JSON and the panel honest.
|
||||
"""
|
||||
num = _number(value)
|
||||
return {"value": num, "severity": classify(metric, num)}
|
||||
|
||||
|
||||
def with_severity(metric: str, cell: Mapping[str, Any] | None,
|
||||
scale: float = 1.0) -> dict[str, Any]:
|
||||
"""Copy of a `{value, source, derived}` cell (LLM metrics) with `severity` added.
|
||||
|
||||
Additive-only: every existing key keeps its name and meaning, and a cell that is
|
||||
missing or holds a non-numeric value still comes back with ``severity: null``. The
|
||||
input mapping is never mutated.
|
||||
|
||||
``scale`` rescales the value **for the classification only**: the KV-cache gauges are
|
||||
published as 0-1 ratios (`llamacpp:kv_cache_usage_ratio`, `vllm:kv_cache_usage_perc`)
|
||||
while the table is written in percent, so classifying the raw ratio would read "fine"
|
||||
forever. The cell's own `value` is left exactly as the upstream published it — the
|
||||
conversion is a unit, not a new measurement, and the page still prints what it got.
|
||||
"""
|
||||
out: dict[str, Any] = dict(cell) if cell is not None else {}
|
||||
raw = out.get("value")
|
||||
value = raw if raw is None or isinstance(raw, (int, float)) else None
|
||||
out["severity"] = classify(metric, None if value is None else value * scale)
|
||||
return out
|
||||
@@ -21,6 +21,7 @@ from typing import Any
|
||||
|
||||
from ..poller import Result
|
||||
from ..rates import RateEstimator
|
||||
from ..severity import classify
|
||||
from ..store import Sample
|
||||
|
||||
# /proc/stat jiffies we treat as "not doing useful work".
|
||||
@@ -231,6 +232,7 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl
|
||||
temp_c = max(temps.values()) if temps else None
|
||||
pools_used = None if vram_used is None else vram_used + (gtt_used or 0.0)
|
||||
pools_total = None if vram_total is None else vram_total + (gtt_total or 0.0)
|
||||
pool_used_pct = _ratio_percent(pools_used, pools_total)
|
||||
row = {
|
||||
"index": index,
|
||||
"card": gpu.get("card"),
|
||||
@@ -250,11 +252,14 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl
|
||||
# this is the honest whole-addressable-pool view, kept beside the two pools
|
||||
"mem_used_bytes": pools_used,
|
||||
"mem_total_bytes": pools_total,
|
||||
"mem_used_pct": _ratio_percent(pools_used, pools_total),
|
||||
"mem_used_pct": pool_used_pct,
|
||||
"power_watts": gpu.get("power_watts"),
|
||||
"temps": temps,
|
||||
"temp_c": temp_c,
|
||||
"clocks": gpu.get("clocks") or {},
|
||||
# the verdict travels with the card; keys mirror the rated fields beside them
|
||||
"severity": {"busy_pct": classify("gpu.busy_pct", gpu.get("busy_pct")),
|
||||
"mem_used_pct": classify("gpu.mem_used_pct", pool_used_pct)},
|
||||
}
|
||||
gpus.append(row)
|
||||
for metric, value in (("busy_pct", row["busy_pct"]), ("vram_used_bytes", vram_used),
|
||||
@@ -324,6 +329,10 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl
|
||||
"procs_running": cpu_raw.get("procs_running"),
|
||||
"procs_blocked": cpu_raw.get("procs_blocked"),
|
||||
"temp_c": cpu_temp,
|
||||
# Additive: the pages colour from this verdict instead of comparing the number
|
||||
# to a threshold of their own. Keys mirror the rated fields in this section.
|
||||
"severity": {"usage_pct": classify("cpu.usage_pct", usage_pct),
|
||||
"temp_c": classify("cpu.temp_c", cpu_temp)},
|
||||
},
|
||||
"memory": {
|
||||
"total_bytes": mem_total,
|
||||
@@ -336,6 +345,7 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl
|
||||
"swap_used_bytes": swap_used,
|
||||
"swap_used_pct": swap_used_pct,
|
||||
"committed_bytes": mem.get("committed_bytes"),
|
||||
"severity": {"used_pct": classify("mem.used_pct", mem_used_pct)},
|
||||
},
|
||||
"disks": {
|
||||
"filesystems": filesystems,
|
||||
@@ -347,6 +357,8 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl
|
||||
"writes_per_s": writes_total if any_io else None,
|
||||
"util_pct": util_max,
|
||||
"devices": io_devices,
|
||||
"severity": {"worst_used_pct": classify(
|
||||
"disk.used_pct", worst.get("used_pct") if worst else None)},
|
||||
},
|
||||
"network": {
|
||||
"rx_bytes_per_s": rx_total if any_net else None,
|
||||
|
||||
@@ -31,6 +31,7 @@ from typing import Any
|
||||
from ..poller import Result
|
||||
from ..prom import Prom, field
|
||||
from ..rates import RateEstimator
|
||||
from ..severity import with_severity
|
||||
from ..store import Sample
|
||||
|
||||
GUFO_SERIES: dict[str, tuple[tuple[str, ...], str]] = {
|
||||
@@ -91,6 +92,23 @@ VLLM_HISTOGRAMS: dict[str, tuple[str, ...]] = {
|
||||
"e2e_request_latency_seconds": ("vllm:e2e_request_latency_seconds",),
|
||||
}
|
||||
|
||||
#: Cells that have a row in `severity.THRESHOLDS`, mapped to (metric, scale). Both engines
|
||||
#: publish KV pressure as a 0-1 ratio while the threshold table is written in percent, so
|
||||
#: the scale converts for the *verdict only* — `value` stays what the engine published
|
||||
#: (rule 2: no invented number), and 0.85 is `warn` rather than a permanent `ok`.
|
||||
SEVERITY_CELLS: dict[str, tuple[str, float]] = {
|
||||
"kv_cache_usage_ratio": ("llm.kv_cache_usage_pct", 100.0), # gufo / llama.cpp
|
||||
"gpu_cache_usage": ("llm.kv_cache_usage_pct", 100.0), # vLLM
|
||||
}
|
||||
|
||||
|
||||
def attach_cell_severity(metrics: dict[str, Any]) -> None:
|
||||
"""Add `severity` to the rated cells of a parsed metrics dict, in place, additively."""
|
||||
for key, (metric, scale) in SEVERITY_CELLS.items():
|
||||
if key in metrics:
|
||||
metrics[key] = with_severity(metric, metrics[key], scale=scale)
|
||||
|
||||
|
||||
#: history series per engine, in the order the LLM page charts them
|
||||
HISTORY_METRICS: dict[str, tuple[str, ...]] = {
|
||||
"decode_tokens_per_second": ("decode_tokens_per_second",),
|
||||
@@ -165,6 +183,7 @@ def parse_gufo_metrics(text: str, rates: RateEstimator | None = None,
|
||||
out["spec_drafts_per_request"] = _ratio(out["spec_drafts_total"],
|
||||
out["spec_draft_tokens_total"],
|
||||
"computed:drafts/draft_tokens")
|
||||
attach_cell_severity(out)
|
||||
return out
|
||||
|
||||
|
||||
@@ -244,6 +263,7 @@ def parse_vllm_metrics(text: str, rates: RateEstimator | None = None,
|
||||
out["prefix_cache_hit_ratio"] = _ratio(out["prefix_cache_hits_total"],
|
||||
out["prefix_cache_queries_total"],
|
||||
"computed:hits/queries")
|
||||
attach_cell_severity(out)
|
||||
return out
|
||||
|
||||
|
||||
|
||||
@@ -34,6 +34,8 @@ import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from ..severity import classify
|
||||
|
||||
DEFAULT_PORT = 3493
|
||||
|
||||
|
||||
@@ -741,6 +743,12 @@ def payload(reading: UpsReading, *, now: float | None = None, **assess_kwargs: A
|
||||
"alarms": reading.alarms,
|
||||
"on_battery": reading.on_battery,
|
||||
"numbers": numbers,
|
||||
# Severity answers "how bad is this number" for the three hero values; `health`
|
||||
# and its `checks[].status` answer "is this condition satisfied". Both stay.
|
||||
"severity": {"charge_pct": classify("ups.charge_pct", numbers.get("charge_pct")),
|
||||
"load_pct": classify("ups.load_pct", numbers.get("load_pct")),
|
||||
"battery_runtime": classify("ups.battery_runtime",
|
||||
numbers.get("battery_runtime"))},
|
||||
"health": health,
|
||||
"advertised_commands": reading.commands,
|
||||
"writable": reading.writable,
|
||||
|
||||
@@ -102,6 +102,16 @@ const Dash = (() => {
|
||||
el.classList.add(threshold);
|
||||
}
|
||||
|
||||
/**
|
||||
* FALLBACK ONLY: a colour class derived from thresholds the *caller* passes.
|
||||
*
|
||||
* Thresholds live in `src/dashboards/severity.py` and travel in the payload as
|
||||
* `severity`; `severityFrom()` below is how a page reads them. This function survives
|
||||
* only because `page-index.js`, `page-system.js`, `page-llm.js` and `page-ups.js` still
|
||||
* call it with their own numbers (phases 02–05 convert them page by page, so each page's
|
||||
* change is reviewable on its own). Do not add a new call site: a threshold written in
|
||||
* JS is a threshold that can disagree with the other three pages.
|
||||
*/
|
||||
function severityFor(value, warnAt, failAt) {
|
||||
const warn = warnAt == null ? 75 : warnAt;
|
||||
const fail = failAt == null ? 90 : failAt;
|
||||
@@ -111,6 +121,105 @@ const Dash = (() => {
|
||||
return 'ok';
|
||||
}
|
||||
|
||||
// ---- severity & hero numbers -------------------------------------------
|
||||
/** The only colour vocabulary in the app — matches `dash.css` and severity.py. */
|
||||
const SEVERITY_CLASSES = ['ok', 'warn', 'fail'];
|
||||
|
||||
/**
|
||||
* The colour class for a value, read from the payload instead of a threshold.
|
||||
*
|
||||
* Accepts a payload cell (`{value, severity}`), the bare `severity` string, or `null`,
|
||||
* and returns `ok` / `warn` / `fail` / `''`. Anything the API does not rate — no field,
|
||||
* `null`, a word from another vocabulary — returns `''`, which means *no colour*: an
|
||||
* unknown number must never look green.
|
||||
*/
|
||||
function severityFrom(cellOrSeverity) {
|
||||
let value = cellOrSeverity;
|
||||
if (value && typeof value === 'object') value = value.severity;
|
||||
if (typeof value !== 'string') return '';
|
||||
const word = value.trim().toLowerCase();
|
||||
return SEVERITY_CLASSES.indexOf(word) >= 0 ? word : '';
|
||||
}
|
||||
|
||||
/** First direct child with `className`, or null. */
|
||||
function childOf(parent, className) {
|
||||
for (const node of Array.from(parent.children)) {
|
||||
if (node.classList && node.classList.contains(className)) return node;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** The child with `className`, created (before `before`) on first use. */
|
||||
function ensureChild(parent, className, tag, before) {
|
||||
const existing = childOf(parent, className);
|
||||
if (existing) return existing;
|
||||
const node = document.createElement(tag);
|
||||
node.className = className;
|
||||
parent.insertBefore(node, before || null);
|
||||
return node;
|
||||
}
|
||||
|
||||
/** Text for a label or footnote: `null` prints nothing (unlike `setTxt`'s `n/a`). */
|
||||
function assignText(el, text) {
|
||||
const value = text == null ? '' : String(text);
|
||||
if (el.textContent !== value) el.textContent = value;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render — or update in place — one big colour-coded number.
|
||||
*
|
||||
* `hero(el, {label, value, unit, digits, severity, sub, bar})`
|
||||
* - `el`: an element or a selector (a missing element is a silent no-op, like the other
|
||||
* setters, so a page can call every hero it knows about);
|
||||
* - `severity`: taken from the payload (`{value, severity}` cell or the bare string) and
|
||||
* resolved by `severityFrom` — this function never compares a number to a threshold;
|
||||
* - a `null` / non-finite `value` prints `n/a`, faint, **with no severity class**, so a
|
||||
* value nobody measured cannot keep last poll's green;
|
||||
* - `bar`: `true` for a 0-100 value, or a number for another max — adds/updates a `.bar`
|
||||
* under the number (the `.hero-bar` variant of `dash.css`).
|
||||
*
|
||||
* The DOM is built once and then patched text-by-text, so a poll does not restart CSS
|
||||
* transitions, drop the tabular-numeral layout, or steal focus.
|
||||
*/
|
||||
function hero(el, options) {
|
||||
const node = typeof el === 'string' ? $(el) : el;
|
||||
if (!node) return null;
|
||||
const opts = options || {};
|
||||
node.classList.add('hero');
|
||||
assignText(ensureChild(node, 'label', 'div'), opts.label);
|
||||
const valueEl = ensureChild(node, 'value', 'div');
|
||||
const numberEl = ensureChild(valueEl, 'num', 'span');
|
||||
const unitEl = ensureChild(valueEl, 'unit', 'span');
|
||||
|
||||
const raw = opts.value;
|
||||
const measured = raw != null && (typeof raw === 'string' ? raw !== '' : isFinite(raw));
|
||||
const sev = measured ? severityFrom(opts.severity) : '';
|
||||
SEVERITY_CLASSES.forEach((name) => node.classList.toggle(name, name === sev));
|
||||
|
||||
if (measured) {
|
||||
const text = typeof raw === 'string' ? raw : num(raw, opts.digits);
|
||||
if (numberEl.textContent !== text) numberEl.textContent = text;
|
||||
numberEl.classList.remove('is-null', 'na');
|
||||
valueEl.classList.remove('is-null');
|
||||
} else {
|
||||
if (numberEl.textContent !== naText) numberEl.textContent = naText;
|
||||
numberEl.classList.add('is-null', 'na');
|
||||
valueEl.classList.add('is-null');
|
||||
}
|
||||
assignText(unitEl, measured ? opts.unit : '');
|
||||
|
||||
if (opts.bar) {
|
||||
const bar = ensureChild(node, 'bar', 'div', childOf(node, 'sub'));
|
||||
const fill = $('i', bar) || bar.appendChild(document.createElement('i'));
|
||||
const max = typeof opts.bar === 'number' && opts.bar > 0 ? opts.bar : 100;
|
||||
const width = typeof raw === 'number' ? Math.max(0, Math.min(100, (raw / max) * 100)) : 0;
|
||||
fill.style.width = `${width}%`;
|
||||
SEVERITY_CLASSES.forEach((name) => bar.classList.toggle(name, name === sev));
|
||||
}
|
||||
assignText(ensureChild(node, 'sub', 'div'), opts.sub);
|
||||
return node;
|
||||
}
|
||||
|
||||
function setDot(el, status) {
|
||||
if (!el) return;
|
||||
el.classList.remove('ok', 'warn', 'fail', 'idle');
|
||||
@@ -335,7 +444,7 @@ const Dash = (() => {
|
||||
|
||||
return { $, $$, fetchJSON, poll, fmtBytes, bytes, rate, pct, num, duration, age, setTxt,
|
||||
setValue, setBar, setDot, chip, initChrome, rangeControl, toast, bindPause,
|
||||
staleness, markStale, severityFor, ageSeconds, naText };
|
||||
staleness, markStale, severityFor, severityFrom, hero, ageSeconds, naText };
|
||||
})();
|
||||
|
||||
document.addEventListener('DOMContentLoaded', () => Dash.initChrome());
|
||||
|
||||
@@ -238,6 +238,54 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
|
||||
|
||||
.na { color: var(--text-faint); }
|
||||
|
||||
/* ---- hero numbers --------------------------------------------------------
|
||||
* The big glanceable number: label, value + unit, and optional bar and footnote.
|
||||
* Colour comes from the *same* `ok|warn|fail` classes the rest of the stylesheet uses
|
||||
* (`.bar.ok`, `.dot.warn`, `.chip.fail`) — there is deliberately no parallel `.sev-*`
|
||||
* family, because one word per colour is what keeps four pages agreeing. `Dash.hero()`
|
||||
* applies the class from the `severity` the API sent, so nothing here (or in JS) knows a
|
||||
* threshold. No class at all means no measurement: the number renders faint, never green.
|
||||
*/
|
||||
.hero {
|
||||
display: grid;
|
||||
gap: .18rem;
|
||||
min-width: 0;
|
||||
}
|
||||
.hero .label {
|
||||
font-size: 10.5px; text-transform: uppercase; letter-spacing: .06em; color: var(--text-faint);
|
||||
white-space: nowrap; overflow: hidden; text-overflow: ellipsis;
|
||||
}
|
||||
.hero .value {
|
||||
font-family: var(--mono); font-size: 2.4rem; font-weight: 650; line-height: 1.05;
|
||||
font-variant-numeric: tabular-nums; /* fixed digit width: no jitter between polls */
|
||||
display: flex; align-items: baseline; gap: .3rem; flex-wrap: wrap;
|
||||
color: var(--text);
|
||||
}
|
||||
.hero .value > span { white-space: nowrap; }
|
||||
.hero .unit { font-size: .38em; font-weight: 550; color: var(--text-muted); }
|
||||
.hero .sub {
|
||||
font-family: var(--mono); font-size: 11px; color: var(--text-faint);
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
/* an empty label/sub takes no space, so a hero without a footnote is not taller */
|
||||
.hero .label:empty, .hero .sub:empty { display: none; }
|
||||
|
||||
.hero.ok .value { color: var(--ok); }
|
||||
.hero.warn .value { color: var(--warn); }
|
||||
.hero.fail .value { color: var(--fail); }
|
||||
/* after the verdicts on purpose: `n/a` is not a measurement, so it is never coloured */
|
||||
.hero .value.is-null, .hero .value .is-null { color: var(--text-faint); font-size: 1.25rem; font-weight: 550; }
|
||||
|
||||
/* number + bar: the per-host cards and the UPS hero row both want the pair, and the bar
|
||||
takes its colour from the hero so markup does not have to repeat the verdict */
|
||||
.hero-bar { gap: .28rem; }
|
||||
.hero-bar .bar { width: 100%; }
|
||||
.hero-bar.ok .bar > i { background: var(--ok); }
|
||||
.hero-bar.warn .bar > i { background: var(--warn); }
|
||||
.hero-bar.fail .bar > i { background: var(--fail); }
|
||||
/* with no verdict on either the hero or the bar the fill stays the neutral accent: the
|
||||
bar shows *how much*, never a judgement nobody measured. */
|
||||
|
||||
/* ---- bars, dots, chips -------------------------------------------------- */
|
||||
.bar {
|
||||
position: relative; height: 7px; border-radius: 999px; background: var(--bg-sunken);
|
||||
|
||||
@@ -7,6 +7,8 @@ asset the network can fail to deliver.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import re
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
@@ -17,6 +19,9 @@ from fastapi.testclient import TestClient
|
||||
|
||||
from dashboards.app import create_app
|
||||
from dashboards.config import Settings, Target
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources.host import HostCollector, parse_host_snapshot
|
||||
from dashboards.store import Event, Sample, SampleStore
|
||||
|
||||
# refused immediately, and never dials out of the test environment
|
||||
@@ -63,6 +68,107 @@ def wait_for_payload(client: TestClient, path: str, *, timeout: float = 5.0) ->
|
||||
pytest.fail(f"{path} never became available")
|
||||
|
||||
|
||||
def host_payload(agent_snapshot: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Two scrapes of the real driveripper capture, nudged onto known sides of the table.
|
||||
|
||||
Driven from the fixture rather than hand-written so a schema drift breaks here first.
|
||||
"""
|
||||
snap = copy.deepcopy(agent_snapshot)
|
||||
snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95 # mem.used_pct -> fail
|
||||
snap["cpu_temperature_celsius"] = 91.0 # cpu.temp_c -> fail
|
||||
for fs in snap["filesystems"]:
|
||||
fs["used_pct"] = 93.0 # disk -> fail
|
||||
collector = HostCollector(RateEstimator(span=60.0))
|
||||
parse_host_snapshot("driveripper", snap, collector, now=1_000.0)
|
||||
second = copy.deepcopy(snap)
|
||||
second["ts"] = 1_010.0
|
||||
modes = second["cpu"]["modes"]
|
||||
modes["total"] += 1_000.0
|
||||
modes["idle"] += 900.0 # 10 % busy over the window -> ok
|
||||
return parse_host_snapshot("driveripper", second, collector, now=1_010.0).payload
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
agent_snapshot: dict[str, Any]) -> Iterator[TestClient]:
|
||||
"""Overview-shaped app: one host payload in the hub plus the simulated UPSes.
|
||||
|
||||
The host job points at a dead port, so a poll can never overwrite the injected payload
|
||||
(a failed poll keeps the last good one) — the numbers below are ours, not the fleet's.
|
||||
"""
|
||||
monkeypatch.setenv("NUT_SIMULATE", "1")
|
||||
db = str(tmp_path / "summary.sqlite3")
|
||||
store = SampleStore(db)
|
||||
app = create_app(Settings(db_path=db, nut_host="127.0.0.1",
|
||||
host_targets=(Target("driveripper", DEAD_URL),)), store)
|
||||
test_client = TestClient(app)
|
||||
test_client.__enter__()
|
||||
try:
|
||||
app.state.ctx.hub.jobs["host:driveripper"].state.payload = host_payload(agent_snapshot)
|
||||
wait_for_payload(test_client, "/api/ups/snapshot")
|
||||
yield test_client
|
||||
finally:
|
||||
test_client.__exit__(None, None, None)
|
||||
store.close()
|
||||
|
||||
|
||||
# ---- severity travels with the value --------------------------------------- #
|
||||
|
||||
def test_summary_carries_the_verdict_for_every_column_the_page_colours(data_client) -> None:
|
||||
body = data_client.get("/api/summary").json()
|
||||
row = body["system"]["hosts"][0]
|
||||
assert row["severity"] == {"cpu_usage_pct": "ok", "mem_used_pct": "fail",
|
||||
"disk_worst_used_pct": "fail", "temp_c": "fail"}
|
||||
# the fleet card is judged by the same table, so amber means the same thing at both sizes
|
||||
assert body["system"]["fleet"]["severity"] == {
|
||||
"cpu_usage_pct_avg": "ok", "memory_used_pct": "fail",
|
||||
"disk_worst_used_pct": "fail", "gpu_busy_pct_avg": "ok",
|
||||
}
|
||||
for ups in body["ups"]["ups"]:
|
||||
assert set(ups["severity"]) == {"charge_pct", "load_pct", "runtime_seconds"}
|
||||
assert ups["severity"]["charge_pct"] == classify("ups.charge_pct", ups["charge_pct"])
|
||||
assert ups["severity"]["charge_pct"] == "ok", "the simulator starts at float charge"
|
||||
|
||||
|
||||
def test_summary_shape_did_not_shrink(data_client) -> None:
|
||||
"""Additive-only anchor: /api/summary gained `severity` and lost nothing.
|
||||
|
||||
The overview, the live tests, and any external consumer read the flat keys; this test is
|
||||
the tripwire for a well-meaning refactor that "tidies" them away.
|
||||
"""
|
||||
body = data_client.get("/api/summary").json()
|
||||
fleet_keys = {"hosts_total", "hosts_live", "cpu_cores", "cpu_usage_pct_avg",
|
||||
"memory_total_bytes", "memory_used_bytes", "memory_used_pct",
|
||||
"disk_worst_used_pct", "load1_total", "net_rx_bytes_per_s",
|
||||
"net_tx_bytes_per_s", "gpu_count", "gpu_busy_pct_avg", "power_estimate_w"}
|
||||
host_keys = {"name", "os", "cpu_usage_pct", "load1", "cores", "mem_used_pct",
|
||||
"disk_worst_used_pct", "gpu_busy_pct", "temp_c", "uptime_seconds"}
|
||||
ups_keys = {"name", "overall", "status", "charge_pct", "load_pct", "runtime_seconds"}
|
||||
assert fleet_keys <= set(body["system"]["fleet"]), fleet_keys - set(body["system"]["fleet"])
|
||||
assert host_keys <= set(body["system"]["hosts"][0]), host_keys - set(body["system"]["hosts"][0])
|
||||
assert ups_keys <= set(body["ups"]["ups"][0]), ups_keys - set(body["ups"]["ups"][0])
|
||||
assert set(body) == {"ts", "system", "llm", "ups", "recent_events"}
|
||||
# ...and the new keys are there beside the old ones, not instead of them
|
||||
assert "severity" in body["system"]["fleet"]
|
||||
assert "severity" in body["system"]["hosts"][0]
|
||||
assert "severity" in body["ups"]["ups"][0]
|
||||
|
||||
|
||||
def test_snapshot_payloads_carry_their_own_severity_sections(data_client) -> None:
|
||||
"""/api/system/snapshot and /api/ups/snapshot ship the same verdicts the summary reuses."""
|
||||
system = data_client.get("/api/system/snapshot").json()
|
||||
host = system["hosts"][0]
|
||||
assert set(host["cpu"]["severity"]) == {"usage_pct", "temp_c"}
|
||||
assert set(host["memory"]["severity"]) == {"used_pct"}
|
||||
assert set(host["disks"]["severity"]) == {"worst_used_pct"}
|
||||
assert set(host["gpus"][0]["severity"]) == {"busy_pct", "mem_used_pct"}
|
||||
ups = data_client.get("/api/ups/snapshot").json()
|
||||
for card in ups["ups"]:
|
||||
assert set(card["severity"]) == {"charge_pct", "load_pct", "battery_runtime"}
|
||||
# numbers keep their own shape; severity is a sibling, never a replacement
|
||||
assert isinstance(card["numbers"], dict) and "charge_pct" in card["numbers"]
|
||||
|
||||
|
||||
# ---- unconfigured: say so, and do not dial -------------------------------- #
|
||||
|
||||
def test_config_reports_every_source_as_off(client) -> None:
|
||||
@@ -175,6 +281,104 @@ def test_static_assets_are_served(client) -> None:
|
||||
assert client.get(asset).status_code == 200, asset
|
||||
|
||||
|
||||
# ---- the severity language in the static assets --------------------------- #
|
||||
#
|
||||
# There is no JS test runner (a locked anchor: adding one would mean a build step).
|
||||
# What *can* be asserted end-to-end, from the served bytes, is the contract the pages
|
||||
# depend on: the hero style exists, its colour comes from the shared ok/warn/fail tokens,
|
||||
# no parallel `.sev-*` vocabulary appeared, and `Dash.hero()` reads the verdict instead of
|
||||
# comparing the number to a threshold in JavaScript.
|
||||
|
||||
ASSETS = ("/static/dash.css", "/static/app.js", "/static/charts.js",
|
||||
"/static/page-index.js", "/static/page-system.js", "/static/page-llm.js",
|
||||
"/static/page-ups.js")
|
||||
|
||||
|
||||
def _css(client: TestClient) -> str:
|
||||
return client.get("/static/dash.css").text
|
||||
|
||||
|
||||
def _hero_css(css: str) -> str:
|
||||
"""Just the `hero numbers` section of the stylesheet."""
|
||||
return css.split("/* ---- hero numbers", 1)[1].split("/* ---- bars", 1)[0]
|
||||
|
||||
|
||||
def test_css_ships_the_hero_number_style(client) -> None:
|
||||
css = _css(client)
|
||||
hero_block = _hero_css(css)
|
||||
for rule in (".hero {", ".hero .label", ".hero .value", ".hero .unit", ".hero .sub",
|
||||
".hero.ok .value", ".hero.warn .value", ".hero.fail .value",
|
||||
".hero-bar", ".hero-bar.ok .bar"):
|
||||
assert rule in hero_block, f"dash.css lost the {rule} rule"
|
||||
# a display size with fixed-width digits, so the number does not jitter between polls
|
||||
assert "2.4rem" in hero_block
|
||||
assert "tabular-nums" in hero_block
|
||||
|
||||
|
||||
def test_hero_colour_comes_from_the_shared_verdict_tokens(client) -> None:
|
||||
"""`.hero` reuses `--ok/--warn/--fail`; it does not invent a palette of its own."""
|
||||
hero_block = _hero_css(_css(client))
|
||||
for token in ("var(--ok)", "var(--warn)", "var(--fail)", "var(--text)",
|
||||
"var(--text-muted)", "var(--text-faint)"):
|
||||
assert token in hero_block, f"the hero style no longer uses {token}"
|
||||
|
||||
|
||||
def test_no_stylesheet_rule_uses_an_undeclared_token(client) -> None:
|
||||
"""A typo'd `var(--x)` renders unstyled and reads as a broken panel, not an error."""
|
||||
css = _css(client)
|
||||
root = re.search(r":root\s*\{(.*?)\n\}", css, re.S)
|
||||
assert root is not None
|
||||
declared = set(re.findall(r"(--[a-z0-9-]+)\s*:", root.group(1)))
|
||||
used = set(re.findall(r"var\(\s*(--[a-z0-9-]+)", css))
|
||||
assert not used - declared
|
||||
|
||||
|
||||
def _code(body: str) -> str:
|
||||
"""An asset with its comments stripped — prose may *name* a class the code must not define."""
|
||||
body = re.sub(r"/\*.*?\*/", " ", body, flags=re.S)
|
||||
body = re.sub(r"<!--.*?-->", " ", body, flags=re.S)
|
||||
return re.sub(r"^\s*//.*$", " ", body, flags=re.M)
|
||||
|
||||
|
||||
def test_no_parallel_severity_vocabulary_exists_anywhere(client) -> None:
|
||||
"""One word per colour: `ok|warn|fail`, as in severity.py and the existing components."""
|
||||
for asset in (*ASSETS, "/", "/system", "/llm", "/ups"):
|
||||
body = _code(client.get(asset).text)
|
||||
assert ".sev-" not in body, f"{asset} invented a .sev-* class"
|
||||
assert not re.search(r"classList\.(?:add|toggle)\(\s*[\"'](?:danger|good|warning)[\"']",
|
||||
body), f"{asset} invented a synonym for fail"
|
||||
|
||||
|
||||
def test_app_js_exposes_the_hero_helpers(client) -> None:
|
||||
js = client.get("/static/app.js").text
|
||||
assert "function hero(" in js
|
||||
assert "function severityFrom(" in js
|
||||
exports = js.rsplit("return {", 1)[1].split("}", 1)[0]
|
||||
for name in ("hero", "severityFrom", "severityFor"):
|
||||
assert re.search(rf"\b{name}\b", exports), f"Dash.{name} is not exported"
|
||||
|
||||
|
||||
def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
|
||||
"""`hero()` may only colour what the API rated — no thresholds re-derived in JS."""
|
||||
js = client.get("/static/app.js").text
|
||||
start = js.index("function hero(")
|
||||
body = js[start:js.index("\n function setDot", start)]
|
||||
assert "severityFrom(" in body
|
||||
assert "severityFor(" not in body
|
||||
# an unknown value gets n/a and no colour class at all
|
||||
assert "naText" in body
|
||||
assert re.search(r"raw != null", body)
|
||||
|
||||
|
||||
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
|
||||
"""The old call sites still work until phases 02-05 convert each page."""
|
||||
js = client.get("/static/app.js").text
|
||||
assert "FALLBACK ONLY" in js
|
||||
for page in ("/static/page-index.js", "/static/page-system.js",
|
||||
"/static/page-llm.js", "/static/page-ups.js"):
|
||||
assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site"
|
||||
|
||||
|
||||
def test_api_responses_are_not_cacheable(client) -> None:
|
||||
for path in ("/api/config", "/api/health", "/api/summary"):
|
||||
assert "no-store" in client.get(path).headers.get("cache-control", "")
|
||||
|
||||
@@ -13,6 +13,7 @@ from typing import Any
|
||||
import pytest
|
||||
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources.host import HostCollector, fleet_summary, parse_host_snapshot
|
||||
|
||||
TICKS_PER_SECOND = 100.0
|
||||
@@ -234,6 +235,80 @@ def test_fleet_summary_worst_disk_and_memory_totals(agent_snapshot) -> None:
|
||||
base["memory"]["used_bytes"] / base["memory"]["total_bytes"] * 100)
|
||||
|
||||
|
||||
# ---- severity: the verdict travels with the value ------------------------- #
|
||||
|
||||
def test_payloads_carry_a_severity_beside_every_rated_value(agent_snapshot) -> None:
|
||||
"""The page paints a class; it must never hold its own opinion about where amber starts."""
|
||||
_first, payload = parse_twice(agent_snapshot)
|
||||
assert payload["cpu"]["severity"] == {
|
||||
"usage_pct": classify("cpu.usage_pct", payload["cpu"]["usage_pct"]),
|
||||
"temp_c": classify("cpu.temp_c", payload["cpu"]["temp_c"]),
|
||||
}
|
||||
assert payload["cpu"]["severity"]["usage_pct"] == "ok" # the window is 10 % busy
|
||||
assert payload["cpu"]["severity"]["temp_c"] == "ok" # 68 °C in the capture
|
||||
assert payload["memory"]["severity"]["used_pct"] == "ok" # 68 % of 128 GB
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] == "ok" # 74.6 % worst mount
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["severity"] == {
|
||||
"busy_pct": classify("gpu.busy_pct", gpu["busy_pct"]),
|
||||
"mem_used_pct": classify("gpu.mem_used_pct", gpu["mem_used_pct"]),
|
||||
}
|
||||
assert gpu["severity"]["busy_pct"] == "ok"
|
||||
|
||||
|
||||
def test_severity_trips_on_the_ratified_boundaries(agent_snapshot) -> None:
|
||||
"""Values pushed onto the table's edges in the real fixture, not on a hand-drawn snapshot."""
|
||||
snap = copy.deepcopy(agent_snapshot)
|
||||
snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.92 # mem.used_pct danger
|
||||
snap["cpu_temperature_celsius"] = 90.0 # cpu.temp_c danger
|
||||
for fs in snap["filesystems"]:
|
||||
fs["used_pct"] = 80.0 # disk.used_pct warn
|
||||
snap["gpus"][0]["busy_pct"] = 95.0 # gpu.busy_pct danger
|
||||
snap["gpus"][0]["gtt_used_bytes"] = 65e9 # whole-pool usage well past danger
|
||||
payload = parse_host_snapshot("driveripper", snap, make_collector(), now=1_000.0).payload
|
||||
assert payload["memory"]["severity"]["used_pct"] == "fail"
|
||||
assert payload["cpu"]["severity"]["temp_c"] == "fail"
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] == "warn"
|
||||
assert payload["gpus"][0]["severity"]["busy_pct"] == "fail"
|
||||
assert payload["gpus"][0]["severity"]["mem_used_pct"] == "fail"
|
||||
|
||||
|
||||
def test_a_value_the_agent_did_not_report_gets_no_colour(agent_snapshot) -> None:
|
||||
"""Unknown is not ok: `severity: null` renders `n/a`, a green pass would be a lie."""
|
||||
snap = copy.deepcopy(agent_snapshot)
|
||||
snap["cpu_temperature_celsius"] = None
|
||||
snap["gpus"][0]["busy_pct"] = None
|
||||
snap["filesystems"] = []
|
||||
payload = parse_host_snapshot("driveripper", snap, make_collector(), now=1_000.0).payload
|
||||
assert payload["cpu"]["severity"]["temp_c"] is None
|
||||
assert payload["cpu"]["severity"]["usage_pct"] is None, "one scrape is not a rate"
|
||||
assert payload["gpus"][0]["severity"]["busy_pct"] is None
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] is None
|
||||
# the numbers themselves are unchanged by the verdict sitting next to them
|
||||
assert payload["memory"]["used_pct"] == pytest.approx(
|
||||
snap["memory"]["used_bytes"] / snap["memory"]["total_bytes"] * 100)
|
||||
|
||||
|
||||
def test_attaching_severity_renamed_or_removed_nothing(agent_snapshot) -> None:
|
||||
"""Additive-only anchor, at the source: the flat fields the pages read are all still there."""
|
||||
payload = parse_host_snapshot("driveripper", agent_snapshot, make_collector(),
|
||||
now=1_000.0).payload
|
||||
expected = {
|
||||
"cpu": {"usage_pct", "modes_pct", "load1", "load5", "load15", "cores",
|
||||
"core_usage_pct", "freq_mhz", "freq_max_mhz", "context_switches_per_s",
|
||||
"forks_per_s", "procs_running", "procs_blocked", "temp_c"},
|
||||
"memory": {"total_bytes", "used_bytes", "available_bytes", "used_pct", "cached_bytes",
|
||||
"buffers_bytes", "swap_total_bytes", "swap_used_bytes", "swap_used_pct",
|
||||
"committed_bytes"},
|
||||
"disks": {"filesystems", "worst_used_pct", "worst_mount", "read_bytes_per_s",
|
||||
"write_bytes_per_s", "reads_per_s", "writes_per_s", "util_pct", "devices"},
|
||||
}
|
||||
for section, keys in expected.items():
|
||||
assert keys <= set(payload[section]), f"{section} lost {keys - set(payload[section])}"
|
||||
for gpu in payload["gpus"]:
|
||||
assert {"busy_pct", "mem_used_pct"} <= set(gpu)
|
||||
|
||||
|
||||
def test_fleet_summary_of_nothing_is_empty_not_zero() -> None:
|
||||
fleet = fleet_summary([])
|
||||
assert fleet["hosts_total"] == 0
|
||||
|
||||
@@ -100,6 +100,35 @@ def test_absent_gufo_metrics_are_none_with_no_source() -> None:
|
||||
assert metrics.get("spec_accept_rate", {}).get("value") is None
|
||||
|
||||
|
||||
# ---- severity: KV pressure arrives with a verdict ------------------------- #
|
||||
|
||||
def test_gufo_kv_cache_cell_keeps_provenance_and_gains_severity(metrics_text) -> None:
|
||||
metrics = parse_gufo_metrics(metrics_text("gufo"))
|
||||
cell = metrics["kv_cache_usage_ratio"]
|
||||
assert cell["value"] == 0.0
|
||||
assert cell["source"] == "llamacpp:kv_cache_usage_ratio"
|
||||
assert cell["derived"] is False, "the verdict is additive; the cell still says what it was"
|
||||
assert cell["severity"] == "ok"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ratio,expected", [
|
||||
("0.74", "ok"), ("0.75", "warn"), ("0.89", "warn"), ("0.9", "fail"), ("1.0", "fail"),
|
||||
])
|
||||
def test_kv_ratio_is_classified_on_a_percent_scale(metrics_text, ratio, expected) -> None:
|
||||
"""0.9 of a KV cache is 90 %, not 0.9 % — classifying the raw ratio is a permanent green."""
|
||||
text = re.sub(r"^(llamacpp:kv_cache_usage_ratio) \S+$", rf"\g<1> {ratio}",
|
||||
metrics_text("gufo"), count=1, flags=re.M)
|
||||
metrics = parse_gufo_metrics(text)
|
||||
assert metrics["kv_cache_usage_ratio"]["severity"] == expected
|
||||
assert metrics["kv_cache_usage_ratio"]["value"] == pytest.approx(float(ratio)), \
|
||||
"the scale is for the verdict only; the published value is untouched"
|
||||
|
||||
|
||||
def test_absent_kv_cell_gets_no_verdict() -> None:
|
||||
metrics = parse_gufo_metrics("# nothing here\n")
|
||||
assert metrics["kv_cache_usage_ratio"]["severity"] is None
|
||||
|
||||
|
||||
# ---- vLLM ----------------------------------------------------------------- #
|
||||
|
||||
def test_vllm_counters_and_gauges(metrics_text) -> None:
|
||||
@@ -112,6 +141,23 @@ def test_vllm_counters_and_gauges(metrics_text) -> None:
|
||||
assert metrics["num_preemptions_total"]["value"] is not None
|
||||
|
||||
|
||||
def test_vllm_kv_cache_cell_carries_a_severity(metrics_text) -> None:
|
||||
assert parse_vllm_metrics(metrics_text("vllm"))["gpu_cache_usage"]["severity"] == "ok"
|
||||
pressed = bump_metric(metrics_text("vllm"), "vllm:kv_cache_usage_perc", 0.95)
|
||||
cell = parse_vllm_metrics(pressed)["gpu_cache_usage"]
|
||||
assert cell["value"] == pytest.approx(0.95)
|
||||
assert cell["severity"] == "fail"
|
||||
assert cell["source"] == "vllm:kv_cache_usage_perc"
|
||||
|
||||
|
||||
def test_vllm_absent_kv_cell_gets_no_verdict(metrics_text) -> None:
|
||||
stripped = "\n".join(line for line in metrics_text("vllm").splitlines()
|
||||
if "kv_cache_usage" not in line and "gpu_cache_usage" not in line)
|
||||
metrics = parse_vllm_metrics(stripped)
|
||||
assert metrics["gpu_cache_usage"]["value"] is None
|
||||
assert metrics["gpu_cache_usage"]["severity"] is None
|
||||
|
||||
|
||||
def test_vllm_latency_histograms(metrics_text) -> None:
|
||||
metrics = parse_vllm_metrics(metrics_text("vllm"))
|
||||
ttft = metrics["time_to_first_token_seconds"]
|
||||
|
||||
@@ -435,6 +435,61 @@ def test_payload_carries_numbers_health_and_what_the_driver_offers() -> None:
|
||||
assert payload["extra"] or payload["status"] == "OL CHRG"
|
||||
|
||||
|
||||
def test_payload_severity_colours_the_hero_battery_numbers() -> None:
|
||||
"""charge / load / runtime carry the verdict so the page paints a class, not a threshold."""
|
||||
payload = nut.payload(reading(), now=1_767_225_600.0) # charge 100, runtime 3720 s, load 34 %
|
||||
assert payload["severity"] == {"charge_pct": "ok", "load_pct": "ok", "battery_runtime": "ok"}
|
||||
# the verdict sits beside the numbers; it does not replace or rename any of them
|
||||
assert payload["numbers"]["charge_pct"] == 100.0
|
||||
assert payload["numbers"]["battery_runtime"] == 3720.0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("overrides,expected", [
|
||||
({"battery.charge": "15"}, {"charge_pct": "fail"}),
|
||||
({"battery.charge": "50"}, {"charge_pct": "warn"}), # reaching warn trips it
|
||||
({"battery.charge": "50.1"}, {"charge_pct": "ok"}),
|
||||
({"battery.runtime": "240"}, {"battery_runtime": "fail"}),
|
||||
({"battery.runtime": "600"}, {"battery_runtime": "warn"}),
|
||||
({"ups.load": "95"}, {"load_pct": "fail"}),
|
||||
({"ups.load": "70"}, {"load_pct": "warn"}),
|
||||
])
|
||||
def test_severity_trips_the_ratified_ups_thresholds(overrides, expected) -> None:
|
||||
payload = nut.payload(reading(**overrides), now=1_767_225_600.0)
|
||||
for key, verdict in expected.items():
|
||||
assert payload["severity"][key] == verdict, f"{key} with {overrides}"
|
||||
|
||||
|
||||
def test_severity_is_null_when_the_driver_publishes_nothing() -> None:
|
||||
"""Neither real UPS reports a battery temperature and either can drop `battery.runtime`;
|
||||
an absent number gets no colour, and the health check keeps saying `unknown`."""
|
||||
values = {key: value for key, value in VAR_ROWS.items()
|
||||
if key not in ("battery.runtime", "battery.temperature")}
|
||||
payload = nut.payload(nut.UpsReading(name="apc", vars=values, commands=[]),
|
||||
now=1_767_225_600.0)
|
||||
assert payload["numbers"]["battery_runtime"] is None
|
||||
assert payload["severity"]["battery_runtime"] is None
|
||||
assert payload["numbers"]["battery_temperature"] is None
|
||||
temp = next(c for c in payload["health"]["checks"] if c["name"] == "Battery temperature")
|
||||
assert temp["status"] == "unknown", "severity is additive: assess() keeps its own judgement"
|
||||
|
||||
|
||||
def test_severity_does_not_touch_health_checks_or_numbers() -> None:
|
||||
ups = reading()
|
||||
payload = nut.payload(ups, now=1_767_225_600.0)
|
||||
assert all(check["status"] in ("ok", "warn", "fail", "unknown")
|
||||
for check in payload["health"]["checks"])
|
||||
assert payload["health"] == nut.assess(ups, now=1_767_225_600.0), \
|
||||
"the condition verdict and the value verdict must not contaminate each other"
|
||||
|
||||
|
||||
async def test_simulated_ups_cards_carry_severity_too() -> None:
|
||||
"""The UPS page is developable without a battery, and still gets real verdicts."""
|
||||
for ups in await NutSimulator().read():
|
||||
card = nut.payload(ups, now=1_767_225_600.0)
|
||||
assert set(card["severity"]) == {"charge_pct", "load_pct", "battery_runtime"}
|
||||
assert card["severity"]["charge_pct"] == "ok", "the sim starts at float charge"
|
||||
|
||||
|
||||
def test_history_samples_only_include_real_numbers() -> None:
|
||||
values = {key: value for key, value in VAR_ROWS.items() if key != "battery.runtime"}
|
||||
samples = nut.history_samples(nut.UpsReading(name="cyberpower", vars=values, commands=[]))
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
"""The single threshold table and its classifier.
|
||||
|
||||
Colour is a judgement, and a judgement made in four places drifts into four meanings.
|
||||
These tests pin the boundaries (reaching a threshold trips it, in both directions), the
|
||||
"unknown is not ok" rule, and the shape of the two payload helpers that put `severity`
|
||||
next to a value.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
|
||||
import pytest
|
||||
|
||||
from dashboards import severity
|
||||
from dashboards.severity import FAIL, INVERTED, OK, THRESHOLDS, WARN, attach, classify, with_severity
|
||||
|
||||
# The ratified table (chat, 2026-10-16). Edit a number here and in THRESHOLDS together —
|
||||
# that coupling is the point: thresholds are a contract with the pages, not a tuning knob.
|
||||
RATIFIED = {
|
||||
"cpu.usage_pct": (75.0, 90.0),
|
||||
"mem.used_pct": (80.0, 92.0),
|
||||
"disk.used_pct": (80.0, 90.0),
|
||||
"cpu.temp_c": (75.0, 90.0),
|
||||
"gpu.busy_pct": (85.0, 95.0),
|
||||
"gpu.mem_used_pct": (85.0, 95.0),
|
||||
"llm.kv_cache_usage_pct": (75.0, 90.0),
|
||||
"ups.charge_pct": (50.0, 20.0),
|
||||
"ups.battery_runtime": (600.0, 300.0),
|
||||
"ups.load_pct": (70.0, 90.0),
|
||||
}
|
||||
|
||||
|
||||
def _samples(metric: str) -> list[tuple[float, str]]:
|
||||
"""(value, expected) walking across the boundaries of one metric, in its own direction."""
|
||||
warn, danger = THRESHOLDS[metric]
|
||||
middle = (warn + danger) / 2
|
||||
if metric in INVERTED: # lower is worse: reaching warn trips it, going below danger fails
|
||||
step = 0.1
|
||||
return [
|
||||
(warn + step, OK),
|
||||
(warn, WARN),
|
||||
(middle, WARN),
|
||||
(danger, FAIL),
|
||||
(danger - 1.0, FAIL),
|
||||
]
|
||||
return [
|
||||
(warn - 0.1, OK),
|
||||
(warn, WARN),
|
||||
(middle, WARN),
|
||||
(danger, FAIL),
|
||||
(danger + 1.0, FAIL),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("metric", sorted(THRESHOLDS))
|
||||
def test_boundaries_hold_for_every_metric_in_the_table(metric: str) -> None:
|
||||
for value, expected in _samples(metric):
|
||||
assert classify(metric, value) == expected, f"{metric}={value} should be {expected}"
|
||||
|
||||
|
||||
def test_completion_criteria_examples() -> None:
|
||||
assert classify("cpu.usage_pct", 95) == "fail"
|
||||
assert classify("cpu.usage_pct", 60) == "ok"
|
||||
assert classify("cpu.usage_pct", None) is None
|
||||
assert classify("ups.charge_pct", 15) == "fail"
|
||||
|
||||
|
||||
def test_warn_edge_is_inclusive_and_just_below_is_still_good() -> None:
|
||||
assert classify("cpu.usage_pct", 74.9) == "ok"
|
||||
assert classify("cpu.usage_pct", 75) == "warn"
|
||||
assert classify("cpu.usage_pct", 75.0001) == "warn"
|
||||
assert classify("cpu.usage_pct", 89.9) == "warn"
|
||||
assert classify("cpu.usage_pct", 90) == "fail"
|
||||
|
||||
|
||||
def test_inverted_metrics_read_the_other_way() -> None:
|
||||
"""A flat battery and a flat runtime are the bad end, not the good one."""
|
||||
assert classify("ups.charge_pct", 100) == "ok"
|
||||
assert classify("ups.charge_pct", 50) == "warn"
|
||||
assert classify("ups.charge_pct", 19.9) == "fail"
|
||||
assert classify("ups.battery_runtime", 3600) == "ok"
|
||||
assert classify("ups.battery_runtime", 600) == "warn"
|
||||
assert classify("ups.battery_runtime", 120) == "fail"
|
||||
|
||||
|
||||
def test_a_metric_is_only_in_one_direction() -> None:
|
||||
"""Sanity: bounds are ordered for their direction, so an edit cannot invert a metric."""
|
||||
for metric, (warn, danger) in THRESHOLDS.items():
|
||||
assert math.isfinite(warn) and math.isfinite(danger), metric
|
||||
if metric in INVERTED:
|
||||
assert warn > danger, f"{metric} is inverted: warn must sit above danger"
|
||||
else:
|
||||
assert warn < danger, f"{metric} is not inverted: warn must sit below danger"
|
||||
|
||||
|
||||
def test_inverted_is_a_subset_of_the_table() -> None:
|
||||
assert set(THRESHOLDS) >= INVERTED, "an inverted metric with no thresholds has no colour"
|
||||
|
||||
|
||||
def test_the_ratified_numbers_are_the_ones_in_the_table() -> None:
|
||||
assert THRESHOLDS == RATIFIED
|
||||
|
||||
|
||||
def test_unknown_metric_is_none_rather_than_an_exception() -> None:
|
||||
assert classify("gpu.vram_used_pct", 99) is None
|
||||
assert classify("nope.nope", 0) is None
|
||||
assert classify("", 100) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("metric", sorted(THRESHOLDS))
|
||||
def test_none_is_never_ok(metric: str) -> None:
|
||||
assert classify(metric, None) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("metric", sorted(THRESHOLDS))
|
||||
@pytest.mark.parametrize("value", [float("nan"), float("inf"), float("-inf")])
|
||||
def test_non_finite_values_are_treated_as_missing(metric: str, value: float) -> None:
|
||||
assert classify(metric, value) is None
|
||||
|
||||
|
||||
def test_per_device_streams_inherit_their_base_metric() -> None:
|
||||
assert classify("disk.used_pct@/srv", 95) == "fail"
|
||||
assert classify("disk.used_pct@/srv", 10) == "ok"
|
||||
assert classify("gpu.busy_pct@0", 84) == "ok"
|
||||
assert classify("gpu.busy_pct@0", 90) == "warn"
|
||||
assert classify("gpu.busy_pct@1", 95) == "fail"
|
||||
assert classify("cpu.usage_pct@core7", 80) == "warn"
|
||||
|
||||
|
||||
def test_vocabulary_is_exactly_ok_warn_fail_or_none() -> None:
|
||||
seen = {classify(metric, value) for metric in THRESHOLDS for value, _ in _samples(metric)}
|
||||
assert seen == {OK, WARN, FAIL}
|
||||
inverted = severity.INVERTED
|
||||
assert inverted == {"ups.charge_pct", "ups.battery_runtime"}
|
||||
|
||||
|
||||
def test_attach_pairs_a_plain_number_with_its_verdict() -> None:
|
||||
assert attach("cpu.usage_pct", 95) == {"value": 95.0, "severity": "fail"}
|
||||
assert attach("mem.used_pct", 10) == {"value": 10.0, "severity": "ok"}
|
||||
assert attach("ups.load_pct", 70) == {"value": 70.0, "severity": "warn"}
|
||||
|
||||
|
||||
def test_attach_keeps_the_value_when_it_has_no_opinion() -> None:
|
||||
assert attach("cpu.usage_pct", None) == {"value": None, "severity": None}
|
||||
assert attach("telemetry.mystery", 42) == {"value": 42.0, "severity": None}
|
||||
|
||||
|
||||
def test_attach_never_puts_a_non_finite_number_in_a_payload() -> None:
|
||||
for value in (float("nan"), float("inf"), float("-inf")):
|
||||
assert attach("cpu.usage_pct", value) == {"value": None, "severity": None}
|
||||
assert json.dumps(attach("cpu.usage_pct", float("nan"))) # would raise otherwise
|
||||
|
||||
|
||||
def test_with_severity_is_additive_on_a_llm_shaped_cell() -> None:
|
||||
cell = {"value": 82.0, "source": "llamacpp:kv_cache_usage_ratio", "derived": False}
|
||||
out = with_severity("llm.kv_cache_usage_pct", cell)
|
||||
assert out["source"] == "llamacpp:kv_cache_usage_ratio"
|
||||
assert out["derived"] is False
|
||||
assert out["value"] == 82.0
|
||||
assert out["severity"] == "warn"
|
||||
assert cell == {"value": 82.0, "source": "llamacpp:kv_cache_usage_ratio", "derived": False}, \
|
||||
"the caller's mapping must not be mutated"
|
||||
|
||||
|
||||
def test_with_severity_handles_missing_and_non_numeric_cells() -> None:
|
||||
# purely additive: a missing cell gains `severity` and nothing else
|
||||
assert with_severity("llm.kv_cache_usage_pct", None) == {"severity": None}
|
||||
assert with_severity("llm.kv_cache_usage_pct", {"value": None}) == {"value": None, "severity": None}
|
||||
assert with_severity("llm.decode_tps", {"value": "n/a", "source": None}) == {
|
||||
"value": "n/a", "source": None, "severity": None,
|
||||
}
|
||||
# a value the table has no thresholds for still keeps its other keys
|
||||
out = with_severity("llm.ttft_seconds", {"value": 1.2, "source": "computed"})
|
||||
assert out == {"value": 1.2, "source": "computed", "severity": None}
|
||||
Reference in New Issue
Block a user