diff --git a/.agents/phases/todo/01_severity_language/00_phase.md b/.agents/phases/complete/01_severity_language/00_phase.md similarity index 100% rename from .agents/phases/todo/01_severity_language/00_phase.md rename to .agents/phases/complete/01_severity_language/00_phase.md diff --git a/.agents/phases/todo/01_severity_language/01_severity_module.md b/.agents/phases/complete/01_severity_language/01_severity_module.md similarity index 100% rename from .agents/phases/todo/01_severity_language/01_severity_module.md rename to .agents/phases/complete/01_severity_language/01_severity_module.md diff --git a/.agents/phases/todo/01_severity_language/02_severity_in_payloads.md b/.agents/phases/complete/01_severity_language/02_severity_in_payloads.md similarity index 100% rename from .agents/phases/todo/01_severity_language/02_severity_in_payloads.md rename to .agents/phases/complete/01_severity_language/02_severity_in_payloads.md diff --git a/.agents/phases/todo/01_severity_language/03_hero_number_style.md b/.agents/phases/complete/01_severity_language/03_hero_number_style.md similarity index 100% rename from .agents/phases/todo/01_severity_language/03_hero_number_style.md rename to .agents/phases/complete/01_severity_language/03_hero_number_style.md diff --git a/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.err b/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.md b/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.md new file mode 100644 index 0000000..ff4bd7a --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.md @@ -0,0 +1,13 @@ +**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail). + +**Verification (all green)** +- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean +- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty +- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓ +- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png` + +**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them. + +**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05. + +**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`). diff --git a/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.md.status b/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.validate b/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.validate new file mode 100644 index 0000000..b2802f7 --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__00_phase.a1.validate @@ -0,0 +1,217 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 19%] +.....................................................................sss [ 39%] +sss..................................................................... [ 59%] +........................................................................ [ 79%] +........................................................................ [ 99%] +../usr/lib64/python3.14/collections/__init__.py:453: ResourceWarning: unclosed database in + @classmethod +ResourceWarning: Enable tracemalloc to get the object allocation traceback +/usr/lib64/python3.14/collections/__init__.py:455: ResourceWarning: unclosed database in + result = tuple_new(cls, iterable) +ResourceWarning: Enable tracemalloc to get the object allocation traceback + [100%]/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/_pytest/unraisableexception.py:33: ResourceWarning: unclosed database in + gc.collect() +ResourceWarning: Enable tracemalloc to get the object allocation traceback +/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/_pytest/unraisableexception.py:33: ResourceWarning: unclosed database in + gc.collect() +ResourceWarning: Enable tracemalloc to get the object allocation traceback +/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/_pytest/unraisableexception.py:33: ResourceWarning: unclosed database in + gc.collect() +ResourceWarning: Enable tracemalloc to get the object allocation traceback +/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/_pytest/unraisableexception.py:33: ResourceWarning: unclosed database in + gc.collect() +ResourceWarning: Enable tracemalloc to get the object allocation traceback + +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +tests/test_api.py::test_unconfigured_source_is_404_configured_false[/api/ups/snapshot] + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/telemetry/_runtime.py:213: ResourceWarning: unclosed database in + async def wrapped_receive() -> Message: + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_api.py::test_unconfigured_source_is_404_configured_false[/api/ups/snapshot] + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/telemetry/_runtime.py:213: ResourceWarning: unclosed database in + async def wrapped_receive() -> Message: + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_api.py::test_unconfigured_source_is_404_configured_false[/api/ups/snapshot] + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/telemetry/_runtime.py:213: ResourceWarning: unclosed database in + async def wrapped_receive() -> Message: + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_api.py::test_summary_still_answers_with_no_sources + /usr/lib64/python3.14/urllib/request.py:1888: ResourceWarning: unclosed database in + if len(name) > 5 and name[-6] == "_" and name[-5:].lower() == "proxy": + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_api.py::test_agent_tokens_never_reach_the_browser + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/utils.py:43: ResourceWarning: unclosed database in + def get_path_param_names(path: str) -> set[str]: + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +tests/test_main.py::test_main_serves_the_assembled_app + :0: ResourceWarning: unclosed database in + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 28 +src/dashboards/api.py 190 28 85% 59-60, 62, 280-286, 303-311, 318-322, 406, 411-412, 418-429 +src/dashboards/app.py 109 7 94% 50-51, 56, 58, 60, 97, 131 +src/dashboards/collectors.py 140 9 94% 78-79, 92-93, 97-98, 115, 232, 236 +src/dashboards/config.py 110 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 139 1 99% 226 +src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164 +src/dashboards/rates.py 37 1 97% 51 +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/host.py 199 2 99% 122, 147 +src/dashboards/sources/llm.py 150 1 99% 226 +src/dashboards/sources/nut.py 367 22 94% 118, 132, 138, 150, 156, 168, 176, 181, 252-253, 255, 414, 458, 470, 505, 538, 542, 595, 599-600, 605, 617 +src/dashboards/sources/nut_sim.py 120 21 82% 129, 155-173, 214 +src/dashboards/store.py 178 3 98% 204, 243-244 +------------------------------------------------------------------ +TOTAL 1924 101 95% +Required test coverage of 94% reached. Total coverage: 94.75% +356 passed, 6 skipped, 31 warnings in 39.85s +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == diff --git a/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.err b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.md b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.md new file mode 100644 index 0000000..e4cdb7f --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.md @@ -0,0 +1,17 @@ +**Implemented** +- `src/dashboards/severity.py` (new leaf module): `THRESHOLDS` (the 10 ratified rows), `INVERTED` (`ups.charge_pct`, `ups.battery_runtime`), `classify()` → `ok|warn|fail|None`, `attach()` for bare numbers, `with_severity()` for `{value, source, derived}` cells; docstring states "unknown is not ok" + "a threshold exists nowhere else" and that `_pct` metrics are 0-100. +- `tests/test_severity.py` (new, 64 tests): every metric walked across both boundaries, warn/danger edges inclusive, both directions for inverted metrics, table-sanity test (bound ordering per direction, `INVERTED ⊆ THRESHOLDS`), pinned ratified table, unknown metric / `None` / NaN / ±inf → `None`, `@stream` inheritance (`disk.used_pct@/srv`, `gpu.busy_pct@0`), attach/with_severity shapes incl. no caller-mutation and JSON-safety. +- No page, API, CSS, or payload code touched — severity is not consumed yet. No live infra touched. + +**Verification** +- `uv run pytest tests/test_severity.py -v` → 64 passed +- `uv run pytest` → **322 passed, 6 skipped** (baseline 258+6, no regressions); `uv run pytest --cov` → `severity.py` **100%**, total **94%** +- `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors +- `uv run python -c "from dashboards.severity import classify as c; print(c('cpu.usage_pct',95), c('cpu.usage_pct',60), c('cpu.usage_pct',None), c('ups.charge_pct',15))"` → `fail ok None fail` +- Deps unchanged → `uv pip list --outdated` not applicable. + +**Docs**: `AGENTS.md` — one `severity.py` line added to the module-responsibility list. No README impact (no new setup/command/API surface yet). + +**Decisions**: reaching a threshold trips it in both directions (`cpu.usage_pct` 75 → warn, 90 → fail; `ups.charge_pct` 50 → warn, 20 → fail); `metric@stream` classifies with its base metric's row; `attach()` normalises non-finite values to `None` so payloads stay valid JSON, `with_severity()` is strictly additive (a missing cell gains only `severity`). No screenshot taken — this task changes no UI; the phase's render-unchanged screenshot belongs with task 03. + +**Next pending task**: `.agents/phases/todo/01_severity_language/02_severity_in_payloads.md` diff --git a/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.md.status b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.validate b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.validate new file mode 100644 index 0000000..df406d4 --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a1.validate @@ -0,0 +1,16 @@ +........................................................................ [ 21%] +.......................................................ssssss........... [ 43%] +........................................................................ [ 65%] +........................................................................ [ 87%] +........................................ [100%] +=============================== warnings summary =============================== +../../../.local/lib/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/.local/lib/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +All checks passed! +repo hygiene: TRACKED files look like log drops / key material (untrack with 'git rm --cached ' and add a .gitignore rule): + .env.dev.example +.env.example +validation FAILED (see output above) diff --git a/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.err b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.md b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.md new file mode 100644 index 0000000..4e4c117 --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.md @@ -0,0 +1,17 @@ +**Fixed (rejection cause)** +- `validate.sh` gate #2 flagged the repo's tracked env templates `.env.example` / `.env.dev.example` — they are tracked **by design** (`.gitignore` ignores `.env.*` and un-ignores exactly those two so a clone has documented setup; `cp .env.example .env` is in README/AGENTS.md). Untracking them would break the documented workflow, so I adapted the gate instead (its header says to adapt it per project): + - narrowly allow-list `(^|/)\.env[^/]*\.(example|template|sample)$` — real `.env`, `.env.dev`, `*.pem/key/crt`, `*.log`, `*.jsonl` still fail (verified against a synthetic file list). + - added gate **2b** so the allow-list can't smuggle a secret: every tracked `*.example` template's `*_PASSWORD|*_TOKEN|*_SECRET|*_KEY` assignment must be empty or a placeholder word. Passes on both templates (`replace-me-on-the-host`, `SECRET`, empty), fails on a planted real-looking value. +- No test/lint/code defects found in this or prior work; `severity.py` + tests unchanged from the accepted implementation. + +**Verification** +- `./.agents/validate.sh` → **rc=0** (pytest 322 passed / 6 skipped; `ruff check .` All checks passed; hygiene gates silent) +- `uv run pytest` → **322 passed, 6 skipped**; `uv run pytest --cov` → `severity.py` **100%**, total **94%** +- `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors +- `classify` criteria one-liner → `fail ok None fail` + +**Docs**: `AGENTS.md` — `severity.py` line in the module-responsibility list (from the prior pass, still accurate). Gate adaptation is documented inline in `.agents/validate.sh`; no README change needed (setup steps unchanged). + +**Notable decisions**: kept the templates tracked rather than untracking, because the hygiene gate's intent (no key material / log drops) is preserved and strengthened by gate 2b; no `git rm --cached` used (not sanctioned by this task file). + +**Next pending task**: `.agents/phases/todo/01_severity_language/02_severity_in_payloads.md` diff --git a/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.md.status b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.validate b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.validate new file mode 100644 index 0000000..ba5ed3c --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__01_severity_module.a2.validate @@ -0,0 +1,12 @@ +........................................................................ [ 21%] +.......................................................ssssss........... [ 43%] +........................................................................ [ 65%] +........................................................................ [ 87%] +........................................ [100%] +=============================== warnings summary =============================== +../../../.local/lib/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/.local/lib/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +All checks passed! diff --git a/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.err b/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.md b/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.md new file mode 100644 index 0000000..aa7f2e1 --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.md @@ -0,0 +1,13 @@ +**Task 02 — Severity in the payloads: done** + +- `sources/host.py`: added `severity` sub-dicts — `cpu.severity{usage_pct,temp_c}`, `memory.severity{used_pct}`, `disks.severity{worst_used_pct}`, `gpus[].severity{busy_pct,mem_used_pct}`; all existing keys untouched. +- `sources/nut.py`: each UPS card now has `severity{charge_pct,load_pct,battery_runtime}`; `health`/`checks[].status`/`numbers` unchanged. +- `sources/llm.py`: `attach_cell_severity()` puts `severity` inside the KV cells (`kv_cache_usage_ratio`, `gpu_cache_usage`) via `severity.with_severity`; `value/source/derived` intact. +- `api.py` `/api/summary`: `severity` per host row, per UPS row, and on the fleet aggregate (`/api/health`, `/api/config` untouched). +- `severity.with_severity` gained an optional `scale` (ratio→percent, **verdict only**) so 0.9 KV pressure is `fail`, not a permanent green; task-01 tests unchanged. +- Regression caught and fixed by the new tests: a local-name collision in `parse_host_snapshot` had overwritten `memory.used_pct` with the GPU pool %; test asserts `memory.used_pct == used/total`. +- Tests: `uv run pytest --cov` → **349 passed, 6 skipped, TOTAL 95%** (severity.py 100%, api.py/host.py/nut.py/llm.py gaps all pre-existing route bodies); `ruff check src tests agent` clean; `pyright` 0 errors; `uv pip list --outdated` empty. +- Live: `uv run pytest -m live --live` → **6 passed**; dev server restarted, curl confirms severity on `/api/system/snapshot` (gateway `temp_c: null`), `/api/ups/snapshot` (apc + cyberpower ok; `battery_temperature` null with health check still `unknown`), `/api/summary`, `/api/llm/snapshot`. +- Docs: AGENTS.md severity bullet now states the additive payload contract; test count 258→349; README architecture gained a `severity.py` bullet. +- Note: `deploy/nut-sudoers` was modified in the working tree by something outside this task (grant line, mtime mid-session) — I left it untouched for the harness to judge. +- Next pending task: `01_severity_language/03_hero_number_style.md`. diff --git a/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.md.status b/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.validate b/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.validate new file mode 100644 index 0000000..e5f2bfd --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__02_severity_in_payloads.a1.validate @@ -0,0 +1,12 @@ +........................................................................ [ 20%] +..............................................................ssssss.... [ 40%] +........................................................................ [ 60%] +........................................................................ [ 81%] +................................................................... [100%] +=============================== warnings summary =============================== +../../../.local/lib/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/.local/lib/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +All checks passed! diff --git a/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.err b/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.md b/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.md new file mode 100644 index 0000000..694094a --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.md @@ -0,0 +1,10 @@ +**Task 03 — Hero-number style and the JS side of severity: complete** + +- `dash.css`: new `hero numbers` block (`.hero`, `.hero .label/.value/.unit/.sub`, `.hero.ok|.warn|.fail .value`, `.hero-bar` + `.hero-bar.ok .bar > i`) built only from existing tokens; 2.4rem tabular-numeral value, faint `n/a` placed after the verdict rules so an unmeasured number is never coloured. +- `app.js`: `Dash.hero(el,{label,value,unit,digits,severity,sub,bar})` renders/patches a hero in place (`n/a` + **no** severity class when the value is null/non-finite, colour class dropped on the transition); `Dash.severityFrom(cell|string|null)` → `ok|warn|fail|''`; `severityFor` kept, documented as the fallback with "do not add a call site"; both new helpers exported. Pages untouched (phases 02–05 convert them). +- Tests (7 new, `tests/test_api.py`): served CSS has every hero rule + the shared verdict tokens; no `.sev-*`/synonym classes in any asset or page (comments stripped first); `hero()` body must call `severityFrom` and must not contain `severityFor`; no stylesheet rule uses an undeclared `var(--token)`; `severityFor` still exported for the four pages. +- Live browser verification (real Chromium against `:8090`, live fleet/engines/upsd): `severityFrom` cell/bare/null/junk → `fail`/`warn`/`''`/`''`; hero classes `hero ok|hero warn|hero fail|hero`, fail computed colour `rgb(207,34,46)` at 38.4px (2.4rem), bar width+class from the verdict, null value = `n/a` with `is-null`; all four pages render unchanged. Screenshots: `data/logs/hero-number-preview.png`, `hero-task-{index,system,llm,ups}.png`. +- Green: `uv run pytest` → **356 passed, 6 skipped**; `uv run pytest --cov` → TOTAL **95%**, `severity.py` 100%; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` empty; `curl -s localhost:8090/static/dash.css | grep -c "\.hero"` → 17. +- Docs: AGENTS.md — `web/` responsibility names `Dash.hero`/`Dash.severityFrom`, UI conventions describe `.hero`/`.hero-bar` and the no-`.sev-*` rule, counts refreshed (356 tests, ~95%); README — `web/` bullet documents the hero helper and "no verdict → `n/a`, no colour". +- Decision: `hero()`'s optional `bar` colours itself only from the payload verdict (never `severityFor`), and `.hero-bar.ok .bar > i` lets hand-written markup inherit the verdict from the hero, so no component can reintroduce a JS threshold. +- Next pending task: `02_overview_fleet_and_alerts/01_alerts_builder.md` (phase 01 tasks all done). diff --git a/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.md.status b/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.validate b/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.validate new file mode 100644 index 0000000..a651841 --- /dev/null +++ b/.agents/reports/01_severity_language/01_severity_language__03_hero_number_style.a1.validate @@ -0,0 +1,13 @@ +........................................................................ [ 19%] +.....................................................................sss [ 39%] +sss..................................................................... [ 59%] +........................................................................ [ 79%] +........................................................................ [ 99%] +.. [100%] +=============================== warnings summary =============================== +../../../.local/lib/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/.local/lib/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +All checks passed! diff --git a/.env.dev.example b/.env.dev.example index 1e97b33..50c187b 100644 --- a/.env.dev.example +++ b/.env.dev.example @@ -10,7 +10,9 @@ # scripts/dev-tunnels.sh forwards each one to 127.0.0.1: so the dashboard on # the laptop reads live fleet data with nothing listening on the hosts. export DEV_TUNNELS="9121:driveripper-ops:9120 9122:deskwork-ops:9120 9123:gateway-ops:9120" -export DEV_FORWARDS="3494:driveripper-ops:3493" +# The remote port is whatever upsd actually listens on — 3493 is NUT's stock default, and +# this fleet's driveripper answers on 3051, so check before copying the number. +export DEV_FORWARDS="3494:driveripper-ops:3051" # Do not assume upsd is on 3493 — nonstandard ports are common and a refused connection then # reads as "upsd is broken" while the service is fine. Ask the host what actually speaks NUT: @@ -20,8 +22,12 @@ export DEV_FORWARDS="3494:driveripper-ops:3493" # Point the dashboard at those tunnels. export HOST_TARGETS="driveripper=http://127.0.0.1:9121 deskwork=http://127.0.0.1:9122 gateway=http://127.0.0.1:9123" -export GUFO_BASE_URL=http://127.0.0.1:8731 -export VLLM_BASE_URL=http://127.0.0.1:8000 +# The engines are reached directly on their hostnames (nothing tunnels them here). If the +# laptop cannot route to them, add them to DEV_FORWARDS above +# (e.g. `8731:deskwork-ops:8731 8000:driveripper-ops:8000`) and point these at 127.0.0.1. +# The host names below are placeholders (RFC 2606) — use your own. +export GUFO_BASE_URL=http://deskwork.example.net:8731 +export VLLM_BASE_URL=http://driveripper.example.net:8000 # The UPS source: the forwarded upsd above. NUT_PORT is the LOCAL port (3494), not 3493. # Leave NUT_SIMULATE=1 only while upsd is unreachable — it labels every panel SIMULATED. export NUT_HOST=127.0.0.1 diff --git a/AGENTS.md b/AGENTS.md index 86002fc..4e46fff 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ file — plus an overview page that says which of the three you need: | `/` | Is anything wrong, and where do I look first? | all three, in one request | | `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host | | `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo (llama.cpp family) + vLLM `/metrics` | -| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` on 3493 (our own wire-protocol client) | +| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) | History is a first-class feature, not a sparkline afterthought: every number worth graphing is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`. @@ -115,11 +115,25 @@ Responsibilities, one line each: battery. It must never be the default, and every payload it produces says `simulated`. - `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own). - `rates.py` — Δ/window estimator, keyed by resolved series name. +- `severity.py` — the only place thresholds live: `THRESHOLDS`/`INVERTED` per metric, + `classify()` → `ok`/`warn`/`fail`/`None`, plus `attach()`/`with_severity()` to put the verdict + beside a value. No measurement, no verdict: `None` in → `None` out, never a green pass for + an unknown. The verdicts travel **additively** in payloads — `cpu.severity`, + `memory.severity`, `disks.severity`, `gpus[].severity`, each UPS card's `severity` + (charge/load/runtime), `severity` inside rated LLM cells (KV pressure, classified on a + percent scale because the engines publish a 0-1 ratio), and `severity` per host/fleet/UPS + row in `/api/summary`. Pages render the class; they never compare a number to a threshold. - `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning. - `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary. - `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app with an in-memory DB and no targets. -- `web/` — four static pages, one stylesheet, `charts.js`. No framework. +- `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the + colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an + `ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and + `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a + `.hero` big number (plus an optional `.bar`) from that class — it never compares a number + to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for + values the API does not rate; converting its remaining call sites is phases 02–05's job. Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`, `gpu.busy_pct@0`, `llm.decode_tps`, `ups.charge_pct`), `@stream` for per-device series. @@ -141,11 +155,13 @@ That scheme is the history API's key — treat it as a public interface. *Connection refused* and it looks like a dead service — `ss -ltnH` plus `printf 'LIST UPS\n' | nc 127.0.0.1 3051` (answers `BEGIN LIST UPS`) is how that was pinned down. Two UPSes are exported over USB and read live through `DEV_FORWARDS="3494:driveripper-ops:3051"`: - `apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U. + `apc` = APC Back-UPS RS 1500MS, `cyberpower` = CyberPower PR1500LCDRT2U (that is + `ups.description`, the operator's label — the driver's own `identity.model` answers + `GX1500U`, so the description is what identifies the box). Real-hardware quirks the parsers already handle: `battery.mfr.date` is the useful date (`battery.date` is often an unset placeholder like `2001-09-25`), CyberPower reports `battery.mfr.date: CPS` (not a date → `unknown`, never a fake age), neither driver reports - a battery temperature, and `ups.test.result: No test initiated` is the factory state. + a battery temperature, and `ups.test.result: No test initiated` is the factory state on both. - Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the work that is still blocked). @@ -162,16 +178,24 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets ./scripts/dev-server.sh status # per-source ok/DOWN ./scripts/dev-server.sh logs -uv run pytest # hermetic: 258 tests, no network -uv run pytest --cov # coverage gate (~94%) +uv run pytest # hermetic: 356 tests, no network +uv run pytest --cov # coverage gate (94.75%; floor 94%) uv run pytest -m live --live # opt-in: real fleet / engines / upsd uv run ruff check src tests agent && uv run pyright ./scripts/capture-fixtures.sh # re-record ground truth + +bash .agents/validate.sh # the phase gate: the four checks above (pytest + # --cov --cov-fail-under=94, ruff, pyright, plus a + # node --check of web/static) + repo hygiene. + # Always `uv run` — a bare `python3`/`ruff` on PATH + # tests an interpreter nobody deploys. ``` Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`, SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh` -per monitored host (systemd unit, own user, `--token-file`, `ProtectSystem=strict`). +per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/ +`--status`/`--uninstall`; it stores the token in a root-owned 0600 file and hands the agent +`--token-file`, never argv, and runs under `ProtectSystem=strict`). The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other service here. @@ -190,8 +214,10 @@ service here. test on hardware something is plugged into is not a test. - The agent's real-`/proc` test (`tests/test_agent.py`) touches only the box running pytest, so it stays in the default run and is *not* marked `live`. -- Coverage gate: 80% is the floor, ~94% is where we are; keep it there. API tests use - `create_app()` with an in-memory DB, never a real target. +- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces + 94% (`--cov-fail-under`), and ~95% is where we are; keep it there. New/modified Python + still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB, + never a real target. ## UI conventions @@ -201,6 +227,12 @@ dots in the header, window chips (1h…90d) on history pages. A panel that canno why in the panel — the operator should never have to open devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`. +A big number is `Dash.hero()` on `.hero` / `.hero-bar` (`dash.css`): 2.4rem tabular numerals, +muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the payload's +`severity` through the existing `.ok` / `.warn` / `.fail` classes — there is no `.sev-*` +family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS +rule reaches for a token that `:root` does not declare. + ## Known gaps / next work 1. **UPS instant commands are read-only in practice.** Reads work against real upsd (see @@ -208,15 +240,22 @@ unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`. `actions` in upsd.users" because no such user exists yet. Creating one safely needs a *read* of `/etc/ups/upsd.users` first — `sudo tee` can write the file but overwriting it blind would delete the users upsmon already authenticates as, which can take the box down - at the wrong moment. `deploy/nut-sudoers` lists the read rules; once they are in, add - `[dashboards]` with `instcmds` limited to the battery tests, put the credentials in - `.env.dev`, and the buttons come alive. Never grant `actions = -`. + at the wrong moment. `deploy/nut-sudoers` is the installable ruleset that makes that + read (and only a scoped write + reload) possible — it carries the + `ops ALL=(root) NOPASSWD: NUT_READ, NUT_WRITE` grant and is installed with + `sudo install -m 0440 -o root -g root deploy/nut-sudoers /etc/sudoers.d/dashboards-nut`, + but it is **repo-side only so far**: the live UPS cards still answer "NUT_USER / + NUT_PASSWORD must name a user with `actions` in upsd.users" (verified through + `/api/ups/snapshot`). Once it is installed: read `upsd.users`, add `[dashboards]` with + `instcmds` limited to the battery tests, put the credentials in `.env.dev`, and the + buttons come alive. Never grant `actions = -`, and never write the file blind. 2. **GPU coverage is AMD only.** `host_agent.gpus()` reads amdgpu DRM sysfs (`gpu_busy_percent`, `mem_info_vram_*`, hwmon temps, clocks). That covers the current fleet (R9700 + Strix Halo iGPU; gateway has none). An NVIDIA host would report no GPU at all — needs a `nvidia-smi`/NVML or DCGM path, and a fixture from a real card before the field map is written. -3. **No remote yet.** Two commits, no `origin` — the user is providing it. +3. **No remote yet.** No `origin` is configured (`git remote -v` is empty) — the user is + providing it, so every phase so far is local-only history. 4. **UPS history depth.** Events and the numeric series are stored; long-window battery health (capacity fade over months) has the rollups but no dedicated view. @@ -240,6 +279,9 @@ unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`. ## Definition of done (per change) +The first three are exactly what `bash .agents/validate.sh` (the phase gate) checks, so no +phase may be called done on a subset of them. + `uv run pytest` green · coverage not down · `ruff` + `pyright` clean · `uv pip list --outdated` empty if deps moved · dev server restarted and `/api/health` sane · UI change screenshotted in a browser against **live** data · fixtures re-captured if an upstream contract changed · diff --git a/README.md b/README.md index b7472c1..8ff748a 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Three dashboards, one domain, one process, one SQLite file: | `/` | Is anything wrong, and which dashboard do I need? | all of the below, one request | | `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host | | `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) + vLLM `/metrics`, read-only | -| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd` on 3493 (its own wire protocol client) | +| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) | No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at runtime; every asset is served by this app. @@ -39,6 +39,8 @@ for host in ("driveripper-ops", "deskwork-ops", "gateway-ops"): PY ./scripts/dev-tunnels.sh start # 9121/9122/9123 -> each host's agent, # + DEV_FORWARDS -> upsd on 127.0.0.1:3494 + # (the remote port is upsd's real one — 3051 + # on driveripper, not the 3493 default) set -a; . ./.env.dev; set +a uv run dashboards # foreground ./scripts/dev-server.sh restart # or backgrounded, with a pidfile + logs @@ -51,7 +53,9 @@ that already listens there (upsd) — that is how the UPS page reads a real batt ## The host agent `agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`, -`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` as JSON. +`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one +JSON document), `GET /info` (slow-changing identity: OS, kernel, CPU model, GPU list), and +`GET /health` (the one route that never needs the token) as JSON. ```bash python3 host_agent.py --once # print one snapshot, exit @@ -84,15 +88,20 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev | Variable | Meaning | | ------------------------------------- | -------------------------------------------------------------- | | `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) | -| `HOST_INTERVAL_SECONDS` | poll interval for hosts (default 10) | | `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears | | `RATE_WINDOW_SECONDS` | the Δ window behind every measured token rate (default 10) | +| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) | +| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) | +| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others | +| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server | | `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT | | `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) | | `NUT_ALLOW_ACTIONS` | `1` enables the one write endpoint in the app | | `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) | | `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags | | `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) | +| `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer | +| `DEBUGPY` / `DEBUGPY_PORT` | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup | | `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning | A source that is not configured answers `404 {"configured": false}` **without dialling**, @@ -119,8 +128,19 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ── blanks or blocks another source. * `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are pruned, so a year of history is a few hundred MB rather than a disk emergency. +* `severity.py` — the single green/amber/red table (`THRESHOLDS`, plus `INVERTED` for the + metrics where lower is worse). Rated values in the API payloads travel with a sibling + `severity` (`cpu.severity.usage_pct`, `memory.severity`, `disks.severity`, + `gpus[].severity`, a UPS card's `severity.charge_pct`, `severity` inside the rated LLM + cells such as KV pressure, `/api/summary` rows), computed server-side, so every page + colours the same way and a value nobody reported renders `n/a` instead of green. A metric + published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays + what the upstream published. * `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No - framework, no CDN, no bundler. + framework, no CDN, no bundler. A big colour-coded number is `Dash.hero(el, {...})` on the + `.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via + `Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value + with no verdict renders `n/a` with no colour at all. ## Testing diff --git a/deploy/nut-sudoers b/deploy/nut-sudoers index 7e6f52b..b07ad2c 100644 --- a/deploy/nut-sudoers +++ b/deploy/nut-sudoers @@ -45,6 +45,12 @@ Cmnd_Alias NUT_WRITE = \ /usr/bin/systemctl restart nut-server, \ /usr/bin/systemctl reload nut-server +# ---- grants --------------------------------------------------------------------- # +# A Cmnd_Alias grants nothing on its own — this line is the point of the file. +# Scope: read the NUT config/state, add/replace a scoped upsd.users entry, and +# restart/reload the server so upsd re-reads it. Nothing else. +ops ALL=(root) NOPASSWD: NUT_READ, NUT_WRITE + # ---- deliberately NOT granted -------------------------------------------------------- # # /usr/bin/upscmd, /usr/bin/upsrw — the dashboard's gated POST /api/ups/command is the # ONLY command path in this project, by design diff --git a/src/dashboards/api.py b/src/dashboards/api.py index 3566d97..1c89709 100644 --- a/src/dashboards/api.py +++ b/src/dashboards/api.py @@ -21,6 +21,7 @@ from fastapi import APIRouter, Body, Depends, HTTPException, Query, Request from .collectors import NutMonitor from .config import Settings from .poller import Hub +from .severity import classify from .sources import nut from .sources.host import fleet_summary from .sources.llm import fleet_totals @@ -74,6 +75,43 @@ def _history(store: SampleStore, source: str, host: str, metric: str, span: floa } +def _host_severity(host: dict[str, Any]) -> dict[str, str | None]: + """Verdicts for the overview's fleet-table columns, from the one threshold table. + + The row flattens `cpu.usage_pct` into `cpu_usage_pct`; the verdict is looked up under + the row's own key so the page never has to know which section a number came from. + """ + cpu = host.get("cpu") or {} + return { + "cpu_usage_pct": classify("cpu.usage_pct", cpu.get("usage_pct")), + "mem_used_pct": classify("mem.used_pct", (host.get("memory") or {}).get("used_pct")), + "disk_worst_used_pct": classify("disk.used_pct", + (host.get("disks") or {}).get("worst_used_pct")), + "temp_c": classify("cpu.temp_c", cpu.get("temp_c")), + } + + +def _fleet_severity(fleet: dict[str, Any]) -> dict[str, str | None]: + """Fleet aggregates judged by the same rows as the per-host values — no fleet-only + thresholds, so "the fleet is amber" means the same thing as one amber host.""" + return { + "cpu_usage_pct_avg": classify("cpu.usage_pct", fleet.get("cpu_usage_pct_avg")), + "memory_used_pct": classify("mem.used_pct", fleet.get("memory_used_pct")), + "disk_worst_used_pct": classify("disk.used_pct", fleet.get("disk_worst_used_pct")), + "gpu_busy_pct_avg": classify("gpu.busy_pct", fleet.get("gpu_busy_pct_avg")), + } + + +def _ups_severity(card: dict[str, Any]) -> dict[str, str | None]: + """The three hero battery numbers, keyed as the overview row names them.""" + numbers = card.get("numbers") or {} + return { + "charge_pct": classify("ups.charge_pct", numbers.get("charge_pct")), + "load_pct": classify("ups.load_pct", numbers.get("load_pct")), + "runtime_seconds": classify("ups.battery_runtime", numbers.get("battery_runtime")), + } + + # --------------------------------------------------------------------------- # # Common routes # --------------------------------------------------------------------------- # @@ -161,10 +199,13 @@ def common_router() -> APIRouter: nut_job = ctx.hub.jobs.get("nut") if nut_job and nut_job.state.payload: ups_payload = nut_job.state.payload + fleet = fleet_summary(hosts) if hosts else None + if fleet is not None: + fleet["severity"] = _fleet_severity(fleet) return { "ts": time.time(), "system": {"enabled": bool(ctx.settings.host_targets), - "fleet": fleet_summary(hosts) if hosts else None, + "fleet": fleet, "hosts": [{"name": h["name"], "os": (h.get("info") or {}).get("os_pretty"), "cpu_usage_pct": h["cpu"]["usage_pct"], "load1": h["cpu"].get("load1"), @@ -173,7 +214,8 @@ def common_router() -> APIRouter: "disk_worst_used_pct": h["disks"]["worst_used_pct"], "gpu_busy_pct": [g.get("busy_pct") for g in h.get("gpus") or []], "temp_c": h["cpu"].get("temp_c"), - "uptime_seconds": (h.get("info") or {}).get("uptime_seconds")} + "uptime_seconds": (h.get("info") or {}).get("uptime_seconds"), + "severity": _host_severity(h)} for h in hosts], "states": ctx.hub.states("system")}, "llm": {"enabled": bool(ctx.settings.gufo_url or ctx.settings.vllm_url), @@ -190,7 +232,8 @@ def common_router() -> APIRouter: "status": card["status"], "charge_pct": card["numbers"].get("charge_pct"), "load_pct": card["numbers"].get("load_pct"), - "runtime_seconds": card["numbers"].get("battery_runtime")} + "runtime_seconds": card["numbers"].get("battery_runtime"), + "severity": _ups_severity(card)} for card in (ups_payload or {}).get("ups", [])], "states": ctx.hub.states("ups")}, "recent_events": [event.as_json() for event in ctx.store.recent_events(12)], diff --git a/src/dashboards/severity.py b/src/dashboards/severity.py new file mode 100644 index 0000000..8d6ccf1 --- /dev/null +++ b/src/dashboards/severity.py @@ -0,0 +1,144 @@ +"""The one place that decides whether a number is fine, worrying, or bad. + +Two rules, and the whole module exists to keep them true: + +1. **Unknown is not ok.** A value the upstream does not publish — `None`, a NaN that + leaked through a parser, a metric with no entry in :data:`THRESHOLDS` — classifies to + ``None``, never to ``ok``. Green means "measured and good", the same promise the + `AGENTS.md` "never invent a number" rule makes about the value itself. The page renders + `n/a` with no colour class. +2. **A threshold exists nowhere else in the app.** Pages render the class the API sent and + must not compare a number to a magic value; the JS helpers in ``web/static`` read the + ``severity`` field instead of re-deriving it. Four pages with four opinions about what + red means is the bug this module prevents. + +Vocabulary is ``ok`` / ``warn`` / ``fail`` — the words ``dash.css`` (``.bar.ok``, +``.dot.warn``, ``.chip.fail``) and ``sources/nut.py``'s ``health.checks[].status`` already +use. "Danger" in the spec means ``fail``; there is no synonym for the same colour. + +Reaching a threshold trips it, in both directions: for ``cpu.usage_pct`` (warn 75, danger +90) 75 is ``warn`` and 90 is ``fail``; for the inverted ``ups.charge_pct`` (warn 50, +danger 20) 50 is ``warn`` and 20 is ``fail``, because there a *smaller* number is worse. +:data:`INVERTED` is the only place that direction is recorded, so a caller cannot get it +backwards by accident. + +Every `_pct` metric is on a 0-100 scale, never 0-1: an upstream that publishes a ratio +(`llamacpp:kv_cache_usage_ratio`) is converted to percent *before* it is classified, or it +will read as "fine" forever. +""" + +from __future__ import annotations + +import math +from collections.abc import Mapping +from typing import Any + +OK = "ok" +WARN = "warn" +FAIL = "fail" + +#: metric name -> (warn, danger). The numbers were ratified on 2026-10-16 and match what +#: the old JS call sites passed, so turning the thresholds into data changes colour on the +#: pages only where they had already drifted. The two UPS metrics are new: a battery at +#: half charge or with ten minutes of runtime left is worth amber before it is worth red. +#: Keys are the dotted metric names of the history namespace; ``@stream`` suffixes +#: (``disk.used_pct@/srv``, ``gpu.busy_pct@0``) inherit their base metric's row. +THRESHOLDS: dict[str, tuple[float, float]] = { + "cpu.usage_pct": (75.0, 90.0), + "mem.used_pct": (80.0, 92.0), + "disk.used_pct": (80.0, 90.0), + "cpu.temp_c": (75.0, 90.0), + "gpu.busy_pct": (85.0, 95.0), + "gpu.mem_used_pct": (85.0, 95.0), + "llm.kv_cache_usage_pct": (75.0, 90.0), + "ups.charge_pct": (50.0, 20.0), + "ups.battery_runtime": (600.0, 300.0), # seconds of runtime left + "ups.load_pct": (70.0, 90.0), +} + +#: Metrics where **lower is worse**, so ``classify`` owns the direction. Every entry must +#: also be in :data:`THRESHOLDS` (a test enforces it, along with bound ordering). +INVERTED: set[str] = {"ups.charge_pct", "ups.battery_runtime"} + + +def _number(value: float | None) -> float | None: + """The value as a finite float, or ``None`` when there is no number to classify. + + NaN and ±inf are not renderable and not JSON-serialisable; they become the same + `n/a` the app uses for a missing upstream value rather than a colour nobody can justify. + """ + if value is None: + return None + num = float(value) + return num if math.isfinite(num) else None + + +def _resolve(metric: str) -> tuple[float, float, bool] | None: + """(warn, danger, inverted) for a metric, or ``None`` if the table does not know it. + + ``metric@stream`` resolves to ``metric``: a per-core, per-mount, or per-card series is + judged by the same rule as the aggregate it came from. + """ + base = metric.split("@", 1)[0] + pair = THRESHOLDS.get(base) + if pair is None: + return None + warn, danger = pair + return warn, danger, base in INVERTED + + +def classify(metric: str, value: float | None) -> str | None: + """`ok`, `warn`, `fail` — or ``None`` when nothing honest can be said. + + ``None`` in → ``None`` out, always: no measurement, no verdict. An unknown metric name + is the same case (no threshold, no colour) and never raises. + """ + num = _number(value) + if num is None: + return None + bounds = _resolve(metric) + if bounds is None: + return None + warn, danger, inverted = bounds + if inverted: + if num <= danger: + return FAIL + if num <= warn: + return WARN + return OK + if num >= danger: + return FAIL + if num >= warn: + return WARN + return OK + + +def attach(metric: str, value: float | None) -> dict[str, Any]: + """`{"value": …, "severity": …}` for payloads where the metric is a bare number. + + The only mutation of the caller's value is turning a non-finite float into ``None``, + which keeps the payload valid JSON and the panel honest. + """ + num = _number(value) + return {"value": num, "severity": classify(metric, num)} + + +def with_severity(metric: str, cell: Mapping[str, Any] | None, + scale: float = 1.0) -> dict[str, Any]: + """Copy of a `{value, source, derived}` cell (LLM metrics) with `severity` added. + + Additive-only: every existing key keeps its name and meaning, and a cell that is + missing or holds a non-numeric value still comes back with ``severity: null``. The + input mapping is never mutated. + + ``scale`` rescales the value **for the classification only**: the KV-cache gauges are + published as 0-1 ratios (`llamacpp:kv_cache_usage_ratio`, `vllm:kv_cache_usage_perc`) + while the table is written in percent, so classifying the raw ratio would read "fine" + forever. The cell's own `value` is left exactly as the upstream published it — the + conversion is a unit, not a new measurement, and the page still prints what it got. + """ + out: dict[str, Any] = dict(cell) if cell is not None else {} + raw = out.get("value") + value = raw if raw is None or isinstance(raw, (int, float)) else None + out["severity"] = classify(metric, None if value is None else value * scale) + return out diff --git a/src/dashboards/sources/host.py b/src/dashboards/sources/host.py index 44a7939..b7a09e4 100644 --- a/src/dashboards/sources/host.py +++ b/src/dashboards/sources/host.py @@ -21,6 +21,7 @@ from typing import Any from ..poller import Result from ..rates import RateEstimator +from ..severity import classify from ..store import Sample # /proc/stat jiffies we treat as "not doing useful work". @@ -231,6 +232,7 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl temp_c = max(temps.values()) if temps else None pools_used = None if vram_used is None else vram_used + (gtt_used or 0.0) pools_total = None if vram_total is None else vram_total + (gtt_total or 0.0) + pool_used_pct = _ratio_percent(pools_used, pools_total) row = { "index": index, "card": gpu.get("card"), @@ -250,11 +252,14 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl # this is the honest whole-addressable-pool view, kept beside the two pools "mem_used_bytes": pools_used, "mem_total_bytes": pools_total, - "mem_used_pct": _ratio_percent(pools_used, pools_total), + "mem_used_pct": pool_used_pct, "power_watts": gpu.get("power_watts"), "temps": temps, "temp_c": temp_c, "clocks": gpu.get("clocks") or {}, + # the verdict travels with the card; keys mirror the rated fields beside them + "severity": {"busy_pct": classify("gpu.busy_pct", gpu.get("busy_pct")), + "mem_used_pct": classify("gpu.mem_used_pct", pool_used_pct)}, } gpus.append(row) for metric, value in (("busy_pct", row["busy_pct"]), ("vram_used_bytes", vram_used), @@ -324,6 +329,10 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl "procs_running": cpu_raw.get("procs_running"), "procs_blocked": cpu_raw.get("procs_blocked"), "temp_c": cpu_temp, + # Additive: the pages colour from this verdict instead of comparing the number + # to a threshold of their own. Keys mirror the rated fields in this section. + "severity": {"usage_pct": classify("cpu.usage_pct", usage_pct), + "temp_c": classify("cpu.temp_c", cpu_temp)}, }, "memory": { "total_bytes": mem_total, @@ -336,6 +345,7 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl "swap_used_bytes": swap_used, "swap_used_pct": swap_used_pct, "committed_bytes": mem.get("committed_bytes"), + "severity": {"used_pct": classify("mem.used_pct", mem_used_pct)}, }, "disks": { "filesystems": filesystems, @@ -347,6 +357,8 @@ def parse_host_snapshot(name: str, snapshot: dict[str, Any], collector: HostColl "writes_per_s": writes_total if any_io else None, "util_pct": util_max, "devices": io_devices, + "severity": {"worst_used_pct": classify( + "disk.used_pct", worst.get("used_pct") if worst else None)}, }, "network": { "rx_bytes_per_s": rx_total if any_net else None, diff --git a/src/dashboards/sources/llm.py b/src/dashboards/sources/llm.py index bffa4c5..09ddaf8 100644 --- a/src/dashboards/sources/llm.py +++ b/src/dashboards/sources/llm.py @@ -31,6 +31,7 @@ from typing import Any from ..poller import Result from ..prom import Prom, field from ..rates import RateEstimator +from ..severity import with_severity from ..store import Sample GUFO_SERIES: dict[str, tuple[tuple[str, ...], str]] = { @@ -91,6 +92,23 @@ VLLM_HISTOGRAMS: dict[str, tuple[str, ...]] = { "e2e_request_latency_seconds": ("vllm:e2e_request_latency_seconds",), } +#: Cells that have a row in `severity.THRESHOLDS`, mapped to (metric, scale). Both engines +#: publish KV pressure as a 0-1 ratio while the threshold table is written in percent, so +#: the scale converts for the *verdict only* — `value` stays what the engine published +#: (rule 2: no invented number), and 0.85 is `warn` rather than a permanent `ok`. +SEVERITY_CELLS: dict[str, tuple[str, float]] = { + "kv_cache_usage_ratio": ("llm.kv_cache_usage_pct", 100.0), # gufo / llama.cpp + "gpu_cache_usage": ("llm.kv_cache_usage_pct", 100.0), # vLLM +} + + +def attach_cell_severity(metrics: dict[str, Any]) -> None: + """Add `severity` to the rated cells of a parsed metrics dict, in place, additively.""" + for key, (metric, scale) in SEVERITY_CELLS.items(): + if key in metrics: + metrics[key] = with_severity(metric, metrics[key], scale=scale) + + #: history series per engine, in the order the LLM page charts them HISTORY_METRICS: dict[str, tuple[str, ...]] = { "decode_tokens_per_second": ("decode_tokens_per_second",), @@ -165,6 +183,7 @@ def parse_gufo_metrics(text: str, rates: RateEstimator | None = None, out["spec_drafts_per_request"] = _ratio(out["spec_drafts_total"], out["spec_draft_tokens_total"], "computed:drafts/draft_tokens") + attach_cell_severity(out) return out @@ -244,6 +263,7 @@ def parse_vllm_metrics(text: str, rates: RateEstimator | None = None, out["prefix_cache_hit_ratio"] = _ratio(out["prefix_cache_hits_total"], out["prefix_cache_queries_total"], "computed:hits/queries") + attach_cell_severity(out) return out diff --git a/src/dashboards/sources/nut.py b/src/dashboards/sources/nut.py index 647c203..384a968 100644 --- a/src/dashboards/sources/nut.py +++ b/src/dashboards/sources/nut.py @@ -34,6 +34,8 @@ import time from dataclasses import dataclass, field from typing import Any +from ..severity import classify + DEFAULT_PORT = 3493 @@ -741,6 +743,12 @@ def payload(reading: UpsReading, *, now: float | None = None, **assess_kwargs: A "alarms": reading.alarms, "on_battery": reading.on_battery, "numbers": numbers, + # Severity answers "how bad is this number" for the three hero values; `health` + # and its `checks[].status` answer "is this condition satisfied". Both stay. + "severity": {"charge_pct": classify("ups.charge_pct", numbers.get("charge_pct")), + "load_pct": classify("ups.load_pct", numbers.get("load_pct")), + "battery_runtime": classify("ups.battery_runtime", + numbers.get("battery_runtime"))}, "health": health, "advertised_commands": reading.commands, "writable": reading.writable, diff --git a/src/dashboards/web/static/app.js b/src/dashboards/web/static/app.js index 37e4fba..7e8bd69 100644 --- a/src/dashboards/web/static/app.js +++ b/src/dashboards/web/static/app.js @@ -102,6 +102,16 @@ const Dash = (() => { el.classList.add(threshold); } + /** + * FALLBACK ONLY: a colour class derived from thresholds the *caller* passes. + * + * Thresholds live in `src/dashboards/severity.py` and travel in the payload as + * `severity`; `severityFrom()` below is how a page reads them. This function survives + * only because `page-index.js`, `page-system.js`, `page-llm.js` and `page-ups.js` still + * call it with their own numbers (phases 02–05 convert them page by page, so each page's + * change is reviewable on its own). Do not add a new call site: a threshold written in + * JS is a threshold that can disagree with the other three pages. + */ function severityFor(value, warnAt, failAt) { const warn = warnAt == null ? 75 : warnAt; const fail = failAt == null ? 90 : failAt; @@ -111,6 +121,105 @@ const Dash = (() => { return 'ok'; } + // ---- severity & hero numbers ------------------------------------------- + /** The only colour vocabulary in the app — matches `dash.css` and severity.py. */ + const SEVERITY_CLASSES = ['ok', 'warn', 'fail']; + + /** + * The colour class for a value, read from the payload instead of a threshold. + * + * Accepts a payload cell (`{value, severity}`), the bare `severity` string, or `null`, + * and returns `ok` / `warn` / `fail` / `''`. Anything the API does not rate — no field, + * `null`, a word from another vocabulary — returns `''`, which means *no colour*: an + * unknown number must never look green. + */ + function severityFrom(cellOrSeverity) { + let value = cellOrSeverity; + if (value && typeof value === 'object') value = value.severity; + if (typeof value !== 'string') return ''; + const word = value.trim().toLowerCase(); + return SEVERITY_CLASSES.indexOf(word) >= 0 ? word : ''; + } + + /** First direct child with `className`, or null. */ + function childOf(parent, className) { + for (const node of Array.from(parent.children)) { + if (node.classList && node.classList.contains(className)) return node; + } + return null; + } + + /** The child with `className`, created (before `before`) on first use. */ + function ensureChild(parent, className, tag, before) { + const existing = childOf(parent, className); + if (existing) return existing; + const node = document.createElement(tag); + node.className = className; + parent.insertBefore(node, before || null); + return node; + } + + /** Text for a label or footnote: `null` prints nothing (unlike `setTxt`'s `n/a`). */ + function assignText(el, text) { + const value = text == null ? '' : String(text); + if (el.textContent !== value) el.textContent = value; + } + + /** + * Render — or update in place — one big colour-coded number. + * + * `hero(el, {label, value, unit, digits, severity, sub, bar})` + * - `el`: an element or a selector (a missing element is a silent no-op, like the other + * setters, so a page can call every hero it knows about); + * - `severity`: taken from the payload (`{value, severity}` cell or the bare string) and + * resolved by `severityFrom` — this function never compares a number to a threshold; + * - a `null` / non-finite `value` prints `n/a`, faint, **with no severity class**, so a + * value nobody measured cannot keep last poll's green; + * - `bar`: `true` for a 0-100 value, or a number for another max — adds/updates a `.bar` + * under the number (the `.hero-bar` variant of `dash.css`). + * + * The DOM is built once and then patched text-by-text, so a poll does not restart CSS + * transitions, drop the tabular-numeral layout, or steal focus. + */ + function hero(el, options) { + const node = typeof el === 'string' ? $(el) : el; + if (!node) return null; + const opts = options || {}; + node.classList.add('hero'); + assignText(ensureChild(node, 'label', 'div'), opts.label); + const valueEl = ensureChild(node, 'value', 'div'); + const numberEl = ensureChild(valueEl, 'num', 'span'); + const unitEl = ensureChild(valueEl, 'unit', 'span'); + + const raw = opts.value; + const measured = raw != null && (typeof raw === 'string' ? raw !== '' : isFinite(raw)); + const sev = measured ? severityFrom(opts.severity) : ''; + SEVERITY_CLASSES.forEach((name) => node.classList.toggle(name, name === sev)); + + if (measured) { + const text = typeof raw === 'string' ? raw : num(raw, opts.digits); + if (numberEl.textContent !== text) numberEl.textContent = text; + numberEl.classList.remove('is-null', 'na'); + valueEl.classList.remove('is-null'); + } else { + if (numberEl.textContent !== naText) numberEl.textContent = naText; + numberEl.classList.add('is-null', 'na'); + valueEl.classList.add('is-null'); + } + assignText(unitEl, measured ? opts.unit : ''); + + if (opts.bar) { + const bar = ensureChild(node, 'bar', 'div', childOf(node, 'sub')); + const fill = $('i', bar) || bar.appendChild(document.createElement('i')); + const max = typeof opts.bar === 'number' && opts.bar > 0 ? opts.bar : 100; + const width = typeof raw === 'number' ? Math.max(0, Math.min(100, (raw / max) * 100)) : 0; + fill.style.width = `${width}%`; + SEVERITY_CLASSES.forEach((name) => bar.classList.toggle(name, name === sev)); + } + assignText(ensureChild(node, 'sub', 'div'), opts.sub); + return node; + } + function setDot(el, status) { if (!el) return; el.classList.remove('ok', 'warn', 'fail', 'idle'); @@ -335,7 +444,7 @@ const Dash = (() => { return { $, $$, fetchJSON, poll, fmtBytes, bytes, rate, pct, num, duration, age, setTxt, setValue, setBar, setDot, chip, initChrome, rangeControl, toast, bindPause, - staleness, markStale, severityFor, ageSeconds, naText }; + staleness, markStale, severityFor, severityFrom, hero, ageSeconds, naText }; })(); document.addEventListener('DOMContentLoaded', () => Dash.initChrome()); diff --git a/src/dashboards/web/static/dash.css b/src/dashboards/web/static/dash.css index 9f57f3a..147ee6c 100644 --- a/src/dashboards/web/static/dash.css +++ b/src/dashboards/web/static/dash.css @@ -238,6 +238,54 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; } .na { color: var(--text-faint); } +/* ---- hero numbers -------------------------------------------------------- + * The big glanceable number: label, value + unit, and optional bar and footnote. + * Colour comes from the *same* `ok|warn|fail` classes the rest of the stylesheet uses + * (`.bar.ok`, `.dot.warn`, `.chip.fail`) — there is deliberately no parallel `.sev-*` + * family, because one word per colour is what keeps four pages agreeing. `Dash.hero()` + * applies the class from the `severity` the API sent, so nothing here (or in JS) knows a + * threshold. No class at all means no measurement: the number renders faint, never green. + */ +.hero { + display: grid; + gap: .18rem; + min-width: 0; +} +.hero .label { + font-size: 10.5px; text-transform: uppercase; letter-spacing: .06em; color: var(--text-faint); + white-space: nowrap; overflow: hidden; text-overflow: ellipsis; +} +.hero .value { + font-family: var(--mono); font-size: 2.4rem; font-weight: 650; line-height: 1.05; + font-variant-numeric: tabular-nums; /* fixed digit width: no jitter between polls */ + display: flex; align-items: baseline; gap: .3rem; flex-wrap: wrap; + color: var(--text); +} +.hero .value > span { white-space: nowrap; } +.hero .unit { font-size: .38em; font-weight: 550; color: var(--text-muted); } +.hero .sub { + font-family: var(--mono); font-size: 11px; color: var(--text-faint); + overflow-wrap: anywhere; +} +/* an empty label/sub takes no space, so a hero without a footnote is not taller */ +.hero .label:empty, .hero .sub:empty { display: none; } + +.hero.ok .value { color: var(--ok); } +.hero.warn .value { color: var(--warn); } +.hero.fail .value { color: var(--fail); } +/* after the verdicts on purpose: `n/a` is not a measurement, so it is never coloured */ +.hero .value.is-null, .hero .value .is-null { color: var(--text-faint); font-size: 1.25rem; font-weight: 550; } + +/* number + bar: the per-host cards and the UPS hero row both want the pair, and the bar + takes its colour from the hero so markup does not have to repeat the verdict */ +.hero-bar { gap: .28rem; } +.hero-bar .bar { width: 100%; } +.hero-bar.ok .bar > i { background: var(--ok); } +.hero-bar.warn .bar > i { background: var(--warn); } +.hero-bar.fail .bar > i { background: var(--fail); } +/* with no verdict on either the hero or the bar the fill stays the neutral accent: the + bar shows *how much*, never a judgement nobody measured. */ + /* ---- bars, dots, chips -------------------------------------------------- */ .bar { position: relative; height: 7px; border-radius: 999px; background: var(--bg-sunken); diff --git a/tests/test_api.py b/tests/test_api.py index 0875c95..bf76430 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -7,6 +7,8 @@ asset the network can fail to deliver. """ from __future__ import annotations +import copy +import re import time from collections.abc import Iterator from pathlib import Path @@ -17,6 +19,9 @@ from fastapi.testclient import TestClient from dashboards.app import create_app from dashboards.config import Settings, Target +from dashboards.rates import RateEstimator +from dashboards.severity import classify +from dashboards.sources.host import HostCollector, parse_host_snapshot from dashboards.store import Event, Sample, SampleStore # refused immediately, and never dials out of the test environment @@ -63,6 +68,107 @@ def wait_for_payload(client: TestClient, path: str, *, timeout: float = 5.0) -> pytest.fail(f"{path} never became available") +def host_payload(agent_snapshot: dict[str, Any]) -> dict[str, Any]: + """Two scrapes of the real driveripper capture, nudged onto known sides of the table. + + Driven from the fixture rather than hand-written so a schema drift breaks here first. + """ + snap = copy.deepcopy(agent_snapshot) + snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95 # mem.used_pct -> fail + snap["cpu_temperature_celsius"] = 91.0 # cpu.temp_c -> fail + for fs in snap["filesystems"]: + fs["used_pct"] = 93.0 # disk -> fail + collector = HostCollector(RateEstimator(span=60.0)) + parse_host_snapshot("driveripper", snap, collector, now=1_000.0) + second = copy.deepcopy(snap) + second["ts"] = 1_010.0 + modes = second["cpu"]["modes"] + modes["total"] += 1_000.0 + modes["idle"] += 900.0 # 10 % busy over the window -> ok + return parse_host_snapshot("driveripper", second, collector, now=1_010.0).payload + + +@pytest.fixture +def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, + agent_snapshot: dict[str, Any]) -> Iterator[TestClient]: + """Overview-shaped app: one host payload in the hub plus the simulated UPSes. + + The host job points at a dead port, so a poll can never overwrite the injected payload + (a failed poll keeps the last good one) — the numbers below are ours, not the fleet's. + """ + monkeypatch.setenv("NUT_SIMULATE", "1") + db = str(tmp_path / "summary.sqlite3") + store = SampleStore(db) + app = create_app(Settings(db_path=db, nut_host="127.0.0.1", + host_targets=(Target("driveripper", DEAD_URL),)), store) + test_client = TestClient(app) + test_client.__enter__() + try: + app.state.ctx.hub.jobs["host:driveripper"].state.payload = host_payload(agent_snapshot) + wait_for_payload(test_client, "/api/ups/snapshot") + yield test_client + finally: + test_client.__exit__(None, None, None) + store.close() + + +# ---- severity travels with the value --------------------------------------- # + +def test_summary_carries_the_verdict_for_every_column_the_page_colours(data_client) -> None: + body = data_client.get("/api/summary").json() + row = body["system"]["hosts"][0] + assert row["severity"] == {"cpu_usage_pct": "ok", "mem_used_pct": "fail", + "disk_worst_used_pct": "fail", "temp_c": "fail"} + # the fleet card is judged by the same table, so amber means the same thing at both sizes + assert body["system"]["fleet"]["severity"] == { + "cpu_usage_pct_avg": "ok", "memory_used_pct": "fail", + "disk_worst_used_pct": "fail", "gpu_busy_pct_avg": "ok", + } + for ups in body["ups"]["ups"]: + assert set(ups["severity"]) == {"charge_pct", "load_pct", "runtime_seconds"} + assert ups["severity"]["charge_pct"] == classify("ups.charge_pct", ups["charge_pct"]) + assert ups["severity"]["charge_pct"] == "ok", "the simulator starts at float charge" + + +def test_summary_shape_did_not_shrink(data_client) -> None: + """Additive-only anchor: /api/summary gained `severity` and lost nothing. + + The overview, the live tests, and any external consumer read the flat keys; this test is + the tripwire for a well-meaning refactor that "tidies" them away. + """ + body = data_client.get("/api/summary").json() + fleet_keys = {"hosts_total", "hosts_live", "cpu_cores", "cpu_usage_pct_avg", + "memory_total_bytes", "memory_used_bytes", "memory_used_pct", + "disk_worst_used_pct", "load1_total", "net_rx_bytes_per_s", + "net_tx_bytes_per_s", "gpu_count", "gpu_busy_pct_avg", "power_estimate_w"} + host_keys = {"name", "os", "cpu_usage_pct", "load1", "cores", "mem_used_pct", + "disk_worst_used_pct", "gpu_busy_pct", "temp_c", "uptime_seconds"} + ups_keys = {"name", "overall", "status", "charge_pct", "load_pct", "runtime_seconds"} + assert fleet_keys <= set(body["system"]["fleet"]), fleet_keys - set(body["system"]["fleet"]) + assert host_keys <= set(body["system"]["hosts"][0]), host_keys - set(body["system"]["hosts"][0]) + assert ups_keys <= set(body["ups"]["ups"][0]), ups_keys - set(body["ups"]["ups"][0]) + assert set(body) == {"ts", "system", "llm", "ups", "recent_events"} + # ...and the new keys are there beside the old ones, not instead of them + assert "severity" in body["system"]["fleet"] + assert "severity" in body["system"]["hosts"][0] + assert "severity" in body["ups"]["ups"][0] + + +def test_snapshot_payloads_carry_their_own_severity_sections(data_client) -> None: + """/api/system/snapshot and /api/ups/snapshot ship the same verdicts the summary reuses.""" + system = data_client.get("/api/system/snapshot").json() + host = system["hosts"][0] + assert set(host["cpu"]["severity"]) == {"usage_pct", "temp_c"} + assert set(host["memory"]["severity"]) == {"used_pct"} + assert set(host["disks"]["severity"]) == {"worst_used_pct"} + assert set(host["gpus"][0]["severity"]) == {"busy_pct", "mem_used_pct"} + ups = data_client.get("/api/ups/snapshot").json() + for card in ups["ups"]: + assert set(card["severity"]) == {"charge_pct", "load_pct", "battery_runtime"} + # numbers keep their own shape; severity is a sibling, never a replacement + assert isinstance(card["numbers"], dict) and "charge_pct" in card["numbers"] + + # ---- unconfigured: say so, and do not dial -------------------------------- # def test_config_reports_every_source_as_off(client) -> None: @@ -175,6 +281,104 @@ def test_static_assets_are_served(client) -> None: assert client.get(asset).status_code == 200, asset +# ---- the severity language in the static assets --------------------------- # +# +# There is no JS test runner (a locked anchor: adding one would mean a build step). +# What *can* be asserted end-to-end, from the served bytes, is the contract the pages +# depend on: the hero style exists, its colour comes from the shared ok/warn/fail tokens, +# no parallel `.sev-*` vocabulary appeared, and `Dash.hero()` reads the verdict instead of +# comparing the number to a threshold in JavaScript. + +ASSETS = ("/static/dash.css", "/static/app.js", "/static/charts.js", + "/static/page-index.js", "/static/page-system.js", "/static/page-llm.js", + "/static/page-ups.js") + + +def _css(client: TestClient) -> str: + return client.get("/static/dash.css").text + + +def _hero_css(css: str) -> str: + """Just the `hero numbers` section of the stylesheet.""" + return css.split("/* ---- hero numbers", 1)[1].split("/* ---- bars", 1)[0] + + +def test_css_ships_the_hero_number_style(client) -> None: + css = _css(client) + hero_block = _hero_css(css) + for rule in (".hero {", ".hero .label", ".hero .value", ".hero .unit", ".hero .sub", + ".hero.ok .value", ".hero.warn .value", ".hero.fail .value", + ".hero-bar", ".hero-bar.ok .bar"): + assert rule in hero_block, f"dash.css lost the {rule} rule" + # a display size with fixed-width digits, so the number does not jitter between polls + assert "2.4rem" in hero_block + assert "tabular-nums" in hero_block + + +def test_hero_colour_comes_from_the_shared_verdict_tokens(client) -> None: + """`.hero` reuses `--ok/--warn/--fail`; it does not invent a palette of its own.""" + hero_block = _hero_css(_css(client)) + for token in ("var(--ok)", "var(--warn)", "var(--fail)", "var(--text)", + "var(--text-muted)", "var(--text-faint)"): + assert token in hero_block, f"the hero style no longer uses {token}" + + +def test_no_stylesheet_rule_uses_an_undeclared_token(client) -> None: + """A typo'd `var(--x)` renders unstyled and reads as a broken panel, not an error.""" + css = _css(client) + root = re.search(r":root\s*\{(.*?)\n\}", css, re.S) + assert root is not None + declared = set(re.findall(r"(--[a-z0-9-]+)\s*:", root.group(1))) + used = set(re.findall(r"var\(\s*(--[a-z0-9-]+)", css)) + assert not used - declared + + +def _code(body: str) -> str: + """An asset with its comments stripped — prose may *name* a class the code must not define.""" + body = re.sub(r"/\*.*?\*/", " ", body, flags=re.S) + body = re.sub(r"", " ", body, flags=re.S) + return re.sub(r"^\s*//.*$", " ", body, flags=re.M) + + +def test_no_parallel_severity_vocabulary_exists_anywhere(client) -> None: + """One word per colour: `ok|warn|fail`, as in severity.py and the existing components.""" + for asset in (*ASSETS, "/", "/system", "/llm", "/ups"): + body = _code(client.get(asset).text) + assert ".sev-" not in body, f"{asset} invented a .sev-* class" + assert not re.search(r"classList\.(?:add|toggle)\(\s*[\"'](?:danger|good|warning)[\"']", + body), f"{asset} invented a synonym for fail" + + +def test_app_js_exposes_the_hero_helpers(client) -> None: + js = client.get("/static/app.js").text + assert "function hero(" in js + assert "function severityFrom(" in js + exports = js.rsplit("return {", 1)[1].split("}", 1)[0] + for name in ("hero", "severityFrom", "severityFor"): + assert re.search(rf"\b{name}\b", exports), f"Dash.{name} is not exported" + + +def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None: + """`hero()` may only colour what the API rated — no thresholds re-derived in JS.""" + js = client.get("/static/app.js").text + start = js.index("function hero(") + body = js[start:js.index("\n function setDot", start)] + assert "severityFrom(" in body + assert "severityFor(" not in body + # an unknown value gets n/a and no colour class at all + assert "naText" in body + assert re.search(r"raw != null", body) + + +def test_severity_for_is_kept_as_the_documented_fallback(client) -> None: + """The old call sites still work until phases 02-05 convert each page.""" + js = client.get("/static/app.js").text + assert "FALLBACK ONLY" in js + for page in ("/static/page-index.js", "/static/page-system.js", + "/static/page-llm.js", "/static/page-ups.js"): + assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site" + + def test_api_responses_are_not_cacheable(client) -> None: for path in ("/api/config", "/api/health", "/api/summary"): assert "no-store" in client.get(path).headers.get("cache-control", "") diff --git a/tests/test_host_source.py b/tests/test_host_source.py index 5bc7b3e..75f6b34 100644 --- a/tests/test_host_source.py +++ b/tests/test_host_source.py @@ -13,6 +13,7 @@ from typing import Any import pytest from dashboards.rates import RateEstimator +from dashboards.severity import classify from dashboards.sources.host import HostCollector, fleet_summary, parse_host_snapshot TICKS_PER_SECOND = 100.0 @@ -234,6 +235,80 @@ def test_fleet_summary_worst_disk_and_memory_totals(agent_snapshot) -> None: base["memory"]["used_bytes"] / base["memory"]["total_bytes"] * 100) +# ---- severity: the verdict travels with the value ------------------------- # + +def test_payloads_carry_a_severity_beside_every_rated_value(agent_snapshot) -> None: + """The page paints a class; it must never hold its own opinion about where amber starts.""" + _first, payload = parse_twice(agent_snapshot) + assert payload["cpu"]["severity"] == { + "usage_pct": classify("cpu.usage_pct", payload["cpu"]["usage_pct"]), + "temp_c": classify("cpu.temp_c", payload["cpu"]["temp_c"]), + } + assert payload["cpu"]["severity"]["usage_pct"] == "ok" # the window is 10 % busy + assert payload["cpu"]["severity"]["temp_c"] == "ok" # 68 °C in the capture + assert payload["memory"]["severity"]["used_pct"] == "ok" # 68 % of 128 GB + assert payload["disks"]["severity"]["worst_used_pct"] == "ok" # 74.6 % worst mount + gpu = payload["gpus"][0] + assert gpu["severity"] == { + "busy_pct": classify("gpu.busy_pct", gpu["busy_pct"]), + "mem_used_pct": classify("gpu.mem_used_pct", gpu["mem_used_pct"]), + } + assert gpu["severity"]["busy_pct"] == "ok" + + +def test_severity_trips_on_the_ratified_boundaries(agent_snapshot) -> None: + """Values pushed onto the table's edges in the real fixture, not on a hand-drawn snapshot.""" + snap = copy.deepcopy(agent_snapshot) + snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.92 # mem.used_pct danger + snap["cpu_temperature_celsius"] = 90.0 # cpu.temp_c danger + for fs in snap["filesystems"]: + fs["used_pct"] = 80.0 # disk.used_pct warn + snap["gpus"][0]["busy_pct"] = 95.0 # gpu.busy_pct danger + snap["gpus"][0]["gtt_used_bytes"] = 65e9 # whole-pool usage well past danger + payload = parse_host_snapshot("driveripper", snap, make_collector(), now=1_000.0).payload + assert payload["memory"]["severity"]["used_pct"] == "fail" + assert payload["cpu"]["severity"]["temp_c"] == "fail" + assert payload["disks"]["severity"]["worst_used_pct"] == "warn" + assert payload["gpus"][0]["severity"]["busy_pct"] == "fail" + assert payload["gpus"][0]["severity"]["mem_used_pct"] == "fail" + + +def test_a_value_the_agent_did_not_report_gets_no_colour(agent_snapshot) -> None: + """Unknown is not ok: `severity: null` renders `n/a`, a green pass would be a lie.""" + snap = copy.deepcopy(agent_snapshot) + snap["cpu_temperature_celsius"] = None + snap["gpus"][0]["busy_pct"] = None + snap["filesystems"] = [] + payload = parse_host_snapshot("driveripper", snap, make_collector(), now=1_000.0).payload + assert payload["cpu"]["severity"]["temp_c"] is None + assert payload["cpu"]["severity"]["usage_pct"] is None, "one scrape is not a rate" + assert payload["gpus"][0]["severity"]["busy_pct"] is None + assert payload["disks"]["severity"]["worst_used_pct"] is None + # the numbers themselves are unchanged by the verdict sitting next to them + assert payload["memory"]["used_pct"] == pytest.approx( + snap["memory"]["used_bytes"] / snap["memory"]["total_bytes"] * 100) + + +def test_attaching_severity_renamed_or_removed_nothing(agent_snapshot) -> None: + """Additive-only anchor, at the source: the flat fields the pages read are all still there.""" + payload = parse_host_snapshot("driveripper", agent_snapshot, make_collector(), + now=1_000.0).payload + expected = { + "cpu": {"usage_pct", "modes_pct", "load1", "load5", "load15", "cores", + "core_usage_pct", "freq_mhz", "freq_max_mhz", "context_switches_per_s", + "forks_per_s", "procs_running", "procs_blocked", "temp_c"}, + "memory": {"total_bytes", "used_bytes", "available_bytes", "used_pct", "cached_bytes", + "buffers_bytes", "swap_total_bytes", "swap_used_bytes", "swap_used_pct", + "committed_bytes"}, + "disks": {"filesystems", "worst_used_pct", "worst_mount", "read_bytes_per_s", + "write_bytes_per_s", "reads_per_s", "writes_per_s", "util_pct", "devices"}, + } + for section, keys in expected.items(): + assert keys <= set(payload[section]), f"{section} lost {keys - set(payload[section])}" + for gpu in payload["gpus"]: + assert {"busy_pct", "mem_used_pct"} <= set(gpu) + + def test_fleet_summary_of_nothing_is_empty_not_zero() -> None: fleet = fleet_summary([]) assert fleet["hosts_total"] == 0 diff --git a/tests/test_llm_source.py b/tests/test_llm_source.py index 55534f1..73304f0 100644 --- a/tests/test_llm_source.py +++ b/tests/test_llm_source.py @@ -100,6 +100,35 @@ def test_absent_gufo_metrics_are_none_with_no_source() -> None: assert metrics.get("spec_accept_rate", {}).get("value") is None +# ---- severity: KV pressure arrives with a verdict ------------------------- # + +def test_gufo_kv_cache_cell_keeps_provenance_and_gains_severity(metrics_text) -> None: + metrics = parse_gufo_metrics(metrics_text("gufo")) + cell = metrics["kv_cache_usage_ratio"] + assert cell["value"] == 0.0 + assert cell["source"] == "llamacpp:kv_cache_usage_ratio" + assert cell["derived"] is False, "the verdict is additive; the cell still says what it was" + assert cell["severity"] == "ok" + + +@pytest.mark.parametrize("ratio,expected", [ + ("0.74", "ok"), ("0.75", "warn"), ("0.89", "warn"), ("0.9", "fail"), ("1.0", "fail"), +]) +def test_kv_ratio_is_classified_on_a_percent_scale(metrics_text, ratio, expected) -> None: + """0.9 of a KV cache is 90 %, not 0.9 % — classifying the raw ratio is a permanent green.""" + text = re.sub(r"^(llamacpp:kv_cache_usage_ratio) \S+$", rf"\g<1> {ratio}", + metrics_text("gufo"), count=1, flags=re.M) + metrics = parse_gufo_metrics(text) + assert metrics["kv_cache_usage_ratio"]["severity"] == expected + assert metrics["kv_cache_usage_ratio"]["value"] == pytest.approx(float(ratio)), \ + "the scale is for the verdict only; the published value is untouched" + + +def test_absent_kv_cell_gets_no_verdict() -> None: + metrics = parse_gufo_metrics("# nothing here\n") + assert metrics["kv_cache_usage_ratio"]["severity"] is None + + # ---- vLLM ----------------------------------------------------------------- # def test_vllm_counters_and_gauges(metrics_text) -> None: @@ -112,6 +141,23 @@ def test_vllm_counters_and_gauges(metrics_text) -> None: assert metrics["num_preemptions_total"]["value"] is not None +def test_vllm_kv_cache_cell_carries_a_severity(metrics_text) -> None: + assert parse_vllm_metrics(metrics_text("vllm"))["gpu_cache_usage"]["severity"] == "ok" + pressed = bump_metric(metrics_text("vllm"), "vllm:kv_cache_usage_perc", 0.95) + cell = parse_vllm_metrics(pressed)["gpu_cache_usage"] + assert cell["value"] == pytest.approx(0.95) + assert cell["severity"] == "fail" + assert cell["source"] == "vllm:kv_cache_usage_perc" + + +def test_vllm_absent_kv_cell_gets_no_verdict(metrics_text) -> None: + stripped = "\n".join(line for line in metrics_text("vllm").splitlines() + if "kv_cache_usage" not in line and "gpu_cache_usage" not in line) + metrics = parse_vllm_metrics(stripped) + assert metrics["gpu_cache_usage"]["value"] is None + assert metrics["gpu_cache_usage"]["severity"] is None + + def test_vllm_latency_histograms(metrics_text) -> None: metrics = parse_vllm_metrics(metrics_text("vllm")) ttft = metrics["time_to_first_token_seconds"] diff --git a/tests/test_nut.py b/tests/test_nut.py index 4c71107..416eb4f 100644 --- a/tests/test_nut.py +++ b/tests/test_nut.py @@ -435,6 +435,61 @@ def test_payload_carries_numbers_health_and_what_the_driver_offers() -> None: assert payload["extra"] or payload["status"] == "OL CHRG" +def test_payload_severity_colours_the_hero_battery_numbers() -> None: + """charge / load / runtime carry the verdict so the page paints a class, not a threshold.""" + payload = nut.payload(reading(), now=1_767_225_600.0) # charge 100, runtime 3720 s, load 34 % + assert payload["severity"] == {"charge_pct": "ok", "load_pct": "ok", "battery_runtime": "ok"} + # the verdict sits beside the numbers; it does not replace or rename any of them + assert payload["numbers"]["charge_pct"] == 100.0 + assert payload["numbers"]["battery_runtime"] == 3720.0 + + +@pytest.mark.parametrize("overrides,expected", [ + ({"battery.charge": "15"}, {"charge_pct": "fail"}), + ({"battery.charge": "50"}, {"charge_pct": "warn"}), # reaching warn trips it + ({"battery.charge": "50.1"}, {"charge_pct": "ok"}), + ({"battery.runtime": "240"}, {"battery_runtime": "fail"}), + ({"battery.runtime": "600"}, {"battery_runtime": "warn"}), + ({"ups.load": "95"}, {"load_pct": "fail"}), + ({"ups.load": "70"}, {"load_pct": "warn"}), +]) +def test_severity_trips_the_ratified_ups_thresholds(overrides, expected) -> None: + payload = nut.payload(reading(**overrides), now=1_767_225_600.0) + for key, verdict in expected.items(): + assert payload["severity"][key] == verdict, f"{key} with {overrides}" + + +def test_severity_is_null_when_the_driver_publishes_nothing() -> None: + """Neither real UPS reports a battery temperature and either can drop `battery.runtime`; + an absent number gets no colour, and the health check keeps saying `unknown`.""" + values = {key: value for key, value in VAR_ROWS.items() + if key not in ("battery.runtime", "battery.temperature")} + payload = nut.payload(nut.UpsReading(name="apc", vars=values, commands=[]), + now=1_767_225_600.0) + assert payload["numbers"]["battery_runtime"] is None + assert payload["severity"]["battery_runtime"] is None + assert payload["numbers"]["battery_temperature"] is None + temp = next(c for c in payload["health"]["checks"] if c["name"] == "Battery temperature") + assert temp["status"] == "unknown", "severity is additive: assess() keeps its own judgement" + + +def test_severity_does_not_touch_health_checks_or_numbers() -> None: + ups = reading() + payload = nut.payload(ups, now=1_767_225_600.0) + assert all(check["status"] in ("ok", "warn", "fail", "unknown") + for check in payload["health"]["checks"]) + assert payload["health"] == nut.assess(ups, now=1_767_225_600.0), \ + "the condition verdict and the value verdict must not contaminate each other" + + +async def test_simulated_ups_cards_carry_severity_too() -> None: + """The UPS page is developable without a battery, and still gets real verdicts.""" + for ups in await NutSimulator().read(): + card = nut.payload(ups, now=1_767_225_600.0) + assert set(card["severity"]) == {"charge_pct", "load_pct", "battery_runtime"} + assert card["severity"]["charge_pct"] == "ok", "the sim starts at float charge" + + def test_history_samples_only_include_real_numbers() -> None: values = {key: value for key, value in VAR_ROWS.items() if key != "battery.runtime"} samples = nut.history_samples(nut.UpsReading(name="cyberpower", vars=values, commands=[])) diff --git a/tests/test_severity.py b/tests/test_severity.py new file mode 100644 index 0000000..aaa1c9c --- /dev/null +++ b/tests/test_severity.py @@ -0,0 +1,176 @@ +"""The single threshold table and its classifier. + +Colour is a judgement, and a judgement made in four places drifts into four meanings. +These tests pin the boundaries (reaching a threshold trips it, in both directions), the +"unknown is not ok" rule, and the shape of the two payload helpers that put `severity` +next to a value. +""" + +from __future__ import annotations + +import json +import math + +import pytest + +from dashboards import severity +from dashboards.severity import FAIL, INVERTED, OK, THRESHOLDS, WARN, attach, classify, with_severity + +# The ratified table (chat, 2026-10-16). Edit a number here and in THRESHOLDS together — +# that coupling is the point: thresholds are a contract with the pages, not a tuning knob. +RATIFIED = { + "cpu.usage_pct": (75.0, 90.0), + "mem.used_pct": (80.0, 92.0), + "disk.used_pct": (80.0, 90.0), + "cpu.temp_c": (75.0, 90.0), + "gpu.busy_pct": (85.0, 95.0), + "gpu.mem_used_pct": (85.0, 95.0), + "llm.kv_cache_usage_pct": (75.0, 90.0), + "ups.charge_pct": (50.0, 20.0), + "ups.battery_runtime": (600.0, 300.0), + "ups.load_pct": (70.0, 90.0), +} + + +def _samples(metric: str) -> list[tuple[float, str]]: + """(value, expected) walking across the boundaries of one metric, in its own direction.""" + warn, danger = THRESHOLDS[metric] + middle = (warn + danger) / 2 + if metric in INVERTED: # lower is worse: reaching warn trips it, going below danger fails + step = 0.1 + return [ + (warn + step, OK), + (warn, WARN), + (middle, WARN), + (danger, FAIL), + (danger - 1.0, FAIL), + ] + return [ + (warn - 0.1, OK), + (warn, WARN), + (middle, WARN), + (danger, FAIL), + (danger + 1.0, FAIL), + ] + + +@pytest.mark.parametrize("metric", sorted(THRESHOLDS)) +def test_boundaries_hold_for_every_metric_in_the_table(metric: str) -> None: + for value, expected in _samples(metric): + assert classify(metric, value) == expected, f"{metric}={value} should be {expected}" + + +def test_completion_criteria_examples() -> None: + assert classify("cpu.usage_pct", 95) == "fail" + assert classify("cpu.usage_pct", 60) == "ok" + assert classify("cpu.usage_pct", None) is None + assert classify("ups.charge_pct", 15) == "fail" + + +def test_warn_edge_is_inclusive_and_just_below_is_still_good() -> None: + assert classify("cpu.usage_pct", 74.9) == "ok" + assert classify("cpu.usage_pct", 75) == "warn" + assert classify("cpu.usage_pct", 75.0001) == "warn" + assert classify("cpu.usage_pct", 89.9) == "warn" + assert classify("cpu.usage_pct", 90) == "fail" + + +def test_inverted_metrics_read_the_other_way() -> None: + """A flat battery and a flat runtime are the bad end, not the good one.""" + assert classify("ups.charge_pct", 100) == "ok" + assert classify("ups.charge_pct", 50) == "warn" + assert classify("ups.charge_pct", 19.9) == "fail" + assert classify("ups.battery_runtime", 3600) == "ok" + assert classify("ups.battery_runtime", 600) == "warn" + assert classify("ups.battery_runtime", 120) == "fail" + + +def test_a_metric_is_only_in_one_direction() -> None: + """Sanity: bounds are ordered for their direction, so an edit cannot invert a metric.""" + for metric, (warn, danger) in THRESHOLDS.items(): + assert math.isfinite(warn) and math.isfinite(danger), metric + if metric in INVERTED: + assert warn > danger, f"{metric} is inverted: warn must sit above danger" + else: + assert warn < danger, f"{metric} is not inverted: warn must sit below danger" + + +def test_inverted_is_a_subset_of_the_table() -> None: + assert set(THRESHOLDS) >= INVERTED, "an inverted metric with no thresholds has no colour" + + +def test_the_ratified_numbers_are_the_ones_in_the_table() -> None: + assert THRESHOLDS == RATIFIED + + +def test_unknown_metric_is_none_rather_than_an_exception() -> None: + assert classify("gpu.vram_used_pct", 99) is None + assert classify("nope.nope", 0) is None + assert classify("", 100) is None + + +@pytest.mark.parametrize("metric", sorted(THRESHOLDS)) +def test_none_is_never_ok(metric: str) -> None: + assert classify(metric, None) is None + + +@pytest.mark.parametrize("metric", sorted(THRESHOLDS)) +@pytest.mark.parametrize("value", [float("nan"), float("inf"), float("-inf")]) +def test_non_finite_values_are_treated_as_missing(metric: str, value: float) -> None: + assert classify(metric, value) is None + + +def test_per_device_streams_inherit_their_base_metric() -> None: + assert classify("disk.used_pct@/srv", 95) == "fail" + assert classify("disk.used_pct@/srv", 10) == "ok" + assert classify("gpu.busy_pct@0", 84) == "ok" + assert classify("gpu.busy_pct@0", 90) == "warn" + assert classify("gpu.busy_pct@1", 95) == "fail" + assert classify("cpu.usage_pct@core7", 80) == "warn" + + +def test_vocabulary_is_exactly_ok_warn_fail_or_none() -> None: + seen = {classify(metric, value) for metric in THRESHOLDS for value, _ in _samples(metric)} + assert seen == {OK, WARN, FAIL} + inverted = severity.INVERTED + assert inverted == {"ups.charge_pct", "ups.battery_runtime"} + + +def test_attach_pairs_a_plain_number_with_its_verdict() -> None: + assert attach("cpu.usage_pct", 95) == {"value": 95.0, "severity": "fail"} + assert attach("mem.used_pct", 10) == {"value": 10.0, "severity": "ok"} + assert attach("ups.load_pct", 70) == {"value": 70.0, "severity": "warn"} + + +def test_attach_keeps_the_value_when_it_has_no_opinion() -> None: + assert attach("cpu.usage_pct", None) == {"value": None, "severity": None} + assert attach("telemetry.mystery", 42) == {"value": 42.0, "severity": None} + + +def test_attach_never_puts_a_non_finite_number_in_a_payload() -> None: + for value in (float("nan"), float("inf"), float("-inf")): + assert attach("cpu.usage_pct", value) == {"value": None, "severity": None} + assert json.dumps(attach("cpu.usage_pct", float("nan"))) # would raise otherwise + + +def test_with_severity_is_additive_on_a_llm_shaped_cell() -> None: + cell = {"value": 82.0, "source": "llamacpp:kv_cache_usage_ratio", "derived": False} + out = with_severity("llm.kv_cache_usage_pct", cell) + assert out["source"] == "llamacpp:kv_cache_usage_ratio" + assert out["derived"] is False + assert out["value"] == 82.0 + assert out["severity"] == "warn" + assert cell == {"value": 82.0, "source": "llamacpp:kv_cache_usage_ratio", "derived": False}, \ + "the caller's mapping must not be mutated" + + +def test_with_severity_handles_missing_and_non_numeric_cells() -> None: + # purely additive: a missing cell gains `severity` and nothing else + assert with_severity("llm.kv_cache_usage_pct", None) == {"severity": None} + assert with_severity("llm.kv_cache_usage_pct", {"value": None}) == {"value": None, "severity": None} + assert with_severity("llm.decode_tps", {"value": "n/a", "source": None}) == { + "value": "n/a", "source": None, "severity": None, + } + # a value the table has no thresholds for still keeps its other keys + out = with_severity("llm.ttft_seconds", {"value": 1.2, "source": "computed"}) + assert out == {"value": 1.2, "source": "computed", "severity": None}