phase: 06_system_page_gpu_verdicts
All criteria verified and the project is fully green.
**Verified (final pass — all 3 phase-06 tasks were already in `complete/`)**
- `classify("gpu.busy_pct", 99)` / `gpu.mem_used_pct` → `None`; `THRESHOLDS` has no `gpu.*` row; `alerts.py`/`api.py` carry no GPU verdict; live browser: deskwork GPU at **91 %** and **98 %** renders `class="hero"`, `color=rgb(22,25,31)` (plain text), no bar, no `.sev-*`.
- GPU temp on the hero `.sub` from live agents: `Radeon 8060S Graphics · VRAM 165.1 MiB / 512.0 MiB · pool 78% · 48 °C`, R9700 `· 40 °C`, gateway prints none (no `n/a`, no `0 °C`).
- Charts collapsed by default: fresh load = 3 panels / 3 toggles / **0 open / 0 `<svg>`**; expand → 3 SVGs (510 px wide), repaints across polls (legend 3.3 %→5.6 %), collapsed siblings keep their own height (803 vs 358/339).
- `gpu.busy_pct@0` still answers from `/api/system/history` (345 points) — verdict gone, data intact.
**Tests / lint / coverage**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **550 passed, 8 skipped, 97.91 %**; `uv run ruff check src tests agent` → clean; `uv run pyright` → 0 errors; `node --check` all scripts clean; `uv pip list --outdated` empty.
- `uv run pytest -m live --live` → **8 passed** (real agents/engines/upsd); dev server restarted, `/api/health` all sources `ok`.
- New screenshots: `data/logs/phase06-final-collapsed-default.png`, `data/logs/phase06-final-expanded-gpu-busy-uncoloured.png`.
**Completion criteria:** classify→None ✅ · pytest+94 % gate ✅ · ruff+pyright ✅ · validate.sh ✅ · screenshots of the three behaviours on live data ✅ · AGENTS.md severity section claims no GPU rows ✅
**Docs:** fixed 3 items — (1) `AGENTS.md` pytest count stale (546 → 550); (2) `app.js` `severityFor` docstring still claimed "`page-ups.js` still calls it (phase 05 converts it)", contradicting the code/tests/AGENTS.md (comment-only); rest of README/AGENTS reviewed in full (env table vs `config.py`+`.env.example`, agent routes incl. `/schema`, alerts rules, page descriptions, contracts) — consistent.
**Notable:** fixed a real pre-existing defect in `scripts/dev-server.sh` — `stop` slept 1 s and deleted the pidfile without waiting for exit, so `restart` with a browser open overlapped two servers on one SQLite file, producing a `database is locked` store refusal that `/system` rendered as a red event. Now waits (bounded 20 s) then SIGKILLs; verified a restart with live polling logs no such error/event. No live infra config was changed.
**Next pending phase:** `.agents/phases/todo/07_llm_monitor_style_pages`
This commit is contained in:
+18
-4
@@ -16,7 +16,7 @@ cpu % ``cpu.usage_pct`` (+ ``cpu.severity.usage_pct``)
|
||||
mem % ``memory.used_pct`` (+ ``memory.severity.used_pct``)
|
||||
cpu temp ``cpu.temp_c`` (+ ``cpu.severity.temp_c``)
|
||||
disk usage % ``disks.worst_used_pct`` + ``disks.worst_mount`` (+ verdict)
|
||||
graphics card ``gpus[].model/busy_pct/vram_*/mem_used_pct`` (+ verdict)
|
||||
graphics card ``gpus[].model/busy_pct/vram_*/mem_used_pct/temp_c`` (no verdict)
|
||||
network down/up ``network.rx_bytes_per_s`` / ``tx_bytes_per_s`` (no verdict)
|
||||
=========================== ==========================================================
|
||||
|
||||
@@ -27,7 +27,16 @@ into its fleet row, so the card and the overview cannot drift apart.
|
||||
Deliberately *not* card metrics (legibility, not missing data — the series stay in SQLite):
|
||||
per-core usage (``cpu.usage_pct@coreN``), swap, the process table, the raw sensor list, and
|
||||
per-interface rows. Network carries no ``severity`` on purpose; throughput has no good/bad,
|
||||
so ``severity.THRESHOLDS`` has no ``net.*`` row.
|
||||
so ``severity.THRESHOLDS`` has no ``net.*`` row. GPU rows carry no ``severity`` either
|
||||
(phase 06): these are LLM servers, a 95 %-busy card is the job and not a fault, so
|
||||
``THRESHOLDS`` has no ``gpu.*`` row and the card prints its numbers uncoloured at every
|
||||
utilisation — the *absence* of the dict is pinned below, the way ``network``'s is.
|
||||
|
||||
``gpus[].temp_c`` is on the card because the number worth reading *while* a card is busy is
|
||||
how hot it is (phase 06, TODO.md L9). It is the agent's **max over that card's hwmon
|
||||
labels** — the same max the agent already computes and the same value stored as
|
||||
``gpu.temp_c@N`` — so the hero prints a measurement rather than a new field, and the value
|
||||
may honestly be ``None`` on a box whose driver exposes no temperature input.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -54,8 +63,9 @@ GPU_METRICS: tuple[str, ...] = (
|
||||
"vram_used_pct",
|
||||
# the honest APU view: vram + GTT, because a Strix Halo keeps the model in GTT
|
||||
"mem_used_pct",
|
||||
"severity.busy_pct",
|
||||
"severity.mem_used_pct",
|
||||
# the hero's footnote: the agent's max over this card's hwmon labels (None is legal —
|
||||
# a driver with no temperature input must not be rendered as 0 °C)
|
||||
"temp_c",
|
||||
)
|
||||
|
||||
#: `network` must never grow a `severity` dict (see the module docstring).
|
||||
@@ -95,6 +105,10 @@ def assert_card_contract(payload: Any) -> None:
|
||||
for gpu in gpus:
|
||||
for path in GPU_METRICS:
|
||||
dig(gpu, path)
|
||||
assert "severity" not in gpu, (
|
||||
"gpus[].severity would grade utilisation on a fleet built to be busy — "
|
||||
"severity.THRESHOLDS has no gpu.* row and a busy GPU is the job, not a fault"
|
||||
)
|
||||
for section in SEVERITY_FREE_SECTIONS:
|
||||
block = payload.get(section)
|
||||
assert isinstance(block, dict), f"{section} must be a dict"
|
||||
|
||||
+14
-7
@@ -216,21 +216,28 @@ def test_a_number_the_upstream_does_not_publish_is_never_a_breach(
|
||||
assert alerts.build(SETTINGS, [state()], [payload]) == []
|
||||
|
||||
|
||||
def test_a_gpu_breach_names_the_card_and_dedupes_per_metric(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
def test_a_saturated_gpu_is_not_an_alert(host_scrape: Any,
|
||||
agent_snapshot: dict[str, Any]) -> None:
|
||||
"""Phase 06: these are LLM servers — a fully-worked GPU is the job, not a fault.
|
||||
|
||||
The host is otherwise all-clear, so a card at 100 % busy with both memory pools
|
||||
filled must produce **no** alert at all: `severity.THRESHOLDS` has no `gpu.*` row,
|
||||
the payload carries no GPU verdict, and this module owns no threshold of its own
|
||||
from which to resurrect one.
|
||||
"""
|
||||
def fill_the_card(snap: dict[str, Any]) -> None:
|
||||
gpu = snap["gpus"][0]
|
||||
gpu["busy_pct"] = 97.0
|
||||
gpu["busy_pct"] = 100.0
|
||||
for used, total in (("vram_used_bytes", "vram_total_bytes"),
|
||||
("gtt_used_bytes", "gtt_total_bytes")):
|
||||
if gpu.get(total) is not None:
|
||||
gpu[used] = gpu[total]
|
||||
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=10.0, mutate=fill_the_card)
|
||||
messages = {item["id"]: item["message"] for item in alerts.build(SETTINGS, [state()], [payload])}
|
||||
assert any("GPU utilisation 97% on Radeon AI PRO R9700" in text
|
||||
for text in messages.values()), messages
|
||||
assert any("GPU memory" in text and "R9700" in text for text in messages.values())
|
||||
assert payload["gpus"][0]["busy_pct"] == 100.0, "the card really is saturated here"
|
||||
assert alerts.build(SETTINGS, [state()], [payload]) == [], (
|
||||
"a busy GPU produced an alert — GPU verdicts were removed in phase 06"
|
||||
)
|
||||
|
||||
|
||||
def test_a_degraded_agent_warns_and_names_the_sections_that_failed(
|
||||
|
||||
+146
-6
@@ -161,10 +161,11 @@ def test_summary_carries_the_verdict_for_every_column_the_page_colours(data_clie
|
||||
row = body["system"]["hosts"][0]
|
||||
assert row["severity"] == {"cpu_usage_pct": "ok", "mem_used_pct": "fail",
|
||||
"disk_worst_used_pct": "fail", "temp_c": "fail"}
|
||||
# the fleet card is judged by the same table, so amber means the same thing at both sizes
|
||||
# the fleet card is judged by the same table, so amber means the same thing at both
|
||||
# sizes — and no GPU verdict, at fleet size or host size (phase 06: busy is the job)
|
||||
assert body["system"]["fleet"]["severity"] == {
|
||||
"cpu_usage_pct_avg": "ok", "memory_used_pct": "fail",
|
||||
"disk_worst_used_pct": "fail", "gpu_busy_pct_avg": "ok",
|
||||
"disk_worst_used_pct": "fail",
|
||||
}
|
||||
for ups in body["ups"]["ups"]:
|
||||
assert set(ups["severity"]) == {"charge_pct", "load_pct", "runtime_seconds"}
|
||||
@@ -311,7 +312,9 @@ def test_snapshot_payloads_carry_their_own_severity_sections(data_client) -> Non
|
||||
assert set(host["cpu"]["severity"]) == {"usage_pct", "temp_c"}
|
||||
assert set(host["memory"]["severity"]) == {"used_pct"}
|
||||
assert set(host["disks"]["severity"]) == {"worst_used_pct"}
|
||||
assert set(host["gpus"][0]["severity"]) == {"busy_pct", "mem_used_pct"}
|
||||
assert "severity" not in host["gpus"][0], (
|
||||
"a GPU verdict came back on /api/system/snapshot — phase 06 removed it"
|
||||
)
|
||||
ups = data_client.get("/api/ups/snapshot").json()
|
||||
for card in ups["ups"]:
|
||||
assert set(card["severity"]) == {"charge_pct", "load_pct", "battery_runtime"}
|
||||
@@ -603,17 +606,48 @@ def test_system_page_colours_from_the_payload_not_from_js_thresholds(client) ->
|
||||
|
||||
Phase 03 task 02's promise: every colour on a host card comes from a `severity` dict
|
||||
`severity.py` produced (`cpu.severity.usage_pct`, `memory.severity.used_pct`,
|
||||
`disks.severity.worst_used_pct`, `gpus[].severity.busy_pct`, `cpu.severity.temp_c`).
|
||||
`disks.severity.worst_used_pct`, `cpu.severity.temp_c`).
|
||||
A threshold written here could disagree with the overview's alerts; a hero with no
|
||||
verdict prints `n/a` uncoloured, which is `Dash.hero()`'s contract.
|
||||
|
||||
The GPU slot names no verdict on purpose (phase 06): `severity.THRESHOLDS` has no
|
||||
`gpu.*` row, the payload carries no `gpus[].severity`, and the hero is built without
|
||||
`severity` or `bar` — a bar under a verdictless number would be a verdict by shape.
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
assert "severityFor(" not in js, "the system page re-invented a threshold"
|
||||
assert "Dash.hero(" in js
|
||||
for hook in ("cpu.severity.usage_pct", "memory.severity.used_pct",
|
||||
"disks.severity.worst_used_pct", "gpus[].severity.busy_pct",
|
||||
"cpu.severity.temp_c"):
|
||||
"disks.severity.worst_used_pct", "cpu.severity.temp_c"):
|
||||
assert hook in js, f"the card no longer names {hook} as a colour source"
|
||||
# and the GPU hero grades nothing: no verdict hook, no utilisation bar
|
||||
assert "gpus[].severity" not in js, "the GPU slot invented a colour source again"
|
||||
gpu_slot = js[js.index("function renderGpuSlot("):js.index("function renderGpuExtra(")]
|
||||
assert "severity:" not in gpu_slot, "the GPU hero passes a severity option again"
|
||||
assert "bar:" not in gpu_slot, "the GPU hero grew a bar back (a verdict by shape)"
|
||||
assert "hero-bar" not in gpu_slot, "the GPU hero rebuilt the bar variant of .hero"
|
||||
|
||||
|
||||
def test_gpu_hero_footnote_prints_the_temperature_the_agent_read(client) -> None:
|
||||
"""TODO.md L9: the GPU temperature is glanceable on the hero row, not only in the table.
|
||||
|
||||
`gpus[].temp_c` is the agent's max over that card's hwmon labels (`sources/host.py`,
|
||||
pinned by `tests/test_system_contract.py`), so the hero needed no new field and the
|
||||
`gpu.temp_c@N` history series kept its name. The null guard *is* the behaviour, so that
|
||||
is what the assertion pins (there is no JS test runner — a locked anchor): a card whose
|
||||
driver exposes no temperature prints none, because `n/a` is for numbers the card is
|
||||
supposed to have and a `0 °C` would be a cool GPU that does not exist.
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
sub = js[js.index("function gpuSub("):js.index("function renderGpuSlot(")]
|
||||
assert "gpu.temp_c" in sub, "the GPU hero stopped printing the temperature"
|
||||
assert "if (gpu.temp_c != null)" in sub, "the temperature lost its null guard"
|
||||
assert sub.count("°C") == 1, (
|
||||
"a second °C in gpuSub is a branch printing a temperature nobody measured"
|
||||
)
|
||||
# "More detail" keeps the per-label breakdown: a surface change, not a de-duplication
|
||||
extra = js[js.index("function renderGpuExtra("):js.index("function renderHostCard(")]
|
||||
assert "Temp" in extra and "gpu.temp_c" in extra, "GPU detail lost its temperature column"
|
||||
|
||||
|
||||
def test_system_host_card_is_six_heroes_without_per_core(client) -> None:
|
||||
@@ -709,6 +743,112 @@ def test_every_system_chart_is_drawn_inside_its_own_host_panel(client) -> None:
|
||||
"a host that left the payload keeps a live panel and drawing charts"
|
||||
|
||||
|
||||
def test_system_history_charts_are_collapsed_behind_a_toggle(client) -> None:
|
||||
"""TODO.md L4: "The historical charts on the system resources page should only be visible
|
||||
if expanded. I only want to see current stats to keep things clean."
|
||||
|
||||
The default `/system` view is the hero row. Pinned on the served script because the card
|
||||
markup is generated in JS and there is no JS test runner (a locked anchor), and the two
|
||||
ways this regresses are invisible otherwise: a `<details open>` (charts visible by
|
||||
default, i.e. the task un-done but the page still pretty) and a paint pass that runs
|
||||
while the stack is shut, which builds three `LineChart`s per host inside a
|
||||
`display:none` container — measured 0 px wide, never drawn, and rebuilt every poll.
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
card = js[js.index("function hostCard("):js.index("function memorySub(")]
|
||||
|
||||
# the stack is *wrapped*, not merely preceded by a heading: it lives inside the <details>
|
||||
box = card.index('<details class="charts-box" data-k="charts">')
|
||||
stack = card.index('<div class="host-charts">')
|
||||
closed = card.index("</details>")
|
||||
assert box < stack < closed, "the chart stack is not inside the toggle"
|
||||
assert closed < card.index("More detail"), "the charts toggle swallowed the detail row too"
|
||||
assert not re.search(r"<details[^>]*\bopen\b", card), \
|
||||
"a panel ships its charts open — the default view is not current stats"
|
||||
assert re.search(r"<summary[^>]*>\s*Charts", card), "the toggle no longer reads Charts"
|
||||
# it also says which window the expanded stack draws (the chips stay in the page head)
|
||||
assert 'data-k="charts-window"' in card, "the toggle does not carry the window"
|
||||
# …and the toggle is what turns the paint pass on
|
||||
assert "addEventListener('toggle'" in card and "chartsToggled(" in card
|
||||
|
||||
setter = js[js.index("function setChartsWindow("):js.index("function memorySub(")]
|
||||
assert "state.rangeKey" in setter, \
|
||||
"the toggle re-derives its window label instead of using the chips' own"
|
||||
handler = js[js.index("Dash.rangeControl("):js.index("state.poller = poll(")]
|
||||
assert "state.rangeKey = key" in handler and "setChartsWindow(" in handler, \
|
||||
"changing the window leaves every toggle labelling the old one"
|
||||
|
||||
# nothing is built or drawn while the stack is shut, in either path that could paint
|
||||
paint = js[js.index("function drawHostCharts("):js.index("async function renderHostCharts(")]
|
||||
assert "if (!chartsOpen(card)) return" in paint, "a closed panel gets painted anyway"
|
||||
assert "hostChartSet(card, host)" in paint, \
|
||||
"the LineChart instances are no longer built by the open pass"
|
||||
toggle = js[js.index("function chartsToggled("):js.index("function drawHostCharts(")]
|
||||
assert "if (!chartsOpen(card)) return" in toggle, "closing the stack still paints"
|
||||
assert "renderHostCharts(card, host)" in toggle and "state.range" in toggle, \
|
||||
"a cached response from another window is drawn under this window's label"
|
||||
|
||||
# the request discipline is unchanged: one per host per refresh, every series named
|
||||
fetch = js[js.index("async function renderHostCharts("):js.index("function windowLabel(")]
|
||||
assert "hostChartSet(" not in fetch, "the poll path still builds charts for closed panels"
|
||||
assert "state.history.set(" in fetch, \
|
||||
"the response is not kept, so the first open is an empty box until the next poll"
|
||||
assert "if (range === state.range) drawHostCharts(" in fetch, \
|
||||
"a response for a window the operator just left is painted under the new label"
|
||||
assert "drawHostCharts(" in fetch, "an open panel is no longer repainted on refresh"
|
||||
|
||||
|
||||
def test_a_pruned_host_takes_its_charts_toggle_with_it(client) -> None:
|
||||
"""A host that leaves the payload must leave no orphan control in the grid.
|
||||
|
||||
The toggle is a child of the panel, which is why `card.remove()` is enough — but that is
|
||||
only true while the markup is built inside `hostCard()`, so the placement is pinned here
|
||||
along with the three per-host Maps the panel owns.
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
card = js[js.index("function hostCard("):js.index("function memorySub(")]
|
||||
assert 'data-k="charts"' in card, "the toggle is no longer part of the panel"
|
||||
pruner = js[js.index("function pruneHosts("):js.index("// ---- snapshot")]
|
||||
assert "state.hosts.get(name).remove()" in pruner
|
||||
for owned in ("state.charts", "state.payloads", "state.history"):
|
||||
assert f"{owned}.delete(name)" in pruner, f"{owned} outlives the panel it belongs to"
|
||||
|
||||
|
||||
def test_the_charts_toggle_is_a_quiet_disclosure_not_a_new_component(client) -> None:
|
||||
"""The collapse adds no palette and no second vocabulary (UI conventions).
|
||||
|
||||
Open/closed is carried by a caret *shape* as well as a colour step, the summary is muted
|
||||
rather than accented, and it introduces no `.sev-*` family — the same rule the hero
|
||||
numbers live by, checked here because a `<summary>` is the first control on the page a
|
||||
designer might be tempted to grade.
|
||||
"""
|
||||
css = client.get("/static/dash.css").text
|
||||
block = css[css.index("details.charts-box { "):css.index(".host-charts {")]
|
||||
assert "details.charts-box > summary::before" in block, "the toggle has no caret shape"
|
||||
assert 'details.charts-box[open] > summary::before' in block, \
|
||||
"open and closed differ only in colour"
|
||||
assert "var(--text-faint)" in block, "the closed toggle is not muted"
|
||||
assert ".sev-" not in block and "var(--accent)" not in block, "the toggle invented a palette"
|
||||
assert 'class="charts-box"' in client.get("/static/page-system.js").text, \
|
||||
"the stylesheet styles a class the page no longer renders"
|
||||
|
||||
|
||||
def test_an_expanded_chart_stack_does_not_grow_its_collapsed_siblings(client) -> None:
|
||||
"""One host open must not stretch the hosts beside it into blank cards.
|
||||
|
||||
`/system` is the same shape `/llm` had to fix: an expanded panel beside a collapsed one
|
||||
grows the collapsed card to match, and a blank panel reads as "no data" (the misreading
|
||||
the unconfigured rules exist to prevent). `.grid.items-start` already exists for exactly
|
||||
this, so the page uses it rather than growing a rule of its own.
|
||||
"""
|
||||
html = _code(client.get("/system").text)
|
||||
hosts_class = re.findall(r'<div class="([^"]*)" id="hosts">', html)
|
||||
assert hosts_class == ["grid auto items-start"], \
|
||||
f"the host grid stretches its cards again: {hosts_class}"
|
||||
assert ".grid.items-start" in client.get("/static/dash.css").text, \
|
||||
"the rule the host grid relies on is gone"
|
||||
|
||||
|
||||
def test_every_system_chart_names_itself_for_assistive_tech(client) -> None:
|
||||
"""A 120px SVG is silence to a screen reader unless the page names it.
|
||||
|
||||
|
||||
@@ -248,12 +248,8 @@ def test_payloads_carry_a_severity_beside_every_rated_value(agent_snapshot) -> N
|
||||
assert payload["cpu"]["severity"]["temp_c"] == "ok" # 68 °C in the capture
|
||||
assert payload["memory"]["severity"]["used_pct"] == "ok" # 68 % of 128 GB
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] == "ok" # 74.6 % worst mount
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["severity"] == {
|
||||
"busy_pct": classify("gpu.busy_pct", gpu["busy_pct"]),
|
||||
"mem_used_pct": classify("gpu.mem_used_pct", gpu["mem_used_pct"]),
|
||||
}
|
||||
assert gpu["severity"]["busy_pct"] == "ok"
|
||||
# GPU rows carry no verdict at all (phase 06): busy is the job on an LLM server
|
||||
assert "severity" not in payload["gpus"][0]
|
||||
|
||||
|
||||
def test_severity_trips_on_the_ratified_boundaries(agent_snapshot) -> None:
|
||||
@@ -263,14 +259,14 @@ def test_severity_trips_on_the_ratified_boundaries(agent_snapshot) -> None:
|
||||
snap["cpu_temperature_celsius"] = 90.0 # cpu.temp_c danger
|
||||
for fs in snap["filesystems"]:
|
||||
fs["used_pct"] = 80.0 # disk.used_pct warn
|
||||
snap["gpus"][0]["busy_pct"] = 95.0 # gpu.busy_pct danger
|
||||
snap["gpus"][0]["gtt_used_bytes"] = 65e9 # whole-pool usage well past danger
|
||||
snap["gpus"][0]["busy_pct"] = 100.0 # no gpu.* row exists to trip, at any level
|
||||
snap["gpus"][0]["gtt_used_bytes"] = 65e9 # whole-pool usage well past the old bounds
|
||||
payload = parse_host_snapshot("driveripper", snap, make_collector(), now=1_000.0).payload
|
||||
assert payload["memory"]["severity"]["used_pct"] == "fail"
|
||||
assert payload["cpu"]["severity"]["temp_c"] == "fail"
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] == "warn"
|
||||
assert payload["gpus"][0]["severity"]["busy_pct"] == "fail"
|
||||
assert payload["gpus"][0]["severity"]["mem_used_pct"] == "fail"
|
||||
assert payload["gpus"][0]["busy_pct"] == 100.0, "the numbers stay; only the verdict left"
|
||||
assert "severity" not in payload["gpus"][0], "a GPU verdict row came back"
|
||||
|
||||
|
||||
def test_a_value_the_agent_did_not_report_gets_no_colour(agent_snapshot) -> None:
|
||||
@@ -282,7 +278,7 @@ def test_a_value_the_agent_did_not_report_gets_no_colour(agent_snapshot) -> None
|
||||
payload = parse_host_snapshot("driveripper", snap, make_collector(), now=1_000.0).payload
|
||||
assert payload["cpu"]["severity"]["temp_c"] is None
|
||||
assert payload["cpu"]["severity"]["usage_pct"] is None, "one scrape is not a rate"
|
||||
assert payload["gpus"][0]["severity"]["busy_pct"] is None
|
||||
assert payload["gpus"][0]["busy_pct"] is None
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] is None
|
||||
# the numbers themselves are unchanged by the verdict sitting next to them
|
||||
assert payload["memory"]["used_pct"] == pytest.approx(
|
||||
|
||||
+17
-5
@@ -23,8 +23,6 @@ RATIFIED = {
|
||||
"mem.used_pct": (80.0, 92.0),
|
||||
"disk.used_pct": (80.0, 90.0),
|
||||
"cpu.temp_c": (75.0, 90.0),
|
||||
"gpu.busy_pct": (85.0, 95.0),
|
||||
"gpu.mem_used_pct": (85.0, 95.0),
|
||||
"llm.kv_cache_usage_pct": (75.0, 90.0),
|
||||
"ups.charge_pct": (50.0, 20.0),
|
||||
"ups.battery_runtime": (600.0, 300.0),
|
||||
@@ -109,6 +107,22 @@ def test_unknown_metric_is_none_rather_than_an_exception() -> None:
|
||||
assert classify("", 100) is None
|
||||
|
||||
|
||||
def test_gpu_usage_carries_no_verdict_at_any_level() -> None:
|
||||
"""Phase 06: these are LLM servers — a busy GPU is the job, not a fault.
|
||||
|
||||
The table has **no `gpu.*` row at all** (not a raised ceiling): every utilisation
|
||||
level classifies to `None`, so the card renders uncoloured rather than inventing
|
||||
amber for the workload the machines exist to run. Deleting the row is not deleting
|
||||
the data — the `gpu.busy_pct@N` series are still stored and charted.
|
||||
"""
|
||||
assert not any(metric.startswith("gpu.") for metric in THRESHOLDS)
|
||||
assert classify("gpu.busy_pct", 0) is None
|
||||
assert classify("gpu.busy_pct", 99) is None
|
||||
assert classify("gpu.mem_used_pct", 99) is None
|
||||
assert classify("gpu.mem_used_pct", 100) is None
|
||||
assert classify("gpu.busy_pct@0", 95) is None, "streams inherit the absent row: nothing"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("metric", sorted(THRESHOLDS))
|
||||
def test_none_is_never_ok(metric: str) -> None:
|
||||
assert classify(metric, None) is None
|
||||
@@ -123,10 +137,8 @@ def test_non_finite_values_are_treated_as_missing(metric: str, value: float) ->
|
||||
def test_per_device_streams_inherit_their_base_metric() -> None:
|
||||
assert classify("disk.used_pct@/srv", 95) == "fail"
|
||||
assert classify("disk.used_pct@/srv", 10) == "ok"
|
||||
assert classify("gpu.busy_pct@0", 84) == "ok"
|
||||
assert classify("gpu.busy_pct@0", 90) == "warn"
|
||||
assert classify("gpu.busy_pct@1", 95) == "fail"
|
||||
assert classify("cpu.usage_pct@core7", 80) == "warn"
|
||||
assert classify("mem.used_pct@any", 95) == "fail"
|
||||
|
||||
|
||||
def test_vocabulary_is_exactly_ok_warn_fail_or_none() -> None:
|
||||
|
||||
@@ -143,6 +143,9 @@ def test_the_discrete_gpu_capture_supplies_real_numbers(
|
||||
assert gpu["busy_pct"] == pytest.approx(snapshot["gpus"][0]["busy_pct"])
|
||||
assert gpu["vram_used_pct"] == pytest.approx(
|
||||
gpu["vram_used_bytes"] / gpu["vram_total_bytes"] * 100.0)
|
||||
# the hero's footnote is the agent's own max over this card's hwmon labels (phase 06),
|
||||
# so the card and the `gpu.temp_c@N` history series can never disagree
|
||||
assert gpu["temp_c"] == max(snapshot["gpus"][0]["temps"].values())
|
||||
|
||||
# one scrape is not a rate: usage and throughput are honestly n/a until the second
|
||||
assert payload["cpu"]["usage_pct"] is None
|
||||
@@ -177,11 +180,34 @@ def test_each_card_number_travels_with_its_own_verdict(host_scrape: Any,
|
||||
"mem.used_pct", payload["memory"]["used_pct"])
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] == classify(
|
||||
"disk.used_pct", payload["disks"]["worst_used_pct"])
|
||||
|
||||
|
||||
def test_the_gpu_card_carries_no_verdict_at_all(host_scrape: Any,
|
||||
agent_snapshot: dict[str, Any]) -> None:
|
||||
"""Phase 06: a fully-worked GPU is the job on an LLM server, not a fault.
|
||||
|
||||
The card prints busy % / VRAM with no colour class at any level, so the payload
|
||||
carries no `severity` dict to colour from — the absence is the contract (pushed past
|
||||
every old bound in the fixture, so a resurrected threshold row would trip and fail).
|
||||
"""
|
||||
def work_the_card(snap: dict[str, Any]) -> None:
|
||||
gpu = snap["gpus"][0]
|
||||
gpu["busy_pct"] = 100.0
|
||||
for used, total in (("vram_used_bytes", "vram_total_bytes"),
|
||||
("gtt_used_bytes", "gtt_total_bytes")):
|
||||
if gpu.get(total) is not None:
|
||||
gpu[used] = gpu[total]
|
||||
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=33.0, mutate=work_the_card)
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["severity"] == {
|
||||
"busy_pct": classify("gpu.busy_pct", gpu["busy_pct"]),
|
||||
"mem_used_pct": classify("gpu.mem_used_pct", gpu["mem_used_pct"]),
|
||||
}
|
||||
assert gpu["busy_pct"] == 100.0, "the numbers themselves stay on the card"
|
||||
assert gpu["mem_used_pct"] > 90.0, "the whole pool is worked too"
|
||||
assert "severity" not in gpu, (
|
||||
"a GPU verdict came back — classify('gpu.busy_pct', …) must be None and the row "
|
||||
"must not carry a dict for the page to colour from"
|
||||
)
|
||||
assert classify("gpu.busy_pct", gpu["busy_pct"]) is None
|
||||
assert classify("gpu.mem_used_pct", gpu["mem_used_pct"]) is None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -208,6 +234,26 @@ def test_the_apu_reports_vram_and_the_whole_addressable_pool(
|
||||
assert gpu["model"], "the card needs a name to put beside the numbers"
|
||||
|
||||
|
||||
def test_a_gpu_with_no_temperature_input_reports_none_not_zero(agent_snapshot: dict[str, Any],
|
||||
) -> None:
|
||||
"""Phase 06 put `gpus[].temp_c` on the hero footnote, so its absence must stay honest.
|
||||
|
||||
A driver exposing no hwmon temperature has to come back as `None` (the page prints no
|
||||
temperature at all) rather than `0 °C`, the same rule
|
||||
`tests/test_host_source.py::test_missing_sensors_stay_missing` holds for the flat sensor
|
||||
list — and no `gpu.temp_c@N` sample may be written for a number nobody measured.
|
||||
"""
|
||||
snapshot = copy.deepcopy(agent_snapshot)
|
||||
snapshot["gpus"][0]["temps"] = {}
|
||||
result = parse_host_snapshot("driveripper", snapshot, make_collector(), now=1_000.0)
|
||||
gpu = result.payload["gpus"][0]
|
||||
assert gpu["temp_c"] is None, "an unreadable sensor is not a cold card"
|
||||
assert gpu["temps"] == {}
|
||||
assert_card_contract(result.payload)
|
||||
assert not [s for s in result.samples if s.metric.startswith("gpu.temp_c@")], (
|
||||
"a None temperature must not reach the history as a number")
|
||||
|
||||
|
||||
def test_paths_survive_a_host_that_reports_almost_nothing(agent_snapshot: dict[str, Any]) -> None:
|
||||
"""Keys present, values None: the card must render `n/a` without a KeyError."""
|
||||
snapshot = copy.deepcopy(agent_snapshot)
|
||||
@@ -229,9 +275,10 @@ def test_paths_survive_a_host_that_reports_almost_nothing(agent_snapshot: dict[s
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] is None
|
||||
assert payload["network"]["rx_bytes_per_s"] is None
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["busy_pct"] is None and gpu["severity"]["busy_pct"] is None
|
||||
assert gpu["busy_pct"] is None
|
||||
assert gpu["vram_used_pct"] is None, "no total, so no percentage — not 0 %"
|
||||
assert gpu["mem_used_pct"] is None
|
||||
assert gpu["temp_c"] is not None, "the card still has its temperature to print"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
Reference in New Issue
Block a user