Files
dashboards/tests/card_contract.py
T
ducoterra de325caa49 phase: 06_system_page_gpu_verdicts
All criteria verified and the project is fully green.

**Verified (final pass — all 3 phase-06 tasks were already in `complete/`)**
- `classify("gpu.busy_pct", 99)` / `gpu.mem_used_pct` → `None`; `THRESHOLDS` has no `gpu.*` row; `alerts.py`/`api.py` carry no GPU verdict; live browser: deskwork GPU at **91 %** and **98 %** renders `class="hero"`, `color=rgb(22,25,31)` (plain text), no bar, no `.sev-*`.
- GPU temp on the hero `.sub` from live agents: `Radeon 8060S Graphics · VRAM 165.1 MiB / 512.0 MiB · pool 78% · 48 °C`, R9700 `· 40 °C`, gateway prints none (no `n/a`, no `0 °C`).
- Charts collapsed by default: fresh load = 3 panels / 3 toggles / **0 open / 0 `<svg>`**; expand → 3 SVGs (510 px wide), repaints across polls (legend 3.3 %→5.6 %), collapsed siblings keep their own height (803 vs 358/339).
- `gpu.busy_pct@0` still answers from `/api/system/history` (345 points) — verdict gone, data intact.

**Tests / lint / coverage**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **550 passed, 8 skipped, 97.91 %**; `uv run ruff check src tests agent` → clean; `uv run pyright` → 0 errors; `node --check` all scripts clean; `uv pip list --outdated` empty.
- `uv run pytest -m live --live` → **8 passed** (real agents/engines/upsd); dev server restarted, `/api/health` all sources `ok`.
- New screenshots: `data/logs/phase06-final-collapsed-default.png`, `data/logs/phase06-final-expanded-gpu-busy-uncoloured.png`.

**Completion criteria:** classify→None ✅ · pytest+94 % gate ✅ · ruff+pyright ✅ · validate.sh ✅ · screenshots of the three behaviours on live data ✅ · AGENTS.md severity section claims no GPU rows ✅

**Docs:** fixed 3 items — (1) `AGENTS.md` pytest count stale (546 → 550); (2) `app.js` `severityFor` docstring still claimed "`page-ups.js` still calls it (phase 05 converts it)", contradicting the code/tests/AGENTS.md (comment-only); rest of README/AGENTS reviewed in full (env table vs `config.py`+`.env.example`, agent routes incl. `/schema`, alerts rules, page descriptions, contracts) — consistent.

**Notable:** fixed a real pre-existing defect in `scripts/dev-server.sh` — `stop` slept 1 s and deleted the pidfile without waiting for exit, so `restart` with a browser open overlapped two servers on one SQLite file, producing a `database is locked` store refusal that `/system` rendered as a red event. Now waits (bounded 20 s) then SIGKILLs; verified a restart with live polling logs no such error/event. No live infra config was changed.

**Next pending phase:** `.agents/phases/todo/07_llm_monitor_style_pages`
2026-10-17 14:04:52 -04:00

119 lines
6.2 KiB
Python

"""The `/system` hero-card payload contract, shared by the hermetic and the live tests.
Phase 03 makes `/system` one panel per host with six big colour-coded numbers. The paths
below are the payload fields those cards are allowed to read; they were read out of
`parse_host_snapshot()` and are pinned hermetically in `tests/test_system_contract.py`
(driven from `tests/fixtures/agent_*.json`) and against the real fleet in
`tests/test_live_fleet.py::test_every_host_serves_the_hero_card_contract`.
Keeping the list here rather than in one test file means the live assertions cannot drift
away from the fixture assertions: there is one table, and it is the thing under test.
=========================== ==========================================================
Card Path in `/api/system/snapshot` `hosts[]`
=========================== ==========================================================
cpu % ``cpu.usage_pct`` (+ ``cpu.severity.usage_pct``)
mem % ``memory.used_pct`` (+ ``memory.severity.used_pct``)
cpu temp ``cpu.temp_c`` (+ ``cpu.severity.temp_c``)
disk usage % ``disks.worst_used_pct`` + ``disks.worst_mount`` (+ verdict)
graphics card ``gpus[].model/busy_pct/vram_*/mem_used_pct/temp_c`` (no verdict)
network down/up ``network.rx_bytes_per_s`` / ``tx_bytes_per_s`` (no verdict)
=========================== ==========================================================
CPU temp comes from ``cpu.temp_c`` rather than ``temperatures.cpu_c``: both are the agent's
own ``cpu_temperature_celsius``, and ``cpu.temp_c`` is the field ``/api/summary`` flattens
into its fleet row, so the card and the overview cannot drift apart.
Deliberately *not* card metrics (legibility, not missing data — the series stay in SQLite):
per-core usage (``cpu.usage_pct@coreN``), swap, the process table, the raw sensor list, and
per-interface rows. Network carries no ``severity`` on purpose; throughput has no good/bad,
so ``severity.THRESHOLDS`` has no ``net.*`` row. GPU rows carry no ``severity`` either
(phase 06): these are LLM servers, a 95 %-busy card is the job and not a fault, so
``THRESHOLDS`` has no ``gpu.*`` row and the card prints its numbers uncoloured at every
utilisation — the *absence* of the dict is pinned below, the way ``network``'s is.
``gpus[].temp_c`` is on the card because the number worth reading *while* a card is busy is
how hot it is (phase 06, TODO.md L9). It is the agent's **max over that card's hwmon
labels** — the same max the agent already computes and the same value stored as
``gpu.temp_c@N`` — so the hero prints a measurement rather than a new field, and the value
may honestly be ``None`` on a box whose driver exposes no temperature input.
"""
from __future__ import annotations
from typing import Any
#: Card metric -> dotted payload paths. Present is the contract; a value may be ``None``
#: on a host that genuinely does not have the number (gateway reports no CPU temperature).
CARD_METRICS: dict[str, tuple[str, ...]] = {
"cpu %": ("cpu.usage_pct", "cpu.severity.usage_pct"),
"mem %": ("memory.used_pct", "memory.severity.used_pct"),
"cpu temp": ("cpu.temp_c", "cpu.severity.temp_c", "temperatures.cpu_c"),
# the mount is what makes the percentage actionable: 88 % used — of which disk?
"disk %": ("disks.worst_used_pct", "disks.worst_mount", "disks.severity.worst_used_pct"),
"network down/up": ("network.rx_bytes_per_s", "network.tx_bytes_per_s"),
}
#: Per-GPU paths, checked on every row of `gpus[]` — and on none of them when the list is
#: empty, which is how a host without a GPU stays honest (no card, not a card at 0 % busy).
GPU_METRICS: tuple[str, ...] = (
"model",
"busy_pct",
"vram_used_bytes",
"vram_total_bytes",
"vram_used_pct",
# the honest APU view: vram + GTT, because a Strix Halo keeps the model in GTT
"mem_used_pct",
# the hero's footnote: the agent's max over this card's hwmon labels (None is legal —
# a driver with no temperature input must not be rendered as 0 °C)
"temp_c",
)
#: `network` must never grow a `severity` dict (see the module docstring).
SEVERITY_FREE_SECTIONS: tuple[str, ...] = ("network",)
#: The host shapes the captures in `tests/fixtures/` cover, and therefore the shapes a card
#: must handle: a discrete GPU, an APU (VRAM *and* GTT), and no GPU at all. A new capture
#: has to be added here — `test_the_contract_covers_every_captured_host` fails otherwise.
HOST_SHAPES: tuple[str, ...] = ("driveripper", "deskwork", "gateway")
def dig(node: Any, path: str) -> Any:
"""Follow a dotted payload path, failing with the path in the message.
A missing key is a contract break; a ``None`` value is not (that renders `n/a`), which is
why this returns the value instead of asserting on it.
"""
walked = ""
for part in path.split("."):
assert isinstance(node, dict), f"{path}: hit a non-dict at {walked or 'payload'}"
assert part in node, (
f"{path}: no key {part!r} under {walked or 'payload'} — a hero card would print "
f"n/a for a number the API used to serve"
)
node = node[part]
walked = f"{walked}.{part}" if walked else part
return node
def assert_card_contract(payload: Any) -> None:
"""Every card path exists on this host payload, GPU rows included."""
for paths in CARD_METRICS.values():
for path in paths:
dig(payload, path)
gpus = payload.get("gpus")
assert isinstance(gpus, list), "gpus must be a list — [] when the agent reports no GPU"
for gpu in gpus:
for path in GPU_METRICS:
dig(gpu, path)
assert "severity" not in gpu, (
"gpus[].severity would grade utilisation on a fleet built to be busy — "
"severity.THRESHOLDS has no gpu.* row and a busy GPU is the job, not a fault"
)
for section in SEVERITY_FREE_SECTIONS:
block = payload.get(section)
assert isinstance(block, dict), f"{section} must be a dict"
assert "severity" not in block, (
f"{section}.severity would be a colour nobody measured — throughput has no "
f"good/bad and severity.THRESHOLDS has no {section}.* row"
)