All criteria verified and the project is fully green.
**Verified (final pass — all 3 phase-06 tasks were already in `complete/`)**
- `classify("gpu.busy_pct", 99)` / `gpu.mem_used_pct` → `None`; `THRESHOLDS` has no `gpu.*` row; `alerts.py`/`api.py` carry no GPU verdict; live browser: deskwork GPU at **91 %** and **98 %** renders `class="hero"`, `color=rgb(22,25,31)` (plain text), no bar, no `.sev-*`.
- GPU temp on the hero `.sub` from live agents: `Radeon 8060S Graphics · VRAM 165.1 MiB / 512.0 MiB · pool 78% · 48 °C`, R9700 `· 40 °C`, gateway prints none (no `n/a`, no `0 °C`).
- Charts collapsed by default: fresh load = 3 panels / 3 toggles / **0 open / 0 `<svg>`**; expand → 3 SVGs (510 px wide), repaints across polls (legend 3.3 %→5.6 %), collapsed siblings keep their own height (803 vs 358/339).
- `gpu.busy_pct@0` still answers from `/api/system/history` (345 points) — verdict gone, data intact.
**Tests / lint / coverage**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **550 passed, 8 skipped, 97.91 %**; `uv run ruff check src tests agent` → clean; `uv run pyright` → 0 errors; `node --check` all scripts clean; `uv pip list --outdated` empty.
- `uv run pytest -m live --live` → **8 passed** (real agents/engines/upsd); dev server restarted, `/api/health` all sources `ok`.
- New screenshots: `data/logs/phase06-final-collapsed-default.png`, `data/logs/phase06-final-expanded-gpu-busy-uncoloured.png`.
**Completion criteria:** classify→None ✅ · pytest+94 % gate ✅ · ruff+pyright ✅ · validate.sh ✅ · screenshots of the three behaviours on live data ✅ · AGENTS.md severity section claims no GPU rows ✅
**Docs:** fixed 3 items — (1) `AGENTS.md` pytest count stale (546 → 550); (2) `app.js` `severityFor` docstring still claimed "`page-ups.js` still calls it (phase 05 converts it)", contradicting the code/tests/AGENTS.md (comment-only); rest of README/AGENTS reviewed in full (env table vs `config.py`+`.env.example`, agent routes incl. `/schema`, alerts rules, page descriptions, contracts) — consistent.
**Notable:** fixed a real pre-existing defect in `scripts/dev-server.sh` — `stop` slept 1 s and deleted the pidfile without waiting for exit, so `restart` with a browser open overlapped two servers on one SQLite file, producing a `database is locked` store refusal that `/system` rendered as a red event. Now waits (bounded 20 s) then SIGKILLs; verified a restart with live polling logs no such error/event. No live infra config was changed.
**Next pending phase:** `.agents/phases/todo/07_llm_monitor_style_pages`
119 lines
6.2 KiB
Python
119 lines
6.2 KiB
Python
"""The `/system` hero-card payload contract, shared by the hermetic and the live tests.
|
|
|
|
Phase 03 makes `/system` one panel per host with six big colour-coded numbers. The paths
|
|
below are the payload fields those cards are allowed to read; they were read out of
|
|
`parse_host_snapshot()` and are pinned hermetically in `tests/test_system_contract.py`
|
|
(driven from `tests/fixtures/agent_*.json`) and against the real fleet in
|
|
`tests/test_live_fleet.py::test_every_host_serves_the_hero_card_contract`.
|
|
|
|
Keeping the list here rather than in one test file means the live assertions cannot drift
|
|
away from the fixture assertions: there is one table, and it is the thing under test.
|
|
|
|
=========================== ==========================================================
|
|
Card Path in `/api/system/snapshot` `hosts[]`
|
|
=========================== ==========================================================
|
|
cpu % ``cpu.usage_pct`` (+ ``cpu.severity.usage_pct``)
|
|
mem % ``memory.used_pct`` (+ ``memory.severity.used_pct``)
|
|
cpu temp ``cpu.temp_c`` (+ ``cpu.severity.temp_c``)
|
|
disk usage % ``disks.worst_used_pct`` + ``disks.worst_mount`` (+ verdict)
|
|
graphics card ``gpus[].model/busy_pct/vram_*/mem_used_pct/temp_c`` (no verdict)
|
|
network down/up ``network.rx_bytes_per_s`` / ``tx_bytes_per_s`` (no verdict)
|
|
=========================== ==========================================================
|
|
|
|
CPU temp comes from ``cpu.temp_c`` rather than ``temperatures.cpu_c``: both are the agent's
|
|
own ``cpu_temperature_celsius``, and ``cpu.temp_c`` is the field ``/api/summary`` flattens
|
|
into its fleet row, so the card and the overview cannot drift apart.
|
|
|
|
Deliberately *not* card metrics (legibility, not missing data — the series stay in SQLite):
|
|
per-core usage (``cpu.usage_pct@coreN``), swap, the process table, the raw sensor list, and
|
|
per-interface rows. Network carries no ``severity`` on purpose; throughput has no good/bad,
|
|
so ``severity.THRESHOLDS`` has no ``net.*`` row. GPU rows carry no ``severity`` either
|
|
(phase 06): these are LLM servers, a 95 %-busy card is the job and not a fault, so
|
|
``THRESHOLDS`` has no ``gpu.*`` row and the card prints its numbers uncoloured at every
|
|
utilisation — the *absence* of the dict is pinned below, the way ``network``'s is.
|
|
|
|
``gpus[].temp_c`` is on the card because the number worth reading *while* a card is busy is
|
|
how hot it is (phase 06, TODO.md L9). It is the agent's **max over that card's hwmon
|
|
labels** — the same max the agent already computes and the same value stored as
|
|
``gpu.temp_c@N`` — so the hero prints a measurement rather than a new field, and the value
|
|
may honestly be ``None`` on a box whose driver exposes no temperature input.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
|
|
#: Card metric -> dotted payload paths. Present is the contract; a value may be ``None``
|
|
#: on a host that genuinely does not have the number (gateway reports no CPU temperature).
|
|
CARD_METRICS: dict[str, tuple[str, ...]] = {
|
|
"cpu %": ("cpu.usage_pct", "cpu.severity.usage_pct"),
|
|
"mem %": ("memory.used_pct", "memory.severity.used_pct"),
|
|
"cpu temp": ("cpu.temp_c", "cpu.severity.temp_c", "temperatures.cpu_c"),
|
|
# the mount is what makes the percentage actionable: 88 % used — of which disk?
|
|
"disk %": ("disks.worst_used_pct", "disks.worst_mount", "disks.severity.worst_used_pct"),
|
|
"network down/up": ("network.rx_bytes_per_s", "network.tx_bytes_per_s"),
|
|
}
|
|
|
|
#: Per-GPU paths, checked on every row of `gpus[]` — and on none of them when the list is
|
|
#: empty, which is how a host without a GPU stays honest (no card, not a card at 0 % busy).
|
|
GPU_METRICS: tuple[str, ...] = (
|
|
"model",
|
|
"busy_pct",
|
|
"vram_used_bytes",
|
|
"vram_total_bytes",
|
|
"vram_used_pct",
|
|
# the honest APU view: vram + GTT, because a Strix Halo keeps the model in GTT
|
|
"mem_used_pct",
|
|
# the hero's footnote: the agent's max over this card's hwmon labels (None is legal —
|
|
# a driver with no temperature input must not be rendered as 0 °C)
|
|
"temp_c",
|
|
)
|
|
|
|
#: `network` must never grow a `severity` dict (see the module docstring).
|
|
SEVERITY_FREE_SECTIONS: tuple[str, ...] = ("network",)
|
|
|
|
#: The host shapes the captures in `tests/fixtures/` cover, and therefore the shapes a card
|
|
#: must handle: a discrete GPU, an APU (VRAM *and* GTT), and no GPU at all. A new capture
|
|
#: has to be added here — `test_the_contract_covers_every_captured_host` fails otherwise.
|
|
HOST_SHAPES: tuple[str, ...] = ("driveripper", "deskwork", "gateway")
|
|
|
|
|
|
def dig(node: Any, path: str) -> Any:
|
|
"""Follow a dotted payload path, failing with the path in the message.
|
|
|
|
A missing key is a contract break; a ``None`` value is not (that renders `n/a`), which is
|
|
why this returns the value instead of asserting on it.
|
|
"""
|
|
walked = ""
|
|
for part in path.split("."):
|
|
assert isinstance(node, dict), f"{path}: hit a non-dict at {walked or 'payload'}"
|
|
assert part in node, (
|
|
f"{path}: no key {part!r} under {walked or 'payload'} — a hero card would print "
|
|
f"n/a for a number the API used to serve"
|
|
)
|
|
node = node[part]
|
|
walked = f"{walked}.{part}" if walked else part
|
|
return node
|
|
|
|
|
|
def assert_card_contract(payload: Any) -> None:
|
|
"""Every card path exists on this host payload, GPU rows included."""
|
|
for paths in CARD_METRICS.values():
|
|
for path in paths:
|
|
dig(payload, path)
|
|
gpus = payload.get("gpus")
|
|
assert isinstance(gpus, list), "gpus must be a list — [] when the agent reports no GPU"
|
|
for gpu in gpus:
|
|
for path in GPU_METRICS:
|
|
dig(gpu, path)
|
|
assert "severity" not in gpu, (
|
|
"gpus[].severity would grade utilisation on a fleet built to be busy — "
|
|
"severity.THRESHOLDS has no gpu.* row and a busy GPU is the job, not a fault"
|
|
)
|
|
for section in SEVERITY_FREE_SECTIONS:
|
|
block = payload.get(section)
|
|
assert isinstance(block, dict), f"{section} must be a dict"
|
|
assert "severity" not in block, (
|
|
f"{section}.severity would be a colour nobody measured — throughput has no "
|
|
f"good/bad and severity.THRESHOLDS has no {section}.* row"
|
|
)
|