phase: 03_system_per_host_panels
**Phase 03 final verification pass — all 4 tasks already in `complete/`; verified criteria, fixed 3 pre-existing defects.**
- **Verified live** (headless Chromium + curl against `:8090`, all 6 sources `ok`): 3 panels × six heroes in order `cpu,mem,temp,disk,gpus,net`; charts per panel `util,net[,gpu]` (gateway: no GPU chart/hole); legends name metrics only — no legend names a foreign host; every `<svg>` `role="img"` + host-scoped `aria-label`; 2 sections, 0 `<select>`; `grep -rn "renderCores|core_usage" src/dashboards/web/` → nothing; console error-free; new shot `data/logs/final-system-panels.png`.
- **Defect fixed (credential hygiene, invariant 6)**: `collectors.py` put the *dialed* URL into every `UpstreamError`, and that string lands in the health record (`/api/health`) and the `event` table (`/api/events` → page); `/api/config` also echoed `GUFO_BASE_URL`/`VLLM_BASE_URL` verbatim. Added `config.strip_userinfo()` (now what `Target.display_url` uses) and redacted all message/config URLs. Tests: `test_a_failing_host_never_echoes_its_token_in_the_message` (×3 paths), `test_a_failing_engine_never_echoes_credentials_in_its_url`, `test_a_dead_host_never_writes_its_token_into_the_feed`, `test_engine_urls_are_redacted_the_same_way_hosts_are`, `test_strip_userinfo_is_the_one_redaction_every_url_leaves_by`. (`parse_targets` already moves `HOST_TARGETS` userinfo into a header, so this is a backstop for hosts and a real fix for engine URLs.)
- **Defect fixed (invariant 2)**: `/system` footnote printed `Δ{HOST_INTERVAL}` as the measurement; it now prints each host's measured `window_seconds` (range when they disagree) — `rates over Δ10.0s · poll 10s`. Pinned by `test_system_footnote_prints_the_window_it_actually_measured`. Also fixed the page's stale "window in each header" note and the empty-state's `--token` (argv) advice → `--token-file`.
- `bash .agents/validate.sh` → **EXIT=0**: **472 passed, 7 skipped**, coverage **96.87%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` clean · `uv run pytest -m live --live` → **7 passed** · `uv pip list --outdated` empty.
- Criteria: no multi-host legend ✅ · no per-core visual ✅ · six heroes/panel incl. no-GPU `n/a` uncoloured ✅ · per-core history still queryable ✅ (354 pts) · ≥3 shape screenshots ✅ (`system-driveripper/deskwork/gateway.png` + dark/light/stale) · pytest+coverage+lint ✅ · phases 01/02 unchanged (their pin tests + `/api/summary` shape tests pass; live UPS gap in AGENTS.md re-confirmed read-only against real upsd) ✅.
- **Docs**: AGENTS.md — invariant 6 rewritten to cover message/event redaction, `collectors.py` line, new "Δ is a measurement" rule, "hosts answering total" → "nine-figure fleet-totals panel" (the subtitle still counts hosts), test count 452→472, coverage 96.8→96.9, rewrapped an over-long line. README.md — same fleet-totals wording, `RATE_WINDOW_SECONDS` now covers every measured rate (not just tokens), agent route list completed (`/`, `/schema`, `?top=N`), `LIST CMD` in the diagram, new paragraph on URL redaction.
- **Deviation**: the criterion's literal `curl …/api/system/history?host=…&metric=cpu.usage_pct@core7` uses singular params that endpoint has never accepted (it takes `hosts=`/`metrics=` and ignores unknown ones). Verified 354 points via `…system/history?hosts=driveripper&metrics=cpu.usage_pct@core7&range=1h` and `/api/history?source=host&host=…&metric=…`; I did **not** add singular param aliases (unauthorized API surface). No live infra changes made — reads only.
- **Next pending phase**: `04_llm_chart_readability`.
This commit is contained in:
@@ -0,0 +1,104 @@
|
||||
"""The `/system` hero-card payload contract, shared by the hermetic and the live tests.
|
||||
|
||||
Phase 03 makes `/system` one panel per host with six big colour-coded numbers. The paths
|
||||
below are the payload fields those cards are allowed to read; they were read out of
|
||||
`parse_host_snapshot()` and are pinned hermetically in `tests/test_system_contract.py`
|
||||
(driven from `tests/fixtures/agent_*.json`) and against the real fleet in
|
||||
`tests/test_live_fleet.py::test_every_host_serves_the_hero_card_contract`.
|
||||
|
||||
Keeping the list here rather than in one test file means the live assertions cannot drift
|
||||
away from the fixture assertions: there is one table, and it is the thing under test.
|
||||
|
||||
=========================== ==========================================================
|
||||
Card Path in `/api/system/snapshot` `hosts[]`
|
||||
=========================== ==========================================================
|
||||
cpu % ``cpu.usage_pct`` (+ ``cpu.severity.usage_pct``)
|
||||
mem % ``memory.used_pct`` (+ ``memory.severity.used_pct``)
|
||||
cpu temp ``cpu.temp_c`` (+ ``cpu.severity.temp_c``)
|
||||
disk usage % ``disks.worst_used_pct`` + ``disks.worst_mount`` (+ verdict)
|
||||
graphics card ``gpus[].model/busy_pct/vram_*/mem_used_pct`` (+ verdict)
|
||||
network down/up ``network.rx_bytes_per_s`` / ``tx_bytes_per_s`` (no verdict)
|
||||
=========================== ==========================================================
|
||||
|
||||
CPU temp comes from ``cpu.temp_c`` rather than ``temperatures.cpu_c``: both are the agent's
|
||||
own ``cpu_temperature_celsius``, and ``cpu.temp_c`` is the field ``/api/summary`` flattens
|
||||
into its fleet row, so the card and the overview cannot drift apart.
|
||||
|
||||
Deliberately *not* card metrics (legibility, not missing data — the series stay in SQLite):
|
||||
per-core usage (``cpu.usage_pct@coreN``), swap, the process table, the raw sensor list, and
|
||||
per-interface rows. Network carries no ``severity`` on purpose; throughput has no good/bad,
|
||||
so ``severity.THRESHOLDS`` has no ``net.*`` row.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
#: Card metric -> dotted payload paths. Present is the contract; a value may be ``None``
|
||||
#: on a host that genuinely does not have the number (gateway reports no CPU temperature).
|
||||
CARD_METRICS: dict[str, tuple[str, ...]] = {
|
||||
"cpu %": ("cpu.usage_pct", "cpu.severity.usage_pct"),
|
||||
"mem %": ("memory.used_pct", "memory.severity.used_pct"),
|
||||
"cpu temp": ("cpu.temp_c", "cpu.severity.temp_c", "temperatures.cpu_c"),
|
||||
# the mount is what makes the percentage actionable: 88 % used — of which disk?
|
||||
"disk %": ("disks.worst_used_pct", "disks.worst_mount", "disks.severity.worst_used_pct"),
|
||||
"network down/up": ("network.rx_bytes_per_s", "network.tx_bytes_per_s"),
|
||||
}
|
||||
|
||||
#: Per-GPU paths, checked on every row of `gpus[]` — and on none of them when the list is
|
||||
#: empty, which is how a host without a GPU stays honest (no card, not a card at 0 % busy).
|
||||
GPU_METRICS: tuple[str, ...] = (
|
||||
"model",
|
||||
"busy_pct",
|
||||
"vram_used_bytes",
|
||||
"vram_total_bytes",
|
||||
"vram_used_pct",
|
||||
# the honest APU view: vram + GTT, because a Strix Halo keeps the model in GTT
|
||||
"mem_used_pct",
|
||||
"severity.busy_pct",
|
||||
"severity.mem_used_pct",
|
||||
)
|
||||
|
||||
#: `network` must never grow a `severity` dict (see the module docstring).
|
||||
SEVERITY_FREE_SECTIONS: tuple[str, ...] = ("network",)
|
||||
|
||||
#: The host shapes the captures in `tests/fixtures/` cover, and therefore the shapes a card
|
||||
#: must handle: a discrete GPU, an APU (VRAM *and* GTT), and no GPU at all. A new capture
|
||||
#: has to be added here — `test_the_contract_covers_every_captured_host` fails otherwise.
|
||||
HOST_SHAPES: tuple[str, ...] = ("driveripper", "deskwork", "gateway")
|
||||
|
||||
|
||||
def dig(node: Any, path: str) -> Any:
|
||||
"""Follow a dotted payload path, failing with the path in the message.
|
||||
|
||||
A missing key is a contract break; a ``None`` value is not (that renders `n/a`), which is
|
||||
why this returns the value instead of asserting on it.
|
||||
"""
|
||||
walked = ""
|
||||
for part in path.split("."):
|
||||
assert isinstance(node, dict), f"{path}: hit a non-dict at {walked or 'payload'}"
|
||||
assert part in node, (
|
||||
f"{path}: no key {part!r} under {walked or 'payload'} — a hero card would print "
|
||||
f"n/a for a number the API used to serve"
|
||||
)
|
||||
node = node[part]
|
||||
walked = f"{walked}.{part}" if walked else part
|
||||
return node
|
||||
|
||||
|
||||
def assert_card_contract(payload: Any) -> None:
|
||||
"""Every card path exists on this host payload, GPU rows included."""
|
||||
for paths in CARD_METRICS.values():
|
||||
for path in paths:
|
||||
dig(payload, path)
|
||||
gpus = payload.get("gpus")
|
||||
assert isinstance(gpus, list), "gpus must be a list — [] when the agent reports no GPU"
|
||||
for gpu in gpus:
|
||||
for path in GPU_METRICS:
|
||||
dig(gpu, path)
|
||||
for section in SEVERITY_FREE_SECTIONS:
|
||||
block = payload.get(section)
|
||||
assert isinstance(block, dict), f"{section} must be a dict"
|
||||
assert "severity" not in block, (
|
||||
f"{section}.severity would be a colour nobody measured — throughput has no "
|
||||
f"good/bad and severity.THRESHOLDS has no {section}.* row"
|
||||
)
|
||||
+292
-5
@@ -388,6 +388,60 @@ def test_agent_tokens_never_reach_the_browser(tmp_path: Path, monkeypatch: pytes
|
||||
store.close()
|
||||
|
||||
|
||||
def test_a_dead_host_never_writes_its_token_into_the_feed(tmp_path: Path) -> None:
|
||||
"""The second path a credential can take to a browser: the failure message.
|
||||
|
||||
`parse_targets` moves a userinfo token out of the URL and into a bearer header, so the
|
||||
documented `HOST_TARGETS` path never puts one in `Target.url` — this is the backstop for
|
||||
any `Target` that does (a hand-built one, a future parser), and the message path is where
|
||||
it would matter: the poller keeps an `UpstreamError` string in the health record and logs
|
||||
it as an event, so a userinfo-bearing URL there would sit in SQLite and be served on
|
||||
`/api/events` forever (AGENTS.md invariant 6). Asserted on every endpoint a page reads,
|
||||
not just `/api/config`.
|
||||
"""
|
||||
db = str(tmp_path / "dead-host-token.sqlite3")
|
||||
store = SampleStore(db)
|
||||
settings = Settings(
|
||||
db_path=db,
|
||||
host_interval=0.05,
|
||||
host_targets=(Target(name="driveripper", url="http://SEKRIT-AGENT-TOKEN@127.0.0.1:1"),),
|
||||
)
|
||||
with TestClient(create_app(settings, store)) as client:
|
||||
deadline = time.time() + 10
|
||||
while time.time() < deadline:
|
||||
health = client.get("/api/health").json()
|
||||
if health["sources"] and health["sources"][0]["error"]:
|
||||
break
|
||||
time.sleep(0.05)
|
||||
else:
|
||||
pytest.fail("the dead host never recorded an error")
|
||||
for path in ("/api/health", "/api/events", "/api/system/snapshot", "/api/summary"):
|
||||
body = client.get(path).text
|
||||
assert "SEKRIT-AGENT-TOKEN" not in body, f"{path} leaked the agent token"
|
||||
# and the outage is still legible — redaction is not the same as silence
|
||||
assert "driveripper" in client.get("/api/health").text
|
||||
assert "unreachable" in client.get("/api/events").text
|
||||
store.close()
|
||||
|
||||
|
||||
def test_engine_urls_are_redacted_the_same_way_hosts_are(tmp_path: Path) -> None:
|
||||
"""`GUFO_BASE_URL`/`VLLM_BASE_URL` go to the browser verbatim in `/api/config`, and unlike
|
||||
`HOST_TARGETS` nothing parses a userinfo token out of them into a header — so a URL that
|
||||
happens to carry credentials has to be redacted where it is served (invariant 6).
|
||||
"""
|
||||
db = str(tmp_path / "engine-redact.sqlite3")
|
||||
settings = Settings(db_path=db, gufo_url="http://SEKRIT-ENGINE-TOKEN@deskwork:8731",
|
||||
vllm_url="http://driveripper:8000")
|
||||
store = SampleStore(db)
|
||||
with TestClient(create_app(settings, store)) as client:
|
||||
body = client.get("/api/config")
|
||||
assert "SEKRIT-ENGINE-TOKEN" not in body.text
|
||||
llm = body.json()["llm"]
|
||||
assert llm["gufo"]["base_url"] == "http://deskwork:8731"
|
||||
assert llm["vllm"]["base_url"] == "http://driveripper:8000" # untouched when clean
|
||||
store.close()
|
||||
|
||||
|
||||
def wait_for_state(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
@@ -522,13 +576,14 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
|
||||
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
|
||||
"""The old call sites still work until the remaining phases convert each page.
|
||||
|
||||
`/` was converted in phase 02: the overview colours from the `severity` verdicts
|
||||
`/api/summary` already sends, so it no longer carries a threshold of its own. The
|
||||
three pages still to come must keep compiling against the fallback until they are.
|
||||
`/` was converted in phase 02 and `/system` in phase 03: both colour themselves from
|
||||
the `severity` verdicts the API already sends, so they carry no threshold of their
|
||||
own. The two pages still to come must keep compiling against the fallback until they
|
||||
are converted — deleting it early would break a page mid-migration.
|
||||
"""
|
||||
js = client.get("/static/app.js").text
|
||||
assert "FALLBACK ONLY" in js
|
||||
for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"):
|
||||
for page in ("/static/page-llm.js", "/static/page-ups.js"):
|
||||
assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site"
|
||||
|
||||
|
||||
@@ -539,6 +594,198 @@ def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> Non
|
||||
assert "severityFor" not in js
|
||||
|
||||
|
||||
def test_system_page_colours_from_the_payload_not_from_js_thresholds(client) -> None:
|
||||
"""`page-system.js` hero cards paint payload verdicts, not thresholds of their own.
|
||||
|
||||
Phase 03 task 02's promise: every colour on a host card comes from a `severity` dict
|
||||
`severity.py` produced (`cpu.severity.usage_pct`, `memory.severity.used_pct`,
|
||||
`disks.severity.worst_used_pct`, `gpus[].severity.busy_pct`, `cpu.severity.temp_c`).
|
||||
A threshold written here could disagree with the overview's alerts; a hero with no
|
||||
verdict prints `n/a` uncoloured, which is `Dash.hero()`'s contract.
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
assert "severityFor(" not in js, "the system page re-invented a threshold"
|
||||
assert "Dash.hero(" in js
|
||||
for hook in ("cpu.severity.usage_pct", "memory.severity.used_pct",
|
||||
"disks.severity.worst_used_pct", "gpus[].severity.busy_pct",
|
||||
"cpu.severity.temp_c"):
|
||||
assert hook in js, f"the card no longer names {hook} as a colour source"
|
||||
|
||||
|
||||
def test_system_host_card_is_six_heroes_without_per_core(client) -> None:
|
||||
"""Each host card leads with exactly six heroes: CPU, MEM, TEMP, DISK, GPU, NET.
|
||||
|
||||
Asserted on the served script because there is no JS test runner (a locked anchor).
|
||||
The two dense `.metrics` rows and the per-core heat bars are exactly what made the
|
||||
card unreadable, and "no per-core anywhere on /system" is a promise a stray
|
||||
`renderCores` call would silently break — the series stays stored, the page just
|
||||
stopped drawing it (PLAN.md *Dropped visuals*).
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
card = js[js.index("function hostCard("):js.index("function renderHostCard(")]
|
||||
slots = re.findall(r'<div class="(?:hero|hero-slot)[^"]*" data-k="([^"]+)"', card)
|
||||
assert slots == ["cpu", "mem", "temp", "disk", "gpus", "net"], \
|
||||
"the hero row is no longer the six numbers in the agreed order"
|
||||
# per-core visualisation is gone from the card code entirely
|
||||
assert "cores" not in card, "a per-core visual came back to the card"
|
||||
assert "renderCores" not in js and "core_usage" not in js
|
||||
# the NET slot is a slot with two lines and no verdict: a byte rate has no good/bad
|
||||
assert 'data-k="net"' in card
|
||||
assert not re.search(r'data-k="net"[^>]*data-sev', card), "the NET slot invented a colour"
|
||||
# what the glance leaves out stays reachable, collapsed by default
|
||||
assert "More detail" in card
|
||||
for key in ("load", "freq", "dr", "dw", "dutil", "uptime", "details"):
|
||||
assert f'data-k="{key}"' in card, f"{key} left the page instead of the glance"
|
||||
# the panel owns its charts: the slots live inside the card, not on the page
|
||||
for chart in ("util", "net"):
|
||||
assert f'data-chart="{chart}"' in card, f"the panel lost its {chart} chart slot"
|
||||
|
||||
|
||||
def test_system_page_keeps_no_fleet_wide_chart_or_metric_picker(client) -> None:
|
||||
"""Nothing on `/system` plots two hosts, and nothing needs a selection to make sense.
|
||||
|
||||
TODO.md L4 is the instruction ("The system page doesn't need shared charts for
|
||||
anything") and the overview's alerts + fleet table are what made removing them safe
|
||||
(phase 02). Asserted on the served bytes because the only thing that would otherwise
|
||||
notice a resurrected `<select>` is a browser: there is no JS test runner (a locked
|
||||
anchor), and `test_no_page_carries_an_id_nothing_reads` cannot see ids that were deleted
|
||||
correctly from the page but are still queried by the script.
|
||||
"""
|
||||
html = _code(client.get("/system").text)
|
||||
for gone in ('id="chart-cpu"', 'id="chart-mem"', 'id="chart-disk"', 'id="chart-net"',
|
||||
'id="chart-gpu"', 'id="chart-temp"', 'id="chart-detail"', 'id="detail-host"',
|
||||
'id="detail-metric"', 'id="f-cpu"', 'id="f-load"', 'id="f-gpu"',
|
||||
"Fleet totals", "Fleet charts", "Series detail", "<select"):
|
||||
assert gone not in html, f"/system still carries {gone}"
|
||||
# the two panels that are left are the per-host grid and the event feed
|
||||
assert html.count("<section") == 2, "/system grew a panel back on top of the host grid"
|
||||
assert 'id="hosts"' in html and 'id="events"' in html
|
||||
|
||||
# and the script must not reach for the deleted endpoints/ids either — a `#detail-host`
|
||||
# query against a removed node is a silent null, so the picker would just never appear
|
||||
js = client.get("/static/page-system.js").text
|
||||
for gone in ("#chart-", "detail-host", "detail-metric", "detail-note",
|
||||
"renderFleet", "makeCharts", "refreshGpuChart", "populateDetailMetrics"):
|
||||
assert gone not in js, f"page-system.js still references {gone}"
|
||||
|
||||
|
||||
def test_every_system_chart_is_drawn_inside_its_own_host_panel(client) -> None:
|
||||
"""One host per chart: the legends of `/system` can never list two machine names.
|
||||
|
||||
The construction is a factory that takes the card, so a chart cannot be built against a
|
||||
page-level id; the metric list is per host, so the GPU streams cannot leak from one
|
||||
panel into another; and a panel is torn down (with its charts) when its host leaves the
|
||||
payload, which is the bug class a keyed-by-host chart map exists to avoid.
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
builds = re.findall(r"new DashCharts\.LineChart\(([^,]+),", js)
|
||||
assert builds, "the system page builds no charts at all"
|
||||
# every chart is built from a `[data-chart]` slot inside the host card, never from a
|
||||
# page-level id — which is what makes "one host per chart" structural, not a convention
|
||||
assert all(re.search(r"(?:card|slot)\.querySelector\(.\[data-chart=", b) for b in builds), \
|
||||
f"a chart is built outside a host panel: {builds}"
|
||||
# three construction sites, one per slot (util, net, gpu) — never one per metric
|
||||
assert len(builds) == 3, f"charts are not built one per panel slot: {builds}"
|
||||
assert sorted(re.findall(r'data-chart=\\?"(\w+)"', " ".join(builds))) == ["gpu", "net", "util"]
|
||||
|
||||
factory = js[js.index("async function renderHostCharts("):js.index("function pruneHosts(")]
|
||||
assert "hosts=${encodeURIComponent(host.name)}" in factory, \
|
||||
"the per-host chart query is not scoped to one host"
|
||||
assert "&metrics=" in factory and "state.range" in factory, \
|
||||
"a host's charts need one metric list and the selected window"
|
||||
# GPU series are named from the payload's own `gpus[].index`, never discovered/guessed
|
||||
assert "gpu.busy_pct@" in js and "gpu.index" in js
|
||||
# the GPU chart exists only while the host reports cards
|
||||
gpu_slot = js[js.index("function hostChartSet("):js.index("function seriesFor(")]
|
||||
assert re.search(r"if \(!wantsGpu && set\.gpu\)", gpu_slot), \
|
||||
"a host that stops reporting GPUs keeps a GPU chart it has no data for"
|
||||
|
||||
pruner = js[js.index("function pruneHosts("):js.index("// ---- snapshot")]
|
||||
assert "state.hosts.delete(name)" in pruner and "state.charts.delete(name)" in pruner, \
|
||||
"a host that left the payload keeps a live panel and drawing charts"
|
||||
|
||||
|
||||
def test_every_system_chart_names_itself_for_assistive_tech(client) -> None:
|
||||
"""A 120px SVG is silence to a screen reader unless the page names it.
|
||||
|
||||
The phase-03 accessibility pass (task 04) found the per-host charts drawing anonymous
|
||||
plots: the legend is a sibling element and labels nothing, so the `<svg>` itself must
|
||||
carry `role="img"` + `aria-label` that says which host and which metrics. Pinned on the
|
||||
served bytes because there is no JS test runner (a locked anchor).
|
||||
"""
|
||||
charts_js = client.get("/static/charts.js").text
|
||||
# the shared LineChart applies the name to the svg, and only when the caller opted in
|
||||
assert "if (this.opts.ariaLabel)" in charts_js, "LineChart stopped honouring ariaLabel"
|
||||
assert "setAttribute('role', 'img')" in charts_js
|
||||
assert "setAttribute('aria-label', String(this.opts.ariaLabel))" in charts_js
|
||||
|
||||
js = client.get("/static/page-system.js").text
|
||||
# all three slots name themselves, and each name starts with the host the panel belongs
|
||||
# to — an aria-label that cannot say *which machine* is not an accessible name here
|
||||
assert js.count("ariaLabel: `") == 3, "a per-host chart slot lost its accessible name"
|
||||
for slot in ("utilisation", "network traffic", "GPU busy"):
|
||||
assert f"${{host.name}} \u2014 {slot}" in js or f"${{host.name}} — {slot}" in js, \
|
||||
f"the {slot} chart has no host-scoped accessible name"
|
||||
|
||||
|
||||
def test_system_event_feed_queries_the_source_transitions_are_stored_under(client) -> None:
|
||||
"""Host transitions are stored under the *dashboard* name, not the sample source.
|
||||
|
||||
`poller.py` logs `Event(ts, self.dashboard, …)` — host jobs write `system`, the way
|
||||
the UPS job writes `ups` — while *samples* use `host`. The `/system` feed queried
|
||||
`source=host` since the first commit and therefore always rendered "no transitions
|
||||
yet", even with a real outage (a gateway disconnect) sitting in the store. Pinned on
|
||||
the served bytes because there is no JS test runner (a locked anchor).
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
feed = js[js.index("async function refreshEvents("):]
|
||||
url = re.search(r"fetchJSON\('(/api/events[^']*)'\)", feed)
|
||||
assert url, "the /system page no longer fetches the event feed"
|
||||
assert "source=system" in url.group(1), \
|
||||
"the /system event feed must ask for the source the poller stores events under"
|
||||
assert "source=host" not in url.group(1), \
|
||||
"source=host matches no stored event — the feed is empty forever"
|
||||
|
||||
|
||||
def test_system_footnote_prints_the_window_it_actually_measured(client) -> None:
|
||||
"""`Δ<window>` is the page's promise that a rate is a measurement (invariant 2), so the
|
||||
number behind it has to come from the measurement.
|
||||
|
||||
The `window_seconds` beside the fleet is `HOST_INTERVAL_SECONDS`, while the window behind a
|
||||
rate is the estimator's `RATE_WINDOW_SECONDS` — equal at the defaults, which is why the
|
||||
footnote could quote the poll interval and look right. Set `RATE_WINDOW_SECONDS=30` and it
|
||||
would have labelled every rate on the page with a window it never used. The footnote now
|
||||
reads each host's own measured `window_seconds`, and prints a range when they disagree.
|
||||
"""
|
||||
js = client.get("/static/page-system.js").text
|
||||
assert "function windowLabel(" in js, "the footnote lost its measured-window helper"
|
||||
footnote = js[js.index("$('#foot-note').textContent"):js.index("await Promise.all(chartJobs)")]
|
||||
assert "windowLabel(" in footnote
|
||||
assert "host.window_seconds" in footnote, \
|
||||
"the footnote is quoting the poll interval as a measurement again"
|
||||
# a footnote that says "window" and a page that says "poll" must stay two separate claims
|
||||
assert "poll ${Math.round(state.pollMs / 1000)}s" in footnote
|
||||
|
||||
|
||||
def test_no_per_core_visual_library_survives_the_page(client) -> None:
|
||||
"""The per-core heat bars left `/system` and `charts.js` with them (PLAN.md *Dropped
|
||||
visuals* is about what the pages draw, not what the store keeps).
|
||||
|
||||
`renderCores` had one caller — the hero-card per-core block deleted in task 02 — so
|
||||
keeping it would be dead markup generating unstyled `.core` spans; `.cores` was the CSS
|
||||
that block needed. Both are checked absent here so neither can come back as a
|
||||
half-change, while `sparkline`/`LineChart` stay for phase 04 and the UPS page.
|
||||
"""
|
||||
assert "renderCores" not in client.get("/static/charts.js").text
|
||||
css = client.get("/static/dash.css").text
|
||||
assert ".cores" not in css
|
||||
for asset in ASSETS:
|
||||
assert "core_usage" not in _code(client.get(asset).text), f"{asset} plots per-core"
|
||||
# what phase 04 and the other pages still chart from
|
||||
for name in ("LineChart", "sparkline", "renderLegend"):
|
||||
assert name in client.get("/static/charts.js").text
|
||||
|
||||
|
||||
def test_overview_is_exactly_two_panels(client) -> None:
|
||||
"""The landing page answers one question: what is wrong, and what is the fleet doing.
|
||||
|
||||
@@ -682,7 +929,14 @@ def test_parse_range_refuses_a_window_it_cannot_interpret(raw: str) -> None:
|
||||
|
||||
def test_system_history_charts_every_configured_host_in_one_request(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""N hosts x M metrics, one round trip — the chart endpoint's whole reason to exist."""
|
||||
"""N hosts x M metrics, one round trip.
|
||||
|
||||
`/system` no longer uses the N-hosts half of this (every chart belongs to one host and
|
||||
the page issues one request per host, pinned by
|
||||
`test_every_system_chart_is_drawn_inside_its_own_host_panel`); the shape is kept because
|
||||
the metric-list half is exactly what the per-host query costs one request with, and
|
||||
because the endpoint is a public interface for programmatic consumers.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
host_targets=(Target("driveripper", DEAD_URL),
|
||||
Target("deskwork", DEAD_URL))) as test_client:
|
||||
@@ -702,6 +956,39 @@ def test_system_history_charts_every_configured_host_in_one_request(
|
||||
assert narrowed["hosts"] == ["deskwork"]
|
||||
|
||||
|
||||
def test_one_request_per_host_answers_every_per_host_chart(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The exact query `/system` issues per host returns every series that panel draws.
|
||||
|
||||
`page-system.js` sends one request per host carrying the utilisation trio
|
||||
(`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct`), both network rates, and one
|
||||
`gpu.busy_pct@<index>` per card the payload reported — so a name missing here is a chart
|
||||
that silently reads "no samples yet" in the browser, and a renamed series in
|
||||
`sources/host.py` fails here rather than in a screenshot. This is the metric-name half of
|
||||
the contract; `tests/test_system_contract.py` holds the payload-path half.
|
||||
"""
|
||||
with build_client(tmp_path, monkeypatch,
|
||||
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
|
||||
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
|
||||
now = time.time()
|
||||
metrics = ["cpu.usage_pct", "mem.used_pct", "disk.used_pct",
|
||||
"net.rx_bytes_per_s", "net.tx_bytes_per_s", "gpu.busy_pct@0"]
|
||||
for metric in metrics:
|
||||
store.add("host", "driveripper", metric, 12.5, ts=now - 60)
|
||||
store.add("host", "driveripper", metric, 25.0, ts=now - 30)
|
||||
|
||||
body = test_client.get("/api/system/history", params={
|
||||
"hosts": "driveripper", "metrics": ",".join(metrics), "range": "1h"}).json()
|
||||
assert body["hosts"] == ["driveripper"], "one panel, one host"
|
||||
assert [item["metric"] for item in body["series"]] == metrics
|
||||
assert [len(item["points"]) for item in body["series"]] == [2] * len(metrics)
|
||||
|
||||
# a second card the host does not have answers empty, never zero and never 404
|
||||
gap = test_client.get("/api/system/history",
|
||||
params={"hosts": "driveripper", "metrics": "gpu.busy_pct@1"}).json()
|
||||
assert gap["series"][0]["points"] == []
|
||||
|
||||
|
||||
def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
|
||||
|
||||
@@ -120,6 +120,44 @@ async def test_a_refused_connection_is_an_upstream_error_not_a_traceback() -> No
|
||||
HostCollector(RateEstimator()))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", ["refused", "timeout", "http-500"])
|
||||
async def test_a_failing_host_never_echoes_its_token_in_the_message(failure: str) -> None:
|
||||
"""An `UpstreamError` message is not just a log line: the poller keeps it in the health
|
||||
record (`/api/health`) and writes it to the `event` table (`/api/events`), both of which
|
||||
the browser reads — so the URL it names has to be the redacted one, on every failure path
|
||||
(AGENTS.md invariant 6). `parse_targets` normally moves a userinfo token into a header, so
|
||||
this is the backstop that keeps the guarantee true for any `Target` that still carries one.
|
||||
"""
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if failure == "refused":
|
||||
raise httpx.ConnectError("connection refused")
|
||||
if failure == "timeout":
|
||||
raise httpx.ReadTimeout("too slow")
|
||||
return httpx.Response(500, text="boom")
|
||||
|
||||
target = Target("gw", "http://SEKRIT-AGENT-TOKEN@agent.test")
|
||||
async with client_for(handler) as client:
|
||||
with pytest.raises(UpstreamError) as excinfo:
|
||||
await fetch_host_snapshot(client, target, HostCollector(RateEstimator()))
|
||||
message = excinfo.value.message
|
||||
assert "SEKRIT-AGENT-TOKEN" not in message, f"the token leaked through {failure}"
|
||||
# …and the message still names the machine it failed to reach
|
||||
assert "http://agent.test/snapshot" in message
|
||||
|
||||
|
||||
async def test_a_failing_engine_never_echoes_credentials_in_its_url() -> None:
|
||||
"""Same rule for the engines: the URL in the message is the redacted one."""
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
raise httpx.ConnectError("connection refused")
|
||||
|
||||
async with client_for(handler) as client:
|
||||
with pytest.raises(UpstreamError, match="unreachable") as excinfo:
|
||||
await fetch_engine(client, "gufo", "http://SEKRIT-ENGINE-TOKEN@agent.test", "",
|
||||
RateEstimator())
|
||||
assert "SEKRIT-ENGINE-TOKEN" not in excinfo.value.message
|
||||
assert "http://agent.test/metrics" in excinfo.value.message
|
||||
|
||||
|
||||
async def test_a_slow_agent_is_a_gateway_timeout() -> None:
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
raise httpx.ReadTimeout("too slow")
|
||||
|
||||
@@ -47,6 +47,21 @@ def test_token_in_url_becomes_a_bearer_header_and_stays_out_of_the_display_url()
|
||||
assert "SEKRIT" not in target.display_url
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw,shown", [
|
||||
("http://SEKRIT@host:9120", "http://host:9120"),
|
||||
("https://user:pass@host:8000/metrics", "https://host:8000/metrics"),
|
||||
("HTTP://SEKRIT@host:1", "HTTP://host:1"),
|
||||
("http://host:9120", "http://host:9120"), # clean URLs pass through untouched
|
||||
("", ""),
|
||||
])
|
||||
def test_strip_userinfo_is_the_one_redaction_every_url_leaves_by(raw: str, shown: str) -> None:
|
||||
"""`strip_userinfo` is what `/api/config` and every `UpstreamError` message use, so it has
|
||||
to survive the shapes a URL actually comes in: both schemes, `user:pass@`, a path, and the
|
||||
common no-credentials case where it must change nothing (invariant 6).
|
||||
"""
|
||||
assert config.strip_userinfo(raw) == shown
|
||||
|
||||
|
||||
def test_target_names_are_restricted_because_they_are_db_keys() -> None:
|
||||
"""Names key the history DB and appear in URLs, so they are validated, not guessed.
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@ import os
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from card_contract import assert_card_contract
|
||||
from dashboards import collectors
|
||||
from dashboards.config import Settings, load
|
||||
from dashboards.rates import RateEstimator
|
||||
@@ -70,6 +71,27 @@ async def test_every_host_agent_answers_and_parses(settings: Settings,
|
||||
)
|
||||
|
||||
|
||||
async def test_every_host_serves_the_hero_card_contract(settings: Settings,
|
||||
host_collector: HostCollector) -> None:
|
||||
"""The phase-03 `/system` card paths exist on the **real** agents, not just the captures.
|
||||
|
||||
Values may honestly be `n/a` on real hardware (gateway reports no CPU sensor and no GPU
|
||||
at all); the *shape* is what phase 03 renders, so this is the test that catches an agent
|
||||
upgrade renaming a field. Hermetic twin: `tests/test_system_contract.py`.
|
||||
"""
|
||||
if not settings.host_targets:
|
||||
pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh")
|
||||
async with httpx.AsyncClient(timeout=settings.host_timeout) as client:
|
||||
for target in settings.host_targets:
|
||||
await collectors.fetch_host_snapshot(client, target, host_collector)
|
||||
await asyncio.sleep(1.2)
|
||||
result = await collectors.fetch_host_snapshot(client, target, host_collector)
|
||||
assert_card_contract(result.payload)
|
||||
assert result.payload["cpu"]["temp_c"] == result.payload["temperatures"]["cpu_c"], (
|
||||
f"{target.name}: the card and /api/summary would report different CPU temps"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("engine", ["gufo", "vllm"])
|
||||
async def test_engine_still_publishes_the_series_we_parse(engine: str,
|
||||
settings: Settings) -> None:
|
||||
|
||||
@@ -0,0 +1,288 @@
|
||||
"""The payload contract behind the `/system` hero cards (phase 03, task 01).
|
||||
|
||||
Phase 03 turns `/system` into one panel per host with six big colour-coded numbers. Before
|
||||
any of it is rendered, the data has to be proven present — otherwise a card that shows `n/a`
|
||||
gets "fixed" by inventing a field, and the field it invents is a number the agent never
|
||||
reported. This file drives the verbatim captures in `tests/fixtures/` through
|
||||
`parse_host_snapshot()` (and through a real app), so a rename breaks here rather than in a
|
||||
browser at 2am (`AGENTS.md` invariant 7).
|
||||
|
||||
The path table itself lives in `tests/card_contract.py` so the live suite asserts the same
|
||||
list against real agents (`tests/test_live_fleet.py::test_every_host_serves_the_hero_card_contract`)
|
||||
and the two cannot drift apart.
|
||||
|
||||
What is **deliberately not on the card** — a legibility decision, not a missing feature:
|
||||
per-core usage (still stored per logical CPU as `cpu.usage_pct@coreN`, see
|
||||
`test_visuals_the_page_drops_are_still_stored`), swap, the process table, the raw sensor
|
||||
list, and per-interface rows. Removing a plot is not removing data (PLAN.md *Dropped
|
||||
visuals*), so the decision stays reversible.
|
||||
|
||||
Two choices are recorded here because the task asked for them in writing:
|
||||
|
||||
* **CPU temp comes from `cpu.temp_c`**, not `temperatures.cpu_c`. They are the same
|
||||
measurement (the agent's own `cpu_temperature_celsius`) on every capture, `cpu.temp_c` is
|
||||
what `/api/summary` flattens into its fleet row, and a test below keeps them equal.
|
||||
* **The disk verdict lives at `disks.severity.worst_used_pct`** (phase 01 attaches severity
|
||||
as a sibling dict per section, so there is no nested `disks.worst` object to hang it on).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import math
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from card_contract import GPU_METRICS, HOST_SHAPES, assert_card_contract, dig
|
||||
from dashboards.app import create_app
|
||||
from dashboards.config import Settings, Target
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources.host import HostCollector, parse_host_snapshot
|
||||
from dashboards.store import SampleStore
|
||||
|
||||
#: One scrape is not a rate, so the API-level app points at a port that refuses instantly.
|
||||
DEAD_URL = "http://127.0.0.1:1"
|
||||
|
||||
|
||||
def make_collector() -> HostCollector:
|
||||
return HostCollector(RateEstimator(span=60.0))
|
||||
|
||||
|
||||
def _bump_modes(modes: dict[str, Any], delta: float) -> None:
|
||||
"""Move a /proc/stat mode set so the window reads `delta` ticks, 10 % of them busy."""
|
||||
if not isinstance(modes.get("total"), (int, float)):
|
||||
return
|
||||
modes["total"] += delta
|
||||
for idle_mode, share in (("idle", 0.8), ("iowait", 0.1)):
|
||||
if isinstance(modes.get(idle_mode), (int, float)):
|
||||
modes[idle_mode] += delta * share
|
||||
|
||||
|
||||
def _advance(snapshot: dict[str, Any], dt: float = 10.0) -> dict[str, Any]:
|
||||
"""The same scrape `dt` later with counters moved — the second point every rate needs."""
|
||||
moved = copy.deepcopy(snapshot)
|
||||
moved["ts"] = float(snapshot["ts"]) + dt
|
||||
_bump_modes(moved["cpu"]["modes"], 1_000.0)
|
||||
for core in moved["cpu"].get("cores") or []:
|
||||
_bump_modes(core.get("modes") or {}, 1_000.0)
|
||||
for iface in (moved.get("network") or {}).get("interfaces") or []:
|
||||
for key, value in list(iface.items()):
|
||||
if key.endswith("_total") and isinstance(value, (int, float)):
|
||||
iface[key] = value + 50_000.0
|
||||
return moved
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def payloads(agent_snapshots: dict[str, dict[str, Any]]) -> dict[str, dict[str, Any]]:
|
||||
"""Each capture parsed once, first scrape — the shape the cards must tolerate."""
|
||||
out: dict[str, dict[str, Any]] = {}
|
||||
for name, snapshot in agent_snapshots.items():
|
||||
out[name] = parse_host_snapshot(name, snapshot, make_collector(), now=1_000.0).payload
|
||||
return out
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def api_client(tmp_path: Any, agent_snapshot: dict[str, Any],
|
||||
host_scrape: Any) -> Iterator[TestClient]:
|
||||
"""A running app serving one host payload, so the contract is checked over HTTP too."""
|
||||
db = str(tmp_path / "contract.sqlite3")
|
||||
store = SampleStore(db)
|
||||
app = create_app(Settings(db_path=db, host_targets=(Target("driveripper", DEAD_URL),)),
|
||||
store)
|
||||
client = TestClient(app)
|
||||
client.__enter__()
|
||||
try:
|
||||
# the job's port refuses, so a poll can never overwrite what is injected here
|
||||
app.state.ctx.hub.jobs["host:driveripper"].state.payload = host_scrape(
|
||||
agent_snapshot, cpu_busy=10.0)
|
||||
yield client
|
||||
finally:
|
||||
client.__exit__(None, None, None)
|
||||
store.close()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Every card metric has a path, on every host shape
|
||||
# --------------------------------------------------------------------------- #
|
||||
@pytest.mark.parametrize("name", HOST_SHAPES)
|
||||
def test_every_card_metric_has_a_payload_path_on_every_host(
|
||||
payloads: dict[str, dict[str, Any]], name: str) -> None:
|
||||
assert_card_contract(payloads[name])
|
||||
|
||||
|
||||
def test_the_contract_covers_every_captured_host(agent_snapshots: dict[str, dict[str, Any]],
|
||||
payloads: dict[str, dict[str, Any]]) -> None:
|
||||
"""A fourth capture must join `HOST_SHAPES`, or its host shape goes uncontracted."""
|
||||
assert set(HOST_SHAPES) == set(payloads) == set(agent_snapshots)
|
||||
|
||||
|
||||
def test_the_discrete_gpu_capture_supplies_real_numbers(
|
||||
payloads: dict[str, dict[str, Any]], agent_snapshots: dict[str, dict[str, Any]]) -> None:
|
||||
"""Not just present: correct, on the host that has all six metrics."""
|
||||
payload = payloads["driveripper"]
|
||||
snapshot = agent_snapshots["driveripper"]
|
||||
|
||||
assert payload["cpu"]["temp_c"] == snapshot["cpu_temperature_celsius"]
|
||||
# the two places the CPU temperature is exposed are the same measurement, always
|
||||
assert payload["temperatures"]["cpu_c"] == payload["cpu"]["temp_c"]
|
||||
|
||||
memory = snapshot["memory"]
|
||||
assert payload["memory"]["used_pct"] == pytest.approx(
|
||||
memory["used_bytes"] / memory["total_bytes"] * 100.0)
|
||||
|
||||
worst = max(snapshot["filesystems"], key=lambda fs: fs["used_pct"])
|
||||
assert payload["disks"]["worst_mount"] == worst["mountpoint"]
|
||||
assert payload["disks"]["worst_used_pct"] == pytest.approx(worst["used_pct"])
|
||||
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["model"]
|
||||
assert gpu["busy_pct"] == pytest.approx(snapshot["gpus"][0]["busy_pct"])
|
||||
assert gpu["vram_used_pct"] == pytest.approx(
|
||||
gpu["vram_used_bytes"] / gpu["vram_total_bytes"] * 100.0)
|
||||
|
||||
# one scrape is not a rate: usage and throughput are honestly n/a until the second
|
||||
assert payload["cpu"]["usage_pct"] is None
|
||||
assert payload["network"]["rx_bytes_per_s"] is None
|
||||
|
||||
|
||||
def test_measured_values_appear_on_the_second_scrape(host_scrape: Any,
|
||||
agent_snapshot: dict[str, Any]) -> None:
|
||||
"""The payload the API actually serves (after two scrapes) has numbers in all six places."""
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=33.0)
|
||||
assert payload["cpu"]["usage_pct"] == pytest.approx(33.0)
|
||||
assert math.isfinite(payload["network"]["rx_bytes_per_s"])
|
||||
assert math.isfinite(payload["network"]["tx_bytes_per_s"])
|
||||
assert payload["memory"]["used_pct"] is not None
|
||||
assert payload["disks"]["worst_used_pct"] is not None
|
||||
assert payload["cpu"]["temp_c"] is not None
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["busy_pct"] is not None
|
||||
assert gpu["vram_used_pct"] is not None
|
||||
assert gpu["mem_used_pct"] is not None
|
||||
|
||||
|
||||
def test_each_card_number_travels_with_its_own_verdict(host_scrape: Any,
|
||||
agent_snapshot: dict[str, Any]) -> None:
|
||||
"""The card colours a verdict from the payload; it never holds a threshold (phase 01)."""
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=33.0)
|
||||
assert payload["cpu"]["severity"] == {
|
||||
"usage_pct": classify("cpu.usage_pct", payload["cpu"]["usage_pct"]),
|
||||
"temp_c": classify("cpu.temp_c", payload["cpu"]["temp_c"]),
|
||||
}
|
||||
assert payload["memory"]["severity"]["used_pct"] == classify(
|
||||
"mem.used_pct", payload["memory"]["used_pct"])
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] == classify(
|
||||
"disk.used_pct", payload["disks"]["worst_used_pct"])
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["severity"] == {
|
||||
"busy_pct": classify("gpu.busy_pct", gpu["busy_pct"]),
|
||||
"mem_used_pct": classify("gpu.mem_used_pct", gpu["mem_used_pct"]),
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The two host shapes that are not "a box with a discrete GPU"
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_the_box_with_no_gpu_reports_an_empty_list_not_a_missing_key(
|
||||
payloads: dict[str, dict[str, Any]]) -> None:
|
||||
"""gateway has no GPU: `gpus == []`, never a missing key and never a card at 0 % busy."""
|
||||
payload = payloads["gateway"]
|
||||
assert "gpus" in payload, "a host with no GPU still has the key; the page says 'no GPU'"
|
||||
assert payload["gpus"] == []
|
||||
# the other five numbers keep their paths, with the honest None where nothing is measured
|
||||
assert_card_contract(payload)
|
||||
assert payload["cpu"]["temp_c"] is None, "this agent reports no CPU sensor"
|
||||
assert payload["cpu"]["severity"]["temp_c"] is None, "unknown is not ok"
|
||||
|
||||
|
||||
def test_the_apu_reports_vram_and_the_whole_addressable_pool(
|
||||
payloads: dict[str, dict[str, Any]]) -> None:
|
||||
"""deskwork (Strix Halo) is why `mem_used_pct` exists: 3 % VRAM and 78 % memory, both true."""
|
||||
gpu = payloads["deskwork"]["gpus"][0]
|
||||
assert gpu["vram_total_bytes"] < gpu["mem_total_bytes"]
|
||||
assert gpu["vram_used_pct"] < gpu["mem_used_pct"]
|
||||
assert gpu["model"], "the card needs a name to put beside the numbers"
|
||||
|
||||
|
||||
def test_paths_survive_a_host_that_reports_almost_nothing(agent_snapshot: dict[str, Any]) -> None:
|
||||
"""Keys present, values None: the card must render `n/a` without a KeyError."""
|
||||
snapshot = copy.deepcopy(agent_snapshot)
|
||||
snapshot["cpu_temperature_celsius"] = None
|
||||
snapshot["filesystems"] = []
|
||||
snapshot["network"]["interfaces"] = []
|
||||
snapshot["gpus"][0]["busy_pct"] = None
|
||||
snapshot["gpus"][0]["vram_total_bytes"] = None
|
||||
|
||||
payload = parse_host_snapshot("driveripper", snapshot, make_collector(),
|
||||
now=1_000.0).payload
|
||||
assert_card_contract(payload)
|
||||
for path in GPU_METRICS:
|
||||
dig(payload["gpus"][0], path)
|
||||
|
||||
assert payload["cpu"]["temp_c"] is None and payload["cpu"]["severity"]["temp_c"] is None
|
||||
assert payload["disks"]["worst_used_pct"] is None
|
||||
assert payload["disks"]["worst_mount"] is None
|
||||
assert payload["disks"]["severity"]["worst_used_pct"] is None
|
||||
assert payload["network"]["rx_bytes_per_s"] is None
|
||||
gpu = payload["gpus"][0]
|
||||
assert gpu["busy_pct"] is None and gpu["severity"]["busy_pct"] is None
|
||||
assert gpu["vram_used_pct"] is None, "no total, so no percentage — not 0 %"
|
||||
assert gpu["mem_used_pct"] is None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# What the cards drop stays in the database
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_visuals_the_page_drops_are_still_stored(agent_snapshot: dict[str, Any]) -> None:
|
||||
"""Per-core / per-mount / per-interface / sensor / process data leaves the page.
|
||||
|
||||
It must still be *sampled*: PLAN.md's *Dropped visuals* anchor says the page stops
|
||||
drawing what the operator stopped asking for while history keeps the series, so the
|
||||
decision stays reversible.
|
||||
"""
|
||||
collector = make_collector()
|
||||
parse_host_snapshot("driveripper", agent_snapshot, collector, now=1_000.0)
|
||||
result = parse_host_snapshot("driveripper", _advance(agent_snapshot), collector,
|
||||
now=1_010.0)
|
||||
stored = {sample.metric for sample in result.samples}
|
||||
|
||||
assert result.payload["cpu"]["usage_pct"] == pytest.approx(10.0), "the counters moved"
|
||||
cores = {metric for metric in stored if metric.startswith("cpu.usage_pct@core")}
|
||||
logical_cpus = result.payload["info"]["cpu_cores_logical"]
|
||||
assert len(cores) == logical_cpus, "one series per logical CPU, plotted or not"
|
||||
assert "cpu.usage_pct@core0" in stored and f"cpu.usage_pct@core{logical_cpus - 1}" in stored
|
||||
assert "disk.used_pct@/" in stored, "per-mount usage is still stored; the card shows worst"
|
||||
assert any(metric.startswith("net.rx_bytes_per_s@") for metric in stored), (
|
||||
"per-interface throughput is still stored; the card shows the totals")
|
||||
assert "temp.max_c" in stored, "the sensor list is not on the card but stays in history"
|
||||
assert "host.processes" in stored
|
||||
# swap is not a card metric either; it is absent here only because this capture has no
|
||||
# swap device (swap_total_bytes == 0), not because storing it stopped
|
||||
assert result.payload["memory"]["swap_used_pct"] is None
|
||||
assert "mem.swap_used_pct" not in stored
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# The same contract over HTTP
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_snapshot_endpoint_serves_the_card_paths(api_client: TestClient) -> None:
|
||||
"""The cards read `/api/system/snapshot`, so the contract is checked on that body too."""
|
||||
hosts = api_client.get("/api/system/snapshot").json()["hosts"]
|
||||
assert len(hosts) == 1
|
||||
assert_card_contract(hosts[0])
|
||||
|
||||
|
||||
def test_history_endpoint_still_answers_for_a_per_core_metric(api_client: TestClient) -> None:
|
||||
"""`cpu.usage_pct@core7` is off the page but still queryable — dropping the plot is
|
||||
not dropping the series."""
|
||||
store = api_client.app.state.ctx.store # type: ignore[attr-defined]
|
||||
store.add("host", "driveripper", "cpu.usage_pct@core7", 42.0, time.time())
|
||||
body = api_client.get("/api/system/history",
|
||||
params={"metrics": "cpu.usage_pct@core7", "hosts": "driveripper",
|
||||
"range": "1h"}).json()
|
||||
assert body["series"], "the per-core series must still be retrievable"
|
||||
assert [point["value"] for point in body["series"][0]["points"]] == [42.0]
|
||||
Reference in New Issue
Block a user