services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 02_overview_fleet_and_alerts

Browse Source 
**Phase 02 final verification pass — all three tasks already complete; verified criteria, fixed 3 defects found live.**

- Verified: `/` = exactly 2 `<section>`s (Warnings & errors + Fleet), alerts render from `/api/summary` verdicts with each alert's own `href`; `/system`, `/llm`, `/ups` unchanged (charts + data live).
- Fixed (`src/dashboards/alerts.py`): a dead source's message printed the name twice — `gateway: UpstreamError: gateway: http://…`; the poller's `UpstreamError:` wrapper is now stripped (other exception names kept). Tests: 2 new in `tests/test_alerts.py`.
- Fixed (`src/dashboards/api.py`): `/api/summary` dropped a configured host/engine that had never answered post-start, so the fleet said `2/2 hosts` while a third was down. Stub rows (`unreachable: true`, all `n/a`) now match `/api/*/snapshot`; `hosts_total=3 / hosts_live=2`. Test: `test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet`.
- Fixed (`web/static/page-index.js`): fleet row of a non-answering host is now dimmed with the shared `.stale` class (invariant 4 on `/`). Test: `test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering`.
- `bash .agents/validate.sh` exit 0: `uv run pytest --cov --cov-fail-under=94` → **439 passed, 6 skipped, 96.82%** (`alerts.py` 100%, `api.py` 100%); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check web/static` clean; hygiene clean; `uv pip list --outdated` empty; `uv run pytest -m live --live` → **6 passed**.
- Live proof (tunnel kill, then restored — all 6 sources `ok` again): killing the gateway forward produced exactly **one** alert, `fail gateway /system … unreachable (ConnectError)`; screenshots `data/logs/phase02-final-{alert,system,llm,ups,recovered}.png`.
- Completion criteria: two panels ✅ · `[]` when healthy ✅ (hermetic `healthy_client` + a live gateway-only instance returned `[]`; the real fleet can never be `[]` — its UPSes permanently warn on battery age/self-test) · one alert per dead host with `/system` link ✅ · pytest/live/coverage/ruff/pyright ✅ · no change on `/system`,`/llm`,`/ups` ✅ · phase 01 verdict vocabulary intact ✅.
- Docs: README `web/` bullet now states the dimmed `n/a` row; AGENTS.md invariant 4, `alerts.py`/`api.py` bullets, UI-conventions stale-row + window-chip ranges (5m…30d vs 1h…90d), metric example `llm.decode_tps` → `llm.decode_tokens_per_second` (verified against `/api/metrics`), test count 435 → 439. Env table, Containerfile, agent/install-script claims, fleet facts re-checked and consistent.
- Notable: left the fleet's real hostnames/domain in AGENTS.md's "real environment" section (deliberate, prior-phase content) even though rule 6 discourages personal hostnames in tracked files — worth a separate decision; no live config was changed beyond stopping/starting a local dev ssh forward.
- Next pending phase: `.agents/phases/todo/03_system_per_host_panels/`.
This commit is contained in:
ducoterra
2026-10-17 00:59:09 -04:00
parent ad993c7b03
commit 8b048599f3
34 changed files with 1955 additions and 225 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_api.py
+494 -28
View File
@@ -7,30 +7,38 @@ asset the network can fail to deliver.
"""
from __future__ import annotations
import copy
import re
import time
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path
from typing import Any
import pytest
from fastapi import HTTPException
from fastapi.testclient import TestClient
from dashboards import alerts
from dashboards.api import RANGE_SECONDS, parse_range
from dashboards.app import create_app
from dashboards.config import Settings, Target
from dashboards.rates import RateEstimator
from dashboards.severity import classify
from dashboards.sources.host import HostCollector, parse_host_snapshot
from dashboards.sources import nut
from dashboards.store import Event, Sample, SampleStore
# refused immediately, and never dials out of the test environment
DEAD_URL = "http://127.0.0.1:1"
@contextmanager
def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: bool = False,
**overrides: Any) -> TestClient:
"""A TestClient whose lifespan runs, against a private DB and no real network."""
**overrides: Any) -> Iterator[TestClient]:
"""A TestClient whose lifespan runs, against a private DB and no real network.
The store is injected, so `create_app()` deliberately does not close it (production
owns its own); the caller here does, or every test leaks a SQLite handle and the
`--cov` log fills with ResourceWarnings that bury the warnings worth reading.
"""
if simulate:
monkeypatch.setenv("NUT_SIMULATE", "1")
else:
@@ -38,10 +46,13 @@ def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: b
db = str(tmp_path / "api.sqlite3")
settings = Settings(db_path=db, **overrides)
store = SampleStore(db)
app = create_app(settings, store)
client = TestClient(app)
client = TestClient(create_app(settings, store))
client.__enter__()
return client
try:
yield client
finally:
client.__exit__(None, None, None)
store.close()
@pytest.fixture
@@ -68,29 +79,23 @@ def wait_for_payload(client: TestClient, path: str, *, timeout: float = 5.0) ->
pytest.fail(f"{path} never became available")
def host_payload(agent_snapshot: dict[str, Any]) -> dict[str, Any]:
def failing_host_payload(scrape: Any, agent_snapshot: dict[str, Any]) -> dict[str, Any]:
"""Two scrapes of the real driveripper capture, nudged onto known sides of the table.
Driven from the fixture rather than hand-written so a schema drift breaks here first.
"""
snap = copy.deepcopy(agent_snapshot)
snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95 # mem.used_pct -> fail
snap["cpu_temperature_celsius"] = 91.0 # cpu.temp_c -> fail
for fs in snap["filesystems"]:
fs["used_pct"] = 93.0 # disk -> fail
collector = HostCollector(RateEstimator(span=60.0))
parse_host_snapshot("driveripper", snap, collector, now=1_000.0)
second = copy.deepcopy(snap)
second["ts"] = 1_010.0
modes = second["cpu"]["modes"]
modes["total"] += 1_000.0
modes["idle"] += 900.0 # 10 % busy over the window -> ok
return parse_host_snapshot("driveripper", second, collector, now=1_010.0).payload
def nudge(snap: dict[str, Any]) -> None:
snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95 # mem.used_pct -> fail
snap["cpu_temperature_celsius"] = 91.0 # cpu.temp_c -> fail
for fs in snap["filesystems"]:
fs["used_pct"] = 93.0 # disk -> fail
return scrape(agent_snapshot, cpu_busy=10.0, mutate=nudge)
@pytest.fixture
def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
agent_snapshot: dict[str, Any]) -> Iterator[TestClient]:
agent_snapshot: dict[str, Any], host_scrape: Any) -> Iterator[TestClient]:
"""Overview-shaped app: one host payload in the hub plus the simulated UPSes.
The host job points at a dead port, so a poll can never overwrite the injected payload
@@ -104,7 +109,8 @@ def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
test_client = TestClient(app)
test_client.__enter__()
try:
app.state.ctx.hub.jobs["host:driveripper"].state.payload = host_payload(agent_snapshot)
app.state.ctx.hub.jobs["host:driveripper"].state.payload = failing_host_payload(
host_scrape, agent_snapshot)
wait_for_payload(test_client, "/api/ups/snapshot")
yield test_client
finally:
@@ -112,6 +118,41 @@ def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
store.close()
@pytest.fixture
def healthy_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
agent_snapshot: dict[str, Any],
host_scrape: Any) -> Iterator[TestClient]:
"""The all-clear app: one host answering, nothing past a threshold, no UPS configured.
The job's collector is swapped for one that hands back the injected payload, so the
health record says `ok` truthfully instead of being overridden behind the poller.
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
from dashboards.poller import Result # noqa: PLC0415 - only this fixture needs it
payload = host_scrape(agent_snapshot, cpu_busy=10.0)
async def collect() -> Result:
return Result(payload=payload, samples=[])
job = test_client.app.state.ctx.hub.jobs["host:driveripper"] # type: ignore[attr-defined]
job.fn = collect
# let the poll already in flight against the dead port land first (it fails in
# microseconds), then take the health record over: the alerts must describe a host
# that is answering, not the half-second of startup where it was not
deadline = time.time() + 2.0
while time.time() < deadline and job.state.attempts == 0:
time.sleep(0.02)
time.sleep(0.05)
job.state.payload = payload
job.state.ok = True
job.state.error = None
job.state.consecutive_failures = 0
job.state.last_ok = time.time()
yield test_client
# ---- severity travels with the value --------------------------------------- #
def test_summary_carries_the_verdict_for_every_column_the_page_colours(data_client) -> None:
@@ -147,13 +188,121 @@ def test_summary_shape_did_not_shrink(data_client) -> None:
assert fleet_keys <= set(body["system"]["fleet"]), fleet_keys - set(body["system"]["fleet"])
assert host_keys <= set(body["system"]["hosts"][0]), host_keys - set(body["system"]["hosts"][0])
assert ups_keys <= set(body["ups"]["ups"][0]), ups_keys - set(body["ups"]["ups"][0])
assert set(body) == {"ts", "system", "llm", "ups", "recent_events"}
assert set(body) == {"ts", "system", "llm", "ups", "recent_events", "alerts"}
# ...and the new keys are there beside the old ones, not instead of them
assert "severity" in body["system"]["fleet"]
assert "severity" in body["system"]["hosts"][0]
assert "severity" in body["ups"]["ups"][0]
# `/` paints two panels; the payload is still the whole-fleet contract. Every path here is
# read by something that is not the overview page: the live suite, an external probe, a
# notifier later. Deleting a card must never delete a key.
SUMMARY_CONTRACT = ("system.enabled", "system.fleet", "system.hosts",
"llm.engines", "ups.ups", "recent_events", "alerts")
def dig(body: dict[str, Any], path: str) -> Any:
"""`body["a"]["b"]` that names the missing path when it is not there."""
node: Any = body
for part in path.split("."):
assert isinstance(node, dict) and part in node, f"/api/summary lost `{path}`"
node = node[part]
return node
@pytest.mark.parametrize("fixture", ["client", "data_client"])
def test_summary_serves_every_path_another_consumer_reads(request, fixture: str) -> None:
"""The keys survive the overview cleanup whether or not anything is configured.
Run twice on purpose: `client` has no targets at all, so it proves the keys are present
as `None`/`[]` rather than omitted when a source is unconfigured (invariant 3 is about
the payload shape as much as about colour); `data_client` proves the same paths carry
the fleet once there is one.
"""
test_client = request.getfixturevalue(fixture)
body = test_client.get("/api/summary").json()
populated = fixture == "data_client"
for path in SUMMARY_CONTRACT:
dig(body, path) # presence is the assertion; a missing path raises with its name
assert dig(body, "system.enabled") is populated
assert (dig(body, "system.fleet") is not None) is populated
names = {host["name"] for host in dig(body, "system.hosts")}
assert names == ({"driveripper"} if populated else set())
batteries = {item["name"] for item in dig(body, "ups.ups")}
assert batteries == ({"apc", "cyberpower"} if populated else set())
assert dig(body, "llm.engines") == [], "neither fixture configures an engine"
assert isinstance(dig(body, "recent_events"), list)
assert isinstance(dig(body, "alerts"), list)
# and the payload as a whole still carries every top-level section it always did
assert set(body) >= {"ts", "system", "llm", "ups", "recent_events", "alerts"}
# ---- alerts: the landing page's answer to "what is wrong" ------------------- #
def test_summary_is_empty_of_alerts_when_everything_is_healthy(healthy_client) -> None:
"""`[]`, not a green banner: an empty list is the clearest possible all-clear."""
body = healthy_client.get("/api/summary").json()
assert body["alerts"] == []
assert body["system"]["fleet"]["hosts_live"] == 1
def test_summary_raises_one_alert_for_the_host_that_stopped_answering(data_client) -> None:
"""One dead host is one line naming it, with the link to the page that explains it."""
body = data_client.get("/api/summary").json()
down = [item for item in body["alerts"] if item["source"] == "driveripper"]
assert len(down) == 1, "its stale payload must not add amber lines on top of the red one"
assert down[0]["severity"] == "fail"
assert down[0]["href"] == "/system"
assert "unreachable" in down[0]["message"]
def test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A dead agent is a red row in the fleet, not a fleet that quietly got smaller.
`/api/system/snapshot` already stubs a job with no payload; `/api/summary` has to agree
with it, or a target that never answered since startup leaves the table at "2/2 hosts"
while the panel above says a third box is down (`AGENTS.md` invariants 3 and 4).
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
body = test_client.get("/api/summary").json()
rows = body["system"]["hosts"]
assert [row["name"] for row in rows] == ["driveripper"]
assert rows[0]["unreachable"] is True
assert rows[0]["cpu_usage_pct"] is None, "n/a, never a 0.0 nobody measured"
assert body["system"]["fleet"]["hosts_total"] == 1
assert body["system"]["fleet"]["hosts_live"] == 0
# and the alert list stays exactly one line for it, not a row plus a stale payload
assert [item["source"] for item in body["alerts"]] == ["driveripper"]
def test_summary_alerts_are_worst_first_and_carry_the_contract(data_client) -> None:
body = data_client.get("/api/summary").json()
alerts_list = body["alerts"]
assert alerts_list, "a dead host plus simulated UPS data cannot read as all-clear"
for item in alerts_list:
assert set(item) == {"id", "severity", "source", "message", "href", "at"}
assert item["severity"] in ("warn", "fail")
ranks = [0 if item["severity"] == "fail" else 1 for item in alerts_list]
assert ranks == sorted(ranks), "fails before warns"
assert any("simulat" in item["message"].lower() and item["href"] == "/ups"
for item in alerts_list), "NUT_SIMULATE must be said out loud on the landing page"
def test_summary_reuses_the_alert_builder_verbatim(data_client) -> None:
"""The endpoint is a thin call: what `alerts.build` says over the same state is the body."""
body = data_client.get("/api/summary").json()
hub = data_client.app.state.ctx.hub # type: ignore[attr-defined]
settings = data_client.app.state.ctx.settings # type: ignore[attr-defined]
hosts = [job.state.payload for job in hub.jobs.values()
if job.dashboard == "system" and job.state.payload]
ups_payload = hub.jobs["nut"].state.payload
expected = alerts.build(settings, hub.states(), hosts, [], ups_payload, [])
assert [item["id"] for item in body["alerts"]] == [item["id"] for item in expected]
def test_snapshot_payloads_carry_their_own_severity_sections(data_client) -> None:
"""/api/system/snapshot and /api/ups/snapshot ship the same verdicts the summary reuses."""
system = data_client.get("/api/system/snapshot").json()
@@ -371,14 +520,103 @@ def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
"""The old call sites still work until phases 02-05 convert each page."""
"""The old call sites still work until the remaining phases convert each page.
`/` was converted in phase 02: the overview colours from the `severity` verdicts
`/api/summary` already sends, so it no longer carries a threshold of its own. The
three pages still to come must keep compiling against the fallback until they are.
"""
js = client.get("/static/app.js").text
assert "FALLBACK ONLY" in js
for page in ("/static/page-index.js", "/static/page-system.js",
"/static/page-llm.js", "/static/page-ups.js"):
for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"):
assert "severityFor" in client.get(page).text, f"{page} lost its fallback call site"
def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> None:
"""`page-index.js` reads verdicts; it does not compare a number to a bound."""
js = client.get("/static/page-index.js").text
assert "severityFrom" in js
assert "severityFor" not in js
def test_overview_is_exactly_two_panels(client) -> None:
"""The landing page answers one question: what is wrong, and what is the fleet doing.
Asserted on the served bytes because there is no JS test runner (a locked anchor):
the summary cards and the raw event feed are what made this page busy, and once a
panel is gone a browser is the only thing that would notice its return.
"""
html = client.get("/").text
assert html.count("<section") == 2, "the overview grew a third panel"
assert 'id="alerts"' in html and 'id="alerts-clear"' in html
assert 'aria-live="polite"' in html
assert 'id="fleet-table"' in html
for gone in ("sys-cpu", "sys-mem", "sys-disk", "sys-gpus", "llm-decode", "llm-kv",
"ups-list", 'id="events"'):
assert gone not in html, f"the overview still renders {gone}"
# the header's source dots are chrome owned by app.js, not the deleted card dots
assert 'id="dot-system"' not in html and 'id="nav-status"' in html
def test_overview_alert_rows_link_to_the_dashboard_that_explains_them(client) -> None:
"""Every alert carries the `href` its row turns into a link; the page invents none."""
page_js = client.get("/static/page-index.js").text
assert "alert.href" in page_js
css = client.get("/static/dash.css").text
for selector in ("ul.alerts li.ok .src", "ul.alerts li.warn .src", "ul.alerts li.fail .src",
"table.data td.warn", "table.data td.fail"):
assert selector in css, f"dash.css lost {selector}"
# the swap that puts a live problem above the fleet table
assert 'data-state="alerting"' in css
def test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering(client) -> None:
"""A dead host keeps its last good numbers, so the row has to say they are stale.
`AGENTS.md` invariant 4: keep the payload and mark it stale, never zero it. The
overview's table is the one place those last good numbers still print, so it reads the
per-source verdicts `/api/summary` sends (`system.states`) and applies the shared
`.stale` class the other pages already use — no new palette, no JS threshold.
"""
page_js = client.get("/static/page-index.js").text
assert "system.states" in page_js, "the fleet table stopped reading reachability"
assert 'class="stale"' in page_js
assert ".stale" in client.get("/static/dash.css").text
# Which script owns which page, for the orphan check below. `charts.js` and `app.js` are
# loaded by all four pages, so an id either page code or shared chrome may read.
PAGE_SCRIPTS = {
"/": "page-index.js",
"/system": "page-system.js",
"/llm": "page-llm.js",
"/ups": "page-ups.js",
}
SHARED_SCRIPTS = ("app.js", "charts.js")
def test_no_page_carries_an_id_nothing_reads(client) -> None:
"""An id no script queries, no stylesheet styles and no attribute points at is dead markup.
The summary cards were retired this way — a `<span id="sys-cpu">` left behind after its
writer was deleted is the same mistake in slower motion. Asserted on served bytes: the
browser console is otherwise the only thing that would notice.
"""
scripts = {name: client.get(f"/static/{name}").text for name in
(*SHARED_SCRIPTS, *PAGE_SCRIPTS.values(), "dash.css")}
for path, own_script in PAGE_SCRIPTS.items():
html = _code(client.get(path).text)
corpus = "".join(_code(scripts[name])
for name in (*SHARED_SCRIPTS, own_script, "dash.css"))
# ids the markup uses itself: labels, targets, `<label for>`
anchors = set(re.findall(
r'(?:aria-labelledby|aria-describedby|aria-controls|for)="([^"]+)"', html))
for element_id in re.findall(r'id="([^"]+)"', html):
quoted = (f'"{element_id}"' in corpus or f"'{element_id}'" in corpus
or f"#{element_id}" in corpus or element_id in anchors)
assert quoted, f"{path} has id=\"{element_id}\" that nothing reads"
def test_api_responses_are_not_cacheable(client) -> None:
for path in ("/api/config", "/api/health", "/api/summary"):
assert "no-store" in client.get(path).headers.get("cache-control", "")
@@ -413,6 +651,87 @@ def test_history_requires_the_series_coordinates(client) -> None:
assert client.get("/api/history").status_code == 422
@pytest.mark.parametrize(("raw", "seconds"), [
*[(key, float(seconds)) for key, seconds in RANGE_SECONDS.items()],
("45s", 45.0), ("13h", 46_800.0), ("7200", 7_200.0), (" 2D ", 172_800.0),
(None, 3_600.0), ("", 3_600.0),
])
def test_parse_range_reads_every_window_the_ui_offers(raw: Any, seconds: float) -> None:
"""Every chip key (5m…90d), any `<n><s|m|h|d>`, raw seconds, and the 1h default.
`parse_range` is the only thing between a typo'd query string and a chart of the wrong
window, and a wrong window is worse than an error: it still looks like data.
"""
assert parse_range(raw) == seconds
@pytest.mark.parametrize("raw", ["banana", "1w", "h", "-4h", "one"])
def test_parse_range_refuses_a_window_it_cannot_interpret(raw: str) -> None:
"""A 400 that names the value, never a silent fallback to 1h."""
with pytest.raises(HTTPException) as excinfo:
parse_range(raw)
assert excinfo.value.status_code == 400
assert raw in str(excinfo.value.detail)
# ---- the routes the *other* pages chart from -------------------------------- #
#
# `/` renders two panels now; `/system`, `/llm` and `/ups` still need these, and so does
# anything outside the browser. Retiring overview code must not retire them, so the shapes
# are pinned here rather than discovered by a page three phases from now.
def test_system_history_charts_every_configured_host_in_one_request(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""N hosts x M metrics, one round trip — the chart endpoint's whole reason to exist."""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),
Target("deskwork", DEAD_URL))) as test_client:
body = test_client.get("/api/system/history",
params={"metrics": "cpu.usage_pct,mem.used_pct",
"range": "1h"}).json()
assert body["hosts"] == ["driveripper", "deskwork"]
assert body["range_seconds"] == 3_600.0
assert len(body["series"]) == 4
assert {(item["host"], item["metric"]) for item in body["series"]} == {
("driveripper", "cpu.usage_pct"), ("driveripper", "mem.used_pct"),
("deskwork", "cpu.usage_pct"), ("deskwork", "mem.used_pct")}
# an explicit `hosts` list narrows it instead of adding to the default
narrowed = test_client.get("/api/system/history",
params={"metrics": "cpu.usage_pct",
"hosts": "deskwork"}).json()
assert narrowed["hosts"] == ["deskwork"]
def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
# a host target too, so the snapshot's job filter has to skip a non-LLM job
with build_client(tmp_path, monkeypatch, vllm_url=DEAD_URL,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
body = test_client.get("/api/llm/snapshot").json()
assert [engine["engine"] for engine in body["engines"]] == ["vLLM"]
assert body["engines"][0]["unreachable"] is True
assert body["fleet"]["engines_total"] == 1
assert body["fleet"]["engines_live"] == 0
assert [state["key"] for state in body["states"]] == ["llm:vllm"]
def test_llm_history_charts_both_engines_by_default(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
body = test_client.get("/api/llm/history",
params={"metrics": "decode_tokens_per_second,kv_cache_usage_ratio",
"range": "15m"}).json()
assert body["engines"] == ["gufo", "vllm"]
assert body["range_seconds"] == 900.0
assert {(item["host"], item["metric"]) for item in body["series"]} == {
("gufo", "llm.decode_tokens_per_second"),
("gufo", "llm.kv_cache_usage_ratio"),
("vllm", "llm.decode_tokens_per_second"),
("vllm", "llm.kv_cache_usage_ratio")}
def test_metrics_endpoint_lists_what_actually_has_history(client) -> None:
body = client.get("/api/metrics").json()
assert body["metrics"] == []
@@ -474,6 +793,153 @@ def test_a_dangerous_command_needs_the_confirmation_word(ups_client) -> None:
json={"ups": name, "command": deep, "confirm": "DANGER"}).status_code == 200
def test_a_command_the_driver_does_not_advertise_is_refused_before_it_is_sent(
ups_client) -> None:
"""Being in the risk catalog is not permission; `LIST CMD` from the driver is (invariant 1).
`shutdown.return` is a real NUT command the catalog knows and this UPS does not offer,
so the refusal must come from the advertisement check and not from the confirmation gate.
"""
name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
response = ups_client.post("/api/ups/command",
json={"ups": name, "command": "shutdown.return"})
assert response.status_code == 400
assert "does not advertise" in response.json()["detail"]
def test_cutting_the_output_is_refused_while_the_ups_is_already_on_battery(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Invariant 1, the one that matters most: no power cut while the pack is carrying the load.
Driven through the simulator's own state machine — a quick battery test puts the pack on
battery — so the 409 comes from the same poll the buttons are drawn from, with a 0.5s
interval to keep the wait short. Nothing here can touch real hardware: `NUT_SIMULATE=1`.
"""
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
nut_allow_actions=True, nut_interval=0.5) as test_client:
name = wait_for_payload(test_client, "/api/ups/snapshot")["ups"][0]["name"]
start = test_client.post("/api/ups/command",
json={"ups": name, "command": "test.battery.start.quick"})
assert start.status_code == 200
deadline = time.time() + 6.0
card: dict[str, Any] = {}
while time.time() < deadline:
card = next(item for item in test_client.get("/api/ups/snapshot").json()["ups"]
if item["name"] == name)
if card["on_battery"]:
break
time.sleep(0.05)
assert card.get("on_battery") is True, "the quick test must put the simulated pack on battery"
refusal = test_client.post("/api/ups/command",
json={"ups": name, "command": "load.off",
"confirm": "DANGER"})
assert refusal.status_code == 409
assert "on battery" in refusal.json()["detail"]
# and the same command is still offered once the pack is back on line power
assert not any(item["command"] == "load.off" and item["enabled"]
for item in card["commands"]), "the panel must not offer a blocked button"
class ScriptedNutClient:
"""`nut.NutClient` with the socket left out: the real code path, no battery attached.
`AGENTS.md` keeps live tests read-only, so this is where the one write endpoint's real
branch gets proven. The variable names are the ones `VAR_MAP` already pins from real
upsd captures; only the replies are scripted (a fake for behaviour, per the test policy).
"""
#: set per test: an exception instance to raise from `instcmd`, e.g. a refusal
refusal: Exception | None = None
sent: list[tuple[str, str]]
logged_in: list[str]
def __init__(self, host: str, port: int, *, timeout: float = 4.0,
username: str | None = None, password: str | None = None) -> None:
self.credentials = (username, password)
async def __aenter__(self) -> ScriptedNutClient:
return self
async def __aexit__(self, *_exc: object) -> None:
return None
async def list_ups(self) -> list[dict[str, str]]:
return [{"name": "apc", "description": "APC Back-UPS RS 1500MS"}]
async def list_vars(self, ups: str) -> dict[str, str]:
return {"ups.mfr": "APC", "ups.model": "Back-UPS RS 1500MS",
"ups.status": "OL", "battery.charge": "100", "ups.load": "18",
"battery.voltage": "13.6", "input.voltage": "230",
"battery.mfr.date": "2022-04", "ups.test.result": "Ok last test passed"}
async def list_cmds(self, ups: str) -> list[str]:
return ["test.panel.start", "beeper.mute", "load.off"]
async def list_rw_vars(self, ups: str) -> list[str]:
return []
async def login(self) -> None:
assert self.credentials[0], "a command send must authenticate as a named user"
ScriptedNutClient.logged_in.append(str(self.credentials[0]))
async def instcmd(self, ups: str, command: str) -> str:
if ScriptedNutClient.refusal is not None:
raise ScriptedNutClient.refusal
ScriptedNutClient.sent.append((ups, command))
return "OK Start Panel Test"
@pytest.fixture
def real_upsd_client(monkeypatch: pytest.MonkeyPatch) -> Iterator[type[ScriptedNutClient]]:
"""Swap the module-level `NutClient` so the collector and the send use the same stand-in."""
ScriptedNutClient.sent = []
ScriptedNutClient.logged_in = []
ScriptedNutClient.refusal = None
monkeypatch.setattr(nut, "NutClient", ScriptedNutClient)
yield ScriptedNutClient
@pytest.fixture
def nut_client_app(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
real_upsd_client: type[ScriptedNutClient]) -> Iterator[TestClient]:
"""A non-simulated UPS app: real code paths, scripted upsd, 0.5s polls."""
with build_client(tmp_path, monkeypatch, nut_host="127.0.0.1", nut_user="dashboards",
nut_password="not-the-real-one", nut_allow_actions=True,
nut_interval=0.5) as test_client:
yield test_client
def test_a_real_upsd_send_authenticates_sends_and_audits(nut_client_app,
real_upsd_client) -> None:
body = wait_for_payload(nut_client_app, "/api/ups/snapshot")
assert body["simulated"] is False, "this test is about the branch that talks to upsd"
name = body["ups"][0]["name"]
response = nut_client_app.post("/api/ups/command",
json={"ups": name, "command": "test.panel.start"})
assert response.status_code == 200, response.text
assert response.json()["simulated"] is False
assert real_upsd_client.sent == [(name, "test.panel.start")]
assert real_upsd_client.logged_in, "instcmd requires a LOGIN first (upsd refuses otherwise)"
events = nut_client_app.get("/api/events").json()["events"]
assert any(event["kind"] == "action" and name in event["message"] for event in events)
def test_a_refusal_from_upsd_is_a_502_and_an_error_event(nut_client_app,
real_upsd_client) -> None:
real_upsd_client.refusal = nut.NutError("Access denied", code="ACCESS-DENIED")
name = wait_for_payload(nut_client_app, "/api/ups/snapshot")["ups"][0]["name"]
response = nut_client_app.post("/api/ups/command",
json={"ups": name, "command": "beeper.mute"})
assert response.status_code == 502
detail = response.json()["detail"]
assert "ACCESS-DENIED" in detail or "Access denied" in detail, detail
assert real_upsd_client.sent == [], "a refused command was never accepted"
events = nut_client_app.get("/api/events").json()["events"]
failures = [event for event in events if event["kind"] == "action"
and event["severity"] == "error"]
assert failures, "an upsd refusal has to leave a trace, not just an HTTP status"
def test_unknown_commands_are_refused_even_with_actions_on(ups_client) -> None:
name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
for command in ("load.format", "rm -rf /", "INSTCMD x y"):