Files
dashboards/tests/test_api.py
T
ducoterra 28139df6d6 phase: 12_nut_self_test_verdict
Final state verified. Report:

**Verified (phase 12 — all 3 tasks already complete; this was the verification pass)**
- Re-ran the wire capture live: `ups.test.result` = `No test initiated` (apc) / `Done and passed` (cyberpower) — matches `LIVE_SELF_TEST_WORDS` + both fixtures; `/api/summary` has **no** self-test row, `/api/ups/snapshot` reads `cyberpower ok | Done and passed`, `apc unknown | …run a battery test`; both screenshots read back correct.
- Fixed a defect found in prior work while checking the "unrelated live noise" note: `NutMonitor.collect()` reused one upsd socket after a *partially-read* reply, so a 4 s stall produced bogus `expected 'BEGIN LIST VAR', got 'RW apc battery.charge.low "10"'` errors (same hazard on `INSTCMD`, where a late `OK` could be read as a different command's success). `NutClient` now marks a socket whose reply stopped early as `UNUSABLE`, naming the command that stalled; a complete `ERR …` line still leaves it usable (the per-UPS tolerance `collect()` depends on). +3 hermetic pins against the fake upsd; live re-check (09:33–09:44) now logs one line per stall, no cascade.

**Tests / lint / coverage (all green)**
- `bash .agents/validate.sh` → **exit 0**; `uv run pytest --cov --cov-fail-under=94` → **1172 passed, 11 skipped**, coverage **98.63 %** (floor 94); `uv run ruff check src tests agent` clean; `uv run pyright` 0 errors; `node --check` clean. Modified modules: `sources/nut.py` 95 %, `collectors.py` 97 % (no new line uncovered).
- `uv run pytest -m live --live` → **8 passed / 3 failed**, all three the pre-existing `driveripper` agent-transport failures AGENTS.md records (verified still true: unit `inactive`, no installed file, nothing on `:9120`).
- `./scripts/dev-server.sh status`: `nut ok`, `llm:* ok`, `host/disks:deskwork|gateway ok`; the two `driveripper` jobs DOWN.

**Completion criteria**
- Live capture recorded + asserts `ok` — met (re-confirmed from the wire today).
- `/ups` renders `ok`, overview silent, screenshots present — met.
- pytest / cov / ruff / pyright / validate — met (above).
- No other `health.checks[]` behaviour changed — met (diff confined to the `test_result` branch; all prior nut/ups-contract tests unmodified).

**Documentation**: README verified consistent (self-test wording, config table cross-checked against `config.py`, all documented commands/flags exist) — no changes needed beyond phase tasks' edit. AGENTS.md fixes: stale test count (1140 → 1172), `nvme0n1` power-on hours 19 158 → 19 160, gap 4 store-error claim corrected (last stored `store-error` 2026-10-18 06:33, none in the previous 24 h), new `collectors.py`/`sources/nut.py` socket-hygiene note, new gap-1 bullet on the upsd stall, plus live evidence appended to `data/logs/NOTES-self-test.md`.

**Deviations**: made no live *changes* (read-only probes only) — `driveripper`'s agent install needs root (`ops` has no installer/visudo grant) and a battery test stays the operator's call; the upsd 4 s stall itself is upstream-side. `uv pip list --outdated` still advisory (certifi/fastapi/starlette) — dependencies untouched this phase.

**Next pending phase**: `13_llm_page_and_decode_rate`.
2026-10-19 09:54:03 -04:00

3992 lines
216 KiB
Python

"""HTTP contract: what the pages may assume, and what must never leak.
The behaviours here are the ones a screenshot cannot show: an unconfigured source answering
404 `configured: false` instead of dialling a hostname that does not exist; a bearer token
that stops at the backend; the single write endpoint refusing on every gate; no external
asset the network can fail to deliver.
"""
from __future__ import annotations
import re
import time
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path
from typing import Any
import httpx
import pytest
from fastapi import HTTPException
from fastapi.testclient import TestClient
from auth_client import sign_in
from auth_contract import ADMIN_PASSWORD, FAST_PBKDF2
from dashboards import alerts, auth
from dashboards.api import MIN_PASSWORD_LENGTH, RANGE_SECONDS, parse_range
from dashboards.app import create_app
from dashboards.collectors import UpstreamError
from dashboards.config import Settings, Target
from dashboards.severity import classify
from dashboards.sources import nut
from dashboards.sources.disks import SMART_SERIES, DisksCollector, parse_disks_snapshot
from dashboards.sources.llm import (
HISTORY_METRICS,
parse_gufo_metrics,
parse_vllm_metrics,
)
from dashboards.store import Event, Sample, SampleStore
from slots_contract import (
CANONICAL_TASK_ID_FIELD,
OPTIONAL_NEXT_TOKEN_FIELDS,
OPTIONAL_ROW_FIELDS,
REQUIRED_NEXT_TOKEN_FIELDS,
REQUIRED_ROW_FIELDS,
TASK_ID_ALIASES,
USER_CONTENT_FIELDS,
load_slots_capture,
)
# refused immediately, and never dials out of the test environment
DEAD_URL = "http://127.0.0.1:1"
#: Every dashboard route, in nav order. The markup pins below (no CDN, no `.sev-*`, no
#: narration, no orphan `id`) run across this list rather than a hand-written four, so a
#: fifth page cannot join the masthead and skip the checks the others had to pass.
PAGES = ("/", "/system", "/disks", "/llm", "/ups")
#: `/settings` is a **page** and not a dashboard (phase 11): it carries the same masthead with
#: no `aria-current` tab, and the account link in every dashboard's masthead is how you reach
#: it. Kept out of `PAGES` because the pins over that tuple assert tab structure — one current
#: tab, a fixed tab order — which this page deliberately does not have. `MARKUP_PAGES` is the
#: list for the checks that are about the bytes rather than the nav.
SETTINGS_PAGE = "/settings"
MARKUP_PAGES = (*PAGES, SETTINGS_PAGE)
@contextmanager
def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: bool = False,
**overrides: Any) -> Iterator[TestClient]:
"""A TestClient whose lifespan runs, against a private DB and no real network.
The store is injected, so `create_app()` deliberately does not close it (production
owns its own); the caller here does, or every test leaks a SQLite handle and the
`--cov` log fills with ResourceWarnings that bury the warnings worth reading.
"""
if simulate:
monkeypatch.setenv("NUT_SIMULATE", "1")
else:
monkeypatch.delenv("NUT_SIMULATE", raising=False)
db = str(tmp_path / "api.sqlite3")
settings = Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2, **overrides)
store = SampleStore(db)
client = TestClient(create_app(settings, store))
client.__enter__()
try:
sign_in(client)
yield client
finally:
client.__exit__(None, None, None)
store.close()
@pytest.fixture
def client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[TestClient]:
with build_client(tmp_path, monkeypatch) as test_client:
yield test_client
@pytest.fixture
def ups_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[TestClient]:
"""NUT on the labelled simulator, actions allowed — the dev configuration."""
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
nut_allow_actions=True) as c:
yield c
def wait_for_payload(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
deadline = time.time() + timeout
while time.time() < deadline:
response = client.get(path)
if response.status_code == 200:
return response.json()
time.sleep(0.1)
pytest.fail(f"{path} never became available")
def failing_host_payload(scrape: Any, agent_snapshot: dict[str, Any]) -> dict[str, Any]:
"""Two scrapes of the real driveripper capture, nudged onto known sides of the table.
Driven from the fixture rather than hand-written so a schema drift breaks here first.
"""
def nudge(snap: dict[str, Any]) -> None:
snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95 # mem.used_pct -> fail
snap["cpu_temperature_celsius"] = 91.0 # cpu.temp_c -> fail
for fs in snap["filesystems"]:
fs["used_pct"] = 93.0 # disk -> fail
return scrape(agent_snapshot, cpu_busy=10.0, mutate=nudge)
@pytest.fixture
def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
agent_snapshot: dict[str, Any], host_scrape: Any) -> Iterator[TestClient]:
"""Overview-shaped app: one host payload in the hub plus the simulated UPSes.
The host job points at a dead port, so a poll can never overwrite the injected payload
(a failed poll keeps the last good one) — the numbers below are ours, not the fleet's.
"""
monkeypatch.setenv("NUT_SIMULATE", "1")
db = str(tmp_path / "summary.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db, nut_host="127.0.0.1",
admin_password=ADMIN_PASSWORD, pbkdf2_iterations=FAST_PBKDF2,
host_targets=(Target("driveripper", DEAD_URL),)), store)
test_client = TestClient(app)
test_client.__enter__()
try:
sign_in(test_client)
app.state.ctx.hub.jobs["host:driveripper"].state.payload = failing_host_payload(
host_scrape, agent_snapshot)
wait_for_payload(test_client, "/api/ups/snapshot")
yield test_client
finally:
test_client.__exit__(None, None, None)
store.close()
@pytest.fixture
def healthy_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
agent_snapshot: dict[str, Any],
host_scrape: Any) -> Iterator[TestClient]:
"""The all-clear app: one host answering on both of its jobs, nothing past a threshold.
The job's collector is swapped for one that hands back the injected payload, so the
health record says `ok` truthfully instead of being overridden behind the poller — and
that has to happen for the `disks` job too, which polls the same dead port: a host is
either answering or it is not, and a fixture that claims all-clear cannot leave one of
its two readers failing. Each reader gets **its own** projection of the same capture
(`parse_disks_snapshot` for the drive page), because a `disks` job holding a `system`
payload would make `/api/summary`'s drive block answer from the wrong shape and read as
a passing test that proved nothing.
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
from dashboards.poller import Result # noqa: PLC0415 - only this fixture needs it
payload = host_scrape(agent_snapshot, cpu_busy=10.0)
disks_payload = parse_disks_snapshot("driveripper", agent_snapshot,
DisksCollector()).payload
async def collect() -> Result:
return Result(payload=payload, samples=[])
async def collect_disks() -> Result:
return Result(payload=disks_payload, samples=[])
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
job = hub.jobs["host:driveripper"]
disks_job = hub.jobs["disks:driveripper"]
job.fn = collect
disks_job.fn = collect_disks
# let the poll already in flight against the dead port land first (it fails in
# microseconds), then take the health record over: the alerts must describe a host
# that is answering, not the half-second of startup where it was not
deadline = time.time() + 2.0
while time.time() < deadline and job.state.attempts == 0:
time.sleep(0.02)
time.sleep(0.05)
for reader in hub.jobs.values():
if reader.dashboard not in ("system", "disks"):
continue
reader.state.payload = disks_payload if reader is disks_job else payload
reader.state.ok = True
reader.state.error = None
reader.state.consecutive_failures = 0
reader.state.last_ok = time.time()
yield test_client
# ---- severity travels with the value --------------------------------------- #
def test_summary_carries_the_verdict_for_every_column_the_page_colours(data_client) -> None:
body = data_client.get("/api/summary").json()
row = body["system"]["hosts"][0]
assert row["severity"] == {"cpu_usage_pct": "ok", "mem_used_pct": "fail",
"disk_worst_used_pct": "fail", "temp_c": "fail"}
# the fleet card is judged by the same table, so amber means the same thing at both
# sizes — and no GPU verdict, at fleet size or host size (phase 06: busy is the job)
assert body["system"]["fleet"]["severity"] == {
"cpu_usage_pct_avg": "ok", "memory_used_pct": "fail",
"disk_worst_used_pct": "fail",
}
for ups in body["ups"]["ups"]:
assert set(ups["severity"]) == {"charge_pct", "load_pct", "runtime_seconds"}
assert ups["severity"]["charge_pct"] == classify("ups.charge_pct", ups["charge_pct"])
assert ups["severity"]["charge_pct"] == "ok", "the simulator starts at float charge"
def test_summary_shape_did_not_shrink(data_client) -> None:
"""Additive-only anchor: /api/summary gained `severity` and lost nothing.
The overview, the live tests, and any external consumer read the flat keys; this test is
the tripwire for a well-meaning refactor that "tidies" them away.
"""
body = data_client.get("/api/summary").json()
fleet_keys = {"hosts_total", "hosts_live", "cpu_cores", "cpu_usage_pct_avg",
"memory_total_bytes", "memory_used_bytes", "memory_used_pct",
"disk_worst_used_pct", "load1_total", "net_rx_bytes_per_s",
"net_tx_bytes_per_s", "gpu_count", "gpu_busy_pct_avg", "power_estimate_w"}
host_keys = {"name", "os", "cpu_usage_pct", "load1", "cores", "mem_used_pct",
"disk_worst_used_pct", "gpu_busy_pct", "temp_c", "uptime_seconds"}
ups_keys = {"name", "overall", "status", "charge_pct", "load_pct", "runtime_seconds"}
assert fleet_keys <= set(body["system"]["fleet"]), fleet_keys - set(body["system"]["fleet"])
assert host_keys <= set(body["system"]["hosts"][0]), host_keys - set(body["system"]["hosts"][0])
assert ups_keys <= set(body["ups"]["ups"][0]), ups_keys - set(body["ups"]["ups"][0])
# `disks` joined in phase 09 and changed nothing else: the set grows, never shrinks
assert set(body) == {"ts", "system", "disks", "llm", "ups", "recent_events", "alerts"}
# ...and the new keys are there beside the old ones, not instead of them
assert "severity" in body["system"]["fleet"]
assert "severity" in body["system"]["hosts"][0]
assert "severity" in body["ups"]["ups"][0]
# `/` paints two panels; the payload is still the whole-fleet contract. Every path here is
# read by something that is not the overview page: the live suite, an external probe, a
# notifier later. Deleting a card must never delete a key.
SUMMARY_CONTRACT = ("system.enabled", "system.fleet", "system.hosts",
"disks.enabled", "disks.fleet", "disks.hosts",
"llm.engines", "ups.ups", "recent_events", "alerts")
def dig(body: dict[str, Any], path: str) -> Any:
"""`body["a"]["b"]` that names the missing path when it is not there."""
node: Any = body
for part in path.split("."):
assert isinstance(node, dict) and part in node, f"/api/summary lost `{path}`"
node = node[part]
return node
@pytest.mark.parametrize("fixture", ["client", "data_client"])
def test_summary_serves_every_path_another_consumer_reads(request, fixture: str) -> None:
"""The keys survive the overview cleanup whether or not anything is configured.
Run twice on purpose: `client` has no targets at all, so it proves the keys are present
as `None`/`[]` rather than omitted when a source is unconfigured (invariant 3 is about
the payload shape as much as about colour); `data_client` proves the same paths carry
the fleet once there is one.
"""
test_client = request.getfixturevalue(fixture)
body = test_client.get("/api/summary").json()
populated = fixture == "data_client"
for path in SUMMARY_CONTRACT:
dig(body, path) # presence is the assertion; a missing path raises with its name
assert dig(body, "system.enabled") is populated
assert (dig(body, "system.fleet") is not None) is populated
names = {host["name"] for host in dig(body, "system.hosts")}
assert names == ({"driveripper"} if populated else set())
# the drive-health block is configured exactly when the agents it reads are, and lists the
# same hosts: a `disks` job exists for every `HOST_TARGETS` entry and for no other reason
assert dig(body, "disks.enabled") is populated
assert (dig(body, "disks.fleet") is not None) is populated
assert {host["name"] for host in dig(body, "disks.hosts")} == names
batteries = {item["name"] for item in dig(body, "ups.ups")}
assert batteries == ({"apc", "cyberpower"} if populated else set())
assert dig(body, "llm.engines") == [], "neither fixture configures an engine"
assert isinstance(dig(body, "recent_events"), list)
assert isinstance(dig(body, "alerts"), list)
# and the payload as a whole still carries every top-level section it always did
assert set(body) >= {"ts", "system", "llm", "ups", "recent_events", "alerts"}
# ---- alerts: the landing page's answer to "what is wrong" ------------------- #
def test_summary_is_empty_of_alerts_when_everything_is_healthy(healthy_client) -> None:
"""`[]`, not a green banner: an empty list is the clearest possible all-clear."""
body = healthy_client.get("/api/summary").json()
assert body["alerts"] == []
assert body["system"]["fleet"]["hosts_live"] == 1
def test_summary_raises_one_alert_for_the_host_that_stopped_answering(data_client) -> None:
"""One dead host is one line naming it, with the link to the page that explains it."""
body = data_client.get("/api/summary").json()
down = [item for item in body["alerts"] if item["source"] == "driveripper"]
assert len(down) == 1, "its stale payload must not add amber lines on top of the red one"
assert down[0]["severity"] == "fail"
assert down[0]["href"] == "/system"
assert "unreachable" in down[0]["message"]
def test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A dead agent is a red row in the fleet, not a fleet that quietly got smaller.
`/api/system/snapshot` already stubs a job with no payload; `/api/summary` has to agree
with it, or a target that never answered since startup leaves the table at "2/2 hosts"
while the panel above says a third box is down (`AGENTS.md` invariants 3 and 4).
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
body = test_client.get("/api/summary").json()
rows = body["system"]["hosts"]
assert [row["name"] for row in rows] == ["driveripper"]
assert rows[0]["unreachable"] is True
assert rows[0]["cpu_usage_pct"] is None, "n/a, never a 0.0 nobody measured"
assert body["system"]["fleet"]["hosts_total"] == 1
assert body["system"]["fleet"]["hosts_live"] == 0
# and the alert list stays exactly one line for it, not a row plus a stale payload
assert [item["source"] for item in body["alerts"]] == ["driveripper"]
def test_summary_alerts_are_worst_first_and_carry_the_contract(data_client) -> None:
body = data_client.get("/api/summary").json()
alerts_list = body["alerts"]
assert alerts_list, "a dead host plus simulated UPS data cannot read as all-clear"
for item in alerts_list:
assert set(item) == {"id", "severity", "source", "message", "href", "at"}
assert item["severity"] in ("warn", "fail")
ranks = [0 if item["severity"] == "fail" else 1 for item in alerts_list]
assert ranks == sorted(ranks), "fails before warns"
assert any("simulat" in item["message"].lower() and item["href"] == "/ups"
for item in alerts_list), "NUT_SIMULATE must be said out loud on the landing page"
def test_summary_reuses_the_alert_builder_verbatim(data_client) -> None:
"""The endpoint is a thin call: what `alerts.build` says over the same state is the body."""
body = data_client.get("/api/summary").json()
hub = data_client.app.state.ctx.hub # type: ignore[attr-defined]
settings = data_client.app.state.ctx.settings # type: ignore[attr-defined]
hosts = [job.state.payload for job in hub.jobs.values()
if job.dashboard == "system" and job.state.payload]
ups_payload = hub.jobs["nut"].state.payload
expected = alerts.build(settings, hub.states(), hosts, [], ups_payload, [])
assert [item["id"] for item in body["alerts"]] == [item["id"] for item in expected]
def test_snapshot_payloads_carry_their_own_severity_sections(data_client) -> None:
"""/api/system/snapshot and /api/ups/snapshot ship the same verdicts the summary reuses."""
system = data_client.get("/api/system/snapshot").json()
host = system["hosts"][0]
assert set(host["cpu"]["severity"]) == {"usage_pct", "temp_c"}
assert set(host["memory"]["severity"]) == {"used_pct"}
assert set(host["disks"]["severity"]) == {"worst_used_pct"}
assert "severity" not in host["gpus"][0], (
"a GPU verdict came back on /api/system/snapshot — phase 06 removed it"
)
ups = data_client.get("/api/ups/snapshot").json()
for card in ups["ups"]:
assert set(card["severity"]) == {"charge_pct", "load_pct", "battery_runtime"}
# numbers keep their own shape; severity is a sibling, never a replacement
assert isinstance(card["numbers"], dict) and "charge_pct" in card["numbers"]
# ---- the self-test verdict on the two surfaces that were wrong (phase 12) --- #
#: The `ups.test.result` sentences as the hardware answers them. `Done and passed` is the
#: CyberPower's verbatim word from the 2026-10-19 capture (`tests/test_nut.py
#: ::LIVE_SELF_TEST_WORDS`); `FAILED` is what a driver says when a test ran and did not pass,
#: and is the reason the check exists at all. `tests/test_nut.py` pins the *word map*; what is
#: pinned here is the surface `TODO.md:3` names — a warning in the overview's Warnings panel —
#: because a verdict that grades correctly inside `assess()` and still reaches the alert feed
#: as a warning is the same bug wearing a different hat.
SELF_TEST_PASS_WORD = "Done and passed"
SELF_TEST_FAIL_WORD = "FAILED"
#: How `/api/summary` says the check out loud (`alerts._ups_alerts`): `"<check name>: <detail>"`.
SELF_TEST_ALERT_TEXT = "Last self-test"
def set_self_test_verdict(payload: dict[str, Any], status: str, detail: str) -> None:
"""Rewrite every card's `Last self-test` check in the hub's own UPS payload.
Mutating the payload the job actually serves (instead of hand-building a fake one) is what
makes this an end-to-end pin: `/api/ups/snapshot` and `alerts.build()` then read the *same*
object, so the test cannot pass by having the two surfaces disagree about the verdict.
"""
graded = 0
for card in payload["ups"]:
for check in card["health"]["checks"]:
if check["name"] == SELF_TEST_ALERT_TEXT:
check["status"] = status
check["detail"] = detail
graded += 1
assert graded, "no `Last self-test` check in the UPS payload — nothing was tested"
@contextmanager
def self_test_app(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
) -> Iterator[tuple[TestClient, dict[str, Any]]]:
"""A UPS app whose served payload cannot be replaced by a poll mid-assertion.
The simulated UPSes answer `PASSED`, so the pass direction would survive a race with the
poll loop and the *fail* direction would not — a test that flakes on a poll is a test of
timing. Hence `healthy_client`'s trick for the `nut` job: the collector hands back the very
dict this harness holds, so the health record says `ok` truthfully (invariant 4) and the
payload stays ours. The interval is shortened for the same reason — waiting a real
`NUT_INTERVAL_SECONDS` per test would only make the suite slower at proving the same thing.
"""
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
nut_interval=0.5) as test_client:
from dashboards.poller import Result # noqa: PLC0415 - only this harness needs it
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
job = hub.jobs["nut"]
wait_for_payload(test_client, "/api/ups/snapshot")
payload = job.state.payload
async def collect() -> Result:
return Result(payload=payload, samples=[])
job.fn = collect
# let one poll of *our* collector land before the caller mutates the payload, so the
# health record and the served bytes are both demonstrably ours
baseline = job.state.attempts
deadline = time.time() + 5.0
while time.time() < deadline and job.state.attempts <= baseline:
time.sleep(0.02)
assert job.state.payload is payload and job.state.ok, (
"the nut job is not serving the payload under test — the pin below would be fiction"
)
yield test_client, payload
def self_test_alerts(alerts_list: list[dict[str, Any]]) -> list[dict[str, Any]]:
return [item for item in alerts_list
if SELF_TEST_ALERT_TEXT.lower() in item["message"].lower()
or SELF_TEST_ALERT_TEXT.lower() in item["id"].lower()]
def test_a_passed_self_test_is_quiet_on_the_overview_and_ok_on_the_ups_page(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The TODO's sentence, end to end: `Done and passed` is not a warning anywhere.
Both halves of the fix are asserted on one payload: `/api/ups/snapshot` carries the `ok`
entry the `/ups` check list renders verbatim, and `/api/summary` — the only call the
overview makes — carries **no** row naming it. The simulated card still contributes its
`NUT_SIMULATE` line, so the list is not empty and this cannot be a vacuous pass.
"""
with self_test_app(tmp_path, monkeypatch) as (test_client, payload):
set_self_test_verdict(payload, "ok", SELF_TEST_PASS_WORD)
cards = wait_for_payload(test_client, "/api/ups/snapshot")["ups"]
for card in cards:
check = next(c for c in card["health"]["checks"]
if c["name"] == SELF_TEST_ALERT_TEXT)
assert check["status"] == "ok", f"{card['name']} reports a pass as {check['status']}"
assert check["detail"] == SELF_TEST_PASS_WORD, "the driver's word is the useful half"
rows = self_test_alerts(test_client.get("/api/summary").json()["alerts"])
assert rows == [], f"a passed battery test is on the overview as {rows}"
assert any(item["source"] == "ups" for item in
test_client.get("/api/summary").json()["alerts"]), (
"the simulated-UPS line disappeared, so this assertion has nothing left to be quiet about"
)
def test_a_never_tested_battery_is_not_an_alert_either(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The APC's factory `No test initiated` reaches the overview as silence, not amber.
This is the state the live hardware is actually in on one of the two UPSes, so the
screenshot proves the absence of a warning for a *never-run* test while the payload pin
above proves it for a passed one. `unknown` is not a verdict an operator can act on, and
`alerts._RANK` admits only `fail` and `warn`.
"""
with self_test_app(tmp_path, monkeypatch) as (test_client, payload):
set_self_test_verdict(payload, "unknown", nut.NEVER_TESTED_DETAIL)
cards = wait_for_payload(test_client, "/api/ups/snapshot")["ups"]
assert all(next(c for c in card["health"]["checks"]
if c["name"] == SELF_TEST_ALERT_TEXT)["status"] == "unknown"
for card in cards)
assert self_test_alerts(test_client.get("/api/summary").json()["alerts"]) == []
def test_a_failed_self_test_still_reaches_the_overview(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The check keeps its reason for existing: a test that failed is a red line, once per UPS.
Without this direction the absence test above is satisfiable by deleting the check, which
is the failure mode the phase's completion criterion calls out.
"""
with self_test_app(tmp_path, monkeypatch) as (test_client, payload):
set_self_test_verdict(payload, "fail", SELF_TEST_FAIL_WORD)
names = {card["name"] for card in
wait_for_payload(test_client, "/api/ups/snapshot")["ups"]}
assert names, "no UPS card to fail"
rows = self_test_alerts(test_client.get("/api/summary").json()["alerts"])
assert {row["source"] for row in rows} == names
for row in rows:
assert row["severity"] == "fail", f"a failed battery test reads {row['severity']}"
assert row["href"] == "/ups", "the alert must link to the card that names the UPS"
assert SELF_TEST_ALERT_TEXT in row["message"] and SELF_TEST_FAIL_WORD in row["message"]
assert row["message"].count(SELF_TEST_ALERT_TEXT) == 1, (
"the row names the check twice — see alerts._clean_error")
assert all(row["id"] == f"{row['source']}:health:{SELF_TEST_ALERT_TEXT}" for row in rows)
def test_the_ups_check_list_needs_no_new_markup_for_a_passed_verdict(client) -> None:
"""Task 03 expected no page change, and this pins *why* there is none.
`page-ups.js` puts `check.status` straight on the `<li>`, and `dash.css` colours only the
`.mark` glyph from the shared `--ok`/`--warn`/`--fail` tokens while the check's name and
detail stay `--text`/`--text-muted` — so an `ok` row is quiet ink with a tick, and a verdict
fixed in `assess()` is a verdict rendered. Nothing on either asset names the self-test
check, which is the point: a special case added here would be a second opinion about the
driver's word, and the API already made it.
"""
js = _code(client.get("/static/page-ups.js").text)
assert '<li class="${check.status}">' in js, "the row no longer wears the payload verdict as its class"
for verdict in ("ok", "warn", "fail", "unknown"):
assert f"{verdict}: '" in js[js.index("const MARKS"):], f"MARKS lost the {verdict} glyph"
rows = js[js.index("const health = item.health"):js.index("const commands = item.commands")]
assert "self-test" not in rows.lower(), "the check list grew an opinion about one check by name"
css = _code(_css(client))
ok_rule = re.search(r"ul\.checks li\.ok \.mark \{([^}]*)\}", css)
assert ok_rule and "var(--ok)" in ok_rule.group(1), "the ok tick is not the shared token"
assert "ul.checks li.ok {" not in css, "the whole ok row is coloured — a pass is not a banner"
assert "self-test" not in css.lower(), "the stylesheet special-cases one check by name"
# ---- unconfigured: say so, and do not dial -------------------------------- #
def test_config_reports_every_source_as_off(client) -> None:
body = client.get("/api/config").json()
assert body["system"]["enabled"] is False
assert body["system"]["targets"] == []
assert body["disks"] == {"enabled": False, "interval_seconds": 300.0,
"timeout_seconds": 30.0}, (
"the drive-health page is configured exactly when the host agents are")
assert body["llm"]["gufo"]["enabled"] is False
assert body["llm"]["vllm"]["enabled"] is False
assert body["ups"]["enabled"] is False
assert body["ups"]["actions_enabled"] is False
assert body["ranges"], "the page builds its window buttons from this list"
def test_health_with_nothing_configured_is_not_a_failure(client) -> None:
"""Nothing configured is 'empty', not 'broken' — and the page must not go red."""
body = client.get("/api/health").json()
assert body["status"] in ("ok", "empty")
assert body["ok"] is True
assert body["sources"] == []
def test_the_disk_source_has_its_own_health_record_per_host(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""`disks` is a source of its own, not a field of `system` (phase 09).
Its own record is what gives the page its own dot and its own staleness, and it is
also what makes `alerts.py` need a rule: two jobs read the same agent, so one dead
host must still be one line on the overview (see `test_summary_raises_one_alert...`).
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
sources = {row["key"]: row for row in test_client.get("/api/health").json()["sources"]}
assert "disks:driveripper" in sources, "no separate record, no separate dot"
assert sources["disks:driveripper"]["dashboard"] == "disks"
assert sources["disks:driveripper"]["label"] == "driveripper"
assert sources["disks:driveripper"]["configured"] is True
assert sources["host:driveripper"]["dashboard"] == "system", (
"the system job is still its own source: one slow SMART poll must not "
"decide whether the host looks alive")
def test_no_targets_means_no_disk_job_that_could_look_broken(client) -> None:
"""No host agents, no `disks` task at all — so nothing red for an unconfigured page."""
assert client.get("/api/health").json()["sources"] == []
@pytest.mark.parametrize("path", ["/api/system/snapshot", "/api/llm/snapshot",
"/api/ups/snapshot", "/api/ups/commands"])
def test_unconfigured_source_is_404_configured_false(client, path) -> None:
response = client.get(path)
assert response.status_code == 404
assert response.json()["configured"] is False
assert "detail" in response.json()
def test_summary_still_answers_with_no_sources(client) -> None:
body = client.get("/api/summary").json()
assert body["system"]["enabled"] is False
assert body["system"]["fleet"] is None
assert body["llm"]["enabled"] is False
assert body["ups"]["enabled"] is False
def test_unknown_route_is_json_not_html(client) -> None:
response = client.get("/api/nope")
assert response.status_code == 404
assert response.headers["content-type"].startswith("application/json")
assert "no such route" in response.json()["detail"]
def test_an_upstream_failure_escaping_a_route_keeps_its_own_status(tmp_path: Path) -> None:
"""`collectors.UpstreamError` is the one error type a route may raise to say "upstream".
The handler turns it into `{ok: false, detail}` with the collector's status (502, or 504
for a timeout) instead of a 500 traceback, and the message is the collector's already
redacted sentence. Nothing dials here: the error is raised by the route under test.
"""
db = str(tmp_path / "upstream-error.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2), store)
@app.get("/api/_boom")
async def boom() -> None:
raise UpstreamError("driveripper: http://127.0.0.1:9120/snapshot timed out", status=504)
with TestClient(app) as test_client:
sign_in(test_client)
response = test_client.get("/api/_boom")
assert response.status_code == 504, "a timeout stays a timeout, not a 500"
assert response.json() == {"ok": False,
"detail": "driveripper: http://127.0.0.1:9120/snapshot "
"timed out"}
assert "no-store" in response.headers["cache-control"]
store.close()
# ---- tokens --------------------------------------------------------------- #
def test_agent_tokens_never_reach_the_browser(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("NUT_SIMULATE", "")
targets = [Target(name="driveripper", url="http://driveripper:9120",
headers={"Authorization": "Bearer SEKRIT-TOKEN"})]
db = str(tmp_path / "tokens.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2,
host_targets=tuple(targets)), store)
with TestClient(app) as client:
sign_in(client)
body = client.get("/api/config")
assert "SEKRIT-TOKEN" not in body.text
target = body.json()["system"]["targets"][0]
assert target["name"] == "driveripper"
assert "SEKRIT" not in target["url"] and "@" not in target["url"]
# and a host that never answered stays in the list, marked, with the fleet card intact
snapshot = wait_for_state(client, "/api/system/snapshot")
assert snapshot["hosts"][0]["name"] == "driveripper"
assert snapshot["hosts"][0].get("unreachable") is True
assert snapshot["fleet"]["hosts_total"] == 1
assert snapshot["fleet"]["hosts_live"] == 0
assert snapshot["fleet"]["memory_total_bytes"] is None
store.close()
def test_a_dead_host_never_writes_its_token_into_the_feed(tmp_path: Path) -> None:
"""The second path a credential can take to a browser: the failure message.
`parse_targets` moves a userinfo token out of the URL and into a bearer header, so the
documented `HOST_TARGETS` path never puts one in `Target.url` — this is the backstop for
any `Target` that does (a hand-built one, a future parser), and the message path is where
it would matter: the poller keeps an `UpstreamError` string in the health record and logs
it as an event, so a userinfo-bearing URL there would sit in SQLite and be served on
`/api/events` forever (AGENTS.md invariant 6). Asserted on every endpoint a page reads,
not just `/api/config`.
"""
db = str(tmp_path / "dead-host-token.sqlite3")
store = SampleStore(db)
settings = Settings(
db_path=db,
admin_password=ADMIN_PASSWORD, pbkdf2_iterations=FAST_PBKDF2,
host_interval=0.05,
host_targets=(Target(name="driveripper", url="http://SEKRIT-AGENT-TOKEN@127.0.0.1:1"),),
)
with TestClient(create_app(settings, store)) as client:
sign_in(client)
deadline = time.time() + 10
while time.time() < deadline:
health = client.get("/api/health").json()
if health["sources"] and health["sources"][0]["error"]:
break
time.sleep(0.05)
else:
pytest.fail("the dead host never recorded an error")
for path in ("/api/health", "/api/events", "/api/system/snapshot",
"/api/disks/snapshot", "/api/summary"):
body = client.get(path).text
assert "SEKRIT-AGENT-TOKEN" not in body, f"{path} leaked the agent token"
# and the outage is still legible — redaction is not the same as silence
assert "driveripper" in client.get("/api/health").text
assert "unreachable" in client.get("/api/events").text
# the `disks` job dials the same token-bearing URL on its own cadence, so its
# sentence is a second copy of the same risk: `fetch_snapshot` is the one place
# either dashboard builds a failure message, and it uses the redacted URL
disks_state = next(row for row in client.get("/api/health").json()["sources"]
if row["key"] == "disks:driveripper")
assert disks_state["error"] and "SEKRIT-AGENT-TOKEN" not in disks_state["error"]
assert "127.0.0.1:1/snapshot" in disks_state["error"], "the failure still names where"
store.close()
def test_engine_urls_are_redacted_the_same_way_hosts_are(tmp_path: Path) -> None:
"""`GUFO_BASE_URL`/`VLLM_BASE_URL` go to the browser verbatim in `/api/config`, and unlike
`HOST_TARGETS` nothing parses a userinfo token out of them into a header — so a URL that
happens to carry credentials has to be redacted where it is served (invariant 6).
"""
db = str(tmp_path / "engine-redact.sqlite3")
settings = Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2,
gufo_url="http://SEKRIT-ENGINE-TOKEN@deskwork:8731",
vllm_url="http://driveripper:8000")
store = SampleStore(db)
with TestClient(create_app(settings, store)) as client:
sign_in(client)
body = client.get("/api/config")
assert "SEKRIT-ENGINE-TOKEN" not in body.text
llm = body.json()["llm"]
assert llm["gufo"]["base_url"] == "http://deskwork:8731"
assert llm["vllm"]["base_url"] == "http://driveripper:8000" # untouched when clean
store.close()
def wait_for_state(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
deadline = time.time() + timeout
while time.time() < deadline:
response = client.get(path)
if response.status_code == 200:
return response.json()
time.sleep(0.1)
pytest.fail(f"{path} never answered")
def test_target_problems_are_surfaced_not_swallowed(client) -> None:
body = client.get("/api/config").json()
assert body["system"]["target_problems"] == []
# ---- the credential the server refuses to start without (phase 11) --------- #
def test_no_verifier_and_no_seed_means_the_server_does_not_come_up(tmp_path: Path) -> None:
"""Fail closed, at the one place a mistake is unmissable: startup.
Task 03 is what demands a session on every route; this is the half before it — a deployment
with neither a stored `auth.verifier` row nor `DASH_ADMIN_PASSWORD` must not run open, so
the lifespan raises instead of publishing a context, starting the pollers, or serving a
page. Asserting that nothing was written is part of it: "refused" that seeded a row on the
way out would leave the next start half-configured.
"""
db = str(tmp_path / "no-credential.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db, admin_password=""), store)
try:
with pytest.raises(RuntimeError) as caught, TestClient(app):
pytest.fail("the lifespan yielded with no admin credential")
message = str(caught.value)
assert "DASH_ADMIN_PASSWORD" in message, "the operator's fix is in the exit output"
assert "auth.verifier" in message and "DELETE FROM meta" in message
assert not hasattr(app.state, "ctx"), "nothing is published, so nothing can serve"
assert store.get_meta("auth.verifier") is None
assert store.get_meta("auth.session_secret") is None
finally:
store.close()
def test_the_same_server_starts_once_the_environment_seeds_a_password(tmp_path: Path) -> None:
"""The other side of the gate, so the failure above cannot be satisfied by a broken app.
With a seed the process comes up, the state hangs off the context (task 03 reads it on
every request), and both rows are durable — which is also what lets the operator delete
`DASH_ADMIN_PASSWORD` from the deployment after the first boot.
"""
db = str(tmp_path / "seeded.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2), store)
try:
with TestClient(app) as client:
assert client.get("/api/health").status_code == 200 # public: the healthcheck
state = app.state.ctx.auth
assert state is not None
assert state.seeded is True and state.env_ignored is False
assert state.iterations == FAST_PBKDF2
assert auth.verify_password(ADMIN_PASSWORD, state.verifier) is True
assert store.get_meta("auth.verifier") == state.verifier
assert store.get_meta("auth.session_secret") == state.secret
assert ADMIN_PASSWORD not in state.verifier + state.secret
finally:
store.close()
# ---- pages ---------------------------------------------------------------- #
@pytest.mark.parametrize("path,title", [
("/", "overview"), ("/system", "system"), ("/disks", "disks"), ("/llm", "llm"),
("/ups", "ups"),
])
def test_pages_serve_and_declare_themselves(client, path, title) -> None:
response = client.get(path)
assert response.status_code == 200
assert "text/html" in response.headers["content-type"]
assert title in response.text.lower()
def test_no_page_loads_anything_from_the_internet(client) -> None:
"""No CDN, no webfonts: this dashboard has to work with no route to the internet.
`/login` is not in `MARKUP_PAGES` and cannot be: this `client` is signed in, a signed-in
`GET /login` is a `303`, and following it would audit `/`'s bytes under the wrong name. The
same fragments are checked against the real login page, unsigned, by
`tests/test_auth_api.py::test_the_login_page_loads_nothing_from_the_internet`.
"""
for path in MARKUP_PAGES:
html = client.get(path).text
for fragment in ("//cdn.", "https://fonts.", "//unpkg.com", "//cdnjs"):
assert fragment not in html, f"{path} references {fragment}"
def test_static_assets_are_served(client) -> None:
for asset in ASSETS:
assert client.get(asset).status_code == 200, asset
# ---- /settings: the account page (phase 11 task 06) ----------------------- #
#
# `/settings` is the operator's own surface: the session this browser holds, a sign-out, and the
# one form that changes the admin password. Its HTTP surface is tested twice on purpose — the
# route-enumeration test in `tests/test_auth_api.py` proves nothing outside the public list
# answers anonymously (which is how `/settings` is gated for free by being in `PAGES` in
# `app.py`), and the assertions here pin the **page**: the form fields a password manager needs,
# the outcome line that is a live region, the precedence sentence that stops the "I edited the
# env file and nothing happened" support call, and the masthead link that is the only way in.
def test_the_settings_page_carries_the_session_and_the_form(client) -> None:
response = client.get(SETTINGS_PAGE)
assert response.status_code == 200
assert "text/html" in response.headers["content-type"]
html = response.text
# the session panel and its way out
for fragment in ('id="sign-out"', 'id="session-user"', 'id="session-expires"',
'id="session-left"'):
assert fragment in html, f"/settings has no {fragment}"
# the change-password form: three fields, each labelled, each with the floor spelled out,
# and the autocomplete values a password manager needs to fill the right one
for fragment in ('id="current-password"', 'autocomplete="current-password"',
'id="new-password"', 'autocomplete="new-password"',
'id="confirm-password"', 'id="change-password"',
'<label for="current-password">', '<label for="new-password">',
'<label for="confirm-password">', 'aria-describedby="password-hint"'):
assert fragment in html, f"/settings' form is missing {fragment}"
assert "8 characters" in html, "the form stopped naming the length floor it enforces"
# the outcome line is announced, because the operator asked for it and nothing else will
assert 'id="settings-result"' in html and 'role="status"' in html
def test_the_settings_page_states_the_precedence_rule(client) -> None:
"""The sentence that makes the env file's losing half legible on the page that wins.
`README.md` documents the same rule, but the operator reading it there is deciding what to
write in a file; the operator reading *this* page just changed a password and needs to know
why a container env will not undo it.
"""
html = client.get(SETTINGS_PAGE).text
assert "DASH_ADMIN_PASSWORD" in html
assert "auth.verifier" in html
assert "never overwrites" in html
def test_the_settings_page_is_not_one_of_the_dashboards(client) -> None:
"""The tabs are there to leave with; nothing marks itself current, because nothing is.
A tab marked current on a page that is not in the tab list claims a location the app does not
have, and the four dashboards' own masthead pin (`aria-current` exactly once) relies on that
marker meaning something.
"""
html = client.get(SETTINGS_PAGE).text
assert '<nav class="tabs"' in html
assert 'aria-current="page"' not in html
for path in PAGES:
assert f'href="{path}"' in html, f"/settings cannot navigate to {path}"
assert 'id="nav-status"' not in html, "/settings grew fleet dots it has nothing to fill"
def test_the_settings_page_refuses_a_caller_with_no_session(tmp_path: Path) -> None:
"""One page-level assertion of the gate, not a second enumeration of it.
`tests/test_auth_api.py::test_every_route_that_is_not_public_requires_a_session` is the
general proof; this one exists so a regression in `/settings` names `/settings`.
"""
db = str(tmp_path / "settings-gate.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2), store)
try:
with TestClient(app, follow_redirects=False) as anon:
page = anon.get(SETTINGS_PAGE, headers={"accept": "text/html"})
assert page.status_code == 303
assert page.headers["location"] == f"/login?next={SETTINGS_PAGE}"
data = anon.get(SETTINGS_PAGE, headers={"accept": "application/json"})
assert data.status_code == 401
assert data.json() == {"ok": False, "detail": "sign in required",
"login": "/login"}
finally:
store.close()
def test_every_dashboard_page_links_to_settings(client) -> None:
"""The `⚙ Account` link is the only way to the session page, so it is on every dashboard.
It sits in `.masthead-right` before the theme toggle and **outside** `.tabs`, because the tab
strip's structure — five dashboards, one of them current — is pinned by
`test_the_disks_tab_is_in_the_masthead_of_every_page` and a sixth entry with no
`aria-current` would break what that marker means.
"""
link = '<a class="ghost account" href="/settings"'
for path in PAGES:
html = client.get(path).text
assert link in html, f"{path} has no route to the account page"
tabs = html[html.index('<nav class="tabs"'):html.index("</nav>")]
assert "/settings" not in tabs, f"{path} made Account a dashboard tab"
assert html.index(link) < html.index('id="theme-toggle"'), \
f"{path} put the account link after the theme toggle"
# the stylesheet styles the link with the button's own rule, not a copy of it
css = client.get("/static/dash.css").text
assert "a.ghost" in css
def test_the_password_floor_the_form_checks_is_the_one_the_api_enforces(client) -> None:
"""The client-side length check is a courtesy; the server's is the rule.
There is no build step to share the constant (a locked anchor), so the two literals are
pinned together here: a floor raised in `api.py` alone would leave the form promising eight
characters to a request the server answers `422`.
"""
js = client.get("/static/page-settings.js").text
found = re.search(r"const MIN_PASSWORD_LENGTH = (\d+);", js)
assert found, "page-settings.js no longer names its length floor"
assert int(found.group(1)) == MIN_PASSWORD_LENGTH
def test_a_rotated_secret_clears_the_cached_session_every_page_uses(client) -> None:
"""`Dash.resetAuth()` is the difference between a password change and a 403 dashboard.
`POST /api/auth/password` rotates the signing secret on purpose (`auth.update_verifier`),
which invalidates the CSRF token `ensureAuth()` memoised. Every page writes through
`fetchJSON`, which reads that cache — so unless the change clears it, the operator changes
the password, clicks *beeper disable* on `/ups`, and gets `missing or invalid CSRF token`
until they reload the page. A bounce on a `401` clears it for the same reason: the session it
described no longer exists.
"""
app_js = client.get("/static/app.js").text
assert "function resetAuth(" in app_js
exports = app_js.rsplit("return {", 1)[1].split("}", 1)[0]
assert re.search(r"\bresetAuth\b", exports), "Dash.resetAuth is not exported"
reset = app_js[app_js.index("function resetAuth("):app_js.index("function bounceToLogin(")]
for field in ("loaded: false", "authenticated: false", "csrf: null"):
assert field in reset, f"resetAuth() left `{field}` cached"
assert "authPromise = null" in reset, \
"resetAuth() left an in-flight status call memoised — the fresh write would reuse it"
bounce = app_js[app_js.index("function bounceToLogin("):app_js.index("const READ_METHODS")]
assert "resetAuth()" in bounce, "a 401 bounce kept the dead session's token cached"
js = client.get("/static/page-settings.js").text
assert js.count("Dash.resetAuth()") >= 2, \
"the settings page must reset the cached session after both a change and a sign-out"
assert "textContent" in js and "innerHTML" not in js, \
"/settings renders a server sentence as markup"
# ---- the severity language in the static assets --------------------------- #
#
# There is no JS test runner (a locked anchor: adding one would mean a build step).
# What *can* be asserted end-to-end, from the served bytes, is the contract the pages
# depend on: the hero style exists, its colour comes from the shared ok/warn/fail tokens,
# no parallel `.sev-*` vocabulary appeared, and `Dash.hero()` reads the verdict instead of
# comparing the number to a threshold in JavaScript.
ASSETS = ("/static/dash.css", "/static/app.js", "/static/charts.js",
"/static/page-index.js", "/static/page-disks.js", "/static/page-system.js",
"/static/page-llm.js", "/static/page-ups.js", "/static/page-login.js",
"/static/page-settings.js")
def _css(client: TestClient) -> str:
return client.get("/static/dash.css").text
def _css_section(css: str, name: str) -> str:
"""One `/* ---- name … */` section of the stylesheet, up to the **next** section header.
Slicing to a named neighbour worked while the sheet had five sections. Every page has since
added a component section, and a slice that runs across someone else's rules accuses the
fleet block of colouring numbers it never coloured — so the boundary is structural: the end
of *this* section, whoever comes next.
"""
start = css.index(f"/* ---- {name}")
return css[start:css.index("/* ----", start + 10)]
def _hero_css(css: str) -> str:
"""Just the `hero numbers` section of the stylesheet."""
return css.split("/* ---- hero numbers", 1)[1].split("/* ---- bars", 1)[0]
def test_css_ships_the_hero_number_style(client) -> None:
css = _css(client)
hero_block = _hero_css(css)
for rule in (".hero {", ".hero .label", ".hero .value", ".hero .unit", ".hero .sub",
".hero.ok .value", ".hero.warn .value", ".hero.fail .value",
".hero-bar", ".hero-bar.ok .bar"):
assert rule in hero_block, f"dash.css lost the {rule} rule"
# a display size with fixed-width digits, so the number does not jitter between polls
assert "2.4rem" in hero_block
assert "tabular-nums" in hero_block
def test_hero_colour_comes_from_the_shared_verdict_tokens(client) -> None:
"""`.hero` reuses `--ok/--warn/--fail`; it does not invent a palette of its own."""
hero_block = _hero_css(_css(client))
for token in ("var(--ok)", "var(--warn)", "var(--fail)", "var(--text)",
"var(--text-muted)", "var(--text-faint)"):
assert token in hero_block, f"the hero style no longer uses {token}"
def test_no_stylesheet_rule_uses_an_undeclared_token(client) -> None:
"""A typo'd `var(--x)` renders unstyled and reads as a broken panel, not an error."""
css = _css(client)
root = re.search(r":root\s*\{(.*?)\n\}", css, re.S)
assert root is not None
declared = set(re.findall(r"(--[a-z0-9-]+)\s*:", root.group(1)))
used = set(re.findall(r"var\(\s*(--[a-z0-9-]+)", css))
assert not used - declared
def _code(body: str) -> str:
"""An asset with its comments stripped — prose may *name* a class the code must not define."""
body = re.sub(r"/\*.*?\*/", " ", body, flags=re.S)
body = re.sub(r"<!--.*?-->", " ", body, flags=re.S)
return re.sub(r"^\s*//.*$", " ", body, flags=re.M)
# The `fleet` object paths `/llm`'s aggregate block reads. `engines_total` drives the
# "hidden below two engines" rule and the tag; `engines_live` is what makes a partial sum
# visible; the rest are the row itself.
FLEET_BLOCK_PATHS = frozenset({
"engines_total", "engines_live", "requests_active", "requests_waiting",
"prompt_tokens_per_second", "decode_tokens_per_second", "tokens_total",
})
# The summed half of that list — the ones that must be null, not 0, with nothing answering.
FLEET_SUM_PATHS = ("requests_active", "requests_waiting", "prompt_tokens_per_second",
"decode_tokens_per_second", "tokens_total")
def _fleet_tag(html: str) -> str:
"""The aggregate block's opening tag — the one place its `hidden` default lives."""
start = html.index('<section class="card flat fleet"')
return html[start:html.index(">", start)]
def _fleet_render_path(js: str) -> str:
"""Just the body of `renderFleet()` — the aggregate block's whole code path, comments out.
Everything the block writes goes through here, so it is the slice that can be asserted
verdict-free without tripping over the engine panels' rated KV sparkline next door.
"""
js = _code(js)
return js[js.index("function renderFleet("):js.index("async function refresh(")]
def _fleet_field_metrics(js: str) -> list[str]:
"""The `fleet` keys the block's rows are bound to, in the order the block prints them."""
js = _code(js)
start = js.index("const FLEET_FIELDS = [")
table = js[start:js.index("];", start) + 2]
return re.findall(r"metric: '([a-z_]+)'", table)
def test_no_parallel_severity_vocabulary_exists_anywhere(client) -> None:
"""One word per colour: `ok|warn|fail`, as in severity.py and the existing components."""
for asset in (*ASSETS, *MARKUP_PAGES):
body = _code(client.get(asset).text)
assert ".sev-" not in body, f"{asset} invented a .sev-* class"
assert not re.search(r"classList\.(?:add|toggle)\(\s*[\"'](?:danger|good|warning)[\"']",
body), f"{asset} invented a synonym for fail"
def test_app_js_exposes_the_hero_helpers(client) -> None:
js = client.get("/static/app.js").text
assert "function hero(" in js
assert "function severityFrom(" in js
exports = js.rsplit("return {", 1)[1].split("}", 1)[0]
for name in ("hero", "severityFrom", "severityFor"):
assert re.search(rf"\b{name}\b", exports), f"Dash.{name} is not exported"
def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
"""`hero()` may only colour what the API rated — no thresholds re-derived in JS."""
js = client.get("/static/app.js").text
start = js.index("function hero(")
body = js[start:js.index("\n function setDot", start)]
assert "severityFrom(" in body
assert "severityFor(" not in body
# an unknown value gets n/a and no colour class at all
assert "naText" in body
assert re.search(r"raw != null", body)
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
"""Every page call site is gone now that phase 05 converted `/ups`.
`/` was converted in phase 02, `/system` in phase 03, `/llm` in phase 04 and `/ups` in
phase 05, so no page colours anything from a threshold it wrote itself. The function
itself stays: it is the documented fallback `setBar()` uses when a caller names no
verdict, and a page that ever needs a verdict the API does not send should add it to
`severity.py` rather than here — which is what the docstring on it says.
"""
js = client.get("/static/app.js").text
assert "FALLBACK ONLY" in js
assert "severityFor" in js, "the documented fallback itself was deleted"
for page in PAGE_SCRIPTS.values():
assert "severityFor" not in client.get(f"/static/{page}").text, \
f"{page} colours something from a threshold of its own"
def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> None:
"""`page-index.js` reads verdicts; it does not compare a number to a bound."""
js = client.get("/static/page-index.js").text
assert "severityFrom" in js
assert "severityFor" not in js
def test_system_page_colours_from_the_payload_not_from_js_thresholds(client) -> None:
"""`page-system.js` hero cards paint payload verdicts, not thresholds of their own.
Phase 03 task 02's promise: every colour on a host card comes from a `severity` dict
`severity.py` produced (`cpu.severity.usage_pct`, `memory.severity.used_pct`,
`disks.severity.worst_used_pct`, `cpu.severity.temp_c`).
A threshold written here could disagree with the overview's alerts; a hero with no
verdict prints `n/a` uncoloured, which is `Dash.hero()`'s contract.
The GPU slot names no verdict on purpose (phase 06): `severity.THRESHOLDS` has no
`gpu.*` row, the payload carries no `gpus[].severity`, and the hero is built without
`severity` or `bar` — a bar under a verdictless number would be a verdict by shape.
"""
js = client.get("/static/page-system.js").text
assert "severityFor(" not in js, "the system page re-invented a threshold"
assert "Dash.hero(" in js
for hook in ("cpu.severity.usage_pct", "memory.severity.used_pct",
"disks.severity.worst_used_pct", "cpu.severity.temp_c"):
assert hook in js, f"the card no longer names {hook} as a colour source"
# and the GPU hero grades nothing: no verdict hook, no utilisation bar
assert "gpus[].severity" not in js, "the GPU slot invented a colour source again"
gpu_slot = js[js.index("function renderGpuSlot("):js.index("function renderGpuExtra(")]
assert "severity:" not in gpu_slot, "the GPU hero passes a severity option again"
assert "bar:" not in gpu_slot, "the GPU hero grew a bar back (a verdict by shape)"
assert "hero-bar" not in gpu_slot, "the GPU hero rebuilt the bar variant of .hero"
def test_gpu_hero_footnote_prints_the_temperature_the_agent_read(client) -> None:
"""TODO.md L9: the GPU temperature is glanceable on the hero row, not only in the table.
`gpus[].temp_c` is the agent's max over that card's hwmon labels (`sources/host.py`,
pinned by `tests/test_system_contract.py`), so the hero needed no new field and the
`gpu.temp_c@N` history series kept its name. The null guard *is* the behaviour, so that
is what the assertion pins (there is no JS test runner — a locked anchor): a card whose
driver exposes no temperature prints none, because `n/a` is for numbers the card is
supposed to have and a `0 °C` would be a cool GPU that does not exist.
"""
js = client.get("/static/page-system.js").text
sub = js[js.index("function gpuSub("):js.index("function renderGpuSlot(")]
assert "gpu.temp_c" in sub, "the GPU hero stopped printing the temperature"
assert "if (gpu.temp_c != null)" in sub, "the temperature lost its null guard"
assert sub.count("°C") == 1, (
"a second °C in gpuSub is a branch printing a temperature nobody measured"
)
# "More detail" keeps the per-label breakdown: a surface change, not a de-duplication
extra = js[js.index("function renderGpuExtra("):js.index("function renderHostCard(")]
assert "Temp" in extra and "gpu.temp_c" in extra, "GPU detail lost its temperature column"
def test_system_host_card_is_six_heroes_without_per_core(client) -> None:
"""Each host card leads with exactly six heroes: CPU, MEM, TEMP, DISK, GPU, NET.
Asserted on the served script because there is no JS test runner (a locked anchor).
The two dense `.metrics` rows and the per-core heat bars are exactly what made the
card unreadable, and "no per-core anywhere on /system" is a promise a stray
`renderCores` call would silently break — the series stays stored, the page just
stopped drawing it (PLAN.md *Dropped visuals*).
"""
js = client.get("/static/page-system.js").text
card = js[js.index("function hostCard("):js.index("function renderHostCard(")]
slots = re.findall(r'<div class="(?:hero|hero-slot)[^"]*" data-k="([^"]+)"', card)
assert slots == ["cpu", "mem", "temp", "disk", "gpus", "net"], \
"the hero row is no longer the six numbers in the agreed order"
# per-core visualisation is gone from the card code entirely
assert "cores" not in card, "a per-core visual came back to the card"
assert "renderCores" not in js and "core_usage" not in js
# the NET slot is a slot with two lines and no verdict: a byte rate has no good/bad
assert 'data-k="net"' in card
assert not re.search(r'data-k="net"[^>]*data-sev', card), "the NET slot invented a colour"
# what the glance leaves out stays reachable, collapsed by default
assert "More detail" in card
for key in ("load", "freq", "dr", "dw", "dutil", "uptime", "details"):
assert f'data-k="{key}"' in card, f"{key} left the page instead of the glance"
# the panel owns its charts: the slots live inside the card, not on the page
for chart in ("util", "net"):
assert f'data-chart="{chart}"' in card, f"the panel lost its {chart} chart slot"
def test_system_page_keeps_no_fleet_wide_chart_or_metric_picker(client) -> None:
"""Nothing on `/system` plots two hosts, and nothing needs a selection to make sense.
TODO.md L4 is the instruction ("The system page doesn't need shared charts for
anything") and the overview's alerts + fleet table are what made removing them safe
(phase 02). Asserted on the served bytes because the only thing that would otherwise
notice a resurrected `<select>` is a browser: there is no JS test runner (a locked
anchor), and `test_no_page_carries_an_id_nothing_reads` cannot see ids that were deleted
correctly from the page but are still queried by the script.
"""
html = _code(client.get("/system").text)
for gone in ('id="chart-cpu"', 'id="chart-mem"', 'id="chart-disk"', 'id="chart-net"',
'id="chart-gpu"', 'id="chart-temp"', 'id="chart-detail"', 'id="detail-host"',
'id="detail-metric"', 'id="f-cpu"', 'id="f-load"', 'id="f-gpu"',
"Fleet totals", "Fleet charts", "Series detail", "<select"):
assert gone not in html, f"/system still carries {gone}"
# the two panels that are left are the per-host grid and the event feed
assert html.count("<section") == 2, "/system grew a panel back on top of the host grid"
assert 'id="hosts"' in html and 'id="events"' in html
# and the script must not reach for the deleted endpoints/ids either — a `#detail-host`
# query against a removed node is a silent null, so the picker would just never appear
js = client.get("/static/page-system.js").text
for gone in ("#chart-", "detail-host", "detail-metric", "detail-note",
"renderFleet", "makeCharts", "refreshGpuChart", "populateDetailMetrics"):
assert gone not in js, f"page-system.js still references {gone}"
def test_every_system_chart_is_drawn_inside_its_own_host_panel(client) -> None:
"""One host per chart: the legends of `/system` can never list two machine names.
The construction is a factory that takes the card, so a chart cannot be built against a
page-level id; the metric list is per host, so the GPU streams cannot leak from one
panel into another; and a panel is torn down (with its charts) when its host leaves the
payload, which is the bug class a keyed-by-host chart map exists to avoid.
"""
js = client.get("/static/page-system.js").text
builds = re.findall(r"new DashCharts\.LineChart\(([^,]+),", js)
assert builds, "the system page builds no charts at all"
# every chart is built from a `[data-chart]` slot inside the host card, never from a
# page-level id — which is what makes "one host per chart" structural, not a convention
assert all(re.search(r"(?:card|slot)\.querySelector\(.\[data-chart=", b) for b in builds), \
f"a chart is built outside a host panel: {builds}"
# three construction sites, one per slot (util, net, gpu) — never one per metric
assert len(builds) == 3, f"charts are not built one per panel slot: {builds}"
assert sorted(re.findall(r'data-chart=\\?"(\w+)"', " ".join(builds))) == ["gpu", "net", "util"]
factory = js[js.index("async function renderHostCharts("):js.index("function pruneHosts(")]
assert "hosts=${encodeURIComponent(host.name)}" in factory, \
"the per-host chart query is not scoped to one host"
assert "&metrics=" in factory and "state.range" in factory, \
"a host's charts need one metric list and the selected window"
# GPU series are named from the payload's own `gpus[].index`, never discovered/guessed
assert "gpu.busy_pct@" in js and "gpu.index" in js
# the GPU chart exists only while the host reports cards
gpu_slot = js[js.index("function hostChartSet("):js.index("function seriesFor(")]
assert re.search(r"if \(!wantsGpu && set\.gpu\)", gpu_slot), \
"a host that stops reporting GPUs keeps a GPU chart it has no data for"
pruner = js[js.index("function pruneHosts("):js.index("// ---- snapshot")]
assert "state.hosts.delete(name)" in pruner and "state.charts.delete(name)" in pruner, \
"a host that left the payload keeps a live panel and drawing charts"
def test_system_history_charts_are_collapsed_behind_a_toggle(client) -> None:
"""TODO.md L4: "The historical charts on the system resources page should only be visible
if expanded. I only want to see current stats to keep things clean."
The default `/system` view is the hero row. Pinned on the served script because the card
markup is generated in JS and there is no JS test runner (a locked anchor), and the two
ways this regresses are invisible otherwise: a `<details open>` (charts visible by
default, i.e. the task un-done but the page still pretty) and a paint pass that runs
while the stack is shut, which builds three `LineChart`s per host inside a
`display:none` container — measured 0 px wide, never drawn, and rebuilt every poll.
"""
js = client.get("/static/page-system.js").text
card = js[js.index("function hostCard("):js.index("function memorySub(")]
# the stack is *wrapped*, not merely preceded by a heading: it lives inside the <details>
box = card.index('<details class="charts-box" data-k="charts">')
stack = card.index('<div class="host-charts">')
closed = card.index("</details>")
assert box < stack < closed, "the chart stack is not inside the toggle"
assert closed < card.index("More detail"), "the charts toggle swallowed the detail row too"
assert not re.search(r"<details[^>]*\bopen\b", card), \
"a panel ships its charts open — the default view is not current stats"
assert re.search(r"<summary[^>]*>\s*Charts", card), "the toggle no longer reads Charts"
# it also says which window the expanded stack draws (the chips stay in the page head)
assert 'data-k="charts-window"' in card, "the toggle does not carry the window"
# …and the toggle is what turns the paint pass on
assert "addEventListener('toggle'" in card and "chartsToggled(" in card
setter = js[js.index("function setChartsWindow("):js.index("function memorySub(")]
assert "state.rangeKey" in setter, \
"the toggle re-derives its window label instead of using the chips' own"
handler = js[js.index("Dash.rangeControl("):js.index("state.poller = poll(")]
assert "state.rangeKey = key" in handler and "setChartsWindow(" in handler, \
"changing the window leaves every toggle labelling the old one"
# nothing is built or drawn while the stack is shut, in either path that could paint
paint = js[js.index("function drawHostCharts("):js.index("async function renderHostCharts(")]
assert "if (!chartsOpen(card)) return" in paint, "a closed panel gets painted anyway"
assert "hostChartSet(card, host)" in paint, \
"the LineChart instances are no longer built by the open pass"
toggle = js[js.index("function chartsToggled("):js.index("function drawHostCharts(")]
assert "if (!chartsOpen(card)) return" in toggle, "closing the stack still paints"
assert "renderHostCharts(card, host)" in toggle and "state.range" in toggle, \
"a cached response from another window is drawn under this window's label"
# the request discipline is unchanged: one per host per refresh, every series named
fetch = js[js.index("async function renderHostCharts("):js.index("function windowLabel(")]
assert "hostChartSet(" not in fetch, "the poll path still builds charts for closed panels"
assert "state.history.set(" in fetch, \
"the response is not kept, so the first open is an empty box until the next poll"
assert "if (range === state.range) drawHostCharts(" in fetch, \
"a response for a window the operator just left is painted under the new label"
assert "drawHostCharts(" in fetch, "an open panel is no longer repainted on refresh"
def test_a_pruned_host_takes_its_charts_toggle_with_it(client) -> None:
"""A host that leaves the payload must leave no orphan control in the grid.
The toggle is a child of the panel, which is why `card.remove()` is enough — but that is
only true while the markup is built inside `hostCard()`, so the placement is pinned here
along with the three per-host Maps the panel owns.
"""
js = client.get("/static/page-system.js").text
card = js[js.index("function hostCard("):js.index("function memorySub(")]
assert 'data-k="charts"' in card, "the toggle is no longer part of the panel"
pruner = js[js.index("function pruneHosts("):js.index("// ---- snapshot")]
assert "state.hosts.get(name).remove()" in pruner
for owned in ("state.charts", "state.payloads", "state.history"):
assert f"{owned}.delete(name)" in pruner, f"{owned} outlives the panel it belongs to"
def test_the_charts_toggle_is_a_quiet_disclosure_not_a_new_component(client) -> None:
"""The collapse adds no palette and no second vocabulary (UI conventions).
Open/closed is carried by a caret *shape* as well as a colour step, the summary is muted
rather than accented, and it introduces no `.sev-*` family — the same rule the hero
numbers live by, checked here because a `<summary>` is the first control on the page a
designer might be tempted to grade.
"""
css = client.get("/static/dash.css").text
block = css[css.index("details.charts-box { "):css.index(".host-charts {")]
assert "details.charts-box > summary::before" in block, "the toggle has no caret shape"
assert 'details.charts-box[open] > summary::before' in block, \
"open and closed differ only in colour"
assert "var(--text-faint)" in block, "the closed toggle is not muted"
assert ".sev-" not in block and "var(--accent)" not in block, "the toggle invented a palette"
assert 'class="charts-box"' in client.get("/static/page-system.js").text, \
"the stylesheet styles a class the page no longer renders"
def test_an_expanded_chart_stack_does_not_grow_its_collapsed_siblings(client) -> None:
"""One host open must not stretch the hosts beside it into blank cards.
`/system` is the same shape `/llm` had to fix: an expanded panel beside a collapsed one
grows the collapsed card to match, and a blank panel reads as "no data" (the misreading
the unconfigured rules exist to prevent). `.grid.items-start` already exists for exactly
this, so the page uses it rather than growing a rule of its own.
"""
html = _code(client.get("/system").text)
hosts_class = re.findall(r'<div class="([^"]*)" id="hosts">', html)
assert hosts_class == ["grid auto items-start"], \
f"the host grid stretches its cards again: {hosts_class}"
assert ".grid.items-start" in client.get("/static/dash.css").text, \
"the rule the host grid relies on is gone"
def test_every_system_chart_names_itself_for_assistive_tech(client) -> None:
"""A 120px SVG is silence to a screen reader unless the page names it.
The phase-03 accessibility pass (task 04) found the per-host charts drawing anonymous
plots: the legend is a sibling element and labels nothing, so the `<svg>` itself must
carry `role="img"` + `aria-label` that says which host and which metrics. Pinned on the
served bytes because there is no JS test runner (a locked anchor).
"""
charts_js = client.get("/static/charts.js").text
# the shared LineChart applies the name to the svg, and only when the caller opted in
assert "if (this.opts.ariaLabel)" in charts_js, "LineChart stopped honouring ariaLabel"
assert "setAttribute('role', 'img')" in charts_js
assert "setAttribute('aria-label', String(this.opts.ariaLabel))" in charts_js
js = client.get("/static/page-system.js").text
# all three slots name themselves, and each name starts with the host the panel belongs
# to — an aria-label that cannot say *which machine* is not an accessible name here
assert js.count("ariaLabel: `") == 3, "a per-host chart slot lost its accessible name"
for slot in ("utilisation", "network traffic", "GPU busy"):
assert f"${{host.name}} \u2014 {slot}" in js or f"${{host.name}} — {slot}" in js, \
f"the {slot} chart has no host-scoped accessible name"
# ---- event feeds: a bounded, paginated window ----------------------------- #
#
# Every `ul.events` feed is one component (`Dash.eventsPager`) because a pager that
# works on only one page is the bug this shape prevents: the `event` table grows for as long
# as the fleet runs, and an unbounded feed turns `/system` into a scroll to nowhere
# (`TODO.md` L6 — "All the events tables need to be scrollable and paginated"). There is no
# JS test runner (a locked anchor), so what is pinned here is the shape every feed must keep:
# one shared renderer, a fetch bigger than one page, a footer that counts and labels itself,
# and a container that scrolls instead of growing the page.
# page -> (its script, the `source=` the poller stores that page's events under)
EVENT_FEEDS = {
"/system": ("page-system.js", "system"),
"/disks": ("page-disks.js", "disks"),
"/llm": ("page-llm.js", "llm"),
"/ups": ("page-ups.js", "ups"),
}
def _event_feed_path(js: str) -> str:
"""Just the `refreshEvents()` half of a page script — the feed's whole code path, comments out."""
js = _code(js)
return js[js.index("async function refreshEvents("):]
def _feed_fetch_url(js: str) -> str:
"""The `/api/events` URL a feed asks for, quoted or templated."""
found = re.search(r"fetchJSON\([`'](/api/events[^`']*)[`']\)", _event_feed_path(js))
assert found, "this page no longer fetches the event feed"
return found.group(1)
def test_the_event_feeds_are_one_shared_pager_not_one_per_page(client) -> None:
"""`Dash.eventsPager()` renders every feed, and no page grows its own pager.
One copy per page is one chance per page to keep the old behaviour — and `/system` is the
one whose feed was already too tall when this was written down. `/disks` joined as a
fourth call site of the same helper, not as a fifth implementation of it.
"""
app_js = _code(client.get("/static/app.js").text)
assert app_js.count("function eventsPager(") == 1
exports = client.get("/static/app.js").text.rsplit("return {", 1)[1].split("}", 1)[0]
assert "eventsPager" in exports, "Dash.eventsPager is not exported"
for path, (script, _) in EVENT_FEEDS.items():
js = _code(client.get(f"/static/{script}").text)
assert "Dash.eventsPager($('#events')" in js, f"{path} does not render its feed through the pager"
assert "function eventsPager(" not in js, f"{path} invented its own pager"
assert "events.slice(" not in js, f"{path} windows its own events instead of asking the pager"
assert "row:" in _event_feed_path(js), f"{path} passes no row renderer"
def test_the_pager_pages_50_events_and_fetches_more_than_one_page(client) -> None:
"""The two numbers that make a client-side pager work, and the relation between them.
`EVENT_PAGE_SIZE` is the 50 the operator asked for; `EVENT_FEED_LIMIT` is what the feed
asks `/api/events` for. A limit at or below the page size is a pager that silently never
shows — there is never a second page to reach — and the feeds used to fetch 20/25/40.
"""
app_js = client.get("/static/app.js").text
page_size = re.search(r"const EVENT_PAGE_SIZE = (\d+);", app_js)
feed_limit = re.search(r"const EVENT_FEED_LIMIT = (\d+);", app_js)
assert page_size and int(page_size.group(1)) == 50, "a feed no longer pages 50 events"
assert feed_limit, "the feeds' event limit is no longer a named constant"
assert int(feed_limit.group(1)) > int(page_size.group(1)), \
"a fetch at or below one page means page 2 can never exist"
for path, (script, source) in EVENT_FEEDS.items():
url = _feed_fetch_url(client.get(f"/static/{script}").text)
assert "EVENT_FEED_LIMIT" in url, f"{path} hardcodes its own event limit"
assert f"source={source}" in url, f"{path} asks for the wrong event source"
def test_the_pager_footer_counts_and_labels_its_controls(client) -> None:
"""The footer counts (`1\u201350 of 137`), and prev/next are labelled and disabled at the ends.
The count is what tells the operator the window is a window; a control that stays live at
the last page claims there is more history than there is.
"""
app_js = _code(client.get("/static/app.js").text)
assert "${first}\u2013${last} of ${total}" in app_js, "the footer no longer counts"
assert 'data-pager="prev"' in app_js and 'data-pager="next"' in app_js
assert "prev.setAttribute('aria-label'" in app_js
assert "next.setAttribute('aria-label'" in app_js
assert "prev.disabled = pager.page === 0" in app_js
assert "next.disabled = pager.page >= pager.pages - 1" in app_js
# pagination only applies when there is more than one page of data
assert "foot.hidden = !paginating" in app_js
css = client.get("/static/dash.css").text
assert ".events-pager {" in css, "the pager footer has no style"
assert "button.ghost:disabled" in css, "a disabled pager control looks live"
def test_the_event_feeds_scroll_instead_of_growing_the_page(client) -> None:
"""The `<ul>` is the scrollable element, bounded in `dash.css`; the page stays short.
Pinned on the served bytes because there is no JS test runner (a locked anchor): a
stylesheet edit that drops `max-height` is invisible until the feed is long, and by then
the page is massive again.
"""
css = client.get("/static/dash.css").text
block = css[css.index("ul.events {"):css.index("}", css.index("ul.events {"))]
assert "max-height" in block and "overflow-y: auto" in block, \
"the event feed is unbounded again — the page grows with history"
for path in EVENT_FEEDS:
html = client.get(path).text
assert '<ul class="events" id="events"' in html, f"{path} lost its feed's id"
def test_the_pager_returns_to_page_1_when_the_events_change(client) -> None:
"""New events reset the window; the same events do not.
Resetting on every poll would snap `next` back within one interval, so page 2 could never
be read; never resetting lands an operator on page 3 of a list that changed underneath
them. The page key is the set of events, not the clock.
"""
app_js = _code(client.get("/static/app.js").text)
body = app_js[app_js.index("function eventsPager("):]
assert "signature !== pager.signature" in body, "the pager no longer notices new events"
reset = body[body.index("signature !== pager.signature"):]
assert reset.index("pager.page = 0") < reset.index("paintEventFeed"), \
"new events do not reset the page before the feed is drawn"
assert "if (pager.markup !== markup)" in app_js, \
"every poll rewrites 50 rows and throws away the reader's scroll position"
def test_system_event_feed_queries_the_source_transitions_are_stored_under(client) -> None:
"""Host transitions are stored under the *dashboard* name, not the sample source.
`poller.py` logs `Event(ts, self.dashboard, …)` — host jobs write `system`, the way
the UPS job writes `ups` — while *samples* use `host`. The `/system` feed queried
`source=host` since the first commit and therefore always rendered "no transitions
yet", even with a real outage (a gateway disconnect) sitting in the store. Pinned on
the served bytes because there is no JS test runner (a locked anchor).
"""
js = client.get("/static/page-system.js").text
feed = _event_feed_path(js)
url = re.search(r"fetchJSON\([`'](/api/events[^`']*)[`']\)", feed)
assert url, "the /system page no longer fetches the event feed"
assert "source=system" in url.group(1), \
"the /system event feed must ask for the source the poller stores events under"
assert "source=host" not in url.group(1), \
"source=host matches no stored event — the feed is empty forever"
def test_system_footnote_prints_the_window_it_actually_measured(client) -> None:
"""`Δ<window>` is the page's promise that a rate is a measurement (invariant 2), so the
number behind it has to come from the measurement.
The `window_seconds` beside the fleet is `HOST_INTERVAL_SECONDS`, while the window behind a
rate is the estimator's `RATE_WINDOW_SECONDS` — equal at the defaults, which is why the
footnote could quote the poll interval and look right. Set `RATE_WINDOW_SECONDS=30` and it
would have labelled every rate on the page with a window it never used. The footnote now
reads each host's own measured `window_seconds`, and prints a range when they disagree.
"""
js = client.get("/static/page-system.js").text
assert "function windowLabel(" in js, "the footnote lost its measured-window helper"
footnote = js[js.index("$('#foot-note').textContent"):js.index("await Promise.all(chartJobs)")]
assert "windowLabel(" in footnote
assert "host.window_seconds" in footnote, \
"the footnote is quoting the poll interval as a measurement again"
# a footnote that says "window" and a page that says "poll" must stay two separate claims
assert "poll ${Math.round(state.pollMs / 1000)}s" in footnote
def test_no_per_core_visual_library_survives_the_page(client) -> None:
"""The per-core heat bars left `/system` and `charts.js` with them (PLAN.md *Dropped
visuals* is about what the pages draw, not what the store keeps).
`renderCores` had one caller — the hero-card per-core block deleted in task 02 — so
keeping it would be dead markup generating unstyled `.core` spans; `.cores` was the CSS
that block needed. Both are checked absent here so neither can come back as a
half-change, while `sparkline`/`LineChart` stay for phase 04 and the UPS page.
"""
assert "renderCores" not in client.get("/static/charts.js").text
css = client.get("/static/dash.css").text
assert ".cores" not in css
for asset in ASSETS:
assert "core_usage" not in _code(client.get(asset).text), f"{asset} plots per-core"
# what phase 04 and the other pages still chart from
for name in ("LineChart", "sparkline", "renderLegend"):
assert name in client.get("/static/charts.js").text
def test_sparkline_ships_the_fixed_scale_and_verdict_options(client) -> None:
"""Phase 04 turns `/llm` cells into sparkline rows, so the shared `sparkline()` must
behave like `llm_monitor`'s `drawSpark` (the task-02 contract, NOTES-chart-set).
Pinned on the served bytes because there is no JS test runner (a locked anchor). What
matters is not the option names but the semantics behind each one: a percent metric
that silently rescaled to its own data would make an 8 % KV cache and a 92 % spike the
same squiggle, and a stroke coloured from a page-invented threshold would break the
one-colour-language rule the same way.
"""
charts_js = client.get("/static/charts.js").text
body = charts_js[charts_js.index("function sparkline("):charts_js.index("window.DashCharts")]
# the phase grep criterion, as a test: the options are really in the shipped file
hits = [ln for ln in charts_js.splitlines()
if re.search(r"fixedMax|align|severity", ln)]
assert len(hits) >= 3, "the sparkline upgrade is not in the served charts.js"
for opt in ("fixedMax", "align", "labels", "severity", "minPoints", "span", "format",
"minSpan"):
assert opt in body, f"sparkline() lost its `{opt}` option"
# fixedMax: the data clamps to the axis, never the axis to the data
assert "Math.min(Math.max(v, lo), hi)" in body, \
"a fixedMax sparkline rescales to its own data again"
# align:'right' grows within a real time window (span seconds), not a sample count
assert "x1 - o.span" in body, "align:'right' forgot the window it grows inside"
# verdict colour comes from the theme tokens only; unrated is faint grey, never green
for token in ("cssVar('--ok')", "cssVar('--warn')", "cssVar('--fail')",
"cssVar('--text-faint')"):
assert token in body, f"sparkline() stopped taking its colour from {token}"
assert ".sev-" not in body, "sparkline invented a parallel severity vocabulary"
# the `now / max` foot llm_monitor prints, via the caller's format callback
assert "now ${fmt(last.value)}" in body and "max ${fmt(dataMax)}" in body
# never a confident line through one point, and the foot quotes data, not the padded axis
assert "minPoints: 2" in body and "no history" in body
# ASSUMPTION (task 02): no animation at all, so prefers-reduced-motion holds by design
assert "transition:" not in body and "animation:" not in body
# ---- the battery glyph (phase 05 task 02) ---------------------------------
#
# `DashCharts.battery()` is the one visual in this app whose *shape* is the data: the fill
# is the charge. That makes the honesty rules heavier here than anywhere else — a fill
# rounded up to look reassuring, a colour invented for an unrated cell, or a state only
# readable as green-vs-red would each turn a glance at the battery into a wrong claim.
# There is no JS test runner (a locked anchor), so the construction is pinned on the served
# bytes and shown in a browser (`data/logs/ups-glyph-*.png`).
def _battery_js(client: TestClient) -> str:
"""The glyph's code as served (state table + body), comments stripped.
Prose cannot satisfy these assertions, and the phrase table the accessible name is
built from is part of the component, so the slice starts above `function battery`.
"""
charts = client.get("/static/charts.js").text
body = charts[charts.index("const BATTERY_STATES"):charts.index("window.DashCharts")]
assert "function battery(container, opts)" in body
return _code(body)
def test_battery_glyph_is_exported_from_the_shared_chart_library(client) -> None:
"""The phase's own grep criterion, as a test: exactly one glyph, and it is public."""
charts = client.get("/static/charts.js").text
assert charts.count("function battery") == 1
exports = charts[charts.index("window.DashCharts"):]
assert re.search(r"\bbattery\b", exports), "DashCharts.battery is not exported"
# the wrapper the stylesheet sizes, added by the component rather than each caller
assert ".battery {" in _css(client) and ".battery svg" in _css(client)
def test_battery_fill_is_the_charge_and_is_never_padded(client) -> None:
"""2 % must look nearly empty, and 0 % must leave an empty shell, not a stray pixel."""
body = _battery_js(client)
# clamped to the scale, and the clamp is the only thing that touches the number: a
# minimum visible width would be a reassuring battery over a flat one
assert "Math.max(0, Math.min(100," in body
assert "(pctValue / 100) * inner.w" in body, "the fill stopped being a fraction of charge"
assert not re.search(r"Math\.max\(\s*[0-9.]*[1-9]\s*,", body), \
"the charge fill got a minimum width — a small charge would look bigger than it is"
# nothing is drawn for a cell at zero, and the plate it fills is a clip, so a low fill
# cannot leak outside the rounded interior
assert "pctValue > 0" in body
assert "<clipPath" in body and 'clip-path="url(#' in body
# an empty cell is the sunken plate, never a transparent hole in the card
assert "cssVar('--bg-sunken')" in body
def test_battery_colours_come_from_the_verdict_tokens_only(client) -> None:
"""The verdict paints; an unrated cell draws a grey shape and no colour at all."""
body = _battery_js(client)
for token in ("cssVar('--ok')", "cssVar('--warn')", "cssVar('--fail')"):
assert token in body, f"the glyph stopped taking its colour from {token}"
# no verdict → muted outline, and a measured charge in faint grey at half strength: a
# shape, not a judgement (an unknown number must never look green)
assert "cssVar('--text-muted')" in body and "cssVar('--text-faint')" in body
assert ' opacity=".5"' in body, "an unrated charge is drawn with the same weight as a rated one"
assert ".sev-" not in body, "the glyph invented a parallel severity vocabulary"
assert "severityFor(" not in body, "the glyph compares the charge to a threshold itself"
def test_battery_states_are_readable_without_colour(client) -> None:
"""Every state has a *shape*, so the glyph survives monochrome and colour-blindness.
on battery = dashed outline, charging = bolt, unknown = a dash pattern through the
body, online = a plain solid shell. A word the page did not intend falls back to
`unknown` rather than to a confident "on line".
"""
body = _battery_js(client)
assert 'BATTERY_STATES.indexOf(o.state) >= 0' in body, \
"an unrecognised state would be drawn as some other state"
assert 'state === \'onbattery\'' in body and 'stroke-dasharray="7 5"' in body, \
"the on-battery outline is no longer dashed"
assert "state === 'charging'" in body and 'd="M13 2 L3 14' in body, "the charging bolt is gone"
assert "state === 'unknown'" in body and 'stroke-dasharray="5 6"' in body, \
"an unknown charge no longer draws a dash through the body"
# and the unknown case really is *no fill*: the shape has to say it on its own
assert "pctValue != null" in body
def test_battery_glyph_names_itself_and_can_say_it_once(client) -> None:
"""`role="img"` + `<title>` built from the inputs, with an explicit `aria-hidden` opt-out."""
body = _battery_js(client)
assert 'role="img"' in body and "<title>" in body and "aria-label=" in body
for phrase in ("Charge ", " percent", "Charge unknown", "on line", "on battery",
"charging", "state unknown"):
assert phrase in body, f"the accessible name lost {phrase!r}"
# the label is a UPS name from upsd, so it goes through the one escaper
assert body.count("escapeHtml(name)") == 2, "<title> or aria-label stopped escaping the label"
assert 'aria-hidden="true"' in body, "the decorative opt-out for a duplicate hero number is gone"
def test_battery_glyph_scales_with_its_container_and_never_animates(client) -> None:
"""One `viewBox` + `preserveAspectRatio` for a phone and a 27" monitor; no motion at all."""
body = _battery_js(client)
assert 'viewBox="0 0 ${W} ${H}"' in body
assert 'preserveAspectRatio="xMidYMid meet"' in body, "the glyph stopped scaling with its box"
assert "ResizeObserver" not in body, "the glyph redraws on resize instead of scaling"
assert "transition" not in body and "animation" not in body
css = _code(_css(client))
wrapper = re.search(r"\.battery \{([^}]*)\}", css)
inner = re.search(r"\.battery svg \{([^}]*)\}", css)
assert wrapper and inner, "dash.css no longer sizes the glyph box"
assert "width: 100%" in wrapper.group(1) and "max-width" in wrapper.group(1), \
"the glyph is not capped inside its container"
assert "width: 100%" in inner.group(1) and "height: auto" in inner.group(1), \
"the svg would stretch instead of keeping the viewBox ratio"
both = wrapper.group(1) + inner.group(1)
assert "transition" not in both and "animation" not in both
# ---- the `/ups` hero row (phase 05 task 03) ------------------------------
#
# The card used to be nine equal `.metric` cells in which charge, load and runtime were the
# same size as the battery amperage. These pin the shape that replaced it on the served
# bytes (a locked anchor: no JS test runner), because the whole point of the row is a size
# and colour hierarchy a later "tidy up" can quietly flatten back to nine equal cells.
def _ups_card_js(client: TestClient) -> str:
"""The card markup as served, from `upsCard()` to `renderCard()`."""
js = client.get("/static/page-ups.js").text
return js[js.index("function upsCard("):js.index("function renderCard(")]
def test_ups_card_opens_with_the_glyph_and_three_heroes(client) -> None:
"""Charge / Load / Runtime are the card's three largest elements, next to the glyph.
`data-sev` names the verdict path each slot colours itself from, the same convention the
`/system` host card uses — so "which number is coloured by what" is readable in the
markup, and a rename that disconnects the page from `/api/ups/snapshot` fails here
instead of shipping a card of `n/a`.
"""
card = _ups_card_js(client)
assert '<div class="ups-hero">' in card, "the hero row left the top of the card"
assert 'data-k="glyph"' in card, "the battery glyph is not part of the hero row"
slots = re.findall(r'<div class="hero[^"]*" data-k="([^"]+)"', card)
assert slots == ["charge", "load", "runtime"], \
"the hero row is no longer charge / load / runtime in the agreed order"
for hook in ("severity.charge_pct", "severity.load_pct", "severity.battery_runtime"):
assert hook in card, f"the card no longer names {hook} as a colour source"
# the hero row is the first thing in the card body, above the health checks and commands
assert card.index("ups-hero") < card.index('data-k="checks"')
assert card.index("ups-hero") < card.index('data-k="commands"')
def test_ups_heroes_take_their_colour_from_the_payload(client) -> None:
"""Three `Dash.hero()` calls, three payload verdicts, no JS threshold.
The charge hero carries **no** `.bar`: the glyph beside it is that same value drawn as a
shape, and one value drawn twice in one row is the pattern this app avoids (load, which
has no other shape, keeps its bar). Runtime prose comes from `runtimeText()` alone, and
a runtime nobody estimated is passed as `null` so `hero()` renders the faint, uncoloured
unknown rather than the string `n/a` styled like a measurement.
"""
js = client.get("/static/page-ups.js").text
body = js[js.index("function renderCard("):js.index("// ---- command dialog")]
assert body.count("Dash.hero(q(") == 3, "the hero row is not three hero numbers"
for verdict in ("verdicts.charge_pct", "verdicts.load_pct", "verdicts.battery_runtime"):
assert verdict in body, f"{verdict} never reaches a hero"
charge = body[body.index("Dash.hero(q('charge')"):body.index("Dash.hero(q('load')")]
assert "bar" not in charge, "charge grew a second shape when the glyph already is one"
load = body[body.index("Dash.hero(q('load')"):body.index("Dash.hero(q('runtime')")]
assert "bar: true" in load, "load lost its bar"
assert "runtimeText(n.battery_runtime)" in body, "runtime prose bypasses runtimeText()"
runtime = body[body.index("Dash.hero(q('runtime')"):]
assert "n.battery_runtime == null ? null" in runtime, \
"an unmeasured runtime is rendered as a measurement"
assert "severityFor(" not in body
def test_ups_secondary_electrical_row_is_demoted_not_deleted(client) -> None:
"""Line in, output, battery volts and amps, battery °C: still there, visibly second.
These are the numbers you reach for *while* the UPS is on battery, so they were demoted
rather than folded into the disclosure — and they stay one step down (smaller, muted) so
the three verdicts remain the largest elements, which is also what keeps the card from
growing so tall that two stop fitting side by side on a laptop.
"""
card = _ups_card_js(client)
assert 'class="metrics ups-electrical"' in card
for key in ("line", "out", "batt", "amps", "temp"):
assert f'data-k="{key}"' in card, f"the electrical row lost {key}"
assert card.index("ups-electrical") > card.index("ups-hero")
css = _code(_css(client))
rule = re.search(r"\.ups-electrical \.metric \.value\.sm \{([^}]*)\}", css)
assert rule, "dash.css no longer styles the secondary row"
assert "var(--text-muted)" in rule.group(1), "the secondary row is not muted"
sizes = re.findall(r"font-size: ([\d.]+)rem", rule.group(1))
assert sizes and float(sizes[0]) < 1.3, "the secondary row competes with the heroes"
# `n/a` is fainter than a muted reading — neither real driver reports a battery temperature
null_rule = re.search(r"\.ups-electrical \.metric \.value\.sm\.is-null \{([^}]*)\}", css)
assert null_rule and "var(--text-faint)" in null_rule.group(1), \
"an unmeasured secondary value is not faint, so it reads like a number"
def test_ups_glyph_state_comes_from_the_api_fields(client) -> None:
"""`glyphState()` maps `on_battery` + `charger_status` — and `full` is **not** charging.
The mapping is presentation, so it lives in the page (task 02's assumption), but it is
still a mapping the real drivers exercise: neither UPS here exports
`battery.charger.status`, so the fallback that reads a state the API rated `ok` is what
draws a solid shell on both of them. Nothing here re-reads `ups.status` —
`test_the_page_does_not_re_parse_nut_status_flags` holds that line.
"""
js = _code(client.get("/static/page-ups.js").text)
glyph = js[js.index("function glyphState("):js.index("function drawGlyph(")]
assert "item.on_battery" in glyph and "'onbattery'" in glyph
assert "charger === 'chg'" in glyph and "'charging'" in glyph
assert "charger === 'dchg'" in glyph, "a discharging charger token is on battery too"
assert "charger === 'full'" in glyph and "'online'" in glyph, \
"a full, online battery must not be drawn as charging"
assert "item.states" in glyph, "a driver with no rich status lost its state entirely"
assert "'unknown'" in glyph, "an unknown state now degrades to a confident 'on line'"
def test_ups_glyph_repaints_only_when_something_visible_changes(client) -> None:
"""Charge jitters by a tenth of a percent every poll; the `<svg>` does not rebuild on it.
Rebuilding would re-resolve the theme tokens and re-announce the accessible name every
`NUT_INTERVAL_SECONDS` for a difference no eye can see. `decorative: true` is the
component's documented opt-out — the charge hero and the state chips already say
"Charge 100 percent, on line", and a screen reader should not hear it twice.
"""
js = _code(client.get("/static/page-ups.js").text)
draw = js[js.index("function drawGlyph("):js.index("function upsCard(")]
assert "DashCharts.battery(" in draw
assert "cell.dataset.drawn === drawn" in draw, "the glyph rebuilds on every poll"
assert "decorative: true" in draw, "the glyph duplicates the hero number in a11y"
assert "glyphState(item)" in draw
def test_ups_hero_row_makes_room_for_the_glyph(client) -> None:
"""The glyph gets a real column, and the row stacks rather than clipping on a phone.
The track floor is the widest runtime prose the page prints (`9h 59m`) — a number that
cannot wrap needs the floor to be its own width, or the row grows a clipped `1h 28`.
"""
css = _code(_css(client))
row = re.search(r"\.ups-hero \{([^}]*)\}", css)
assert row, "dash.css lost the UPS hero row"
assert "minmax(112px, 156px) minmax(0, 1fr)" in row.group(1), "the glyph has no column"
assert "align-items: center" in row.group(1)
heroes = re.search(r"\.ups-hero \.heroes \{([^}]*)\}", css)
assert heroes and "minmax(min(140px, 100%), 1fr)" in heroes.group(1), \
"a hero track can now clip the widest runtime prose"
at = css.index("@media (max-width: 620px)")
block = css[at:at + 240]
assert ".ups-hero" in block and "minmax(0, 1fr)" in block, "the row never stacks"
def test_every_ups_chart_names_itself_for_assistive_tech(client) -> None:
"""The four trend plots on `/ups` carry an accessible name, like every other chart here.
`AGENTS.md` states the convention as *every* chart names itself: `LineChart` puts
`role="img"` + `aria-label` on the `<svg>` when the caller passes `ariaLabel`, because
the legend is a sibling that labels nothing and an unlabelled plot is silence. `/system`
(phase 03) and `/llm` (phase 04) were pinned; these four were the stragglers, so the
rule is now pinned on every page that draws one.
"""
js = client.get("/static/page-ups.js").text
charts = js[js.index("function makeCharts("):js.index("function points(")]
assert charts.count("ariaLabel:") == 4, "a `/ups` chart lost its accessible name"
for phrase in ("Battery charge", "Battery runtime", "Battery and line voltage", "UPS load"):
assert phrase in charts, f"the {phrase!r} chart names nothing readable"
# ---- the `/llm` layout (phase 04 task 03) ---------------------------------
#
# The drift this phase fixes was a 14-cell grid where everything was the same size and
# nothing showed recent shape. These tests pin the *shape* the page committed to on the
# served bytes (a locked anchor: no JS test runner), so a regression to "all cells equal"
# fails a test rather than a screenshot.
def test_llm_page_colours_from_the_payload_not_from_js_thresholds(client) -> None:
"""`page-llm.js` paints the KV verdict `severity.py` produced; it owns no threshold.
Both `severityFor(value, 75, 90)` call sites are gone (the task's explicit work item):
the KV row takes `severity` from the rated ratio cell, the fleet KV bar was deleted
with the summed card, and every unrated number — throughput above all, because 0 tok/s
on an idle server is healthy — draws no colour at all.
"""
js = client.get("/static/page-llm.js").text
assert "severityFor(" not in js, "the LLM page re-invented a threshold"
assert "severityFrom" in js
# the KV verdict comes from the payload cells `sources/llm.py` rates
kv = js[js.index("case 'cache':"):js.index("case 'ttft':")]
assert "kv_cache_usage_ratio" in kv and "gpu_cache_usage" in kv and "severity" in kv
def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
"""Decode and prompt are the largest elements per engine; the rest is visibly secondary.
The hero row is `Dash.hero()` markup (2.4rem via `.hero`), built from the `hero: true`
flags in `CELLS`, and throughput carries **no** severity — `severity.py` has no
throughput row and the page invents none (the neutral-ink decision in NOTES-chart-set,
mirroring the `/system` NET precedent). The provenance tag stays in the hero's `.sub`
with the exact `source` as the tooltip: removing it to look cleaner is a regression.
"""
js = client.get("/static/page-llm.js").text
panel = js[js.index("function enginePanel("):js.index("function renderPanel(")]
assert 'data-hero="${cell.key}"' in panel, "the hero row is not flag-driven markup"
assert '<div class="heroes">' in panel, "the hero row left the engine panel"
heroes = re.findall(r"{ key: '(\w+)'[^}]*?hero: true", js)
assert heroes == ["decode", "prompt"], "the hero row is no longer the two throughputs"
# driven by flags, not fourteen identical cells. Phase 07 task 03 lifted the summary rows
# out of the panel into the stats strip above it, so what is left here is the pair plus
# the latency rows the strip does not print — never a fourteen-cell grid again.
assert "SPARK_CELLS" in js and "ALARM_CELLS" in js
assert js.count("spark: '") >= 2, "the panel lost the latency sparkline rows it kept"
assert js.count("hero: true") == 2, "the hero row is no longer exactly two numbers"
# the hero call must not pass a severity — unrated by design
hero_call = js[js.index("const node = hero("):js.index("if (node) node.title")]
assert "severity" not in hero_call, "throughput invented a verdict"
# provenance intact: the tag is generated from the cell's `source`, never hardcoded
assert "provenance" in js and "Δ" in js
assert "node.title = value.source" in js
def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client) -> None:
"""Each panel row draws its own sparkline over the *selected* window and prints mean
**and** p95 without a hover.
The window is honest time (`align: 'right'` + `span`) and the p95 comes from the
snapshot's histogram cells — p95 needs no stored series. The `only` scopes are what keep
a gufo panel from showing a latency mystery and a vLLM strip card from showing a
spec-decode row it cannot fill.
Phase 07 task 03 moved the percent and count rows to the stats strip, so the plotted
percent series left the cell list and the axis guarantee moved with it: the KV trend
chart pins `yMax: 100` (an 8 % window that once held a 92 % spike must not read flat),
and the strip converts its 0-1 ratios to per cent in `cellValue()` — a bar cannot
autoscale, so it cannot lie that way either. The `fixedMax`/`minSpan` options stay
plumbed through `drawSparks()` and are pinned on the component itself by
`test_sparkline_ships_the_fixed_scale_and_verdict_options`: a count sparkline that comes
back must not get an axis finer than one request.
"""
js = client.get("/static/page-llm.js").text
sparks = js[js.index("function drawSparks("):js.index("function renderFleet(")]
assert "DashCharts.sparkline(" in sparks
for opt in ("align: 'right'", "span: state.range", "fixedMax", "labels: { max: true }"):
assert opt in sparks, f"cell sparklines lost `{opt}`"
# the one percent series still plotted owns a fixed 0-100 axis; seconds stay ms
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
assert "yMax: 100" in charts, "the KV trend chart no longer pins its 0-100 axis"
assert "scale: 1000" in js and "cell.scale || 1" in sparks
# every spark cell declares its history series in the markup it is built from
assert 'data-spark data-metric="${cell.spark}"' in js
# percentile foot lines: mean big in the row, p95 beneath, n/a when nothing measured
assert "function p95Value(" in js
for name in ("time_to_first_token_seconds__p95", "time_per_output_token_seconds__p95",
"request_queue_time_seconds__p95"):
assert name in js, f"{name} no longer feeds a foot line"
assert "p95 n/a" in js
assert 'data-foot' in js
# engine-scoped rows: gufo never grows a TTFT row, vLLM never a spec-accept row
assert "only: ['vllm']" in js and "only: ['gufo']" in js
scoped = js[js.index("function sparkCellsFor("):js.index("function enginePanel(")]
assert "cell.only" in scoped
assert "minSpan: cell.minSpan" in sparks, "cell minSpan never reaches the sparkline"
def test_llm_sparklines_and_charts_cost_one_batched_history_request(client) -> None:
"""One `/api/llm/history` call per refresh feeds charts *and* every sparkline row.
The endpoint answers a metric list for every engine, and unknown names come back as
empty series on purpose — so the page asks for the full cell set once instead of one
XHR per cell, and the two series phase 04 added (`tpot_seconds`, `spec_accept_rate`)
must be in that list or those rows are permanent "no history" slots.
"""
js = client.get("/static/page-llm.js").text
# comments aside, exactly one fetch hits the endpoint: one request, every metric
calls = re.findall(r"fetchJSON\(\s*`/api/llm/history", _code(js))
assert len(calls) == 1, "the page makes more than one history request again"
assert "HISTORY_SERIES.join" in js and "SPARK_CELLS.map((cell) => cell.spark)" in js
request = js[js.index("async function refreshHistory("):js.index("function drawSparks(")]
assert "metrics=${HISTORY_SERIES.join" in request
assert "range=${state.range}" in request
# and the endpoint really answers every name the page asks for
for name in ("tpot_seconds", "spec_accept_rate", "ttft_seconds", "queue_seconds",
"kv_cache_usage", "requests_running", "requests_waiting",
"prefix_cache_hit_ratio"):
assert name in HISTORY_METRICS, f"llm.{name} is not stored, so its spark is dead"
def test_llm_alarm_cells_exist_only_while_there_is_an_incident(client) -> None:
"""`preemptions` / `device_lost` are absent at zero and red chips above zero.
The rule is the *existence of an incident*, not a graded threshold — `sources/llm.py`
already treats any non-zero `gufo_device_lost_total` as one. The chip reuses the shared
`.chip.fail` vocabulary; nothing new is painted, and a container emptied at zero means
the row does not exist rather than showing a permanent ignored 0.
Phase 07 task 03 renders them on the engine's strip card (the summary you read first)
instead of inside the panel, so the two `alarm: true` rows live in `STRIP_ROWS` and
`ALARM_CELLS` filters them out of it — counted with the comments stripped, because the
docstring on that table explains the flag by name.
"""
js = client.get("/static/page-llm.js").text
alarms = js[js.index("function renderAlarms("):js.index("function makeCharts(")]
assert "class=\"chip fail\"" in alarms, "the alarm chip left the shared fail vocabulary"
assert "value.value > 0" in alarms
assert "host.hidden = chips.length === 0" in alarms
assert 'data-alarm="${cell.key}"' in alarms
assert _code(js).count("alarm: true") == 2
assert "ALARM_CELLS = STRIP_ROWS.filter" in js, "the alarm rows are not strip rows"
# the panel no longer carries a second container for the same incident
panel = js[js.index("function enginePanel("):js.index("function renderPanel(")]
assert 'data-k="alarms"' not in panel, "the alarm chips are drawn twice"
strip = js[js.index("function stripCard("):js.index("function renderStripCard(")]
assert 'data-k="alarms"' in strip, "the alarm chips left the strip card"
# ---- the `/llm` stats strip (phase 07 task 03) ---------------------------
#
# `llm_monitor`'s shared strip (`index.html` ~L743), ported: one card per configured engine
# between the fleet block and the panels. Two things make it a summary instead of a second
# set of panels, and both are pinned below: every row is a cell `/api/llm/snapshot` really
# publishes, and no row is printed in the strip *and* in the panel under it.
#: row key -> the cells `cellValue()` resolves it from (`sessions.*` is the gufo slot count,
#: which is not a metrics cell). The served `STRIP_ROWS` table must match these keys exactly.
STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
"prompt": ("prompt_tokens_per_second",),
"decode": ("decode_tokens_per_second",),
"tokens_total": ("tokens_total",),
"slots_busy": ("sessions.list", "sessions.active"),
"running": ("requests_processing", "num_requests_running"),
"waiting": ("requests_deferred", "num_requests_waiting"),
"cache_hit": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"),
"spec_accept": ("spec_accept_rate",),
"queue_mean": ("request_queue_time_seconds__mean",),
"preemptions": ("num_preemptions_total",),
"device_lost": ("device_lost_total",),
}
#: rows that are incidents, not stats: absent where the engine does not count them.
ALARM_ROW_KEYS = ("preemptions", "device_lost")
def _strip_rows(js: str) -> dict[str, str | None]:
"""The served `STRIP_ROWS` table: row key -> its `only` scope (`None` = every engine)."""
body = _code(js)
start = body.index("const STRIP_ROWS = [")
table = body[start:body.index("\n];", start)]
rows: dict[str, str | None] = {}
for chunk in table.split("{ key: '")[1:]:
scoped = re.search(r"only: \[([^\]]*)\]", chunk)
rows[chunk.split("'")[0]] = scoped.group(1) if scoped else None
return rows
def _strip_render_path(js: str) -> str:
"""Just `renderStripCard()` — the whole code path that paints one strip card."""
js = _code(js)
return js[js.index("function renderStripCard("):js.index("function pruneEngines(")]
def test_llm_stats_strip_sits_between_the_fleet_block_and_the_panels(client) -> None:
"""The strip is there, ships hidden, names itself, and appears only with engines.
An unconfigured engine gets no card and no strip (invariant 3) — the section is empty in
the markup and `page-llm.js` unhides it only when the payload has an engine to draw.
"""
html = _code(client.get("/llm").text)
assert 'id="strip"' in html, "/llm lost the per-engine stats strip"
assert html.index('id="fleet"') < html.index('id="strip"') < html.index('id="engines"'), \
"the strip is no longer between the fleet block and the engine panels"
start = html.index('<section class="strip"')
tag = html[start:html.index(">", start)]
assert "hidden" in tag, "the strip no longer ships hidden"
assert 'aria-label="Engine summary"' in tag, "the strip is a region that names nothing"
js = client.get("/static/page-llm.js").text
assert "$('#strip').hidden = payloads.length === 0" in js
def test_llm_strip_card_is_one_card_per_engine_of_shared_vocabulary(client) -> None:
"""`stripCard()` = `.card` + `.kv` + `.bar` + `.chip.fail`: a layout, not a new component.
Dot, name and age in the head (reachability, the same distinction the panels and the
fleet block draw), the KV bar, the rows keyed by payload field, the alarm container.
"""
js = client.get("/static/page-llm.js").text
card = js[js.index("function stripCard("):js.index("function renderStripCard(")]
assert "card.className = 'card stat'" in card
assert "card.dataset.engine = engine" in card, "the card stopped naming its engine"
assert '<span class="dot idle"></span>' in card
assert '<span class="faint mono updated"></span>' in card, "the card lost its timestamp"
assert '<span class="bar" data-kv-bar><i></i></span>' in card, "the KV bar left `.bar`"
assert '<dl class="kv">' in card, "the strip rows left the shared `.kv` vocabulary"
assert 'data-row="${row.key}"' in card
assert 'data-k="alarms"' in card
# the wrapper is why the shared `.kv` grid has to move *inside* it: applied to the
# wrappers, `max-content 1fr` lays two rows side by side and the card reads as a
# column of orphaned labels (found in the browser, not in a test failure)
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- engine stats strip"):css.index("/* ---- fleet totals")]
assert ".strip .kv { margin: 0; grid-template-columns: 1fr; }" in section
assert ".strip .kv > div {" in section
assert "grid-template-columns: max-content minmax(0, 1fr)" in section
refresh = js[js.index("async function refresh("):js.index("async function refreshEvents(")]
assert "state.strips.get(payload.engine)" in refresh, "the strip rebuilds its cards per poll"
assert "$('#strip').appendChild(strip)" in refresh
assert "renderStripCard(strip, payload, stateRow)" in refresh
prune = js[js.index("function pruneEngines("):js.index("function makeCharts(")]
assert "state.strips" in prune and "state.panels" in prune, \
"an engine that leaves the payload leaves a card behind"
def test_llm_strip_scopes_its_rows_to_the_engines_that_publish_them(client) -> None:
"""gufo gets slots and speculative decoding; vLLM gets running and queue time."""
rows = _strip_rows(client.get("/static/page-llm.js").text)
assert rows["slots_busy"] == "'gufo'" and rows["spec_accept"] == "'gufo'"
assert rows["running"] == "'vllm'" and rows["queue_mean"] == "'vllm'"
assert rows["waiting"] is None and rows["cache_hit"] is None and rows["tokens_total"] is None
js = client.get("/static/page-llm.js").text
scoped = js[js.index("function stripRowsFor("):js.index("function stripCard(")]
assert "row.only" in scoped and "row.only.indexOf(engine)" in scoped
def test_llm_strip_rows_name_cells_the_engines_publish(client, metrics_text) -> None:
"""The strip renders what the payload sends — no row whose cell no engine produces.
`STRIP_ROW_PATHS` mirrors the JS table; every metric cell is checked against the
capture-parsed metrics of each engine that would render that row, so a rename in
`sources/llm.py` fails here instead of printing a permanent `n/a` in the browser, and a
row scoped to one engine cannot leak into the other's card.
"""
rows = _strip_rows(client.get("/static/page-llm.js").text)
assert set(rows) == set(STRIP_ROW_PATHS), \
f"the served strip rows drifted: {set(rows) ^ set(STRIP_ROW_PATHS)}"
parsed = {"gufo": parse_gufo_metrics(metrics_text("gufo")),
"vllm": parse_vllm_metrics(metrics_text("vllm"))}
for key, only in rows.items():
paths = STRIP_ROW_PATHS[key]
if paths[0].startswith("sessions."):
continue # its own payload object; pinned on the JS below
shown = {engine: metrics for engine, metrics in parsed.items()
if not only or engine in only}
present = [engine for engine in shown if any(path in shown[engine] for path in paths)]
if key in ALARM_ROW_KEYS:
# an alarm the engine does not count simply is not there: the chip renders only
# while non-zero, so absence *is* the zero state rather than a dead row
assert present, f"no engine publishes {'/'.join(paths)} for the {key} alarm"
continue
assert len(present) == len(shown), (
f"the {sorted(set(shown) - set(present))} card would print `{key}` as n/a forever "
f"({'/'.join(paths)} is not published)")
def test_llm_strip_slot_count_counts_published_slots_and_never_invents_zero(client) -> None:
"""`slots busy` is a count of `/slots` rows; when /slots did not answer it is `n/a`.
Counting rows the server published is a count of published facts. An empty list from a
server that never answered is not a measurement of an idle engine, so the row falls back
to the published `sessions.active` carrying the API's own `detail` string (rule 2).
"""
js = _code(client.get("/static/page-llm.js").text)
slots = js[js.index("case 'slots_busy':"):js.index("case 'cache':")]
assert "sessions.source" in slots and "sessions.list" in slots
assert "is_processing" in slots, "the count ignores the authoritative busy verdict"
assert "sessions.detail" in slots, "the fallback lost the API's own provenance"
def test_llm_strip_kv_bar_is_coloured_by_the_payload_verdict_only(client) -> None:
"""The strip's KV bar is the page's one rated cell: `severity` in, class out.
`sources/llm.py` rates `llm.kv_cache_usage_pct`, the ratio→percent conversion the panel
cell used to make rides with it, and `Dash.severityFrom` is the only thing that decides a
colour — no threshold in the page, and the stylesheet adds no palette (`ok` stays quiet
ink, exactly like the `.cell` rows and the fleet table).
"""
body = _strip_render_path(client.get("/static/page-llm.js").text)
assert "severityFrom(kv.severity)" in body
assert "cellValue(engine, metrics, sessions, 'cache')" in body
assert "Math.max(0, Math.min(100, kv.value))" in body, "the fill stopped clamping"
assert body.count("severityFrom") == 1, "a second strip row invented a verdict"
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- engine stats strip"):css.index("/* ---- fleet totals")]
assert ".stat-bar.warn .value { color: var(--warn); }" in section
assert ".stat-bar.fail .value { color: var(--fail); }" in section
assert ".stat-bar.ok" not in section, "a healthy KV reading must stay quiet ink"
assert ".prov:empty" in section, "an empty provenance tag draws a pill of nothing"
def test_llm_strip_throughput_carries_provenance_without_colour(client) -> None:
"""`prompt tok/s` and `decode tok/s` keep the `Δ<window>` tag and stay ink.
The tag is generated from the cell's `source` (never hardcoded), the exact source rides
in the row's tooltip, and `severity.py` rates no throughput — so nothing here may paint
it (0 tok/s on an idle engine is healthy).
"""
js = client.get("/static/page-llm.js").text
body = _strip_render_path(js)
assert "row.rate ? provenance(value) : ''" in body
assert "notes.join" in body and "value.source" in body
assert "severityFor" not in js
def test_llm_strip_and_panel_print_no_value_twice(client) -> None:
"""One value, one home — the UPS hero row's rule, applied across two components.
Task 03 lifted sessions, queue depth, KV pressure, the cache-hit and spec-accept percents,
the lifetime token counter and the alarm chips out of the panel cell list rather than
drawing them twice. The two throughput heroes are the exemption the task names: the panel
keeps its hero row (so does `llm_monitor`, which carries the pair in both places), and
everything else appears once. Queue time moved with its p95 foot, so the move lost nothing.
"""
js = _code(client.get("/static/page-llm.js").text)
start = js.index("const CELLS = [")
panel_keys = set(re.findall(r"\{ key: '(\w+)'", js[start:js.index("\n];", start)]))
assert panel_keys == {"decode", "prompt", "ttft", "tpot"}, \
f"the panel cell list changed shape: {sorted(panel_keys)}"
shared = set(_strip_rows(js)) & panel_keys
assert shared == {"decode", "prompt"}, f"printed in the strip and the panel: {sorted(shared)}"
body = _strip_render_path(js)
assert "p95Value(metrics, row.key)" in body, "queue time lost its p95 in the move"
def test_llm_strip_prints_no_peak_context_number(client) -> None:
"""The phase's assumption, pinned: `llm_monitor`'s peak-ctx row does not come over.
Its cell is `llamacpp:n_tokens_max`, a since-start maximum rather than a glance number,
and the task says this strip gets no peak-context cell. The series stays stored and
queryable from `/api/llm/history` — not plotting it is not removing it.
"""
js = _code(client.get("/static/page-llm.js").text)
assert "n_tokens_max" not in js, "/llm reads the peak-context gauge again"
assert not [key for key in _strip_rows(js) if "peak" in key]
def test_llm_fleet_block_leads_the_page_with_the_engine_sums(client) -> None:
"""The page opens with llm_monitor's aggregate block: the sums no panel can say.
Phase 04 retired these numbers to counts; the operator asked for the aggregate
throughput back (`TODO.md:5`), and phase 07 restores it in `llm_monitor`'s shape —
status dot, "all engines", an `N of M engines` tag, the note that the sums cover only
the engines answering now, then running / waiting / prompt / decode / total tokens.
The counts-only card it replaces (`#totals`, `f-live`, `f-active`) is gone, ids and all:
an element nothing writes is the orphan this repo pins against.
"""
html = _code(client.get("/llm").text)
assert 'id="fleet"' in html, "/llm lost the aggregate block"
assert html.index('id="fleet"') < html.index('id="engines"'), \
"the fleet block is no longer above the engine panels"
for part in ("fleet-dot", "fleet-count", "f-running", "f-waiting", "f-prompt",
"f-decode", "f-tokens"):
assert f'id="{part}"' in html, f"/llm lost the fleet block's {part}"
for words in ("all engines", "sums only engines answering now"):
assert words in html, f"/llm lost the fleet block's `{words}` line"
for gone in ('id="totals"', "f-live", "f-total", "f-active", "f-kv", "seam-note"):
assert gone not in html, f"/llm still carries {gone} from the retired counts card"
js = client.get("/static/page-llm.js").text
body = _fleet_render_path(js)
assert "block.hidden = total < 2" in body, "the fleet block no longer hides for one engine"
assert "hidden" in _fleet_tag(html), \
"the block no longer ships hidden, so a single engine would show a fleet row"
def test_llm_fleet_block_reads_only_paths_the_api_publishes(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Every number the block prints is a `fleet` key of `/api/llm/snapshot`.
The page sums nothing itself — `fleet_totals()` does — so the field table in
`page-llm.js` is a list of API paths, and a rename in `sources/llm.py` must fail here
rather than print a permanent `n/a` in the browser. Pinned with both engines configured
(the keys exist, empty-handed because nothing answered) and with neither (404
`configured: false` without dialling, invariant 3 — and the shipped block stays hidden).
"""
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
fleet = test_client.get("/api/llm/snapshot").json()["fleet"]
assert set(fleet) >= FLEET_BLOCK_PATHS, FLEET_BLOCK_PATHS - set(fleet)
# the served field table is exactly the sum rows, in llm_monitor's order
js = test_client.get("/static/page-llm.js").text
table = _fleet_field_metrics(js)
assert table == ["requests_active", "requests_waiting",
"prompt_tokens_per_second", "decode_tokens_per_second",
"tokens_total"], "the fleet row changed shape without the API"
assert set(FLEET_SUM_PATHS) == set(table), "the block totals something the API does not"
# and the tag's two numbers are API paths too, not arithmetic the page invented
body = _fleet_render_path(js)
assert "fleet.engines_total" in body and "fleet.engines_live" in body
with build_client(tmp_path, monkeypatch) as bare:
response = bare.get("/api/llm/snapshot")
assert response.status_code == 404
assert response.json()["configured"] is False
html = _code(bare.get("/llm").text)
assert 'id="fleet"' in html and "hidden" in _fleet_tag(html), \
"an unconfigured engine must not leave a fleet row on the page"
def test_llm_fleet_totals_drop_an_engine_that_stopped_answering(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A total presented as current is current: a dead engine contributes nothing.
`fleet_totals()`'s docstring is the contract and the block is where it is read. The
injected payload survives the dead-port polls (a failed poll keeps the last good one), so
these are the numbers a browser would get with gufo answering and vLLM down — the live
engine's sums verbatim, `engines_live` one of two, and not one zero where a number the
dead engine never published would sit (rule 2).
"""
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
hub.jobs["llm:gufo"].state.payload = {
"engine": "gufo",
"metrics": {"decode_tokens_per_second": {"value": 41.0},
"prompt_tokens_per_second": {"value": 17.5},
"requests_processing": {"value": 2.0},
"requests_deferred": {"value": 1.0},
"tokens_total": {"value": 512.0}},
"sessions": {"active": 2, "list": []}}
fleet = test_client.get("/api/llm/snapshot").json()["fleet"]
assert fleet["engines_total"] == 2 and fleet["engines_live"] == 1
assert fleet["decode_tokens_per_second"] == pytest.approx(41.0)
assert fleet["prompt_tokens_per_second"] == pytest.approx(17.5)
assert fleet["requests_active"] == pytest.approx(2.0)
assert fleet["requests_waiting"] == pytest.approx(1.0)
assert fleet["tokens_total"] == pytest.approx(512.0)
# and with neither engine answering, every sum is null, never a confident 0
hub.jobs["llm:gufo"].state.payload = {"engine": "gufo", "metrics": {},
"sessions": {"active": None, "list": []}}
empty = test_client.get("/api/llm/snapshot").json()["fleet"]
assert empty["engines_live"] == 0
assert [empty[key] for key in FLEET_SUM_PATHS] == [None] * len(FLEET_SUM_PATHS)
def test_llm_fleet_block_proves_where_its_sums_come_from(client) -> None:
"""The seam tooltip, the excluded-engine tooltip, and an `n/a` with no unit.
Two provenance rules the rest of the app holds to, on the one block that adds engines
together: the prompt sum carries `fleet.seams` — the API's own words on why gufo's and
vLLM's prompt counters are not the same quantity — and the tag's tooltip names the
engines `fleet_totals()` left out, so a partial total says so in words and not only by
being dimmed. And a sum the API did not publish prints `n/a` with its unit cleared: "n/a
tok/s" would read as a measurement of nothing.
"""
js = _code(client.get("/static/page-llm.js").text)
body = _fleet_render_path(js)
assert "fleet.seams" in body, "the prompt sum lost the API's seam note"
assert "not answering — excluded from every sum below" in body, \
"the tag no longer names the engines left out of the totals"
assert "value == null ? ''" in body, "an n/a sum can still print a unit"
assert "value == null ? ''" in body, "an n/a sum can still print a unit"
def test_llm_fleet_block_colours_no_number(client) -> None:
"""No verdict on the aggregate row — and no dimming of anything but the tag.
`severity.py` rates no throughput and no queue depth (0 tok/s on an idle fleet is
healthy), so the block renders ink: the render path touches no `severity`, calls no
`Dash.hero()`, and the stylesheet section for the block reaches for no verdict token.
The one coloured thing on it is the shared **reachability** dot — the same distinction
the engine panels draw between their numbers and their header dot — and the partial-fleet
signal is the shared `.stale` dim on the count tag, which is opacity, not a verdict.
"""
js = _code(client.get("/static/page-llm.js").text)
body = _fleet_render_path(js)
assert "severity" not in body, "the fleet block invented a verdict"
assert "hero(" not in body, "the fleet block drew a hero number it cannot rate"
assert "setDot(" in body and "markStale(tag" in body, \
"a dead engine no longer dims the tag or darkens the block's dot"
css = client.get("/static/dash.css").text
section = _css_section(css, "fleet totals")
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
assert token not in section, f"the fleet block coloured a number with {token}"
assert ".fleet-head" in section and ".fleet-note" in section, \
"the fleet block's head row lost its styling"
# ---- the `/llm` gufo slot grid (phase 07 task 04) -------------------------
#
# `llm_monitor`'s slot cards (`index.html` ~L1179), rebuilt against the capture instead of a
# memory: one keyed card per gufo slot, every cell naming **one** key of
# `tests/fixtures/slots_gufo.json`. Two things are pinned below, because they are the two ways
# this component failed before: a card that reads a key the server does not publish (dashes
# forever), and a card that is rebuilt every poll so a busy slot flickers in front of you.
#: The keys a slot card reads, and where they live: `field` (and its optional `sub` /
#: `ratioAgainst` partners) index the row, or its first `next_token` entry when `path` says so.
SLOT_ROW_PATHS = ("field", "sub", "ratioAgainst")
def _slot_table(js: str) -> list[dict[str, str | None]]:
"""The served `SLOT_FIELDS` table, one dict per printed cell."""
body = _code(js)
start = body.index("const SLOT_FIELDS = [")
table = body[start:body.index("\n];", start)]
cells: list[dict[str, str | None]] = []
for chunk in table.split("{ key: '")[1:]:
cell: dict[str, str | None] = {"key": chunk.split("'")[0]}
for option in (*SLOT_ROW_PATHS, "path"):
found = re.search(rf"{option}: '([a-z_0-9]+)'", chunk)
cell[option] = found.group(1) if found else None
cells.append(cell)
return cells
def _slot_js(js: str) -> str:
"""The slot grid's whole code path, comments out: `slotValue()` to the end of the block."""
body = _code(js)
return body[body.index("function slotValue("):body.index("function makeCharts(")]
def _slot_markup(js: str) -> str:
"""Just `slotSection()` — the markup the grid, its label and its empty states are built from."""
body = _code(js)
return body[body.index("function slotSection("):body.index("function renderPanel(")]
def test_llm_gufo_panel_holds_the_slot_grid_under_its_label(client) -> None:
"""`sessions (GET /slots)`, then the keyed grid, then the note about what it cannot contain.
The label is `llm_monitor`'s, the grid ships `hidden` (the empty state is a sentence, never
a card of nothing), and the note stays because a queued (deferred) request is not in
`/slots` at all — without that line an idle grid reads as "nothing waiting".
"""
js = client.get("/static/page-llm.js").text
markup = _slot_markup(js)
assert "sessions (GET /slots)" in markup, "/llm lost the slot grid's label"
# label, grid, empty state, note — in that order, and the grid and the empty state both
# ship hidden so nothing paints before the payload says which of them is true
order = ['class="section-label" data-slots-label', "sessions (GET /slots)",
'<div class="slots" data-slots role="list" hidden>',
'class="slots-empty" data-slots-empty hidden',
'class="slots-note" data-slots-note hidden']
position = -1
for needle in order:
found = markup.index(needle)
assert found > position, f"`{needle}` is out of order in the slot section"
position = found
assert "queued (deferred) requests are not in /slots" in js, "the queued-request note is gone"
assert "renderSlotGrid(card, enginePayload)" in js, "nothing draws the grid any more"
# the label is a heading and the grid a list: a screen reader gets the block's name and
# its item count, which a stack of divs does not say
assert '<h4 class="section-label" data-slots-label>' in markup, \
"the slot section's label stopped being a heading"
assert 'data-slots role="list"' in markup, "the grid lost its list semantics"
# the grid's own markup: a keyed card, the state pill, and one dd per printed cell
card = js[js.index("function slotCard("):js.index("function slotSub(")]
assert "setAttribute('role', 'listitem')" in card, "the slot cards left the list"
assert 'data-slot-cell="${cell.key}"' in card, "the card cells are not table-driven"
assert 'data-pill' in card and 'data-mark' in card, "the card lost its state pill or glyph"
def test_llm_slot_grid_style_is_a_marker_and_not_a_verdict(client) -> None:
"""A busy slot is a border, a glyph and a word — `severity.py` rates no slot.
Phase 06 settled this argument for GPU utilisation on an LLM server: a slot generating
tokens is the job, so the accent colour says *this is the live one* and no `--ok/--warn/
--fail` token appears anywhere in the section. The glyph (`○`/`●`) and the pill's word mean
the marker survives monochrome, which is the battery glyph's rule applied here.
"""
css = client.get("/static/dash.css").text
section = _css_section(css, "engine slot grid")
for selector in (".slots {", ".slot {", ".slot.busy", ".slot-top", ".slot-id",
".slot-model", ".slot dt", ".slot dd", ".slots-empty", ".slots-note",
".section-label"):
assert selector in section, f"dash.css lost the {selector} rule"
assert "var(--accent)" in section, "the busy marker stopped using the non-verdict accent"
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
assert token not in section, f"the slot grid coloured a state with {token}"
js = client.get("/static/page-llm.js").text
assert "'\\u25cf'" in js or "'●'" in js, "the busy glyph left the card"
assert "processing" in _slot_js(js), "the pill stopped saying the state in a word"
def test_llm_slot_cells_each_name_one_key_the_capture_publishes(client) -> None:
"""Every printed cell traces to a key of `tests/fixtures/slots_gufo.json`.
One `field` per cell and no `a ?? b`: the deleted table's `slot.n_prompt ??
slot.prompt_tokens` guessing is what made it print dashes and then plausible noise. A key
the capture does not have fails here, and the alias the capture answers beside `id_task`
must not appear in the page at all — the contract names the canonical one.
"""
# comments out: this test bans names the prose is allowed to explain
js = _code(client.get("/static/page-llm.js").text)
cells = _slot_table(js)
assert cells, "the slot grid has no cell table to read"
assert [cell["key"] for cell in cells] == ["req", "prompt", "cache", "generated",
"remain", "ctx"], \
"the slot card's cell set changed without the capture"
for cell in cells:
assert cell["field"], f"the {cell['key']} cell names no row key"
rows = load_slots_capture()
assert rows, "the capture is empty, so this test would prove nothing"
for cell in cells:
for option in SLOT_ROW_PATHS:
key = cell[option]
if key is None:
continue
in_next = cell["path"] == "next_token"
required = REQUIRED_NEXT_TOKEN_FIELDS if in_next else REQUIRED_ROW_FIELDS
optional = OPTIONAL_NEXT_TOKEN_FIELDS if in_next else OPTIONAL_ROW_FIELDS
assert key in required or key in optional, (
f"the {cell['key']} cell reads {key!r}, which no contract table declares — the "
"page would be reading a field nobody pinned")
for row in rows:
source = row["next_token"][0] if in_next else row
if key in required:
assert key in source, (
f"the {cell['key']} cell reads {key!r}, which "
f"{'next_token[0]' if in_next else 'the row'} does not publish — this is "
"how the deleted table rendered dashes")
elif key in source: # optional: `prob` is on no captured row, and that is legal
assert isinstance(source[key], (int, float)), (
f"{key!r} is {source[key]!r}, which the cell cannot format")
assert cells[0]["field"] == CANONICAL_TASK_ID_FIELD, \
"the request cell stopped reading the canonical task id"
for alias in TASK_ID_ALIASES:
assert alias not in js, f"page-llm.js renders the {alias} alias the contract does not name"
# the old table's invented names stay invented, and so does its per-slot speed
for gone in ("n_predict", "decode_per_second", "cache_usable", "n_ctx_used"):
assert gone not in js, f"page-llm.js reads the invented {gone} again"
def test_llm_slot_card_prints_no_speed_it_cannot_measure(client) -> None:
"""No per-slot speed cell: the capture publishes no per-slot rate key.
`llm_monitor`'s grid has one, and so did the deleted table (`decode_per_second`) — which is
why it was dashes. A page-side Δn_decoded/Δt would be a measurement with no estimator, no
`Δ<window>` label and no `source` string, i.e. rule 2 with a confident number printed on it,
so the number stays on the engine's strip card and panel, where `rates.py` owns it.
"""
js = _code(client.get("/static/page-llm.js").text)
assert not [cell for cell in _slot_table(js)
if cell["key"] in {"speed", "tok_s", "decode"}], "a slot cell grew a rate"
assert "decode_per_second" not in js
def test_llm_slot_sentinels_print_a_word_or_na_never_a_negative_number(client) -> None:
"""`-1` is llama.cpp's "nothing in this slot", and it never becomes a countdown.
An idle slot answers `id_task: -1` and `next_token[0].n_remain: -1` (both in the capture).
The request cell says `idle` in a word; the remaining cell prints `n/a`. Neither prints a
negative number, and neither prints the `0` that would read as "one token from done".
"""
body = _slot_js(client.get("/static/page-llm.js").text)
assert "value < 0 ? 'idle'" in body, "the request cell prints the -1 sentinel again"
assert "value >= 0 ? value : null" in body, "the remaining cell prints the -1 sentinel"
def test_llm_slot_card_header_reads_only_fields_the_contract_declares(client) -> None:
"""The card top prints `model`, `speculative`, `id` and the state pair — and nothing else.
`id` keys the card (escaped once at build time), `is_processing` is the authoritative pill,
`state` labels a number the boolean cannot express, and `model` + `speculative` are the
footnote beside them. Each is declared in a contract table, so a rename fails there. The
user-content field a busy row carries (`prompt`) is the one key this page must never read.
"""
js = client.get("/static/page-llm.js").text
header = js[js.index("function renderSlotCard("):js.index("function renderSlotGrid(")]
read = {"is_processing", "state", "model", "speculative"}
for key in read:
assert f"row.{key}" in header, f"the slot card header stopped reading {key}"
declared = set(REQUIRED_ROW_FIELDS) | set(OPTIONAL_ROW_FIELDS)
assert read <= declared, "the header reads a field no contract table declares"
for field in USER_CONTENT_FIELDS:
assert field not in header, f"the slot card prints the user's {field}"
assert "escapeHtml(id)" in js, "the slot id reached the markup unescaped"
def test_llm_slot_cards_are_keyed_created_once_and_patched_in_place(client) -> None:
"""A busy slot does not get rebuilt every poll — and a slot that leaves loses its card.
`appendChild` of the existing node keeps DOM order == slot order; everything else is a text
patch, so a poll cannot flicker a card or re-announce it to assistive tech for a difference
of a few tokens. An engine that leaves the payload loses its card map with its panel, or the
refs of removed nodes would be re-appended when it came back.
"""
js = client.get("/static/page-llm.js").text
body = _slot_js(js)
assert "state.slots.get(enginePayload.engine)" in body, "the grid stopped keying its cards"
assert "if (!refs) {" in body and "refs = slotCard(id)" in body
assert "grid.appendChild(refs.root)" in body, "slot order is no longer kept by moving nodes"
assert "byId.delete(id)" in body, "a slot that left keeps its card forever"
assert "byId.forEach((refs) => refs.root.remove())" in body, \
"the empty state left cards on the page"
assert "setTxt(box.value" in body, "the cell values stopped being patched in place"
assert "innerHTML" not in body.split("function renderSlotCard(")[1], \
"a card is rebuilt from markup on a poll"
prune = js[js.index("function pruneEngines("):js.index("/* ---- the gufo slot grid")]
assert "state.slots.delete(engine)" in prune, "a pruned engine kept its slot cards"
def test_llm_slot_grid_says_why_there_are_no_rows(client) -> None:
"""Two honest empty states, and neither of them is a card or a 0-count claim.
`/slots` did not answer → the API's own `sessions.detail` sentence (the same words the
strip's session row carries). It answered with no rows → "engine reports 0 slots", which is
a fact about that answer. `sessions.source` is the gate that tells the two apart.
"""
body = _slot_js(client.get("/static/page-llm.js").text)
assert "Boolean(sessions.source)" in body, "the grid cannot tell no-answer from no-rows"
assert "engine reports 0 slots" in body
assert "sessions.detail ||" in body, "the no-answer state lost the API's own sentence"
assert "no /slots answer from this engine" in body, "the no-answer state can print undefined"
def test_llm_vllm_panel_says_there_is_no_per_slot_json(client) -> None:
"""vLLM gets one line and no grid — no fake rows for a server that cannot answer.
`sources/llm.py`'s `_sessions()` publishes counts for vLLM and says so in `detail`; the
panel turns that into the `llm_monitor` line, with the API's sentence as the tooltip. The
gate is the engine's name (`SLOT_ENGINES`), not `sessions.list.length`, so an engine that
has not answered yet still shows its label and the honest empty sentence.
"""
js = client.get("/static/page-llm.js").text
markup = _slot_markup(js)
branch = markup.split("if (SLOT_ENGINES.indexOf(engine) < 0) {")[1].split("}")[0]
assert 'class="slots-note" data-slots-note' in branch, "the vLLM line lost its element"
assert "NO_SLOT_JSON" in branch and "vLLM exposes no per-slot JSON" in js
assert '<div class="slots"' not in branch, "the vLLM panel grew a grid it cannot fill"
body = _slot_js(js)
assert body.startswith("function slotValue("), "the grid's code path moved"
assert "if (!grid) {" in body, "renderSlotGrid no longer leaves a panel with no grid alone"
assert "note.title = enginePayload.sessions.detail" in body, \
"the vLLM line stopped carrying the API's own words"
def test_the_captured_slot_rows_fill_the_cells_the_grid_prints(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The capture, through the API, into the keys the served cell table reads.
The fixture rows are injected as a live job payload (the `/slots` passthrough verbatim), so
this is the render path's input as the browser would receive it: `sessions.source` set (the
grid's gate), every row present, and every key `SLOT_FIELDS` names resolvable in it. A
rename in `sources/llm.py` or in gufo fails here rather than leaving a card of `n/a`s.
"""
rows = load_slots_capture()
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
hub.jobs["llm:gufo"].state.payload = {
"engine": "gufo", "metrics": {"requests_processing": {"value": 1.0}},
"sessions": {"active": 1, "list": rows, "source": "gufo /slots", "detail": None},
}
engines = test_client.get("/api/llm/snapshot").json()["engines"]
gufo = next(row for row in engines if row["engine"] == "gufo")
sessions = gufo["sessions"]
assert sessions["source"] == "gufo /slots", "the grid's gate is gone from the payload"
assert sessions["list"] == rows, "the rows are no longer the passthrough the grid reads"
cells = _slot_table(test_client.get("/static/page-llm.js").text)
for cell in cells:
for option in SLOT_ROW_PATHS:
key = cell[option]
if key is None:
continue
in_next = cell["path"] == "next_token"
required = REQUIRED_NEXT_TOKEN_FIELDS if in_next else REQUIRED_ROW_FIELDS
for row in sessions["list"]:
source = row["next_token"][0] if in_next else row
if key not in required and key not in source:
continue # an optional sub-line this build does not publish
value = source[key]
assert isinstance(value, (int, float)) and not isinstance(value, bool), (
f"the {cell['key']} cell reads {key!r} = {value!r}, which the number "
"formatter cannot print")
def test_llm_charts_are_secondary_and_named(client) -> None:
"""The four trend charts stay, at half height, *below* the engine panels, each named.
Half of the old 165px and moved after `#engines` is the "cells read first" decision;
every `LineChart` gets an `ariaLabel` because the legend is a sibling that labels
nothing (`/system` did this in phase 03, `/llm` never had).
"""
html = _code(client.get("/llm").text)
assert html.index('id="engines"') < html.index('id="chart-decode"'), \
"the charts are back on top of the engine panels"
for chart in ("chart-decode", "chart-prompt", "chart-kv", "chart-queue"):
assert f'id="{chart}"' in html, f"/llm lost the {chart} chart"
js = client.get("/static/page-llm.js").text
assert "height: 110" in js and "height: 165" not in js
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
assert charts.count("ariaLabel:") == 4, "an /llm chart lost its accessible name"
def test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling(client) -> None:
"""A 5-row gufo panel beside a 7-row vLLM panel keeps its own height.
Found while comparing against `llm_monitor`: the shorter engine card grew 172 px of
blank card, and a blank panel reads as "no data" — the exact misreading the unconfigured
rules exist to prevent. Pinned on the served markup so the `.grid.items-start` rule in
`dash.css` cannot silently become dead CSS (and the void cannot silently come back).
"""
html = _code(client.get("/llm").text)
engines_div = html[html.index('<div class="grid'):html.index('id="engines"') + 40]
assert "items-start" in engines_div, "the engine grid stretches its cards again"
css = client.get("/static/dash.css").text
assert ".grid.items-start" in css, "the rule the engine grid relies on is gone"
def test_overview_is_exactly_two_panels(client) -> None:
"""The landing page answers one question: what is wrong, and what is the fleet doing.
Asserted on the served bytes because there is no JS test runner (a locked anchor):
the summary cards and the raw event feed are what made this page busy, and once a
panel is gone a browser is the only thing that would notice its return.
"""
html = client.get("/").text
assert html.count("<section") == 2, "the overview grew a third panel"
assert 'id="alerts"' in html and 'id="alerts-clear"' in html
assert 'aria-live="polite"' in html
assert 'id="fleet-table"' in html
for gone in ("sys-cpu", "sys-mem", "sys-disk", "sys-gpus", "llm-decode", "llm-kv",
"ups-list", 'id="events"'):
assert gone not in html, f"the overview still renders {gone}"
# the header's source dots are chrome owned by app.js, not the deleted card dots
assert 'id="dot-system"' not in html and 'id="nav-status"' in html
def test_overview_alert_rows_link_to_the_dashboard_that_explains_them(client) -> None:
"""Every alert carries the `href` its row turns into a link; the page invents none."""
page_js = client.get("/static/page-index.js").text
assert "alert.href" in page_js
css = client.get("/static/dash.css").text
for selector in ("ul.alerts li.ok .src", "ul.alerts li.warn .src", "ul.alerts li.fail .src",
"table.data td.warn", "table.data td.fail"):
assert selector in css, f"dash.css lost {selector}"
# the swap that puts a live problem above the fleet table
assert 'data-state="alerting"' in css
def test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering(client) -> None:
"""A dead host keeps its last good numbers, so the row has to say they are stale.
`AGENTS.md` invariant 4: keep the payload and mark it stale, never zero it. The
overview's table is the one place those last good numbers still print, so it reads the
per-source verdicts `/api/summary` sends (`system.states`) and applies the shared
`.stale` class the other pages already use — no new palette, no JS threshold.
"""
page_js = client.get("/static/page-index.js").text
assert "system.states" in page_js, "the fleet table stopped reading reachability"
assert 'class="stale"' in page_js
assert ".stale" in client.get("/static/dash.css").text
# Which script owns which page, for the orphan check below. `charts.js` and `app.js` are
# loaded by every page, so an id either page code or shared chrome may read.
PAGE_SCRIPTS = {
"/": "page-index.js",
"/system": "page-system.js",
"/disks": "page-disks.js",
"/llm": "page-llm.js",
"/ups": "page-ups.js",
# `/settings` is audited like a dashboard even though it is not one: an id nothing reads is
# dead markup whatever page it sits on, and this page has five new ones.
SETTINGS_PAGE: "page-settings.js",
}
# `/login` is deliberately **not** in this table even though it is audited (phase 11): a
# signed-in client asking for it gets a 303 to the dashboard, so this helper's own
# redirect-following `get` would audit `/`'s markup and call it `/login`. The login page's
# orphan-`id`, CDN and masthead checks live in `tests/test_auth_api.py`, fetched unsigned, and
# `/static/page-login.js` is in `ASSETS` above.
SHARED_SCRIPTS = ("app.js", "charts.js")
def test_no_page_carries_an_id_nothing_reads(client) -> None:
"""An id no script queries, no stylesheet styles and no attribute points at is dead markup.
The summary cards were retired this way — a `<span id="sys-cpu">` left behind after its
writer was deleted is the same mistake in slower motion. Asserted on served bytes: the
browser console is otherwise the only thing that would notice.
"""
scripts = {name: client.get(f"/static/{name}").text for name in
(*SHARED_SCRIPTS, *PAGE_SCRIPTS.values(), "dash.css")}
for path, own_script in PAGE_SCRIPTS.items():
html = _code(client.get(path).text)
corpus = "".join(_code(scripts[name])
for name in (*SHARED_SCRIPTS, own_script, "dash.css"))
# ids the markup uses itself: labels, targets, `<label for>`
anchors = set(re.findall(
r'(?:aria-labelledby|aria-describedby|aria-controls|for)="([^"]+)"', html))
for element_id in re.findall(r'id="([^"]+)"', html):
quoted = (f'"{element_id}"' in corpus or f"'{element_id}'" in corpus
or f"#{element_id}" in corpus or element_id in anchors)
assert quoted, f"{path} has id=\"{element_id}\" that nothing reads"
# ---- the flavour-text audit (phase 08 task 02) ----------------------------
#
# `TODO.md:7`: "Get rid of most of the flavor text and add a note to AGENTS.md to not add it
# unless necessary. It just comes off as LLM generated slop, not helpful information." The
# test is one question per string — *would the operator act differently without it?* — and
# the answer decided the deletion. Copy that explains how the app builds its numbers is not
# helpful information; a provenance tag, a measured Δ window, and the sentence beside an
# empty panel are. `AGENTS.md` → *UI conventions* now carries the rule; the pins below are
# what stop the deleted sentences crawling back, because a screenshot only shows them once
# somebody already knows to look.
#: Fragments deleted from the served pages. The first two are the lines the operator quoted.
DELETED_FLAVOUR = (
# the overview's footer
"every number is scraped from the box",
"Sources the operator never configured are not problems",
# /system's "Per host" sub-line and its footer
"agent-reported values",
"each panel's charts cover only that host",
"numbers are measured, not inferred",
# /llm's footer, the one half-sentence of it that lived in the script too
"each throughput number carries its provenance",
"rates over the Δ window in each tag",
"a server you do not run",
# /ups's footer and its empty state
"commands are the only write path in this app",
"The dashboard speaks the NUT protocol",
# the three event feeds' header captions, all duplicated by an empty-state sentence
"unreachable / reachable / reboots",
"engine up / down",
"state changes, self-test results, commands sent",
)
@pytest.mark.parametrize("fragment", DELETED_FLAVOUR)
def test_no_page_narrates_its_own_pipeline(client, fragment: str) -> None:
"""The deleted copy is gone from every served byte, markup and script alike.
Asserted across all four pages and all five scripts rather than the file it came from:
the same sentence migrating into another page's card is the same slop, and a pin that
only guards its original file would let that through.
"""
for path in (*MARKUP_PAGES, *ASSETS):
body = client.get(path).text
assert fragment not in body, f"{path} narrates the app again: {fragment!r}"
def test_pages_leave_no_empty_narration_container(client) -> None:
"""A deleted sub-line removes its container, not just its words.
An empty `<span class="sub">` is markup that reserves a line for nothing — the page head
renders a gap where the sentence used to be. Elements the scripts fill (`#sub`,
`#alerts-count`) carry an `id` and are not in this category.
"""
for path in MARKUP_PAGES:
html = _code(client.get(path).text)
for cls in ("sub", "faint", "mono"):
assert f'<span class="{cls}"></span>' not in html, \
f"{path} keeps an empty .{cls} container"
def test_the_copy_the_operator_acts_on_survives_the_audit(client) -> None:
"""The other half of the deletion: functional copy stays.
Stripping flavour must not become stripping information. Every string here answers a
question the operator is asking while looking at the page — why is this panel empty,
is this number a measurement, is this battery real, is this host's data last-good —
and each one is the reason the audit deletes sentences and not labels.
"""
pages = {path: client.get(path).text for path in PAGES}
scripts = {name: client.get(f"/static/{name}").text for name in PAGE_SCRIPTS.values()}
# an empty panel says why, in terms the operator can act on
assert "No hosts configured" in pages["/"]
assert "HOST_TARGETS" in pages["/system"]
# …including the fourth page's: a `n/a` drive card is only actionable if the page names
# the two things that make SMART readable (the binary and the device-read grant).
assert "smartmontools" in pages["/disks"] + scripts["page-disks.js"]
assert "disk group" in scripts["page-disks.js"]
assert "No engines configured" in pages["/llm"]
assert "No UPS reporting" in pages["/ups"]
assert "instcmds" in pages["/ups"], "the UPS setup note stopped naming the grant that matters"
assert "3493" in pages["/ups"], "the page stopped warning that upsd's port is not always 3493"
# …and the setup copy names the grant NUT actually checks. `README.md` is explicit: a
# dashboard user gets `instcmds`, never `actions = -`, so a page that told the operator
# to write that was sending them to fix the wrong line in `upsd.users`.
assert "actions = -" not in pages["/ups"] + scripts["page-ups.js"] + scripts["page-index.js"]
assert "instcmds" in scripts["page-ups.js"] and "instcmds" in scripts["page-index.js"]
# a Δ label is a measurement, and the rate's provenance travels with the rate
assert "windowLabel(" in scripts["page-system.js"]
assert "Δ" in scripts["page-llm.js"] and "function provenance(" in scripts["page-llm.js"]
assert "at the current load" in scripts["page-ups.js"]
# modelled data is labelled, and so is a feed that cannot contain what you are hunting
assert "SIMULATED" in scripts["page-index.js"] and "SIMULATED" in scripts["page-ups.js"]
assert "queued (deferred) requests are not in /slots" in scripts["page-llm.js"]
assert "Last good values" in scripts["page-index.js"]
def test_api_responses_are_not_cacheable(client) -> None:
for path in ("/api/config", "/api/health", "/api/summary"):
assert "no-store" in client.get(path).headers.get("cache-control", "")
# ---- history -------------------------------------------------------------- #
def test_history_returns_seeded_samples(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
db = str(tmp_path / "hist.sqlite3")
store = SampleStore(db)
now = time.time()
for i in range(30):
store.add("host", "driveripper", "cpu.usage_pct", 10.0 + i, ts=now - 300 + i)
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2,
host_targets=(Target("driveripper", DEAD_URL),)), store)
with TestClient(app) as client:
sign_in(client)
body = client.get("/api/history", params={"source": "host", "host": "driveripper",
"metric": "cpu.usage_pct", "range": "15m"})
assert response_ok(body)
points = body.json()["points"]
assert len(points) >= 25
assert points[-1]["value"] == pytest.approx(39.0)
store.close()
def test_history_rejects_an_unknown_range(client) -> None:
response = client.get("/api/history", params={"source": "host", "host": "h",
"metric": "m", "range": "banana"})
assert response.status_code == 400
def test_history_requires_the_series_coordinates(client) -> None:
assert client.get("/api/history").status_code == 422
@pytest.mark.parametrize(("raw", "seconds"), [
*[(key, float(seconds)) for key, seconds in RANGE_SECONDS.items()],
("45s", 45.0), ("13h", 46_800.0), ("7200", 7_200.0), (" 2D ", 172_800.0),
(None, 3_600.0), ("", 3_600.0),
])
def test_parse_range_reads_every_window_the_ui_offers(raw: Any, seconds: float) -> None:
"""Every chip key (5m…90d), any `<n><s|m|h|d>`, raw seconds, and the 1h default.
`parse_range` is the only thing between a typo'd query string and a chart of the wrong
window, and a wrong window is worse than an error: it still looks like data.
"""
assert parse_range(raw) == seconds
@pytest.mark.parametrize("raw", ["banana", "1w", "h", "-4h", "one"])
def test_parse_range_refuses_a_window_it_cannot_interpret(raw: str) -> None:
"""A 400 that names the value, never a silent fallback to 1h."""
with pytest.raises(HTTPException) as excinfo:
parse_range(raw)
assert excinfo.value.status_code == 400
assert raw in str(excinfo.value.detail)
# ---- the routes the *other* pages chart from -------------------------------- #
#
# `/` renders two panels now; `/system`, `/llm` and `/ups` still need these, and so does
# anything outside the browser. Retiring overview code must not retire them, so the shapes
# are pinned here rather than discovered by a page three phases from now.
def test_system_history_charts_every_configured_host_in_one_request(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""N hosts x M metrics, one round trip.
`/system` no longer uses the N-hosts half of this (every chart belongs to one host and
the page issues one request per host, pinned by
`test_every_system_chart_is_drawn_inside_its_own_host_panel`); the shape is kept because
the metric-list half is exactly what the per-host query costs one request with, and
because the endpoint is a public interface for programmatic consumers.
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),
Target("deskwork", DEAD_URL))) as test_client:
body = test_client.get("/api/system/history",
params={"metrics": "cpu.usage_pct,mem.used_pct",
"range": "1h"}).json()
assert body["hosts"] == ["driveripper", "deskwork"]
assert body["range_seconds"] == 3_600.0
assert len(body["series"]) == 4
assert {(item["host"], item["metric"]) for item in body["series"]} == {
("driveripper", "cpu.usage_pct"), ("driveripper", "mem.used_pct"),
("deskwork", "cpu.usage_pct"), ("deskwork", "mem.used_pct")}
# an explicit `hosts` list narrows it instead of adding to the default
narrowed = test_client.get("/api/system/history",
params={"metrics": "cpu.usage_pct",
"hosts": "deskwork"}).json()
assert narrowed["hosts"] == ["deskwork"]
def test_one_request_per_host_answers_every_per_host_chart(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The exact query `/system` issues per host returns every series that panel draws.
`page-system.js` sends one request per host carrying the utilisation trio
(`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct`), both network rates, and one
`gpu.busy_pct@<index>` per card the payload reported — so a name missing here is a chart
that silently reads "no samples yet" in the browser, and a renamed series in
`sources/host.py` fails here rather than in a screenshot. This is the metric-name half of
the contract; `tests/test_system_contract.py` holds the payload-path half.
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
now = time.time()
metrics = ["cpu.usage_pct", "mem.used_pct", "disk.used_pct",
"net.rx_bytes_per_s", "net.tx_bytes_per_s", "gpu.busy_pct@0"]
for metric in metrics:
store.add("host", "driveripper", metric, 12.5, ts=now - 60)
store.add("host", "driveripper", metric, 25.0, ts=now - 30)
body = test_client.get("/api/system/history", params={
"hosts": "driveripper", "metrics": ",".join(metrics), "range": "1h"}).json()
assert body["hosts"] == ["driveripper"], "one panel, one host"
assert [item["metric"] for item in body["series"]] == metrics
assert [len(item["points"]) for item in body["series"]] == [2] * len(metrics)
# a second card the host does not have answers empty, never zero and never 404
gap = test_client.get("/api/system/history",
params={"hosts": "driveripper", "metrics": "gpu.busy_pct@1"}).json()
assert gap["series"][0]["points"] == []
def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
# a host target too, so the snapshot's job filter has to skip a non-LLM job
with build_client(tmp_path, monkeypatch, vllm_url=DEAD_URL,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
body = test_client.get("/api/llm/snapshot").json()
assert [engine["engine"] for engine in body["engines"]] == ["vLLM"]
assert body["engines"][0]["unreachable"] is True
assert body["fleet"]["engines_total"] == 1
assert body["fleet"]["engines_live"] == 0
assert [state["key"] for state in body["states"]] == ["llm:vllm"]
def test_llm_history_charts_both_engines_by_default(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
body = test_client.get("/api/llm/history",
params={"metrics": "decode_tokens_per_second,kv_cache_usage_ratio",
"range": "15m"}).json()
assert body["engines"] == ["gufo", "vllm"]
assert body["range_seconds"] == 900.0
assert {(item["host"], item["metric"]) for item in body["series"]} == {
("gufo", "llm.decode_tokens_per_second"),
("gufo", "llm.kv_cache_usage_ratio"),
("vllm", "llm.decode_tokens_per_second"),
("vllm", "llm.kv_cache_usage_ratio")}
def test_one_batched_llm_history_request_answers_every_cell_series(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""The exact query `/llm` issues per refresh returns every chart *and* sparkline series.
Phase 04 sends one request carrying the chart trio and the whole spark cell list —
including the two series the layout required (`tpot_seconds` for the vLLM TPOT row,
`spec_accept_rate` for the gufo spec row). A name missing here is a cell that reads
"no history" forever in the browser, which is the dead-cell anti-pattern the phase
wrote NOTES-chart-set to prevent; a rename in `sources/llm.py` fails here first.
"""
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
vllm_url=DEAD_URL) as test_client:
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
now = time.time()
metrics = ["decode_tokens_per_second", "prompt_tokens_per_second",
"kv_cache_usage", "requests_running", "requests_waiting",
"ttft_seconds", "tpot_seconds", "queue_seconds",
"prefix_cache_hit_ratio", "spec_accept_rate"]
for metric in metrics:
store.add("llm", "gufo", f"llm.{metric}", 1.5, ts=now - 60)
store.add("llm", "vllm", f"llm.{metric}", 2.5, ts=now - 30)
body = test_client.get("/api/llm/history", params={
"engines": "gufo,vllm", "metrics": ",".join(metrics), "range": "1h"}).json()
assert body["range_seconds"] == 3_600.0
by_key = {(item["host"], item["metric"]) for item in body["series"]}
assert by_key == {(engine, f"llm.{metric}")
for engine in ("gufo", "vllm") for metric in metrics}
points = {(item["host"], item["metric"]): item["points"] for item in body["series"]}
assert points[("gufo", "llm.spec_accept_rate")][0]["value"] == pytest.approx(1.5)
assert points[("vllm", "llm.tpot_seconds")][0]["value"] == pytest.approx(2.5)
def test_metrics_endpoint_lists_what_actually_has_history(client) -> None:
body = client.get("/api/metrics").json()
assert body["metrics"] == []
# ---- the `/disks` API (phase 09 task 03) ---------------------------------- #
#
# The drive-health surface answers like every other dashboard's: GETs only, the same
# stale/stub semantics for a host that is not answering, the same history mechanism for the
# `disk.*@dev` series, and the same 404-`configured: false` sentence when no agent is
# configured at all — the last one *without dialling*, because "not deployed here" must not
# cost a connect timeout or a red dot (invariant 3).
@pytest.fixture
def disks_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
disk_snapshot: Any) -> Iterator[TestClient]:
"""Two hosts configured; one of them answering SMART, through the production parser.
The payload and the stored samples are what `parse_disks_snapshot` makes of the
fixture-shaped capture (the agent's whole-disk rows plus the recorded `smartctl` shapes),
so every path a `/disks` card will read is a path the source really emits. The other host
keeps its `unreachable` stub, which is the shape a dead agent must keep (invariant 4).
Both targets point at a dead port, so nothing dials.
"""
db = str(tmp_path / "disks.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2,
host_targets=(Target("driveripper", DEAD_URL),
Target("deskwork", DEAD_URL))), store)
with TestClient(app) as test_client:
sign_in(test_client)
now = time.time()
snapshot = disk_snapshot("driveripper")
snapshot["ts"] = now # "the agent answered just now", so charts are in range
result = parse_disks_snapshot("driveripper", snapshot, DisksCollector(), now=now)
store.add_many(result.samples)
app.state.ctx.hub.jobs["disks:driveripper"].state.payload = result.payload
yield test_client
store.close()
#: (`smart` field, severity metric) for the two drive numbers the table rates — the same
#: pair `sources/disks.py:SEVERITY_FIELDS` declares, restated here so a rename that only
#: touches one side fails rather than colouring nothing.
RATED_CELLS: tuple[tuple[str, str], ...] = (
("temp_c", "disk.temp_c"),
("percentage_used", "disk.percentage_used"),
)
def disk_rows(body: dict[str, Any], host: str = "driveripper") -> list[dict[str, Any]]:
"""The disk rows of one host out of a `/api/disks/snapshot` body."""
return next(row for row in body["hosts"] if row["name"] == host)["disks"]
def flat_routes(app: Any) -> list[Any]:
"""Every route the app serves, with `include_router()` results flattened to their routes.
The FastAPI this repo pins keeps an included router as an `_IncludedRouter` wrapper with
no `path` of its own, so walking `app.routes` alone sees five anonymous objects instead of
the API surface — and the verb assertion below is only as good as this walk.
"""
out: list[Any] = []
for route in app.routes:
inner = getattr(route, "original_router", None)
out.extend(inner.routes if inner is not None else [route])
return out
def series_source(body: dict[str, Any]) -> set[str]:
"""Which sample family a history answer came from: `disks`, never the 10-second `host`."""
return {series["source"] for series in body["series"]}
def test_disks_snapshot_answers_one_entry_per_host_with_its_physical_disks(
disks_client) -> None:
"""`GET /api/disks/snapshot` is per host: rows + `smart` + `smartctl_available`."""
body = disks_client.get("/api/disks/snapshot")
assert response_ok(body)
payload = body.json()
assert [host["name"] for host in payload["hosts"]] == ["deskwork", "driveripper"], (
"every configured host is a row, in name order, answering or not")
rows = disk_rows(payload)
assert rows, "the capture has 16 whole-disk devices, so the page has 16 cards"
assert all("/" not in row["device"] for row in rows), "a mount leaked into the disk list"
first = rows[0]
assert first["path"] == f"/dev/{first['device']}"
assert set(first) >= {"device", "path", "model", "size_bytes", "rotational",
"health", "smart", "severity"}
assert first["smart"]["power_on_hours"] > 0, "the drive's own counters travel verbatim"
live = next(host for host in payload["hosts"] if host["name"] == "driveripper")
assert live["smartctl_available"] is True
assert live["smart_errors"] == []
assert live["summary"]["disks"] == len(rows)
assert "unreachable" not in live, "answering is the absence of the flag, as on /system"
assert [state["key"] for state in payload["states"]] == ["disks:driveripper",
"disks:deskwork"], (
"the page's dots come from the disks jobs, never the system ones")
def test_disks_snapshot_carries_the_verdicts_the_page_will_paint(disks_client) -> None:
"""Colour is in the payload: each row's `severity` is what `severity.classify` says.
This is the pin that keeps the threshold out of the page — `page-disks.js` reads these
and never compares a drive temperature to a bound.
"""
rows = disk_rows(disks_client.get("/api/disks/snapshot").json())
assert "ok" in {row["severity"]["temp_c"] for row in rows}, "the capture is healthy"
for row in rows:
smart = row["smart"] or {}
assert set(row["severity"]) == {field for field, _metric in RATED_CELLS} | {"health"}
for field, metric in RATED_CELLS:
assert row["severity"][field] == classify(metric, smart.get(field)), (
f"{row['device']}.{field} is coloured by something other than the one table")
assert not {key for row in rows for key in row["severity"]} & {
"reallocated_sector_ct", "pending", "uncorrectable", "power_on_hours"}, (
"a counter is an alarm chip on presence, never a graded cell")
def test_a_configured_host_with_no_payload_yet_is_a_row_of_nas(disks_client) -> None:
"""The stub shape `/api/system/snapshot` hands out, so `n/a` is never a missing row.
A host in `HOST_TARGETS` that has not answered is still in the fleet (invariants 3 and
4) with its counts `None` rather than `0`: "unknown" and "no drives" are different
sentences (rule 2), and the fleet header counts it without summing it.
"""
payload = disks_client.get("/api/disks/snapshot").json()
stub = next(host for host in payload["hosts"] if host["name"] == "deskwork")
assert stub["unreachable"] is True
assert stub["disks"] == [], "nothing to draw, but the host is still listed"
assert stub["smartctl_available"] is None, "an agent that never answered says nothing"
assert stub["smart_errors"] == []
assert set(stub["summary"]) == {"disks", "with_smart", "health_failed", "alarm_disks"}
assert all(value is None for value in stub["summary"].values()), "no invented zeros"
fleet = payload["fleet"]
assert (fleet["hosts_total"], fleet["hosts_live"]) == (2, 1)
assert fleet["disks"] == len(disk_rows(payload))
def test_the_disks_fleet_header_is_summed_by_the_api_not_the_page(disks_client) -> None:
"""`fleet` is the header's only arithmetic source, and it carries no verdict."""
payload = disks_client.get("/api/disks/snapshot").json()
live = next(host for host in payload["hosts"] if host["name"] == "driveripper")
fleet = payload["fleet"]
assert fleet["disks"] == live["summary"]["disks"]
assert fleet["alarm_disks"] == live["summary"]["alarm_disks"] > 0, (
"the recorded ATA shape carries reallocated sectors, so the fleet has chips to draw")
assert "severity" not in fleet, "a count of drives is not a graded number"
def test_the_disks_surface_is_gets_only_and_the_write_stays_the_ups_one(
disks_client) -> None:
"""Invariant 1: the app writes to an **upstream** from exactly one route."""
routes = flat_routes(disks_client.app)
disks = [route for route in routes
if str(getattr(route, "path", "")).startswith("/api/disks")]
assert {route.path for route in disks} == {"/api/disks/snapshot", "/api/disks/history"}
verbs = {verb for route in disks for verb in (getattr(route, "methods", None) or set())}
assert verbs == {"GET"}, "a write appeared on the drive-health surface"
for verb in ("POST", "PUT", "PATCH", "DELETE"):
response = disks_client.request(verb, "/api/disks/snapshot", json={})
assert response.status_code == 405, f"{verb} /api/disks/snapshot was accepted"
writes = sorted(str(route.path) for route in routes
if "POST" in (getattr(route, "methods", None) or set()))
# Invariant 1 is about the fleet, and phase 11 did not touch the fleet: the three `/auth/`
# writes create, re-credential, and clear a session in this process and dial nothing. They
# are named here instead of filtered out of sight, because the failure this assertion catches
# is a POST arriving quietly — a list the reader recognises is worth more than a short one.
upstream = [path for path in writes if not path.startswith("/api/auth/")]
session = [path for path in writes if path.startswith("/api/auth/")]
assert upstream == ["/api/ups/command"], "one write endpoint, four gates, unchanged"
assert session == ["/api/auth/login", "/api/auth/logout", "/api/auth/password"]
# …and nothing anywhere grew the verbs this app has no use for, session included.
assert not sorted(path for route in routes
for path in [str(route.path)]
if set(getattr(route, "methods", None) or ()) & {"PUT", "PATCH", "DELETE"})
# The session writes are not upstream writes, but they are still the only ones: `/login`
# is the page, not a POST target, and `/api/auth/status` stays a read.
pages = sorted(str(route.path) for route in routes
if "GET" in (getattr(route, "methods", None) or set())
and not str(route.path).startswith("/api/"))
# `/settings` (phase 11) is a fifth dashboard-shaped **page** and it grew no verb: a page
# route is a GET, so the write surface this test guards stays the one UPS command.
assert pages == ["/", "/disks", "/llm", "/login", "/settings", "/system", "/ups"]
def test_config_prints_the_disks_cadence_the_page_explains_itself_with(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A card four minutes old is behaving, not broken — and the interval is the proof.
`/disks` has no upstream of its own, so `enabled` follows `HOST_TARGETS` and the cadence
is its own. Both numbers come from config rather than a constant in the page, so changing
`DISKS_INTERVAL_SECONDS` moves the sentence and the poller together.
"""
with build_client(tmp_path, monkeypatch, host_targets=(Target("driveripper", DEAD_URL),),
disks_interval=120.0, disks_timeout=45.0) as test_client:
disks = test_client.get("/api/config").json()["disks"]
assert disks["enabled"] is True
assert disks["interval_seconds"] == 120.0
assert disks["timeout_seconds"] == 45.0
def test_disks_snapshot_404s_without_dialling_a_host_that_was_never_configured(
client, monkeypatch: pytest.MonkeyPatch) -> None:
"""The gate runs before anything that could reach the network (invariant 3).
Both spies raise, so a route that constructed a client or fetched a snapshot before
checking `HOST_TARGETS` fails here instead of costing a timeout and a red dot. The hub
assertion is the structural half: with no targets there is no `disks` job to dial with.
"""
calls: list[str] = []
def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
calls.append(f"GET {url}")
raise AssertionError(f"the unconfigured disks route dialled {url}")
def spy_fetch(*args: Any, **kwargs: Any) -> Any:
calls.append("fetch_snapshot")
raise AssertionError("the unconfigured disks route fetched an agent snapshot")
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
monkeypatch.setattr("dashboards.collectors.fetch_snapshot", spy_fetch)
response = client.get("/api/disks/snapshot")
assert response.status_code == 404
body = response.json()
assert body["configured"] is False
assert "HOST_TARGETS" in body["detail"], "the sentence names the fix"
assert calls == [], "an unconfigured page must not touch the network"
assert client.app.state.ctx.hub.jobs == {}, (
"no targets means no disks job, so nothing can age out or go red")
def test_disks_history_serves_the_dev_series_the_page_charts(disks_client) -> None:
"""One request, every series a host's cards could draw, points from the stored samples."""
rows = disk_rows(disks_client.get("/api/disks/snapshot").json())
metrics = [f"{base}@{row['device']}" for row in rows for _field, base in SMART_SERIES]
body = disks_client.get("/api/disks/history", params={
"hosts": "driveripper", "metrics": ",".join(metrics), "range": "1h"}).json()
assert body["hosts"] == ["driveripper"], "one host, one panel"
assert body["range_seconds"] == 3_600.0
assert [series["metric"] for series in body["series"]] == metrics
assert series_source(body) == {"disks"}, "the disks family, never the 10-second host one"
points = {(series["metric"]): series["points"] for series in body["series"]}
assert points["disk.temp_c@nvme0n1"][0]["value"] == pytest.approx(38.0)
assert points["disk.power_on_hours@nvme0n1"][0]["value"] == pytest.approx(8_760.0)
def test_disks_history_answers_empty_for_a_series_a_drive_does_not_publish(
disks_client) -> None:
"""Unknown names are an empty series, not a 404 — the /system and /llm contract.
The recorded ATA shape reports no `percentage_used`, so that drive's wear plot has
nothing to draw, and a device name the agent never listed behaves the same way. The card
asks for every series it could draw and reads `n/a` where there is nothing, instead of
the request failing.
"""
rows = disk_rows(disks_client.get("/api/disks/snapshot").json())
unworn = next(row["device"] for row in rows
if (row["smart"] or {}).get("percentage_used") is None)
body = disks_client.get("/api/disks/history", params={
"hosts": "driveripper",
"metrics": f"disk.percentage_used@{unworn},disk.temp_c@nosuchdisk,"
"disk.temp_c@nvme0n1",
"range": "1h"}).json()
by_metric = {series["metric"]: series["points"] for series in body["series"]}
assert by_metric[f"disk.percentage_used@{unworn}"] == [], "no measurement, no line"
assert by_metric["disk.temp_c@nosuchdisk"] == []
assert by_metric["disk.temp_c@nvme0n1"], "the series that does exist still answers"
def test_disks_history_charts_every_configured_host_in_one_request(disks_client) -> None:
"""With no `hosts` the answer covers every configured agent, dead ones included (empty)."""
body = disks_client.get("/api/disks/history",
params={"metrics": "disk.temp_c@nvme0n1,disk.temp_c@sda",
"range": "24h"}).json()
assert body["hosts"] == ["driveripper", "deskwork"]
assert body["range_seconds"] == 86_400.0
assert {(series["host"], series["metric"]) for series in body["series"]} == {
("driveripper", "disk.temp_c@nvme0n1"), ("driveripper", "disk.temp_c@sda"),
("deskwork", "disk.temp_c@nvme0n1"), ("deskwork", "disk.temp_c@sda")}
assert [series for series in body["series"] if series["host"] == "deskwork"], (
"a host with no history is an empty series, never a missing one")
def test_the_metrics_endpoint_answers_for_the_disks_family(disks_client) -> None:
""""Why is this chart empty" has an answer per page: /api/metrics?source=disks."""
body = disks_client.get("/api/metrics", params={"source": "disks"}).json()
assert "disk.temp_c@nvme0n1" in body["metrics"]
assert not [name for name in body["metrics"] if not name.startswith("disk.")], (
"the disks family holds only disk series")
assert not [name for name in disks_client.get(
"/api/metrics", params={"source": "host"}).json()["metrics"]
if name.startswith("disk.temp_c")], "SMART never mixes into the system family"
# ---- the `/disks` page (phase 09 task 04) ---------------------------------- #
#
# The fourth dashboard, asserted the way the other three are: on the served bytes, because a
# JS test runner would be a build step (a locked anchor). What is pinned is not that the page
# exists but that it keeps the four promises the phase wrote down — one card per **physical**
# drive and never one per partition, alarm chips that exist only while a counter is non-zero,
# a host without `smartctl` dimmed rather than reddened, and charts that belong to one host,
# stay collapsed until opened, and cost one history request per host per refresh.
def _disks_js(client: TestClient) -> str:
return client.get("/static/page-disks.js").text
def test_the_disks_tab_is_in_the_masthead_of_every_page(client) -> None:
"""Four tabs became five, on every page, with exactly one marking itself current.
The masthead is the app's only "which dashboard do I need" surface since phase 02 deleted
the overview's table, so a tab missing from one page is a page you cannot get to, and a
tab marked current on the wrong page is a lie about where you are.
"""
for path in PAGES:
html = client.get(path).text
assert '<a href="/disks"' in html, f"{path}'s nav has no Disks tab"
assert html.count('aria-current="page"') == 1, f"{path} marks more than one tab current"
assert f'<a href="{path}" aria-current="page">' in html, f"{path} does not mark itself"
assert (html.index('href="/system"') < html.index('href="/disks"')
< html.index('href="/llm"')), "tab order is machine, drive, engine, power"
def test_the_header_status_dot_picks_up_the_disks_job(client) -> None:
"""`disks` gets its own dot instead of hiding inside `hosts`.
The same agent can answer the 10-second utilisation poll and have a wedged SMART pass;
one dot for both would have to be right about neither.
"""
app_js = client.get("/static/app.js").text
assert "{ key: 'disks', label: 'disks', href: '/disks' }" in app_js
assert "s.dashboard === job.key" in app_js, "the dot stopped filtering by dashboard"
def test_disks_page_draws_one_card_per_physical_drive_from_the_api(client) -> None:
"""The card is a drive, keyed by the agent's device name, patched in place.
A mount or a partition would need a filesystem field, and `/system` already answers "is
the filesystem full"; if one leaks back in here, the page stops being about the drive.
"""
html = client.get("/disks").text
js = _disks_js(client)
assert 'id="hosts"' in html and 'id="hosts-empty"' in html
assert "fetchJSON('/api/disks/snapshot')" in js
assert "row.path" in js and "/dev/" in js, "the card stopped naming the device it is about"
# identity + verdict lead the card: no hero row, no big single number
assert 'class="heroes"' not in html, "the disk card grew a hero row"
assert "Dash.hero(" not in js
for filesystem in ("mountpoint", "fstype", "free_bytes", "worst_mount"):
assert filesystem not in js, f"{filesystem} leaked into the drive page"
# built once per device and patched, so a 16-drive host does not rebuild 16 panels a poll
assert "seen.get(row.device)" in js and "renderDiskCard(panel, row)" in js
assert "pruneHosts(" in js, "a drive that left the host keeps its card forever"
def test_disk_cards_print_the_drive_numbers_and_colour_only_rated_ones(client) -> None:
"""Temp, power-on hours and wear, coloured by the API's verdicts and nothing else.
The odometer is the deliberate exception: `severity.THRESHOLDS` has no `disk.power_on_hours`
row, so the cell is written with a `null` verdict and cannot go green at 100 h or red at
60 000 h. A drive that publishes no wear (the recorded HDD shape) prints `n/a` because
`setTxt` is the only writer.
"""
js = _disks_js(client)
for field in ("smart.temp_c", "smart.power_on_hours", "smart.percentage_used",
"row.health"):
assert field in js, f"{field} is no longer on the card"
for verdict in ("severity.temp_c", "severity.percentage_used", "severity.health"):
assert verdict in js, f"the card stopped rendering {verdict}"
assert "severityFor" not in js, "the drive page wrote a threshold of its own"
body = js[js.index("function renderDiskCard("):js.index("/** Text that prints nothing")]
assert "severity.temp_c" in body and "severity.percentage_used" in body
assert re.search(r"setMetric\(card, 'hours',[^;]*null\)", body), (
"the odometer is being graded by something severity.py does not have")
# the drive's own word, and `unknown` for a drive that gave none — never a green pass
assert "'FAILED'" in body and "'unknown'" in body
def test_disk_alarm_chips_exist_only_while_a_counter_is_non_zero(client) -> None:
"""Zero and unknown both draw no chip; only a non-zero counter draws a red one.
The `/llm` preemption rule: a permanent "0 pending sectors" row trains the eye to ignore
it, and grading a counter green would call a quiet drive an achievement. `severity.py`
rates none of these, so the chip's presence is the entire signal here.
"""
js = _disks_js(client)
for counter in ("reallocated_sector_ct", "current_pending_sector_ct",
"offline_uncorrectable"):
assert counter in js, f"{counter} is no longer an alarm"
assert "value <= 0) continue" in js, "a zero now draws a chip"
assert "alarms.hidden = chips.length === 0" in js, "the empty chip row stays on screen"
assert 'class="chip fail"' in js
css = client.get("/static/dash.css").text
assert ".alarms {" in css and ".chip.fail" in css
def test_a_host_without_smartctl_is_dimmed_with_a_one_line_why_and_never_red(client) -> None:
"""Unconfigured is not broken (invariant 3), and the three answers stay three.
`smartctl_available` is `false` (install smartmontools — the agent said so, so the page
may say it), `null` (nobody told us either way, so name the fix without claiming to know
which half is missing), or the agent is not answering at all, which is a different
sentence again. None of the three reddens anything.
"""
js = _disks_js(client)
html = client.get("/disks").text
assert "smartctl not available on" in js
assert "disk group" in js, "the why stopped naming the grant that fixes it"
assert "smartctl_available === false" in js and "smartctl_available === true" in js
assert "host.unreachable" in js, "a dead agent must not be reported as a missing binary"
assert "smart_errors" in js, "the agent's own sentence is no longer printed"
assert "classList.toggle('stale'" in js
assert 'class="banner info why" hidden' in js, "the why-line has no home on the card"
assert "banner fail" not in js and html.count("banner") == 0, (
"a missing binary reddens the page")
def test_disks_charts_are_collapsed_lazy_and_cost_one_history_request_per_host(client) -> None:
"""`/system`'s chart discipline, reused: collapsed, painted only while open, one XHR.
The request still names every series the host could draw (temperature and wear per
drive), because `/api/disks/history` answers an unknown name as an empty series and a
panel should never have to discover series it did not ask for.
"""
js = _disks_js(client)
assert '<details class="charts-box"' in js
guard = js[js.index("function drawHostCharts("):js.index("async function renderHostCharts(")]
assert "if (!chartsOpen(card)) return;" in guard, (
"a collapsed stack is being built and drawn every poll")
history = js[js.index("async function renderHostCharts("):js.index("/**\n * Drop cards")]
assert history.count("fetchJSON(") == 1, "more than one history request for one host"
assert "/api/disks/history" in history
assert "streams.temp.concat(streams.wear)" in history, (
"the request no longer names every series the panel could draw")
prune = js[js.index("function pruneHosts("):js.index("// ---- fleet header")]
for entry in ("state.hosts.delete", "state.disks.delete", "state.charts.delete",
"state.history.delete", "state.payloads.delete"):
assert entry in prune, f"a pruned host keeps its {entry}"
def test_every_disks_chart_names_itself_for_assistive_tech(client) -> None:
"""An unlabelled plot is silence, and the legend is a sibling that labels nothing."""
js = _code(_disks_js(client))
labels = re.findall(r"ariaLabel: `([^`]+)`", js)
assert len(labels) == 2, "every drive chart must name itself (temperature, wear)"
assert all(label.startswith("${host.name}") for label in labels), (
"a chart that does not say which host it plots belongs on no page")
def test_the_disks_fleet_header_reads_the_api_sums_and_invents_none(client) -> None:
"""The header's arithmetic is the API's (`fleet_disk_summary`), never the page's.
Hidden below two hosts, because with one box in `HOST_TARGETS` these sums only duplicate
that host's own cards — `/llm`'s fleet-block rule. And no cell is coloured: a count of
drives is not a graded number (`severity.py` has no row for one).
"""
html = client.get("/disks").text
js = _disks_js(client)
for element in ("fleet", "fleet-dot", "fleet-count", "fleet-disks", "fleet-smart",
"fleet-failed", "fleet-alarms"):
assert f'id="{element}"' in html, f"the header lost #{element}"
fleet = js[js.index("function renderFleet("):js.index("// ---- snapshot")]
for path in ("fleet.hosts_live", "fleet.disks", "fleet.with_smart",
"fleet.health_failed", "fleet.alarm_disks"):
assert path in fleet, f"the header no longer reads {path}"
assert "reduce(" not in fleet and "+=" not in fleet, "the page is summing drives itself"
assert "severity" not in fleet, "a drive count grew a verdict"
assert "block.hidden = true" in fleet, "one host's sums duplicate its own cards"
def test_the_disk_panel_style_is_existing_vocabulary_with_no_new_palette(client) -> None:
"""`.disk` is the `.slot` shape, and the only colours it reaches for are the verdicts."""
css = _css(client)
block = _css_section(css, "drive panels")
for selector in (".disks {", ".disk {", ".disk-head", ".disk-dev", ".disk-model",
".disk .metrics", ".disk .alarms", ".disk .metric.warn .value",
".disk .metric.fail .value"):
assert selector in block, f"dash.css lost {selector}"
for token in ("var(--warn)", "var(--fail)", "var(--border)", "var(--bg-sunken)",
"var(--text)"):
assert token in block, f"the drive panel no longer uses {token}"
assert "--disk" not in css, "a drive-only colour token appeared"
assert ".stale" in css, "the dim that marks an unreadable host is gone"
def test_the_overview_prints_the_fourth_source_cadence(client) -> None:
"""`/` names every source's interval, so a drive card four minutes old reads as behaving."""
js = client.get("/static/page-index.js").text
assert "config.disks.interval_seconds" in js
@pytest.mark.parametrize("fixture", ["client", "data_client"])
def test_summary_carries_the_drive_block_for_programmatic_consumers(
request, fixture: str) -> None:
"""`disks` is additive: the same configured-ness, the same hosts, no key retired.
`client` has no targets, so it proves the block answers as `False`/`None`/`[]` instead of
disappearing; `data_client` proves it lists the host the fleet has.
"""
test_client = request.getfixturevalue(fixture)
body = test_client.get("/api/summary").json()
populated = fixture == "data_client"
assert body["disks"]["enabled"] is populated
assert (body["disks"]["fleet"] is not None) is populated
assert body["disks"]["hosts"] == ([] if not populated else [
{"name": "driveripper", "disks": None, "with_smart": None, "health_failed": None,
"alarm_disks": None, "smartctl_available": None, "failed_devices": [],
"unreachable": True}]), "a configured host with no payload is a row of nones"
assert [row["name"] for row in body["disks"]["hosts"]] == [
row["name"] for row in body["system"]["hosts"]]
def test_summary_serves_the_same_counts_the_disks_page_prints(disks_client) -> None:
"""One projection, two consumers: the summary's numbers *are* the snapshot's."""
body = disks_client.get("/api/summary").json()
snapshot = disks_client.get("/api/disks/snapshot").json()
disks = body["disks"]
assert disks["fleet"] == snapshot["fleet"], "the header would disagree with the page"
rows = {row["name"]: row for row in disks["hosts"]}
live = next(host for host in snapshot["hosts"] if host["name"] == "driveripper")
assert (rows["driveripper"]["disks"], rows["driveripper"]["with_smart"],
rows["driveripper"]["health_failed"], rows["driveripper"]["alarm_disks"]) == (
live["summary"]["disks"], live["summary"]["with_smart"],
live["summary"]["health_failed"], live["summary"]["alarm_disks"])
assert rows["driveripper"]["smartctl_available"] is True
assert rows["driveripper"]["unreachable"] is False
assert rows["deskwork"]["unreachable"] is True
assert rows["deskwork"]["disks"] is None, "unknown, never 0 (rule 2)"
assert sorted(state["key"] for state in disks["states"]) == ["disks:deskwork",
"disks:driveripper"]
def test_summary_names_the_drive_the_health_verdict_called_failed(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, disk_snapshot: Any) -> None:
"""Which drive the hardware itself called `FAILED` is part of the fleet answer.
A notifier reading `/api/summary` should not have to fetch a second endpoint to name the
device, and the count and the list have to agree or one of them is wrong.
"""
db = str(tmp_path / "summary-failed.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2,
host_targets=(Target("driveripper", DEAD_URL),)), store)
with TestClient(app) as test_client:
sign_in(test_client)
now = time.time()
snapshot = disk_snapshot("driveripper", mode="failed")
snapshot["ts"] = now
result = parse_disks_snapshot("driveripper", snapshot, DisksCollector(), now=now)
app.state.ctx.hub.jobs["disks:driveripper"].state.payload = result.payload
row = test_client.get("/api/summary").json()["disks"]["hosts"][0]
assert len(row["failed_devices"]) == 1
assert row["failed_devices"][0] == result.payload["disks"][0]["device"]
assert row["health_failed"] == result.payload["summary"]["health_failed"] == 1
store.close()
# ---- events --------------------------------------------------------------- #
def test_events_are_returned_in_json(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
db = str(tmp_path / "events.sqlite3")
store = SampleStore(db)
store.log_event(Event(ts=time.time(), source="host", host="deskwork", kind="down",
severity="error", message="deskwork is not answering"))
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2), store)
with TestClient(app) as client:
sign_in(client)
events = client.get("/api/events").json()["events"]
assert events[0]["host"] == "deskwork"
assert events[0]["severity"] == "error"
store.close()
def test_event_rows_say_their_subject_once(client) -> None:
"""`poller.py` messages name their own subject, and the row prints a host chip too.
"gufo gufo is answering again after 1 failure" reads as two problems — the same defect
`alerts._clean_error()` already fixes for the overview's alert list. `/ups` had the
mirror-image bug: it `replace()`d `host + ':'` *anywhere*, so "Quick battery test sent
to apc: OK" printed as "Quick battery test sent to OK". All three feeds now share
`Dash.eventText()`, which strips the wrapper and a **leading** name only.
"""
app_js = client.get("/static/app.js").text
helper = app_js[app_js.index("function eventText("):app_js.index("// ---- DOM helpers")]
assert "eventText" in app_js[app_js.index("return {"):], "eventText is not exported"
assert "^UpstreamError:\\s+" in helper, "the wrapper `alerts.py` drops is back"
assert "new RegExp" in helper and "stripped.trim() ? stripped : out" in helper, \
"the host strip is no longer anchored to the front, or can empty a message"
for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"):
js = client.get(page).text
assert "eventText(event)" in js, f"{page} prints the raw message again"
assert "replace(event.host" not in client.get("/static/page-ups.js").text, \
"/ups is mangling the host name mid-message again"
# ---- the one write endpoint ------------------------------------------------ #
def test_command_endpoint_is_post_only(ups_client) -> None:
assert ups_client.post("/api/ups/command", json={}).status_code in (400, 403, 404)
assert ups_client.put("/api/ups/command", json={}).status_code == 405
assert ups_client.delete("/api/ups/command").status_code == 405
def test_commands_are_offered_when_actions_are_enabled(ups_client) -> None:
body = wait_for_payload(ups_client, "/api/ups/snapshot")
assert body["simulated"] is True, "the simulator must label itself"
assert body["actions_enabled"] is True
ups = body["ups"][0]
enabled = [item for item in ups["commands"] if item["enabled"]]
assert enabled, "with NUT_ALLOW_ACTIONS=1 and a simulator, safe commands are sendable"
assert all(item["risk"] in ("safe", "caution", "dangerous") for item in ups["commands"])
def test_a_safe_command_actually_reaches_the_driver(ups_client) -> None:
body = wait_for_payload(ups_client, "/api/ups/snapshot")
name = body["ups"][0]["name"]
response = ups_client.post("/api/ups/command",
json={"ups": name, "command": "test.panel.start"})
assert response.status_code == 200, response.text
payload = response.json()
assert payload["ok"] is True and payload["simulated"] is True
events = ups_client.get("/api/events").json()["events"]
assert any("test.panel.start" in event["message"] or "Panel test" in event["message"]
for event in events), "every sent command must leave an audit trail"
def test_a_dangerous_command_needs_the_confirmation_word(ups_client) -> None:
name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
deep = "test.battery.start.deep"
refusal = ups_client.post("/api/ups/command", json={"ups": name, "command": deep})
assert refusal.status_code == 422
assert "DANGER" in refusal.json()["detail"]
assert ups_client.post("/api/ups/command",
json={"ups": name, "command": deep, "confirm": "DANGER"}).status_code == 200
def test_a_command_the_driver_does_not_advertise_is_refused_before_it_is_sent(
ups_client) -> None:
"""Being in the risk catalog is not permission; `LIST CMD` from the driver is (invariant 1).
`shutdown.return` is a real NUT command the catalog knows and this UPS does not offer,
so the refusal must come from the advertisement check and not from the confirmation gate.
"""
name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
response = ups_client.post("/api/ups/command",
json={"ups": name, "command": "shutdown.return"})
assert response.status_code == 400
assert "does not advertise" in response.json()["detail"]
def test_cutting_the_output_is_refused_while_the_ups_is_already_on_battery(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Invariant 1, the one that matters most: no power cut while the pack is carrying the load.
Driven through the simulator's own state machine — a quick battery test puts the pack on
battery — so the 409 comes from the same poll the buttons are drawn from, with a 0.5s
interval to keep the wait short. Nothing here can touch real hardware: `NUT_SIMULATE=1`.
"""
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
nut_allow_actions=True, nut_interval=0.5) as test_client:
name = wait_for_payload(test_client, "/api/ups/snapshot")["ups"][0]["name"]
start = test_client.post("/api/ups/command",
json={"ups": name, "command": "test.battery.start.quick"})
assert start.status_code == 200
deadline = time.time() + 6.0
card: dict[str, Any] = {}
while time.time() < deadline:
card = next(item for item in test_client.get("/api/ups/snapshot").json()["ups"]
if item["name"] == name)
if card["on_battery"]:
break
time.sleep(0.05)
assert card.get("on_battery") is True, "the quick test must put the simulated pack on battery"
refusal = test_client.post("/api/ups/command",
json={"ups": name, "command": "load.off",
"confirm": "DANGER"})
assert refusal.status_code == 409
assert "on battery" in refusal.json()["detail"]
# and the same command is still offered once the pack is back on line power
assert not any(item["command"] == "load.off" and item["enabled"]
for item in card["commands"]), "the panel must not offer a blocked button"
class ScriptedNutClient:
"""`nut.NutClient` with the socket left out: the real code path, no battery attached.
`AGENTS.md` keeps live tests read-only, so this is where the one write endpoint's real
branch gets proven. The variable names are the ones `VAR_MAP` already pins from real
upsd captures; only the replies are scripted (a fake for behaviour, per the test policy).
"""
#: set per test: an exception instance to raise from `instcmd`, e.g. a refusal
refusal: Exception | None = None
sent: list[tuple[str, str]]
logged_in: list[str]
def __init__(self, host: str, port: int, *, timeout: float = 4.0,
username: str | None = None, password: str | None = None) -> None:
self.credentials = (username, password)
async def __aenter__(self) -> ScriptedNutClient:
return self
async def __aexit__(self, *_exc: object) -> None:
return None
async def list_ups(self) -> list[dict[str, str]]:
return [{"name": "apc", "description": "APC Back-UPS RS 1500MS"}]
async def list_vars(self, ups: str) -> dict[str, str]:
return {"ups.mfr": "APC", "ups.model": "Back-UPS RS 1500MS",
"ups.status": "OL", "battery.charge": "100", "ups.load": "18",
"battery.voltage": "13.6", "input.voltage": "230",
"battery.mfr.date": "2022-04", "ups.test.result": "Ok last test passed"}
async def list_cmds(self, ups: str) -> list[str]:
return ["test.panel.start", "beeper.mute", "load.off"]
async def list_rw_vars(self, ups: str) -> list[str]:
return []
async def login(self) -> None:
assert self.credentials[0], "a command send must authenticate as a named user"
ScriptedNutClient.logged_in.append(str(self.credentials[0]))
async def instcmd(self, ups: str, command: str) -> str:
if ScriptedNutClient.refusal is not None:
raise ScriptedNutClient.refusal
ScriptedNutClient.sent.append((ups, command))
return "OK Start Panel Test"
@pytest.fixture
def real_upsd_client(monkeypatch: pytest.MonkeyPatch) -> Iterator[type[ScriptedNutClient]]:
"""Swap the module-level `NutClient` so the collector and the send use the same stand-in."""
ScriptedNutClient.sent = []
ScriptedNutClient.logged_in = []
ScriptedNutClient.refusal = None
monkeypatch.setattr(nut, "NutClient", ScriptedNutClient)
yield ScriptedNutClient
@pytest.fixture
def nut_client_app(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
real_upsd_client: type[ScriptedNutClient]) -> Iterator[TestClient]:
"""A non-simulated UPS app: real code paths, scripted upsd, 0.5s polls."""
with build_client(tmp_path, monkeypatch, nut_host="127.0.0.1", nut_user="dashboards",
nut_password="not-the-real-one", nut_allow_actions=True,
nut_interval=0.5) as test_client:
yield test_client
def test_a_real_upsd_send_authenticates_sends_and_audits(nut_client_app,
real_upsd_client) -> None:
body = wait_for_payload(nut_client_app, "/api/ups/snapshot")
assert body["simulated"] is False, "this test is about the branch that talks to upsd"
name = body["ups"][0]["name"]
response = nut_client_app.post("/api/ups/command",
json={"ups": name, "command": "test.panel.start"})
assert response.status_code == 200, response.text
assert response.json()["simulated"] is False
assert real_upsd_client.sent == [(name, "test.panel.start")]
assert real_upsd_client.logged_in, "instcmd requires a LOGIN first (upsd refuses otherwise)"
events = nut_client_app.get("/api/events").json()["events"]
assert any(event["kind"] == "action" and name in event["message"] for event in events)
def test_a_refusal_from_upsd_is_a_502_and_an_error_event(nut_client_app,
real_upsd_client) -> None:
real_upsd_client.refusal = nut.NutError("Access denied", code="ACCESS-DENIED")
name = wait_for_payload(nut_client_app, "/api/ups/snapshot")["ups"][0]["name"]
response = nut_client_app.post("/api/ups/command",
json={"ups": name, "command": "beeper.mute"})
assert response.status_code == 502
detail = response.json()["detail"]
assert "ACCESS-DENIED" in detail or "Access denied" in detail, detail
assert real_upsd_client.sent == [], "a refused command was never accepted"
events = nut_client_app.get("/api/events").json()["events"]
failures = [event for event in events if event["kind"] == "action"
and event["severity"] == "error"]
assert failures, "an upsd refusal has to leave a trace, not just an HTTP status"
def test_unknown_commands_are_refused_even_with_actions_on(ups_client) -> None:
name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
for command in ("load.format", "rm -rf /", "INSTCMD x y"):
response = ups_client.post("/api/ups/command", json={"ups": name, "command": command})
assert response.status_code == 400, command
def test_commands_for_an_unknown_ups_are_404(ups_client) -> None:
wait_for_payload(ups_client, "/api/ups/snapshot")
response = ups_client.post("/api/ups/command",
json={"ups": "not-real", "command": "beeper.mute"})
assert response.status_code == 404
def test_missing_fields_are_a_400(ups_client) -> None:
assert ups_client.post("/api/ups/command", json={"ups": ""}).status_code == 400
def test_actions_flag_off_blocks_every_command(tmp_path: Path,
monkeypatch: pytest.MonkeyPatch) -> None:
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1") as client:
body = wait_for_payload(client, "/api/ups/snapshot")
name = body["ups"][0]["name"]
assert body["actions_enabled"] is False
response = client.post("/api/ups/command",
json={"ups": name, "command": "beeper.mute"})
assert response.status_code == 403
assert "NUT_ALLOW_ACTIONS" in response.json()["detail"]
def test_read_only_beats_the_actions_flag(tmp_path: Path,
monkeypatch: pytest.MonkeyPatch) -> None:
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
nut_allow_actions=True, read_only=True) as client:
body = wait_for_payload(client, "/api/ups/snapshot")
name = body["ups"][0]["name"]
response = client.post("/api/ups/command",
json={"ups": name, "command": "beeper.mute"})
assert response.status_code == 403
commands = wait_for_payload(client, "/api/ups/commands")
assert commands["read_only"] is True
assert all(item["enabled"] is False for ups in commands["ups"]
for item in ups["commands"])
def test_ups_history_answers_for_simulated_ups(ups_client) -> None:
wait_for_payload(ups_client, "/api/ups/snapshot")
time.sleep(1.2)
body = ups_client.get("/api/ups/history", params={"metrics": "ups.charge_pct",
"range": "1h"}).json()
assert body["ups"], "the UPS page needs a name to chart before any history exists"
def response_ok(response) -> bool:
assert response.status_code == 200, response.text
return True
def test_samples_written_by_the_poller_are_queryable(tmp_path: Path,
monkeypatch: pytest.MonkeyPatch) -> None:
"""The write path (poller -> store) and the read path (/api/history) must agree on names."""
db = str(tmp_path / "roundtrip.sqlite3")
store = SampleStore(db)
store.add_many([Sample("llm", "gufo", "llm.decode_tokens_per_second", 40.0, time.time() - 60),
Sample("llm", "gufo", "llm.decode_tokens_per_second", 45.0, time.time() - 30)])
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
pbkdf2_iterations=FAST_PBKDF2), store)
with TestClient(app) as client:
sign_in(client)
metrics = client.get("/api/metrics", params={"source": "llm"}).json()["metrics"]
assert "llm.decode_tokens_per_second" in metrics
points = client.get("/api/history", params={"source": "llm", "host": "gufo",
"metric": "llm.decode_tokens_per_second",
"range": "15m"}).json()["points"]
assert [p["value"] for p in points] == [40.0, 45.0]
store.close()