All green. Final verification summary:
**Verified / implemented (phase 13 final pass)**
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.
**Tests / lint (exact, all 100 % green)**
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.
**Completion criteria**
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
- Task-02 findings file names the confirmed mechanism — met.
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
- Full suite + validate green; no phase-07/04 behaviour change — met.
**Documentation**
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.
**Notable decisions / deviations**
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.
**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).
4082 lines
221 KiB
Python
4082 lines
221 KiB
Python
"""HTTP contract: what the pages may assume, and what must never leak.
|
|
|
|
The behaviours here are the ones a screenshot cannot show: an unconfigured source answering
|
|
404 `configured: false` instead of dialling a hostname that does not exist; a bearer token
|
|
that stops at the backend; the single write endpoint refusing on every gate; no external
|
|
asset the network can fail to deliver.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
import time
|
|
from collections.abc import Iterator
|
|
from contextlib import contextmanager
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import httpx
|
|
import pytest
|
|
from fastapi import HTTPException
|
|
from fastapi.testclient import TestClient
|
|
|
|
from auth_client import sign_in
|
|
from auth_contract import ADMIN_PASSWORD, FAST_PBKDF2
|
|
from dashboards import alerts, auth
|
|
from dashboards.api import MIN_PASSWORD_LENGTH, RANGE_SECONDS, parse_range
|
|
from dashboards.app import create_app
|
|
from dashboards.collectors import UpstreamError
|
|
from dashboards.config import Settings, Target
|
|
from dashboards.severity import classify
|
|
from dashboards.sources import nut
|
|
from dashboards.sources.disks import SMART_SERIES, DisksCollector, parse_disks_snapshot
|
|
from dashboards.sources.llm import (
|
|
HISTORY_METRICS,
|
|
parse_gufo_metrics,
|
|
parse_vllm_metrics,
|
|
)
|
|
from dashboards.store import Event, Sample, SampleStore
|
|
from slots_contract import (
|
|
CANONICAL_TASK_ID_FIELD,
|
|
OPTIONAL_NEXT_TOKEN_FIELDS,
|
|
OPTIONAL_ROW_FIELDS,
|
|
REQUIRED_NEXT_TOKEN_FIELDS,
|
|
REQUIRED_ROW_FIELDS,
|
|
TASK_ID_ALIASES,
|
|
USER_CONTENT_FIELDS,
|
|
load_slots_capture,
|
|
)
|
|
|
|
# refused immediately, and never dials out of the test environment
|
|
DEAD_URL = "http://127.0.0.1:1"
|
|
|
|
#: Every dashboard route, in nav order. The markup pins below (no CDN, no `.sev-*`, no
|
|
#: narration, no orphan `id`) run across this list rather than a hand-written four, so a
|
|
#: fifth page cannot join the masthead and skip the checks the others had to pass.
|
|
PAGES = ("/", "/system", "/disks", "/llm", "/ups")
|
|
|
|
#: `/settings` is a **page** and not a dashboard (phase 11): it carries the same masthead with
|
|
#: no `aria-current` tab, and the account link in every dashboard's masthead is how you reach
|
|
#: it. Kept out of `PAGES` because the pins over that tuple assert tab structure — one current
|
|
#: tab, a fixed tab order — which this page deliberately does not have. `MARKUP_PAGES` is the
|
|
#: list for the checks that are about the bytes rather than the nav.
|
|
SETTINGS_PAGE = "/settings"
|
|
MARKUP_PAGES = (*PAGES, SETTINGS_PAGE)
|
|
|
|
|
|
@contextmanager
|
|
def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: bool = False,
|
|
**overrides: Any) -> Iterator[TestClient]:
|
|
"""A TestClient whose lifespan runs, against a private DB and no real network.
|
|
|
|
The store is injected, so `create_app()` deliberately does not close it (production
|
|
owns its own); the caller here does, or every test leaks a SQLite handle and the
|
|
`--cov` log fills with ResourceWarnings that bury the warnings worth reading.
|
|
"""
|
|
if simulate:
|
|
monkeypatch.setenv("NUT_SIMULATE", "1")
|
|
else:
|
|
monkeypatch.delenv("NUT_SIMULATE", raising=False)
|
|
db = str(tmp_path / "api.sqlite3")
|
|
settings = Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2, **overrides)
|
|
store = SampleStore(db)
|
|
client = TestClient(create_app(settings, store))
|
|
client.__enter__()
|
|
try:
|
|
sign_in(client)
|
|
yield client
|
|
finally:
|
|
client.__exit__(None, None, None)
|
|
store.close()
|
|
|
|
|
|
@pytest.fixture
|
|
def client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[TestClient]:
|
|
with build_client(tmp_path, monkeypatch) as test_client:
|
|
yield test_client
|
|
|
|
|
|
@pytest.fixture
|
|
def ups_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[TestClient]:
|
|
"""NUT on the labelled simulator, actions allowed — the dev configuration."""
|
|
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
|
|
nut_allow_actions=True) as c:
|
|
yield c
|
|
|
|
|
|
def wait_for_payload(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
|
|
deadline = time.time() + timeout
|
|
while time.time() < deadline:
|
|
response = client.get(path)
|
|
if response.status_code == 200:
|
|
return response.json()
|
|
time.sleep(0.1)
|
|
pytest.fail(f"{path} never became available")
|
|
|
|
|
|
def failing_host_payload(scrape: Any, agent_snapshot: dict[str, Any]) -> dict[str, Any]:
|
|
"""Two scrapes of the real driveripper capture, nudged onto known sides of the table.
|
|
|
|
Driven from the fixture rather than hand-written so a schema drift breaks here first.
|
|
"""
|
|
def nudge(snap: dict[str, Any]) -> None:
|
|
snap["memory"]["used_bytes"] = snap["memory"]["total_bytes"] * 0.95 # mem.used_pct -> fail
|
|
snap["cpu_temperature_celsius"] = 91.0 # cpu.temp_c -> fail
|
|
for fs in snap["filesystems"]:
|
|
fs["used_pct"] = 93.0 # disk -> fail
|
|
|
|
return scrape(agent_snapshot, cpu_busy=10.0, mutate=nudge)
|
|
|
|
|
|
@pytest.fixture
|
|
def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
|
agent_snapshot: dict[str, Any], host_scrape: Any) -> Iterator[TestClient]:
|
|
"""Overview-shaped app: one host payload in the hub plus the simulated UPSes.
|
|
|
|
The host job points at a dead port, so a poll can never overwrite the injected payload
|
|
(a failed poll keeps the last good one) — the numbers below are ours, not the fleet's.
|
|
"""
|
|
monkeypatch.setenv("NUT_SIMULATE", "1")
|
|
db = str(tmp_path / "summary.sqlite3")
|
|
store = SampleStore(db)
|
|
app = create_app(Settings(db_path=db, nut_host="127.0.0.1",
|
|
admin_password=ADMIN_PASSWORD, pbkdf2_iterations=FAST_PBKDF2,
|
|
host_targets=(Target("driveripper", DEAD_URL),)), store)
|
|
test_client = TestClient(app)
|
|
test_client.__enter__()
|
|
try:
|
|
sign_in(test_client)
|
|
app.state.ctx.hub.jobs["host:driveripper"].state.payload = failing_host_payload(
|
|
host_scrape, agent_snapshot)
|
|
wait_for_payload(test_client, "/api/ups/snapshot")
|
|
yield test_client
|
|
finally:
|
|
test_client.__exit__(None, None, None)
|
|
store.close()
|
|
|
|
|
|
@pytest.fixture
|
|
def healthy_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
|
agent_snapshot: dict[str, Any],
|
|
host_scrape: Any) -> Iterator[TestClient]:
|
|
"""The all-clear app: one host answering on both of its jobs, nothing past a threshold.
|
|
|
|
The job's collector is swapped for one that hands back the injected payload, so the
|
|
health record says `ok` truthfully instead of being overridden behind the poller — and
|
|
that has to happen for the `disks` job too, which polls the same dead port: a host is
|
|
either answering or it is not, and a fixture that claims all-clear cannot leave one of
|
|
its two readers failing. Each reader gets **its own** projection of the same capture
|
|
(`parse_disks_snapshot` for the drive page), because a `disks` job holding a `system`
|
|
payload would make `/api/summary`'s drive block answer from the wrong shape and read as
|
|
a passing test that proved nothing.
|
|
"""
|
|
with build_client(tmp_path, monkeypatch,
|
|
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
|
|
from dashboards.poller import Result # noqa: PLC0415 - only this fixture needs it
|
|
|
|
payload = host_scrape(agent_snapshot, cpu_busy=10.0)
|
|
disks_payload = parse_disks_snapshot("driveripper", agent_snapshot,
|
|
DisksCollector()).payload
|
|
|
|
async def collect() -> Result:
|
|
return Result(payload=payload, samples=[])
|
|
|
|
async def collect_disks() -> Result:
|
|
return Result(payload=disks_payload, samples=[])
|
|
|
|
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
|
|
job = hub.jobs["host:driveripper"]
|
|
disks_job = hub.jobs["disks:driveripper"]
|
|
job.fn = collect
|
|
disks_job.fn = collect_disks
|
|
# let the poll already in flight against the dead port land first (it fails in
|
|
# microseconds), then take the health record over: the alerts must describe a host
|
|
# that is answering, not the half-second of startup where it was not
|
|
deadline = time.time() + 2.0
|
|
while time.time() < deadline and job.state.attempts == 0:
|
|
time.sleep(0.02)
|
|
time.sleep(0.05)
|
|
for reader in hub.jobs.values():
|
|
if reader.dashboard not in ("system", "disks"):
|
|
continue
|
|
reader.state.payload = disks_payload if reader is disks_job else payload
|
|
reader.state.ok = True
|
|
reader.state.error = None
|
|
reader.state.consecutive_failures = 0
|
|
reader.state.last_ok = time.time()
|
|
yield test_client
|
|
|
|
|
|
# ---- severity travels with the value --------------------------------------- #
|
|
|
|
def test_summary_carries_the_verdict_for_every_column_the_page_colours(data_client) -> None:
|
|
body = data_client.get("/api/summary").json()
|
|
row = body["system"]["hosts"][0]
|
|
assert row["severity"] == {"cpu_usage_pct": "ok", "mem_used_pct": "fail",
|
|
"disk_worst_used_pct": "fail", "temp_c": "fail"}
|
|
# the fleet card is judged by the same table, so amber means the same thing at both
|
|
# sizes — and no GPU verdict, at fleet size or host size (phase 06: busy is the job)
|
|
assert body["system"]["fleet"]["severity"] == {
|
|
"cpu_usage_pct_avg": "ok", "memory_used_pct": "fail",
|
|
"disk_worst_used_pct": "fail",
|
|
}
|
|
for ups in body["ups"]["ups"]:
|
|
assert set(ups["severity"]) == {"charge_pct", "load_pct", "runtime_seconds"}
|
|
assert ups["severity"]["charge_pct"] == classify("ups.charge_pct", ups["charge_pct"])
|
|
assert ups["severity"]["charge_pct"] == "ok", "the simulator starts at float charge"
|
|
|
|
|
|
def test_summary_shape_did_not_shrink(data_client) -> None:
|
|
"""Additive-only anchor: /api/summary gained `severity` and lost nothing.
|
|
|
|
The overview, the live tests, and any external consumer read the flat keys; this test is
|
|
the tripwire for a well-meaning refactor that "tidies" them away.
|
|
"""
|
|
body = data_client.get("/api/summary").json()
|
|
fleet_keys = {"hosts_total", "hosts_live", "cpu_cores", "cpu_usage_pct_avg",
|
|
"memory_total_bytes", "memory_used_bytes", "memory_used_pct",
|
|
"disk_worst_used_pct", "load1_total", "net_rx_bytes_per_s",
|
|
"net_tx_bytes_per_s", "gpu_count", "gpu_busy_pct_avg", "power_estimate_w"}
|
|
host_keys = {"name", "os", "cpu_usage_pct", "load1", "cores", "mem_used_pct",
|
|
"disk_worst_used_pct", "gpu_busy_pct", "temp_c", "uptime_seconds"}
|
|
ups_keys = {"name", "overall", "status", "charge_pct", "load_pct", "runtime_seconds"}
|
|
assert fleet_keys <= set(body["system"]["fleet"]), fleet_keys - set(body["system"]["fleet"])
|
|
assert host_keys <= set(body["system"]["hosts"][0]), host_keys - set(body["system"]["hosts"][0])
|
|
assert ups_keys <= set(body["ups"]["ups"][0]), ups_keys - set(body["ups"]["ups"][0])
|
|
# `disks` joined in phase 09 and changed nothing else: the set grows, never shrinks
|
|
assert set(body) == {"ts", "system", "disks", "llm", "ups", "recent_events", "alerts"}
|
|
# ...and the new keys are there beside the old ones, not instead of them
|
|
assert "severity" in body["system"]["fleet"]
|
|
assert "severity" in body["system"]["hosts"][0]
|
|
assert "severity" in body["ups"]["ups"][0]
|
|
|
|
|
|
# `/` paints two panels; the payload is still the whole-fleet contract. Every path here is
|
|
# read by something that is not the overview page: the live suite, an external probe, a
|
|
# notifier later. Deleting a card must never delete a key.
|
|
SUMMARY_CONTRACT = ("system.enabled", "system.fleet", "system.hosts",
|
|
"disks.enabled", "disks.fleet", "disks.hosts",
|
|
"llm.engines", "ups.ups", "recent_events", "alerts")
|
|
|
|
|
|
def dig(body: dict[str, Any], path: str) -> Any:
|
|
"""`body["a"]["b"]` that names the missing path when it is not there."""
|
|
node: Any = body
|
|
for part in path.split("."):
|
|
assert isinstance(node, dict) and part in node, f"/api/summary lost `{path}`"
|
|
node = node[part]
|
|
return node
|
|
|
|
|
|
@pytest.mark.parametrize("fixture", ["client", "data_client"])
|
|
def test_summary_serves_every_path_another_consumer_reads(request, fixture: str) -> None:
|
|
"""The keys survive the overview cleanup whether or not anything is configured.
|
|
|
|
Run twice on purpose: `client` has no targets at all, so it proves the keys are present
|
|
as `None`/`[]` rather than omitted when a source is unconfigured (invariant 3 is about
|
|
the payload shape as much as about colour); `data_client` proves the same paths carry
|
|
the fleet once there is one.
|
|
"""
|
|
test_client = request.getfixturevalue(fixture)
|
|
body = test_client.get("/api/summary").json()
|
|
populated = fixture == "data_client"
|
|
for path in SUMMARY_CONTRACT:
|
|
dig(body, path) # presence is the assertion; a missing path raises with its name
|
|
assert dig(body, "system.enabled") is populated
|
|
assert (dig(body, "system.fleet") is not None) is populated
|
|
names = {host["name"] for host in dig(body, "system.hosts")}
|
|
assert names == ({"driveripper"} if populated else set())
|
|
# the drive-health block is configured exactly when the agents it reads are, and lists the
|
|
# same hosts: a `disks` job exists for every `HOST_TARGETS` entry and for no other reason
|
|
assert dig(body, "disks.enabled") is populated
|
|
assert (dig(body, "disks.fleet") is not None) is populated
|
|
assert {host["name"] for host in dig(body, "disks.hosts")} == names
|
|
batteries = {item["name"] for item in dig(body, "ups.ups")}
|
|
assert batteries == ({"apc", "cyberpower"} if populated else set())
|
|
assert dig(body, "llm.engines") == [], "neither fixture configures an engine"
|
|
assert isinstance(dig(body, "recent_events"), list)
|
|
assert isinstance(dig(body, "alerts"), list)
|
|
# and the payload as a whole still carries every top-level section it always did
|
|
assert set(body) >= {"ts", "system", "llm", "ups", "recent_events", "alerts"}
|
|
|
|
|
|
# ---- alerts: the landing page's answer to "what is wrong" ------------------- #
|
|
|
|
def test_summary_is_empty_of_alerts_when_everything_is_healthy(healthy_client) -> None:
|
|
"""`[]`, not a green banner: an empty list is the clearest possible all-clear."""
|
|
body = healthy_client.get("/api/summary").json()
|
|
assert body["alerts"] == []
|
|
assert body["system"]["fleet"]["hosts_live"] == 1
|
|
|
|
|
|
def test_summary_raises_one_alert_for_the_host_that_stopped_answering(data_client) -> None:
|
|
"""One dead host is one line naming it, with the link to the page that explains it."""
|
|
body = data_client.get("/api/summary").json()
|
|
down = [item for item in body["alerts"] if item["source"] == "driveripper"]
|
|
assert len(down) == 1, "its stale payload must not add amber lines on top of the red one"
|
|
assert down[0]["severity"] == "fail"
|
|
assert down[0]["href"] == "/system"
|
|
assert "unreachable" in down[0]["message"]
|
|
|
|
|
|
def test_a_host_that_has_never_answered_is_still_a_row_in_the_fleet(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""A dead agent is a red row in the fleet, not a fleet that quietly got smaller.
|
|
|
|
`/api/system/snapshot` already stubs a job with no payload; `/api/summary` has to agree
|
|
with it, or a target that never answered since startup leaves the table at "2/2 hosts"
|
|
while the panel above says a third box is down (`AGENTS.md` invariants 3 and 4).
|
|
"""
|
|
with build_client(tmp_path, monkeypatch,
|
|
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
|
|
body = test_client.get("/api/summary").json()
|
|
rows = body["system"]["hosts"]
|
|
assert [row["name"] for row in rows] == ["driveripper"]
|
|
assert rows[0]["unreachable"] is True
|
|
assert rows[0]["cpu_usage_pct"] is None, "n/a, never a 0.0 nobody measured"
|
|
assert body["system"]["fleet"]["hosts_total"] == 1
|
|
assert body["system"]["fleet"]["hosts_live"] == 0
|
|
# and the alert list stays exactly one line for it, not a row plus a stale payload
|
|
assert [item["source"] for item in body["alerts"]] == ["driveripper"]
|
|
|
|
|
|
def test_summary_alerts_are_worst_first_and_carry_the_contract(data_client) -> None:
|
|
body = data_client.get("/api/summary").json()
|
|
alerts_list = body["alerts"]
|
|
assert alerts_list, "a dead host plus simulated UPS data cannot read as all-clear"
|
|
for item in alerts_list:
|
|
assert set(item) == {"id", "severity", "source", "message", "href", "at"}
|
|
assert item["severity"] in ("warn", "fail")
|
|
ranks = [0 if item["severity"] == "fail" else 1 for item in alerts_list]
|
|
assert ranks == sorted(ranks), "fails before warns"
|
|
assert any("simulat" in item["message"].lower() and item["href"] == "/ups"
|
|
for item in alerts_list), "NUT_SIMULATE must be said out loud on the landing page"
|
|
|
|
|
|
def test_summary_reuses_the_alert_builder_verbatim(data_client) -> None:
|
|
"""The endpoint is a thin call: what `alerts.build` says over the same state is the body."""
|
|
body = data_client.get("/api/summary").json()
|
|
hub = data_client.app.state.ctx.hub # type: ignore[attr-defined]
|
|
settings = data_client.app.state.ctx.settings # type: ignore[attr-defined]
|
|
hosts = [job.state.payload for job in hub.jobs.values()
|
|
if job.dashboard == "system" and job.state.payload]
|
|
ups_payload = hub.jobs["nut"].state.payload
|
|
expected = alerts.build(settings, hub.states(), hosts, [], ups_payload, [])
|
|
assert [item["id"] for item in body["alerts"]] == [item["id"] for item in expected]
|
|
|
|
|
|
def test_snapshot_payloads_carry_their_own_severity_sections(data_client) -> None:
|
|
"""/api/system/snapshot and /api/ups/snapshot ship the same verdicts the summary reuses."""
|
|
system = data_client.get("/api/system/snapshot").json()
|
|
host = system["hosts"][0]
|
|
assert set(host["cpu"]["severity"]) == {"usage_pct", "temp_c"}
|
|
assert set(host["memory"]["severity"]) == {"used_pct"}
|
|
assert set(host["disks"]["severity"]) == {"worst_used_pct"}
|
|
assert "severity" not in host["gpus"][0], (
|
|
"a GPU verdict came back on /api/system/snapshot — phase 06 removed it"
|
|
)
|
|
ups = data_client.get("/api/ups/snapshot").json()
|
|
for card in ups["ups"]:
|
|
assert set(card["severity"]) == {"charge_pct", "load_pct", "battery_runtime"}
|
|
# numbers keep their own shape; severity is a sibling, never a replacement
|
|
assert isinstance(card["numbers"], dict) and "charge_pct" in card["numbers"]
|
|
|
|
|
|
# ---- the self-test verdict on the two surfaces that were wrong (phase 12) --- #
|
|
|
|
#: The `ups.test.result` sentences as the hardware answers them. `Done and passed` is the
|
|
#: CyberPower's verbatim word from the 2026-10-19 capture (`tests/test_nut.py
|
|
#: ::LIVE_SELF_TEST_WORDS`); `FAILED` is what a driver says when a test ran and did not pass,
|
|
#: and is the reason the check exists at all. `tests/test_nut.py` pins the *word map*; what is
|
|
#: pinned here is the surface `TODO.md:3` names — a warning in the overview's Warnings panel —
|
|
#: because a verdict that grades correctly inside `assess()` and still reaches the alert feed
|
|
#: as a warning is the same bug wearing a different hat.
|
|
SELF_TEST_PASS_WORD = "Done and passed"
|
|
SELF_TEST_FAIL_WORD = "FAILED"
|
|
|
|
#: How `/api/summary` says the check out loud (`alerts._ups_alerts`): `"<check name>: <detail>"`.
|
|
SELF_TEST_ALERT_TEXT = "Last self-test"
|
|
|
|
|
|
def set_self_test_verdict(payload: dict[str, Any], status: str, detail: str) -> None:
|
|
"""Rewrite every card's `Last self-test` check in the hub's own UPS payload.
|
|
|
|
Mutating the payload the job actually serves (instead of hand-building a fake one) is what
|
|
makes this an end-to-end pin: `/api/ups/snapshot` and `alerts.build()` then read the *same*
|
|
object, so the test cannot pass by having the two surfaces disagree about the verdict.
|
|
"""
|
|
graded = 0
|
|
for card in payload["ups"]:
|
|
for check in card["health"]["checks"]:
|
|
if check["name"] == SELF_TEST_ALERT_TEXT:
|
|
check["status"] = status
|
|
check["detail"] = detail
|
|
graded += 1
|
|
assert graded, "no `Last self-test` check in the UPS payload — nothing was tested"
|
|
|
|
|
|
@contextmanager
|
|
def self_test_app(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
|
) -> Iterator[tuple[TestClient, dict[str, Any]]]:
|
|
"""A UPS app whose served payload cannot be replaced by a poll mid-assertion.
|
|
|
|
The simulated UPSes answer `PASSED`, so the pass direction would survive a race with the
|
|
poll loop and the *fail* direction would not — a test that flakes on a poll is a test of
|
|
timing. Hence `healthy_client`'s trick for the `nut` job: the collector hands back the very
|
|
dict this harness holds, so the health record says `ok` truthfully (invariant 4) and the
|
|
payload stays ours. The interval is shortened for the same reason — waiting a real
|
|
`NUT_INTERVAL_SECONDS` per test would only make the suite slower at proving the same thing.
|
|
"""
|
|
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
|
|
nut_interval=0.5) as test_client:
|
|
from dashboards.poller import Result # noqa: PLC0415 - only this harness needs it
|
|
|
|
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
|
|
job = hub.jobs["nut"]
|
|
wait_for_payload(test_client, "/api/ups/snapshot")
|
|
payload = job.state.payload
|
|
|
|
async def collect() -> Result:
|
|
return Result(payload=payload, samples=[])
|
|
|
|
job.fn = collect
|
|
# let one poll of *our* collector land before the caller mutates the payload, so the
|
|
# health record and the served bytes are both demonstrably ours
|
|
baseline = job.state.attempts
|
|
deadline = time.time() + 5.0
|
|
while time.time() < deadline and job.state.attempts <= baseline:
|
|
time.sleep(0.02)
|
|
assert job.state.payload is payload and job.state.ok, (
|
|
"the nut job is not serving the payload under test — the pin below would be fiction"
|
|
)
|
|
yield test_client, payload
|
|
|
|
|
|
def self_test_alerts(alerts_list: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
|
return [item for item in alerts_list
|
|
if SELF_TEST_ALERT_TEXT.lower() in item["message"].lower()
|
|
or SELF_TEST_ALERT_TEXT.lower() in item["id"].lower()]
|
|
|
|
|
|
def test_a_passed_self_test_is_quiet_on_the_overview_and_ok_on_the_ups_page(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The TODO's sentence, end to end: `Done and passed` is not a warning anywhere.
|
|
|
|
Both halves of the fix are asserted on one payload: `/api/ups/snapshot` carries the `ok`
|
|
entry the `/ups` check list renders verbatim, and `/api/summary` — the only call the
|
|
overview makes — carries **no** row naming it. The simulated card still contributes its
|
|
`NUT_SIMULATE` line, so the list is not empty and this cannot be a vacuous pass.
|
|
"""
|
|
with self_test_app(tmp_path, monkeypatch) as (test_client, payload):
|
|
set_self_test_verdict(payload, "ok", SELF_TEST_PASS_WORD)
|
|
cards = wait_for_payload(test_client, "/api/ups/snapshot")["ups"]
|
|
for card in cards:
|
|
check = next(c for c in card["health"]["checks"]
|
|
if c["name"] == SELF_TEST_ALERT_TEXT)
|
|
assert check["status"] == "ok", f"{card['name']} reports a pass as {check['status']}"
|
|
assert check["detail"] == SELF_TEST_PASS_WORD, "the driver's word is the useful half"
|
|
rows = self_test_alerts(test_client.get("/api/summary").json()["alerts"])
|
|
assert rows == [], f"a passed battery test is on the overview as {rows}"
|
|
assert any(item["source"] == "ups" for item in
|
|
test_client.get("/api/summary").json()["alerts"]), (
|
|
"the simulated-UPS line disappeared, so this assertion has nothing left to be quiet about"
|
|
)
|
|
|
|
|
|
def test_a_never_tested_battery_is_not_an_alert_either(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The APC's factory `No test initiated` reaches the overview as silence, not amber.
|
|
|
|
This is the state the live hardware is actually in on one of the two UPSes, so the
|
|
screenshot proves the absence of a warning for a *never-run* test while the payload pin
|
|
above proves it for a passed one. `unknown` is not a verdict an operator can act on, and
|
|
`alerts._RANK` admits only `fail` and `warn`.
|
|
"""
|
|
with self_test_app(tmp_path, monkeypatch) as (test_client, payload):
|
|
set_self_test_verdict(payload, "unknown", nut.NEVER_TESTED_DETAIL)
|
|
cards = wait_for_payload(test_client, "/api/ups/snapshot")["ups"]
|
|
assert all(next(c for c in card["health"]["checks"]
|
|
if c["name"] == SELF_TEST_ALERT_TEXT)["status"] == "unknown"
|
|
for card in cards)
|
|
assert self_test_alerts(test_client.get("/api/summary").json()["alerts"]) == []
|
|
|
|
|
|
def test_a_failed_self_test_still_reaches_the_overview(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The check keeps its reason for existing: a test that failed is a red line, once per UPS.
|
|
|
|
Without this direction the absence test above is satisfiable by deleting the check, which
|
|
is the failure mode the phase's completion criterion calls out.
|
|
"""
|
|
with self_test_app(tmp_path, monkeypatch) as (test_client, payload):
|
|
set_self_test_verdict(payload, "fail", SELF_TEST_FAIL_WORD)
|
|
names = {card["name"] for card in
|
|
wait_for_payload(test_client, "/api/ups/snapshot")["ups"]}
|
|
assert names, "no UPS card to fail"
|
|
rows = self_test_alerts(test_client.get("/api/summary").json()["alerts"])
|
|
assert {row["source"] for row in rows} == names
|
|
for row in rows:
|
|
assert row["severity"] == "fail", f"a failed battery test reads {row['severity']}"
|
|
assert row["href"] == "/ups", "the alert must link to the card that names the UPS"
|
|
assert SELF_TEST_ALERT_TEXT in row["message"] and SELF_TEST_FAIL_WORD in row["message"]
|
|
assert row["message"].count(SELF_TEST_ALERT_TEXT) == 1, (
|
|
"the row names the check twice — see alerts._clean_error")
|
|
assert all(row["id"] == f"{row['source']}:health:{SELF_TEST_ALERT_TEXT}" for row in rows)
|
|
|
|
|
|
def test_the_ups_check_list_needs_no_new_markup_for_a_passed_verdict(client) -> None:
|
|
"""Task 03 expected no page change, and this pins *why* there is none.
|
|
|
|
`page-ups.js` puts `check.status` straight on the `<li>`, and `dash.css` colours only the
|
|
`.mark` glyph from the shared `--ok`/`--warn`/`--fail` tokens while the check's name and
|
|
detail stay `--text`/`--text-muted` — so an `ok` row is quiet ink with a tick, and a verdict
|
|
fixed in `assess()` is a verdict rendered. Nothing on either asset names the self-test
|
|
check, which is the point: a special case added here would be a second opinion about the
|
|
driver's word, and the API already made it.
|
|
"""
|
|
js = _code(client.get("/static/page-ups.js").text)
|
|
assert '<li class="${check.status}">' in js, "the row no longer wears the payload verdict as its class"
|
|
for verdict in ("ok", "warn", "fail", "unknown"):
|
|
assert f"{verdict}: '" in js[js.index("const MARKS"):], f"MARKS lost the {verdict} glyph"
|
|
rows = js[js.index("const health = item.health"):js.index("const commands = item.commands")]
|
|
assert "self-test" not in rows.lower(), "the check list grew an opinion about one check by name"
|
|
css = _code(_css(client))
|
|
ok_rule = re.search(r"ul\.checks li\.ok \.mark \{([^}]*)\}", css)
|
|
assert ok_rule and "var(--ok)" in ok_rule.group(1), "the ok tick is not the shared token"
|
|
assert "ul.checks li.ok {" not in css, "the whole ok row is coloured — a pass is not a banner"
|
|
assert "self-test" not in css.lower(), "the stylesheet special-cases one check by name"
|
|
|
|
|
|
# ---- unconfigured: say so, and do not dial -------------------------------- #
|
|
|
|
def test_config_reports_every_source_as_off(client) -> None:
|
|
body = client.get("/api/config").json()
|
|
assert body["system"]["enabled"] is False
|
|
assert body["system"]["targets"] == []
|
|
assert body["disks"] == {"enabled": False, "interval_seconds": 300.0,
|
|
"timeout_seconds": 30.0}, (
|
|
"the drive-health page is configured exactly when the host agents are")
|
|
assert body["llm"]["gufo"]["enabled"] is False
|
|
assert body["llm"]["vllm"]["enabled"] is False
|
|
assert body["ups"]["enabled"] is False
|
|
assert body["ups"]["actions_enabled"] is False
|
|
assert body["ranges"], "the page builds its window buttons from this list"
|
|
|
|
|
|
def test_health_with_nothing_configured_is_not_a_failure(client) -> None:
|
|
"""Nothing configured is 'empty', not 'broken' — and the page must not go red."""
|
|
body = client.get("/api/health").json()
|
|
assert body["status"] in ("ok", "empty")
|
|
assert body["ok"] is True
|
|
assert body["sources"] == []
|
|
|
|
|
|
def test_the_disk_source_has_its_own_health_record_per_host(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""`disks` is a source of its own, not a field of `system` (phase 09).
|
|
|
|
Its own record is what gives the page its own dot and its own staleness, and it is
|
|
also what makes `alerts.py` need a rule: two jobs read the same agent, so one dead
|
|
host must still be one line on the overview (see `test_summary_raises_one_alert...`).
|
|
"""
|
|
with build_client(tmp_path, monkeypatch,
|
|
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
|
|
sources = {row["key"]: row for row in test_client.get("/api/health").json()["sources"]}
|
|
assert "disks:driveripper" in sources, "no separate record, no separate dot"
|
|
assert sources["disks:driveripper"]["dashboard"] == "disks"
|
|
assert sources["disks:driveripper"]["label"] == "driveripper"
|
|
assert sources["disks:driveripper"]["configured"] is True
|
|
assert sources["host:driveripper"]["dashboard"] == "system", (
|
|
"the system job is still its own source: one slow SMART poll must not "
|
|
"decide whether the host looks alive")
|
|
|
|
|
|
def test_no_targets_means_no_disk_job_that_could_look_broken(client) -> None:
|
|
"""No host agents, no `disks` task at all — so nothing red for an unconfigured page."""
|
|
assert client.get("/api/health").json()["sources"] == []
|
|
|
|
|
|
@pytest.mark.parametrize("path", ["/api/system/snapshot", "/api/llm/snapshot",
|
|
"/api/ups/snapshot", "/api/ups/commands"])
|
|
def test_unconfigured_source_is_404_configured_false(client, path) -> None:
|
|
response = client.get(path)
|
|
assert response.status_code == 404
|
|
assert response.json()["configured"] is False
|
|
assert "detail" in response.json()
|
|
|
|
|
|
def test_summary_still_answers_with_no_sources(client) -> None:
|
|
body = client.get("/api/summary").json()
|
|
assert body["system"]["enabled"] is False
|
|
assert body["system"]["fleet"] is None
|
|
assert body["llm"]["enabled"] is False
|
|
assert body["ups"]["enabled"] is False
|
|
|
|
|
|
def test_unknown_route_is_json_not_html(client) -> None:
|
|
response = client.get("/api/nope")
|
|
assert response.status_code == 404
|
|
assert response.headers["content-type"].startswith("application/json")
|
|
assert "no such route" in response.json()["detail"]
|
|
|
|
def test_an_upstream_failure_escaping_a_route_keeps_its_own_status(tmp_path: Path) -> None:
|
|
"""`collectors.UpstreamError` is the one error type a route may raise to say "upstream".
|
|
|
|
The handler turns it into `{ok: false, detail}` with the collector's status (502, or 504
|
|
for a timeout) instead of a 500 traceback, and the message is the collector's already
|
|
redacted sentence. Nothing dials here: the error is raised by the route under test.
|
|
"""
|
|
db = str(tmp_path / "upstream-error.sqlite3")
|
|
store = SampleStore(db)
|
|
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2), store)
|
|
|
|
@app.get("/api/_boom")
|
|
async def boom() -> None:
|
|
raise UpstreamError("driveripper: http://127.0.0.1:9120/snapshot timed out", status=504)
|
|
|
|
with TestClient(app) as test_client:
|
|
sign_in(test_client)
|
|
response = test_client.get("/api/_boom")
|
|
assert response.status_code == 504, "a timeout stays a timeout, not a 500"
|
|
assert response.json() == {"ok": False,
|
|
"detail": "driveripper: http://127.0.0.1:9120/snapshot "
|
|
"timed out"}
|
|
assert "no-store" in response.headers["cache-control"]
|
|
store.close()
|
|
|
|
|
|
# ---- tokens --------------------------------------------------------------- #
|
|
|
|
def test_agent_tokens_never_reach_the_browser(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
monkeypatch.setenv("NUT_SIMULATE", "")
|
|
targets = [Target(name="driveripper", url="http://driveripper:9120",
|
|
headers={"Authorization": "Bearer SEKRIT-TOKEN"})]
|
|
db = str(tmp_path / "tokens.sqlite3")
|
|
store = SampleStore(db)
|
|
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2,
|
|
host_targets=tuple(targets)), store)
|
|
with TestClient(app) as client:
|
|
sign_in(client)
|
|
body = client.get("/api/config")
|
|
assert "SEKRIT-TOKEN" not in body.text
|
|
target = body.json()["system"]["targets"][0]
|
|
assert target["name"] == "driveripper"
|
|
assert "SEKRIT" not in target["url"] and "@" not in target["url"]
|
|
# and a host that never answered stays in the list, marked, with the fleet card intact
|
|
snapshot = wait_for_state(client, "/api/system/snapshot")
|
|
assert snapshot["hosts"][0]["name"] == "driveripper"
|
|
assert snapshot["hosts"][0].get("unreachable") is True
|
|
assert snapshot["fleet"]["hosts_total"] == 1
|
|
assert snapshot["fleet"]["hosts_live"] == 0
|
|
assert snapshot["fleet"]["memory_total_bytes"] is None
|
|
store.close()
|
|
|
|
|
|
def test_a_dead_host_never_writes_its_token_into_the_feed(tmp_path: Path) -> None:
|
|
"""The second path a credential can take to a browser: the failure message.
|
|
|
|
`parse_targets` moves a userinfo token out of the URL and into a bearer header, so the
|
|
documented `HOST_TARGETS` path never puts one in `Target.url` — this is the backstop for
|
|
any `Target` that does (a hand-built one, a future parser), and the message path is where
|
|
it would matter: the poller keeps an `UpstreamError` string in the health record and logs
|
|
it as an event, so a userinfo-bearing URL there would sit in SQLite and be served on
|
|
`/api/events` forever (AGENTS.md invariant 6). Asserted on every endpoint a page reads,
|
|
not just `/api/config`.
|
|
"""
|
|
db = str(tmp_path / "dead-host-token.sqlite3")
|
|
store = SampleStore(db)
|
|
settings = Settings(
|
|
db_path=db,
|
|
admin_password=ADMIN_PASSWORD, pbkdf2_iterations=FAST_PBKDF2,
|
|
host_interval=0.05,
|
|
host_targets=(Target(name="driveripper", url="http://SEKRIT-AGENT-TOKEN@127.0.0.1:1"),),
|
|
)
|
|
with TestClient(create_app(settings, store)) as client:
|
|
sign_in(client)
|
|
deadline = time.time() + 10
|
|
while time.time() < deadline:
|
|
health = client.get("/api/health").json()
|
|
if health["sources"] and health["sources"][0]["error"]:
|
|
break
|
|
time.sleep(0.05)
|
|
else:
|
|
pytest.fail("the dead host never recorded an error")
|
|
for path in ("/api/health", "/api/events", "/api/system/snapshot",
|
|
"/api/disks/snapshot", "/api/summary"):
|
|
body = client.get(path).text
|
|
assert "SEKRIT-AGENT-TOKEN" not in body, f"{path} leaked the agent token"
|
|
# and the outage is still legible — redaction is not the same as silence
|
|
assert "driveripper" in client.get("/api/health").text
|
|
assert "unreachable" in client.get("/api/events").text
|
|
# the `disks` job dials the same token-bearing URL on its own cadence, so its
|
|
# sentence is a second copy of the same risk: `fetch_snapshot` is the one place
|
|
# either dashboard builds a failure message, and it uses the redacted URL
|
|
disks_state = next(row for row in client.get("/api/health").json()["sources"]
|
|
if row["key"] == "disks:driveripper")
|
|
assert disks_state["error"] and "SEKRIT-AGENT-TOKEN" not in disks_state["error"]
|
|
assert "127.0.0.1:1/snapshot" in disks_state["error"], "the failure still names where"
|
|
store.close()
|
|
|
|
|
|
def test_engine_urls_are_redacted_the_same_way_hosts_are(tmp_path: Path) -> None:
|
|
"""`GUFO_BASE_URL`/`VLLM_BASE_URL` go to the browser verbatim in `/api/config`, and unlike
|
|
`HOST_TARGETS` nothing parses a userinfo token out of them into a header — so a URL that
|
|
happens to carry credentials has to be redacted where it is served (invariant 6).
|
|
"""
|
|
db = str(tmp_path / "engine-redact.sqlite3")
|
|
settings = Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2,
|
|
gufo_url="http://SEKRIT-ENGINE-TOKEN@deskwork:8731",
|
|
vllm_url="http://driveripper:8000")
|
|
store = SampleStore(db)
|
|
with TestClient(create_app(settings, store)) as client:
|
|
sign_in(client)
|
|
body = client.get("/api/config")
|
|
assert "SEKRIT-ENGINE-TOKEN" not in body.text
|
|
llm = body.json()["llm"]
|
|
assert llm["gufo"]["base_url"] == "http://deskwork:8731"
|
|
assert llm["vllm"]["base_url"] == "http://driveripper:8000" # untouched when clean
|
|
store.close()
|
|
|
|
|
|
def wait_for_state(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
|
|
deadline = time.time() + timeout
|
|
while time.time() < deadline:
|
|
response = client.get(path)
|
|
if response.status_code == 200:
|
|
return response.json()
|
|
time.sleep(0.1)
|
|
pytest.fail(f"{path} never answered")
|
|
|
|
|
|
def test_target_problems_are_surfaced_not_swallowed(client) -> None:
|
|
body = client.get("/api/config").json()
|
|
assert body["system"]["target_problems"] == []
|
|
|
|
|
|
# ---- the credential the server refuses to start without (phase 11) --------- #
|
|
|
|
|
|
def test_no_verifier_and_no_seed_means_the_server_does_not_come_up(tmp_path: Path) -> None:
|
|
"""Fail closed, at the one place a mistake is unmissable: startup.
|
|
|
|
Task 03 is what demands a session on every route; this is the half before it — a deployment
|
|
with neither a stored `auth.verifier` row nor `DASH_ADMIN_PASSWORD` must not run open, so
|
|
the lifespan raises instead of publishing a context, starting the pollers, or serving a
|
|
page. Asserting that nothing was written is part of it: "refused" that seeded a row on the
|
|
way out would leave the next start half-configured.
|
|
"""
|
|
db = str(tmp_path / "no-credential.sqlite3")
|
|
store = SampleStore(db)
|
|
app = create_app(Settings(db_path=db, admin_password=""), store)
|
|
try:
|
|
with pytest.raises(RuntimeError) as caught, TestClient(app):
|
|
pytest.fail("the lifespan yielded with no admin credential")
|
|
message = str(caught.value)
|
|
assert "DASH_ADMIN_PASSWORD" in message, "the operator's fix is in the exit output"
|
|
assert "auth.verifier" in message and "DELETE FROM meta" in message
|
|
assert not hasattr(app.state, "ctx"), "nothing is published, so nothing can serve"
|
|
assert store.get_meta("auth.verifier") is None
|
|
assert store.get_meta("auth.session_secret") is None
|
|
finally:
|
|
store.close()
|
|
|
|
|
|
def test_the_same_server_starts_once_the_environment_seeds_a_password(tmp_path: Path) -> None:
|
|
"""The other side of the gate, so the failure above cannot be satisfied by a broken app.
|
|
|
|
With a seed the process comes up, the state hangs off the context (task 03 reads it on
|
|
every request), and both rows are durable — which is also what lets the operator delete
|
|
`DASH_ADMIN_PASSWORD` from the deployment after the first boot.
|
|
"""
|
|
db = str(tmp_path / "seeded.sqlite3")
|
|
store = SampleStore(db)
|
|
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2), store)
|
|
try:
|
|
with TestClient(app) as client:
|
|
assert client.get("/api/health").status_code == 200 # public: the healthcheck
|
|
state = app.state.ctx.auth
|
|
assert state is not None
|
|
assert state.seeded is True and state.env_ignored is False
|
|
assert state.iterations == FAST_PBKDF2
|
|
assert auth.verify_password(ADMIN_PASSWORD, state.verifier) is True
|
|
assert store.get_meta("auth.verifier") == state.verifier
|
|
assert store.get_meta("auth.session_secret") == state.secret
|
|
assert ADMIN_PASSWORD not in state.verifier + state.secret
|
|
finally:
|
|
store.close()
|
|
|
|
|
|
# ---- pages ---------------------------------------------------------------- #
|
|
|
|
@pytest.mark.parametrize("path,title", [
|
|
("/", "overview"), ("/system", "system"), ("/disks", "disks"), ("/llm", "llm"),
|
|
("/ups", "ups"),
|
|
])
|
|
def test_pages_serve_and_declare_themselves(client, path, title) -> None:
|
|
response = client.get(path)
|
|
assert response.status_code == 200
|
|
assert "text/html" in response.headers["content-type"]
|
|
assert title in response.text.lower()
|
|
|
|
|
|
def test_no_page_loads_anything_from_the_internet(client) -> None:
|
|
"""No CDN, no webfonts: this dashboard has to work with no route to the internet.
|
|
|
|
`/login` is not in `MARKUP_PAGES` and cannot be: this `client` is signed in, a signed-in
|
|
`GET /login` is a `303`, and following it would audit `/`'s bytes under the wrong name. The
|
|
same fragments are checked against the real login page, unsigned, by
|
|
`tests/test_auth_api.py::test_the_login_page_loads_nothing_from_the_internet`.
|
|
"""
|
|
for path in MARKUP_PAGES:
|
|
html = client.get(path).text
|
|
for fragment in ("//cdn.", "https://fonts.", "//unpkg.com", "//cdnjs"):
|
|
assert fragment not in html, f"{path} references {fragment}"
|
|
|
|
|
|
def test_static_assets_are_served(client) -> None:
|
|
for asset in ASSETS:
|
|
assert client.get(asset).status_code == 200, asset
|
|
|
|
|
|
# ---- /settings: the account page (phase 11 task 06) ----------------------- #
|
|
#
|
|
# `/settings` is the operator's own surface: the session this browser holds, a sign-out, and the
|
|
# one form that changes the admin password. Its HTTP surface is tested twice on purpose — the
|
|
# route-enumeration test in `tests/test_auth_api.py` proves nothing outside the public list
|
|
# answers anonymously (which is how `/settings` is gated for free by being in `PAGES` in
|
|
# `app.py`), and the assertions here pin the **page**: the form fields a password manager needs,
|
|
# the outcome line that is a live region, the precedence sentence that stops the "I edited the
|
|
# env file and nothing happened" support call, and the masthead link that is the only way in.
|
|
|
|
def test_the_settings_page_carries_the_session_and_the_form(client) -> None:
|
|
response = client.get(SETTINGS_PAGE)
|
|
assert response.status_code == 200
|
|
assert "text/html" in response.headers["content-type"]
|
|
html = response.text
|
|
# the session panel and its way out
|
|
for fragment in ('id="sign-out"', 'id="session-user"', 'id="session-expires"',
|
|
'id="session-left"'):
|
|
assert fragment in html, f"/settings has no {fragment}"
|
|
# the change-password form: three fields, each labelled, each with the floor spelled out,
|
|
# and the autocomplete values a password manager needs to fill the right one
|
|
for fragment in ('id="current-password"', 'autocomplete="current-password"',
|
|
'id="new-password"', 'autocomplete="new-password"',
|
|
'id="confirm-password"', 'id="change-password"',
|
|
'<label for="current-password">', '<label for="new-password">',
|
|
'<label for="confirm-password">', 'aria-describedby="password-hint"'):
|
|
assert fragment in html, f"/settings' form is missing {fragment}"
|
|
assert "8 characters" in html, "the form stopped naming the length floor it enforces"
|
|
# the outcome line is announced, because the operator asked for it and nothing else will
|
|
assert 'id="settings-result"' in html and 'role="status"' in html
|
|
|
|
|
|
def test_the_settings_page_states_the_precedence_rule(client) -> None:
|
|
"""The sentence that makes the env file's losing half legible on the page that wins.
|
|
|
|
`README.md` documents the same rule, but the operator reading it there is deciding what to
|
|
write in a file; the operator reading *this* page just changed a password and needs to know
|
|
why a container env will not undo it.
|
|
"""
|
|
html = client.get(SETTINGS_PAGE).text
|
|
assert "DASH_ADMIN_PASSWORD" in html
|
|
assert "auth.verifier" in html
|
|
assert "never overwrites" in html
|
|
|
|
|
|
def test_the_settings_page_is_not_one_of_the_dashboards(client) -> None:
|
|
"""The tabs are there to leave with; nothing marks itself current, because nothing is.
|
|
|
|
A tab marked current on a page that is not in the tab list claims a location the app does not
|
|
have, and the four dashboards' own masthead pin (`aria-current` exactly once) relies on that
|
|
marker meaning something.
|
|
"""
|
|
html = client.get(SETTINGS_PAGE).text
|
|
assert '<nav class="tabs"' in html
|
|
assert 'aria-current="page"' not in html
|
|
for path in PAGES:
|
|
assert f'href="{path}"' in html, f"/settings cannot navigate to {path}"
|
|
assert 'id="nav-status"' not in html, "/settings grew fleet dots it has nothing to fill"
|
|
|
|
|
|
def test_the_settings_page_refuses_a_caller_with_no_session(tmp_path: Path) -> None:
|
|
"""One page-level assertion of the gate, not a second enumeration of it.
|
|
|
|
`tests/test_auth_api.py::test_every_route_that_is_not_public_requires_a_session` is the
|
|
general proof; this one exists so a regression in `/settings` names `/settings`.
|
|
"""
|
|
db = str(tmp_path / "settings-gate.sqlite3")
|
|
store = SampleStore(db)
|
|
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2), store)
|
|
try:
|
|
with TestClient(app, follow_redirects=False) as anon:
|
|
page = anon.get(SETTINGS_PAGE, headers={"accept": "text/html"})
|
|
assert page.status_code == 303
|
|
assert page.headers["location"] == f"/login?next={SETTINGS_PAGE}"
|
|
data = anon.get(SETTINGS_PAGE, headers={"accept": "application/json"})
|
|
assert data.status_code == 401
|
|
assert data.json() == {"ok": False, "detail": "sign in required",
|
|
"login": "/login"}
|
|
finally:
|
|
store.close()
|
|
|
|
|
|
def test_every_dashboard_page_links_to_settings(client) -> None:
|
|
"""The `⚙ Account` link is the only way to the session page, so it is on every dashboard.
|
|
|
|
It sits in `.masthead-right` before the theme toggle and **outside** `.tabs`, because the tab
|
|
strip's structure — five dashboards, one of them current — is pinned by
|
|
`test_the_disks_tab_is_in_the_masthead_of_every_page` and a sixth entry with no
|
|
`aria-current` would break what that marker means.
|
|
"""
|
|
link = '<a class="ghost account" href="/settings"'
|
|
for path in PAGES:
|
|
html = client.get(path).text
|
|
assert link in html, f"{path} has no route to the account page"
|
|
tabs = html[html.index('<nav class="tabs"'):html.index("</nav>")]
|
|
assert "/settings" not in tabs, f"{path} made Account a dashboard tab"
|
|
assert html.index(link) < html.index('id="theme-toggle"'), \
|
|
f"{path} put the account link after the theme toggle"
|
|
# the stylesheet styles the link with the button's own rule, not a copy of it
|
|
css = client.get("/static/dash.css").text
|
|
assert "a.ghost" in css
|
|
|
|
|
|
def test_the_password_floor_the_form_checks_is_the_one_the_api_enforces(client) -> None:
|
|
"""The client-side length check is a courtesy; the server's is the rule.
|
|
|
|
There is no build step to share the constant (a locked anchor), so the two literals are
|
|
pinned together here: a floor raised in `api.py` alone would leave the form promising eight
|
|
characters to a request the server answers `422`.
|
|
"""
|
|
js = client.get("/static/page-settings.js").text
|
|
found = re.search(r"const MIN_PASSWORD_LENGTH = (\d+);", js)
|
|
assert found, "page-settings.js no longer names its length floor"
|
|
assert int(found.group(1)) == MIN_PASSWORD_LENGTH
|
|
|
|
|
|
def test_a_rotated_secret_clears_the_cached_session_every_page_uses(client) -> None:
|
|
"""`Dash.resetAuth()` is the difference between a password change and a 403 dashboard.
|
|
|
|
`POST /api/auth/password` rotates the signing secret on purpose (`auth.update_verifier`),
|
|
which invalidates the CSRF token `ensureAuth()` memoised. Every page writes through
|
|
`fetchJSON`, which reads that cache — so unless the change clears it, the operator changes
|
|
the password, clicks *beeper disable* on `/ups`, and gets `missing or invalid CSRF token`
|
|
until they reload the page. A bounce on a `401` clears it for the same reason: the session it
|
|
described no longer exists.
|
|
"""
|
|
app_js = client.get("/static/app.js").text
|
|
assert "function resetAuth(" in app_js
|
|
exports = app_js.rsplit("return {", 1)[1].split("}", 1)[0]
|
|
assert re.search(r"\bresetAuth\b", exports), "Dash.resetAuth is not exported"
|
|
reset = app_js[app_js.index("function resetAuth("):app_js.index("function bounceToLogin(")]
|
|
for field in ("loaded: false", "authenticated: false", "csrf: null"):
|
|
assert field in reset, f"resetAuth() left `{field}` cached"
|
|
assert "authPromise = null" in reset, \
|
|
"resetAuth() left an in-flight status call memoised — the fresh write would reuse it"
|
|
bounce = app_js[app_js.index("function bounceToLogin("):app_js.index("const READ_METHODS")]
|
|
assert "resetAuth()" in bounce, "a 401 bounce kept the dead session's token cached"
|
|
|
|
js = client.get("/static/page-settings.js").text
|
|
assert js.count("Dash.resetAuth()") >= 2, \
|
|
"the settings page must reset the cached session after both a change and a sign-out"
|
|
assert "textContent" in js and "innerHTML" not in js, \
|
|
"/settings renders a server sentence as markup"
|
|
|
|
|
|
# ---- the severity language in the static assets --------------------------- #
|
|
#
|
|
# There is no JS test runner (a locked anchor: adding one would mean a build step).
|
|
# What *can* be asserted end-to-end, from the served bytes, is the contract the pages
|
|
# depend on: the hero style exists, its colour comes from the shared ok/warn/fail tokens,
|
|
# no parallel `.sev-*` vocabulary appeared, and `Dash.hero()` reads the verdict instead of
|
|
# comparing the number to a threshold in JavaScript.
|
|
|
|
ASSETS = ("/static/dash.css", "/static/app.js", "/static/charts.js",
|
|
"/static/page-index.js", "/static/page-disks.js", "/static/page-system.js",
|
|
"/static/page-llm.js", "/static/page-ups.js", "/static/page-login.js",
|
|
"/static/page-settings.js")
|
|
|
|
|
|
def _css(client: TestClient) -> str:
|
|
return client.get("/static/dash.css").text
|
|
|
|
|
|
def _css_section(css: str, name: str) -> str:
|
|
"""One `/* ---- name … */` section of the stylesheet, up to the **next** section header.
|
|
|
|
Slicing to a named neighbour worked while the sheet had five sections. Every page has since
|
|
added a component section, and a slice that runs across someone else's rules accuses the
|
|
fleet block of colouring numbers it never coloured — so the boundary is structural: the end
|
|
of *this* section, whoever comes next.
|
|
"""
|
|
start = css.index(f"/* ---- {name}")
|
|
return css[start:css.index("/* ----", start + 10)]
|
|
|
|
|
|
def _hero_css(css: str) -> str:
|
|
"""Just the `hero numbers` section of the stylesheet."""
|
|
return css.split("/* ---- hero numbers", 1)[1].split("/* ---- bars", 1)[0]
|
|
|
|
|
|
def test_css_ships_the_hero_number_style(client) -> None:
|
|
css = _css(client)
|
|
hero_block = _hero_css(css)
|
|
for rule in (".hero {", ".hero .label", ".hero .value", ".hero .unit", ".hero .sub",
|
|
".hero.ok .value", ".hero.warn .value", ".hero.fail .value",
|
|
".hero-bar", ".hero-bar.ok .bar"):
|
|
assert rule in hero_block, f"dash.css lost the {rule} rule"
|
|
# a display size with fixed-width digits, so the number does not jitter between polls
|
|
assert "2.4rem" in hero_block
|
|
assert "tabular-nums" in hero_block
|
|
|
|
|
|
def test_hero_colour_comes_from_the_shared_verdict_tokens(client) -> None:
|
|
"""`.hero` reuses `--ok/--warn/--fail`; it does not invent a palette of its own."""
|
|
hero_block = _hero_css(_css(client))
|
|
for token in ("var(--ok)", "var(--warn)", "var(--fail)", "var(--text)",
|
|
"var(--text-muted)", "var(--text-faint)"):
|
|
assert token in hero_block, f"the hero style no longer uses {token}"
|
|
|
|
|
|
def test_no_stylesheet_rule_uses_an_undeclared_token(client) -> None:
|
|
"""A typo'd `var(--x)` renders unstyled and reads as a broken panel, not an error."""
|
|
css = _css(client)
|
|
root = re.search(r":root\s*\{(.*?)\n\}", css, re.S)
|
|
assert root is not None
|
|
declared = set(re.findall(r"(--[a-z0-9-]+)\s*:", root.group(1)))
|
|
used = set(re.findall(r"var\(\s*(--[a-z0-9-]+)", css))
|
|
assert not used - declared
|
|
|
|
|
|
def _code(body: str) -> str:
|
|
"""An asset with its comments stripped — prose may *name* a class the code must not define."""
|
|
body = re.sub(r"/\*.*?\*/", " ", body, flags=re.S)
|
|
body = re.sub(r"<!--.*?-->", " ", body, flags=re.S)
|
|
return re.sub(r"^\s*//.*$", " ", body, flags=re.M)
|
|
|
|
|
|
# The `fleet` object paths `/llm`'s aggregate block reads. `engines_total` drives the
|
|
# "hidden below two engines" rule and the tag; `engines_live` is what makes a partial sum
|
|
# visible; the rest are the row itself.
|
|
FLEET_BLOCK_PATHS = frozenset({
|
|
"engines_total", "engines_live", "requests_active", "requests_waiting",
|
|
"prompt_tokens_per_second", "decode_tokens_per_second", "tokens_total",
|
|
})
|
|
# The summed half of that list — the ones that must be null, not 0, with nothing answering.
|
|
FLEET_SUM_PATHS = ("requests_active", "requests_waiting", "prompt_tokens_per_second",
|
|
"decode_tokens_per_second", "tokens_total")
|
|
|
|
|
|
def _fleet_tag(html: str) -> str:
|
|
"""The aggregate block's opening tag — the one place its `hidden` default lives."""
|
|
start = html.index('<section class="card flat fleet"')
|
|
return html[start:html.index(">", start)]
|
|
|
|
|
|
def _fleet_render_path(js: str) -> str:
|
|
"""Just the body of `renderFleet()` — the aggregate block's whole code path, comments out.
|
|
|
|
Everything the block writes goes through here, so it is the slice that can be asserted
|
|
verdict-free without tripping over the engine panels' rated KV sparkline next door.
|
|
"""
|
|
js = _code(js)
|
|
return js[js.index("function renderFleet("):js.index("async function refresh(")]
|
|
|
|
|
|
def _fleet_field_metrics(js: str) -> list[str]:
|
|
"""The `fleet` keys the block's rows are bound to, in the order the block prints them."""
|
|
js = _code(js)
|
|
start = js.index("const FLEET_FIELDS = [")
|
|
table = js[start:js.index("];", start) + 2]
|
|
return re.findall(r"metric: '([a-z_]+)'", table)
|
|
|
|
|
|
def test_no_parallel_severity_vocabulary_exists_anywhere(client) -> None:
|
|
"""One word per colour: `ok|warn|fail`, as in severity.py and the existing components."""
|
|
for asset in (*ASSETS, *MARKUP_PAGES):
|
|
body = _code(client.get(asset).text)
|
|
assert ".sev-" not in body, f"{asset} invented a .sev-* class"
|
|
assert not re.search(r"classList\.(?:add|toggle)\(\s*[\"'](?:danger|good|warning)[\"']",
|
|
body), f"{asset} invented a synonym for fail"
|
|
|
|
|
|
def test_app_js_exposes_the_hero_helpers(client) -> None:
|
|
js = client.get("/static/app.js").text
|
|
assert "function hero(" in js
|
|
assert "function severityFrom(" in js
|
|
exports = js.rsplit("return {", 1)[1].split("}", 1)[0]
|
|
for name in ("hero", "severityFrom", "severityFor"):
|
|
assert re.search(rf"\b{name}\b", exports), f"Dash.{name} is not exported"
|
|
|
|
|
|
def test_hero_reads_the_verdict_instead_of_a_threshold(client) -> None:
|
|
"""`hero()` may only colour what the API rated — no thresholds re-derived in JS."""
|
|
js = client.get("/static/app.js").text
|
|
start = js.index("function hero(")
|
|
body = js[start:js.index("\n function setDot", start)]
|
|
assert "severityFrom(" in body
|
|
assert "severityFor(" not in body
|
|
# an unknown value gets n/a and no colour class at all
|
|
assert "naText" in body
|
|
assert re.search(r"raw != null", body)
|
|
|
|
|
|
def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
|
|
"""Every page call site is gone now that phase 05 converted `/ups`.
|
|
|
|
`/` was converted in phase 02, `/system` in phase 03, `/llm` in phase 04 and `/ups` in
|
|
phase 05, so no page colours anything from a threshold it wrote itself. The function
|
|
itself stays: it is the documented fallback `setBar()` uses when a caller names no
|
|
verdict, and a page that ever needs a verdict the API does not send should add it to
|
|
`severity.py` rather than here — which is what the docstring on it says.
|
|
"""
|
|
js = client.get("/static/app.js").text
|
|
assert "FALLBACK ONLY" in js
|
|
assert "severityFor" in js, "the documented fallback itself was deleted"
|
|
for page in PAGE_SCRIPTS.values():
|
|
assert "severityFor" not in client.get(f"/static/{page}").text, \
|
|
f"{page} colours something from a threshold of its own"
|
|
|
|
|
|
def test_overview_colours_from_the_payload_not_from_js_thresholds(client) -> None:
|
|
"""`page-index.js` reads verdicts; it does not compare a number to a bound."""
|
|
js = client.get("/static/page-index.js").text
|
|
assert "severityFrom" in js
|
|
assert "severityFor" not in js
|
|
|
|
|
|
def test_system_page_colours_from_the_payload_not_from_js_thresholds(client) -> None:
|
|
"""`page-system.js` hero cards paint payload verdicts, not thresholds of their own.
|
|
|
|
Phase 03 task 02's promise: every colour on a host card comes from a `severity` dict
|
|
`severity.py` produced (`cpu.severity.usage_pct`, `memory.severity.used_pct`,
|
|
`disks.severity.worst_used_pct`, `cpu.severity.temp_c`).
|
|
A threshold written here could disagree with the overview's alerts; a hero with no
|
|
verdict prints `n/a` uncoloured, which is `Dash.hero()`'s contract.
|
|
|
|
The GPU slot names no verdict on purpose (phase 06): `severity.THRESHOLDS` has no
|
|
`gpu.*` row, the payload carries no `gpus[].severity`, and the hero is built without
|
|
`severity` or `bar` — a bar under a verdictless number would be a verdict by shape.
|
|
"""
|
|
js = client.get("/static/page-system.js").text
|
|
assert "severityFor(" not in js, "the system page re-invented a threshold"
|
|
assert "Dash.hero(" in js
|
|
for hook in ("cpu.severity.usage_pct", "memory.severity.used_pct",
|
|
"disks.severity.worst_used_pct", "cpu.severity.temp_c"):
|
|
assert hook in js, f"the card no longer names {hook} as a colour source"
|
|
# and the GPU hero grades nothing: no verdict hook, no utilisation bar
|
|
assert "gpus[].severity" not in js, "the GPU slot invented a colour source again"
|
|
gpu_slot = js[js.index("function renderGpuSlot("):js.index("function renderGpuExtra(")]
|
|
assert "severity:" not in gpu_slot, "the GPU hero passes a severity option again"
|
|
assert "bar:" not in gpu_slot, "the GPU hero grew a bar back (a verdict by shape)"
|
|
assert "hero-bar" not in gpu_slot, "the GPU hero rebuilt the bar variant of .hero"
|
|
|
|
|
|
def test_gpu_hero_footnote_prints_the_temperature_the_agent_read(client) -> None:
|
|
"""TODO.md L9: the GPU temperature is glanceable on the hero row, not only in the table.
|
|
|
|
`gpus[].temp_c` is the agent's max over that card's hwmon labels (`sources/host.py`,
|
|
pinned by `tests/test_system_contract.py`), so the hero needed no new field and the
|
|
`gpu.temp_c@N` history series kept its name. The null guard *is* the behaviour, so that
|
|
is what the assertion pins (there is no JS test runner — a locked anchor): a card whose
|
|
driver exposes no temperature prints none, because `n/a` is for numbers the card is
|
|
supposed to have and a `0 °C` would be a cool GPU that does not exist.
|
|
"""
|
|
js = client.get("/static/page-system.js").text
|
|
sub = js[js.index("function gpuSub("):js.index("function renderGpuSlot(")]
|
|
assert "gpu.temp_c" in sub, "the GPU hero stopped printing the temperature"
|
|
assert "if (gpu.temp_c != null)" in sub, "the temperature lost its null guard"
|
|
assert sub.count("°C") == 1, (
|
|
"a second °C in gpuSub is a branch printing a temperature nobody measured"
|
|
)
|
|
# "More detail" keeps the per-label breakdown: a surface change, not a de-duplication
|
|
extra = js[js.index("function renderGpuExtra("):js.index("function renderHostCard(")]
|
|
assert "Temp" in extra and "gpu.temp_c" in extra, "GPU detail lost its temperature column"
|
|
|
|
|
|
def test_system_host_card_is_six_heroes_without_per_core(client) -> None:
|
|
"""Each host card leads with exactly six heroes: CPU, MEM, TEMP, DISK, GPU, NET.
|
|
|
|
Asserted on the served script because there is no JS test runner (a locked anchor).
|
|
The two dense `.metrics` rows and the per-core heat bars are exactly what made the
|
|
card unreadable, and "no per-core anywhere on /system" is a promise a stray
|
|
`renderCores` call would silently break — the series stays stored, the page just
|
|
stopped drawing it (PLAN.md *Dropped visuals*).
|
|
"""
|
|
js = client.get("/static/page-system.js").text
|
|
card = js[js.index("function hostCard("):js.index("function renderHostCard(")]
|
|
slots = re.findall(r'<div class="(?:hero|hero-slot)[^"]*" data-k="([^"]+)"', card)
|
|
assert slots == ["cpu", "mem", "temp", "disk", "gpus", "net"], \
|
|
"the hero row is no longer the six numbers in the agreed order"
|
|
# per-core visualisation is gone from the card code entirely
|
|
assert "cores" not in card, "a per-core visual came back to the card"
|
|
assert "renderCores" not in js and "core_usage" not in js
|
|
# the NET slot is a slot with two lines and no verdict: a byte rate has no good/bad
|
|
assert 'data-k="net"' in card
|
|
assert not re.search(r'data-k="net"[^>]*data-sev', card), "the NET slot invented a colour"
|
|
# what the glance leaves out stays reachable, collapsed by default
|
|
assert "More detail" in card
|
|
for key in ("load", "freq", "dr", "dw", "dutil", "uptime", "details"):
|
|
assert f'data-k="{key}"' in card, f"{key} left the page instead of the glance"
|
|
# the panel owns its charts: the slots live inside the card, not on the page
|
|
for chart in ("util", "net"):
|
|
assert f'data-chart="{chart}"' in card, f"the panel lost its {chart} chart slot"
|
|
|
|
|
|
def test_system_page_keeps_no_fleet_wide_chart_or_metric_picker(client) -> None:
|
|
"""Nothing on `/system` plots two hosts, and nothing needs a selection to make sense.
|
|
|
|
TODO.md L4 is the instruction ("The system page doesn't need shared charts for
|
|
anything") and the overview's alerts + fleet table are what made removing them safe
|
|
(phase 02). Asserted on the served bytes because the only thing that would otherwise
|
|
notice a resurrected `<select>` is a browser: there is no JS test runner (a locked
|
|
anchor), and `test_no_page_carries_an_id_nothing_reads` cannot see ids that were deleted
|
|
correctly from the page but are still queried by the script.
|
|
"""
|
|
html = _code(client.get("/system").text)
|
|
for gone in ('id="chart-cpu"', 'id="chart-mem"', 'id="chart-disk"', 'id="chart-net"',
|
|
'id="chart-gpu"', 'id="chart-temp"', 'id="chart-detail"', 'id="detail-host"',
|
|
'id="detail-metric"', 'id="f-cpu"', 'id="f-load"', 'id="f-gpu"',
|
|
"Fleet totals", "Fleet charts", "Series detail", "<select"):
|
|
assert gone not in html, f"/system still carries {gone}"
|
|
# the two panels that are left are the per-host grid and the event feed
|
|
assert html.count("<section") == 2, "/system grew a panel back on top of the host grid"
|
|
assert 'id="hosts"' in html and 'id="events"' in html
|
|
|
|
# and the script must not reach for the deleted endpoints/ids either — a `#detail-host`
|
|
# query against a removed node is a silent null, so the picker would just never appear
|
|
js = client.get("/static/page-system.js").text
|
|
for gone in ("#chart-", "detail-host", "detail-metric", "detail-note",
|
|
"renderFleet", "makeCharts", "refreshGpuChart", "populateDetailMetrics"):
|
|
assert gone not in js, f"page-system.js still references {gone}"
|
|
|
|
|
|
def test_every_system_chart_is_drawn_inside_its_own_host_panel(client) -> None:
|
|
"""One host per chart: the legends of `/system` can never list two machine names.
|
|
|
|
The construction is a factory that takes the card, so a chart cannot be built against a
|
|
page-level id; the metric list is per host, so the GPU streams cannot leak from one
|
|
panel into another; and a panel is torn down (with its charts) when its host leaves the
|
|
payload, which is the bug class a keyed-by-host chart map exists to avoid.
|
|
"""
|
|
js = client.get("/static/page-system.js").text
|
|
builds = re.findall(r"new DashCharts\.LineChart\(([^,]+),", js)
|
|
assert builds, "the system page builds no charts at all"
|
|
# every chart is built from a `[data-chart]` slot inside the host card, never from a
|
|
# page-level id — which is what makes "one host per chart" structural, not a convention
|
|
assert all(re.search(r"(?:card|slot)\.querySelector\(.\[data-chart=", b) for b in builds), \
|
|
f"a chart is built outside a host panel: {builds}"
|
|
# three construction sites, one per slot (util, net, gpu) — never one per metric
|
|
assert len(builds) == 3, f"charts are not built one per panel slot: {builds}"
|
|
assert sorted(re.findall(r'data-chart=\\?"(\w+)"', " ".join(builds))) == ["gpu", "net", "util"]
|
|
|
|
factory = js[js.index("async function renderHostCharts("):js.index("function pruneHosts(")]
|
|
assert "hosts=${encodeURIComponent(host.name)}" in factory, \
|
|
"the per-host chart query is not scoped to one host"
|
|
assert "&metrics=" in factory and "state.range" in factory, \
|
|
"a host's charts need one metric list and the selected window"
|
|
# GPU series are named from the payload's own `gpus[].index`, never discovered/guessed
|
|
assert "gpu.busy_pct@" in js and "gpu.index" in js
|
|
# the GPU chart exists only while the host reports cards
|
|
gpu_slot = js[js.index("function hostChartSet("):js.index("function seriesFor(")]
|
|
assert re.search(r"if \(!wantsGpu && set\.gpu\)", gpu_slot), \
|
|
"a host that stops reporting GPUs keeps a GPU chart it has no data for"
|
|
|
|
pruner = js[js.index("function pruneHosts("):js.index("// ---- snapshot")]
|
|
assert "state.hosts.delete(name)" in pruner and "state.charts.delete(name)" in pruner, \
|
|
"a host that left the payload keeps a live panel and drawing charts"
|
|
|
|
|
|
def test_system_history_charts_are_collapsed_behind_a_toggle(client) -> None:
|
|
"""TODO.md L4: "The historical charts on the system resources page should only be visible
|
|
if expanded. I only want to see current stats to keep things clean."
|
|
|
|
The default `/system` view is the hero row. Pinned on the served script because the card
|
|
markup is generated in JS and there is no JS test runner (a locked anchor), and the two
|
|
ways this regresses are invisible otherwise: a `<details open>` (charts visible by
|
|
default, i.e. the task un-done but the page still pretty) and a paint pass that runs
|
|
while the stack is shut, which builds three `LineChart`s per host inside a
|
|
`display:none` container — measured 0 px wide, never drawn, and rebuilt every poll.
|
|
"""
|
|
js = client.get("/static/page-system.js").text
|
|
card = js[js.index("function hostCard("):js.index("function memorySub(")]
|
|
|
|
# the stack is *wrapped*, not merely preceded by a heading: it lives inside the <details>
|
|
box = card.index('<details class="charts-box" data-k="charts">')
|
|
stack = card.index('<div class="host-charts">')
|
|
closed = card.index("</details>")
|
|
assert box < stack < closed, "the chart stack is not inside the toggle"
|
|
assert closed < card.index("More detail"), "the charts toggle swallowed the detail row too"
|
|
assert not re.search(r"<details[^>]*\bopen\b", card), \
|
|
"a panel ships its charts open — the default view is not current stats"
|
|
assert re.search(r"<summary[^>]*>\s*Charts", card), "the toggle no longer reads Charts"
|
|
# it also says which window the expanded stack draws (the chips stay in the page head)
|
|
assert 'data-k="charts-window"' in card, "the toggle does not carry the window"
|
|
# …and the toggle is what turns the paint pass on
|
|
assert "addEventListener('toggle'" in card and "chartsToggled(" in card
|
|
|
|
setter = js[js.index("function setChartsWindow("):js.index("function memorySub(")]
|
|
assert "state.rangeKey" in setter, \
|
|
"the toggle re-derives its window label instead of using the chips' own"
|
|
handler = js[js.index("Dash.rangeControl("):js.index("state.poller = poll(")]
|
|
assert "state.rangeKey = key" in handler and "setChartsWindow(" in handler, \
|
|
"changing the window leaves every toggle labelling the old one"
|
|
|
|
# nothing is built or drawn while the stack is shut, in either path that could paint
|
|
paint = js[js.index("function drawHostCharts("):js.index("async function renderHostCharts(")]
|
|
assert "if (!chartsOpen(card)) return" in paint, "a closed panel gets painted anyway"
|
|
assert "hostChartSet(card, host)" in paint, \
|
|
"the LineChart instances are no longer built by the open pass"
|
|
toggle = js[js.index("function chartsToggled("):js.index("function drawHostCharts(")]
|
|
assert "if (!chartsOpen(card)) return" in toggle, "closing the stack still paints"
|
|
assert "renderHostCharts(card, host)" in toggle and "state.range" in toggle, \
|
|
"a cached response from another window is drawn under this window's label"
|
|
|
|
# the request discipline is unchanged: one per host per refresh, every series named
|
|
fetch = js[js.index("async function renderHostCharts("):js.index("function windowLabel(")]
|
|
assert "hostChartSet(" not in fetch, "the poll path still builds charts for closed panels"
|
|
assert "state.history.set(" in fetch, \
|
|
"the response is not kept, so the first open is an empty box until the next poll"
|
|
assert "if (range === state.range) drawHostCharts(" in fetch, \
|
|
"a response for a window the operator just left is painted under the new label"
|
|
assert "drawHostCharts(" in fetch, "an open panel is no longer repainted on refresh"
|
|
|
|
|
|
def test_a_pruned_host_takes_its_charts_toggle_with_it(client) -> None:
|
|
"""A host that leaves the payload must leave no orphan control in the grid.
|
|
|
|
The toggle is a child of the panel, which is why `card.remove()` is enough — but that is
|
|
only true while the markup is built inside `hostCard()`, so the placement is pinned here
|
|
along with the three per-host Maps the panel owns.
|
|
"""
|
|
js = client.get("/static/page-system.js").text
|
|
card = js[js.index("function hostCard("):js.index("function memorySub(")]
|
|
assert 'data-k="charts"' in card, "the toggle is no longer part of the panel"
|
|
pruner = js[js.index("function pruneHosts("):js.index("// ---- snapshot")]
|
|
assert "state.hosts.get(name).remove()" in pruner
|
|
for owned in ("state.charts", "state.payloads", "state.history"):
|
|
assert f"{owned}.delete(name)" in pruner, f"{owned} outlives the panel it belongs to"
|
|
|
|
|
|
def test_the_charts_toggle_is_a_quiet_disclosure_not_a_new_component(client) -> None:
|
|
"""The collapse adds no palette and no second vocabulary (UI conventions).
|
|
|
|
Open/closed is carried by a caret *shape* as well as a colour step, the summary is muted
|
|
rather than accented, and it introduces no `.sev-*` family — the same rule the hero
|
|
numbers live by, checked here because a `<summary>` is the first control on the page a
|
|
designer might be tempted to grade.
|
|
"""
|
|
css = client.get("/static/dash.css").text
|
|
block = css[css.index("details.charts-box { "):css.index(".host-charts {")]
|
|
assert "details.charts-box > summary::before" in block, "the toggle has no caret shape"
|
|
assert 'details.charts-box[open] > summary::before' in block, \
|
|
"open and closed differ only in colour"
|
|
assert "var(--text-faint)" in block, "the closed toggle is not muted"
|
|
assert ".sev-" not in block and "var(--accent)" not in block, "the toggle invented a palette"
|
|
assert 'class="charts-box"' in client.get("/static/page-system.js").text, \
|
|
"the stylesheet styles a class the page no longer renders"
|
|
|
|
|
|
def test_an_expanded_chart_stack_does_not_grow_its_collapsed_siblings(client) -> None:
|
|
"""One host open must not stretch the hosts beside it into blank cards.
|
|
|
|
`/system` is the same shape `/llm` had to fix: an expanded panel beside a collapsed one
|
|
grows the collapsed card to match, and a blank panel reads as "no data" (the misreading
|
|
the unconfigured rules exist to prevent). `.grid.items-start` already exists for exactly
|
|
this, so the page uses it rather than growing a rule of its own.
|
|
"""
|
|
html = _code(client.get("/system").text)
|
|
hosts_class = re.findall(r'<div class="([^"]*)" id="hosts">', html)
|
|
assert hosts_class == ["grid auto items-start"], \
|
|
f"the host grid stretches its cards again: {hosts_class}"
|
|
assert ".grid.items-start" in client.get("/static/dash.css").text, \
|
|
"the rule the host grid relies on is gone"
|
|
|
|
|
|
def test_every_system_chart_names_itself_for_assistive_tech(client) -> None:
|
|
"""A 120px SVG is silence to a screen reader unless the page names it.
|
|
|
|
The phase-03 accessibility pass (task 04) found the per-host charts drawing anonymous
|
|
plots: the legend is a sibling element and labels nothing, so the `<svg>` itself must
|
|
carry `role="img"` + `aria-label` that says which host and which metrics. Pinned on the
|
|
served bytes because there is no JS test runner (a locked anchor).
|
|
"""
|
|
charts_js = client.get("/static/charts.js").text
|
|
# the shared LineChart applies the name to the svg, and only when the caller opted in
|
|
assert "if (this.opts.ariaLabel)" in charts_js, "LineChart stopped honouring ariaLabel"
|
|
assert "setAttribute('role', 'img')" in charts_js
|
|
assert "setAttribute('aria-label', String(this.opts.ariaLabel))" in charts_js
|
|
|
|
js = client.get("/static/page-system.js").text
|
|
# all three slots name themselves, and each name starts with the host the panel belongs
|
|
# to — an aria-label that cannot say *which machine* is not an accessible name here
|
|
assert js.count("ariaLabel: `") == 3, "a per-host chart slot lost its accessible name"
|
|
for slot in ("utilisation", "network traffic", "GPU busy"):
|
|
assert f"${{host.name}} \u2014 {slot}" in js or f"${{host.name}} — {slot}" in js, \
|
|
f"the {slot} chart has no host-scoped accessible name"
|
|
|
|
|
|
# ---- event feeds: a bounded, paginated window ----------------------------- #
|
|
#
|
|
# Every `ul.events` feed is one component (`Dash.eventsPager`) because a pager that
|
|
# works on only one page is the bug this shape prevents: the `event` table grows for as long
|
|
# as the fleet runs, and an unbounded feed turns `/system` into a scroll to nowhere
|
|
# (`TODO.md` L6 — "All the events tables need to be scrollable and paginated"). There is no
|
|
# JS test runner (a locked anchor), so what is pinned here is the shape every feed must keep:
|
|
# one shared renderer, a fetch bigger than one page, a footer that counts and labels itself,
|
|
# and a container that scrolls instead of growing the page.
|
|
|
|
# page -> (its script, the `source=` the poller stores that page's events under)
|
|
EVENT_FEEDS = {
|
|
"/system": ("page-system.js", "system"),
|
|
"/disks": ("page-disks.js", "disks"),
|
|
"/llm": ("page-llm.js", "llm"),
|
|
"/ups": ("page-ups.js", "ups"),
|
|
}
|
|
|
|
|
|
def _event_feed_path(js: str) -> str:
|
|
"""Just the `refreshEvents()` half of a page script — the feed's whole code path, comments out."""
|
|
js = _code(js)
|
|
return js[js.index("async function refreshEvents("):]
|
|
|
|
|
|
def _feed_fetch_url(js: str) -> str:
|
|
"""The `/api/events` URL a feed asks for, quoted or templated."""
|
|
found = re.search(r"fetchJSON\([`'](/api/events[^`']*)[`']\)", _event_feed_path(js))
|
|
assert found, "this page no longer fetches the event feed"
|
|
return found.group(1)
|
|
|
|
|
|
def test_the_event_feeds_are_one_shared_pager_not_one_per_page(client) -> None:
|
|
"""`Dash.eventsPager()` renders every feed, and no page grows its own pager.
|
|
|
|
One copy per page is one chance per page to keep the old behaviour — and `/system` is the
|
|
one whose feed was already too tall when this was written down. `/disks` joined as a
|
|
fourth call site of the same helper, not as a fifth implementation of it.
|
|
"""
|
|
app_js = _code(client.get("/static/app.js").text)
|
|
assert app_js.count("function eventsPager(") == 1
|
|
exports = client.get("/static/app.js").text.rsplit("return {", 1)[1].split("}", 1)[0]
|
|
assert "eventsPager" in exports, "Dash.eventsPager is not exported"
|
|
|
|
for path, (script, _) in EVENT_FEEDS.items():
|
|
js = _code(client.get(f"/static/{script}").text)
|
|
assert "Dash.eventsPager($('#events')" in js, f"{path} does not render its feed through the pager"
|
|
assert "function eventsPager(" not in js, f"{path} invented its own pager"
|
|
assert "events.slice(" not in js, f"{path} windows its own events instead of asking the pager"
|
|
assert "row:" in _event_feed_path(js), f"{path} passes no row renderer"
|
|
|
|
|
|
def test_the_pager_pages_50_events_and_fetches_more_than_one_page(client) -> None:
|
|
"""The two numbers that make a client-side pager work, and the relation between them.
|
|
|
|
`EVENT_PAGE_SIZE` is the 50 the operator asked for; `EVENT_FEED_LIMIT` is what the feed
|
|
asks `/api/events` for. A limit at or below the page size is a pager that silently never
|
|
shows — there is never a second page to reach — and the feeds used to fetch 20/25/40.
|
|
"""
|
|
app_js = client.get("/static/app.js").text
|
|
page_size = re.search(r"const EVENT_PAGE_SIZE = (\d+);", app_js)
|
|
feed_limit = re.search(r"const EVENT_FEED_LIMIT = (\d+);", app_js)
|
|
assert page_size and int(page_size.group(1)) == 50, "a feed no longer pages 50 events"
|
|
assert feed_limit, "the feeds' event limit is no longer a named constant"
|
|
assert int(feed_limit.group(1)) > int(page_size.group(1)), \
|
|
"a fetch at or below one page means page 2 can never exist"
|
|
|
|
for path, (script, source) in EVENT_FEEDS.items():
|
|
url = _feed_fetch_url(client.get(f"/static/{script}").text)
|
|
assert "EVENT_FEED_LIMIT" in url, f"{path} hardcodes its own event limit"
|
|
assert f"source={source}" in url, f"{path} asks for the wrong event source"
|
|
|
|
|
|
def test_the_pager_footer_counts_and_labels_its_controls(client) -> None:
|
|
"""The footer counts (`1\u201350 of 137`), and prev/next are labelled and disabled at the ends.
|
|
|
|
The count is what tells the operator the window is a window; a control that stays live at
|
|
the last page claims there is more history than there is.
|
|
"""
|
|
app_js = _code(client.get("/static/app.js").text)
|
|
assert "${first}\u2013${last} of ${total}" in app_js, "the footer no longer counts"
|
|
assert 'data-pager="prev"' in app_js and 'data-pager="next"' in app_js
|
|
assert "prev.setAttribute('aria-label'" in app_js
|
|
assert "next.setAttribute('aria-label'" in app_js
|
|
assert "prev.disabled = pager.page === 0" in app_js
|
|
assert "next.disabled = pager.page >= pager.pages - 1" in app_js
|
|
# pagination only applies when there is more than one page of data
|
|
assert "foot.hidden = !paginating" in app_js
|
|
|
|
css = client.get("/static/dash.css").text
|
|
assert ".events-pager {" in css, "the pager footer has no style"
|
|
assert "button.ghost:disabled" in css, "a disabled pager control looks live"
|
|
|
|
|
|
def test_the_event_feeds_scroll_instead_of_growing_the_page(client) -> None:
|
|
"""The `<ul>` is the scrollable element, bounded in `dash.css`; the page stays short.
|
|
|
|
Pinned on the served bytes because there is no JS test runner (a locked anchor): a
|
|
stylesheet edit that drops `max-height` is invisible until the feed is long, and by then
|
|
the page is massive again.
|
|
"""
|
|
css = client.get("/static/dash.css").text
|
|
block = css[css.index("ul.events {"):css.index("}", css.index("ul.events {"))]
|
|
assert "max-height" in block and "overflow-y: auto" in block, \
|
|
"the event feed is unbounded again — the page grows with history"
|
|
|
|
for path in EVENT_FEEDS:
|
|
html = client.get(path).text
|
|
assert '<ul class="events" id="events"' in html, f"{path} lost its feed's id"
|
|
|
|
|
|
def test_the_pager_returns_to_page_1_when_the_events_change(client) -> None:
|
|
"""New events reset the window; the same events do not.
|
|
|
|
Resetting on every poll would snap `next` back within one interval, so page 2 could never
|
|
be read; never resetting lands an operator on page 3 of a list that changed underneath
|
|
them. The page key is the set of events, not the clock.
|
|
"""
|
|
app_js = _code(client.get("/static/app.js").text)
|
|
body = app_js[app_js.index("function eventsPager("):]
|
|
assert "signature !== pager.signature" in body, "the pager no longer notices new events"
|
|
reset = body[body.index("signature !== pager.signature"):]
|
|
assert reset.index("pager.page = 0") < reset.index("paintEventFeed"), \
|
|
"new events do not reset the page before the feed is drawn"
|
|
assert "if (pager.markup !== markup)" in app_js, \
|
|
"every poll rewrites 50 rows and throws away the reader's scroll position"
|
|
|
|
|
|
def test_system_event_feed_queries_the_source_transitions_are_stored_under(client) -> None:
|
|
"""Host transitions are stored under the *dashboard* name, not the sample source.
|
|
|
|
`poller.py` logs `Event(ts, self.dashboard, …)` — host jobs write `system`, the way
|
|
the UPS job writes `ups` — while *samples* use `host`. The `/system` feed queried
|
|
`source=host` since the first commit and therefore always rendered "no transitions
|
|
yet", even with a real outage (a gateway disconnect) sitting in the store. Pinned on
|
|
the served bytes because there is no JS test runner (a locked anchor).
|
|
"""
|
|
js = client.get("/static/page-system.js").text
|
|
feed = _event_feed_path(js)
|
|
url = re.search(r"fetchJSON\([`'](/api/events[^`']*)[`']\)", feed)
|
|
assert url, "the /system page no longer fetches the event feed"
|
|
assert "source=system" in url.group(1), \
|
|
"the /system event feed must ask for the source the poller stores events under"
|
|
assert "source=host" not in url.group(1), \
|
|
"source=host matches no stored event — the feed is empty forever"
|
|
|
|
|
|
def test_system_footnote_prints_the_window_it_actually_measured(client) -> None:
|
|
"""`Δ<window>` is the page's promise that a rate is a measurement (invariant 2), so the
|
|
number behind it has to come from the measurement.
|
|
|
|
The `window_seconds` beside the fleet is `HOST_INTERVAL_SECONDS`, while the window behind a
|
|
rate is the estimator's `RATE_WINDOW_SECONDS` — equal at the defaults, which is why the
|
|
footnote could quote the poll interval and look right. Set `RATE_WINDOW_SECONDS=30` and it
|
|
would have labelled every rate on the page with a window it never used. The footnote now
|
|
reads each host's own measured `window_seconds`, and prints a range when they disagree.
|
|
"""
|
|
js = client.get("/static/page-system.js").text
|
|
assert "function windowLabel(" in js, "the footnote lost its measured-window helper"
|
|
footnote = js[js.index("$('#foot-note').textContent"):js.index("await Promise.all(chartJobs)")]
|
|
assert "windowLabel(" in footnote
|
|
assert "host.window_seconds" in footnote, \
|
|
"the footnote is quoting the poll interval as a measurement again"
|
|
# a footnote that says "window" and a page that says "poll" must stay two separate claims
|
|
assert "poll ${Math.round(state.pollMs / 1000)}s" in footnote
|
|
|
|
|
|
def test_no_per_core_visual_library_survives_the_page(client) -> None:
|
|
"""The per-core heat bars left `/system` and `charts.js` with them (PLAN.md *Dropped
|
|
visuals* is about what the pages draw, not what the store keeps).
|
|
|
|
`renderCores` had one caller — the hero-card per-core block deleted in task 02 — so
|
|
keeping it would be dead markup generating unstyled `.core` spans; `.cores` was the CSS
|
|
that block needed. Both are checked absent here so neither can come back as a
|
|
half-change, while `sparkline`/`LineChart` stay for phase 04 and the UPS page.
|
|
"""
|
|
assert "renderCores" not in client.get("/static/charts.js").text
|
|
css = client.get("/static/dash.css").text
|
|
assert ".cores" not in css
|
|
for asset in ASSETS:
|
|
assert "core_usage" not in _code(client.get(asset).text), f"{asset} plots per-core"
|
|
# what phase 04 and the other pages still chart from
|
|
for name in ("LineChart", "sparkline", "renderLegend"):
|
|
assert name in client.get("/static/charts.js").text
|
|
|
|
|
|
def test_sparkline_ships_the_fixed_scale_and_verdict_options(client) -> None:
|
|
"""Phase 04 turns `/llm` cells into sparkline rows, so the shared `sparkline()` must
|
|
behave like `llm_monitor`'s `drawSpark` (the task-02 contract, NOTES-chart-set).
|
|
|
|
Pinned on the served bytes because there is no JS test runner (a locked anchor). What
|
|
matters is not the option names but the semantics behind each one: a percent metric
|
|
that silently rescaled to its own data would make an 8 % KV cache and a 92 % spike the
|
|
same squiggle, and a stroke coloured from a page-invented threshold would break the
|
|
one-colour-language rule the same way.
|
|
"""
|
|
charts_js = client.get("/static/charts.js").text
|
|
body = charts_js[charts_js.index("function sparkline("):charts_js.index("window.DashCharts")]
|
|
|
|
# the phase grep criterion, as a test: the options are really in the shipped file
|
|
hits = [ln for ln in charts_js.splitlines()
|
|
if re.search(r"fixedMax|align|severity", ln)]
|
|
assert len(hits) >= 3, "the sparkline upgrade is not in the served charts.js"
|
|
for opt in ("fixedMax", "align", "labels", "severity", "minPoints", "span", "format",
|
|
"minSpan"):
|
|
assert opt in body, f"sparkline() lost its `{opt}` option"
|
|
|
|
# fixedMax: the data clamps to the axis, never the axis to the data
|
|
assert "Math.min(Math.max(v, lo), hi)" in body, \
|
|
"a fixedMax sparkline rescales to its own data again"
|
|
# align:'right' grows within a real time window (span seconds), not a sample count
|
|
assert "x1 - o.span" in body, "align:'right' forgot the window it grows inside"
|
|
# verdict colour comes from the theme tokens only; unrated is faint grey, never green
|
|
for token in ("cssVar('--ok')", "cssVar('--warn')", "cssVar('--fail')",
|
|
"cssVar('--text-faint')"):
|
|
assert token in body, f"sparkline() stopped taking its colour from {token}"
|
|
assert ".sev-" not in body, "sparkline invented a parallel severity vocabulary"
|
|
# the `now / max` foot llm_monitor prints, via the caller's format callback
|
|
assert "now ${fmt(last.value)}" in body and "max ${fmt(dataMax)}" in body
|
|
# never a confident line through one point, and the foot quotes data, not the padded axis
|
|
assert "minPoints: 2" in body and "no history" in body
|
|
# ASSUMPTION (task 02): no animation at all, so prefers-reduced-motion holds by design
|
|
assert "transition:" not in body and "animation:" not in body
|
|
|
|
|
|
# ---- the battery glyph (phase 05 task 02) ---------------------------------
|
|
#
|
|
# `DashCharts.battery()` is the one visual in this app whose *shape* is the data: the fill
|
|
# is the charge. That makes the honesty rules heavier here than anywhere else — a fill
|
|
# rounded up to look reassuring, a colour invented for an unrated cell, or a state only
|
|
# readable as green-vs-red would each turn a glance at the battery into a wrong claim.
|
|
# There is no JS test runner (a locked anchor), so the construction is pinned on the served
|
|
# bytes and shown in a browser (`data/logs/ups-glyph-*.png`).
|
|
|
|
def _battery_js(client: TestClient) -> str:
|
|
"""The glyph's code as served (state table + body), comments stripped.
|
|
|
|
Prose cannot satisfy these assertions, and the phrase table the accessible name is
|
|
built from is part of the component, so the slice starts above `function battery`.
|
|
"""
|
|
charts = client.get("/static/charts.js").text
|
|
body = charts[charts.index("const BATTERY_STATES"):charts.index("window.DashCharts")]
|
|
assert "function battery(container, opts)" in body
|
|
return _code(body)
|
|
|
|
|
|
def test_battery_glyph_is_exported_from_the_shared_chart_library(client) -> None:
|
|
"""The phase's own grep criterion, as a test: exactly one glyph, and it is public."""
|
|
charts = client.get("/static/charts.js").text
|
|
assert charts.count("function battery") == 1
|
|
exports = charts[charts.index("window.DashCharts"):]
|
|
assert re.search(r"\bbattery\b", exports), "DashCharts.battery is not exported"
|
|
# the wrapper the stylesheet sizes, added by the component rather than each caller
|
|
assert ".battery {" in _css(client) and ".battery svg" in _css(client)
|
|
|
|
|
|
def test_battery_fill_is_the_charge_and_is_never_padded(client) -> None:
|
|
"""2 % must look nearly empty, and 0 % must leave an empty shell, not a stray pixel."""
|
|
body = _battery_js(client)
|
|
# clamped to the scale, and the clamp is the only thing that touches the number: a
|
|
# minimum visible width would be a reassuring battery over a flat one
|
|
assert "Math.max(0, Math.min(100," in body
|
|
assert "(pctValue / 100) * inner.w" in body, "the fill stopped being a fraction of charge"
|
|
assert not re.search(r"Math\.max\(\s*[0-9.]*[1-9]\s*,", body), \
|
|
"the charge fill got a minimum width — a small charge would look bigger than it is"
|
|
# nothing is drawn for a cell at zero, and the plate it fills is a clip, so a low fill
|
|
# cannot leak outside the rounded interior
|
|
assert "pctValue > 0" in body
|
|
assert "<clipPath" in body and 'clip-path="url(#' in body
|
|
# an empty cell is the sunken plate, never a transparent hole in the card
|
|
assert "cssVar('--bg-sunken')" in body
|
|
|
|
|
|
def test_battery_colours_come_from_the_verdict_tokens_only(client) -> None:
|
|
"""The verdict paints; an unrated cell draws a grey shape and no colour at all."""
|
|
body = _battery_js(client)
|
|
for token in ("cssVar('--ok')", "cssVar('--warn')", "cssVar('--fail')"):
|
|
assert token in body, f"the glyph stopped taking its colour from {token}"
|
|
# no verdict → muted outline, and a measured charge in faint grey at half strength: a
|
|
# shape, not a judgement (an unknown number must never look green)
|
|
assert "cssVar('--text-muted')" in body and "cssVar('--text-faint')" in body
|
|
assert ' opacity=".5"' in body, "an unrated charge is drawn with the same weight as a rated one"
|
|
assert ".sev-" not in body, "the glyph invented a parallel severity vocabulary"
|
|
assert "severityFor(" not in body, "the glyph compares the charge to a threshold itself"
|
|
|
|
|
|
def test_battery_states_are_readable_without_colour(client) -> None:
|
|
"""Every state has a *shape*, so the glyph survives monochrome and colour-blindness.
|
|
|
|
on battery = dashed outline, charging = bolt, unknown = a dash pattern through the
|
|
body, online = a plain solid shell. A word the page did not intend falls back to
|
|
`unknown` rather than to a confident "on line".
|
|
"""
|
|
body = _battery_js(client)
|
|
assert 'BATTERY_STATES.indexOf(o.state) >= 0' in body, \
|
|
"an unrecognised state would be drawn as some other state"
|
|
assert 'state === \'onbattery\'' in body and 'stroke-dasharray="7 5"' in body, \
|
|
"the on-battery outline is no longer dashed"
|
|
assert "state === 'charging'" in body and 'd="M13 2 L3 14' in body, "the charging bolt is gone"
|
|
assert "state === 'unknown'" in body and 'stroke-dasharray="5 6"' in body, \
|
|
"an unknown charge no longer draws a dash through the body"
|
|
# and the unknown case really is *no fill*: the shape has to say it on its own
|
|
assert "pctValue != null" in body
|
|
|
|
|
|
def test_battery_glyph_names_itself_and_can_say_it_once(client) -> None:
|
|
"""`role="img"` + `<title>` built from the inputs, with an explicit `aria-hidden` opt-out."""
|
|
body = _battery_js(client)
|
|
assert 'role="img"' in body and "<title>" in body and "aria-label=" in body
|
|
for phrase in ("Charge ", " percent", "Charge unknown", "on line", "on battery",
|
|
"charging", "state unknown"):
|
|
assert phrase in body, f"the accessible name lost {phrase!r}"
|
|
# the label is a UPS name from upsd, so it goes through the one escaper
|
|
assert body.count("escapeHtml(name)") == 2, "<title> or aria-label stopped escaping the label"
|
|
assert 'aria-hidden="true"' in body, "the decorative opt-out for a duplicate hero number is gone"
|
|
|
|
|
|
def test_battery_glyph_scales_with_its_container_and_never_animates(client) -> None:
|
|
"""One `viewBox` + `preserveAspectRatio` for a phone and a 27" monitor; no motion at all."""
|
|
body = _battery_js(client)
|
|
assert 'viewBox="0 0 ${W} ${H}"' in body
|
|
assert 'preserveAspectRatio="xMidYMid meet"' in body, "the glyph stopped scaling with its box"
|
|
assert "ResizeObserver" not in body, "the glyph redraws on resize instead of scaling"
|
|
assert "transition" not in body and "animation" not in body
|
|
css = _code(_css(client))
|
|
wrapper = re.search(r"\.battery \{([^}]*)\}", css)
|
|
inner = re.search(r"\.battery svg \{([^}]*)\}", css)
|
|
assert wrapper and inner, "dash.css no longer sizes the glyph box"
|
|
assert "width: 100%" in wrapper.group(1) and "max-width" in wrapper.group(1), \
|
|
"the glyph is not capped inside its container"
|
|
assert "width: 100%" in inner.group(1) and "height: auto" in inner.group(1), \
|
|
"the svg would stretch instead of keeping the viewBox ratio"
|
|
both = wrapper.group(1) + inner.group(1)
|
|
assert "transition" not in both and "animation" not in both
|
|
|
|
|
|
# ---- the `/ups` hero row (phase 05 task 03) ------------------------------
|
|
#
|
|
# The card used to be nine equal `.metric` cells in which charge, load and runtime were the
|
|
# same size as the battery amperage. These pin the shape that replaced it on the served
|
|
# bytes (a locked anchor: no JS test runner), because the whole point of the row is a size
|
|
# and colour hierarchy a later "tidy up" can quietly flatten back to nine equal cells.
|
|
|
|
def _ups_card_js(client: TestClient) -> str:
|
|
"""The card markup as served, from `upsCard()` to `renderCard()`."""
|
|
js = client.get("/static/page-ups.js").text
|
|
return js[js.index("function upsCard("):js.index("function renderCard(")]
|
|
|
|
|
|
def test_ups_card_opens_with_the_glyph_and_three_heroes(client) -> None:
|
|
"""Charge / Load / Runtime are the card's three largest elements, next to the glyph.
|
|
|
|
`data-sev` names the verdict path each slot colours itself from, the same convention the
|
|
`/system` host card uses — so "which number is coloured by what" is readable in the
|
|
markup, and a rename that disconnects the page from `/api/ups/snapshot` fails here
|
|
instead of shipping a card of `n/a`.
|
|
"""
|
|
card = _ups_card_js(client)
|
|
assert '<div class="ups-hero">' in card, "the hero row left the top of the card"
|
|
assert 'data-k="glyph"' in card, "the battery glyph is not part of the hero row"
|
|
slots = re.findall(r'<div class="hero[^"]*" data-k="([^"]+)"', card)
|
|
assert slots == ["charge", "load", "runtime"], \
|
|
"the hero row is no longer charge / load / runtime in the agreed order"
|
|
for hook in ("severity.charge_pct", "severity.load_pct", "severity.battery_runtime"):
|
|
assert hook in card, f"the card no longer names {hook} as a colour source"
|
|
# the hero row is the first thing in the card body, above the health checks and commands
|
|
assert card.index("ups-hero") < card.index('data-k="checks"')
|
|
assert card.index("ups-hero") < card.index('data-k="commands"')
|
|
|
|
|
|
def test_ups_heroes_take_their_colour_from_the_payload(client) -> None:
|
|
"""Three `Dash.hero()` calls, three payload verdicts, no JS threshold.
|
|
|
|
The charge hero carries **no** `.bar`: the glyph beside it is that same value drawn as a
|
|
shape, and one value drawn twice in one row is the pattern this app avoids (load, which
|
|
has no other shape, keeps its bar). Runtime prose comes from `runtimeText()` alone, and
|
|
a runtime nobody estimated is passed as `null` so `hero()` renders the faint, uncoloured
|
|
unknown rather than the string `n/a` styled like a measurement.
|
|
"""
|
|
js = client.get("/static/page-ups.js").text
|
|
body = js[js.index("function renderCard("):js.index("// ---- command dialog")]
|
|
assert body.count("Dash.hero(q(") == 3, "the hero row is not three hero numbers"
|
|
for verdict in ("verdicts.charge_pct", "verdicts.load_pct", "verdicts.battery_runtime"):
|
|
assert verdict in body, f"{verdict} never reaches a hero"
|
|
charge = body[body.index("Dash.hero(q('charge')"):body.index("Dash.hero(q('load')")]
|
|
assert "bar" not in charge, "charge grew a second shape when the glyph already is one"
|
|
load = body[body.index("Dash.hero(q('load')"):body.index("Dash.hero(q('runtime')")]
|
|
assert "bar: true" in load, "load lost its bar"
|
|
assert "runtimeText(n.battery_runtime)" in body, "runtime prose bypasses runtimeText()"
|
|
runtime = body[body.index("Dash.hero(q('runtime')"):]
|
|
assert "n.battery_runtime == null ? null" in runtime, \
|
|
"an unmeasured runtime is rendered as a measurement"
|
|
assert "severityFor(" not in body
|
|
|
|
|
|
def test_ups_secondary_electrical_row_is_demoted_not_deleted(client) -> None:
|
|
"""Line in, output, battery volts and amps, battery °C: still there, visibly second.
|
|
|
|
These are the numbers you reach for *while* the UPS is on battery, so they were demoted
|
|
rather than folded into the disclosure — and they stay one step down (smaller, muted) so
|
|
the three verdicts remain the largest elements, which is also what keeps the card from
|
|
growing so tall that two stop fitting side by side on a laptop.
|
|
"""
|
|
card = _ups_card_js(client)
|
|
assert 'class="metrics ups-electrical"' in card
|
|
for key in ("line", "out", "batt", "amps", "temp"):
|
|
assert f'data-k="{key}"' in card, f"the electrical row lost {key}"
|
|
assert card.index("ups-electrical") > card.index("ups-hero")
|
|
css = _code(_css(client))
|
|
rule = re.search(r"\.ups-electrical \.metric \.value\.sm \{([^}]*)\}", css)
|
|
assert rule, "dash.css no longer styles the secondary row"
|
|
assert "var(--text-muted)" in rule.group(1), "the secondary row is not muted"
|
|
sizes = re.findall(r"font-size: ([\d.]+)rem", rule.group(1))
|
|
assert sizes and float(sizes[0]) < 1.3, "the secondary row competes with the heroes"
|
|
# `n/a` is fainter than a muted reading — neither real driver reports a battery temperature
|
|
null_rule = re.search(r"\.ups-electrical \.metric \.value\.sm\.is-null \{([^}]*)\}", css)
|
|
assert null_rule and "var(--text-faint)" in null_rule.group(1), \
|
|
"an unmeasured secondary value is not faint, so it reads like a number"
|
|
|
|
|
|
def test_ups_glyph_state_comes_from_the_api_fields(client) -> None:
|
|
"""`glyphState()` maps `on_battery` + `charger_status` — and `full` is **not** charging.
|
|
|
|
The mapping is presentation, so it lives in the page (task 02's assumption), but it is
|
|
still a mapping the real drivers exercise: neither UPS here exports
|
|
`battery.charger.status`, so the fallback that reads a state the API rated `ok` is what
|
|
draws a solid shell on both of them. Nothing here re-reads `ups.status` —
|
|
`test_the_page_does_not_re_parse_nut_status_flags` holds that line.
|
|
"""
|
|
js = _code(client.get("/static/page-ups.js").text)
|
|
glyph = js[js.index("function glyphState("):js.index("function drawGlyph(")]
|
|
assert "item.on_battery" in glyph and "'onbattery'" in glyph
|
|
assert "charger === 'chg'" in glyph and "'charging'" in glyph
|
|
assert "charger === 'dchg'" in glyph, "a discharging charger token is on battery too"
|
|
assert "charger === 'full'" in glyph and "'online'" in glyph, \
|
|
"a full, online battery must not be drawn as charging"
|
|
assert "item.states" in glyph, "a driver with no rich status lost its state entirely"
|
|
assert "'unknown'" in glyph, "an unknown state now degrades to a confident 'on line'"
|
|
|
|
|
|
def test_ups_glyph_repaints_only_when_something_visible_changes(client) -> None:
|
|
"""Charge jitters by a tenth of a percent every poll; the `<svg>` does not rebuild on it.
|
|
|
|
Rebuilding would re-resolve the theme tokens and re-announce the accessible name every
|
|
`NUT_INTERVAL_SECONDS` for a difference no eye can see. `decorative: true` is the
|
|
component's documented opt-out — the charge hero and the state chips already say
|
|
"Charge 100 percent, on line", and a screen reader should not hear it twice.
|
|
"""
|
|
js = _code(client.get("/static/page-ups.js").text)
|
|
draw = js[js.index("function drawGlyph("):js.index("function upsCard(")]
|
|
assert "DashCharts.battery(" in draw
|
|
assert "cell.dataset.drawn === drawn" in draw, "the glyph rebuilds on every poll"
|
|
assert "decorative: true" in draw, "the glyph duplicates the hero number in a11y"
|
|
assert "glyphState(item)" in draw
|
|
|
|
|
|
def test_ups_hero_row_makes_room_for_the_glyph(client) -> None:
|
|
"""The glyph gets a real column, and the row stacks rather than clipping on a phone.
|
|
|
|
The track floor is the widest runtime prose the page prints (`9h 59m`) — a number that
|
|
cannot wrap needs the floor to be its own width, or the row grows a clipped `1h 28`.
|
|
"""
|
|
css = _code(_css(client))
|
|
row = re.search(r"\.ups-hero \{([^}]*)\}", css)
|
|
assert row, "dash.css lost the UPS hero row"
|
|
assert "minmax(112px, 156px) minmax(0, 1fr)" in row.group(1), "the glyph has no column"
|
|
assert "align-items: center" in row.group(1)
|
|
heroes = re.search(r"\.ups-hero \.heroes \{([^}]*)\}", css)
|
|
assert heroes and "minmax(min(140px, 100%), 1fr)" in heroes.group(1), \
|
|
"a hero track can now clip the widest runtime prose"
|
|
at = css.index("@media (max-width: 620px)")
|
|
block = css[at:at + 240]
|
|
assert ".ups-hero" in block and "minmax(0, 1fr)" in block, "the row never stacks"
|
|
|
|
|
|
def test_every_ups_chart_names_itself_for_assistive_tech(client) -> None:
|
|
"""The four trend plots on `/ups` carry an accessible name, like every other chart here.
|
|
|
|
`AGENTS.md` states the convention as *every* chart names itself: `LineChart` puts
|
|
`role="img"` + `aria-label` on the `<svg>` when the caller passes `ariaLabel`, because
|
|
the legend is a sibling that labels nothing and an unlabelled plot is silence. `/system`
|
|
(phase 03) and `/llm` (phase 04) were pinned; these four were the stragglers, so the
|
|
rule is now pinned on every page that draws one.
|
|
"""
|
|
js = client.get("/static/page-ups.js").text
|
|
charts = js[js.index("function makeCharts("):js.index("function points(")]
|
|
assert charts.count("ariaLabel:") == 4, "a `/ups` chart lost its accessible name"
|
|
for phrase in ("Battery charge", "Battery runtime", "Battery and line voltage", "UPS load"):
|
|
assert phrase in charts, f"the {phrase!r} chart names nothing readable"
|
|
|
|
|
|
# ---- the `/llm` layout (phase 04 task 03) ---------------------------------
|
|
#
|
|
# The drift this phase fixes was a 14-cell grid where everything was the same size and
|
|
# nothing showed recent shape. These tests pin the *shape* the page committed to on the
|
|
# served bytes (a locked anchor: no JS test runner), so a regression to "all cells equal"
|
|
# fails a test rather than a screenshot.
|
|
|
|
def test_llm_page_colours_from_the_payload_not_from_js_thresholds(client) -> None:
|
|
"""`page-llm.js` paints the KV verdict `severity.py` produced; it owns no threshold.
|
|
|
|
Both `severityFor(value, 75, 90)` call sites are gone (the task's explicit work item):
|
|
the KV row takes `severity` from the rated ratio cell, the fleet KV bar was deleted
|
|
with the summed card, and every unrated number — throughput above all, because 0 tok/s
|
|
on an idle server is healthy — draws no colour at all.
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
assert "severityFor(" not in js, "the LLM page re-invented a threshold"
|
|
assert "severityFrom" in js
|
|
# the KV verdict comes from the payload cells `sources/llm.py` rates
|
|
kv = js[js.index("case 'cache':"):js.index("case 'ttft':")]
|
|
assert "kv_cache_usage_ratio" in kv and "gpu_cache_usage" in kv and "severity" in kv
|
|
|
|
|
|
def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
|
|
"""Decode and prompt are the largest elements per engine; the rest is visibly secondary.
|
|
|
|
The hero row is `Dash.hero()` markup (2.4rem via `.hero`), built from the `hero: true`
|
|
flags in `CELLS`, and throughput carries **no** severity — `severity.py` has no
|
|
throughput row and the page invents none (the neutral-ink decision in NOTES-chart-set,
|
|
mirroring the `/system` NET precedent). The provenance tag stays in the hero's `.sub`
|
|
with the exact `source` as the tooltip: removing it to look cleaner is a regression.
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
panel = js[js.index("function enginePanel("):js.index("function renderPanel(")]
|
|
assert 'data-hero="${cell.key}"' in panel, "the hero row is not flag-driven markup"
|
|
assert '<div class="heroes">' in panel, "the hero row left the engine panel"
|
|
heroes = re.findall(r"{ key: '(\w+)'[^}]*?hero: true", js)
|
|
assert heroes == ["decode", "prompt"], "the hero row is no longer the two throughputs"
|
|
# driven by flags, not fourteen identical cells. Phase 07 task 03 lifted the summary rows
|
|
# out of the panel into the stats strip above it, so what is left here is the pair plus
|
|
# the latency rows the strip does not print — never a fourteen-cell grid again.
|
|
assert "SPARK_CELLS" in js and "ALARM_CELLS" in js
|
|
assert js.count("spark: '") >= 2, "the panel lost the latency sparkline rows it kept"
|
|
assert js.count("hero: true") == 2, "the hero row is no longer exactly two numbers"
|
|
# the hero call must not pass a severity — unrated by design
|
|
hero_call = js[js.index("const node = hero("):js.index("if (node) node.title")]
|
|
assert "severity" not in hero_call, "throughput invented a verdict"
|
|
# provenance intact: the tag is generated from the cell's `source`, never hardcoded
|
|
assert "provenance" in js and "Δ" in js
|
|
assert "node.title = value.source" in js
|
|
|
|
|
|
def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client) -> None:
|
|
"""Each panel row draws its own sparkline over the *selected* window and prints mean
|
|
**and** p95 without a hover.
|
|
|
|
The window is honest time (`align: 'right'` + `span`) and the p95 comes from the
|
|
snapshot's histogram cells — p95 needs no stored series. The `only` scopes are what keep
|
|
a gufo panel from showing a latency mystery and a vLLM strip card from showing a
|
|
spec-decode row it cannot fill.
|
|
|
|
Phase 07 task 03 moved the percent and count rows to the stats strip, so the plotted
|
|
percent series left the cell list and the axis guarantee moved with it: the KV trend
|
|
chart pins `yMax: 100` (an 8 % window that once held a 92 % spike must not read flat),
|
|
and the strip converts its 0-1 ratios to per cent in `cellValue()` — a bar cannot
|
|
autoscale, so it cannot lie that way either. The `fixedMax`/`minSpan` options stay
|
|
plumbed through `drawSparks()` and are pinned on the component itself by
|
|
`test_sparkline_ships_the_fixed_scale_and_verdict_options`: a count sparkline that comes
|
|
back must not get an axis finer than one request.
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
sparks = js[js.index("function drawSparks("):js.index("function renderFleet(")]
|
|
assert "DashCharts.sparkline(" in sparks
|
|
for opt in ("align: 'right'", "span: state.range", "fixedMax", "labels: { max: true }"):
|
|
assert opt in sparks, f"cell sparklines lost `{opt}`"
|
|
# the one percent series still plotted owns a fixed 0-100 axis; seconds stay ms
|
|
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
|
|
assert "yMax: 100" in charts, "the KV trend chart no longer pins its 0-100 axis"
|
|
assert "scale: 1000" in js and "cell.scale || 1" in sparks
|
|
# every spark cell declares its history series in the markup it is built from
|
|
assert 'data-spark data-metric="${cell.spark}"' in js
|
|
# percentile foot lines: mean big in the row, p95 beneath, n/a when nothing measured
|
|
assert "function p95Value(" in js
|
|
for name in ("time_to_first_token_seconds__p95", "time_per_output_token_seconds__p95",
|
|
"request_queue_time_seconds__p95"):
|
|
assert name in js, f"{name} no longer feeds a foot line"
|
|
assert "p95 n/a" in js
|
|
assert 'data-foot' in js
|
|
# engine-scoped rows: gufo never grows a TTFT row, vLLM never a spec-accept row
|
|
assert "only: ['vllm']" in js and "only: ['gufo']" in js
|
|
scoped = js[js.index("function sparkCellsFor("):js.index("function enginePanel(")]
|
|
assert "cell.only" in scoped
|
|
assert "minSpan: cell.minSpan" in sparks, "cell minSpan never reaches the sparkline"
|
|
|
|
|
|
def test_llm_sparklines_and_charts_cost_one_batched_history_request(client) -> None:
|
|
"""One `/api/llm/history` call per refresh feeds charts *and* every sparkline row.
|
|
|
|
The endpoint answers a metric list for every engine, and unknown names come back as
|
|
empty series on purpose — so the page asks for the full cell set once instead of one
|
|
XHR per cell, and the two series phase 04 added (`tpot_seconds`, `spec_accept_rate`)
|
|
must be in that list or those rows are permanent "no history" slots.
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
# comments aside, exactly one fetch hits the endpoint: one request, every metric
|
|
calls = re.findall(r"fetchJSON\(\s*`/api/llm/history", _code(js))
|
|
assert len(calls) == 1, "the page makes more than one history request again"
|
|
assert "HISTORY_SERIES.join" in js and "SPARK_CELLS.map((cell) => cell.spark)" in js
|
|
request = js[js.index("async function refreshHistory("):js.index("function drawSparks(")]
|
|
assert "metrics=${HISTORY_SERIES.join" in request
|
|
assert "range=${state.range}" in request
|
|
# and the endpoint really answers every name the page asks for
|
|
for name in ("tpot_seconds", "spec_accept_rate", "ttft_seconds", "queue_seconds",
|
|
"kv_cache_usage", "requests_running", "requests_waiting",
|
|
"prefix_cache_hit_ratio"):
|
|
assert name in HISTORY_METRICS, f"llm.{name} is not stored, so its spark is dead"
|
|
|
|
|
|
def test_llm_alarm_cells_exist_only_while_there_is_an_incident(client) -> None:
|
|
"""`preemptions` / `device_lost` are absent at zero and red chips above zero.
|
|
|
|
The rule is the *existence of an incident*, not a graded threshold — `sources/llm.py`
|
|
already treats any non-zero `gufo_device_lost_total` as one. The chip reuses the shared
|
|
`.chip.fail` vocabulary; nothing new is painted, and a container emptied at zero means
|
|
the row does not exist rather than showing a permanent ignored 0.
|
|
|
|
Phase 07 task 03 renders them on the engine's strip card (the summary you read first)
|
|
instead of inside the panel, so the two `alarm: true` rows live in `STRIP_ROWS` and
|
|
`ALARM_CELLS` filters them out of it — counted with the comments stripped, because the
|
|
docstring on that table explains the flag by name.
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
alarms = js[js.index("function renderAlarms("):js.index("function makeCharts(")]
|
|
assert "class=\"chip fail\"" in alarms, "the alarm chip left the shared fail vocabulary"
|
|
assert "value.value > 0" in alarms
|
|
assert "host.hidden = chips.length === 0" in alarms
|
|
assert 'data-alarm="${cell.key}"' in alarms
|
|
assert _code(js).count("alarm: true") == 2
|
|
assert "ALARM_CELLS = STRIP_ROWS.filter" in js, "the alarm rows are not strip rows"
|
|
# the panel no longer carries a second container for the same incident
|
|
panel = js[js.index("function enginePanel("):js.index("function renderPanel(")]
|
|
assert 'data-k="alarms"' not in panel, "the alarm chips are drawn twice"
|
|
strip = js[js.index("function stripCard("):js.index("function renderStripCard(")]
|
|
assert 'data-k="alarms"' in strip, "the alarm chips left the strip card"
|
|
|
|
|
|
# ---- the `/llm` stats strip (phase 07 task 03) ---------------------------
|
|
#
|
|
# `llm_monitor`'s shared strip (`index.html` ~L743), ported: one card per configured engine
|
|
# between the fleet block and the panels. Two things make it a summary instead of a second
|
|
# set of panels, and both are pinned below: every row is a cell `/api/llm/snapshot` really
|
|
# publishes, and no row is printed in the strip *and* in the panel under it.
|
|
|
|
#: row key -> the cells `cellValue()` resolves it from (`sessions.*` is the gufo slot count,
|
|
#: which is not a metrics cell). The served `STRIP_ROWS` table must match these keys exactly.
|
|
STRIP_ROW_PATHS: dict[str, tuple[str, ...]] = {
|
|
"prompt": ("prompt_tokens_per_second",),
|
|
"decode": ("decode_tokens_per_second",),
|
|
"tokens_total": ("tokens_total",),
|
|
"slots_busy": ("sessions.list", "sessions.active"),
|
|
"running": ("requests_processing", "num_requests_running"),
|
|
"waiting": ("requests_deferred", "num_requests_waiting"),
|
|
"cache_hit": ("prefix_cache_hit_ratio", "prompt_cache_hit_ratio"),
|
|
"spec_accept": ("spec_accept_rate",),
|
|
"queue_mean": ("request_queue_time_seconds__mean",),
|
|
"preemptions": ("num_preemptions_total",),
|
|
"device_lost": ("device_lost_total",),
|
|
}
|
|
|
|
#: rows that are incidents, not stats: absent where the engine does not count them.
|
|
ALARM_ROW_KEYS = ("preemptions", "device_lost")
|
|
|
|
|
|
def _strip_rows(js: str) -> dict[str, str | None]:
|
|
"""The served `STRIP_ROWS` table: row key -> its `only` scope (`None` = every engine)."""
|
|
body = _code(js)
|
|
start = body.index("const STRIP_ROWS = [")
|
|
table = body[start:body.index("\n];", start)]
|
|
rows: dict[str, str | None] = {}
|
|
for chunk in table.split("{ key: '")[1:]:
|
|
scoped = re.search(r"only: \[([^\]]*)\]", chunk)
|
|
rows[chunk.split("'")[0]] = scoped.group(1) if scoped else None
|
|
return rows
|
|
|
|
|
|
def _strip_render_path(js: str) -> str:
|
|
"""Just `renderStripCard()` — the whole code path that paints one strip card."""
|
|
js = _code(js)
|
|
return js[js.index("function renderStripCard("):js.index("function pruneEngines(")]
|
|
|
|
|
|
def test_llm_stats_strip_sits_between_the_fleet_block_and_the_panels(client) -> None:
|
|
"""The strip is there, ships hidden, names itself, and appears only with engines.
|
|
|
|
An unconfigured engine gets no card and no strip (invariant 3) — the section is empty in
|
|
the markup and `page-llm.js` unhides it only when the payload has an engine to draw.
|
|
"""
|
|
html = _code(client.get("/llm").text)
|
|
assert 'id="strip"' in html, "/llm lost the per-engine stats strip"
|
|
assert html.index('id="fleet"') < html.index('id="strip"') < html.index('id="engines"'), \
|
|
"the strip is no longer between the fleet block and the engine panels"
|
|
start = html.index('<section class="strip"')
|
|
tag = html[start:html.index(">", start)]
|
|
assert "hidden" in tag, "the strip no longer ships hidden"
|
|
assert 'aria-label="Engine summary"' in tag, "the strip is a region that names nothing"
|
|
js = client.get("/static/page-llm.js").text
|
|
assert "$('#strip').hidden = payloads.length === 0" in js
|
|
|
|
|
|
def test_llm_strip_card_is_one_card_per_engine_of_shared_vocabulary(client) -> None:
|
|
"""`stripCard()` = `.card` + `.kv` + `.bar` + `.chip.fail`: a layout, not a new component.
|
|
|
|
Dot, name and age in the head (reachability, the same distinction the panels and the
|
|
fleet block draw), the KV bar, the rows keyed by payload field, the alarm container.
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
card = js[js.index("function stripCard("):js.index("function renderStripCard(")]
|
|
assert "card.className = 'card stat'" in card
|
|
assert "card.dataset.engine = engine" in card, "the card stopped naming its engine"
|
|
assert '<span class="dot idle"></span>' in card
|
|
assert '<span class="faint mono updated"></span>' in card, "the card lost its timestamp"
|
|
assert '<span class="bar" data-kv-bar><i></i></span>' in card, "the KV bar left `.bar`"
|
|
assert '<dl class="kv">' in card, "the strip rows left the shared `.kv` vocabulary"
|
|
assert 'data-row="${row.key}"' in card
|
|
assert 'data-k="alarms"' in card
|
|
# the wrapper is why the shared `.kv` grid has to move *inside* it: applied to the
|
|
# wrappers, `max-content 1fr` lays two rows side by side and the card reads as a
|
|
# column of orphaned labels (found in the browser, not in a test failure)
|
|
css = client.get("/static/dash.css").text
|
|
section = css[css.index("/* ---- engine stats strip"):css.index("/* ---- fleet totals")]
|
|
assert ".strip .kv { margin: 0; grid-template-columns: 1fr; }" in section
|
|
assert ".strip .kv > div {" in section
|
|
assert "grid-template-columns: max-content minmax(0, 1fr)" in section
|
|
refresh = js[js.index("async function refresh("):js.index("async function refreshEvents(")]
|
|
assert "state.strips.get(payload.engine)" in refresh, "the strip rebuilds its cards per poll"
|
|
assert "$('#strip').appendChild(strip)" in refresh
|
|
assert "renderStripCard(strip, payload, stateRow)" in refresh
|
|
prune = js[js.index("function pruneEngines("):js.index("function makeCharts(")]
|
|
assert "state.strips" in prune and "state.panels" in prune, \
|
|
"an engine that leaves the payload leaves a card behind"
|
|
|
|
|
|
def test_llm_strip_scopes_its_rows_to_the_engines_that_publish_them(client) -> None:
|
|
"""gufo gets slots and speculative decoding; vLLM gets running and queue time."""
|
|
rows = _strip_rows(client.get("/static/page-llm.js").text)
|
|
assert rows["slots_busy"] == "'gufo'" and rows["spec_accept"] == "'gufo'"
|
|
assert rows["running"] == "'vllm'" and rows["queue_mean"] == "'vllm'"
|
|
assert rows["waiting"] is None and rows["cache_hit"] is None and rows["tokens_total"] is None
|
|
js = client.get("/static/page-llm.js").text
|
|
scoped = js[js.index("function stripRowsFor("):js.index("function stripCard(")]
|
|
assert "row.only" in scoped and "row.only.indexOf(engine)" in scoped
|
|
|
|
|
|
def test_llm_strip_rows_name_cells_the_engines_publish(client, metrics_text) -> None:
|
|
"""The strip renders what the payload sends — no row whose cell no engine produces.
|
|
|
|
`STRIP_ROW_PATHS` mirrors the JS table; every metric cell is checked against the
|
|
capture-parsed metrics of each engine that would render that row, so a rename in
|
|
`sources/llm.py` fails here instead of printing a permanent `n/a` in the browser, and a
|
|
row scoped to one engine cannot leak into the other's card.
|
|
"""
|
|
rows = _strip_rows(client.get("/static/page-llm.js").text)
|
|
assert set(rows) == set(STRIP_ROW_PATHS), \
|
|
f"the served strip rows drifted: {set(rows) ^ set(STRIP_ROW_PATHS)}"
|
|
parsed = {"gufo": parse_gufo_metrics(metrics_text("gufo")),
|
|
"vllm": parse_vllm_metrics(metrics_text("vllm"))}
|
|
for key, only in rows.items():
|
|
paths = STRIP_ROW_PATHS[key]
|
|
if paths[0].startswith("sessions."):
|
|
continue # its own payload object; pinned on the JS below
|
|
shown = {engine: metrics for engine, metrics in parsed.items()
|
|
if not only or engine in only}
|
|
present = [engine for engine in shown if any(path in shown[engine] for path in paths)]
|
|
if key in ALARM_ROW_KEYS:
|
|
# an alarm the engine does not count simply is not there: the chip renders only
|
|
# while non-zero, so absence *is* the zero state rather than a dead row
|
|
assert present, f"no engine publishes {'/'.join(paths)} for the {key} alarm"
|
|
continue
|
|
assert len(present) == len(shown), (
|
|
f"the {sorted(set(shown) - set(present))} card would print `{key}` as n/a forever "
|
|
f"({'/'.join(paths)} is not published)")
|
|
|
|
|
|
def test_llm_strip_slot_count_counts_published_slots_and_never_invents_zero(client) -> None:
|
|
"""`slots busy` is a count of `/slots` rows; when /slots did not answer it is `n/a`.
|
|
|
|
Counting rows the server published is a count of published facts. An empty list from a
|
|
server that never answered is not a measurement of an idle engine, so the row falls back
|
|
to the published `sessions.active` carrying the API's own `detail` string (rule 2).
|
|
"""
|
|
js = _code(client.get("/static/page-llm.js").text)
|
|
slots = js[js.index("case 'slots_busy':"):js.index("case 'cache':")]
|
|
assert "sessions.source" in slots and "sessions.list" in slots
|
|
assert "is_processing" in slots, "the count ignores the authoritative busy verdict"
|
|
assert "sessions.detail" in slots, "the fallback lost the API's own provenance"
|
|
|
|
|
|
def test_llm_strip_kv_bar_is_coloured_by_the_payload_verdict_only(client) -> None:
|
|
"""The strip's KV bar is the page's one rated cell: `severity` in, class out.
|
|
|
|
`sources/llm.py` rates `llm.kv_cache_usage_pct`, the ratio→percent conversion the panel
|
|
cell used to make rides with it, and `Dash.severityFrom` is the only thing that decides a
|
|
colour — no threshold in the page, and the stylesheet adds no palette (`ok` stays quiet
|
|
ink, exactly like the `.cell` rows and the fleet table).
|
|
"""
|
|
body = _strip_render_path(client.get("/static/page-llm.js").text)
|
|
assert "severityFrom(kv.severity)" in body
|
|
assert "cellValue(engine, metrics, sessions, 'cache')" in body
|
|
assert "Math.max(0, Math.min(100, kv.value))" in body, "the fill stopped clamping"
|
|
assert body.count("severityFrom") == 1, "a second strip row invented a verdict"
|
|
css = client.get("/static/dash.css").text
|
|
section = css[css.index("/* ---- engine stats strip"):css.index("/* ---- fleet totals")]
|
|
assert ".stat-bar.warn .value { color: var(--warn); }" in section
|
|
assert ".stat-bar.fail .value { color: var(--fail); }" in section
|
|
assert ".stat-bar.ok" not in section, "a healthy KV reading must stay quiet ink"
|
|
assert ".prov:empty" in section, "an empty provenance tag draws a pill of nothing"
|
|
|
|
|
|
def test_llm_strip_throughput_carries_provenance_without_colour(client) -> None:
|
|
"""`prompt tok/s` and `decode tok/s` keep the `Δ<window>` tag and stay ink.
|
|
|
|
The tag is generated from the cell's `source` (never hardcoded), the exact source rides
|
|
in the row's tooltip, and `severity.py` rates no throughput — so nothing here may paint
|
|
it (0 tok/s on an idle engine is healthy).
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
body = _strip_render_path(js)
|
|
assert "row.rate ? provenance(value) : ''" in body
|
|
assert "notes.join" in body and "value.source" in body
|
|
assert "severityFor" not in js
|
|
|
|
|
|
def test_llm_strip_and_panel_print_no_value_twice(client) -> None:
|
|
"""One value, one home — the UPS hero row's rule, applied across two components.
|
|
|
|
Task 03 lifted sessions, queue depth, KV pressure, the cache-hit and spec-accept percents,
|
|
the lifetime token counter and the alarm chips out of the panel cell list rather than
|
|
drawing them twice. The two throughput heroes are the exemption the task names: the panel
|
|
keeps its hero row (so does `llm_monitor`, which carries the pair in both places), and
|
|
everything else appears once. Queue time moved with its p95 foot, so the move lost nothing.
|
|
"""
|
|
js = _code(client.get("/static/page-llm.js").text)
|
|
start = js.index("const CELLS = [")
|
|
panel_keys = set(re.findall(r"\{ key: '(\w+)'", js[start:js.index("\n];", start)]))
|
|
assert panel_keys == {"decode", "prompt", "ttft", "tpot"}, \
|
|
f"the panel cell list changed shape: {sorted(panel_keys)}"
|
|
shared = set(_strip_rows(js)) & panel_keys
|
|
assert shared == {"decode", "prompt"}, f"printed in the strip and the panel: {sorted(shared)}"
|
|
body = _strip_render_path(js)
|
|
assert "p95Value(metrics, row.key)" in body, "queue time lost its p95 in the move"
|
|
|
|
|
|
def test_llm_strip_prints_no_peak_context_number(client) -> None:
|
|
"""The phase's assumption, pinned: `llm_monitor`'s peak-ctx row does not come over.
|
|
|
|
Its cell is `llamacpp:n_tokens_max`, a since-start maximum rather than a glance number,
|
|
and the task says this strip gets no peak-context cell. The series stays stored and
|
|
queryable from `/api/llm/history` — not plotting it is not removing it.
|
|
"""
|
|
js = _code(client.get("/static/page-llm.js").text)
|
|
assert "n_tokens_max" not in js, "/llm reads the peak-context gauge again"
|
|
assert not [key for key in _strip_rows(js) if "peak" in key]
|
|
|
|
|
|
def test_llm_fleet_block_leads_the_page_with_the_engine_sums(client) -> None:
|
|
"""The page opens with llm_monitor's aggregate block: the sums no panel can say.
|
|
|
|
Phase 04 retired these numbers to counts; the operator asked for the aggregate
|
|
throughput back (`TODO.md:5`), and phase 07 restores it in `llm_monitor`'s shape —
|
|
status dot, "all engines", an `N of M engines` tag, the note that the sums cover only
|
|
the engines answering now, then running / waiting / prompt / decode / total tokens.
|
|
The counts-only card it replaces (`#totals`, `f-live`, `f-active`) is gone, ids and all:
|
|
an element nothing writes is the orphan this repo pins against.
|
|
"""
|
|
html = _code(client.get("/llm").text)
|
|
assert 'id="fleet"' in html, "/llm lost the aggregate block"
|
|
assert html.index('id="fleet"') < html.index('id="engines"'), \
|
|
"the fleet block is no longer above the engine panels"
|
|
for part in ("fleet-dot", "fleet-count", "f-running", "f-waiting", "f-prompt",
|
|
"f-decode", "f-tokens"):
|
|
assert f'id="{part}"' in html, f"/llm lost the fleet block's {part}"
|
|
for words in ("all engines", "sums only engines answering now"):
|
|
assert words in html, f"/llm lost the fleet block's `{words}` line"
|
|
for gone in ('id="totals"', "f-live", "f-total", "f-active", "f-kv", "seam-note"):
|
|
assert gone not in html, f"/llm still carries {gone} from the retired counts card"
|
|
|
|
js = client.get("/static/page-llm.js").text
|
|
body = _fleet_render_path(js)
|
|
assert "block.hidden = total < 2" in body, "the fleet block no longer hides for one engine"
|
|
assert "hidden" in _fleet_tag(html), \
|
|
"the block no longer ships hidden, so a single engine would show a fleet row"
|
|
|
|
|
|
def test_llm_fleet_block_reads_only_paths_the_api_publishes(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Every number the block prints is a `fleet` key of `/api/llm/snapshot`.
|
|
|
|
The page sums nothing itself — `fleet_totals()` does — so the field table in
|
|
`page-llm.js` is a list of API paths, and a rename in `sources/llm.py` must fail here
|
|
rather than print a permanent `n/a` in the browser. Pinned with both engines configured
|
|
(the keys exist, empty-handed because nothing answered) and with neither (404
|
|
`configured: false` without dialling, invariant 3 — and the shipped block stays hidden).
|
|
"""
|
|
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
|
|
vllm_url=DEAD_URL) as test_client:
|
|
fleet = test_client.get("/api/llm/snapshot").json()["fleet"]
|
|
assert set(fleet) >= FLEET_BLOCK_PATHS, FLEET_BLOCK_PATHS - set(fleet)
|
|
# the served field table is exactly the sum rows, in llm_monitor's order
|
|
js = test_client.get("/static/page-llm.js").text
|
|
table = _fleet_field_metrics(js)
|
|
assert table == ["requests_active", "requests_waiting",
|
|
"prompt_tokens_per_second", "decode_tokens_per_second",
|
|
"tokens_total"], "the fleet row changed shape without the API"
|
|
assert set(FLEET_SUM_PATHS) == set(table), "the block totals something the API does not"
|
|
# and the tag's two numbers are API paths too, not arithmetic the page invented
|
|
body = _fleet_render_path(js)
|
|
assert "fleet.engines_total" in body and "fleet.engines_live" in body
|
|
|
|
with build_client(tmp_path, monkeypatch) as bare:
|
|
response = bare.get("/api/llm/snapshot")
|
|
assert response.status_code == 404
|
|
assert response.json()["configured"] is False
|
|
html = _code(bare.get("/llm").text)
|
|
assert 'id="fleet"' in html and "hidden" in _fleet_tag(html), \
|
|
"an unconfigured engine must not leave a fleet row on the page"
|
|
|
|
|
|
def test_llm_fleet_totals_drop_an_engine_that_stopped_answering(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""A total presented as current is current: a dead engine contributes nothing.
|
|
|
|
`fleet_totals()`'s docstring is the contract and the block is where it is read. The
|
|
injected payload survives the dead-port polls (a failed poll keeps the last good one), so
|
|
these are the numbers a browser would get with gufo answering and vLLM down — the live
|
|
engine's sums verbatim, `engines_live` one of two, and not one zero where a number the
|
|
dead engine never published would sit (rule 2).
|
|
"""
|
|
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
|
|
vllm_url=DEAD_URL) as test_client:
|
|
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
|
|
hub.jobs["llm:gufo"].state.payload = {
|
|
"engine": "gufo",
|
|
"metrics": {"decode_tokens_per_second": {"value": 41.0},
|
|
"prompt_tokens_per_second": {"value": 17.5},
|
|
"requests_processing": {"value": 2.0},
|
|
"requests_deferred": {"value": 1.0},
|
|
"tokens_total": {"value": 512.0}},
|
|
"sessions": {"active": 2, "list": []}}
|
|
fleet = test_client.get("/api/llm/snapshot").json()["fleet"]
|
|
assert fleet["engines_total"] == 2 and fleet["engines_live"] == 1
|
|
assert fleet["decode_tokens_per_second"] == pytest.approx(41.0)
|
|
assert fleet["prompt_tokens_per_second"] == pytest.approx(17.5)
|
|
assert fleet["requests_active"] == pytest.approx(2.0)
|
|
assert fleet["requests_waiting"] == pytest.approx(1.0)
|
|
assert fleet["tokens_total"] == pytest.approx(512.0)
|
|
|
|
# and with neither engine answering, every sum is null, never a confident 0
|
|
hub.jobs["llm:gufo"].state.payload = {"engine": "gufo", "metrics": {},
|
|
"sessions": {"active": None, "list": []}}
|
|
empty = test_client.get("/api/llm/snapshot").json()["fleet"]
|
|
assert empty["engines_live"] == 0
|
|
assert [empty[key] for key in FLEET_SUM_PATHS] == [None] * len(FLEET_SUM_PATHS)
|
|
|
|
|
|
def test_llm_fleet_block_proves_where_its_sums_come_from(client) -> None:
|
|
"""The seam tooltip, the excluded-engine tooltip, and an `n/a` with no unit.
|
|
|
|
Two provenance rules the rest of the app holds to, on the one block that adds engines
|
|
together: the prompt sum carries `fleet.seams` — the API's own words on why gufo's and
|
|
vLLM's prompt counters are not the same quantity — and the tag's tooltip names the
|
|
engines `fleet_totals()` left out, so a partial total says so in words and not only by
|
|
being dimmed. And a sum the API did not publish prints `n/a` with its unit cleared: "n/a
|
|
tok/s" would read as a measurement of nothing.
|
|
"""
|
|
js = _code(client.get("/static/page-llm.js").text)
|
|
body = _fleet_render_path(js)
|
|
assert "fleet.seams" in body, "the prompt sum lost the API's seam note"
|
|
assert "not answering — excluded from every sum below" in body, \
|
|
"the tag no longer names the engines left out of the totals"
|
|
assert "value == null ? ''" in body, "an n/a sum can still print a unit"
|
|
|
|
assert "value == null ? ''" in body, "an n/a sum can still print a unit"
|
|
|
|
|
|
def test_llm_fleet_block_colours_no_number(client) -> None:
|
|
"""No verdict on the aggregate row — and no dimming of anything but the tag.
|
|
|
|
`severity.py` rates no throughput and no queue depth (0 tok/s on an idle fleet is
|
|
healthy), so the block renders ink: the render path touches no `severity`, calls no
|
|
`Dash.hero()`, and the stylesheet section for the block reaches for no verdict token.
|
|
The one coloured thing on it is the shared **reachability** dot — the same distinction
|
|
the engine panels draw between their numbers and their header dot — and the partial-fleet
|
|
signal is the shared `.stale` dim on the count tag, which is opacity, not a verdict.
|
|
"""
|
|
js = _code(client.get("/static/page-llm.js").text)
|
|
body = _fleet_render_path(js)
|
|
assert "severity" not in body, "the fleet block invented a verdict"
|
|
assert "hero(" not in body, "the fleet block drew a hero number it cannot rate"
|
|
assert "setDot(" in body and "markStale(tag" in body, \
|
|
"a dead engine no longer dims the tag or darkens the block's dot"
|
|
|
|
css = client.get("/static/dash.css").text
|
|
section = _css_section(css, "fleet totals")
|
|
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
|
|
assert token not in section, f"the fleet block coloured a number with {token}"
|
|
assert ".fleet-head" in section and ".fleet-note" in section, \
|
|
"the fleet block's head row lost its styling"
|
|
|
|
|
|
#: Elements the browser treats as empty, so they never open a nesting level.
|
|
_VOID_TAGS = {"meta", "link", "br", "img", "input", "hr", "source", "track", "wbr"}
|
|
|
|
|
|
def _depth_at(html: str, needle: str) -> int:
|
|
"""How deeply the `needle` sits inside `<main>` (1 = a direct child of it).
|
|
|
|
The `~` in `#banners:empty ~ #fleet` only resolves between **siblings**, so the gap rule
|
|
depends on the markup as much as on the stylesheet. Counting tags is enough here and keeps
|
|
the check on the served bytes: no HTML parser, no dependency, and comments are already
|
|
gone by the time `_code` hands the page over.
|
|
"""
|
|
body = html[html.index("<main>"):html.index(needle, html.index("<main>"))]
|
|
depth = 0
|
|
for closing, tag, _attrs, self_closed in re.findall(
|
|
r"<(/?)([a-zA-Z][\w-]*)([^>]*?)(/?)>", body):
|
|
if self_closed or tag.lower() in _VOID_TAGS:
|
|
continue
|
|
depth += -1 if closing else 1
|
|
return depth
|
|
|
|
|
|
def test_llm_section_stack_keeps_the_shared_vertical_rhythm(client) -> None:
|
|
"""Fleet block → engine summaries → engine panels sit the shared `.85rem` apart.
|
|
|
|
`TODO.md:4`: the fleet section had no spacing between it and the engine summaries. Every
|
|
boundary below it was already handled inline — the section under `#strip` carries
|
|
`margin-top:.85rem` — so `#fleet` and `#strip` skipped the rhythm and the top of the page
|
|
read as one unbroken wall. Measured in Chromium at the 16px root (`data/logs/`
|
|
`llm-fleet-spacing*.png`), before → after:
|
|
|
|
| gap | before | after |
|
|
|-----|--------|-------|
|
|
| `#fleet` → `#strip` | **0 (flush)** | 13.59px |
|
|
| `#strip` → engine panels | 13.59px | 13.59px |
|
|
| banner slot → `#fleet`, no banner | 13.59px | 13.59px |
|
|
| banner slot → `#fleet`, one banner up | 13.59px | 13.59px (`margin-top: 0px`) |
|
|
|
|
One rendered distance moves, and it is the one the operator named. What the *shape* of the
|
|
two rules buys is that no other state can put it back:
|
|
|
|
* `#strip`'s gap is **unconditional**, because below two engines `#fleet` is `[hidden]`
|
|
(`display:none !important`, so it contributes nothing) and the strip is then the first
|
|
visible section: the 13.59px below the banner slot in that state is now the strip's own
|
|
margin rather than one borrowed from a slot above it, which is what `#fleet ~ #strip`
|
|
would have been;
|
|
* `#fleet`'s gap is guarded by `#banners:empty`, so the block owns its top gap exactly when
|
|
nothing occupies the slot above it, and a banner's own `margin-bottom:.85rem` stays the
|
|
single gap under it. An unguarded `#fleet { margin-top }` is a second rule for one gap:
|
|
margin collapsing hides it at `.85rem`/`.85rem` today, and it doubles the moment either
|
|
margin is changed. The empty slot keeps its inline bottom margin, so head → fleet stays
|
|
the 1rem + .85rem the other two banner pages (`/`, `/ups`) already render.
|
|
|
|
Pinned on the served bytes because a stylesheet "cleanup" that drops either rule is
|
|
invisible in a diff and only shows up as a page that reads worse.
|
|
"""
|
|
raw = _css(client)
|
|
section = _css_section(raw, "fleet totals")
|
|
assert "#strip { margin-top: .85rem; }" in section, \
|
|
"the fleet block and the engine summaries render flush again (TODO.md:4)"
|
|
assert "#banners:empty ~ #fleet { margin-top: .85rem; }" in section, (
|
|
"the gap above the fleet block stopped belonging to the block: a banner-less page "
|
|
"would then have it only as long as the empty banner slot keeps its inline margin")
|
|
|
|
# one rule per element: a second one (or an unguarded `#fleet`) is the double gap
|
|
css = _code(raw)
|
|
for element, expected in (("#strip", {"#strip"}),
|
|
("#fleet", {"#banners:empty ~ #fleet"})):
|
|
selectors = {s.strip() for s in re.findall(rf"([^{{}}]*{element}[^{{}}]*)\{{", css)}
|
|
assert selectors == expected, (
|
|
f"dash.css styles {element} with {sorted(selectors)} — exactly one rule, the "
|
|
"guarded one, or the page grows a double gap or loses the single one")
|
|
|
|
html = _code(client.get("/llm").text)
|
|
for element in ("banners", "fleet", "strip"):
|
|
assert _depth_at(html, f'id="{element}"') == 1, (
|
|
f"`#{element}` is no longer a direct child of <main>, so the `~` gap rule cannot "
|
|
"match: rewrite the selectors for the new structure instead of dropping the gap")
|
|
assert html.index('id="banners"') < html.index('id="fleet"') < html.index('id="strip"') \
|
|
< html.index('id="engines"'), "the section stack was re-ordered"
|
|
|
|
# the inline half of the rhythm, which the two rules are written around
|
|
assert 'style="margin-bottom:.85rem"' in html[:html.index('id="fleet"')], (
|
|
"#banners lost the inline bottom margin that is the banner → fleet gap, and today the "
|
|
"only thing spacing the page head from the block")
|
|
after_strip = html[html.index('id="strip"'):]
|
|
assert after_strip.count('style="margin-top:.85rem"') == 3, \
|
|
"a section below the strip lost the inline gap the strip's rule hands off to"
|
|
|
|
|
|
# ---- the `/llm` gufo slot grid (phase 07 task 04) -------------------------
|
|
#
|
|
# `llm_monitor`'s slot cards (`index.html` ~L1179), rebuilt against the capture instead of a
|
|
# memory: one keyed card per gufo slot, every cell naming **one** key of
|
|
# `tests/fixtures/slots_gufo.json`. Two things are pinned below, because they are the two ways
|
|
# this component failed before: a card that reads a key the server does not publish (dashes
|
|
# forever), and a card that is rebuilt every poll so a busy slot flickers in front of you.
|
|
|
|
#: The keys a slot card reads, and where they live: `field` (and its optional `sub` /
|
|
#: `ratioAgainst` partners) index the row, or its first `next_token` entry when `path` says so.
|
|
SLOT_ROW_PATHS = ("field", "sub", "ratioAgainst")
|
|
|
|
|
|
def _slot_table(js: str) -> list[dict[str, str | None]]:
|
|
"""The served `SLOT_FIELDS` table, one dict per printed cell."""
|
|
body = _code(js)
|
|
start = body.index("const SLOT_FIELDS = [")
|
|
table = body[start:body.index("\n];", start)]
|
|
cells: list[dict[str, str | None]] = []
|
|
for chunk in table.split("{ key: '")[1:]:
|
|
cell: dict[str, str | None] = {"key": chunk.split("'")[0]}
|
|
for option in (*SLOT_ROW_PATHS, "path"):
|
|
found = re.search(rf"{option}: '([a-z_0-9]+)'", chunk)
|
|
cell[option] = found.group(1) if found else None
|
|
cells.append(cell)
|
|
return cells
|
|
|
|
|
|
def _slot_js(js: str) -> str:
|
|
"""The slot grid's whole code path, comments out: `slotValue()` to the end of the block."""
|
|
body = _code(js)
|
|
return body[body.index("function slotValue("):body.index("function makeCharts(")]
|
|
|
|
|
|
def _slot_markup(js: str) -> str:
|
|
"""Just `slotSection()` — the markup the grid, its label and its empty states are built from."""
|
|
body = _code(js)
|
|
return body[body.index("function slotSection("):body.index("function renderPanel(")]
|
|
|
|
|
|
def test_llm_gufo_panel_holds_the_slot_grid_under_its_label(client) -> None:
|
|
"""`sessions (GET /slots)`, then the keyed grid, then the note about what it cannot contain.
|
|
|
|
The label is `llm_monitor`'s, the grid ships `hidden` (the empty state is a sentence, never
|
|
a card of nothing), and the note stays because a queued (deferred) request is not in
|
|
`/slots` at all — without that line an idle grid reads as "nothing waiting".
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
markup = _slot_markup(js)
|
|
assert "sessions (GET /slots)" in markup, "/llm lost the slot grid's label"
|
|
# label, grid, empty state, note — in that order, and the grid and the empty state both
|
|
# ship hidden so nothing paints before the payload says which of them is true
|
|
order = ['class="section-label" data-slots-label', "sessions (GET /slots)",
|
|
'<div class="slots" data-slots role="list" hidden>',
|
|
'class="slots-empty" data-slots-empty hidden',
|
|
'class="slots-note" data-slots-note hidden']
|
|
position = -1
|
|
for needle in order:
|
|
found = markup.index(needle)
|
|
assert found > position, f"`{needle}` is out of order in the slot section"
|
|
position = found
|
|
assert "queued (deferred) requests are not in /slots" in js, "the queued-request note is gone"
|
|
assert "renderSlotGrid(card, enginePayload)" in js, "nothing draws the grid any more"
|
|
# the label is a heading and the grid a list: a screen reader gets the block's name and
|
|
# its item count, which a stack of divs does not say
|
|
assert '<h4 class="section-label" data-slots-label>' in markup, \
|
|
"the slot section's label stopped being a heading"
|
|
assert 'data-slots role="list"' in markup, "the grid lost its list semantics"
|
|
# the grid's own markup: a keyed card, the state pill, and one dd per printed cell
|
|
card = js[js.index("function slotCard("):js.index("function slotSub(")]
|
|
assert "setAttribute('role', 'listitem')" in card, "the slot cards left the list"
|
|
assert 'data-slot-cell="${cell.key}"' in card, "the card cells are not table-driven"
|
|
assert 'data-pill' in card and 'data-mark' in card, "the card lost its state pill or glyph"
|
|
|
|
|
|
def test_llm_slot_grid_style_is_a_marker_and_not_a_verdict(client) -> None:
|
|
"""A busy slot is a border, a glyph and a word — `severity.py` rates no slot.
|
|
|
|
Phase 06 settled this argument for GPU utilisation on an LLM server: a slot generating
|
|
tokens is the job, so the accent colour says *this is the live one* and no `--ok/--warn/
|
|
--fail` token appears anywhere in the section. The glyph (`○`/`●`) and the pill's word mean
|
|
the marker survives monochrome, which is the battery glyph's rule applied here.
|
|
"""
|
|
css = client.get("/static/dash.css").text
|
|
section = _css_section(css, "engine slot grid")
|
|
for selector in (".slots {", ".slot {", ".slot.busy", ".slot-top", ".slot-id",
|
|
".slot-model", ".slot dt", ".slot dd", ".slots-empty", ".slots-note",
|
|
".section-label"):
|
|
assert selector in section, f"dash.css lost the {selector} rule"
|
|
assert "var(--accent)" in section, "the busy marker stopped using the non-verdict accent"
|
|
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
|
|
assert token not in section, f"the slot grid coloured a state with {token}"
|
|
js = client.get("/static/page-llm.js").text
|
|
assert "'\\u25cf'" in js or "'●'" in js, "the busy glyph left the card"
|
|
assert "processing" in _slot_js(js), "the pill stopped saying the state in a word"
|
|
|
|
|
|
def test_llm_slot_cells_each_name_one_key_the_capture_publishes(client) -> None:
|
|
"""Every printed cell traces to a key of `tests/fixtures/slots_gufo.json`.
|
|
|
|
One `field` per cell and no `a ?? b`: the deleted table's `slot.n_prompt ??
|
|
slot.prompt_tokens` guessing is what made it print dashes and then plausible noise. A key
|
|
the capture does not have fails here, and the alias the capture answers beside `id_task`
|
|
must not appear in the page at all — the contract names the canonical one.
|
|
"""
|
|
# comments out: this test bans names the prose is allowed to explain
|
|
js = _code(client.get("/static/page-llm.js").text)
|
|
cells = _slot_table(js)
|
|
assert cells, "the slot grid has no cell table to read"
|
|
assert [cell["key"] for cell in cells] == ["req", "prompt", "cache", "generated",
|
|
"remain", "ctx"], \
|
|
"the slot card's cell set changed without the capture"
|
|
for cell in cells:
|
|
assert cell["field"], f"the {cell['key']} cell names no row key"
|
|
|
|
rows = load_slots_capture()
|
|
assert rows, "the capture is empty, so this test would prove nothing"
|
|
for cell in cells:
|
|
for option in SLOT_ROW_PATHS:
|
|
key = cell[option]
|
|
if key is None:
|
|
continue
|
|
in_next = cell["path"] == "next_token"
|
|
required = REQUIRED_NEXT_TOKEN_FIELDS if in_next else REQUIRED_ROW_FIELDS
|
|
optional = OPTIONAL_NEXT_TOKEN_FIELDS if in_next else OPTIONAL_ROW_FIELDS
|
|
assert key in required or key in optional, (
|
|
f"the {cell['key']} cell reads {key!r}, which no contract table declares — the "
|
|
"page would be reading a field nobody pinned")
|
|
for row in rows:
|
|
source = row["next_token"][0] if in_next else row
|
|
if key in required:
|
|
assert key in source, (
|
|
f"the {cell['key']} cell reads {key!r}, which "
|
|
f"{'next_token[0]' if in_next else 'the row'} does not publish — this is "
|
|
"how the deleted table rendered dashes")
|
|
elif key in source: # optional: `prob` is on no captured row, and that is legal
|
|
assert isinstance(source[key], (int, float)), (
|
|
f"{key!r} is {source[key]!r}, which the cell cannot format")
|
|
|
|
assert cells[0]["field"] == CANONICAL_TASK_ID_FIELD, \
|
|
"the request cell stopped reading the canonical task id"
|
|
for alias in TASK_ID_ALIASES:
|
|
assert alias not in js, f"page-llm.js renders the {alias} alias the contract does not name"
|
|
# the old table's invented names stay invented, and so does its per-slot speed
|
|
for gone in ("n_predict", "decode_per_second", "cache_usable", "n_ctx_used"):
|
|
assert gone not in js, f"page-llm.js reads the invented {gone} again"
|
|
|
|
|
|
def test_llm_slot_card_prints_no_speed_it_cannot_measure(client) -> None:
|
|
"""No per-slot speed cell: the capture publishes no per-slot rate key.
|
|
|
|
`llm_monitor`'s grid has one, and so did the deleted table (`decode_per_second`) — which is
|
|
why it was dashes. A page-side Δn_decoded/Δt would be a measurement with no estimator, no
|
|
`Δ<window>` label and no `source` string, i.e. rule 2 with a confident number printed on it,
|
|
so the number stays on the engine's strip card and panel, where `rates.py` owns it.
|
|
"""
|
|
js = _code(client.get("/static/page-llm.js").text)
|
|
assert not [cell for cell in _slot_table(js)
|
|
if cell["key"] in {"speed", "tok_s", "decode"}], "a slot cell grew a rate"
|
|
assert "decode_per_second" not in js
|
|
|
|
|
|
def test_llm_slot_sentinels_print_a_word_or_na_never_a_negative_number(client) -> None:
|
|
"""`-1` is llama.cpp's "nothing in this slot", and it never becomes a countdown.
|
|
|
|
An idle slot answers `id_task: -1` and `next_token[0].n_remain: -1` (both in the capture).
|
|
The request cell says `idle` in a word; the remaining cell prints `n/a`. Neither prints a
|
|
negative number, and neither prints the `0` that would read as "one token from done".
|
|
"""
|
|
body = _slot_js(client.get("/static/page-llm.js").text)
|
|
assert "value < 0 ? 'idle'" in body, "the request cell prints the -1 sentinel again"
|
|
assert "value >= 0 ? value : null" in body, "the remaining cell prints the -1 sentinel"
|
|
|
|
|
|
def test_llm_slot_card_header_reads_only_fields_the_contract_declares(client) -> None:
|
|
"""The card top prints `model`, `speculative`, `id` and the state pair — and nothing else.
|
|
|
|
`id` keys the card (escaped once at build time), `is_processing` is the authoritative pill,
|
|
`state` labels a number the boolean cannot express, and `model` + `speculative` are the
|
|
footnote beside them. Each is declared in a contract table, so a rename fails there. The
|
|
user-content field a busy row carries (`prompt`) is the one key this page must never read.
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
header = js[js.index("function renderSlotCard("):js.index("function renderSlotGrid(")]
|
|
read = {"is_processing", "state", "model", "speculative"}
|
|
for key in read:
|
|
assert f"row.{key}" in header, f"the slot card header stopped reading {key}"
|
|
declared = set(REQUIRED_ROW_FIELDS) | set(OPTIONAL_ROW_FIELDS)
|
|
assert read <= declared, "the header reads a field no contract table declares"
|
|
for field in USER_CONTENT_FIELDS:
|
|
assert field not in header, f"the slot card prints the user's {field}"
|
|
assert "escapeHtml(id)" in js, "the slot id reached the markup unescaped"
|
|
|
|
|
|
def test_llm_slot_cards_are_keyed_created_once_and_patched_in_place(client) -> None:
|
|
"""A busy slot does not get rebuilt every poll — and a slot that leaves loses its card.
|
|
|
|
`appendChild` of the existing node keeps DOM order == slot order; everything else is a text
|
|
patch, so a poll cannot flicker a card or re-announce it to assistive tech for a difference
|
|
of a few tokens. An engine that leaves the payload loses its card map with its panel, or the
|
|
refs of removed nodes would be re-appended when it came back.
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
body = _slot_js(js)
|
|
assert "state.slots.get(enginePayload.engine)" in body, "the grid stopped keying its cards"
|
|
assert "if (!refs) {" in body and "refs = slotCard(id)" in body
|
|
assert "grid.appendChild(refs.root)" in body, "slot order is no longer kept by moving nodes"
|
|
assert "byId.delete(id)" in body, "a slot that left keeps its card forever"
|
|
assert "byId.forEach((refs) => refs.root.remove())" in body, \
|
|
"the empty state left cards on the page"
|
|
assert "setTxt(box.value" in body, "the cell values stopped being patched in place"
|
|
assert "innerHTML" not in body.split("function renderSlotCard(")[1], \
|
|
"a card is rebuilt from markup on a poll"
|
|
prune = js[js.index("function pruneEngines("):js.index("/* ---- the gufo slot grid")]
|
|
assert "state.slots.delete(engine)" in prune, "a pruned engine kept its slot cards"
|
|
|
|
|
|
def test_llm_slot_grid_says_why_there_are_no_rows(client) -> None:
|
|
"""Two honest empty states, and neither of them is a card or a 0-count claim.
|
|
|
|
`/slots` did not answer → the API's own `sessions.detail` sentence (the same words the
|
|
strip's session row carries). It answered with no rows → "engine reports 0 slots", which is
|
|
a fact about that answer. `sessions.source` is the gate that tells the two apart.
|
|
"""
|
|
body = _slot_js(client.get("/static/page-llm.js").text)
|
|
assert "Boolean(sessions.source)" in body, "the grid cannot tell no-answer from no-rows"
|
|
assert "engine reports 0 slots" in body
|
|
assert "sessions.detail ||" in body, "the no-answer state lost the API's own sentence"
|
|
assert "no /slots answer from this engine" in body, "the no-answer state can print undefined"
|
|
|
|
|
|
def test_llm_vllm_panel_says_there_is_no_per_slot_json(client) -> None:
|
|
"""vLLM gets one line and no grid — no fake rows for a server that cannot answer.
|
|
|
|
`sources/llm.py`'s `_sessions()` publishes counts for vLLM and says so in `detail`; the
|
|
panel turns that into the `llm_monitor` line, with the API's sentence as the tooltip. The
|
|
gate is the engine's name (`SLOT_ENGINES`), not `sessions.list.length`, so an engine that
|
|
has not answered yet still shows its label and the honest empty sentence.
|
|
"""
|
|
js = client.get("/static/page-llm.js").text
|
|
markup = _slot_markup(js)
|
|
branch = markup.split("if (SLOT_ENGINES.indexOf(engine) < 0) {")[1].split("}")[0]
|
|
assert 'class="slots-note" data-slots-note' in branch, "the vLLM line lost its element"
|
|
assert "NO_SLOT_JSON" in branch and "vLLM exposes no per-slot JSON" in js
|
|
assert '<div class="slots"' not in branch, "the vLLM panel grew a grid it cannot fill"
|
|
body = _slot_js(js)
|
|
assert body.startswith("function slotValue("), "the grid's code path moved"
|
|
assert "if (!grid) {" in body, "renderSlotGrid no longer leaves a panel with no grid alone"
|
|
assert "note.title = enginePayload.sessions.detail" in body, \
|
|
"the vLLM line stopped carrying the API's own words"
|
|
|
|
|
|
def test_the_captured_slot_rows_fill_the_cells_the_grid_prints(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The capture, through the API, into the keys the served cell table reads.
|
|
|
|
The fixture rows are injected as a live job payload (the `/slots` passthrough verbatim), so
|
|
this is the render path's input as the browser would receive it: `sessions.source` set (the
|
|
grid's gate), every row present, and every key `SLOT_FIELDS` names resolvable in it. A
|
|
rename in `sources/llm.py` or in gufo fails here rather than leaving a card of `n/a`s.
|
|
"""
|
|
rows = load_slots_capture()
|
|
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
|
|
vllm_url=DEAD_URL) as test_client:
|
|
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
|
|
hub.jobs["llm:gufo"].state.payload = {
|
|
"engine": "gufo", "metrics": {"requests_processing": {"value": 1.0}},
|
|
"sessions": {"active": 1, "list": rows, "source": "gufo /slots", "detail": None},
|
|
}
|
|
engines = test_client.get("/api/llm/snapshot").json()["engines"]
|
|
gufo = next(row for row in engines if row["engine"] == "gufo")
|
|
sessions = gufo["sessions"]
|
|
assert sessions["source"] == "gufo /slots", "the grid's gate is gone from the payload"
|
|
assert sessions["list"] == rows, "the rows are no longer the passthrough the grid reads"
|
|
|
|
cells = _slot_table(test_client.get("/static/page-llm.js").text)
|
|
for cell in cells:
|
|
for option in SLOT_ROW_PATHS:
|
|
key = cell[option]
|
|
if key is None:
|
|
continue
|
|
in_next = cell["path"] == "next_token"
|
|
required = REQUIRED_NEXT_TOKEN_FIELDS if in_next else REQUIRED_ROW_FIELDS
|
|
for row in sessions["list"]:
|
|
source = row["next_token"][0] if in_next else row
|
|
if key not in required and key not in source:
|
|
continue # an optional sub-line this build does not publish
|
|
value = source[key]
|
|
assert isinstance(value, (int, float)) and not isinstance(value, bool), (
|
|
f"the {cell['key']} cell reads {key!r} = {value!r}, which the number "
|
|
"formatter cannot print")
|
|
|
|
|
|
def test_llm_charts_are_secondary_and_named(client) -> None:
|
|
"""The four trend charts stay, at half height, *below* the engine panels, each named.
|
|
|
|
Half of the old 165px and moved after `#engines` is the "cells read first" decision;
|
|
every `LineChart` gets an `ariaLabel` because the legend is a sibling that labels
|
|
nothing (`/system` did this in phase 03, `/llm` never had).
|
|
"""
|
|
html = _code(client.get("/llm").text)
|
|
assert html.index('id="engines"') < html.index('id="chart-decode"'), \
|
|
"the charts are back on top of the engine panels"
|
|
for chart in ("chart-decode", "chart-prompt", "chart-kv", "chart-queue"):
|
|
assert f'id="{chart}"' in html, f"/llm lost the {chart} chart"
|
|
js = client.get("/static/page-llm.js").text
|
|
assert "height: 110" in js and "height: 165" not in js
|
|
charts = js[js.index("function makeCharts("):js.index("function pointsFor(")]
|
|
assert charts.count("ariaLabel:") == 4, "an /llm chart lost its accessible name"
|
|
|
|
|
|
def test_llm_engine_panels_are_not_stretched_to_their_tallest_sibling(client) -> None:
|
|
"""A 5-row gufo panel beside a 7-row vLLM panel keeps its own height.
|
|
|
|
Found while comparing against `llm_monitor`: the shorter engine card grew 172 px of
|
|
blank card, and a blank panel reads as "no data" — the exact misreading the unconfigured
|
|
rules exist to prevent. Pinned on the served markup so the `.grid.items-start` rule in
|
|
`dash.css` cannot silently become dead CSS (and the void cannot silently come back).
|
|
"""
|
|
html = _code(client.get("/llm").text)
|
|
engines_div = html[html.index('<div class="grid'):html.index('id="engines"') + 40]
|
|
assert "items-start" in engines_div, "the engine grid stretches its cards again"
|
|
css = client.get("/static/dash.css").text
|
|
assert ".grid.items-start" in css, "the rule the engine grid relies on is gone"
|
|
|
|
|
|
def test_overview_is_exactly_two_panels(client) -> None:
|
|
"""The landing page answers one question: what is wrong, and what is the fleet doing.
|
|
|
|
Asserted on the served bytes because there is no JS test runner (a locked anchor):
|
|
the summary cards and the raw event feed are what made this page busy, and once a
|
|
panel is gone a browser is the only thing that would notice its return.
|
|
"""
|
|
html = client.get("/").text
|
|
assert html.count("<section") == 2, "the overview grew a third panel"
|
|
assert 'id="alerts"' in html and 'id="alerts-clear"' in html
|
|
assert 'aria-live="polite"' in html
|
|
assert 'id="fleet-table"' in html
|
|
for gone in ("sys-cpu", "sys-mem", "sys-disk", "sys-gpus", "llm-decode", "llm-kv",
|
|
"ups-list", 'id="events"'):
|
|
assert gone not in html, f"the overview still renders {gone}"
|
|
# the header's source dots are chrome owned by app.js, not the deleted card dots
|
|
assert 'id="dot-system"' not in html and 'id="nav-status"' in html
|
|
|
|
|
|
def test_overview_alert_rows_link_to_the_dashboard_that_explains_them(client) -> None:
|
|
"""Every alert carries the `href` its row turns into a link; the page invents none."""
|
|
page_js = client.get("/static/page-index.js").text
|
|
assert "alert.href" in page_js
|
|
css = client.get("/static/dash.css").text
|
|
for selector in ("ul.alerts li.ok .src", "ul.alerts li.warn .src", "ul.alerts li.fail .src",
|
|
"table.data td.warn", "table.data td.fail"):
|
|
assert selector in css, f"dash.css lost {selector}"
|
|
# the swap that puts a live problem above the fleet table
|
|
assert 'data-state="alerting"' in css
|
|
|
|
|
|
def test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering(client) -> None:
|
|
"""A dead host keeps its last good numbers, so the row has to say they are stale.
|
|
|
|
`AGENTS.md` invariant 4: keep the payload and mark it stale, never zero it. The
|
|
overview's table is the one place those last good numbers still print, so it reads the
|
|
per-source verdicts `/api/summary` sends (`system.states`) and applies the shared
|
|
`.stale` class the other pages already use — no new palette, no JS threshold.
|
|
"""
|
|
page_js = client.get("/static/page-index.js").text
|
|
assert "system.states" in page_js, "the fleet table stopped reading reachability"
|
|
assert 'class="stale"' in page_js
|
|
assert ".stale" in client.get("/static/dash.css").text
|
|
|
|
|
|
# Which script owns which page, for the orphan check below. `charts.js` and `app.js` are
|
|
# loaded by every page, so an id either page code or shared chrome may read.
|
|
PAGE_SCRIPTS = {
|
|
"/": "page-index.js",
|
|
"/system": "page-system.js",
|
|
"/disks": "page-disks.js",
|
|
"/llm": "page-llm.js",
|
|
"/ups": "page-ups.js",
|
|
# `/settings` is audited like a dashboard even though it is not one: an id nothing reads is
|
|
# dead markup whatever page it sits on, and this page has five new ones.
|
|
SETTINGS_PAGE: "page-settings.js",
|
|
}
|
|
# `/login` is deliberately **not** in this table even though it is audited (phase 11): a
|
|
# signed-in client asking for it gets a 303 to the dashboard, so this helper's own
|
|
# redirect-following `get` would audit `/`'s markup and call it `/login`. The login page's
|
|
# orphan-`id`, CDN and masthead checks live in `tests/test_auth_api.py`, fetched unsigned, and
|
|
# `/static/page-login.js` is in `ASSETS` above.
|
|
|
|
SHARED_SCRIPTS = ("app.js", "charts.js")
|
|
|
|
|
|
def test_no_page_carries_an_id_nothing_reads(client) -> None:
|
|
"""An id no script queries, no stylesheet styles and no attribute points at is dead markup.
|
|
|
|
The summary cards were retired this way — a `<span id="sys-cpu">` left behind after its
|
|
writer was deleted is the same mistake in slower motion. Asserted on served bytes: the
|
|
browser console is otherwise the only thing that would notice.
|
|
"""
|
|
scripts = {name: client.get(f"/static/{name}").text for name in
|
|
(*SHARED_SCRIPTS, *PAGE_SCRIPTS.values(), "dash.css")}
|
|
for path, own_script in PAGE_SCRIPTS.items():
|
|
html = _code(client.get(path).text)
|
|
corpus = "".join(_code(scripts[name])
|
|
for name in (*SHARED_SCRIPTS, own_script, "dash.css"))
|
|
# ids the markup uses itself: labels, targets, `<label for>`
|
|
anchors = set(re.findall(
|
|
r'(?:aria-labelledby|aria-describedby|aria-controls|for)="([^"]+)"', html))
|
|
for element_id in re.findall(r'id="([^"]+)"', html):
|
|
quoted = (f'"{element_id}"' in corpus or f"'{element_id}'" in corpus
|
|
or f"#{element_id}" in corpus or element_id in anchors)
|
|
assert quoted, f"{path} has id=\"{element_id}\" that nothing reads"
|
|
|
|
|
|
# ---- the flavour-text audit (phase 08 task 02) ----------------------------
|
|
#
|
|
# `TODO.md:7`: "Get rid of most of the flavor text and add a note to AGENTS.md to not add it
|
|
# unless necessary. It just comes off as LLM generated slop, not helpful information." The
|
|
# test is one question per string — *would the operator act differently without it?* — and
|
|
# the answer decided the deletion. Copy that explains how the app builds its numbers is not
|
|
# helpful information; a provenance tag, a measured Δ window, and the sentence beside an
|
|
# empty panel are. `AGENTS.md` → *UI conventions* now carries the rule; the pins below are
|
|
# what stop the deleted sentences crawling back, because a screenshot only shows them once
|
|
# somebody already knows to look.
|
|
|
|
#: Fragments deleted from the served pages. The first two are the lines the operator quoted.
|
|
DELETED_FLAVOUR = (
|
|
# the overview's footer
|
|
"every number is scraped from the box",
|
|
"Sources the operator never configured are not problems",
|
|
# /system's "Per host" sub-line and its footer
|
|
"agent-reported values",
|
|
"each panel's charts cover only that host",
|
|
"numbers are measured, not inferred",
|
|
# /llm's footer, the one half-sentence of it that lived in the script too
|
|
"each throughput number carries its provenance",
|
|
"rates over the Δ window in each tag",
|
|
"a server you do not run",
|
|
# /ups's footer and its empty state
|
|
"commands are the only write path in this app",
|
|
"The dashboard speaks the NUT protocol",
|
|
# the three event feeds' header captions, all duplicated by an empty-state sentence
|
|
"unreachable / reachable / reboots",
|
|
"engine up / down",
|
|
"state changes, self-test results, commands sent",
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("fragment", DELETED_FLAVOUR)
|
|
def test_no_page_narrates_its_own_pipeline(client, fragment: str) -> None:
|
|
"""The deleted copy is gone from every served byte, markup and script alike.
|
|
|
|
Asserted across all four pages and all five scripts rather than the file it came from:
|
|
the same sentence migrating into another page's card is the same slop, and a pin that
|
|
only guards its original file would let that through.
|
|
"""
|
|
for path in (*MARKUP_PAGES, *ASSETS):
|
|
body = client.get(path).text
|
|
assert fragment not in body, f"{path} narrates the app again: {fragment!r}"
|
|
|
|
|
|
def test_pages_leave_no_empty_narration_container(client) -> None:
|
|
"""A deleted sub-line removes its container, not just its words.
|
|
|
|
An empty `<span class="sub">` is markup that reserves a line for nothing — the page head
|
|
renders a gap where the sentence used to be. Elements the scripts fill (`#sub`,
|
|
`#alerts-count`) carry an `id` and are not in this category.
|
|
"""
|
|
for path in MARKUP_PAGES:
|
|
html = _code(client.get(path).text)
|
|
for cls in ("sub", "faint", "mono"):
|
|
assert f'<span class="{cls}"></span>' not in html, \
|
|
f"{path} keeps an empty .{cls} container"
|
|
|
|
|
|
def test_the_copy_the_operator_acts_on_survives_the_audit(client) -> None:
|
|
"""The other half of the deletion: functional copy stays.
|
|
|
|
Stripping flavour must not become stripping information. Every string here answers a
|
|
question the operator is asking while looking at the page — why is this panel empty,
|
|
is this number a measurement, is this battery real, is this host's data last-good —
|
|
and each one is the reason the audit deletes sentences and not labels.
|
|
"""
|
|
pages = {path: client.get(path).text for path in PAGES}
|
|
scripts = {name: client.get(f"/static/{name}").text for name in PAGE_SCRIPTS.values()}
|
|
|
|
# an empty panel says why, in terms the operator can act on
|
|
assert "No hosts configured" in pages["/"]
|
|
assert "HOST_TARGETS" in pages["/system"]
|
|
# …including the fourth page's: a `n/a` drive card is only actionable if the page names
|
|
# the two things that make SMART readable (the binary and the device-read grant).
|
|
assert "smartmontools" in pages["/disks"] + scripts["page-disks.js"]
|
|
assert "disk group" in scripts["page-disks.js"]
|
|
assert "No engines configured" in pages["/llm"]
|
|
assert "No UPS reporting" in pages["/ups"]
|
|
assert "instcmds" in pages["/ups"], "the UPS setup note stopped naming the grant that matters"
|
|
assert "3493" in pages["/ups"], "the page stopped warning that upsd's port is not always 3493"
|
|
# …and the setup copy names the grant NUT actually checks. `README.md` is explicit: a
|
|
# dashboard user gets `instcmds`, never `actions = -`, so a page that told the operator
|
|
# to write that was sending them to fix the wrong line in `upsd.users`.
|
|
assert "actions = -" not in pages["/ups"] + scripts["page-ups.js"] + scripts["page-index.js"]
|
|
assert "instcmds" in scripts["page-ups.js"] and "instcmds" in scripts["page-index.js"]
|
|
|
|
# a Δ label is a measurement, and the rate's provenance travels with the rate
|
|
assert "windowLabel(" in scripts["page-system.js"]
|
|
assert "Δ" in scripts["page-llm.js"] and "function provenance(" in scripts["page-llm.js"]
|
|
assert "at the current load" in scripts["page-ups.js"]
|
|
|
|
# modelled data is labelled, and so is a feed that cannot contain what you are hunting
|
|
assert "SIMULATED" in scripts["page-index.js"] and "SIMULATED" in scripts["page-ups.js"]
|
|
assert "queued (deferred) requests are not in /slots" in scripts["page-llm.js"]
|
|
assert "Last good values" in scripts["page-index.js"]
|
|
|
|
|
|
def test_api_responses_are_not_cacheable(client) -> None:
|
|
for path in ("/api/config", "/api/health", "/api/summary"):
|
|
assert "no-store" in client.get(path).headers.get("cache-control", "")
|
|
|
|
|
|
# ---- history -------------------------------------------------------------- #
|
|
|
|
def test_history_returns_seeded_samples(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
db = str(tmp_path / "hist.sqlite3")
|
|
store = SampleStore(db)
|
|
now = time.time()
|
|
for i in range(30):
|
|
store.add("host", "driveripper", "cpu.usage_pct", 10.0 + i, ts=now - 300 + i)
|
|
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2,
|
|
host_targets=(Target("driveripper", DEAD_URL),)), store)
|
|
with TestClient(app) as client:
|
|
sign_in(client)
|
|
body = client.get("/api/history", params={"source": "host", "host": "driveripper",
|
|
"metric": "cpu.usage_pct", "range": "15m"})
|
|
assert response_ok(body)
|
|
points = body.json()["points"]
|
|
assert len(points) >= 25
|
|
assert points[-1]["value"] == pytest.approx(39.0)
|
|
store.close()
|
|
|
|
|
|
def test_history_rejects_an_unknown_range(client) -> None:
|
|
response = client.get("/api/history", params={"source": "host", "host": "h",
|
|
"metric": "m", "range": "banana"})
|
|
assert response.status_code == 400
|
|
|
|
|
|
def test_history_requires_the_series_coordinates(client) -> None:
|
|
assert client.get("/api/history").status_code == 422
|
|
|
|
|
|
@pytest.mark.parametrize(("raw", "seconds"), [
|
|
*[(key, float(seconds)) for key, seconds in RANGE_SECONDS.items()],
|
|
("45s", 45.0), ("13h", 46_800.0), ("7200", 7_200.0), (" 2D ", 172_800.0),
|
|
(None, 3_600.0), ("", 3_600.0),
|
|
])
|
|
def test_parse_range_reads_every_window_the_ui_offers(raw: Any, seconds: float) -> None:
|
|
"""Every chip key (5m…90d), any `<n><s|m|h|d>`, raw seconds, and the 1h default.
|
|
|
|
`parse_range` is the only thing between a typo'd query string and a chart of the wrong
|
|
window, and a wrong window is worse than an error: it still looks like data.
|
|
"""
|
|
assert parse_range(raw) == seconds
|
|
|
|
|
|
@pytest.mark.parametrize("raw", ["banana", "1w", "h", "-4h", "one"])
|
|
def test_parse_range_refuses_a_window_it_cannot_interpret(raw: str) -> None:
|
|
"""A 400 that names the value, never a silent fallback to 1h."""
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
parse_range(raw)
|
|
assert excinfo.value.status_code == 400
|
|
assert raw in str(excinfo.value.detail)
|
|
|
|
|
|
# ---- the routes the *other* pages chart from -------------------------------- #
|
|
#
|
|
# `/` renders two panels now; `/system`, `/llm` and `/ups` still need these, and so does
|
|
# anything outside the browser. Retiring overview code must not retire them, so the shapes
|
|
# are pinned here rather than discovered by a page three phases from now.
|
|
|
|
def test_system_history_charts_every_configured_host_in_one_request(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""N hosts x M metrics, one round trip.
|
|
|
|
`/system` no longer uses the N-hosts half of this (every chart belongs to one host and
|
|
the page issues one request per host, pinned by
|
|
`test_every_system_chart_is_drawn_inside_its_own_host_panel`); the shape is kept because
|
|
the metric-list half is exactly what the per-host query costs one request with, and
|
|
because the endpoint is a public interface for programmatic consumers.
|
|
"""
|
|
with build_client(tmp_path, monkeypatch,
|
|
host_targets=(Target("driveripper", DEAD_URL),
|
|
Target("deskwork", DEAD_URL))) as test_client:
|
|
body = test_client.get("/api/system/history",
|
|
params={"metrics": "cpu.usage_pct,mem.used_pct",
|
|
"range": "1h"}).json()
|
|
assert body["hosts"] == ["driveripper", "deskwork"]
|
|
assert body["range_seconds"] == 3_600.0
|
|
assert len(body["series"]) == 4
|
|
assert {(item["host"], item["metric"]) for item in body["series"]} == {
|
|
("driveripper", "cpu.usage_pct"), ("driveripper", "mem.used_pct"),
|
|
("deskwork", "cpu.usage_pct"), ("deskwork", "mem.used_pct")}
|
|
# an explicit `hosts` list narrows it instead of adding to the default
|
|
narrowed = test_client.get("/api/system/history",
|
|
params={"metrics": "cpu.usage_pct",
|
|
"hosts": "deskwork"}).json()
|
|
assert narrowed["hosts"] == ["deskwork"]
|
|
|
|
|
|
def test_one_request_per_host_answers_every_per_host_chart(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The exact query `/system` issues per host returns every series that panel draws.
|
|
|
|
`page-system.js` sends one request per host carrying the utilisation trio
|
|
(`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct`), both network rates, and one
|
|
`gpu.busy_pct@<index>` per card the payload reported — so a name missing here is a chart
|
|
that silently reads "no samples yet" in the browser, and a renamed series in
|
|
`sources/host.py` fails here rather than in a screenshot. This is the metric-name half of
|
|
the contract; `tests/test_system_contract.py` holds the payload-path half.
|
|
"""
|
|
with build_client(tmp_path, monkeypatch,
|
|
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
|
|
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
|
|
now = time.time()
|
|
metrics = ["cpu.usage_pct", "mem.used_pct", "disk.used_pct",
|
|
"net.rx_bytes_per_s", "net.tx_bytes_per_s", "gpu.busy_pct@0"]
|
|
for metric in metrics:
|
|
store.add("host", "driveripper", metric, 12.5, ts=now - 60)
|
|
store.add("host", "driveripper", metric, 25.0, ts=now - 30)
|
|
|
|
body = test_client.get("/api/system/history", params={
|
|
"hosts": "driveripper", "metrics": ",".join(metrics), "range": "1h"}).json()
|
|
assert body["hosts"] == ["driveripper"], "one panel, one host"
|
|
assert [item["metric"] for item in body["series"]] == metrics
|
|
assert [len(item["points"]) for item in body["series"]] == [2] * len(metrics)
|
|
|
|
# a second card the host does not have answers empty, never zero and never 404
|
|
gap = test_client.get("/api/system/history",
|
|
params={"hosts": "driveripper", "metrics": "gpu.busy_pct@1"}).json()
|
|
assert gap["series"][0]["points"] == []
|
|
|
|
|
|
def test_llm_snapshot_keeps_an_engine_that_never_answered_in_the_list(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""A configured-but-dead engine is listed as unreachable, not dropped (invariant 4)."""
|
|
# a host target too, so the snapshot's job filter has to skip a non-LLM job
|
|
with build_client(tmp_path, monkeypatch, vllm_url=DEAD_URL,
|
|
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
|
|
body = test_client.get("/api/llm/snapshot").json()
|
|
assert [engine["engine"] for engine in body["engines"]] == ["vLLM"]
|
|
assert body["engines"][0]["unreachable"] is True
|
|
assert body["fleet"]["engines_total"] == 1
|
|
assert body["fleet"]["engines_live"] == 0
|
|
assert [state["key"] for state in body["states"]] == ["llm:vllm"]
|
|
|
|
|
|
def test_llm_history_charts_both_engines_by_default(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
|
|
vllm_url=DEAD_URL) as test_client:
|
|
body = test_client.get("/api/llm/history",
|
|
params={"metrics": "decode_tokens_per_second,kv_cache_usage_ratio",
|
|
"range": "15m"}).json()
|
|
assert body["engines"] == ["gufo", "vllm"]
|
|
assert body["range_seconds"] == 900.0
|
|
assert {(item["host"], item["metric"]) for item in body["series"]} == {
|
|
("gufo", "llm.decode_tokens_per_second"),
|
|
("gufo", "llm.kv_cache_usage_ratio"),
|
|
("vllm", "llm.decode_tokens_per_second"),
|
|
("vllm", "llm.kv_cache_usage_ratio")}
|
|
|
|
|
|
def test_one_batched_llm_history_request_answers_every_cell_series(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The exact query `/llm` issues per refresh returns every chart *and* sparkline series.
|
|
|
|
Phase 04 sends one request carrying the chart trio and the whole spark cell list —
|
|
including the two series the layout required (`tpot_seconds` for the vLLM TPOT row,
|
|
`spec_accept_rate` for the gufo spec row). A name missing here is a cell that reads
|
|
"no history" forever in the browser, which is the dead-cell anti-pattern the phase
|
|
wrote NOTES-chart-set to prevent; a rename in `sources/llm.py` fails here first.
|
|
"""
|
|
with build_client(tmp_path, monkeypatch, gufo_url=DEAD_URL,
|
|
vllm_url=DEAD_URL) as test_client:
|
|
store = test_client.app.state.ctx.store # type: ignore[attr-defined]
|
|
now = time.time()
|
|
metrics = ["decode_tokens_per_second", "prompt_tokens_per_second",
|
|
"kv_cache_usage", "requests_running", "requests_waiting",
|
|
"ttft_seconds", "tpot_seconds", "queue_seconds",
|
|
"prefix_cache_hit_ratio", "spec_accept_rate"]
|
|
for metric in metrics:
|
|
store.add("llm", "gufo", f"llm.{metric}", 1.5, ts=now - 60)
|
|
store.add("llm", "vllm", f"llm.{metric}", 2.5, ts=now - 30)
|
|
body = test_client.get("/api/llm/history", params={
|
|
"engines": "gufo,vllm", "metrics": ",".join(metrics), "range": "1h"}).json()
|
|
assert body["range_seconds"] == 3_600.0
|
|
by_key = {(item["host"], item["metric"]) for item in body["series"]}
|
|
assert by_key == {(engine, f"llm.{metric}")
|
|
for engine in ("gufo", "vllm") for metric in metrics}
|
|
points = {(item["host"], item["metric"]): item["points"] for item in body["series"]}
|
|
assert points[("gufo", "llm.spec_accept_rate")][0]["value"] == pytest.approx(1.5)
|
|
assert points[("vllm", "llm.tpot_seconds")][0]["value"] == pytest.approx(2.5)
|
|
|
|
|
|
def test_metrics_endpoint_lists_what_actually_has_history(client) -> None:
|
|
body = client.get("/api/metrics").json()
|
|
assert body["metrics"] == []
|
|
|
|
|
|
# ---- the `/disks` API (phase 09 task 03) ---------------------------------- #
|
|
#
|
|
# The drive-health surface answers like every other dashboard's: GETs only, the same
|
|
# stale/stub semantics for a host that is not answering, the same history mechanism for the
|
|
# `disk.*@dev` series, and the same 404-`configured: false` sentence when no agent is
|
|
# configured at all — the last one *without dialling*, because "not deployed here" must not
|
|
# cost a connect timeout or a red dot (invariant 3).
|
|
|
|
@pytest.fixture
|
|
def disks_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
|
disk_snapshot: Any) -> Iterator[TestClient]:
|
|
"""Two hosts configured; one of them answering SMART, through the production parser.
|
|
|
|
The payload and the stored samples are what `parse_disks_snapshot` makes of the
|
|
fixture-shaped capture (the agent's whole-disk rows plus the recorded `smartctl` shapes),
|
|
so every path a `/disks` card will read is a path the source really emits. The other host
|
|
keeps its `unreachable` stub, which is the shape a dead agent must keep (invariant 4).
|
|
Both targets point at a dead port, so nothing dials.
|
|
"""
|
|
db = str(tmp_path / "disks.sqlite3")
|
|
store = SampleStore(db)
|
|
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2,
|
|
host_targets=(Target("driveripper", DEAD_URL),
|
|
Target("deskwork", DEAD_URL))), store)
|
|
with TestClient(app) as test_client:
|
|
sign_in(test_client)
|
|
now = time.time()
|
|
snapshot = disk_snapshot("driveripper")
|
|
snapshot["ts"] = now # "the agent answered just now", so charts are in range
|
|
result = parse_disks_snapshot("driveripper", snapshot, DisksCollector(), now=now)
|
|
store.add_many(result.samples)
|
|
app.state.ctx.hub.jobs["disks:driveripper"].state.payload = result.payload
|
|
yield test_client
|
|
store.close()
|
|
|
|
|
|
#: (`smart` field, severity metric) for the two drive numbers the table rates — the same
|
|
#: pair `sources/disks.py:SEVERITY_FIELDS` declares, restated here so a rename that only
|
|
#: touches one side fails rather than colouring nothing.
|
|
RATED_CELLS: tuple[tuple[str, str], ...] = (
|
|
("temp_c", "disk.temp_c"),
|
|
("percentage_used", "disk.percentage_used"),
|
|
)
|
|
|
|
|
|
def disk_rows(body: dict[str, Any], host: str = "driveripper") -> list[dict[str, Any]]:
|
|
"""The disk rows of one host out of a `/api/disks/snapshot` body."""
|
|
return next(row for row in body["hosts"] if row["name"] == host)["disks"]
|
|
|
|
|
|
def flat_routes(app: Any) -> list[Any]:
|
|
"""Every route the app serves, with `include_router()` results flattened to their routes.
|
|
|
|
The FastAPI this repo pins keeps an included router as an `_IncludedRouter` wrapper with
|
|
no `path` of its own, so walking `app.routes` alone sees five anonymous objects instead of
|
|
the API surface — and the verb assertion below is only as good as this walk.
|
|
"""
|
|
out: list[Any] = []
|
|
for route in app.routes:
|
|
inner = getattr(route, "original_router", None)
|
|
out.extend(inner.routes if inner is not None else [route])
|
|
return out
|
|
|
|
|
|
def series_source(body: dict[str, Any]) -> set[str]:
|
|
"""Which sample family a history answer came from: `disks`, never the 10-second `host`."""
|
|
return {series["source"] for series in body["series"]}
|
|
|
|
|
|
def test_disks_snapshot_answers_one_entry_per_host_with_its_physical_disks(
|
|
disks_client) -> None:
|
|
"""`GET /api/disks/snapshot` is per host: rows + `smart` + `smartctl_available`."""
|
|
body = disks_client.get("/api/disks/snapshot")
|
|
assert response_ok(body)
|
|
payload = body.json()
|
|
assert [host["name"] for host in payload["hosts"]] == ["deskwork", "driveripper"], (
|
|
"every configured host is a row, in name order, answering or not")
|
|
rows = disk_rows(payload)
|
|
assert rows, "the capture has 16 whole-disk devices, so the page has 16 cards"
|
|
assert all("/" not in row["device"] for row in rows), "a mount leaked into the disk list"
|
|
first = rows[0]
|
|
assert first["path"] == f"/dev/{first['device']}"
|
|
assert set(first) >= {"device", "path", "model", "size_bytes", "rotational",
|
|
"health", "smart", "severity"}
|
|
assert first["smart"]["power_on_hours"] > 0, "the drive's own counters travel verbatim"
|
|
live = next(host for host in payload["hosts"] if host["name"] == "driveripper")
|
|
assert live["smartctl_available"] is True
|
|
assert live["smart_errors"] == []
|
|
assert live["summary"]["disks"] == len(rows)
|
|
assert "unreachable" not in live, "answering is the absence of the flag, as on /system"
|
|
assert [state["key"] for state in payload["states"]] == ["disks:driveripper",
|
|
"disks:deskwork"], (
|
|
"the page's dots come from the disks jobs, never the system ones")
|
|
|
|
|
|
def test_disks_snapshot_carries_the_verdicts_the_page_will_paint(disks_client) -> None:
|
|
"""Colour is in the payload: each row's `severity` is what `severity.classify` says.
|
|
|
|
This is the pin that keeps the threshold out of the page — `page-disks.js` reads these
|
|
and never compares a drive temperature to a bound.
|
|
"""
|
|
rows = disk_rows(disks_client.get("/api/disks/snapshot").json())
|
|
assert "ok" in {row["severity"]["temp_c"] for row in rows}, "the capture is healthy"
|
|
for row in rows:
|
|
smart = row["smart"] or {}
|
|
assert set(row["severity"]) == {field for field, _metric in RATED_CELLS} | {"health"}
|
|
for field, metric in RATED_CELLS:
|
|
assert row["severity"][field] == classify(metric, smart.get(field)), (
|
|
f"{row['device']}.{field} is coloured by something other than the one table")
|
|
assert not {key for row in rows for key in row["severity"]} & {
|
|
"reallocated_sector_ct", "pending", "uncorrectable", "power_on_hours"}, (
|
|
"a counter is an alarm chip on presence, never a graded cell")
|
|
|
|
|
|
def test_a_configured_host_with_no_payload_yet_is_a_row_of_nas(disks_client) -> None:
|
|
"""The stub shape `/api/system/snapshot` hands out, so `n/a` is never a missing row.
|
|
|
|
A host in `HOST_TARGETS` that has not answered is still in the fleet (invariants 3 and
|
|
4) with its counts `None` rather than `0`: "unknown" and "no drives" are different
|
|
sentences (rule 2), and the fleet header counts it without summing it.
|
|
"""
|
|
payload = disks_client.get("/api/disks/snapshot").json()
|
|
stub = next(host for host in payload["hosts"] if host["name"] == "deskwork")
|
|
assert stub["unreachable"] is True
|
|
assert stub["disks"] == [], "nothing to draw, but the host is still listed"
|
|
assert stub["smartctl_available"] is None, "an agent that never answered says nothing"
|
|
assert stub["smart_errors"] == []
|
|
assert set(stub["summary"]) == {"disks", "with_smart", "health_failed", "alarm_disks"}
|
|
assert all(value is None for value in stub["summary"].values()), "no invented zeros"
|
|
fleet = payload["fleet"]
|
|
assert (fleet["hosts_total"], fleet["hosts_live"]) == (2, 1)
|
|
assert fleet["disks"] == len(disk_rows(payload))
|
|
|
|
|
|
def test_the_disks_fleet_header_is_summed_by_the_api_not_the_page(disks_client) -> None:
|
|
"""`fleet` is the header's only arithmetic source, and it carries no verdict."""
|
|
payload = disks_client.get("/api/disks/snapshot").json()
|
|
live = next(host for host in payload["hosts"] if host["name"] == "driveripper")
|
|
fleet = payload["fleet"]
|
|
assert fleet["disks"] == live["summary"]["disks"]
|
|
assert fleet["alarm_disks"] == live["summary"]["alarm_disks"] > 0, (
|
|
"the recorded ATA shape carries reallocated sectors, so the fleet has chips to draw")
|
|
assert "severity" not in fleet, "a count of drives is not a graded number"
|
|
|
|
|
|
def test_the_disks_surface_is_gets_only_and_the_write_stays_the_ups_one(
|
|
disks_client) -> None:
|
|
"""Invariant 1: the app writes to an **upstream** from exactly one route."""
|
|
routes = flat_routes(disks_client.app)
|
|
disks = [route for route in routes
|
|
if str(getattr(route, "path", "")).startswith("/api/disks")]
|
|
assert {route.path for route in disks} == {"/api/disks/snapshot", "/api/disks/history"}
|
|
verbs = {verb for route in disks for verb in (getattr(route, "methods", None) or set())}
|
|
assert verbs == {"GET"}, "a write appeared on the drive-health surface"
|
|
for verb in ("POST", "PUT", "PATCH", "DELETE"):
|
|
response = disks_client.request(verb, "/api/disks/snapshot", json={})
|
|
assert response.status_code == 405, f"{verb} /api/disks/snapshot was accepted"
|
|
writes = sorted(str(route.path) for route in routes
|
|
if "POST" in (getattr(route, "methods", None) or set()))
|
|
# Invariant 1 is about the fleet, and phase 11 did not touch the fleet: the three `/auth/`
|
|
# writes create, re-credential, and clear a session in this process and dial nothing. They
|
|
# are named here instead of filtered out of sight, because the failure this assertion catches
|
|
# is a POST arriving quietly — a list the reader recognises is worth more than a short one.
|
|
upstream = [path for path in writes if not path.startswith("/api/auth/")]
|
|
session = [path for path in writes if path.startswith("/api/auth/")]
|
|
assert upstream == ["/api/ups/command"], "one write endpoint, four gates, unchanged"
|
|
assert session == ["/api/auth/login", "/api/auth/logout", "/api/auth/password"]
|
|
# …and nothing anywhere grew the verbs this app has no use for, session included.
|
|
assert not sorted(path for route in routes
|
|
for path in [str(route.path)]
|
|
if set(getattr(route, "methods", None) or ()) & {"PUT", "PATCH", "DELETE"})
|
|
# The session writes are not upstream writes, but they are still the only ones: `/login`
|
|
# is the page, not a POST target, and `/api/auth/status` stays a read.
|
|
pages = sorted(str(route.path) for route in routes
|
|
if "GET" in (getattr(route, "methods", None) or set())
|
|
and not str(route.path).startswith("/api/"))
|
|
# `/settings` (phase 11) is a fifth dashboard-shaped **page** and it grew no verb: a page
|
|
# route is a GET, so the write surface this test guards stays the one UPS command.
|
|
assert pages == ["/", "/disks", "/llm", "/login", "/settings", "/system", "/ups"]
|
|
|
|
|
|
def test_config_prints_the_disks_cadence_the_page_explains_itself_with(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""A card four minutes old is behaving, not broken — and the interval is the proof.
|
|
|
|
`/disks` has no upstream of its own, so `enabled` follows `HOST_TARGETS` and the cadence
|
|
is its own. Both numbers come from config rather than a constant in the page, so changing
|
|
`DISKS_INTERVAL_SECONDS` moves the sentence and the poller together.
|
|
"""
|
|
with build_client(tmp_path, monkeypatch, host_targets=(Target("driveripper", DEAD_URL),),
|
|
disks_interval=120.0, disks_timeout=45.0) as test_client:
|
|
disks = test_client.get("/api/config").json()["disks"]
|
|
assert disks["enabled"] is True
|
|
assert disks["interval_seconds"] == 120.0
|
|
assert disks["timeout_seconds"] == 45.0
|
|
|
|
|
|
def test_disks_snapshot_404s_without_dialling_a_host_that_was_never_configured(
|
|
client, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The gate runs before anything that could reach the network (invariant 3).
|
|
|
|
Both spies raise, so a route that constructed a client or fetched a snapshot before
|
|
checking `HOST_TARGETS` fails here instead of costing a timeout and a red dot. The hub
|
|
assertion is the structural half: with no targets there is no `disks` job to dial with.
|
|
"""
|
|
calls: list[str] = []
|
|
|
|
def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
|
|
calls.append(f"GET {url}")
|
|
raise AssertionError(f"the unconfigured disks route dialled {url}")
|
|
|
|
def spy_fetch(*args: Any, **kwargs: Any) -> Any:
|
|
calls.append("fetch_snapshot")
|
|
raise AssertionError("the unconfigured disks route fetched an agent snapshot")
|
|
|
|
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
|
|
monkeypatch.setattr("dashboards.collectors.fetch_snapshot", spy_fetch)
|
|
|
|
response = client.get("/api/disks/snapshot")
|
|
assert response.status_code == 404
|
|
body = response.json()
|
|
assert body["configured"] is False
|
|
assert "HOST_TARGETS" in body["detail"], "the sentence names the fix"
|
|
assert calls == [], "an unconfigured page must not touch the network"
|
|
assert client.app.state.ctx.hub.jobs == {}, (
|
|
"no targets means no disks job, so nothing can age out or go red")
|
|
|
|
|
|
def test_disks_history_serves_the_dev_series_the_page_charts(disks_client) -> None:
|
|
"""One request, every series a host's cards could draw, points from the stored samples."""
|
|
rows = disk_rows(disks_client.get("/api/disks/snapshot").json())
|
|
metrics = [f"{base}@{row['device']}" for row in rows for _field, base in SMART_SERIES]
|
|
body = disks_client.get("/api/disks/history", params={
|
|
"hosts": "driveripper", "metrics": ",".join(metrics), "range": "1h"}).json()
|
|
assert body["hosts"] == ["driveripper"], "one host, one panel"
|
|
assert body["range_seconds"] == 3_600.0
|
|
assert [series["metric"] for series in body["series"]] == metrics
|
|
assert series_source(body) == {"disks"}, "the disks family, never the 10-second host one"
|
|
points = {(series["metric"]): series["points"] for series in body["series"]}
|
|
assert points["disk.temp_c@nvme0n1"][0]["value"] == pytest.approx(38.0)
|
|
assert points["disk.power_on_hours@nvme0n1"][0]["value"] == pytest.approx(8_760.0)
|
|
|
|
|
|
def test_disks_history_answers_empty_for_a_series_a_drive_does_not_publish(
|
|
disks_client) -> None:
|
|
"""Unknown names are an empty series, not a 404 — the /system and /llm contract.
|
|
|
|
The recorded ATA shape reports no `percentage_used`, so that drive's wear plot has
|
|
nothing to draw, and a device name the agent never listed behaves the same way. The card
|
|
asks for every series it could draw and reads `n/a` where there is nothing, instead of
|
|
the request failing.
|
|
"""
|
|
rows = disk_rows(disks_client.get("/api/disks/snapshot").json())
|
|
unworn = next(row["device"] for row in rows
|
|
if (row["smart"] or {}).get("percentage_used") is None)
|
|
body = disks_client.get("/api/disks/history", params={
|
|
"hosts": "driveripper",
|
|
"metrics": f"disk.percentage_used@{unworn},disk.temp_c@nosuchdisk,"
|
|
"disk.temp_c@nvme0n1",
|
|
"range": "1h"}).json()
|
|
by_metric = {series["metric"]: series["points"] for series in body["series"]}
|
|
assert by_metric[f"disk.percentage_used@{unworn}"] == [], "no measurement, no line"
|
|
assert by_metric["disk.temp_c@nosuchdisk"] == []
|
|
assert by_metric["disk.temp_c@nvme0n1"], "the series that does exist still answers"
|
|
|
|
|
|
def test_disks_history_charts_every_configured_host_in_one_request(disks_client) -> None:
|
|
"""With no `hosts` the answer covers every configured agent, dead ones included (empty)."""
|
|
body = disks_client.get("/api/disks/history",
|
|
params={"metrics": "disk.temp_c@nvme0n1,disk.temp_c@sda",
|
|
"range": "24h"}).json()
|
|
assert body["hosts"] == ["driveripper", "deskwork"]
|
|
assert body["range_seconds"] == 86_400.0
|
|
assert {(series["host"], series["metric"]) for series in body["series"]} == {
|
|
("driveripper", "disk.temp_c@nvme0n1"), ("driveripper", "disk.temp_c@sda"),
|
|
("deskwork", "disk.temp_c@nvme0n1"), ("deskwork", "disk.temp_c@sda")}
|
|
assert [series for series in body["series"] if series["host"] == "deskwork"], (
|
|
"a host with no history is an empty series, never a missing one")
|
|
|
|
|
|
def test_the_metrics_endpoint_answers_for_the_disks_family(disks_client) -> None:
|
|
""""Why is this chart empty" has an answer per page: /api/metrics?source=disks."""
|
|
body = disks_client.get("/api/metrics", params={"source": "disks"}).json()
|
|
assert "disk.temp_c@nvme0n1" in body["metrics"]
|
|
assert not [name for name in body["metrics"] if not name.startswith("disk.")], (
|
|
"the disks family holds only disk series")
|
|
assert not [name for name in disks_client.get(
|
|
"/api/metrics", params={"source": "host"}).json()["metrics"]
|
|
if name.startswith("disk.temp_c")], "SMART never mixes into the system family"
|
|
|
|
|
|
# ---- the `/disks` page (phase 09 task 04) ---------------------------------- #
|
|
#
|
|
# The fourth dashboard, asserted the way the other three are: on the served bytes, because a
|
|
# JS test runner would be a build step (a locked anchor). What is pinned is not that the page
|
|
# exists but that it keeps the four promises the phase wrote down — one card per **physical**
|
|
# drive and never one per partition, alarm chips that exist only while a counter is non-zero,
|
|
# a host without `smartctl` dimmed rather than reddened, and charts that belong to one host,
|
|
# stay collapsed until opened, and cost one history request per host per refresh.
|
|
|
|
def _disks_js(client: TestClient) -> str:
|
|
return client.get("/static/page-disks.js").text
|
|
|
|
|
|
def test_the_disks_tab_is_in_the_masthead_of_every_page(client) -> None:
|
|
"""Four tabs became five, on every page, with exactly one marking itself current.
|
|
|
|
The masthead is the app's only "which dashboard do I need" surface since phase 02 deleted
|
|
the overview's table, so a tab missing from one page is a page you cannot get to, and a
|
|
tab marked current on the wrong page is a lie about where you are.
|
|
"""
|
|
for path in PAGES:
|
|
html = client.get(path).text
|
|
assert '<a href="/disks"' in html, f"{path}'s nav has no Disks tab"
|
|
assert html.count('aria-current="page"') == 1, f"{path} marks more than one tab current"
|
|
assert f'<a href="{path}" aria-current="page">' in html, f"{path} does not mark itself"
|
|
assert (html.index('href="/system"') < html.index('href="/disks"')
|
|
< html.index('href="/llm"')), "tab order is machine, drive, engine, power"
|
|
|
|
|
|
def test_the_header_status_dot_picks_up_the_disks_job(client) -> None:
|
|
"""`disks` gets its own dot instead of hiding inside `hosts`.
|
|
|
|
The same agent can answer the 10-second utilisation poll and have a wedged SMART pass;
|
|
one dot for both would have to be right about neither.
|
|
"""
|
|
app_js = client.get("/static/app.js").text
|
|
assert "{ key: 'disks', label: 'disks', href: '/disks' }" in app_js
|
|
assert "s.dashboard === job.key" in app_js, "the dot stopped filtering by dashboard"
|
|
|
|
|
|
def test_disks_page_draws_one_card_per_physical_drive_from_the_api(client) -> None:
|
|
"""The card is a drive, keyed by the agent's device name, patched in place.
|
|
|
|
A mount or a partition would need a filesystem field, and `/system` already answers "is
|
|
the filesystem full"; if one leaks back in here, the page stops being about the drive.
|
|
"""
|
|
html = client.get("/disks").text
|
|
js = _disks_js(client)
|
|
assert 'id="hosts"' in html and 'id="hosts-empty"' in html
|
|
assert "fetchJSON('/api/disks/snapshot')" in js
|
|
assert "row.path" in js and "/dev/" in js, "the card stopped naming the device it is about"
|
|
# identity + verdict lead the card: no hero row, no big single number
|
|
assert 'class="heroes"' not in html, "the disk card grew a hero row"
|
|
assert "Dash.hero(" not in js
|
|
for filesystem in ("mountpoint", "fstype", "free_bytes", "worst_mount"):
|
|
assert filesystem not in js, f"{filesystem} leaked into the drive page"
|
|
# built once per device and patched, so a 16-drive host does not rebuild 16 panels a poll
|
|
assert "seen.get(row.device)" in js and "renderDiskCard(panel, row)" in js
|
|
assert "pruneHosts(" in js, "a drive that left the host keeps its card forever"
|
|
|
|
|
|
def test_disk_cards_print_the_drive_numbers_and_colour_only_rated_ones(client) -> None:
|
|
"""Temp, power-on hours and wear, coloured by the API's verdicts and nothing else.
|
|
|
|
The odometer is the deliberate exception: `severity.THRESHOLDS` has no `disk.power_on_hours`
|
|
row, so the cell is written with a `null` verdict and cannot go green at 100 h or red at
|
|
60 000 h. A drive that publishes no wear (the recorded HDD shape) prints `n/a` because
|
|
`setTxt` is the only writer.
|
|
"""
|
|
js = _disks_js(client)
|
|
for field in ("smart.temp_c", "smart.power_on_hours", "smart.percentage_used",
|
|
"row.health"):
|
|
assert field in js, f"{field} is no longer on the card"
|
|
for verdict in ("severity.temp_c", "severity.percentage_used", "severity.health"):
|
|
assert verdict in js, f"the card stopped rendering {verdict}"
|
|
assert "severityFor" not in js, "the drive page wrote a threshold of its own"
|
|
body = js[js.index("function renderDiskCard("):js.index("/** Text that prints nothing")]
|
|
assert "severity.temp_c" in body and "severity.percentage_used" in body
|
|
assert re.search(r"setMetric\(card, 'hours',[^;]*null\)", body), (
|
|
"the odometer is being graded by something severity.py does not have")
|
|
# the drive's own word, and `unknown` for a drive that gave none — never a green pass
|
|
assert "'FAILED'" in body and "'unknown'" in body
|
|
|
|
|
|
def test_disk_alarm_chips_exist_only_while_a_counter_is_non_zero(client) -> None:
|
|
"""Zero and unknown both draw no chip; only a non-zero counter draws a red one.
|
|
|
|
The `/llm` preemption rule: a permanent "0 pending sectors" row trains the eye to ignore
|
|
it, and grading a counter green would call a quiet drive an achievement. `severity.py`
|
|
rates none of these, so the chip's presence is the entire signal here.
|
|
"""
|
|
js = _disks_js(client)
|
|
for counter in ("reallocated_sector_ct", "current_pending_sector_ct",
|
|
"offline_uncorrectable"):
|
|
assert counter in js, f"{counter} is no longer an alarm"
|
|
assert "value <= 0) continue" in js, "a zero now draws a chip"
|
|
assert "alarms.hidden = chips.length === 0" in js, "the empty chip row stays on screen"
|
|
assert 'class="chip fail"' in js
|
|
css = client.get("/static/dash.css").text
|
|
assert ".alarms {" in css and ".chip.fail" in css
|
|
|
|
|
|
def test_a_host_without_smartctl_is_dimmed_with_a_one_line_why_and_never_red(client) -> None:
|
|
"""Unconfigured is not broken (invariant 3), and the three answers stay three.
|
|
|
|
`smartctl_available` is `false` (install smartmontools — the agent said so, so the page
|
|
may say it), `null` (nobody told us either way, so name the fix without claiming to know
|
|
which half is missing), or the agent is not answering at all, which is a different
|
|
sentence again. None of the three reddens anything.
|
|
"""
|
|
js = _disks_js(client)
|
|
html = client.get("/disks").text
|
|
assert "smartctl not available on" in js
|
|
assert "disk group" in js, "the why stopped naming the grant that fixes it"
|
|
assert "smartctl_available === false" in js and "smartctl_available === true" in js
|
|
assert "host.unreachable" in js, "a dead agent must not be reported as a missing binary"
|
|
assert "smart_errors" in js, "the agent's own sentence is no longer printed"
|
|
assert "classList.toggle('stale'" in js
|
|
assert 'class="banner info why" hidden' in js, "the why-line has no home on the card"
|
|
assert "banner fail" not in js and html.count("banner") == 0, (
|
|
"a missing binary reddens the page")
|
|
|
|
|
|
def test_disks_charts_are_collapsed_lazy_and_cost_one_history_request_per_host(client) -> None:
|
|
"""`/system`'s chart discipline, reused: collapsed, painted only while open, one XHR.
|
|
|
|
The request still names every series the host could draw (temperature and wear per
|
|
drive), because `/api/disks/history` answers an unknown name as an empty series and a
|
|
panel should never have to discover series it did not ask for.
|
|
"""
|
|
js = _disks_js(client)
|
|
assert '<details class="charts-box"' in js
|
|
guard = js[js.index("function drawHostCharts("):js.index("async function renderHostCharts(")]
|
|
assert "if (!chartsOpen(card)) return;" in guard, (
|
|
"a collapsed stack is being built and drawn every poll")
|
|
history = js[js.index("async function renderHostCharts("):js.index("/**\n * Drop cards")]
|
|
assert history.count("fetchJSON(") == 1, "more than one history request for one host"
|
|
assert "/api/disks/history" in history
|
|
assert "streams.temp.concat(streams.wear)" in history, (
|
|
"the request no longer names every series the panel could draw")
|
|
prune = js[js.index("function pruneHosts("):js.index("// ---- fleet header")]
|
|
for entry in ("state.hosts.delete", "state.disks.delete", "state.charts.delete",
|
|
"state.history.delete", "state.payloads.delete"):
|
|
assert entry in prune, f"a pruned host keeps its {entry}"
|
|
|
|
|
|
def test_every_disks_chart_names_itself_for_assistive_tech(client) -> None:
|
|
"""An unlabelled plot is silence, and the legend is a sibling that labels nothing."""
|
|
js = _code(_disks_js(client))
|
|
labels = re.findall(r"ariaLabel: `([^`]+)`", js)
|
|
assert len(labels) == 2, "every drive chart must name itself (temperature, wear)"
|
|
assert all(label.startswith("${host.name}") for label in labels), (
|
|
"a chart that does not say which host it plots belongs on no page")
|
|
|
|
|
|
def test_the_disks_fleet_header_reads_the_api_sums_and_invents_none(client) -> None:
|
|
"""The header's arithmetic is the API's (`fleet_disk_summary`), never the page's.
|
|
|
|
Hidden below two hosts, because with one box in `HOST_TARGETS` these sums only duplicate
|
|
that host's own cards — `/llm`'s fleet-block rule. And no cell is coloured: a count of
|
|
drives is not a graded number (`severity.py` has no row for one).
|
|
"""
|
|
html = client.get("/disks").text
|
|
js = _disks_js(client)
|
|
for element in ("fleet", "fleet-dot", "fleet-count", "fleet-disks", "fleet-smart",
|
|
"fleet-failed", "fleet-alarms"):
|
|
assert f'id="{element}"' in html, f"the header lost #{element}"
|
|
fleet = js[js.index("function renderFleet("):js.index("// ---- snapshot")]
|
|
for path in ("fleet.hosts_live", "fleet.disks", "fleet.with_smart",
|
|
"fleet.health_failed", "fleet.alarm_disks"):
|
|
assert path in fleet, f"the header no longer reads {path}"
|
|
assert "reduce(" not in fleet and "+=" not in fleet, "the page is summing drives itself"
|
|
assert "severity" not in fleet, "a drive count grew a verdict"
|
|
assert "block.hidden = true" in fleet, "one host's sums duplicate its own cards"
|
|
|
|
|
|
def test_the_disk_panel_style_is_existing_vocabulary_with_no_new_palette(client) -> None:
|
|
"""`.disk` is the `.slot` shape, and the only colours it reaches for are the verdicts."""
|
|
css = _css(client)
|
|
block = _css_section(css, "drive panels")
|
|
for selector in (".disks {", ".disk {", ".disk-head", ".disk-dev", ".disk-model",
|
|
".disk .metrics", ".disk .alarms", ".disk .metric.warn .value",
|
|
".disk .metric.fail .value"):
|
|
assert selector in block, f"dash.css lost {selector}"
|
|
for token in ("var(--warn)", "var(--fail)", "var(--border)", "var(--bg-sunken)",
|
|
"var(--text)"):
|
|
assert token in block, f"the drive panel no longer uses {token}"
|
|
assert "--disk" not in css, "a drive-only colour token appeared"
|
|
assert ".stale" in css, "the dim that marks an unreadable host is gone"
|
|
|
|
|
|
def test_the_overview_prints_the_fourth_source_cadence(client) -> None:
|
|
"""`/` names every source's interval, so a drive card four minutes old reads as behaving."""
|
|
js = client.get("/static/page-index.js").text
|
|
assert "config.disks.interval_seconds" in js
|
|
|
|
|
|
@pytest.mark.parametrize("fixture", ["client", "data_client"])
|
|
def test_summary_carries_the_drive_block_for_programmatic_consumers(
|
|
request, fixture: str) -> None:
|
|
"""`disks` is additive: the same configured-ness, the same hosts, no key retired.
|
|
|
|
`client` has no targets, so it proves the block answers as `False`/`None`/`[]` instead of
|
|
disappearing; `data_client` proves it lists the host the fleet has.
|
|
"""
|
|
test_client = request.getfixturevalue(fixture)
|
|
body = test_client.get("/api/summary").json()
|
|
populated = fixture == "data_client"
|
|
assert body["disks"]["enabled"] is populated
|
|
assert (body["disks"]["fleet"] is not None) is populated
|
|
assert body["disks"]["hosts"] == ([] if not populated else [
|
|
{"name": "driveripper", "disks": None, "with_smart": None, "health_failed": None,
|
|
"alarm_disks": None, "smartctl_available": None, "failed_devices": [],
|
|
"unreachable": True}]), "a configured host with no payload is a row of nones"
|
|
assert [row["name"] for row in body["disks"]["hosts"]] == [
|
|
row["name"] for row in body["system"]["hosts"]]
|
|
|
|
|
|
def test_summary_serves_the_same_counts_the_disks_page_prints(disks_client) -> None:
|
|
"""One projection, two consumers: the summary's numbers *are* the snapshot's."""
|
|
body = disks_client.get("/api/summary").json()
|
|
snapshot = disks_client.get("/api/disks/snapshot").json()
|
|
disks = body["disks"]
|
|
assert disks["fleet"] == snapshot["fleet"], "the header would disagree with the page"
|
|
rows = {row["name"]: row for row in disks["hosts"]}
|
|
live = next(host for host in snapshot["hosts"] if host["name"] == "driveripper")
|
|
assert (rows["driveripper"]["disks"], rows["driveripper"]["with_smart"],
|
|
rows["driveripper"]["health_failed"], rows["driveripper"]["alarm_disks"]) == (
|
|
live["summary"]["disks"], live["summary"]["with_smart"],
|
|
live["summary"]["health_failed"], live["summary"]["alarm_disks"])
|
|
assert rows["driveripper"]["smartctl_available"] is True
|
|
assert rows["driveripper"]["unreachable"] is False
|
|
assert rows["deskwork"]["unreachable"] is True
|
|
assert rows["deskwork"]["disks"] is None, "unknown, never 0 (rule 2)"
|
|
assert sorted(state["key"] for state in disks["states"]) == ["disks:deskwork",
|
|
"disks:driveripper"]
|
|
|
|
|
|
def test_summary_names_the_drive_the_health_verdict_called_failed(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, disk_snapshot: Any) -> None:
|
|
"""Which drive the hardware itself called `FAILED` is part of the fleet answer.
|
|
|
|
A notifier reading `/api/summary` should not have to fetch a second endpoint to name the
|
|
device, and the count and the list have to agree or one of them is wrong.
|
|
"""
|
|
db = str(tmp_path / "summary-failed.sqlite3")
|
|
store = SampleStore(db)
|
|
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2,
|
|
host_targets=(Target("driveripper", DEAD_URL),)), store)
|
|
with TestClient(app) as test_client:
|
|
sign_in(test_client)
|
|
now = time.time()
|
|
snapshot = disk_snapshot("driveripper", mode="failed")
|
|
snapshot["ts"] = now
|
|
result = parse_disks_snapshot("driveripper", snapshot, DisksCollector(), now=now)
|
|
app.state.ctx.hub.jobs["disks:driveripper"].state.payload = result.payload
|
|
row = test_client.get("/api/summary").json()["disks"]["hosts"][0]
|
|
assert len(row["failed_devices"]) == 1
|
|
assert row["failed_devices"][0] == result.payload["disks"][0]["device"]
|
|
assert row["health_failed"] == result.payload["summary"]["health_failed"] == 1
|
|
store.close()
|
|
|
|
|
|
# ---- events --------------------------------------------------------------- #
|
|
|
|
def test_events_are_returned_in_json(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
db = str(tmp_path / "events.sqlite3")
|
|
store = SampleStore(db)
|
|
store.log_event(Event(ts=time.time(), source="host", host="deskwork", kind="down",
|
|
severity="error", message="deskwork is not answering"))
|
|
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2), store)
|
|
with TestClient(app) as client:
|
|
sign_in(client)
|
|
events = client.get("/api/events").json()["events"]
|
|
assert events[0]["host"] == "deskwork"
|
|
assert events[0]["severity"] == "error"
|
|
store.close()
|
|
|
|
|
|
def test_event_rows_say_their_subject_once(client) -> None:
|
|
"""`poller.py` messages name their own subject, and the row prints a host chip too.
|
|
|
|
"gufo gufo is answering again after 1 failure" reads as two problems — the same defect
|
|
`alerts._clean_error()` already fixes for the overview's alert list. `/ups` had the
|
|
mirror-image bug: it `replace()`d `host + ':'` *anywhere*, so "Quick battery test sent
|
|
to apc: OK" printed as "Quick battery test sent to OK". All three feeds now share
|
|
`Dash.eventText()`, which strips the wrapper and a **leading** name only.
|
|
"""
|
|
app_js = client.get("/static/app.js").text
|
|
helper = app_js[app_js.index("function eventText("):app_js.index("// ---- DOM helpers")]
|
|
assert "eventText" in app_js[app_js.index("return {"):], "eventText is not exported"
|
|
assert "^UpstreamError:\\s+" in helper, "the wrapper `alerts.py` drops is back"
|
|
assert "new RegExp" in helper and "stripped.trim() ? stripped : out" in helper, \
|
|
"the host strip is no longer anchored to the front, or can empty a message"
|
|
for page in ("/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js"):
|
|
js = client.get(page).text
|
|
assert "eventText(event)" in js, f"{page} prints the raw message again"
|
|
assert "replace(event.host" not in client.get("/static/page-ups.js").text, \
|
|
"/ups is mangling the host name mid-message again"
|
|
|
|
|
|
# ---- the one write endpoint ------------------------------------------------ #
|
|
|
|
def test_command_endpoint_is_post_only(ups_client) -> None:
|
|
assert ups_client.post("/api/ups/command", json={}).status_code in (400, 403, 404)
|
|
assert ups_client.put("/api/ups/command", json={}).status_code == 405
|
|
assert ups_client.delete("/api/ups/command").status_code == 405
|
|
|
|
|
|
def test_commands_are_offered_when_actions_are_enabled(ups_client) -> None:
|
|
body = wait_for_payload(ups_client, "/api/ups/snapshot")
|
|
assert body["simulated"] is True, "the simulator must label itself"
|
|
assert body["actions_enabled"] is True
|
|
ups = body["ups"][0]
|
|
enabled = [item for item in ups["commands"] if item["enabled"]]
|
|
assert enabled, "with NUT_ALLOW_ACTIONS=1 and a simulator, safe commands are sendable"
|
|
assert all(item["risk"] in ("safe", "caution", "dangerous") for item in ups["commands"])
|
|
|
|
|
|
def test_a_safe_command_actually_reaches_the_driver(ups_client) -> None:
|
|
body = wait_for_payload(ups_client, "/api/ups/snapshot")
|
|
name = body["ups"][0]["name"]
|
|
response = ups_client.post("/api/ups/command",
|
|
json={"ups": name, "command": "test.panel.start"})
|
|
assert response.status_code == 200, response.text
|
|
payload = response.json()
|
|
assert payload["ok"] is True and payload["simulated"] is True
|
|
events = ups_client.get("/api/events").json()["events"]
|
|
assert any("test.panel.start" in event["message"] or "Panel test" in event["message"]
|
|
for event in events), "every sent command must leave an audit trail"
|
|
|
|
|
|
def test_a_dangerous_command_needs_the_confirmation_word(ups_client) -> None:
|
|
name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
|
|
deep = "test.battery.start.deep"
|
|
refusal = ups_client.post("/api/ups/command", json={"ups": name, "command": deep})
|
|
assert refusal.status_code == 422
|
|
assert "DANGER" in refusal.json()["detail"]
|
|
assert ups_client.post("/api/ups/command",
|
|
json={"ups": name, "command": deep, "confirm": "DANGER"}).status_code == 200
|
|
|
|
|
|
def test_a_command_the_driver_does_not_advertise_is_refused_before_it_is_sent(
|
|
ups_client) -> None:
|
|
"""Being in the risk catalog is not permission; `LIST CMD` from the driver is (invariant 1).
|
|
|
|
`shutdown.return` is a real NUT command the catalog knows and this UPS does not offer,
|
|
so the refusal must come from the advertisement check and not from the confirmation gate.
|
|
"""
|
|
name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
|
|
response = ups_client.post("/api/ups/command",
|
|
json={"ups": name, "command": "shutdown.return"})
|
|
assert response.status_code == 400
|
|
assert "does not advertise" in response.json()["detail"]
|
|
|
|
|
|
def test_cutting_the_output_is_refused_while_the_ups_is_already_on_battery(
|
|
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""Invariant 1, the one that matters most: no power cut while the pack is carrying the load.
|
|
|
|
Driven through the simulator's own state machine — a quick battery test puts the pack on
|
|
battery — so the 409 comes from the same poll the buttons are drawn from, with a 0.5s
|
|
interval to keep the wait short. Nothing here can touch real hardware: `NUT_SIMULATE=1`.
|
|
"""
|
|
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
|
|
nut_allow_actions=True, nut_interval=0.5) as test_client:
|
|
name = wait_for_payload(test_client, "/api/ups/snapshot")["ups"][0]["name"]
|
|
start = test_client.post("/api/ups/command",
|
|
json={"ups": name, "command": "test.battery.start.quick"})
|
|
assert start.status_code == 200
|
|
deadline = time.time() + 6.0
|
|
card: dict[str, Any] = {}
|
|
while time.time() < deadline:
|
|
card = next(item for item in test_client.get("/api/ups/snapshot").json()["ups"]
|
|
if item["name"] == name)
|
|
if card["on_battery"]:
|
|
break
|
|
time.sleep(0.05)
|
|
assert card.get("on_battery") is True, "the quick test must put the simulated pack on battery"
|
|
refusal = test_client.post("/api/ups/command",
|
|
json={"ups": name, "command": "load.off",
|
|
"confirm": "DANGER"})
|
|
assert refusal.status_code == 409
|
|
assert "on battery" in refusal.json()["detail"]
|
|
# and the same command is still offered once the pack is back on line power
|
|
assert not any(item["command"] == "load.off" and item["enabled"]
|
|
for item in card["commands"]), "the panel must not offer a blocked button"
|
|
|
|
|
|
class ScriptedNutClient:
|
|
"""`nut.NutClient` with the socket left out: the real code path, no battery attached.
|
|
|
|
`AGENTS.md` keeps live tests read-only, so this is where the one write endpoint's real
|
|
branch gets proven. The variable names are the ones `VAR_MAP` already pins from real
|
|
upsd captures; only the replies are scripted (a fake for behaviour, per the test policy).
|
|
"""
|
|
|
|
#: set per test: an exception instance to raise from `instcmd`, e.g. a refusal
|
|
refusal: Exception | None = None
|
|
sent: list[tuple[str, str]]
|
|
logged_in: list[str]
|
|
|
|
def __init__(self, host: str, port: int, *, timeout: float = 4.0,
|
|
username: str | None = None, password: str | None = None) -> None:
|
|
self.credentials = (username, password)
|
|
|
|
async def __aenter__(self) -> ScriptedNutClient:
|
|
return self
|
|
|
|
async def __aexit__(self, *_exc: object) -> None:
|
|
return None
|
|
|
|
async def list_ups(self) -> list[dict[str, str]]:
|
|
return [{"name": "apc", "description": "APC Back-UPS RS 1500MS"}]
|
|
|
|
async def list_vars(self, ups: str) -> dict[str, str]:
|
|
return {"ups.mfr": "APC", "ups.model": "Back-UPS RS 1500MS",
|
|
"ups.status": "OL", "battery.charge": "100", "ups.load": "18",
|
|
"battery.voltage": "13.6", "input.voltage": "230",
|
|
"battery.mfr.date": "2022-04", "ups.test.result": "Ok last test passed"}
|
|
|
|
async def list_cmds(self, ups: str) -> list[str]:
|
|
return ["test.panel.start", "beeper.mute", "load.off"]
|
|
|
|
async def list_rw_vars(self, ups: str) -> list[str]:
|
|
return []
|
|
|
|
async def login(self) -> None:
|
|
assert self.credentials[0], "a command send must authenticate as a named user"
|
|
ScriptedNutClient.logged_in.append(str(self.credentials[0]))
|
|
|
|
async def instcmd(self, ups: str, command: str) -> str:
|
|
if ScriptedNutClient.refusal is not None:
|
|
raise ScriptedNutClient.refusal
|
|
ScriptedNutClient.sent.append((ups, command))
|
|
return "OK Start Panel Test"
|
|
|
|
|
|
@pytest.fixture
|
|
def real_upsd_client(monkeypatch: pytest.MonkeyPatch) -> Iterator[type[ScriptedNutClient]]:
|
|
"""Swap the module-level `NutClient` so the collector and the send use the same stand-in."""
|
|
ScriptedNutClient.sent = []
|
|
ScriptedNutClient.logged_in = []
|
|
ScriptedNutClient.refusal = None
|
|
monkeypatch.setattr(nut, "NutClient", ScriptedNutClient)
|
|
yield ScriptedNutClient
|
|
|
|
|
|
@pytest.fixture
|
|
def nut_client_app(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
|
real_upsd_client: type[ScriptedNutClient]) -> Iterator[TestClient]:
|
|
"""A non-simulated UPS app: real code paths, scripted upsd, 0.5s polls."""
|
|
with build_client(tmp_path, monkeypatch, nut_host="127.0.0.1", nut_user="dashboards",
|
|
nut_password="not-the-real-one", nut_allow_actions=True,
|
|
nut_interval=0.5) as test_client:
|
|
yield test_client
|
|
|
|
|
|
def test_a_real_upsd_send_authenticates_sends_and_audits(nut_client_app,
|
|
real_upsd_client) -> None:
|
|
body = wait_for_payload(nut_client_app, "/api/ups/snapshot")
|
|
assert body["simulated"] is False, "this test is about the branch that talks to upsd"
|
|
name = body["ups"][0]["name"]
|
|
response = nut_client_app.post("/api/ups/command",
|
|
json={"ups": name, "command": "test.panel.start"})
|
|
assert response.status_code == 200, response.text
|
|
assert response.json()["simulated"] is False
|
|
assert real_upsd_client.sent == [(name, "test.panel.start")]
|
|
assert real_upsd_client.logged_in, "instcmd requires a LOGIN first (upsd refuses otherwise)"
|
|
events = nut_client_app.get("/api/events").json()["events"]
|
|
assert any(event["kind"] == "action" and name in event["message"] for event in events)
|
|
|
|
|
|
def test_a_refusal_from_upsd_is_a_502_and_an_error_event(nut_client_app,
|
|
real_upsd_client) -> None:
|
|
real_upsd_client.refusal = nut.NutError("Access denied", code="ACCESS-DENIED")
|
|
name = wait_for_payload(nut_client_app, "/api/ups/snapshot")["ups"][0]["name"]
|
|
response = nut_client_app.post("/api/ups/command",
|
|
json={"ups": name, "command": "beeper.mute"})
|
|
assert response.status_code == 502
|
|
detail = response.json()["detail"]
|
|
assert "ACCESS-DENIED" in detail or "Access denied" in detail, detail
|
|
assert real_upsd_client.sent == [], "a refused command was never accepted"
|
|
events = nut_client_app.get("/api/events").json()["events"]
|
|
failures = [event for event in events if event["kind"] == "action"
|
|
and event["severity"] == "error"]
|
|
assert failures, "an upsd refusal has to leave a trace, not just an HTTP status"
|
|
|
|
|
|
def test_unknown_commands_are_refused_even_with_actions_on(ups_client) -> None:
|
|
name = wait_for_payload(ups_client, "/api/ups/snapshot")["ups"][0]["name"]
|
|
for command in ("load.format", "rm -rf /", "INSTCMD x y"):
|
|
response = ups_client.post("/api/ups/command", json={"ups": name, "command": command})
|
|
assert response.status_code == 400, command
|
|
|
|
|
|
def test_commands_for_an_unknown_ups_are_404(ups_client) -> None:
|
|
wait_for_payload(ups_client, "/api/ups/snapshot")
|
|
response = ups_client.post("/api/ups/command",
|
|
json={"ups": "not-real", "command": "beeper.mute"})
|
|
assert response.status_code == 404
|
|
|
|
|
|
def test_missing_fields_are_a_400(ups_client) -> None:
|
|
assert ups_client.post("/api/ups/command", json={"ups": ""}).status_code == 400
|
|
|
|
|
|
def test_actions_flag_off_blocks_every_command(tmp_path: Path,
|
|
monkeypatch: pytest.MonkeyPatch) -> None:
|
|
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1") as client:
|
|
body = wait_for_payload(client, "/api/ups/snapshot")
|
|
name = body["ups"][0]["name"]
|
|
assert body["actions_enabled"] is False
|
|
response = client.post("/api/ups/command",
|
|
json={"ups": name, "command": "beeper.mute"})
|
|
assert response.status_code == 403
|
|
assert "NUT_ALLOW_ACTIONS" in response.json()["detail"]
|
|
|
|
|
|
def test_read_only_beats_the_actions_flag(tmp_path: Path,
|
|
monkeypatch: pytest.MonkeyPatch) -> None:
|
|
with build_client(tmp_path, monkeypatch, simulate=True, nut_host="127.0.0.1",
|
|
nut_allow_actions=True, read_only=True) as client:
|
|
body = wait_for_payload(client, "/api/ups/snapshot")
|
|
name = body["ups"][0]["name"]
|
|
response = client.post("/api/ups/command",
|
|
json={"ups": name, "command": "beeper.mute"})
|
|
assert response.status_code == 403
|
|
commands = wait_for_payload(client, "/api/ups/commands")
|
|
assert commands["read_only"] is True
|
|
assert all(item["enabled"] is False for ups in commands["ups"]
|
|
for item in ups["commands"])
|
|
|
|
|
|
def test_ups_history_answers_for_simulated_ups(ups_client) -> None:
|
|
wait_for_payload(ups_client, "/api/ups/snapshot")
|
|
time.sleep(1.2)
|
|
body = ups_client.get("/api/ups/history", params={"metrics": "ups.charge_pct",
|
|
"range": "1h"}).json()
|
|
assert body["ups"], "the UPS page needs a name to chart before any history exists"
|
|
|
|
|
|
def response_ok(response) -> bool:
|
|
assert response.status_code == 200, response.text
|
|
return True
|
|
|
|
|
|
def test_samples_written_by_the_poller_are_queryable(tmp_path: Path,
|
|
monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""The write path (poller -> store) and the read path (/api/history) must agree on names."""
|
|
db = str(tmp_path / "roundtrip.sqlite3")
|
|
store = SampleStore(db)
|
|
store.add_many([Sample("llm", "gufo", "llm.decode_tokens_per_second", 40.0, time.time() - 60),
|
|
Sample("llm", "gufo", "llm.decode_tokens_per_second", 45.0, time.time() - 30)])
|
|
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
|
pbkdf2_iterations=FAST_PBKDF2), store)
|
|
with TestClient(app) as client:
|
|
sign_in(client)
|
|
metrics = client.get("/api/metrics", params={"source": "llm"}).json()["metrics"]
|
|
assert "llm.decode_tokens_per_second" in metrics
|
|
points = client.get("/api/history", params={"source": "llm", "host": "gufo",
|
|
"metric": "llm.decode_tokens_per_second",
|
|
"range": "15m"}).json()["points"]
|
|
assert [p["value"] for p in points] == [40.0, 45.0]
|
|
store.close()
|