phase: 09_disks_smartctl

**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**

- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
This commit is contained in:
2026-10-18 00:45:01 -04:00
parent 19cfedfddf
commit fefbeb8227
81 changed files with 5525 additions and 134 deletions
+672 -30
View File
@@ -14,6 +14,7 @@ from contextlib import contextmanager
from pathlib import Path
from typing import Any
import httpx
import pytest
from fastapi import HTTPException
from fastapi.testclient import TestClient
@@ -21,9 +22,11 @@ from fastapi.testclient import TestClient
from dashboards import alerts
from dashboards.api import RANGE_SECONDS, parse_range
from dashboards.app import create_app
from dashboards.collectors import UpstreamError
from dashboards.config import Settings, Target
from dashboards.severity import classify
from dashboards.sources import nut
from dashboards.sources.disks import SMART_SERIES, DisksCollector, parse_disks_snapshot
from dashboards.sources.llm import (
HISTORY_METRICS,
parse_gufo_metrics,
@@ -44,6 +47,11 @@ from slots_contract import (
# refused immediately, and never dials out of the test environment
DEAD_URL = "http://127.0.0.1:1"
#: Every dashboard route, in nav order. The markup pins below (no CDN, no `.sev-*`, no
#: narration, no orphan `id`) run across this list rather than a hand-written four, so a
#: fifth page cannot join the masthead and skip the checks the others had to pass.
PAGES = ("/", "/system", "/disks", "/llm", "/ups")
@contextmanager
def build_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, *, simulate: bool = False,
@@ -137,22 +145,36 @@ def data_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
def healthy_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
agent_snapshot: dict[str, Any],
host_scrape: Any) -> Iterator[TestClient]:
"""The all-clear app: one host answering, nothing past a threshold, no UPS configured.
"""The all-clear app: one host answering on both of its jobs, nothing past a threshold.
The job's collector is swapped for one that hands back the injected payload, so the
health record says `ok` truthfully instead of being overridden behind the poller.
health record says `ok` truthfully instead of being overridden behind the poller — and
that has to happen for the `disks` job too, which polls the same dead port: a host is
either answering or it is not, and a fixture that claims all-clear cannot leave one of
its two readers failing. Each reader gets **its own** projection of the same capture
(`parse_disks_snapshot` for the drive page), because a `disks` job holding a `system`
payload would make `/api/summary`'s drive block answer from the wrong shape and read as
a passing test that proved nothing.
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
from dashboards.poller import Result # noqa: PLC0415 - only this fixture needs it
payload = host_scrape(agent_snapshot, cpu_busy=10.0)
disks_payload = parse_disks_snapshot("driveripper", agent_snapshot,
DisksCollector()).payload
async def collect() -> Result:
return Result(payload=payload, samples=[])
job = test_client.app.state.ctx.hub.jobs["host:driveripper"] # type: ignore[attr-defined]
async def collect_disks() -> Result:
return Result(payload=disks_payload, samples=[])
hub = test_client.app.state.ctx.hub # type: ignore[attr-defined]
job = hub.jobs["host:driveripper"]
disks_job = hub.jobs["disks:driveripper"]
job.fn = collect
disks_job.fn = collect_disks
# let the poll already in flight against the dead port land first (it fails in
# microseconds), then take the health record over: the alerts must describe a host
# that is answering, not the half-second of startup where it was not
@@ -160,11 +182,14 @@ def healthy_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
while time.time() < deadline and job.state.attempts == 0:
time.sleep(0.02)
time.sleep(0.05)
job.state.payload = payload
job.state.ok = True
job.state.error = None
job.state.consecutive_failures = 0
job.state.last_ok = time.time()
for reader in hub.jobs.values():
if reader.dashboard not in ("system", "disks"):
continue
reader.state.payload = disks_payload if reader is disks_job else payload
reader.state.ok = True
reader.state.error = None
reader.state.consecutive_failures = 0
reader.state.last_ok = time.time()
yield test_client
@@ -204,7 +229,8 @@ def test_summary_shape_did_not_shrink(data_client) -> None:
assert fleet_keys <= set(body["system"]["fleet"]), fleet_keys - set(body["system"]["fleet"])
assert host_keys <= set(body["system"]["hosts"][0]), host_keys - set(body["system"]["hosts"][0])
assert ups_keys <= set(body["ups"]["ups"][0]), ups_keys - set(body["ups"]["ups"][0])
assert set(body) == {"ts", "system", "llm", "ups", "recent_events", "alerts"}
# `disks` joined in phase 09 and changed nothing else: the set grows, never shrinks
assert set(body) == {"ts", "system", "disks", "llm", "ups", "recent_events", "alerts"}
# ...and the new keys are there beside the old ones, not instead of them
assert "severity" in body["system"]["fleet"]
assert "severity" in body["system"]["hosts"][0]
@@ -215,6 +241,7 @@ def test_summary_shape_did_not_shrink(data_client) -> None:
# read by something that is not the overview page: the live suite, an external probe, a
# notifier later. Deleting a card must never delete a key.
SUMMARY_CONTRACT = ("system.enabled", "system.fleet", "system.hosts",
"disks.enabled", "disks.fleet", "disks.hosts",
"llm.engines", "ups.ups", "recent_events", "alerts")
@@ -245,6 +272,11 @@ def test_summary_serves_every_path_another_consumer_reads(request, fixture: str)
assert (dig(body, "system.fleet") is not None) is populated
names = {host["name"] for host in dig(body, "system.hosts")}
assert names == ({"driveripper"} if populated else set())
# the drive-health block is configured exactly when the agents it reads are, and lists the
# same hosts: a `disks` job exists for every `HOST_TARGETS` entry and for no other reason
assert dig(body, "disks.enabled") is populated
assert (dig(body, "disks.fleet") is not None) is populated
assert {host["name"] for host in dig(body, "disks.hosts")} == names
batteries = {item["name"] for item in dig(body, "ups.ups")}
assert batteries == ({"apc", "cyberpower"} if populated else set())
assert dig(body, "llm.engines") == [], "neither fixture configures an engine"
@@ -342,6 +374,9 @@ def test_config_reports_every_source_as_off(client) -> None:
body = client.get("/api/config").json()
assert body["system"]["enabled"] is False
assert body["system"]["targets"] == []
assert body["disks"] == {"enabled": False, "interval_seconds": 300.0,
"timeout_seconds": 30.0}, (
"the drive-health page is configured exactly when the host agents are")
assert body["llm"]["gufo"]["enabled"] is False
assert body["llm"]["vllm"]["enabled"] is False
assert body["ups"]["enabled"] is False
@@ -357,6 +392,31 @@ def test_health_with_nothing_configured_is_not_a_failure(client) -> None:
assert body["sources"] == []
def test_the_disk_source_has_its_own_health_record_per_host(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""`disks` is a source of its own, not a field of `system` (phase 09).
Its own record is what gives the page its own dot and its own staleness, and it is
also what makes `alerts.py` need a rule: two jobs read the same agent, so one dead
host must still be one line on the overview (see `test_summary_raises_one_alert...`).
"""
with build_client(tmp_path, monkeypatch,
host_targets=(Target("driveripper", DEAD_URL),)) as test_client:
sources = {row["key"]: row for row in test_client.get("/api/health").json()["sources"]}
assert "disks:driveripper" in sources, "no separate record, no separate dot"
assert sources["disks:driveripper"]["dashboard"] == "disks"
assert sources["disks:driveripper"]["label"] == "driveripper"
assert sources["disks:driveripper"]["configured"] is True
assert sources["host:driveripper"]["dashboard"] == "system", (
"the system job is still its own source: one slow SMART poll must not "
"decide whether the host looks alive")
def test_no_targets_means_no_disk_job_that_could_look_broken(client) -> None:
"""No host agents, no `disks` task at all — so nothing red for an unconfigured page."""
assert client.get("/api/health").json()["sources"] == []
@pytest.mark.parametrize("path", ["/api/system/snapshot", "/api/llm/snapshot",
"/api/ups/snapshot", "/api/ups/commands"])
def test_unconfigured_source_is_404_configured_false(client, path) -> None:
@@ -380,6 +440,30 @@ def test_unknown_route_is_json_not_html(client) -> None:
assert response.headers["content-type"].startswith("application/json")
assert "no such route" in response.json()["detail"]
def test_an_upstream_failure_escaping_a_route_keeps_its_own_status(tmp_path: Path) -> None:
"""`collectors.UpstreamError` is the one error type a route may raise to say "upstream".
The handler turns it into `{ok: false, detail}` with the collector's status (502, or 504
for a timeout) instead of a 500 traceback, and the message is the collector's already
redacted sentence. Nothing dials here: the error is raised by the route under test.
"""
db = str(tmp_path / "upstream-error.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db), store)
@app.get("/api/_boom")
async def boom() -> None:
raise UpstreamError("driveripper: http://127.0.0.1:9121/snapshot timed out", status=504)
with TestClient(app) as test_client:
response = test_client.get("/api/_boom")
assert response.status_code == 504, "a timeout stays a timeout, not a 500"
assert response.json() == {"ok": False,
"detail": "driveripper: http://127.0.0.1:9121/snapshot "
"timed out"}
assert "no-store" in response.headers["cache-control"]
store.close()
# ---- tokens --------------------------------------------------------------- #
@@ -433,12 +517,20 @@ def test_a_dead_host_never_writes_its_token_into_the_feed(tmp_path: Path) -> Non
time.sleep(0.05)
else:
pytest.fail("the dead host never recorded an error")
for path in ("/api/health", "/api/events", "/api/system/snapshot", "/api/summary"):
for path in ("/api/health", "/api/events", "/api/system/snapshot",
"/api/disks/snapshot", "/api/summary"):
body = client.get(path).text
assert "SEKRIT-AGENT-TOKEN" not in body, f"{path} leaked the agent token"
# and the outage is still legible — redaction is not the same as silence
assert "driveripper" in client.get("/api/health").text
assert "unreachable" in client.get("/api/events").text
# the `disks` job dials the same token-bearing URL on its own cadence, so its
# sentence is a second copy of the same risk: `fetch_snapshot` is the one place
# either dashboard builds a failure message, and it uses the redacted URL
disks_state = next(row for row in client.get("/api/health").json()["sources"]
if row["key"] == "disks:driveripper")
assert disks_state["error"] and "SEKRIT-AGENT-TOKEN" not in disks_state["error"]
assert "127.0.0.1:1/snapshot" in disks_state["error"], "the failure still names where"
store.close()
@@ -478,7 +570,8 @@ def test_target_problems_are_surfaced_not_swallowed(client) -> None:
# ---- pages ---------------------------------------------------------------- #
@pytest.mark.parametrize("path,title", [
("/", "overview"), ("/system", "system"), ("/llm", "llm"), ("/ups", "ups"),
("/", "overview"), ("/system", "system"), ("/disks", "disks"), ("/llm", "llm"),
("/ups", "ups"),
])
def test_pages_serve_and_declare_themselves(client, path, title) -> None:
response = client.get(path)
@@ -489,16 +582,14 @@ def test_pages_serve_and_declare_themselves(client, path, title) -> None:
def test_no_page_loads_anything_from_the_internet(client) -> None:
"""No CDN, no webfonts: this dashboard has to work with no route to the internet."""
for path in ("/", "/system", "/llm", "/ups"):
for path in PAGES:
html = client.get(path).text
for fragment in ("//cdn.", "https://fonts.", "//unpkg.com", "//cdnjs"):
assert fragment not in html, f"{path} references {fragment}"
def test_static_assets_are_served(client) -> None:
for asset in ("/static/dash.css", "/static/app.js", "/static/charts.js",
"/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js",
"/static/page-index.js"):
for asset in ASSETS:
assert client.get(asset).status_code == 200, asset
@@ -511,14 +602,26 @@ def test_static_assets_are_served(client) -> None:
# comparing the number to a threshold in JavaScript.
ASSETS = ("/static/dash.css", "/static/app.js", "/static/charts.js",
"/static/page-index.js", "/static/page-system.js", "/static/page-llm.js",
"/static/page-ups.js")
"/static/page-index.js", "/static/page-disks.js", "/static/page-system.js",
"/static/page-llm.js", "/static/page-ups.js")
def _css(client: TestClient) -> str:
return client.get("/static/dash.css").text
def _css_section(css: str, name: str) -> str:
"""One `/* ---- name … */` section of the stylesheet, up to the **next** section header.
Slicing to a named neighbour worked while the sheet had five sections. Every page has since
added a component section, and a slice that runs across someone else's rules accuses the
fleet block of colouring numbers it never coloured — so the boundary is structural: the end
of *this* section, whoever comes next.
"""
start = css.index(f"/* ---- {name}")
return css[start:css.index("/* ----", start + 10)]
def _hero_css(css: str) -> str:
"""Just the `hero numbers` section of the stylesheet."""
return css.split("/* ---- hero numbers", 1)[1].split("/* ---- bars", 1)[0]
@@ -599,7 +702,7 @@ def _fleet_field_metrics(js: str) -> list[str]:
def test_no_parallel_severity_vocabulary_exists_anywhere(client) -> None:
"""One word per colour: `ok|warn|fail`, as in severity.py and the existing components."""
for asset in (*ASSETS, "/", "/system", "/llm", "/ups"):
for asset in (*ASSETS, *PAGES):
body = _code(client.get(asset).text)
assert ".sev-" not in body, f"{asset} invented a .sev-* class"
assert not re.search(r"classList\.(?:add|toggle)\(\s*[\"'](?:danger|good|warning)[\"']",
@@ -639,7 +742,7 @@ def test_severity_for_is_kept_as_the_documented_fallback(client) -> None:
js = client.get("/static/app.js").text
assert "FALLBACK ONLY" in js
assert "severityFor" in js, "the documented fallback itself was deleted"
for page in ("page-index.js", "page-system.js", "page-llm.js", "page-ups.js"):
for page in PAGE_SCRIPTS.values():
assert "severityFor" not in client.get(f"/static/{page}").text, \
f"{page} colours something from a threshold of its own"
@@ -924,7 +1027,7 @@ def test_every_system_chart_names_itself_for_assistive_tech(client) -> None:
# ---- event feeds: a bounded, paginated window ----------------------------- #
#
# The three `ul.events` feeds are one component (`Dash.eventsPager`) because a pager that
# Every `ul.events` feed is one component (`Dash.eventsPager`) because a pager that
# works on only one page is the bug this shape prevents: the `event` table grows for as long
# as the fleet runs, and an unbounded feed turns `/system` into a scroll to nowhere
# (`TODO.md` L6 — "All the events tables need to be scrollable and paginated"). There is no
@@ -935,6 +1038,7 @@ def test_every_system_chart_names_itself_for_assistive_tech(client) -> None:
# page -> (its script, the `source=` the poller stores that page's events under)
EVENT_FEEDS = {
"/system": ("page-system.js", "system"),
"/disks": ("page-disks.js", "disks"),
"/llm": ("page-llm.js", "llm"),
"/ups": ("page-ups.js", "ups"),
}
@@ -953,11 +1057,12 @@ def _feed_fetch_url(js: str) -> str:
return found.group(1)
def test_the_event_feeds_are_one_shared_pager_not_three(client) -> None:
"""`Dash.eventsPager()` renders all three feeds, and no page grows its own pager.
def test_the_event_feeds_are_one_shared_pager_not_one_per_page(client) -> None:
"""`Dash.eventsPager()` renders every feed, and no page grows its own pager.
Three copies of a pager is three chances for one page to keep the old behaviour — and
`/system` is the one whose feed was already too tall when this was written down.
One copy per page is one chance per page to keep the old behaviour — and `/system` is the
one whose feed was already too tall when this was written down. `/disks` joined as a
fourth call site of the same helper, not as a fifth implementation of it.
"""
app_js = _code(client.get("/static/app.js").text)
assert app_js.count("function eventsPager(") == 1
@@ -1913,7 +2018,7 @@ def test_llm_fleet_block_colours_no_number(client) -> None:
"a dead engine no longer dims the tag or darkens the block's dot"
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- fleet totals"):css.index("/* ---- hero numbers")]
section = _css_section(css, "fleet totals")
for token in ("var(--ok)", "var(--warn)", "var(--fail)"):
assert token not in section, f"the fleet block coloured a number with {token}"
assert ".fleet-head" in section and ".fleet-note" in section, \
@@ -2004,7 +2109,7 @@ def test_llm_slot_grid_style_is_a_marker_and_not_a_verdict(client) -> None:
the marker survives monochrome, which is the battery glyph's rule applied here.
"""
css = client.get("/static/dash.css").text
section = css[css.index("/* ---- engine slot grid"):css.index("/* ---- fleet totals")]
section = _css_section(css, "engine slot grid")
for selector in (".slots {", ".slot {", ".slot.busy", ".slot-top", ".slot-id",
".slot-model", ".slot dt", ".slot dd", ".slots-empty", ".slots-note",
".section-label"):
@@ -2292,10 +2397,11 @@ def test_overview_dims_the_fleet_row_of_a_host_that_stopped_answering(client) ->
# Which script owns which page, for the orphan check below. `charts.js` and `app.js` are
# loaded by all four pages, so an id either page code or shared chrome may read.
# loaded by every page, so an id either page code or shared chrome may read.
PAGE_SCRIPTS = {
"/": "page-index.js",
"/system": "page-system.js",
"/disks": "page-disks.js",
"/llm": "page-llm.js",
"/ups": "page-ups.js",
}
@@ -2366,7 +2472,7 @@ def test_no_page_narrates_its_own_pipeline(client, fragment: str) -> None:
the same sentence migrating into another page's card is the same slop, and a pin that
only guards its original file would let that through.
"""
for path in ("/", "/system", "/llm", "/ups", *ASSETS):
for path in (*PAGES, *ASSETS):
body = client.get(path).text
assert fragment not in body, f"{path} narrates the app again: {fragment!r}"
@@ -2378,7 +2484,7 @@ def test_pages_leave_no_empty_narration_container(client) -> None:
renders a gap where the sentence used to be. Elements the scripts fill (`#sub`,
`#alerts-count`) carry an `id` and are not in this category.
"""
for path in ("/", "/system", "/llm", "/ups"):
for path in PAGES:
html = _code(client.get(path).text)
for cls in ("sub", "faint", "mono"):
assert f'<span class="{cls}"></span>' not in html, \
@@ -2393,12 +2499,16 @@ def test_the_copy_the_operator_acts_on_survives_the_audit(client) -> None:
is this number a measurement, is this battery real, is this host's data last-good —
and each one is the reason the audit deletes sentences and not labels.
"""
pages = {path: client.get(path).text for path in ("/", "/system", "/llm", "/ups")}
pages = {path: client.get(path).text for path in PAGES}
scripts = {name: client.get(f"/static/{name}").text for name in PAGE_SCRIPTS.values()}
# an empty panel says why, in terms the operator can act on
assert "No hosts configured" in pages["/"]
assert "HOST_TARGETS" in pages["/system"]
# …including the fourth page's: a `n/a` drive card is only actionable if the page names
# the two things that make SMART readable (the binary and the device-read grant).
assert "smartmontools" in pages["/disks"] + scripts["page-disks.js"]
assert "disk group" in scripts["page-disks.js"]
assert "No engines configured" in pages["/llm"]
assert "No UPS reporting" in pages["/ups"]
assert "instcmds" in pages["/ups"], "the UPS setup note stopped naming the grant that matters"
@@ -2612,6 +2722,538 @@ def test_metrics_endpoint_lists_what_actually_has_history(client) -> None:
assert body["metrics"] == []
# ---- the `/disks` API (phase 09 task 03) ---------------------------------- #
#
# The drive-health surface answers like every other dashboard's: GETs only, the same
# stale/stub semantics for a host that is not answering, the same history mechanism for the
# `disk.*@dev` series, and the same 404-`configured: false` sentence when no agent is
# configured at all — the last one *without dialling*, because "not deployed here" must not
# cost a connect timeout or a red dot (invariant 3).
@pytest.fixture
def disks_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
disk_snapshot: Any) -> Iterator[TestClient]:
"""Two hosts configured; one of them answering SMART, through the production parser.
The payload and the stored samples are what `parse_disks_snapshot` makes of the
fixture-shaped capture (the agent's whole-disk rows plus the recorded `smartctl` shapes),
so every path a `/disks` card will read is a path the source really emits. The other host
keeps its `unreachable` stub, which is the shape a dead agent must keep (invariant 4).
Both targets point at a dead port, so nothing dials.
"""
db = str(tmp_path / "disks.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db, host_targets=(Target("driveripper", DEAD_URL),
Target("deskwork", DEAD_URL))), store)
with TestClient(app) as test_client:
now = time.time()
snapshot = disk_snapshot("driveripper")
snapshot["ts"] = now # "the agent answered just now", so charts are in range
result = parse_disks_snapshot("driveripper", snapshot, DisksCollector(), now=now)
store.add_many(result.samples)
app.state.ctx.hub.jobs["disks:driveripper"].state.payload = result.payload
yield test_client
store.close()
#: (`smart` field, severity metric) for the two drive numbers the table rates — the same
#: pair `sources/disks.py:SEVERITY_FIELDS` declares, restated here so a rename that only
#: touches one side fails rather than colouring nothing.
RATED_CELLS: tuple[tuple[str, str], ...] = (
("temp_c", "disk.temp_c"),
("percentage_used", "disk.percentage_used"),
)
def disk_rows(body: dict[str, Any], host: str = "driveripper") -> list[dict[str, Any]]:
"""The disk rows of one host out of a `/api/disks/snapshot` body."""
return next(row for row in body["hosts"] if row["name"] == host)["disks"]
def flat_routes(app: Any) -> list[Any]:
"""Every route the app serves, with `include_router()` results flattened to their routes.
The FastAPI this repo pins keeps an included router as an `_IncludedRouter` wrapper with
no `path` of its own, so walking `app.routes` alone sees five anonymous objects instead of
the API surface — and the verb assertion below is only as good as this walk.
"""
out: list[Any] = []
for route in app.routes:
inner = getattr(route, "original_router", None)
out.extend(inner.routes if inner is not None else [route])
return out
def series_source(body: dict[str, Any]) -> set[str]:
"""Which sample family a history answer came from: `disks`, never the 10-second `host`."""
return {series["source"] for series in body["series"]}
def test_disks_snapshot_answers_one_entry_per_host_with_its_physical_disks(
disks_client) -> None:
"""`GET /api/disks/snapshot` is per host: rows + `smart` + `smartctl_available`."""
body = disks_client.get("/api/disks/snapshot")
assert response_ok(body)
payload = body.json()
assert [host["name"] for host in payload["hosts"]] == ["deskwork", "driveripper"], (
"every configured host is a row, in name order, answering or not")
rows = disk_rows(payload)
assert rows, "the capture has 16 whole-disk devices, so the page has 16 cards"
assert all("/" not in row["device"] for row in rows), "a mount leaked into the disk list"
first = rows[0]
assert first["path"] == f"/dev/{first['device']}"
assert set(first) >= {"device", "path", "model", "size_bytes", "rotational",
"health", "smart", "severity"}
assert first["smart"]["power_on_hours"] > 0, "the drive's own counters travel verbatim"
live = next(host for host in payload["hosts"] if host["name"] == "driveripper")
assert live["smartctl_available"] is True
assert live["smart_errors"] == []
assert live["summary"]["disks"] == len(rows)
assert "unreachable" not in live, "answering is the absence of the flag, as on /system"
assert [state["key"] for state in payload["states"]] == ["disks:driveripper",
"disks:deskwork"], (
"the page's dots come from the disks jobs, never the system ones")
def test_disks_snapshot_carries_the_verdicts_the_page_will_paint(disks_client) -> None:
"""Colour is in the payload: each row's `severity` is what `severity.classify` says.
This is the pin that keeps the threshold out of the page — `page-disks.js` reads these
and never compares a drive temperature to a bound.
"""
rows = disk_rows(disks_client.get("/api/disks/snapshot").json())
assert "ok" in {row["severity"]["temp_c"] for row in rows}, "the capture is healthy"
for row in rows:
smart = row["smart"] or {}
assert set(row["severity"]) == {field for field, _metric in RATED_CELLS} | {"health"}
for field, metric in RATED_CELLS:
assert row["severity"][field] == classify(metric, smart.get(field)), (
f"{row['device']}.{field} is coloured by something other than the one table")
assert not {key for row in rows for key in row["severity"]} & {
"reallocated_sector_ct", "pending", "uncorrectable", "power_on_hours"}, (
"a counter is an alarm chip on presence, never a graded cell")
def test_a_configured_host_with_no_payload_yet_is_a_row_of_nas(disks_client) -> None:
"""The stub shape `/api/system/snapshot` hands out, so `n/a` is never a missing row.
A host in `HOST_TARGETS` that has not answered is still in the fleet (invariants 3 and
4) with its counts `None` rather than `0`: "unknown" and "no drives" are different
sentences (rule 2), and the fleet header counts it without summing it.
"""
payload = disks_client.get("/api/disks/snapshot").json()
stub = next(host for host in payload["hosts"] if host["name"] == "deskwork")
assert stub["unreachable"] is True
assert stub["disks"] == [], "nothing to draw, but the host is still listed"
assert stub["smartctl_available"] is None, "an agent that never answered says nothing"
assert stub["smart_errors"] == []
assert set(stub["summary"]) == {"disks", "with_smart", "health_failed", "alarm_disks"}
assert all(value is None for value in stub["summary"].values()), "no invented zeros"
fleet = payload["fleet"]
assert (fleet["hosts_total"], fleet["hosts_live"]) == (2, 1)
assert fleet["disks"] == len(disk_rows(payload))
def test_the_disks_fleet_header_is_summed_by_the_api_not_the_page(disks_client) -> None:
"""`fleet` is the header's only arithmetic source, and it carries no verdict."""
payload = disks_client.get("/api/disks/snapshot").json()
live = next(host for host in payload["hosts"] if host["name"] == "driveripper")
fleet = payload["fleet"]
assert fleet["disks"] == live["summary"]["disks"]
assert fleet["alarm_disks"] == live["summary"]["alarm_disks"] > 0, (
"the recorded ATA shape carries reallocated sectors, so the fleet has chips to draw")
assert "severity" not in fleet, "a count of drives is not a graded number"
def test_the_disks_surface_is_gets_only_and_the_write_stays_the_ups_one(
disks_client) -> None:
"""Invariant 1: the app has exactly one POST, and a fourth dashboard does not add one."""
routes = flat_routes(disks_client.app)
disks = [route for route in routes
if str(getattr(route, "path", "")).startswith("/api/disks")]
assert {route.path for route in disks} == {"/api/disks/snapshot", "/api/disks/history"}
verbs = {verb for route in disks for verb in (getattr(route, "methods", None) or set())}
assert verbs == {"GET"}, "a write appeared on the drive-health surface"
for verb in ("POST", "PUT", "PATCH", "DELETE"):
response = disks_client.request(verb, "/api/disks/snapshot", json={})
assert response.status_code == 405, f"{verb} /api/disks/snapshot was accepted"
writes = sorted(str(route.path) for route in routes
if "POST" in (getattr(route, "methods", None) or set()))
assert writes == ["/api/ups/command"], "one write endpoint, four gates, unchanged"
def test_config_prints_the_disks_cadence_the_page_explains_itself_with(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""A card four minutes old is behaving, not broken — and the interval is the proof.
`/disks` has no upstream of its own, so `enabled` follows `HOST_TARGETS` and the cadence
is its own. Both numbers come from config rather than a constant in the page, so changing
`DISKS_INTERVAL_SECONDS` moves the sentence and the poller together.
"""
with build_client(tmp_path, monkeypatch, host_targets=(Target("driveripper", DEAD_URL),),
disks_interval=120.0, disks_timeout=45.0) as test_client:
disks = test_client.get("/api/config").json()["disks"]
assert disks["enabled"] is True
assert disks["interval_seconds"] == 120.0
assert disks["timeout_seconds"] == 45.0
def test_disks_snapshot_404s_without_dialling_a_host_that_was_never_configured(
client, monkeypatch: pytest.MonkeyPatch) -> None:
"""The gate runs before anything that could reach the network (invariant 3).
Both spies raise, so a route that constructed a client or fetched a snapshot before
checking `HOST_TARGETS` fails here instead of costing a timeout and a red dot. The hub
assertion is the structural half: with no targets there is no `disks` job to dial with.
"""
calls: list[str] = []
def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
calls.append(f"GET {url}")
raise AssertionError(f"the unconfigured disks route dialled {url}")
def spy_fetch(*args: Any, **kwargs: Any) -> Any:
calls.append("fetch_snapshot")
raise AssertionError("the unconfigured disks route fetched an agent snapshot")
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
monkeypatch.setattr("dashboards.collectors.fetch_snapshot", spy_fetch)
response = client.get("/api/disks/snapshot")
assert response.status_code == 404
body = response.json()
assert body["configured"] is False
assert "HOST_TARGETS" in body["detail"], "the sentence names the fix"
assert calls == [], "an unconfigured page must not touch the network"
assert client.app.state.ctx.hub.jobs == {}, (
"no targets means no disks job, so nothing can age out or go red")
def test_disks_history_serves_the_dev_series_the_page_charts(disks_client) -> None:
"""One request, every series a host's cards could draw, points from the stored samples."""
rows = disk_rows(disks_client.get("/api/disks/snapshot").json())
metrics = [f"{base}@{row['device']}" for row in rows for _field, base in SMART_SERIES]
body = disks_client.get("/api/disks/history", params={
"hosts": "driveripper", "metrics": ",".join(metrics), "range": "1h"}).json()
assert body["hosts"] == ["driveripper"], "one host, one panel"
assert body["range_seconds"] == 3_600.0
assert [series["metric"] for series in body["series"]] == metrics
assert series_source(body) == {"disks"}, "the disks family, never the 10-second host one"
points = {(series["metric"]): series["points"] for series in body["series"]}
assert points["disk.temp_c@nvme0n1"][0]["value"] == pytest.approx(38.0)
assert points["disk.power_on_hours@nvme0n1"][0]["value"] == pytest.approx(8_760.0)
def test_disks_history_answers_empty_for_a_series_a_drive_does_not_publish(
disks_client) -> None:
"""Unknown names are an empty series, not a 404 — the /system and /llm contract.
The recorded ATA shape reports no `percentage_used`, so that drive's wear plot has
nothing to draw, and a device name the agent never listed behaves the same way. The card
asks for every series it could draw and reads `n/a` where there is nothing, instead of
the request failing.
"""
rows = disk_rows(disks_client.get("/api/disks/snapshot").json())
unworn = next(row["device"] for row in rows
if (row["smart"] or {}).get("percentage_used") is None)
body = disks_client.get("/api/disks/history", params={
"hosts": "driveripper",
"metrics": f"disk.percentage_used@{unworn},disk.temp_c@nosuchdisk,"
"disk.temp_c@nvme0n1",
"range": "1h"}).json()
by_metric = {series["metric"]: series["points"] for series in body["series"]}
assert by_metric[f"disk.percentage_used@{unworn}"] == [], "no measurement, no line"
assert by_metric["disk.temp_c@nosuchdisk"] == []
assert by_metric["disk.temp_c@nvme0n1"], "the series that does exist still answers"
def test_disks_history_charts_every_configured_host_in_one_request(disks_client) -> None:
"""With no `hosts` the answer covers every configured agent, dead ones included (empty)."""
body = disks_client.get("/api/disks/history",
params={"metrics": "disk.temp_c@nvme0n1,disk.temp_c@sda",
"range": "24h"}).json()
assert body["hosts"] == ["driveripper", "deskwork"]
assert body["range_seconds"] == 86_400.0
assert {(series["host"], series["metric"]) for series in body["series"]} == {
("driveripper", "disk.temp_c@nvme0n1"), ("driveripper", "disk.temp_c@sda"),
("deskwork", "disk.temp_c@nvme0n1"), ("deskwork", "disk.temp_c@sda")}
assert [series for series in body["series"] if series["host"] == "deskwork"], (
"a host with no history is an empty series, never a missing one")
def test_the_metrics_endpoint_answers_for_the_disks_family(disks_client) -> None:
""""Why is this chart empty" has an answer per page: /api/metrics?source=disks."""
body = disks_client.get("/api/metrics", params={"source": "disks"}).json()
assert "disk.temp_c@nvme0n1" in body["metrics"]
assert not [name for name in body["metrics"] if not name.startswith("disk.")], (
"the disks family holds only disk series")
assert not [name for name in disks_client.get(
"/api/metrics", params={"source": "host"}).json()["metrics"]
if name.startswith("disk.temp_c")], "SMART never mixes into the system family"
# ---- the `/disks` page (phase 09 task 04) ---------------------------------- #
#
# The fourth dashboard, asserted the way the other three are: on the served bytes, because a
# JS test runner would be a build step (a locked anchor). What is pinned is not that the page
# exists but that it keeps the four promises the phase wrote down — one card per **physical**
# drive and never one per partition, alarm chips that exist only while a counter is non-zero,
# a host without `smartctl` dimmed rather than reddened, and charts that belong to one host,
# stay collapsed until opened, and cost one history request per host per refresh.
def _disks_js(client: TestClient) -> str:
return client.get("/static/page-disks.js").text
def test_the_disks_tab_is_in_the_masthead_of_every_page(client) -> None:
"""Four tabs became five, on every page, with exactly one marking itself current.
The masthead is the app's only "which dashboard do I need" surface since phase 02 deleted
the overview's table, so a tab missing from one page is a page you cannot get to, and a
tab marked current on the wrong page is a lie about where you are.
"""
for path in PAGES:
html = client.get(path).text
assert '<a href="/disks"' in html, f"{path}'s nav has no Disks tab"
assert html.count('aria-current="page"') == 1, f"{path} marks more than one tab current"
assert f'<a href="{path}" aria-current="page">' in html, f"{path} does not mark itself"
assert (html.index('href="/system"') < html.index('href="/disks"')
< html.index('href="/llm"')), "tab order is machine, drive, engine, power"
def test_the_header_status_dot_picks_up_the_disks_job(client) -> None:
"""`disks` gets its own dot instead of hiding inside `hosts`.
The same agent can answer the 10-second utilisation poll and have a wedged SMART pass;
one dot for both would have to be right about neither.
"""
app_js = client.get("/static/app.js").text
assert "{ key: 'disks', label: 'disks', href: '/disks' }" in app_js
assert "s.dashboard === job.key" in app_js, "the dot stopped filtering by dashboard"
def test_disks_page_draws_one_card_per_physical_drive_from_the_api(client) -> None:
"""The card is a drive, keyed by the agent's device name, patched in place.
A mount or a partition would need a filesystem field, and `/system` already answers "is
the filesystem full"; if one leaks back in here, the page stops being about the drive.
"""
html = client.get("/disks").text
js = _disks_js(client)
assert 'id="hosts"' in html and 'id="hosts-empty"' in html
assert "fetchJSON('/api/disks/snapshot')" in js
assert "row.path" in js and "/dev/" in js, "the card stopped naming the device it is about"
# identity + verdict lead the card: no hero row, no big single number
assert 'class="heroes"' not in html, "the disk card grew a hero row"
assert "Dash.hero(" not in js
for filesystem in ("mountpoint", "fstype", "free_bytes", "worst_mount"):
assert filesystem not in js, f"{filesystem} leaked into the drive page"
# built once per device and patched, so a 16-drive host does not rebuild 16 panels a poll
assert "seen.get(row.device)" in js and "renderDiskCard(panel, row)" in js
assert "pruneHosts(" in js, "a drive that left the host keeps its card forever"
def test_disk_cards_print_the_drive_numbers_and_colour_only_rated_ones(client) -> None:
"""Temp, power-on hours and wear, coloured by the API's verdicts and nothing else.
The odometer is the deliberate exception: `severity.THRESHOLDS` has no `disk.power_on_hours`
row, so the cell is written with a `null` verdict and cannot go green at 100 h or red at
60 000 h. A drive that publishes no wear (the recorded HDD shape) prints `n/a` because
`setTxt` is the only writer.
"""
js = _disks_js(client)
for field in ("smart.temp_c", "smart.power_on_hours", "smart.percentage_used",
"row.health"):
assert field in js, f"{field} is no longer on the card"
for verdict in ("severity.temp_c", "severity.percentage_used", "severity.health"):
assert verdict in js, f"the card stopped rendering {verdict}"
assert "severityFor" not in js, "the drive page wrote a threshold of its own"
body = js[js.index("function renderDiskCard("):js.index("/** Text that prints nothing")]
assert "severity.temp_c" in body and "severity.percentage_used" in body
assert re.search(r"setMetric\(card, 'hours',[^;]*null\)", body), (
"the odometer is being graded by something severity.py does not have")
# the drive's own word, and `unknown` for a drive that gave none — never a green pass
assert "'FAILED'" in body and "'unknown'" in body
def test_disk_alarm_chips_exist_only_while_a_counter_is_non_zero(client) -> None:
"""Zero and unknown both draw no chip; only a non-zero counter draws a red one.
The `/llm` preemption rule: a permanent "0 pending sectors" row trains the eye to ignore
it, and grading a counter green would call a quiet drive an achievement. `severity.py`
rates none of these, so the chip's presence is the entire signal here.
"""
js = _disks_js(client)
for counter in ("reallocated_sector_ct", "current_pending_sector_ct",
"offline_uncorrectable"):
assert counter in js, f"{counter} is no longer an alarm"
assert "value <= 0) continue" in js, "a zero now draws a chip"
assert "alarms.hidden = chips.length === 0" in js, "the empty chip row stays on screen"
assert 'class="chip fail"' in js
css = client.get("/static/dash.css").text
assert ".alarms {" in css and ".chip.fail" in css
def test_a_host_without_smartctl_is_dimmed_with_a_one_line_why_and_never_red(client) -> None:
"""Unconfigured is not broken (invariant 3), and the three answers stay three.
`smartctl_available` is `false` (install smartmontools — the agent said so, so the page
may say it), `null` (nobody told us either way, so name the fix without claiming to know
which half is missing), or the agent is not answering at all, which is a different
sentence again. None of the three reddens anything.
"""
js = _disks_js(client)
html = client.get("/disks").text
assert "smartctl not available on" in js
assert "disk group" in js, "the why stopped naming the grant that fixes it"
assert "smartctl_available === false" in js and "smartctl_available === true" in js
assert "host.unreachable" in js, "a dead agent must not be reported as a missing binary"
assert "smart_errors" in js, "the agent's own sentence is no longer printed"
assert "classList.toggle('stale'" in js
assert 'class="banner info why" hidden' in js, "the why-line has no home on the card"
assert "banner fail" not in js and html.count("banner") == 0, (
"a missing binary reddens the page")
def test_disks_charts_are_collapsed_lazy_and_cost_one_history_request_per_host(client) -> None:
"""`/system`'s chart discipline, reused: collapsed, painted only while open, one XHR.
The request still names every series the host could draw (temperature and wear per
drive), because `/api/disks/history` answers an unknown name as an empty series and a
panel should never have to discover series it did not ask for.
"""
js = _disks_js(client)
assert '<details class="charts-box"' in js
guard = js[js.index("function drawHostCharts("):js.index("async function renderHostCharts(")]
assert "if (!chartsOpen(card)) return;" in guard, (
"a collapsed stack is being built and drawn every poll")
history = js[js.index("async function renderHostCharts("):js.index("/**\n * Drop cards")]
assert history.count("fetchJSON(") == 1, "more than one history request for one host"
assert "/api/disks/history" in history
assert "streams.temp.concat(streams.wear)" in history, (
"the request no longer names every series the panel could draw")
prune = js[js.index("function pruneHosts("):js.index("// ---- fleet header")]
for entry in ("state.hosts.delete", "state.disks.delete", "state.charts.delete",
"state.history.delete", "state.payloads.delete"):
assert entry in prune, f"a pruned host keeps its {entry}"
def test_every_disks_chart_names_itself_for_assistive_tech(client) -> None:
"""An unlabelled plot is silence, and the legend is a sibling that labels nothing."""
js = _code(_disks_js(client))
labels = re.findall(r"ariaLabel: `([^`]+)`", js)
assert len(labels) == 2, "every drive chart must name itself (temperature, wear)"
assert all(label.startswith("${host.name}") for label in labels), (
"a chart that does not say which host it plots belongs on no page")
def test_the_disks_fleet_header_reads_the_api_sums_and_invents_none(client) -> None:
"""The header's arithmetic is the API's (`fleet_disk_summary`), never the page's.
Hidden below two hosts, because with one box in `HOST_TARGETS` these sums only duplicate
that host's own cards — `/llm`'s fleet-block rule. And no cell is coloured: a count of
drives is not a graded number (`severity.py` has no row for one).
"""
html = client.get("/disks").text
js = _disks_js(client)
for element in ("fleet", "fleet-dot", "fleet-count", "fleet-disks", "fleet-smart",
"fleet-failed", "fleet-alarms"):
assert f'id="{element}"' in html, f"the header lost #{element}"
fleet = js[js.index("function renderFleet("):js.index("// ---- snapshot")]
for path in ("fleet.hosts_live", "fleet.disks", "fleet.with_smart",
"fleet.health_failed", "fleet.alarm_disks"):
assert path in fleet, f"the header no longer reads {path}"
assert "reduce(" not in fleet and "+=" not in fleet, "the page is summing drives itself"
assert "severity" not in fleet, "a drive count grew a verdict"
assert "block.hidden = true" in fleet, "one host's sums duplicate its own cards"
def test_the_disk_panel_style_is_existing_vocabulary_with_no_new_palette(client) -> None:
"""`.disk` is the `.slot` shape, and the only colours it reaches for are the verdicts."""
css = _css(client)
block = _css_section(css, "drive panels")
for selector in (".disks {", ".disk {", ".disk-head", ".disk-dev", ".disk-model",
".disk .metrics", ".disk .alarms", ".disk .metric.warn .value",
".disk .metric.fail .value"):
assert selector in block, f"dash.css lost {selector}"
for token in ("var(--warn)", "var(--fail)", "var(--border)", "var(--bg-sunken)",
"var(--text)"):
assert token in block, f"the drive panel no longer uses {token}"
assert "--disk" not in css, "a drive-only colour token appeared"
assert ".stale" in css, "the dim that marks an unreadable host is gone"
def test_the_overview_prints_the_fourth_source_cadence(client) -> None:
"""`/` names every source's interval, so a drive card four minutes old reads as behaving."""
js = client.get("/static/page-index.js").text
assert "config.disks.interval_seconds" in js
@pytest.mark.parametrize("fixture", ["client", "data_client"])
def test_summary_carries_the_drive_block_for_programmatic_consumers(
request, fixture: str) -> None:
"""`disks` is additive: the same configured-ness, the same hosts, no key retired.
`client` has no targets, so it proves the block answers as `False`/`None`/`[]` instead of
disappearing; `data_client` proves it lists the host the fleet has.
"""
test_client = request.getfixturevalue(fixture)
body = test_client.get("/api/summary").json()
populated = fixture == "data_client"
assert body["disks"]["enabled"] is populated
assert (body["disks"]["fleet"] is not None) is populated
assert body["disks"]["hosts"] == ([] if not populated else [
{"name": "driveripper", "disks": None, "with_smart": None, "health_failed": None,
"alarm_disks": None, "smartctl_available": None, "failed_devices": [],
"unreachable": True}]), "a configured host with no payload is a row of nones"
assert [row["name"] for row in body["disks"]["hosts"]] == [
row["name"] for row in body["system"]["hosts"]]
def test_summary_serves_the_same_counts_the_disks_page_prints(disks_client) -> None:
"""One projection, two consumers: the summary's numbers *are* the snapshot's."""
body = disks_client.get("/api/summary").json()
snapshot = disks_client.get("/api/disks/snapshot").json()
disks = body["disks"]
assert disks["fleet"] == snapshot["fleet"], "the header would disagree with the page"
rows = {row["name"]: row for row in disks["hosts"]}
live = next(host for host in snapshot["hosts"] if host["name"] == "driveripper")
assert (rows["driveripper"]["disks"], rows["driveripper"]["with_smart"],
rows["driveripper"]["health_failed"], rows["driveripper"]["alarm_disks"]) == (
live["summary"]["disks"], live["summary"]["with_smart"],
live["summary"]["health_failed"], live["summary"]["alarm_disks"])
assert rows["driveripper"]["smartctl_available"] is True
assert rows["driveripper"]["unreachable"] is False
assert rows["deskwork"]["unreachable"] is True
assert rows["deskwork"]["disks"] is None, "unknown, never 0 (rule 2)"
assert sorted(state["key"] for state in disks["states"]) == ["disks:deskwork",
"disks:driveripper"]
def test_summary_names_the_drive_the_health_verdict_called_failed(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, disk_snapshot: Any) -> None:
"""Which drive the hardware itself called `FAILED` is part of the fleet answer.
A notifier reading `/api/summary` should not have to fetch a second endpoint to name the
device, and the count and the list have to agree or one of them is wrong.
"""
db = str(tmp_path / "summary-failed.sqlite3")
store = SampleStore(db)
app = create_app(Settings(db_path=db,
host_targets=(Target("driveripper", DEAD_URL),)), store)
with TestClient(app) as test_client:
now = time.time()
snapshot = disk_snapshot("driveripper", mode="failed")
snapshot["ts"] = now
result = parse_disks_snapshot("driveripper", snapshot, DisksCollector(), now=now)
app.state.ctx.hub.jobs["disks:driveripper"].state.payload = result.payload
row = test_client.get("/api/summary").json()["disks"]["hosts"][0]
assert len(row["failed_devices"]) == 1
assert row["failed_devices"][0] == result.payload["disks"][0]["device"]
assert row["health_failed"] == result.payload["summary"]["health_failed"] == 1
store.close()
# ---- events --------------------------------------------------------------- #
def test_events_are_returned_in_json(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: