phase: 09_disks_smartctl
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**
- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
This commit is contained in:
@@ -32,6 +32,7 @@ from dashboards.config import Settings, load
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources import nut
|
||||
from dashboards.sources.disks import SMART_SERIES, DisksCollector
|
||||
from dashboards.sources.host import HostCollector
|
||||
from slots_contract import assert_slot_shape
|
||||
from ups_contract import assert_ups_hero_contract
|
||||
@@ -95,6 +96,90 @@ async def test_every_host_serves_the_hero_card_contract(settings: Settings,
|
||||
)
|
||||
|
||||
|
||||
async def test_agent_disk_rows_carry_the_smart_shape(settings: Settings) -> None:
|
||||
"""Phase-09: a redeployed agent answers `smart` per physical disk of `/snapshot`.
|
||||
|
||||
Agents are redeployed by hand, so an agent whose rows carry no `smart` key at all
|
||||
SKIPS this host rather than failing it. `smart: null` is a documented state (no
|
||||
smartctl binary, or the agent user lacks device read access) — what must hold is the
|
||||
shape of a non-null block and that unknowns are null, never 0. Read-only: nothing
|
||||
here sends `smartctl -t`; a self-test on hardware something is plugged into is the
|
||||
operator's call, not a unit test.
|
||||
"""
|
||||
if not settings.host_targets:
|
||||
pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh")
|
||||
seen_smart = 0
|
||||
async with httpx.AsyncClient(timeout=settings.host_timeout) as client:
|
||||
for target in settings.host_targets:
|
||||
response = await client.get(f"{target.url}/snapshot", headers=target.headers)
|
||||
if response.status_code >= 400:
|
||||
continue # reachability has its own live test
|
||||
devices = (response.json().get("disk_io") or {}).get("devices") or []
|
||||
if devices and not all("smart" in device for device in devices):
|
||||
continue # agent predates the smart pass — redeploy it to test this
|
||||
for device in devices:
|
||||
seen_smart += 1
|
||||
smart = device.get("smart")
|
||||
if smart is None:
|
||||
continue # degraded by design; the errors[] entry says why
|
||||
# the literal set is the pin: a rename on the agent must fail here too
|
||||
assert set(smart) == {"model", "serial", "overall_health", "temp_c",
|
||||
"power_on_hours", "reallocated_sector_ct",
|
||||
"current_pending_sector_ct",
|
||||
"offline_uncorrectable", "percentage_used"}, \
|
||||
f"{target.name}/{device['name']}"
|
||||
if smart["power_on_hours"] is not None:
|
||||
assert smart["power_on_hours"] > 0
|
||||
if smart["temp_c"] is not None:
|
||||
assert -20.0 < smart["temp_c"] < 125.0
|
||||
assert smart["overall_health"] in (None, "PASSED", "FAILED", "UNKNOWN")
|
||||
if not seen_smart:
|
||||
pytest.skip("no agent redeployed with the smart pass yet (deploy/install-agent.sh)")
|
||||
|
||||
|
||||
async def test_disks_source_charts_the_physical_drives_the_agent_reports(
|
||||
settings: Settings) -> None:
|
||||
"""Phase 09 task 02: the `disks` source over a real agent snapshot.
|
||||
|
||||
This is the production code path — the fetch the poller runs, not a hand-built dict —
|
||||
so what fails here is a rename between the agent and the page, not a typo in a fixture.
|
||||
The assertions hold whether or not this fleet can read SMART yet: one row per whole
|
||||
disk, `disk.*@<dev>` series only for values that exist, and no invented numbers.
|
||||
"""
|
||||
if not settings.host_targets:
|
||||
pytest.skip("HOST_TARGETS is unset — start ./scripts/dev-tunnels.sh")
|
||||
collector = DisksCollector()
|
||||
checked = 0
|
||||
async with httpx.AsyncClient(timeout=settings.disks_timeout) as client:
|
||||
for target in settings.host_targets:
|
||||
try:
|
||||
result = await collectors.fetch_disk_snapshot(client, target, collector,
|
||||
timeout=settings.disks_timeout)
|
||||
except collectors.UpstreamError:
|
||||
continue # reachability has its own live test
|
||||
checked += 1
|
||||
payload = result.payload
|
||||
assert payload["name"] == target.name
|
||||
assert payload["disks"], f"{target.name}: no physical disks at all?"
|
||||
for row in payload["disks"]:
|
||||
assert row["path"] == f"/dev/{row['device']}"
|
||||
assert "/" not in row["device"], f"partition {row['device']} in a disk list"
|
||||
assert row["health"] in (None, "ok", "failed")
|
||||
devices = {row["device"] for row in payload["disks"]}
|
||||
bases = {base for _field, base in SMART_SERIES}
|
||||
for sample in result.samples:
|
||||
assert sample.source == "disks", "the wrong family would chart nowhere"
|
||||
base, _, device = sample.metric.partition("@")
|
||||
assert base in bases and device in devices, (
|
||||
f"{target.name}: {sample.metric} is not a pinned disk series")
|
||||
assert sample.value == sample.value and abs(sample.value) != float("inf"), \
|
||||
"NaN must never reach the store"
|
||||
if any(row["smart"] for row in payload["disks"]):
|
||||
assert payload["smartctl_available"] is True
|
||||
if not checked:
|
||||
pytest.skip("no host agent answered — start ./scripts/dev-tunnels.sh")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("engine", ["gufo", "vllm"])
|
||||
async def test_engine_still_publishes_the_series_we_parse(engine: str,
|
||||
settings: Settings) -> None:
|
||||
|
||||
Reference in New Issue
Block a user