phase: 09_disks_smartctl
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**
- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
This commit is contained in:
@@ -166,6 +166,98 @@ def test_a_dashboard_the_builder_does_not_know_about_still_gets_a_verdict_and_a_
|
||||
assert "every 60s" in out[0]["message"]
|
||||
|
||||
|
||||
def test_the_disk_job_is_stale_only_after_three_of_its_own_five_minute_polls() -> None:
|
||||
"""SMART is a 300 s signal: judging it by the 10 s system interval would amber `/`
|
||||
every five minutes for a source that is behaving exactly as configured."""
|
||||
disks = Settings(disks_interval=300.0)
|
||||
fresh = state(key="disks:driveripper", dashboard="disks", label="driveripper", age=600.0)
|
||||
assert alerts.build(disks, [fresh]) == [], "600s is two polls, not an outage"
|
||||
stale = {**fresh, "age_seconds": 1_000.0}
|
||||
out = alerts.build(disks, [stale])
|
||||
assert out[0]["severity"] == "warn"
|
||||
assert "every 300s" in out[0]["message"], "the message must print the disks interval"
|
||||
|
||||
|
||||
def test_a_disk_job_that_failed_because_its_agent_is_down_is_one_line_not_two() -> None:
|
||||
"""`disks` reads the same agent snapshot `system` does.
|
||||
|
||||
A dead host would otherwise print the same sentence twice (`alerts` rule: one dead
|
||||
source, one line), with only the internal id telling the two apart — which is a lie of
|
||||
the same shape as a duplicated host name.
|
||||
"""
|
||||
out = alerts.build(SETTINGS, [
|
||||
state(key="disks:driveripper", dashboard="disks", label="driveripper",
|
||||
ok=False, error="UpstreamError: driveripper: http://h:9121/snapshot "
|
||||
"unreachable (ConnectError)"),
|
||||
state(ok=False, error="UpstreamError: driveripper: http://h:9121/snapshot "
|
||||
"unreachable (ConnectError)"),
|
||||
], now=2_000.0)
|
||||
assert [item["source"] for item in out] == ["driveripper"]
|
||||
assert len(out) == 1, "the echo of the same outage must not read as two problems"
|
||||
assert out[0]["id"].startswith("driveripper:host:")
|
||||
|
||||
|
||||
def test_a_disk_job_that_fails_while_its_host_answers_is_the_only_voice_it_has() -> None:
|
||||
"""SMART wedged inside the agent is a `disks`-only fault, and must still be said."""
|
||||
out = alerts.build(SETTINGS, [
|
||||
state(key="disks:driveripper", dashboard="disks", label="driveripper",
|
||||
ok=False, error="UpstreamError: driveripper: timed out"),
|
||||
state(),
|
||||
], now=2_000.0)
|
||||
assert len(out) == 1
|
||||
assert out[0]["id"].startswith("driveripper:disks:")
|
||||
assert "timed out" in out[0]["message"]
|
||||
|
||||
|
||||
def test_a_disk_only_alert_links_to_the_drive_page() -> None:
|
||||
"""The overview's row is a link, so it has to point at the page that explains it.
|
||||
|
||||
A wedged SMART pass has no other voice (the host itself is answering), and `/system`
|
||||
cannot tell the operator which drive stopped reporting — `/disks` can. Before that route
|
||||
existed this link pointed at `/` on purpose; now it must not point anywhere else.
|
||||
"""
|
||||
out = alerts.build(SETTINGS, [
|
||||
state(key="disks:driveripper", dashboard="disks", label="driveripper",
|
||||
ok=False, error="UpstreamError: driveripper: timed out"),
|
||||
state(),
|
||||
], now=2_000.0)
|
||||
assert len(out) == 1
|
||||
assert out[0]["href"] == "/disks"
|
||||
|
||||
|
||||
def test_an_unconfigured_disk_source_says_nothing_even_when_its_job_says_down() -> None:
|
||||
"""No `HOST_TARGETS` means no `disks` job exists; a reported-but-unconfigured one must
|
||||
still be silence (invariant 3).
|
||||
|
||||
The builder is fed job states rather than settings, so a hub that ever hands over a
|
||||
`disks` record with `configured: false` — a config that arrives between two polls, a
|
||||
future target list of its own — must not amber `/` over a page nobody installed.
|
||||
"""
|
||||
out = alerts.build(Settings(disks_interval=300.0), [
|
||||
state(key="disks:driveripper", dashboard="disks", label="driveripper",
|
||||
ok=False, configured=False, error="no hosts configured"),
|
||||
], now=2_000.0)
|
||||
assert out == []
|
||||
|
||||
|
||||
def test_a_host_threshold_reads_the_system_job_not_the_disk_job(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
"""Two jobs share a host label, so the lookup has to be filtered by dashboard.
|
||||
|
||||
`disks` is registered after `system`, but the builder must not depend on that: if an
|
||||
unfiltered lookup found the failing disk job first, the host would be called
|
||||
unreachable and its real threshold breach would vanish.
|
||||
"""
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=95.0) # cpu.usage_pct -> fail
|
||||
out = alerts.build(SETTINGS, [
|
||||
state(key="disks:driveripper", dashboard="disks", label="driveripper",
|
||||
ok=False, error="timed out"),
|
||||
state(),
|
||||
], [payload], [], None, now=2_000.0)
|
||||
ids = [item["id"] for item in out]
|
||||
assert "driveripper:cpu.usage_pct" in ids, "the breach is real; the disk job is not it"
|
||||
|
||||
|
||||
def test_a_failing_source_says_when_the_poller_first_noticed() -> None:
|
||||
events = [{"ts": 1_500.0, "source": "system", "host": "driveripper", "kind": "unreachable",
|
||||
"severity": "error", "message": "ConnectError", "data": None},
|
||||
|
||||
Reference in New Issue
Block a user