phase: 09_disks_smartctl

**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**

- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
This commit is contained in:
2026-10-18 00:45:01 -04:00
parent 19cfedfddf
commit fefbeb8227
81 changed files with 5525 additions and 134 deletions
+308 -6
View File
@@ -21,8 +21,19 @@ Design rules
monitoring agent is not allowed to make.
* **Every read is guarded.** A vanished /proc entry (process exited mid-scan) skips that
item; the snapshot still succeeds. The poll is degraded, never failed.
* **The only state** is a small pid->cpu-time cache so `processes.top_cpu` can answer a
question /proc cannot (CPU needs two samples). It is bounded and cleared when stale.
* **SMART is a best-effort add-on.** The per-disk `smart` block runs `smartctl -i -A -H`
in a subprocess: bounded 5 s per device and 20 s per pass, cached 240 s so a fast
dashboard poll cannot re-issue ATA commands every few seconds. Refreshing an expired
cache happens on a background thread while the request is served the previous result —
20 s of wedged disk must never cost a poll more than its own timeout. No binary, a denied
device or a wedged disk degrades *that disk's* `smart` to `null` plus one `errors`
entry (a missing binary produces exactly one entry for the whole pass) and never
fails the snapshot or `/health`. Absence is a documented state, not a red panel:
install `smartmontools` and give the agent user read access to the block devices
(typically the `disk` group), or pass `--no-smart` to opt out silently.
* **The state** is a small pid->cpu-time cache so `processes.top_cpu` can answer a
question /proc cannot (CPU needs two samples), bounded and cleared when stale, plus
the short-TTL SMART result cache above.
SCHEMA (GET /snapshot)
----------------------
@@ -46,8 +57,17 @@ SCHEMA (GET /snapshot)
disk_io {devices: [{name, model, rotational, size_bytes, reads_total,
writes_total, read_bytes_total, write_bytes_total,
in_flight, io_time_ms_total, weighted_io_ms_total,
discard_bytes_total}],
discard_bytes_total,
smart: {model, serial, overall_health, temp_c,
power_on_hours, reallocated_sector_ct,
current_pending_sector_ct, offline_uncorrectable,
percentage_used} | null}],
device_names: [str]} # whole disks from /sys/block, no partitions
# `smart` is `smartctl -i -A -H` parsed; null with an errors[] entry
# when the binary is missing or that device cannot be read. Fields
# the drive does not publish stay null (e.g. NVMe has no
# reallocated_sector_ct), and NVMe `offline_uncorrectable` carries
# the health log's "Media and Data Integrity Errors" counter.
network {interfaces: [{name, rx_bytes_total, tx_bytes_total, rx_packets_total,
tx_packets_total, rx_errors_total, tx_errors_total,
rx_dropped_total, tx_dropped_total, speed_mbps}]}
@@ -70,6 +90,7 @@ import json
import os
import re
import socket
import subprocess
import threading
import time
from collections.abc import Callable
@@ -100,6 +121,41 @@ VIRTUAL_NET_PREFIXES = ("lo", "veth", "docker", "br-", "virbr", "br", "tap", "vn
# or LUKS volume is carved out of a disk we already count, so charting both double-counts.
VIRTUAL_DISK_PREFIXES = ("loop", "ram", "zram", "sr", "dm-", "md127")
# SMART collection bounds. 5 s per device is generous for a healthy drive and caps how
# long one wedged device can hold a snapshot hostage; 20 s caps the whole pass so even a
# shelf of bad disks cannot stall the HTTP handler; 240 s is below the dashboard's
# 5-minute disks cadence, so the slow disks job gets fresh data while the fast system
# poll never re-runs smartctl every 10 s.
SMARTCTL_TIMEOUT_SECONDS = 5.0
SMARTCTL_BUDGET_SECONDS = 20.0
SMARTCTL_CACHE_SECONDS = 240.0
# The smart block's field set — the shape the `/disks` page (phase 09) consumes. A field
# the drive does not publish stays None; the page renders `n/a`, never 0.
SMART_FIELDS = ("model", "serial", "overall_health", "temp_c", "power_on_hours",
"reallocated_sector_ct", "current_pending_sector_ct",
"offline_uncorrectable", "percentage_used")
_SMART_HEALTH_RE = re.compile(r"^SMART overall-health self-assessment test result:\s*(\S+)")
# The number a SMART attribute RAW_VALUE column leads with. Captured from the deployed
# smartmontools 7.5 on this fleet: a Seagate IronWolf writes `35 (Min/Max 34/44)`, a
# Western Digital writes its packed hours as `43894h+43m+32.519s`, and most drives write a
# bare `30423`. In every shape the drive's own figure is the leading decimal group.
_SMART_RAW_NUMBER_RE = re.compile(r"^\d[\d,]*(?:\.\d+)?")
# ATA SMART attribute table rows, keyed by the exact attribute names the deployed
# smartmontools 7.5 prints (verified against its own drive database, not from memory).
_SMART_ATTR_FIELDS = {
"Temperature_Celsius": "temp_c",
"Airflow_Temperature_Cel": "temp_c",
"Power_On_Hours": "power_on_hours",
"Reallocated_Sector_Ct": "reallocated_sector_ct",
"Current_Pending_Sector": "current_pending_sector_ct",
"Offline_Uncorrectable": "offline_uncorrectable",
"Percent_Lifetime_Used": "percentage_used",
"Percent_Life_Used": "percentage_used",
"Percentage_Lifetime_Use": "percentage_used",
}
# pci slot -> human model, filled on first look (lspci output cannot change at runtime).
_GPU_MODEL_CACHE: dict[str, str | None] = {}
@@ -167,10 +223,20 @@ class Collector:
of fixture files instead of a live machine."""
def __init__(self, root: str | os.PathLike[str] = "/", *, top_processes: int = 5,
net_all: bool = False) -> None:
net_all: bool = False, smartctl: str = "smartctl",
smart: bool = True) -> None:
self.root = Path(root)
self.top_processes = top_processes
self.net_all = net_all
self.smartctl = smartctl
self.smart_enabled = smart
self.smart_timeout_seconds = SMARTCTL_TIMEOUT_SECONDS
self.smart_budget_seconds = SMARTCTL_BUDGET_SECONDS
self.smart_cache_seconds = SMARTCTL_CACHE_SECONDS
self._smart_lock = threading.Lock() # ThreadingHTTPServer may poll while a pass runs
self._smart_cache: tuple[float, dict[str, dict[str, Any] | None],
list[str]] | None = None
self._smart_refreshing = False # at most one background refresh at a time
self._lock = threading.Lock() # one HTTP handler at a time may touch the pid cache
self._pid_cpu: dict[int, tuple[float, int, str]] = {} # pid -> (ts, jiffies, name)
self._boot_ts: float | None = None
@@ -443,6 +509,129 @@ class Collector:
})
return {"devices": devices, "device_names": names}
# -- smart (smartctl) ------------------------------------------------------ #
def smart(self, device_names: list[str]) -> tuple[dict[str, dict[str, Any] | None],
list[str]]:
"""Best-effort `smartctl -i -A -H <device>` per whole disk; never raises.
Returns (per-device parsed dict or None, error lines). Failure semantics (phase
09 task 01): the binary missing produces **one** error entry for the whole pass
and every disk `None`; one denied/wedged device produces its own error entry and
nulls only that disk. Results are cached for `smart_cache_seconds` so the fast
system poll cannot re-issue ATA commands every few seconds; the cache replays the
same error lines, so a snapshot always states why `smart` is null.
An **expired** cache is refreshed on a background thread and the request is served
the previous result, because the pass budget (20 s) is five times the dashboard's
`HOST_TIMEOUT_SECONDS` (4 s): run inline on the request path, one wedged disk would
turn the *system* source red every `smart_cache_seconds` for a fault in an add-on
(AGENTS.md invariant 4 — SMART must not blink the page that does not use it). The
very first pass still runs inline, so a freshly started agent answers `/snapshot`
with real SMART fields instead of a page of `n/a` until someone polls it twice.
"""
now = time.monotonic()
with self._smart_lock:
cached = self._smart_cache
if cached is not None:
if now - cached[0] < self.smart_cache_seconds or self._smart_refreshing:
return dict(cached[1]), list(cached[2])
self._smart_refreshing = True
threading.Thread(target=self._refresh_smart, args=(list(device_names),),
name="smart-refresh", daemon=True).start()
return dict(cached[1]), list(cached[2])
return self._smart_pass(device_names)
def _refresh_smart(self, device_names: list[str]) -> None:
"""The expired-cache pass, off the request path (see `smart()`).
A failure here keeps the previous cache (and its age) rather than clearing it: the
snapshot keeps answering with the last truth it had, and the retry comes on the next
expiry instead of hammering a disk that just failed.
"""
try:
self._smart_pass(device_names)
except Exception as exc: # noqa: BLE001 - the thread must always clear its own flag
with self._smart_lock:
previous = self._smart_cache
results = dict(previous[1]) if previous else dict.fromkeys(device_names)
errors = list(previous[2]) if previous else []
self._smart_cache = (time.monotonic(), results,
[*errors,
f"smart: background refresh failed: "
f"{type(exc).__name__}: {exc}"])
finally:
with self._smart_lock:
self._smart_refreshing = False
def _smart_pass(self, device_names: list[str]) -> tuple[dict[str, dict[str, Any] | None],
list[str]]:
"""One `smartctl` pass over the whole disks, and the cache write that goes with it."""
now = time.monotonic()
results: dict[str, dict[str, Any] | None] = {}
errors: list[str] = []
deadline = now + self.smart_budget_seconds
for name in device_names:
left = deadline - time.monotonic()
if left <= 0:
remaining = [n for n in device_names if n not in results]
for skipped_name in remaining:
results[skipped_name] = None
errors.append(f"smart: pass budget of {self.smart_budget_seconds:.0f}s "
f"exhausted, {len(remaining)} disk(s) skipped")
break
try:
proc = subprocess.run([self.smartctl, "-i", "-A", "-H", f"/dev/{name}"],
capture_output=True, text=True,
timeout=min(left, self.smart_timeout_seconds),
check=False)
except FileNotFoundError:
# one entry for the pass, not one per disk: a missing binary is one fact
errors.append(f"smart: {self.smartctl} not found — install smartmontools "
"and give this user device read access (disk group) for "
"SMART data")
for missing_name in device_names:
results.setdefault(missing_name, None)
break
except subprocess.TimeoutExpired:
results[name] = None
errors.append(f"smart: {name}: smartctl timed out after "
f"{self.smart_timeout_seconds:.0f}s (device wedged?)")
continue
except OSError as exc:
results[name] = None
errors.append(f"smart: {name}: smartctl could not run: {exc}")
continue
parsed = parse_smart_output(proc.stdout)
if any(value is not None for value in parsed.values()):
results[name] = parsed # partial output is still what the drive published
else:
results[name] = None
errors.append(f"smart: {name}: "
f"{smart_failure_line(proc.returncode, proc.stdout, proc.stderr)}")
with self._smart_lock:
self._smart_cache = (time.monotonic(), results, errors)
return dict(results), list(errors)
def attach_smart(self, disk_io: dict[str, Any], errors: list[str]) -> None:
"""Put `smart` on every disk row; `--no-smart` leaves them all null silently."""
devices = disk_io.get("devices") or []
for device in devices:
device["smart"] = None
if not devices or not self.smart_enabled:
return
results, smart_errors = self.smart([str(device.get("name")) for device in devices])
errors.extend(smart_errors)
for device in devices:
parsed = results.get(str(device.get("name")))
if not parsed:
continue
smart = dict(parsed)
if not smart.get("model"):
# smartctl's identity block can be absent (some bridges refuse -i while
# answering the health log); the agent's sysfs model is the fallback.
smart["model"] = device.get("model")
device["smart"] = smart
# -- network ------------------------------------------------------------- #
def network(self) -> dict[str, Any]:
interfaces: list[dict[str, Any]] = []
@@ -787,6 +976,10 @@ class Collector:
processes = self._safe(self.processes, "processes", errors) or {"total": None,
"top_rss": [],
"top_cpu": []}
disk_io = self._safe(self.disk_io, "disk_io", errors) or {"devices": [], "device_names": []}
# best-effort, after the cheap reads: a wedged disk can cost the pass budget, not
# the snapshot (see the module docstring and smart() for the bounds)
self._safe(lambda: self.attach_smart(disk_io, errors), "smart", errors)
return {
"schema": SCHEMA_VERSION,
"agent_version": AGENT_VERSION,
@@ -796,7 +989,7 @@ class Collector:
"cpu": self._safe(self.cpu, "cpu", errors) or {},
"memory": self._safe(self.memory, "memory", errors) or {},
"filesystems": self._safe(self.filesystems, "filesystems", errors) or [],
"disk_io": self._safe(self.disk_io, "disk_io", errors) or {"devices": [], "device_names": []},
"disk_io": disk_io,
"network": self._safe(self.network, "network", errors) or {"interfaces": []},
"gpus": self._safe(self.gpus, "gpus", errors) or [],
"temperatures": temps,
@@ -820,6 +1013,108 @@ MEMORY_KEYS = ("total_bytes", "free_bytes", "available_bytes", "used_bytes", "bu
"cached_bytes", "dirty_bytes", "swap_total_bytes", "swap_used_bytes",
"committed_bytes")
_SMART_FAILURE_HINT_RE = re.compile(r"failed|denied|error|unable|unsupported|not supported", re.I)
def smart_failure_line(returncode: int, stdout: str | None, stderr: str | None) -> str:
"""The sentence that says *why* smartctl answered nothing.
Never its first line: smartctl opens every run with its version banner, and errors
land on stdout, on stderr, or both. Take the first line that reads like a failure,
else the last non-empty line, else the exit status — an error entry that quotes the
copyright line is worse than no error entry.
"""
blobs = [stderr or "", stdout or ""]
for blob in blobs:
for line in blob.splitlines():
if line.strip() and _SMART_FAILURE_HINT_RE.search(line):
return line.strip()
for blob in blobs: # nothing that reads like a failure: last real line of stderr first
lines = [line.strip() for line in blob.splitlines() if line.strip()]
if lines:
return lines[-1]
return f"exit status {returncode}"
def _smart_number(text: str) -> float | None:
"""`8,760` and `38` are numbers; a hex-packed vendor raw is not one we can decode
without a drive-specific table, so it stays None — an invented sector count is the
exact failure mode this agent exists to avoid."""
return _to_float(text.strip().replace(",", ""))
def _smart_raw_number(raw: str) -> float | None:
"""The value of a SMART attribute row's RAW_VALUE column (column 9, `raw`).
smartctl prints one bare number for most attributes and the shapes above for the rest,
all of which are read here; `0x...` is a packed raw, and a packed raw read as its
leading `0` would report a healthy-looking zero for a value we never decoded, so that
one answers None."""
token = raw.strip().split(" ")[0]
if not token or token.lower().startswith("0x"):
return None
match = _SMART_RAW_NUMBER_RE.match(token)
return _smart_number(match.group(0)) if match else None
def parse_smart_output(text: str | None) -> dict[str, Any]:
"""Parse `smartctl -i -A -H` stdout (ATA, NVMe and SCSI share this entry point).
Only labels the deployed smartmontools 7.5 actually prints are matched (they were
verified against that binary's own format strings and drive database, and are pinned
by tests/fixtures/smartctl_*.txt). Values the drive or the parser cannot supply stay
None. `overall_health` keeps smartctl's own verdict word (`PASSED`, `FAILED`,
`UNKNOWN`, trailing `!` stripped); NVMe `offline_uncorrectable` carries the health
log's `Media and Data Integrity Errors` counter — same meaning, different name, and
the ATA `Offline_Uncorrectable` attribute when both exist wins because they are the
drive's own words for it. ATA rows are read from the RAW_VALUE column rather than the
end of the line, because this binary annotates that column — `35 (Min/Max 34/44)` on a
Seagate, `43894h+43m+32.519s` on a Western Digital — and the last token of such a row
is not the value.
"""
out: dict[str, Any] = dict.fromkeys(SMART_FIELDS)
if not text:
return out
for line in text.splitlines():
stripped = line.strip()
if not stripped:
continue
label, sep, value = stripped.partition(":")
if sep:
label, value = label.strip(), value.strip()
if label in ("Device Model", "Model Number"):
if value:
out["model"] = value
elif label == "Serial Number":
if value:
out["serial"] = value
elif label == "Temperature": # NVMe health log: `38 Celsius`
number = _smart_number(value.split()[0]) if value.split() else None
if number is not None:
out["temp_c"] = number
elif label == "Power On Hours":
out["power_on_hours"] = _smart_number(value)
elif label == "Percentage Used":
out["percentage_used"] = _smart_number(value.rstrip("%"))
elif label == "Media and Data Integrity Errors" and out["offline_uncorrectable"] is None:
out["offline_uncorrectable"] = _smart_number(value)
health = _SMART_HEALTH_RE.match(stripped)
if health:
out["overall_health"] = health.group(1).rstrip("!")
continue
fields = stripped.split()
if fields and fields[0].isdigit() and len(fields) >= 6:
attribute = _SMART_ATTR_FIELDS.get(fields[1])
if attribute is not None:
# ID NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE —
# the raw is column 9, and reading the *last* token instead loses every
# drive that annotates it: `35 (Min/Max 34/44)` ends in `44)`.
number = _smart_raw_number(fields[9] if len(fields) > 9 else fields[-1])
if number is not None:
out[attribute] = number
return out
# /proc/<pid>/stat comm is in parens and may itself contain spaces and parens.
_STAT_COMM_RE = re.compile(r"^\d+ \((?P<comm>.*)\) (?P<rest>.*)$", re.S)
@@ -969,10 +1264,17 @@ def main(argv: list[str] | None = None) -> int:
parser.add_argument("--root", default="/", help="filesystem root to read (for tests)")
parser.add_argument("--top", type=int, default=5, help="processes per top-N list")
parser.add_argument("--net-all", action="store_true", help="include veth/bridge interfaces")
parser.add_argument("--smartctl", default="smartctl",
help="smartctl binary; missing is a documented degradation "
"(every disk's smart is null, one errors entry)")
parser.add_argument("--no-smart", action="store_true",
help="skip the smartctl pass entirely (disks report smart: null, "
"no error entries)")
parser.add_argument("--once", action="store_true", help="print one snapshot as JSON and exit")
args = parser.parse_args(argv)
collector = Collector(args.root, top_processes=args.top, net_all=args.net_all)
collector = Collector(args.root, top_processes=args.top, net_all=args.net_all,
smartctl=args.smartctl, smart=not args.no_smart)
if args.once:
print(json.dumps({"ok": True, **collector.snapshot()}, indent=2, default=str))
return 0