phase: 09_disks_smartctl
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**
- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
This commit is contained in:
@@ -107,6 +107,112 @@ def host_scrape() -> Any:
|
||||
return scrape
|
||||
|
||||
|
||||
# ---- phase 09: physical disks (the /disks source and page) ----------------- #
|
||||
|
||||
def _load_host_agent() -> Any:
|
||||
"""The stdlib agent module, loaded by path: it is not an installed package (and must
|
||||
not become one — it ships as a single file to every host). Same loader as
|
||||
tests/test_agent.py, kept private here because only the disk fixtures need it."""
|
||||
import importlib.util # noqa: PLC0415 - only the disk fixtures load the agent
|
||||
|
||||
path = FIXTURES.parent.parent / "agent" / "host_agent.py"
|
||||
spec = importlib.util.spec_from_file_location("host_agent", path)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules.setdefault("host_agent", module)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
#: The two `smartctl` shapes the fleet has: NVMe (health-log fields) and ATA (the vendor
|
||||
#: attribute table). `tests/fixtures/smartctl_reference_*.txt` — labels verbatim from the
|
||||
#: deployed smartmontools 7.5, values synthetic (see the header of each file).
|
||||
SMART_SHAPES = ("nvme", "ata_hdd")
|
||||
|
||||
#: The agent's own words for the way SMART goes missing that the page must name: the
|
||||
#: binary is not installed. Mirrors `host_agent.Collector._smart_pass()`; the agent test
|
||||
#: `test_missing_smartctl_binary_is_one_error_for_the_whole_pass` pins that the line it
|
||||
#: really writes still contains the word `sources.disks.MISSING_BINARY_WORD` looks for, so
|
||||
#: a rewording fails a test instead of turning "install smartmontools" into "unknown".
|
||||
SMART_MISSING_BINARY = ("smart: /usr/sbin/smartctl not found — install smartmontools and "
|
||||
"give this user device read access (disk group) for SMART data")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def smart_shape() -> Any:
|
||||
"""`smart_shape('nvme')` / `smart_shape('ata_hdd')` -> the capture, parsed by the agent.
|
||||
|
||||
Fixture-driven on purpose (AGENTS.md rule 7): the `smart` blocks a `/disks` test hands
|
||||
to the source are the recorded `smartctl` text run through the agent's own parser, so a
|
||||
field rename on either side of the wire fails a test instead of printing a dash.
|
||||
"""
|
||||
agent = _load_host_agent()
|
||||
|
||||
def shape(kind: str = "nvme") -> dict[str, Any]:
|
||||
text = (FIXTURES / f"smartctl_reference_{kind}.txt").read_text()
|
||||
return agent.parse_smart_output(text)
|
||||
|
||||
return shape
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def disk_snapshot(agent_snapshots: dict[str, dict[str, Any]], smart_shape: Any) -> Any:
|
||||
"""`build(host="driveripper", mode="healthy")` — a real capture with SMART attached.
|
||||
|
||||
The fleet captures predate the agent's SMART pass, so there is no single capture with
|
||||
both halves. This assembles the two that exist: the physical disk rows come from the
|
||||
host capture verbatim, the `smart` blocks from the recorded smartmontools shapes, and
|
||||
the failure modes are the ones `host_agent.smart()` actually produces.
|
||||
|
||||
Modes:
|
||||
|
||||
* `healthy` — every disk reads, alternating the NVMe and ATA shapes
|
||||
* `no_binary` — every `smart` is null plus the agent's single "not found" line
|
||||
* `denied` — the last disk is null with its own permission line (SMART is installed)
|
||||
* `silent` — every `smart` is null with no error line (`--no-smart`, or an agent that
|
||||
predates the pass: the dashboard cannot tell why, and must not guess)
|
||||
* `failed` — one drive answering `FAILED`, the rest healthy
|
||||
"""
|
||||
def build(host: str = "driveripper", *, mode: str = "healthy",
|
||||
mutate: Any = None) -> dict[str, Any]:
|
||||
import copy # noqa: PLC0415 - every build starts from the untouched capture
|
||||
|
||||
snapshot = copy.deepcopy(agent_snapshots[host])
|
||||
devices = (snapshot.get("disk_io") or {}).get("devices") or []
|
||||
errors: list[str] = list(snapshot.get("errors") or [])
|
||||
for index, device in enumerate(devices):
|
||||
if mode == "no_binary":
|
||||
device["smart"] = None
|
||||
continue
|
||||
if mode == "silent":
|
||||
device["smart"] = None
|
||||
continue
|
||||
if mode == "denied" and index == len(devices) - 1:
|
||||
device["smart"] = None
|
||||
continue
|
||||
shape = smart_shape(SMART_SHAPES[index % len(SMART_SHAPES)])
|
||||
block = dict(shape)
|
||||
# the identity in the capture is the disk the row is about; the shape files
|
||||
# describe one drive each, so the row keeps its own model and the counters
|
||||
# come from the shape
|
||||
block["model"] = block.get("model") or device.get("model")
|
||||
if mode == "failed" and index == 0:
|
||||
block["overall_health"] = "FAILED"
|
||||
device["smart"] = block
|
||||
if mode == "no_binary":
|
||||
errors.append(SMART_MISSING_BINARY)
|
||||
if mode == "denied" and devices:
|
||||
name = str(devices[-1].get("name"))
|
||||
errors.append(f"smart: {name}: Smartctl open device: /dev/{name} failed: "
|
||||
"Permission denied")
|
||||
snapshot["errors"] = errors
|
||||
if mutate is not None:
|
||||
mutate(snapshot)
|
||||
return snapshot
|
||||
|
||||
return build
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def metrics_text() -> Any:
|
||||
"""metrics_text('gufo') / metrics_text('vllm') -> the captured exposition verbatim."""
|
||||
|
||||
Reference in New Issue
Block a user