phase: 09_disks_smartctl

**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**

- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
This commit is contained in:
2026-10-18 00:45:01 -04:00
parent 19cfedfddf
commit fefbeb8227
81 changed files with 5525 additions and 134 deletions
@@ -0,0 +1,53 @@
# Phase 09 — A /disks page for physical disk health (smartctl)
**Source:** `TODO.md` L8 — "a truly physical disk monitoring page": "Add a 'disks' tab dedicated to smartctl and per-disk metrics. Note: not per-partition, but truly a physical disk monitoring page that collects data about all my physical disks across my fleet (should only be deskwork and driveripper right now)."
**Story:** n/a
**Context:** `agent/host_agent.py` (physical whole disks via `/sys/block`, ~L400 — already partition-free; stdlib-only, `ProtectSystem=strict`), `src/dashboards/config.py` (env only, sources default to unset), `src/dashboards/poller.py` (one task per source, stale/last-good, events under the dashboard name), `src/dashboards/store.py` (WAL, rollups raw→1m→1h, retention — rule 9), `src/dashboards/api.py` (GETs, 404-when-unconfigured without dialling), the three existing dashboards + `/` overview (a new tab joins the nav and the overview's "which of the three" table)
## Objective
A fourth dashboard: `/disks` — one card per **physical** disk (the agent's whole-
`/sys/block` enumeration, never partitions), each carrying `smartctl` health: overall
health, temperature, power-on hours, and the sector/wear alarms. Its own `disks` source
and poller task at a slow 5-minute cadence (SMART is not a per-second signal), its own
API (`/api/disks/snapshot`, `/api/disks/history`), rollups and retention, and a page that
reads `n/a` for what `smartctl` cannot see — never red for a missing binary.
- ASSUMPTION (phase, confirmed): the agent reads SMART **directly** (`smartctl` as the
agent user, e.g. `disk` group) — no sudo — and `DISKS_INTERVAL_SECONDS` defaults to
**300** s. `smartctl` absent or a disk unreadable → that disk's `smart` is `null`
with an error entry; the page says "smartctl not available" dimmed (invariants 3/4:
unconfigured is not broken; a dead source must not blink another).
## Dependencies
- `08_events_paging_and_copy` (todo) — sequence only: it settles `app.js`/`dash.css`
(the pager + copy audit) that the new page builds on; no hard code dependency.
## Tasks
1. `01_agent_smartctl_collection.md` — `host_agent.py` gains a `smartctl` pass over its existing whole-disk list (per-device `smartctl -i -A -H`, stdlib `subprocess`, timeout-bounded); fixture from the real disks; deploy docs.
2. `02_disks_source_poller_store.md` — the `disks` source (payload + `disk.*@dev` samples), its own poller job at `DISKS_INTERVAL_SECONDS` (default 300), rollups + retention in `store.py`.
3. `03_disks_api_and_config.md` — `/api/disks/snapshot` + `/api/disks/history`, config wiring, `404 {"configured": false}` without dialling, severity rows for the health/temperature metrics.
4. `04_disks_page_and_overview.md` — the `/disks` page (per-disk cards, alarm chips, per-host history charts), nav tab + overview entry + header status dot.
## Testing & Quality
- Hermetic: fixture-driven agent tests (`tests/test_agent.py` + recorded `smartctl`
output), `tests/test_disks_source.py` (new), `tests/test_api.py` key-path pins (with
and without config), `node --check`.
- Live: `uv run pytest -m live --live` — the agent on deskwork/driveripper answers
`smart` fields for its real disks; **live tests must not write** (no `smartctl -t`, no
self-test on hardware something is plugged into).
- Browser (DoD): screenshots in `data/logs/` — per-disk cards on live data, the
"smartctl not available" state, the overview with the new tab.
- Coverage: **>90%** on new/modified code, ≥94% overall gate maintained.
## Completion Criteria
- [ ] `GET /api/disks/snapshot` answers per-disk `smart` data for every physical disk the agents report; `404 {"configured": false}` with no agents configured, without dialling
- [ ] `/api/disks/history` serves the `disk.*@dev` series through the existing rollups
- [ ] the `/disks` page renders one card per physical disk (never per-partition) with health, temp, power-on hours and non-zero alarm chips
- [ ] a host without `smartctl` (or a denied read) shows dimmed `n/a` cards, never a red page
- [ ] `bash .agents/validate.sh` green; screenshots in `data/logs/`
- [ ] you have verified with real smartctl data from driveripper. The `ops` user has `sudo smartctl*` access via sudoers.