services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 09_disks_smartctl

Browse Source 
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**

- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
This commit is contained in:
ducoterra
2026-10-18 00:45:01 -04:00
parent 19cfedfddf
commit fefbeb8227
81 changed files with 5525 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+77 -10
View File
@@ -1,11 +1,12 @@
# Homelab Dashboards
Three dashboards, one domain, one process, one SQLite file:
Four dashboards, one domain, one process, one SQLite file:
| Page | What it answers | Source |
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
| `/` | Is anything wrong, and which dashboard do I need? | fleet + alerts in one request (`/api/summary`) |
| `/` | Is anything wrong, and which dashboard do I need? | all four, in one request (`/api/summary`) |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
| `/disks` | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount) |
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
@@ -71,6 +72,22 @@ It is read-only by construction: there is no code path that writes, and the HTTP
only answers `GET`. Counters are reported as counters — the dashboard computes rates over
a labelled window, so the agent never has to remember who polled it last.
Per-disk `smart` health is a best-effort add-on on the whole-disk `/sys/block` list (no
partitions): the agent runs `smartctl -i -A -H` per physical disk — 5 s bound per device,
20 s per pass, results cached 4 minutes — and parses health, temperature, power-on hours,
the sector alarms and SSD wear into `disk_io.devices[].smart`. Refreshing an expired cache
happens on a background thread while the request is served the previous result, so a wedged
disk can never cost a poll more than its own timeout (only the very first pass after the
agent starts runs inline). A host without
`smartmontools`, or whose agent user cannot read the block devices, is *not broken*: every
disk reports `smart: null` with one explanatory `errors` line, and that line never makes
the system page amber. Enable SMART with `smartmontools` plus the `disk` group (which
`deploy/install-agent.sh` grants when it exists) — the agent shells `smartctl` as itself and
never calls `sudo`, so a sudoers rule alone is not enough — and **redeploy the agent**, since
one installed before phase 09 has no SMART pass to run. Opt out silently with `--no-smart`.
The drive truth this is all developed against lives in `tests/fixtures/smartctl_driveripper_*.txt`:
16 real drives, recorded with serials scrubbed, each one parsed by a test.
Binding off-loopback without a token is refused (exit 2) rather than warned about. Prefer
`--token-file` over `--token`: argv is world-readable through `ps`, and this token guards
hostname, mounts, disk and GPU models, and process names. `deploy/install-agent.sh` creates
@@ -94,7 +111,9 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and `/system`'s footnote prints that, not this default |
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
| `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:<host>` jobs exist exactly when `HOST_TARGETS` does |
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and 20 s of wedged disk must not read as a dead host |
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
@@ -102,7 +121,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
| `NUT_SIMULATE` | `1` replaces upsd with a **labelled** simulator (development only) |
| `DASH_READ_ONLY` | `1` disables every mutating endpoint regardless of other flags |
| `DASH_DB` | SQLite path (default `data/dashboards.sqlite3`) |
| `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer |
| `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer. `/disks` has no `POLL_MS_*` of its own and polls at a fixed 30 s: its payload only moves when the `disks` job re-reads the agents |
| `DEBUGPY` / `DEBUGPY_PORT` | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup |
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |
@@ -120,12 +139,12 @@ unreachable host writes its failure message into the event table and `/api/event
```
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
│ │ + /slots (gufo) │
│ (twice: system 10s, disks 5m) │ + /slots (gufo) │
▼ ▼ ▼
src/dashboards/collectors.py ────────────────────────────────────────────┐
│ parse + Δcounter/Δt (rates.py) │
▼ ▼
sources/host.py · sources/llm.py · sources/nut.py poller.py (one task per source)
sources/host.py · disks.py · llm.py · nut.py poller.py (one task per source)
│ samples events │
▼ ▼
store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ── api.py → web/
@@ -133,7 +152,11 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
* `poller.py` — every source gets its own asyncio task, its own health record, and its own
error handling. A dead host keeps its last good values and is labelled stale; it never
blanks or blocks another source. And it never quietly *stops*: the task is that source's only
blanks or blocks another source. Disk health is a source of its own (`disks:<host>` in
`/api/health`, 5-minute cadence, its own fetch timeout) reading the *same* agent
`/snapshot` the system page reads — a separate task is what keeps a SMART poll out of the
10-second utilisation poll and lets drive history go stale without blinking `/system`, and
`alerts.py` is why one dead agent is still one line on the overview rather than two. And it never quietly *stops*: the task is that source's only
poller, so a SQLite write that gets refused is reported (`ok: false`, the real reason, the
numbers aging, one line in the event log) instead of freezing the source under a green dot.
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
@@ -147,7 +170,12 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
published as a 0-1 ratio is rescaled *for the verdict only* — the number printed stays
what the upstream published. There is no `gpu.*` row: these are LLM servers, a busy GPU
is the job, so the GPU card (and the fleet GPU column) renders uncoloured at every
utilisation — the `gpu.*` history series are still stored and charted.
utilisation — the `gpu.*` history series are still stored and charted. Drive health gets two
rows: `disk.temp_c` (amber 50 °C, red 70 °C) and `disk.percentage_used` (amber 80 %, red
100 % — an SSD that has spent its rated wear is a replacement). The sector counters
(`disk.reallocated`, `disk.pending`, `disk.uncorrectable`) and `disk.power_on_hours` have
**no row on purpose**: a non-zero counter is an alarm chip on presence, zero is the absence
of a chip and never a green "healthy drive", and an odometer has no bad value.
* `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states,
host_payloads, engine_payloads, ups_payload, recent_events)` is a pure projection of the
state the poller already keeps plus the verdicts `severity.py` and the UPS health checks
@@ -157,7 +185,27 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than
one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a
marker that says how many it hid.
* `web/` — four static pages, one stylesheet, hand-rolled SVG charts (`charts.js`). No
* `api.py` — the JSON every page reads: `/api/config`, `/api/health`, `/api/events`,
`/api/history`, `/api/metrics`, `/api/summary`, and one `snapshot` + `history` pair per
dashboard (`/api/system/*`, `/api/disks/*`, `/api/llm/*`, `/api/ups/*`). GETs everywhere
except the single `POST /api/ups/command`. Two rules make the pages legible: a source with
no env config answers `404 {"configured": false}` **without dialling** (so a page says
"not configured" instead of going red), and a history query for a series that does not
exist answers an empty series instead of an error (so a card asks for every plot it could
draw and reads `n/a` for the ones the drive never reported). `/api/disks/snapshot` is the
drive-health surface: one entry per host with its physical disk rows, the agent's `smart`
block verbatim, `smartctl_available` (`true` / `false` = install smartmontools / `null` =
the agent never said), a `severity` beside each row's rated numbers, and a `fleet` block
summed here so the page does no arithmetic — a host that has not answered yet is listed as
an `unreachable` stub of `n/a`s and counted, never summed. `/api/config` carries
`disks.interval_seconds`, which is why a drive card four minutes old is behaving, not
broken. `/api/summary` grew a `disks` block with that page (the same `fleet` sums plus one row
per host — counts, `smartctl_available`, and `failed_devices`, so a notifier learns *which*
drive the hardware called `FAILED` without a second call); it is additive, exactly as `alerts`
was, and the key paths of both are pinned with and without targets configured.
* `web/` — five static pages (`/`, `/system`, `/disks`, `/llm`, `/ups`), one stylesheet, hand-rolled
SVG charts (`charts.js`), and a masthead status dot per dashboard job (`disks` has its own:
the same agent can answer the 10-second utilisation poll and have a wedged SMART pass). No
framework, no CDN, no bundler, and no narration: a page never explains its own pipeline, so
the words on screen are the ones you act on — a unit, a `Δ10.0s` provenance tag, a
`SIMULATED` badge, or the sentence beside an empty panel. The overview (`/`) is exactly two panels — a
@@ -196,7 +244,23 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
processes — folds into a collapsed
"More detail" disclosure. Each panel pulls its series in **one** `/api/system/history` request
(`hosts=<that host>&metrics=…&range=…`) every refresh, stack open or not, and a host that stops
reporting takes its panel, its Charts toggle and its charts off the page. `/llm` opens with
reporting takes its panel, its Charts toggle and its charts off the page.
`/disks` is the drive-health page: **one card per physical drive** (the agent's
whole-`/sys/block` list — never a partition, never a mount), led by the device path and the
drive's own verdict word (`OK` / `FAILED` / `unknown`, coloured only by `severity.health`), then
three equal cells — **Temp**, **Power-on** hours, **Wear** — coloured by `disk.temp_c` and
`disk.percentage_used` from the API, with the odometer deliberately uncoloured because
`severity.py` rates no power-on hours. The sector counters appear as **red chips that exist only
while non-zero**, so a healthy drive shows no alarm row at all rather than a wall of zeroes.
A host whose SMART cannot be read keeps its cards, dimmed with every number `n/a` and one
`.banner` line saying why — `smartctl not available on <host>` when the agent reported the binary
missing, its own sentence when it reported something else, and "not answering" when the agent is
down — never a red panel, because a missing binary is unconfigured, not broken. The header's
drive counts are the API's `fleet` block (hidden below two hosts, so one box's sums never
duplicate its own cards), and each host's **Charts** disclosure behaves exactly like `/system`'s:
collapsed, built only while open, and fed by one `/api/disks/history` request per host per
refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive it has.
`/llm` opens with
the **fleet**: one block above the panels — `ALL ENGINES`, an `N of M engines` tag, the note
*sums only engines answering now*, then **running / waiting / prompt tok/s / decode tok/s /
total tokens** — and every number on it is `fleet_totals()`, so the page sums nothing itself.
@@ -240,7 +304,7 @@ store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event ──
battery °C) kept below them in smaller muted type because they are what you read *while* the
UPS is on battery. Health checks, the battery-test buttons (with their `DANGER`
confirmation) and the NUT variable dump are unchanged, just second.
The three event feeds (`/system`, `/llm`, `/ups`) are a **paged window** over the `event`
The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event`
table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a
page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath.
One helper, `Dash.eventsPager()`, renders every feed and each keeps its own row markup; the fetch
@@ -258,6 +322,9 @@ uv run pytest -m live --live # optional: real fleet / engines
uv run ruff check src tests agent && uv run pyright # lint + types
./scripts/capture-fixtures.sh # re-record ground truth: host agents, UPS replies, gufo /slots
./scripts/capture-fixtures.sh --with-llm # ... and the engines' /metrics while they are up
SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh
# ... and smartctl per physical disk: plain smartctl,
# `sudo -n` if that is denied; scrubs serials/WWNs
bash .agents/validate.sh # all of the above + `node --check` on every page
# script + repo hygiene, with the 94% coverage floor
```