services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 09_disks_smartctl

Browse Source 
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**

- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
This commit is contained in:
ducoterra
2026-10-18 00:45:01 -04:00
parent 19cfedfddf
commit fefbeb8227
81 changed files with 5525 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+204 -29
View File
@@ -19,13 +19,14 @@ keep its numbers honest, and the state of the fleet it watches.
## What this is
Three dashboards for one homelab, behind one domain, in one process, backed by one SQLite
file — plus an overview page that says which of the three you need:
Four dashboards for one homelab, behind one domain, in one process, backed by one SQLite
file — plus an overview page that says which of the four you need:
| Route | Question it answers | Source |
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
| `/` | Is anything wrong, and where do I look first? | all three, in one request |
| `/` | Is anything wrong, and where do I look first? | all four, in one request |
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent/host |
| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) |
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
@@ -101,7 +102,7 @@ agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
collectors.py ── fetch, one UpstreamError type, no writes except run_ups_command ──┐
│ parse + Δcounter/Δt │
▼ ▼
sources/host.py · sources/llm.py · sources/nut.py poller.py (task per source)
sources/host.py · disks.py · llm.py · nut.py poller.py (task per source)
│ Result(payload, samples, events) │
▼ ▼
store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ──► api.py ──► web/
@@ -114,20 +115,69 @@ Responsibilities, one line each:
HTML status page at `GET /`. Refuses to bind off-loopback without a token (exit 2).
Counters stay counters: the dashboard computes
rates so the agent never remembers who polled it last.
SMART (phase 09): `smart()` runs `smartctl -i -A -H` per whole `/sys/block` disk
(`parse_smart_output` covers ATA + NVMe; NVMe media errors map to
`offline_uncorrectable`, hex-packed vendor raws are never decoded, and an attribute row is
read from its **RAW_VALUE column** rather than the end of the line because this binary
annotates that column — `35 (Min/Max 34/44)`, `43894h+43m+32.519s` — so its last token is
not the value), 5 s per device /
20 s per pass / 240 s cache, attached as `disk_io.devices[].smart`. Missing binary →
**one** `errors` line for the pass (the word `not found` in it is what
`sources/disks.py` reads to answer "is smartctl installed", and
`tests/test_agent.py` pins that the line still says it); one denied/wedged device → that
disk's `smart` is `null` with its own line; `--no-smart` opts out silently. Never fails
the snapshot. **An expired cache refreshes on a background thread while the request is
served the previous result** — only the first pass after startup runs inline — because 20 s
of wedged disk on the request path blows `HOST_TIMEOUT_SECONDS` and would turn the
*system* source red every four minutes over an add-on that page never draws. The refresher
is single-flight and keeps the old cache on failure, so a bad pass retries on the next
expiry instead of hammering the disk.
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
env var always wins. Defaults are safe except the sources, which default to unset.
- `poller.py` — job loop, health records, stale/last-good semantics, transition events. The
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
`/disks` is a **source of its own** (`disks:<host>`, `DISKS_INTERVAL_SECONDS` 300 s,
`DISKS_TIMEOUT_SECONDS` 30 s, events under the dashboard name `disks`) reading the same
agents the `system` jobs read: the separate task is what keeps a SMART poll out of the
10-second utilisation loop and lets `/disks` age on its own instead of blinking `/system`.
The job list is built once (`app.build_hub()`) because it is wired twice — production and
the injected-store test path — and two hand-written copies drifting is how a source ends up
polling in tests and not in the container. The
success path records (payload → samples → events) and only *then* moves `last_ok`, so a
refused SQLite write ages the source and alerts instead of leaving it current with charts
that have silently stopped drawing.
Events are stored under the **dashboard** name (`system`/`llm`/`ups`), not the sample
Events are stored under the **dashboard** name (`system`/`disks`/`llm`/`ups`), not the sample
source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
(found on `/system` during phase 03 browser verification).
- `collectors.py` — the fetch half. `NutMonitor` owns per-UPS transition bookkeeping. Failure
messages name the **redacted** URL (`strip_userinfo` / `Target.display_url`), never the one
that was dialled, because they become health records and event rows.
that was dialled, because they become health records and event rows. `fetch_snapshot()` is
the one agent `/snapshot` read that `system` and `disks` share, so a dead host says the
same sentence on both pages and only the `timeout` differs (the disks poll has to outlast
the agent's 20 s smartctl pass, and `None` is translated into httpx's sentinel because a
literal `None` means *no timeout at all*). Agent `errors`
become the payload's `degraded` **except** the `smart:` add-on lines — a host without
smartmontools or device read access is unconfigured, not broken (invariant 3); they stay
in `agent.errors` for the `/disks` page to explain.
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
(`gpu.busy_pct@<index>`).
- `sources/disks.py` — the same snapshot projected onto **physical** drives for `/disks`
(phase 09): one row per whole-`/sys/block` disk (never a partition, never a mount), the
agent's `smart` block carried through verbatim, `health` the drive's own word normalised to
`ok`/`failed`/`None`, and `smartctl_available` three-way on purpose — `True` (a disk
answered), `False` (the agent said the binary is not installed, so the page can say *install
smartmontools*), `None` (an agent predating the pass, or `--no-smart`: nobody knows, so
`n/a` plus the agent's own `smart_errors` line and never a guessed verdict). **Nothing here
is a rate**: power-on hours and the sector counters are stored counters-as-reported, so the
source takes no `RateEstimator` and its payload carries no `window_seconds` (rule 2 — a rate
with no estimator, no `Δ<window>` label and no `source` string is an invented number).
`disk.health_ok@dev` is not a series either: a verdict is not a number, so the drive's word
travels in the payload only. Each row does carry a `severity` dict (`SEVERITY_FIELDS` = the
two rated numbers, `temp_c`/`percentage_used`, plus `health` through `HEALTH_SEVERITY`,
which translates the drive's own word instead of grading it), and `UNGRADED_FIELDS` names
the numbers that must never be graded. `fleet_disk_summary()` sums the per-host `summary`
counts for the page header and drops a host with no payload from the sums while still
counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does
arithmetic and a dead agent never shrinks a fleet total.
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
`VLLM_HISTOGRAMS`), measured Δ rates, prefill speed as
`mean(prompt tokens)/mean(prefill time)`, session list from gufo `/slots`.
@@ -170,6 +220,12 @@ Responsibilities, one line each:
a busy GPU is the job at every level, so the GPU card and the fleet GPU column render
uncoloured — like `net.*`, whose colourlessness predates it. Removing the verdict is not
removing the data: `gpu.busy_pct@N` / `gpu.mem_used_pct@N` are still stored and charted.
The `/disks` rows (phase 09) are `disk.temp_c` (50/70) and `disk.percentage_used` (80/100)
— an SSD at 100 % of its rated wear is a replacement, not a warning — attached per row by
`sources/disks.py`; the sector counters and `disk.power_on_hours` deliberately have **no
row**, because a non-zero counter is an alarm chip on presence and zero is not an
achievement (the `/llm` preemption-chip rule), so `classify("disk.reallocated", 1)` is
`None` and the page shows a chip instead of a colour.
- `alerts.py` — `build(settings, job_states, host_payloads, engine_payloads, ups_payload,
recent_events)` → the "what is wrong right now" list `/api/summary` serves as `alerts`
(`{id, severity, source, message, href, at}`, fail first then warn, then by source). A
@@ -177,39 +233,74 @@ Responsibilities, one line each:
reachability, the `severity` dicts for breaches, `health.checks[]` for UPS conditions.
Rules: an **unconfigured** source never alerts (invariant 3); a source that is ok but
older than 3× its own poll interval is `warn` while a source that stopped answering is
`fail`; a dead source gets exactly one line (its stale payload adds nothing) that names it
once — the poller's `"UpstreamError: "` wrapper and the duplicated host name are stripped,
because a message that says `gateway` twice reads as two problems; `unknown` and missing
numbers are silence; one alert per (source, metric) and at most 12, the last being a marker
that says how many it hid. This is why the overview can be two panels.
`fail` (the interval is the job's own dashboard's, so `disks` goes amber after 15 quiet
minutes rather than 30 seconds); a dead source gets exactly one line (its stale payload
adds nothing) that names it once — the poller's `"UpstreamError: "` wrapper and the
duplicated host name are stripped, because a message that says `gateway` twice reads as two
problems, and the same rule silences the **`disks` echo** of a dead agent (the disk job
dials the same `/snapshot`, so its failure carries no new information — while a `disks` job
failing *while its host answers*, SMART wedged inside the handler, is the only voice that
fault has); `unknown` and missing numbers are silence; one alert per (source, metric) and at most 12, the last being a marker
that says how many it hid. This is why the overview can be two panels. There is **no
disk-health rule and none is wanted**: a drive the hardware itself called `FAILED` is a red
chip on its `/disks` card and a line in that page's event feed, not an overview alert —
`severity.py` rates no sector counter and this module may not have a threshold of its own, so
grading a drive here would be the bug, not the fix (phase 09). Its `HREF` table does carry
`/disks`, so the one alert a disk job *can* raise — SMART failing while its host answers —
links to the page that names the drive.
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning.
- `api.py` — GETs everywhere plus the one POST. `/api/config` is redaction boundary.
`/api/disks/*` (phase 09) is the drive-health pair: `/snapshot` answers one entry per
configured host (disk rows + `smart` + `smartctl_available` + per-row `severity` + a
`fleet` block summed by the API, with a configured-but-never-answered host as an
`unreachable` stub whose counts are `None`, not `0`), and `/history` serves the
`disk.*@dev` series with unknown names answering as empty series like the other two
history endpoints. The configured gate runs **before** anything that could reach the
network — `HOST_TARGETS` empty is `404 {"configured": false}` with no client, no fetch and
no `disks` job in the hub at all (invariant 3), and `tests/test_api.py` proves it with
spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot`. The new surface grows no
write of its own — a fourth dashboard must not quietly become a second one, so the app's
only POST stays `/api/ups/command` (invariant 1, pinned by enumerating every served route's
verbs).
`/api/summary` is the overview's single call and the cheapest whole-fleet probe there is:
the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `llm.engines`,
`ups.ups`, the `states` maps and `recent_events` stay for programmatic consumers (live
tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events` are unchanged
and `alerts` is additive — `tests/test_api.py` pins every one of those key paths, with and
without targets configured, so retiring a panel can never shrink the payload. A configured
the page renders only `alerts` plus `system.fleet`/`system.hosts`, while `disks.hosts`,
`llm.engines`, `ups.ups`, the `states` maps and `recent_events` stay for programmatic
consumers (live tests, an external check, a notifier). `system`/`llm`/`ups`/`recent_events`
are unchanged and `alerts` (phase 02) and `disks` (phase 09) are additive — the `disks`
block is `fleet_disk_summary` verbatim plus one row per host (`disks`/`with_smart`/
`health_failed`/`alarm_disks`/`smartctl_available`/`failed_devices`), so a notifier learns
*which* drive the hardware called `FAILED` without a second call, and
`tests/test_api.py` pins every one of those key paths, with and without targets configured,
so retiring a panel can never shrink the payload. A configured
job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
`system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
counts the machines there are and `hosts_live` the ones answering.
- `app.py` — assembly; `create_app()` takes injectable settings/store so tests build an app
with an in-memory DB and no targets.
- `web/` — four static pages, one stylesheet, `charts.js`. No framework. `app.js` owns the
- `web/` — five static pages (`/`, `/system`, `/disks`, `/llm`, `/ups`), one stylesheet,
`charts.js`. No framework. `app.js`'s masthead status dots are one per dashboard job —
including `disks`, because the same agent can answer the 10-second utilisation poll and
have a wedged SMART pass, and one dot for both would be right about neither. `app.js` owns the
colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
`ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
`.hero` big number (plus an optional `.bar`) from that class — it never compares a number
to a threshold. `Dash.severityFor(value, warnAt, failAt)` is the documented fallback for
values the API does not rate, and **no page script calls it any more** — `/` was converted
in phase 02, `/system` in phase 03, `/llm` in phase 04 and `/ups` in phase 05, so every
colour on all four pages comes from a `severity` verdict the API sent
in phase 02, `/system` in phase 03, `/llm` in phase 04, `/ups` in phase 05 and `/disks`
shipped that way from its first commit, so every colour on all five pages comes from a
`severity` verdict the API sent
(`tests/test_api.py` fails if a page script names it again). A value the API should rate
gets a row in `severity.py`, never a bound in JS.
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
series — `/api/metrics` answers with exactly these names.
series — `/api/metrics` answers with exactly these names. The `sample.source` column is the
dashboard family — `host`, `llm`, `ups`, and `disks` for the SMART series
(`disk.temp_c@nvme0n1`, `disk.power_on_hours@…`, `disk.reallocated@…`, `disk.pending@…`,
`disk.uncorrectable@…`, `disk.percentage_used@…`). Those keep their own family instead of
sitting beside `host`'s `disk.used_pct@/mount` because the cadences differ thirty-fold and
`GET /api/metrics?source=disks` has to answer "what does *this page* have history for".
That scheme is the history API's key — treat it as a public interface.
## The fleet (real environment, verified 2026-10)
@@ -243,6 +334,42 @@ That scheme is the history API's key — treat it as a public interface.
- Host access is via ssh keys as `ops`. Passwordless sudo exists but is command-scoped; the
set we rely on is documented in `deploy/nut-sudoers` (additions are listed there for the
work that is still blocked).
- **SMART is captured from real hardware; the live agents still cannot read it.** `smartctl`
7.5 is installed on deskwork and driveripper. The `ops` account (which runs the dev agents)
is in no `disk` group; driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and
deskwork grants nothing, so the agent's own unprivileged `smartctl -i -A -H` — it shells the
binary and never calls sudo — answers *Permission denied* (exit 2) on every host and `smart`
stays `null` end to end: the designed degradation, not a bug. The three deployed agents also
predate the SMART pass, so the `disks` source today yields one row per physical disk (16 on
driveripper, 1 on deskwork, `vda` on gateway) with `smart: null`, `smart_errors: []` and
`smartctl_available: null` — "nobody told us either way", which is why that flag is
three-way and not a boolean.
**Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
every one — which is what found the parser's one real bug: this binary annotates the
attribute table's RAW_VALUE column (`35 (Min/Max 34/44)` on every Seagate, and one Western
Digital writes its hours as `43894h+43m+32.519s`), so reading the *end* of the row lost the
temperature of all eight HDDs and one power-on count. The captures are also the first
honest look at this fleet's drives: `nvme5n1` reports 13 media-and-data-integrity errors and
`nvme1n1` is at 58 % of its rated wear, so the alarm chip and the amber wear bar on `/disks`
are real features with real data behind them, not demo furniture.
**Lighting up the live page is two operator actions that travel together:** give the agent
user device read access — the `disk` group (`sudo usermod -aG disk <agent-user>`, which is
what `deploy/install-agent.sh` does on install), *not* a sudo rule, because the agent must
read the device itself — and redeploy `agent/host_agent.py` (never redeploy without the
grant: 16 permission-denied lines would age nothing but are pure noise, while with the grant
they are 16 drives' worth of temperature, hours and sector counters). Until then `/disks`
renders the designed degradation: one dimmed card per physical drive, every number `n/a`, and
the one-line why — which is also what gateway shows permanently, being a VM with no physical
disk to read and no smartmontools at all.
`tests/fixtures/smartctl_reference_*.txt` stay as what they are: **reference shapes** (labels
verbatim from the deployed binary, values synthetic), kept beside the real captures so the
synthetic edge cases stay pinned. Re-record the real ones with
`SMART_HOSTS="driveripper-ops" ./scripts/capture-fixtures.sh`: it tries plain `smartctl`
first (so a fixture can never read healthier than the agent does), falls back to `sudo -n`,
refuses to overwrite good fixtures with unreadable output, and scrubs
Serial/WWN/EUI-64 (`test_smartctl_captures_carry_no_unscrubbed_serials` fails the day it
stops).
## Commands
@@ -256,11 +383,14 @@ cp .env.dev.example .env.dev # dev-only: ssh tunnels + live targets
./scripts/dev-server.sh status # per-source ok/DOWN
./scripts/dev-server.sh logs
uv run pytest # hermetic: 624 passed + 9 live-skipped, no network
uv run pytest --cov # coverage gate (97.9%; floor 94%)
uv run pytest # hermetic: 769 passed + 11 live-skipped, no network
uv run pytest --cov # coverage gate (98.33%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
uv run ruff check src tests agent && uv run pyright
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
# ... plus smartctl per physical disk (needs device read
# access: plain smartctl, else `sudo -n`; scrubs serials)
bash .agents/validate.sh # the phase gate: the four checks above (pytest
# --cov --cov-fail-under=94, ruff, pyright, plus a
@@ -284,7 +414,11 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
- Anything that must match an upstream — `/proc` layouts, vLLM series names, NUT framing —
is asserted against a verbatim capture in `tests/fixtures/`. If the schema drifts, a
fixture test fails before a dashboard does.
fixture test fails before a dashboard does. `smartctl_driveripper_<dev>.txt` is the whole
16-drive fleet read through a `smartctl` sudo rule (Serial/WWN/EUI-64 scrubbed), and
`test_the_real_drives_answer_every_number_the_disks_page_prints` asserts every number
`/disks` prints comes out of real drive output — that test is what caught the parser
reading the end of an attribute row instead of its annotated RAW_VALUE column.
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
- **Page payloads that a rewrite must not break are pinned as a table.**
@@ -337,9 +471,11 @@ measured rates, `n/a` for the unknown, `SIMULATED` badges on modelled data, per-
dots in the header. Every chart names itself for assistive tech: pages pass `ariaLabel` to
`LineChart`, which puts `role="img"` + `aria-label` on the `<svg>` (an unlabelled plot is
silence, and the legend is a sibling that labels nothing). Every `LineChart` on every page has
one — `/system`'s three per-host slots, `/llm`'s four trend plots, `/ups`'s four battery
plots — each pinned by `tests/test_api.py`, so a chart that ships anonymous fails a test.
Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups`; the API accepts 5m…90d). A
one — `/system`'s three per-host slots, `/disks`'s two per-host plots, `/llm`'s four trend
plots, `/ups`'s four battery plots — each pinned by `tests/test_api.py`, so a chart that ships
anonymous fails a test.
Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups` and `/disks`; the API accepts
5m…90d — SMART lands every 300 s, so a 5-minute window on `/disks` is one dot). A
panel that cannot get data says why in the panel — the operator should never have to open
devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
@@ -386,6 +522,34 @@ muted unit, mono `.sub` footnote (`Δ10.0s`, "at current load"), coloured by the
family and no second palette, and `tests/test_api.py` fails if either appears or if a CSS
rule reaches for a token that `:root` does not declare.
**The `/disks` card is one panel per *physical* drive, led by identity and verdict, not by a
hero** (phase 09): a drive has no one-number story the way a UPS has charge %, so the card opens
with the device path (`/dev/nvme0n1`) and the drive's own word (`OK` / `FAILED` / `unknown`, a
`.chip` coloured only by `severity.health` — an unreadable drive gets grey `unknown`, never a
green pass), then the identity line (`model · size · SSD|HDD`, nothing invented where a field is
absent) and three equal `.metric` cells: **Temp**, **Power-on**, **Wear**. Colour comes from
`severity.temp_c` / `severity.percentage_used` and nowhere else; `power_on_hours` is an odometer,
written with a `null` verdict, so it cannot read green at 100 h or red at 60 000 h. The sector
counters are **`.chip.fail` alarm chips that exist only while non-zero** (the `/llm` preemption
rule, and why `severity.py` rates no counter): a drive at zero draws no chip row at all, because a
permanent "0 pending sectors" line trains the eye to ignore it and a green one would call a quiet
drive an achievement. Per host the three-way `smartctl_available` decides one `.banner.info`
why-line and nothing else — `false` says *install smartmontools and grant the disk group* (the
agent said so, so the page may), `null` prints the agent's own `smart_errors` line or names both
fixes without claiming to know which half is missing, and an agent that is not answering says
that instead — and in every case the drive grid takes the shared `.stale` dim while the host card
stays uncoloured (invariant 3: a missing binary is unconfigured, not broken, and a red panel here
would train the operator to ignore red on the drives that do report). Nothing on the page is per
partition or per mount: the rows are the agent's whole-`/sys/block` list, and a filesystem field
naming its way into `page-disks.js` fails a test. The fleet header renders
`fleet_disk_summary()` — hidden below two hosts (one box's sums duplicate its own cards) and
coloured nowhere, because a count of drives is not a graded number — and the per-host `Charts`
disclosure follows `/system` exactly: collapsed, lazy-built, **one** `/api/disks/history` request
per host per refresh naming `disk.temp_c@dev` and `disk.percentage_used@dev` for every drive (the
endpoint answers unknown names as empty series on purpose), the wear plot created only once some
drive publishes the field, legends naming drives because the card heading already says who, and
`pruneHosts` taking the cards, the toggle and the plots off the page together.
A cell-scale trend is `DashCharts.sparkline()` — the `llm_monitor` `drawSpark` idiom, so a
percent metric is plotted against `fixedMax` (**never** autoscaled: an 8 % KV window that once
held a 92 % spike must not read flat, and the axis gets top/bottom scale marks so "a third of
@@ -442,7 +606,7 @@ repaints only when charge / verdict / state change (`drawGlyph()` keys a `data-d
charge float jitters every `NUT_INTERVAL_SECONDS`, and rebuilding the `<svg>` on that re-announces
an accessible name for a difference no eye can see.
An event row names its subject **once**. Every feed (`/system`, `/llm`, `/ups`) renders the
An event row names its subject **once**. Every feed (`/system`, `/disks`, `/llm`, `/ups`) renders the
message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a
*leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py`
messages start with their own label, and each row already prints that label as a chip, so
@@ -452,7 +616,8 @@ This is presentation only — the stored event and `/api/events` keep every word
error string still surfaces in `/api/health`.
An event feed is a **window**, not a printout (`Dash.eventsPager(el, events, opts)` in
`app.js`, one helper with three call sites — a pager that works on one page is the bug this
`app.js`, one helper with four call sites (`/system`, `/disks`, `/llm`, `/ups`) — a pager that
works on one page is the bug this
shape prevents). `ul.events` is a bounded 320px scroll container (`overscroll-behavior:
contain`) and the pager puts `EVENT_PAGE_SIZE` = **50** events in it, with a `.events-pager`
strip under the list: `1–50 of 200` between `‹ prev` and `next ›`, both `aria-label`led for
@@ -658,6 +823,16 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
field map is written.
3. **UPS history depth.** Events and the numeric series are stored; long-window battery
health (capacity fade over months) has the rollups but no dedicated view.
4. **The dev DB is big enough to refuse a write now and then.** The dev store (~590 MB, WAL
grown to ~70 MB) logs `store-error: OperationalError: database is locked` a few times an
hour across `system`/`llm`/`ups`; the poller handles it exactly as invariant 4 requires —
the source goes red with the real reason, ages, and writes `recording` when the next poll
lands — so nothing freezes, but those polls' samples are lost. The one cause this repo can
prove is two servers on one file, which is why `scripts/dev-server.sh stop` waits for the
old pid to die (its comment records the incident). A controlled 25 s external read
transaction during this pass produced none, so checkpoint starvation is not yet shown.
Retention/`wal_checkpoint` tuning for a long-running store is its own phase, not a
phase-09 follow-up.
## Locked decisions (and why)