phase: 09_disks_smartctl
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**
- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
This commit is contained in:
@@ -4,9 +4,15 @@
|
||||
# ./scripts/capture-fixtures.sh # needs the tunnels up: scripts/dev-tunnels.sh start
|
||||
# ./scripts/capture-fixtures.sh --with-llm # also scrape the live engines' /metrics
|
||||
# # (gufo /slots is captured without the flag)
|
||||
# SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
|
||||
# # also capture smartctl per physical disk
|
||||
#
|
||||
# Fixtures are verbatim, not hand-written: a fixture you typed is a test of your memory.
|
||||
# When a host's kernel or an engine's metric names change, re-run this and read the diff.
|
||||
# Exception on record: tests/fixtures/smartctl_reference_*.txt are reference shapes, not
|
||||
# field captures (no host could reach a block device when they were written). The SMART_HOSTS
|
||||
# pass below writes hardware truth beside them as smartctl_<host>_<dev>.txt, so the reference
|
||||
# shapes stay what they are: synthetic edge cases with verbatim labels.
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
@@ -106,6 +112,67 @@ else
|
||||
echo " (skipped UPS captures — NUT_HOST unset; see .env.dev.example)"
|
||||
fi
|
||||
|
||||
# --- SMART (smartctl -i -A -H per physical disk) --------------------------------
|
||||
# The phase-09 `/disks` page parses these. smartctl must be able to READ the block
|
||||
# devices, and a fleet grants that one of two ways: the agent user in the `disk` group
|
||||
# (`sudo usermod -aG disk <user>`, what deploy/install-agent.sh does) or a sudoers rule
|
||||
# for smartctl alone (what this fleet has). Plain smartctl is tried first because that is
|
||||
# exactly what agent/host_agent.py shells — if the capture needs sudo and the agent does
|
||||
# not have it, the fixtures would read healthier than the fleet does. When plain smartctl
|
||||
# answers "Permission denied" the capture retries under `sudo -n`. A drive that answers
|
||||
# neither way is deliberately NOT recorded (a good fixture is never replaced by a dead
|
||||
# reader's output).
|
||||
# Identifiers never go in the repo: Serial Number, LU WWN Device Id and the NVMe namespace
|
||||
# EUI-64 (which encodes the serial) are replaced with SCRUBBED* markers; everything else is
|
||||
# verbatim, and tests/test_agent.py fails the day this stops happening.
|
||||
# Skipped when SMART_HOSTS is unset, so a laptop without fleet ssh still re-records the
|
||||
# rest. Whole disks only, the same virtual-prefix skip agent/host_agent.py applies —
|
||||
# partitions and dm-* would double-capture the same platter.
|
||||
if [ -n "${SMART_HOSTS:-}" ]; then
|
||||
for host in $SMART_HOSTS; do
|
||||
name="${host%-ops}" # fixtures are named for the host, not the ssh alias, like agent_*.json
|
||||
devs="$(ssh -o BatchMode=yes -o ConnectTimeout=8 "$host" \
|
||||
'for d in /sys/block/*; do n=${d##*/}; case "$n" in loop*|ram*|zram*|sr*|dm-*|md*) ;; *) echo "$n";; esac; done' 2>/dev/null)"
|
||||
if [ -z "$devs" ]; then
|
||||
echo " ✗ smartctl_${name}_* — ssh did not answer (is $host an ssh alias?)"
|
||||
failed=1
|
||||
continue
|
||||
fi
|
||||
for dev in $devs; do
|
||||
tmp="$FIX/.smartctl_${name}_${dev}.txt.tmp"
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=8 "$host" "smartctl -i -A -H /dev/$dev" >"$tmp" 2>&1
|
||||
rc=$?
|
||||
if grep -qi "Permission denied" "$tmp"; then
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=8 "$host" "sudo -n smartctl -i -A -H /dev/$dev" >"$tmp" 2>&1
|
||||
rc=$?
|
||||
fi
|
||||
# A drive that cannot answer health at all (no SMART support, virtual HBA tricks)
|
||||
# is the same fixture contract as an unreadable one: the parse tests require model +
|
||||
# health in every recorded capture, so do not record what cannot be parsed.
|
||||
if [ ! -s "$tmp" ] || grep -qiE "Permission denied|failed to open|not found|does not support SMART|SMART support is: Unavailable|not smart-capable" "$tmp"; then
|
||||
rm -f "$tmp"
|
||||
echo " ✗ smartctl_${name}_${dev} — no readable SMART (rc $rc): install smartmontools and grant this user the disk group or a smartctl sudo rule"
|
||||
failed=1
|
||||
else
|
||||
sed -Ei \
|
||||
-e 's/^([[:space:]]*Serial [Nn]umber:)[[:space:]]*.*/\1 SCRUBBEDSERIAL/' \
|
||||
-e 's/^([[:space:]]*LU WWN Device Id:)[[:space:]]*.*/\1 SCRUBBEDWWN/' \
|
||||
-e 's/^([[:space:]]*(Namespace [0-9]+ )?IEEE EUI-?64:)[[:space:]]*.*/\1 SCRUBBEDEUI64/' \
|
||||
"$tmp"
|
||||
{ echo "# Verbatim smartctl -i -A -H /dev/$dev from $name, recorded by"
|
||||
echo "# ./scripts/capture-fixtures.sh — Serial/WWN/EUI-64 values are SCRUBBED,"
|
||||
echo "# every other byte is what the drive answered. Volatile rows (the"
|
||||
echo "# Temperature raws, Local Time) move; the parse tests read shape."
|
||||
cat "$tmp"; } > "$FIX/smartctl_${name}_${dev}.txt"
|
||||
rm -f "$tmp"
|
||||
echo " ✓ smartctl_${name}_${dev}.txt"
|
||||
fi
|
||||
done
|
||||
done
|
||||
else
|
||||
echo " (skipped SMART captures — SMART_HOSTS unset; see .env.dev.example)"
|
||||
fi
|
||||
|
||||
# --- gufo /slots (per-slot rows) ---------------------------------------------
|
||||
# `tests/slots_contract.py` pins the shape of the `/llm` slot grid (task 04) against
|
||||
# `tests/fixtures/slots_gufo.json`, so this capture is NOT behind `--with-llm`: it is one
|
||||
@@ -176,6 +243,13 @@ changed = []
|
||||
# upsd quotes a field only when it has to, so both `VAR "apc" "ups.serial" "x"` and
|
||||
# `VAR apc ups.serial "x"` are real wire shapes; the value is always quoted.
|
||||
serial_re = re.compile(r'(^VAR\s+\S+\s+"?(?:device|ups)\.serial"?\s+")([^"\n]*)(")', re.M)
|
||||
# smartctl identities: the serial identifies the physical unit and the LU WWN encodes
|
||||
# the vendor's serial — both scrubbed, same deterministic markers as the UPS serials.
|
||||
# "Local Time is:" is capture time, not device truth: normalised so re-captures do not
|
||||
# churn the diff.
|
||||
smart_serial_re = re.compile(r"(?im)^(Serial Number:\s+)(\S.*?)(\s*)$")
|
||||
smart_wwn_re = re.compile(r"(?im)^(LU WWN Device Id:\s+)(\S.*?)(\s*)$")
|
||||
smart_time_re = re.compile(r"(?im)^Local Time is:.*$")
|
||||
for path in sorted(fix.iterdir()):
|
||||
if path.suffix not in (".json", ".txt", ".vars"):
|
||||
continue
|
||||
@@ -191,6 +265,7 @@ for path in sorted(fix.iterdir()):
|
||||
seen[value] = f"00000000-0000-4000-8000-{len(seen) + 1:012d}"
|
||||
return seen[value]
|
||||
|
||||
seen_wwns: dict[str, str] = {}
|
||||
text = uuid_re.sub(swap, text)
|
||||
# "hostname": "gw.lab.example.com" -> "gw": the dashboard's host key is the target name
|
||||
# from HOST_TARGETS, never whatever the box calls itself
|
||||
@@ -210,6 +285,18 @@ for path in sorted(fix.iterdir()):
|
||||
return match.group(1) + serials[body] + pad + match.group(3)
|
||||
|
||||
text = serial_re.sub(scrub_serial, text)
|
||||
if path.name.startswith("smartctl_"):
|
||||
def scrub_field(match: re.Match[str], prefix: str, store: dict[str, str]) -> str:
|
||||
body = match.group(2).strip()
|
||||
if body.startswith(prefix):
|
||||
return match.group(0)
|
||||
if body not in store:
|
||||
store[body] = f"{prefix}{len(store) + 1}"
|
||||
return match.group(1) + store[body] + match.group(3)
|
||||
|
||||
text = smart_serial_re.sub(lambda m: scrub_field(m, "SCRUBBEDSERIAL", serials), text)
|
||||
text = smart_wwn_re.sub(lambda m: scrub_field(m, "SCRUBBEDWWN", seen_wwns), text)
|
||||
text = smart_time_re.sub("Local Time is: (time of capture removed by scripts/capture-fixtures.sh)", text)
|
||||
if text != path.read_text():
|
||||
path.write_text(text)
|
||||
changed.append(path.name)
|
||||
|
||||
Reference in New Issue
Block a user