phase: 16_agent_smart_pass_budget

**Report — phase 16 final verification pass** (`16_agent_smart_pass_budget`)

- Verified both tasks' work is present and non-tautological: `_smart_pass` scales `max(20 s, 5 s × disks)` with the override winning; `tests/test_agent.py` pins the 16-disk scale (no truncation), the 20 s floor, and the exact `pass budget of 80s exhausted` line, plus the untouched `0.0`-override test. No code change this pass (only the docstring reflow already in the tree).
- Completion criteria:
  - `uv run pytest tests/test_agent.py` → **105 passed** (4 pass-budget tests green, `-k pass_budget` → 4 passed) ✅
  - `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` → **no matches** (exit 1) ✅
  - `bash .agents/validate.sh` → **EXIT=0**: `uv run pytest --cov --cov-fail-under=94` **1427 passed, 15 skipped, coverage 98.94 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` + repo hygiene clean ✅
  - `uv run pytest -m live --live` (with `.env.dev`) → **11 passed**; `./scripts/dev-server.sh status` → nine sources `ok`; live `/api/disks/snapshot` → driveripper **16/16 drives with temps, no budget line** ✅
  - driveripper md5 byte-identity → **not met, deliberately no live change**: deployed copy is 1.0.0 and differs from the repo only by phase 15's TLS block (verified by diff); `ops` there *is* sudo-granted a staged `install-agent.sh` (new finding), but re-running it rewrites that host's hand-diverged `User=root`+`CapabilityBoundingSet` unit and re-breaks the 16-drive SMART read (gap 10) — needs root + phase 17's template fix. Recorded in AGENTS.md gap 11.
- Docs (full README + AGENTS review, incl. pre-existing drift): fixed 4 inaccuracies — AGENTS fleet record claimed *neither* host sudo-grants the installer (driveripper does, and that grant is the trap); `/health` publishes `version`, not `agent_version`; deskwork's "55 °C amber" marked as a dated capture (48 °C `ok` on re-read); gap 10's title still read as an open defect though resolved the same day. README verified accurate end-to-end (SMART budget wording, TLS rollout, config table, routes, test philosophy) — no further change.
- Notable: left `NOTES.md` (untracked operator scratch) alone; its "admin buttons disappear on `/certs`/`/settings`" item is deliberate, test-pinned behaviour (`test_every_page_links_to_both_admin_pages`) and belongs to a future phase, not this one.
- Next pending phase: `17_installer_device_filter`.
This commit is contained in:
2026-10-20 11:14:35 -04:00
parent 4b0f53c499
commit 7e367a9014
23 changed files with 337 additions and 17 deletions
+36 -13
View File
@@ -854,10 +854,16 @@ That scheme is the history API's key — treat it as a public interface.
`upsc`, `udevadm`, the `/etc/ups` reads, and the `tee /etc/ups/upsd.users|upsd.conf` +
`systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork`
grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`).
Neither host grants `ops` the installer or `visudo` (re-checked 2026-10-19), so every install
that has happened on those two went through root by the operator — `driveripper`'s included,
which is why its unit is not the one this repo writes (see the SMART block: `DevicePolicy=auto`,
no `DeviceAllow`). Worth knowing before the next install: the allow-list `install-agent.sh`
`deskwork` grants `ops` neither the installer nor `visudo`, so its install went through root by
the operator. **`driveripper` does grant the installer** — `(root) NOPASSWD:
/home/ops/agent-install/deploy/install-agent.sh`, an Oct-18 snapshot of the script staged under
the operator's home (`sudo -n -l`, re-checked 2026-10-20) — and that grant is the trap, not the
shortcut: the installer rewrites the unit unconditionally, so running it there today replaces the
`User=root` + `CapabilityBoundingSet` unit that gap 10 resolved with this repo's
`User=dashboards-agent` + `DevicePolicy=strict` template and re-breaks all 16 SMART reads. The
live unit is not the one this repo writes (no `DevicePolicy`/`DeviceAllow` lines at all —
systemd's default `auto` — `PrivateDevices=false`, and the two named capabilities).
Worth knowing before the next install: the allow-list `install-agent.sh`
does write (`/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`) has **no `/dev/ng*n*` and no
`/dev/bsg/*`**, the nodes smartctl 7.5 reaches for NVMe admin and SATA pass-through, so a
`strict` unit there would deny exactly what a SMART read needs — whatever phase takes
@@ -870,9 +876,9 @@ That scheme is the history API's key — treat it as a public interface.
What changed by 2026-10-19 is that `deskwork`'s **installed** service runs as
`dashboards-agent`, which is in the `disk` group (`getent group disk` → `disk:dashboards-agent`)
on an agent installed after the SMART pass: `/disks` shows real numbers there —
`smartctl_available: true`, `nvme0n1` healthy at 55 °C (amber, past `disk.temp_c`'s 50 °C
threshold), 19 166 power-on hours (an odometer,
so that figure is a capture date, not a fact about the drive), 11 % of rated
`smartctl_available: true`, `nvme0n1` healthy at 55 °C that day — amber, past `disk.temp_c`'s
50 °C threshold, and a plain 48 °C `ok` on the 2026-10-20 re-read — 19 166 power-on hours (an
odometer, so that figure is a capture date, not a fact about the drive), 11 % of rated
wear — so the wear bar and the alarm chips are drawn from live hardware, not only fixtures.
`gateway` answers `smartctl_available: false` (no smartmontools in its VM: the *install it*
half of the three-way flag), and **`driveripper` is the fourth case the flag was built for:
@@ -1705,7 +1711,8 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
(`test_a_fallback_inside_the_documented_ceiling_still_publishes`), so a span-relative floor
would be a re-decision of that phase, not a bug fix. Do not "fix" the residual by hiding the
short window or its label.
10. **`driveripper`'s 16 drives read `n/a`: kernel 7.2's unprivileged-passthrough gate (2026-10-19).**
10. **`driveripper`'s 16 drives read `n/a`: kernel 7.2's unprivileged-passthrough gate
(2026-10-19, resolved the same day — the installer template is what is left).**
The agent is installed, the `disk` group grant is in place, the block nodes open, and the
page renders all 16 real drive rows — but kernel 7.2.11 blocks the *agent user's* `smartctl`
at the command level (not the open; the probe matrix is in
@@ -1716,7 +1723,7 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
read needs the drive's ACEL to advertise the page (the 980 PRO's `5B2QGXA7`, 2020-era,
does not: `NVME_IOCTL_ADMIN_CMD: Permission denied`) or `CAP_SYS_ADMIN`; `deskwork` reads
live because its firmware does. Options, in order of narrowness: (a) accept the designed
degradation (what `/disks` renders now — the identity is real, the values are `n/a` with a
degradation (what `/disks` rendered then — the identity is real, the values are `n/a` with a
why-line); (b) `AmbientCapabilities=CAP_SYS_RAWIO` (+ `CapabilityBoundingSet`) on the unit
— unblocks exactly the 8 HDDs, is the capability the kernel names for the gate, but a
compromised agent could then write raw block I/O — test it in a `systemd-run --scope`
@@ -1731,10 +1738,16 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
`CapEff` came back `0000000000220000` (SYS_ADMIN + systemd's auto CHROOT; RAWIO in the
bounding set but not effective) — the 8 SATA drives' reads were unblocked by SYS_ADMIN
alone, so the effective grant is {SYS_ADMIN}; the RAWIO in the line stays as the named
intent for the `SG_IO` gate. The agent also gained the pass-budget scale-up (20 s floor
→ 5 s × disk count; an explicit override still wins, the budget test is untouched),
because a 16-disk pass takes more than 20 s when the drives answer. The `deploy/install-agent.sh`
template still ships the `DevicePolicy`/`DeviceAllow` lines that misbehaved on this host
intent for the `SG_IO` gate. The agent also gained the pass-budget scale-up — 20 s floor, 5 s ×
disk count, an explicit override still winning — because a 16-disk pass takes more than 20 s
when the drives answer, and truncating it would null the tail drives. Phase 16 pinned that
default in `tests/test_agent.py` (`test_the_pass_budget_scales_with_the_disk_count`,
`test_the_pass_budget_floor_is_twenty_seconds`,
`test_the_pass_budget_error_line_names_the_scaled_figure` — which demands the line read `80s`
for 16 disks, not the floor; the override test stays). Re-read live on 2026-10-20: the installed
agent answers 16/16 drives with temperatures and no `pass budget … exhausted` line. The
`deploy/install-agent.sh` template still ships the `DevicePolicy`/`DeviceAllow` lines that
misbehaved on this host
(a deny-all filter; the `nvme*` glob matches nothing on this kernel because
`/proc/devices` lists NVMe as `blkext`) — the template change is its own phase.
`tests/fixtures/smartctl_driveripper_*.txt` (captured via `sudo`) remain the ground truth
@@ -1803,6 +1816,16 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
stays `http://` here until somebody installs a bundle on a host — which needs root on that host
(`ops` has no installer grant on `deskwork`, and `driveripper`'s unit must not be rewritten, gap
10), and flipping the URL before the bundle is installed turns that host red by construction.
Re-checked from the dev box the same day, and worth knowing before that rollout: all three
agents answer `/health` with `version` **1.0.0** (`agent_version` is the `/snapshot` spelling),
so none of them carries task 02's TLS code either — the rollout ships the repo's
`agent/host_agent.py` (1.1.0) *together with* the
bundle, never the bundle alone. `driveripper`'s installed copy differs from the repo's by
exactly that TLS block; its `_smart_pass` and the phase-16 budget scale-up are already there.
Copying 1.1.0 into place is not an `install-agent.sh` re-run on that host (gap 10) — even
though `ops` there is sudo-granted a staged copy of the installer (see the fleet record above),
which is the one route that must not be taken until phase 17 fixes the unit template — and no
live change was made while pinning phase 16.
## Locked decisions (and why)