services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 17_installer_device_filter

Browse Source 
**Phase 17 — final verification pass: complete, all green.**

Verified (criteria):
- `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` → no match; `PrivateDevices=false` retained (line 472) ✓
- `install-agent.sh --help` names `--as-root`/`--caps` with the kernel-gate sentence; `--force` documented ✓
- Fresh fake-root install = pre-phase unit minus the filter lines: `test_a_default_install_unit_is_the_pre_phase_unit_minus_the_filter_lines` + `test_the_template_unit_has_no_device_filter_lines` pass ✓
- Re-run that differs → exit 1 with diff on stdout, `--force` applies, identical re-run silent: `test_a_re_run_that_differs_refuses_without_force_and_applies_with_it` (+ 3 sibling guard pins) pass ✓
- driveripper reproduction — verified **read-only**, no live change: `systemctl cat` shows `User=root`, `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, no filter, same `ExecStart`; `active`, `CapEff 0000000000220000`; `/api/disks/snapshot` → 16/16 drives with temperatures ✓
- `uv run pytest` → **1451 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** ≥ 94, `ruff` clean, `pyright` 0 errors, `node --check` + hygiene clean); `uv run pytest -m live --live` → **11 passed**; dev server 9/9 sources `ok` (one initial live failure was a transient gufo timeout, clean on re-run)

Docs (inconsistencies found + fixed, incl. pre-existing):
- `AGENTS.md` fleet record said driveripper's unit runs `User=dashboards-agent` while the same bullet (and the live host) says `User=root` — rewritten to separate the installed user from the running one.
- `AGENTS.md` gap 10 header still claimed "the installer template is what is left" → now records phase 17 as the fix.
- `README.md` "with no capabilities" → "with an empty `CapabilityBoundingSet` (unless `--caps`…)", since this phase added the capability flag.
- `README.md` battery-test "gated four times" now notes the two session gates in front, matching AGENTS invariant 1.
- `.env.example` was missing three README-documented settings (`DASH_ENV_FILE`, `DASH_RELOAD`, `NUT_SIMULATE`) and `AGENT_CERT_BUNDLE` in the agent-variables note — added, annotated; all template values re-checked against `config.py` defaults.

Deviations: no live `--force` run on driveripper (needs the Oct-18 sudo-staged installer re-staged — it lacks the flags — and would push agent 1.1.0 mid phase-15 rollout); command + expected diff + post-checks recorded in AGENTS/gap 10. `uv pip list --outdated` still advisory (fastapi 0.153→0.156), untouched as in tasks 02/04 — this phase moved no dependency.

Next pending phase: `.agents/phases/todo/18_nav_hamburger/`.
This commit is contained in:
ducoterra
2026-10-20 13:48:40 -04:00
parent 7e367a9014
commit b02c13f7d3
30 changed files with 1502 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+92 -35
View File
@@ -782,13 +782,18 @@ That scheme is the history API's key — treat it as a public interface.
**`ok`** and the `live` suite is **11 passed / 0 failed** (it was 9/2 on the missing token that
morning, and 8/3 on the ConnectError before that). The unit reads
`--host 10.1.0.126 --port 9120 --token-file /etc/dashboards-agent.token` (`640
root:dashboards-agent`, the mode `deploy/install-agent.sh` writes) with `User=dashboards-agent`,
and `getent group disk` → `disk:x:6:dashboards-agent`.
**The deployed unit has since diverged from the template in the three ways the SMART
block below records** (the `DevicePolicy`/`DeviceAllow` lines removed, `User=root`,
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`) — re-running `deploy/install-agent.sh`
on driveripper would clobber the working unit, because it rewrites the unit unconditionally
(gap 10).
root:dashboards-agent`, the mode `deploy/install-agent.sh` writes) and is still in the `disk`
group (`getent group disk` → `disk:x:6:dashboards-agent`) — but it does **not** run as that
user: gap 10's hand-fix set `User=root`, and re-read from the host on 2026-10-20 the installed
unit is `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` with no device filter.
**Those two extra lines are now template output, and the unit's rewrite is no longer
silent**: phase 17 took the
`DevicePolicy`/`DeviceAllow` lines out of the template (task 01), gave it `--as-root` and
`--caps` (task 02), and made a re-run whose generated unit differs from the installed one print
the diff and exit 1 unless `--force` was passed (task 03), so
`sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force`
reproduces it and a re-run that would quietly replace a hand-fixed unit is now a refusal with a
diff attached rather than a clobber (gap 10).
`/api/health` today reads **all nine sources `ok`**: `host:driveripper`, `host:deskwork`,
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
`llm:vllm`.
@@ -857,17 +862,19 @@ That scheme is the history API's key — treat it as a public interface.
`deskwork` grants `ops` neither the installer nor `visudo`, so its install went through root by
the operator. **`driveripper` does grant the installer** — `(root) NOPASSWD:
/home/ops/agent-install/deploy/install-agent.sh`, an Oct-18 snapshot of the script staged under
the operator's home (`sudo -n -l`, re-checked 2026-10-20) — and that grant is the trap, not the
shortcut: the installer rewrites the unit unconditionally, so running it there today replaces the
`User=root` + `CapabilityBoundingSet` unit that gap 10 resolved with this repo's
`User=dashboards-agent` + `DevicePolicy=strict` template and re-breaks all 16 SMART reads. The
live unit is not the one this repo writes (no `DevicePolicy`/`DeviceAllow` lines at all —
systemd's default `auto` — `PrivateDevices=false`, and the two named capabilities).
Worth knowing before the next install: the allow-list `install-agent.sh`
does write (`/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`) has **no `/dev/ng*n*` and no
`/dev/bsg/*`**, the nodes smartctl 7.5 reaches for NVMe admin and SATA pass-through, so a
`strict` unit there would deny exactly what a SMART read needs — whatever phase takes
driveripper's SMART on has to settle the node list and the grant together.
the operator's home (`sudo -n -l`, re-checked 2026-10-20) — and that grant is still the trap,
not the shortcut: it points at that Oct-18 **snapshot**, which carries neither this phase's
flags, nor the filter removal, nor the re-run guard, so running the granted path replaces the
`User=root` + `CapabilityBoundingSet` unit that gap 10 resolved with the snapshot's own
`User=dashboards-agent`, capability-less unit and re-breaks all 16 SMART reads — **silently**,
because the guard that refuses a differing re-run (task 03) lives only in this repo's script,
which therefore has to be re-staged before any of it means anything there. The template now writes the live unit's shape: no device filter at all
(phase 17 task 01 — systemd's default `auto`, with `PrivateDevices=false`) and `--as-root`
plus `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` for the `User=root` and
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` lines (task 02). The old allow-list's
missing nodes are no longer a concern for the next install: with no filter, `/dev/ng*n*` and
`/dev/bsg/*` are reachable the same way `/dev/nvme*` and `/dev/sd*` are, subject to DAC and
SELinux.
- **SMART: real captures, and one live host finally reads them.** `smartctl` 7.5 is installed on
deskwork and driveripper. The `ops` account (which is who runs a hand-started dev agent) is in
no `disk` group, driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and deskwork
@@ -953,7 +960,7 @@ set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no
./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health
./scripts/dev-server.sh logs
uv run pytest # hermetic: 1427 passed + 15 skipped (11 live,
uv run pytest # hermetic: 1451 passed + 15 skipped (11 live,
# 4 store_perf), no network
uv run pytest --cov # coverage gate (98.94%; floor 94%)
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
@@ -993,23 +1000,42 @@ bash .agents/validate.sh # the phase gate: the four checks abov
Deployment: `Containerfile` (two-stage, `uv sync --frozen`, runs as uid 10001 on `0.0.0.0:8090`,
SQLite under `/var/lib/dashboards` — put a volume there) and `deploy/install-agent.sh`
per monitored host (systemd unit, own user, `--bind`/`--port`/`--token`/`--rotate-token`/
`--cert-bundle`/`--status`/`--uninstall`; it stores the token `0640 root:dashboards-agent` — the
`--cert-bundle`/`--as-root`/`--caps`/`--force`/`--status`/`--uninstall`; it stores the token
`0640 root:dashboards-agent` — the
unit hands the agent the *path* and the agent opens it as its own user, so that user must be able
to read it and no other account may, and a `--cert-bundle` file is installed to
`/etc/dashboards-agent.certs` at that same mode after a shape check that runs before anything is
touched and never dials the dashboard (the agent re-reads the bundle live for the denylist, so
0600 would fail the unit exactly like it fails the token file; the task file's `0600` is the
intent — *root and the agent's own user only* — and 0640 is the mode that achieves it) — never
puts a token on argv, and runs under `ProtectSystem=strict` with
`DevicePolicy=strict` reopening only the disk nodes plus the `/dev/null` family, because strict
without `/dev/null` kills the unit at spawn (`status=208/STDIN`); a bundle-installed unit also
gets `--cert-bundle` plus `RuntimeDirectory=dashboards-agent` (mode 0700 — the bundle unpacks its
puts a token on argv, and runs under `ProtectSystem=strict` with **no device
filter** — SMART device access is DAC via the `disk` group + SELinux, because kernel 7.2 matches
a device-filter glob by name against `/proc/devices` (where NVMe is `blkext`), so the old
allow-list denied the whole namespace; `PrivateDevices=false` stays explicit because a private
`/dev` with no device list kills the unit at spawn (`status=208/STDIN`). Where DAC is not enough,
`--as-root` writes `User=root` and `--caps CAP1,CAP2` writes that `CapabilityBoundingSet` line
(space-separated, as systemd spells it): kernel 7.2's unprivileged-passthrough gate checks
capabilities rather than uid, so `--as-root` alone is capability-less (`CapEff 0`) and prints a
note saying so, while `--caps` without `--as-root` is refused with exit 1 before `need_root` and
before any write. Both default off, and neither touches the token file's mode or the `--status`
probe. A bundle-installed
unit also gets `--cert-bundle` plus `RuntimeDirectory=dashboards-agent` (mode 0700 — the bundle unpacks its
PEM blocks there and the agent deletes the key once the SSL context holds it), keeps the
`--token-file` line for a host-by-host rollout, and `--status` then probes `https` on the
bundle's CA with **no** client certificate via a `python3` `ssl` one-liner — task 02's
`CERT_OPTIONAL` + `/health` exemption is what makes that probeable, and there is no external TLS
binary. A re-run without the flag leaves an installed bundle in place, and `--uninstall` removes
it with the token. No root shell? The one sudoers
it with the token — and neither is guarded, because both are deliberate. What is guarded is the
unit itself (phase 17 task 03): it is rendered to a temp file first, and if an installed `$UNIT`
differs the run prints `refusing to overwrite …` plus the `diff -u` on stdout and exits 1 before
the unit write, the `daemon-reload` and the `enable`, so a redeploy cannot undo a hand-fix and an
identical re-run stays the silent zero-noise path. `--force` prints the same diff and applies it —
which is what the TLS rollout's `--cert-bundle` step needs on a token-mode host, because that step
*is* a unit change. What it applies is the **file**: `systemctl enable --now` only *starts* a unit,
so a run that rewrote `ExecStart` leaves the already-running agent on its old settings until
`systemctl restart dashboards-agent` — and `--status` chooses http from https by reading the unit
file, so probing before that restart reports "no answer" for a change that landed correctly (the
rollout in README restarts for exactly this reason). No root shell? The one sudoers
line that lets `ops` run the installer is `deploy/agent-sudoers`; run the script without root and
it prints that line with its own resolved path.)
The quadlet unit + env template + nginx/SNI live in the deployment repo, like every other
@@ -1712,7 +1738,8 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
would be a re-decision of that phase, not a bug fix. Do not "fix" the residual by hiding the
short window or its label.
10. **`driveripper`'s 16 drives read `n/a`: kernel 7.2's unprivileged-passthrough gate
(2026-10-19, resolved the same day — the installer template is what is left).**
(2026-10-19, resolved the same day; the installer template that shipped the broken device
filter was cleared in phase 17).**
The agent is installed, the `disk` group grant is in place, the block nodes open, and the
page renders all 16 real drive rows — but kernel 7.2.11 blocks the *agent user's* `smartctl`
at the command level (not the open; the probe matrix is in
@@ -1745,11 +1772,39 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
`test_the_pass_budget_floor_is_twenty_seconds`,
`test_the_pass_budget_error_line_names_the_scaled_figure` — which demands the line read `80s`
for 16 disks, not the floor; the override test stays). Re-read live on 2026-10-20: the installed
agent answers 16/16 drives with temperatures and no `pass budget … exhausted` line. The
`deploy/install-agent.sh` template still ships the `DevicePolicy`/`DeviceAllow` lines that
misbehaved on this host
(a deny-all filter; the `nvme*` glob matches nothing on this kernel because
`/proc/devices` lists NVMe as `blkext`) — the template change is its own phase.
agent answers 16/16 drives with temperatures and no `pass budget … exhausted` line.
The `deploy/install-agent.sh` template no longer ships the device-filter lines that
misbehaved on this host — phase 17 task 01 removed them (the `nvme*` glob matches nothing
on this kernel because `/proc/devices` lists NVMe as `blkext`, and a strict device policy
denies everything a list fails to match), and the unit pins in `tests/test_deploy_scripts.py`
now assert their absence. Task 02 added the two flags that reproduce this host's remaining
lines — `--as-root` → `User=root`, `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` →
`CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, both off by default, `--caps` refused
without `--as-root`, `--as-root` alone accepted with the CapEff-0 note — pinned by
`tests/test_deploy_scripts.py` (exact `User=` / `CapabilityBoundingSet=` lines in a fake-root
sandbox, the refusal before any write, the `--help` recipe). Task 03 removed the unconditional
rewrite: the unit is rendered to a temp file, and a re-run whose render differs from the
installed unit prints the diff on stdout and exits 1 unless `--force` is passed, so the flag
line above now needs `--force` to land and a routine re-run can no longer clobber this host's
unit (pinned by the refuse/force/silent sequence in `tests/test_deploy_scripts.py`, including
the `--force` leg of this host's own recipe — the same
`--bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` line the
fleet record above prints). Task 04 verified the reproduction **read-only** rather
than by running it: the unit installed on `driveripper` (`systemctl cat`, 2026-10-20) differs
from what this template renders with that exact flag line in **comment lines only** — every
directive matches, `User=root`, `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`, no device
filter, `PrivateDevices=false`, the same `ExecStart` on the token file — and the host still
answers `active` with `CapEff: 0000000000220000` and 16/16 drives with temperatures in
`/snapshot`. So the operator's `--force` run lands the new rationale prose and changes nothing
systemd acts on (the three captures and the exact probe list are in
`data/logs/NOTES-17-installer-repro.md`; no live change was made while pinning this phase). Two facts that came out of reading the live host, both load-bearing for the
next install: the sudo-granted `/home/ops/agent-install` snapshot is the Oct-18 file (no
`--as-root`, no `--caps`, no `--force`, no `--cert-bundle`, still nine `DeviceAllow` lines) and
it **refuses** the recipe with `unknown argument: --as-root` and exit 2 before `need_root`, so
that path cannot apply this recipe at all until the repo's script is re-staged — while a
re-run of it *without* the flags is still the silent clobber the guard exists for. (The other
fact from that read — an applied change reaches the file but not the running process until
something restarts it — is in the Deployment paragraph, and README's rollout now restarts.)
`tests/fixtures/smartctl_driveripper_*.txt` (captured via `sudo`) remain the ground truth
for the drives' shape.
11. **The CA is durable, the collector dials with it, and no bundle is installed with one**
@@ -1814,8 +1869,9 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
place (`data/logs/verify-15-tls-e2e.py`, transcript `verify-15-tls-e2e.out`). What is left of
the phase is the live rollout (07): the fleet's agents still run token mode, so `HOST_TARGETS`
stays `http://` here until somebody installs a bundle on a host — which needs root on that host
(`ops` has no installer grant on `deskwork`, and `driveripper`'s unit must not be rewritten, gap
10), and flipping the URL before the bundle is installed turns that host red by construction.
(`ops` has no installer grant on `deskwork`, and `driveripper`'s unit is reproducible only with
phase 17's `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` line, gap 10), and flipping
the URL before the bundle is installed turns that host red by construction.
Re-checked from the dev box the same day, and worth knowing before that rollout: all three
agents answer `/health` with `version` **1.0.0** (`agent_version` is the `/snapshot` spelling),
so none of them carries task 02's TLS code either — the rollout ships the repo's
@@ -1824,8 +1880,9 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
exactly that TLS block; its `_smart_pass` and the phase-16 budget scale-up are already there.
Copying 1.1.0 into place is not an `install-agent.sh` re-run on that host (gap 10) — even
though `ops` there is sudo-granted a staged copy of the installer (see the fleet record above),
which is the one route that must not be taken until phase 17 fixes the unit template — and no
live change was made while pinning phase 16.
which is the one route that must not be taken until that snapshot is re-staged from this repo:
the Oct-18 copy predates phase 17's template *and* its re-run guard, so it would rewrite that
unit silently — and no live change was made while pinning phase 16.
## Locked decisions (and why)