phase: 14_store_performance
**Phase 14 `store_performance` — final verification pass (all 4 tasks already in `complete/`)** - Re-verified every task's work in the tree: read caches (`series_names`/`stats`), capped compound `pick_res_many`, `history_many` + `api._history_batch`, windowed `maintain()` + per-resolution prune + `PRAGMA optimize`/`automatic_analyze=0`/WAL pragmas, `store_perf.BUDGETS` (14 ceilings), probe, live watch. - **Completion criteria:** baseline+after side by side in `data/logs/NOTES-store-perf.md` ✅ · budget tests green (`uv run pytest -m store_perf --store-perf` → **4 passed**, 14/14 inside) ✅ · 25-min dev-server watch re-run **independently** (74 passes, `scripts/watch-store-perf.py`): **0 `store-error`** (total still 190, newest 2026‑10‑18 06:33:30), WAL 7.77 MB ≤ 67.1 MB, medians 1.7–3.2 ms, p95 720/613 ms = the documented cold-cache window; a 3rd 27-pass watch on a freshly restarted server reproduced it (3.1/708, 1.8/572) ✅ · `maintain()` 5,368 ms → 2.7–6.8 ms and WAL bounded across two cycles (test + notes) ✅ · no behaviour change (all four history routes pinned against `query()`) ✅ - **Full gates:** `uv run pytest` → **1267 passed, 15 skipped**; `bash .agents/validate.sh` → **exit 0**, coverage **98.75%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` ok, repo hygiene ok; `uv run pytest -m live --live` → **11 passed**; dev server up, 9/9 sources `ok`, pages 200 in 1.5–3.3 ms. - **Docs fixed:** README/AGENTS still claimed a flat "20 s per pass" while `host_agent.py` (in this same uncommitted change) scales the budget to 5 s × disk count — corrected in both plus the two illustrative "20 s of wedged disk" sentences; broken bold (`` `driveripper** ``) repaired; stale gap-4 title + "tuning is its own phase" wording updated to say phase 14 ran; README's absolute "nothing … is answered by a whole-table scan" qualified to *per call* with the cold-window cost named; "third opt-in marker" → second; conftest `--store-perf` help named the real marker; **`data/logs/watch-store-perf.py` moved to `scripts/watch-store-perf.py`** (both docs told a clone to run a gitignored path) with root-anchored paths, `DASH_WATCH_OUT`, and dead code removed. - **New finding recorded:** gap 9's "~10 s process-wide stall is still gap 4's / `14_store_performance`'s problem" is now closed — coincident >8 s `llm` + >12 s `ups` raw-stream gaps went 393 (00:00–12:00) and 198 (12:00–21:36) to **0** across 110+ min of the phase-14 store; written into AGENTS.md gaps 4 and 9 and the notes table. - **Deviations:** did *not* pull either deferred cost knob (`STATS_CACHE_TTL_SECONDS`, warming caches from the maintenance tick) — AGENTS.md records that decision and its reasoning; `uv pip list --outdated` still reports certifi/fastapi/starlette/websockets (pre-existing advisory, no dependency touched this phase, gate treats it as advisory). Overwrote nothing: task 04's watch preserved as `store-perf-watch.task04.*`, my two runs as `.verify25.*` / `.current-code.*`. - **Next pending phase:** `15_agent_mtls`, then `16_agent_smart_pass_budget` (owns the scaled-budget tests + its doc criterion, now partly pre-satisfied by this pass), `17_installer_device_filter`.
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
# Phase 16 — Pin the agent's scaled smart-pass budget
|
||||
|
||||
**Source:** chat 2026-10-19 — the `driveripper` 16-drive host: the flat 20 s smart-pass
|
||||
budget truncates a 16-device pass (a real read takes ~30–80 s), and because the deadline
|
||||
resets each pass, the tail drives would have stayed `smart: null` every pass. The fix —
|
||||
budget = 20 s floor scaled to 5 s × disk count, an explicit override still winning — was
|
||||
applied to the working tree on 2026-10-19 (operator-confirmed, verified on 16/16 live
|
||||
drives). This phase pins that already-applied change with tests and documentation.
|
||||
**Story:** n/a
|
||||
**Context:** `agent/host_agent.py` (`_smart_pass` — the scaled `budget = max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names)) when the default is in force, else the explicit override; the `pass budget of {budget:.0f}s exhausted` error line; the module docstring "5 s per device and a pass budget of 20 s scaled to the disk count"), `tests/test_agent.py::test_the_pass_budget_skips_the_rest_with_one_error` (sets `collector.smart_budget_seconds = 0.0` — the override-wins branch), `AGENTS.md` line 185 ("5 s per device / 20 s per pass / 240 s cache") and the `deploy/install-agent.sh` header comment (line 25 area, "bounded 5 s per device and 20 s per pass" if still present there), `README.md` line 94 ("20 s per pass, results cached 4 minutes")
|
||||
|
||||
## Objective
|
||||
|
||||
The smart pass budget scale-up that is already in the working tree is pinned: a test proves the *default* scales (16 devices → 80 s deadline, one device → 20 s floor, the exhaustion line names the scaled figure), the override-wins branch stays covered, and every doc that still says "20 s per pass" says the floor-scaled truth. No new behaviour — the code change is already deployed on `driveripper` and verified live.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `09_disks_smartctl` (complete) — the smart pass, its bounds, and `tests/test_agent.py`'s in-process agent harness
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_pin_the_scaled_budget.md` — the tests that the default budget scales with the disk count and that the error line names the scaled figure
|
||||
2. `02_docs_and_gate.md` — every "20 s per pass" mention in `AGENTS.md`/`README.md`/`agent/host_agent.py` says the floor-scaled truth; the gate is green
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/test_agent.py` — the scaled default (a default-attribute collector over 16 device names exhausts only when the 80 s budget runs out; over one device the floor is 20 s; the exhaustion error line carries the scaled figure, not the floor)
|
||||
- Coverage: **>90%** on the modified code (the `_smart_pass` branch)
|
||||
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the smart-pass bounds are a documented contract of `host_agent.py`).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_agent.py` green, including the new scaled-budget tests and the untouched override test
|
||||
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing (the wording is floor-scaled everywhere)
|
||||
- [ ] `bash .agents/validate.sh` green (pytest --cov --cov-fail-under=94, ruff, pyright, node --check)
|
||||
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
|
||||
- [ ] no behavior change in completed work: the deployed agent on `driveripper` is byte-identical to the repo's `agent/host_agent.py` (md5), the live suite is green
|
||||
@@ -0,0 +1,26 @@
|
||||
# Task 01 — Pin the scaled smart-pass budget
|
||||
|
||||
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The default smart-pass budget (20 s floor, scaled to 5 s × disk count) is proven by tests: the deadline actually moves with the disk count, the exhaustion error line names the *scaled* figure, and the existing override-wins branch (a test sets `smart_budget_seconds = 0.0`) keeps passing untouched. The code change is already in `agent/host_agent.py::_smart_pass` — this task adds the tests, it does not rewrite the code.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_agent.py` — add a helper that builds the in-process collector with **default** smart attributes (do not touch `smart_budget_seconds`/`smart_timeout_seconds`) and a `smartctl` fake that records each device's deadline window, in the same style as `test_the_pass_budget_skips_the_rest_with_one_error` (which stays untouched):
|
||||
- `test_the_pass_budget_scales_with_the_disk_count`: 16 device names → the last device's `smartctl` run happens with `left` (its deadline margin) consistent with a ~80 s budget (assert the deadline the code computes: `max(20, 5 × 16)`; the fake binary is instant, so assert via the recorded `min(left, 5.0)` timeout being 5.0 for a late device and the pass *not* truncating — every device got a `smartctl` run); 1 device name → budget 20 s (assert the recorded timeout is 5.0 and, with the fake made slow enough to exhaust a 20 s but not an 80 s budget in a *separate* case if feasible — if the harness cannot fake wall time, assert the computed `budget` expression via the error-line route instead: force exhaustion with a `time.monotonic` monkeypatch offset).
|
||||
- `test_the_pass_budget_error_line_names_the_scaled_figure`: force exhaustion on 16 devices (monkeypatch the clock forward past the 80 s deadline between two device runs, as the existing test's `0.0` override does for its 0 s budget) → the error line reads `smart: pass budget of 80s exhausted, N disk(s) skipped` — the scaled figure, not the 20 s floor.
|
||||
2. `agent/host_agent.py` — **no code change expected.** If a test exposes a real bug in the already-applied change (e.g. the `== SMARTCTL_BUDGET_SECONDS` default check), fix it minimally and record the diff in the task's completion note; do not redesign.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: the two new tests above + the untouched `test_the_pass_budget_skips_the_rest_with_one_error` (override wins: `0.0` still exhausts immediately on 3 devices with one error line)
|
||||
- Coverage: **>90%** on `agent/host_agent.py::_smart_pass`'s budget branch (the default-scaled path must now be exercised, not just the override)
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_agent.py -q` green: the two new tests, the untouched override test, and every pre-existing agent test
|
||||
- [ ] the 80 s case asserts the *scaled* figure in the error line (a test that passed with a 20 s budget would fail on the scaled code — no tautological assertions)
|
||||
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: `agent/host_agent.py` unchanged unless a real bug was found (then the fix is minimal and recorded)
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 02 — Docs and gate for the scaled budget
|
||||
|
||||
**Phase:** `16_agent_smart_pass_budget` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
Every place that still says the smart pass is "20 s per pass" says the floor-scaled truth (20 s floor, 5 s × disk count), and the phase gate is green. The agent's module docstring was already updated on 2026-10-19 to the scaled wording — this task verifies it and fixes the two remaining doc lines.
|
||||
|
||||
## Work
|
||||
|
||||
1. `AGENTS.md` line ~185 (the `host_agent.py` bullet): `5 s per device / 20 s per pass / 240 s cache` → `5 s per device / a pass budget of 20 s scaled to 5 s × disk count / 240 s cache` (keep the bullet's voice; one line).
|
||||
2. `README.md` line ~94: `20 s per pass, results cached 4 minutes` → `a pass budget of 20 s scaled to the disk count (5 s each), results cached 4 minutes` (one line).
|
||||
3. `agent/host_agent.py` module docstring — verify it reads `bounded 5 s per device and a pass budget of 20 s scaled to the disk count, cached 240 s`; fix only if it drifted.
|
||||
4. `bash .agents/validate.sh` — the four checks (pytest --cov --cov-fail-under=94, ruff, pyright, node --check) + repo hygiene, all `uv run`.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- No new logic in this task; the gate run is the check.
|
||||
- Coverage: **>90%** gate holds (task 01's tests carry the new branch).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -rn "20 s per pass" AGENTS.md README.md agent/ deploy/` returns nothing
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
- [ ] the two edited doc lines read the floor-scaled truth and nothing else changed in either file
|
||||
@@ -0,0 +1,59 @@
|
||||
# Phase 17 — The installer's broken device filter and the kernel-gate variant
|
||||
|
||||
**Source:** chat 2026-10-19 — the `driveripper` install: the template's
|
||||
`DevicePolicy=strict` + `DeviceAllow` lines produced a **deny-all device filter** on
|
||||
kernel 7.2 (systemd resolves `DeviceAllow` globs by name against `/proc/devices`, where
|
||||
this kernel lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and the whole
|
||||
device namespace was locked). The operator removed the lines by hand; the unit now also
|
||||
runs `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` (kernel 7.2's
|
||||
unprivileged-passthrough gate — `data/logs/NOTES-driveripper-agent.md`). The template
|
||||
still ships the broken lines, `User=$AGENT_USER`, and an empty bounding set, and re-running
|
||||
the installer on `driveripper` would **clobber the working unit** (unconditional
|
||||
`cat > "$UNIT"`). This phase makes the template the single source of the working unit.
|
||||
**Story:** n/a
|
||||
**Context:** `deploy/install-agent.sh` (`do_install` — the unconditional unit heredoc with `DevicePolicy=strict` + nine `DeviceAllow` lines, the `PrivateDevices=false` rationale comment block, `User=$AGENT_USER`, `CapabilityBoundingSet=`, the `--bind/--port/--token/--rotate-token/--status/--uninstall` flag set), `tests/test_deploy_scripts.py` (the fake-root install sandbox `test_install_path_installs_into_its_own_paths_and_prints_the_token_once`, the token-file permission test, the `bash -n` + function-naming pins), `AGENTS.md` (the fleet bullet "The deployed unit has since diverged from the template in the three ways the SMART block below records", gap 10's "the template change is its own phase"), `README.md` (the install section, line ~46 `sudo deploy/install-agent.sh --bind <addr> --port 9120`)
|
||||
|
||||
## Objective
|
||||
|
||||
Re-running `deploy/install-agent.sh` on any host — including `driveripper`, with
|
||||
`--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — produces the working unit: no
|
||||
`DevicePolicy`/`DeviceAllow` lines (device access falls back to DAC via the `disk` group
|
||||
+ SELinux, the standard SMART-monitoring arrangement), the kernel-gate variant expressible
|
||||
by flags, and a re-run that would change an existing unit prints the diff and refuses
|
||||
without `--force`. The default install (dedicated user, empty bounding set) is otherwise
|
||||
byte-identical to what the template wrote before this phase.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `09_disks_smartctl` (complete) — the smart pass the unit exists to serve
|
||||
- `15_agent_mtls` (todo) — **shared-file, run-order note**: 15's task 06 adds `--cert-bundle`/`RuntimeDirectory` to the same `do_install`; apply on top of the file state at the time it runs, never restore a line the other phase owns. If 15 lands first, its new lines survive this phase's filter-line removal untouched.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_drop_the_broken_device_filter.md` — remove `DevicePolicy=strict` + all nine `DeviceAllow` lines and rewrite the rationale comment block
|
||||
2. `02_root_and_caps_flags.md` — `--as-root` and `--caps CAP1,CAP2` write the kernel-gate unit lines
|
||||
3. `03_no_silent_clobber.md` — a re-run that would change an existing unit prints the diff and refuses without `--force`
|
||||
4. `04_tests_docs_gate.md` — the unit pins in `tests/test_deploy_scripts.py`, `AGENTS.md`/`README.md` updated, gate green
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: `tests/test_deploy_scripts.py` — the template unit carries **no** `DevicePolicy`/`DeviceAllow` line; a default-install sandbox unit is byte-identical to the pre-phase template unit minus the filter lines; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` writes `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; the re-run guard refuses (exit 1, diff on stdout) and overwrites with `--force`; `bash -n` and the existing function-naming pin still pass
|
||||
- Live (task 04, operator-verified): `driveripper`'s deployed unit is reproduced by `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` — `systemctl diff` empty, `CapEff` still `0000000000220000`, 16/16 drives still live; `deskwork`/`gateway` untouched (their units are not re-run)
|
||||
- Coverage: the script itself is pinned by text assertions + `bash -n` (house pattern; >90% applies to the new Python test code)
|
||||
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's)
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the new flags are operator-facing; gap 10's "the template change is its own phase" becomes "done in phase 17" with the `driveripper` flag line).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing (the `PrivateDevices=false` line stays)
|
||||
- [ ] `sudo deploy/install-agent.sh --help` names `--as-root` and `--caps` with the kernel-gate sentence
|
||||
- [ ] a fresh sandbox install (fake root, no flags) produces a unit byte-identical to the pre-phase template unit minus the filter lines (pinned in `tests/test_deploy_scripts.py`)
|
||||
- [ ] a re-run that would change an existing unit exits 1 with the diff on stdout; with `--force` it overwrites
|
||||
- [ ] operator-verified on `driveripper`: the flags reproduce the working unit (`systemctl diff` empty, 16/16 drives live after restart)
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass)
|
||||
- [ ] no behavior change in completed work: the default install's unit (minus the filter lines) and every existing flag behave as before; the live suite is green
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 01 — Drop the broken device filter
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The template unit no longer carries `DevicePolicy=strict` or any `DeviceAllow` line, and the comment block that justified them is rewritten to say why the *absence* is the fix. The `PrivateDevices=false` line stays (explicit, default behaviour, and the rest of the lockdown is unchanged).
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` — in the `do_install` unit heredoc: delete the `DevicePolicy=strict` line and all nine `DeviceAllow=…` lines (`/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom`, `/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`).
|
||||
2. `deploy/install-agent.sh` — rewrite the comment block above `PrivateDevices=false` (the "The pseudo nodes are not decoration…" paragraph) to the new truth: **no device filter on purpose** — systemd resolves `DeviceAllow` globs by name against `/proc/devices` (fnmatch in `bpf_devices_allow_list_major`), where kernel 7.2 lists NVMe as `blkext` (major 259), so `nvme*` matched nothing and `DevicePolicy=strict` denied the *whole* device namespace (the `driveripper` install went deny-all, root included; `data/logs/NOTES-driveripper-agent.md`). Without the policy, device access falls back to DAC (the `disk` group the script already grants) + SELinux — the standard SMART-monitoring arrangement. Keep the one operational fact that survives: `PrivateDevices=false` is the default (a `true` sandbox with no `DeviceAllow` list is what dies at spawn with `status=208/STDIN` — keep that warning sentence, it is still why the line is written explicitly).
|
||||
3. `deploy/install-agent.sh` — the script's top-of-file header comment (line ~25: "the unit device-allows the disk nodes") → "the unit applies no device filter (kernel 7.2's `/proc/devices` name matching made the old `nvme*` glob deny-all); device access is DAC via the `disk` group + SELinux".
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: `bash -n deploy/install-agent.sh` clean (the existing pin covers it); a quick sandbox run (the `test_install_path_installs_into_its_own_paths_and_prints_the_token_once` pattern, run manually) shows no `DevicePolicy`/`DeviceAllow` in the written unit — the formal pins are task 04, this task keeps the change small and reviewable.
|
||||
- Coverage: **>90%** n/a (shell script; the pins are text assertions per the house pattern).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
|
||||
- [ ] `bash -n deploy/install-agent.sh` clean
|
||||
- [ ] a manual sandbox install (fake root, `--bind 127.0.0.1 --port 9199`) writes a unit with the filter lines gone, `PrivateDevices=false` present, and every other line byte-identical to the pre-phase template unit
|
||||
- [ ] the rewritten comment names the `/proc/devices`/`blkext` mechanism in one sentence (an executor reading only the script can understand *why* there is no filter)
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 02 — `--as-root` and `--caps` flags
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The kernel-gate variant of the unit (what `driveripper` runs) is expressible from the template: `--as-root` writes `User=root` instead of `User=$AGENT_USER`, and `--caps CAP1,CAP2` (comma-separated, e.g. `--caps CAP_SYS_RAWIO,CAP_SYS_ADMIN`) writes `CapabilityBoundingSet=CAP1 CAP2` instead of the empty `CapabilityBoundingSet=` line. Both default off; a flag without its friend is refused with a message.
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` — flag parsing: add `--as-root` (sets `AS_ROOT=1`) and `--caps <list>` (sets `CAPS="<list>"`, validates non-empty, rejects an empty list with exit 1) in the existing `case`/flag block, and `--help` text:
|
||||
- `--as-root run the service as root (for a kernel-gated host: kernel 7.2 gates unprivileged smartctl passthrough behind capabilities, not uid — a root process under the default empty bounding set is capability-less, so this flag needs --caps to mean anything)`
|
||||
- `--caps LIST CapabilityBoundingSet line for the unit (comma-separated, e.g. CAP_SYS_RAWIO,CAP_SYS_ADMIN); default: empty (capability-less service)`
|
||||
2. `deploy/install-agent.sh` — validation: `--as-root` without `--caps` → print "note: --as-root with the default empty CapabilityBoundingSet leaves the process capability-less (the kernel gates check capabilities, not uid); pair it with --caps for a kernel-gated host" and **still proceed** (a capability-less root is a legitimate, if unusual, choice — do not hard-refuse; the note is the operator's reminder). `--caps` without `--as-root` → refuse, exit 1 ("--caps needs --as-root: a capability grant on the service user is not the kernel-gate fix").
|
||||
3. `deploy/install-agent.sh` — the unit heredoc: `User=$AGENT_USER` → `User=$( [ "$AS_ROOT" -eq 1 ] && echo root || echo "$AGENT_USER" )`; the empty `CapabilityBoundingSet=` line → the `CAPS` list converted comma→space when set. Add one comment line above: `# --caps: kernel 7.2 gates unprivileged smartctl passthrough (NVMe admin log: CAP_SYS_ADMIN; SG_IO allow-list: CAP_SYS_RAWIO); a root process needs them named, because an empty bounding set bounds even root to CapEff 0`.
|
||||
4. `deploy/install-agent.sh` — the `--status` probe is unchanged (it probes the agent over HTTP; the user the unit runs as is invisible to it). The token file stays `0640 root:$AGENT_USER` — with `--as-root` the group read is irrelevant (root reads it), so do **not** change the token-file logic.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: manual sandbox runs (fake root): default → `User=dashboards-agent` + empty `CapabilityBoundingSet=`; `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` + `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN`; `--caps` alone → exit 1 with the message; `--as-root` alone → proceeds with the note on stdout. (Formal pins: task 04.)
|
||||
- Coverage: **>90%** n/a (shell script; text pins per the house pattern).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `sudo deploy/install-agent.sh --help` names both flags with the kernel-gate sentence (a fresh clone's executor can learn the `driveripper` recipe from `--help` alone)
|
||||
- [ ] the two sandbox units match the exact lines above (byte-for-byte on the `User=` and `CapabilityBoundingSet=` lines)
|
||||
- [ ] `bash -n deploy/install-agent.sh` clean; every pre-existing flag (`--bind/--port/--token/--rotate-token/--status/--uninstall`) behaves as before in a sandbox run
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 03 — No silent clobber on re-run
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
A re-run that would change an already-installed unit stops: it prints the diff between the installed unit and the unit the script would write, and exits 1 without touching the file. `--force` proceeds with the overwrite. A re-run whose generated unit is identical to the installed one proceeds silently (idempotent re-runs stay the zero-noise path the script has always had).
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` — add a `--force` flag to the flag block (`FORCE=1`) and `--help` text: `--force overwrite an existing unit even if the generated unit differs (prints the diff first)`.
|
||||
2. `deploy/install-agent.sh` — in `do_install`, immediately before the `cat > "$UNIT"` heredoc write: build the would-be unit into a temp file (the heredoc already exists — render it once to `$(mktemp)`), and if `"$UNIT"` exists and `! cmp -s "$UNIT" "$tmp_unit"`:
|
||||
- if `FORCE=1`: print the diff (`diff -u "$UNIT" "$tmp_unit"` on stdout) and proceed with the overwrite;
|
||||
- else: print `refusing to overwrite $UNIT (generated unit differs); run with --force to apply:` + the same diff, and `exit 1` **before** the token-file write, the `systemctl daemon-reload`, and the `systemctl enable --now` — a refused run must leave the host exactly as it was (token file, unit, and running service all untouched; note the token-reuse logic runs earlier in `do_install` and only *reads* the existing token, so ordering the guard before the unit write is enough).
|
||||
- identical unit → no output, normal proceed.
|
||||
3. `deploy/install-agent.sh` — `--uninstall` is unchanged (it already removes the unit deliberately; the guard applies to `do_install` only).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: manual sandbox sequence (fake root): install → modify the installed unit by one line → re-run with the same flags → exit 1, diff on stdout, installed unit byte-identical to the modified version, service still the old one; re-run with `--force` → diff printed, unit overwritten; re-run again unchanged → silent, exit 0. (Formal pins: task 04.)
|
||||
- Coverage: **>90%** n/a (shell script; text/behaviour pins per the house pattern).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] the three-way sequence above (refuse → force → silent) holds in the sandbox, with the token file and the running service untouched by a refused run
|
||||
- [ ] `--force` prints the diff *before* overwriting (an operator sees exactly what changed)
|
||||
- [ ] `bash -n deploy/install-agent.sh` clean; a fresh install (no existing unit) is unaffected by the guard
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 04 — Pins, docs, gate
|
||||
|
||||
**Phase:** `17_installer_device_filter` · **Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The template's new shape is pinned in `tests/test_deploy_scripts.py` (no filter lines, the flags' exact lines, the clobber guard's refuse/force/silent sequence), `AGENTS.md` and `README.md` name the `driveripper` recipe, and the gate is green. The operator-verified `driveripper` reproduction (the completion criterion in `00_phase.md`) is the last check.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/test_deploy_scripts.py` — extend the existing sandbox pattern:
|
||||
- `test_the_template_unit_has_no_device_filter_lines` — read `deploy/install-agent.sh`'s unit heredoc region and assert no line starts with `DevicePolicy=` or `DeviceAllow=`, and `PrivateDevices=false` is still present (the filter's absence is the fix; a regression that re-adds `DeviceAllow=/dev/nvme*` is the deny-all returning).
|
||||
- `test_a_default_install_unit_is_the_pre_phase_unit_minus_the_filter_lines` — the sandbox-installed unit equals the known pre-phase template unit text minus the ten filter lines (pin the whole unit, so a future template line drift fails here).
|
||||
- `test_as_root_and_caps_write_the_kernel_gate_unit_lines` — `--as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN` → `User=root` and `CapabilityBoundingSet=CAP_SYS_RAWIO CAP_SYS_ADMIN` exactly; `--caps` alone → exit 1 with the `--as-root` message; `--as-root` alone → proceeds with the capability-less note on stdout.
|
||||
- `test_a_re_run_that_differs_refuses_without_force_and_applies_with_it` — the refuse/force/silent sequence (exit 1 + diff on stdout + installed unit untouched; `--force` overwrites with the diff printed; an unchanged re-run is silent).
|
||||
2. `AGENTS.md` — the fleet bullet's "re-running `deploy/install-agent.sh` on driveripper would clobber the working unit, because it rewrites the unit unconditionally (gap 10)" → the post-phase truth: the installer no longer writes the device-filter lines, and the `driveripper` unit is reproducible with `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` (a re-run that would change an existing unit refuses without `--force`). Update gap 10's "the template change is its own phase" → "done in phase 17" with the flag line.
|
||||
3. `README.md` — the install section: one line naming the flags' purpose (the kernel-gated-host recipe), keeping the how-to voice.
|
||||
4. `bash .agents/validate.sh` — all four checks green.
|
||||
5. **Operator-verified (live, read-only + one restart):** on `driveripper`, `sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force` → `systemctl diff dashboards-agent` empty, `systemctl is-active` `active`, `CapEff` still `0000000000220000`, and 16/16 drives still live in the next `/snapshot` (the restart is the only fleet action; `deskwork`/`gateway` are not re-run). Screenshot `data/logs/disks-17-installer-repro.png` per invariant 10.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Integration: the four new pins above + the pre-existing pins (sudoers, `bash -n`, function naming, token-file permissions) all green.
|
||||
- Coverage: **>90%** on the new test code; the 94% gate holds.
|
||||
- Live: the operator-verified reproduction (item 5) — the phase's `driveripper` criterion.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] the four new pins pass and every pre-existing `test_deploy_scripts.py` pin passes
|
||||
- [ ] `grep -n "DevicePolicy\|DeviceAllow" deploy/install-agent.sh` returns nothing
|
||||
- [ ] `AGENTS.md` fleet bullet + gap 10 and `README.md` name the `driveripper` flag recipe
|
||||
- [ ] the operator-verified reproduction: `systemctl diff` empty, 16/16 drives live, screenshot in `data/logs/`
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] the live suite (`set -a; . ./.env.dev; set +a; uv run pytest -m live --live`) is green
|
||||
Reference in New Issue
Block a user