Files
dashboards/.agents/phases/complete/15_agent_mtls/07_live_rollout.md
T
ducoterra cffbac5771 phase: 15_agent_mtls
All green. Report:

**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.

**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.

**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.

**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.

**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
2026-10-20 10:02:16 -04:00

3.6 KiB

Task 07 — The fleet runs TLS: install, flip, verify, document

Phase: 15_agent_mtls · Source: TODO.md:3 — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."

Objective

The two live agents (deskwork, gateway) present the dashboard-issued server cert and require the dashboard's client cert; the dashboard dials https://; the health table, the live suite, and a /certs screenshot prove it — and the still-missing driveripper install is documented to carry a bundle from day one.

Work

  1. Fleet (operator root actions, the deploy/agent-sudoers grant): for deskwork and gateway: GET /api/certs/<name>/bundle (a signed-in curl), scp the bundle to the host, sudo ./deploy/install-agent.sh --cert-bundle <file>, systemctl restart dashboards-agent, ./deploy/install-agent.sh --status green over TLS. Record the exact commands + output in data/logs/NOTES-agent-install.md (an append to the file's existing per-host record; local evidence, not tracked, the AGENTS.md rule).
  2. The flip: .env.dev — the HOST_TARGETS entries become https://<same address>:9120 (no userinfo — the cert is the auth; the token goes away for a TLS host), ./scripts/dev-server.sh restart, ./scripts/dev-server.sh status shows host:deskwork ok, host:gateway ok, disks:deskwork ok, disks:gateway ok (the two driveripper jobs stay DOWN — unchanged, the designed degradation).
  3. Proof (invariant 10): uv run pytest -m live --live — the host/disks tests over the https path; a /certs screenshot (the table showing both rows active, issued rev == deployed rev) and a /system screenshot (the host rows not dimmed) in data/logs/; a data/logs/NOTES-verify.md-style note: the curl codes (unsigned /api/certs refused, signed-in 200), one line for a wrong-CA client's handshake failure (the redacted URL only).
  4. The driveripper procedure: an append to data/logs/NOTES-agent-install.md — when the agent is finally installed (the outstanding operator action, the AGENTS.md fleet section), install the bundle in the same invocation (sudo ./deploy/install-agent.sh --bind … --cert-bundle …) and point HOST_TARGETS at https:// from day one; a token-only install after this phase is a host the https-dialing dashboard cannot reach.
  5. Docs (the AGENTS.md definition of done): the AGENTS.md fleet section (the transport fact: the agent is now mTLS, the bundle file, the /certs page) + the commands section (the bundle-fetch line); a short "Protecting the agent" section in README.md (the operator's how-to: fetch the bundle, install, flip the URL — three sentences, no flavour).

Testing & Quality

  • the live suite is read-only (the repo's rule) — no bundle mutations on the hardware; the revocation/rotation proof happens on a dev-box agent, not the fleet's.
  • invariant 10: the dev server stays up throughout; screenshots in data/logs/.

Completion Criteria

  • ./scripts/dev-server.sh status: host:deskwork ok, host:gateway ok, disks:deskwork ok, disks:gateway ok over the https:// dials
  • uv run pytest -m live --live green for the host/disks paths
  • the /certs + /system screenshots in data/logs/ show active, issued rev == deployed rev, the host rows not dimmed
  • the data/logs/ note records the exact commands + the curl codes + the wrong-CA failure line
  • AGENTS.md + README.md updated (the definition of done)
  • bash .agents/validate.sh green