dashboards/deploy/install-agent.sh

#!/usr/bin/env bash
# Install agent/host_agent.py as a systemd service on a monitored host.
#
#   sudo ./deploy/install-agent.sh                       # loopback only (read it locally)
#   sudo ./deploy/install-agent.sh --bind 10.0.0.5       # serve the LAN, token required
#   sudo ./deploy/install-agent.sh --cert-bundle agent.bundle   # serve TLS (phase 15)
#   sudo ./deploy/install-agent.sh --fetch-bundle https://dashboards.example.com \
#        --cert-name deskwork --session <signed-in token> --bind 10.1.0.51 --port 9120
#                                                        # registration fetches its own bundle
#   sudo ./deploy/install-agent.sh --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN
#                                                        # a kernel-gated host (see below)
#   sudo ./deploy/install-agent.sh --with-routectl      # also install the route-switch helper
#                                                        # and its two units (the gateway only)
#   ./deploy/install-agent.sh --status                   # unit state + /health, no root
#   sudo ./deploy/install-agent.sh --uninstall
#   sudo ./deploy/install-agent.sh --without-routectl   # remove helper + units, keep config and state
#   sudo ./deploy/install-agent.sh --force               # re-apply over a unit this would change
#
# No root shell? `deploy/agent-sudoers` is the single sudoers line that lets the fleet's
# unprivileged `ops` account run this script as root. Run it without root and it prints that
# exact line with this script's resolved path, then exits 1.
#
# The token is stored in a file only root and the agent's own service user can read, and is
# passed to the agent as --token-file, never on the command line: argv is world-readable
# through `ps`, and this token guards hostname, mounts, disk models, GPU model and process
# names. Re-running the installer reuses the existing token — use --rotate-token when the old
# one leaked.
#
# The certificate bundle (--cert-bundle <file>, fetched by the operator from the dashboard's
# GET /api/certs/<name>/bundle or the /certs page) is the TLS install: one PEM holding this
# host's server certificate and key, the dashboard's root CA, a `# BUNDLE-REV <n>` line and
# the revocation denylist. With a local file this script dials nothing at all. The bundle is
# validated before anything is touched — a bundle the agent could not load must stop the
# install with its reason, not leave a unit that exits at startup — and then installed to
# /etc/dashboards-agent.certs, the token-file idiom: the agent opens the bundle *itself* at
# startup and re-reads it when its stamp changes (that is how a re-deployed revocation list
# lands without a restart), so the mode must let the agent's own service user read it and no
# other account. The unit then runs https with --cert-bundle and a RuntimeDirectory for the
# PEM blocks the agent unpacks (it deletes the key once the SSL context holds it), and
# --status probes /health over TLS with the bundle's CA and no client certificate — the path
# the agent exempts for probes. Re-running without the flag leaves an installed bundle in
# place, the token-reuse idiom.
#
# --fetch-bundle is that same install with the download done for you, because registration is
# the moment a client should get its identity (TODO.md:3 — "distribute them to clients during
# registration so we never need non-mtls servers"). It pulls
# GET <url>/api/certs/<cert-name>/bundle and hands the bytes to the path above, so there is no
# curl-a-key-across-the-network step for an operator to forget; the shape check, the 0640
# install, the unit's TLS lines and the --status probe all then run on the fetched bytes
# unchanged.
#
#   --fetch-bundle URL   the dashboard's own base URL — this is the one dial this script makes
#   --cert-name NAME     the HOST_TARGETS name the certificate is issued for
#   --session TOKEN      the signed-in session token, a **fetch credential only**
#
# The three are all-or-nothing, and they are refused together with a local --cert-bundle: one
# install, one bundle source. The token is used for that one request and nothing else — never
# printed, never written to the token file, the unit, the bundle or a log, and unset as soon as
# the bundle is in hand (it is a bearer credential for a signed-in surface, so `ps` exposure
# for the seconds of the fetch is the accepted cost on a single-admin box; the file it buys is
# the artifact, not the credential). The fetch and the shape check both run before the root
# boundary, so a dashboard that answers 401 — or says nothing is issued for that name — stops
# the run with its own sentence and no partial install:
#
#   sudo ./deploy/install-agent.sh --bind <addr> --port 9120 \
#        --fetch-bundle https://dashboards.example.com --cert-name <HOST_TARGETS name> \
#        --session <signed-in session token>
#
# The agent is read-only by construction (it answers GET and writes nothing), runs as its own
# system user (unless --as-root says otherwise, below), and gets the usual systemd lockdown. It
# still needs to read /proc, /sys and statvfs(), which the hardening below leaves alone.
#
# Disk health (`smartctl -i -A -H`, the `/disks` page) is the one add-on that needs more
# than /proc and /sys: smartctl must open the disk device nodes. The installer therefore
# adds the agent user to the `disk` group when one exists, and the unit applies no device
# filter at all: kernel 7.2 matches a device-filter glob by name against /proc/devices, where
# NVMe is listed as `blkext`, so the old `/dev/nvme*` pattern denied the whole device
# namespace instead of opening four nodes. Device access is DAC via the `disk` group, with
# SELinux still in front of it — the standard arrangement for SMART monitoring, granted
# deliberately. Its absence is a documented degradation, never an install failure: no
# smartmontools (or no group) simply means every disk reports `smart: null` with an
# explanatory error entry.
#
# A kernel gate is a different grant, and --as-root / --caps are how the unit gets it. Kernel
# 7.2 gates smartctl's raw passthrough behind *capabilities*, not uid: an NVMe SMART-log read
# through the block node needs CAP_SYS_ADMIN unless the drive's firmware advertises the page,
# and SATA SMART behind a SAT layer is an ATA pass-through the unprivileged SG_IO allow-list
# does not include (CAP_SYS_RAWIO is what the kernel names for that). Those checks run against
# the process's capabilities, so a root service under the default empty CapabilityBoundingSet
# is capability-less (CapEff 0) and reads nothing — which is why the two flags belong together.
#
#   --as-root      run the service as root (for a kernel-gated host: kernel 7.2 gates
#                  unprivileged smartctl passthrough behind capabilities, not uid — a root
#                  process under the default empty bounding set is capability-less, so this
#                  flag needs --caps to mean anything)
#   --caps LIST    CapabilityBoundingSet line for the unit (comma-separated, e.g.
#                  CAP_SYS_RAWIO,CAP_SYS_ADMIN); default: empty (capability-less service)
#
# Both default off, because a host whose drives answer through the disk group needs neither.
# --caps without --as-root is refused (exit 1): a capability grant on the dedicated service
# user is not the kernel-gate fix. --as-root alone proceeds with a note, since a
# capability-less root is a legitimate if unusual choice. The recipe for the host this exists
# for — 16 drives, NVMe firmware without the SMART-log page and SATA behind SAT — is:
#
#   sudo ./deploy/install-agent.sh --bind <addr> --port 9120 \
#        --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN
#
# --with-routectl (gateway hosts only) installs the one narrow write the dashboard can make:
# the root route-switch helper as a socket-activated service. The grant is a file mode systemd
# enforces — the socket is 0600 and owned by the agent's own user, and the helper's peer check
# (--allow-uid) is the second look — not a sudoers line, and the agent's unit is not changed by
# it: an install with the flag renders the agent unit byte-identically to one without it. The
# helper edits exactly one route (a #off <name> marker the agent's parser reads), validates the
# file with the proxy's own checker before it means anything, restores the previous bytes on a
# checker failure, and reloads — never restarts. --without-routectl removes the helper and its
# two units, leaves the proxy configs and the helper's state directory alone, and prints the
# list of still-off routes first, because those are the operator's to switch back on.
#
# A re-run is the zero-noise path it has always been as long as the unit this script would
# write is the unit already on disk. When the two differ the run stops: the diff between the
# installed unit and the generated one is printed and the exit code is 1, before the unit is
# written, before daemon-reload and before the service is re-enabled — so a routine redeploy
# cannot reset a unit somebody fixed by hand. driveripper's root + CapabilityBoundingSet unit
# is exactly that hand-fixed case (and the flags above are what reproduce it), which is why
# the refusal, not the overwrite, is the default. --force applies the change, with the same
# diff printed first so the operator sees what they are committing to.
#
#   --force        overwrite an existing unit even if the generated unit differs (prints the
#                  diff first)
#
# --allow-writes (gateway hosts) adds the agent's own --allow-writes to the unit's
# ExecStart — the agent-side half of the write path; --with-routectl installs the
# helper it dials, and the two are independent (a host with the flag but without the
# helper answers a write with 503, not with a write). It is the *only* change this
# script makes to the agent's unit for the write path: a re-run diff that shows the
# security directives moving is a stop, not a --force.
set -euo pipefail

SERVICE="dashboards-agent"
AGENT_USER="dashboards-agent"
AGENT_DIR="/usr/local/lib/dashboards-agent"
AGENT_TARGET="$AGENT_DIR/host_agent.py"
TOKEN_FILE="/etc/dashboards-agent.token"
CERT_FILE="/etc/dashboards-agent.certs"
UNIT="/etc/systemd/system/$SERVICE.service"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
AGENT_SOURCE="$SCRIPT_DIR/../agent/host_agent.py"
# Absolute, symlink-resolved path of this script: the sudoers grant in deploy/agent-sudoers
# is matched on a path, so the non-root hint below prints the line that actually matches
# wherever the script happens to live (the canonical staging path is
# /home/ops/agent-install/deploy/install-agent.sh).
SCRIPT_PATH="$SCRIPT_DIR/$(basename "$0")"
if command -v realpath >/dev/null 2>&1; then
  SCRIPT_PATH="$(realpath "$0" 2>/dev/null || printf '%s' "$SCRIPT_PATH")"
fi
SUDOERS_SOURCE="$(dirname "$SCRIPT_PATH")/agent-sudoers"
# The route-switch helper (phase 33) and its two units. The helper is the only writer this
# phase adds: a root, socket-activated service the agent may talk to and nothing else. The
# grant is a file mode systemd enforces (the socket is 0600, owned by the agent's user), not
# a sudoers line, and the agent's own unit is not touched by installing it — the rendered
# unit is byte-identical with and without --with-routectl, and the re-run guard below applies
# to the helper and both units exactly as it does to the agent's.
ROUTECTL_SOURCE="$SCRIPT_DIR/dashboards-routectl.py"
ROUTECTL_SOCKET_UNIT_SRC="$SCRIPT_DIR/dashboards-routectl.socket"
ROUTECTL_SERVICE_UNIT_SRC="$SCRIPT_DIR/dashboards-routectl@.service"
ROUTECTL_TARGET="/usr/local/sbin/dashboards-routectl.py"
ROUTECTL_SOCKET_UNIT="/etc/systemd/system/dashboards-routectl.socket"
ROUTECTL_SERVICE_UNIT="/etc/systemd/system/dashboards-routectl@.service"
ROUTECTL_SOCKET_PATH="/run/dashboards-routectl.sock"
ROUTECTL_STATE_DIR="/var/lib/dashboards-routectl"

BIND="127.0.0.1"
PORT="9120"
TOKEN="${AGENT_TOKEN:-}"
ROTATE=0
BUNDLE_SRC=""
# Which flag the bytes in $BUNDLE_SRC came from, so a shape refusal names the flag the operator
# typed. Empty means `--cert-bundle` (the default inside `validate_bundle`).
BUNDLE_LABEL=""
# --fetch-bundle: the three parts of "fetch my bundle in this install run". All three or none
# (checked below, before any write), and never together with a local --cert-bundle.
FETCH_URL=""
CERT_NAME=""
SESSION_TOKEN=""
# The temp file the fetched bytes land in, and the trap that guarantees it is gone whatever the
# rest of the run does. Its name is never printed.
FETCH_TMP=""
# --as-root / --caps: the kernel-gate variant of the unit (phase 17 task 02). Both off means
# the default install — dedicated user, empty CapabilityBoundingSet — byte-for-byte what this
# script wrote before the flags existed.
AS_ROOT=0
CAPS=""
# --force: the re-run guard's opt-in (phase 17 task 03). 0 means a re-run whose generated unit
# differs from the installed one refuses after printing the diff.
FORCE=0
# --with-routectl / --without-routectl (phase 33): the route-switch helper and its two units,
# installed beside the agent and never inside its unit. Both off is the default: an install
# without the flag is byte-identical to one before the flag existed.
WITH_ROUTECTL=0
# --allow-writes (phase 33): appends the agent's own --allow-writes to the unit's ExecStart.
# It is the agent-side gate for POST /route; the unit's security directives do not move
# with it, which is the phase's central safety claim in one flag.
ALLOW_WRITES=0
MODE="install"

usage() {
  # the header comment *is* the help text, so it cannot drift from the flags
  awk 'NR > 1 { if (/^#/) { sub(/^# ?/, ""); print } else { exit } }' "$0"
  exit "${1:-0}"
}

while [ $# -gt 0 ]; do
  case "$1" in
    --bind) BIND="$2"; shift 2 ;;
    --port) PORT="$2"; shift 2 ;;
    --token) TOKEN="$2"; shift 2 ;;
    --rotate-token) ROTATE=1; shift ;;
    --cert-bundle) BUNDLE_SRC="$2"; shift 2 ;;
    --fetch-bundle) FETCH_URL="$2"; shift 2 ;;
    --cert-name) CERT_NAME="$2"; shift 2 ;;
    --session) SESSION_TOKEN="$2"; shift 2 ;;
    --as-root) AS_ROOT=1; shift ;;
    --caps)
      [ $# -ge 2 ] || {
        echo "--caps needs a capability list (e.g. --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN)" >&2
        exit 1
      }
      # An empty list is a typo, not "the default": the operator typed the flag, so refusing
      # is kinder than writing the capability-less unit they did not ask for.
      if [ -z "$(printf '%s' "$2" | tr -d '[:space:],')" ]; then
        echo "--caps needs at least one capability name (e.g. --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN)" >&2
        exit 1
      fi
      CAPS="$2"
      shift 2
      ;;
    --force) FORCE=1; shift ;;
    --allow-writes) ALLOW_WRITES=1; shift ;;
    --with-routectl) WITH_ROUTECTL=1; shift ;;
    --without-routectl) MODE="routectl-off"; shift ;;
    --status) MODE="status"; shift ;;
    --uninstall) MODE="uninstall"; shift ;;
    -h|--help) usage 0 ;;
    *) echo "unknown argument: $1" >&2; usage 2 ;;
  esac
done

# Refused here, before need_root and before any write, for the bundle validator's reason: a
# wrong flag combination is answered with its own sentence rather than sending the operator to
# add a sudoers line for a mistake in the flags. --caps on the dedicated service user is not
# the kernel-gate fix — the gate checks the process's capabilities, and a non-root user cannot
# hold them across exec — so the grant only means anything on a root service.
if [ -n "$CAPS" ] && [ "$AS_ROOT" -ne 1 ]; then
  echo "--caps needs --as-root: a capability grant on the service user is not the kernel-gate fix" >&2
  exit 1
fi

# The same reasoning for the fetch: a flag combination is a mistake about what to install, and
# it gets its own sentence before anything else happens. Three parts because a URL without a
# name is a route nobody can guess and a name without a token is a 401 the operator would have
# to diagnose — so all three or none. And `--fetch-bundle` beside `--cert-bundle` is two
# answers to "where is the bundle?", which is a contradiction rather than a preference.
FETCH_GIVEN=0
for _flag in "$FETCH_URL" "$CERT_NAME" "$SESSION_TOKEN"; do
  # an `if`, not `[ -n … ] && FETCH_GIVEN=…`: under `set -e` a failing test as the last command
  # of the loop body would end the run on the first unset flag instead of counting them.
  if [ -n "$_flag" ]; then
    FETCH_GIVEN=$((FETCH_GIVEN + 1))
  fi
done
if [ "$FETCH_GIVEN" -gt 0 ] && [ "$FETCH_GIVEN" -lt 3 ]; then
  echo "--fetch-bundle, --cert-name and --session go together (see --help)" >&2
  exit 1
fi
if [ "$FETCH_GIVEN" -eq 3 ] && [ -n "$BUNDLE_SRC" ]; then
  echo "--fetch-bundle and --cert-bundle are two bundle sources; pick one" >&2
  exit 1
fi
# The fetch belongs to an install. `--status` probes a port and `--uninstall` deletes files; a
# run of either would fetch a private key for nothing and leave it in a temp file.
if [ "$FETCH_GIVEN" -eq 3 ] && [ "$MODE" != "install" ]; then
  echo "--fetch-bundle needs an install (it means nothing to --status, --uninstall or --without-routectl)" >&2
  exit 1
fi
# One flag each direction, like --as-root/--caps: a run that installs the helper and removes
# it is a mistake about what to install, and it gets its own sentence before any write.
if [ "$WITH_ROUTECTL" -eq 1 ] && [ "$MODE" = "routectl-off" ]; then
  echo "--with-routectl and --without-routectl are one flag each direction; run them separately" >&2
  exit 1
fi
# --with-routectl installs files; --status probes a port and --uninstall deletes the agent.
if [ "$WITH_ROUTECTL" -eq 1 ] && [ "$MODE" != "install" ]; then
  echo "--with-routectl needs an install (it means nothing to --status or --uninstall)" >&2
  exit 1
fi
# --allow-writes rewrites the agent's unit; none of the non-install modes render a unit.
if [ "$ALLOW_WRITES" -eq 1 ] && [ "$MODE" != "install" ]; then
  echo "--allow-writes needs an install (it is an argument to the agent's unit, not to --status, --uninstall or --without-routectl)" >&2
  exit 1
fi

cleanup_fetch() {
  # The fetched bytes are somebody's private key. Registered as an EXIT trap when the fetch
  # starts, so no path — a refused shape, a refused unit re-run, a failing systemctl — leaves
  # them behind in /tmp. Runs before the trap is armed too, and does nothing then.
  if [ -n "$FETCH_TMP" ] && [ -e "$FETCH_TMP" ]; then
    rm -f "$FETCH_TMP"
  fi
}

need_root() {
  if [ "$(id -u)" -ne 0 ]; then
    # The dev workflow deploys over ssh as the unprivileged `ops` account, so "run as root"
    # is not an action the operator can take. Print the one sudoers line that makes this
    # script installable — the same line as deploy/agent-sudoers, with this script's own
    # resolved path — plus the two-stage install it documents, then exit 1 as before.
    cat >&2 <<HINT
$SCRIPT_PATH has to run as root: it writes $UNIT,
creates the $AGENT_USER user and its disk-group membership, and stores
the token in $TOKEN_FILE (and, with --cert-bundle, the certificate bundle in $CERT_FILE).

Add this single sudoers line on this host (it is the whole grant; see $SUDOERS_SOURCE):

ops ALL=(root) NOPASSWD: $SCRIPT_PATH

Install it the two-stage way every sudoers change here uses, as root — the command-scoped
ops sudo on these hosts does not include visudo, so this step is the operator's:

  sudo visudo -cf $SUDOERS_SOURCE
  sudo install -m 0440 -o root -g root $SUDOERS_SOURCE /etc/sudoers.d/dashboards-agent
  sudo visudo -cf /etc/sudoers.d/dashboards-agent

Then run this same command again:

  sudo $SCRIPT_PATH $*
HINT
    exit 1
  fi
}

status() {
  systemctl status "$SERVICE" --no-pager 2>&1 || true
  echo
  if [ -r "$UNIT" ] && grep -q -- '--cert-bundle' "$UNIT"; then
    # TLS install (phase 15): probe over https with the bundle's root CA and *no* client
    # certificate. The agent's CERT_OPTIONAL handshake plus the /health exemption is what
    # makes this possible — the installer holds no client identity, and a service that
    # cannot health-check is a dead install. The CA is the bundle's last certificate block
    # (the format is positional: agent cert, agent key, root CA). Chain-to-our-root is the
    # claim; the leaf names the dial addresses rather than loopback, so the name check is
    # off — forging the chain would still need the root key. python3 is the unit's own
    # interpreter: there is no external TLS binary to install on a monitored host.
    printf 'health: '
    python3 - "$BIND" "$PORT" "$CERT_FILE" <<'PY' || echo "(no answer)"
import re
import ssl
import sys
import urllib.request

host, port, bundle = sys.argv[1], sys.argv[2], sys.argv[3]
try:
    with open(bundle, encoding="utf-8", errors="replace") as handle:
        text = handle.read()
except OSError as exc:
    print(f"(cannot read {bundle}: {exc.strerror or exc} — run --status as root)")
    raise SystemExit(0)
blocks = re.findall(r"-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----", text, re.DOTALL)
if len(blocks) < 2:
    print(f"(no root CA block in {bundle})")
    raise SystemExit(0)
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.check_hostname = False
context.verify_mode = ssl.CERT_REQUIRED
context.load_verify_locations(cadata=blocks[-1])
try:
    with urllib.request.urlopen(
        f"https://{host}:{port}/health", timeout=4, context=context
    ) as response:
        print(response.read().decode("utf-8", "replace").strip())
except Exception as exc:  # the probe reports, it never propagates
    print(f"(no answer: {exc})")
PY
  elif command -v curl >/dev/null 2>&1; then
    # token mode: the plain probe, unchanged
    printf 'health: '
    curl -s --max-time 4 "http://$BIND:$PORT/health" || echo "(no answer)"
    echo
  fi
}

uninstall() {
  need_root --uninstall
  systemctl disable --now "$SERVICE" 2>/dev/null || true
  rm -f "$UNIT" "$TOKEN_FILE" "$CERT_FILE" "$AGENT_TARGET"
  rmdir "$AGENT_DIR" 2>/dev/null || true
  systemctl daemon-reload
  echo "removed $SERVICE (the $AGENT_USER user was left in place; remove it with userdel)"
}

routectl_off() {
  # The inverse of --with-routectl: the helper and its two units go; the proxy configs and
  # the helper's state directory stay — the state is the memory of what was switched, and
  # a route the helper left off is the operator's to switch back on, so the still-off list
  # prints first and the removal ends with the reminder, not the file list.
  need_root --without-routectl
  if [ -f "$ROUTECTL_TARGET" ]; then
    echo "routes the helper has switched off (these are still off after this removal — switch them back on):"
    "$ROUTECTL_TARGET" --state "$ROUTECTL_STATE_DIR/state.json" list || true
  else
    echo "the helper is not installed; nothing to list"
  fi
  systemctl disable --now dashboards-routectl.socket 2>/dev/null || true
  systemctl disable --now 'dashboards-routectl@.service' 2>/dev/null || true
  rm -f "$ROUTECTL_SOCKET_UNIT" "$ROUTECTL_SERVICE_UNIT" "$ROUTECTL_TARGET"
  systemctl daemon-reload
  echo "removed dashboards-routectl (the proxy configs and $ROUTECTL_STATE_DIR are left in place)"
  echo "reminder: a route still off in the list above is still off — switch it back on"
}

new_token() {
  # urlsafe and no padding: the token goes into HOST_TARGETS as URL userinfo
  # (`name=http://TOKEN@host:9120`), where '+' and '=' are annoying to quote.
  python3 -c 'import secrets; print(secrets.token_urlsafe(32).rstrip("="))'
}

validate_bundle() {
  # The `resolve_bundle` idiom from agent/host_agent.py, run before anything is touched:
  # a bundle the agent could not load must stop the install with its reason, not leave a
  # unit that exits at startup. Shape only, by text — the agent parses the same way (it is
  # stdlib; a monitored host has no cryptography) and the dashboard is the only signer, so
  # a second copy of this shape check would only ever disagree about nothing. The three PEM
  # blocks are positional: agent certificate, agent key, root CA certificate.
  #
  # $2 names the flag the bytes came from, so a refusal says what the operator typed rather than
  # a temp file they did not. One validator, two bundle sources: a local --cert-bundle file and
  # the fetched bytes of --fetch-bundle have to satisfy exactly the same shape, because the agent
  # cannot tell them apart either.
  if ! command -v python3 >/dev/null 2>&1; then
    echo "${2:---cert-bundle} needs python3 (3.10 or newer) to validate the bundle" >&2
    exit 2
  fi
  if python3 - "$1" "${2:---cert-bundle}" <<'PY' 1>&2
import re
import sys

path, flag = sys.argv[1], sys.argv[2]
try:
    with open(path, encoding="utf-8", errors="replace") as handle:
        text = handle.read()
except OSError as exc:
    print(f"{flag} {path} cannot be read: {exc.strerror or exc}")
    raise SystemExit(1)
if not text.strip():
    print(f"{flag} {path} is empty")
    raise SystemExit(1)
revision = None
for line in text.splitlines():
    stripped = line.strip()
    if stripped.startswith("# BUNDLE-REV "):
        fields = stripped[len("# BUNDLE-REV "):].split()
        if fields:
            try:
                revision = int(fields[0])
            except ValueError:
                revision = None
        break
if revision is None:
    print(f"{flag} {path} has no `# BUNDLE-REV <n>` line — that line is how the"
          " dashboard tells the bundle it issued from a stale one, so a bundle without it"
          " is not one this app signed")
    raise SystemExit(1)
blocks = re.findall(r"-----BEGIN [A-Z0-9 ]+-----.*?-----END [A-Z0-9 ]+-----", text, re.DOTALL)
certificates = [b for b in blocks if b.startswith("-----BEGIN CERTIFICATE-----")]
keys = [b for b in blocks if not b.startswith("-----BEGIN CERTIFICATE-----")]
if len(certificates) != 2 or len(keys) != 1:
    print(f"{flag} {path} must hold exactly the agent certificate, the agent key and"
          f" the root CA certificate — found {len(certificates)} certificates and"
          f" {len(keys)} keys. Re-download it from GET /api/certs/<name>/bundle")
    raise SystemExit(1)
PY
  then
    return 0
  fi
  exit 2
}

fetch_bundle() {
  # Registration is where a client gets its identity, so the install run pulls this host's
  # bundle from the dashboard instead of an operator carrying a file across the network by hand
  # (phase 24: "distribute them to clients during registration so we never need non-mtls
  # servers"). Two properties are load-bearing, and both are the validator's reasoning.
  #
  # * **Before `need_root`.** A dashboard that answers 401, or says nothing is issued for that
  #   name, or does not answer at all, stops the run here with its own sentence — no user, no
  #   files, no unit. A refusal is not a sudoers problem, and a half-install is worse than none.
  # * **Then the existing shape check.** The bytes land in a temp file and become `$BUNDLE_SRC`,
  #   so the PEM check, the `0640` install, the unit's TLS lines and the `--status` probe see a
  #   fetched bundle exactly as they see a downloaded one. One install path, two sources.
  #
  # python3 is already required (the shape check and the TLS probe both use it), so this adds no
  # dependency and no assumption that a monitored host has `curl`. The token reaches python in
  # the **environment**, never on its argv: argv is world-readable through `ps`, which is the
  # same reason the agent's own token is a file rather than a flag — a second copy of a bearer
  # credential on a command line would only widen what `ps` shows.
  if ! command -v python3 >/dev/null 2>&1; then
    echo "--fetch-bundle needs python3 (3.10 or newer) to fetch the bundle" >&2
    exit 2
  fi
  FETCH_TMP="$(mktemp "${TMPDIR:-/tmp}/dashboards-agent-bundle.XXXXXX")"
  trap cleanup_fetch EXIT
  if ! FETCH_SESSION="$SESSION_TOKEN" python3 - "$FETCH_URL" "$CERT_NAME" "$FETCH_TMP" <<'PY' 1>&2
import json
import os
import sys
import urllib.error
import urllib.parse
import urllib.request

base, name, destination = sys.argv[1].rstrip("/"), sys.argv[2], sys.argv[3]
token = os.environ.get("FETCH_SESSION", "")
route = f"/api/certs/{urllib.parse.quote(name, safe='')}/bundle"
request = urllib.request.Request(f"{base}{route}",
                                 headers={"authorization": f"Bearer {token}"})
try:
    with urllib.request.urlopen(request, timeout=20) as response:
        status, body = response.status, response.read()
except urllib.error.HTTPError as refused:
    # A refusal is the answer, and it carries the reason: this app's 404 body says whether the
    # name is configured and whether anything was issued, which are three different operator
    # actions (fix the URL, issue a certificate, re-sign in).
    status, body = refused.code, refused.read()
except (urllib.error.URLError, TimeoutError, OSError) as unreachable:
    reason = getattr(unreachable, "reason", None) or unreachable
    print(f"--fetch-bundle: no dashboard answered at {base}: {reason}")
    raise SystemExit(1)

try:
    parsed = json.loads(body.decode("utf-8", "replace"))
except ValueError:
    parsed = None
payload = parsed if isinstance(parsed, dict) else None


def detail() -> str:
    """The dashboard's own sentence, or the first bytes of whatever it sent instead.

    Every refusal this prints is text the app writes for an operator, and no response body of
    a *successful* fetch ever reaches the output — only the bytes land in the file.
    """
    if isinstance(payload, dict) and isinstance(payload.get("detail"), str) and payload["detail"]:
        return str(payload["detail"])
    text = body.decode("utf-8", "replace").strip().replace("\n", " ")
    return text[:160] or "no body"


if status == 200:
    if not body.strip():
        print(f"--fetch-bundle: GET {route} answered 200 with an empty body — nothing to install")
        raise SystemExit(1)
    with open(destination, "wb") as handle:
        handle.write(body)
    raise SystemExit(0)
if (status == 404 and isinstance(payload, dict)
        and payload.get("configured") is True and payload.get("issued") is False):
    # The invariant-3 shape: a configured host with no certificate yet. Its own sentence, because
    # the fix is a click on /certs and not a re-typed URL or a new token.
    print(f"--fetch-bundle: no bundle issued for {name} — issue it from /certs first"
          f" (POST /api/certs/{name}/rotate mints one; GET {route} answered 404)")
    raise SystemExit(1)
print(f"--fetch-bundle: GET {route} answered {status}: {detail()}")
raise SystemExit(1)
PY
  then
    exit 2
  fi
  # A fetch credential only: from here the install needs the bytes, not the session. The env
  # assignment above was per-command, so the only copy left is the shell variable.
  unset SESSION_TOKEN
  BUNDLE_SRC="$FETCH_TMP"
  BUNDLE_LABEL="--fetch-bundle"
  echo "  fetched the certificate bundle for $CERT_NAME from $FETCH_URL"
}

# NOT named `install`: bash resolves a function before an external command of the same name,
# so an `install()` here turns the `install -d` / `install -m` calls below into infinite
# recursion instead of creating $AGENT_DIR. Phase 10 task 02 found exactly that on a live
# host: the run looped on the SMART block forever and installed nothing.
do_install() {
  # Registration fetches its own bundle, before the root boundary for the same reason the shape
  # check is there: a dashboard that refuses is answered with its own sentence, and a run that
  # cannot get a bundle writes nothing at all. On success this is just another $BUNDLE_SRC.
  if [ -n "$FETCH_URL" ]; then
    fetch_bundle
  fi
  # Validate before need_root: a bad bundle is refused with its own reason whether or not
  # this shell is root, so a wrong file says "wrong file" and not "add a sudoers line".
  if [ -n "$BUNDLE_SRC" ]; then
    validate_bundle "$BUNDLE_SRC" "$BUNDLE_LABEL"
  fi
  need_root --bind "$BIND" --port "$PORT"
  [ -f "$AGENT_SOURCE" ] || { echo "no agent next to this script: $AGENT_SOURCE" >&2; exit 1; }
  if ! command -v python3 >/dev/null 2>&1; then
    echo "python3 is required on the monitored host (3.10 or newer)" >&2
    exit 1
  fi
  # The agent needs no privileges, so give it a user with nothing to lose.
  if ! id "$AGENT_USER" >/dev/null 2>&1; then
    useradd --system --home-dir / --shell /usr/sbin/nologin "$AGENT_USER"
  fi

  # SMART (phase 09): the usual distro answer for reading drive health is the `disk`
  # group — no sudoers entry, no root. Re-runs are idempotent.
  if getent group disk >/dev/null 2>&1; then
    if id -nG "$AGENT_USER" 2>/dev/null | tr ' ' '\n' | grep -qx disk; then
      echo "  $AGENT_USER is already in the disk group (SMART reads enabled)"
    else
      usermod -aG disk "$AGENT_USER"
      echo "  added $AGENT_USER to the disk group — raw device reads, which is what smartctl needs"
    fi
  else
    echo "  no disk group on this host — SMART data will read null (harmless; see README)"
  fi
  if ! command -v smartctl >/dev/null 2>&1; then
    echo "  note: smartctl is not installed — every disk reports smart: null until smartmontools is"
  fi

  install -d -m 0755 "$AGENT_DIR"
  install -m 0755 "$AGENT_SOURCE" "$AGENT_TARGET"

  if [ -f "$TOKEN_FILE" ] && [ "$ROTATE" -eq 0 ] && [ -z "$TOKEN" ]; then
    TOKEN="$(cat "$TOKEN_FILE")"
    echo "reusing the existing token in $TOKEN_FILE (--rotate-token for a new one)"
  elif [ -z "$TOKEN" ]; then
    TOKEN="$(new_token)"
  fi
  # 0640 root:$AGENT_USER. The unit passes `--token-file` as a *path*: systemd does not read
  # the secret for the service, the agent process (running as $AGENT_USER) opens it itself,
  # so a 0600 root-owned file makes the agent exit "cannot read --token-file" and the unit
  # never starts. Group ownership is the smallest grant that lets the service read it, and
  # every other account — ops included — still cannot.
  (umask 0177; printf '%s\n' "$TOKEN" > "$TOKEN_FILE")
  chmod 0640 "$TOKEN_FILE"
  chown root:"$AGENT_USER" "$TOKEN_FILE"

  # The certificate bundle is the token's twin and the mode is the whole design. It holds
  # the agent's *private key*, so every other account — ops included — must not read it;
  # but the agent process (running as $AGENT_USER) opens the bundle itself at startup and
  # re-reads it when its stamp changes (that live re-read is how a re-deployed denylist
  # lands without a restart), so a 0600 root-owned file would fail the unit exactly the way
  # it would fail the token file. 0640 root:$AGENT_USER is the smallest grant that works.
  # An existing bundle survives a re-run that was not given the flag — the token-reuse
  # idiom — and this script never dials the dashboard to fetch one.
  if [ -n "$BUNDLE_SRC" ]; then
    install -m 0640 "$BUNDLE_SRC" "$CERT_FILE"
    chown root:"$AGENT_USER" "$CERT_FILE"
    echo "  installed the certificate bundle in $CERT_FILE"
  elif [ -f "$CERT_FILE" ]; then
    echo "reusing the installed certificate bundle in $CERT_FILE (--cert-bundle <file> for a new one)"
  fi

  # TLS mode: the unit runs https with the installed bundle. The --token-file line stays —
  # the agent skips the token check on the TLS path — so a rollout happens host by host
  # instead of on a flag day.
  TLS_EXEC_ARGS=""
  TLS_UNIT_LINES=""
  if [ -f "$CERT_FILE" ]; then
    TLS_EXEC_ARGS=" --cert-bundle $CERT_FILE"
    TLS_UNIT_LINES="# The TLS bundle unpacks its PEM blocks here: tmpfs, private to the unit,
# gone when it stops. ProtectSystem=strict leaves the agent nothing else writable, and the
# agent deletes the key the moment the SSL context holds it.
RuntimeDirectory=dashboards-agent
RuntimeDirectoryMode=0700"
  fi

  # The write path's one unit change: the flag on ExecStart, and nothing else. The four
  # security directives below are rendered exactly as before; a diff that shows them moving
  # is a stop, not a --force (the phase's central safety claim, in one flag).
  WRITES_EXEC_ARGS=""
  if [ "$ALLOW_WRITES" -eq 1 ]; then
    WRITES_EXEC_ARGS=" --allow-writes"
  fi

  # The kernel-gate variant (--as-root / --caps). Default: the dedicated user and an empty
  # CapabilityBoundingSet, i.e. the capability-less unit this script has always written.
  UNIT_USER="$AGENT_USER"
  if [ "$AS_ROOT" -eq 1 ]; then
    UNIT_USER="root"
  fi
  # systemd wants the capabilities space-separated; the flag takes the comma-separated form
  # that reads like a list on a command line. Empty entries are dropped so `A,,B` cannot put
  # a double space in the unit file.
  CAPS_LINE=""
  set -f  # a capability name is a word, never a glob: keep `*` out of the unit file
  for _cap in $(printf '%s' "$CAPS" | tr ',' ' '); do
    CAPS_LINE="${CAPS_LINE:+$CAPS_LINE }$_cap"
  done
  set +f
  if [ "$AS_ROOT" -eq 1 ] && [ -z "$CAPS_LINE" ]; then
    # Not a refusal: a capability-less root is a legitimate if unusual choice, and the operator
    # may have a reason this script cannot see. The note is the reminder gap 10 paid for — the
    # first root-user attempt on the kernel-7.2 host still read n/a on all 16 drives, and this
    # empty bounding set was the only thing that had changed.
    echo "note: --as-root with the default empty CapabilityBoundingSet leaves the process" \
      "capability-less (the kernel gates check capabilities, not uid); pair it with --caps" \
      "for a kernel-gated host"
  fi

  # The unit is rendered to a temp file rather than straight to $UNIT so the guard below can
  # compare the guard's input with what would actually be written — byte for byte, with every
  # flag, bundle and capability decision already applied, not a model of them. The prefix is so
  # the +++ header of the diff the guard prints reads as the generated unit and not as an
  # anonymous /tmp file.
  TMP_UNIT="$(mktemp "${TMPDIR:-/tmp}/dashboards-agent-unit.XXXXXX")"
  cat > "$TMP_UNIT" <<UNIT_FILE
[Unit]
Description=Homelab host agent (read-only system metrics)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=$UNIT_USER
ExecStart=/usr/bin/env python3 $AGENT_TARGET --host $BIND --port $PORT --token-file ${TOKEN_FILE}${TLS_EXEC_ARGS}${WRITES_EXEC_ARGS}
Restart=on-failure
RestartSec=5
Environment=PYTHONUNBUFFERED=1
SyslogIdentifier=$SERVICE
$TLS_UNIT_LINES
# The agent reads /proc, /sys and statvfs() and writes nothing, so everything writable can
# be taken away. smartctl is the one thing that needs a real /dev node, and it gets one the
# way every SMART monitor on Linux does: DAC — the installer adds the service user to the
# disk group — with SELinux still in front of it. The unit therefore carries no device
# filter of its own, and the code (and smartctl -i -A -H) stays read-only by design.
#
# No filter is the fix, not an omission. systemd resolves a device-filter glob by name
# against /proc/devices (fnmatch in bpf_devices_allow_list_major), and kernel 7.2 lists NVMe
# there as blkext (major 259) rather than nvme, so the old /dev/nvme* glob matched nothing —
# and a strict device policy denies every node the list failed to match, so on driveripper
# that unit locked the whole device namespace, root included
# (data/logs/NOTES-driveripper-agent.md). A glob that silently matches nothing is the failure
# mode; having no filter has none, and it is what the live hosts run today.
#
# PrivateDevices=false is the systemd default and is written out for one reason: a private
# /dev with no device list is what dies at spawn with "Failed at step STDIN" (status 208),
# because systemd opens /dev/null for the service stdin before it ever execs python — that is
# how the first phase-10 install failed on a live host running systemd 257.
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=false
ProtectSystem=strict
ProtectHome=read-only
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictNamespaces=true
RestrictRealtime=true
LockPersonality=true
MemoryDenyWriteExecute=true
# --caps: kernel 7.2 gates unprivileged smartctl passthrough (NVMe admin log: CAP_SYS_ADMIN;
# SG_IO allow-list: CAP_SYS_RAWIO) and those checks look at capabilities, not uid — so a root
# service needs them named here, because an empty bounding set bounds even root to CapEff 0.
CapabilityBoundingSet=$CAPS_LINE
SystemCallFilter=@system-service

[Install]
WantedBy=multi-user.target
UNIT_FILE

  # The helper and its two units land beside the agent's unit and never inside it: the
  # re-run guard is the same one, per file — a differing installed helper or unit is a diff
  # plus exit 1 unless --force, so a routine redeploy cannot reset a helper somebody fixed by
  # hand. The *guard* for the three runs here, before the agent unit guard, and the *writes*
  # run after the agent unit write: a refusal does nothing after it — no routectl file, no
  # agent unit, no daemon-reload, no enable — and an apply writes both before the one reload.
  ROUTECTL_REFUSED=0
  if [ "$WITH_ROUTECTL" -eq 1 ]; then
    for _pair in "$ROUTECTL_SOURCE $ROUTECTL_TARGET" \
                "$ROUTECTL_SOCKET_UNIT_SRC $ROUTECTL_SOCKET_UNIT" \
                "$ROUTECTL_SERVICE_UNIT_SRC $ROUTECTL_SERVICE_UNIT"; do
      _src="${_pair%% *}"
      _dst="${_pair##* }"
      [ -f "$_src" ] || { echo "no $_src beside this script" >&2; exit 1; }
      if [ -f "$_dst" ] && ! cmp -s "$_dst" "$_src"; then
        if [ "$FORCE" -eq 1 ]; then
          diff -u "$_dst" "$_src" || true
        else
          echo "refusing to overwrite $_dst (installed copy differs); run with --force to apply:"
          diff -u "$_dst" "$_src" || true
          ROUTECTL_REFUSED=1
          break
        fi
      fi
    done
  fi

  # No silent clobber (phase 17 task 03). $UNIT is the operator's file as much as this
  # template's output: it gets edited on the host to fix something the template did not know
  # (driveripper: User=root plus the two capabilities the kernel-7.2 passthrough gate needs),
  # and an unconditional rewrite would undo that fix while looking like a routine redeploy. So
  # a re-run whose unit differs stops, with the diff on stdout so the difference is readable
  # rather than guessed, and --force is the opt-in to apply it anyway. The diff is printed on
  # both paths and on stdout, not stderr: it is the payload of the answer, and splitting it
  # across two streams interleaves it with the sentence above it. The writes before this point
  # are the idempotent ones — the service user, its disk-group membership, the agent file, and
  # the token/bundle reuse, which re-reads the existing secret instead of replacing it — and a
  # refusal does nothing after it: no unit write, no daemon-reload, no enable, so the unit on
  # disk and the service running against it stay exactly as they were.
  if [ -f "$UNIT" ] && ! cmp -s "$UNIT" "$TMP_UNIT"; then
    if [ "$FORCE" -eq 1 ]; then
      diff -u "$UNIT" "$TMP_UNIT" || true
    else
      echo "refusing to overwrite $UNIT (generated unit differs); run with --force to apply:"
      diff -u "$UNIT" "$TMP_UNIT" || true
      rm -f "$TMP_UNIT"
      exit 1
    fi
  fi
  # Both guards have passed now: the writes are one step. The routectl refusal flag exists
  # only so this write can be skipped when a routectl file refused above — but the agent
  # unit guard below exits the run, so a routectl refusal alone must stop the run too:
  # the refusal is the answer, and a half-apply (agent unit written, routectl not, and no
  # reload) is the failure it exists to prevent.
  if [ "$ROUTECTL_REFUSED" -eq 1 ]; then
    rm -f "$TMP_UNIT"
    exit 1
  fi
  if [ "$WITH_ROUTECTL" -eq 1 ]; then
    install -d -m 0755 "$(dirname "$ROUTECTL_TARGET")"
    install -m 0755 "$ROUTECTL_SOURCE" "$ROUTECTL_TARGET"
    install -m 0644 "$ROUTECTL_SOCKET_UNIT_SRC" "$ROUTECTL_SOCKET_UNIT"
    install -m 0644 "$ROUTECTL_SERVICE_UNIT_SRC" "$ROUTECTL_SERVICE_UNIT"
    echo "  installed the route-switch helper in $ROUTECTL_TARGET and its two units"
  fi
  cat "$TMP_UNIT" > "$UNIT"
  rm -f "$TMP_UNIT"

  systemctl daemon-reload
  systemctl enable --now "$SERVICE"
  sleep 1
  if systemctl is-active --quiet "$SERVICE"; then
    echo "  $SERVICE is running on $BIND:$PORT"
  else
    echo "  $SERVICE failed to start — last lines:" >&2
    journalctl -u "$SERVICE" -n 20 --no-pager >&2
    exit 1
  fi
  if [ "$WITH_ROUTECTL" -eq 1 ]; then
    systemctl enable --now dashboards-routectl.socket
    if ! systemctl is-active --quiet dashboards-routectl.socket; then
      echo "  dashboards-routectl.socket failed to start — last lines:" >&2
      journalctl -u dashboards-routectl.socket -n 20 --no-pager >&2
      exit 1
    fi
    # The reachability check the install owes: a socket the agent's own uid cannot connect
    # to is a feature that fails at 03:00 during an incident, so the check is part of the
    # install and its output is printed. $UNIT_USER is the service user (the agent runs as
    # root under --as-root, and root may always connect — the check then passes trivially).
    if sudo -u "$UNIT_USER" python3 -c 'import socket, sys; s = socket.socket(socket.AF_UNIX); s.connect(sys.argv[1])' "$ROUTECTL_SOCKET_PATH"; then
      echo "  $UNIT_USER can connect to $ROUTECTL_SOCKET_PATH (the whole grant at the file layer)"
    else
      echo "  $UNIT_USER cannot connect to $ROUTECTL_SOCKET_PATH — the helper is installed but the agent cannot reach it" >&2
      exit 1
    fi
  fi
  if [ -n "$TLS_EXEC_ARGS" ]; then
    cat <<NEXT

The unit now serves https with the certificate bundle in $CERT_FILE. Point the dashboard at
it over TLS — the bearer token plays no part on the certificate path, so the URL carries
none:

  HOST_TARGETS="hostname=https://$BIND:$PORT"

After a rotate or a root rotation on the dashboard, fetch the new bundle and re-run this
script: --fetch-bundle <url> --cert-name <name> --session <token> does the download and the
install in one go, and --cert-bundle <file> still works from a file already on this box. The
agent picks a changed denylist up without a restart, but a new certificate needs one, and the
run above only *starts* the unit — restart the service to put the new bundle into effect.
NEXT
  else
    cat <<NEXT

Point the dashboard at it (the token is the URL userinfo, so it never needs a variable of
its own, and the API strips it before anything reaches a browser):

  HOST_TARGETS="hostname=http://$TOKEN@$BIND:$PORT"

Paste the token now if this is the first install — it is not printed again without
--rotate-token, and only root and the agent's own service user can read it on this box:

  $TOKEN
NEXT
  fi
}

case "$MODE" in
  install) do_install ;;
  status) status ;;
  uninstall) uninstall ;;
  routectl-off) routectl_off ;;
esac