homelab/active/device_raspberry_pi/systemd/root-ssh-key.md

# Runbook: self-healing root SSH key

`ensure-root-ssh-key.service` re-installs the managed SSH public key(s) into
`/root/.ssh/authorized_keys` on every boot, and `ensure-root-ssh-key.path`
re-applies it at runtime whenever that file is edited, deleted, or replaced.
Useful when the key gets wiped by imaging, first-boot tooling, or config
management — or when you need the key to come back after someone else edits the
file.

## Files on the host

| Path | Purpose |
| --- | --- |
| `/etc/systemd/system/ensure-root-ssh-key.service` | `Type=oneshot`, `WantedBy=multi-user.target`, ordered `Before=ssh.service` so the key is in place before the daemon starts |
| `/etc/systemd/system/ensure-root-ssh-key.path` | `PathChanged=` + `PathModified=` on `/root/.ssh/authorized_keys`, triggers the service |
| `/usr/local/sbin/ensure-root-ssh-key` | Idempotent POSIX `sh` installer |
| `/usr/local/share/root-ssh-keys/authorized_keys` | Source of truth — one public key per line, `#` comments allowed |

## Behaviour

- Appends any managed key that is missing; never removes or rewrites foreign keys.
- Creates `/root/.ssh` (`0700`) and `authorized_keys` (`0600`, root:root) if absent.
- Lines whose first field is not a known key algorithm are skipped with a warning.
- Skipped, not failed, if the managed keys file is deleted.
- Runtime self-healing: deleting, truncating, or renaming `authorized_keys` puts
  the managed key(s) back within ~1s. That also means a managed key cannot be
  revoked from the host alone — delete it from
  `/usr/local/share/root-ssh-keys/authorized_keys` first.

> Gotcha: systemd has no `PathMissing=` key — it is silently ignored with a
> warning in `systemd-analyze verify`. `PathChanged=` is what fires on
> delete/rename; `PathModified=` covers in-place appends and truncations. Set both.

## Install

```bash
HOST=root@<pi-address>
D=$PWD        # run the snippets from a checkout of this directory
PUB=$HOME/.ssh/id_ed25519.pub          # the key that must always work

ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /usr/local/share/root-ssh-keys'
tar -C "$D" -cf - ensure-root-ssh-key ensure-root-ssh-key.service ensure-root-ssh-key.path | \
ssh "$HOST" 'tar -xf - -C /root &&
  install -m 0755 /root/ensure-root-ssh-key /usr/local/sbin/ &&
  install -m 0644 /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path /etc/systemd/system/ &&
  rm -f /root/ensure-root-ssh-key /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path &&
  systemctl daemon-reload'
cat "$PUB" | ssh "$HOST" 'install -m 0644 /dev/stdin /usr/local/share/root-ssh-keys/authorized_keys'
ssh "$HOST" 'systemctl enable --now ensure-root-ssh-key.service ensure-root-ssh-key.path'
```

Use `root-ssh-keys/authorized_keys.example` instead of `$PUB` when the host
should carry more than one key.

## Verify

One-shot smoke test that proves both units without touching a live session
(existing SSH sessions are unaffected; only new logins during the window would
fail):

```bash
ssh "$HOST" 'cp -a /root/.ssh/authorized_keys /root/.ssh/authorized_keys.testbak
  rm -f /root/.ssh/authorized_keys
  systemctl start ensure-root-ssh-key.service && cat /root/.ssh/authorized_keys
  rm -f /root/.ssh/authorized_keys.testbak'
```

Status and history:

```bash
ssh "$HOST" 'systemctl is-enabled ensure-root-ssh-key.service ensure-root-ssh-key.path
  systemctl status ensure-root-ssh-key.service --no-pager | head -8
  journalctl -u ensure-root-ssh-key.service -b --no-pager | tail'
```

The installer honours `KEY_DIR` and `MANAGED_KEYS`, so the create-if-missing and
preserve-foreign-keys paths can be exercised in a scratch directory instead of
`/root/.ssh`:

```bash
ssh "$HOST" 'install -d -m 0700 /tmp/k/src && cp /usr/local/share/root-ssh-keys/authorized_keys /tmp/k/src/
  KEY_DIR=/tmp/k/root MANAGED_KEYS=/tmp/k/src/authorized_keys /usr/local/sbin/ensure-root-ssh-key
  ls -ld /tmp/k/root /tmp/k/root/authorized_keys; rm -rf /tmp/k'
```

Checklist that has passed on a reference host:

- `systemd-analyze verify` clean for both units, both `enabled`.
- Repeated runs add no duplicates; a wiped file and a deleted file are both restored.
- Missing `/root/.ssh` is created `0700`, file `0600`.
- Foreign keys survive; no trigger loop once the path unit settles.
- Boot ordering: `Before=ssh.service` is honoured inside the boot transaction;
  prove it after a reboot with `systemd-analyze critical-chain ssh.service`.

## Manage keys

```bash
ssh "$HOST" 'echo "<public key line>" >> /usr/local/share/root-ssh-keys/authorized_keys
  systemctl restart ensure-root-ssh-key.service'
```

## Uninstall

```bash
ssh "$HOST" 'systemctl disable --now ensure-root-ssh-key.path ensure-root-ssh-key.service
  rm -f /etc/systemd/system/ensure-root-ssh-key.{service,path} /usr/local/sbin/ensure-root-ssh-key
  rm -rf /usr/local/share/root-ssh-keys
  systemctl daemon-reload'
```