add README index for the Raspberry Pi systemd runbooks

Ties the two runbooks together as one host-agnostic directory: which runbook
solves which problem, a file -> target-path map for the units, and the shared
deploy pattern (one tar | ssh pipe into /root, then install with explicit
modes rather than copying a whole tree into /etc).

States the conventions both units follow - configs under /usr/local as the
source of truth with thin systemd wrappers, idempotent installers that re-derive
desired state and never delete what they did not create, and gap-limited
self-healing so a broken dependency cannot make a unit flap.

Also documents the redaction recipe for pasting a WireGuard config: match
`PrivateKey[[:space:]]*=` because wg-quick strip emits spaces around the `=`,
which a `PrivateKey=` regex misses.

LLM-Generated: true

active/device_raspberry_pi/systemd/README.md

@@ -0,0 +1,60 @@
+# Raspberry Pi — boot-time hardening units
+
+Two reusable runbooks for a headless Pi that must stay reachable over SSH.
+Deliberately host-agnostic: **no addresses, key material, or peer identities
+live in this repo** — those stay on each host. Set `HOST` and fill in the
+placeholders from the runbook before running anything.
+
+| Runbook | Solves |
+| --- | --- |
+| [`root-ssh-key.md`](root-ssh-key.md) | root `authorized_keys` gets wiped (imaging, config management) → key is re-installed at boot and re-applied on change |
+| [`wireguard.md`](wireguard.md) | SSH through a WireGuard tunnel address fails after reboot / NAT timeout / roaming to another network |
+
+Verified against Debian 13 (trixie) with systemd 257 and NetworkManager. The
+units are plain POSIX `sh` + systemd, so they carry to any other distribution
+with the same systemd version or newer.
+
+## Layout → target path on the host
+
+| File in this directory | Installs to |
+| --- | --- |
+| `ensure-root-ssh-key` | `/usr/local/sbin/ensure-root-ssh-key` (`0755`) |
+| `ensure-root-ssh-key.service` | `/etc/systemd/system/` |
+| `ensure-root-ssh-key.path` | `/etc/systemd/system/` |
+| `root-ssh-keys/authorized_keys.example` | `/usr/local/share/root-ssh-keys/authorized_keys` (`0644`) — copy, fill in your public key(s) |
+| `wg0-watchdog` | `/usr/local/sbin/wg0-watchdog` (`0755`) |
+| `wg0-watchdog.service` | `/etc/systemd/system/` |
+| `wg0-watchdog.timer` | `/etc/systemd/system/` |
+| `wg-quick@wg0.service.d-10-boot-resilience.conf` | `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` |
+| `wg0.conf.template` | documents `/etc/wireguard/wg0.conf` (`0600`) — the real file is never copied here |
+
+## Deploy pattern
+
+Every command in both runbooks follows the same shape, so a fresh Pi needs only
+`HOST` and the placeholders:
+
+```bash
+HOST=root@<pi-address>
+D=$PWD   # run the snippets from a checkout of this directory
+```
+
+Files travel over one `tar | ssh` pipe into `/root`, then `install` moves them
+into place with explicit modes (no `scp` of a whole tree into `/etc`, no
+ambiguous ownership). Reload and enable once at the end:
+
+```bash
+ssh "$HOST" 'systemctl daemon-reload && systemctl enable --now <units>'
+```
+
+## Conventions worth keeping on any host
+
+- Config files under `/usr/local/{sbin,share}` are the source of truth; systemd
+  units stay thin wrappers so they can be re-imaged independently.
+- Idempotent installers over clever state: every script re-derives the desired
+  state from its own config file and never deletes what it did not create.
+- Anything that must heal itself is gap-limited (a minimum interval between
+  restarts) so a broken dependency cannot make a unit flap.
+- Private keys and addresses are excluded from the repo; when dumping a
+  WireGuard config, redact with a regex that tolerates `PrivateKey = ` spacing
+  (`wg-quick strip` emits spaces, a `PrivateKey=` regex misses them):
+  `sed -E 's/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/'`

active/device_raspberry_pi/systemd/ensure-root-ssh-key

@@ -0,0 +1,61 @@
+#!/bin/sh
+# ensure-root-ssh-key - idempotently install managed SSH public keys into
+# /root/.ssh/authorized_keys. Runs at boot (ensure-root-ssh-key.service) and
+# can be run by hand at any time.
+#
+# Source of truth: /usr/local/share/root-ssh-keys/authorized_keys
+# One key per line, same format as an authorized_keys file. Blank lines and
+# lines starting with '#' are ignored. Keys already present are left alone;
+# foreign keys in authorized_keys are never removed.
+
+set -eu
+
+KEY_DIR="${KEY_DIR:-/root/.ssh}"
+AUTH_KEYS="$KEY_DIR/authorized_keys"
+MANAGED_KEYS="${MANAGED_KEYS:-/usr/local/share/root-ssh-keys/authorized_keys}"
+
+if [ ! -r "$MANAGED_KEYS" ]; then
+    echo "ensure-root-ssh-key: cannot read $MANAGED_KEYS" >&2
+    exit 1
+fi
+
+umask 077
+install -d -m 0700 "$KEY_DIR"
+[ -e "$AUTH_KEYS" ] || : > "$AUTH_KEYS"
+chmod 0600 "$AUTH_KEYS"
+chown root:root "$KEY_DIR" "$AUTH_KEYS"
+
+added=0
+checked=0
+while IFS= read -r line <&3; do
+    case "$line" in
+        '' | \#*) continue ;;
+    esac
+
+    # Basic shape check: <algo> <base64> [comment]
+    algo=${line%% *}
+    case "$algo" in
+        ssh-ed25519 | ssh-rsa | ssh-dss | ecdsa-sha2-* | sk-ecdsa-sha2-* | sk-ssh-ed25519-* | ssh-ed448) ;;
+        *)
+            echo "ensure-root-ssh-key: skipping unrecognised line: $line" >&2
+            continue
+            ;;
+    esac
+
+    checked=$((checked + 1))
+    if grep -qxF -- "$line" "$AUTH_KEYS" 2>/dev/null; then
+        continue
+    fi
+    printf '%s\n' "$line" >> "$AUTH_KEYS"
+    added=$((added + 1))
+    echo "ensure-root-ssh-key: installed key (${algo}, ${#line} chars)"
+done 3< "$MANAGED_KEYS"
+
+if [ "$added" -eq 0 ]; then
+    echo "ensure-root-ssh-key: $checked managed key(s) already present in $AUTH_KEYS"
+else
+    echo "ensure-root-ssh-key: added $added key(s), $checked managed key(s) total"
+fi
+
+chmod 0600 "$AUTH_KEYS"
+exit 0

active/device_raspberry_pi/systemd/ensure-root-ssh-key.path

@@ -0,0 +1,16 @@
+[Unit]
+Description=Watch /root/.ssh/authorized_keys and re-install the managed root SSH key
+Documentation=man:sshd(8)
+After=local-fs.target
+
+[Path]
+# PathChanged covers deletion / rename / creation of the file,
+# PathModified covers in-place edits (key stripped or file truncated).
+# ensure-root-ssh-key.service only writes when a managed key is missing, so
+# its own writes settle after one extra no-op run instead of looping.
+PathChanged=/root/.ssh/authorized_keys
+PathModified=/root/.ssh/authorized_keys
+Unit=ensure-root-ssh-key.service
+
+[Install]
+WantedBy=multi-user.target

active/device_raspberry_pi/systemd/ensure-root-ssh-key.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=Ensure managed SSH public keys are installed in /root/.ssh/authorized_keys
+Documentation=man:sshd(8)
+After=local-fs.target
+Before=ssh.service sshd.service
+ConditionPathExists=/usr/local/share/root-ssh-keys/authorized_keys
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/ensure-root-ssh-key
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target

active/device_raspberry_pi/systemd/root-ssh-key.md

@@ -0,0 +1,109 @@
+# Runbook: self-healing root SSH key
+
+`ensure-root-ssh-key.service` re-installs the managed SSH public key(s) into
+`/root/.ssh/authorized_keys` on every boot, and `ensure-root-ssh-key.path`
+re-applies it at runtime whenever that file is edited, deleted, or replaced.
+Useful when the key gets wiped by imaging, first-boot tooling, or config
+management — or when you need the key to come back after someone else edits the
+file.
+
+## Files on the host
+
+| Path | Purpose |
+| --- | --- |
+| `/etc/systemd/system/ensure-root-ssh-key.service` | `Type=oneshot`, `WantedBy=multi-user.target`, ordered `Before=ssh.service` so the key is in place before the daemon starts |
+| `/etc/systemd/system/ensure-root-ssh-key.path` | `PathChanged=` + `PathModified=` on `/root/.ssh/authorized_keys`, triggers the service |
+| `/usr/local/sbin/ensure-root-ssh-key` | Idempotent POSIX `sh` installer |
+| `/usr/local/share/root-ssh-keys/authorized_keys` | Source of truth — one public key per line, `#` comments allowed |
+
+## Behaviour
+
+- Appends any managed key that is missing; never removes or rewrites foreign keys.
+- Creates `/root/.ssh` (`0700`) and `authorized_keys` (`0600`, root:root) if absent.
+- Lines whose first field is not a known key algorithm are skipped with a warning.
+- Skipped, not failed, if the managed keys file is deleted.
+- Runtime self-healing: deleting, truncating, or renaming `authorized_keys` puts
+  the managed key(s) back within ~1s. That also means a managed key cannot be
+  revoked from the host alone — delete it from
+  `/usr/local/share/root-ssh-keys/authorized_keys` first.
+
+> Gotcha: systemd has no `PathMissing=` key — it is silently ignored with a
+> warning in `systemd-analyze verify`. `PathChanged=` is what fires on
+> delete/rename; `PathModified=` covers in-place appends and truncations. Set both.
+
+## Install
+
+```bash
+HOST=root@<pi-address>
+D=$PWD        # run the snippets from a checkout of this directory
+PUB=$HOME/.ssh/id_ed25519.pub          # the key that must always work
+
+ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /usr/local/share/root-ssh-keys'
+tar -C "$D" -cf - ensure-root-ssh-key ensure-root-ssh-key.service ensure-root-ssh-key.path | \
+ssh "$HOST" 'tar -xf - -C /root &&
+  install -m 0755 /root/ensure-root-ssh-key /usr/local/sbin/ &&
+  install -m 0644 /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path /etc/systemd/system/ &&
+  rm -f /root/ensure-root-ssh-key /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path &&
+  systemctl daemon-reload'
+cat "$PUB" | ssh "$HOST" 'install -m 0644 /dev/stdin /usr/local/share/root-ssh-keys/authorized_keys'
+ssh "$HOST" 'systemctl enable --now ensure-root-ssh-key.service ensure-root-ssh-key.path'
+```
+
+Use `root-ssh-keys/authorized_keys.example` instead of `$PUB` when the host
+should carry more than one key.
+
+## Verify
+
+One-shot smoke test that proves both units without touching a live session
+(existing SSH sessions are unaffected; only new logins during the window would
+fail):
+
+```bash
+ssh "$HOST" 'cp -a /root/.ssh/authorized_keys /root/.ssh/authorized_keys.testbak
+  rm -f /root/.ssh/authorized_keys
+  systemctl start ensure-root-ssh-key.service && cat /root/.ssh/authorized_keys
+  rm -f /root/.ssh/authorized_keys.testbak'
+```
+
+Status and history:
+
+```bash
+ssh "$HOST" 'systemctl is-enabled ensure-root-ssh-key.service ensure-root-ssh-key.path
+  systemctl status ensure-root-ssh-key.service --no-pager | head -8
+  journalctl -u ensure-root-ssh-key.service -b --no-pager | tail'
+```
+
+The installer honours `KEY_DIR` and `MANAGED_KEYS`, so the create-if-missing and
+preserve-foreign-keys paths can be exercised in a scratch directory instead of
+`/root/.ssh`:
+
+```bash
+ssh "$HOST" 'install -d -m 0700 /tmp/k/src && cp /usr/local/share/root-ssh-keys/authorized_keys /tmp/k/src/
+  KEY_DIR=/tmp/k/root MANAGED_KEYS=/tmp/k/src/authorized_keys /usr/local/sbin/ensure-root-ssh-key
+  ls -ld /tmp/k/root /tmp/k/root/authorized_keys; rm -rf /tmp/k'
+```
+
+Checklist that has passed on a reference host:
+
+- `systemd-analyze verify` clean for both units, both `enabled`.
+- Repeated runs add no duplicates; a wiped file and a deleted file are both restored.
+- Missing `/root/.ssh` is created `0700`, file `0600`.
+- Foreign keys survive; no trigger loop once the path unit settles.
+- Boot ordering: `Before=ssh.service` is honoured inside the boot transaction;
+  prove it after a reboot with `systemd-analyze critical-chain ssh.service`.
+
+## Manage keys
+
+```bash
+ssh "$HOST" 'echo "<public key line>" >> /usr/local/share/root-ssh-keys/authorized_keys
+  systemctl restart ensure-root-ssh-key.service'
+```
+
+## Uninstall
+
+```bash
+ssh "$HOST" 'systemctl disable --now ensure-root-ssh-key.path ensure-root-ssh-key.service
+  rm -f /etc/systemd/system/ensure-root-ssh-key.{service,path} /usr/local/sbin/ensure-root-ssh-key
+  rm -rf /usr/local/share/root-ssh-keys
+  systemctl daemon-reload'
+```

active/device_raspberry_pi/systemd/root-ssh-keys/authorized_keys.example

@@ -0,0 +1,11 @@
+# Template for /usr/local/share/root-ssh-keys/authorized_keys on the host.
+# Keys listed here are (re)installed into /root/.ssh/authorized_keys at boot and
+# whenever that file changes. Copy to `authorized_keys`, add your public keys,
+# and keep the real file on the host - do not commit it.
+#
+# One key per line, authorized_keys format. Blank lines and lines starting with
+# '#' are ignored; a line whose first field is not a known key algorithm is
+# skipped with a warning.
+#
+# ssh-ed25519 AAAA<base64> comment@host
+# ssh-rsa     AAAA<base64> comment@host

active/device_raspberry_pi/systemd/wg-quick@wg0.service.d-10-boot-resilience.conf

@@ -0,0 +1,17 @@
+# /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf
+# The stock unit already has After=/Wants=network-online.target nss-lookup.target.
+# This makes a boot-time failure (DHCP/DNS not ready yet, endpoint host offline)
+# retry forever instead of leaving the tunnel down until someone logs in.
+[Unit]
+# Unlimited retries: do not let the start rate limiter give up on us.
+StartLimitIntervalSec=0
+
+[Service]
+# Fail fast instead of hanging forever on endpoint resolution (stock value is
+# infinity), then let Restart= try again. Same variable as the stock unit,
+# later assignment wins.
+Environment=WG_ENDPOINT_RESOLUTION_RETRIES=60
+TimeoutStartSec=120
+TimeoutStopSec=30
+Restart=on-failure
+RestartSec=15

active/device_raspberry_pi/systemd/wg0-watchdog

@@ -0,0 +1,151 @@
+#!/bin/sh
+# wg0-watchdog - keep the WireGuard client tunnel up so SSH to the tunnel
+# address works after boot, after roaming to a new network, and after the NAT
+# mapping or endpoint IP goes stale.
+#
+# Decisions, oldest symptom first:
+#   interface or peer gone            -> restart wg-quick@wg0.service
+#   no PersistentKeepalive in kernel  -> set it (covers a config edited without reload)
+#   handshake older than STALE        -> re-point the peer at the endpoint
+#                                        (re-resolves the hostname, forces a handshake)
+#   handshake older than STALE_RESTART-> restart wg-quick@wg0.service
+#
+# Restarts are gap-limited so a broken network cannot make the unit flap.
+# Never removes peers, addresses, or routes it did not create.
+
+set -eu
+
+IFACE="${WG_WD_IFACE:-wg0}"
+UNIT="${WG_WD_UNIT:-wg-quick@wg0.service}"
+CONF="${WG_WD_CONF:-/etc/wireguard/$IFACE.conf}"
+KEEPALIVE="${WG_WD_KEEPALIVE:-25}"
+STALE="${WG_WD_STALE:-120}"              # seconds: poke the endpoint
+STALE_RESTART="${WG_WD_STALE_RESTART:-300}"   # seconds: restart the unit
+MIN_RESTART_GAP="${WG_WD_MIN_RESTART_GAP:-180}"  # seconds between restarts
+STATE_DIR="${WG_WD_STATE_DIR:-/var/lib/wg0-watchdog}"
+
+DRY=0
+[ "${1:-}" = "--dry-run" ] && DRY=1
+
+now=$(date +%s)
+log() { printf 'wg0-watchdog: %s\n' "$*"; }
+run() {
+    if [ "$DRY" -eq 1 ]; then
+        printf 'wg0-watchdog: DRY-RUN would run:'
+        printf ' %s' "$@"
+        printf '\n'
+    else
+        "$@"
+    fi
+}
+
+read_state() { cat "$1" 2>/dev/null || echo 0; }
+write_state() {
+    if [ "$DRY" -eq 1 ]; then
+        return 0
+    fi
+    mkdir -p "$STATE_DIR"
+    printf '%s\n' "$2" > "$1"
+}
+
+restart_unit() {
+    reason=$1
+    last=$(read_state "$STATE_DIR/last-restart")
+    case "$last" in ''|*[!0-9]*) last=0 ;; esac
+    since=$((now - last))
+    if [ "$since" -lt "$MIN_RESTART_GAP" ]; then
+        log "restart of $UNIT suppressed ($reason); last restart ${since}s ago"
+        return 0
+    fi
+    log "restarting $UNIT ($reason)"
+    write_state "$STATE_DIR/last-restart" "$now"
+    # --no-block: a wg-quick up stuck on endpoint resolution must not wedge the
+    # watchdog; the next run reports whether the restart worked.
+    run systemctl restart --no-block "$UNIT"
+}
+
+# --- 1. interface / peer present -------------------------------------------
+if ! ip link show "$IFACE" >/dev/null 2>&1; then
+    restart_unit "interface $IFACE is missing"
+    exit 0
+fi
+
+peer=$(wg show "$IFACE" peers 2>/dev/null | head -n 1)
+if [ -z "${peer:-}" ]; then
+    restart_unit "no peer configured on $IFACE"
+    exit 0
+fi
+
+# --- 2. persistent keepalive present ---------------------------------------
+live_ka=$(wg show "$IFACE" persistent-keepalive 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
+[ -n "${live_ka:-}" ] || live_ka=0
+if [ "$live_ka" != "$KEEPALIVE" ]; then
+    log "persistent-keepalive is $live_ka, setting $KEEPALIVE"
+    run wg set "$IFACE" peer "$peer" persistent-keepalive "$KEEPALIVE"
+fi
+
+# --- 3. handshake age -------------------------------------------------------
+hs=$(wg show "$IFACE" latest-handshakes 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
+case "${hs:-}" in ''|*[!0-9]*) hs=0 ;; esac
+if [ "$hs" -eq 0 ]; then
+    age=999999
+else
+    age=$((now - hs))
+fi
+
+# endpoint currently in use by the kernel, and the cached last-known-good one
+live_ep=$(wg show "$IFACE" endpoints 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
+cache_file="$STATE_DIR/endpoint"
+cached_ep=""
+[ -f "$cache_file" ] && cached_ep=$(cat "$cache_file" 2>/dev/null || true)
+
+if [ "$age" -lt "$STALE" ]; then
+    if [ -n "$live_ep" ]; then
+        write_state "$cache_file" "$live_ep"
+    fi
+    log "healthy: handshake ${age}s ago, endpoint ${live_ep:-unknown}, keepalive ${KEEPALIVE}s"
+    exit 0
+fi
+
+log "stale: last handshake ${age}s ago"
+
+# Prefer the endpoint from the config file, freshly resolved. Fall back to the
+# kernel's current endpoint, then to the cache - important because a DNS server
+# from the tunnel config may be reachable only through this very tunnel.
+target_ep=""
+conf_ep=$(awk -F= 'tolower($1) ~ /^[[:space:]]*endpoint[[:space:]]*$/ { gsub(/[[:space:]]/, "", $2); print $2 }' "$CONF" 2>/dev/null | tail -n 1)
+if [ -n "${conf_ep:-}" ]; then
+    host=${conf_ep%:*}
+    port=${conf_ep##*:}
+    case "$host" in
+        *[!0-9.]*)
+            resolved=$(getent hosts "$host" 2>/dev/null | awk '{print $1; exit}' || true)
+            ;;
+        *)
+            resolved=$host
+            ;;
+    esac
+    if [ -n "${resolved:-}" ]; then
+        target_ep="$resolved:$port"
+    else
+        log "could not resolve $host (tunnel DNS?), falling back"
+    fi
+fi
+[ -n "$target_ep" ] || target_ep=${live_ep:-${cached_ep:-}}
+
+if [ "$age" -ge "$STALE_RESTART" ]; then
+    restart_unit "handshake ${age}s old"
+    exit 0
+fi
+
+if [ -z "$target_ep" ]; then
+    log "no endpoint to re-point at; will retry next run"
+    exit 0
+fi
+
+if [ "$target_ep" = "$live_ep" ]; then
+    log "endpoint already $target_ep; forcing a handshake"
+fi
+log "re-pointing peer at $target_ep"
+run wg set "$IFACE" peer "$peer" endpoint "$target_ep"
+exit 0

active/device_raspberry_pi/systemd/wg0-watchdog.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Keep the WireGuard wg0 tunnel reachable for SSH
+Documentation=man:wg-quick(8)
+
+[Service]
+Type=oneshot
+# Never let a hung systemctl restart inside the script wedge the watchdog.
+TimeoutStartSec=120
+ExecStart=/usr/local/sbin/wg0-watchdog
+StandardOutput=journal
+StandardError=journal
+Nice=10

active/device_raspberry_pi/systemd/wg0-watchdog.timer

@@ -0,0 +1,14 @@
+[Unit]
+Description=Check the WireGuard wg0 tunnel every minute and heal it if needed
+
+[Timer]
+# First check shortly after boot, then every minute. Persistent=true catches up
+# if the machine was suspended/offline.
+OnBootSec=30s
+OnUnitActiveSec=60s
+AccuracySec=15s
+RandomizedDelaySec=5s
+Persistent=true
+
+[Install]
+WantedBy=timers.target

active/device_raspberry_pi/systemd/wg0.conf.template

@@ -0,0 +1,20 @@
+# wg0.conf template — /etc/wireguard/wg0.conf on the host, mode 0600 root:root.
+# Never commit a real copy: this file holds the tunnel private key.
+# Generate the key with `umask 077; wg genkey | tee privatekey | wg pubkey`.
+
+[Interface]
+PrivateKey = <tunnel private key>
+Address = <TUNNEL_ADDR>
+# Deliberately no `DNS` key: any resolver listed here is pushed as a system
+# upstream while the tunnel is up, which makes all name resolution depend on the
+# tunnel. Omitting it keeps the LAN's DHCP resolvers. Debian's wg-quick has no
+# `DNS = none` - a non-IP value becomes a search domain.
+
+[Peer]
+PublicKey = <PEER_PUBKEY>
+AllowedIPs = <PEER_NETWORK>
+Endpoint = <WG_SERVER_HOST:PORT>
+# Required when the client is behind NAT and must be reachable inbound (SSH):
+# without it the mapping expires and the server cannot initiate. 25s survives
+# typical 30-60s NAT UDP timeouts.
+PersistentKeepalive = 25

active/device_raspberry_pi/systemd/wireguard.md

@@ -0,0 +1,162 @@
+# Runbook: WireGuard client that survives reboot, NAT, and roaming
+
+For a Pi whose SSH address is a **WireGuard tunnel address** rather than a LAN
+address: the physical interface is ordinary DHCP and the Pi moves between
+networks, while the tunnel peer is a reachable server. Reachability then depends
+on three separate things, each of which fails independently — the unit starting
+at boot, the NAT mapping staying open, and the peer hostname still resolving.
+
+Placeholders used below:
+
+| Placeholder | Meaning |
+| --- | --- |
+| `$HOST` | `root@<address you can already reach>` |
+| `<TUNNEL_ADDR>` | this Pi's tunnel address, e.g. `<addr>/32` |
+| `<PEER_NETWORK>` | network routed through the tunnel, e.g. `<net>/<cidr>` |
+| `<WG_SERVER_HOST:PORT>` | tunnel server endpoint |
+| `<PEER_PUBKEY>` | the server's WireGuard public key |
+
+## Files on the host
+
+| Path | Purpose |
+| --- | --- |
+| `/etc/wireguard/wg0.conf` | tunnel config, mode `0600` — see `wg0.conf.template`, never stored in this repo |
+| `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` | retry a failed boot-time `up`: `Restart=on-failure`, `RestartSec=15`, `StartLimitIntervalSec=0`, `WG_ENDPOINT_RESOLUTION_RETRIES=60`, `TimeoutStartSec=120` |
+| `/usr/local/sbin/wg0-watchdog` | heals the tunnel once a minute |
+| `/etc/systemd/system/wg0-watchdog.service` | oneshot that runs the script |
+| `/etc/systemd/system/wg0-watchdog.timer` | `OnBootSec=30s`, `OnUnitActiveSec=60s`, `Persistent=true` |
+| `/var/lib/wg0-watchdog/endpoint` | last-known-good endpoint IP, used when DNS is unavailable |
+
+## Failure modes this closes
+
+1. **No `PersistentKeepalive`.** Behind NAT the server can only send to the Pi
+   while the mapping lives, so once it expires the tunnel looks configured but
+   inbound SSH times out. `PersistentKeepalive = 25` punches UDP state often
+   enough for typical 30-60s NAT timeouts.
+2. **A boot-time failure is permanent.** `wg-quick up` fails outright if the
+   endpoint hostname does not resolve yet or the server is down, and the unit is
+   not retried — so the tunnel stays dead until someone logs in locally.
+   `Restart=on-failure` with `StartLimitIntervalSec=0` retries forever;
+   `WG_ENDPOINT_RESOLUTION_RETRIES` is lowered from the stock `infinity`
+   (which hangs the unit indefinitely) so a retry can actually happen.
+3. **Nothing heals a stale tunnel.** Roaming to a new network keeps the old
+   endpoint IP in the kernel, and a peer whose endpoint is unreachable simply
+   stops handshaking without the unit ever noticing, because the unit already
+   "succeeded". The watchdog acts on handshake age.
+
+## Install
+
+```bash
+HOST=root@<pi-address>
+D=$PWD        # run the snippets from a checkout of this directory
+
+ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /etc/systemd/system/wg-quick@wg0.service.d'
+tar -C "$D" -cf - wg0-watchdog wg0-watchdog.service wg0-watchdog.timer \
+    'wg-quick@wg0.service.d-10-boot-resilience.conf' | \
+ssh "$HOST" 'tar -xf - -C /root &&
+  install -m 0755 /root/wg0-watchdog /usr/local/sbin/ &&
+  install -m 0644 /root/wg0-watchdog.service /root/wg0-watchdog.timer /etc/systemd/system/ &&
+  install -m 0644 "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" \
+    /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf &&
+  rm -f /root/wg0-watchdog /root/wg0-watchdog.service /root/wg0-watchdog.timer \
+    "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" &&
+  systemctl daemon-reload && systemctl enable --now wg-quick@wg0.service wg0-watchdog.timer'
+```
+
+Keep `PersistentKeepalive = 25` in `[Peer]` (see `wg0.conf.template`) and check
+the config file is `0600` — a world-readable `wg0.conf` exposes the tunnel
+private key to every local account:
+
+```bash
+ssh "$HOST" 'ls -l /etc/wireguard/wg0.conf; wg show wg0 persistent-keepalive'
+```
+
+## Watchdog behaviour
+
+Oldest symptom first, one action per run:
+
+1. interface or peer gone → `systemctl restart --no-block wg-quick@wg0.service`
+2. no `PersistentKeepalive` in the kernel → `wg set ... persistent-keepalive <n>`
+   (covers a config edited without a reload)
+3. handshake older than `WG_WD_STALE` (default 120s) → re-point the peer at the
+   endpoint, preferring a fresh resolution of the hostname from the config file,
+   then the kernel's current endpoint, then the cached IP
+4. handshake older than `WG_WD_STALE_RESTART` (default 300s) → restart the unit
+
+Restarts are gap-limited to one per `WG_WD_MIN_RESTART_GAP` (default 180s) so a
+dead network cannot make the unit flap. `--no-block` keeps a hung `wg-quick up`
+from wedging the watchdog. Peers, addresses, and routes are only ever added.
+All thresholds and the interface/state paths are environment overrides, and
+`--dry-run` prints the decision without acting.
+
+The endpoint resolution order matters when the tunnel config also pushes a DNS
+server: that resolver is reachable only through the tunnel, so a dead tunnel can
+mean the endpoint hostname cannot be resolved exactly when it is needed — hence
+the cached-IP fallback.
+
+## DNS
+
+A `DNS = <resolver>` line in the interface section is pushed through
+`resolvconf`/`systemd-resolved` while the tunnel is up, which can silently make
+*all* name resolution on the host depend on the tunnel. If the Pi must resolve
+names on whatever network it lands on, omit the `DNS` key entirely and let the
+LAN's DHCP resolvers stand.
+
+Two gotchas:
+
+- Debian's `wg-quick` does **not** implement `DNS = none` — any non-IP value
+  becomes a search domain, so the line has to be absent, not set to `none`.
+- A record already pushed by a previous `wg-quick up` survives the config edit;
+  withdraw it (`resolvconf -d <record>`, name visible in `resolvconf --list`) and
+  confirm the local caching resolver's upstreams before believing it is gone.
+
+## Taking wg0 down on purpose
+
+Stop the watchdog first or the tunnel comes straight back:
+
+```bash
+ssh "$HOST" 'systemctl stop wg0-watchdog.timer wg-quick@wg0.service'
+ssh "$HOST" 'systemctl start wg0-watchdog.timer'      # resume
+```
+
+## Check on it
+
+```bash
+ssh "$HOST" 'wg show wg0
+  systemctl status wg-quick@wg0 --no-pager | head -6
+  systemctl list-timers "wg0-watchdog*" --no-pager | head -3
+  journalctl -u wg0-watchdog.service -b --no-pager | tail -15'
+```
+
+Dry-run the decision logic without touching the tunnel:
+
+```bash
+ssh "$HOST" 'WG_WD_STATE_DIR=/tmp/wd WG_WD_STALE=0 /usr/local/sbin/wg0-watchdog --dry-run; rm -rf /tmp/wd'
+```
+
+Validate an edited config without disturbing the running tunnel:
+
+```bash
+ssh "$HOST" 'wg-quick strip /etc/wireguard/wg0.conf | sed -E "s/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/"'
+```
+
+## Diagnosis notes (generic, worth checking on any host)
+
+- `wg-quick@<iface>` can be `enabled` yet `inactive` while the interface exists
+  and passes traffic — it may have been created by hand or taken over by
+  NetworkManager as an assumed profile (`nmcli dev status` shows such a device
+  as `connected (externally)`). Nothing owns it: `systemctl reload` fails with
+  "unit not active", so apply changes with `wg set` or a full restart, and let
+  `systemctl restart` take ownership before trusting a reboot.
+- `Restart=on-failure` does work for `Type=oneshot` on systemd 257 — worth
+  proving on a new distribution with a throwaway unit under
+  `/run/systemd/system` before relying on it: `Type=oneshot` +
+  `ExecStart=/bin/false`, then check `systemctl show -p NRestarts`.
+- `systemd-analyze verify <unit>` catches mistyped unit keys, and it only warns —
+  read the output instead of trusting the exit status.
+- NetworkManager does not fight `wg-quick` as long as no keyfile exists for the
+  interface and its assumed profile keeps `autoconnect no`; check
+  `/etc/NetworkManager/system-connections/` before assuming that.
+- The real end-to-end test is a reboot: DHCP → endpoint resolution →
+  `wg-quick up` → inbound SSH through the tunnel. `systemd-analyze
+  critical-chain ssh.service` after boot shows whether the ordering held.