homelab/active/device_raspberry_pi/systemd/wireguard.md

Runbook: WireGuard client that survives reboot, NAT, and roaming

For a Pi whose SSH address is a WireGuard tunnel address rather than a LAN address: the physical interface is ordinary DHCP and the Pi moves between networks, while the tunnel peer is a reachable server. Reachability then depends on three separate things, each of which fails independently — the unit starting at boot, the NAT mapping staying open, and the peer hostname still resolving.

Placeholders used below:

Placeholder	Meaning
$HOST	root@<address you can already reach>
<TUNNEL_ADDR>	this Pi's tunnel address, e.g. <addr>/32
<PEER_NETWORK>	network routed through the tunnel, e.g. <net>/<cidr>
<WG_SERVER_HOST:PORT>	tunnel server endpoint
<PEER_PUBKEY>	the server's WireGuard public key

Files on the host

Path	Purpose
/etc/wireguard/wg0.conf	tunnel config, mode 0600 — see wg0.conf.template, never stored in this repo
/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf	retry a failed boot-time up: Restart=on-failure, RestartSec=15, StartLimitIntervalSec=0, WG_ENDPOINT_RESOLUTION_RETRIES=60, TimeoutStartSec=120
/usr/local/sbin/wg0-watchdog	heals the tunnel once a minute
/etc/systemd/system/wg0-watchdog.service	oneshot that runs the script
/etc/systemd/system/wg0-watchdog.timer	OnBootSec=30s, OnUnitActiveSec=60s, Persistent=true
/var/lib/wg0-watchdog/endpoint	last-known-good endpoint IP, used when DNS is unavailable

Failure modes this closes

1. No PersistentKeepalive. Behind NAT the server can only send to the Pi while the mapping lives, so once it expires the tunnel looks configured but inbound SSH times out. PersistentKeepalive = 25 punches UDP state often enough for typical 30-60s NAT timeouts.
2. A boot-time failure is permanent. wg-quick up fails outright if the endpoint hostname does not resolve yet or the server is down, and the unit is not retried — so the tunnel stays dead until someone logs in locally. Restart=on-failure with StartLimitIntervalSec=0 retries forever; WG_ENDPOINT_RESOLUTION_RETRIES is lowered from the stock infinity (which hangs the unit indefinitely) so a retry can actually happen.
3. Nothing heals a stale tunnel. Roaming to a new network keeps the old endpoint IP in the kernel, and a peer whose endpoint is unreachable simply stops handshaking without the unit ever noticing, because the unit already "succeeded". The watchdog acts on handshake age.

Install

HOST=root@<pi-address>
D=$PWD        # run the snippets from a checkout of this directory

ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /etc/systemd/system/wg-quick@wg0.service.d'
tar -C "$D" -cf - wg0-watchdog wg0-watchdog.service wg0-watchdog.timer \
    'wg-quick@wg0.service.d-10-boot-resilience.conf' | \
ssh "$HOST" 'tar -xf - -C /root &&
  install -m 0755 /root/wg0-watchdog /usr/local/sbin/ &&
  install -m 0644 /root/wg0-watchdog.service /root/wg0-watchdog.timer /etc/systemd/system/ &&
  install -m 0644 "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" \
    /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf &&
  rm -f /root/wg0-watchdog /root/wg0-watchdog.service /root/wg0-watchdog.timer \
    "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" &&
  systemctl daemon-reload && systemctl enable --now wg-quick@wg0.service wg0-watchdog.timer'

Keep PersistentKeepalive = 25 in [Peer] (see wg0.conf.template) and check the config file is 0600 — a world-readable wg0.conf exposes the tunnel private key to every local account:

ssh "$HOST" 'ls -l /etc/wireguard/wg0.conf; wg show wg0 persistent-keepalive'

Watchdog behaviour

Oldest symptom first, one action per run:

1. interface or peer gone → systemctl restart --no-block wg-quick@wg0.service
2. no PersistentKeepalive in the kernel → wg set ... persistent-keepalive <n> (covers a config edited without a reload)
3. handshake older than WG_WD_STALE (default 120s) → re-point the peer at the endpoint, preferring a fresh resolution of the hostname from the config file, then the kernel's current endpoint, then the cached IP
4. handshake older than WG_WD_STALE_RESTART (default 300s) → restart the unit

Restarts are gap-limited to one per WG_WD_MIN_RESTART_GAP (default 180s) so a dead network cannot make the unit flap. --no-block keeps a hung wg-quick up from wedging the watchdog. Peers, addresses, and routes are only ever added. All thresholds and the interface/state paths are environment overrides, and --dry-run prints the decision without acting.

The endpoint resolution order matters when the tunnel config also pushes a DNS server: that resolver is reachable only through the tunnel, so a dead tunnel can mean the endpoint hostname cannot be resolved exactly when it is needed — hence the cached-IP fallback.

DNS

A DNS = <resolver> line in the interface section is pushed through resolvconf/systemd-resolved while the tunnel is up, which can silently make all name resolution on the host depend on the tunnel. If the Pi must resolve names on whatever network it lands on, omit the DNS key entirely and let the LAN's DHCP resolvers stand.

Two gotchas:

• Debian's wg-quick does not implement DNS = none — any non-IP value becomes a search domain, so the line has to be absent, not set to none.
• A record already pushed by a previous wg-quick up survives the config edit; withdraw it (resolvconf -d <record>, name visible in resolvconf --list) and confirm the local caching resolver's upstreams before believing it is gone.

Taking wg0 down on purpose

Stop the watchdog first or the tunnel comes straight back:

ssh "$HOST" 'systemctl stop wg0-watchdog.timer wg-quick@wg0.service'
ssh "$HOST" 'systemctl start wg0-watchdog.timer'      # resume

Check on it

ssh "$HOST" 'wg show wg0
  systemctl status wg-quick@wg0 --no-pager | head -6
  systemctl list-timers "wg0-watchdog*" --no-pager | head -3
  journalctl -u wg0-watchdog.service -b --no-pager | tail -15'

Dry-run the decision logic without touching the tunnel:

ssh "$HOST" 'WG_WD_STATE_DIR=/tmp/wd WG_WD_STALE=0 /usr/local/sbin/wg0-watchdog --dry-run; rm -rf /tmp/wd'

Validate an edited config without disturbing the running tunnel:

ssh "$HOST" 'wg-quick strip /etc/wireguard/wg0.conf | sed -E "s/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/"'

Diagnosis notes (generic, worth checking on any host)

• wg-quick@<iface> can be enabled yet inactive while the interface exists and passes traffic — it may have been created by hand or taken over by NetworkManager as an assumed profile (nmcli dev status shows such a device as connected (externally)). Nothing owns it: systemctl reload fails with "unit not active", so apply changes with wg set or a full restart, and let systemctl restart take ownership before trusting a reboot.
• Restart=on-failure does work for Type=oneshot on systemd 257 — worth proving on a new distribution with a throwaway unit under /run/systemd/system before relying on it: Type=oneshot + ExecStart=/bin/false, then check systemctl show -p NRestarts.
• systemd-analyze verify <unit> catches mistyped unit keys, and it only warns — read the output instead of trusting the exit status.
• NetworkManager does not fight wg-quick as long as no keyfile exists for the interface and its assumed profile keeps autoconnect no; check /etc/NetworkManager/system-connections/ before assuming that.
• The real end-to-end test is a reboot: DHCP → endpoint resolution → wg-quick up → inbound SSH through the tunnel. systemd-analyze critical-chain ssh.service after boot shows whether the ordering held.