homelab/active/device_raspberry_pi/systemd/wireguard.md

# Runbook: WireGuard client that survives reboot, NAT, and roaming

For a Pi whose SSH address is a **WireGuard tunnel address** rather than a LAN
address: the physical interface is ordinary DHCP and the Pi moves between
networks, while the tunnel peer is a reachable server. Reachability then depends
on three separate things, each of which fails independently — the unit starting
at boot, the NAT mapping staying open, and the peer hostname still resolving.

Placeholders used below:

| Placeholder | Meaning |
| --- | --- |
| `$HOST` | `root@<address you can already reach>` |
| `<TUNNEL_ADDR>` | this Pi's tunnel address, e.g. `<addr>/32` |
| `<PEER_NETWORK>` | network routed through the tunnel, e.g. `<net>/<cidr>` |
| `<WG_SERVER_HOST:PORT>` | tunnel server endpoint |
| `<PEER_PUBKEY>` | the server's WireGuard public key |

## Files on the host

| Path | Purpose |
| --- | --- |
| `/etc/wireguard/wg0.conf` | tunnel config, mode `0600` — see `wg0.conf.template`, never stored in this repo |
| `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` | retry a failed boot-time `up`: `Restart=on-failure`, `RestartSec=15`, `StartLimitIntervalSec=0`, `WG_ENDPOINT_RESOLUTION_RETRIES=60`, `TimeoutStartSec=120` |
| `/usr/local/sbin/wg0-watchdog` | heals the tunnel once a minute |
| `/etc/systemd/system/wg0-watchdog.service` | oneshot that runs the script |
| `/etc/systemd/system/wg0-watchdog.timer` | `OnBootSec=30s`, `OnUnitActiveSec=60s`, `Persistent=true` |
| `/var/lib/wg0-watchdog/endpoint` | last-known-good endpoint IP, used when DNS is unavailable |

## Failure modes this closes

1. **No `PersistentKeepalive`.** Behind NAT the server can only send to the Pi
   while the mapping lives, so once it expires the tunnel looks configured but
   inbound SSH times out. `PersistentKeepalive = 25` punches UDP state often
   enough for typical 30-60s NAT timeouts.
2. **A boot-time failure is permanent.** `wg-quick up` fails outright if the
   endpoint hostname does not resolve yet or the server is down, and the unit is
   not retried — so the tunnel stays dead until someone logs in locally.
   `Restart=on-failure` with `StartLimitIntervalSec=0` retries forever;
   `WG_ENDPOINT_RESOLUTION_RETRIES` is lowered from the stock `infinity`
   (which hangs the unit indefinitely) so a retry can actually happen.
3. **Nothing heals a stale tunnel.** Roaming to a new network keeps the old
   endpoint IP in the kernel, and a peer whose endpoint is unreachable simply
   stops handshaking without the unit ever noticing, because the unit already
   "succeeded". The watchdog acts on handshake age.

## Install

```bash
HOST=root@<pi-address>
D=$PWD        # run the snippets from a checkout of this directory

ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /etc/systemd/system/wg-quick@wg0.service.d'
tar -C "$D" -cf - wg0-watchdog wg0-watchdog.service wg0-watchdog.timer \
    'wg-quick@wg0.service.d-10-boot-resilience.conf' | \
ssh "$HOST" 'tar -xf - -C /root &&
  install -m 0755 /root/wg0-watchdog /usr/local/sbin/ &&
  install -m 0644 /root/wg0-watchdog.service /root/wg0-watchdog.timer /etc/systemd/system/ &&
  install -m 0644 "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" \
    /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf &&
  rm -f /root/wg0-watchdog /root/wg0-watchdog.service /root/wg0-watchdog.timer \
    "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" &&
  systemctl daemon-reload && systemctl enable --now wg-quick@wg0.service wg0-watchdog.timer'
```

Keep `PersistentKeepalive = 25` in `[Peer]` (see `wg0.conf.template`) and check
the config file is `0600` — a world-readable `wg0.conf` exposes the tunnel
private key to every local account:

```bash
ssh "$HOST" 'ls -l /etc/wireguard/wg0.conf; wg show wg0 persistent-keepalive'
```

## Watchdog behaviour

Oldest symptom first, one action per run:

1. interface or peer gone → `systemctl restart --no-block wg-quick@wg0.service`
2. no `PersistentKeepalive` in the kernel → `wg set ... persistent-keepalive <n>`
   (covers a config edited without a reload)
3. handshake older than `WG_WD_STALE` (default 120s) → re-point the peer at the
   endpoint, preferring a fresh resolution of the hostname from the config file,
   then the kernel's current endpoint, then the cached IP
4. handshake older than `WG_WD_STALE_RESTART` (default 300s) → restart the unit

Restarts are gap-limited to one per `WG_WD_MIN_RESTART_GAP` (default 180s) so a
dead network cannot make the unit flap. `--no-block` keeps a hung `wg-quick up`
from wedging the watchdog. Peers, addresses, and routes are only ever added.
All thresholds and the interface/state paths are environment overrides, and
`--dry-run` prints the decision without acting.

The endpoint resolution order matters when the tunnel config also pushes a DNS
server: that resolver is reachable only through the tunnel, so a dead tunnel can
mean the endpoint hostname cannot be resolved exactly when it is needed — hence
the cached-IP fallback.

## DNS

A `DNS = <resolver>` line in the interface section is pushed through
`resolvconf`/`systemd-resolved` while the tunnel is up, which can silently make
*all* name resolution on the host depend on the tunnel. If the Pi must resolve
names on whatever network it lands on, omit the `DNS` key entirely and let the
LAN's DHCP resolvers stand.

Two gotchas:

- Debian's `wg-quick` does **not** implement `DNS = none` — any non-IP value
  becomes a search domain, so the line has to be absent, not set to `none`.
- A record already pushed by a previous `wg-quick up` survives the config edit;
  withdraw it (`resolvconf -d <record>`, name visible in `resolvconf --list`) and
  confirm the local caching resolver's upstreams before believing it is gone.

## Taking wg0 down on purpose

Stop the watchdog first or the tunnel comes straight back:

```bash
ssh "$HOST" 'systemctl stop wg0-watchdog.timer wg-quick@wg0.service'
ssh "$HOST" 'systemctl start wg0-watchdog.timer'      # resume
```

## Check on it

```bash
ssh "$HOST" 'wg show wg0
  systemctl status wg-quick@wg0 --no-pager | head -6
  systemctl list-timers "wg0-watchdog*" --no-pager | head -3
  journalctl -u wg0-watchdog.service -b --no-pager | tail -15'
```

Dry-run the decision logic without touching the tunnel:

```bash
ssh "$HOST" 'WG_WD_STATE_DIR=/tmp/wd WG_WD_STALE=0 /usr/local/sbin/wg0-watchdog --dry-run; rm -rf /tmp/wd'
```

Validate an edited config without disturbing the running tunnel:

```bash
ssh "$HOST" 'wg-quick strip /etc/wireguard/wg0.conf | sed -E "s/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/"'
```

## Diagnosis notes (generic, worth checking on any host)

- `wg-quick@<iface>` can be `enabled` yet `inactive` while the interface exists
  and passes traffic — it may have been created by hand or taken over by
  NetworkManager as an assumed profile (`nmcli dev status` shows such a device
  as `connected (externally)`). Nothing owns it: `systemctl reload` fails with
  "unit not active", so apply changes with `wg set` or a full restart, and let
  `systemctl restart` take ownership before trusting a reboot.
- `Restart=on-failure` does work for `Type=oneshot` on systemd 257 — worth
  proving on a new distribution with a throwaway unit under
  `/run/systemd/system` before relying on it: `Type=oneshot` +
  `ExecStart=/bin/false`, then check `systemctl show -p NRestarts`.
- `systemd-analyze verify <unit>` catches mistyped unit keys, and it only warns —
  read the output instead of trusting the exit status.
- NetworkManager does not fight `wg-quick` as long as no keyfile exists for the
  interface and its assumed profile keeps `autoconnect no`; check
  `/etc/NetworkManager/system-connections/` before assuming that.
- The real end-to-end test is a reboot: DHCP → endpoint resolution →
  `wg-quick up` → inbound SSH through the tunnel. `systemd-analyze
  critical-chain ssh.service` after boot shows whether the ordering held.