Compare commits
5
Commits
3f14b8ce14
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f9b5d293ba | ||
|
|
8bcee5c17f | ||
|
|
e94d4f718c | ||
|
|
3cdbb405ab | ||
|
|
bdee581480 |
@@ -0,0 +1,60 @@
|
|||||||
|
# Raspberry Pi — boot-time hardening units
|
||||||
|
|
||||||
|
Two reusable runbooks for a headless Pi that must stay reachable over SSH.
|
||||||
|
Deliberately host-agnostic: **no addresses, key material, or peer identities
|
||||||
|
live in this repo** — those stay on each host. Set `HOST` and fill in the
|
||||||
|
placeholders from the runbook before running anything.
|
||||||
|
|
||||||
|
| Runbook | Solves |
|
||||||
|
| --- | --- |
|
||||||
|
| [`root-ssh-key.md`](root-ssh-key.md) | root `authorized_keys` gets wiped (imaging, config management) → key is re-installed at boot and re-applied on change |
|
||||||
|
| [`wireguard.md`](wireguard.md) | SSH through a WireGuard tunnel address fails after reboot / NAT timeout / roaming to another network |
|
||||||
|
|
||||||
|
Verified against Debian 13 (trixie) with systemd 257 and NetworkManager. The
|
||||||
|
units are plain POSIX `sh` + systemd, so they carry to any other distribution
|
||||||
|
with the same systemd version or newer.
|
||||||
|
|
||||||
|
## Layout → target path on the host
|
||||||
|
|
||||||
|
| File in this directory | Installs to |
|
||||||
|
| --- | --- |
|
||||||
|
| `ensure-root-ssh-key` | `/usr/local/sbin/ensure-root-ssh-key` (`0755`) |
|
||||||
|
| `ensure-root-ssh-key.service` | `/etc/systemd/system/` |
|
||||||
|
| `ensure-root-ssh-key.path` | `/etc/systemd/system/` |
|
||||||
|
| `root-ssh-keys/authorized_keys.example` | `/usr/local/share/root-ssh-keys/authorized_keys` (`0644`) — copy, fill in your public key(s) |
|
||||||
|
| `wg0-watchdog` | `/usr/local/sbin/wg0-watchdog` (`0755`) |
|
||||||
|
| `wg0-watchdog.service` | `/etc/systemd/system/` |
|
||||||
|
| `wg0-watchdog.timer` | `/etc/systemd/system/` |
|
||||||
|
| `wg-quick@wg0.service.d-10-boot-resilience.conf` | `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` |
|
||||||
|
| `wg0.conf.template` | documents `/etc/wireguard/wg0.conf` (`0600`) — the real file is never copied here |
|
||||||
|
|
||||||
|
## Deploy pattern
|
||||||
|
|
||||||
|
Every command in both runbooks follows the same shape, so a fresh Pi needs only
|
||||||
|
`HOST` and the placeholders:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
HOST=root@<pi-address>
|
||||||
|
D=$PWD # run the snippets from a checkout of this directory
|
||||||
|
```
|
||||||
|
|
||||||
|
Files travel over one `tar | ssh` pipe into `/root`, then `install` moves them
|
||||||
|
into place with explicit modes (no `scp` of a whole tree into `/etc`, no
|
||||||
|
ambiguous ownership). Reload and enable once at the end:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'systemctl daemon-reload && systemctl enable --now <units>'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Conventions worth keeping on any host
|
||||||
|
|
||||||
|
- Config files under `/usr/local/{sbin,share}` are the source of truth; systemd
|
||||||
|
units stay thin wrappers so they can be re-imaged independently.
|
||||||
|
- Idempotent installers over clever state: every script re-derives the desired
|
||||||
|
state from its own config file and never deletes what it did not create.
|
||||||
|
- Anything that must heal itself is gap-limited (a minimum interval between
|
||||||
|
restarts) so a broken dependency cannot make a unit flap.
|
||||||
|
- Private keys and addresses are excluded from the repo; when dumping a
|
||||||
|
WireGuard config, redact with a regex that tolerates `PrivateKey = ` spacing
|
||||||
|
(`wg-quick strip` emits spaces, a `PrivateKey=` regex misses them):
|
||||||
|
`sed -E 's/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/'`
|
||||||
+61
@@ -0,0 +1,61 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# ensure-root-ssh-key - idempotently install managed SSH public keys into
|
||||||
|
# /root/.ssh/authorized_keys. Runs at boot (ensure-root-ssh-key.service) and
|
||||||
|
# can be run by hand at any time.
|
||||||
|
#
|
||||||
|
# Source of truth: /usr/local/share/root-ssh-keys/authorized_keys
|
||||||
|
# One key per line, same format as an authorized_keys file. Blank lines and
|
||||||
|
# lines starting with '#' are ignored. Keys already present are left alone;
|
||||||
|
# foreign keys in authorized_keys are never removed.
|
||||||
|
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
KEY_DIR="${KEY_DIR:-/root/.ssh}"
|
||||||
|
AUTH_KEYS="$KEY_DIR/authorized_keys"
|
||||||
|
MANAGED_KEYS="${MANAGED_KEYS:-/usr/local/share/root-ssh-keys/authorized_keys}"
|
||||||
|
|
||||||
|
if [ ! -r "$MANAGED_KEYS" ]; then
|
||||||
|
echo "ensure-root-ssh-key: cannot read $MANAGED_KEYS" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
umask 077
|
||||||
|
install -d -m 0700 "$KEY_DIR"
|
||||||
|
[ -e "$AUTH_KEYS" ] || : > "$AUTH_KEYS"
|
||||||
|
chmod 0600 "$AUTH_KEYS"
|
||||||
|
chown root:root "$KEY_DIR" "$AUTH_KEYS"
|
||||||
|
|
||||||
|
added=0
|
||||||
|
checked=0
|
||||||
|
while IFS= read -r line <&3; do
|
||||||
|
case "$line" in
|
||||||
|
'' | \#*) continue ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# Basic shape check: <algo> <base64> [comment]
|
||||||
|
algo=${line%% *}
|
||||||
|
case "$algo" in
|
||||||
|
ssh-ed25519 | ssh-rsa | ssh-dss | ecdsa-sha2-* | sk-ecdsa-sha2-* | sk-ssh-ed25519-* | ssh-ed448) ;;
|
||||||
|
*)
|
||||||
|
echo "ensure-root-ssh-key: skipping unrecognised line: $line" >&2
|
||||||
|
continue
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
checked=$((checked + 1))
|
||||||
|
if grep -qxF -- "$line" "$AUTH_KEYS" 2>/dev/null; then
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
printf '%s\n' "$line" >> "$AUTH_KEYS"
|
||||||
|
added=$((added + 1))
|
||||||
|
echo "ensure-root-ssh-key: installed key (${algo}, ${#line} chars)"
|
||||||
|
done 3< "$MANAGED_KEYS"
|
||||||
|
|
||||||
|
if [ "$added" -eq 0 ]; then
|
||||||
|
echo "ensure-root-ssh-key: $checked managed key(s) already present in $AUTH_KEYS"
|
||||||
|
else
|
||||||
|
echo "ensure-root-ssh-key: added $added key(s), $checked managed key(s) total"
|
||||||
|
fi
|
||||||
|
|
||||||
|
chmod 0600 "$AUTH_KEYS"
|
||||||
|
exit 0
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Watch /root/.ssh/authorized_keys and re-install the managed root SSH key
|
||||||
|
Documentation=man:sshd(8)
|
||||||
|
After=local-fs.target
|
||||||
|
|
||||||
|
[Path]
|
||||||
|
# PathChanged covers deletion / rename / creation of the file,
|
||||||
|
# PathModified covers in-place edits (key stripped or file truncated).
|
||||||
|
# ensure-root-ssh-key.service only writes when a managed key is missing, so
|
||||||
|
# its own writes settle after one extra no-op run instead of looping.
|
||||||
|
PathChanged=/root/.ssh/authorized_keys
|
||||||
|
PathModified=/root/.ssh/authorized_keys
|
||||||
|
Unit=ensure-root-ssh-key.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Ensure managed SSH public keys are installed in /root/.ssh/authorized_keys
|
||||||
|
Documentation=man:sshd(8)
|
||||||
|
After=local-fs.target
|
||||||
|
Before=ssh.service sshd.service
|
||||||
|
ConditionPathExists=/usr/local/share/root-ssh-keys/authorized_keys
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart=/usr/local/sbin/ensure-root-ssh-key
|
||||||
|
StandardOutput=journal
|
||||||
|
StandardError=journal
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
# Runbook: self-healing root SSH key
|
||||||
|
|
||||||
|
`ensure-root-ssh-key.service` re-installs the managed SSH public key(s) into
|
||||||
|
`/root/.ssh/authorized_keys` on every boot, and `ensure-root-ssh-key.path`
|
||||||
|
re-applies it at runtime whenever that file is edited, deleted, or replaced.
|
||||||
|
Useful when the key gets wiped by imaging, first-boot tooling, or config
|
||||||
|
management — or when you need the key to come back after someone else edits the
|
||||||
|
file.
|
||||||
|
|
||||||
|
## Files on the host
|
||||||
|
|
||||||
|
| Path | Purpose |
|
||||||
|
| --- | --- |
|
||||||
|
| `/etc/systemd/system/ensure-root-ssh-key.service` | `Type=oneshot`, `WantedBy=multi-user.target`, ordered `Before=ssh.service` so the key is in place before the daemon starts |
|
||||||
|
| `/etc/systemd/system/ensure-root-ssh-key.path` | `PathChanged=` + `PathModified=` on `/root/.ssh/authorized_keys`, triggers the service |
|
||||||
|
| `/usr/local/sbin/ensure-root-ssh-key` | Idempotent POSIX `sh` installer |
|
||||||
|
| `/usr/local/share/root-ssh-keys/authorized_keys` | Source of truth — one public key per line, `#` comments allowed |
|
||||||
|
|
||||||
|
## Behaviour
|
||||||
|
|
||||||
|
- Appends any managed key that is missing; never removes or rewrites foreign keys.
|
||||||
|
- Creates `/root/.ssh` (`0700`) and `authorized_keys` (`0600`, root:root) if absent.
|
||||||
|
- Lines whose first field is not a known key algorithm are skipped with a warning.
|
||||||
|
- Skipped, not failed, if the managed keys file is deleted.
|
||||||
|
- Runtime self-healing: deleting, truncating, or renaming `authorized_keys` puts
|
||||||
|
the managed key(s) back within ~1s. That also means a managed key cannot be
|
||||||
|
revoked from the host alone — delete it from
|
||||||
|
`/usr/local/share/root-ssh-keys/authorized_keys` first.
|
||||||
|
|
||||||
|
> Gotcha: systemd has no `PathMissing=` key — it is silently ignored with a
|
||||||
|
> warning in `systemd-analyze verify`. `PathChanged=` is what fires on
|
||||||
|
> delete/rename; `PathModified=` covers in-place appends and truncations. Set both.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
HOST=root@<pi-address>
|
||||||
|
D=$PWD # run the snippets from a checkout of this directory
|
||||||
|
PUB=$HOME/.ssh/id_ed25519.pub # the key that must always work
|
||||||
|
|
||||||
|
ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /usr/local/share/root-ssh-keys'
|
||||||
|
tar -C "$D" -cf - ensure-root-ssh-key ensure-root-ssh-key.service ensure-root-ssh-key.path | \
|
||||||
|
ssh "$HOST" 'tar -xf - -C /root &&
|
||||||
|
install -m 0755 /root/ensure-root-ssh-key /usr/local/sbin/ &&
|
||||||
|
install -m 0644 /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path /etc/systemd/system/ &&
|
||||||
|
rm -f /root/ensure-root-ssh-key /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path &&
|
||||||
|
systemctl daemon-reload'
|
||||||
|
cat "$PUB" | ssh "$HOST" 'install -m 0644 /dev/stdin /usr/local/share/root-ssh-keys/authorized_keys'
|
||||||
|
ssh "$HOST" 'systemctl enable --now ensure-root-ssh-key.service ensure-root-ssh-key.path'
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `root-ssh-keys/authorized_keys.example` instead of `$PUB` when the host
|
||||||
|
should carry more than one key.
|
||||||
|
|
||||||
|
## Verify
|
||||||
|
|
||||||
|
One-shot smoke test that proves both units without touching a live session
|
||||||
|
(existing SSH sessions are unaffected; only new logins during the window would
|
||||||
|
fail):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'cp -a /root/.ssh/authorized_keys /root/.ssh/authorized_keys.testbak
|
||||||
|
rm -f /root/.ssh/authorized_keys
|
||||||
|
systemctl start ensure-root-ssh-key.service && cat /root/.ssh/authorized_keys
|
||||||
|
rm -f /root/.ssh/authorized_keys.testbak'
|
||||||
|
```
|
||||||
|
|
||||||
|
Status and history:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'systemctl is-enabled ensure-root-ssh-key.service ensure-root-ssh-key.path
|
||||||
|
systemctl status ensure-root-ssh-key.service --no-pager | head -8
|
||||||
|
journalctl -u ensure-root-ssh-key.service -b --no-pager | tail'
|
||||||
|
```
|
||||||
|
|
||||||
|
The installer honours `KEY_DIR` and `MANAGED_KEYS`, so the create-if-missing and
|
||||||
|
preserve-foreign-keys paths can be exercised in a scratch directory instead of
|
||||||
|
`/root/.ssh`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'install -d -m 0700 /tmp/k/src && cp /usr/local/share/root-ssh-keys/authorized_keys /tmp/k/src/
|
||||||
|
KEY_DIR=/tmp/k/root MANAGED_KEYS=/tmp/k/src/authorized_keys /usr/local/sbin/ensure-root-ssh-key
|
||||||
|
ls -ld /tmp/k/root /tmp/k/root/authorized_keys; rm -rf /tmp/k'
|
||||||
|
```
|
||||||
|
|
||||||
|
Checklist that has passed on a reference host:
|
||||||
|
|
||||||
|
- `systemd-analyze verify` clean for both units, both `enabled`.
|
||||||
|
- Repeated runs add no duplicates; a wiped file and a deleted file are both restored.
|
||||||
|
- Missing `/root/.ssh` is created `0700`, file `0600`.
|
||||||
|
- Foreign keys survive; no trigger loop once the path unit settles.
|
||||||
|
- Boot ordering: `Before=ssh.service` is honoured inside the boot transaction;
|
||||||
|
prove it after a reboot with `systemd-analyze critical-chain ssh.service`.
|
||||||
|
|
||||||
|
## Manage keys
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'echo "<public key line>" >> /usr/local/share/root-ssh-keys/authorized_keys
|
||||||
|
systemctl restart ensure-root-ssh-key.service'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Uninstall
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'systemctl disable --now ensure-root-ssh-key.path ensure-root-ssh-key.service
|
||||||
|
rm -f /etc/systemd/system/ensure-root-ssh-key.{service,path} /usr/local/sbin/ensure-root-ssh-key
|
||||||
|
rm -rf /usr/local/share/root-ssh-keys
|
||||||
|
systemctl daemon-reload'
|
||||||
|
```
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
# Template for /usr/local/share/root-ssh-keys/authorized_keys on the host.
|
||||||
|
# Keys listed here are (re)installed into /root/.ssh/authorized_keys at boot and
|
||||||
|
# whenever that file changes. Copy to `authorized_keys`, add your public keys,
|
||||||
|
# and keep the real file on the host - do not commit it.
|
||||||
|
#
|
||||||
|
# One key per line, authorized_keys format. Blank lines and lines starting with
|
||||||
|
# '#' are ignored; a line whose first field is not a known key algorithm is
|
||||||
|
# skipped with a warning.
|
||||||
|
#
|
||||||
|
# ssh-ed25519 AAAA<base64> comment@host
|
||||||
|
# ssh-rsa AAAA<base64> comment@host
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf
|
||||||
|
# The stock unit already has After=/Wants=network-online.target nss-lookup.target.
|
||||||
|
# This makes a boot-time failure (DHCP/DNS not ready yet, endpoint host offline)
|
||||||
|
# retry forever instead of leaving the tunnel down until someone logs in.
|
||||||
|
[Unit]
|
||||||
|
# Unlimited retries: do not let the start rate limiter give up on us.
|
||||||
|
StartLimitIntervalSec=0
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
# Fail fast instead of hanging forever on endpoint resolution (stock value is
|
||||||
|
# infinity), then let Restart= try again. Same variable as the stock unit,
|
||||||
|
# later assignment wins.
|
||||||
|
Environment=WG_ENDPOINT_RESOLUTION_RETRIES=60
|
||||||
|
TimeoutStartSec=120
|
||||||
|
TimeoutStopSec=30
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=15
|
||||||
+151
@@ -0,0 +1,151 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# wg0-watchdog - keep the WireGuard client tunnel up so SSH to the tunnel
|
||||||
|
# address works after boot, after roaming to a new network, and after the NAT
|
||||||
|
# mapping or endpoint IP goes stale.
|
||||||
|
#
|
||||||
|
# Decisions, oldest symptom first:
|
||||||
|
# interface or peer gone -> restart wg-quick@wg0.service
|
||||||
|
# no PersistentKeepalive in kernel -> set it (covers a config edited without reload)
|
||||||
|
# handshake older than STALE -> re-point the peer at the endpoint
|
||||||
|
# (re-resolves the hostname, forces a handshake)
|
||||||
|
# handshake older than STALE_RESTART-> restart wg-quick@wg0.service
|
||||||
|
#
|
||||||
|
# Restarts are gap-limited so a broken network cannot make the unit flap.
|
||||||
|
# Never removes peers, addresses, or routes it did not create.
|
||||||
|
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
IFACE="${WG_WD_IFACE:-wg0}"
|
||||||
|
UNIT="${WG_WD_UNIT:-wg-quick@wg0.service}"
|
||||||
|
CONF="${WG_WD_CONF:-/etc/wireguard/$IFACE.conf}"
|
||||||
|
KEEPALIVE="${WG_WD_KEEPALIVE:-25}"
|
||||||
|
STALE="${WG_WD_STALE:-120}" # seconds: poke the endpoint
|
||||||
|
STALE_RESTART="${WG_WD_STALE_RESTART:-300}" # seconds: restart the unit
|
||||||
|
MIN_RESTART_GAP="${WG_WD_MIN_RESTART_GAP:-180}" # seconds between restarts
|
||||||
|
STATE_DIR="${WG_WD_STATE_DIR:-/var/lib/wg0-watchdog}"
|
||||||
|
|
||||||
|
DRY=0
|
||||||
|
[ "${1:-}" = "--dry-run" ] && DRY=1
|
||||||
|
|
||||||
|
now=$(date +%s)
|
||||||
|
log() { printf 'wg0-watchdog: %s\n' "$*"; }
|
||||||
|
run() {
|
||||||
|
if [ "$DRY" -eq 1 ]; then
|
||||||
|
printf 'wg0-watchdog: DRY-RUN would run:'
|
||||||
|
printf ' %s' "$@"
|
||||||
|
printf '\n'
|
||||||
|
else
|
||||||
|
"$@"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
read_state() { cat "$1" 2>/dev/null || echo 0; }
|
||||||
|
write_state() {
|
||||||
|
if [ "$DRY" -eq 1 ]; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
mkdir -p "$STATE_DIR"
|
||||||
|
printf '%s\n' "$2" > "$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
restart_unit() {
|
||||||
|
reason=$1
|
||||||
|
last=$(read_state "$STATE_DIR/last-restart")
|
||||||
|
case "$last" in ''|*[!0-9]*) last=0 ;; esac
|
||||||
|
since=$((now - last))
|
||||||
|
if [ "$since" -lt "$MIN_RESTART_GAP" ]; then
|
||||||
|
log "restart of $UNIT suppressed ($reason); last restart ${since}s ago"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
log "restarting $UNIT ($reason)"
|
||||||
|
write_state "$STATE_DIR/last-restart" "$now"
|
||||||
|
# --no-block: a wg-quick up stuck on endpoint resolution must not wedge the
|
||||||
|
# watchdog; the next run reports whether the restart worked.
|
||||||
|
run systemctl restart --no-block "$UNIT"
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- 1. interface / peer present -------------------------------------------
|
||||||
|
if ! ip link show "$IFACE" >/dev/null 2>&1; then
|
||||||
|
restart_unit "interface $IFACE is missing"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
peer=$(wg show "$IFACE" peers 2>/dev/null | head -n 1)
|
||||||
|
if [ -z "${peer:-}" ]; then
|
||||||
|
restart_unit "no peer configured on $IFACE"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- 2. persistent keepalive present ---------------------------------------
|
||||||
|
live_ka=$(wg show "$IFACE" persistent-keepalive 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
|
||||||
|
[ -n "${live_ka:-}" ] || live_ka=0
|
||||||
|
if [ "$live_ka" != "$KEEPALIVE" ]; then
|
||||||
|
log "persistent-keepalive is $live_ka, setting $KEEPALIVE"
|
||||||
|
run wg set "$IFACE" peer "$peer" persistent-keepalive "$KEEPALIVE"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- 3. handshake age -------------------------------------------------------
|
||||||
|
hs=$(wg show "$IFACE" latest-handshakes 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
|
||||||
|
case "${hs:-}" in ''|*[!0-9]*) hs=0 ;; esac
|
||||||
|
if [ "$hs" -eq 0 ]; then
|
||||||
|
age=999999
|
||||||
|
else
|
||||||
|
age=$((now - hs))
|
||||||
|
fi
|
||||||
|
|
||||||
|
# endpoint currently in use by the kernel, and the cached last-known-good one
|
||||||
|
live_ep=$(wg show "$IFACE" endpoints 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
|
||||||
|
cache_file="$STATE_DIR/endpoint"
|
||||||
|
cached_ep=""
|
||||||
|
[ -f "$cache_file" ] && cached_ep=$(cat "$cache_file" 2>/dev/null || true)
|
||||||
|
|
||||||
|
if [ "$age" -lt "$STALE" ]; then
|
||||||
|
if [ -n "$live_ep" ]; then
|
||||||
|
write_state "$cache_file" "$live_ep"
|
||||||
|
fi
|
||||||
|
log "healthy: handshake ${age}s ago, endpoint ${live_ep:-unknown}, keepalive ${KEEPALIVE}s"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "stale: last handshake ${age}s ago"
|
||||||
|
|
||||||
|
# Prefer the endpoint from the config file, freshly resolved. Fall back to the
|
||||||
|
# kernel's current endpoint, then to the cache - important because a DNS server
|
||||||
|
# from the tunnel config may be reachable only through this very tunnel.
|
||||||
|
target_ep=""
|
||||||
|
conf_ep=$(awk -F= 'tolower($1) ~ /^[[:space:]]*endpoint[[:space:]]*$/ { gsub(/[[:space:]]/, "", $2); print $2 }' "$CONF" 2>/dev/null | tail -n 1)
|
||||||
|
if [ -n "${conf_ep:-}" ]; then
|
||||||
|
host=${conf_ep%:*}
|
||||||
|
port=${conf_ep##*:}
|
||||||
|
case "$host" in
|
||||||
|
*[!0-9.]*)
|
||||||
|
resolved=$(getent hosts "$host" 2>/dev/null | awk '{print $1; exit}' || true)
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
resolved=$host
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
if [ -n "${resolved:-}" ]; then
|
||||||
|
target_ep="$resolved:$port"
|
||||||
|
else
|
||||||
|
log "could not resolve $host (tunnel DNS?), falling back"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
[ -n "$target_ep" ] || target_ep=${live_ep:-${cached_ep:-}}
|
||||||
|
|
||||||
|
if [ "$age" -ge "$STALE_RESTART" ]; then
|
||||||
|
restart_unit "handshake ${age}s old"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ -z "$target_ep" ]; then
|
||||||
|
log "no endpoint to re-point at; will retry next run"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$target_ep" = "$live_ep" ]; then
|
||||||
|
log "endpoint already $target_ep; forcing a handshake"
|
||||||
|
fi
|
||||||
|
log "re-pointing peer at $target_ep"
|
||||||
|
run wg set "$IFACE" peer "$peer" endpoint "$target_ep"
|
||||||
|
exit 0
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Keep the WireGuard wg0 tunnel reachable for SSH
|
||||||
|
Documentation=man:wg-quick(8)
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
# Never let a hung systemctl restart inside the script wedge the watchdog.
|
||||||
|
TimeoutStartSec=120
|
||||||
|
ExecStart=/usr/local/sbin/wg0-watchdog
|
||||||
|
StandardOutput=journal
|
||||||
|
StandardError=journal
|
||||||
|
Nice=10
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Check the WireGuard wg0 tunnel every minute and heal it if needed
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
# First check shortly after boot, then every minute. Persistent=true catches up
|
||||||
|
# if the machine was suspended/offline.
|
||||||
|
OnBootSec=30s
|
||||||
|
OnUnitActiveSec=60s
|
||||||
|
AccuracySec=15s
|
||||||
|
RandomizedDelaySec=5s
|
||||||
|
Persistent=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# wg0.conf template — /etc/wireguard/wg0.conf on the host, mode 0600 root:root.
|
||||||
|
# Never commit a real copy: this file holds the tunnel private key.
|
||||||
|
# Generate the key with `umask 077; wg genkey | tee privatekey | wg pubkey`.
|
||||||
|
|
||||||
|
[Interface]
|
||||||
|
PrivateKey = <tunnel private key>
|
||||||
|
Address = <TUNNEL_ADDR>
|
||||||
|
# Deliberately no `DNS` key: any resolver listed here is pushed as a system
|
||||||
|
# upstream while the tunnel is up, which makes all name resolution depend on the
|
||||||
|
# tunnel. Omitting it keeps the LAN's DHCP resolvers. Debian's wg-quick has no
|
||||||
|
# `DNS = none` - a non-IP value becomes a search domain.
|
||||||
|
|
||||||
|
[Peer]
|
||||||
|
PublicKey = <PEER_PUBKEY>
|
||||||
|
AllowedIPs = <PEER_NETWORK>
|
||||||
|
Endpoint = <WG_SERVER_HOST:PORT>
|
||||||
|
# Required when the client is behind NAT and must be reachable inbound (SSH):
|
||||||
|
# without it the mapping expires and the server cannot initiate. 25s survives
|
||||||
|
# typical 30-60s NAT UDP timeouts.
|
||||||
|
PersistentKeepalive = 25
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
# Runbook: WireGuard client that survives reboot, NAT, and roaming
|
||||||
|
|
||||||
|
For a Pi whose SSH address is a **WireGuard tunnel address** rather than a LAN
|
||||||
|
address: the physical interface is ordinary DHCP and the Pi moves between
|
||||||
|
networks, while the tunnel peer is a reachable server. Reachability then depends
|
||||||
|
on three separate things, each of which fails independently — the unit starting
|
||||||
|
at boot, the NAT mapping staying open, and the peer hostname still resolving.
|
||||||
|
|
||||||
|
Placeholders used below:
|
||||||
|
|
||||||
|
| Placeholder | Meaning |
|
||||||
|
| --- | --- |
|
||||||
|
| `$HOST` | `root@<address you can already reach>` |
|
||||||
|
| `<TUNNEL_ADDR>` | this Pi's tunnel address, e.g. `<addr>/32` |
|
||||||
|
| `<PEER_NETWORK>` | network routed through the tunnel, e.g. `<net>/<cidr>` |
|
||||||
|
| `<WG_SERVER_HOST:PORT>` | tunnel server endpoint |
|
||||||
|
| `<PEER_PUBKEY>` | the server's WireGuard public key |
|
||||||
|
|
||||||
|
## Files on the host
|
||||||
|
|
||||||
|
| Path | Purpose |
|
||||||
|
| --- | --- |
|
||||||
|
| `/etc/wireguard/wg0.conf` | tunnel config, mode `0600` — see `wg0.conf.template`, never stored in this repo |
|
||||||
|
| `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` | retry a failed boot-time `up`: `Restart=on-failure`, `RestartSec=15`, `StartLimitIntervalSec=0`, `WG_ENDPOINT_RESOLUTION_RETRIES=60`, `TimeoutStartSec=120` |
|
||||||
|
| `/usr/local/sbin/wg0-watchdog` | heals the tunnel once a minute |
|
||||||
|
| `/etc/systemd/system/wg0-watchdog.service` | oneshot that runs the script |
|
||||||
|
| `/etc/systemd/system/wg0-watchdog.timer` | `OnBootSec=30s`, `OnUnitActiveSec=60s`, `Persistent=true` |
|
||||||
|
| `/var/lib/wg0-watchdog/endpoint` | last-known-good endpoint IP, used when DNS is unavailable |
|
||||||
|
|
||||||
|
## Failure modes this closes
|
||||||
|
|
||||||
|
1. **No `PersistentKeepalive`.** Behind NAT the server can only send to the Pi
|
||||||
|
while the mapping lives, so once it expires the tunnel looks configured but
|
||||||
|
inbound SSH times out. `PersistentKeepalive = 25` punches UDP state often
|
||||||
|
enough for typical 30-60s NAT timeouts.
|
||||||
|
2. **A boot-time failure is permanent.** `wg-quick up` fails outright if the
|
||||||
|
endpoint hostname does not resolve yet or the server is down, and the unit is
|
||||||
|
not retried — so the tunnel stays dead until someone logs in locally.
|
||||||
|
`Restart=on-failure` with `StartLimitIntervalSec=0` retries forever;
|
||||||
|
`WG_ENDPOINT_RESOLUTION_RETRIES` is lowered from the stock `infinity`
|
||||||
|
(which hangs the unit indefinitely) so a retry can actually happen.
|
||||||
|
3. **Nothing heals a stale tunnel.** Roaming to a new network keeps the old
|
||||||
|
endpoint IP in the kernel, and a peer whose endpoint is unreachable simply
|
||||||
|
stops handshaking without the unit ever noticing, because the unit already
|
||||||
|
"succeeded". The watchdog acts on handshake age.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
HOST=root@<pi-address>
|
||||||
|
D=$PWD # run the snippets from a checkout of this directory
|
||||||
|
|
||||||
|
ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /etc/systemd/system/wg-quick@wg0.service.d'
|
||||||
|
tar -C "$D" -cf - wg0-watchdog wg0-watchdog.service wg0-watchdog.timer \
|
||||||
|
'wg-quick@wg0.service.d-10-boot-resilience.conf' | \
|
||||||
|
ssh "$HOST" 'tar -xf - -C /root &&
|
||||||
|
install -m 0755 /root/wg0-watchdog /usr/local/sbin/ &&
|
||||||
|
install -m 0644 /root/wg0-watchdog.service /root/wg0-watchdog.timer /etc/systemd/system/ &&
|
||||||
|
install -m 0644 "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" \
|
||||||
|
/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf &&
|
||||||
|
rm -f /root/wg0-watchdog /root/wg0-watchdog.service /root/wg0-watchdog.timer \
|
||||||
|
"/root/wg-quick@wg0.service.d-10-boot-resilience.conf" &&
|
||||||
|
systemctl daemon-reload && systemctl enable --now wg-quick@wg0.service wg0-watchdog.timer'
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep `PersistentKeepalive = 25` in `[Peer]` (see `wg0.conf.template`) and check
|
||||||
|
the config file is `0600` — a world-readable `wg0.conf` exposes the tunnel
|
||||||
|
private key to every local account:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'ls -l /etc/wireguard/wg0.conf; wg show wg0 persistent-keepalive'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Watchdog behaviour
|
||||||
|
|
||||||
|
Oldest symptom first, one action per run:
|
||||||
|
|
||||||
|
1. interface or peer gone → `systemctl restart --no-block wg-quick@wg0.service`
|
||||||
|
2. no `PersistentKeepalive` in the kernel → `wg set ... persistent-keepalive <n>`
|
||||||
|
(covers a config edited without a reload)
|
||||||
|
3. handshake older than `WG_WD_STALE` (default 120s) → re-point the peer at the
|
||||||
|
endpoint, preferring a fresh resolution of the hostname from the config file,
|
||||||
|
then the kernel's current endpoint, then the cached IP
|
||||||
|
4. handshake older than `WG_WD_STALE_RESTART` (default 300s) → restart the unit
|
||||||
|
|
||||||
|
Restarts are gap-limited to one per `WG_WD_MIN_RESTART_GAP` (default 180s) so a
|
||||||
|
dead network cannot make the unit flap. `--no-block` keeps a hung `wg-quick up`
|
||||||
|
from wedging the watchdog. Peers, addresses, and routes are only ever added.
|
||||||
|
All thresholds and the interface/state paths are environment overrides, and
|
||||||
|
`--dry-run` prints the decision without acting.
|
||||||
|
|
||||||
|
The endpoint resolution order matters when the tunnel config also pushes a DNS
|
||||||
|
server: that resolver is reachable only through the tunnel, so a dead tunnel can
|
||||||
|
mean the endpoint hostname cannot be resolved exactly when it is needed — hence
|
||||||
|
the cached-IP fallback.
|
||||||
|
|
||||||
|
## DNS
|
||||||
|
|
||||||
|
A `DNS = <resolver>` line in the interface section is pushed through
|
||||||
|
`resolvconf`/`systemd-resolved` while the tunnel is up, which can silently make
|
||||||
|
*all* name resolution on the host depend on the tunnel. If the Pi must resolve
|
||||||
|
names on whatever network it lands on, omit the `DNS` key entirely and let the
|
||||||
|
LAN's DHCP resolvers stand.
|
||||||
|
|
||||||
|
Two gotchas:
|
||||||
|
|
||||||
|
- Debian's `wg-quick` does **not** implement `DNS = none` — any non-IP value
|
||||||
|
becomes a search domain, so the line has to be absent, not set to `none`.
|
||||||
|
- A record already pushed by a previous `wg-quick up` survives the config edit;
|
||||||
|
withdraw it (`resolvconf -d <record>`, name visible in `resolvconf --list`) and
|
||||||
|
confirm the local caching resolver's upstreams before believing it is gone.
|
||||||
|
|
||||||
|
## Taking wg0 down on purpose
|
||||||
|
|
||||||
|
Stop the watchdog first or the tunnel comes straight back:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'systemctl stop wg0-watchdog.timer wg-quick@wg0.service'
|
||||||
|
ssh "$HOST" 'systemctl start wg0-watchdog.timer' # resume
|
||||||
|
```
|
||||||
|
|
||||||
|
## Check on it
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'wg show wg0
|
||||||
|
systemctl status wg-quick@wg0 --no-pager | head -6
|
||||||
|
systemctl list-timers "wg0-watchdog*" --no-pager | head -3
|
||||||
|
journalctl -u wg0-watchdog.service -b --no-pager | tail -15'
|
||||||
|
```
|
||||||
|
|
||||||
|
Dry-run the decision logic without touching the tunnel:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'WG_WD_STATE_DIR=/tmp/wd WG_WD_STALE=0 /usr/local/sbin/wg0-watchdog --dry-run; rm -rf /tmp/wd'
|
||||||
|
```
|
||||||
|
|
||||||
|
Validate an edited config without disturbing the running tunnel:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh "$HOST" 'wg-quick strip /etc/wireguard/wg0.conf | sed -E "s/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/"'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Diagnosis notes (generic, worth checking on any host)
|
||||||
|
|
||||||
|
- `wg-quick@<iface>` can be `enabled` yet `inactive` while the interface exists
|
||||||
|
and passes traffic — it may have been created by hand or taken over by
|
||||||
|
NetworkManager as an assumed profile (`nmcli dev status` shows such a device
|
||||||
|
as `connected (externally)`). Nothing owns it: `systemctl reload` fails with
|
||||||
|
"unit not active", so apply changes with `wg set` or a full restart, and let
|
||||||
|
`systemctl restart` take ownership before trusting a reboot.
|
||||||
|
- `Restart=on-failure` does work for `Type=oneshot` on systemd 257 — worth
|
||||||
|
proving on a new distribution with a throwaway unit under
|
||||||
|
`/run/systemd/system` before relying on it: `Type=oneshot` +
|
||||||
|
`ExecStart=/bin/false`, then check `systemctl show -p NRestarts`.
|
||||||
|
- `systemd-analyze verify <unit>` catches mistyped unit keys, and it only warns —
|
||||||
|
read the output instead of trusting the exit status.
|
||||||
|
- NetworkManager does not fight `wg-quick` as long as no keyfile exists for the
|
||||||
|
interface and its assumed profile keeps `autoconnect no`; check
|
||||||
|
`/etc/NetworkManager/system-connections/` before assuming that.
|
||||||
|
- The real end-to-end test is a reboot: DHCP → endpoint resolution →
|
||||||
|
`wg-quick up` → inbound SSH through the tunnel. `systemd-analyze
|
||||||
|
critical-chain ssh.service` after boot shows whether the ordering held.
|
||||||
@@ -36,7 +36,7 @@ This is my ~/.pi/agent/models.json
|
|||||||
"input": ["text", "image"],
|
"input": ["text", "image"],
|
||||||
"reasoning": true,
|
"reasoning": true,
|
||||||
"contextWindow": 262144,
|
"contextWindow": 262144,
|
||||||
"maxTokens": 32000,
|
"maxTokens": 32768,
|
||||||
"thinkingLevelMap": {
|
"thinkingLevelMap": {
|
||||||
"off": "none",
|
"off": "none",
|
||||||
"minimal": null,
|
"minimal": null,
|
||||||
@@ -52,7 +52,7 @@ This is my ~/.pi/agent/models.json
|
|||||||
"input": ["text", "image"],
|
"input": ["text", "image"],
|
||||||
"reasoning": true,
|
"reasoning": true,
|
||||||
"contextWindow": 262144,
|
"contextWindow": 262144,
|
||||||
"maxTokens": 32000,
|
"maxTokens": 32768,
|
||||||
"thinkingLevelMap": {
|
"thinkingLevelMap": {
|
||||||
"off": "none",
|
"off": "none",
|
||||||
"minimal": null,
|
"minimal": null,
|
||||||
@@ -68,3 +68,72 @@ This is my ~/.pi/agent/models.json
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
This is my ~/.pi/agent/settings.json
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"lastChangelogVersion": "1.3.0",
|
||||||
|
"theme": "dark/dark",
|
||||||
|
"defaultProvider": "reeseapps-aipi",
|
||||||
|
"defaultModel": "turbo",
|
||||||
|
"httpIdleTimeoutMs": 0,
|
||||||
|
"enableInstallTelemetry": false,
|
||||||
|
"doubleEscapeAction": "tree",
|
||||||
|
"images": {
|
||||||
|
"autoResize": false
|
||||||
|
},
|
||||||
|
"editorPaddingX": 0,
|
||||||
|
"defaultThinkingLevel": "xhigh",
|
||||||
|
"hideThinkingBlock": false,
|
||||||
|
"thinkingBudgets": {
|
||||||
|
"minimal": 1024,
|
||||||
|
"low": 4096,
|
||||||
|
"medium": 10240,
|
||||||
|
"xhigh": 65536
|
||||||
|
},
|
||||||
|
"steeringMode": "all"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Installing for sysadmin work
|
||||||
|
|
||||||
|
I would absolutely recommend installing pi under a special user (I call mine
|
||||||
|
`ops`) with limited sudoers access that you tightly control. Pair that with the
|
||||||
|
following AGENTS.md at `/home/ops/AGENTS.md`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
# AGENTS.md
|
||||||
|
|
||||||
|
## Sudo policy
|
||||||
|
|
||||||
|
The `ops` user on this machine has **limited sudoers capabilities**. The exact
|
||||||
|
allowlist is discoverable at any time with:
|
||||||
|
|
||||||
|
sudo -l
|
||||||
|
|
||||||
|
Allowed (NOPASSWD) commands currently include: `hostnamectl hostname`, `reboot`,
|
||||||
|
`dnf update/upgrade/install/remove/clean/makecache/autoremove`,
|
||||||
|
`dnf config-manager addrepo`, `systemctl start/restart/reload/enable`,
|
||||||
|
`firewall-cmd`, the ansible shim, `qemu-img`, `virsh`, `virt-install`,
|
||||||
|
`composer-cli`, `getfacl`, specific `setfacl` invocations,
|
||||||
|
`/usr/local/sbin/osbuild-auto.sh`, and `btrfs scrub`.
|
||||||
|
|
||||||
|
## Rule for agents
|
||||||
|
|
||||||
|
If you (the agent) believe you need a permission that is **not** in the
|
||||||
|
`sudo -l` allowlist — i.e., a command fails with a sudo permission error, or a
|
||||||
|
task requires root access you do not have — you must:
|
||||||
|
|
||||||
|
1. **Stop** and do not attempt workarounds that bypass sudo (no password
|
||||||
|
guessing, no escalating via other users, no trying to write to root-owned
|
||||||
|
paths without permission).
|
||||||
|
2. **Ask the human** to update sudoers with the specific command(s) you need,
|
||||||
|
stating exactly which command(s) and arguments should be added so the user
|
||||||
|
can review and approve them. Requests must be correct and copy-pastable,
|
||||||
|
using full command paths in standard sudoers format, e.g.
|
||||||
|
`ops ALL=(root) NOPASSWD: /usr/sbin/btrfs scrub*`
|
||||||
|
|
||||||
|
Do not proceed with root-only operations until the user has confirmed the
|
||||||
|
sudoers entries have been added.
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user