Compare commits

...
5 Commits
Author SHA1 Message Date
ducoterra f9b5d293ba update pi.md with real pi agent config
Sync the documented models.json example with the actual
~/.pi/agent/models.json (maxTokens 32000 -> 32768) and add the
full ~/.pi/agent/settings.json to the doc.

LLM-Generated: true
2026-10-21 00:00:07 -04:00
ducoterra 8bcee5c17f document ops-user pi install and sudoers escalation policy
Adds the sysadmin setup recipe to the pi notes: run pi as a dedicated `ops`
user with a tightly controlled sudoers allowlist instead of root, so an agent
can do host work without unbounded privilege.

Includes the AGENTS.md to drop at /home/ops/AGENTS.md. It records that the
allowlist is discovered with `sudo -l` (hostnamectl, reboot, the dnf and
systemctl subcommands, firewall-cmd, the ansible shim, libvirt/virsh and
composer-cli, getfacl/setfacl, osbuild-auto.sh, btrfs scrub) and sets the hard
rule for agents: when a command is not on that list, stop, do not attempt
escalation workarounds, and ask the human to add the exact full-path sudoers
line for review before proceeding.

LLM-Generated: true
2026-10-17 12:26:40 -04:00
ducoterra e94d4f718c add README index for the Raspberry Pi systemd runbooks
Ties the two runbooks together as one host-agnostic directory: which runbook
solves which problem, a file -> target-path map for the units, and the shared
deploy pattern (one tar | ssh pipe into /root, then install with explicit
modes rather than copying a whole tree into /etc).

States the conventions both units follow - configs under /usr/local as the
source of truth with thin systemd wrappers, idempotent installers that re-derive
desired state and never delete what they did not create, and gap-limited
self-healing so a broken dependency cannot make a unit flap.

Also documents the redaction recipe for pasting a WireGuard config: match
`PrivateKey[[:space:]]*=` because wg-quick strip emits spaces around the `=`,
which a `PrivateKey=` regex misses.

LLM-Generated: true
2026-10-16 09:08:16 -04:00
ducoterra 3cdbb405ab add WireGuard wg0 watchdog units for always-on SSH
When SSH to a Pi targets a WireGuard tunnel address, reachability depends on
three things that each fail independently: the unit starting at boot, the NAT
mapping staying open, and the peer hostname still resolving. Rebooting onto a
network with no route to the endpoint previously meant no way back in.

- wg-quick@wg0.service.d/10-boot-resilience.conf: retry a failed boot-time up
  forever (Restart=on-failure, RestartSec=15, StartLimitIntervalSec=0) and cap
  endpoint resolution (WG_ENDPOINT_RESOLUTION_RETRIES=60, TimeoutStartSec=120)
  instead of hanging on the stock infinity.
- wg0-watchdog: POSIX sh healer, oldest symptom first - missing interface or
  peer restarts the unit, a missing PersistentKeepalive is set directly, a
  stale handshake re-points the peer at a freshly resolved endpoint, and a
  very stale one restarts. Restarts are gap-limited so a broken network cannot
  make the unit flap, endpoint resolution falls back to the kernel endpoint
  and then to a last-known-good cache (tunnel DNS may only be reachable
  through the tunnel), and nothing it did not create is ever removed.
  Supports --dry-run.
- wg0-watchdog.service/.timer: oneshot on a 60s timer, first check 30s after
  boot, Persistent=true to catch up after downtime, TimeoutStartSec=120 and
  Nice=10 so a restart cannot wedge the watchdog.
- wg0.conf.template: documents /etc/wireguard/wg0.conf with placeholders only.
  Deliberately omits the DNS key so name resolution does not depend on the
  tunnel (Debian wg-quick has no `DNS = none`), and sets PersistentKeepalive
  25 to hold the NAT mapping for inbound SSH.
- wireguard.md: runbook for the placeholder table, failure modes, deploy, and
  verification.

The real wg0.conf and key material never enter the repo.

LLM-Generated: true
2026-10-16 09:07:56 -04:00
ducoterra bdee581480 add self-healing root SSH key systemd units
Headless Pis lose /root/.ssh/authorized_keys to imaging, first-boot tooling,
or config management, which locks out the only SSH account that can fix it.

- ensure-root-ssh-key: idempotent POSIX sh installer. Appends managed public
  keys from /usr/local/share/root-ssh-keys/authorized_keys, never removes or
  rewrites foreign keys, skips lines whose first field is not a known key
  algorithm, and enforces 0700 / 0600 root:root on the key dir and file.
- ensure-root-ssh-key.service: oneshot, ordered Before=ssh.service so the key
  lands before the daemon starts; ConditionPathExists skips it when the
  managed keys file is absent.
- ensure-root-ssh-key.path: PathChanged= + PathModified= on authorized_keys so
  deletes, renames, and in-place edits are healed at runtime, not just at boot.
  Writes only when a key is missing, so the unit settles after one no-op run.
- authorized_keys.example: template with commented placeholder keys only; the
  real file stays on the host.
- root-ssh-key.md: runbook with deploy, verification, revocation steps, and
  the gotcha that systemd has no PathMissing= key.

LLM-Generated: true
2026-10-16 09:05:54 -04:00
13 changed files with 719 additions and 2 deletions
@@ -0,0 +1,60 @@
# Raspberry Pi — boot-time hardening units
Two reusable runbooks for a headless Pi that must stay reachable over SSH.
Deliberately host-agnostic: **no addresses, key material, or peer identities
live in this repo** — those stay on each host. Set `HOST` and fill in the
placeholders from the runbook before running anything.
| Runbook | Solves |
| --- | --- |
| [`root-ssh-key.md`](root-ssh-key.md) | root `authorized_keys` gets wiped (imaging, config management) → key is re-installed at boot and re-applied on change |
| [`wireguard.md`](wireguard.md) | SSH through a WireGuard tunnel address fails after reboot / NAT timeout / roaming to another network |
Verified against Debian 13 (trixie) with systemd 257 and NetworkManager. The
units are plain POSIX `sh` + systemd, so they carry to any other distribution
with the same systemd version or newer.
## Layout → target path on the host
| File in this directory | Installs to |
| --- | --- |
| `ensure-root-ssh-key` | `/usr/local/sbin/ensure-root-ssh-key` (`0755`) |
| `ensure-root-ssh-key.service` | `/etc/systemd/system/` |
| `ensure-root-ssh-key.path` | `/etc/systemd/system/` |
| `root-ssh-keys/authorized_keys.example` | `/usr/local/share/root-ssh-keys/authorized_keys` (`0644`) — copy, fill in your public key(s) |
| `wg0-watchdog` | `/usr/local/sbin/wg0-watchdog` (`0755`) |
| `wg0-watchdog.service` | `/etc/systemd/system/` |
| `wg0-watchdog.timer` | `/etc/systemd/system/` |
| `wg-quick@wg0.service.d-10-boot-resilience.conf` | `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` |
| `wg0.conf.template` | documents `/etc/wireguard/wg0.conf` (`0600`) — the real file is never copied here |
## Deploy pattern
Every command in both runbooks follows the same shape, so a fresh Pi needs only
`HOST` and the placeholders:
```bash
HOST=root@<pi-address>
D=$PWD # run the snippets from a checkout of this directory
```
Files travel over one `tar | ssh` pipe into `/root`, then `install` moves them
into place with explicit modes (no `scp` of a whole tree into `/etc`, no
ambiguous ownership). Reload and enable once at the end:
```bash
ssh "$HOST" 'systemctl daemon-reload && systemctl enable --now <units>'
```
## Conventions worth keeping on any host
- Config files under `/usr/local/{sbin,share}` are the source of truth; systemd
units stay thin wrappers so they can be re-imaged independently.
- Idempotent installers over clever state: every script re-derives the desired
state from its own config file and never deletes what it did not create.
- Anything that must heal itself is gap-limited (a minimum interval between
restarts) so a broken dependency cannot make a unit flap.
- Private keys and addresses are excluded from the repo; when dumping a
WireGuard config, redact with a regex that tolerates `PrivateKey = ` spacing
(`wg-quick strip` emits spaces, a `PrivateKey=` regex misses them):
`sed -E 's/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/'`
+61
View File
@@ -0,0 +1,61 @@
#!/bin/sh
# ensure-root-ssh-key - idempotently install managed SSH public keys into
# /root/.ssh/authorized_keys. Runs at boot (ensure-root-ssh-key.service) and
# can be run by hand at any time.
#
# Source of truth: /usr/local/share/root-ssh-keys/authorized_keys
# One key per line, same format as an authorized_keys file. Blank lines and
# lines starting with '#' are ignored. Keys already present are left alone;
# foreign keys in authorized_keys are never removed.
set -eu
KEY_DIR="${KEY_DIR:-/root/.ssh}"
AUTH_KEYS="$KEY_DIR/authorized_keys"
MANAGED_KEYS="${MANAGED_KEYS:-/usr/local/share/root-ssh-keys/authorized_keys}"
if [ ! -r "$MANAGED_KEYS" ]; then
echo "ensure-root-ssh-key: cannot read $MANAGED_KEYS" >&2
exit 1
fi
umask 077
install -d -m 0700 "$KEY_DIR"
[ -e "$AUTH_KEYS" ] || : > "$AUTH_KEYS"
chmod 0600 "$AUTH_KEYS"
chown root:root "$KEY_DIR" "$AUTH_KEYS"
added=0
checked=0
while IFS= read -r line <&3; do
case "$line" in
'' | \#*) continue ;;
esac
# Basic shape check: <algo> <base64> [comment]
algo=${line%% *}
case "$algo" in
ssh-ed25519 | ssh-rsa | ssh-dss | ecdsa-sha2-* | sk-ecdsa-sha2-* | sk-ssh-ed25519-* | ssh-ed448) ;;
*)
echo "ensure-root-ssh-key: skipping unrecognised line: $line" >&2
continue
;;
esac
checked=$((checked + 1))
if grep -qxF -- "$line" "$AUTH_KEYS" 2>/dev/null; then
continue
fi
printf '%s\n' "$line" >> "$AUTH_KEYS"
added=$((added + 1))
echo "ensure-root-ssh-key: installed key (${algo}, ${#line} chars)"
done 3< "$MANAGED_KEYS"
if [ "$added" -eq 0 ]; then
echo "ensure-root-ssh-key: $checked managed key(s) already present in $AUTH_KEYS"
else
echo "ensure-root-ssh-key: added $added key(s), $checked managed key(s) total"
fi
chmod 0600 "$AUTH_KEYS"
exit 0
@@ -0,0 +1,16 @@
[Unit]
Description=Watch /root/.ssh/authorized_keys and re-install the managed root SSH key
Documentation=man:sshd(8)
After=local-fs.target
[Path]
# PathChanged covers deletion / rename / creation of the file,
# PathModified covers in-place edits (key stripped or file truncated).
# ensure-root-ssh-key.service only writes when a managed key is missing, so
# its own writes settle after one extra no-op run instead of looping.
PathChanged=/root/.ssh/authorized_keys
PathModified=/root/.ssh/authorized_keys
Unit=ensure-root-ssh-key.service
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,15 @@
[Unit]
Description=Ensure managed SSH public keys are installed in /root/.ssh/authorized_keys
Documentation=man:sshd(8)
After=local-fs.target
Before=ssh.service sshd.service
ConditionPathExists=/usr/local/share/root-ssh-keys/authorized_keys
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/ensure-root-ssh-key
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,109 @@
# Runbook: self-healing root SSH key
`ensure-root-ssh-key.service` re-installs the managed SSH public key(s) into
`/root/.ssh/authorized_keys` on every boot, and `ensure-root-ssh-key.path`
re-applies it at runtime whenever that file is edited, deleted, or replaced.
Useful when the key gets wiped by imaging, first-boot tooling, or config
management — or when you need the key to come back after someone else edits the
file.
## Files on the host
| Path | Purpose |
| --- | --- |
| `/etc/systemd/system/ensure-root-ssh-key.service` | `Type=oneshot`, `WantedBy=multi-user.target`, ordered `Before=ssh.service` so the key is in place before the daemon starts |
| `/etc/systemd/system/ensure-root-ssh-key.path` | `PathChanged=` + `PathModified=` on `/root/.ssh/authorized_keys`, triggers the service |
| `/usr/local/sbin/ensure-root-ssh-key` | Idempotent POSIX `sh` installer |
| `/usr/local/share/root-ssh-keys/authorized_keys` | Source of truth — one public key per line, `#` comments allowed |
## Behaviour
- Appends any managed key that is missing; never removes or rewrites foreign keys.
- Creates `/root/.ssh` (`0700`) and `authorized_keys` (`0600`, root:root) if absent.
- Lines whose first field is not a known key algorithm are skipped with a warning.
- Skipped, not failed, if the managed keys file is deleted.
- Runtime self-healing: deleting, truncating, or renaming `authorized_keys` puts
the managed key(s) back within ~1s. That also means a managed key cannot be
revoked from the host alone — delete it from
`/usr/local/share/root-ssh-keys/authorized_keys` first.
> Gotcha: systemd has no `PathMissing=` key — it is silently ignored with a
> warning in `systemd-analyze verify`. `PathChanged=` is what fires on
> delete/rename; `PathModified=` covers in-place appends and truncations. Set both.
## Install
```bash
HOST=root@<pi-address>
D=$PWD # run the snippets from a checkout of this directory
PUB=$HOME/.ssh/id_ed25519.pub # the key that must always work
ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /usr/local/share/root-ssh-keys'
tar -C "$D" -cf - ensure-root-ssh-key ensure-root-ssh-key.service ensure-root-ssh-key.path | \
ssh "$HOST" 'tar -xf - -C /root &&
install -m 0755 /root/ensure-root-ssh-key /usr/local/sbin/ &&
install -m 0644 /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path /etc/systemd/system/ &&
rm -f /root/ensure-root-ssh-key /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path &&
systemctl daemon-reload'
cat "$PUB" | ssh "$HOST" 'install -m 0644 /dev/stdin /usr/local/share/root-ssh-keys/authorized_keys'
ssh "$HOST" 'systemctl enable --now ensure-root-ssh-key.service ensure-root-ssh-key.path'
```
Use `root-ssh-keys/authorized_keys.example` instead of `$PUB` when the host
should carry more than one key.
## Verify
One-shot smoke test that proves both units without touching a live session
(existing SSH sessions are unaffected; only new logins during the window would
fail):
```bash
ssh "$HOST" 'cp -a /root/.ssh/authorized_keys /root/.ssh/authorized_keys.testbak
rm -f /root/.ssh/authorized_keys
systemctl start ensure-root-ssh-key.service && cat /root/.ssh/authorized_keys
rm -f /root/.ssh/authorized_keys.testbak'
```
Status and history:
```bash
ssh "$HOST" 'systemctl is-enabled ensure-root-ssh-key.service ensure-root-ssh-key.path
systemctl status ensure-root-ssh-key.service --no-pager | head -8
journalctl -u ensure-root-ssh-key.service -b --no-pager | tail'
```
The installer honours `KEY_DIR` and `MANAGED_KEYS`, so the create-if-missing and
preserve-foreign-keys paths can be exercised in a scratch directory instead of
`/root/.ssh`:
```bash
ssh "$HOST" 'install -d -m 0700 /tmp/k/src && cp /usr/local/share/root-ssh-keys/authorized_keys /tmp/k/src/
KEY_DIR=/tmp/k/root MANAGED_KEYS=/tmp/k/src/authorized_keys /usr/local/sbin/ensure-root-ssh-key
ls -ld /tmp/k/root /tmp/k/root/authorized_keys; rm -rf /tmp/k'
```
Checklist that has passed on a reference host:
- `systemd-analyze verify` clean for both units, both `enabled`.
- Repeated runs add no duplicates; a wiped file and a deleted file are both restored.
- Missing `/root/.ssh` is created `0700`, file `0600`.
- Foreign keys survive; no trigger loop once the path unit settles.
- Boot ordering: `Before=ssh.service` is honoured inside the boot transaction;
prove it after a reboot with `systemd-analyze critical-chain ssh.service`.
## Manage keys
```bash
ssh "$HOST" 'echo "<public key line>" >> /usr/local/share/root-ssh-keys/authorized_keys
systemctl restart ensure-root-ssh-key.service'
```
## Uninstall
```bash
ssh "$HOST" 'systemctl disable --now ensure-root-ssh-key.path ensure-root-ssh-key.service
rm -f /etc/systemd/system/ensure-root-ssh-key.{service,path} /usr/local/sbin/ensure-root-ssh-key
rm -rf /usr/local/share/root-ssh-keys
systemctl daemon-reload'
```
@@ -0,0 +1,11 @@
# Template for /usr/local/share/root-ssh-keys/authorized_keys on the host.
# Keys listed here are (re)installed into /root/.ssh/authorized_keys at boot and
# whenever that file changes. Copy to `authorized_keys`, add your public keys,
# and keep the real file on the host - do not commit it.
#
# One key per line, authorized_keys format. Blank lines and lines starting with
# '#' are ignored; a line whose first field is not a known key algorithm is
# skipped with a warning.
#
# ssh-ed25519 AAAA<base64> comment@host
# ssh-rsa AAAA<base64> comment@host
@@ -0,0 +1,17 @@
# /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf
# The stock unit already has After=/Wants=network-online.target nss-lookup.target.
# This makes a boot-time failure (DHCP/DNS not ready yet, endpoint host offline)
# retry forever instead of leaving the tunnel down until someone logs in.
[Unit]
# Unlimited retries: do not let the start rate limiter give up on us.
StartLimitIntervalSec=0
[Service]
# Fail fast instead of hanging forever on endpoint resolution (stock value is
# infinity), then let Restart= try again. Same variable as the stock unit,
# later assignment wins.
Environment=WG_ENDPOINT_RESOLUTION_RETRIES=60
TimeoutStartSec=120
TimeoutStopSec=30
Restart=on-failure
RestartSec=15
+151
View File
@@ -0,0 +1,151 @@
#!/bin/sh
# wg0-watchdog - keep the WireGuard client tunnel up so SSH to the tunnel
# address works after boot, after roaming to a new network, and after the NAT
# mapping or endpoint IP goes stale.
#
# Decisions, oldest symptom first:
# interface or peer gone -> restart wg-quick@wg0.service
# no PersistentKeepalive in kernel -> set it (covers a config edited without reload)
# handshake older than STALE -> re-point the peer at the endpoint
# (re-resolves the hostname, forces a handshake)
# handshake older than STALE_RESTART-> restart wg-quick@wg0.service
#
# Restarts are gap-limited so a broken network cannot make the unit flap.
# Never removes peers, addresses, or routes it did not create.
set -eu
IFACE="${WG_WD_IFACE:-wg0}"
UNIT="${WG_WD_UNIT:-wg-quick@wg0.service}"
CONF="${WG_WD_CONF:-/etc/wireguard/$IFACE.conf}"
KEEPALIVE="${WG_WD_KEEPALIVE:-25}"
STALE="${WG_WD_STALE:-120}" # seconds: poke the endpoint
STALE_RESTART="${WG_WD_STALE_RESTART:-300}" # seconds: restart the unit
MIN_RESTART_GAP="${WG_WD_MIN_RESTART_GAP:-180}" # seconds between restarts
STATE_DIR="${WG_WD_STATE_DIR:-/var/lib/wg0-watchdog}"
DRY=0
[ "${1:-}" = "--dry-run" ] && DRY=1
now=$(date +%s)
log() { printf 'wg0-watchdog: %s\n' "$*"; }
run() {
if [ "$DRY" -eq 1 ]; then
printf 'wg0-watchdog: DRY-RUN would run:'
printf ' %s' "$@"
printf '\n'
else
"$@"
fi
}
read_state() { cat "$1" 2>/dev/null || echo 0; }
write_state() {
if [ "$DRY" -eq 1 ]; then
return 0
fi
mkdir -p "$STATE_DIR"
printf '%s\n' "$2" > "$1"
}
restart_unit() {
reason=$1
last=$(read_state "$STATE_DIR/last-restart")
case "$last" in ''|*[!0-9]*) last=0 ;; esac
since=$((now - last))
if [ "$since" -lt "$MIN_RESTART_GAP" ]; then
log "restart of $UNIT suppressed ($reason); last restart ${since}s ago"
return 0
fi
log "restarting $UNIT ($reason)"
write_state "$STATE_DIR/last-restart" "$now"
# --no-block: a wg-quick up stuck on endpoint resolution must not wedge the
# watchdog; the next run reports whether the restart worked.
run systemctl restart --no-block "$UNIT"
}
# --- 1. interface / peer present -------------------------------------------
if ! ip link show "$IFACE" >/dev/null 2>&1; then
restart_unit "interface $IFACE is missing"
exit 0
fi
peer=$(wg show "$IFACE" peers 2>/dev/null | head -n 1)
if [ -z "${peer:-}" ]; then
restart_unit "no peer configured on $IFACE"
exit 0
fi
# --- 2. persistent keepalive present ---------------------------------------
live_ka=$(wg show "$IFACE" persistent-keepalive 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
[ -n "${live_ka:-}" ] || live_ka=0
if [ "$live_ka" != "$KEEPALIVE" ]; then
log "persistent-keepalive is $live_ka, setting $KEEPALIVE"
run wg set "$IFACE" peer "$peer" persistent-keepalive "$KEEPALIVE"
fi
# --- 3. handshake age -------------------------------------------------------
hs=$(wg show "$IFACE" latest-handshakes 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
case "${hs:-}" in ''|*[!0-9]*) hs=0 ;; esac
if [ "$hs" -eq 0 ]; then
age=999999
else
age=$((now - hs))
fi
# endpoint currently in use by the kernel, and the cached last-known-good one
live_ep=$(wg show "$IFACE" endpoints 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
cache_file="$STATE_DIR/endpoint"
cached_ep=""
[ -f "$cache_file" ] && cached_ep=$(cat "$cache_file" 2>/dev/null || true)
if [ "$age" -lt "$STALE" ]; then
if [ -n "$live_ep" ]; then
write_state "$cache_file" "$live_ep"
fi
log "healthy: handshake ${age}s ago, endpoint ${live_ep:-unknown}, keepalive ${KEEPALIVE}s"
exit 0
fi
log "stale: last handshake ${age}s ago"
# Prefer the endpoint from the config file, freshly resolved. Fall back to the
# kernel's current endpoint, then to the cache - important because a DNS server
# from the tunnel config may be reachable only through this very tunnel.
target_ep=""
conf_ep=$(awk -F= 'tolower($1) ~ /^[[:space:]]*endpoint[[:space:]]*$/ { gsub(/[[:space:]]/, "", $2); print $2 }' "$CONF" 2>/dev/null | tail -n 1)
if [ -n "${conf_ep:-}" ]; then
host=${conf_ep%:*}
port=${conf_ep##*:}
case "$host" in
*[!0-9.]*)
resolved=$(getent hosts "$host" 2>/dev/null | awk '{print $1; exit}' || true)
;;
*)
resolved=$host
;;
esac
if [ -n "${resolved:-}" ]; then
target_ep="$resolved:$port"
else
log "could not resolve $host (tunnel DNS?), falling back"
fi
fi
[ -n "$target_ep" ] || target_ep=${live_ep:-${cached_ep:-}}
if [ "$age" -ge "$STALE_RESTART" ]; then
restart_unit "handshake ${age}s old"
exit 0
fi
if [ -z "$target_ep" ]; then
log "no endpoint to re-point at; will retry next run"
exit 0
fi
if [ "$target_ep" = "$live_ep" ]; then
log "endpoint already $target_ep; forcing a handshake"
fi
log "re-pointing peer at $target_ep"
run wg set "$IFACE" peer "$peer" endpoint "$target_ep"
exit 0
@@ -0,0 +1,12 @@
[Unit]
Description=Keep the WireGuard wg0 tunnel reachable for SSH
Documentation=man:wg-quick(8)
[Service]
Type=oneshot
# Never let a hung systemctl restart inside the script wedge the watchdog.
TimeoutStartSec=120
ExecStart=/usr/local/sbin/wg0-watchdog
StandardOutput=journal
StandardError=journal
Nice=10
@@ -0,0 +1,14 @@
[Unit]
Description=Check the WireGuard wg0 tunnel every minute and heal it if needed
[Timer]
# First check shortly after boot, then every minute. Persistent=true catches up
# if the machine was suspended/offline.
OnBootSec=30s
OnUnitActiveSec=60s
AccuracySec=15s
RandomizedDelaySec=5s
Persistent=true
[Install]
WantedBy=timers.target
@@ -0,0 +1,20 @@
# wg0.conf template — /etc/wireguard/wg0.conf on the host, mode 0600 root:root.
# Never commit a real copy: this file holds the tunnel private key.
# Generate the key with `umask 077; wg genkey | tee privatekey | wg pubkey`.
[Interface]
PrivateKey = <tunnel private key>
Address = <TUNNEL_ADDR>
# Deliberately no `DNS` key: any resolver listed here is pushed as a system
# upstream while the tunnel is up, which makes all name resolution depend on the
# tunnel. Omitting it keeps the LAN's DHCP resolvers. Debian's wg-quick has no
# `DNS = none` - a non-IP value becomes a search domain.
[Peer]
PublicKey = <PEER_PUBKEY>
AllowedIPs = <PEER_NETWORK>
Endpoint = <WG_SERVER_HOST:PORT>
# Required when the client is behind NAT and must be reachable inbound (SSH):
# without it the mapping expires and the server cannot initiate. 25s survives
# typical 30-60s NAT UDP timeouts.
PersistentKeepalive = 25
@@ -0,0 +1,162 @@
# Runbook: WireGuard client that survives reboot, NAT, and roaming
For a Pi whose SSH address is a **WireGuard tunnel address** rather than a LAN
address: the physical interface is ordinary DHCP and the Pi moves between
networks, while the tunnel peer is a reachable server. Reachability then depends
on three separate things, each of which fails independently — the unit starting
at boot, the NAT mapping staying open, and the peer hostname still resolving.
Placeholders used below:
| Placeholder | Meaning |
| --- | --- |
| `$HOST` | `root@<address you can already reach>` |
| `<TUNNEL_ADDR>` | this Pi's tunnel address, e.g. `<addr>/32` |
| `<PEER_NETWORK>` | network routed through the tunnel, e.g. `<net>/<cidr>` |
| `<WG_SERVER_HOST:PORT>` | tunnel server endpoint |
| `<PEER_PUBKEY>` | the server's WireGuard public key |
## Files on the host
| Path | Purpose |
| --- | --- |
| `/etc/wireguard/wg0.conf` | tunnel config, mode `0600` — see `wg0.conf.template`, never stored in this repo |
| `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` | retry a failed boot-time `up`: `Restart=on-failure`, `RestartSec=15`, `StartLimitIntervalSec=0`, `WG_ENDPOINT_RESOLUTION_RETRIES=60`, `TimeoutStartSec=120` |
| `/usr/local/sbin/wg0-watchdog` | heals the tunnel once a minute |
| `/etc/systemd/system/wg0-watchdog.service` | oneshot that runs the script |
| `/etc/systemd/system/wg0-watchdog.timer` | `OnBootSec=30s`, `OnUnitActiveSec=60s`, `Persistent=true` |
| `/var/lib/wg0-watchdog/endpoint` | last-known-good endpoint IP, used when DNS is unavailable |
## Failure modes this closes
1. **No `PersistentKeepalive`.** Behind NAT the server can only send to the Pi
while the mapping lives, so once it expires the tunnel looks configured but
inbound SSH times out. `PersistentKeepalive = 25` punches UDP state often
enough for typical 30-60s NAT timeouts.
2. **A boot-time failure is permanent.** `wg-quick up` fails outright if the
endpoint hostname does not resolve yet or the server is down, and the unit is
not retried — so the tunnel stays dead until someone logs in locally.
`Restart=on-failure` with `StartLimitIntervalSec=0` retries forever;
`WG_ENDPOINT_RESOLUTION_RETRIES` is lowered from the stock `infinity`
(which hangs the unit indefinitely) so a retry can actually happen.
3. **Nothing heals a stale tunnel.** Roaming to a new network keeps the old
endpoint IP in the kernel, and a peer whose endpoint is unreachable simply
stops handshaking without the unit ever noticing, because the unit already
"succeeded". The watchdog acts on handshake age.
## Install
```bash
HOST=root@<pi-address>
D=$PWD # run the snippets from a checkout of this directory
ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /etc/systemd/system/wg-quick@wg0.service.d'
tar -C "$D" -cf - wg0-watchdog wg0-watchdog.service wg0-watchdog.timer \
'wg-quick@wg0.service.d-10-boot-resilience.conf' | \
ssh "$HOST" 'tar -xf - -C /root &&
install -m 0755 /root/wg0-watchdog /usr/local/sbin/ &&
install -m 0644 /root/wg0-watchdog.service /root/wg0-watchdog.timer /etc/systemd/system/ &&
install -m 0644 "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" \
/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf &&
rm -f /root/wg0-watchdog /root/wg0-watchdog.service /root/wg0-watchdog.timer \
"/root/wg-quick@wg0.service.d-10-boot-resilience.conf" &&
systemctl daemon-reload && systemctl enable --now wg-quick@wg0.service wg0-watchdog.timer'
```
Keep `PersistentKeepalive = 25` in `[Peer]` (see `wg0.conf.template`) and check
the config file is `0600` — a world-readable `wg0.conf` exposes the tunnel
private key to every local account:
```bash
ssh "$HOST" 'ls -l /etc/wireguard/wg0.conf; wg show wg0 persistent-keepalive'
```
## Watchdog behaviour
Oldest symptom first, one action per run:
1. interface or peer gone → `systemctl restart --no-block wg-quick@wg0.service`
2. no `PersistentKeepalive` in the kernel → `wg set ... persistent-keepalive <n>`
(covers a config edited without a reload)
3. handshake older than `WG_WD_STALE` (default 120s) → re-point the peer at the
endpoint, preferring a fresh resolution of the hostname from the config file,
then the kernel's current endpoint, then the cached IP
4. handshake older than `WG_WD_STALE_RESTART` (default 300s) → restart the unit
Restarts are gap-limited to one per `WG_WD_MIN_RESTART_GAP` (default 180s) so a
dead network cannot make the unit flap. `--no-block` keeps a hung `wg-quick up`
from wedging the watchdog. Peers, addresses, and routes are only ever added.
All thresholds and the interface/state paths are environment overrides, and
`--dry-run` prints the decision without acting.
The endpoint resolution order matters when the tunnel config also pushes a DNS
server: that resolver is reachable only through the tunnel, so a dead tunnel can
mean the endpoint hostname cannot be resolved exactly when it is needed — hence
the cached-IP fallback.
## DNS
A `DNS = <resolver>` line in the interface section is pushed through
`resolvconf`/`systemd-resolved` while the tunnel is up, which can silently make
*all* name resolution on the host depend on the tunnel. If the Pi must resolve
names on whatever network it lands on, omit the `DNS` key entirely and let the
LAN's DHCP resolvers stand.
Two gotchas:
- Debian's `wg-quick` does **not** implement `DNS = none` — any non-IP value
becomes a search domain, so the line has to be absent, not set to `none`.
- A record already pushed by a previous `wg-quick up` survives the config edit;
withdraw it (`resolvconf -d <record>`, name visible in `resolvconf --list`) and
confirm the local caching resolver's upstreams before believing it is gone.
## Taking wg0 down on purpose
Stop the watchdog first or the tunnel comes straight back:
```bash
ssh "$HOST" 'systemctl stop wg0-watchdog.timer wg-quick@wg0.service'
ssh "$HOST" 'systemctl start wg0-watchdog.timer' # resume
```
## Check on it
```bash
ssh "$HOST" 'wg show wg0
systemctl status wg-quick@wg0 --no-pager | head -6
systemctl list-timers "wg0-watchdog*" --no-pager | head -3
journalctl -u wg0-watchdog.service -b --no-pager | tail -15'
```
Dry-run the decision logic without touching the tunnel:
```bash
ssh "$HOST" 'WG_WD_STATE_DIR=/tmp/wd WG_WD_STALE=0 /usr/local/sbin/wg0-watchdog --dry-run; rm -rf /tmp/wd'
```
Validate an edited config without disturbing the running tunnel:
```bash
ssh "$HOST" 'wg-quick strip /etc/wireguard/wg0.conf | sed -E "s/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/"'
```
## Diagnosis notes (generic, worth checking on any host)
- `wg-quick@<iface>` can be `enabled` yet `inactive` while the interface exists
and passes traffic — it may have been created by hand or taken over by
NetworkManager as an assumed profile (`nmcli dev status` shows such a device
as `connected (externally)`). Nothing owns it: `systemctl reload` fails with
"unit not active", so apply changes with `wg set` or a full restart, and let
`systemctl restart` take ownership before trusting a reboot.
- `Restart=on-failure` does work for `Type=oneshot` on systemd 257 — worth
proving on a new distribution with a throwaway unit under
`/run/systemd/system` before relying on it: `Type=oneshot` +
`ExecStart=/bin/false`, then check `systemctl show -p NRestarts`.
- `systemd-analyze verify <unit>` catches mistyped unit keys, and it only warns —
read the output instead of trusting the exit status.
- NetworkManager does not fight `wg-quick` as long as no keyfile exists for the
interface and its assumed profile keeps `autoconnect no`; check
`/etc/NetworkManager/system-connections/` before assuming that.
- The real end-to-end test is a reboot: DHCP → endpoint resolution →
`wg-quick up` → inbound SSH through the tunnel. `systemd-analyze
critical-chain ssh.service` after boot shows whether the ordering held.
+71 -2
View File
@@ -36,7 +36,7 @@ This is my ~/.pi/agent/models.json
"input": ["text", "image"],
"reasoning": true,
"contextWindow": 262144,
"maxTokens": 32000,
"maxTokens": 32768,
"thinkingLevelMap": {
"off": "none",
"minimal": null,
@@ -52,7 +52,7 @@ This is my ~/.pi/agent/models.json
"input": ["text", "image"],
"reasoning": true,
"contextWindow": 262144,
"maxTokens": 32000,
"maxTokens": 32768,
"thinkingLevelMap": {
"off": "none",
"minimal": null,
@@ -68,3 +68,72 @@ This is my ~/.pi/agent/models.json
}
}
```
This is my ~/.pi/agent/settings.json
```json
{
"lastChangelogVersion": "1.3.0",
"theme": "dark/dark",
"defaultProvider": "reeseapps-aipi",
"defaultModel": "turbo",
"httpIdleTimeoutMs": 0,
"enableInstallTelemetry": false,
"doubleEscapeAction": "tree",
"images": {
"autoResize": false
},
"editorPaddingX": 0,
"defaultThinkingLevel": "xhigh",
"hideThinkingBlock": false,
"thinkingBudgets": {
"minimal": 1024,
"low": 4096,
"medium": 10240,
"xhigh": 65536
},
"steeringMode": "all"
}
```
## Installing for sysadmin work
I would absolutely recommend installing pi under a special user (I call mine
`ops`) with limited sudoers access that you tightly control. Pair that with the
following AGENTS.md at `/home/ops/AGENTS.md`:
```text
# AGENTS.md
## Sudo policy
The `ops` user on this machine has **limited sudoers capabilities**. The exact
allowlist is discoverable at any time with:
sudo -l
Allowed (NOPASSWD) commands currently include: `hostnamectl hostname`, `reboot`,
`dnf update/upgrade/install/remove/clean/makecache/autoremove`,
`dnf config-manager addrepo`, `systemctl start/restart/reload/enable`,
`firewall-cmd`, the ansible shim, `qemu-img`, `virsh`, `virt-install`,
`composer-cli`, `getfacl`, specific `setfacl` invocations,
`/usr/local/sbin/osbuild-auto.sh`, and `btrfs scrub`.
## Rule for agents
If you (the agent) believe you need a permission that is **not** in the
`sudo -l` allowlist — i.e., a command fails with a sudo permission error, or a
task requires root access you do not have — you must:
1. **Stop** and do not attempt workarounds that bypass sudo (no password
guessing, no escalating via other users, no trying to write to root-owned
paths without permission).
2. **Ask the human** to update sudoers with the specific command(s) you need,
stating exactly which command(s) and arguments should be added so the user
can review and approve them. Requests must be correct and copy-pastable,
using full command paths in standard sudoers format, e.g.
`ops ALL=(root) NOPASSWD: /usr/sbin/btrfs scrub*`
Do not proceed with root-only operations until the user has confirmed the
sudoers entries have been added.
```