feat(kiosk): self-recover a hung wall and page when it goes quiet
An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel alive but userspace unable to start a process: ssh completed the handshake and closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM lines rule out memory. Separately, the service knew at 14:03Z that nothing was watching (healthz viewers:0) and nothing was reading it. It was discovered by walking over to the panel. Hang recovery, two independent layers (install step 7/9, asserted by check-kiosk.sh): - /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any oops escalate to a panic, and kernel.panic=30 reboots instead of parking. - /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking the iTCO hardware watchdog or the chipset resets the box, which is the layer that covers a kernel too wedged to notice. Clamped to the driver's own 30s because systemd kicks at a third of whatever value it is handed. Presence alerting: docs/wall-watch/ runs on the server host, probes loopback /healthz (exempt from the address gate) every minute, debounces three checks, pages over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the title, body and priority so a drill can never be mistaken for an incident. Verified against a fake ntfy: caller environment wins over the config file, both transitions fire, and the debounce holds. Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and stays open; the TCP half was every high port on a box whose only access control is its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s inbound with the TCP range closed, and tcp/8080 stopped answering from a same-subnet host. Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its built-in default URL, so re-running it to repair a wall silently repointed it at the production default. An existing URL now wins and KIOSK_URL= is an explicit change. Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to capture, and this image has grdctl rather than grdc), and why no userspace RDP would have survived the hang either. LLM-Generated: true
This commit is contained in:
@@ -157,6 +157,53 @@ if [ -n "$HOME_DIR" ]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
printf '\n%s\n' '--- hang recovery ---'
|
||||
# An unattended wall has to end a hang with a reboot, not with a frozen frame for
|
||||
# a week. Two independent layers; ../kiosk-silverblue.md ("When the wall hangs").
|
||||
sc() { sysctl -n "$1" 2>/dev/null || echo missing; }
|
||||
for k in kernel.softlockup_panic kernel.hardlockup_panic kernel.panic_on_oops; do
|
||||
v=$(sc "$k")
|
||||
case "$v" in
|
||||
1) ok "$k = 1" ;;
|
||||
missing) warn "$k does not exist in this kernel build"
|
||||
warn " (kernel.hung_task_panic is absent from Fedora's kernel; the watchdog covers it)" ;;
|
||||
*) bad "$k = $v, expected 1 — fix /etc/sysctl.d/90-mirror-wall-selfheal.conf, then sudo sysctl --system" ;;
|
||||
esac
|
||||
done
|
||||
P=$(sc kernel.panic)
|
||||
case "$P" in
|
||||
''|0|missing) bad "kernel.panic = ${P:-unset}: after a panic the box sits there forever" ;;
|
||||
*) ok "kernel.panic = $P (reboots $P s after a panic)" ;;
|
||||
esac
|
||||
if [ -e /dev/watchdog ]; then
|
||||
WD=$(systemctl show -p RuntimeWatchdogUSec --value 2>/dev/null || echo 0)
|
||||
WID=$(cat /sys/class/watchdog/watchdog0/identity 2>/dev/null || echo '?')
|
||||
case "$WD" in
|
||||
0|''|infinity) bad "PID 1 is not kicking the $WID watchdog (RuntimeWatchdogUSec=$WD) — a wedged kernel here never recovers" ;;
|
||||
*) ok "hardware watchdog armed: $WID at $WD (no kick from PID 1 for $WD = chipset reset)" ;;
|
||||
esac
|
||||
else
|
||||
info "no /dev/watchdog on this box — the lockup panics are the only recovery layer"
|
||||
fi
|
||||
|
||||
printf '\n%s\n' '--- firewall ---'
|
||||
if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then
|
||||
ZONE=$(firewall-cmd --get-default-zone)
|
||||
PORTS=$(firewall-cmd --zone="$ZONE" --list-ports | tr ' ' '\n')
|
||||
if printf '%s\n' "$PORTS" | grep -qx '1025-65535/tcp'; then
|
||||
bad "$ZONE opens 1025-65535/tcp — on a box whose only access control is its address, that is every high port"
|
||||
else
|
||||
ok "$ZONE: 1025-65535/tcp closed"
|
||||
fi
|
||||
if printf '%s\n' "$PORTS" | grep -qx '1025-65535/udp'; then
|
||||
ok "$ZONE: 1025-65535/udp open (WebRTC media needs it — do not close it)"
|
||||
else
|
||||
bad "$ZONE: 1025-65535/udp closed — peer-to-peer media cannot flow, every tile reads 'unreachable'"
|
||||
fi
|
||||
else
|
||||
info "firewalld is not running — nothing listening, nothing to tighten"
|
||||
fi
|
||||
|
||||
printf '\n%s\n' '--- what the screen will show ---'
|
||||
if [ -n "$URL" ]; then
|
||||
BODY=$(curl -fsSL -m 8 "$URL" 2>/dev/null)
|
||||
|
||||
Reference in New Issue
Block a user