feat(kiosk): self-recover a hung wall and page when it goes quiet

An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel
alive but userspace unable to start a process: ssh completed the handshake and
closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM
lines rule out memory. Separately, the service knew at 14:03Z that nothing was
watching (healthz viewers:0) and nothing was reading it. It was discovered by
walking over to the panel.

Hang recovery, two independent layers (install step 7/9, asserted by
check-kiosk.sh):

- /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any
  oops escalate to a panic, and kernel.panic=30 reboots instead of parking.
- /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking
  the iTCO hardware watchdog or the chipset resets the box, which is the layer that
  covers a kernel too wedged to notice. Clamped to the driver's own 30s because
  systemd kicks at a third of whatever value it is handed.

Presence alerting: docs/wall-watch/ runs on the server host, probes loopback
/healthz (exempt from the address gate) every minute, debounces three checks, pages
over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the
title, body and priority so a drill can never be mistaken for an incident.
Verified against a fake ntfy: caller environment wins over the config file, both
transitions fire, and the debounce holds.

Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half
is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and
stays open; the TCP half was every high port on a box whose only access control is
its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s
inbound with the TCP range closed, and tcp/8080 stopped answering from a
same-subnet host.

Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its
built-in default URL, so re-running it to repair a wall silently repointed it at
the production default. An existing URL now wins and KIOSK_URL= is an explicit
change.

Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk
session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to
capture, and this image has grdctl rather than grdc), and why no userspace RDP
would have survived the hang either.

LLM-Generated: true
This commit is contained in:
2026-11-02 09:53:13 -05:00
parent bf027672b5
commit 8fefd83ef8
12 changed files with 796 additions and 11 deletions
+47
View File
@@ -157,6 +157,53 @@ if [ -n "$HOME_DIR" ]; then
fi
fi
printf '\n%s\n' '--- hang recovery ---'
# An unattended wall has to end a hang with a reboot, not with a frozen frame for
# a week. Two independent layers; ../kiosk-silverblue.md ("When the wall hangs").
sc() { sysctl -n "$1" 2>/dev/null || echo missing; }
for k in kernel.softlockup_panic kernel.hardlockup_panic kernel.panic_on_oops; do
v=$(sc "$k")
case "$v" in
1) ok "$k = 1" ;;
missing) warn "$k does not exist in this kernel build"
warn " (kernel.hung_task_panic is absent from Fedora's kernel; the watchdog covers it)" ;;
*) bad "$k = $v, expected 1 — fix /etc/sysctl.d/90-mirror-wall-selfheal.conf, then sudo sysctl --system" ;;
esac
done
P=$(sc kernel.panic)
case "$P" in
''|0|missing) bad "kernel.panic = ${P:-unset}: after a panic the box sits there forever" ;;
*) ok "kernel.panic = $P (reboots $P s after a panic)" ;;
esac
if [ -e /dev/watchdog ]; then
WD=$(systemctl show -p RuntimeWatchdogUSec --value 2>/dev/null || echo 0)
WID=$(cat /sys/class/watchdog/watchdog0/identity 2>/dev/null || echo '?')
case "$WD" in
0|''|infinity) bad "PID 1 is not kicking the $WID watchdog (RuntimeWatchdogUSec=$WD) — a wedged kernel here never recovers" ;;
*) ok "hardware watchdog armed: $WID at $WD (no kick from PID 1 for $WD = chipset reset)" ;;
esac
else
info "no /dev/watchdog on this box — the lockup panics are the only recovery layer"
fi
printf '\n%s\n' '--- firewall ---'
if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then
ZONE=$(firewall-cmd --get-default-zone)
PORTS=$(firewall-cmd --zone="$ZONE" --list-ports | tr ' ' '\n')
if printf '%s\n' "$PORTS" | grep -qx '1025-65535/tcp'; then
bad "$ZONE opens 1025-65535/tcp — on a box whose only access control is its address, that is every high port"
else
ok "$ZONE: 1025-65535/tcp closed"
fi
if printf '%s\n' "$PORTS" | grep -qx '1025-65535/udp'; then
ok "$ZONE: 1025-65535/udp open (WebRTC media needs it — do not close it)"
else
bad "$ZONE: 1025-65535/udp closed — peer-to-peer media cannot flow, every tile reads 'unreachable'"
fi
else
info "firewalld is not running — nothing listening, nothing to tighten"
fi
printf '\n%s\n' '--- what the screen will show ---'
if [ -n "$URL" ]; then
BODY=$(curl -fsSL -m 8 "$URL" 2>/dev/null)