feat(kiosk): self-recover a hung wall and page when it goes quiet

An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel
alive but userspace unable to start a process: ssh completed the handshake and
closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM
lines rule out memory. Separately, the service knew at 14:03Z that nothing was
watching (healthz viewers:0) and nothing was reading it. It was discovered by
walking over to the panel.

Hang recovery, two independent layers (install step 7/9, asserted by
check-kiosk.sh):

- /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any
  oops escalate to a panic, and kernel.panic=30 reboots instead of parking.
- /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking
  the iTCO hardware watchdog or the chipset resets the box, which is the layer that
  covers a kernel too wedged to notice. Clamped to the driver's own 30s because
  systemd kicks at a third of whatever value it is handed.

Presence alerting: docs/wall-watch/ runs on the server host, probes loopback
/healthz (exempt from the address gate) every minute, debounces three checks, pages
over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the
title, body and priority so a drill can never be mistaken for an incident.
Verified against a fake ntfy: caller environment wins over the config file, both
transitions fire, and the debounce holds.

Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half
is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and
stays open; the TCP half was every high port on a box whose only access control is
its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s
inbound with the TCP range closed, and tcp/8080 stopped answering from a
same-subnet host.

Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its
built-in default URL, so re-running it to repair a wall silently repointed it at
the production default. An existing URL now wins and KIOSK_URL= is an explicit
change.

Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk
session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to
capture, and this image has grdctl rather than grdc), and why no userspace RDP
would have survived the hang either.

LLM-Generated: true
This commit is contained in:
2026-11-02 09:53:13 -05:00
parent bf027672b5
commit 8fefd83ef8
12 changed files with 796 additions and 11 deletions
@@ -0,0 +1,29 @@
# 90-mirror-wall-watchdog.conf — PID 1 must kick the hardware watchdog.
#
# Installed by install-kiosk.sh to /etc/systemd/system.conf.d/, but ONLY when the
# box actually has a watchdog device (/dev/watchdog). Takes effect on
# `systemctl daemon-reexec`, which the installer runs; a reboot also picks it up.
#
# This is the layer that catches what the sysctls cannot. The soft/hard lockup
# detectors need kernel work to keep happening; a hardware watchdog needs the
# opposite — if PID 1 stops writing /dev/watchdog0 for one whole timeout, the
# chipset resets the machine no matter what the kernel thinks it is doing. That
# includes the case we hit on 2026-11-02, where the box answered ARP and TCP
# handshakes from kernel state while no userspace process could be started.
#
# 30s is not an arbitrary number: it is the timeout iTCO_wdt initialised itself
# with ("heartbeat=30 sec"). systemd kicks at a third of the configured timeout,
# so 30s means a kick every 10s. Asking for more than the device supports would
# make systemd kick less often than the hardware expires — a self-inflicted
# reboot loop — so install-kiosk.sh keeps this at or below the driver's value.
#
# Cost: a PID-1 stall longer than 30s (a genuinely stuck disk, say) reboots the
# wall. On an unattended display that is the intended behaviour. To back off,
# raise the number to something under the device timeout, or set 0 and rely on
# the sysctls alone:
#
# printf '[Manager]\nRuntimeWatchdogSec=0\n' | sudo tee /etc/systemd/system.conf.d/91-mirror-wall-watchdog-off.conf
# sudo systemctl daemon-reexec
[Manager]
RuntimeWatchdogSec=30s