feat(kiosk): self-recover a hung wall and page when it goes quiet
An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel alive but userspace unable to start a process: ssh completed the handshake and closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM lines rule out memory. Separately, the service knew at 14:03Z that nothing was watching (healthz viewers:0) and nothing was reading it. It was discovered by walking over to the panel. Hang recovery, two independent layers (install step 7/9, asserted by check-kiosk.sh): - /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any oops escalate to a panic, and kernel.panic=30 reboots instead of parking. - /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking the iTCO hardware watchdog or the chipset resets the box, which is the layer that covers a kernel too wedged to notice. Clamped to the driver's own 30s because systemd kicks at a third of whatever value it is handed. Presence alerting: docs/wall-watch/ runs on the server host, probes loopback /healthz (exempt from the address gate) every minute, debounces three checks, pages over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the title, body and priority so a drill can never be mistaken for an incident. Verified against a fake ntfy: caller environment wins over the config file, both transitions fire, and the debounce holds. Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and stays open; the TCP half was every high port on a box whose only access control is its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s inbound with the TCP range closed, and tcp/8080 stopped answering from a same-subnet host. Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its built-in default URL, so re-running it to repair a wall silently repointed it at the production default. An existing URL now wins and KIOSK_URL= is an explicit change. Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to capture, and this image has grdctl rather than grdc), and why no userspace RDP would have survived the hang either. LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,37 @@
|
||||
# 90-mirror-wall-selfheal.conf — make a hung wall recover on its own.
|
||||
#
|
||||
# Installed by install-kiosk.sh to /etc/sysctl.d/. Applied at boot by
|
||||
# systemd-sysctl, and immediately by `sysctl --system` in the installer.
|
||||
#
|
||||
# The trade this makes: a wall whose kernel has oopsed, deadlocked or soft-locked
|
||||
# is a panel showing a frozen frame forever, in a room nobody is in. Rebooting
|
||||
# into itself is always the better outcome, so every one of these escalates to a
|
||||
# reboot. Reasoning and the incident behind it: ../kiosk-silverblue.md
|
||||
# ("When the wall hangs").
|
||||
#
|
||||
# To revert a single line: drop a file with a higher-numbered name in the same
|
||||
# directory, e.g. /etc/sysctl.d/91-mirror-wall-quiet.conf containing
|
||||
# "kernel.panic_on_oops = 0", then `sudo sysctl --system`.
|
||||
|
||||
# The lockup detectors themselves (default on, pinned here so a future base
|
||||
# image change cannot silently turn them off).
|
||||
kernel.watchdog = 1
|
||||
|
||||
# A soft lockup (a CPU spinning in kernel mode with preemption off) or a hard
|
||||
# lockup (interrupts disabled; caught by the NMI watchdog) panics instead of
|
||||
# only printing "watchdog: BUG: soft lockup" into a journal nobody reads.
|
||||
kernel.softlockup_panic = 1
|
||||
kernel.hardlockup_panic = 1
|
||||
|
||||
# An oops used to leave the box nominally up with its display driver dead — the
|
||||
# worst case, because PID 1 stays healthy and so nothing else would reset it.
|
||||
kernel.panic_on_oops = 1
|
||||
|
||||
# Panic, then reboot 30s later. Long enough for the console to be read and for
|
||||
# the filesystems to be sync'ed by the panic path; short enough that nobody
|
||||
# waits for it. 0 would mean "sit there forever", which is the bug.
|
||||
kernel.panic = 30
|
||||
|
||||
# Not set here on purpose: kernel.hung_task_panic does not exist in the Fedora
|
||||
# 44 kernel build (CONFIG_DETECT_HUNG_TASK is off). A task blocked on I/O is
|
||||
# caught by the watchdog in 90-mirror-wall-watchdog.conf instead.
|
||||
Reference in New Issue
Block a user