feat(kiosk): self-recover a hung wall and page when it goes quiet
An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel alive but userspace unable to start a process: ssh completed the handshake and closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM lines rule out memory. Separately, the service knew at 14:03Z that nothing was watching (healthz viewers:0) and nothing was reading it. It was discovered by walking over to the panel. Hang recovery, two independent layers (install step 7/9, asserted by check-kiosk.sh): - /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any oops escalate to a panic, and kernel.panic=30 reboots instead of parking. - /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking the iTCO hardware watchdog or the chipset resets the box, which is the layer that covers a kernel too wedged to notice. Clamped to the driver's own 30s because systemd kicks at a third of whatever value it is handed. Presence alerting: docs/wall-watch/ runs on the server host, probes loopback /healthz (exempt from the address gate) every minute, debounces three checks, pages over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the title, body and priority so a drill can never be mistaken for an incident. Verified against a fake ntfy: caller environment wins over the config file, both transitions fire, and the debounce holds. Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and stays open; the TCP half was every high port on a box whose only access control is its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s inbound with the TCP range closed, and tcp/8080 stopped answering from a same-subnet host. Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its built-in default URL, so re-running it to repair a wall silently repointed it at the production default. An existing URL now wins and KIOSK_URL= is an explicit change. Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to capture, and this image has grdctl rather than grdc), and why no userspace RDP would have survived the hang either. LLM-Generated: true
This commit is contained in:
@@ -71,7 +71,7 @@ Environment variables (all optional; a `.env` file is loaded if present):
|
||||
| `MIRROR_MAX_MIRRORS` | `0` | Cap on concurrent mirrors (`0` = unlimited). |
|
||||
| `MIRROR_MAX_BITRATE` | `25000000` | Per-leg WebRTC bitrate cap in bits/s — the mesh uploads N × bitrate per sharer, so this is the ceiling. 25 Mbit/s suits a wired LAN at 1440p; a Wi-Fi sharer with many viewers may need less, not more. |
|
||||
| `MIRROR_PING_INTERVAL` | `20` | Seconds between socket liveness pings. |
|
||||
| `MIRROR_ALLOW_CIDRS` | *(unset)* | Comma-separated source networks allowed to connect, e.g. `10.2.0.0/22,fd00::/64`. Unset means unrestricted. |
|
||||
| `MIRROR_ALLOW_CIDRS` | *(unset)* | Comma-separated source networks allowed to connect, e.g. `10.2.0.0/22,fd00::/64`. Unset means unrestricted. A wall needs this range too, and its address is DHCP — give it a reservation. |
|
||||
| `MIRROR_TRUSTED_PROXIES` | `127.0.0.0/8, ::1/128` | Which peers may speak for a client via `X-Forwarded-For`. Only a proxy that overwrites the header belongs here. |
|
||||
| `DEBUGPY` | `0` | `1` opens a debugpy listener on `DEBUGPY_PORT`. |
|
||||
| `DEBUGPY_PORT` | `5678` | debugpy port. |
|
||||
@@ -147,6 +147,13 @@ peer state for the media, and a probe page for the rendered resolution. The
|
||||
runbook records why each file lives where it does on an
|
||||
otherwise read-only OS.
|
||||
|
||||
**When it hangs.** A wall is unattended, so a hang has to end without a person:
|
||||
the kernel turns lockups and oopses into a reboot, and PID 1 must keep kicking the
|
||||
hardware watchdog or the chipset resets the box. Being *quiet* is a different
|
||||
failure than being hung, so the server watches `/healthz` for that and pages over
|
||||
ntfy ([`docs/wall-watch/`](docs/wall-watch/)). Both are installed by the commands
|
||||
above and asserted by `check-kiosk.sh`.
|
||||
|
||||
**Keys.** The whole wall works from a keyboard — and the box needs no keyboard
|
||||
attached to be operated *or* debugged; the keys are there when one is plugged
|
||||
in. On `/watch`:
|
||||
|
||||
Reference in New Issue
Block a user