feat(kiosk): self-recover a hung wall and page when it goes quiet
An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel alive but userspace unable to start a process: ssh completed the handshake and closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM lines rule out memory. Separately, the service knew at 14:03Z that nothing was watching (healthz viewers:0) and nothing was reading it. It was discovered by walking over to the panel. Hang recovery, two independent layers (install step 7/9, asserted by check-kiosk.sh): - /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any oops escalate to a panic, and kernel.panic=30 reboots instead of parking. - /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking the iTCO hardware watchdog or the chipset resets the box, which is the layer that covers a kernel too wedged to notice. Clamped to the driver's own 30s because systemd kicks at a third of whatever value it is handed. Presence alerting: docs/wall-watch/ runs on the server host, probes loopback /healthz (exempt from the address gate) every minute, debounces three checks, pages over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the title, body and priority so a drill can never be mistaken for an incident. Verified against a fake ntfy: caller environment wins over the config file, both transitions fire, and the debounce holds. Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and stays open; the TCP half was every high port on a box whose only access control is its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s inbound with the TCP range closed, and tcp/8080 stopped answering from a same-subnet host. Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its built-in default URL, so re-running it to repair a wall silently repointed it at the production default. An existing URL now wins and KIOSK_URL= is an explicit change. Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to capture, and this image has grdctl rather than grdc), and why no userspace RDP would have survived the hang either. LLM-Generated: true
This commit is contained in:
+184
-2
@@ -20,6 +20,11 @@ or to change its URL.
|
||||
| `mirror-kiosk-console-guard` | `/usr/local/libexec/` | reclaims the console if the wall loses it |
|
||||
| `mirror-kiosk-console-guard.{service,timer}` | `/etc/systemd/system/` | runs that check every minute |
|
||||
| `dim-probe.py` | run on any reachable host | measures what the kiosk actually renders |
|
||||
| `90-mirror-wall-selfheal.conf` | `/etc/sysctl.d/` | lockups and oopses become a reboot, not a frozen frame |
|
||||
| `90-mirror-wall-watchdog.conf` | `/etc/systemd/system.conf.d/` | PID 1 must kick the hardware watchdog or the chipset resets the box |
|
||||
|
||||
Presence alerting lives beside this, not inside it: [`../wall-watch/`](../wall-watch/)
|
||||
runs on the **server** host and pages you when a wall goes quiet.
|
||||
|
||||
---
|
||||
|
||||
@@ -325,6 +330,176 @@ convenient IP.
|
||||
|
||||
---
|
||||
|
||||
## When the wall hangs: making it come back by itself
|
||||
|
||||
On 2026-11-02 the reference wall hung solid at 14:03Z. Nothing said so, nothing
|
||||
recovered it, and it stayed that way until someone walked over and powered it off.
|
||||
Everything in this section exists so the next one is a 90-second blink instead of
|
||||
a drive.
|
||||
|
||||
**What the hang actually looked like** — worth reading, because every easy
|
||||
diagnosis was wrong:
|
||||
|
||||
| Signal | Reading |
|
||||
|---|---|
|
||||
| `ping` | answered, 0% loss, RTT jittering 1.9–8.2 ms on a LAN |
|
||||
| `ssh` | **`connect()` succeeded**, then the socket closed 6–13 ms later with **zero bytes**. No SSH banner ever, and `sshd` never logged an attempt. |
|
||||
| the service | the wall's `/ws` dropped at 14:03:04Z after 1212 s and never reconnected; `/healthz` → `viewers:0` (it is normally `viewers:1, watching:1`) |
|
||||
| journal | every unit goes silent at 09:01:59 and stays silent for the 13 minutes until the power cut |
|
||||
| memory | 62 GiB installed, 2.5 GiB in use, **zero OOM lines** in the previous boot |
|
||||
| `pstore` | empty; the next boot reported `system.journal corrupted or uncleanly shut down` — the hard reset, and the reason the journal's tail is lost |
|
||||
|
||||
The trap: **ICMP replies and TCP handshakes are kernel work.** "It pings" and
|
||||
"port 22 is open" say nothing about userspace. Fedora starts `sshd` per
|
||||
connection, so a master that cannot fork answers the SYN from the listen socket
|
||||
and then hangs up without a byte — which is exactly what happened. Put together:
|
||||
the kernel was alive and no userspace process could be started. Not memory, not a
|
||||
full disk, not a fail2ban ban (a ban refuses the SYN, it does not complete the
|
||||
handshake) — a hard hang that left no trace.
|
||||
|
||||
**Two layers, because neither one covers the other:**
|
||||
|
||||
| Layer | File | Catches |
|
||||
|---|---|---|
|
||||
| kernel escalates to a reboot | `/etc/sysctl.d/90-mirror-wall-selfheal.conf` | soft lockup, hard lockup (NMI watchdog), any oops → panic; `kernel.panic = 30` reboots 30 s later |
|
||||
| the chipset reboots the kernel | `/etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf` | PID 1 stops kicking `/dev/watchdog` — i.e. the case above, where the kernel is too wedged to notice it is wedged |
|
||||
|
||||
Verify all of it (or just run `check-kiosk.sh`, which asserts every line):
|
||||
|
||||
```bash
|
||||
ssh mirror 'systemctl show -p RuntimeWatchdogUSec --value' # 30s
|
||||
ssh mirror 'cat /sys/class/watchdog/watchdog0/status' # 0x8100 on the reference box
|
||||
ssh mirror 'sysctl kernel.softlockup_panic kernel.hardlockup_panic kernel.panic_on_oops kernel.panic'
|
||||
# 1 1 1 30
|
||||
```
|
||||
|
||||
`status` is a bitmask of driver features *and* current state, so do not compare it
|
||||
for equality: the reference box reports `0x8100` = `WDIOF_KEEPALIVEPING` (PID 1 is
|
||||
currently kicking it) plus `WDIOF_MAGICCLOSE` (a clean shutdown stops it, so the
|
||||
watchdog cannot reset a box that is deliberately powered off). The bit that matters
|
||||
is `0x8000`. `check-kiosk.sh` asserts `RuntimeWatchdogUSec` instead, which is not
|
||||
ambiguous.
|
||||
|
||||
**Why 30 seconds.** It is the timeout `iTCO_wdt` initialises itself with
|
||||
(`heartbeat=30 sec`), and systemd kicks at a third of whatever it is handed. Asking
|
||||
a device for more than it supports would make systemd kick *less* often than the
|
||||
hardware expires — a reboot loop of our own making — so `install-kiosk.sh` clamps
|
||||
to the driver's own value. Boxes with no watchdog at all (most VMs) get the sysctls
|
||||
and a line saying so.
|
||||
|
||||
**The cost, stated plainly.** Any stall of PID 1 longer than 30 s now reboots the
|
||||
wall — a genuinely stuck disk, say. On an unattended display that is the intended
|
||||
trade, but it is a trade; to back off raise the value or set it to 0 and rely on
|
||||
the sysctls alone (the drop-in documents both).
|
||||
|
||||
**What it still cannot do.** `kernel.hung_task_panic` does not exist in Fedora's
|
||||
kernel build (`CONFIG_DETECT_HUNG_TASK` is off), so a pure blocked-on-I/O hang is
|
||||
caught by the watchdog rather than by a targeted panic — and a machine that is
|
||||
*off* (dead PSU, tripped plug, firmware that never reaches the bootloader) cannot
|
||||
watchdog anything. That remains an ops decision.
|
||||
|
||||
---
|
||||
|
||||
## The wall's firewall
|
||||
|
||||
Fedora's `FedoraWorkstation` zone — the default on Silverblue — ships with:
|
||||
|
||||
```
|
||||
ports: 1025-65535/udp 1025-65535/tcp 443/tcp 80/tcp
|
||||
```
|
||||
|
||||
The **UDP** half is load-bearing: media is peer-to-peer and mirror deliberately
|
||||
uses no STUN/TURN, so a wall that cannot receive UDP on high ports shows
|
||||
`unreachable` tiles. The **TCP** half is pure exposure on a box whose only access
|
||||
control is its address (A19) — it made every high port on the wall connectable
|
||||
from anywhere the wall was reachable, and a wall never serves anything.
|
||||
|
||||
`install-kiosk.sh` step 8/9 removes `1025-65535/tcp`, leaves the UDP alone, and
|
||||
says which of the two it found; `check-kiosk.sh` FAILs if either half moves.
|
||||
|
||||
This was measured, not assumed. With the TCP range gone, the wall's WebRTC socket
|
||||
process still had ICE candidates bound on LAN IPv4 and public IPv6, and media kept
|
||||
flowing inbound:
|
||||
|
||||
```
|
||||
wall NIC over 6s: rx 459.6 kbit/s tx 16.6 kbit/s
|
||||
```
|
||||
|
||||
while inbound probes from a same-subnet host went `tcp/22` reachable and
|
||||
`tcp/8080`, `tcp/9999` silent. (To see the ICE sockets, grep for
|
||||
`Socket Process` — Firefox's WebRTC sockets are not owned by a process named
|
||||
`firefox`.)
|
||||
|
||||
Still open and not needed by a wall: `80/tcp`, `443/tcp`, `samba-client`. Left
|
||||
alone because they are not this change.
|
||||
|
||||
---
|
||||
|
||||
## The wall went quiet and nobody noticed
|
||||
|
||||
`/healthz` knew at 14:03:04Z that nothing was watching. It is a JSON endpoint and
|
||||
nothing was reading it, so the outage was discovered by looking at the panel. The
|
||||
other half of running an unattended wall is being told about it —
|
||||
[`../wall-watch/`](../wall-watch/) checks presence on the server host every minute
|
||||
and pages over ntfy, then announces the recovery.
|
||||
|
||||
```bash
|
||||
tar czf - -C docs wall-watch | ssh server \
|
||||
'cd /var/tmp && tar xzf - && cd wall-watch && sudo sh install-wall-watch.sh'
|
||||
```
|
||||
|
||||
Test it the honest way — a **drill**, labelled as one in the title, body and
|
||||
priority so it can never be mistaken for an incident:
|
||||
|
||||
```bash
|
||||
sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=http://127.0.0.1:1/healthz \
|
||||
WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test sh /usr/local/libexec/mirror-wall-watch.sh'
|
||||
```
|
||||
|
||||
Two things worth knowing: it probes loopback `/healthz`, which works only because
|
||||
that path is exempt from the address gate (see `AGENTS.md` §3); and it cannot
|
||||
detect the **server** being down, because it lives there. `scp` to some of these
|
||||
hosts dies with `lost connection` (no sftp subsystem) — the tar pipe above works
|
||||
everywhere.
|
||||
|
||||
---
|
||||
|
||||
## Why there is no RDP on the wall
|
||||
|
||||
You cannot see the panel, so of course the idea is to look at it remotely.
|
||||
`gnome-remote-desktop` is already in the Silverblue base image, so it looks free.
|
||||
It does not work here, for a structural reason:
|
||||
|
||||
**`grd` does not capture a screen.** It is a *client* of mutter's remote-desktop
|
||||
D-Bus API, and gnome-kiosk embeds libmutter without exporting it. Ask the wall's
|
||||
own session bus:
|
||||
|
||||
```bash
|
||||
sudo -u kiosk XDG_RUNTIME_DIR=/run/user/1001 \
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus \
|
||||
busctl call org.gnome.Mutter.RemoteDesktop /org/gnome/Mutter/RemoteDesktop \
|
||||
org.freedesktop.DBus.Peer Ping
|
||||
# Call failed: The name is not activatable (same for .ScreenCast and .DisplayConfig)
|
||||
```
|
||||
|
||||
Not a missing package: gnome-shell 50.6 *is* installed and is what exports those
|
||||
names; gnome-kiosk 50.1 does not. So `grdctl rdp enable` and
|
||||
`grdctl rdp set-credentials` would give you a running daemon with nothing to
|
||||
share. For whoever tries next:
|
||||
|
||||
- this image ships **`grdctl`, not `grdc`**, and has **no `org.gnome.RemoteDesktop`
|
||||
gsettings schema** — configuration is grdctl/JSON, so the `DCONF_PROFILE` trap
|
||||
above does not even apply to it.
|
||||
- the only way to really see the panel over RDP is to move the wall onto a
|
||||
gnome-shell-based session. That replaces the script session this whole runbook
|
||||
is built on — the never-exit loop, `window-config.ini`, the greeter trap — and is
|
||||
a `PLAN.md` decision with a full re-validation, not a config tweak.
|
||||
- **and it would not have survived the hang above.** An RDP server is userspace;
|
||||
the failure was userspace being unable to start. RDP is a way to *look*, never a
|
||||
rescue path. The rescue path is the watchdog.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause and fix |
|
||||
@@ -341,6 +516,10 @@ convenient IP.
|
||||
| Firefox shows a restore-session or first-run page | the policy file is missing or unreadable — `sudo cat /etc/firefox/policies/policies.json` |
|
||||
| Tile says `unreachable` | the sharer and the kiosk could not find each other over ICE. Both must be on a network where they can send UDP directly; mirror uses no STUN/TURN by design |
|
||||
| Wall shows the "not allowed" page | the kiosk's address is outside `MIRROR_ALLOW_CIDRS` — this is admission control working, see `SECURITY.md` |
|
||||
| **`ssh` connects, then closes in milliseconds with no banner** | Not a firewall ban — a ban refuses the SYN. The kernel is answering from the listen socket while `sshd` cannot spawn a session: the box is **hung**, and nothing userspace on it, including this checker, will run. The hardware watchdog should reset it within a minute. |
|
||||
| It pings and the port is open, so why is it broken? | ICMP replies and TCP handshakes are produced by the kernel. They say nothing about userspace. Trust `viewers` on `/healthz`, not `ping`. |
|
||||
| Wall reboots on its own | Either the kernel panicked or PID 1 missed the watchdog — that is the design working. Find out which before it happens again: `journalctl -b -1 -p err`, `journalctl -b -1 -k \| tail`, and `ls /sys/fs/pstore` (this box has no ERST backend, so a panic trace may not survive). |
|
||||
| `install-kiosk.sh` pointed the wall somewhere else | It no longer can: an existing `kiosk.conf` URL wins over the built-in default. Pass `KIOSK_URL=...` to change it deliberately. |
|
||||
|
||||
Debugging rule for this project applies: reproduce at the lowest layer that can
|
||||
fail. `curl` the URL from the client, check `/api/streams` on the server, and
|
||||
@@ -367,8 +546,11 @@ decision, not an oversight.
|
||||
`rpm-ostreed-automatic.timer`.
|
||||
* **SELinux is left enforcing, unmodified.** Nothing here needed a policy tweak;
|
||||
treat a new AVC as a bug in this runbook.
|
||||
* **No reboot watchdog.** If the wall is remote enough that a hang means driving
|
||||
to it, that is an ops decision (greenboot, or a smart PDU), not a client one.
|
||||
* **No power cycle.** A hang is now recovered by the kernel and the chipset
|
||||
watchdogs above. A machine that is *off* — dead PSU, tripped plug, firmware that
|
||||
never reaches the bootloader — cannot reboot itself, and if the wall is remote
|
||||
enough for that to matter it is still an ops decision (greenboot, or a smart
|
||||
PDU), not a client one.
|
||||
|
||||
## Naming on the reference network
|
||||
|
||||
|
||||
Reference in New Issue
Block a user