An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel
alive but userspace unable to start a process: ssh completed the handshake and
closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM
lines rule out memory. Separately, the service knew at 14:03Z that nothing was
watching (healthz viewers:0) and nothing was reading it. It was discovered by
walking over to the panel.
Hang recovery, two independent layers (install step 7/9, asserted by
check-kiosk.sh):
- /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any
oops escalate to a panic, and kernel.panic=30 reboots instead of parking.
- /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking
the iTCO hardware watchdog or the chipset resets the box, which is the layer that
covers a kernel too wedged to notice. Clamped to the driver's own 30s because
systemd kicks at a third of whatever value it is handed.
Presence alerting: docs/wall-watch/ runs on the server host, probes loopback
/healthz (exempt from the address gate) every minute, debounces three checks, pages
over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the
title, body and priority so a drill can never be mistaken for an incident.
Verified against a fake ntfy: caller environment wins over the config file, both
transitions fire, and the debounce holds.
Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half
is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and
stays open; the TCP half was every high port on a box whose only access control is
its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s
inbound with the TCP range closed, and tcp/8080 stopped answering from a
same-subnet host.
Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its
built-in default URL, so re-running it to repair a wall silently repointed it at
the production default. An existing URL now wins and KIOSK_URL= is an explicit
change.
Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk
session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to
capture, and this image has grdctl rather than grdc), and why no userspace RDP
would have survived the hang either.
LLM-Generated: true
Auto-login succeeded and the wall kept streaming, but about ten seconds
after the kiosk session came up GDM autospawned a greeter on the same GPU
and a login screen covered it. gnome-kiosk ships X-GDM-SessionRegisters=false,
so nothing ever registers and GDM concludes the display is still free — which
is why every remote check (loginctl, the mirror roster) stayed green while
the physical screen showed a password prompt.
Override that session file with the flag set to true, keeping the file name
and session id identical: inventing an id makes GDM quietly fall back to the
plain GNOME session, top bar and all.
Add a per-minute guard that reclaims the foreground VT, because this class
of failure is invisible from the outside, and have the checker assert both
the foreground VT and the greeter-session count. That count had a silent
false green: journalctl has no --no-legend, so the check read an empty
stream and passed unconditionally.
Also keep a copy of the payload in /usr/local/share/mirror-kiosk. /tmp is
cleared at every boot, which deletes the checker just after the one event
you are most likely to want to re-run it for.
Verified on the reference client: a cold boot now creates no greeter session
at all, the kiosk compositor and Firefox come up undisturbed, and a console
stolen with chvt is taken back within a minute.
LLM-Generated: true
A wall display is now a target client: Fedora Silverblue running GNOME
Kiosk, auto-logged in as kiosk, showing /watch fullscreen in Firefox.
docs/kiosk-silverblue/ carries the idempotent installer, a read-only
checker and a probe that measures what the browser actually renders, so
a second client is a copy-paste and nothing about it needs a keyboard.
The runbook records what the upstream tutorial leaves out: gsettings
only commits from inside the session (gnome-kiosk sets DCONF_PROFILE,
so a system dconf db is never read at all), window-config.ini wins by
being found first and therefore silently loses the shipped notification
rules, AccountsService caches Session= until its daemon restarts, and
/watch survives a bare LAN IP while /mirror cannot.
Verified on the reference client: cold boot back to a subscribed wall in
under 60s, viewport == window == screen at 3840x2160, and the sharer's
own peer connection to it reports connected.
LLM-Generated: true
ducoterra