feat(kiosk): self-recover a hung wall and page when it goes quiet
An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel alive but userspace unable to start a process: ssh completed the handshake and closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM lines rule out memory. Separately, the service knew at 14:03Z that nothing was watching (healthz viewers:0) and nothing was reading it. It was discovered by walking over to the panel. Hang recovery, two independent layers (install step 7/9, asserted by check-kiosk.sh): - /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any oops escalate to a panic, and kernel.panic=30 reboots instead of parking. - /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking the iTCO hardware watchdog or the chipset resets the box, which is the layer that covers a kernel too wedged to notice. Clamped to the driver's own 30s because systemd kicks at a third of whatever value it is handed. Presence alerting: docs/wall-watch/ runs on the server host, probes loopback /healthz (exempt from the address gate) every minute, debounces three checks, pages over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the title, body and priority so a drill can never be mistaken for an incident. Verified against a fake ntfy: caller environment wins over the config file, both transitions fire, and the debounce holds. Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and stays open; the TCP half was every high port on a box whose only access control is its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s inbound with the TCP range closed, and tcp/8080 stopped answering from a same-subnet host. Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its built-in default URL, so re-running it to repair a wall silently repointed it at the production default. An existing URL now wins and KIOSK_URL= is an explicit change. Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to capture, and this image has grdctl rather than grdc), and why no userspace RDP would have survived the hang either. LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,57 @@
|
||||
# wall-watch — presence alert for the mirror wall
|
||||
|
||||
Runs on the **mirror server host**, not on the kiosk. Once a minute it reads the
|
||||
service's own `/healthz` and pages you over ntfy when nobody is watching, or when
|
||||
the service stops answering — and tells you when the wall comes back.
|
||||
|
||||
Rationale and the incident that produced it:
|
||||
[`../kiosk-silverblue.md`](../kiosk-silverblue.md) → *"The wall went quiet and
|
||||
nobody noticed"*.
|
||||
|
||||
## Install (on the server host)
|
||||
|
||||
```bash
|
||||
scp -r docs/wall-watch server:/var/tmp/ && ssh server \
|
||||
'cd /var/tmp/wall-watch && sudo sh install-wall-watch.sh'
|
||||
sudo vi /etc/mirror-wall-watch.env # set NTFY_TOKEN
|
||||
```
|
||||
|
||||
## Operate
|
||||
|
||||
```bash
|
||||
systemctl status mirror-wall-watch.timer # is it still ticking?
|
||||
journalctl -u mirror-wall-watch.service -f # one line per check, forever
|
||||
sudo sh /usr/local/libexec/mirror-wall-watch.sh # run it by hand
|
||||
|
||||
# Page yourself on purpose, without touching the wall's real state. WALL_DRILL=1
|
||||
# makes the notification say "[drill]" and drop to priority 3, so a test can never
|
||||
# be mistaken for an outage:
|
||||
sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=https://127.0.0.1/nope \
|
||||
WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test \
|
||||
sh /usr/local/libexec/mirror-wall-watch.sh'
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
| File | Lands at |
|
||||
|---|---|
|
||||
| `wall-watch.sh` | `/usr/local/libexec/mirror-wall-watch.sh` |
|
||||
| `mirror-wall-watch.service` / `.timer` | `/etc/systemd/system/` |
|
||||
| `wall-watch.env.example` | `/etc/mirror-wall-watch.env` (0600, seeded once, never overwritten) |
|
||||
| everything here | `/usr/local/share/mirror-wall-watch/` |
|
||||
|
||||
## Deliberate choices
|
||||
|
||||
- **Probes `/healthz` over loopback**, because that endpoint is exempt from the
|
||||
address gate — the same exemption that lets the container healthcheck work from
|
||||
loopback when `MIRROR_ALLOW_CIDRS` excludes it.
|
||||
- **First run cannot alert.** Installing a watcher must not page you about a wall
|
||||
that was already down before you had one.
|
||||
- **A failed notification fails the unit** and does *not* bank the re-notify
|
||||
timestamp, so an ntfy outage is retried on the next tick instead of eating the
|
||||
alert.
|
||||
- **A drill is labelled as one.** `WALL_DRILL=1` prefixes the title with `[drill]`,
|
||||
lowers the priority and says "nothing is wrong" in the body. A test page that is
|
||||
indistinguishable from an incident teaches people to ignore the page.
|
||||
- **It cannot detect the server being down.** It lives there. That gap belongs to
|
||||
whatever monitors this host.
|
||||
Reference in New Issue
Block a user