feat(kiosk): self-recover a hung wall and page when it goes quiet

An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel
alive but userspace unable to start a process: ssh completed the handshake and
closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM
lines rule out memory. Separately, the service knew at 14:03Z that nothing was
watching (healthz viewers:0) and nothing was reading it. It was discovered by
walking over to the panel.

Hang recovery, two independent layers (install step 7/9, asserted by
check-kiosk.sh):

- /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any
  oops escalate to a panic, and kernel.panic=30 reboots instead of parking.
- /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking
  the iTCO hardware watchdog or the chipset resets the box, which is the layer that
  covers a kernel too wedged to notice. Clamped to the driver's own 30s because
  systemd kicks at a third of whatever value it is handed.

Presence alerting: docs/wall-watch/ runs on the server host, probes loopback
/healthz (exempt from the address gate) every minute, debounces three checks, pages
over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the
title, body and priority so a drill can never be mistaken for an incident.
Verified against a fake ntfy: caller environment wins over the config file, both
transitions fire, and the debounce holds.

Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half
is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and
stays open; the TCP half was every high port on a box whose only access control is
its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s
inbound with the TCP range closed, and tcp/8080 stopped answering from a
same-subnet host.

Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its
built-in default URL, so re-running it to repair a wall silently repointed it at
the production default. An existing URL now wins and KIOSK_URL= is an explicit
change.

Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk
session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to
capture, and this image has grdctl rather than grdc), and why no userspace RDP
would have survived the hang either.

LLM-Generated: true

docs/wall-watch/README.md

@@ -0,0 +1,57 @@
+# wall-watch — presence alert for the mirror wall
+
+Runs on the **mirror server host**, not on the kiosk. Once a minute it reads the
+service's own `/healthz` and pages you over ntfy when nobody is watching, or when
+the service stops answering — and tells you when the wall comes back.
+
+Rationale and the incident that produced it:
+[`../kiosk-silverblue.md`](../kiosk-silverblue.md) → *"The wall went quiet and
+nobody noticed"*.
+
+## Install (on the server host)
+
+```bash
+scp -r docs/wall-watch server:/var/tmp/ && ssh server \
+  'cd /var/tmp/wall-watch && sudo sh install-wall-watch.sh'
+sudo vi /etc/mirror-wall-watch.env          # set NTFY_TOKEN
+```
+
+## Operate
+
+```bash
+systemctl status mirror-wall-watch.timer            # is it still ticking?
+journalctl -u mirror-wall-watch.service -f          # one line per check, forever
+sudo sh /usr/local/libexec/mirror-wall-watch.sh     # run it by hand
+
+# Page yourself on purpose, without touching the wall's real state. WALL_DRILL=1
+# makes the notification say "[drill]" and drop to priority 3, so a test can never
+# be mistaken for an outage:
+sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=https://127.0.0.1/nope \
+  WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test \
+  sh /usr/local/libexec/mirror-wall-watch.sh'
+```
+
+## Files
+
+| File | Lands at |
+|---|---|
+| `wall-watch.sh` | `/usr/local/libexec/mirror-wall-watch.sh` |
+| `mirror-wall-watch.service` / `.timer` | `/etc/systemd/system/` |
+| `wall-watch.env.example` | `/etc/mirror-wall-watch.env` (0600, seeded once, never overwritten) |
+| everything here | `/usr/local/share/mirror-wall-watch/` |
+
+## Deliberate choices
+
+- **Probes `/healthz` over loopback**, because that endpoint is exempt from the
+  address gate — the same exemption that lets the container healthcheck work from
+  loopback when `MIRROR_ALLOW_CIDRS` excludes it.
+- **First run cannot alert.** Installing a watcher must not page you about a wall
+  that was already down before you had one.
+- **A failed notification fails the unit** and does *not* bank the re-notify
+  timestamp, so an ntfy outage is retried on the next tick instead of eating the
+  alert.
+- **A drill is labelled as one.** `WALL_DRILL=1` prefixes the title with `[drill]`, 
+  lowers the priority and says "nothing is wrong" in the body. A test page that is
+  indistinguishable from an incident teaches people to ignore the page.
+- **It cannot detect the server being down.** It lives there. That gap belongs to
+  whatever monitors this host.

docs/wall-watch/install-wall-watch.sh

@@ -0,0 +1,76 @@
+#!/bin/sh
+# install-wall-watch.sh — install the wall-presence alert on the MIRROR SERVER host.
+#
+# Run ON THE SERVER, as root, from this directory:
+#
+#   sudo sh install-wall-watch.sh
+#
+# Idempotent. It never overwrites /etc/mirror-wall-watch.env once it exists, so
+# the ntfy token you pasted in survives a re-run.
+#
+# What it does, and why this exists at all: ../kiosk-silverblue.md
+# ("The wall went quiet and nobody noticed").
+
+set -eu
+
+HERE=$(cd "$(dirname "$0")" && pwd)
+CONF=/etc/mirror-wall-watch.env
+say() { printf '%s\n' "$*"; }
+die() { printf 'error: %s\n' "$*" >&2; exit 1; }
+
+[ "$(id -u)" -eq 0 ] || die "run me as root (sudo sh $0)"
+for c in curl logger systemctl; do
+    command -v "$c" >/dev/null 2>&1 || die "missing required command: $c"
+done
+
+say "== 1/4 program =="
+install -D -m 0755 "$HERE/wall-watch.sh" /usr/local/libexec/mirror-wall-watch.sh
+install -d -m 0755 /usr/local/share/mirror-wall-watch
+cp -a "$HERE"/. /usr/local/share/mirror-wall-watch/
+rm -f /usr/local/share/mirror-wall-watch/install-wall-watch.sh
+say "  /usr/local/libexec/mirror-wall-watch.sh"
+say "  /usr/local/share/mirror-wall-watch (docs, survives reboot)"
+
+say "== 2/4 config =="
+if [ ! -f "$CONF" ]; then
+    install -D -m 0600 "$HERE/wall-watch.env.example" "$CONF"
+    say "  created $CONF (0600) — NOW SET NTFY_TOKEN IN IT:"
+    say "      sudo vi $CONF"
+else
+    perm=$(stat -c '%a' "$CONF")
+    [ "$perm" = "600" ] || say "  warning: $CONF is mode $perm, expected 0600 (it holds a token)"
+    say "  kept existing $CONF"
+fi
+
+say "== 3/4 units =="
+install -D -m 0644 "$HERE/mirror-wall-watch.service" /etc/systemd/system/mirror-wall-watch.service
+install -D -m 0644 "$HERE/mirror-wall-watch.timer" /etc/systemd/system/mirror-wall-watch.timer
+systemctl daemon-reload
+systemctl enable --now mirror-wall-watch.timer
+say "  mirror-wall-watch.timer enabled (every minute)"
+
+say "== 4/4 prove it =="
+# One run now, so a broken config fails in front of you instead of silently in a
+# timer at 03:00. The first run only records the baseline; it cannot alert, which
+# is deliberate — installing the watcher must not page you about a wall that was
+# already down before you had a watcher.
+if systemctl start --wait mirror-wall-watch.service 2>/dev/null; then
+    say "  first check passed:"
+    journalctl -u mirror-wall-watch.service --no-pager -n 1 2>/dev/null | sed 's/^/      /' || :
+else
+    say "  first check FAILED — read this before trusting the timer:"
+    journalctl -u mirror-wall-watch.service --no-pager -n 12 2>/dev/null | sed 's/^/      /' || :
+fi
+# An AVC here means SELinux disagreed with something above; say so rather than
+# leaving a silently broken alert.
+if command -v ausearch >/dev/null 2>&1; then
+    if ausearch -m avc -ts recent >/dev/null 2>&1; then
+        say "  warning: recent AVCs found — run: sudo ausearch -m avc -ts recent"
+    fi
+fi
+
+say ""
+say "Installed. Fire a test notification (page yourself on purpose):"
+say "    sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=https://127.0.0.1/nope WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test sh /usr/local/libexec/mirror-wall-watch.sh'"
+say "Then watch the timeline:"
+say "    journalctl -u mirror-wall-watch.service -f"

docs/wall-watch/mirror-wall-watch.service

@@ -0,0 +1,16 @@
+[Unit]
+Description=Check that the mirror wall is still watching
+Documentation=file:///usr/local/share/mirror-wall-watch/README.md
+# The probe is only meaningful once nginx (or whatever fronts the service) is up.
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/sh /usr/local/libexec/mirror-wall-watch.sh
+EnvironmentFile=-/etc/mirror-wall-watch.env
+# It only ever writes /var/lib/mirror-wall-watch; keep the rest read-only.
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectSystem=full
+ProtectHome=read-only

docs/wall-watch/mirror-wall-watch.timer

@@ -0,0 +1,17 @@
+[Unit]
+Description=Run the mirror wall presence check every minute
+Documentation=file:///usr/local/share/mirror-wall-watch/README.md
+
+[Timer]
+# Check from a minute after boot — the service needs a moment to come up, and a
+# wall has its own relaunch loop — and then every minute. The jitter keeps a
+# fleet of these from all waking up on the same second.
+OnBootSec=1min
+OnUnitActiveSec=1min
+RandomizedDelaySec=5s
+AccuracySec=1s
+# If the host was suspended or the timer missed its window, run once anyway.
+Persistent=true
+
+[Install]
+WantedBy=timers.target

docs/wall-watch/wall-watch.env.example

@@ -0,0 +1,28 @@
+# /etc/mirror-wall-watch.env — config for mirror-wall-watch.timer.
+# chmod 0600, root:root. It holds the ntfy token. Never commit a filled-in copy.
+# install-wall-watch.sh seeds this from the example and then NEVER overwrites it.
+
+# The presence probe. /healthz is exempt from the address gate, so loopback works
+# even when MIRROR_ALLOW_CIDRS excludes 127.0.0.0/8 — which is why the container
+# healthcheck works too. Use whatever loopback URL reaches this deployment.
+WALL_HEALTH_URL=https://127.0.0.1/healthz
+
+# Alert when fewer than this many viewers are attached. Set 0 to stop alerting on
+# "nobody is watching" and only alert when the service itself stops answering
+# (useful while a fleet has no wall plugged in).
+WALL_EXPECT_VIEWERS=1
+
+# Consecutive bad checks before the first notification (~1 min per check). Three
+# rides out a Firefox relaunch on the wall; raise it if that is normal for you.
+WALL_THRESHOLD=3
+
+# Keep shouting while it stays down. 0 = alert once per incident.
+WALL_RENOTIFY_SECS=3600
+
+# Optional: the URL a notification opens when tapped.
+WALL_STATUS_URL=https://mirror.reeseapps.com/healthz
+
+# ntfy (outbound only — this never reads anything back).
+NTFY_URL=https://ntfy.reeseapps.com
+NTFY_TOKEN=
+NTFY_TOPIC=pi

docs/wall-watch/wall-watch.sh

@@ -0,0 +1,213 @@
+#!/bin/sh
+# wall-watch.sh — know within minutes that the mirror wall stopped watching.
+#
+# Run ON THE MIRROR SERVER HOST from mirror-wall-watch.timer (once a minute).
+# It reads the service's own /healthz over loopback and posts to ntfy when no
+# viewer is attached, or when the service stops answering at all — and again when
+# the wall comes back.
+#
+# Why it exists: on 2026-11-02 the reference wall hard-hung at 14:03Z. The
+# service knew the instant it happened — /healthz reported viewers:0 — and said
+# nothing, because nothing was listening. The failure was eventually found by
+# walking up to the panel, which was in another room of another building.
+#
+# /healthz is the right probe precisely because it is exempt from the address
+# gate (AGENTS.md §3): a loopback check works even when MIRROR_ALLOW_CIDRS
+# excludes loopback, which is also why the container healthcheck can work.
+#
+# Config: /etc/mirror-wall-watch.env (root-owned 0600 — it holds the ntfy token)
+#   WALL_HEALTH_URL       default https://127.0.0.1/healthz
+#   WALL_EXPECT_VIEWERS   default 1; set 0 to stop alerting on "nobody watching"
+#   WALL_THRESHOLD        default 3 consecutive bad checks (~3 min) before shouting
+#   WALL_RENOTIFY_SECS    default 3600; keeps shouting while it stays down
+#   WALL_STATUS_URL       optional; the URL a notification opens
+#   NTFY_URL NTFY_TOKEN NTFY_TOPIC
+#
+# Manual run (the timer runs it the same way):
+#   sudo sh /usr/local/libexec/mirror-wall-watch.sh
+#   sudo sh -c 'WALL_HEALTH_URL=https://127.0.0.1/nope WALL_THRESHOLD=1 sh /usr/local/libexec/mirror-wall-watch.sh'   # fire a test alert
+#
+# Honest limit: this runs on the server, so it cannot tell you the *server* is
+# down. That is a different alert, and it belongs to whatever watches this host.
+
+set -eu
+
+CONF="${WALL_WATCH_CONF:-/etc/mirror-wall-watch.env}"
+STATE_DIR="${WALL_STATE_DIR:-/var/lib/mirror-wall-watch}"
+STATE_FILE="$STATE_DIR/state"
+
+log() {
+    printf '%s\n' "$*"
+    # Under systemd stdout already IS the journal, so writing to /dev/log as well
+    # puts two copies of every check in it. Keep the syslog trail for interactive
+    # runs, where stdout is a terminal and nothing else would record it.
+    [ -t 1 ] && logger -t mirror-wall-watch -- "$*" 2>/dev/null
+    return 0
+}
+die() { log "error: $*"; exit 1; }
+
+command -v curl >/dev/null 2>&1 || die "curl is required"
+# Root is not inherently required, only write access to the state directory — but
+# the default lives next to root-only config, so a non-root run must say where it
+# is going to fail rather than fail at the interesting moment.
+if ! install -d -m 0700 "$STATE_DIR" 2>/dev/null; then
+    die "cannot create $STATE_DIR — run me as root, or set WALL_STATE_DIR somewhere writable"
+fi
+
+# Apply KEY=VALUE from the config file, but only for variables the caller did not
+# already set. An explicit environment must win over the file — the same precedence
+# systemd gives Environment= over EnvironmentFile= — otherwise "run the checker
+# against a different URL for a moment" silently does nothing, which is exactly the
+# kind of thing you only discover when the alert you were testing never arrives.
+load_conf() {
+    [ -r "$1" ] || return 0
+    while IFS= read -r line || [ -n "$line" ]; do
+        case "$line" in ''|'#'*) continue ;; esac
+        key=${line%%=*}
+        val=${line#*=}
+        key=$(printf '%s' "$key" | tr -d '[:space:]')
+        val=$(printf '%s' "$val" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
+        case "$val" in '\"'*'"') val=${val#\"}; val=${val%\"} ;; esac
+        # Only sane shell identifiers get assigned.
+        case "$key" in ''|*[!A-Za-z0-9_]*|[0-9]*) continue ;; esac
+        eval "cur=\${$key:-}"
+        [ -n "$cur" ] || eval "$key=\$val"
+    done <"$1"
+}
+load_conf "$CONF"
+
+: "${WALL_HEALTH_URL:=https://127.0.0.1/healthz}"
+: "${WALL_EXPECT_VIEWERS:=1}"
+: "${WALL_THRESHOLD:=3}"
+: "${WALL_RENOTIFY_SECS:=3600}"
+
+# A drill is an alert fired on purpose to prove the path works. It must never be
+# mistaken for an incident, so it says so in the title, the body and the priority
+# — an alerting tool whose test page looks like a real page trains people to
+# ignore it.
+: "${WALL_DRILL:=0}"
+if [ "$WALL_DRILL" = "1" ]; then
+    DRILL_TAG="[drill] "
+    DRILL_NOTE="> **This is a drill.** Someone ran the test command on \`$(hostname)\`; nothing is wrong with the wall. What it checked is the path: probe -> ntfy -> your phone."
+    DOWN_PRIORITY=3
+else
+    DRILL_TAG=""
+    DRILL_NOTE=""
+    DOWN_PRIORITY=4
+fi
+
+# Posts one notification. Returns non-zero instead of exiting: a failed alert
+# must not stop the state file being written, or the next run loses the counter
+# and the wall pages once a minute forever.
+notify() {
+    prio=$1 tags=$2 title=$3 body=$4
+    if [ -z "${NTFY_URL:-}" ] || [ -z "${NTFY_TOPIC:-}" ]; then
+        log "error: ntfy is not configured (NTFY_URL/NTFY_TOPIC in $CONF)"
+        return 1
+    fi
+
+    # Collect the headers in the positional parameters, and only the ones that
+    # have values, so ntfy never receives an empty Authorization or X-Click.
+    set -- -H "X-Priority: $prio" -H "X-Tags: $tags" -H "X-Title: $title" -H "X-Markdown: yes"
+    [ -n "${NTFY_TOKEN:-}" ] && set -- "$@" -H "Authorization: Bearer $NTFY_TOKEN"
+    [ -n "${WALL_STATUS_URL:-}" ] && set -- "$@" -H "X-Click: $WALL_STATUS_URL"
+
+    if curl -fsS -m 10 "$@" -d "$body" "$NTFY_URL/$NTFY_TOPIC" >/dev/null 2>&1; then
+        log "notified (priority $prio): $title"
+    else
+        log "error: could not reach $NTFY_URL — the alert did not go out"
+        return 1
+    fi
+}
+
+# ---- read the service's own truth -------------------------------------------------
+probe=""
+viewers=""
+if probe=$(curl -fsS -m 8 -k "$WALL_HEALTH_URL" 2>/dev/null); then
+    if command -v jq >/dev/null 2>&1; then
+        viewers=$(printf '%s' "$probe" | jq -r '.viewers // empty' 2>/dev/null || :)
+    else
+        viewers=$(printf '%s' "$probe" |
+            sed -n 's/.*"viewers"[[:space:]]*:[[:space:]]*\([0-9][0-9]*\).*/\1/p' | head -1)
+    fi
+    # 200 but no parseable viewer count is a broken answer, not a healthy one.
+    [ -n "$viewers" ] || probe=""
+fi
+
+if [ -z "$probe" ]; then
+    condition="unreachable"
+    summary="the mirror service did not answer \`$WALL_HEALTH_URL\`"
+elif [ "$WALL_EXPECT_VIEWERS" -gt 0 ] && [ "$viewers" -lt "$WALL_EXPECT_VIEWERS" ]; then
+    condition="quiet"
+    summary="the service is up but **nothing is watching** (viewers=$viewers)"
+else
+    condition="ok"
+    summary="viewers=$viewers"
+fi
+
+# ---- state machine ----------------------------------------------------------------
+state="ok"
+bad=0
+last_alert=0
+if [ -r "$STATE_FILE" ]; then
+    # shellcheck disable=SC1090
+    . "$STATE_FILE" 2>/dev/null || :
+fi
+now=$(date +%s)
+
+NOTIFY_FAILED=0
+
+if [ "$condition" = "ok" ]; then
+    if [ "$state" != "ok" ]; then
+        notify 3 "heavy_check_mark,recovered" "${DRILL_TAG}Wall back" \
+"**${DRILL_TAG}The mirror wall is watching again.**
+
+was: \`$state\`
+now: $summary
+host: \`$(hostname)\` · $(date -u '+%Y-%m-%d %H:%M:%SZ')
+
+$DRILL_NOTE" || NOTIFY_FAILED=1
+    fi
+    state="ok"; bad=0; last_alert=0
+else
+    if [ "$state" = "$condition" ]; then bad=$((bad + 1)); else bad=1; fi
+    if [ "$bad" -ge "$WALL_THRESHOLD" ] &&
+       { [ "$state" != "$condition" ] || [ $((now - last_alert)) -ge "$WALL_RENOTIFY_SECS" ]; }; then
+        hint="Recover it: \`ssh <wall>\` then \`sudo sh /usr/local/share/mirror-kiosk/check-kiosk.sh\`; if ssh itself will not open a session the box is hung, not offline — the watchdog should reset it within a minute of this, and if it does not, power-cycle it."
+        [ "$condition" = "unreachable" ] &&
+            hint="Check the service first (\`systemctl status\`, container logs) before the wall: this host may be the thing that is broken."
+        notify "$DOWN_PRIORITY" "rotating_light,wall-down" "${DRILL_TAG}Wall down" \
+"**${DRILL_TAG}Mirror wall: $condition**
+
+$summary
+
+\`\`\`
+$probe
+\`\`\`
+
+$hint
+
+host: \`$(hostname)\` · $(date -u '+%Y-%m-%d %H:%M:%SZ') · bad check #$bad
+
+$DRILL_NOTE" || NOTIFY_FAILED=1
+        # Only bank the re-notify timestamp if it actually went out, so a ntfy
+        # outage is retried on the next tick rather than swallowed.
+        [ "$NOTIFY_FAILED" -eq 0 ] && last_alert=$now
+    fi
+    state="$condition"
+fi
+
+umask 077
+cat >"$STATE_FILE" <<EOF
+state=$state
+bad=$bad
+last_alert=$last_alert
+EOF
+
+# One greppable line per run, so `journalctl -u mirror-wall-watch` reads as a
+# timeline of the wall's health rather than as silence.
+log "$condition: $summary (state=$state bad=$bad threshold=$WALL_THRESHOLD)"
+
+# A needed-but-undeliverable alert fails the unit, so `systemctl show`
+# mirror-wall-watch.service shows why nobody got paged.
+exit "$NOTIFY_FAILED"