ducoterra
8fefd83ef8
An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel alive but userspace unable to start a process: ssh completed the handshake and closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM lines rule out memory. Separately, the service knew at 14:03Z that nothing was watching (healthz viewers:0) and nothing was reading it. It was discovered by walking over to the panel. Hang recovery, two independent layers (install step 7/9, asserted by check-kiosk.sh): - /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any oops escalate to a panic, and kernel.panic=30 reboots instead of parking. - /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking the iTCO hardware watchdog or the chipset resets the box, which is the layer that covers a kernel too wedged to notice. Clamped to the driver's own 30s because systemd kicks at a third of whatever value it is handed. Presence alerting: docs/wall-watch/ runs on the server host, probes loopback /healthz (exempt from the address gate) every minute, debounces three checks, pages over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the title, body and priority so a drill can never be mistaken for an incident. Verified against a fake ntfy: caller environment wins over the config file, both transitions fire, and the debounce holds. Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and stays open; the TCP half was every high port on a box whose only access control is its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s inbound with the TCP range closed, and tcp/8080 stopped answering from a same-subnet host. Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its built-in default URL, so re-running it to repair a wall silently repointed it at the production default. An existing URL now wins and KIOSK_URL= is an explicit change. Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to capture, and this image has grdctl rather than grdc), and why no userspace RDP would have survived the hang either. LLM-Generated: true
77 lines
3.2 KiB
Bash
77 lines
3.2 KiB
Bash
#!/bin/sh
|
|
# install-wall-watch.sh — install the wall-presence alert on the MIRROR SERVER host.
|
|
#
|
|
# Run ON THE SERVER, as root, from this directory:
|
|
#
|
|
# sudo sh install-wall-watch.sh
|
|
#
|
|
# Idempotent. It never overwrites /etc/mirror-wall-watch.env once it exists, so
|
|
# the ntfy token you pasted in survives a re-run.
|
|
#
|
|
# What it does, and why this exists at all: ../kiosk-silverblue.md
|
|
# ("The wall went quiet and nobody noticed").
|
|
|
|
set -eu
|
|
|
|
HERE=$(cd "$(dirname "$0")" && pwd)
|
|
CONF=/etc/mirror-wall-watch.env
|
|
say() { printf '%s\n' "$*"; }
|
|
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
|
|
|
|
[ "$(id -u)" -eq 0 ] || die "run me as root (sudo sh $0)"
|
|
for c in curl logger systemctl; do
|
|
command -v "$c" >/dev/null 2>&1 || die "missing required command: $c"
|
|
done
|
|
|
|
say "== 1/4 program =="
|
|
install -D -m 0755 "$HERE/wall-watch.sh" /usr/local/libexec/mirror-wall-watch.sh
|
|
install -d -m 0755 /usr/local/share/mirror-wall-watch
|
|
cp -a "$HERE"/. /usr/local/share/mirror-wall-watch/
|
|
rm -f /usr/local/share/mirror-wall-watch/install-wall-watch.sh
|
|
say " /usr/local/libexec/mirror-wall-watch.sh"
|
|
say " /usr/local/share/mirror-wall-watch (docs, survives reboot)"
|
|
|
|
say "== 2/4 config =="
|
|
if [ ! -f "$CONF" ]; then
|
|
install -D -m 0600 "$HERE/wall-watch.env.example" "$CONF"
|
|
say " created $CONF (0600) — NOW SET NTFY_TOKEN IN IT:"
|
|
say " sudo vi $CONF"
|
|
else
|
|
perm=$(stat -c '%a' "$CONF")
|
|
[ "$perm" = "600" ] || say " warning: $CONF is mode $perm, expected 0600 (it holds a token)"
|
|
say " kept existing $CONF"
|
|
fi
|
|
|
|
say "== 3/4 units =="
|
|
install -D -m 0644 "$HERE/mirror-wall-watch.service" /etc/systemd/system/mirror-wall-watch.service
|
|
install -D -m 0644 "$HERE/mirror-wall-watch.timer" /etc/systemd/system/mirror-wall-watch.timer
|
|
systemctl daemon-reload
|
|
systemctl enable --now mirror-wall-watch.timer
|
|
say " mirror-wall-watch.timer enabled (every minute)"
|
|
|
|
say "== 4/4 prove it =="
|
|
# One run now, so a broken config fails in front of you instead of silently in a
|
|
# timer at 03:00. The first run only records the baseline; it cannot alert, which
|
|
# is deliberate — installing the watcher must not page you about a wall that was
|
|
# already down before you had a watcher.
|
|
if systemctl start --wait mirror-wall-watch.service 2>/dev/null; then
|
|
say " first check passed:"
|
|
journalctl -u mirror-wall-watch.service --no-pager -n 1 2>/dev/null | sed 's/^/ /' || :
|
|
else
|
|
say " first check FAILED — read this before trusting the timer:"
|
|
journalctl -u mirror-wall-watch.service --no-pager -n 12 2>/dev/null | sed 's/^/ /' || :
|
|
fi
|
|
# An AVC here means SELinux disagreed with something above; say so rather than
|
|
# leaving a silently broken alert.
|
|
if command -v ausearch >/dev/null 2>&1; then
|
|
if ausearch -m avc -ts recent >/dev/null 2>&1; then
|
|
say " warning: recent AVCs found — run: sudo ausearch -m avc -ts recent"
|
|
fi
|
|
fi
|
|
|
|
say ""
|
|
say "Installed. Fire a test notification (page yourself on purpose):"
|
|
say " sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=https://127.0.0.1/nope WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test sh /usr/local/libexec/mirror-wall-watch.sh'"
|
|
say "Then watch the timeline:"
|
|
say " journalctl -u mirror-wall-watch.service -f"
|