feat(kiosk): self-recover a hung wall and page when it goes quiet
An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel alive but userspace unable to start a process: ssh completed the handshake and closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM lines rule out memory. Separately, the service knew at 14:03Z that nothing was watching (healthz viewers:0) and nothing was reading it. It was discovered by walking over to the panel. Hang recovery, two independent layers (install step 7/9, asserted by check-kiosk.sh): - /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any oops escalate to a panic, and kernel.panic=30 reboots instead of parking. - /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking the iTCO hardware watchdog or the chipset resets the box, which is the layer that covers a kernel too wedged to notice. Clamped to the driver's own 30s because systemd kicks at a third of whatever value it is handed. Presence alerting: docs/wall-watch/ runs on the server host, probes loopback /healthz (exempt from the address gate) every minute, debounces three checks, pages over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the title, body and priority so a drill can never be mistaken for an incident. Verified against a fake ntfy: caller environment wins over the config file, both transitions fire, and the debounce holds. Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and stays open; the TCP half was every high port on a box whose only access control is its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s inbound with the TCP range closed, and tcp/8080 stopped answering from a same-subnet host. Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its built-in default URL, so re-running it to repair a wall silently repointed it at the production default. An existing URL now wins and KIOSK_URL= is an explicit change. Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to capture, and this image has grdctl rather than grdc), and why no userspace RDP would have survived the hang either. LLM-Generated: true
This commit is contained in:
@@ -71,7 +71,7 @@ Environment variables (all optional; a `.env` file is loaded if present):
|
||||
| `MIRROR_MAX_MIRRORS` | `0` | Cap on concurrent mirrors (`0` = unlimited). |
|
||||
| `MIRROR_MAX_BITRATE` | `25000000` | Per-leg WebRTC bitrate cap in bits/s — the mesh uploads N × bitrate per sharer, so this is the ceiling. 25 Mbit/s suits a wired LAN at 1440p; a Wi-Fi sharer with many viewers may need less, not more. |
|
||||
| `MIRROR_PING_INTERVAL` | `20` | Seconds between socket liveness pings. |
|
||||
| `MIRROR_ALLOW_CIDRS` | *(unset)* | Comma-separated source networks allowed to connect, e.g. `10.2.0.0/22,fd00::/64`. Unset means unrestricted. |
|
||||
| `MIRROR_ALLOW_CIDRS` | *(unset)* | Comma-separated source networks allowed to connect, e.g. `10.2.0.0/22,fd00::/64`. Unset means unrestricted. A wall needs this range too, and its address is DHCP — give it a reservation. |
|
||||
| `MIRROR_TRUSTED_PROXIES` | `127.0.0.0/8, ::1/128` | Which peers may speak for a client via `X-Forwarded-For`. Only a proxy that overwrites the header belongs here. |
|
||||
| `DEBUGPY` | `0` | `1` opens a debugpy listener on `DEBUGPY_PORT`. |
|
||||
| `DEBUGPY_PORT` | `5678` | debugpy port. |
|
||||
@@ -147,6 +147,13 @@ peer state for the media, and a probe page for the rendered resolution. The
|
||||
runbook records why each file lives where it does on an
|
||||
otherwise read-only OS.
|
||||
|
||||
**When it hangs.** A wall is unattended, so a hang has to end without a person:
|
||||
the kernel turns lockups and oopses into a reboot, and PID 1 must keep kicking the
|
||||
hardware watchdog or the chipset resets the box. Being *quiet* is a different
|
||||
failure than being hung, so the server watches `/healthz` for that and pages over
|
||||
ntfy ([`docs/wall-watch/`](docs/wall-watch/)). Both are installed by the commands
|
||||
above and asserted by `check-kiosk.sh`.
|
||||
|
||||
**Keys.** The whole wall works from a keyboard — and the box needs no keyboard
|
||||
attached to be operated *or* debugged; the keys are there when one is plugged
|
||||
in. On `/watch`:
|
||||
|
||||
+184
-2
@@ -20,6 +20,11 @@ or to change its URL.
|
||||
| `mirror-kiosk-console-guard` | `/usr/local/libexec/` | reclaims the console if the wall loses it |
|
||||
| `mirror-kiosk-console-guard.{service,timer}` | `/etc/systemd/system/` | runs that check every minute |
|
||||
| `dim-probe.py` | run on any reachable host | measures what the kiosk actually renders |
|
||||
| `90-mirror-wall-selfheal.conf` | `/etc/sysctl.d/` | lockups and oopses become a reboot, not a frozen frame |
|
||||
| `90-mirror-wall-watchdog.conf` | `/etc/systemd/system.conf.d/` | PID 1 must kick the hardware watchdog or the chipset resets the box |
|
||||
|
||||
Presence alerting lives beside this, not inside it: [`../wall-watch/`](../wall-watch/)
|
||||
runs on the **server** host and pages you when a wall goes quiet.
|
||||
|
||||
---
|
||||
|
||||
@@ -325,6 +330,176 @@ convenient IP.
|
||||
|
||||
---
|
||||
|
||||
## When the wall hangs: making it come back by itself
|
||||
|
||||
On 2026-11-02 the reference wall hung solid at 14:03Z. Nothing said so, nothing
|
||||
recovered it, and it stayed that way until someone walked over and powered it off.
|
||||
Everything in this section exists so the next one is a 90-second blink instead of
|
||||
a drive.
|
||||
|
||||
**What the hang actually looked like** — worth reading, because every easy
|
||||
diagnosis was wrong:
|
||||
|
||||
| Signal | Reading |
|
||||
|---|---|
|
||||
| `ping` | answered, 0% loss, RTT jittering 1.9–8.2 ms on a LAN |
|
||||
| `ssh` | **`connect()` succeeded**, then the socket closed 6–13 ms later with **zero bytes**. No SSH banner ever, and `sshd` never logged an attempt. |
|
||||
| the service | the wall's `/ws` dropped at 14:03:04Z after 1212 s and never reconnected; `/healthz` → `viewers:0` (it is normally `viewers:1, watching:1`) |
|
||||
| journal | every unit goes silent at 09:01:59 and stays silent for the 13 minutes until the power cut |
|
||||
| memory | 62 GiB installed, 2.5 GiB in use, **zero OOM lines** in the previous boot |
|
||||
| `pstore` | empty; the next boot reported `system.journal corrupted or uncleanly shut down` — the hard reset, and the reason the journal's tail is lost |
|
||||
|
||||
The trap: **ICMP replies and TCP handshakes are kernel work.** "It pings" and
|
||||
"port 22 is open" say nothing about userspace. Fedora starts `sshd` per
|
||||
connection, so a master that cannot fork answers the SYN from the listen socket
|
||||
and then hangs up without a byte — which is exactly what happened. Put together:
|
||||
the kernel was alive and no userspace process could be started. Not memory, not a
|
||||
full disk, not a fail2ban ban (a ban refuses the SYN, it does not complete the
|
||||
handshake) — a hard hang that left no trace.
|
||||
|
||||
**Two layers, because neither one covers the other:**
|
||||
|
||||
| Layer | File | Catches |
|
||||
|---|---|---|
|
||||
| kernel escalates to a reboot | `/etc/sysctl.d/90-mirror-wall-selfheal.conf` | soft lockup, hard lockup (NMI watchdog), any oops → panic; `kernel.panic = 30` reboots 30 s later |
|
||||
| the chipset reboots the kernel | `/etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf` | PID 1 stops kicking `/dev/watchdog` — i.e. the case above, where the kernel is too wedged to notice it is wedged |
|
||||
|
||||
Verify all of it (or just run `check-kiosk.sh`, which asserts every line):
|
||||
|
||||
```bash
|
||||
ssh mirror 'systemctl show -p RuntimeWatchdogUSec --value' # 30s
|
||||
ssh mirror 'cat /sys/class/watchdog/watchdog0/status' # 0x8100 on the reference box
|
||||
ssh mirror 'sysctl kernel.softlockup_panic kernel.hardlockup_panic kernel.panic_on_oops kernel.panic'
|
||||
# 1 1 1 30
|
||||
```
|
||||
|
||||
`status` is a bitmask of driver features *and* current state, so do not compare it
|
||||
for equality: the reference box reports `0x8100` = `WDIOF_KEEPALIVEPING` (PID 1 is
|
||||
currently kicking it) plus `WDIOF_MAGICCLOSE` (a clean shutdown stops it, so the
|
||||
watchdog cannot reset a box that is deliberately powered off). The bit that matters
|
||||
is `0x8000`. `check-kiosk.sh` asserts `RuntimeWatchdogUSec` instead, which is not
|
||||
ambiguous.
|
||||
|
||||
**Why 30 seconds.** It is the timeout `iTCO_wdt` initialises itself with
|
||||
(`heartbeat=30 sec`), and systemd kicks at a third of whatever it is handed. Asking
|
||||
a device for more than it supports would make systemd kick *less* often than the
|
||||
hardware expires — a reboot loop of our own making — so `install-kiosk.sh` clamps
|
||||
to the driver's own value. Boxes with no watchdog at all (most VMs) get the sysctls
|
||||
and a line saying so.
|
||||
|
||||
**The cost, stated plainly.** Any stall of PID 1 longer than 30 s now reboots the
|
||||
wall — a genuinely stuck disk, say. On an unattended display that is the intended
|
||||
trade, but it is a trade; to back off raise the value or set it to 0 and rely on
|
||||
the sysctls alone (the drop-in documents both).
|
||||
|
||||
**What it still cannot do.** `kernel.hung_task_panic` does not exist in Fedora's
|
||||
kernel build (`CONFIG_DETECT_HUNG_TASK` is off), so a pure blocked-on-I/O hang is
|
||||
caught by the watchdog rather than by a targeted panic — and a machine that is
|
||||
*off* (dead PSU, tripped plug, firmware that never reaches the bootloader) cannot
|
||||
watchdog anything. That remains an ops decision.
|
||||
|
||||
---
|
||||
|
||||
## The wall's firewall
|
||||
|
||||
Fedora's `FedoraWorkstation` zone — the default on Silverblue — ships with:
|
||||
|
||||
```
|
||||
ports: 1025-65535/udp 1025-65535/tcp 443/tcp 80/tcp
|
||||
```
|
||||
|
||||
The **UDP** half is load-bearing: media is peer-to-peer and mirror deliberately
|
||||
uses no STUN/TURN, so a wall that cannot receive UDP on high ports shows
|
||||
`unreachable` tiles. The **TCP** half is pure exposure on a box whose only access
|
||||
control is its address (A19) — it made every high port on the wall connectable
|
||||
from anywhere the wall was reachable, and a wall never serves anything.
|
||||
|
||||
`install-kiosk.sh` step 8/9 removes `1025-65535/tcp`, leaves the UDP alone, and
|
||||
says which of the two it found; `check-kiosk.sh` FAILs if either half moves.
|
||||
|
||||
This was measured, not assumed. With the TCP range gone, the wall's WebRTC socket
|
||||
process still had ICE candidates bound on LAN IPv4 and public IPv6, and media kept
|
||||
flowing inbound:
|
||||
|
||||
```
|
||||
wall NIC over 6s: rx 459.6 kbit/s tx 16.6 kbit/s
|
||||
```
|
||||
|
||||
while inbound probes from a same-subnet host went `tcp/22` reachable and
|
||||
`tcp/8080`, `tcp/9999` silent. (To see the ICE sockets, grep for
|
||||
`Socket Process` — Firefox's WebRTC sockets are not owned by a process named
|
||||
`firefox`.)
|
||||
|
||||
Still open and not needed by a wall: `80/tcp`, `443/tcp`, `samba-client`. Left
|
||||
alone because they are not this change.
|
||||
|
||||
---
|
||||
|
||||
## The wall went quiet and nobody noticed
|
||||
|
||||
`/healthz` knew at 14:03:04Z that nothing was watching. It is a JSON endpoint and
|
||||
nothing was reading it, so the outage was discovered by looking at the panel. The
|
||||
other half of running an unattended wall is being told about it —
|
||||
[`../wall-watch/`](../wall-watch/) checks presence on the server host every minute
|
||||
and pages over ntfy, then announces the recovery.
|
||||
|
||||
```bash
|
||||
tar czf - -C docs wall-watch | ssh server \
|
||||
'cd /var/tmp && tar xzf - && cd wall-watch && sudo sh install-wall-watch.sh'
|
||||
```
|
||||
|
||||
Test it the honest way — a **drill**, labelled as one in the title, body and
|
||||
priority so it can never be mistaken for an incident:
|
||||
|
||||
```bash
|
||||
sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=http://127.0.0.1:1/healthz \
|
||||
WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test sh /usr/local/libexec/mirror-wall-watch.sh'
|
||||
```
|
||||
|
||||
Two things worth knowing: it probes loopback `/healthz`, which works only because
|
||||
that path is exempt from the address gate (see `AGENTS.md` §3); and it cannot
|
||||
detect the **server** being down, because it lives there. `scp` to some of these
|
||||
hosts dies with `lost connection` (no sftp subsystem) — the tar pipe above works
|
||||
everywhere.
|
||||
|
||||
---
|
||||
|
||||
## Why there is no RDP on the wall
|
||||
|
||||
You cannot see the panel, so of course the idea is to look at it remotely.
|
||||
`gnome-remote-desktop` is already in the Silverblue base image, so it looks free.
|
||||
It does not work here, for a structural reason:
|
||||
|
||||
**`grd` does not capture a screen.** It is a *client* of mutter's remote-desktop
|
||||
D-Bus API, and gnome-kiosk embeds libmutter without exporting it. Ask the wall's
|
||||
own session bus:
|
||||
|
||||
```bash
|
||||
sudo -u kiosk XDG_RUNTIME_DIR=/run/user/1001 \
|
||||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus \
|
||||
busctl call org.gnome.Mutter.RemoteDesktop /org/gnome/Mutter/RemoteDesktop \
|
||||
org.freedesktop.DBus.Peer Ping
|
||||
# Call failed: The name is not activatable (same for .ScreenCast and .DisplayConfig)
|
||||
```
|
||||
|
||||
Not a missing package: gnome-shell 50.6 *is* installed and is what exports those
|
||||
names; gnome-kiosk 50.1 does not. So `grdctl rdp enable` and
|
||||
`grdctl rdp set-credentials` would give you a running daemon with nothing to
|
||||
share. For whoever tries next:
|
||||
|
||||
- this image ships **`grdctl`, not `grdc`**, and has **no `org.gnome.RemoteDesktop`
|
||||
gsettings schema** — configuration is grdctl/JSON, so the `DCONF_PROFILE` trap
|
||||
above does not even apply to it.
|
||||
- the only way to really see the panel over RDP is to move the wall onto a
|
||||
gnome-shell-based session. That replaces the script session this whole runbook
|
||||
is built on — the never-exit loop, `window-config.ini`, the greeter trap — and is
|
||||
a `PLAN.md` decision with a full re-validation, not a config tweak.
|
||||
- **and it would not have survived the hang above.** An RDP server is userspace;
|
||||
the failure was userspace being unable to start. RDP is a way to *look*, never a
|
||||
rescue path. The rescue path is the watchdog.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause and fix |
|
||||
@@ -341,6 +516,10 @@ convenient IP.
|
||||
| Firefox shows a restore-session or first-run page | the policy file is missing or unreadable — `sudo cat /etc/firefox/policies/policies.json` |
|
||||
| Tile says `unreachable` | the sharer and the kiosk could not find each other over ICE. Both must be on a network where they can send UDP directly; mirror uses no STUN/TURN by design |
|
||||
| Wall shows the "not allowed" page | the kiosk's address is outside `MIRROR_ALLOW_CIDRS` — this is admission control working, see `SECURITY.md` |
|
||||
| **`ssh` connects, then closes in milliseconds with no banner** | Not a firewall ban — a ban refuses the SYN. The kernel is answering from the listen socket while `sshd` cannot spawn a session: the box is **hung**, and nothing userspace on it, including this checker, will run. The hardware watchdog should reset it within a minute. |
|
||||
| It pings and the port is open, so why is it broken? | ICMP replies and TCP handshakes are produced by the kernel. They say nothing about userspace. Trust `viewers` on `/healthz`, not `ping`. |
|
||||
| Wall reboots on its own | Either the kernel panicked or PID 1 missed the watchdog — that is the design working. Find out which before it happens again: `journalctl -b -1 -p err`, `journalctl -b -1 -k \| tail`, and `ls /sys/fs/pstore` (this box has no ERST backend, so a panic trace may not survive). |
|
||||
| `install-kiosk.sh` pointed the wall somewhere else | It no longer can: an existing `kiosk.conf` URL wins over the built-in default. Pass `KIOSK_URL=...` to change it deliberately. |
|
||||
|
||||
Debugging rule for this project applies: reproduce at the lowest layer that can
|
||||
fail. `curl` the URL from the client, check `/api/streams` on the server, and
|
||||
@@ -367,8 +546,11 @@ decision, not an oversight.
|
||||
`rpm-ostreed-automatic.timer`.
|
||||
* **SELinux is left enforcing, unmodified.** Nothing here needed a policy tweak;
|
||||
treat a new AVC as a bug in this runbook.
|
||||
* **No reboot watchdog.** If the wall is remote enough that a hang means driving
|
||||
to it, that is an ops decision (greenboot, or a smart PDU), not a client one.
|
||||
* **No power cycle.** A hang is now recovered by the kernel and the chipset
|
||||
watchdogs above. A machine that is *off* — dead PSU, tripped plug, firmware that
|
||||
never reaches the bootloader — cannot reboot itself, and if the wall is remote
|
||||
enough for that to matter it is still an ops decision (greenboot, or a smart
|
||||
PDU), not a client one.
|
||||
|
||||
## Naming on the reference network
|
||||
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# 90-mirror-wall-selfheal.conf — make a hung wall recover on its own.
|
||||
#
|
||||
# Installed by install-kiosk.sh to /etc/sysctl.d/. Applied at boot by
|
||||
# systemd-sysctl, and immediately by `sysctl --system` in the installer.
|
||||
#
|
||||
# The trade this makes: a wall whose kernel has oopsed, deadlocked or soft-locked
|
||||
# is a panel showing a frozen frame forever, in a room nobody is in. Rebooting
|
||||
# into itself is always the better outcome, so every one of these escalates to a
|
||||
# reboot. Reasoning and the incident behind it: ../kiosk-silverblue.md
|
||||
# ("When the wall hangs").
|
||||
#
|
||||
# To revert a single line: drop a file with a higher-numbered name in the same
|
||||
# directory, e.g. /etc/sysctl.d/91-mirror-wall-quiet.conf containing
|
||||
# "kernel.panic_on_oops = 0", then `sudo sysctl --system`.
|
||||
|
||||
# The lockup detectors themselves (default on, pinned here so a future base
|
||||
# image change cannot silently turn them off).
|
||||
kernel.watchdog = 1
|
||||
|
||||
# A soft lockup (a CPU spinning in kernel mode with preemption off) or a hard
|
||||
# lockup (interrupts disabled; caught by the NMI watchdog) panics instead of
|
||||
# only printing "watchdog: BUG: soft lockup" into a journal nobody reads.
|
||||
kernel.softlockup_panic = 1
|
||||
kernel.hardlockup_panic = 1
|
||||
|
||||
# An oops used to leave the box nominally up with its display driver dead — the
|
||||
# worst case, because PID 1 stays healthy and so nothing else would reset it.
|
||||
kernel.panic_on_oops = 1
|
||||
|
||||
# Panic, then reboot 30s later. Long enough for the console to be read and for
|
||||
# the filesystems to be sync'ed by the panic path; short enough that nobody
|
||||
# waits for it. 0 would mean "sit there forever", which is the bug.
|
||||
kernel.panic = 30
|
||||
|
||||
# Not set here on purpose: kernel.hung_task_panic does not exist in the Fedora
|
||||
# 44 kernel build (CONFIG_DETECT_HUNG_TASK is off). A task blocked on I/O is
|
||||
# caught by the watchdog in 90-mirror-wall-watchdog.conf instead.
|
||||
@@ -0,0 +1,29 @@
|
||||
# 90-mirror-wall-watchdog.conf — PID 1 must kick the hardware watchdog.
|
||||
#
|
||||
# Installed by install-kiosk.sh to /etc/systemd/system.conf.d/, but ONLY when the
|
||||
# box actually has a watchdog device (/dev/watchdog). Takes effect on
|
||||
# `systemctl daemon-reexec`, which the installer runs; a reboot also picks it up.
|
||||
#
|
||||
# This is the layer that catches what the sysctls cannot. The soft/hard lockup
|
||||
# detectors need kernel work to keep happening; a hardware watchdog needs the
|
||||
# opposite — if PID 1 stops writing /dev/watchdog0 for one whole timeout, the
|
||||
# chipset resets the machine no matter what the kernel thinks it is doing. That
|
||||
# includes the case we hit on 2026-11-02, where the box answered ARP and TCP
|
||||
# handshakes from kernel state while no userspace process could be started.
|
||||
#
|
||||
# 30s is not an arbitrary number: it is the timeout iTCO_wdt initialised itself
|
||||
# with ("heartbeat=30 sec"). systemd kicks at a third of the configured timeout,
|
||||
# so 30s means a kick every 10s. Asking for more than the device supports would
|
||||
# make systemd kick less often than the hardware expires — a self-inflicted
|
||||
# reboot loop — so install-kiosk.sh keeps this at or below the driver's value.
|
||||
#
|
||||
# Cost: a PID-1 stall longer than 30s (a genuinely stuck disk, say) reboots the
|
||||
# wall. On an unattended display that is the intended behaviour. To back off,
|
||||
# raise the number to something under the device timeout, or set 0 and rely on
|
||||
# the sysctls alone:
|
||||
#
|
||||
# printf '[Manager]\nRuntimeWatchdogSec=0\n' | sudo tee /etc/systemd/system.conf.d/91-mirror-wall-watchdog-off.conf
|
||||
# sudo systemctl daemon-reexec
|
||||
|
||||
[Manager]
|
||||
RuntimeWatchdogSec=30s
|
||||
@@ -157,6 +157,53 @@ if [ -n "$HOME_DIR" ]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
printf '\n%s\n' '--- hang recovery ---'
|
||||
# An unattended wall has to end a hang with a reboot, not with a frozen frame for
|
||||
# a week. Two independent layers; ../kiosk-silverblue.md ("When the wall hangs").
|
||||
sc() { sysctl -n "$1" 2>/dev/null || echo missing; }
|
||||
for k in kernel.softlockup_panic kernel.hardlockup_panic kernel.panic_on_oops; do
|
||||
v=$(sc "$k")
|
||||
case "$v" in
|
||||
1) ok "$k = 1" ;;
|
||||
missing) warn "$k does not exist in this kernel build"
|
||||
warn " (kernel.hung_task_panic is absent from Fedora's kernel; the watchdog covers it)" ;;
|
||||
*) bad "$k = $v, expected 1 — fix /etc/sysctl.d/90-mirror-wall-selfheal.conf, then sudo sysctl --system" ;;
|
||||
esac
|
||||
done
|
||||
P=$(sc kernel.panic)
|
||||
case "$P" in
|
||||
''|0|missing) bad "kernel.panic = ${P:-unset}: after a panic the box sits there forever" ;;
|
||||
*) ok "kernel.panic = $P (reboots $P s after a panic)" ;;
|
||||
esac
|
||||
if [ -e /dev/watchdog ]; then
|
||||
WD=$(systemctl show -p RuntimeWatchdogUSec --value 2>/dev/null || echo 0)
|
||||
WID=$(cat /sys/class/watchdog/watchdog0/identity 2>/dev/null || echo '?')
|
||||
case "$WD" in
|
||||
0|''|infinity) bad "PID 1 is not kicking the $WID watchdog (RuntimeWatchdogUSec=$WD) — a wedged kernel here never recovers" ;;
|
||||
*) ok "hardware watchdog armed: $WID at $WD (no kick from PID 1 for $WD = chipset reset)" ;;
|
||||
esac
|
||||
else
|
||||
info "no /dev/watchdog on this box — the lockup panics are the only recovery layer"
|
||||
fi
|
||||
|
||||
printf '\n%s\n' '--- firewall ---'
|
||||
if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then
|
||||
ZONE=$(firewall-cmd --get-default-zone)
|
||||
PORTS=$(firewall-cmd --zone="$ZONE" --list-ports | tr ' ' '\n')
|
||||
if printf '%s\n' "$PORTS" | grep -qx '1025-65535/tcp'; then
|
||||
bad "$ZONE opens 1025-65535/tcp — on a box whose only access control is its address, that is every high port"
|
||||
else
|
||||
ok "$ZONE: 1025-65535/tcp closed"
|
||||
fi
|
||||
if printf '%s\n' "$PORTS" | grep -qx '1025-65535/udp'; then
|
||||
ok "$ZONE: 1025-65535/udp open (WebRTC media needs it — do not close it)"
|
||||
else
|
||||
bad "$ZONE: 1025-65535/udp closed — peer-to-peer media cannot flow, every tile reads 'unreachable'"
|
||||
fi
|
||||
else
|
||||
info "firewalld is not running — nothing listening, nothing to tighten"
|
||||
fi
|
||||
|
||||
printf '\n%s\n' '--- what the screen will show ---'
|
||||
if [ -n "$URL" ]; then
|
||||
BODY=$(curl -fsSL -m 8 "$URL" 2>/dev/null)
|
||||
|
||||
@@ -15,7 +15,10 @@
|
||||
set -eu
|
||||
|
||||
KIOSK_USER="${KIOSK_USER:-kiosk}"
|
||||
KIOSK_URL="${KIOSK_URL:-https://mirror.reeseapps.com/watch}"
|
||||
# The default is for a NEW client only. An existing client keeps its own URL
|
||||
# unless the operator names one — see "the URL is the one per-client value" below.
|
||||
KIOSK_URL_DEFAULT="https://mirror.reeseapps.com/watch"
|
||||
KIOSK_URL="${KIOSK_URL:-}"
|
||||
KIOSK_SESSION="gnome-kiosk-script-wayland"
|
||||
HERE=$(cd "$(dirname "$0")" && pwd)
|
||||
NEED_REBOOT=0
|
||||
@@ -58,7 +61,7 @@ for c in rpm getent restorecon useradd sed awk grep install; do need "$c"; done
|
||||
[ -f /etc/os-release ] && grep -q '^ID=fedora' /etc/os-release ||
|
||||
say "warning: this does not look like Fedora; continuing anyway"
|
||||
|
||||
say "== 1/7 packages =="
|
||||
say "== 1/9 packages =="
|
||||
if rpm -q gnome-kiosk gnome-kiosk-script-session >/dev/null 2>&1; then
|
||||
say "gnome-kiosk + gnome-kiosk-script-session already present"
|
||||
else
|
||||
@@ -69,7 +72,7 @@ else
|
||||
say "layered the kiosk packages — a reboot is required before they exist"
|
||||
fi
|
||||
|
||||
say "== 2/7 kiosk user =="
|
||||
say "== 2/9 kiosk user =="
|
||||
if id -u "$KIOSK_USER" >/dev/null 2>&1; then
|
||||
HOME_DIR=$(getent passwd "$KIOSK_USER" | cut -d: -f6)
|
||||
else
|
||||
@@ -81,7 +84,7 @@ fi
|
||||
KIOSK_GROUP=$(getent passwd "$KIOSK_USER" | cut -d: -f4)
|
||||
say "user $KIOSK_USER uid=$(id -u "$KIOSK_USER") home=$HOME_DIR"
|
||||
|
||||
say "== 3/7 payload =="
|
||||
say "== 3/9 payload =="
|
||||
# Firefox enterprise policy: no update prompts, no account/sync, no telemetry,
|
||||
# no first-run pages. /etc is mutable on Silverblue and survives upgrades.
|
||||
install -D -m 0644 "$HERE/policies.json" /etc/firefox/policies/policies.json
|
||||
@@ -119,6 +122,19 @@ say " $HOME_DIR/.local/bin/gnome-kiosk-script"
|
||||
|
||||
# The URL is the one per-client value; keep it in its own file so everything
|
||||
# else can stay byte-identical across the fleet.
|
||||
#
|
||||
# Re-running this installer is the documented way to REPAIR a client, and a repair
|
||||
# must not move the wall somewhere else. So an existing kiosk.conf wins over the
|
||||
# default; only an explicit KIOSK_URL= changes it.
|
||||
KIOSK_CONF="$HOME_DIR/.config/mirror/kiosk.conf"
|
||||
if [ -z "$KIOSK_URL" ] && [ -f "$KIOSK_CONF" ]; then
|
||||
KIOSK_URL=$(sed -n 's/^[[:space:]]*MIRROR_KIOSK_URL="\{0,1\}\([^"]*\)"\{0,1\}[[:space:]]*$/\1/p' \
|
||||
"$KIOSK_CONF" | head -1)
|
||||
if [ -n "$KIOSK_URL" ]; then
|
||||
say " keeping this client's existing URL (pass KIOSK_URL=... to change it)"
|
||||
fi
|
||||
fi
|
||||
[ -n "$KIOSK_URL" ] || KIOSK_URL="$KIOSK_URL_DEFAULT"
|
||||
sed "s|^MIRROR_KIOSK_URL=.*|MIRROR_KIOSK_URL=\"$KIOSK_URL\"|" "$HERE/kiosk.conf" \
|
||||
| install -m 0644 -o "$KIOSK_USER" -g "$KIOSK_GROUP" /dev/stdin \
|
||||
"$HOME_DIR/.config/mirror/kiosk.conf"
|
||||
@@ -129,7 +145,7 @@ say " $HOME_DIR/.config/mirror/kiosk.conf -> $KIOSK_URL"
|
||||
install -m 0644 -o "$KIOSK_USER" -g "$KIOSK_GROUP" /dev/null \
|
||||
"$HOME_DIR/.config/gnome-initial-setup-done"
|
||||
|
||||
say "== 4/7 default session =="
|
||||
say "== 4/9 default session =="
|
||||
ACCT="/var/lib/AccountsService/users/$KIOSK_USER"
|
||||
if [ -f "$ACCT" ]; then
|
||||
set_ini "$ACCT" User Session "$KIOSK_SESSION"
|
||||
@@ -141,7 +157,7 @@ fi
|
||||
systemctl restart accounts-daemon.service 2>/dev/null || :
|
||||
say " $ACCT Session=$KIOSK_SESSION"
|
||||
|
||||
say "== 5/7 auto-login =="
|
||||
say "== 5/9 auto-login =="
|
||||
if [ ! -d /etc/gdm ]; then
|
||||
say " gdm is not installed — installing it"
|
||||
rpm-ostree install gdm
|
||||
@@ -156,7 +172,7 @@ if [ -d /etc/gdm ]; then
|
||||
say " /etc/gdm/custom.conf: AutomaticLogin=$KIOSK_USER"
|
||||
fi
|
||||
|
||||
say "== 6/7 console guard =="
|
||||
say "== 6/9 console guard =="
|
||||
# Cheap insurance against the class of bug above: if the kiosk session exists but
|
||||
# something else holds the foreground VT, bring the wall back. Stops short of
|
||||
# ever restarting gdm. Pause it while debugging the console with:
|
||||
@@ -168,6 +184,64 @@ systemctl daemon-reload
|
||||
systemctl enable --now mirror-kiosk-console-guard.timer
|
||||
say " mirror-kiosk-console-guard.timer enabled"
|
||||
|
||||
say "== 7/9 hang recovery =="
|
||||
# An unattended wall that hangs must come back by itself. Two independent layers;
|
||||
# reasoning and the incident behind them in ../kiosk-silverblue.md.
|
||||
install -D -m 0644 "$HERE/90-mirror-wall-selfheal.conf" /etc/sysctl.d/90-mirror-wall-selfheal.conf
|
||||
if sysctl --system >/dev/null 2>&1; then
|
||||
say " soft lockup / hard lockup / oops now panic; panic reboots in $(cat /proc/sys/kernel/panic)s"
|
||||
else
|
||||
say " warning: sysctl --system failed — check: sudo sysctl kernel.softlockup_panic kernel.panic"
|
||||
fi
|
||||
|
||||
# The layer that does not need the kernel to be cooperating: if PID 1 stops
|
||||
# kicking /dev/watchdog, the chipset resets the box. Only configured when the
|
||||
# hardware has one (VMs typically do not).
|
||||
if [ -e /dev/watchdog ]; then
|
||||
wd_identity=$(cat /sys/class/watchdog/watchdog0/identity 2>/dev/null || echo unknown)
|
||||
wd_timeout=$(cat /sys/class/watchdog/watchdog0/timeout 2>/dev/null || echo 0)
|
||||
case "$wd_timeout" in ''|*[!0-9]*) wd_timeout=0 ;; esac
|
||||
install -D -m 0644 "$HERE/90-mirror-wall-watchdog.conf" /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf
|
||||
# systemd kicks at a third of the value it is given. Asking for more than the
|
||||
# device supports would kick less often than the device expires — a reboot
|
||||
# loop of our own making — so clamp to what the driver came up with.
|
||||
if [ "$wd_timeout" -gt 0 ] && [ "$wd_timeout" -ne 30 ]; then
|
||||
sed -i "s|^RuntimeWatchdogSec=.*|RuntimeWatchdogSec=${wd_timeout}s|" \
|
||||
/etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf
|
||||
say " clamped to the device's own ${wd_timeout}s timeout ($wd_identity)"
|
||||
fi
|
||||
# PID 1 re-reads system.conf on re-exec; nothing else picks it up short of a
|
||||
# reboot, and re-exec is safe (sockets and running services survive it).
|
||||
systemctl daemon-reexec
|
||||
say " $wd_identity hardware watchdog armed (PID 1 must kick it or the box resets)"
|
||||
else
|
||||
say " no /dev/watchdog here — hardware layer skipped (lockup panics still installed)"
|
||||
fi
|
||||
|
||||
say "== 8/9 firewall =="
|
||||
# Fedora's Workstation zone opens 1025-65535/tcp and /udp. The UDP half is what
|
||||
# WebRTC media needs (peer-to-peer, both directions, and mirror uses no STUN/TURN);
|
||||
# the TCP half is pure exposure on a box whose only access control is its address.
|
||||
# Dropping it is safe for a wall, which never serves: verified against a live
|
||||
# share (media keeps flowing at ~460 kbit/s inbound with it gone).
|
||||
if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then
|
||||
zone=$(firewall-cmd --get-default-zone)
|
||||
if firewall-cmd --zone="$zone" --list-ports | tr ' ' '\n' | grep -qx '1025-65535/tcp'; then
|
||||
firewall-cmd --permanent --zone="$zone" --remove-port=1025-65535/tcp >/dev/null
|
||||
firewall-cmd --reload >/dev/null
|
||||
say " $zone: dropped 1025-65535/tcp (kept 1025-65535/udp for WebRTC media)"
|
||||
else
|
||||
say " $zone: 1025-65535/tcp already closed"
|
||||
fi
|
||||
if firewall-cmd --zone="$zone" --list-ports | tr ' ' '\n' | grep -qx '1025-65535/udp'; then
|
||||
say " 1025-65535/udp still open — required, do not close it or mirrors go black"
|
||||
else
|
||||
say " warning: 1025-65535/udp is NOT open — peer-to-peer media will fail (tile: unreachable)"
|
||||
fi
|
||||
else
|
||||
say " firewalld not running — nothing to tighten (verify your own firewall keeps UDP media)"
|
||||
fi
|
||||
|
||||
# Keep a copy of the payload on the box. /tmp is cleared at every boot, so a
|
||||
# checker staged there is gone precisely when you want to run it again — after a
|
||||
# reboot, or from a rescue session hours later.
|
||||
@@ -177,10 +251,12 @@ if [ "$HERE" != "/usr/local/share/mirror-kiosk" ]; then
|
||||
say " /usr/local/share/mirror-kiosk (survives reboot; run check-kiosk.sh from here)"
|
||||
fi
|
||||
|
||||
say "== 7/7 selinux labels =="
|
||||
say "== 9/9 selinux labels =="
|
||||
restorecon -RF "$HOME_DIR/.local" "$HOME_DIR/.config" 2>/dev/null || :
|
||||
restorecon -RF /etc/firefox /usr/local/share/gnome-kiosk /usr/local/share/wayland-sessions \
|
||||
/usr/local/libexec 2>/dev/null || :
|
||||
restorecon -RF /etc/sysctl.d/90-mirror-wall-selfheal.conf \
|
||||
/etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf 2>/dev/null || :
|
||||
|
||||
say ""
|
||||
say "Installed."
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# wall-watch — presence alert for the mirror wall
|
||||
|
||||
Runs on the **mirror server host**, not on the kiosk. Once a minute it reads the
|
||||
service's own `/healthz` and pages you over ntfy when nobody is watching, or when
|
||||
the service stops answering — and tells you when the wall comes back.
|
||||
|
||||
Rationale and the incident that produced it:
|
||||
[`../kiosk-silverblue.md`](../kiosk-silverblue.md) → *"The wall went quiet and
|
||||
nobody noticed"*.
|
||||
|
||||
## Install (on the server host)
|
||||
|
||||
```bash
|
||||
scp -r docs/wall-watch server:/var/tmp/ && ssh server \
|
||||
'cd /var/tmp/wall-watch && sudo sh install-wall-watch.sh'
|
||||
sudo vi /etc/mirror-wall-watch.env # set NTFY_TOKEN
|
||||
```
|
||||
|
||||
## Operate
|
||||
|
||||
```bash
|
||||
systemctl status mirror-wall-watch.timer # is it still ticking?
|
||||
journalctl -u mirror-wall-watch.service -f # one line per check, forever
|
||||
sudo sh /usr/local/libexec/mirror-wall-watch.sh # run it by hand
|
||||
|
||||
# Page yourself on purpose, without touching the wall's real state. WALL_DRILL=1
|
||||
# makes the notification say "[drill]" and drop to priority 3, so a test can never
|
||||
# be mistaken for an outage:
|
||||
sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=https://127.0.0.1/nope \
|
||||
WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test \
|
||||
sh /usr/local/libexec/mirror-wall-watch.sh'
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
| File | Lands at |
|
||||
|---|---|
|
||||
| `wall-watch.sh` | `/usr/local/libexec/mirror-wall-watch.sh` |
|
||||
| `mirror-wall-watch.service` / `.timer` | `/etc/systemd/system/` |
|
||||
| `wall-watch.env.example` | `/etc/mirror-wall-watch.env` (0600, seeded once, never overwritten) |
|
||||
| everything here | `/usr/local/share/mirror-wall-watch/` |
|
||||
|
||||
## Deliberate choices
|
||||
|
||||
- **Probes `/healthz` over loopback**, because that endpoint is exempt from the
|
||||
address gate — the same exemption that lets the container healthcheck work from
|
||||
loopback when `MIRROR_ALLOW_CIDRS` excludes it.
|
||||
- **First run cannot alert.** Installing a watcher must not page you about a wall
|
||||
that was already down before you had one.
|
||||
- **A failed notification fails the unit** and does *not* bank the re-notify
|
||||
timestamp, so an ntfy outage is retried on the next tick instead of eating the
|
||||
alert.
|
||||
- **A drill is labelled as one.** `WALL_DRILL=1` prefixes the title with `[drill]`,
|
||||
lowers the priority and says "nothing is wrong" in the body. A test page that is
|
||||
indistinguishable from an incident teaches people to ignore the page.
|
||||
- **It cannot detect the server being down.** It lives there. That gap belongs to
|
||||
whatever monitors this host.
|
||||
@@ -0,0 +1,76 @@
|
||||
#!/bin/sh
|
||||
# install-wall-watch.sh — install the wall-presence alert on the MIRROR SERVER host.
|
||||
#
|
||||
# Run ON THE SERVER, as root, from this directory:
|
||||
#
|
||||
# sudo sh install-wall-watch.sh
|
||||
#
|
||||
# Idempotent. It never overwrites /etc/mirror-wall-watch.env once it exists, so
|
||||
# the ntfy token you pasted in survives a re-run.
|
||||
#
|
||||
# What it does, and why this exists at all: ../kiosk-silverblue.md
|
||||
# ("The wall went quiet and nobody noticed").
|
||||
|
||||
set -eu
|
||||
|
||||
HERE=$(cd "$(dirname "$0")" && pwd)
|
||||
CONF=/etc/mirror-wall-watch.env
|
||||
say() { printf '%s\n' "$*"; }
|
||||
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
[ "$(id -u)" -eq 0 ] || die "run me as root (sudo sh $0)"
|
||||
for c in curl logger systemctl; do
|
||||
command -v "$c" >/dev/null 2>&1 || die "missing required command: $c"
|
||||
done
|
||||
|
||||
say "== 1/4 program =="
|
||||
install -D -m 0755 "$HERE/wall-watch.sh" /usr/local/libexec/mirror-wall-watch.sh
|
||||
install -d -m 0755 /usr/local/share/mirror-wall-watch
|
||||
cp -a "$HERE"/. /usr/local/share/mirror-wall-watch/
|
||||
rm -f /usr/local/share/mirror-wall-watch/install-wall-watch.sh
|
||||
say " /usr/local/libexec/mirror-wall-watch.sh"
|
||||
say " /usr/local/share/mirror-wall-watch (docs, survives reboot)"
|
||||
|
||||
say "== 2/4 config =="
|
||||
if [ ! -f "$CONF" ]; then
|
||||
install -D -m 0600 "$HERE/wall-watch.env.example" "$CONF"
|
||||
say " created $CONF (0600) — NOW SET NTFY_TOKEN IN IT:"
|
||||
say " sudo vi $CONF"
|
||||
else
|
||||
perm=$(stat -c '%a' "$CONF")
|
||||
[ "$perm" = "600" ] || say " warning: $CONF is mode $perm, expected 0600 (it holds a token)"
|
||||
say " kept existing $CONF"
|
||||
fi
|
||||
|
||||
say "== 3/4 units =="
|
||||
install -D -m 0644 "$HERE/mirror-wall-watch.service" /etc/systemd/system/mirror-wall-watch.service
|
||||
install -D -m 0644 "$HERE/mirror-wall-watch.timer" /etc/systemd/system/mirror-wall-watch.timer
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now mirror-wall-watch.timer
|
||||
say " mirror-wall-watch.timer enabled (every minute)"
|
||||
|
||||
say "== 4/4 prove it =="
|
||||
# One run now, so a broken config fails in front of you instead of silently in a
|
||||
# timer at 03:00. The first run only records the baseline; it cannot alert, which
|
||||
# is deliberate — installing the watcher must not page you about a wall that was
|
||||
# already down before you had a watcher.
|
||||
if systemctl start --wait mirror-wall-watch.service 2>/dev/null; then
|
||||
say " first check passed:"
|
||||
journalctl -u mirror-wall-watch.service --no-pager -n 1 2>/dev/null | sed 's/^/ /' || :
|
||||
else
|
||||
say " first check FAILED — read this before trusting the timer:"
|
||||
journalctl -u mirror-wall-watch.service --no-pager -n 12 2>/dev/null | sed 's/^/ /' || :
|
||||
fi
|
||||
# An AVC here means SELinux disagreed with something above; say so rather than
|
||||
# leaving a silently broken alert.
|
||||
if command -v ausearch >/dev/null 2>&1; then
|
||||
if ausearch -m avc -ts recent >/dev/null 2>&1; then
|
||||
say " warning: recent AVCs found — run: sudo ausearch -m avc -ts recent"
|
||||
fi
|
||||
fi
|
||||
|
||||
say ""
|
||||
say "Installed. Fire a test notification (page yourself on purpose):"
|
||||
say " sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=https://127.0.0.1/nope WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test sh /usr/local/libexec/mirror-wall-watch.sh'"
|
||||
say "Then watch the timeline:"
|
||||
say " journalctl -u mirror-wall-watch.service -f"
|
||||
@@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=Check that the mirror wall is still watching
|
||||
Documentation=file:///usr/local/share/mirror-wall-watch/README.md
|
||||
# The probe is only meaningful once nginx (or whatever fronts the service) is up.
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/sh /usr/local/libexec/mirror-wall-watch.sh
|
||||
EnvironmentFile=-/etc/mirror-wall-watch.env
|
||||
# It only ever writes /var/lib/mirror-wall-watch; keep the rest read-only.
|
||||
NoNewPrivileges=yes
|
||||
PrivateTmp=yes
|
||||
ProtectSystem=full
|
||||
ProtectHome=read-only
|
||||
@@ -0,0 +1,17 @@
|
||||
[Unit]
|
||||
Description=Run the mirror wall presence check every minute
|
||||
Documentation=file:///usr/local/share/mirror-wall-watch/README.md
|
||||
|
||||
[Timer]
|
||||
# Check from a minute after boot — the service needs a moment to come up, and a
|
||||
# wall has its own relaunch loop — and then every minute. The jitter keeps a
|
||||
# fleet of these from all waking up on the same second.
|
||||
OnBootSec=1min
|
||||
OnUnitActiveSec=1min
|
||||
RandomizedDelaySec=5s
|
||||
AccuracySec=1s
|
||||
# If the host was suspended or the timer missed its window, run once anyway.
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,28 @@
|
||||
# /etc/mirror-wall-watch.env — config for mirror-wall-watch.timer.
|
||||
# chmod 0600, root:root. It holds the ntfy token. Never commit a filled-in copy.
|
||||
# install-wall-watch.sh seeds this from the example and then NEVER overwrites it.
|
||||
|
||||
# The presence probe. /healthz is exempt from the address gate, so loopback works
|
||||
# even when MIRROR_ALLOW_CIDRS excludes 127.0.0.0/8 — which is why the container
|
||||
# healthcheck works too. Use whatever loopback URL reaches this deployment.
|
||||
WALL_HEALTH_URL=https://127.0.0.1/healthz
|
||||
|
||||
# Alert when fewer than this many viewers are attached. Set 0 to stop alerting on
|
||||
# "nobody is watching" and only alert when the service itself stops answering
|
||||
# (useful while a fleet has no wall plugged in).
|
||||
WALL_EXPECT_VIEWERS=1
|
||||
|
||||
# Consecutive bad checks before the first notification (~1 min per check). Three
|
||||
# rides out a Firefox relaunch on the wall; raise it if that is normal for you.
|
||||
WALL_THRESHOLD=3
|
||||
|
||||
# Keep shouting while it stays down. 0 = alert once per incident.
|
||||
WALL_RENOTIFY_SECS=3600
|
||||
|
||||
# Optional: the URL a notification opens when tapped.
|
||||
WALL_STATUS_URL=https://mirror.reeseapps.com/healthz
|
||||
|
||||
# ntfy (outbound only — this never reads anything back).
|
||||
NTFY_URL=https://ntfy.reeseapps.com
|
||||
NTFY_TOKEN=
|
||||
NTFY_TOPIC=pi
|
||||
@@ -0,0 +1,213 @@
|
||||
#!/bin/sh
|
||||
# wall-watch.sh — know within minutes that the mirror wall stopped watching.
|
||||
#
|
||||
# Run ON THE MIRROR SERVER HOST from mirror-wall-watch.timer (once a minute).
|
||||
# It reads the service's own /healthz over loopback and posts to ntfy when no
|
||||
# viewer is attached, or when the service stops answering at all — and again when
|
||||
# the wall comes back.
|
||||
#
|
||||
# Why it exists: on 2026-11-02 the reference wall hard-hung at 14:03Z. The
|
||||
# service knew the instant it happened — /healthz reported viewers:0 — and said
|
||||
# nothing, because nothing was listening. The failure was eventually found by
|
||||
# walking up to the panel, which was in another room of another building.
|
||||
#
|
||||
# /healthz is the right probe precisely because it is exempt from the address
|
||||
# gate (AGENTS.md §3): a loopback check works even when MIRROR_ALLOW_CIDRS
|
||||
# excludes loopback, which is also why the container healthcheck can work.
|
||||
#
|
||||
# Config: /etc/mirror-wall-watch.env (root-owned 0600 — it holds the ntfy token)
|
||||
# WALL_HEALTH_URL default https://127.0.0.1/healthz
|
||||
# WALL_EXPECT_VIEWERS default 1; set 0 to stop alerting on "nobody watching"
|
||||
# WALL_THRESHOLD default 3 consecutive bad checks (~3 min) before shouting
|
||||
# WALL_RENOTIFY_SECS default 3600; keeps shouting while it stays down
|
||||
# WALL_STATUS_URL optional; the URL a notification opens
|
||||
# NTFY_URL NTFY_TOKEN NTFY_TOPIC
|
||||
#
|
||||
# Manual run (the timer runs it the same way):
|
||||
# sudo sh /usr/local/libexec/mirror-wall-watch.sh
|
||||
# sudo sh -c 'WALL_HEALTH_URL=https://127.0.0.1/nope WALL_THRESHOLD=1 sh /usr/local/libexec/mirror-wall-watch.sh' # fire a test alert
|
||||
#
|
||||
# Honest limit: this runs on the server, so it cannot tell you the *server* is
|
||||
# down. That is a different alert, and it belongs to whatever watches this host.
|
||||
|
||||
set -eu
|
||||
|
||||
CONF="${WALL_WATCH_CONF:-/etc/mirror-wall-watch.env}"
|
||||
STATE_DIR="${WALL_STATE_DIR:-/var/lib/mirror-wall-watch}"
|
||||
STATE_FILE="$STATE_DIR/state"
|
||||
|
||||
log() {
|
||||
printf '%s\n' "$*"
|
||||
# Under systemd stdout already IS the journal, so writing to /dev/log as well
|
||||
# puts two copies of every check in it. Keep the syslog trail for interactive
|
||||
# runs, where stdout is a terminal and nothing else would record it.
|
||||
[ -t 1 ] && logger -t mirror-wall-watch -- "$*" 2>/dev/null
|
||||
return 0
|
||||
}
|
||||
die() { log "error: $*"; exit 1; }
|
||||
|
||||
command -v curl >/dev/null 2>&1 || die "curl is required"
|
||||
# Root is not inherently required, only write access to the state directory — but
|
||||
# the default lives next to root-only config, so a non-root run must say where it
|
||||
# is going to fail rather than fail at the interesting moment.
|
||||
if ! install -d -m 0700 "$STATE_DIR" 2>/dev/null; then
|
||||
die "cannot create $STATE_DIR — run me as root, or set WALL_STATE_DIR somewhere writable"
|
||||
fi
|
||||
|
||||
# Apply KEY=VALUE from the config file, but only for variables the caller did not
|
||||
# already set. An explicit environment must win over the file — the same precedence
|
||||
# systemd gives Environment= over EnvironmentFile= — otherwise "run the checker
|
||||
# against a different URL for a moment" silently does nothing, which is exactly the
|
||||
# kind of thing you only discover when the alert you were testing never arrives.
|
||||
load_conf() {
|
||||
[ -r "$1" ] || return 0
|
||||
while IFS= read -r line || [ -n "$line" ]; do
|
||||
case "$line" in ''|'#'*) continue ;; esac
|
||||
key=${line%%=*}
|
||||
val=${line#*=}
|
||||
key=$(printf '%s' "$key" | tr -d '[:space:]')
|
||||
val=$(printf '%s' "$val" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
|
||||
case "$val" in '\"'*'"') val=${val#\"}; val=${val%\"} ;; esac
|
||||
# Only sane shell identifiers get assigned.
|
||||
case "$key" in ''|*[!A-Za-z0-9_]*|[0-9]*) continue ;; esac
|
||||
eval "cur=\${$key:-}"
|
||||
[ -n "$cur" ] || eval "$key=\$val"
|
||||
done <"$1"
|
||||
}
|
||||
load_conf "$CONF"
|
||||
|
||||
: "${WALL_HEALTH_URL:=https://127.0.0.1/healthz}"
|
||||
: "${WALL_EXPECT_VIEWERS:=1}"
|
||||
: "${WALL_THRESHOLD:=3}"
|
||||
: "${WALL_RENOTIFY_SECS:=3600}"
|
||||
|
||||
# A drill is an alert fired on purpose to prove the path works. It must never be
|
||||
# mistaken for an incident, so it says so in the title, the body and the priority
|
||||
# — an alerting tool whose test page looks like a real page trains people to
|
||||
# ignore it.
|
||||
: "${WALL_DRILL:=0}"
|
||||
if [ "$WALL_DRILL" = "1" ]; then
|
||||
DRILL_TAG="[drill] "
|
||||
DRILL_NOTE="> **This is a drill.** Someone ran the test command on \`$(hostname)\`; nothing is wrong with the wall. What it checked is the path: probe -> ntfy -> your phone."
|
||||
DOWN_PRIORITY=3
|
||||
else
|
||||
DRILL_TAG=""
|
||||
DRILL_NOTE=""
|
||||
DOWN_PRIORITY=4
|
||||
fi
|
||||
|
||||
# Posts one notification. Returns non-zero instead of exiting: a failed alert
|
||||
# must not stop the state file being written, or the next run loses the counter
|
||||
# and the wall pages once a minute forever.
|
||||
notify() {
|
||||
prio=$1 tags=$2 title=$3 body=$4
|
||||
if [ -z "${NTFY_URL:-}" ] || [ -z "${NTFY_TOPIC:-}" ]; then
|
||||
log "error: ntfy is not configured (NTFY_URL/NTFY_TOPIC in $CONF)"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Collect the headers in the positional parameters, and only the ones that
|
||||
# have values, so ntfy never receives an empty Authorization or X-Click.
|
||||
set -- -H "X-Priority: $prio" -H "X-Tags: $tags" -H "X-Title: $title" -H "X-Markdown: yes"
|
||||
[ -n "${NTFY_TOKEN:-}" ] && set -- "$@" -H "Authorization: Bearer $NTFY_TOKEN"
|
||||
[ -n "${WALL_STATUS_URL:-}" ] && set -- "$@" -H "X-Click: $WALL_STATUS_URL"
|
||||
|
||||
if curl -fsS -m 10 "$@" -d "$body" "$NTFY_URL/$NTFY_TOPIC" >/dev/null 2>&1; then
|
||||
log "notified (priority $prio): $title"
|
||||
else
|
||||
log "error: could not reach $NTFY_URL — the alert did not go out"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ---- read the service's own truth -------------------------------------------------
|
||||
probe=""
|
||||
viewers=""
|
||||
if probe=$(curl -fsS -m 8 -k "$WALL_HEALTH_URL" 2>/dev/null); then
|
||||
if command -v jq >/dev/null 2>&1; then
|
||||
viewers=$(printf '%s' "$probe" | jq -r '.viewers // empty' 2>/dev/null || :)
|
||||
else
|
||||
viewers=$(printf '%s' "$probe" |
|
||||
sed -n 's/.*"viewers"[[:space:]]*:[[:space:]]*\([0-9][0-9]*\).*/\1/p' | head -1)
|
||||
fi
|
||||
# 200 but no parseable viewer count is a broken answer, not a healthy one.
|
||||
[ -n "$viewers" ] || probe=""
|
||||
fi
|
||||
|
||||
if [ -z "$probe" ]; then
|
||||
condition="unreachable"
|
||||
summary="the mirror service did not answer \`$WALL_HEALTH_URL\`"
|
||||
elif [ "$WALL_EXPECT_VIEWERS" -gt 0 ] && [ "$viewers" -lt "$WALL_EXPECT_VIEWERS" ]; then
|
||||
condition="quiet"
|
||||
summary="the service is up but **nothing is watching** (viewers=$viewers)"
|
||||
else
|
||||
condition="ok"
|
||||
summary="viewers=$viewers"
|
||||
fi
|
||||
|
||||
# ---- state machine ----------------------------------------------------------------
|
||||
state="ok"
|
||||
bad=0
|
||||
last_alert=0
|
||||
if [ -r "$STATE_FILE" ]; then
|
||||
# shellcheck disable=SC1090
|
||||
. "$STATE_FILE" 2>/dev/null || :
|
||||
fi
|
||||
now=$(date +%s)
|
||||
|
||||
NOTIFY_FAILED=0
|
||||
|
||||
if [ "$condition" = "ok" ]; then
|
||||
if [ "$state" != "ok" ]; then
|
||||
notify 3 "heavy_check_mark,recovered" "${DRILL_TAG}Wall back" \
|
||||
"**${DRILL_TAG}The mirror wall is watching again.**
|
||||
|
||||
was: \`$state\`
|
||||
now: $summary
|
||||
host: \`$(hostname)\` · $(date -u '+%Y-%m-%d %H:%M:%SZ')
|
||||
|
||||
$DRILL_NOTE" || NOTIFY_FAILED=1
|
||||
fi
|
||||
state="ok"; bad=0; last_alert=0
|
||||
else
|
||||
if [ "$state" = "$condition" ]; then bad=$((bad + 1)); else bad=1; fi
|
||||
if [ "$bad" -ge "$WALL_THRESHOLD" ] &&
|
||||
{ [ "$state" != "$condition" ] || [ $((now - last_alert)) -ge "$WALL_RENOTIFY_SECS" ]; }; then
|
||||
hint="Recover it: \`ssh <wall>\` then \`sudo sh /usr/local/share/mirror-kiosk/check-kiosk.sh\`; if ssh itself will not open a session the box is hung, not offline — the watchdog should reset it within a minute of this, and if it does not, power-cycle it."
|
||||
[ "$condition" = "unreachable" ] &&
|
||||
hint="Check the service first (\`systemctl status\`, container logs) before the wall: this host may be the thing that is broken."
|
||||
notify "$DOWN_PRIORITY" "rotating_light,wall-down" "${DRILL_TAG}Wall down" \
|
||||
"**${DRILL_TAG}Mirror wall: $condition**
|
||||
|
||||
$summary
|
||||
|
||||
\`\`\`
|
||||
$probe
|
||||
\`\`\`
|
||||
|
||||
$hint
|
||||
|
||||
host: \`$(hostname)\` · $(date -u '+%Y-%m-%d %H:%M:%SZ') · bad check #$bad
|
||||
|
||||
$DRILL_NOTE" || NOTIFY_FAILED=1
|
||||
# Only bank the re-notify timestamp if it actually went out, so a ntfy
|
||||
# outage is retried on the next tick rather than swallowed.
|
||||
[ "$NOTIFY_FAILED" -eq 0 ] && last_alert=$now
|
||||
fi
|
||||
state="$condition"
|
||||
fi
|
||||
|
||||
umask 077
|
||||
cat >"$STATE_FILE" <<EOF
|
||||
state=$state
|
||||
bad=$bad
|
||||
last_alert=$last_alert
|
||||
EOF
|
||||
|
||||
# One greppable line per run, so `journalctl -u mirror-wall-watch` reads as a
|
||||
# timeline of the wall's health rather than as silence.
|
||||
log "$condition: $summary (state=$state bad=$bad threshold=$WALL_THRESHOLD)"
|
||||
|
||||
# A needed-but-undeliverable alert fails the unit, so `systemctl show`
|
||||
# mirror-wall-watch.service shows why nobody got paged.
|
||||
exit "$NOTIFY_FAILED"
|
||||
Reference in New Issue
Block a user