Experimental/mirror
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(kiosk): self-recover a hung wall and page when it goes quiet

Browse Source 
An unattended wall failed in two ways on 2026-11-02. It hard-hung with the kernel
alive but userspace unable to start a process: ssh completed the handshake and
closed 6-13 ms later with zero bytes and no banner, while 62 GiB RAM and zero OOM
lines rule out memory. Separately, the service knew at 14:03Z that nothing was
watching (healthz viewers:0) and nothing was reading it. It was discovered by
walking over to the panel.

Hang recovery, two independent layers (install step 7/9, asserted by
check-kiosk.sh):

- /etc/sysctl.d/90-mirror-wall-selfheal.conf: soft lockup, hard lockup and any
  oops escalate to a panic, and kernel.panic=30 reboots instead of parking.
- /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf: PID 1 must keep kicking
  the iTCO hardware watchdog or the chipset resets the box, which is the layer that
  covers a kernel too wedged to notice. Clamped to the driver's own 30s because
  systemd kicks at a third of whatever value it is handed.

Presence alerting: docs/wall-watch/ runs on the server host, probes loopback
/healthz (exempt from the address gate) every minute, debounces three checks, pages
over ntfy and then announces the recovery. WALL_DRILL=1 labels a test page in the
title, body and priority so a drill can never be mistaken for an incident.
Verified against a fake ntfy: caller environment wins over the config file, both
transitions fire, and the debounce holds.

Firewall (install step 8/9): drop FedoraWorkstation's 1025-65535/tcp. The UDP half
is load-bearing for peer-to-peer media (mirror deliberately uses no STUN/TURN) and
stays open; the TCP half was every high port on a box whose only access control is
its address. Measured, not assumed: a live share kept flowing at 459.6 kbit/s
inbound with the TCP range closed, and tcp/8080 stopped answering from a
same-subnet host.

Also fixed: install-kiosk.sh overwrote an existing client's kiosk.conf with its
built-in default URL, so re-running it to repair a wall silently repointed it at
the production default. An existing URL now wins and KIOSK_URL= is an explicit
change.

Documented with the evidence: why gnome-remote-desktop cannot share a gnome-kiosk
session (gnome-kiosk exports no org.gnome.Mutter.* names, so grd has nothing to
capture, and this image has grdctl rather than grdc), and why no userspace RDP
would have survived the hang either.

LLM-Generated: true
This commit is contained in:
ducoterra
2026-11-02 09:53:13 -05:00
parent bf027672b5
commit 8fefd83ef8
12 changed files with 796 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+8 -1
View File
@@ -71,7 +71,7 @@ Environment variables (all optional; a `.env` file is loaded if present):
| `MIRROR_MAX_MIRRORS` | `0` | Cap on concurrent mirrors (`0` = unlimited). | | `MIRROR_MAX_MIRRORS` | `0` | Cap on concurrent mirrors (`0` = unlimited). |
| `MIRROR_MAX_BITRATE` | `25000000` | Per-leg WebRTC bitrate cap in bits/s — the mesh uploads N × bitrate per sharer, so this is the ceiling. 25 Mbit/s suits a wired LAN at 1440p; a Wi-Fi sharer with many viewers may need less, not more. | | `MIRROR_MAX_BITRATE` | `25000000` | Per-leg WebRTC bitrate cap in bits/s — the mesh uploads N × bitrate per sharer, so this is the ceiling. 25 Mbit/s suits a wired LAN at 1440p; a Wi-Fi sharer with many viewers may need less, not more. |
| `MIRROR_PING_INTERVAL` | `20` | Seconds between socket liveness pings. | | `MIRROR_PING_INTERVAL` | `20` | Seconds between socket liveness pings. |
| `MIRROR_ALLOW_CIDRS` | *(unset)* | Comma-separated source networks allowed to connect, e.g. `10.2.0.0/22,fd00::/64`. Unset means unrestricted. | | `MIRROR_ALLOW_CIDRS` | *(unset)* | Comma-separated source networks allowed to connect, e.g. `10.2.0.0/22,fd00::/64`. Unset means unrestricted. A wall needs this range too, and its address is DHCP — give it a reservation. |
| `MIRROR_TRUSTED_PROXIES` | `127.0.0.0/8, ::1/128` | Which peers may speak for a client via `X-Forwarded-For`. Only a proxy that overwrites the header belongs here. | | `MIRROR_TRUSTED_PROXIES` | `127.0.0.0/8, ::1/128` | Which peers may speak for a client via `X-Forwarded-For`. Only a proxy that overwrites the header belongs here. |
| `DEBUGPY` | `0` | `1` opens a debugpy listener on `DEBUGPY_PORT`. | | `DEBUGPY` | `0` | `1` opens a debugpy listener on `DEBUGPY_PORT`. |
| `DEBUGPY_PORT` | `5678` | debugpy port. | | `DEBUGPY_PORT` | `5678` | debugpy port. |
@@ -147,6 +147,13 @@ peer state for the media, and a probe page for the rendered resolution. The
runbook records why each file lives where it does on an runbook records why each file lives where it does on an
otherwise read-only OS. otherwise read-only OS.
**When it hangs.** A wall is unattended, so a hang has to end without a person:
the kernel turns lockups and oopses into a reboot, and PID 1 must keep kicking the
hardware watchdog or the chipset resets the box. Being *quiet* is a different
failure than being hung, so the server watches `/healthz` for that and pages over
ntfy ([`docs/wall-watch/`](docs/wall-watch/)). Both are installed by the commands
above and asserted by `check-kiosk.sh`.
**Keys.** The whole wall works from a keyboard — and the box needs no keyboard **Keys.** The whole wall works from a keyboard — and the box needs no keyboard
attached to be operated *or* debugged; the keys are there when one is plugged attached to be operated *or* debugged; the keys are there when one is plugged
in. On `/watch`: in. On `/watch`:
docs/kiosk-silverblue.md
+184 -2
View File
@@ -20,6 +20,11 @@ or to change its URL.
| `mirror-kiosk-console-guard` | `/usr/local/libexec/` | reclaims the console if the wall loses it | | `mirror-kiosk-console-guard` | `/usr/local/libexec/` | reclaims the console if the wall loses it |
| `mirror-kiosk-console-guard.{service,timer}` | `/etc/systemd/system/` | runs that check every minute | | `mirror-kiosk-console-guard.{service,timer}` | `/etc/systemd/system/` | runs that check every minute |
| `dim-probe.py` | run on any reachable host | measures what the kiosk actually renders | | `dim-probe.py` | run on any reachable host | measures what the kiosk actually renders |
| `90-mirror-wall-selfheal.conf` | `/etc/sysctl.d/` | lockups and oopses become a reboot, not a frozen frame |
| `90-mirror-wall-watchdog.conf` | `/etc/systemd/system.conf.d/` | PID 1 must kick the hardware watchdog or the chipset resets the box |
Presence alerting lives beside this, not inside it: [`../wall-watch/`](../wall-watch/)
runs on the **server** host and pages you when a wall goes quiet.
--- ---
@@ -325,6 +330,176 @@ convenient IP.
--- ---
## When the wall hangs: making it come back by itself
On 2026-11-02 the reference wall hung solid at 14:03Z. Nothing said so, nothing
recovered it, and it stayed that way until someone walked over and powered it off.
Everything in this section exists so the next one is a 90-second blink instead of
a drive.
**What the hang actually looked like** — worth reading, because every easy
diagnosis was wrong:
| Signal | Reading |
|---|---|
| `ping` | answered, 0% loss, RTT jittering 1.9–8.2 ms on a LAN |
| `ssh` | **`connect()` succeeded**, then the socket closed 6–13 ms later with **zero bytes**. No SSH banner ever, and `sshd` never logged an attempt. |
| the service | the wall's `/ws` dropped at 14:03:04Z after 1212 s and never reconnected; `/healthz` → `viewers:0` (it is normally `viewers:1, watching:1`) |
| journal | every unit goes silent at 09:01:59 and stays silent for the 13 minutes until the power cut |
| memory | 62 GiB installed, 2.5 GiB in use, **zero OOM lines** in the previous boot |
| `pstore` | empty; the next boot reported `system.journal corrupted or uncleanly shut down` — the hard reset, and the reason the journal's tail is lost |
The trap: **ICMP replies and TCP handshakes are kernel work.** "It pings" and
"port 22 is open" say nothing about userspace. Fedora starts `sshd` per
connection, so a master that cannot fork answers the SYN from the listen socket
and then hangs up without a byte — which is exactly what happened. Put together:
the kernel was alive and no userspace process could be started. Not memory, not a
full disk, not a fail2ban ban (a ban refuses the SYN, it does not complete the
handshake) — a hard hang that left no trace.
**Two layers, because neither one covers the other:**
| Layer | File | Catches |
|---|---|---|
| kernel escalates to a reboot | `/etc/sysctl.d/90-mirror-wall-selfheal.conf` | soft lockup, hard lockup (NMI watchdog), any oops → panic; `kernel.panic = 30` reboots 30 s later |
| the chipset reboots the kernel | `/etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf` | PID 1 stops kicking `/dev/watchdog` — i.e. the case above, where the kernel is too wedged to notice it is wedged |
Verify all of it (or just run `check-kiosk.sh`, which asserts every line):
```bash
ssh mirror 'systemctl show -p RuntimeWatchdogUSec --value' # 30s
ssh mirror 'cat /sys/class/watchdog/watchdog0/status' # 0x8100 on the reference box
ssh mirror 'sysctl kernel.softlockup_panic kernel.hardlockup_panic kernel.panic_on_oops kernel.panic'
# 1 1 1 30
```
`status` is a bitmask of driver features *and* current state, so do not compare it
for equality: the reference box reports `0x8100` = `WDIOF_KEEPALIVEPING` (PID 1 is
currently kicking it) plus `WDIOF_MAGICCLOSE` (a clean shutdown stops it, so the
watchdog cannot reset a box that is deliberately powered off). The bit that matters
is `0x8000`. `check-kiosk.sh` asserts `RuntimeWatchdogUSec` instead, which is not
ambiguous.
**Why 30 seconds.** It is the timeout `iTCO_wdt` initialises itself with
(`heartbeat=30 sec`), and systemd kicks at a third of whatever it is handed. Asking
a device for more than it supports would make systemd kick *less* often than the
hardware expires — a reboot loop of our own making — so `install-kiosk.sh` clamps
to the driver's own value. Boxes with no watchdog at all (most VMs) get the sysctls
and a line saying so.
**The cost, stated plainly.** Any stall of PID 1 longer than 30 s now reboots the
wall — a genuinely stuck disk, say. On an unattended display that is the intended
trade, but it is a trade; to back off raise the value or set it to 0 and rely on
the sysctls alone (the drop-in documents both).
**What it still cannot do.** `kernel.hung_task_panic` does not exist in Fedora's
kernel build (`CONFIG_DETECT_HUNG_TASK` is off), so a pure blocked-on-I/O hang is
caught by the watchdog rather than by a targeted panic — and a machine that is
*off* (dead PSU, tripped plug, firmware that never reaches the bootloader) cannot
watchdog anything. That remains an ops decision.
---
## The wall's firewall
Fedora's `FedoraWorkstation` zone — the default on Silverblue — ships with:
```
ports: 1025-65535/udp 1025-65535/tcp 443/tcp 80/tcp
```
The **UDP** half is load-bearing: media is peer-to-peer and mirror deliberately
uses no STUN/TURN, so a wall that cannot receive UDP on high ports shows
`unreachable` tiles. The **TCP** half is pure exposure on a box whose only access
control is its address (A19) — it made every high port on the wall connectable
from anywhere the wall was reachable, and a wall never serves anything.
`install-kiosk.sh` step 8/9 removes `1025-65535/tcp`, leaves the UDP alone, and
says which of the two it found; `check-kiosk.sh` FAILs if either half moves.
This was measured, not assumed. With the TCP range gone, the wall's WebRTC socket
process still had ICE candidates bound on LAN IPv4 and public IPv6, and media kept
flowing inbound:
```
wall NIC over 6s: rx 459.6 kbit/s tx 16.6 kbit/s
```
while inbound probes from a same-subnet host went `tcp/22` reachable and
`tcp/8080`, `tcp/9999` silent. (To see the ICE sockets, grep for
`Socket Process` — Firefox's WebRTC sockets are not owned by a process named
`firefox`.)
Still open and not needed by a wall: `80/tcp`, `443/tcp`, `samba-client`. Left
alone because they are not this change.
---
## The wall went quiet and nobody noticed
`/healthz` knew at 14:03:04Z that nothing was watching. It is a JSON endpoint and
nothing was reading it, so the outage was discovered by looking at the panel. The
other half of running an unattended wall is being told about it —
[`../wall-watch/`](../wall-watch/) checks presence on the server host every minute
and pages over ntfy, then announces the recovery.
```bash
tar czf - -C docs wall-watch | ssh server \
'cd /var/tmp && tar xzf - && cd wall-watch && sudo sh install-wall-watch.sh'
```
Test it the honest way — a **drill**, labelled as one in the title, body and
priority so it can never be mistaken for an incident:
```bash
sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=http://127.0.0.1:1/healthz \
WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test sh /usr/local/libexec/mirror-wall-watch.sh'
```
Two things worth knowing: it probes loopback `/healthz`, which works only because
that path is exempt from the address gate (see `AGENTS.md` §3); and it cannot
detect the **server** being down, because it lives there. `scp` to some of these
hosts dies with `lost connection` (no sftp subsystem) — the tar pipe above works
everywhere.
---
## Why there is no RDP on the wall
You cannot see the panel, so of course the idea is to look at it remotely.
`gnome-remote-desktop` is already in the Silverblue base image, so it looks free.
It does not work here, for a structural reason:
**`grd` does not capture a screen.** It is a *client* of mutter's remote-desktop
D-Bus API, and gnome-kiosk embeds libmutter without exporting it. Ask the wall's
own session bus:
```bash
sudo -u kiosk XDG_RUNTIME_DIR=/run/user/1001 \
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus \
busctl call org.gnome.Mutter.RemoteDesktop /org/gnome/Mutter/RemoteDesktop \
org.freedesktop.DBus.Peer Ping
# Call failed: The name is not activatable (same for .ScreenCast and .DisplayConfig)
```
Not a missing package: gnome-shell 50.6 *is* installed and is what exports those
names; gnome-kiosk 50.1 does not. So `grdctl rdp enable` and
`grdctl rdp set-credentials` would give you a running daemon with nothing to
share. For whoever tries next:
- this image ships **`grdctl`, not `grdc`**, and has **no `org.gnome.RemoteDesktop`
gsettings schema** — configuration is grdctl/JSON, so the `DCONF_PROFILE` trap
above does not even apply to it.
- the only way to really see the panel over RDP is to move the wall onto a
gnome-shell-based session. That replaces the script session this whole runbook
is built on — the never-exit loop, `window-config.ini`, the greeter trap — and is
a `PLAN.md` decision with a full re-validation, not a config tweak.
- **and it would not have survived the hang above.** An RDP server is userspace;
the failure was userspace being unable to start. RDP is a way to *look*, never a
rescue path. The rescue path is the watchdog.
---
## Troubleshooting ## Troubleshooting
| Symptom | Cause and fix | | Symptom | Cause and fix |
@@ -341,6 +516,10 @@ convenient IP.
| Firefox shows a restore-session or first-run page | the policy file is missing or unreadable — `sudo cat /etc/firefox/policies/policies.json` | | Firefox shows a restore-session or first-run page | the policy file is missing or unreadable — `sudo cat /etc/firefox/policies/policies.json` |
| Tile says `unreachable` | the sharer and the kiosk could not find each other over ICE. Both must be on a network where they can send UDP directly; mirror uses no STUN/TURN by design | | Tile says `unreachable` | the sharer and the kiosk could not find each other over ICE. Both must be on a network where they can send UDP directly; mirror uses no STUN/TURN by design |
| Wall shows the "not allowed" page | the kiosk's address is outside `MIRROR_ALLOW_CIDRS` — this is admission control working, see `SECURITY.md` | | Wall shows the "not allowed" page | the kiosk's address is outside `MIRROR_ALLOW_CIDRS` — this is admission control working, see `SECURITY.md` |
| **`ssh` connects, then closes in milliseconds with no banner** | Not a firewall ban — a ban refuses the SYN. The kernel is answering from the listen socket while `sshd` cannot spawn a session: the box is **hung**, and nothing userspace on it, including this checker, will run. The hardware watchdog should reset it within a minute. |
| It pings and the port is open, so why is it broken? | ICMP replies and TCP handshakes are produced by the kernel. They say nothing about userspace. Trust `viewers` on `/healthz`, not `ping`. |
| Wall reboots on its own | Either the kernel panicked or PID 1 missed the watchdog — that is the design working. Find out which before it happens again: `journalctl -b -1 -p err`, `journalctl -b -1 -k \| tail`, and `ls /sys/fs/pstore` (this box has no ERST backend, so a panic trace may not survive). |
| `install-kiosk.sh` pointed the wall somewhere else | It no longer can: an existing `kiosk.conf` URL wins over the built-in default. Pass `KIOSK_URL=...` to change it deliberately. |
Debugging rule for this project applies: reproduce at the lowest layer that can Debugging rule for this project applies: reproduce at the lowest layer that can
fail. `curl` the URL from the client, check `/api/streams` on the server, and fail. `curl` the URL from the client, check `/api/streams` on the server, and
@@ -367,8 +546,11 @@ decision, not an oversight.
`rpm-ostreed-automatic.timer`. `rpm-ostreed-automatic.timer`.
* **SELinux is left enforcing, unmodified.** Nothing here needed a policy tweak; * **SELinux is left enforcing, unmodified.** Nothing here needed a policy tweak;
treat a new AVC as a bug in this runbook. treat a new AVC as a bug in this runbook.
* **No reboot watchdog.** If the wall is remote enough that a hang means driving * **No power cycle.** A hang is now recovered by the kernel and the chipset
to it, that is an ops decision (greenboot, or a smart PDU), not a client one. watchdogs above. A machine that is *off* — dead PSU, tripped plug, firmware that
never reaches the bootloader — cannot reboot itself, and if the wall is remote
enough for that to matter it is still an ops decision (greenboot, or a smart
PDU), not a client one.
## Naming on the reference network ## Naming on the reference network
docs/kiosk-silverblue/90-mirror-wall-selfheal.conf
+37
View File
@@ -0,0 +1,37 @@
# 90-mirror-wall-selfheal.conf — make a hung wall recover on its own.
#
# Installed by install-kiosk.sh to /etc/sysctl.d/. Applied at boot by
# systemd-sysctl, and immediately by `sysctl --system` in the installer.
#
# The trade this makes: a wall whose kernel has oopsed, deadlocked or soft-locked
# is a panel showing a frozen frame forever, in a room nobody is in. Rebooting
# into itself is always the better outcome, so every one of these escalates to a
# reboot. Reasoning and the incident behind it: ../kiosk-silverblue.md
# ("When the wall hangs").
#
# To revert a single line: drop a file with a higher-numbered name in the same
# directory, e.g. /etc/sysctl.d/91-mirror-wall-quiet.conf containing
# "kernel.panic_on_oops = 0", then `sudo sysctl --system`.
# The lockup detectors themselves (default on, pinned here so a future base
# image change cannot silently turn them off).
kernel.watchdog = 1
# A soft lockup (a CPU spinning in kernel mode with preemption off) or a hard
# lockup (interrupts disabled; caught by the NMI watchdog) panics instead of
# only printing "watchdog: BUG: soft lockup" into a journal nobody reads.
kernel.softlockup_panic = 1
kernel.hardlockup_panic = 1
# An oops used to leave the box nominally up with its display driver dead — the
# worst case, because PID 1 stays healthy and so nothing else would reset it.
kernel.panic_on_oops = 1
# Panic, then reboot 30s later. Long enough for the console to be read and for
# the filesystems to be sync'ed by the panic path; short enough that nobody
# waits for it. 0 would mean "sit there forever", which is the bug.
kernel.panic = 30
# Not set here on purpose: kernel.hung_task_panic does not exist in the Fedora
# 44 kernel build (CONFIG_DETECT_HUNG_TASK is off). A task blocked on I/O is
# caught by the watchdog in 90-mirror-wall-watchdog.conf instead.
docs/kiosk-silverblue/90-mirror-wall-watchdog.conf
+29
View File
@@ -0,0 +1,29 @@
# 90-mirror-wall-watchdog.conf — PID 1 must kick the hardware watchdog.
#
# Installed by install-kiosk.sh to /etc/systemd/system.conf.d/, but ONLY when the
# box actually has a watchdog device (/dev/watchdog). Takes effect on
# `systemctl daemon-reexec`, which the installer runs; a reboot also picks it up.
#
# This is the layer that catches what the sysctls cannot. The soft/hard lockup
# detectors need kernel work to keep happening; a hardware watchdog needs the
# opposite — if PID 1 stops writing /dev/watchdog0 for one whole timeout, the
# chipset resets the machine no matter what the kernel thinks it is doing. That
# includes the case we hit on 2026-11-02, where the box answered ARP and TCP
# handshakes from kernel state while no userspace process could be started.
#
# 30s is not an arbitrary number: it is the timeout iTCO_wdt initialised itself
# with ("heartbeat=30 sec"). systemd kicks at a third of the configured timeout,
# so 30s means a kick every 10s. Asking for more than the device supports would
# make systemd kick less often than the hardware expires — a self-inflicted
# reboot loop — so install-kiosk.sh keeps this at or below the driver's value.
#
# Cost: a PID-1 stall longer than 30s (a genuinely stuck disk, say) reboots the
# wall. On an unattended display that is the intended behaviour. To back off,
# raise the number to something under the device timeout, or set 0 and rely on
# the sysctls alone:
#
# printf '[Manager]\nRuntimeWatchdogSec=0\n' | sudo tee /etc/systemd/system.conf.d/91-mirror-wall-watchdog-off.conf
# sudo systemctl daemon-reexec
[Manager]
RuntimeWatchdogSec=30s
docs/kiosk-silverblue/check-kiosk.sh
+47
View File
@@ -157,6 +157,53 @@ if [ -n "$HOME_DIR" ]; then
fi fi
fi fi
printf '\n%s\n' '--- hang recovery ---'
# An unattended wall has to end a hang with a reboot, not with a frozen frame for
# a week. Two independent layers; ../kiosk-silverblue.md ("When the wall hangs").
sc() { sysctl -n "$1" 2>/dev/null || echo missing; }
for k in kernel.softlockup_panic kernel.hardlockup_panic kernel.panic_on_oops; do
v=$(sc "$k")
case "$v" in
1) ok "$k = 1" ;;
missing) warn "$k does not exist in this kernel build"
warn " (kernel.hung_task_panic is absent from Fedora's kernel; the watchdog covers it)" ;;
*) bad "$k = $v, expected 1 — fix /etc/sysctl.d/90-mirror-wall-selfheal.conf, then sudo sysctl --system" ;;
esac
done
P=$(sc kernel.panic)
case "$P" in
''|0|missing) bad "kernel.panic = ${P:-unset}: after a panic the box sits there forever" ;;
*) ok "kernel.panic = $P (reboots $P s after a panic)" ;;
esac
if [ -e /dev/watchdog ]; then
WD=$(systemctl show -p RuntimeWatchdogUSec --value 2>/dev/null || echo 0)
WID=$(cat /sys/class/watchdog/watchdog0/identity 2>/dev/null || echo '?')
case "$WD" in
0|''|infinity) bad "PID 1 is not kicking the $WID watchdog (RuntimeWatchdogUSec=$WD) — a wedged kernel here never recovers" ;;
*) ok "hardware watchdog armed: $WID at $WD (no kick from PID 1 for $WD = chipset reset)" ;;
esac
else
info "no /dev/watchdog on this box — the lockup panics are the only recovery layer"
fi
printf '\n%s\n' '--- firewall ---'
if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then
ZONE=$(firewall-cmd --get-default-zone)
PORTS=$(firewall-cmd --zone="$ZONE" --list-ports | tr ' ' '\n')
if printf '%s\n' "$PORTS" | grep -qx '1025-65535/tcp'; then
bad "$ZONE opens 1025-65535/tcp — on a box whose only access control is its address, that is every high port"
else
ok "$ZONE: 1025-65535/tcp closed"
fi
if printf '%s\n' "$PORTS" | grep -qx '1025-65535/udp'; then
ok "$ZONE: 1025-65535/udp open (WebRTC media needs it — do not close it)"
else
bad "$ZONE: 1025-65535/udp closed — peer-to-peer media cannot flow, every tile reads 'unreachable'"
fi
else
info "firewalld is not running — nothing listening, nothing to tighten"
fi
printf '\n%s\n' '--- what the screen will show ---' printf '\n%s\n' '--- what the screen will show ---'
if [ -n "$URL" ]; then if [ -n "$URL" ]; then
BODY=$(curl -fsSL -m 8 "$URL" 2>/dev/null) BODY=$(curl -fsSL -m 8 "$URL" 2>/dev/null)
docs/kiosk-silverblue/install-kiosk.sh
+84 -8
View File
@@ -15,7 +15,10 @@
set -eu set -eu
KIOSK_USER="${KIOSK_USER:-kiosk}" KIOSK_USER="${KIOSK_USER:-kiosk}"
KIOSK_URL="${KIOSK_URL:-https://mirror.reeseapps.com/watch}" # The default is for a NEW client only. An existing client keeps its own URL
# unless the operator names one — see "the URL is the one per-client value" below.
KIOSK_URL_DEFAULT="https://mirror.reeseapps.com/watch"
KIOSK_URL="${KIOSK_URL:-}"
KIOSK_SESSION="gnome-kiosk-script-wayland" KIOSK_SESSION="gnome-kiosk-script-wayland"
HERE=$(cd "$(dirname "$0")" && pwd) HERE=$(cd "$(dirname "$0")" && pwd)
NEED_REBOOT=0 NEED_REBOOT=0
@@ -58,7 +61,7 @@ for c in rpm getent restorecon useradd sed awk grep install; do need "$c"; done
[ -f /etc/os-release ] && grep -q '^ID=fedora' /etc/os-release || [ -f /etc/os-release ] && grep -q '^ID=fedora' /etc/os-release ||
say "warning: this does not look like Fedora; continuing anyway" say "warning: this does not look like Fedora; continuing anyway"
say "== 1/7 packages ==" say "== 1/9 packages =="
if rpm -q gnome-kiosk gnome-kiosk-script-session >/dev/null 2>&1; then if rpm -q gnome-kiosk gnome-kiosk-script-session >/dev/null 2>&1; then
say "gnome-kiosk + gnome-kiosk-script-session already present" say "gnome-kiosk + gnome-kiosk-script-session already present"
else else
@@ -69,7 +72,7 @@ else
say "layered the kiosk packages — a reboot is required before they exist" say "layered the kiosk packages — a reboot is required before they exist"
fi fi
say "== 2/7 kiosk user ==" say "== 2/9 kiosk user =="
if id -u "$KIOSK_USER" >/dev/null 2>&1; then if id -u "$KIOSK_USER" >/dev/null 2>&1; then
HOME_DIR=$(getent passwd "$KIOSK_USER" | cut -d: -f6) HOME_DIR=$(getent passwd "$KIOSK_USER" | cut -d: -f6)
else else
@@ -81,7 +84,7 @@ fi
KIOSK_GROUP=$(getent passwd "$KIOSK_USER" | cut -d: -f4) KIOSK_GROUP=$(getent passwd "$KIOSK_USER" | cut -d: -f4)
say "user $KIOSK_USER uid=$(id -u "$KIOSK_USER") home=$HOME_DIR" say "user $KIOSK_USER uid=$(id -u "$KIOSK_USER") home=$HOME_DIR"
say "== 3/7 payload ==" say "== 3/9 payload =="
# Firefox enterprise policy: no update prompts, no account/sync, no telemetry, # Firefox enterprise policy: no update prompts, no account/sync, no telemetry,
# no first-run pages. /etc is mutable on Silverblue and survives upgrades. # no first-run pages. /etc is mutable on Silverblue and survives upgrades.
install -D -m 0644 "$HERE/policies.json" /etc/firefox/policies/policies.json install -D -m 0644 "$HERE/policies.json" /etc/firefox/policies/policies.json
@@ -119,6 +122,19 @@ say " $HOME_DIR/.local/bin/gnome-kiosk-script"
# The URL is the one per-client value; keep it in its own file so everything # The URL is the one per-client value; keep it in its own file so everything
# else can stay byte-identical across the fleet. # else can stay byte-identical across the fleet.
#
# Re-running this installer is the documented way to REPAIR a client, and a repair
# must not move the wall somewhere else. So an existing kiosk.conf wins over the
# default; only an explicit KIOSK_URL= changes it.
KIOSK_CONF="$HOME_DIR/.config/mirror/kiosk.conf"
if [ -z "$KIOSK_URL" ] && [ -f "$KIOSK_CONF" ]; then
KIOSK_URL=$(sed -n 's/^[[:space:]]*MIRROR_KIOSK_URL="\{0,1\}\([^"]*\)"\{0,1\}[[:space:]]*$/\1/p' \
"$KIOSK_CONF" | head -1)
if [ -n "$KIOSK_URL" ]; then
say " keeping this client's existing URL (pass KIOSK_URL=... to change it)"
fi
fi
[ -n "$KIOSK_URL" ] || KIOSK_URL="$KIOSK_URL_DEFAULT"
sed "s|^MIRROR_KIOSK_URL=.*|MIRROR_KIOSK_URL=\"$KIOSK_URL\"|" "$HERE/kiosk.conf" \ sed "s|^MIRROR_KIOSK_URL=.*|MIRROR_KIOSK_URL=\"$KIOSK_URL\"|" "$HERE/kiosk.conf" \
| install -m 0644 -o "$KIOSK_USER" -g "$KIOSK_GROUP" /dev/stdin \ | install -m 0644 -o "$KIOSK_USER" -g "$KIOSK_GROUP" /dev/stdin \
"$HOME_DIR/.config/mirror/kiosk.conf" "$HOME_DIR/.config/mirror/kiosk.conf"
@@ -129,7 +145,7 @@ say " $HOME_DIR/.config/mirror/kiosk.conf -> $KIOSK_URL"
install -m 0644 -o "$KIOSK_USER" -g "$KIOSK_GROUP" /dev/null \ install -m 0644 -o "$KIOSK_USER" -g "$KIOSK_GROUP" /dev/null \
"$HOME_DIR/.config/gnome-initial-setup-done" "$HOME_DIR/.config/gnome-initial-setup-done"
say "== 4/7 default session ==" say "== 4/9 default session =="
ACCT="/var/lib/AccountsService/users/$KIOSK_USER" ACCT="/var/lib/AccountsService/users/$KIOSK_USER"
if [ -f "$ACCT" ]; then if [ -f "$ACCT" ]; then
set_ini "$ACCT" User Session "$KIOSK_SESSION" set_ini "$ACCT" User Session "$KIOSK_SESSION"
@@ -141,7 +157,7 @@ fi
systemctl restart accounts-daemon.service 2>/dev/null || : systemctl restart accounts-daemon.service 2>/dev/null || :
say " $ACCT Session=$KIOSK_SESSION" say " $ACCT Session=$KIOSK_SESSION"
say "== 5/7 auto-login ==" say "== 5/9 auto-login =="
if [ ! -d /etc/gdm ]; then if [ ! -d /etc/gdm ]; then
say " gdm is not installed — installing it" say " gdm is not installed — installing it"
rpm-ostree install gdm rpm-ostree install gdm
@@ -156,7 +172,7 @@ if [ -d /etc/gdm ]; then
say " /etc/gdm/custom.conf: AutomaticLogin=$KIOSK_USER" say " /etc/gdm/custom.conf: AutomaticLogin=$KIOSK_USER"
fi fi
say "== 6/7 console guard ==" say "== 6/9 console guard =="
# Cheap insurance against the class of bug above: if the kiosk session exists but # Cheap insurance against the class of bug above: if the kiosk session exists but
# something else holds the foreground VT, bring the wall back. Stops short of # something else holds the foreground VT, bring the wall back. Stops short of
# ever restarting gdm. Pause it while debugging the console with: # ever restarting gdm. Pause it while debugging the console with:
@@ -168,6 +184,64 @@ systemctl daemon-reload
systemctl enable --now mirror-kiosk-console-guard.timer systemctl enable --now mirror-kiosk-console-guard.timer
say " mirror-kiosk-console-guard.timer enabled" say " mirror-kiosk-console-guard.timer enabled"
say "== 7/9 hang recovery =="
# An unattended wall that hangs must come back by itself. Two independent layers;
# reasoning and the incident behind them in ../kiosk-silverblue.md.
install -D -m 0644 "$HERE/90-mirror-wall-selfheal.conf" /etc/sysctl.d/90-mirror-wall-selfheal.conf
if sysctl --system >/dev/null 2>&1; then
say " soft lockup / hard lockup / oops now panic; panic reboots in $(cat /proc/sys/kernel/panic)s"
else
say " warning: sysctl --system failed — check: sudo sysctl kernel.softlockup_panic kernel.panic"
fi
# The layer that does not need the kernel to be cooperating: if PID 1 stops
# kicking /dev/watchdog, the chipset resets the box. Only configured when the
# hardware has one (VMs typically do not).
if [ -e /dev/watchdog ]; then
wd_identity=$(cat /sys/class/watchdog/watchdog0/identity 2>/dev/null || echo unknown)
wd_timeout=$(cat /sys/class/watchdog/watchdog0/timeout 2>/dev/null || echo 0)
case "$wd_timeout" in ''|*[!0-9]*) wd_timeout=0 ;; esac
install -D -m 0644 "$HERE/90-mirror-wall-watchdog.conf" /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf
# systemd kicks at a third of the value it is given. Asking for more than the
# device supports would kick less often than the device expires — a reboot
# loop of our own making — so clamp to what the driver came up with.
if [ "$wd_timeout" -gt 0 ] && [ "$wd_timeout" -ne 30 ]; then
sed -i "s|^RuntimeWatchdogSec=.*|RuntimeWatchdogSec=${wd_timeout}s|" \
/etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf
say " clamped to the device's own ${wd_timeout}s timeout ($wd_identity)"
fi
# PID 1 re-reads system.conf on re-exec; nothing else picks it up short of a
# reboot, and re-exec is safe (sockets and running services survive it).
systemctl daemon-reexec
say " $wd_identity hardware watchdog armed (PID 1 must kick it or the box resets)"
else
say " no /dev/watchdog here — hardware layer skipped (lockup panics still installed)"
fi
say "== 8/9 firewall =="
# Fedora's Workstation zone opens 1025-65535/tcp and /udp. The UDP half is what
# WebRTC media needs (peer-to-peer, both directions, and mirror uses no STUN/TURN);
# the TCP half is pure exposure on a box whose only access control is its address.
# Dropping it is safe for a wall, which never serves: verified against a live
# share (media keeps flowing at ~460 kbit/s inbound with it gone).
if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then
zone=$(firewall-cmd --get-default-zone)
if firewall-cmd --zone="$zone" --list-ports | tr ' ' '\n' | grep -qx '1025-65535/tcp'; then
firewall-cmd --permanent --zone="$zone" --remove-port=1025-65535/tcp >/dev/null
firewall-cmd --reload >/dev/null
say " $zone: dropped 1025-65535/tcp (kept 1025-65535/udp for WebRTC media)"
else
say " $zone: 1025-65535/tcp already closed"
fi
if firewall-cmd --zone="$zone" --list-ports | tr ' ' '\n' | grep -qx '1025-65535/udp'; then
say " 1025-65535/udp still open — required, do not close it or mirrors go black"
else
say " warning: 1025-65535/udp is NOT open — peer-to-peer media will fail (tile: unreachable)"
fi
else
say " firewalld not running — nothing to tighten (verify your own firewall keeps UDP media)"
fi
# Keep a copy of the payload on the box. /tmp is cleared at every boot, so a # Keep a copy of the payload on the box. /tmp is cleared at every boot, so a
# checker staged there is gone precisely when you want to run it again — after a # checker staged there is gone precisely when you want to run it again — after a
# reboot, or from a rescue session hours later. # reboot, or from a rescue session hours later.
@@ -177,10 +251,12 @@ if [ "$HERE" != "/usr/local/share/mirror-kiosk" ]; then
say " /usr/local/share/mirror-kiosk (survives reboot; run check-kiosk.sh from here)" say " /usr/local/share/mirror-kiosk (survives reboot; run check-kiosk.sh from here)"
fi fi
say "== 7/7 selinux labels ==" say "== 9/9 selinux labels =="
restorecon -RF "$HOME_DIR/.local" "$HOME_DIR/.config" 2>/dev/null || : restorecon -RF "$HOME_DIR/.local" "$HOME_DIR/.config" 2>/dev/null || :
restorecon -RF /etc/firefox /usr/local/share/gnome-kiosk /usr/local/share/wayland-sessions \ restorecon -RF /etc/firefox /usr/local/share/gnome-kiosk /usr/local/share/wayland-sessions \
/usr/local/libexec 2>/dev/null || : /usr/local/libexec 2>/dev/null || :
restorecon -RF /etc/sysctl.d/90-mirror-wall-selfheal.conf \
/etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf 2>/dev/null || :
say "" say ""
say "Installed." say "Installed."
docs/wall-watch/README.md
+57
View File
@@ -0,0 +1,57 @@
# wall-watch — presence alert for the mirror wall
Runs on the **mirror server host**, not on the kiosk. Once a minute it reads the
service's own `/healthz` and pages you over ntfy when nobody is watching, or when
the service stops answering — and tells you when the wall comes back.
Rationale and the incident that produced it:
[`../kiosk-silverblue.md`](../kiosk-silverblue.md) → *"The wall went quiet and
nobody noticed"*.
## Install (on the server host)
```bash
scp -r docs/wall-watch server:/var/tmp/ && ssh server \
'cd /var/tmp/wall-watch && sudo sh install-wall-watch.sh'
sudo vi /etc/mirror-wall-watch.env # set NTFY_TOKEN
```
## Operate
```bash
systemctl status mirror-wall-watch.timer # is it still ticking?
journalctl -u mirror-wall-watch.service -f # one line per check, forever
sudo sh /usr/local/libexec/mirror-wall-watch.sh # run it by hand
# Page yourself on purpose, without touching the wall's real state. WALL_DRILL=1
# makes the notification say "[drill]" and drop to priority 3, so a test can never
# be mistaken for an outage:
sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=https://127.0.0.1/nope \
WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test \
sh /usr/local/libexec/mirror-wall-watch.sh'
```
## Files
| File | Lands at |
|---|---|
| `wall-watch.sh` | `/usr/local/libexec/mirror-wall-watch.sh` |
| `mirror-wall-watch.service` / `.timer` | `/etc/systemd/system/` |
| `wall-watch.env.example` | `/etc/mirror-wall-watch.env` (0600, seeded once, never overwritten) |
| everything here | `/usr/local/share/mirror-wall-watch/` |
## Deliberate choices
- **Probes `/healthz` over loopback**, because that endpoint is exempt from the
address gate — the same exemption that lets the container healthcheck work from
loopback when `MIRROR_ALLOW_CIDRS` excludes it.
- **First run cannot alert.** Installing a watcher must not page you about a wall
that was already down before you had one.
- **A failed notification fails the unit** and does *not* bank the re-notify
timestamp, so an ntfy outage is retried on the next tick instead of eating the
alert.
- **A drill is labelled as one.** `WALL_DRILL=1` prefixes the title with `[drill]`,
lowers the priority and says "nothing is wrong" in the body. A test page that is
indistinguishable from an incident teaches people to ignore the page.
- **It cannot detect the server being down.** It lives there. That gap belongs to
whatever monitors this host.
docs/wall-watch/install-wall-watch.sh
+76
View File
@@ -0,0 +1,76 @@
#!/bin/sh
# install-wall-watch.sh — install the wall-presence alert on the MIRROR SERVER host.
#
# Run ON THE SERVER, as root, from this directory:
#
# sudo sh install-wall-watch.sh
#
# Idempotent. It never overwrites /etc/mirror-wall-watch.env once it exists, so
# the ntfy token you pasted in survives a re-run.
#
# What it does, and why this exists at all: ../kiosk-silverblue.md
# ("The wall went quiet and nobody noticed").
set -eu
HERE=$(cd "$(dirname "$0")" && pwd)
CONF=/etc/mirror-wall-watch.env
say() { printf '%s\n' "$*"; }
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
[ "$(id -u)" -eq 0 ] || die "run me as root (sudo sh $0)"
for c in curl logger systemctl; do
command -v "$c" >/dev/null 2>&1 || die "missing required command: $c"
done
say "== 1/4 program =="
install -D -m 0755 "$HERE/wall-watch.sh" /usr/local/libexec/mirror-wall-watch.sh
install -d -m 0755 /usr/local/share/mirror-wall-watch
cp -a "$HERE"/. /usr/local/share/mirror-wall-watch/
rm -f /usr/local/share/mirror-wall-watch/install-wall-watch.sh
say " /usr/local/libexec/mirror-wall-watch.sh"
say " /usr/local/share/mirror-wall-watch (docs, survives reboot)"
say "== 2/4 config =="
if [ ! -f "$CONF" ]; then
install -D -m 0600 "$HERE/wall-watch.env.example" "$CONF"
say " created $CONF (0600) — NOW SET NTFY_TOKEN IN IT:"
say " sudo vi $CONF"
else
perm=$(stat -c '%a' "$CONF")
[ "$perm" = "600" ] || say " warning: $CONF is mode $perm, expected 0600 (it holds a token)"
say " kept existing $CONF"
fi
say "== 3/4 units =="
install -D -m 0644 "$HERE/mirror-wall-watch.service" /etc/systemd/system/mirror-wall-watch.service
install -D -m 0644 "$HERE/mirror-wall-watch.timer" /etc/systemd/system/mirror-wall-watch.timer
systemctl daemon-reload
systemctl enable --now mirror-wall-watch.timer
say " mirror-wall-watch.timer enabled (every minute)"
say "== 4/4 prove it =="
# One run now, so a broken config fails in front of you instead of silently in a
# timer at 03:00. The first run only records the baseline; it cannot alert, which
# is deliberate — installing the watcher must not page you about a wall that was
# already down before you had a watcher.
if systemctl start --wait mirror-wall-watch.service 2>/dev/null; then
say " first check passed:"
journalctl -u mirror-wall-watch.service --no-pager -n 1 2>/dev/null | sed 's/^/ /' || :
else
say " first check FAILED — read this before trusting the timer:"
journalctl -u mirror-wall-watch.service --no-pager -n 12 2>/dev/null | sed 's/^/ /' || :
fi
# An AVC here means SELinux disagreed with something above; say so rather than
# leaving a silently broken alert.
if command -v ausearch >/dev/null 2>&1; then
if ausearch -m avc -ts recent >/dev/null 2>&1; then
say " warning: recent AVCs found — run: sudo ausearch -m avc -ts recent"
fi
fi
say ""
say "Installed. Fire a test notification (page yourself on purpose):"
say " sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=https://127.0.0.1/nope WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test sh /usr/local/libexec/mirror-wall-watch.sh'"
say "Then watch the timeline:"
say " journalctl -u mirror-wall-watch.service -f"
docs/wall-watch/mirror-wall-watch.service
+16
View File
@@ -0,0 +1,16 @@
[Unit]
Description=Check that the mirror wall is still watching
Documentation=file:///usr/local/share/mirror-wall-watch/README.md
# The probe is only meaningful once nginx (or whatever fronts the service) is up.
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/bin/sh /usr/local/libexec/mirror-wall-watch.sh
EnvironmentFile=-/etc/mirror-wall-watch.env
# It only ever writes /var/lib/mirror-wall-watch; keep the rest read-only.
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=full
ProtectHome=read-only
docs/wall-watch/mirror-wall-watch.timer
+17
View File
@@ -0,0 +1,17 @@
[Unit]
Description=Run the mirror wall presence check every minute
Documentation=file:///usr/local/share/mirror-wall-watch/README.md
[Timer]
# Check from a minute after boot — the service needs a moment to come up, and a
# wall has its own relaunch loop — and then every minute. The jitter keeps a
# fleet of these from all waking up on the same second.
OnBootSec=1min
OnUnitActiveSec=1min
RandomizedDelaySec=5s
AccuracySec=1s
# If the host was suspended or the timer missed its window, run once anyway.
Persistent=true
[Install]
WantedBy=timers.target
docs/wall-watch/wall-watch.env.example
+28
View File
@@ -0,0 +1,28 @@
# /etc/mirror-wall-watch.env — config for mirror-wall-watch.timer.
# chmod 0600, root:root. It holds the ntfy token. Never commit a filled-in copy.
# install-wall-watch.sh seeds this from the example and then NEVER overwrites it.
# The presence probe. /healthz is exempt from the address gate, so loopback works
# even when MIRROR_ALLOW_CIDRS excludes 127.0.0.0/8 — which is why the container
# healthcheck works too. Use whatever loopback URL reaches this deployment.
WALL_HEALTH_URL=https://127.0.0.1/healthz
# Alert when fewer than this many viewers are attached. Set 0 to stop alerting on
# "nobody is watching" and only alert when the service itself stops answering
# (useful while a fleet has no wall plugged in).
WALL_EXPECT_VIEWERS=1
# Consecutive bad checks before the first notification (~1 min per check). Three
# rides out a Firefox relaunch on the wall; raise it if that is normal for you.
WALL_THRESHOLD=3
# Keep shouting while it stays down. 0 = alert once per incident.
WALL_RENOTIFY_SECS=3600
# Optional: the URL a notification opens when tapped.
WALL_STATUS_URL=https://mirror.reeseapps.com/healthz
# ntfy (outbound only — this never reads anything back).
NTFY_URL=https://ntfy.reeseapps.com
NTFY_TOKEN=
NTFY_TOPIC=pi
docs/wall-watch/wall-watch.sh
+213
View File
@@ -0,0 +1,213 @@
#!/bin/sh
# wall-watch.sh — know within minutes that the mirror wall stopped watching.
#
# Run ON THE MIRROR SERVER HOST from mirror-wall-watch.timer (once a minute).
# It reads the service's own /healthz over loopback and posts to ntfy when no
# viewer is attached, or when the service stops answering at all — and again when
# the wall comes back.
#
# Why it exists: on 2026-11-02 the reference wall hard-hung at 14:03Z. The
# service knew the instant it happened — /healthz reported viewers:0 — and said
# nothing, because nothing was listening. The failure was eventually found by
# walking up to the panel, which was in another room of another building.
#
# /healthz is the right probe precisely because it is exempt from the address
# gate (AGENTS.md §3): a loopback check works even when MIRROR_ALLOW_CIDRS
# excludes loopback, which is also why the container healthcheck can work.
#
# Config: /etc/mirror-wall-watch.env (root-owned 0600 — it holds the ntfy token)
# WALL_HEALTH_URL default https://127.0.0.1/healthz
# WALL_EXPECT_VIEWERS default 1; set 0 to stop alerting on "nobody watching"
# WALL_THRESHOLD default 3 consecutive bad checks (~3 min) before shouting
# WALL_RENOTIFY_SECS default 3600; keeps shouting while it stays down
# WALL_STATUS_URL optional; the URL a notification opens
# NTFY_URL NTFY_TOKEN NTFY_TOPIC
#
# Manual run (the timer runs it the same way):
# sudo sh /usr/local/libexec/mirror-wall-watch.sh
# sudo sh -c 'WALL_HEALTH_URL=https://127.0.0.1/nope WALL_THRESHOLD=1 sh /usr/local/libexec/mirror-wall-watch.sh' # fire a test alert
#
# Honest limit: this runs on the server, so it cannot tell you the *server* is
# down. That is a different alert, and it belongs to whatever watches this host.
set -eu
CONF="${WALL_WATCH_CONF:-/etc/mirror-wall-watch.env}"
STATE_DIR="${WALL_STATE_DIR:-/var/lib/mirror-wall-watch}"
STATE_FILE="$STATE_DIR/state"
log() {
printf '%s\n' "$*"
# Under systemd stdout already IS the journal, so writing to /dev/log as well
# puts two copies of every check in it. Keep the syslog trail for interactive
# runs, where stdout is a terminal and nothing else would record it.
[ -t 1 ] && logger -t mirror-wall-watch -- "$*" 2>/dev/null
return 0
}
die() { log "error: $*"; exit 1; }
command -v curl >/dev/null 2>&1 || die "curl is required"
# Root is not inherently required, only write access to the state directory — but
# the default lives next to root-only config, so a non-root run must say where it
# is going to fail rather than fail at the interesting moment.
if ! install -d -m 0700 "$STATE_DIR" 2>/dev/null; then
die "cannot create $STATE_DIR — run me as root, or set WALL_STATE_DIR somewhere writable"
fi
# Apply KEY=VALUE from the config file, but only for variables the caller did not
# already set. An explicit environment must win over the file — the same precedence
# systemd gives Environment= over EnvironmentFile= — otherwise "run the checker
# against a different URL for a moment" silently does nothing, which is exactly the
# kind of thing you only discover when the alert you were testing never arrives.
load_conf() {
[ -r "$1" ] || return 0
while IFS= read -r line || [ -n "$line" ]; do
case "$line" in ''|'#'*) continue ;; esac
key=${line%%=*}
val=${line#*=}
key=$(printf '%s' "$key" | tr -d '[:space:]')
val=$(printf '%s' "$val" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')
case "$val" in '\"'*'"') val=${val#\"}; val=${val%\"} ;; esac
# Only sane shell identifiers get assigned.
case "$key" in ''|*[!A-Za-z0-9_]*|[0-9]*) continue ;; esac
eval "cur=\${$key:-}"
[ -n "$cur" ] || eval "$key=\$val"
done <"$1"
}
load_conf "$CONF"
: "${WALL_HEALTH_URL:=https://127.0.0.1/healthz}"
: "${WALL_EXPECT_VIEWERS:=1}"
: "${WALL_THRESHOLD:=3}"
: "${WALL_RENOTIFY_SECS:=3600}"
# A drill is an alert fired on purpose to prove the path works. It must never be
# mistaken for an incident, so it says so in the title, the body and the priority
# — an alerting tool whose test page looks like a real page trains people to
# ignore it.
: "${WALL_DRILL:=0}"
if [ "$WALL_DRILL" = "1" ]; then
DRILL_TAG="[drill] "
DRILL_NOTE="> **This is a drill.** Someone ran the test command on \`$(hostname)\`; nothing is wrong with the wall. What it checked is the path: probe -> ntfy -> your phone."
DOWN_PRIORITY=3
else
DRILL_TAG=""
DRILL_NOTE=""
DOWN_PRIORITY=4
fi
# Posts one notification. Returns non-zero instead of exiting: a failed alert
# must not stop the state file being written, or the next run loses the counter
# and the wall pages once a minute forever.
notify() {
prio=$1 tags=$2 title=$3 body=$4
if [ -z "${NTFY_URL:-}" ] || [ -z "${NTFY_TOPIC:-}" ]; then
log "error: ntfy is not configured (NTFY_URL/NTFY_TOPIC in $CONF)"
return 1
fi
# Collect the headers in the positional parameters, and only the ones that
# have values, so ntfy never receives an empty Authorization or X-Click.
set -- -H "X-Priority: $prio" -H "X-Tags: $tags" -H "X-Title: $title" -H "X-Markdown: yes"
[ -n "${NTFY_TOKEN:-}" ] && set -- "$@" -H "Authorization: Bearer $NTFY_TOKEN"
[ -n "${WALL_STATUS_URL:-}" ] && set -- "$@" -H "X-Click: $WALL_STATUS_URL"
if curl -fsS -m 10 "$@" -d "$body" "$NTFY_URL/$NTFY_TOPIC" >/dev/null 2>&1; then
log "notified (priority $prio): $title"
else
log "error: could not reach $NTFY_URL — the alert did not go out"
return 1
fi
}
# ---- read the service's own truth -------------------------------------------------
probe=""
viewers=""
if probe=$(curl -fsS -m 8 -k "$WALL_HEALTH_URL" 2>/dev/null); then
if command -v jq >/dev/null 2>&1; then
viewers=$(printf '%s' "$probe" | jq -r '.viewers // empty' 2>/dev/null || :)
else
viewers=$(printf '%s' "$probe" |
sed -n 's/.*"viewers"[[:space:]]*:[[:space:]]*\([0-9][0-9]*\).*/\1/p' | head -1)
fi
# 200 but no parseable viewer count is a broken answer, not a healthy one.
[ -n "$viewers" ] || probe=""
fi
if [ -z "$probe" ]; then
condition="unreachable"
summary="the mirror service did not answer \`$WALL_HEALTH_URL\`"
elif [ "$WALL_EXPECT_VIEWERS" -gt 0 ] && [ "$viewers" -lt "$WALL_EXPECT_VIEWERS" ]; then
condition="quiet"
summary="the service is up but **nothing is watching** (viewers=$viewers)"
else
condition="ok"
summary="viewers=$viewers"
fi
# ---- state machine ----------------------------------------------------------------
state="ok"
bad=0
last_alert=0
if [ -r "$STATE_FILE" ]; then
# shellcheck disable=SC1090
. "$STATE_FILE" 2>/dev/null || :
fi
now=$(date +%s)
NOTIFY_FAILED=0
if [ "$condition" = "ok" ]; then
if [ "$state" != "ok" ]; then
notify 3 "heavy_check_mark,recovered" "${DRILL_TAG}Wall back" \
"**${DRILL_TAG}The mirror wall is watching again.**
was: \`$state\`
now: $summary
host: \`$(hostname)\` · $(date -u '+%Y-%m-%d %H:%M:%SZ')
$DRILL_NOTE" || NOTIFY_FAILED=1
fi
state="ok"; bad=0; last_alert=0
else
if [ "$state" = "$condition" ]; then bad=$((bad + 1)); else bad=1; fi
if [ "$bad" -ge "$WALL_THRESHOLD" ] &&
{ [ "$state" != "$condition" ] || [ $((now - last_alert)) -ge "$WALL_RENOTIFY_SECS" ]; }; then
hint="Recover it: \`ssh <wall>\` then \`sudo sh /usr/local/share/mirror-kiosk/check-kiosk.sh\`; if ssh itself will not open a session the box is hung, not offline — the watchdog should reset it within a minute of this, and if it does not, power-cycle it."
[ "$condition" = "unreachable" ] &&
hint="Check the service first (\`systemctl status\`, container logs) before the wall: this host may be the thing that is broken."
notify "$DOWN_PRIORITY" "rotating_light,wall-down" "${DRILL_TAG}Wall down" \
"**${DRILL_TAG}Mirror wall: $condition**
$summary
\`\`\`
$probe
\`\`\`
$hint
host: \`$(hostname)\` · $(date -u '+%Y-%m-%d %H:%M:%SZ') · bad check #$bad
$DRILL_NOTE" || NOTIFY_FAILED=1
# Only bank the re-notify timestamp if it actually went out, so a ntfy
# outage is retried on the next tick rather than swallowed.
[ "$NOTIFY_FAILED" -eq 0 ] && last_alert=$now
fi
state="$condition"
fi
umask 077
cat >"$STATE_FILE" <<EOF
state=$state
bad=$bad
last_alert=$last_alert
EOF
# One greppable line per run, so `journalctl -u mirror-wall-watch` reads as a
# timeline of the wall's health rather than as silence.
log "$condition: $summary (state=$state bad=$bad threshold=$WALL_THRESHOLD)"
# A needed-but-undeliverable alert fails the unit, so `systemctl show`
# mirror-wall-watch.service shows why nobody got paged.
exit "$NOTIFY_FAILED"