diff --git a/README.md b/README.md index a0acb7c..4311c9c 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ Environment variables (all optional; a `.env` file is loaded if present): | `MIRROR_MAX_MIRRORS` | `0` | Cap on concurrent mirrors (`0` = unlimited). | | `MIRROR_MAX_BITRATE` | `25000000` | Per-leg WebRTC bitrate cap in bits/s — the mesh uploads N × bitrate per sharer, so this is the ceiling. 25 Mbit/s suits a wired LAN at 1440p; a Wi-Fi sharer with many viewers may need less, not more. | | `MIRROR_PING_INTERVAL` | `20` | Seconds between socket liveness pings. | -| `MIRROR_ALLOW_CIDRS` | *(unset)* | Comma-separated source networks allowed to connect, e.g. `10.2.0.0/22,fd00::/64`. Unset means unrestricted. | +| `MIRROR_ALLOW_CIDRS` | *(unset)* | Comma-separated source networks allowed to connect, e.g. `10.2.0.0/22,fd00::/64`. Unset means unrestricted. A wall needs this range too, and its address is DHCP — give it a reservation. | | `MIRROR_TRUSTED_PROXIES` | `127.0.0.0/8, ::1/128` | Which peers may speak for a client via `X-Forwarded-For`. Only a proxy that overwrites the header belongs here. | | `DEBUGPY` | `0` | `1` opens a debugpy listener on `DEBUGPY_PORT`. | | `DEBUGPY_PORT` | `5678` | debugpy port. | @@ -147,6 +147,13 @@ peer state for the media, and a probe page for the rendered resolution. The runbook records why each file lives where it does on an otherwise read-only OS. +**When it hangs.** A wall is unattended, so a hang has to end without a person: +the kernel turns lockups and oopses into a reboot, and PID 1 must keep kicking the +hardware watchdog or the chipset resets the box. Being *quiet* is a different +failure than being hung, so the server watches `/healthz` for that and pages over +ntfy ([`docs/wall-watch/`](docs/wall-watch/)). Both are installed by the commands +above and asserted by `check-kiosk.sh`. + **Keys.** The whole wall works from a keyboard — and the box needs no keyboard attached to be operated *or* debugged; the keys are there when one is plugged in. On `/watch`: diff --git a/docs/kiosk-silverblue.md b/docs/kiosk-silverblue.md index f617ca6..6fefab7 100644 --- a/docs/kiosk-silverblue.md +++ b/docs/kiosk-silverblue.md @@ -20,6 +20,11 @@ or to change its URL. | `mirror-kiosk-console-guard` | `/usr/local/libexec/` | reclaims the console if the wall loses it | | `mirror-kiosk-console-guard.{service,timer}` | `/etc/systemd/system/` | runs that check every minute | | `dim-probe.py` | run on any reachable host | measures what the kiosk actually renders | +| `90-mirror-wall-selfheal.conf` | `/etc/sysctl.d/` | lockups and oopses become a reboot, not a frozen frame | +| `90-mirror-wall-watchdog.conf` | `/etc/systemd/system.conf.d/` | PID 1 must kick the hardware watchdog or the chipset resets the box | + +Presence alerting lives beside this, not inside it: [`../wall-watch/`](../wall-watch/) +runs on the **server** host and pages you when a wall goes quiet. --- @@ -325,6 +330,176 @@ convenient IP. --- +## When the wall hangs: making it come back by itself + +On 2026-11-02 the reference wall hung solid at 14:03Z. Nothing said so, nothing +recovered it, and it stayed that way until someone walked over and powered it off. +Everything in this section exists so the next one is a 90-second blink instead of +a drive. + +**What the hang actually looked like** — worth reading, because every easy +diagnosis was wrong: + +| Signal | Reading | +|---|---| +| `ping` | answered, 0% loss, RTT jittering 1.9–8.2 ms on a LAN | +| `ssh` | **`connect()` succeeded**, then the socket closed 6–13 ms later with **zero bytes**. No SSH banner ever, and `sshd` never logged an attempt. | +| the service | the wall's `/ws` dropped at 14:03:04Z after 1212 s and never reconnected; `/healthz` → `viewers:0` (it is normally `viewers:1, watching:1`) | +| journal | every unit goes silent at 09:01:59 and stays silent for the 13 minutes until the power cut | +| memory | 62 GiB installed, 2.5 GiB in use, **zero OOM lines** in the previous boot | +| `pstore` | empty; the next boot reported `system.journal corrupted or uncleanly shut down` — the hard reset, and the reason the journal's tail is lost | + +The trap: **ICMP replies and TCP handshakes are kernel work.** "It pings" and +"port 22 is open" say nothing about userspace. Fedora starts `sshd` per +connection, so a master that cannot fork answers the SYN from the listen socket +and then hangs up without a byte — which is exactly what happened. Put together: +the kernel was alive and no userspace process could be started. Not memory, not a +full disk, not a fail2ban ban (a ban refuses the SYN, it does not complete the +handshake) — a hard hang that left no trace. + +**Two layers, because neither one covers the other:** + +| Layer | File | Catches | +|---|---|---| +| kernel escalates to a reboot | `/etc/sysctl.d/90-mirror-wall-selfheal.conf` | soft lockup, hard lockup (NMI watchdog), any oops → panic; `kernel.panic = 30` reboots 30 s later | +| the chipset reboots the kernel | `/etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf` | PID 1 stops kicking `/dev/watchdog` — i.e. the case above, where the kernel is too wedged to notice it is wedged | + +Verify all of it (or just run `check-kiosk.sh`, which asserts every line): + +```bash +ssh mirror 'systemctl show -p RuntimeWatchdogUSec --value' # 30s +ssh mirror 'cat /sys/class/watchdog/watchdog0/status' # 0x8100 on the reference box +ssh mirror 'sysctl kernel.softlockup_panic kernel.hardlockup_panic kernel.panic_on_oops kernel.panic' +# 1 1 1 30 +``` + +`status` is a bitmask of driver features *and* current state, so do not compare it +for equality: the reference box reports `0x8100` = `WDIOF_KEEPALIVEPING` (PID 1 is +currently kicking it) plus `WDIOF_MAGICCLOSE` (a clean shutdown stops it, so the +watchdog cannot reset a box that is deliberately powered off). The bit that matters +is `0x8000`. `check-kiosk.sh` asserts `RuntimeWatchdogUSec` instead, which is not +ambiguous. + +**Why 30 seconds.** It is the timeout `iTCO_wdt` initialises itself with +(`heartbeat=30 sec`), and systemd kicks at a third of whatever it is handed. Asking +a device for more than it supports would make systemd kick *less* often than the +hardware expires — a reboot loop of our own making — so `install-kiosk.sh` clamps +to the driver's own value. Boxes with no watchdog at all (most VMs) get the sysctls +and a line saying so. + +**The cost, stated plainly.** Any stall of PID 1 longer than 30 s now reboots the +wall — a genuinely stuck disk, say. On an unattended display that is the intended +trade, but it is a trade; to back off raise the value or set it to 0 and rely on +the sysctls alone (the drop-in documents both). + +**What it still cannot do.** `kernel.hung_task_panic` does not exist in Fedora's +kernel build (`CONFIG_DETECT_HUNG_TASK` is off), so a pure blocked-on-I/O hang is +caught by the watchdog rather than by a targeted panic — and a machine that is +*off* (dead PSU, tripped plug, firmware that never reaches the bootloader) cannot +watchdog anything. That remains an ops decision. + +--- + +## The wall's firewall + +Fedora's `FedoraWorkstation` zone — the default on Silverblue — ships with: + +``` +ports: 1025-65535/udp 1025-65535/tcp 443/tcp 80/tcp +``` + +The **UDP** half is load-bearing: media is peer-to-peer and mirror deliberately +uses no STUN/TURN, so a wall that cannot receive UDP on high ports shows +`unreachable` tiles. The **TCP** half is pure exposure on a box whose only access +control is its address (A19) — it made every high port on the wall connectable +from anywhere the wall was reachable, and a wall never serves anything. + +`install-kiosk.sh` step 8/9 removes `1025-65535/tcp`, leaves the UDP alone, and +says which of the two it found; `check-kiosk.sh` FAILs if either half moves. + +This was measured, not assumed. With the TCP range gone, the wall's WebRTC socket +process still had ICE candidates bound on LAN IPv4 and public IPv6, and media kept +flowing inbound: + +``` +wall NIC over 6s: rx 459.6 kbit/s tx 16.6 kbit/s +``` + +while inbound probes from a same-subnet host went `tcp/22` reachable and +`tcp/8080`, `tcp/9999` silent. (To see the ICE sockets, grep for +`Socket Process` — Firefox's WebRTC sockets are not owned by a process named +`firefox`.) + +Still open and not needed by a wall: `80/tcp`, `443/tcp`, `samba-client`. Left +alone because they are not this change. + +--- + +## The wall went quiet and nobody noticed + +`/healthz` knew at 14:03:04Z that nothing was watching. It is a JSON endpoint and +nothing was reading it, so the outage was discovered by looking at the panel. The +other half of running an unattended wall is being told about it — +[`../wall-watch/`](../wall-watch/) checks presence on the server host every minute +and pages over ntfy, then announces the recovery. + +```bash +tar czf - -C docs wall-watch | ssh server \ + 'cd /var/tmp && tar xzf - && cd wall-watch && sudo sh install-wall-watch.sh' +``` + +Test it the honest way — a **drill**, labelled as one in the title, body and +priority so it can never be mistaken for an incident: + +```bash +sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=http://127.0.0.1:1/healthz \ + WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test sh /usr/local/libexec/mirror-wall-watch.sh' +``` + +Two things worth knowing: it probes loopback `/healthz`, which works only because +that path is exempt from the address gate (see `AGENTS.md` §3); and it cannot +detect the **server** being down, because it lives there. `scp` to some of these +hosts dies with `lost connection` (no sftp subsystem) — the tar pipe above works +everywhere. + +--- + +## Why there is no RDP on the wall + +You cannot see the panel, so of course the idea is to look at it remotely. +`gnome-remote-desktop` is already in the Silverblue base image, so it looks free. +It does not work here, for a structural reason: + +**`grd` does not capture a screen.** It is a *client* of mutter's remote-desktop +D-Bus API, and gnome-kiosk embeds libmutter without exporting it. Ask the wall's +own session bus: + +```bash +sudo -u kiosk XDG_RUNTIME_DIR=/run/user/1001 \ + DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus \ + busctl call org.gnome.Mutter.RemoteDesktop /org/gnome/Mutter/RemoteDesktop \ + org.freedesktop.DBus.Peer Ping +# Call failed: The name is not activatable (same for .ScreenCast and .DisplayConfig) +``` + +Not a missing package: gnome-shell 50.6 *is* installed and is what exports those +names; gnome-kiosk 50.1 does not. So `grdctl rdp enable` and +`grdctl rdp set-credentials` would give you a running daemon with nothing to +share. For whoever tries next: + +- this image ships **`grdctl`, not `grdc`**, and has **no `org.gnome.RemoteDesktop` + gsettings schema** — configuration is grdctl/JSON, so the `DCONF_PROFILE` trap + above does not even apply to it. +- the only way to really see the panel over RDP is to move the wall onto a + gnome-shell-based session. That replaces the script session this whole runbook + is built on — the never-exit loop, `window-config.ini`, the greeter trap — and is + a `PLAN.md` decision with a full re-validation, not a config tweak. +- **and it would not have survived the hang above.** An RDP server is userspace; + the failure was userspace being unable to start. RDP is a way to *look*, never a + rescue path. The rescue path is the watchdog. + +--- + ## Troubleshooting | Symptom | Cause and fix | @@ -341,6 +516,10 @@ convenient IP. | Firefox shows a restore-session or first-run page | the policy file is missing or unreadable — `sudo cat /etc/firefox/policies/policies.json` | | Tile says `unreachable` | the sharer and the kiosk could not find each other over ICE. Both must be on a network where they can send UDP directly; mirror uses no STUN/TURN by design | | Wall shows the "not allowed" page | the kiosk's address is outside `MIRROR_ALLOW_CIDRS` — this is admission control working, see `SECURITY.md` | +| **`ssh` connects, then closes in milliseconds with no banner** | Not a firewall ban — a ban refuses the SYN. The kernel is answering from the listen socket while `sshd` cannot spawn a session: the box is **hung**, and nothing userspace on it, including this checker, will run. The hardware watchdog should reset it within a minute. | +| It pings and the port is open, so why is it broken? | ICMP replies and TCP handshakes are produced by the kernel. They say nothing about userspace. Trust `viewers` on `/healthz`, not `ping`. | +| Wall reboots on its own | Either the kernel panicked or PID 1 missed the watchdog — that is the design working. Find out which before it happens again: `journalctl -b -1 -p err`, `journalctl -b -1 -k \| tail`, and `ls /sys/fs/pstore` (this box has no ERST backend, so a panic trace may not survive). | +| `install-kiosk.sh` pointed the wall somewhere else | It no longer can: an existing `kiosk.conf` URL wins over the built-in default. Pass `KIOSK_URL=...` to change it deliberately. | Debugging rule for this project applies: reproduce at the lowest layer that can fail. `curl` the URL from the client, check `/api/streams` on the server, and @@ -367,8 +546,11 @@ decision, not an oversight. `rpm-ostreed-automatic.timer`. * **SELinux is left enforcing, unmodified.** Nothing here needed a policy tweak; treat a new AVC as a bug in this runbook. -* **No reboot watchdog.** If the wall is remote enough that a hang means driving - to it, that is an ops decision (greenboot, or a smart PDU), not a client one. +* **No power cycle.** A hang is now recovered by the kernel and the chipset + watchdogs above. A machine that is *off* — dead PSU, tripped plug, firmware that + never reaches the bootloader — cannot reboot itself, and if the wall is remote + enough for that to matter it is still an ops decision (greenboot, or a smart + PDU), not a client one. ## Naming on the reference network diff --git a/docs/kiosk-silverblue/90-mirror-wall-selfheal.conf b/docs/kiosk-silverblue/90-mirror-wall-selfheal.conf new file mode 100644 index 0000000..b07932c --- /dev/null +++ b/docs/kiosk-silverblue/90-mirror-wall-selfheal.conf @@ -0,0 +1,37 @@ +# 90-mirror-wall-selfheal.conf — make a hung wall recover on its own. +# +# Installed by install-kiosk.sh to /etc/sysctl.d/. Applied at boot by +# systemd-sysctl, and immediately by `sysctl --system` in the installer. +# +# The trade this makes: a wall whose kernel has oopsed, deadlocked or soft-locked +# is a panel showing a frozen frame forever, in a room nobody is in. Rebooting +# into itself is always the better outcome, so every one of these escalates to a +# reboot. Reasoning and the incident behind it: ../kiosk-silverblue.md +# ("When the wall hangs"). +# +# To revert a single line: drop a file with a higher-numbered name in the same +# directory, e.g. /etc/sysctl.d/91-mirror-wall-quiet.conf containing +# "kernel.panic_on_oops = 0", then `sudo sysctl --system`. + +# The lockup detectors themselves (default on, pinned here so a future base +# image change cannot silently turn them off). +kernel.watchdog = 1 + +# A soft lockup (a CPU spinning in kernel mode with preemption off) or a hard +# lockup (interrupts disabled; caught by the NMI watchdog) panics instead of +# only printing "watchdog: BUG: soft lockup" into a journal nobody reads. +kernel.softlockup_panic = 1 +kernel.hardlockup_panic = 1 + +# An oops used to leave the box nominally up with its display driver dead — the +# worst case, because PID 1 stays healthy and so nothing else would reset it. +kernel.panic_on_oops = 1 + +# Panic, then reboot 30s later. Long enough for the console to be read and for +# the filesystems to be sync'ed by the panic path; short enough that nobody +# waits for it. 0 would mean "sit there forever", which is the bug. +kernel.panic = 30 + +# Not set here on purpose: kernel.hung_task_panic does not exist in the Fedora +# 44 kernel build (CONFIG_DETECT_HUNG_TASK is off). A task blocked on I/O is +# caught by the watchdog in 90-mirror-wall-watchdog.conf instead. diff --git a/docs/kiosk-silverblue/90-mirror-wall-watchdog.conf b/docs/kiosk-silverblue/90-mirror-wall-watchdog.conf new file mode 100644 index 0000000..4811acc --- /dev/null +++ b/docs/kiosk-silverblue/90-mirror-wall-watchdog.conf @@ -0,0 +1,29 @@ +# 90-mirror-wall-watchdog.conf — PID 1 must kick the hardware watchdog. +# +# Installed by install-kiosk.sh to /etc/systemd/system.conf.d/, but ONLY when the +# box actually has a watchdog device (/dev/watchdog). Takes effect on +# `systemctl daemon-reexec`, which the installer runs; a reboot also picks it up. +# +# This is the layer that catches what the sysctls cannot. The soft/hard lockup +# detectors need kernel work to keep happening; a hardware watchdog needs the +# opposite — if PID 1 stops writing /dev/watchdog0 for one whole timeout, the +# chipset resets the machine no matter what the kernel thinks it is doing. That +# includes the case we hit on 2026-11-02, where the box answered ARP and TCP +# handshakes from kernel state while no userspace process could be started. +# +# 30s is not an arbitrary number: it is the timeout iTCO_wdt initialised itself +# with ("heartbeat=30 sec"). systemd kicks at a third of the configured timeout, +# so 30s means a kick every 10s. Asking for more than the device supports would +# make systemd kick less often than the hardware expires — a self-inflicted +# reboot loop — so install-kiosk.sh keeps this at or below the driver's value. +# +# Cost: a PID-1 stall longer than 30s (a genuinely stuck disk, say) reboots the +# wall. On an unattended display that is the intended behaviour. To back off, +# raise the number to something under the device timeout, or set 0 and rely on +# the sysctls alone: +# +# printf '[Manager]\nRuntimeWatchdogSec=0\n' | sudo tee /etc/systemd/system.conf.d/91-mirror-wall-watchdog-off.conf +# sudo systemctl daemon-reexec + +[Manager] +RuntimeWatchdogSec=30s diff --git a/docs/kiosk-silverblue/check-kiosk.sh b/docs/kiosk-silverblue/check-kiosk.sh index 55f16a2..b703074 100755 --- a/docs/kiosk-silverblue/check-kiosk.sh +++ b/docs/kiosk-silverblue/check-kiosk.sh @@ -157,6 +157,53 @@ if [ -n "$HOME_DIR" ]; then fi fi +printf '\n%s\n' '--- hang recovery ---' +# An unattended wall has to end a hang with a reboot, not with a frozen frame for +# a week. Two independent layers; ../kiosk-silverblue.md ("When the wall hangs"). +sc() { sysctl -n "$1" 2>/dev/null || echo missing; } +for k in kernel.softlockup_panic kernel.hardlockup_panic kernel.panic_on_oops; do + v=$(sc "$k") + case "$v" in + 1) ok "$k = 1" ;; + missing) warn "$k does not exist in this kernel build" + warn " (kernel.hung_task_panic is absent from Fedora's kernel; the watchdog covers it)" ;; + *) bad "$k = $v, expected 1 — fix /etc/sysctl.d/90-mirror-wall-selfheal.conf, then sudo sysctl --system" ;; + esac +done +P=$(sc kernel.panic) +case "$P" in + ''|0|missing) bad "kernel.panic = ${P:-unset}: after a panic the box sits there forever" ;; + *) ok "kernel.panic = $P (reboots $P s after a panic)" ;; +esac +if [ -e /dev/watchdog ]; then + WD=$(systemctl show -p RuntimeWatchdogUSec --value 2>/dev/null || echo 0) + WID=$(cat /sys/class/watchdog/watchdog0/identity 2>/dev/null || echo '?') + case "$WD" in + 0|''|infinity) bad "PID 1 is not kicking the $WID watchdog (RuntimeWatchdogUSec=$WD) — a wedged kernel here never recovers" ;; + *) ok "hardware watchdog armed: $WID at $WD (no kick from PID 1 for $WD = chipset reset)" ;; + esac +else + info "no /dev/watchdog on this box — the lockup panics are the only recovery layer" +fi + +printf '\n%s\n' '--- firewall ---' +if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then + ZONE=$(firewall-cmd --get-default-zone) + PORTS=$(firewall-cmd --zone="$ZONE" --list-ports | tr ' ' '\n') + if printf '%s\n' "$PORTS" | grep -qx '1025-65535/tcp'; then + bad "$ZONE opens 1025-65535/tcp — on a box whose only access control is its address, that is every high port" + else + ok "$ZONE: 1025-65535/tcp closed" + fi + if printf '%s\n' "$PORTS" | grep -qx '1025-65535/udp'; then + ok "$ZONE: 1025-65535/udp open (WebRTC media needs it — do not close it)" + else + bad "$ZONE: 1025-65535/udp closed — peer-to-peer media cannot flow, every tile reads 'unreachable'" + fi +else + info "firewalld is not running — nothing listening, nothing to tighten" +fi + printf '\n%s\n' '--- what the screen will show ---' if [ -n "$URL" ]; then BODY=$(curl -fsSL -m 8 "$URL" 2>/dev/null) diff --git a/docs/kiosk-silverblue/install-kiosk.sh b/docs/kiosk-silverblue/install-kiosk.sh index f304fea..e3b119d 100755 --- a/docs/kiosk-silverblue/install-kiosk.sh +++ b/docs/kiosk-silverblue/install-kiosk.sh @@ -15,7 +15,10 @@ set -eu KIOSK_USER="${KIOSK_USER:-kiosk}" -KIOSK_URL="${KIOSK_URL:-https://mirror.reeseapps.com/watch}" +# The default is for a NEW client only. An existing client keeps its own URL +# unless the operator names one — see "the URL is the one per-client value" below. +KIOSK_URL_DEFAULT="https://mirror.reeseapps.com/watch" +KIOSK_URL="${KIOSK_URL:-}" KIOSK_SESSION="gnome-kiosk-script-wayland" HERE=$(cd "$(dirname "$0")" && pwd) NEED_REBOOT=0 @@ -58,7 +61,7 @@ for c in rpm getent restorecon useradd sed awk grep install; do need "$c"; done [ -f /etc/os-release ] && grep -q '^ID=fedora' /etc/os-release || say "warning: this does not look like Fedora; continuing anyway" -say "== 1/7 packages ==" +say "== 1/9 packages ==" if rpm -q gnome-kiosk gnome-kiosk-script-session >/dev/null 2>&1; then say "gnome-kiosk + gnome-kiosk-script-session already present" else @@ -69,7 +72,7 @@ else say "layered the kiosk packages — a reboot is required before they exist" fi -say "== 2/7 kiosk user ==" +say "== 2/9 kiosk user ==" if id -u "$KIOSK_USER" >/dev/null 2>&1; then HOME_DIR=$(getent passwd "$KIOSK_USER" | cut -d: -f6) else @@ -81,7 +84,7 @@ fi KIOSK_GROUP=$(getent passwd "$KIOSK_USER" | cut -d: -f4) say "user $KIOSK_USER uid=$(id -u "$KIOSK_USER") home=$HOME_DIR" -say "== 3/7 payload ==" +say "== 3/9 payload ==" # Firefox enterprise policy: no update prompts, no account/sync, no telemetry, # no first-run pages. /etc is mutable on Silverblue and survives upgrades. install -D -m 0644 "$HERE/policies.json" /etc/firefox/policies/policies.json @@ -119,6 +122,19 @@ say " $HOME_DIR/.local/bin/gnome-kiosk-script" # The URL is the one per-client value; keep it in its own file so everything # else can stay byte-identical across the fleet. +# +# Re-running this installer is the documented way to REPAIR a client, and a repair +# must not move the wall somewhere else. So an existing kiosk.conf wins over the +# default; only an explicit KIOSK_URL= changes it. +KIOSK_CONF="$HOME_DIR/.config/mirror/kiosk.conf" +if [ -z "$KIOSK_URL" ] && [ -f "$KIOSK_CONF" ]; then + KIOSK_URL=$(sed -n 's/^[[:space:]]*MIRROR_KIOSK_URL="\{0,1\}\([^"]*\)"\{0,1\}[[:space:]]*$/\1/p' \ + "$KIOSK_CONF" | head -1) + if [ -n "$KIOSK_URL" ]; then + say " keeping this client's existing URL (pass KIOSK_URL=... to change it)" + fi +fi +[ -n "$KIOSK_URL" ] || KIOSK_URL="$KIOSK_URL_DEFAULT" sed "s|^MIRROR_KIOSK_URL=.*|MIRROR_KIOSK_URL=\"$KIOSK_URL\"|" "$HERE/kiosk.conf" \ | install -m 0644 -o "$KIOSK_USER" -g "$KIOSK_GROUP" /dev/stdin \ "$HOME_DIR/.config/mirror/kiosk.conf" @@ -129,7 +145,7 @@ say " $HOME_DIR/.config/mirror/kiosk.conf -> $KIOSK_URL" install -m 0644 -o "$KIOSK_USER" -g "$KIOSK_GROUP" /dev/null \ "$HOME_DIR/.config/gnome-initial-setup-done" -say "== 4/7 default session ==" +say "== 4/9 default session ==" ACCT="/var/lib/AccountsService/users/$KIOSK_USER" if [ -f "$ACCT" ]; then set_ini "$ACCT" User Session "$KIOSK_SESSION" @@ -141,7 +157,7 @@ fi systemctl restart accounts-daemon.service 2>/dev/null || : say " $ACCT Session=$KIOSK_SESSION" -say "== 5/7 auto-login ==" +say "== 5/9 auto-login ==" if [ ! -d /etc/gdm ]; then say " gdm is not installed — installing it" rpm-ostree install gdm @@ -156,7 +172,7 @@ if [ -d /etc/gdm ]; then say " /etc/gdm/custom.conf: AutomaticLogin=$KIOSK_USER" fi -say "== 6/7 console guard ==" +say "== 6/9 console guard ==" # Cheap insurance against the class of bug above: if the kiosk session exists but # something else holds the foreground VT, bring the wall back. Stops short of # ever restarting gdm. Pause it while debugging the console with: @@ -168,6 +184,64 @@ systemctl daemon-reload systemctl enable --now mirror-kiosk-console-guard.timer say " mirror-kiosk-console-guard.timer enabled" +say "== 7/9 hang recovery ==" +# An unattended wall that hangs must come back by itself. Two independent layers; +# reasoning and the incident behind them in ../kiosk-silverblue.md. +install -D -m 0644 "$HERE/90-mirror-wall-selfheal.conf" /etc/sysctl.d/90-mirror-wall-selfheal.conf +if sysctl --system >/dev/null 2>&1; then + say " soft lockup / hard lockup / oops now panic; panic reboots in $(cat /proc/sys/kernel/panic)s" +else + say " warning: sysctl --system failed — check: sudo sysctl kernel.softlockup_panic kernel.panic" +fi + +# The layer that does not need the kernel to be cooperating: if PID 1 stops +# kicking /dev/watchdog, the chipset resets the box. Only configured when the +# hardware has one (VMs typically do not). +if [ -e /dev/watchdog ]; then + wd_identity=$(cat /sys/class/watchdog/watchdog0/identity 2>/dev/null || echo unknown) + wd_timeout=$(cat /sys/class/watchdog/watchdog0/timeout 2>/dev/null || echo 0) + case "$wd_timeout" in ''|*[!0-9]*) wd_timeout=0 ;; esac + install -D -m 0644 "$HERE/90-mirror-wall-watchdog.conf" /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf + # systemd kicks at a third of the value it is given. Asking for more than the + # device supports would kick less often than the device expires — a reboot + # loop of our own making — so clamp to what the driver came up with. + if [ "$wd_timeout" -gt 0 ] && [ "$wd_timeout" -ne 30 ]; then + sed -i "s|^RuntimeWatchdogSec=.*|RuntimeWatchdogSec=${wd_timeout}s|" \ + /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf + say " clamped to the device's own ${wd_timeout}s timeout ($wd_identity)" + fi + # PID 1 re-reads system.conf on re-exec; nothing else picks it up short of a + # reboot, and re-exec is safe (sockets and running services survive it). + systemctl daemon-reexec + say " $wd_identity hardware watchdog armed (PID 1 must kick it or the box resets)" +else + say " no /dev/watchdog here — hardware layer skipped (lockup panics still installed)" +fi + +say "== 8/9 firewall ==" +# Fedora's Workstation zone opens 1025-65535/tcp and /udp. The UDP half is what +# WebRTC media needs (peer-to-peer, both directions, and mirror uses no STUN/TURN); +# the TCP half is pure exposure on a box whose only access control is its address. +# Dropping it is safe for a wall, which never serves: verified against a live +# share (media keeps flowing at ~460 kbit/s inbound with it gone). +if command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state >/dev/null 2>&1; then + zone=$(firewall-cmd --get-default-zone) + if firewall-cmd --zone="$zone" --list-ports | tr ' ' '\n' | grep -qx '1025-65535/tcp'; then + firewall-cmd --permanent --zone="$zone" --remove-port=1025-65535/tcp >/dev/null + firewall-cmd --reload >/dev/null + say " $zone: dropped 1025-65535/tcp (kept 1025-65535/udp for WebRTC media)" + else + say " $zone: 1025-65535/tcp already closed" + fi + if firewall-cmd --zone="$zone" --list-ports | tr ' ' '\n' | grep -qx '1025-65535/udp'; then + say " 1025-65535/udp still open — required, do not close it or mirrors go black" + else + say " warning: 1025-65535/udp is NOT open — peer-to-peer media will fail (tile: unreachable)" + fi +else + say " firewalld not running — nothing to tighten (verify your own firewall keeps UDP media)" +fi + # Keep a copy of the payload on the box. /tmp is cleared at every boot, so a # checker staged there is gone precisely when you want to run it again — after a # reboot, or from a rescue session hours later. @@ -177,10 +251,12 @@ if [ "$HERE" != "/usr/local/share/mirror-kiosk" ]; then say " /usr/local/share/mirror-kiosk (survives reboot; run check-kiosk.sh from here)" fi -say "== 7/7 selinux labels ==" +say "== 9/9 selinux labels ==" restorecon -RF "$HOME_DIR/.local" "$HOME_DIR/.config" 2>/dev/null || : restorecon -RF /etc/firefox /usr/local/share/gnome-kiosk /usr/local/share/wayland-sessions \ /usr/local/libexec 2>/dev/null || : +restorecon -RF /etc/sysctl.d/90-mirror-wall-selfheal.conf \ + /etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf 2>/dev/null || : say "" say "Installed." diff --git a/docs/wall-watch/README.md b/docs/wall-watch/README.md new file mode 100644 index 0000000..b82b17b --- /dev/null +++ b/docs/wall-watch/README.md @@ -0,0 +1,57 @@ +# wall-watch — presence alert for the mirror wall + +Runs on the **mirror server host**, not on the kiosk. Once a minute it reads the +service's own `/healthz` and pages you over ntfy when nobody is watching, or when +the service stops answering — and tells you when the wall comes back. + +Rationale and the incident that produced it: +[`../kiosk-silverblue.md`](../kiosk-silverblue.md) → *"The wall went quiet and +nobody noticed"*. + +## Install (on the server host) + +```bash +scp -r docs/wall-watch server:/var/tmp/ && ssh server \ + 'cd /var/tmp/wall-watch && sudo sh install-wall-watch.sh' +sudo vi /etc/mirror-wall-watch.env # set NTFY_TOKEN +``` + +## Operate + +```bash +systemctl status mirror-wall-watch.timer # is it still ticking? +journalctl -u mirror-wall-watch.service -f # one line per check, forever +sudo sh /usr/local/libexec/mirror-wall-watch.sh # run it by hand + +# Page yourself on purpose, without touching the wall's real state. WALL_DRILL=1 +# makes the notification say "[drill]" and drop to priority 3, so a test can never +# be mistaken for an outage: +sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=https://127.0.0.1/nope \ + WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test \ + sh /usr/local/libexec/mirror-wall-watch.sh' +``` + +## Files + +| File | Lands at | +|---|---| +| `wall-watch.sh` | `/usr/local/libexec/mirror-wall-watch.sh` | +| `mirror-wall-watch.service` / `.timer` | `/etc/systemd/system/` | +| `wall-watch.env.example` | `/etc/mirror-wall-watch.env` (0600, seeded once, never overwritten) | +| everything here | `/usr/local/share/mirror-wall-watch/` | + +## Deliberate choices + +- **Probes `/healthz` over loopback**, because that endpoint is exempt from the + address gate — the same exemption that lets the container healthcheck work from + loopback when `MIRROR_ALLOW_CIDRS` excludes it. +- **First run cannot alert.** Installing a watcher must not page you about a wall + that was already down before you had one. +- **A failed notification fails the unit** and does *not* bank the re-notify + timestamp, so an ntfy outage is retried on the next tick instead of eating the + alert. +- **A drill is labelled as one.** `WALL_DRILL=1` prefixes the title with `[drill]`, + lowers the priority and says "nothing is wrong" in the body. A test page that is + indistinguishable from an incident teaches people to ignore the page. +- **It cannot detect the server being down.** It lives there. That gap belongs to + whatever monitors this host. diff --git a/docs/wall-watch/install-wall-watch.sh b/docs/wall-watch/install-wall-watch.sh new file mode 100644 index 0000000..89e3d41 --- /dev/null +++ b/docs/wall-watch/install-wall-watch.sh @@ -0,0 +1,76 @@ +#!/bin/sh +# install-wall-watch.sh — install the wall-presence alert on the MIRROR SERVER host. +# +# Run ON THE SERVER, as root, from this directory: +# +# sudo sh install-wall-watch.sh +# +# Idempotent. It never overwrites /etc/mirror-wall-watch.env once it exists, so +# the ntfy token you pasted in survives a re-run. +# +# What it does, and why this exists at all: ../kiosk-silverblue.md +# ("The wall went quiet and nobody noticed"). + +set -eu + +HERE=$(cd "$(dirname "$0")" && pwd) +CONF=/etc/mirror-wall-watch.env +say() { printf '%s\n' "$*"; } +die() { printf 'error: %s\n' "$*" >&2; exit 1; } + +[ "$(id -u)" -eq 0 ] || die "run me as root (sudo sh $0)" +for c in curl logger systemctl; do + command -v "$c" >/dev/null 2>&1 || die "missing required command: $c" +done + +say "== 1/4 program ==" +install -D -m 0755 "$HERE/wall-watch.sh" /usr/local/libexec/mirror-wall-watch.sh +install -d -m 0755 /usr/local/share/mirror-wall-watch +cp -a "$HERE"/. /usr/local/share/mirror-wall-watch/ +rm -f /usr/local/share/mirror-wall-watch/install-wall-watch.sh +say " /usr/local/libexec/mirror-wall-watch.sh" +say " /usr/local/share/mirror-wall-watch (docs, survives reboot)" + +say "== 2/4 config ==" +if [ ! -f "$CONF" ]; then + install -D -m 0600 "$HERE/wall-watch.env.example" "$CONF" + say " created $CONF (0600) — NOW SET NTFY_TOKEN IN IT:" + say " sudo vi $CONF" +else + perm=$(stat -c '%a' "$CONF") + [ "$perm" = "600" ] || say " warning: $CONF is mode $perm, expected 0600 (it holds a token)" + say " kept existing $CONF" +fi + +say "== 3/4 units ==" +install -D -m 0644 "$HERE/mirror-wall-watch.service" /etc/systemd/system/mirror-wall-watch.service +install -D -m 0644 "$HERE/mirror-wall-watch.timer" /etc/systemd/system/mirror-wall-watch.timer +systemctl daemon-reload +systemctl enable --now mirror-wall-watch.timer +say " mirror-wall-watch.timer enabled (every minute)" + +say "== 4/4 prove it ==" +# One run now, so a broken config fails in front of you instead of silently in a +# timer at 03:00. The first run only records the baseline; it cannot alert, which +# is deliberate — installing the watcher must not page you about a wall that was +# already down before you had a watcher. +if systemctl start --wait mirror-wall-watch.service 2>/dev/null; then + say " first check passed:" + journalctl -u mirror-wall-watch.service --no-pager -n 1 2>/dev/null | sed 's/^/ /' || : +else + say " first check FAILED — read this before trusting the timer:" + journalctl -u mirror-wall-watch.service --no-pager -n 12 2>/dev/null | sed 's/^/ /' || : +fi +# An AVC here means SELinux disagreed with something above; say so rather than +# leaving a silently broken alert. +if command -v ausearch >/dev/null 2>&1; then + if ausearch -m avc -ts recent >/dev/null 2>&1; then + say " warning: recent AVCs found — run: sudo ausearch -m avc -ts recent" + fi +fi + +say "" +say "Installed. Fire a test notification (page yourself on purpose):" +say " sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=https://127.0.0.1/nope WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test sh /usr/local/libexec/mirror-wall-watch.sh'" +say "Then watch the timeline:" +say " journalctl -u mirror-wall-watch.service -f" diff --git a/docs/wall-watch/mirror-wall-watch.service b/docs/wall-watch/mirror-wall-watch.service new file mode 100644 index 0000000..f475f6c --- /dev/null +++ b/docs/wall-watch/mirror-wall-watch.service @@ -0,0 +1,16 @@ +[Unit] +Description=Check that the mirror wall is still watching +Documentation=file:///usr/local/share/mirror-wall-watch/README.md +# The probe is only meaningful once nginx (or whatever fronts the service) is up. +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=/usr/bin/sh /usr/local/libexec/mirror-wall-watch.sh +EnvironmentFile=-/etc/mirror-wall-watch.env +# It only ever writes /var/lib/mirror-wall-watch; keep the rest read-only. +NoNewPrivileges=yes +PrivateTmp=yes +ProtectSystem=full +ProtectHome=read-only diff --git a/docs/wall-watch/mirror-wall-watch.timer b/docs/wall-watch/mirror-wall-watch.timer new file mode 100644 index 0000000..f676df2 --- /dev/null +++ b/docs/wall-watch/mirror-wall-watch.timer @@ -0,0 +1,17 @@ +[Unit] +Description=Run the mirror wall presence check every minute +Documentation=file:///usr/local/share/mirror-wall-watch/README.md + +[Timer] +# Check from a minute after boot — the service needs a moment to come up, and a +# wall has its own relaunch loop — and then every minute. The jitter keeps a +# fleet of these from all waking up on the same second. +OnBootSec=1min +OnUnitActiveSec=1min +RandomizedDelaySec=5s +AccuracySec=1s +# If the host was suspended or the timer missed its window, run once anyway. +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/docs/wall-watch/wall-watch.env.example b/docs/wall-watch/wall-watch.env.example new file mode 100644 index 0000000..2821ab1 --- /dev/null +++ b/docs/wall-watch/wall-watch.env.example @@ -0,0 +1,28 @@ +# /etc/mirror-wall-watch.env — config for mirror-wall-watch.timer. +# chmod 0600, root:root. It holds the ntfy token. Never commit a filled-in copy. +# install-wall-watch.sh seeds this from the example and then NEVER overwrites it. + +# The presence probe. /healthz is exempt from the address gate, so loopback works +# even when MIRROR_ALLOW_CIDRS excludes 127.0.0.0/8 — which is why the container +# healthcheck works too. Use whatever loopback URL reaches this deployment. +WALL_HEALTH_URL=https://127.0.0.1/healthz + +# Alert when fewer than this many viewers are attached. Set 0 to stop alerting on +# "nobody is watching" and only alert when the service itself stops answering +# (useful while a fleet has no wall plugged in). +WALL_EXPECT_VIEWERS=1 + +# Consecutive bad checks before the first notification (~1 min per check). Three +# rides out a Firefox relaunch on the wall; raise it if that is normal for you. +WALL_THRESHOLD=3 + +# Keep shouting while it stays down. 0 = alert once per incident. +WALL_RENOTIFY_SECS=3600 + +# Optional: the URL a notification opens when tapped. +WALL_STATUS_URL=https://mirror.reeseapps.com/healthz + +# ntfy (outbound only — this never reads anything back). +NTFY_URL=https://ntfy.reeseapps.com +NTFY_TOKEN= +NTFY_TOPIC=pi diff --git a/docs/wall-watch/wall-watch.sh b/docs/wall-watch/wall-watch.sh new file mode 100644 index 0000000..ec11a73 --- /dev/null +++ b/docs/wall-watch/wall-watch.sh @@ -0,0 +1,213 @@ +#!/bin/sh +# wall-watch.sh — know within minutes that the mirror wall stopped watching. +# +# Run ON THE MIRROR SERVER HOST from mirror-wall-watch.timer (once a minute). +# It reads the service's own /healthz over loopback and posts to ntfy when no +# viewer is attached, or when the service stops answering at all — and again when +# the wall comes back. +# +# Why it exists: on 2026-11-02 the reference wall hard-hung at 14:03Z. The +# service knew the instant it happened — /healthz reported viewers:0 — and said +# nothing, because nothing was listening. The failure was eventually found by +# walking up to the panel, which was in another room of another building. +# +# /healthz is the right probe precisely because it is exempt from the address +# gate (AGENTS.md §3): a loopback check works even when MIRROR_ALLOW_CIDRS +# excludes loopback, which is also why the container healthcheck can work. +# +# Config: /etc/mirror-wall-watch.env (root-owned 0600 — it holds the ntfy token) +# WALL_HEALTH_URL default https://127.0.0.1/healthz +# WALL_EXPECT_VIEWERS default 1; set 0 to stop alerting on "nobody watching" +# WALL_THRESHOLD default 3 consecutive bad checks (~3 min) before shouting +# WALL_RENOTIFY_SECS default 3600; keeps shouting while it stays down +# WALL_STATUS_URL optional; the URL a notification opens +# NTFY_URL NTFY_TOKEN NTFY_TOPIC +# +# Manual run (the timer runs it the same way): +# sudo sh /usr/local/libexec/mirror-wall-watch.sh +# sudo sh -c 'WALL_HEALTH_URL=https://127.0.0.1/nope WALL_THRESHOLD=1 sh /usr/local/libexec/mirror-wall-watch.sh' # fire a test alert +# +# Honest limit: this runs on the server, so it cannot tell you the *server* is +# down. That is a different alert, and it belongs to whatever watches this host. + +set -eu + +CONF="${WALL_WATCH_CONF:-/etc/mirror-wall-watch.env}" +STATE_DIR="${WALL_STATE_DIR:-/var/lib/mirror-wall-watch}" +STATE_FILE="$STATE_DIR/state" + +log() { + printf '%s\n' "$*" + # Under systemd stdout already IS the journal, so writing to /dev/log as well + # puts two copies of every check in it. Keep the syslog trail for interactive + # runs, where stdout is a terminal and nothing else would record it. + [ -t 1 ] && logger -t mirror-wall-watch -- "$*" 2>/dev/null + return 0 +} +die() { log "error: $*"; exit 1; } + +command -v curl >/dev/null 2>&1 || die "curl is required" +# Root is not inherently required, only write access to the state directory — but +# the default lives next to root-only config, so a non-root run must say where it +# is going to fail rather than fail at the interesting moment. +if ! install -d -m 0700 "$STATE_DIR" 2>/dev/null; then + die "cannot create $STATE_DIR — run me as root, or set WALL_STATE_DIR somewhere writable" +fi + +# Apply KEY=VALUE from the config file, but only for variables the caller did not +# already set. An explicit environment must win over the file — the same precedence +# systemd gives Environment= over EnvironmentFile= — otherwise "run the checker +# against a different URL for a moment" silently does nothing, which is exactly the +# kind of thing you only discover when the alert you were testing never arrives. +load_conf() { + [ -r "$1" ] || return 0 + while IFS= read -r line || [ -n "$line" ]; do + case "$line" in ''|'#'*) continue ;; esac + key=${line%%=*} + val=${line#*=} + key=$(printf '%s' "$key" | tr -d '[:space:]') + val=$(printf '%s' "$val" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//') + case "$val" in '\"'*'"') val=${val#\"}; val=${val%\"} ;; esac + # Only sane shell identifiers get assigned. + case "$key" in ''|*[!A-Za-z0-9_]*|[0-9]*) continue ;; esac + eval "cur=\${$key:-}" + [ -n "$cur" ] || eval "$key=\$val" + done <"$1" +} +load_conf "$CONF" + +: "${WALL_HEALTH_URL:=https://127.0.0.1/healthz}" +: "${WALL_EXPECT_VIEWERS:=1}" +: "${WALL_THRESHOLD:=3}" +: "${WALL_RENOTIFY_SECS:=3600}" + +# A drill is an alert fired on purpose to prove the path works. It must never be +# mistaken for an incident, so it says so in the title, the body and the priority +# — an alerting tool whose test page looks like a real page trains people to +# ignore it. +: "${WALL_DRILL:=0}" +if [ "$WALL_DRILL" = "1" ]; then + DRILL_TAG="[drill] " + DRILL_NOTE="> **This is a drill.** Someone ran the test command on \`$(hostname)\`; nothing is wrong with the wall. What it checked is the path: probe -> ntfy -> your phone." + DOWN_PRIORITY=3 +else + DRILL_TAG="" + DRILL_NOTE="" + DOWN_PRIORITY=4 +fi + +# Posts one notification. Returns non-zero instead of exiting: a failed alert +# must not stop the state file being written, or the next run loses the counter +# and the wall pages once a minute forever. +notify() { + prio=$1 tags=$2 title=$3 body=$4 + if [ -z "${NTFY_URL:-}" ] || [ -z "${NTFY_TOPIC:-}" ]; then + log "error: ntfy is not configured (NTFY_URL/NTFY_TOPIC in $CONF)" + return 1 + fi + + # Collect the headers in the positional parameters, and only the ones that + # have values, so ntfy never receives an empty Authorization or X-Click. + set -- -H "X-Priority: $prio" -H "X-Tags: $tags" -H "X-Title: $title" -H "X-Markdown: yes" + [ -n "${NTFY_TOKEN:-}" ] && set -- "$@" -H "Authorization: Bearer $NTFY_TOKEN" + [ -n "${WALL_STATUS_URL:-}" ] && set -- "$@" -H "X-Click: $WALL_STATUS_URL" + + if curl -fsS -m 10 "$@" -d "$body" "$NTFY_URL/$NTFY_TOPIC" >/dev/null 2>&1; then + log "notified (priority $prio): $title" + else + log "error: could not reach $NTFY_URL — the alert did not go out" + return 1 + fi +} + +# ---- read the service's own truth ------------------------------------------------- +probe="" +viewers="" +if probe=$(curl -fsS -m 8 -k "$WALL_HEALTH_URL" 2>/dev/null); then + if command -v jq >/dev/null 2>&1; then + viewers=$(printf '%s' "$probe" | jq -r '.viewers // empty' 2>/dev/null || :) + else + viewers=$(printf '%s' "$probe" | + sed -n 's/.*"viewers"[[:space:]]*:[[:space:]]*\([0-9][0-9]*\).*/\1/p' | head -1) + fi + # 200 but no parseable viewer count is a broken answer, not a healthy one. + [ -n "$viewers" ] || probe="" +fi + +if [ -z "$probe" ]; then + condition="unreachable" + summary="the mirror service did not answer \`$WALL_HEALTH_URL\`" +elif [ "$WALL_EXPECT_VIEWERS" -gt 0 ] && [ "$viewers" -lt "$WALL_EXPECT_VIEWERS" ]; then + condition="quiet" + summary="the service is up but **nothing is watching** (viewers=$viewers)" +else + condition="ok" + summary="viewers=$viewers" +fi + +# ---- state machine ---------------------------------------------------------------- +state="ok" +bad=0 +last_alert=0 +if [ -r "$STATE_FILE" ]; then + # shellcheck disable=SC1090 + . "$STATE_FILE" 2>/dev/null || : +fi +now=$(date +%s) + +NOTIFY_FAILED=0 + +if [ "$condition" = "ok" ]; then + if [ "$state" != "ok" ]; then + notify 3 "heavy_check_mark,recovered" "${DRILL_TAG}Wall back" \ +"**${DRILL_TAG}The mirror wall is watching again.** + +was: \`$state\` +now: $summary +host: \`$(hostname)\` · $(date -u '+%Y-%m-%d %H:%M:%SZ') + +$DRILL_NOTE" || NOTIFY_FAILED=1 + fi + state="ok"; bad=0; last_alert=0 +else + if [ "$state" = "$condition" ]; then bad=$((bad + 1)); else bad=1; fi + if [ "$bad" -ge "$WALL_THRESHOLD" ] && + { [ "$state" != "$condition" ] || [ $((now - last_alert)) -ge "$WALL_RENOTIFY_SECS" ]; }; then + hint="Recover it: \`ssh \` then \`sudo sh /usr/local/share/mirror-kiosk/check-kiosk.sh\`; if ssh itself will not open a session the box is hung, not offline — the watchdog should reset it within a minute of this, and if it does not, power-cycle it." + [ "$condition" = "unreachable" ] && + hint="Check the service first (\`systemctl status\`, container logs) before the wall: this host may be the thing that is broken." + notify "$DOWN_PRIORITY" "rotating_light,wall-down" "${DRILL_TAG}Wall down" \ +"**${DRILL_TAG}Mirror wall: $condition** + +$summary + +\`\`\` +$probe +\`\`\` + +$hint + +host: \`$(hostname)\` · $(date -u '+%Y-%m-%d %H:%M:%SZ') · bad check #$bad + +$DRILL_NOTE" || NOTIFY_FAILED=1 + # Only bank the re-notify timestamp if it actually went out, so a ntfy + # outage is retried on the next tick rather than swallowed. + [ "$NOTIFY_FAILED" -eq 0 ] && last_alert=$now + fi + state="$condition" +fi + +umask 077 +cat >"$STATE_FILE" <