**Phase 08 final verification pass — complete, all green.**
**Verified (all 5 tasks already implemented):**
- `static/wall.js` (`createWall` + `installWallKeys`), `kiosk.html`/`kiosk.js`, `GET /kiosk` route — guards: no `$("#…")` in wall.js, one keydown listener, kiosk ids no-collision across 3 pages, no audio field in `arrange`, no header/share link in markup, address gate covers `/kiosk`
- Cross-page e2e: arrival-arrangement, keypress-on-one-display moves the other (asserted on the second display's decoded moving pixels), dead mirror → `data-layout="wall"` on both, `followers` count, pointer-free meta-test
**Gates (exact outcomes):**
- `uv run ruff check . && uv run ruff format --check .` — passed
- `uv run pyright` — 0 errors
- `uv run pytest` — 341 passed, 5 skipped (realdisplay-gated), 96s
- `MIRROR_REALDISPLAY=1 uv run pytest -m realdisplay` — 4 passed, 1 skipped (documented: host has no audio source for real capture)
- Coverage 96% (>90%); only warning is a starlette/httpx third-party deprecation
**Completion criteria:** all 9 ✅ — e2e green incl. `test_kiosk_layout.py` (the one e2e diff is the phase-mandated address-gate extension to `/kiosk`; assertion logic untouched), full suite 100% passing, README documents `/kiosk`, PLAN.md untouched.
**Documentation:**
- README: fixed the Kiosk-client section that implied the display box is permanently pointed at `/watch` — now states the display page is `/kiosk` with the runbook's one-line re-point (installer default still `/watch` this phase, per docs). Rest of README verified against code (routes, config table, CLI flag, presets, key table) — consistent.
- AGENTS.md: fixed 2 stale statements — CDN test moved to `TestPages` (was `TestClientScripts`), suite runtime ~50s → ~95s. No content added.
**Notable:** PLAN.md §6 says "per-leg `maxBitrate` cap (default 12 Mbit/s)" but code/README default is 25 Mbit/s — pre-existing, left untouched (locked file; flagged for a user decision).
**Next pending phase:** `09_wall_remote_page`.
616 lines
31 KiB
Markdown
616 lines
31 KiB
Markdown
# Kiosk client — Fedora Silverblue
|
||
|
||
The target client for a wall display is **Fedora Silverblue running GNOME Kiosk,
|
||
auto-logged in as `kiosk`, showing `https://mirror.reeseapps.com/watch`
|
||
fullscreen in Firefox.** One box takes about five minutes over SSH and needs no
|
||
keyboard, no monitor access, and no console.
|
||
|
||
Everything is in this directory and is idempotent — re-run it to repair a kiosk
|
||
or to change its URL.
|
||
|
||
| File | Lands at | Why |
|
||
|---|---|---|
|
||
| `install-kiosk.sh` | run on the client as root | the whole install, one pass |
|
||
| `check-kiosk.sh` | run on the client as root | read-only report; nothing needs the console |
|
||
| `gnome-kiosk-script` | `~kiosk/.local/bin/gnome-kiosk-script` | the kiosk application + watchdog loop |
|
||
| `kiosk.conf` | `~kiosk/.config/mirror/kiosk.conf` | **the one per-client value: the URL** |
|
||
| `policies.json` | `/etc/firefox/policies/policies.json` | no updates, accounts, telemetry, first-run pages |
|
||
| `window-config.ini` | `/usr/local/share/gnome-kiosk/window-config.ini` | force the Firefox window fullscreen |
|
||
| `gnome-kiosk-script-wayland.desktop` | `/usr/local/share/wayland-sessions/` | **stops GDM covering the wall with a greeter** |
|
||
| `mirror-kiosk-console-guard` | `/usr/local/libexec/` | reclaims the console if the wall loses it |
|
||
| `mirror-kiosk-console-guard.{service,timer}` | `/etc/systemd/system/` | runs that check every minute |
|
||
| `dim-probe.py` | run on any reachable host | measures what the kiosk actually renders |
|
||
| `90-mirror-wall-selfheal.conf` | `/etc/sysctl.d/` | lockups and oopses become a reboot, not a frozen frame |
|
||
| `90-mirror-wall-watchdog.conf` | `/etc/systemd/system.conf.d/` | PID 1 must kick the hardware watchdog or the chipset resets the box |
|
||
|
||
Presence alerting lives beside this, not inside it: [`../wall-watch/`](../wall-watch/)
|
||
runs on the **server** host and pages you when a wall goes quiet.
|
||
|
||
---
|
||
|
||
## Add a client
|
||
|
||
```bash
|
||
# 1. A vanilla Silverblue install. Enable Settings → Sharing → Remote Login.
|
||
# Create an admin user (the first one you make) — that is your SSH account.
|
||
|
||
# 2. Layer the kiosk packages. This is the only step that needs a reboot.
|
||
ssh client 'sudo rpm-ostree install gnome-kiosk gnome-kiosk-script-session && sudo reboot'
|
||
|
||
# 3. Wait ~60s, then install everything else. Stage it in /var/tmp, not /tmp:
|
||
# /tmp is wiped at every boot, so a checker staged there is gone exactly when
|
||
# you want it. The installer keeps a permanent copy of the payload in
|
||
# /usr/local/share/mirror-kiosk.
|
||
scp -r docs/kiosk-silverblue client:/var/tmp/ && ssh client \
|
||
'cd /var/tmp/kiosk-silverblue && sudo sh install-kiosk.sh'
|
||
|
||
# 4. Hand the console to the kiosk.
|
||
ssh client 'sudo systemctl restart gdm'
|
||
|
||
# 5. Prove it, without touching the machine.
|
||
ssh client 'sudo sh /usr/local/share/mirror-kiosk/check-kiosk.sh'
|
||
```
|
||
|
||
`install-kiosk.sh` creates the `kiosk` user if it does not exist, so step 2 can
|
||
be skipped when the packages are already layered. To point a client somewhere
|
||
other than the production wall, pass the URL:
|
||
|
||
```bash
|
||
sudo KIOSK_URL=https://mirror.reeseapps.com/watch sh install-kiosk.sh
|
||
```
|
||
|
||
Only `kiosk.conf` differs between clients; every other file stays byte-identical
|
||
so the fleet is auditable.
|
||
|
||
---
|
||
|
||
## Two endpoints: `/watch` and `/kiosk`
|
||
|
||
The wall has two client pages, and they are not the same page twice:
|
||
|
||
* **`/watch` is a wall a person operates.** The operator's page: the grid,
|
||
the keys, and the chrome a person standing in front of it needs — header,
|
||
share link, counts.
|
||
* **`/kiosk` is a display that obeys a wall.** No chrome at all: no header,
|
||
no counts, no share button — the net pill is the only thing on screen that
|
||
is not a mirror. It subscribes to every mirror like `/watch` does and then
|
||
does what the shared layout state says — on arrival *and* on every change —
|
||
with no input from anyone standing at the box. Point the box at it and it
|
||
arranges itself: whichever mirror is on stage, fullscreen or not, is
|
||
applied before the first tile finishes painting, so a person who points a
|
||
kiosk at `/kiosk` never has to touch it. When a staged mirror goes away,
|
||
every display falls back to the wall in the same frame set; no display sits
|
||
on a dead tile.
|
||
|
||
The state is the boss, locally too: **the wall keys still work on `/kiosk`**
|
||
for the moment a keyboard is plugged in (see [the key table](#operating-the-wall-keys)),
|
||
and a keypress there performs the change *and* publishes the same `arrange`
|
||
intent, so a controller and a display converge on one state instead of
|
||
disagreeing (`m` is local-only — a display never becomes audible because
|
||
another machine asked). The **controller side** — a page a person on another
|
||
machine drives the wall from — arrives in phase 09 (`/remote`); until then a
|
||
controller is any client that can send an `arrange` frame on `/ws`, `websocat`
|
||
included. `GET /api/kiosk` is the state's read surface: what is on stage,
|
||
whether it is fullscreen, and how many displays (`followers`) are obeying it.
|
||
|
||
**The migration rule, stated honestly.** Existing clients keep their own
|
||
`MIRROR_KIOSK_URL`: an existing `kiosk.conf` URL wins over the built-in
|
||
default, and the installer's default still points at `/watch` in this phase.
|
||
So **nothing changes under a box until its `kiosk.conf` is re-pointed and
|
||
`gdm` restarted**:
|
||
|
||
```bash
|
||
sudo sed -i 's|^MIRROR_KIOSK_URL=.*|MIRROR_KIOSK_URL="https://…/kiosk"|' \
|
||
/var/home/kiosk/.config/mirror/kiosk.conf
|
||
sudo systemctl restart gdm
|
||
```
|
||
|
||
(`pkill -u kiosk firefox` is the shorter path — the loop re-sources
|
||
`kiosk.conf` on every relaunch, see [the section below](#the-two-facts-that-make-this-work) —
|
||
`gdm` is the clean restart when you want the whole session back.) The kiosk
|
||
page serves `<title>Wall · mirror`; `check-kiosk.sh`, untouched in this
|
||
phase, still greps for `<title>Watch · mirror`, and the checker is widened to
|
||
accept the kiosk title in phase 09, when the default URL moves with it.
|
||
|
||
---
|
||
|
||
## The two facts that make this work
|
||
|
||
**Silverblue's mutable seams.** `/usr` is read-only and belongs to the OSTree
|
||
commit, but `/etc`, `/var` and `/usr/local` (which *is* `/var/usrlocal`) are
|
||
writable and survive upgrades. That is why the payload goes to `/etc/firefox`,
|
||
`/usr/local/share/gnome-kiosk` and `/var/home/kiosk` and nowhere else. Never
|
||
`ostree admin unlock` to shove files into `/usr` — the next upgrade deletes them.
|
||
|
||
**GNOME Kiosk's script session.** `gnome-kiosk-script-session` starts a locked
|
||
down mutter session (no top bar, no activities, no dock) and runs
|
||
`~/.local/bin/gnome-kiosk-script` as the kiosk application. Two consequences
|
||
drive the design of that file:
|
||
|
||
* It **re-executes itself forever**, so an unguarded launch opens a new window
|
||
every few seconds. The script checks `pgrep -u $(id -u) -x firefox` first.
|
||
* It **must never exit**. `org.gnome.Kiosk.Script.service` carries
|
||
`OnSuccess=gnome-session-shutdown.target`, so the script returning logs the
|
||
kiosk out and the screen falls back to the GDM greeter. Exiting is the failure
|
||
mode, not crashing — which is exactly why a browser that dies is a feature:
|
||
the loop comes back and starts it again.
|
||
|
||
The loop also **re-sources `kiosk.conf` on every relaunch**, so changing the URL
|
||
is:
|
||
|
||
```bash
|
||
sudo sed -i 's|^MIRROR_KIOSK_URL=.*|MIRROR_KIOSK_URL="https://…/watch"|' \
|
||
/var/home/kiosk/.config/mirror/kiosk.conf
|
||
sudo pkill -u kiosk firefox # loop notices within 5s and relaunches
|
||
```
|
||
|
||
---
|
||
|
||
## Operating the wall: keys
|
||
|
||
The whole wall works from a keyboard — and the box needs no keyboard attached
|
||
to be operated *or* debugged: everything about it is checkable over SSH (see
|
||
[Verifying a kiosk you cannot see](#verifying-a-kiosk-you-cannot-see)). The keys
|
||
are there when one is plugged in. On `/watch`:
|
||
|
||
| Key | Action |
|
||
|---|---|
|
||
| `j` / `↓` · `k` / `↑` | select the next / previous mirror (wraps) |
|
||
| `Enter` / `f` | stage the selected mirror fullscreen (`f` re-enters fullscreen) |
|
||
| `g` | back to the wall |
|
||
| `n` / `p` | stage the next / previous mirror |
|
||
| `1`–`9` | stage the Nth mirror |
|
||
| `m` | mute / unmute the staged mirror |
|
||
| `?` | show / hide the key list on screen |
|
||
| `Esc` | close the key list, then leave the stage |
|
||
|
||
On `/mirror`: `s` starts a share, `x` stops one, `Esc` stops while live. Keys
|
||
with `Ctrl`/`Cmd`/`Alt` are never intercepted, and a key typed in the
|
||
display-name field is text, not a command. `?` is the only key that needs
|
||
Shift, and it is matched on the character itself, so on a layout where `?` is a
|
||
dead key the list is not reachable — keep this table to hand.
|
||
|
||
**Sound.** The wall is silent until someone presses `m` (or clicks a tile's
|
||
audio button): only the staged mirror can be audible, and leaving the stage
|
||
re-mutes it. What that sound is depends on what the *sharer's* browser was
|
||
given — Chrome on Linux picks up a tab's audio reliably, system audio only
|
||
where PipeWire offers it, and Firefox's desktop capture does not offer system
|
||
audio at all (SECURITY.md §4 "Audio").
|
||
|
||
---
|
||
|
||
## Why each piece is where it is
|
||
|
||
**Auto-login — `/etc/gdm/custom.conf`**
|
||
|
||
```ini
|
||
[daemon]
|
||
AutomaticLoginEnable=True
|
||
AutomaticLogin=kiosk
|
||
TimedLoginEnable=False
|
||
```
|
||
|
||
`TimedLoginEnable=False` matters: a timed login would hand the console to the
|
||
greeter on its own schedule and you get a login form on your wall.
|
||
|
||
**Default session — `/var/lib/AccountsService/users/kiosk`**
|
||
|
||
```ini
|
||
[User]
|
||
Session=gnome-kiosk-script-wayland
|
||
```
|
||
|
||
This is what GDM logs the kiosk user *into*. Without it auto-login drops the
|
||
user into a normal GNOME session with a top bar. The installer restarts
|
||
`accounts-daemon` afterwards, because the daemon caches these files and happily
|
||
writes the stale value back over your edit.
|
||
|
||
**gsettings — applied inside the session, not at install time**
|
||
|
||
The screen must never blank or lock. This cannot be done from your SSH session:
|
||
`gsettings` only commits when it can reach a D-Bus session bus, and over SSH it
|
||
prints `Cannot autolaunch D-Bus without X11` and silently leaves the default. It
|
||
also cannot live in `/etc/dconf/db/local.d`, because `/usr/bin/gnome-kiosk` sets
|
||
`DCONF_PROFILE=gnomekiosk` and that profile is `user-db:user` plus the shipped
|
||
compiled defaults — it never reads a system db.
|
||
|
||
So the kiosk script applies them itself on first run each boot, stamped by a
|
||
file in `$XDG_RUNTIME_DIR` (which is per-login, so it re-applies if someone
|
||
changes a setting by hand):
|
||
|
||
```sh
|
||
gsettings set org.gnome.desktop.session idle-delay 0
|
||
gsettings set org.gnome.desktop.screensaver lock-enabled false
|
||
gsettings set org.gnome.desktop.interface enable-animations false
|
||
```
|
||
|
||
**Firefox window rules — `/usr/local/share/gnome-kiosk/window-config.ini`**
|
||
|
||
gnome-kiosk reads the **first** `window-config.ini` it finds in
|
||
`$XDG_CONFIG_HOME/gnome-kiosk`, then `/var/lib/flatpak/exports/share`,
|
||
`/usr/local/share`, `/usr/share`. Because ours wins, the shipped file is not
|
||
read at all — anything it defines is lost unless repeated. Ours therefore copies
|
||
the shipped `[notifications]` section verbatim and adds:
|
||
|
||
```ini
|
||
[mirror-browser]
|
||
match-class=org.mozilla.*
|
||
set-fullscreen=true
|
||
```
|
||
|
||
`firefox --kiosk` already goes fullscreen; this is the belt so a window that
|
||
comes back from a resume still fills the output.
|
||
|
||
**Firefox policy — `/etc/firefox/policies/policies.json`**
|
||
|
||
No update prompts, no Firefox Account or Sync, no studies, no telemetry, no
|
||
first-run/restore pages, no default bookmarks, no saved-password prompts.
|
||
`/usr/lib64/firefox/distribution/` is the other documented location and it is
|
||
inside the read-only `/usr`.
|
||
|
||
> Verified on this box: the file installs, is world-readable with a sane SELinux
|
||
> label, and Firefox runs happily with it. What is *not* verified headlessly is
|
||
> that each policy is applied — Firefox's policy engine does not initialise in
|
||
> the `--headless --screenshot`/`--dump-dom` modes, so a probe there proves
|
||
> nothing either way. Open `about:policies` on a real kiosk display to confirm;
|
||
> it lists every policy Firefox read and any it rejected.
|
||
|
||
**Initial-setup guard — `~kiosk/.config/gnome-initial-setup-done`**
|
||
|
||
Without it the GNOME "welcome" wizard claims the session on first login and the
|
||
kiosk never starts. The journal confirms the guard working:
|
||
`gnome-initial-setup-first-login.service … skipped, unmet condition check
|
||
ConditionPathExists=!/var/home/kiosk/.config/gnome-initial-setup-done`.
|
||
|
||
---
|
||
|
||
## The greeter that eats the wall
|
||
|
||
The shipped `gnome-kiosk-script-wayland.desktop` declares:
|
||
|
||
```ini
|
||
X-GDM-SessionRegisters=false
|
||
```
|
||
|
||
That is true of the session itself — gnome-kiosk is a bare mutter kiosk, so
|
||
nothing ever registers with GDM. But GDM reads "never registered" as "this
|
||
display is still mine", and about ten seconds after a perfectly successful
|
||
auto-login it **autospawns a full greeter session on the same GPU**. A login
|
||
screen covers the wall.
|
||
|
||
The part that makes this maddening: the kiosk keeps running underneath. The
|
||
session stays `Active`, Firefox keeps streaming, the mirror roster keeps
|
||
counting the wall as a viewer, `loginctl` reports the kiosk on `seat0` — every
|
||
remote check says nothing is wrong while a password prompt is on the physical
|
||
screen. This is what "it didn't auto-login" actually looks like.
|
||
|
||
From a real boot, `journalctl -b`:
|
||
|
||
```
|
||
19:07:28 PAM:authentication grantors=pam_permit acct="kiosk" ... res=success
|
||
19:07:28 New session '1' of user 'kiosk' with class 'user' and type 'wayland'
|
||
19:07:30 Started org.gnome.Kiosk@wayland.service ← the wall is up
|
||
19:07:38 New session 'c1' of user 'gdm-greeter' with class 'greeter'
|
||
19:07:39 gnome-shell: Registering session with GDM ← the greeter eats it
|
||
19:08:46 Stopped org.gnome.Shell@gdm.service ← 68s of login screen
|
||
```
|
||
|
||
**The fix** is the local override of that one file, changing only
|
||
`X-GDM-SessionRegisters` to `true`, so GDM treats the kiosk session as the owner
|
||
of the display and never brings up a greeter. Two things about it are traps:
|
||
|
||
* **Keep the file name and session id identical to the shipped one.** Inventing
|
||
`mirror-kiosk.desktop` and pointing AccountsService at it does *not* work —
|
||
GDM silently falls back to the plain GNOME session, and you get a normal
|
||
desktop with a top bar instead of a kiosk. Verified the hard way.
|
||
* A normal GNOME session already sets `true`, which is why this bug only ever
|
||
shows up on the kiosk.
|
||
|
||
After the override, the same boot looks like this: `gdm-greeter` appears **zero**
|
||
times, the session registers, and the kiosk compositor and Firefox come up
|
||
undisturbed.
|
||
|
||
## And if the wall loses the console anyway
|
||
|
||
Because the failure above is invisible from the outside, the install also drops
|
||
in a guard: once a minute it checks that the kiosk's `seat0` session is still the
|
||
foreground VT, and if not it runs `loginctl activate` on it and writes one line
|
||
to the journal.
|
||
|
||
It is deliberately timid — it never restarts gdm and never logs anyone out, and
|
||
if there is no kiosk session at all it only reports that, because a display
|
||
manager is a bad thing to fight automatically. To stop it fighting you while you
|
||
debug the console by hand:
|
||
|
||
```bash
|
||
sudo systemctl stop mirror-kiosk-console-guard.timer
|
||
sudo systemctl start mirror-kiosk-console-guard.timer # when finished
|
||
sudo journalctl -u mirror-kiosk-console-guard.service # what it has done
|
||
```
|
||
|
||
Tested by stealing the console with `chvt 1`; the wall came back on its own:
|
||
|
||
```
|
||
mirror-kiosk-guard[4598]: kiosk session 1 brought back to the foreground
|
||
```
|
||
|
||
## Verifying a kiosk you cannot see
|
||
|
||
The console is not an interface. Every claim below is checkable over SSH.
|
||
|
||
| Claim | Evidence |
|
||
|---|---|
|
||
| Auto-login works | `loginctl list-sessions` → a `seat0` session whose user is `kiosk` |
|
||
| The right session | `loginctl show-session N -p Type` → `wayland`; `pgrep -u kiosk -a gnome-kiosk` |
|
||
| The browser is up in kiosk mode | `pgrep -u kiosk -a firefox` → `--kiosk … <url>` |
|
||
| It restarted after a crash | `~kiosk/.local/state/mirror-kiosk.log` shows `exited rc=143` then `launching:` |
|
||
| The page loaded | the **server's** `GET /api/streams` shows the kiosk in `viewers` and its socket in `ss -tn` |
|
||
| Media really flows | the sharer's own page renders each viewer's `RTCPeerConnection.connectionState`; `connected` = ICE+DTLS+SRTP up |
|
||
| It is truly fullscreen | `dim-probe.py`: `viewport == window == screen == available` |
|
||
| The URL is serving *mirror* | `check-kiosk.sh` greps the page for `<title>Watch · mirror` |
|
||
| The kiosk holds the screen | `/sys/class/tty/tty0/active` equals the kiosk session's TTY and the session is `Active` — the one check that catches a covered wall |
|
||
| No greeter is fighting it | `journalctl -b \| grep -c "New session.*gdm-greeter"` is `0`; any non-zero number is a login screen on your wall |
|
||
| It survives a power cut | `sudo systemctl reboot`, then re-run `check-kiosk.sh` |
|
||
|
||
Measured on the reference client, which is attached to a 4K output:
|
||
|
||
```
|
||
viewport=3840x2160 window=3840x2160 screen=3840x2160 available=3840x2160
|
||
```
|
||
|
||
Window equal to viewport means no browser chrome is left over; screen equal to
|
||
available means no panel or bar took any of it.
|
||
|
||
After a cold boot the wall re-joins on its own — reference client back up and
|
||
subscribed to a live mirror in under 60 seconds:
|
||
|
||
```
|
||
{"mirrors": 1, "sharers": 1, "viewers": 1, "connections": 2, "watching": 1}
|
||
Firefox · Linux | Primary Monitor | 2560 x 1600 | viewers 1
|
||
```
|
||
|
||
### One trap worth knowing
|
||
|
||
`/watch` works over plain `http://<lan-ip>`, but **`/mirror` does not**:
|
||
`getDisplayMedia()` needs a secure context, so the Share button is disabled
|
||
there. Pointing a kiosk at a bare-LAN-IP mirror gives you a wall that watches
|
||
fine and cannot share. Production is HTTPS at the edge, so both work — which is
|
||
the main practical reason the kiosk URL should be the `https://` one and not a
|
||
convenient IP.
|
||
|
||
---
|
||
|
||
## When the wall hangs: making it come back by itself
|
||
|
||
On 2026-11-02 the reference wall hung solid at 14:03Z. Nothing said so, nothing
|
||
recovered it, and it stayed that way until someone walked over and powered it off.
|
||
Everything in this section exists so the next one is a 90-second blink instead of
|
||
a drive.
|
||
|
||
**What the hang actually looked like** — worth reading, because every easy
|
||
diagnosis was wrong:
|
||
|
||
| Signal | Reading |
|
||
|---|---|
|
||
| `ping` | answered, 0% loss, RTT jittering 1.9–8.2 ms on a LAN |
|
||
| `ssh` | **`connect()` succeeded**, then the socket closed 6–13 ms later with **zero bytes**. No SSH banner ever, and `sshd` never logged an attempt. |
|
||
| the service | the wall's `/ws` dropped at 14:03:04Z after 1212 s and never reconnected; `/healthz` → `viewers:0` (it is normally `viewers:1, watching:1`) |
|
||
| journal | every unit goes silent at 09:01:59 and stays silent for the 13 minutes until the power cut |
|
||
| memory | 62 GiB installed, 2.5 GiB in use, **zero OOM lines** in the previous boot |
|
||
| `pstore` | empty; the next boot reported `system.journal corrupted or uncleanly shut down` — the hard reset, and the reason the journal's tail is lost |
|
||
|
||
The trap: **ICMP replies and TCP handshakes are kernel work.** "It pings" and
|
||
"port 22 is open" say nothing about userspace. Fedora starts `sshd` per
|
||
connection, so a master that cannot fork answers the SYN from the listen socket
|
||
and then hangs up without a byte — which is exactly what happened. Put together:
|
||
the kernel was alive and no userspace process could be started. Not memory, not a
|
||
full disk, not a fail2ban ban (a ban refuses the SYN, it does not complete the
|
||
handshake) — a hard hang that left no trace.
|
||
|
||
**Two layers, because neither one covers the other:**
|
||
|
||
| Layer | File | Catches |
|
||
|---|---|---|
|
||
| kernel escalates to a reboot | `/etc/sysctl.d/90-mirror-wall-selfheal.conf` | soft lockup, hard lockup (NMI watchdog), any oops → panic; `kernel.panic = 30` reboots 30 s later |
|
||
| the chipset reboots the kernel | `/etc/systemd/system.conf.d/90-mirror-wall-watchdog.conf` | PID 1 stops kicking `/dev/watchdog` — i.e. the case above, where the kernel is too wedged to notice it is wedged |
|
||
|
||
Verify all of it (or just run `check-kiosk.sh`, which asserts every line):
|
||
|
||
```bash
|
||
ssh mirror 'systemctl show -p RuntimeWatchdogUSec --value' # 30s
|
||
ssh mirror 'cat /sys/class/watchdog/watchdog0/status' # 0x8100 on the reference box
|
||
ssh mirror 'sysctl kernel.softlockup_panic kernel.hardlockup_panic kernel.panic_on_oops kernel.panic'
|
||
# 1 1 1 30
|
||
```
|
||
|
||
`status` is a bitmask of driver features *and* current state, so do not compare it
|
||
for equality: the reference box reports `0x8100` = `WDIOF_KEEPALIVEPING` (PID 1 is
|
||
currently kicking it) plus `WDIOF_MAGICCLOSE` (a clean shutdown stops it, so the
|
||
watchdog cannot reset a box that is deliberately powered off). The bit that matters
|
||
is `0x8000`. `check-kiosk.sh` asserts `RuntimeWatchdogUSec` instead, which is not
|
||
ambiguous.
|
||
|
||
**Why 30 seconds.** It is the timeout `iTCO_wdt` initialises itself with
|
||
(`heartbeat=30 sec`), and systemd kicks at a third of whatever it is handed. Asking
|
||
a device for more than it supports would make systemd kick *less* often than the
|
||
hardware expires — a reboot loop of our own making — so `install-kiosk.sh` clamps
|
||
to the driver's own value. Boxes with no watchdog at all (most VMs) get the sysctls
|
||
and a line saying so.
|
||
|
||
**The cost, stated plainly.** Any stall of PID 1 longer than 30 s now reboots the
|
||
wall — a genuinely stuck disk, say. On an unattended display that is the intended
|
||
trade, but it is a trade; to back off raise the value or set it to 0 and rely on
|
||
the sysctls alone (the drop-in documents both).
|
||
|
||
**What it still cannot do.** `kernel.hung_task_panic` does not exist in Fedora's
|
||
kernel build (`CONFIG_DETECT_HUNG_TASK` is off), so a pure blocked-on-I/O hang is
|
||
caught by the watchdog rather than by a targeted panic — and a machine that is
|
||
*off* (dead PSU, tripped plug, firmware that never reaches the bootloader) cannot
|
||
watchdog anything. That remains an ops decision.
|
||
|
||
---
|
||
|
||
## The wall's firewall
|
||
|
||
Fedora's `FedoraWorkstation` zone — the default on Silverblue — ships with:
|
||
|
||
```
|
||
ports: 1025-65535/udp 1025-65535/tcp 443/tcp 80/tcp
|
||
```
|
||
|
||
The **UDP** half is load-bearing: media is peer-to-peer and mirror deliberately
|
||
uses no STUN/TURN, so a wall that cannot receive UDP on high ports shows
|
||
`unreachable` tiles. The **TCP** half is pure exposure on a box whose only access
|
||
control is its address (A19) — it made every high port on the wall connectable
|
||
from anywhere the wall was reachable, and a wall never serves anything.
|
||
|
||
`install-kiosk.sh` step 8/9 removes `1025-65535/tcp`, leaves the UDP alone, and
|
||
says which of the two it found; `check-kiosk.sh` FAILs if either half moves.
|
||
|
||
This was measured, not assumed. With the TCP range gone, the wall's WebRTC socket
|
||
process still had ICE candidates bound on LAN IPv4 and public IPv6, and media kept
|
||
flowing inbound:
|
||
|
||
```
|
||
wall NIC over 6s: rx 459.6 kbit/s tx 16.6 kbit/s
|
||
```
|
||
|
||
while inbound probes from a same-subnet host went `tcp/22` reachable and
|
||
`tcp/8080`, `tcp/9999` silent. (To see the ICE sockets, grep for
|
||
`Socket Process` — Firefox's WebRTC sockets are not owned by a process named
|
||
`firefox`.)
|
||
|
||
Still open and not needed by a wall: `80/tcp`, `443/tcp`, `samba-client`. Left
|
||
alone because they are not this change.
|
||
|
||
---
|
||
|
||
## The wall went quiet and nobody noticed
|
||
|
||
`/healthz` knew at 14:03:04Z that nothing was watching. It is a JSON endpoint and
|
||
nothing was reading it, so the outage was discovered by looking at the panel. The
|
||
other half of running an unattended wall is being told about it —
|
||
[`../wall-watch/`](../wall-watch/) checks presence on the server host every minute
|
||
and pages over ntfy, then announces the recovery.
|
||
|
||
```bash
|
||
tar czf - -C docs wall-watch | ssh server \
|
||
'cd /var/tmp && tar xzf - && cd wall-watch && sudo sh install-wall-watch.sh'
|
||
```
|
||
|
||
Test it the honest way — a **drill**, labelled as one in the title, body and
|
||
priority so it can never be mistaken for an incident:
|
||
|
||
```bash
|
||
sudo sh -c '. /etc/mirror-wall-watch.env; WALL_DRILL=1 WALL_HEALTH_URL=http://127.0.0.1:1/healthz \
|
||
WALL_THRESHOLD=1 WALL_STATE_DIR=/tmp/wall-watch-test sh /usr/local/libexec/mirror-wall-watch.sh'
|
||
```
|
||
|
||
Two things worth knowing: it probes loopback `/healthz`, which works only because
|
||
that path is exempt from the address gate (see `AGENTS.md` §3); and it cannot
|
||
detect the **server** being down, because it lives there. `scp` to some of these
|
||
hosts dies with `lost connection` (no sftp subsystem) — the tar pipe above works
|
||
everywhere.
|
||
|
||
---
|
||
|
||
## Why there is no RDP on the wall
|
||
|
||
You cannot see the panel, so of course the idea is to look at it remotely.
|
||
`gnome-remote-desktop` is already in the Silverblue base image, so it looks free.
|
||
It does not work here, for a structural reason:
|
||
|
||
**`grd` does not capture a screen.** It is a *client* of mutter's remote-desktop
|
||
D-Bus API, and gnome-kiosk embeds libmutter without exporting it. Ask the wall's
|
||
own session bus:
|
||
|
||
```bash
|
||
sudo -u kiosk XDG_RUNTIME_DIR=/run/user/1001 \
|
||
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus \
|
||
busctl call org.gnome.Mutter.RemoteDesktop /org/gnome/Mutter/RemoteDesktop \
|
||
org.freedesktop.DBus.Peer Ping
|
||
# Call failed: The name is not activatable (same for .ScreenCast and .DisplayConfig)
|
||
```
|
||
|
||
Not a missing package: gnome-shell 50.6 *is* installed and is what exports those
|
||
names; gnome-kiosk 50.1 does not. So `grdctl rdp enable` and
|
||
`grdctl rdp set-credentials` would give you a running daemon with nothing to
|
||
share. For whoever tries next:
|
||
|
||
- this image ships **`grdctl`, not `grdc`**, and has **no `org.gnome.RemoteDesktop`
|
||
gsettings schema** — configuration is grdctl/JSON, so the `DCONF_PROFILE` trap
|
||
above does not even apply to it.
|
||
- the only way to really see the panel over RDP is to move the wall onto a
|
||
gnome-shell-based session. That replaces the script session this whole runbook
|
||
is built on — the never-exit loop, `window-config.ini`, the greeter trap — and is
|
||
a `PLAN.md` decision with a full re-validation, not a config tweak.
|
||
- **and it would not have survived the hang above.** An RDP server is userspace;
|
||
the failure was userspace being unable to start. RDP is a way to *look*, never a
|
||
rescue path. The rescue path is the watchdog.
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Cause and fix |
|
||
|---|---|
|
||
| **Login screen on the wall, but every remote check passes** | The session is not registered with GDM, so GDM autospawns a greeter over it. See [The greeter that eats the wall](#the-greeter-that-eats-the-wall); confirm with `ls /usr/local/share/wayland-sessions/` |
|
||
| Wall vanished, console on a blank VT | Someone or something switched VT — the guard reclaims it within a minute; `journalctl -u mirror-kiosk-console-guard.service` |
|
||
| Greeter on screen after boot and `seat0` is another user | `sudo systemctl restart gdm`. GDM reads `custom.conf` at start only |
|
||
| Kiosk logs in to a normal desktop | `Session=` empty in `/var/lib/AccountsService/users/kiosk`; re-run the installer (it restarts `accounts-daemon`) |
|
||
| Wall shows a non-mirror page | the URL serves something else. `check-kiosk.sh` calls this out; a stale vhost/DNS looks like a working kiosk showing a login form |
|
||
| Black screen, no browser at all | the URL is unreachable so the loop is waiting. `curl -fsSL <url>` from the client; then look in `~/.local/state/mirror-kiosk.log` |
|
||
| A new window every few seconds | someone removed the `pgrep` guard from the kiosk script |
|
||
| Blank on wake from idle | the gsettings did not apply (they must run inside the session) — `journalctl _UID=<kiosk uid> \| grep -i permission` |
|
||
| `Permission denied` about keyrings/ICC | `~kiosk/.local` is not owned by kiosk. `sudo chown -R kiosk:kiosk /var/home/kiosk/.local` and re-run the installer |
|
||
| Firefox shows a restore-session or first-run page | the policy file is missing or unreadable — `sudo cat /etc/firefox/policies/policies.json` |
|
||
| Tile says `unreachable` | the sharer and the kiosk could not find each other over ICE. Both must be on a network where they can send UDP directly; mirror uses no STUN/TURN by design |
|
||
| Wall shows the "not allowed" page | the kiosk's address is outside `MIRROR_ALLOW_CIDRS` — this is admission control working, see `SECURITY.md` |
|
||
| **`ssh` connects, then closes in milliseconds with no banner** | Not a firewall ban — a ban refuses the SYN. The kernel is answering from the listen socket while `sshd` cannot spawn a session: the box is **hung**, and nothing userspace on it, including this checker, will run. The hardware watchdog should reset it within a minute. |
|
||
| It pings and the port is open, so why is it broken? | ICMP replies and TCP handshakes are produced by the kernel. They say nothing about userspace. Trust `viewers` on `/healthz`, not `ping`. |
|
||
| Wall reboots on its own | Either the kernel panicked or PID 1 missed the watchdog — that is the design working. Find out which before it happens again: `journalctl -b -1 -p err`, `journalctl -b -1 -k \| tail`, and `ls /sys/fs/pstore` (this box has no ERST backend, so a panic trace may not survive). |
|
||
| `install-kiosk.sh` pointed the wall somewhere else | It no longer can: an existing `kiosk.conf` URL wins over the built-in default. Pass `KIOSK_URL=...` to change it deliberately. |
|
||
|
||
Debugging rule for this project applies: reproduce at the lowest layer that can
|
||
fail. `curl` the URL from the client, check `/api/streams` on the server, and
|
||
only then reach for the browser.
|
||
|
||
---
|
||
|
||
## Deliberately not done here
|
||
|
||
These are real hardening steps for a kiosk in a public space; each is a
|
||
decision, not an oversight.
|
||
|
||
* **The `kiosk` user has no password.** Auto-login does not need one and a
|
||
password does not stop anyone with a keyboard. If you want to be able to log in
|
||
as `kiosk`, set one with `sudo passwd kiosk`.
|
||
* **TTYs are live.** Ctrl-Alt-F3 still gets a console, which is your rescue
|
||
path. `mask`-ing `getty@tty3..6` is the usual lockdown — do it only once the
|
||
kiosk is known good, since it removes the way back in without SSH.
|
||
* **No GRUB password**, so anyone with physical access can boot something else.
|
||
* **No automatic OS updates.** `rpm-ostree` is transactional and the previous
|
||
deployment is kept, so a bad upgrade is undone with `rpm-ostree rollback` (or
|
||
chosen from the GRUB menu if it will not boot). Unattended upgrades of a box
|
||
nobody looks at is its own risk — decide it deliberately, via
|
||
`rpm-ostreed-automatic.timer`.
|
||
* **SELinux is left enforcing, unmodified.** Nothing here needed a policy tweak;
|
||
treat a new AVC as a bug in this runbook.
|
||
* **No power cycle.** A hang is now recovered by the kernel and the chipset
|
||
watchdogs above. A machine that is *off* — dead PSU, tripped plug, firmware that
|
||
never reaches the bootloader — cannot reboot itself, and if the wall is remote
|
||
enough for that to matter it is still an ops decision (greenboot, or a smart
|
||
PDU), not a client one.
|
||
|
||
## Naming on the reference network
|
||
|
||
| Name | Address | What it is |
|
||
|---|---|---|
|
||
| `mirror.reeseapps.com` | 10.1.1.2 | the mirror service (nginx → container), valid TLS cert |
|
||
| `mirror-client.reeseapps.com` | 10.2.1.183 | the reference kiosk client |
|
||
| `mirror` | — | SSH alias for the kiosk client in `~/.ssh/config` |
|
||
|
||
The client's address is **DHCP**. A kiosk wants a reservation: the address is
|
||
what admits it to the service (`MIRROR_ALLOW_CIDRS`, nginx `allow`/`deny`), so a
|
||
renewal into a different range can lock a wall out by itself.
|