update pi.md with real pi agent config

Sync the documented models.json example with the actual
~/.pi/agent/models.json (maxTokens 32000 -> 32768) and add the
full ~/.pi/agent/settings.json to the doc.

LLM-Generated: true

active/device_raspberry_pi/systemd/README.md

@@ -0,0 +1,60 @@
+# Raspberry Pi — boot-time hardening units
+
+Two reusable runbooks for a headless Pi that must stay reachable over SSH.
+Deliberately host-agnostic: **no addresses, key material, or peer identities
+live in this repo** — those stay on each host. Set `HOST` and fill in the
+placeholders from the runbook before running anything.
+
+| Runbook | Solves |
+| --- | --- |
+| [`root-ssh-key.md`](root-ssh-key.md) | root `authorized_keys` gets wiped (imaging, config management) → key is re-installed at boot and re-applied on change |
+| [`wireguard.md`](wireguard.md) | SSH through a WireGuard tunnel address fails after reboot / NAT timeout / roaming to another network |
+
+Verified against Debian 13 (trixie) with systemd 257 and NetworkManager. The
+units are plain POSIX `sh` + systemd, so they carry to any other distribution
+with the same systemd version or newer.
+
+## Layout → target path on the host
+
+| File in this directory | Installs to |
+| --- | --- |
+| `ensure-root-ssh-key` | `/usr/local/sbin/ensure-root-ssh-key` (`0755`) |
+| `ensure-root-ssh-key.service` | `/etc/systemd/system/` |
+| `ensure-root-ssh-key.path` | `/etc/systemd/system/` |
+| `root-ssh-keys/authorized_keys.example` | `/usr/local/share/root-ssh-keys/authorized_keys` (`0644`) — copy, fill in your public key(s) |
+| `wg0-watchdog` | `/usr/local/sbin/wg0-watchdog` (`0755`) |
+| `wg0-watchdog.service` | `/etc/systemd/system/` |
+| `wg0-watchdog.timer` | `/etc/systemd/system/` |
+| `wg-quick@wg0.service.d-10-boot-resilience.conf` | `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` |
+| `wg0.conf.template` | documents `/etc/wireguard/wg0.conf` (`0600`) — the real file is never copied here |
+
+## Deploy pattern
+
+Every command in both runbooks follows the same shape, so a fresh Pi needs only
+`HOST` and the placeholders:
+
+```bash
+HOST=root@<pi-address>
+D=$PWD   # run the snippets from a checkout of this directory
+```
+
+Files travel over one `tar | ssh` pipe into `/root`, then `install` moves them
+into place with explicit modes (no `scp` of a whole tree into `/etc`, no
+ambiguous ownership). Reload and enable once at the end:
+
+```bash
+ssh "$HOST" 'systemctl daemon-reload && systemctl enable --now <units>'
+```
+
+## Conventions worth keeping on any host
+
+- Config files under `/usr/local/{sbin,share}` are the source of truth; systemd
+  units stay thin wrappers so they can be re-imaged independently.
+- Idempotent installers over clever state: every script re-derives the desired
+  state from its own config file and never deletes what it did not create.
+- Anything that must heal itself is gap-limited (a minimum interval between
+  restarts) so a broken dependency cannot make a unit flap.
+- Private keys and addresses are excluded from the repo; when dumping a
+  WireGuard config, redact with a regex that tolerates `PrivateKey = ` spacing
+  (`wg-quick strip` emits spaces, a `PrivateKey=` regex misses them):
+  `sed -E 's/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/'`

active/device_raspberry_pi/systemd/ensure-root-ssh-key

@@ -0,0 +1,61 @@
+#!/bin/sh
+# ensure-root-ssh-key - idempotently install managed SSH public keys into
+# /root/.ssh/authorized_keys. Runs at boot (ensure-root-ssh-key.service) and
+# can be run by hand at any time.
+#
+# Source of truth: /usr/local/share/root-ssh-keys/authorized_keys
+# One key per line, same format as an authorized_keys file. Blank lines and
+# lines starting with '#' are ignored. Keys already present are left alone;
+# foreign keys in authorized_keys are never removed.
+
+set -eu
+
+KEY_DIR="${KEY_DIR:-/root/.ssh}"
+AUTH_KEYS="$KEY_DIR/authorized_keys"
+MANAGED_KEYS="${MANAGED_KEYS:-/usr/local/share/root-ssh-keys/authorized_keys}"
+
+if [ ! -r "$MANAGED_KEYS" ]; then
+    echo "ensure-root-ssh-key: cannot read $MANAGED_KEYS" >&2
+    exit 1
+fi
+
+umask 077
+install -d -m 0700 "$KEY_DIR"
+[ -e "$AUTH_KEYS" ] || : > "$AUTH_KEYS"
+chmod 0600 "$AUTH_KEYS"
+chown root:root "$KEY_DIR" "$AUTH_KEYS"
+
+added=0
+checked=0
+while IFS= read -r line <&3; do
+    case "$line" in
+        '' | \#*) continue ;;
+    esac
+
+    # Basic shape check: <algo> <base64> [comment]
+    algo=${line%% *}
+    case "$algo" in
+        ssh-ed25519 | ssh-rsa | ssh-dss | ecdsa-sha2-* | sk-ecdsa-sha2-* | sk-ssh-ed25519-* | ssh-ed448) ;;
+        *)
+            echo "ensure-root-ssh-key: skipping unrecognised line: $line" >&2
+            continue
+            ;;
+    esac
+
+    checked=$((checked + 1))
+    if grep -qxF -- "$line" "$AUTH_KEYS" 2>/dev/null; then
+        continue
+    fi
+    printf '%s\n' "$line" >> "$AUTH_KEYS"
+    added=$((added + 1))
+    echo "ensure-root-ssh-key: installed key (${algo}, ${#line} chars)"
+done 3< "$MANAGED_KEYS"
+
+if [ "$added" -eq 0 ]; then
+    echo "ensure-root-ssh-key: $checked managed key(s) already present in $AUTH_KEYS"
+else
+    echo "ensure-root-ssh-key: added $added key(s), $checked managed key(s) total"
+fi
+
+chmod 0600 "$AUTH_KEYS"
+exit 0

active/device_raspberry_pi/systemd/ensure-root-ssh-key.path

@@ -0,0 +1,16 @@
+[Unit]
+Description=Watch /root/.ssh/authorized_keys and re-install the managed root SSH key
+Documentation=man:sshd(8)
+After=local-fs.target
+
+[Path]
+# PathChanged covers deletion / rename / creation of the file,
+# PathModified covers in-place edits (key stripped or file truncated).
+# ensure-root-ssh-key.service only writes when a managed key is missing, so
+# its own writes settle after one extra no-op run instead of looping.
+PathChanged=/root/.ssh/authorized_keys
+PathModified=/root/.ssh/authorized_keys
+Unit=ensure-root-ssh-key.service
+
+[Install]
+WantedBy=multi-user.target

active/device_raspberry_pi/systemd/ensure-root-ssh-key.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=Ensure managed SSH public keys are installed in /root/.ssh/authorized_keys
+Documentation=man:sshd(8)
+After=local-fs.target
+Before=ssh.service sshd.service
+ConditionPathExists=/usr/local/share/root-ssh-keys/authorized_keys
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/ensure-root-ssh-key
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target

active/device_raspberry_pi/systemd/root-ssh-key.md

@@ -0,0 +1,109 @@
+# Runbook: self-healing root SSH key
+
+`ensure-root-ssh-key.service` re-installs the managed SSH public key(s) into
+`/root/.ssh/authorized_keys` on every boot, and `ensure-root-ssh-key.path`
+re-applies it at runtime whenever that file is edited, deleted, or replaced.
+Useful when the key gets wiped by imaging, first-boot tooling, or config
+management — or when you need the key to come back after someone else edits the
+file.
+
+## Files on the host
+
+| Path | Purpose |
+| --- | --- |
+| `/etc/systemd/system/ensure-root-ssh-key.service` | `Type=oneshot`, `WantedBy=multi-user.target`, ordered `Before=ssh.service` so the key is in place before the daemon starts |
+| `/etc/systemd/system/ensure-root-ssh-key.path` | `PathChanged=` + `PathModified=` on `/root/.ssh/authorized_keys`, triggers the service |
+| `/usr/local/sbin/ensure-root-ssh-key` | Idempotent POSIX `sh` installer |
+| `/usr/local/share/root-ssh-keys/authorized_keys` | Source of truth — one public key per line, `#` comments allowed |
+
+## Behaviour
+
+- Appends any managed key that is missing; never removes or rewrites foreign keys.
+- Creates `/root/.ssh` (`0700`) and `authorized_keys` (`0600`, root:root) if absent.
+- Lines whose first field is not a known key algorithm are skipped with a warning.
+- Skipped, not failed, if the managed keys file is deleted.
+- Runtime self-healing: deleting, truncating, or renaming `authorized_keys` puts
+  the managed key(s) back within ~1s. That also means a managed key cannot be
+  revoked from the host alone — delete it from
+  `/usr/local/share/root-ssh-keys/authorized_keys` first.
+
+> Gotcha: systemd has no `PathMissing=` key — it is silently ignored with a
+> warning in `systemd-analyze verify`. `PathChanged=` is what fires on
+> delete/rename; `PathModified=` covers in-place appends and truncations. Set both.
+
+## Install
+
+```bash
+HOST=root@<pi-address>
+D=$PWD        # run the snippets from a checkout of this directory
+PUB=$HOME/.ssh/id_ed25519.pub          # the key that must always work
+
+ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /usr/local/share/root-ssh-keys'
+tar -C "$D" -cf - ensure-root-ssh-key ensure-root-ssh-key.service ensure-root-ssh-key.path | \
+ssh "$HOST" 'tar -xf - -C /root &&
+  install -m 0755 /root/ensure-root-ssh-key /usr/local/sbin/ &&
+  install -m 0644 /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path /etc/systemd/system/ &&
+  rm -f /root/ensure-root-ssh-key /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path &&
+  systemctl daemon-reload'
+cat "$PUB" | ssh "$HOST" 'install -m 0644 /dev/stdin /usr/local/share/root-ssh-keys/authorized_keys'
+ssh "$HOST" 'systemctl enable --now ensure-root-ssh-key.service ensure-root-ssh-key.path'
+```
+
+Use `root-ssh-keys/authorized_keys.example` instead of `$PUB` when the host
+should carry more than one key.
+
+## Verify
+
+One-shot smoke test that proves both units without touching a live session
+(existing SSH sessions are unaffected; only new logins during the window would
+fail):
+
+```bash
+ssh "$HOST" 'cp -a /root/.ssh/authorized_keys /root/.ssh/authorized_keys.testbak
+  rm -f /root/.ssh/authorized_keys
+  systemctl start ensure-root-ssh-key.service && cat /root/.ssh/authorized_keys
+  rm -f /root/.ssh/authorized_keys.testbak'
+```
+
+Status and history:
+
+```bash
+ssh "$HOST" 'systemctl is-enabled ensure-root-ssh-key.service ensure-root-ssh-key.path
+  systemctl status ensure-root-ssh-key.service --no-pager | head -8
+  journalctl -u ensure-root-ssh-key.service -b --no-pager | tail'
+```
+
+The installer honours `KEY_DIR` and `MANAGED_KEYS`, so the create-if-missing and
+preserve-foreign-keys paths can be exercised in a scratch directory instead of
+`/root/.ssh`:
+
+```bash
+ssh "$HOST" 'install -d -m 0700 /tmp/k/src && cp /usr/local/share/root-ssh-keys/authorized_keys /tmp/k/src/
+  KEY_DIR=/tmp/k/root MANAGED_KEYS=/tmp/k/src/authorized_keys /usr/local/sbin/ensure-root-ssh-key
+  ls -ld /tmp/k/root /tmp/k/root/authorized_keys; rm -rf /tmp/k'
+```
+
+Checklist that has passed on a reference host:
+
+- `systemd-analyze verify` clean for both units, both `enabled`.
+- Repeated runs add no duplicates; a wiped file and a deleted file are both restored.
+- Missing `/root/.ssh` is created `0700`, file `0600`.
+- Foreign keys survive; no trigger loop once the path unit settles.
+- Boot ordering: `Before=ssh.service` is honoured inside the boot transaction;
+  prove it after a reboot with `systemd-analyze critical-chain ssh.service`.
+
+## Manage keys
+
+```bash
+ssh "$HOST" 'echo "<public key line>" >> /usr/local/share/root-ssh-keys/authorized_keys
+  systemctl restart ensure-root-ssh-key.service'
+```
+
+## Uninstall
+
+```bash
+ssh "$HOST" 'systemctl disable --now ensure-root-ssh-key.path ensure-root-ssh-key.service
+  rm -f /etc/systemd/system/ensure-root-ssh-key.{service,path} /usr/local/sbin/ensure-root-ssh-key
+  rm -rf /usr/local/share/root-ssh-keys
+  systemctl daemon-reload'
+```

active/device_raspberry_pi/systemd/root-ssh-keys/authorized_keys.example

@@ -0,0 +1,11 @@
+# Template for /usr/local/share/root-ssh-keys/authorized_keys on the host.
+# Keys listed here are (re)installed into /root/.ssh/authorized_keys at boot and
+# whenever that file changes. Copy to `authorized_keys`, add your public keys,
+# and keep the real file on the host - do not commit it.
+#
+# One key per line, authorized_keys format. Blank lines and lines starting with
+# '#' are ignored; a line whose first field is not a known key algorithm is
+# skipped with a warning.
+#
+# ssh-ed25519 AAAA<base64> comment@host
+# ssh-rsa     AAAA<base64> comment@host

active/device_raspberry_pi/systemd/wg-quick@wg0.service.d-10-boot-resilience.conf

@@ -0,0 +1,17 @@
+# /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf
+# The stock unit already has After=/Wants=network-online.target nss-lookup.target.
+# This makes a boot-time failure (DHCP/DNS not ready yet, endpoint host offline)
+# retry forever instead of leaving the tunnel down until someone logs in.
+[Unit]
+# Unlimited retries: do not let the start rate limiter give up on us.
+StartLimitIntervalSec=0
+
+[Service]
+# Fail fast instead of hanging forever on endpoint resolution (stock value is
+# infinity), then let Restart= try again. Same variable as the stock unit,
+# later assignment wins.
+Environment=WG_ENDPOINT_RESOLUTION_RETRIES=60
+TimeoutStartSec=120
+TimeoutStopSec=30
+Restart=on-failure
+RestartSec=15

active/device_raspberry_pi/systemd/wg0-watchdog

@@ -0,0 +1,151 @@
+#!/bin/sh
+# wg0-watchdog - keep the WireGuard client tunnel up so SSH to the tunnel
+# address works after boot, after roaming to a new network, and after the NAT
+# mapping or endpoint IP goes stale.
+#
+# Decisions, oldest symptom first:
+#   interface or peer gone            -> restart wg-quick@wg0.service
+#   no PersistentKeepalive in kernel  -> set it (covers a config edited without reload)
+#   handshake older than STALE        -> re-point the peer at the endpoint
+#                                        (re-resolves the hostname, forces a handshake)
+#   handshake older than STALE_RESTART-> restart wg-quick@wg0.service
+#
+# Restarts are gap-limited so a broken network cannot make the unit flap.
+# Never removes peers, addresses, or routes it did not create.
+
+set -eu
+
+IFACE="${WG_WD_IFACE:-wg0}"
+UNIT="${WG_WD_UNIT:-wg-quick@wg0.service}"
+CONF="${WG_WD_CONF:-/etc/wireguard/$IFACE.conf}"
+KEEPALIVE="${WG_WD_KEEPALIVE:-25}"
+STALE="${WG_WD_STALE:-120}"              # seconds: poke the endpoint
+STALE_RESTART="${WG_WD_STALE_RESTART:-300}"   # seconds: restart the unit
+MIN_RESTART_GAP="${WG_WD_MIN_RESTART_GAP:-180}"  # seconds between restarts
+STATE_DIR="${WG_WD_STATE_DIR:-/var/lib/wg0-watchdog}"
+
+DRY=0
+[ "${1:-}" = "--dry-run" ] && DRY=1
+
+now=$(date +%s)
+log() { printf 'wg0-watchdog: %s\n' "$*"; }
+run() {
+    if [ "$DRY" -eq 1 ]; then
+        printf 'wg0-watchdog: DRY-RUN would run:'
+        printf ' %s' "$@"
+        printf '\n'
+    else
+        "$@"
+    fi
+}
+
+read_state() { cat "$1" 2>/dev/null || echo 0; }
+write_state() {
+    if [ "$DRY" -eq 1 ]; then
+        return 0
+    fi
+    mkdir -p "$STATE_DIR"
+    printf '%s\n' "$2" > "$1"
+}
+
+restart_unit() {
+    reason=$1
+    last=$(read_state "$STATE_DIR/last-restart")
+    case "$last" in ''|*[!0-9]*) last=0 ;; esac
+    since=$((now - last))
+    if [ "$since" -lt "$MIN_RESTART_GAP" ]; then
+        log "restart of $UNIT suppressed ($reason); last restart ${since}s ago"
+        return 0
+    fi
+    log "restarting $UNIT ($reason)"
+    write_state "$STATE_DIR/last-restart" "$now"
+    # --no-block: a wg-quick up stuck on endpoint resolution must not wedge the
+    # watchdog; the next run reports whether the restart worked.
+    run systemctl restart --no-block "$UNIT"
+}
+
+# --- 1. interface / peer present -------------------------------------------
+if ! ip link show "$IFACE" >/dev/null 2>&1; then
+    restart_unit "interface $IFACE is missing"
+    exit 0
+fi
+
+peer=$(wg show "$IFACE" peers 2>/dev/null | head -n 1)
+if [ -z "${peer:-}" ]; then
+    restart_unit "no peer configured on $IFACE"
+    exit 0
+fi
+
+# --- 2. persistent keepalive present ---------------------------------------
+live_ka=$(wg show "$IFACE" persistent-keepalive 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
+[ -n "${live_ka:-}" ] || live_ka=0
+if [ "$live_ka" != "$KEEPALIVE" ]; then
+    log "persistent-keepalive is $live_ka, setting $KEEPALIVE"
+    run wg set "$IFACE" peer "$peer" persistent-keepalive "$KEEPALIVE"
+fi
+
+# --- 3. handshake age -------------------------------------------------------
+hs=$(wg show "$IFACE" latest-handshakes 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
+case "${hs:-}" in ''|*[!0-9]*) hs=0 ;; esac
+if [ "$hs" -eq 0 ]; then
+    age=999999
+else
+    age=$((now - hs))
+fi
+
+# endpoint currently in use by the kernel, and the cached last-known-good one
+live_ep=$(wg show "$IFACE" endpoints 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
+cache_file="$STATE_DIR/endpoint"
+cached_ep=""
+[ -f "$cache_file" ] && cached_ep=$(cat "$cache_file" 2>/dev/null || true)
+
+if [ "$age" -lt "$STALE" ]; then
+    if [ -n "$live_ep" ]; then
+        write_state "$cache_file" "$live_ep"
+    fi
+    log "healthy: handshake ${age}s ago, endpoint ${live_ep:-unknown}, keepalive ${KEEPALIVE}s"
+    exit 0
+fi
+
+log "stale: last handshake ${age}s ago"
+
+# Prefer the endpoint from the config file, freshly resolved. Fall back to the
+# kernel's current endpoint, then to the cache - important because a DNS server
+# from the tunnel config may be reachable only through this very tunnel.
+target_ep=""
+conf_ep=$(awk -F= 'tolower($1) ~ /^[[:space:]]*endpoint[[:space:]]*$/ { gsub(/[[:space:]]/, "", $2); print $2 }' "$CONF" 2>/dev/null | tail -n 1)
+if [ -n "${conf_ep:-}" ]; then
+    host=${conf_ep%:*}
+    port=${conf_ep##*:}
+    case "$host" in
+        *[!0-9.]*)
+            resolved=$(getent hosts "$host" 2>/dev/null | awk '{print $1; exit}' || true)
+            ;;
+        *)
+            resolved=$host
+            ;;
+    esac
+    if [ -n "${resolved:-}" ]; then
+        target_ep="$resolved:$port"
+    else
+        log "could not resolve $host (tunnel DNS?), falling back"
+    fi
+fi
+[ -n "$target_ep" ] || target_ep=${live_ep:-${cached_ep:-}}
+
+if [ "$age" -ge "$STALE_RESTART" ]; then
+    restart_unit "handshake ${age}s old"
+    exit 0
+fi
+
+if [ -z "$target_ep" ]; then
+    log "no endpoint to re-point at; will retry next run"
+    exit 0
+fi
+
+if [ "$target_ep" = "$live_ep" ]; then
+    log "endpoint already $target_ep; forcing a handshake"
+fi
+log "re-pointing peer at $target_ep"
+run wg set "$IFACE" peer "$peer" endpoint "$target_ep"
+exit 0

active/device_raspberry_pi/systemd/wg0-watchdog.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Keep the WireGuard wg0 tunnel reachable for SSH
+Documentation=man:wg-quick(8)
+
+[Service]
+Type=oneshot
+# Never let a hung systemctl restart inside the script wedge the watchdog.
+TimeoutStartSec=120
+ExecStart=/usr/local/sbin/wg0-watchdog
+StandardOutput=journal
+StandardError=journal
+Nice=10

active/device_raspberry_pi/systemd/wg0-watchdog.timer

@@ -0,0 +1,14 @@
+[Unit]
+Description=Check the WireGuard wg0 tunnel every minute and heal it if needed
+
+[Timer]
+# First check shortly after boot, then every minute. Persistent=true catches up
+# if the machine was suspended/offline.
+OnBootSec=30s
+OnUnitActiveSec=60s
+AccuracySec=15s
+RandomizedDelaySec=5s
+Persistent=true
+
+[Install]
+WantedBy=timers.target

active/device_raspberry_pi/systemd/wg0.conf.template

@@ -0,0 +1,20 @@
+# wg0.conf template — /etc/wireguard/wg0.conf on the host, mode 0600 root:root.
+# Never commit a real copy: this file holds the tunnel private key.
+# Generate the key with `umask 077; wg genkey | tee privatekey | wg pubkey`.
+
+[Interface]
+PrivateKey = <tunnel private key>
+Address = <TUNNEL_ADDR>
+# Deliberately no `DNS` key: any resolver listed here is pushed as a system
+# upstream while the tunnel is up, which makes all name resolution depend on the
+# tunnel. Omitting it keeps the LAN's DHCP resolvers. Debian's wg-quick has no
+# `DNS = none` - a non-IP value becomes a search domain.
+
+[Peer]
+PublicKey = <PEER_PUBKEY>
+AllowedIPs = <PEER_NETWORK>
+Endpoint = <WG_SERVER_HOST:PORT>
+# Required when the client is behind NAT and must be reachable inbound (SSH):
+# without it the mapping expires and the server cannot initiate. 25s survives
+# typical 30-60s NAT UDP timeouts.
+PersistentKeepalive = 25

active/device_raspberry_pi/systemd/wireguard.md

@@ -0,0 +1,162 @@
+# Runbook: WireGuard client that survives reboot, NAT, and roaming
+
+For a Pi whose SSH address is a **WireGuard tunnel address** rather than a LAN
+address: the physical interface is ordinary DHCP and the Pi moves between
+networks, while the tunnel peer is a reachable server. Reachability then depends
+on three separate things, each of which fails independently — the unit starting
+at boot, the NAT mapping staying open, and the peer hostname still resolving.
+
+Placeholders used below:
+
+| Placeholder | Meaning |
+| --- | --- |
+| `$HOST` | `root@<address you can already reach>` |
+| `<TUNNEL_ADDR>` | this Pi's tunnel address, e.g. `<addr>/32` |
+| `<PEER_NETWORK>` | network routed through the tunnel, e.g. `<net>/<cidr>` |
+| `<WG_SERVER_HOST:PORT>` | tunnel server endpoint |
+| `<PEER_PUBKEY>` | the server's WireGuard public key |
+
+## Files on the host
+
+| Path | Purpose |
+| --- | --- |
+| `/etc/wireguard/wg0.conf` | tunnel config, mode `0600` — see `wg0.conf.template`, never stored in this repo |
+| `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` | retry a failed boot-time `up`: `Restart=on-failure`, `RestartSec=15`, `StartLimitIntervalSec=0`, `WG_ENDPOINT_RESOLUTION_RETRIES=60`, `TimeoutStartSec=120` |
+| `/usr/local/sbin/wg0-watchdog` | heals the tunnel once a minute |
+| `/etc/systemd/system/wg0-watchdog.service` | oneshot that runs the script |
+| `/etc/systemd/system/wg0-watchdog.timer` | `OnBootSec=30s`, `OnUnitActiveSec=60s`, `Persistent=true` |
+| `/var/lib/wg0-watchdog/endpoint` | last-known-good endpoint IP, used when DNS is unavailable |
+
+## Failure modes this closes
+
+1. **No `PersistentKeepalive`.** Behind NAT the server can only send to the Pi
+   while the mapping lives, so once it expires the tunnel looks configured but
+   inbound SSH times out. `PersistentKeepalive = 25` punches UDP state often
+   enough for typical 30-60s NAT timeouts.
+2. **A boot-time failure is permanent.** `wg-quick up` fails outright if the
+   endpoint hostname does not resolve yet or the server is down, and the unit is
+   not retried — so the tunnel stays dead until someone logs in locally.
+   `Restart=on-failure` with `StartLimitIntervalSec=0` retries forever;
+   `WG_ENDPOINT_RESOLUTION_RETRIES` is lowered from the stock `infinity`
+   (which hangs the unit indefinitely) so a retry can actually happen.
+3. **Nothing heals a stale tunnel.** Roaming to a new network keeps the old
+   endpoint IP in the kernel, and a peer whose endpoint is unreachable simply
+   stops handshaking without the unit ever noticing, because the unit already
+   "succeeded". The watchdog acts on handshake age.
+
+## Install
+
+```bash
+HOST=root@<pi-address>
+D=$PWD        # run the snippets from a checkout of this directory
+
+ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /etc/systemd/system/wg-quick@wg0.service.d'
+tar -C "$D" -cf - wg0-watchdog wg0-watchdog.service wg0-watchdog.timer \
+    'wg-quick@wg0.service.d-10-boot-resilience.conf' | \
+ssh "$HOST" 'tar -xf - -C /root &&
+  install -m 0755 /root/wg0-watchdog /usr/local/sbin/ &&
+  install -m 0644 /root/wg0-watchdog.service /root/wg0-watchdog.timer /etc/systemd/system/ &&
+  install -m 0644 "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" \
+    /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf &&
+  rm -f /root/wg0-watchdog /root/wg0-watchdog.service /root/wg0-watchdog.timer \
+    "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" &&
+  systemctl daemon-reload && systemctl enable --now wg-quick@wg0.service wg0-watchdog.timer'
+```
+
+Keep `PersistentKeepalive = 25` in `[Peer]` (see `wg0.conf.template`) and check
+the config file is `0600` — a world-readable `wg0.conf` exposes the tunnel
+private key to every local account:
+
+```bash
+ssh "$HOST" 'ls -l /etc/wireguard/wg0.conf; wg show wg0 persistent-keepalive'
+```
+
+## Watchdog behaviour
+
+Oldest symptom first, one action per run:
+
+1. interface or peer gone → `systemctl restart --no-block wg-quick@wg0.service`
+2. no `PersistentKeepalive` in the kernel → `wg set ... persistent-keepalive <n>`
+   (covers a config edited without a reload)
+3. handshake older than `WG_WD_STALE` (default 120s) → re-point the peer at the
+   endpoint, preferring a fresh resolution of the hostname from the config file,
+   then the kernel's current endpoint, then the cached IP
+4. handshake older than `WG_WD_STALE_RESTART` (default 300s) → restart the unit
+
+Restarts are gap-limited to one per `WG_WD_MIN_RESTART_GAP` (default 180s) so a
+dead network cannot make the unit flap. `--no-block` keeps a hung `wg-quick up`
+from wedging the watchdog. Peers, addresses, and routes are only ever added.
+All thresholds and the interface/state paths are environment overrides, and
+`--dry-run` prints the decision without acting.
+
+The endpoint resolution order matters when the tunnel config also pushes a DNS
+server: that resolver is reachable only through the tunnel, so a dead tunnel can
+mean the endpoint hostname cannot be resolved exactly when it is needed — hence
+the cached-IP fallback.
+
+## DNS
+
+A `DNS = <resolver>` line in the interface section is pushed through
+`resolvconf`/`systemd-resolved` while the tunnel is up, which can silently make
+*all* name resolution on the host depend on the tunnel. If the Pi must resolve
+names on whatever network it lands on, omit the `DNS` key entirely and let the
+LAN's DHCP resolvers stand.
+
+Two gotchas:
+
+- Debian's `wg-quick` does **not** implement `DNS = none` — any non-IP value
+  becomes a search domain, so the line has to be absent, not set to `none`.
+- A record already pushed by a previous `wg-quick up` survives the config edit;
+  withdraw it (`resolvconf -d <record>`, name visible in `resolvconf --list`) and
+  confirm the local caching resolver's upstreams before believing it is gone.
+
+## Taking wg0 down on purpose
+
+Stop the watchdog first or the tunnel comes straight back:
+
+```bash
+ssh "$HOST" 'systemctl stop wg0-watchdog.timer wg-quick@wg0.service'
+ssh "$HOST" 'systemctl start wg0-watchdog.timer'      # resume
+```
+
+## Check on it
+
+```bash
+ssh "$HOST" 'wg show wg0
+  systemctl status wg-quick@wg0 --no-pager | head -6
+  systemctl list-timers "wg0-watchdog*" --no-pager | head -3
+  journalctl -u wg0-watchdog.service -b --no-pager | tail -15'
+```
+
+Dry-run the decision logic without touching the tunnel:
+
+```bash
+ssh "$HOST" 'WG_WD_STATE_DIR=/tmp/wd WG_WD_STALE=0 /usr/local/sbin/wg0-watchdog --dry-run; rm -rf /tmp/wd'
+```
+
+Validate an edited config without disturbing the running tunnel:
+
+```bash
+ssh "$HOST" 'wg-quick strip /etc/wireguard/wg0.conf | sed -E "s/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/"'
+```
+
+## Diagnosis notes (generic, worth checking on any host)
+
+- `wg-quick@<iface>` can be `enabled` yet `inactive` while the interface exists
+  and passes traffic — it may have been created by hand or taken over by
+  NetworkManager as an assumed profile (`nmcli dev status` shows such a device
+  as `connected (externally)`). Nothing owns it: `systemctl reload` fails with
+  "unit not active", so apply changes with `wg set` or a full restart, and let
+  `systemctl restart` take ownership before trusting a reboot.
+- `Restart=on-failure` does work for `Type=oneshot` on systemd 257 — worth
+  proving on a new distribution with a throwaway unit under
+  `/run/systemd/system` before relying on it: `Type=oneshot` +
+  `ExecStart=/bin/false`, then check `systemctl show -p NRestarts`.
+- `systemd-analyze verify <unit>` catches mistyped unit keys, and it only warns —
+  read the output instead of trusting the exit status.
+- NetworkManager does not fight `wg-quick` as long as no keyfile exists for the
+  interface and its assumed profile keeps `autoconnect no`; check
+  `/etc/NetworkManager/system-connections/` before assuming that.
+- The real end-to-end test is a reboot: DHCP → endpoint resolution →
+  `wg-quick up` → inbound SSH through the tunnel. `systemd-analyze
+  critical-chain ssh.service` after boot shows whether the ordering held.

active/os_fedora/sudoers.d/ops

@@ -0,0 +1,59 @@
+# This is the sudoers file for the ops user to give limited access to specific commands without a password
+# Used for automations like ansible and maintenance
+
+# Hostname
+ops ALL=(root) NOPASSWD: /usr/bin/hostnamectl hostname *
+
+# Reboot
+ops ALL=(root) NOPASSWD: /usr/bin/reboot
+
+# System updates
+ops ALL=(root) NOPASSWD: /usr/bin/dnf update*, /usr/bin/dnf upgrade*, /usr/bin/dnf install *, /usr/bin/dnf remove *
+
+# Package cache management
+ops ALL=(root) NOPASSWD: /usr/bin/dnf clean all, /usr/bin/dnf makecache, /usr/bin/dnf autoremove
+
+# Repos
+ops ALL=(root) NOPASSWD: /usr/bin/dnf config-manager addrepo *
+
+# Service management
+ops ALL=(root) NOPASSWD: /usr/bin/systemctl start *, /usr/bin/systemctl restart *, /usr/bin/systemctl reload *, /usr/bin/systemctl enable *
+
+# Firewall
+ops ALL=(root) NOPASSWD: /usr/bin/firewall-cmd *
+
+# Ansible
+ops ALL=(root) NOPASSWD: /bin/sh -c echo BECOME-SUCCESS-* ; /usr/bin/python3 /home/ops/.ansible*
+
+# qemu-img
+ops ALL=(root) NOPASSWD: /usr/bin/qemu-img *
+
+# virsh
+ops ALL=(root) NOPASSWD: /usr/bin/virsh *
+
+# virt-install
+ops ALL=(root) NOPASSWD: /usr/bin/virt-install *
+
+# composer-cli
+ops ALL=(root) NOPASSWD: /usr/bin/composer-cli *
+
+# osbuild
+ops ALL=(root) NOPASSWD: /usr/local/bin/osbuild-auto.sh
+
+# getfacl
+ops ALL=(root) NOPASSWD: /usr/sbin/getfacl*
+
+# set default rwx sbin
+ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -d -m u\:ops\:rwx /usr/local/sbin
+
+# set existing rwx sbin
+ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -m u\:ops\:rwx /usr/local/sbin
+
+# set default rwx /srv
+ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -d -m u\:ops\:rwx /srv/*
+
+# set existing rwx /srv
+ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -m u\:ops\:rwx /srv/*
+
+# Start btrfs scrub
+ops ALL=(root) NOPASSWD: /usr/bin/btrfs scrub*

active/software_pi/pi.md

@@ -36,7 +36,7 @@ This is my ~/.pi/agent/models.json
           "input": ["text", "image"],
           "reasoning": true,
           "contextWindow": 262144,
-          "maxTokens": 32000,
+          "maxTokens": 32768,
           "thinkingLevelMap": {
             "off": "none",
             "minimal": null,
@@ -52,7 +52,7 @@ This is my ~/.pi/agent/models.json
           "input": ["text", "image"],
           "reasoning": true,
           "contextWindow": 262144,
-          "maxTokens": 32000,
+          "maxTokens": 32768,
            "thinkingLevelMap": {
             "off": "none",
             "minimal": null,
@@ -68,3 +68,72 @@ This is my ~/.pi/agent/models.json
   }
 }
 ```
+
+This is my ~/.pi/agent/settings.json
+
+```json
+{
+  "lastChangelogVersion": "1.3.0",
+  "theme": "dark/dark",
+  "defaultProvider": "reeseapps-aipi",
+  "defaultModel": "turbo",
+  "httpIdleTimeoutMs": 0,
+  "enableInstallTelemetry": false,
+  "doubleEscapeAction": "tree",
+  "images": {
+    "autoResize": false
+  },
+  "editorPaddingX": 0,
+  "defaultThinkingLevel": "xhigh",
+  "hideThinkingBlock": false,
+  "thinkingBudgets": {
+    "minimal": 1024,
+    "low": 4096,
+    "medium": 10240,
+    "xhigh": 65536
+  },
+  "steeringMode": "all"
+}
+```
+
+## Installing for sysadmin work
+
+I would absolutely recommend installing pi under a special user (I call mine
+`ops`) with limited sudoers access that you tightly control. Pair that with the
+following AGENTS.md at `/home/ops/AGENTS.md`:
+
+```text
+# AGENTS.md
+
+## Sudo policy
+
+The `ops` user on this machine has **limited sudoers capabilities**. The exact
+allowlist is discoverable at any time with:
+
+sudo -l
+
+Allowed (NOPASSWD) commands currently include: `hostnamectl hostname`, `reboot`,
+`dnf update/upgrade/install/remove/clean/makecache/autoremove`,
+`dnf config-manager addrepo`, `systemctl start/restart/reload/enable`,
+`firewall-cmd`, the ansible shim, `qemu-img`, `virsh`, `virt-install`,
+`composer-cli`, `getfacl`, specific `setfacl` invocations,
+`/usr/local/sbin/osbuild-auto.sh`, and `btrfs scrub`.
+
+## Rule for agents
+
+If you (the agent) believe you need a permission that is **not** in the
+`sudo -l` allowlist — i.e., a command fails with a sudo permission error, or a
+task requires root access you do not have — you must:
+
+1. **Stop** and do not attempt workarounds that bypass sudo (no password
+   guessing, no escalating via other users, no trying to write to root-owned
+   paths without permission).
+2. **Ask the human** to update sudoers with the specific command(s) you need,
+   stating exactly which command(s) and arguments should be added so the user
+   can review and approve them. Requests must be correct and copy-pastable,
+   using full command paths in standard sudoers format, e.g.
+   `ops ALL=(root) NOPASSWD: /usr/sbin/btrfs scrub*`
+
+Do not proceed with root-only operations until the user has confirmed the
+sudoers entries have been added.
+```

active/software_vllm_radiance/vllm_radiance.md

@@ -0,0 +1,281 @@
+# vLLM Radiance on R9700
+
+- [vLLM Radiance on R9700](#vllm-radiance-on-r9700)
+  - [Overview](#overview)
+  - [Deployment Topology](#deployment-topology)
+  - [Capacity Analysis (Single 32 GiB R9700)](#capacity-analysis-single-32-gib-r9700)
+  - [FP8-KV Calibration: The Process](#fp8-kv-calibration-the-process)
+    - [Why: The Uncalibrated Scale Problem](#why-the-uncalibrated-scale-problem)
+    - [Step 1: Host Prep](#step-1-host-prep)
+    - [Step 2: Stop the Service](#step-2-stop-the-service)
+    - [Step 3: Run the Calibration](#step-3-run-the-calibration)
+    - [Step 4: Verify the Artifact (CPU-only)](#step-4-verify-the-artifact-cpu-only)
+    - [Step 5: Edit the Quadlet](#step-5-edit-the-quadlet)
+    - [Step 6: Deploy via Ansible](#step-6-deploy-via-ansible)
+    - [Step 7: Confirm at Runtime](#step-7-confirm-at-runtime)
+  - [What Calibration Actually Does (Benefits)](#what-calibration-actually-does-benefits)
+  - [Operational Notes](#operational-notes)
+  - [Remaining Work: Matched-Control Quality Gate](#remaining-work-matched-control-quality-gate)
+
+## Overview
+
+`vllm-radiance` is a qualified downstream vLLM distribution for the AMD Radeon AI
+PRO R9700 (gfx1201). The upstream-qualified environment is dual-GPU TP2; this
+deployment runs the single-GPU (TP1) configuration on `driveripper-ai` serving
+the AMD Quark MXFP4 27B checkpoint (W4A8: 4-bit weights, dynamic FP8
+activation quantization over RDNA4's native FP8 WMMA path).
+
+Served as `juggernaut` on the `ai-internal` network, port 8000.
+
+## Deployment Topology
+
+| Thing              | Location                                                                                                                                                                                   |
+| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Quadlet (source of |
+| truth)             | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/quadlets/vllm-radiance.container`                                                                                                    |
+| Ansible playbook   | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/playbook.yaml` (run from `~/Deployments`: `ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml`) |
+| Deployed quadlet   | `driveripper-ai:/home/ai/.config/containers/systemd/ai-vllm-radiance.container`                                                                                                            |
+| systemd unit       | **user** unit of `ai` (`systemctl --user status ai-vllm-radiance` — not in system scope)                                                                                                   |
+| Image              | `docker.io/magiccodingman/vllm-radiance:1.0.387`                                                                                                                                           |
+| Model              | `driveripper-ai:/home/ai/vllm-radiance-models/qwen3.8-27b-quark-awq-mxfp4` → `/models` (18.44 GiB on disk, 17.95 GiB in GPU)                                                               |
+| Compile caches     | `driveripper-ai:/home/ai/vllm-radiance-cache` → `/cache`                                                                                                                                   |
+| FP8-KV sidecar     | `driveripper-ai:/home/ai/vllm-radiance-calibration` → `/calibration` (ro)                                                                                                                  |
+
+The playbook copies the quadlet to `~/.config/containers/systemd/` and does
+`systemctl --user restart ai-vllm-radiance` with daemon-reload. **Any edit to
+the local quadlet lands on the next playbook run** — there is no separate
+"apply" step.
+
+Key quadlet settings (as of the calibration deploy, 2026-10-11):
+
+```
+--tensor-parallel-size=1  --max-model-len=154000  --max-num-seqs=2
+--max-num-batched-tokens=4096  --kv-cache-dtype=fp8  --attention-backend=R4D
+--enable-prefix-caching  --mamba-cache-mode=align  --no-async-scheduling
+--gpu-memory-utilization=0.98
+Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
+Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
+HealthStartPeriod=600s  (health check kills on 3x10s failures after the start period)
+```
+
+## Capacity Analysis (Single 32 GiB R9700)
+
+Measured from the server startup logs (the authoritative numbers — do not
+re-derive from weight sizes alone; the GDN hybrid architecture makes
+per-token KV much cheaper than a plain transformer):
+
+```
+Model loading took 17.95 GiB
+Actual usage: 19.38 GiB consumed (weights + non-torch)
+             + 2.02 GiB peak activation
+             + 0.41 GiB CUDA graphs
+Available KV cache memory: 9.82 GiB  (10.14 GiB at the 154K config)
+GPU KV cache size: 308,847 tokens   (309,466 at the 154K config)
+```
+
+Budget math (KV budget ≈ 309K tokens):
+
+| Config                | KV needed | Fits?                                        |
+| --------------------- | --------- | -------------------------------------------- |
+| 150K × 2              | 300,000   | yes, ~3% headroom                            |
+| **154K × 2 (chosen)** | 308,000   | yes — `2.01x` per vLLM, ~1.5K token headroom |
+| 256K × 1              | 262,144   | yes                                          |
+| 256K × 2              | 524,288   | **no** — `1.18x` max concurrency             |
+
+The per-request C2 ceiling is `KV_budget / 2 ≈ 154,423` tokens; `--max-model-len=154000`
+sits exactly at it. A second full 256K request is only reachable via CPU KV
+offload (`RADIANCE_KV_OFFLOADING_SIZE`), which would need to roughly double the
+token budget. `MAX_NUM_SEQS` is an admission ceiling, not a promise that every
+admitted request can simultaneously reach `MAX_MODEL_LEN`.
+
+## FP8-KV Calibration: The Process
+
+### Why: The Uncalibrated Scale Problem
+
+The server runs `--kv-cache-dtype=fp8` with FP8 attention, but the checkpoint
+carries no q/k/v/prob scales. vLLM therefore used placeholder scale **1.0**,
+which the startup log flags:
+
+```
+WARNING ... Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8
+attention. This may cause accuracy issues.
+```
+
+Radiance ships `radiance_kv_calibration.py` (in the image, also in the
+`vllm-radiance` repo) plus the `RADIANCE_FP8_KV_SCALES` runtime loader to fix
+this with a small immutable sidecar. Full reference: `docs/FP8_KV_TUNABLEOP.md`
+in the vllm-radiance repo.
+
+### Step 1: Host Prep
+
+The calibration dir did not exist and the vllm-radiance repo is not on
+`driveripper-ai`, so copy only the corpus:
+
+```bash
+ssh driveripper-ai "mkdir -p /home/ai/vllm-radiance-calibration"
+scp ~/Projects/Experimental/vllm-radiance/benchmarks/fixtures/fp8-kv-calibration.jsonl \
+    driveripper-ai:/home/ai/vllm-radiance-calibration/
+```
+
+(Optionally extend the JSONL with representative traffic first — each line is
+`{"text": ...}` or `{"messages": [...], "tools": [...]}` plus optional `repeat`.)
+
+### Step 2: Stop the Service
+
+Calibration needs the GPU released; the server holds ~97% of VRAM.
+
+```bash
+ssh driveripper-ai "systemctl --user stop ai-vllm-radiance"
+```
+
+### Step 3: Run the Calibration
+
+An eager, BF16 full-precision-KV, `max_tokens=1` prefill sweep with PyTorch
+hooks on every full-attention layer (no source edits — public worker-extension
+API only). Budget ~10–30 min:
+
+```bash
+ssh driveripper-ai "podman run --rm --ipc=host \
+  --device=/dev/kfd --device=/dev/dri \
+  --ulimit memlock=-1:-1 \
+  -e HIP_VISIBLE_DEVICES=0 \
+  -e RADIANCE_MXFP4=1 -e RADIANCE_MXFP4_W4A8=1 -e RADIANCE_MXFP4_W4A8_MIN_M=0 \
+  -v /home/ai/vllm-radiance-models:/models:ro \
+  -v /home/ai/vllm-radiance-calibration:/calibration:rw \
+  -v /home/ai/vllm-radiance-calibration/fp8-kv-calibration.jsonl:/fixtures/fp8-kv-calibration.jsonl:ro \
+  --entrypoint python \
+  docker.io/magiccodingman/vllm-radiance:1.0.387 \
+  -m radiance_kv_calibration calibrate \
+  --model /models/qwen3.8-27b-quark-awq-mxfp4 \
+  --quantization auto \
+  --corpus /fixtures/fp8-kv-calibration.jsonl \
+  --output /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
+  --tensor-parallel-size 1 \
+  --max-model-len 16384 \
+  --batch-size 4 \
+  --seed 17"
+```
+
+Success line: `[radiance] wrote immutable FP8-KV sidecar: /calibration/...`.
+Outputs are **immutable** — a re-run needs a fresh filename. The sidecar and
+its sibling `.manifest.json` (sha256 + full source-checkpoint identity +
+per-layer observed amax + corpus/seed/software provenance) land in the host dir.
+
+### Step 4: Verify the Artifact (CPU-only)
+
+No engine start — checks manifest schema, sidecar sha256, that every tensor is
+a positive finite scalar `*.q|k|v|prob_scale`, and (with `--model`) that it was
+calibrated against this exact checkpoint:
+
+```bash
+ssh driveripper-ai "podman run --rm \
+  -v /home/ai/vllm-radiance-models:/models:ro \
+  -v /home/ai/vllm-radiance-calibration:/calibration:ro \
+  --entrypoint python docker.io/magiccodingman/vllm-radiance:1.0.387 \
+  -m radiance_kv_calibration verify \
+  /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
+  --model /models/qwen3.8-27b-quark-awq-mxfp4"
+```
+
+### Step 5: Edit the Quadlet
+
+Three lines in the local quadlet (source of truth), then nothing else:
+
+```diff
+ Volume=/home/ai/vllm-radiance-models:/models:ro,z
+ Volume=/home/ai/vllm-radiance-cache:/cache:z
++Volume=/home/ai/vllm-radiance-calibration:/calibration:ro,z
+ ...
+ Environment=WEIGHT_QUANTIZATION=auto
++Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
++Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
+```
+
+### Step 6: Deploy via Ansible
+
+```bash
+cd ~/Deployments
+ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml
+```
+
+### Step 7: Confirm at Runtime
+
+All three must be present/absent:
+
+```
+FP8-KV scales = /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors   (banner; was "checkpoint/default")
+Loading Radiance FP8-KV calibration sidecar ... (verify=True)          (checksum + identity passed at load)
+"uncalibrated" warning: gone
+/health: OK
+```
+
+## What Calibration Actually Does (Benefits)
+
+**The problem.** With placeholder scale 1.0, FP8 values are stored as
+`value / 1.0`. The calibration hooks measured the model's real pre-quantization
+activations: peak **23.0 (K, layer 31)** and **59.75 (V, layer 43)** across the
+16 full-attention layers. FP8 E4M3's max magnitude is 448 — so the raw values
+only used **5–13% of the representable range**. E4M3 has just 3 mantissa bits;
+when values cluster near the bottom of the range, the quantization steps
+between representable values are ~8–20× coarser than necessary. That is rounding
+noise injected into every attention score and KV lookup — worst for
+long-context retrieval, where KV detail is exactly what the task depends on.
+
+**The fix.** The calibration writes per-layer
+`scale = observed_amax × 1.05 / 448` (5% headroom for excursions the corpus
+didn't hit) and `prob_scale = 1.05 / 448` (softmax probabilities are bounded by
+1). At load time vLLM divides each tensor by that scale before FP8 conversion,
+so observed maxima land at ~95% of the FP8 range — the full 3 mantissa bits do
+useful work. Result: **~7.5× finer quantization steps on V, ~19.6× finer on K**
+(≈ +2.9 / +4.3 effective mantissa bits), and the "may cause accuracy issues"
+warning is resolved.
+
+**Provenance & safety.** The sidecar is immutable (sha256 in the manifest),
+bound to this exact checkpoint, and re-verified at every startup by
+`VERIFY=1` — a changed model makes the server **fail closed** instead of
+silently using wrong scales.
+
+**What it does NOT do.** No throughput change (expected neutral — a big TPS
+delta means something is wrong, not a win). No new kernels. No weight-path
+change. Purely a precision fix for the FP8 attention/KV arithmetic.
+
+## Operational Notes
+
+- **Sidecar is per-checkpoint.** Any fine-tune or re-download changes
+  activation ranges — re-run calibration (fresh filename) rather than reusing
+  scales by architecture name, and never set `VERIFY=0` as a deployment
+  setting (it exists only to diagnose a deliberate mismatch).
+- **Startup re-hash cost.** The model dir on `driveripper-ai` has no HF
+  download metadata, so the manifest bound full weight sha256s — every startup
+  re-hashes all 18.44 GiB of weights for identity verification. Adds a few
+  minutes on cold restarts; worked fine in practice.
+- **Health check.** `HealthStartPeriod=600s` was chosen deliberately: the
+  deployed (pre-calibration) quadlet had the health check disabled at 30s start
+  period — a cold kernel-compile startup would fail 3×10s of health checks and
+  `HealthOnFailure=kill` + `Restart=always` loops it.
+- **Capacity ceiling.** 154K×2 runs the KV tier to ~99.6% at full context by
+  design. Normal mixed traffic stays well below. If you need two full 256K
+  concurrent requests, that is a CPU-offload decision, not a flag tweak.
+- **Corpus.** The 2026-10-10 sidecar used the stock 8-prompt corpus (tool
+  schema, code, 4.9K-token retrieval, reasoning, multilingual, math, structured
+  data). If the real workload is dominated by content the stock corpus doesn't
+  cover, re-calibrate with an extended corpus. Calibration was run at
+  16K `max_model_len`; amax is largely context-independent, but if long-context
+  numerics ever look off, that's the first thing to re-run with a longer
+  `--max-model-len`.
+
+## Remaining Work: Matched-Control Quality Gate
+
+Per `docs/FP8_KV_TUNABLEOP.md` and the project's no-claim-without-control
+policy: compare **calibrated vs default-scale** runs with identical image,
+target, prompts, seeds, sampling, graphs, and concurrency:
+
+- fixed-prompt greedy outputs (meaningful agreement)
+- sampled tool-schema checks
+- long-context retrieval
+- vision inputs if vision is deployed
+- BetterBench / sustained c1/c2/c4/c8 for the throughput-neutral claim
+
+Success criterion: **fidelity at-or-above the uncalibrated baseline with ~neutral
+throughput.** Only after that gate is this a qualified change, not just a
+plausible one.