update pi.md with real pi agent config

Sync the documented models.json example with the actual ~/.pi/agent/models.json (maxTokens 32000 -> 32768) and add the full ~/.pi/agent/settings.json to the doc. LLM-Generated: true
document ops-user pi install and sudoers escalation policy
2026-10-21 00:00:07 -04:00 · 2026-10-17 12:26:40 -04:00 · 2026-10-16 09:08:16 -04:00 · 2026-10-16 09:07:56 -04:00 · 2026-10-16 09:05:54 -04:00 · 2026-10-12 10:31:43 -04:00
99 changed files with 2814 additions and 1437 deletions
@@ -42,6 +42,7 @@ or give me access to your servers.
    - [bash](#bash)
    - [Bulk File/Folder Renaming](#bulk-filefolder-renaming)
    - [Escaping a Stuck SSH Terminal](#escaping-a-stuck-ssh-terminal)
    - [Curl output hidden by bash prompt](#curl-output-hidden-by-bash-prompt)
  - [SSH Setup](#ssh-setup)
  - [Git GPG Commit Signing](#git-gpg-commit-signing)
  - [Important Dates and Times](#important-dates-and-times)
@@ -133,6 +134,14 @@ done
 Press the following keys: enter + ~ + .
 ### Curl output hidden by bash prompt
 Add this to your ~/.curlrc
 ```conf
 -w "\n"
 ```
 ## SSH Setup
 Generate a key (password protect it!)
@@ -213,12 +222,13 @@ signed you'll see an output. If not, nothing will show.
 ## Important Dates and Times
 | Time  | Day | Description                    |
-| ----- | --- | ---------------------------------- |
+| ----- | --- | ------------------------------ |
 | 00:00 | All | Automated builds               |
 | 00:00 | All | NAS Snapshots                  |
-| 02:00 | All | Backups                            |
+| 01:00 | All | Backups                        |
-| 04:00 | All | Bare Metal Server Security Updates |
+| 03:00 | All | App Updates                    |
-| 05:00 | All | VM Server Security Updates         |
+| 04:00 | All | Bare Metal Server Updates      |
 | 05:00 | All | VM Server Updates              |
 | 05:00 | All | Unifi Protect Firmware Updates |
 | 06:00 | All | Unifi Network Firmware Updates |
@@ -0,0 +1,14 @@
 # AWS Bedrock
 ## Endpoints
 Completions endpoint: <https://bedrock-mantle.us-east-1.api.aws/v1>
 ## Model Analysis
 | Model ID                | Input Price (Mil) | Output Price (Mil) | Doodle Jump Notes                                                                                                                                   |
 | ----------------------- | ----------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- |
 | minimax.minimax-m2.5    | $ 0.30            | $ 1.20             | Way too fast and player moves too slow. Identified issue and fixed after second attempt                                                             |
 | zai.glm-5               | $ 1.00            | $ 3.20             | Perfect game with great graphics.                                                                                                                   |
 | mistral.devstral-2-123b | $ 0.40            | $ 2.00             | Broken game with very basic graphics. Second attempt broken as well.                                                                                |
 | deepseek.v3.2           | $1.35             | $5.40              | Way too fast, platform generation erratic, mobile controls were a nice touch. Fixed speed issue on second attempt but platform generation still bad |
@@ -1,48 +0,0 @@
 services:
  gitea:
    image: docker.gitea.com/gitea:1.25-rootless
    container_name: gitea
    environment:
      - GITEA__database__DB_TYPE=postgres
      - GITEA__database__HOST=postgres:5432
      - GITEA__database__NAME=gitea
      - GITEA__database__USER=gitea
      - GITEA__database__PASSWD=gitea
    security_opt:
      - "label=disable"
    restart: always
    networks:
      - gitea
    volumes:
      - /home/gitea/gitea_data:/data:Z
      - /home/gitea/gitea_etc:/etc/gitea:Z
      - /home/gitea/gitea_custom:/var/lib/gitea/custom:Z
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "2222:2222"
    depends_on:
      - postgres
    labels:
      - "io.containers.autoupdate=registry"
  postgres:
    image: docker.io/library/postgres:15
    container_name: postgres
    security_opt:
      - "label=disable"
    restart: always
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=gitea
      - POSTGRES_DB=gitea
    networks:
      - gitea
    volumes:
      - /home/gitea/gitea_postgres:/var/lib/postgresql/data:Z
    labels:
      - "io.containers.autoupdate=registry"
 networks:
  gitea:
    enable_ipv6: true
@@ -4,7 +4,7 @@ networks:
 services:
  server:
-    image: docker.gitea.com/gitea:1.25.3
+    image: docker.gitea.com/gitea:1.26.1
    container_name: gitea
    environment:
      - USER_UID=1001
@@ -1,22 +0,0 @@
 [Unit]
 Requires=postgres.service
 [Container]
 AutoUpdate=registry
 ContainerName=gitea
 Environment=GITEA__database__DB_TYPE=postgres GITEA__database__HOST=postgres:5432 GITEA__database__NAME=gitea GITEA__database__USER=gitea GITEA__database__PASSWD=gitea
 Image=docker.gitea.com/gitea:1.25-rootless
 Network=gitea.network
 PublishPort=3000:3000
 PublishPort=2222:2222
 SecurityLabelDisable=true
 Volume=/home/gitea/gitea_data:/data:Z
 Volume=/home/gitea/gitea_etc:/etc/gitea:Z
 Volume=/home/gitea/gitea_custom:/var/lib/gitea/custom:Z
 Volume=/etc/localtime:/etc/localtime:ro
 [Service]
 Restart=always
 [Install]
 WantedBy=default.target
@@ -1,14 +0,0 @@
 [Container]
 AutoUpdate=registry
 ContainerName=postgres
 Environment=POSTGRES_USER=gitea POSTGRES_PASSWORD=gitea POSTGRES_DB=gitea
 Image=docker.io/library/postgres:15
 Network=gitea.network
 SecurityLabelDisable=true
 Volume=/home/gitea/gitea_postgres:/var/lib/postgresql/data:Z
 [Service]
 Restart=always
 [Install]
 WantedBy=default.target
@@ -10,7 +10,7 @@ DB_DATA_LOCATION=/srv/immich-db/postgres
 # TZ=Etc/UTC
 # The Immich version to use. You can pin this to a specific version like "v2.1.0"
-IMMICH_VERSION=v2
+IMMICH_VERSION=v3
 # Connection secret for postgres. You should change it to a random password
 # Please use only the characters `A-Za-z0-9`, without special characters or spaces
@@ -1,19 +0,0 @@
 services:
  keycloak:
    container_name: keycloak
    restart: always
    image: quay.io/keycloak/keycloak:26.4.5
    ports:
      - "9443:443"
      - "8443:8443"
    volumes:
      - /home/foobar/data:/var/app/data
    security_opt:
      - label=disable
    userns_mode: keep-id
    command: 
    - "start"
    - "--hostname"
    - "https://keycloak.reeseapps.com"
    - "--hostname-admin"
    - "https://keycloak.reeselink.com:8443"
@@ -1,186 +1,10 @@
 # Podman keycloak
 - [Podman keycloak](#podman-keycloak)
-  - [Setup keycloak Project](#setup-keycloak-project)
+  - [Adding a Client](#adding-a-client)
  - [Install Keycloak with Docker](#install-keycloak-with-docker)
  - [Install Keycloak with Podman](#install-keycloak-with-podman)
    - [Create the keycloak user](#create-the-keycloak-user)
    - [Write the keycloak compose spec](#write-the-keycloak-compose-spec)
      - [A Note on Volumes](#a-note-on-volumes)
    - [Convert keycloak compose spec to quadlets](#convert-keycloak-compose-spec-to-quadlets)
    - [Create any container-mounted directories](#create-any-container-mounted-directories)
    - [Start and enable your systemd quadlet](#start-and-enable-your-systemd-quadlet)
    - [Expose keycloak](#expose-keycloak)
      - [firewalld](#firewalld)
  - [Backup keycloak](#backup-keycloak)
  - [Upgrade keycloak](#upgrade-keycloak)
    - [Upgrade Quadlets](#upgrade-quadlets)
  - [Uninstall](#uninstall)
  - [Notes](#notes)
    - [SELinux](#selinux)
-## Setup keycloak Project
+## Adding a Client
- [ ] Copy and rename this folder to active/container_keycloak
+1. Create a new client
- [ ] Find and replace keycloak with the name of the service.
+2. Clients -> click on client -> Client scopes -> client-frontend-dedicated -> Scope -> turn off Full Scope Allowed
- [ ] Create the rootless user to run the podman containers
+3. 
 - [ ] Write the compose.yaml spec for your service
 - [ ] Convert the compose.yaml spec to a quadlet
 - [ ] Install the quadlet on the podman server
 - [ ] Expose the quadlet service
 - [ ] Install a backup service and timer
 ## Install Keycloak with Docker
 <https://www.keycloak.org/getting-started/getting-started-docker>
 ```bash
 # Test in dev mode
 docker run -p 8080:8080 -e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:26.4.7 start-dev
 ```
 ## Install Keycloak with Podman
 ### Create the keycloak user
 ```bash
 # SSH into your podman server as root
 useradd keycloak
 loginctl enable-linger $(id -u keycloak)
 systemctl --user --machine=keycloak@.host enable podman-restart
 systemctl --user --machine=keycloak@.host enable --now podman.socket
 mkdir -p /home/keycloak/.config/containers/systemd
 ```
 ### Write the keycloak compose spec
 <https://www.keycloak.org/getting-started/getting-started-podman>
 ```bash
 # Based on the example
 podman run \
 -p 127.0.0.1:8080:8080 \
 -e KC_BOOTSTRAP_ADMIN_USERNAME=admin \
 -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
 quay.io/keycloak/keycloak:26.4.5 start-dev
 ```
 #### A Note on Volumes
 Named volumes are stored at `/home/keycloak/.local/share/containers/storage/volumes/`.
 ### Convert keycloak compose spec to quadlets
 Run the following to convert a compose.yaml into the various `.container` files for systemd:
 ```bash
 # Generate the systemd service
 podman run \
 --security-opt label=disable \
 --rm \
 -v $(pwd)/active/container_keycloak/:/compose \
 -v $(pwd)/active/container_keycloak/quadlets:/quadlets \
 quay.io/k9withabone/podlet \
 -f /quadlets \
 -i \
 --overwrite \
 compose /compose/compose.yaml
 # Copy the files to the server
 export PODMAN_SERVER=
 scp -r active/container_keycloak/quadlets/. $PODMAN_SERVER:/home/keycloak/.config/containers/systemd/
 ssh $PODMAN_SERVER chown -R keycloak:keycloak /home/keycloak/.config/containers/systemd/
 ```
 ### Create any container-mounted directories
 SSH into your podman server as root:
 ```bash
 machinectl shell keycloak@
 podman unshare
 mkdir some_volume
 # Chown to the namespaced user with UID 1000
 # This will be some really obscure UID outside the namespace
 # This will also solve most permission denied errors
 chown -R 1000:1000 some_volume
 ```
 ### Start and enable your systemd quadlet
 SSH into your podman server as root:
 ```bash
 machinectl shell keycloak@
 systemctl --user daemon-reload
 systemctl --user restart keycloak
 # Enable auto-update service which will pull new container images automatically every day
 systemctl --user enable --now podman-auto-update.timer
 ```
 ### Expose keycloak
 1. If you need a domain, follow the [DDNS instructions](/active/container_ddns/ddns.md#install-a-new-ddns-service)
 2. For a web service, follow the [Caddy instructions](/active/container_caddy/caddy.md#adding-a-new-caddy-record)
 3. Finally, follow your OS's guide for opening ports via its firewall service.
 #### firewalld
 ```bash
 # command to get current active zone and default zone
 firewall-cmd --get-active-zones
 firewall-cmd --get-default-zone
 # command to open 443 on tcp
 firewall-cmd --permanent --zone=<zone> --add-port=443/tcp
 # command to open 80 and 443 on tcp and udp
 firewall-cmd --permanent --zone=<zone> --add-port={80,443}/{tcp,udp}
 # command to list available services and then open http and https
 firewall-cmd --get-services
 firewall-cmd --permanent --zone=<zone> --add-service={http,https}
 ```
 ## Backup keycloak
 Follow the [Borg Backup instructions](/active/systemd_borg/borg.md#set-up-a-client-for-backup)
 ## Upgrade keycloak
 ### Upgrade Quadlets
 Upgrades should be a repeat of [writing the compose spec](#convert-keycloak-compose-spec-to-quadlets) and [installing the quadlets](#start-and-enable-your-systemd-quadlet)
 ```bash
 export PODMAN_SERVER=
 scp -r quadlets/. $PODMAN_SERVER$:/home/keycloak/.config/containers/systemd/
 ssh keycloak systemctl --user daemon-reload
 ssh keycloak systemctl --user restart keycloak
 ```
 ## Uninstall
 ```bash
 # Stop the user's services
 systemctl --user disable podman-restart
 podman container stop --all
 systemctl --user disable --now podman.socket
 systemctl --user disable --now podman-auto-update.timer
 # Delete the user (this won't delete their home directory)
 # userdel might spit out an error like:
 # userdel: user keycloak is currently used by process 591255
 # kill those processes and try again
 userdel keycloak
 ```
 ## Notes
 ### SELinux
 <https://blog.christophersmart.com/2021/01/31/podman-volumes-and-selinux/>
 :z allows a container to share a mounted volume with all other containers.
 :Z allows a container to reserve a mounted volume and prevents any other container from accessing.
@@ -1,3 +0,0 @@
 # Quadlets
 Put your quadlets here.
@@ -0,0 +1,168 @@
 # Make sure to update the credential placeholders with your own secrets.
 # We mark them with # CHANGEME in the file below.
 # In addition, we recommend to restrict inbound traffic on the host to langfuse-web (port 3000) and minio (port 9090) only.
 # All other components are bound to localhost (127.0.0.1) to only accept connections from the local machine.
 # External connections from other machines will not be able to reach these services directly.
 services:
  langfuse-worker:
    image: docker.io/langfuse/langfuse-worker:3
    restart: always
    ports:
      - 127.0.0.1:3030:3030
    environment: &langfuse-worker-env
      NEXTAUTH_URL: ${NEXTAUTH_URL:-http://localhost:3000}
      DATABASE_URL: ${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/postgres} # CHANGEME
      SALT: ${SALT:-mysalt} # CHANGEME
      ENCRYPTION_KEY: ${ENCRYPTION_KEY:-0000000000000000000000000000000000000000000000000000000000000000} # CHANGEME: generate via `openssl rand -hex 32`
      TELEMETRY_ENABLED: ${TELEMETRY_ENABLED:-true}
      LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: ${LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES:-false}
      CLICKHOUSE_MIGRATION_URL: ${CLICKHOUSE_MIGRATION_URL:-clickhouse://clickhouse:9000}
      CLICKHOUSE_URL: ${CLICKHOUSE_URL:-http://clickhouse:8123}
      CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEME
      CLICKHOUSE_CLUSTER_ENABLED: ${CLICKHOUSE_CLUSTER_ENABLED:-false}
      LANGFUSE_USE_AZURE_BLOB: ${LANGFUSE_USE_AZURE_BLOB:-false}
      LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE: ${LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE:-false}
      LANGFUSE_OCI_AUTH_TYPE: ${LANGFUSE_OCI_AUTH_TYPE:-workload_identity}
      LANGFUSE_S3_EVENT_UPLOAD_BUCKET: ${LANGFUSE_S3_EVENT_UPLOAD_BUCKET:-langfuse}
      LANGFUSE_S3_EVENT_UPLOAD_REGION: ${LANGFUSE_S3_EVENT_UPLOAD_REGION:-auto}
      LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID:-minio}
      LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME
      LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: ${LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT:-http://minio:9000}
      LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE:-true}
      LANGFUSE_S3_EVENT_UPLOAD_PREFIX: ${LANGFUSE_S3_EVENT_UPLOAD_PREFIX:-events/}
      LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: ${LANGFUSE_S3_MEDIA_UPLOAD_BUCKET:-langfuse}
      LANGFUSE_S3_MEDIA_UPLOAD_REGION: ${LANGFUSE_S3_MEDIA_UPLOAD_REGION:-auto}
      LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID:-minio}
      LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME
      LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: ${LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT:-http://localhost:9090}
      LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE:-true}
      LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: ${LANGFUSE_S3_MEDIA_UPLOAD_PREFIX:-media/}
      LANGFUSE_S3_BATCH_EXPORT_ENABLED: ${LANGFUSE_S3_BATCH_EXPORT_ENABLED:-false}
      LANGFUSE_S3_BATCH_EXPORT_BUCKET: ${LANGFUSE_S3_BATCH_EXPORT_BUCKET:-langfuse}
      LANGFUSE_S3_BATCH_EXPORT_PREFIX: ${LANGFUSE_S3_BATCH_EXPORT_PREFIX:-exports/}
      LANGFUSE_S3_BATCH_EXPORT_REGION: ${LANGFUSE_S3_BATCH_EXPORT_REGION:-auto}
      LANGFUSE_S3_BATCH_EXPORT_ENDPOINT: ${LANGFUSE_S3_BATCH_EXPORT_ENDPOINT:-http://minio:9000}
      LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT: ${LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT:-http://localhost:9090}
      LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID: ${LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID:-minio}
      LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY: ${LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME
      LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE: ${LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE:-true}
      LANGFUSE_INGESTION_QUEUE_DELAY_MS: ${LANGFUSE_INGESTION_QUEUE_DELAY_MS:-}
      LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS: ${LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS:-}
      REDIS_HOST: ${REDIS_HOST:-redis}
      REDIS_PORT: ${REDIS_PORT:-6379}
      REDIS_AUTH: ${REDIS_AUTH:-myredissecret} # CHANGEME
      REDIS_TLS_ENABLED: ${REDIS_TLS_ENABLED:-false}
      REDIS_TLS_CA: ${REDIS_TLS_CA:-/certs/ca.crt}
      REDIS_TLS_CERT: ${REDIS_TLS_CERT:-/certs/redis.crt}
      REDIS_TLS_KEY: ${REDIS_TLS_KEY:-/certs/redis.key}
      EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS:-}
      SMTP_CONNECTION_URL: ${SMTP_CONNECTION_URL:-}
  langfuse-web:
    image: docker.io/langfuse/langfuse:3
    restart: always
    ports:
      - 3000:3000
    environment:
      <<: *langfuse-worker-env
      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET:-mysecret} # CHANGEME
      LANGFUSE_INIT_ORG_ID: ${LANGFUSE_INIT_ORG_ID:-}
      LANGFUSE_INIT_ORG_NAME: ${LANGFUSE_INIT_ORG_NAME:-}
      LANGFUSE_INIT_PROJECT_ID: ${LANGFUSE_INIT_PROJECT_ID:-}
      LANGFUSE_INIT_PROJECT_NAME: ${LANGFUSE_INIT_PROJECT_NAME:-}
      LANGFUSE_INIT_PROJECT_PUBLIC_KEY: ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY:-}
      LANGFUSE_INIT_PROJECT_SECRET_KEY: ${LANGFUSE_INIT_PROJECT_SECRET_KEY:-}
      LANGFUSE_INIT_USER_EMAIL: ${LANGFUSE_INIT_USER_EMAIL:-admin@ducoterra.net}
      LANGFUSE_INIT_USER_NAME: ${LANGFUSE_INIT_USER_NAME:-admin}
      LANGFUSE_INIT_USER_PASSWORD: ${LANGFUSE_INIT_USER_PASSWORD:-testpass}
  clickhouse:
    image: docker.io/clickhouse/clickhouse-server
    restart: always
    user: "101:101"
    environment:
      CLICKHOUSE_DB: default
      CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}
      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEME
    volumes:
      - langfuse_clickhouse_data:/var/lib/clickhouse
      - langfuse_clickhouse_logs:/var/log/clickhouse-server
    ports:
      - 127.0.0.1:8123:8123
      - 127.0.0.1:9000:9000
    healthcheck:
      test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
      interval: 5s
      timeout: 5s
      retries: 10
      start_period: 1s
  minio:
    image: cgr.dev/chainguard/minio
    restart: always
    entrypoint: sh
    # create the 'langfuse' bucket before starting the service
    command: -c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data'
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minio}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-miniosecret} # CHANGEME
    ports:
      - 9090:9000
      - 127.0.0.1:9091:9001
    volumes:
      - langfuse_minio_data:/data
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 1s
      timeout: 5s
      retries: 5
      start_period: 1s
  redis:
    image: docker.io/redis:7
    restart: always
    # CHANGEME: row below to secure redis password
    command: >
      --requirepass ${REDIS_AUTH:-myredissecret}
      --maxmemory-policy noeviction
    ports:
      - 127.0.0.1:6379:6379
    volumes:
      - langfuse_redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 3s
      timeout: 10s
      retries: 10
  postgres:
    image: docker.io/postgres:${POSTGRES_VERSION:-17}
    restart: always
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 3s
      timeout: 3s
      retries: 10
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-postgres}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres} # CHANGEME
      POSTGRES_DB: ${POSTGRES_DB:-postgres}
      TZ: UTC
      PGTZ: UTC
    ports:
      - 127.0.0.1:5432:5432
    volumes:
      - langfuse_postgres_data:/var/lib/postgresql/data
 volumes:
  langfuse_postgres_data:
    driver: local
  langfuse_clickhouse_data:
    driver: local
  langfuse_clickhouse_logs:
    driver: local
  langfuse_minio_data:
    driver: local
  langfuse_redis_data:
    driver: local
@@ -0,0 +1,167 @@
 # Podman langfuse
 - [Podman langfuse](#podman-langfuse)
  - [Setup langfuse Project](#setup-langfuse-project)
  - [Install langfuse](#install-langfuse)
    - [Create the langfuse user](#create-the-langfuse-user)
    - [Write the langfuse compose spec](#write-the-langfuse-compose-spec)
      - [A Note on Volumes](#a-note-on-volumes)
    - [Convert langfuse compose spec to quadlets](#convert-langfuse-compose-spec-to-quadlets)
    - [Create any container-mounted directories](#create-any-container-mounted-directories)
    - [Start and enable your systemd quadlet](#start-and-enable-your-systemd-quadlet)
    - [Expose langfuse](#expose-langfuse)
      - [firewalld](#firewalld)
  - [Backup langfuse](#backup-langfuse)
  - [Upgrade langfuse](#upgrade-langfuse)
    - [Upgrade Quadlets](#upgrade-quadlets)
  - [Uninstall](#uninstall)
  - [Notes](#notes)
    - [SELinux](#selinux)
 ## Setup langfuse Project
 - [ ] Copy and rename this folder to active/container_langfuse
 - [ ] Find and replace langfuse with the name of the service.
 - [ ] Create the rootless user to run the podman containers
 - [ ] Write the compose.yaml spec for your service
 - [ ] Convert the compose.yaml spec to a quadlet
 - [ ] Install the quadlet on the podman server
 - [ ] Expose the quadlet service
 - [ ] Install a backup service and timer
 ## Install langfuse
 ### Create the langfuse user
 ```bash
 # SSH into your podman server as root
 useradd langfuse
 loginctl enable-linger $(id -u langfuse)
 systemctl --user --machine=langfuse@.host enable podman-restart
 systemctl --user --machine=langfuse@.host enable --now podman.socket
 mkdir -p /home/langfuse/.config/containers/systemd
 ```
 ### Write the langfuse compose spec
 Edit the compose.yaml at active/container_langfuse/compose/compose.yaml
 #### A Note on Volumes
 Named volumes are stored at `/home/langfuse/.local/share/containers/storage/volumes/`.
 ### Convert langfuse compose spec to quadlets
 Run the following to convert a compose.yaml into the various `.container` files for systemd:
 ```bash
 # Generate the systemd service
 podman run \
 --security-opt label=disable \
 --rm \
 -v $(pwd)/active/container_langfuse/compose:/compose \
 -v $(pwd)/active/container_langfuse/quadlets:/quadlets \
 quay.io/k9withabone/podlet \
 -f /quadlets \
 -i \
 --overwrite \
 compose /compose/compose.yaml
 # Copy the files to the server
 export PODMAN_SERVER=
 scp -r active/container_langfuse/quadlets/. $PODMAN_SERVER:/home/langfuse/.config/containers/systemd/
 ssh $PODMAN_SERVER chown -R langfuse:langfuse /home/langfuse/.config/containers/systemd/
 ```
 ### Create any container-mounted directories
 SSH into your podman server as root:
 ```bash
 machinectl shell langfuse@
 podman unshare
 mkdir some_volume
 # Chown to the namespaced user with UID 1000
 # This will be some really obscure UID outside the namespace
 # This will also solve most permission denied errors
 chown -R 1000:1000 some_volume
 ```
 ### Start and enable your systemd quadlet
 SSH into your podman server as root:
 ```bash
 machinectl shell langfuse@
 systemctl --user daemon-reload
 systemctl --user restart langfuse
 # Enable auto-update service which will pull new container images automatically every day
 systemctl --user enable --now podman-auto-update.timer
 ```
 ### Expose langfuse
 1. If you need a domain, follow the [DDNS instructions](/active/container_ddns/ddns.md#install-a-new-ddns-service)
 2. For a web service, follow the [Caddy instructions](/active/container_caddy/caddy.md#adding-a-new-caddy-record)
 3. Finally, follow your OS's guide for opening ports via its firewall service.
 #### firewalld
 ```bash
 # command to get current active zone and default zone
 firewall-cmd --get-active-zones
 firewall-cmd --get-default-zone
 # command to open 443 on tcp
 firewall-cmd --permanent --zone=<zone> --add-port=443/tcp
 # command to open 80 and 443 on tcp and udp
 firewall-cmd --permanent --zone=<zone> --add-port={80,443}/{tcp,udp}
 # command to list available services and then open http and https
 firewall-cmd --get-services
 firewall-cmd --permanent --zone=<zone> --add-service={http,https}
 ```
 ## Backup langfuse
 Follow the [Borg Backup instructions](/active/systemd_borg/borg.md#set-up-a-client-for-backup)
 ## Upgrade langfuse
 ### Upgrade Quadlets
 Upgrades should be a repeat of [writing the compose spec](#convert-langfuse-compose-spec-to-quadlets) and [installing the quadlets](#start-and-enable-your-systemd-quadlet)
 ```bash
 export PODMAN_SERVER=
 scp -r quadlets/. $PODMAN_SERVER$:/home/langfuse/.config/containers/systemd/
 ssh langfuse systemctl --user daemon-reload
 ssh langfuse systemctl --user restart langfuse
 ```
 ## Uninstall
 ```bash
 # Stop the user's services
 systemctl --user disable podman-restart
 podman container stop --all
 systemctl --user disable --now podman.socket
 systemctl --user disable --now podman-auto-update.timer
 # Delete the user (this won't delete their home directory)
 # userdel might spit out an error like:
 # userdel: user langfuse is currently used by process 591255
 # kill those processes and try again
 userdel langfuse
 ```
 ## Notes
 ### SELinux
 <https://blog.christophersmart.com/2021/01/31/podman-volumes-and-selinux/>
 :z allows a container to share a mounted volume with all other containers.
 :Z allows a container to reserve a mounted volume and prevents any other container from accessing.
@@ -0,0 +1,21 @@
 [Container]
 Pod=langfuse.pod
 ContainerName=langfuse-clickhouse
 Environment=CLICKHOUSE_DB=default
 Environment=CLICKHOUSE_USER=clickhouse
 Environment=CLICKHOUSE_PASSWORD=clickhouse
 User=101:101
 Image=docker.io/clickhouse/clickhouse-server
 Volume=langfuse-clickhouse_data.volume:/var/lib/clickhouse
 Volume=langfuse-clickhouse_logs.volume:/var/log/clickhouse-server
 HealthCmd=wget --no-verbose --tries=1 --spider http://localhost:8123/ping
 HealthInterval=5s
 HealthTimeout=5s
 HealthRetries=10
 HealthStartPeriod=1s
 [Service]
 Restart=always
 [Install]
 WantedBy=default.target
@@ -0,0 +1,5 @@
 [Volume]
 VolumeName=langfuse_clickhouse_data
 [Install]
 WantedBy=default.target
@@ -0,0 +1,5 @@
 [Volume]
 VolumeName=langfuse_clickhouse_logs
 [Install]
 WantedBy=default.target
@@ -0,0 +1,20 @@
 [Container]
 Pod=langfuse.pod
 ContainerName=langfuse-minio
 Environment=MINIO_ROOT_USER=minio
 Environment=MINIO_ROOT_PASSWORD=miniosecret
 Image=cgr.dev/chainguard/minio
 Volume=langfuse-minio_data.volume:/data
 Entrypoint=sh
 Exec=-c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data'
 HealthCmd=mc ready local
 HealthInterval=1s
 HealthTimeout=5s
 HealthRetries=5
 HealthStartPeriod=1s
 [Service]
 Restart=always
 [Install]
 WantedBy=default.target
@@ -0,0 +1,5 @@
 [Volume]
 VolumeName=langfuse_minio_data
 [Install]
 WantedBy=default.target
@@ -0,0 +1,20 @@
 [Container]
 Pod=langfuse.pod
 ContainerName=langfuse-postgres
 Environment=POSTGRES_USER=postgres
 Environment=POSTGRES_PASSWORD=postgres
 Environment=POSTGRES_DB=postgres
 Environment=TZ=UTC
 Environment=PGTZ=UTC
 Image=docker.io/postgres:17
 Volume=langfuse-postgres_data.volume:/var/lib/postgresql/data
 HealthCmd=pg_isready -U postgres
 HealthInterval=3s
 HealthTimeout=3s
 HealthRetries=10
 [Service]
 Restart=always
 [Install]
 WantedBy=default.target
@@ -0,0 +1,5 @@
 [Volume]
 VolumeName=langfuse_postgres_data
 [Install]
 WantedBy=default.target
@@ -0,0 +1,17 @@
 [Container]
 Pod=langfuse.pod
 ContainerName=langfuse-redis
 Environment=REDIS_AUTH=myredissecret
 Image=docker.io/redis:7
 Volume=langfuse-redis_data.volume:/data
 Exec=--requirepass myredissecret --maxmemory-policy noeviction
 HealthCmd=redis-cli ping
 HealthInterval=3s
 HealthTimeout=10s
 HealthRetries=10
 [Service]
 Restart=always
 [Install]
 WantedBy=default.target
@@ -0,0 +1,5 @@
 [Volume]
 VolumeName=langfuse_redis_data
 [Install]
 WantedBy=default.target
@@ -0,0 +1,71 @@
 [Unit]
 Requires=langfuse-postgres.service langfuse-clickhouse.service langfuse-redis.service langfuse-minio.service
 [Container]
 Pod=langfuse.pod
 ContainerName=langfuse-web
 Environment=NEXTAUTH_URL=http://localhost:3000
 Environment=DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
 Environment=SALT=mysalt
 Environment=ENCRYPTION_KEY=0000000000000000000000000000000000000000000000000000000000000000
 Environment=TELEMETRY_ENABLED=true
 Environment=LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES=false
 Environment=CLICKHOUSE_MIGRATION_URL=clickhouse://localhost:9000
 Environment=CLICKHOUSE_URL=http://localhost:8123
 Environment=CLICKHOUSE_USER=clickhouse
 Environment=CLICKHOUSE_PASSWORD=clickhouse
 Environment=CLICKHOUSE_CLUSTER_ENABLED=false
 Environment=LANGFUSE_USE_AZURE_BLOB=false
 Environment=LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=false
 Environment=LANGFUSE_OCI_AUTH_TYPE=workload_identity
 Environment=LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse
 Environment=LANGFUSE_S3_EVENT_UPLOAD_REGION=auto
 Environment=LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=minio
 Environment=LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=miniosecret
 Environment=LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=http://localhost:9000
 Environment=LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true
 Environment=LANGFUSE_S3_EVENT_UPLOAD_PREFIX=events/
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_BUCKET=langfuse
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_REGION=auto
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID=minio
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY=miniosecret
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT=http://localhost:9090
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE=true
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_PREFIX=media/
 Environment=LANGFUSE_S3_BATCH_EXPORT_ENABLED=false
 Environment=LANGFUSE_S3_BATCH_EXPORT_BUCKET=langfuse
 Environment=LANGFUSE_S3_BATCH_EXPORT_PREFIX=exports/
 Environment=LANGFUSE_S3_BATCH_EXPORT_REGION=auto
 Environment=LANGFUSE_S3_BATCH_EXPORT_ENDPOINT=http://localhost:9000
 Environment=LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT=http://localhost:9090
 Environment=LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID=minio
 Environment=LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY=miniosecret
 Environment=LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE=true
 Environment=LANGFUSE_INGESTION_QUEUE_DELAY_MS=
 Environment=LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=
 Environment=REDIS_HOST=localhost
 Environment=REDIS_PORT=6379
 Environment=REDIS_AUTH=myredissecret
 Environment=REDIS_TLS_ENABLED=false
 Environment=REDIS_TLS_CA=/certs/ca.crt
 Environment=REDIS_TLS_CERT=/certs/redis.crt
 Environment=REDIS_TLS_KEY=/certs/redis.key
 Environment=EMAIL_FROM_ADDRESS=
 Environment=SMTP_CONNECTION_URL=
 Environment=NEXTAUTH_SECRET=mysecret
 Environment=LANGFUSE_INIT_ORG_ID=
 Environment=LANGFUSE_INIT_ORG_NAME=
 Environment=LANGFUSE_INIT_PROJECT_ID=
 Environment=LANGFUSE_INIT_PROJECT_NAME=
 Environment=LANGFUSE_INIT_PROJECT_PUBLIC_KEY=
 Environment=LANGFUSE_INIT_PROJECT_SECRET_KEY=
 Environment=LANGFUSE_INIT_USER_EMAIL=admin@ducoterra.net
 Environment=LANGFUSE_INIT_USER_NAME=admin
 Environment=LANGFUSE_INIT_USER_PASSWORD=testpass
 Image=docker.io/langfuse/langfuse:3
 [Service]
 Restart=always
 [Install]
 WantedBy=default.target
@@ -0,0 +1,61 @@
 [Unit]
 Requires=langfuse-postgres.service langfuse-clickhouse.service langfuse-redis.service langfuse-minio.service
 [Container]
 Pod=langfuse.pod
 ContainerName=langfuse-worker
 Environment=NEXTAUTH_URL=http://localhost:3000
 Environment=DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
 Environment=SALT=mysalt
 Environment=ENCRYPTION_KEY=0000000000000000000000000000000000000000000000000000000000000000
 Environment=TELEMETRY_ENABLED=true
 Environment=LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES=false
 Environment=CLICKHOUSE_MIGRATION_URL=clickhouse://localhost:9000
 Environment=CLICKHOUSE_URL=http://localhost:8123
 Environment=CLICKHOUSE_USER=clickhouse
 Environment=CLICKHOUSE_PASSWORD=clickhouse
 Environment=CLICKHOUSE_CLUSTER_ENABLED=false
 Environment=LANGFUSE_USE_AZURE_BLOB=false
 Environment=LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=false
 Environment=LANGFUSE_OCI_AUTH_TYPE=workload_identity
 Environment=LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse
 Environment=LANGFUSE_S3_EVENT_UPLOAD_REGION=auto
 Environment=LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=minio
 Environment=LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=miniosecret
 Environment=LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=http://localhost:9000
 Environment=LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true
 Environment=LANGFUSE_S3_EVENT_UPLOAD_PREFIX=events/
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_BUCKET=langfuse
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_REGION=auto
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID=minio
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY=miniosecret
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT=http://localhost:9090
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE=true
 Environment=LANGFUSE_S3_MEDIA_UPLOAD_PREFIX=media/
 Environment=LANGFUSE_S3_BATCH_EXPORT_ENABLED=false
 Environment=LANGFUSE_S3_BATCH_EXPORT_BUCKET=langfuse
 Environment=LANGFUSE_S3_BATCH_EXPORT_PREFIX=exports/
 Environment=LANGFUSE_S3_BATCH_EXPORT_REGION=auto
 Environment=LANGFUSE_S3_BATCH_EXPORT_ENDPOINT=http://localhost:9000
 Environment=LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT=http://localhost:9090
 Environment=LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID=minio
 Environment=LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY=miniosecret
 Environment=LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE=true
 Environment=LANGFUSE_INGESTION_QUEUE_DELAY_MS=
 Environment=LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=
 Environment=REDIS_HOST=localhost
 Environment=REDIS_PORT=6379
 Environment=REDIS_AUTH=myredissecret
 Environment=REDIS_TLS_ENABLED=false
 Environment=REDIS_TLS_CA=/certs/ca.crt
 Environment=REDIS_TLS_CERT=/certs/redis.crt
 Environment=REDIS_TLS_KEY=/certs/redis.key
 Environment=EMAIL_FROM_ADDRESS=
 Environment=SMTP_CONNECTION_URL=
 Image=docker.io/langfuse/langfuse-worker:3
 [Service]
 Restart=always
 [Install]
 WantedBy=default.target
@@ -0,0 +1,12 @@
 [Pod]
 PublishPort=3000:3000/tcp
 PublishPort=127.0.0.1:3030:3030/tcp
 PublishPort=127.0.0.1:8123:8123/tcp
 PublishPort=127.0.0.1:9000:9000/tcp
 PublishPort=9090:9000/tcp
 PublishPort=127.0.0.1:9091:9001/tcp
 PublishPort=127.0.0.1:6379:6379/tcp
 PublishPort=127.0.0.1:5432:5432/tcp
 [Install]
 WantedBy=default.target
@@ -11,7 +11,7 @@ HealthInterval=30s
 HealthRetries=3
 HealthStartPeriod=40s
 HealthTimeout=10s
-Image=ghcr.io/berriai/litellm-database:v1.83.14-stable.patch.3
+Image=ghcr.io/berriai/litellm-database:main-stable
 Volume=/home/ai/litellm-config.yaml:/app/config.yaml:z
 Exec=--config=/app/config.yaml
@@ -0,0 +1,24 @@
 # NTFY
 ## Running ntfy
 ```bash
 podman run \
 -v ntfy:/var/cache/ntfy \
 -v ./active/container_ntfy/server.yml:/etc/ntfy/server.yml:z \
 -p 8001:80 \
 -it \
 docker.io/binwiederhier/ntfy:latest \
 serve \
 --cache-file /var/cache/ntfy/cache.db
 ```
 ## Testing with Curl
 ```bash
 # Publish a topic
 curl -d "Backup successful" localhost:8001/mytopic
 # Listen on a topic
 curl -s localhost:8001/mytopic
 ```
@@ -0,0 +1 @@
 # base-url:
@@ -0,0 +1,16 @@
 {
  "type": "micro",
  "wake_word": "Alexa",
  "author": "Kevin Ahrendt",
  "website": "https://www.kevinahrendt.com/",
  "model": "alexa.tflite",
  "trained_languages": ["en"],
  "version": 2,
  "micro": {
    "probability_cutoff": 0.9,
    "sliding_window_size": 5,
    "feature_step_size": 10,
    "tensor_arena_size": 22348,
    "minimum_esphome_version": "2024.7.0"
  }
 }
@@ -1,9 +1,12 @@
 esphome:
-  name: tab2
+  name: dickhead
-  friendly_name: M5Stack Tab5 2
+  friendly_name: Dickhead
  on_boot:
-    # Set the charing icon to the correct state on boot
+    # Set the charging icon to the correct state on boot
    - then:
        - logger.log: "Delaying backlight initialization"
        - delay: 2s
        - logger.log: "End delay"
        - if:
            condition:
              lambda: return id(charging).state;
@@ -17,6 +20,7 @@ esphome:
 esp32:
  board: esp32-p4-evboard
  flash_size: 16MB
  cpu_frequency: 360MHz
  framework:
    type: esp-idf
    advanced:
@@ -36,6 +40,7 @@ esp32_hosted:
 logger:
  hardware_uart: USB_SERIAL_JTAG
  level: DEBUG
 psram:
  mode: hex
@@ -53,8 +58,8 @@ ota:
  platform: esphome
 wifi:
-  ssid: !secret wifi_ssid
+  ssid: !secret mobile_wifi_ssid
-  password: !secret wifi_password
+  password: !secret mobile_wifi_pass
  fast_connect: true
  on_connect:
    - lvgl.label.update:
@@ -63,15 +68,50 @@ wifi:
    - select.set:
        id: dac_output
        option: "LINE1"
    - lvgl.label.update:
        id: lbl_ip
        text: !lambda return id(ip_addr).state;
    - lvgl.label.update:
        id: lbl_ap
        text: !lambda return id(ssid).state;
  on_disconnect:
    - lvgl.label.update:
        id: lbl_status
        text: "DISCONNECTED"
-#   ap:
+  ap:
-#     password: !secret hotspot_password
+    password: !secret hotspot_password
-#     ap_timeout: 90s
+    ap_timeout: 30s
-# captive_portal:
+captive_portal:
 text_sensor:
  - platform: wifi_info
    ip_address:
      id: ip_addr
      name: Device IP Address
      address_0:
        name: Device IP Address 0
      address_1:
        name: Device IP Address 1
      address_2:
        name: Device IP Address 2
      address_3:
        name: Device IP Address 3
      address_4:
        name: Device IP Address 4
    ssid:
      id: ssid
      name: Device Connected SSID
    bssid:
      name: Device Connected BSSID
    mac_address:
      name: Device Mac Wifi Address
    scan_results:
      name: Device Latest Scan Results
    dns_address:
      name: Device DNS Address
    power_save_mode:
      name: Device Wifi Power Save Mode
 time:
  - platform: sntp
@@ -82,14 +122,13 @@ time:
      - 1.pool.ntp.org
      - 2.pool.ntp.org
-# wireguard:
+wireguard:
-#   address: !secret tab1_wg_ip
+  address: !secret tab1_wg_ip
-#   private_key: !secret tab1_wg_pk
+  private_key: !secret tab1_wg_pk
-#   peer_endpoint: !secret wg_host
+  peer_endpoint: !secret wg_host
-#   peer_public_key: !secret wg_pubkey
+  peer_public_key: !secret wg_pubkey
-
+  netmask: 0.0.0.0
-#   # Optional keepalive (disabled by default)
+  peer_persistent_keepalive: 25s
 #   peer_persistent_keepalive: 25s
 i2c:
  - id: bsp_bus
@@ -196,6 +235,39 @@ binary_sensor:
    pin:
      pi4ioe5v6408: pi4ioe1
      number: 7
  - platform: lvgl
    widget: volume_up_widget
    name: Volume Up Button
    on_press:
      then:
        - logger.log: "Button pressed"
        - media_player.volume_up:
            id: tab5_media_player
        - delay: 100ms
        - lvgl.label.update:
            id: lbl_volume
            text: !lambda return to_string(int(id(tab5_media_player).volume * 100));
        - light.turn_on:
            id: backlight
            brightness: !lambda |-
              float current_value = id(backlight).current_values.get_brightness();
              return current_value < 0.6 ? 0.6 : current_value + 0.2;
  - platform: lvgl
    widget: volume_down_widget
    name: Volume Down Button
    on_press:
      then:
        - logger.log: "Button pressed"
        - media_player.volume_down:
            id: tab5_media_player
        - lvgl.label.update:
            id: lbl_volume
            text: !lambda return to_string(int(id(tab5_media_player).volume * 100));
        - light.turn_on:
            id: backlight
            brightness: !lambda |-
              float current_value = id(backlight).current_values.get_brightness();
              return current_value < 0.6 ? 0.6 : current_value - 0.2;
 sensor:
  - platform: ina226
@@ -255,8 +327,8 @@ touchscreen:
      - logger.log: "LVGL resuming"
      - lvgl.resume:
      - light.turn_on: backlight
-    on_release:
+    # on_release:
-      - media_player.stop:
+    # - media_player.stop:
 esp_ldo:
  - voltage: 2.5V
@@ -287,7 +359,7 @@ light:
    restore_mode: ALWAYS_ON
    default_transition_length: 250ms
    initial_state:
-      brightness: "100%"
+      brightness: "50%"
 image:
  defaults:
@@ -318,20 +390,41 @@ lvgl:
          transition_length: 15s
      - lvgl.pause:
  widgets:
-    - image:
+    - obj:
-        id: listen_icon_widget
+        align: TOP_MID
-        src: va_idle
+        width: 100%
-        align: CENTER
+        height: 100%
        layout:
          type: flex
          flex_flow: column
          flex_align_main: START
          flex_align_track: center
          flex_align_cross: center
        widgets:
          - label:
              align: TOP_MID
              id: lbl_status
              text_font: montserrat_48
              text: "CONNECTING..."
          - label:
              align: TOP_MID
              id: lbl_ap
              text_font: montserrat_22
              text: "CONNECTING..."
          - label:
              align: TOP_MID
              id: lbl_ip
              text_font: montserrat_22
              text: "CONNECTING..."
    - image:
        id: listen_icon_widget
        src: va_idle
        align: CENTER
    - label:
        align: BOTTOM_LEFT
        id: lbl_version
        text_font: montserrat_12
-        text: "v0.5"
+        text: "v0.6"
    - label:
        align: BOTTOM_RIGHT
        id: lbl_battery
@@ -341,21 +434,38 @@ lvgl:
        id: charging_icon_widget
        src: charging_icon
        align: TOP_RIGHT
-    - slider:
+    - button:
-        id: backlight_slider
+        id: volume_up_widget
        widgets:
          - label:
              text: "\uF028"
              text_font: montserrat_48
              text_align: CENTER
              align: CENTER
        x: 20
-        y: 50
+        y: 20
-        width: 30
+        width: 100
-        height: 220
+        height: 100
        pad_all: 8
-        min_value: 0
+    - button:
-        max_value: 255
+        id: volume_down_widget
-        on_release:
+        widgets:
-          - homeassistant.action:
+          - label:
-              action: light.turn_on
+              text: "\uF027"
-              data:
+              text_font: montserrat_48
-                entity_id: light.backlight
+              text_align: CENTER
-                brightness: !lambda return int(x);
+              align: CENTER
        x: 20
        y: 140
        width: 100
        height: 100
        pad_all: 8
    - label:
        x: 20
        y: 260
        id: lbl_volume
        text_font: montserrat_28
        text: !lambda return "%.1f",to_string(id(tab5_media_player).volume);
 # The DAC Output select needs to be manually (or with an automation) changed to `LINE1` for the onboard speaker
 select:
@@ -414,7 +524,7 @@ speaker:
    audio_dac: es8388_dac
    dac_type: external
    channel: mono
-    buffer_duration: 100ms
+    buffer_duration: 60000ms
    bits_per_sample: 16bit
    sample_rate: 48000
@@ -429,9 +539,12 @@ media_player:
 micro_wake_word:
  id: mww
  models:
-    - okay_nabu
+    - model:
-    - hey_mycroft
+        type: local
-    - hey_jarvis
+        path: alexa.json
      id: alexa
      probability_cutoff: 0.97
      sliding_window_size: 5
  on_wake_word_detected:
    - voice_assistant.start:
        wake_word: !lambda return wake_word;
@@ -56,6 +56,13 @@ Make sure your permissions are set correctly
 sudo usermod -a -G dialout ducoterra
 ```
 or on Kinoite
 ```bash
 grep -E '^dialout:' /usr/lib/group | sudo tee -a /etc/group
 sudo usermod -aG dialout $USER
 ```
 Then "run" your config file
 ```bash
@@ -0,0 +1,16 @@
 {
  "type": "micro",
  "wake_word": "hey dick head",
  "author": "peenfart",
  "website": "https://www.kevinahrendt.com/",
  "model": "./hey_dick_head.tflite",
  "trained_languages": ["en"],
  "version": 2,
  "micro": {
    "probability_cutoff": 0.97,
    "sliding_window_size": 5,
    "feature_step_size": 10,
    "tensor_arena_size": 22860,
    "minimum_esphome_version": "2024.7"
  }
 }
@@ -0,0 +1,12 @@
 # Raspberry Pi
 ## Headless Configuration
 1. Create a file called `ssh` in `bootfs`
 2. Create a file called `userconf.txt` in `bootfs`
 3. Run `echo 'YourPassword' | openssl passwd -6 -stdin` in your terminal to generate your encrypted password
 4. Edit `userconf.txt` and add a single line: `username:encrypted_password`
 5. Save `userconf.txt` and eject the SD card
 6. Plug in the pi
 7. Find the pi with `nmap -sn 10.0.0.0/24`
 8. Login with `ssh username@pi-ip-address`
@@ -0,0 +1,60 @@
 # Raspberry Pi — boot-time hardening units
 Two reusable runbooks for a headless Pi that must stay reachable over SSH.
 Deliberately host-agnostic: **no addresses, key material, or peer identities
 live in this repo** — those stay on each host. Set `HOST` and fill in the
 placeholders from the runbook before running anything.
 | Runbook | Solves |
 | --- | --- |
 | [`root-ssh-key.md`](root-ssh-key.md) | root `authorized_keys` gets wiped (imaging, config management) → key is re-installed at boot and re-applied on change |
 | [`wireguard.md`](wireguard.md) | SSH through a WireGuard tunnel address fails after reboot / NAT timeout / roaming to another network |
 Verified against Debian 13 (trixie) with systemd 257 and NetworkManager. The
 units are plain POSIX `sh` + systemd, so they carry to any other distribution
 with the same systemd version or newer.
 ## Layout → target path on the host
 | File in this directory | Installs to |
 | --- | --- |
 | `ensure-root-ssh-key` | `/usr/local/sbin/ensure-root-ssh-key` (`0755`) |
 | `ensure-root-ssh-key.service` | `/etc/systemd/system/` |
 | `ensure-root-ssh-key.path` | `/etc/systemd/system/` |
 | `root-ssh-keys/authorized_keys.example` | `/usr/local/share/root-ssh-keys/authorized_keys` (`0644`) — copy, fill in your public key(s) |
 | `wg0-watchdog` | `/usr/local/sbin/wg0-watchdog` (`0755`) |
 | `wg0-watchdog.service` | `/etc/systemd/system/` |
 | `wg0-watchdog.timer` | `/etc/systemd/system/` |
 | `wg-quick@wg0.service.d-10-boot-resilience.conf` | `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` |
 | `wg0.conf.template` | documents `/etc/wireguard/wg0.conf` (`0600`) — the real file is never copied here |
 ## Deploy pattern
 Every command in both runbooks follows the same shape, so a fresh Pi needs only
 `HOST` and the placeholders:
 ```bash
 HOST=root@<pi-address>
 D=$PWD   # run the snippets from a checkout of this directory
 ```
 Files travel over one `tar | ssh` pipe into `/root`, then `install` moves them
 into place with explicit modes (no `scp` of a whole tree into `/etc`, no
 ambiguous ownership). Reload and enable once at the end:
 ```bash
 ssh "$HOST" 'systemctl daemon-reload && systemctl enable --now <units>'
 ```
 ## Conventions worth keeping on any host
 - Config files under `/usr/local/{sbin,share}` are the source of truth; systemd
  units stay thin wrappers so they can be re-imaged independently.
 - Idempotent installers over clever state: every script re-derives the desired
  state from its own config file and never deletes what it did not create.
 - Anything that must heal itself is gap-limited (a minimum interval between
  restarts) so a broken dependency cannot make a unit flap.
 - Private keys and addresses are excluded from the repo; when dumping a
  WireGuard config, redact with a regex that tolerates `PrivateKey = ` spacing
  (`wg-quick strip` emits spaces, a `PrivateKey=` regex misses them):
  `sed -E 's/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/'`
@@ -0,0 +1,61 @@
 #!/bin/sh
 # ensure-root-ssh-key - idempotently install managed SSH public keys into
 # /root/.ssh/authorized_keys. Runs at boot (ensure-root-ssh-key.service) and
 # can be run by hand at any time.
 #
 # Source of truth: /usr/local/share/root-ssh-keys/authorized_keys
 # One key per line, same format as an authorized_keys file. Blank lines and
 # lines starting with '#' are ignored. Keys already present are left alone;
 # foreign keys in authorized_keys are never removed.
 set -eu
 KEY_DIR="${KEY_DIR:-/root/.ssh}"
 AUTH_KEYS="$KEY_DIR/authorized_keys"
 MANAGED_KEYS="${MANAGED_KEYS:-/usr/local/share/root-ssh-keys/authorized_keys}"
 if [ ! -r "$MANAGED_KEYS" ]; then
    echo "ensure-root-ssh-key: cannot read $MANAGED_KEYS" >&2
    exit 1
 fi
 umask 077
 install -d -m 0700 "$KEY_DIR"
 [ -e "$AUTH_KEYS" ] || : > "$AUTH_KEYS"
 chmod 0600 "$AUTH_KEYS"
 chown root:root "$KEY_DIR" "$AUTH_KEYS"
 added=0
 checked=0
 while IFS= read -r line <&3; do
    case "$line" in
        '' | \#*) continue ;;
    esac
    # Basic shape check: <algo> <base64> [comment]
    algo=${line%% *}
    case "$algo" in
        ssh-ed25519 | ssh-rsa | ssh-dss | ecdsa-sha2-* | sk-ecdsa-sha2-* | sk-ssh-ed25519-* | ssh-ed448) ;;
        *)
            echo "ensure-root-ssh-key: skipping unrecognised line: $line" >&2
            continue
            ;;
    esac
    checked=$((checked + 1))
    if grep -qxF -- "$line" "$AUTH_KEYS" 2>/dev/null; then
        continue
    fi
    printf '%s\n' "$line" >> "$AUTH_KEYS"
    added=$((added + 1))
    echo "ensure-root-ssh-key: installed key (${algo}, ${#line} chars)"
 done 3< "$MANAGED_KEYS"
 if [ "$added" -eq 0 ]; then
    echo "ensure-root-ssh-key: $checked managed key(s) already present in $AUTH_KEYS"
 else
    echo "ensure-root-ssh-key: added $added key(s), $checked managed key(s) total"
 fi
 chmod 0600 "$AUTH_KEYS"
 exit 0
@@ -0,0 +1,16 @@
 [Unit]
 Description=Watch /root/.ssh/authorized_keys and re-install the managed root SSH key
 Documentation=man:sshd(8)
 After=local-fs.target
 [Path]
 # PathChanged covers deletion / rename / creation of the file,
 # PathModified covers in-place edits (key stripped or file truncated).
 # ensure-root-ssh-key.service only writes when a managed key is missing, so
 # its own writes settle after one extra no-op run instead of looping.
 PathChanged=/root/.ssh/authorized_keys
 PathModified=/root/.ssh/authorized_keys
 Unit=ensure-root-ssh-key.service
 [Install]
 WantedBy=multi-user.target
@@ -0,0 +1,15 @@
 [Unit]
 Description=Ensure managed SSH public keys are installed in /root/.ssh/authorized_keys
 Documentation=man:sshd(8)
 After=local-fs.target
 Before=ssh.service sshd.service
 ConditionPathExists=/usr/local/share/root-ssh-keys/authorized_keys
 [Service]
 Type=oneshot
 ExecStart=/usr/local/sbin/ensure-root-ssh-key
 StandardOutput=journal
 StandardError=journal
 [Install]
 WantedBy=multi-user.target
@@ -0,0 +1,109 @@
 # Runbook: self-healing root SSH key
 `ensure-root-ssh-key.service` re-installs the managed SSH public key(s) into
 `/root/.ssh/authorized_keys` on every boot, and `ensure-root-ssh-key.path`
 re-applies it at runtime whenever that file is edited, deleted, or replaced.
 Useful when the key gets wiped by imaging, first-boot tooling, or config
 management — or when you need the key to come back after someone else edits the
 file.
 ## Files on the host
 | Path | Purpose |
 | --- | --- |
 | `/etc/systemd/system/ensure-root-ssh-key.service` | `Type=oneshot`, `WantedBy=multi-user.target`, ordered `Before=ssh.service` so the key is in place before the daemon starts |
 | `/etc/systemd/system/ensure-root-ssh-key.path` | `PathChanged=` + `PathModified=` on `/root/.ssh/authorized_keys`, triggers the service |
 | `/usr/local/sbin/ensure-root-ssh-key` | Idempotent POSIX `sh` installer |
 | `/usr/local/share/root-ssh-keys/authorized_keys` | Source of truth — one public key per line, `#` comments allowed |
 ## Behaviour
 - Appends any managed key that is missing; never removes or rewrites foreign keys.
 - Creates `/root/.ssh` (`0700`) and `authorized_keys` (`0600`, root:root) if absent.
 - Lines whose first field is not a known key algorithm are skipped with a warning.
 - Skipped, not failed, if the managed keys file is deleted.
 - Runtime self-healing: deleting, truncating, or renaming `authorized_keys` puts
  the managed key(s) back within ~1s. That also means a managed key cannot be
  revoked from the host alone — delete it from
  `/usr/local/share/root-ssh-keys/authorized_keys` first.
 > Gotcha: systemd has no `PathMissing=` key — it is silently ignored with a
 > warning in `systemd-analyze verify`. `PathChanged=` is what fires on
 > delete/rename; `PathModified=` covers in-place appends and truncations. Set both.
 ## Install
 ```bash
 HOST=root@<pi-address>
 D=$PWD        # run the snippets from a checkout of this directory
 PUB=$HOME/.ssh/id_ed25519.pub          # the key that must always work
 ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /usr/local/share/root-ssh-keys'
 tar -C "$D" -cf - ensure-root-ssh-key ensure-root-ssh-key.service ensure-root-ssh-key.path | \
 ssh "$HOST" 'tar -xf - -C /root &&
  install -m 0755 /root/ensure-root-ssh-key /usr/local/sbin/ &&
  install -m 0644 /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path /etc/systemd/system/ &&
  rm -f /root/ensure-root-ssh-key /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path &&
  systemctl daemon-reload'
 cat "$PUB" | ssh "$HOST" 'install -m 0644 /dev/stdin /usr/local/share/root-ssh-keys/authorized_keys'
 ssh "$HOST" 'systemctl enable --now ensure-root-ssh-key.service ensure-root-ssh-key.path'
 ```
 Use `root-ssh-keys/authorized_keys.example` instead of `$PUB` when the host
 should carry more than one key.
 ## Verify
 One-shot smoke test that proves both units without touching a live session
 (existing SSH sessions are unaffected; only new logins during the window would
 fail):
 ```bash
 ssh "$HOST" 'cp -a /root/.ssh/authorized_keys /root/.ssh/authorized_keys.testbak
  rm -f /root/.ssh/authorized_keys
  systemctl start ensure-root-ssh-key.service && cat /root/.ssh/authorized_keys
  rm -f /root/.ssh/authorized_keys.testbak'
 ```
 Status and history:
 ```bash
 ssh "$HOST" 'systemctl is-enabled ensure-root-ssh-key.service ensure-root-ssh-key.path
  systemctl status ensure-root-ssh-key.service --no-pager | head -8
  journalctl -u ensure-root-ssh-key.service -b --no-pager | tail'
 ```
 The installer honours `KEY_DIR` and `MANAGED_KEYS`, so the create-if-missing and
 preserve-foreign-keys paths can be exercised in a scratch directory instead of
 `/root/.ssh`:
 ```bash
 ssh "$HOST" 'install -d -m 0700 /tmp/k/src && cp /usr/local/share/root-ssh-keys/authorized_keys /tmp/k/src/
  KEY_DIR=/tmp/k/root MANAGED_KEYS=/tmp/k/src/authorized_keys /usr/local/sbin/ensure-root-ssh-key
  ls -ld /tmp/k/root /tmp/k/root/authorized_keys; rm -rf /tmp/k'
 ```
 Checklist that has passed on a reference host:
 - `systemd-analyze verify` clean for both units, both `enabled`.
 - Repeated runs add no duplicates; a wiped file and a deleted file are both restored.
 - Missing `/root/.ssh` is created `0700`, file `0600`.
 - Foreign keys survive; no trigger loop once the path unit settles.
 - Boot ordering: `Before=ssh.service` is honoured inside the boot transaction;
  prove it after a reboot with `systemd-analyze critical-chain ssh.service`.
 ## Manage keys
 ```bash
 ssh "$HOST" 'echo "<public key line>" >> /usr/local/share/root-ssh-keys/authorized_keys
  systemctl restart ensure-root-ssh-key.service'
 ```
 ## Uninstall
 ```bash
 ssh "$HOST" 'systemctl disable --now ensure-root-ssh-key.path ensure-root-ssh-key.service
  rm -f /etc/systemd/system/ensure-root-ssh-key.{service,path} /usr/local/sbin/ensure-root-ssh-key
  rm -rf /usr/local/share/root-ssh-keys
  systemctl daemon-reload'
 ```
@@ -0,0 +1,11 @@
 # Template for /usr/local/share/root-ssh-keys/authorized_keys on the host.
 # Keys listed here are (re)installed into /root/.ssh/authorized_keys at boot and
 # whenever that file changes. Copy to `authorized_keys`, add your public keys,
 # and keep the real file on the host - do not commit it.
 #
 # One key per line, authorized_keys format. Blank lines and lines starting with
 # '#' are ignored; a line whose first field is not a known key algorithm is
 # skipped with a warning.
 #
 # ssh-ed25519 AAAA<base64> comment@host
 # ssh-rsa     AAAA<base64> comment@host
@@ -0,0 +1,17 @@
 # /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf
 # The stock unit already has After=/Wants=network-online.target nss-lookup.target.
 # This makes a boot-time failure (DHCP/DNS not ready yet, endpoint host offline)
 # retry forever instead of leaving the tunnel down until someone logs in.
 [Unit]
 # Unlimited retries: do not let the start rate limiter give up on us.
 StartLimitIntervalSec=0
 [Service]
 # Fail fast instead of hanging forever on endpoint resolution (stock value is
 # infinity), then let Restart= try again. Same variable as the stock unit,
 # later assignment wins.
 Environment=WG_ENDPOINT_RESOLUTION_RETRIES=60
 TimeoutStartSec=120
 TimeoutStopSec=30
 Restart=on-failure
 RestartSec=15
@@ -0,0 +1,151 @@
 #!/bin/sh
 # wg0-watchdog - keep the WireGuard client tunnel up so SSH to the tunnel
 # address works after boot, after roaming to a new network, and after the NAT
 # mapping or endpoint IP goes stale.
 #
 # Decisions, oldest symptom first:
 #   interface or peer gone            -> restart wg-quick@wg0.service
 #   no PersistentKeepalive in kernel  -> set it (covers a config edited without reload)
 #   handshake older than STALE        -> re-point the peer at the endpoint
 #                                        (re-resolves the hostname, forces a handshake)
 #   handshake older than STALE_RESTART-> restart wg-quick@wg0.service
 #
 # Restarts are gap-limited so a broken network cannot make the unit flap.
 # Never removes peers, addresses, or routes it did not create.
 set -eu
 IFACE="${WG_WD_IFACE:-wg0}"
 UNIT="${WG_WD_UNIT:-wg-quick@wg0.service}"
 CONF="${WG_WD_CONF:-/etc/wireguard/$IFACE.conf}"
 KEEPALIVE="${WG_WD_KEEPALIVE:-25}"
 STALE="${WG_WD_STALE:-120}"              # seconds: poke the endpoint
 STALE_RESTART="${WG_WD_STALE_RESTART:-300}"   # seconds: restart the unit
 MIN_RESTART_GAP="${WG_WD_MIN_RESTART_GAP:-180}"  # seconds between restarts
 STATE_DIR="${WG_WD_STATE_DIR:-/var/lib/wg0-watchdog}"
 DRY=0
 [ "${1:-}" = "--dry-run" ] && DRY=1
 now=$(date +%s)
 log() { printf 'wg0-watchdog: %s\n' "$*"; }
 run() {
    if [ "$DRY" -eq 1 ]; then
        printf 'wg0-watchdog: DRY-RUN would run:'
        printf ' %s' "$@"
        printf '\n'
    else
        "$@"
    fi
 }
 read_state() { cat "$1" 2>/dev/null || echo 0; }
 write_state() {
    if [ "$DRY" -eq 1 ]; then
        return 0
    fi
    mkdir -p "$STATE_DIR"
    printf '%s\n' "$2" > "$1"
 }
 restart_unit() {
    reason=$1
    last=$(read_state "$STATE_DIR/last-restart")
    case "$last" in ''|*[!0-9]*) last=0 ;; esac
    since=$((now - last))
    if [ "$since" -lt "$MIN_RESTART_GAP" ]; then
        log "restart of $UNIT suppressed ($reason); last restart ${since}s ago"
        return 0
    fi
    log "restarting $UNIT ($reason)"
    write_state "$STATE_DIR/last-restart" "$now"
    # --no-block: a wg-quick up stuck on endpoint resolution must not wedge the
    # watchdog; the next run reports whether the restart worked.
    run systemctl restart --no-block "$UNIT"
 }
 # --- 1. interface / peer present -------------------------------------------
 if ! ip link show "$IFACE" >/dev/null 2>&1; then
    restart_unit "interface $IFACE is missing"
    exit 0
 fi
 peer=$(wg show "$IFACE" peers 2>/dev/null | head -n 1)
 if [ -z "${peer:-}" ]; then
    restart_unit "no peer configured on $IFACE"
    exit 0
 fi
 # --- 2. persistent keepalive present ---------------------------------------
 live_ka=$(wg show "$IFACE" persistent-keepalive 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
 [ -n "${live_ka:-}" ] || live_ka=0
 if [ "$live_ka" != "$KEEPALIVE" ]; then
    log "persistent-keepalive is $live_ka, setting $KEEPALIVE"
    run wg set "$IFACE" peer "$peer" persistent-keepalive "$KEEPALIVE"
 fi
 # --- 3. handshake age -------------------------------------------------------
 hs=$(wg show "$IFACE" latest-handshakes 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
 case "${hs:-}" in ''|*[!0-9]*) hs=0 ;; esac
 if [ "$hs" -eq 0 ]; then
    age=999999
 else
    age=$((now - hs))
 fi
 # endpoint currently in use by the kernel, and the cached last-known-good one
 live_ep=$(wg show "$IFACE" endpoints 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
 cache_file="$STATE_DIR/endpoint"
 cached_ep=""
 [ -f "$cache_file" ] && cached_ep=$(cat "$cache_file" 2>/dev/null || true)
 if [ "$age" -lt "$STALE" ]; then
    if [ -n "$live_ep" ]; then
        write_state "$cache_file" "$live_ep"
    fi
    log "healthy: handshake ${age}s ago, endpoint ${live_ep:-unknown}, keepalive ${KEEPALIVE}s"
    exit 0
 fi
 log "stale: last handshake ${age}s ago"
 # Prefer the endpoint from the config file, freshly resolved. Fall back to the
 # kernel's current endpoint, then to the cache - important because a DNS server
 # from the tunnel config may be reachable only through this very tunnel.
 target_ep=""
 conf_ep=$(awk -F= 'tolower($1) ~ /^[[:space:]]*endpoint[[:space:]]*$/ { gsub(/[[:space:]]/, "", $2); print $2 }' "$CONF" 2>/dev/null | tail -n 1)
 if [ -n "${conf_ep:-}" ]; then
    host=${conf_ep%:*}
    port=${conf_ep##*:}
    case "$host" in
        *[!0-9.]*)
            resolved=$(getent hosts "$host" 2>/dev/null | awk '{print $1; exit}' || true)
            ;;
        *)
            resolved=$host
            ;;
    esac
    if [ -n "${resolved:-}" ]; then
        target_ep="$resolved:$port"
    else
        log "could not resolve $host (tunnel DNS?), falling back"
    fi
 fi
 [ -n "$target_ep" ] || target_ep=${live_ep:-${cached_ep:-}}
 if [ "$age" -ge "$STALE_RESTART" ]; then
    restart_unit "handshake ${age}s old"
    exit 0
 fi
 if [ -z "$target_ep" ]; then
    log "no endpoint to re-point at; will retry next run"
    exit 0
 fi
 if [ "$target_ep" = "$live_ep" ]; then
    log "endpoint already $target_ep; forcing a handshake"
 fi
 log "re-pointing peer at $target_ep"
 run wg set "$IFACE" peer "$peer" endpoint "$target_ep"
 exit 0
@@ -0,0 +1,12 @@
 [Unit]
 Description=Keep the WireGuard wg0 tunnel reachable for SSH
 Documentation=man:wg-quick(8)
 [Service]
 Type=oneshot
 # Never let a hung systemctl restart inside the script wedge the watchdog.
 TimeoutStartSec=120
 ExecStart=/usr/local/sbin/wg0-watchdog
 StandardOutput=journal
 StandardError=journal
 Nice=10
@@ -0,0 +1,14 @@
 [Unit]
 Description=Check the WireGuard wg0 tunnel every minute and heal it if needed
 [Timer]
 # First check shortly after boot, then every minute. Persistent=true catches up
 # if the machine was suspended/offline.
 OnBootSec=30s
 OnUnitActiveSec=60s
 AccuracySec=15s
 RandomizedDelaySec=5s
 Persistent=true
 [Install]
 WantedBy=timers.target
@@ -0,0 +1,20 @@
 # wg0.conf template — /etc/wireguard/wg0.conf on the host, mode 0600 root:root.
 # Never commit a real copy: this file holds the tunnel private key.
 # Generate the key with `umask 077; wg genkey | tee privatekey | wg pubkey`.
 [Interface]
 PrivateKey = <tunnel private key>
 Address = <TUNNEL_ADDR>
 # Deliberately no `DNS` key: any resolver listed here is pushed as a system
 # upstream while the tunnel is up, which makes all name resolution depend on the
 # tunnel. Omitting it keeps the LAN's DHCP resolvers. Debian's wg-quick has no
 # `DNS = none` - a non-IP value becomes a search domain.
 [Peer]
 PublicKey = <PEER_PUBKEY>
 AllowedIPs = <PEER_NETWORK>
 Endpoint = <WG_SERVER_HOST:PORT>
 # Required when the client is behind NAT and must be reachable inbound (SSH):
 # without it the mapping expires and the server cannot initiate. 25s survives
 # typical 30-60s NAT UDP timeouts.
 PersistentKeepalive = 25
@@ -0,0 +1,162 @@
 # Runbook: WireGuard client that survives reboot, NAT, and roaming
 For a Pi whose SSH address is a **WireGuard tunnel address** rather than a LAN
 address: the physical interface is ordinary DHCP and the Pi moves between
 networks, while the tunnel peer is a reachable server. Reachability then depends
 on three separate things, each of which fails independently — the unit starting
 at boot, the NAT mapping staying open, and the peer hostname still resolving.
 Placeholders used below:
 | Placeholder | Meaning |
 | --- | --- |
 | `$HOST` | `root@<address you can already reach>` |
 | `<TUNNEL_ADDR>` | this Pi's tunnel address, e.g. `<addr>/32` |
 | `<PEER_NETWORK>` | network routed through the tunnel, e.g. `<net>/<cidr>` |
 | `<WG_SERVER_HOST:PORT>` | tunnel server endpoint |
 | `<PEER_PUBKEY>` | the server's WireGuard public key |
 ## Files on the host
 | Path | Purpose |
 | --- | --- |
 | `/etc/wireguard/wg0.conf` | tunnel config, mode `0600` — see `wg0.conf.template`, never stored in this repo |
 | `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` | retry a failed boot-time `up`: `Restart=on-failure`, `RestartSec=15`, `StartLimitIntervalSec=0`, `WG_ENDPOINT_RESOLUTION_RETRIES=60`, `TimeoutStartSec=120` |
 | `/usr/local/sbin/wg0-watchdog` | heals the tunnel once a minute |
 | `/etc/systemd/system/wg0-watchdog.service` | oneshot that runs the script |
 | `/etc/systemd/system/wg0-watchdog.timer` | `OnBootSec=30s`, `OnUnitActiveSec=60s`, `Persistent=true` |
 | `/var/lib/wg0-watchdog/endpoint` | last-known-good endpoint IP, used when DNS is unavailable |
 ## Failure modes this closes
 1. **No `PersistentKeepalive`.** Behind NAT the server can only send to the Pi
   while the mapping lives, so once it expires the tunnel looks configured but
   inbound SSH times out. `PersistentKeepalive = 25` punches UDP state often
   enough for typical 30-60s NAT timeouts.
 2. **A boot-time failure is permanent.** `wg-quick up` fails outright if the
   endpoint hostname does not resolve yet or the server is down, and the unit is
   not retried — so the tunnel stays dead until someone logs in locally.
   `Restart=on-failure` with `StartLimitIntervalSec=0` retries forever;
   `WG_ENDPOINT_RESOLUTION_RETRIES` is lowered from the stock `infinity`
   (which hangs the unit indefinitely) so a retry can actually happen.
 3. **Nothing heals a stale tunnel.** Roaming to a new network keeps the old
   endpoint IP in the kernel, and a peer whose endpoint is unreachable simply
   stops handshaking without the unit ever noticing, because the unit already
   "succeeded". The watchdog acts on handshake age.
 ## Install
 ```bash
 HOST=root@<pi-address>
 D=$PWD        # run the snippets from a checkout of this directory
 ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /etc/systemd/system/wg-quick@wg0.service.d'
 tar -C "$D" -cf - wg0-watchdog wg0-watchdog.service wg0-watchdog.timer \
    'wg-quick@wg0.service.d-10-boot-resilience.conf' | \
 ssh "$HOST" 'tar -xf - -C /root &&
  install -m 0755 /root/wg0-watchdog /usr/local/sbin/ &&
  install -m 0644 /root/wg0-watchdog.service /root/wg0-watchdog.timer /etc/systemd/system/ &&
  install -m 0644 "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" \
    /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf &&
  rm -f /root/wg0-watchdog /root/wg0-watchdog.service /root/wg0-watchdog.timer \
    "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" &&
  systemctl daemon-reload && systemctl enable --now wg-quick@wg0.service wg0-watchdog.timer'
 ```
 Keep `PersistentKeepalive = 25` in `[Peer]` (see `wg0.conf.template`) and check
 the config file is `0600` — a world-readable `wg0.conf` exposes the tunnel
 private key to every local account:
 ```bash
 ssh "$HOST" 'ls -l /etc/wireguard/wg0.conf; wg show wg0 persistent-keepalive'
 ```
 ## Watchdog behaviour
 Oldest symptom first, one action per run:
 1. interface or peer gone → `systemctl restart --no-block wg-quick@wg0.service`
 2. no `PersistentKeepalive` in the kernel → `wg set ... persistent-keepalive <n>`
   (covers a config edited without a reload)
 3. handshake older than `WG_WD_STALE` (default 120s) → re-point the peer at the
   endpoint, preferring a fresh resolution of the hostname from the config file,
   then the kernel's current endpoint, then the cached IP
 4. handshake older than `WG_WD_STALE_RESTART` (default 300s) → restart the unit
 Restarts are gap-limited to one per `WG_WD_MIN_RESTART_GAP` (default 180s) so a
 dead network cannot make the unit flap. `--no-block` keeps a hung `wg-quick up`
 from wedging the watchdog. Peers, addresses, and routes are only ever added.
 All thresholds and the interface/state paths are environment overrides, and
 `--dry-run` prints the decision without acting.
 The endpoint resolution order matters when the tunnel config also pushes a DNS
 server: that resolver is reachable only through the tunnel, so a dead tunnel can
 mean the endpoint hostname cannot be resolved exactly when it is needed — hence
 the cached-IP fallback.
 ## DNS
 A `DNS = <resolver>` line in the interface section is pushed through
 `resolvconf`/`systemd-resolved` while the tunnel is up, which can silently make
 *all* name resolution on the host depend on the tunnel. If the Pi must resolve
 names on whatever network it lands on, omit the `DNS` key entirely and let the
 LAN's DHCP resolvers stand.
 Two gotchas:
 - Debian's `wg-quick` does **not** implement `DNS = none` — any non-IP value
  becomes a search domain, so the line has to be absent, not set to `none`.
 - A record already pushed by a previous `wg-quick up` survives the config edit;
  withdraw it (`resolvconf -d <record>`, name visible in `resolvconf --list`) and
  confirm the local caching resolver's upstreams before believing it is gone.
 ## Taking wg0 down on purpose
 Stop the watchdog first or the tunnel comes straight back:
 ```bash
 ssh "$HOST" 'systemctl stop wg0-watchdog.timer wg-quick@wg0.service'
 ssh "$HOST" 'systemctl start wg0-watchdog.timer'      # resume
 ```
 ## Check on it
 ```bash
 ssh "$HOST" 'wg show wg0
  systemctl status wg-quick@wg0 --no-pager | head -6
  systemctl list-timers "wg0-watchdog*" --no-pager | head -3
  journalctl -u wg0-watchdog.service -b --no-pager | tail -15'
 ```
 Dry-run the decision logic without touching the tunnel:
 ```bash
 ssh "$HOST" 'WG_WD_STATE_DIR=/tmp/wd WG_WD_STALE=0 /usr/local/sbin/wg0-watchdog --dry-run; rm -rf /tmp/wd'
 ```
 Validate an edited config without disturbing the running tunnel:
 ```bash
 ssh "$HOST" 'wg-quick strip /etc/wireguard/wg0.conf | sed -E "s/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/"'
 ```
 ## Diagnosis notes (generic, worth checking on any host)
 - `wg-quick@<iface>` can be `enabled` yet `inactive` while the interface exists
  and passes traffic — it may have been created by hand or taken over by
  NetworkManager as an assumed profile (`nmcli dev status` shows such a device
  as `connected (externally)`). Nothing owns it: `systemctl reload` fails with
  "unit not active", so apply changes with `wg set` or a full restart, and let
  `systemctl restart` take ownership before trusting a reboot.
 - `Restart=on-failure` does work for `Type=oneshot` on systemd 257 — worth
  proving on a new distribution with a throwaway unit under
  `/run/systemd/system` before relying on it: `Type=oneshot` +
  `ExecStart=/bin/false`, then check `systemctl show -p NRestarts`.
 - `systemd-analyze verify <unit>` catches mistyped unit keys, and it only warns —
  read the output instead of trusting the exit status.
 - NetworkManager does not fight `wg-quick` as long as no keyfile exists for the
  interface and its assumed profile keeps `autoconnect no`; check
  `/etc/NetworkManager/system-connections/` before assuming that.
 - The real end-to-end test is a reboot: DHCP → endpoint resolution →
  `wg-quick up` → inbound SSH through the tunnel. `systemd-analyze
  critical-chain ssh.service` after boot shows whether the ordering held.
@@ -0,0 +1,48 @@
 # Reese's Workstation
 - [Reese's Workstation](#reeses-workstation)
  - [Virtual Machines](#virtual-machines)
 ## Virtual Machines
 1. Blueprints are stored in active/software_osbuild
 2. Builds are stored in ${HOME}/Images/builds
 ```bash
 # These are saved in ~/.bashrc.d/osbuild.sh
 export OSBUILD_BLUEPRINT_DIR=active/software_osbuild
 export OSBUILD_CLOUD_DIR=${HOME}/Images/cloud
 export OSBUILD_BUILD_DIR=${HOME}/Images/builds
 export OSBUILD_IMAGE_DIR=${HOME}/Images/disks
 # Push a blueprint
 composer-cli blueprints push ${OSBUILD_BLUEPRINT_DIR}/fedora43-base.toml
 # Start a build
 composer-cli compose start fedora43-base qcow2
 # Watch the build
 watch composer-cli compose list running
 # Extract the build
 composer-cli compose image \
 --filename ${OSBUILD_BUILD_DIR}/fedora-43-base.qcow2 \
 92a2f1dd-9238-4d50-ab1d-cd7c8e355cac
 # Convert the build to raw
 qemu-img convert -f qcow2 -O raw \
 ${OSBUILD_BUILD_DIR}/fedora-43-base.qcow2 \
 ${OSBUILD_IMAGE_DIR}/fedora-boot.raw
 # Install (Change password for default user ducoterra!)
 virt-install \
 --boot uefi,firmware.feature0.name=secure-boot,firmware.feature0.enabled=no \
 --cpu host-passthrough --vcpus sockets=1,cores=8,threads=2 \
 --ram=8192 \
 --os-variant=fedora43 \
 --network default \
 --graphics none \
 --console pty,target.type=virtio \
 --name "fedora" \
 --import --disk "path=${OSBUILD_IMAGE_DIR}/fedora-boot.raw,bus=virtio"
 ```
@@ -0,0 +1,89 @@
 # KDE Notes
 ## Fixing alt tab or taskbar icons
 1. System Settings - Window Management - Window Rules - Add New...
 2. Bring the buggy app window side-by-side with system settings.
 3. Click "Detect Window Properties..." in the bottom.
 4. Move the crosshair to the buggy window and click.
 5. Click "Window class (application)".
 6. Click "Add Property..." in the bottm left.
 7. Click "Desktop file name".
 8. Enter the base name of the desktop file
    For flatpak, check `/var/lib/flatpak/exports/share/applications/`
    For system apps, check `/usr/share/applications/`
    For user apps, check `~/.local/share/applications/`
 9. Click "Apply".
 10. The icon should be fixed immediately.
 ## KDE Connect
 Allow KDE Connect via 1714-1764 tcp/udp
 ```bash
 # Set source address to allow connections
 sudo firewall-cmd \
 --zone=drop \
 --permanent \
 --add-port=1714-1764/udp \
 --add-port=1714-1764/tcp
 sudo firewall-cmd --reload
 ```
 ## Automatic Display Switching
 ```bash
 # List displays
 # Builtin: eDP-2
 # Roku: DP-13
 # Lenovo: DP-11
 kscreen-doctor -o
 # Put the builtin display back to normal
 kscreen-doctor \
 output.eDP-2.enable \
 output.eDP-2.position.0,0 \
 output.eDP-2.primary \
 output.eDP-2.mode.2560x1600@165 \
 output.eDP-2.scale.1.25
 # Show on Roku TV and Monitor
 kscreen-doctor \
 output.DP-11.enable \
 output.DP-11.position.0,0 \
 output.DP-11.mode.2560x1440@60 \
 output.DP-11.scale.1 \
 output.DP-13.enable \
 output.DP-13.position.0,0 \
 output.DP-13.mode.2560x1440@100 \
 output.DP-13.scale.1 \
 output.DP-13.primary \
 output.eDP-2.disable
 # Show only on the roku TV
 kscreen-doctor \
 output.DP-11.enable \
 output.DP-11.position.0,0 \
 output.DP-11.primary \
 output.DP-11.mode.3840x2160@60 \
 output.DP-11.scale.2 \
 output.DP-13.disable \
 output.eDP-2.disable
 # Mirror the builtin display to the roku tv
 kscreen-doctor \
 output.DP-11.enable \
 output.DP-11.position.0,0 \
 output.DP-11.primary \
 output.DP-11.mode.3840x2160@60 \
 output.DP-11.scale.2 \
 output.eDP-2.enable \
 output.eDP-2.mode.1920x1080@60 \
 output.eDP-2.scale.1 \
 output.eDP-2.position.0,0
 ```
@@ -1,114 +1,204 @@
 # Fedora Kinoite
 - [Fedora Kinoite](#fedora-kinoite)
-  - [TPM2 Luks Decryption](#tpm2-luks-decryption)
+  - [Adding users to groups (usermod replacement)](#adding-users-to-groups-usermod-replacement)
-  - [Podman](#podman)
+  - [System Prereqs](#system-prereqs)
-    - [Docker Compose and Docker Buildkit with Rootless Podman](#docker-compose-and-docker-buildkit-with-rootless-podman)
+  - [Toolbox](#toolbox)
-  - [rpm-ostree](#rpm-ostree)
+    - [Clipboard Sharing](#clipboard-sharing)
-    - [Git, Vim, etc](#git-vim-etc)
+    - [Running Host Commands](#running-host-commands)
-    - [Libvirt, Qemu, KVM](#libvirt-qemu-kvm)
+    - [Saving a Toolbox](#saving-a-toolbox)
-    - [ROCM](#rocm)
+  - [Apps](#apps)
-    - [Ollama](#ollama)
+    - [Gear Lever](#gear-lever)
-  - [Network](#network)
+    - [VSCodium](#vscodium)
-    - [Hostname](#hostname)
+    - [Ansible](#ansible)
-    - [VLAN Setup with nmcli](#vlan-setup-with-nmcli)
+    - [OpenCode](#opencode)
-  - [GPU Support in Distrobox](#gpu-support-in-distrobox)
+    - [Nextcloud](#nextcloud)
-  - [Automatic Display Switching](#automatic-display-switching)
+    - [UV](#uv)
    - [NVM](#nvm)
  - [Set Hostname](#set-hostname)
  - [rpm-ostree Notes](#rpm-ostree-notes)
  - [Accent keys when holding a key](#accent-keys-when-holding-a-key)
-## TPM2 Luks Decryption
+## Adding users to groups (usermod replacement)
 Mostly taken from here:
 <https://gist.github.com/jdoss/777e8b52c8d88eb87467935769c98a95>
 PCR reference for `--tpm2-pcrs` args
 ```text
 0: System firmware executable
 2: Kernel
 4: Bootloader
 7: Secure boot state
 8: Cmdline
 9: Initrd
 ```
 Basic commands:
 ```bash
-# Show tpm2 devices
+grep -E '^weldr:' /usr/lib/group | sudo tee -a /etc/group
-systemd-cryptenroll --tpm2-device=list
+sudo usermod -aG weldr $USER
 # Show crypto luks block devices
 blkid -t TYPE=crypto_LUKS
 # Enroll the tpm2 device with systemd-cryptenroll
 systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0,2,4,7,8,9 /dev/nvme0n1p3
 # Reenroll
 systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=0,2,4,7,8,9
 # Append to command line args
 rpm-ostree kargs --append=rd.luks.options=tpm2-device=auto
 ```
-When you update you'll need to reenroll. Add this to your ~/.bashrc
+## System Prereqs
 ```bash
-# LUKS TPM2 commands
+# Appimage support
-alias tpm2-reenroll='sudo systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=0,2,4,7,8,9'
+rpm-ostree install fuse fuse-libs
 # Editor
 rpm-ostree install vim
 # Qemu/KVM
 rpm-ostree install virt-manager libvirt virt-install
 # OSBuild
 rpm-ostree install -y osbuild-composer composer-cli openscap-scanner scap-security-guide
 # Distrobox
 rpm-ostree install -y distrobox
 # BTRFS Snapshots
 rpm-ostree install -y snapper
 # Kitty Terminal
 # terminfo prevents the "WARNING: terminal is not fully functional" error
 rpm-ostree install -y kitty kitty-terminfo
 ```
-## Podman
+## Toolbox
-Since you'll be using podman for most container-based services, you'll want to set the
+### Clipboard Sharing
 the podman auth file to somewhere persistent, otherwise it'll get deleted every time you
 reboot.
-Add this to your `.bashrc`:
+Inside the toolbox run:
 ```bash
-# Podman auth file
+sudo dnf install -y wl-clipboard
 export REGISTRY_AUTH_FILE=$HOME/.podman-auth.json
 ```
-Source that and then run `podman login` to create the file.
+### Running Host Commands
-### Docker Compose and Docker Buildkit with Rootless Podman
+You'll want to use toolbox for development. Unfortunately, toolbox doesn't
 integrate well with host system operations.
-Allows you to use podman with full docker-compose compatibility.
+I create a `toolbox.sh` in `~/.bashrc.d` with functions that map to host
-
+commands like so:
 <https://emersion.fr/blog/2025/using-podman-compose-and-buildkit/>
 ```bash
-rpm-ostree install docker-compose docker-buildx
+# Use flatpak-spawn --host podman inside toolbox, regular podman outside
-reboot
+podman() {
-systemctl --user enable --now podman.socket
+    if [ -f /run/.containerenv ]; then
-docker context create podman --docker host=unix://$XDG_RUNTIME_DIR/podman/podman.sock
+        flatpak-spawn --host podman "$@"
-docker context use podman
+    else
        command podman "$@"
    fi
 }
-### Autostarting services with quadlets
+# Notify the user if they're trying to run a command that doesn't work in a toolbox
-
+rpm-ostree() {
-If you want to run something as your user at boot (like a systemd process, think ollama) you can
+    if [ -f /run/.containerenv ]; then
-create a user quadlets like so:
+        echo "Command must be run outside toolbox."
-
+    else
-```bash
+        command rpm-ostree "$@"
-# Generate the .container file
+    fi
-podman run --rm ghcr.io/containers/podlet --install --description "Local AI" \
+}
  podman run \
  -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  --restart always \
  docker.io/ollama/ollama > ~/.config/containers/systemd/ollama.container
 # Verify the service (Note the filename:service, this is required! You will get "Failed to prepare filename" without it)
 systemd-analyze verify ~/.config/containers/systemd/ollama.container:ollama.service
 # Start the service
 systemctl --user daemon-reload
 systemctl --user start ollama
 ```
-## rpm-ostree
+I have the following functions mapped:
 1. podman
 2. rpm-ostree
 3. flatpak
 4. firewall-cmd
 ### Saving a Toolbox
 After setting up a base toolbox you'll want to save it for future use. This is easy with podman commit.
 ```bash
 # 1. Commit the running container to a new image
 podman commit fedora-toolbox-44 my-toolbox-snapshot
 # 2. Save the image to a tar file for later use
 podman save -o ~/my-toolbox-snapshot.tar my-toolbox-snapshot
 # Load the image
 podman load -i ~/my-toolbox-snapshot.tar
 # Create a new toolbox from it
 toolbox create --image my-toolbox-snapshot
 ```
 ## Apps
 Anything not mentioned here is installed via flatpak from flathub.
 ### Gear Lever
 ```bash
 flatpak install it.mijorus.gearlever
 ```
 ### VSCodium
 Install in toolbox.
 <https://vscodium.com/#install-on-fedora-rhel-centos-rockylinux-opensuse-rpm-package>
 ```bash
 sudo tee -a /etc/yum.repos.d/vscodium.repo << 'EOF'
 [gitlab.com_paulcarroty_vscodium_repo]
 name=gitlab.com_paulcarroty_vscodium_repo
 baseurl=https://paulcarroty.gitlab.io/vscodium-deb-rpm-repo/rpms/
 enabled=1
 gpgcheck=1
 repo_gpgcheck=1
 gpgkey=https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg
 metadata_expire=1h
 EOF
 sudo dnf install codium
 ```
 ### Ansible
 ```bash
 # Install pipx
 python3 -m ensurepip --user
 python3 -m pip install --user pipx
 pipx ensurepath
 # Install ansible
 pipx install --include-deps ansible
 ```
 ### OpenCode
 ```bash
 curl -fsSL https://opencode.ai/install | bash
 # Install pyright LSP
 python -m pip install --upgrade pip
 python -m pip install --user pyright
 ```
 ### Nextcloud
 Download the AppImage. Install via Gear Lever.
 <https://github.com/nextcloud-releases/desktop/releases>
 | Source | Repo                       | Release file name |
 | ------ | -------------------------- | ----------------- |
 | Github | nextcloud-releases/desktop | *x86_64.AppImage  |
 ### UV
 ```bash
 curl -LsSf https://astral.sh/uv/install.sh | sh
 ```
 ### NVM
 <https://www.nvmnode.com/guide/installation.html#nvm-install-for-linux-macos>
 ```bash
 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
 ```
 ## Set Hostname
 ```bash
 sudo hostnamectl hostname reesework16
 ```
 ## rpm-ostree Notes
 Note: if you don't need them, remove extra repos
@@ -133,169 +223,6 @@ rpm-ostree apply-live
 rpm-ostree status
 ```
-### Git, Vim, etc
+## Accent keys when holding a key
-Some packages are nice to have at the system level.
+Settings -> keyboard -> virtual keyboard -> disable virtual keyboard
 ```bash
 rpm-ostree install git vim
 ```
 ### Libvirt, Qemu, KVM
 ```bash
 rpm-ostree install virt-manager libvirt
 systemctl enable --now libvirtd
 ```
 ### ROCM
 Credit to:
 <https://medium.com/@seancheo/running-generative-ai-on-amd-in-fedora-40-28aa3bebb187>
 Here are most of the ROCM packages that you'll need.
 Fedora is currently working on a ROCM meta package here:
 <https://packages.fedoraproject.org/pkgs/rocm/rocm/>
 ```bash
 rpm-ostree install \
 hipblas-devel \
 hipblaslt-devel \
 hipcc \
 hipcc-libomp-devel \
 hipcub-devel \
 hipfft-devel \
 hipfort-devel \
 hiprand-devel \
 hiprt-devel \
 hipsolver-devel \
 hipsparse-devel \
 rocalution-devel \
 rocblas-devel \
 rocfft-devel \
 rocm-clang-devel \
 rocm-clang-tools-extra-devel \
 rocm-cmake \
 rocm-comgr-devel \
 rocm-core-devel \
 rocm-hip-devel \
 rocm-libc++-devel \
 rocm-libc++-static \
 rocm-llvm-devel \
 rocm-omp-devel \
 rocm-runtime-devel \
 rocm-rpp-devel \
 rocm-smi-devel \
 rocminfo \
 rocdecode-devel \
 rocjpeg-devel \
 rocprim-devel \
 rocrand-devel \
 rocsolver-devel \
 rocsparse-devel \
 rocthrust-devel \
 roctracer-devel \
 miopen
 ```
 ### Ollama
 <https://ollama.com/download>
 Run the installation script as normal. Make sure you have the [ROCM](#rocm) drivers installed
 for GPU acceleration. The script *should* automatically pull the ROCM drivers after installing
 the base packages. If not, you should install them manually.
 For starting ollama as a service, follow the link below with one exception.
 <https://github.com/ollama/ollama/blob/main/docs/linux.md#adding-ollama-as-a-startup-service-recommended>
 ```bash
 # Change the recommended home dir to /var/home/ollama
 sudo useradd -r -s /bin/false -U -m -d /var/home/ollama ollama
 ```
 ## Network
 ### Hostname
 ```bash
 sudo hostnamectl hostname reesework16
 ```
 ### VLAN Setup with nmcli
 ```bash
 # VLAN 2
 nmcli conn
 export NMCLI_DEVICE=enp195s0f4u1u3
 nmcli connection add type VLAN con-name $NMCLI_DEVICE.2 dev $NMCLI_DEVICE id 2
 ```
 ## GPU Support in Distrobox
 Fix for `error="failed to check permission on /dev/kfd: open /dev/kfd: invalid argument"`
 ```bash
 # You have to create the video and render group to /etc/group before you can use it
 sudo grep -E '^video:' /usr/lib/group | sudo tee -a /etc/group
 sudo grep -E '^render:' /usr/lib/group | sudo tee -a /etc/group
 sudo usermod -aG video $USER
 sudo usermod -aG render $USER
 ```
 Logout and log back in to adopt new groups.
 ## Automatic Display Switching
 ```bash
 # List displays
 # Builtin: eDP-2
 # Roku: DP-13
 # Lenovo: DP-11
 kscreen-doctor -o
 # Put the builtin display back to normal
 kscreen-doctor \
 output.eDP-2.enable \
 output.eDP-2.position.0,0 \
 output.eDP-2.primary \
 output.eDP-2.mode.2560x1600@165 \
 output.eDP-2.scale.1.25
 # Show on Roku TV and Monitor
 kscreen-doctor \
 output.DP-11.enable \
 output.DP-11.position.0,0 \
 output.DP-11.mode.2560x1440@60 \
 output.DP-11.scale.1 \
 output.DP-13.enable \
 output.DP-13.position.0,0 \
 output.DP-13.mode.2560x1440@100 \
 output.DP-13.scale.1 \
 output.DP-13.primary \
 output.eDP-2.disable
 # Show only on the roku TV
 kscreen-doctor \
 output.DP-11.enable \
 output.DP-11.position.0,0 \
 output.DP-11.primary \
 output.DP-11.mode.3840x2160@60 \
 output.DP-11.scale.2 \
 output.DP-13.disable \
 output.eDP-2.disable
 # Mirror the builtin display to the roku tv
 kscreen-doctor \
 output.DP-11.enable \
 output.DP-11.position.0,0 \
 output.DP-11.primary \
 output.DP-11.mode.3840x2160@60 \
 output.DP-11.scale.2 \
 output.eDP-2.enable \
 output.eDP-2.mode.1920x1080@60 \
 output.eDP-2.scale.1 \
 output.eDP-2.position.0,0
 ```
@@ -4,6 +4,7 @@
  - [Installation](#installation)
  - [Setup SSH](#setup-ssh)
  - [DNF](#dnf)
    - [Version lock](#version-lock)
  - [Power Profiles with Tuned](#power-profiles-with-tuned)
  - [Fail2Ban](#fail2ban)
  - [BTRFS Parent Volumes](#btrfs-parent-volumes)
@@ -84,6 +85,7 @@
  - [Sound Devices](#sound-devices)
  - [Install ffmpegthumbnailer, remove totem](#install-ffmpegthumbnailer-remove-totem)
  - [Add compatibility for HEIC to mogrify](#add-compatibility-for-heic-to-mogrify)
  - [Redirecting ports for rootless services](#redirecting-ports-for-rootless-services)
 <https://docs.fedoraproject.org/en-US/fedora-server/installation/postinstallation-tasks/#_manage_system_updates>
@@ -127,6 +129,16 @@ dnf update --refresh -y
 dnf install -y git glances tmux vim python3-libdnf5 borgbackup tpm2-tools
 ```
 ### Version lock
 ```bash
 sudo dnf install dnf-plugin-versionlock
 sudo dnf list <package> --showduplicates
 sudo dnf versionlock add <package_name>-<version>.x86_64
 sudo dnf versionlock list
 sudo dnf versionlock delete <package_name>
 ```
 ## Power Profiles with Tuned
 1. `dnf install tuned`
@@ -1486,3 +1498,29 @@ sudo dnf install ffmpegthumbnailer
 ```bash
 sudo dnf install libheic-freeworld
 ```
 ## Redirecting ports for rootless services
 ```bash
 # ipv4
 sudo firewall-cmd \
 --permanent \
 --add-forward-port=port=80:proto=tcp:toport=8080
 sudo firewall-cmd --add-rich-rule='forward-port port=80 protocol=tcp to-port=8080
 ```
 Or in ansible
 ```yaml
 - name: Forward ports
    ansible.posix.firewalld:
    port_forward:
    - port: "{{ item.from }}"
        proto: tcp
        toport: "{{ item.to }}"
    permanent: true
    state: enabled
    immediate: true
    loop: "{{ forward_ports }}"
 ```
@@ -0,0 +1,59 @@
 # This is the sudoers file for the ops user to give limited access to specific commands without a password
 # Used for automations like ansible and maintenance
 # Hostname
 ops ALL=(root) NOPASSWD: /usr/bin/hostnamectl hostname *
 # Reboot
 ops ALL=(root) NOPASSWD: /usr/bin/reboot
 # System updates
 ops ALL=(root) NOPASSWD: /usr/bin/dnf update*, /usr/bin/dnf upgrade*, /usr/bin/dnf install *, /usr/bin/dnf remove *
 # Package cache management
 ops ALL=(root) NOPASSWD: /usr/bin/dnf clean all, /usr/bin/dnf makecache, /usr/bin/dnf autoremove
 # Repos
 ops ALL=(root) NOPASSWD: /usr/bin/dnf config-manager addrepo *
 # Service management
 ops ALL=(root) NOPASSWD: /usr/bin/systemctl start *, /usr/bin/systemctl restart *, /usr/bin/systemctl reload *, /usr/bin/systemctl enable *
 # Firewall
 ops ALL=(root) NOPASSWD: /usr/bin/firewall-cmd *
 # Ansible
 ops ALL=(root) NOPASSWD: /bin/sh -c echo BECOME-SUCCESS-* ; /usr/bin/python3 /home/ops/.ansible*
 # qemu-img
 ops ALL=(root) NOPASSWD: /usr/bin/qemu-img *
 # virsh
 ops ALL=(root) NOPASSWD: /usr/bin/virsh *
 # virt-install
 ops ALL=(root) NOPASSWD: /usr/bin/virt-install *
 # composer-cli
 ops ALL=(root) NOPASSWD: /usr/bin/composer-cli *
 # osbuild
 ops ALL=(root) NOPASSWD: /usr/local/bin/osbuild-auto.sh
 # getfacl
 ops ALL=(root) NOPASSWD: /usr/sbin/getfacl*
 # set default rwx sbin
 ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -d -m u\:ops\:rwx /usr/local/sbin
 # set existing rwx sbin
 ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -m u\:ops\:rwx /usr/local/sbin
 # set default rwx /srv
 ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -d -m u\:ops\:rwx /srv/*
 # set existing rwx /srv
 ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -m u\:ops\:rwx /srv/*
 # Start btrfs scrub
 ops ALL=(root) NOPASSWD: /usr/bin/btrfs scrub*
@@ -1,2 +0,0 @@
 [Network]
 Internal=true
@@ -1,14 +0,0 @@
 [Pod]
 Network=ai-internal.network
 # llama.cpp server
 PublishPort=8000:8000/tcp
 # llama.cpp embed
 PublishPort=8001:8001/tcp
 # llama.cpp instruct
 PublishPort=8002:8002/tcp
 # llama.cpp tool
 PublishPort=8003:8003/tcp
 # stable-diffusion.cpp gen
 PublishPort=1234:1234/tcp
 # stable-diffusion.cpp edit
 PublishPort=1235:1235/tcp
@@ -3,10 +3,6 @@
 - [Self Hosted AI Stack](#self-hosted-ai-stack)
  - [Notes](#notes)
    - [Podman Volume Locations](#podman-volume-locations)
    - [List of Internal Links](#list-of-internal-links)
  - [Quick Install](#quick-install)
    - [Text Stack](#text-stack)
    - [Image Stack](#image-stack)
  - [Setup](#setup)
    - [Create the AI user](#create-the-ai-user)
    - [Helper aliases](#helper-aliases)
@@ -36,16 +32,7 @@
  - [open-webui](#open-webui)
    - [Upgrade Open Webui](#upgrade-open-webui)
  - [lite-llm](#lite-llm)
  - [Install Services with Quadlets](#install-services-with-quadlets)
    - [API Keys](#api-keys)
    - [Internal and External Pods](#internal-and-external-pods)
    - [Llama CPP Server (Port 8000)](#llama-cpp-server-port-8000)
    - [Llama CPP Embedding Server (Port 8001)](#llama-cpp-embedding-server-port-8001)
    - [Llama CPP Instruct Server (Port 8002)](#llama-cpp-instruct-server-port-8002)
    - [Stable Diffusion CPP (Port 1234 and 1235)](#stable-diffusion-cpp-port-1234-and-1235)
    - [Open Webui (Port 8080)](#open-webui-port-8080)
  - [Install the update script](#install-the-update-script)
    - [Install Guest Open Webui with Start/Stop Services](#install-guest-open-webui-with-startstop-services)
  - [Benchmark Results](#benchmark-results)
  - [Testing with Curl](#testing-with-curl)
    - [OpenAI API](#openai-api)
@@ -77,33 +64,6 @@ hf-download ()
 `~/.local/share/containers/storage/volumes/`
 ### List of Internal Links
 - llama-cpp
 - llama-embed
 - llama-instruct
 - image-gen
 - image-edit
 - openwebui
 ## Quick Install
 ### Text Stack
 ```bash
 ansible-playbook \
 -i ansible/inventory.yaml \
 active/software_ai_stack/install_ai_text_stack.yaml
 ```
 ### Image Stack
 ```bash
 ansible-playbook \
 -i ansible/inventory.yaml \
 active/software_ai_stack/install_ai_image_stack.yaml
 ```
 ## Setup
 ### Create the AI user
@@ -186,8 +146,7 @@ rsync -av --progress /home/ai/models/ /srv/models/
 In my completely subjective opinion: 5 bit quant is usually the sweet spot for
 unsloth models. Q5_K_S is usually just fine.
-I usually download the F16 mmproj files. This is also completely subjective.
+I usually download the BF16 mmproj files. This is also completely subjective.
 BF16 is fine. F32 is overkill.
 #### Text models
@@ -391,20 +350,43 @@ podman run \
 --name llama-server-demo \
 --device=/dev/kfd \
 --device=/dev/dri \
-v /home/ai/models/text:/models:z \
+-v $(pwd):/models:z \
 -p 8010:8000 \
 --ipc host \
 localhost/llama-cpp-vulkan:latest \
 --host 0.0.0.0 \
 --port 8000 \
 -c 128000 \
 --perf \
 --n-gpu-layers all \
 --jinja \
--models-max 1 \
+--model /models/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
--models-dir /models \
+--mmproj /models/mmproj-F16.gguf \
--chat-template-kwargs '{"enable_thinking": false}' \
+--temp 1.0 \
-m /models/qwen3.5-35b-a3b
+--top-p 0.95 \
 --top-k 64 \
 --chat-template-kwargs '{"enable_thinking":true}'
 podman run \
 --rm \
 --name llama-server-demo \
 --device=/dev/kfd \
 --device=/dev/dri \
 -v $(pwd):/models:z \
 -p 8010:8000 \
 --ipc host \
 localhost/llama-cpp-vulkan:latest \
 --host 0.0.0.0 \
 --port 8000 \
 --perf \
 --n-gpu-layers all \
 --jinja \
 --model /models/Qwen3.5-9B-UD-Q4_K_XL.gguf \
 --mmproj /models/mmproj-BF16.gguf \
 --temp 0.6 \
 --top-p 0.95 \
 --top-k 20 \
 --min-p 0.00 \
 --chat-template-kwargs '{"enable_thinking":true}'
 ```
 Embedding models
@@ -595,96 +577,7 @@ podman run \
 -p 4000:4000
 ```
-## Install Services with Quadlets
+## Install the update script
 ### API Keys
 ```bash
 mkdir -p /home/ai/.llama-api
 touch /home/ai/.llama-api/keys.env
 chmod 600 /home/ai/.llama-api/keys.env
 vim /home/ai/.llama-api/keys.env
 LLAMA_API_KEY=
 # Generate keys and append to file, then comma separate the keys
 openssl rand -base64 48 >> keys.env
 openssl rand -base64 48 >> keys.env
 openssl rand -base64 48 >> keys.env
 ```
 ### Internal and External Pods
 These will be used to restrict internet access to our llama.cpp and
 stable-diffusion.cpp services while allowing the frontend services to
 communicate with those containers.
 ```bash
 scp -r active/software_ai_stack/ai-internal.* deskwork-ai:.config/containers/systemd/
 ssh deskwork-ai
 systemctl --user daemon-reload
 systemctl --user start ai-internal-pod.service
 ```
 ### Llama CPP Server (Port 8000)
 Installs the llama.cpp server to run our text models.
 ```bash
 scp -r active/software_ai_stack/llama-think.container deskwork-ai:.config/containers/systemd/
 ssh deskwork-ai
 systemctl --user daemon-reload
 systemctl --user restart ai-internal-pod.service
 ```
 ### Llama CPP Embedding Server (Port 8001)
 Installs the llama.cpp server to run our embedding models
 ```bash
 scp -r active/software_ai_stack/llama-embed.container deskwork-ai:.config/containers/systemd/
 ssh deskwork-ai
 systemctl --user daemon-reload
 systemctl --user restart ai-internal-pod.service
 ```
 ### Llama CPP Instruct Server (Port 8002)
 Installs the llama.cpp server to run a constant instruct (no thinking) model for quick replies
 ```bash
 scp -r active/software_ai_stack/llama-instruct.container deskwork-ai:.config/containers/systemd/
 ssh deskwork-ai
 systemctl --user daemon-reload
 systemctl --user restart ai-internal-pod.service
 ```
 ### Stable Diffusion CPP (Port 1234 and 1235)
 Installs the stable-diffusion.cpp server to run our image models.
 ```bash
 scp -r active/software_ai_stack/quadlets_stable_diffusion/* deskwork-ai:.config/containers/systemd/
 ssh deskwork-ai
 systemctl --user daemon-reload
 systemctl --user restart ai-internal-pod.service
 ```
 ### Open Webui (Port 8080)
 Installs the open webui frontend.
 ```bash
 scp -r active/software_ai_stack/quadlets_openwebui/* deskwork-ai:.config/containers/systemd/
 ssh deskwork-ai
 systemctl --user daemon-reload
 systemctl --user restart ai-external-pod.service
 ```
 Note, all services will be available at `host.containers.internal`. So llama.cpp
 will be up at `http://host.containers.internal:8000`.
 ### Install the update script
 ```bash
 # 1. Builds the latest llama.cpp and stable-diffusion.cpp
@@ -696,18 +589,6 @@ chmod +x update-script.sh
 ./update-script.sh
 ```
 ### Install Guest Open Webui with Start/Stop Services
 Optionally install a guest openwebui service.
 ```bash
 scp -r active/software_ai_stack/systemd/. deskwork-ai:.config/systemd/user/
 ssh deskwork-ai
 systemctl --user daemon-reload
 systemctl --user enable open-webui-guest-start.timer
 systemctl --user enable open-webui-guest-stop.timer
 ```
 ## Benchmark Results
 Benchmarks are run with [unsloth gpt-oss-20b Q8_0](https://huggingface.co/unsloth/gpt-oss-20b-GGUF/blob/main/gpt-oss-20b-Q8_0.gguf)
@@ -1,23 +0,0 @@
 - name: Create Deskwork AI Stack
  hosts: deskwork-ai
  tasks:
    - name: Create /home/ai/.config/containers/systemd
      ansible.builtin.file:
        path: /home/ai/.config/containers/systemd
        state: directory
        mode: "0755"
    - name: Copy Quadlets
      template:
        src: "{{ item }}"
        dest: "/home/ai/.config/containers/systemd/{{ item }}"
      loop:
        - ai-internal.network
        - ai-internal.pod
        - stable-diffusion-gen-server.container
        - stable-diffusion-edit-server.container
    - name: Reload and start the ai-internal-pod service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-internal-pod.service
        daemon_reload: true
        scope: user
@@ -1,24 +0,0 @@
 - name: Create Deskwork AI Stack
  hosts: deskwork-ai
  tasks:
    - name: Create /home/ai/.config/containers/systemd
      ansible.builtin.file:
        path: /home/ai/.config/containers/systemd
        state: directory
        mode: "0755"
    - name: Copy Quadlets
      template:
        src: "{{ item }}"
        dest: "/home/ai/.config/containers/systemd/{{ item }}"
      loop:
        - ai-internal.network
        - ai-internal.pod
        - llama-embed.container
        - llama-instruct.container
        - llama-tool.container
    - name: Reload and start the ai-internal-pod service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-internal-pod.service
        daemon_reload: true
        scope: user
@@ -1,22 +0,0 @@
 - name: Create Deskwork AI Stack
  hosts: driveripper-ai
  tasks:
    - name: Create /home/ai/.config/containers/systemd
      ansible.builtin.file:
        path: /home/ai/.config/containers/systemd
        state: directory
        mode: "0755"
    - name: Copy Quadlets
      template:
        src: "{{ item }}"
        dest: "/home/ai/.config/containers/systemd/{{ item }}"
      loop:
        - ai-internal.network
        - ai-internal.pod
        - llama-turbo.container
    - name: Reload and start the ai-internal-pod service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-internal-pod.service
        daemon_reload: true
        scope: user
@@ -1,46 +0,0 @@
 [Unit]
 Description=A Llama CPP Server For Embedding Models
 [Container]
 # Shared AI internal pod without internet access
 Pod=ai-internal.pod
 # Image is built locally via podman build
 Image=localhost/llama-cpp-vulkan:latest
 # Downloaded models volume
 Volume=/home/ai/models/embedding:/models:z
 # GPU Device
 AddDevice=/dev/kfd
 AddDevice=/dev/dri
 # Server command
 Exec=--port 8001 \
    -c 0 \
    -b 1024 \
    -ub 1024 \
    --perf \
    --n-gpu-layers all \
    --embedding \
    -m /models/emebeddinggemma-300m/embeddinggemma-300M-BF16.gguf \
    --alias embed
 # Health Check
 HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8001/health || exit 1
 HealthInterval=10s
 HealthRetries=3
 HealthStartPeriod=10s
 HealthTimeout=30s
 HealthOnFailure=kill
 # EnvironmentFile=/home/ai/.llama-api/keys.env
 [Service]
 Restart=always
 # Extend Timeout to allow time to pull the image
 TimeoutStartSec=900
 [Install]
 # Start by default on boot
 WantedBy=multi-user.target default.target
@@ -1,54 +0,0 @@
 [Unit]
 Description=A Llama CPP Server Running a Non-Reasoning Model
 [Container]
 # Shared AI internal pod without internet access
 Pod=ai-internal.pod
 # Image is built locally via podman build
 Image=localhost/llama-cpp-vulkan:latest
 # Downloaded models volume
 Volume=/home/ai/models/text:/models:z
 # GPU Device
 AddDevice=/dev/kfd
 AddDevice=/dev/dri
 # Server command
 Exec=--port 8002 \
    -c 262144 \
    -n 32768 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --presence-penalty 1.5 \
    --min-p 0.00 \
    --perf \
    --n-gpu-layers all \
    -fa on \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --jinja \
    --chat-template-kwargs '{"preserve_thinking": false, "enable_thinking": false}' \
    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q5_K_M.gguf \
    --mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F16.gguf \
    --alias instruct
 # Health Check
 HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8002/health || exit 1
 HealthInterval=10s
 HealthRetries=3
 HealthStartPeriod=10s
 HealthTimeout=30s
 HealthOnFailure=kill
 # EnvironmentFile=/home/ai/.llama-api/keys.env
 [Service]
 Restart=always
 # Extend Timeout to allow time to pull the image
 TimeoutStartSec=900
 [Install]
 # Start by default on boot
 WantedBy=multi-user.target default.target
@@ -1,57 +0,0 @@
 [Unit]
 Description=A Llama CPP Server Running a Reasoning Model
 [Container]
 # Shared AI internal pod without internet access
 Pod=ai-internal.pod
 # Image is built locally via podman build
 # latest-mtp is for mtp testing
 # latest is main branch
 Image=localhost/llama-cpp-vulkan:latest
 # Downloaded models volume
 Volume=/home/ai/models/text:/models:z
 # GPU Device
 AddDevice=/dev/kfd
 AddDevice=/dev/dri
 # Server command
 Exec=--port 8003 \
    -c 262144 \
    -n 32768 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0 \
    --presence-penalty 0.0 \
    --repeat-penalty 1.0 \
    --reasoning-budget 4096 \
    --perf \
    --n-gpu-layers all \
    -fa on \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --jinja \
    --chat-template-kwargs '{"preserve_thinking": false}' \
    --chat-template-file /models/qwen3.6-35b-a3b-mtp/chat_template.jinja \
    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q5_K_M.gguf \
    --mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F16.gguf \
    --alias tool
 # Health Check
 HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8003/health || exit 1
 HealthInterval=10s
 HealthRetries=3
 HealthStartPeriod=10s
 HealthTimeout=30s
 HealthOnFailure=kill
 [Service]
 Restart=always
 # Extend Timeout to allow time to pull the image
 TimeoutStartSec=900
 [Install]
 # Start by default on boot
 WantedBy=multi-user.target default.target
@@ -1,57 +0,0 @@
 [Unit]
 Description=A Llama CPP Server Running a Reasoning Model
 [Container]
 # Shared AI internal pod without internet access
 Pod=ai-internal.pod
 # Image is built locally via podman build
 # latest-mtp is for mtp testing
 # latest is main branch
 Image=localhost/llama-cpp-vulkan:latest
 # Downloaded models volume
 Volume=/home/ai/models/text:/models:z
 # GPU Device
 AddDevice=/dev/kfd
 AddDevice=/dev/dri
 # Server command
 Exec=--port 8000 \
    -c 262144 \
    -n 32768 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0 \
    --presence-penalty 0.0 \
    --repeat-penalty 1.0 \
    --reasoning-budget 4096 \
    --perf \
    --n-gpu-layers all \
    -fa on \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --jinja \
    --chat-template-kwargs '{"preserve_thinking": true}' \
    --chat-template-file /models/qwen3.6-35b-a3b-mtp/chat_template.jinja \
    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q5_K_M.gguf \
    --mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F16.gguf \
    --alias turbo
 # Health Check
 HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8000/health || exit 1
 HealthInterval=10s
 HealthRetries=3
 HealthStartPeriod=10s
 HealthTimeout=30s
 HealthOnFailure=kill
 [Service]
 Restart=always
 # Extend Timeout to allow time to pull the image
 TimeoutStartSec=900
 [Install]
 # Start by default on boot
 WantedBy=multi-user.target default.target
@@ -1,24 +0,0 @@
 [Unit]
 Description=An Open Webui Frontend for Local AI Services
 [Container]
 # Shared AI external pod
 PublishPort=8080:8080
 # Open Webui base image
 Image=ghcr.io/open-webui/open-webui:main
 # Nothing too complicated here. Open Webui will basically configure itself.
 Volume=open-webui-data:/app/backend/data
 # WEBUI_SECRET_KEY is required to prevent logout on Restart
 EnvironmentFile=/home/ai/.env/open-webui-env
 [Service]
 Restart=always
 # Extend Timeout to allow time to pull the image
 TimeoutStartSec=900
 [Install]
 # Start by default on boot
 WantedBy=multi-user.target default.target
@@ -1,44 +0,0 @@
 [Unit]
 Description=A Stable Diffusion CPP Server for Editing Images
 [Container]
 # Shared AI internal pod without internet access
 Pod=ai-internal.pod
 # Vulkan image for AMD GPU
 Image=localhost/stable-diffusion-cpp:latest
 # Shared models directory
 Volume=/home/ai/models:/models:z
 # GPU Device
 AddDevice=/dev/kfd
 AddDevice=/dev/dri
 # Override entrypoint to use server
 Entrypoint=/sd-server
 # Server args
 Exec=-l 0.0.0.0 \
    --listen-port 1235 \
    --diffusion-model /models/image/flux2-klein/flux-2-klein-9b-Q8_0.gguf \
    --vae /models/image/flux2-klein/ae.safetensors \
    --llm /models/image/flux2-klein/Qwen3-8B-Q4_K_M.gguf \
    -v \
    --sampling-method euler \
    --cfg-scale 1.0 \
    --vae-conv-direct \
    --diffusion-conv-direct \
    --fa \
    --mmap \
    --seed -1 \
    --steps 5
 [Service]
 Restart=always
 # Extend Timeout to allow time to pull the image
 TimeoutStartSec=900
 [Install]
 # Start by default on boot
 WantedBy=multi-user.target default.target
@@ -1,43 +0,0 @@
 [Unit]
 Description=A Stable Diffusion CPP Server for Generating Images
 [Container]
 # Shared AI internal pod without internet access
 Pod=ai-internal.pod
 # Vulkan image for AMD GPU
 Image=localhost/stable-diffusion-cpp:latest
 # Shared models directory
 Volume=/home/ai/models:/models:z
 # GPU Device
 AddDevice=/dev/kfd
 AddDevice=/dev/dri
 # Override entrypoint to use server
 Entrypoint=/sd-server
 # Server args
 Exec=-l 0.0.0.0 \
    --listen-port 1234 \
    --diffusion-model /models/image/z-turbo/z_image_turbo-Q8_0.gguf \
    --vae /models/image/z-turbo/ae.safetensors  \
    --llm /models/image/z-turbo/Qwen3-4B-Instruct-2507-Q4_K_M.gguf \
    -v \
    --cfg-scale 1.0 \
    --vae-conv-direct \
    --diffusion-conv-direct \
    --fa \
    --mmap \
    --seed -1 \
    --steps 8
 [Service]
 Restart=always
 # Extend Timeout to allow time to pull the image
 TimeoutStartSec=900
 [Install]
 # Start by default on boot
 WantedBy=multi-user.target default.target
@@ -1,8 +0,0 @@
 [Unit]
 Description=Start open-webui-guest on schedule
 Wants=open-webui-guest.service
 After=open-webui-guest.service
 [Service]
 Type=oneshot
 ExecStart=/bin/systemctl --user start open-webui-guest.service
@@ -1,10 +0,0 @@
 [Unit]
 Description=Timer to start open-webui-guest at 08:00 Mon‑Fri
 [Timer]
 OnCalendar=Mon..Fri *-*-* 08:00:00
 Persistent=true
 Unit=open-webui-guest-start.service
 [Install]
 WantedBy=timers.target
@@ -1,7 +0,0 @@
 [Unit]
 Description=Stop open-webui-guest on schedule
 After=open-webui-guest.service
 [Service]
 Type=oneshot
 ExecStart=/bin/systemctl --user stop open-webui-guest.service
@@ -1,10 +0,0 @@
 [Unit]
 Description=Timer to stop open-webui-guest at 18:00 Mon‑Fri
 [Timer]
 OnCalendar=Mon..Fri *-*-* 18:00:00
 Persistent=true
 Unit=open-webui-guest-stop.service
 [Install]
 WantedBy=timers.target
@@ -0,0 +1,12 @@
 # Flatpak
 ## Installing a specific version
 ```bash
 # List the available commits
 flatpak remote-info --log flathub com.orcaslicer.OrcaSlicer
 # Install the commit version you want
 # Orca Slicer, for example
 sudo flatpak update --commit=acd5ab796f5640645938194da3a6822b3699dcf41c61059bfbd6accb798ebc7b com.orcaslicer.OrcaSlicer
 ```
@@ -0,0 +1,5 @@
 # Konsole
 ## Tips
 - To open new tabs with the same profile as the current tab, assign the "clone tab" keyboard shortcut to <ctrl>+<shift>+<T>.
@@ -0,0 +1,79 @@
 name = "fedora44-base"
 description = "Fedora Base Installation"
 version = "0.2.3"
 distro = "fedora-44"
 modules = []
 groups = []
 [customizations]
 hostname = "fedora44-base"
 [customizations.timezone]
 timezone = "America/New_York"
 [[customizations.disk.partitions]]
 type = "plain"
 label = "root"
 mountpoint = "/"
 fs_type = "ext4"
 minsize = "128 GiB"
 [customizations.firewall.services]
 enabled = ["ssh", "cockpit"]
 [customizations.services]
 enabled = ["sshd", "cockpit.socket"]
 [[packages]]
 name = "bash-completion"
 version = "*"
 [[packages]]
 name = "tmux"
 version = "*"
 [[packages]]
 name = "openssh-server"
 version = "*"
 [[packages]]
 name = "vim"
 version = "*"
 [[packages]]
 name = "git"
 version = "*"
 [[packages]]
 name = "cockpit"
 version = "*"
 [[packages]]
 name = "borgbackup"
 version = "*"
 [[packages]]
 name = "policycoreutils-python-utils"
 version = "*"
 [[packages]]
 name = "systemd-container"
 version = "*"
 [[customizations.files]]
 path = "/root/.inputrc"
 mode = "0644"
 user = "root"
 group = "root"
 data = """
 "\\C-h": backward-kill-word
 """
 [[customizations.files]]
 path = "/root/.ssh/authorized_keys"
 mode = "0600"
 user = "root"
 group = "root"
 data = """
 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGQa781Qj8mNlUdRquFFqg0O2ornG9SBHe705y4+1vPI ssh@ducoterra.net
 """
@@ -29,6 +29,8 @@ dnf install -y osbuild-composer composer-cli
 systemctl enable --now osbuild-composer.socket
 # Optional: add your user to the weldr group
 # On fedora kinoite (atomic) you'll need to add the group manually for now
 grep -E '^weldr:' /usr/lib/group | sudo tee -a /etc/group
 sudo usermod -aG weldr $USER
 # Optional: cockpit dependency
@@ -0,0 +1,139 @@
 # Pi AI Harness
 ## Prereqs
 I would recommend using `pass` to store your api key. This is installed with `dnf install pass` on Fedora systems.
 After installing pass, update your `~/.gnupg/gpg-agent.conf` to set the timeout to max so automated processes can use the values in your password store:
 ```conf
 default-cache-ttl 2147483647
 max-cache-ttl 2147483647
 ```
 Then add a `pass.sh` to you `~/.bashrc.d` that looks like
 ```bash
 pass unlock
 ```
 That way every time you log in your pass keyring is unlocked for your session so automations can use it.
 ## Setup
 This is my ~/.pi/agent/models.json
 ```json
 {
  "providers": {
    "reeseapps-aipi": {
      "baseUrl": "https://aipi.reeseapps.com/v1",
      "api": "openai-completions",
      "apiKey": "!pass aipi/dev-token",
      "models": [
        {
          "id": "juggernaut",
          "input": ["text", "image"],
          "reasoning": true,
          "contextWindow": 262144,
          "maxTokens": 32768,
          "thinkingLevelMap": {
            "off": "none",
            "minimal": null,
            "low": "low",
            "medium": "medium",
            "high": null,
            "xhigh": "xhigh",
            "max": null
          }
        },
        {
          "id": "turbo",
          "input": ["text", "image"],
          "reasoning": true,
          "contextWindow": 262144,
          "maxTokens": 32768,
           "thinkingLevelMap": {
            "off": "none",
            "minimal": null,
            "low": "low",
            "medium": "medium",
            "high": null,
            "xhigh": "xhigh",
            "max": null
          }
        }
      ]
    }
  }
 }
 ```
 This is my ~/.pi/agent/settings.json
 ```json
 {
  "lastChangelogVersion": "1.3.0",
  "theme": "dark/dark",
  "defaultProvider": "reeseapps-aipi",
  "defaultModel": "turbo",
  "httpIdleTimeoutMs": 0,
  "enableInstallTelemetry": false,
  "doubleEscapeAction": "tree",
  "images": {
    "autoResize": false
  },
  "editorPaddingX": 0,
  "defaultThinkingLevel": "xhigh",
  "hideThinkingBlock": false,
  "thinkingBudgets": {
    "minimal": 1024,
    "low": 4096,
    "medium": 10240,
    "xhigh": 65536
  },
  "steeringMode": "all"
 }
 ```
 ## Installing for sysadmin work
 I would absolutely recommend installing pi under a special user (I call mine
 `ops`) with limited sudoers access that you tightly control. Pair that with the
 following AGENTS.md at `/home/ops/AGENTS.md`:
 ```text
 # AGENTS.md
 ## Sudo policy
 The `ops` user on this machine has **limited sudoers capabilities**. The exact
 allowlist is discoverable at any time with:
 sudo -l
 Allowed (NOPASSWD) commands currently include: `hostnamectl hostname`, `reboot`,
 `dnf update/upgrade/install/remove/clean/makecache/autoremove`,
 `dnf config-manager addrepo`, `systemctl start/restart/reload/enable`,
 `firewall-cmd`, the ansible shim, `qemu-img`, `virsh`, `virt-install`,
 `composer-cli`, `getfacl`, specific `setfacl` invocations,
 `/usr/local/sbin/osbuild-auto.sh`, and `btrfs scrub`.
 ## Rule for agents
 If you (the agent) believe you need a permission that is **not** in the
 `sudo -l` allowlist — i.e., a command fails with a sudo permission error, or a
 task requires root access you do not have — you must:
 1. **Stop** and do not attempt workarounds that bypass sudo (no password
   guessing, no escalating via other users, no trying to write to root-owned
   paths without permission).
 2. **Ask the human** to update sudoers with the specific command(s) you need,
   stating exactly which command(s) and arguments should be added so the user
   can review and approve them. Requests must be correct and copy-pastable,
   using full command paths in standard sudoers format, e.g.
   `ops ALL=(root) NOPASSWD: /usr/sbin/btrfs scrub*`
 Do not proceed with root-only operations until the user has confirmed the
 sudoers entries have been added.
 ```
@@ -0,0 +1,48 @@
 # Podman
 ## Networks
 ```bash
 # Create a podman network to group shared containers
 podman network create test1
 # Create an iperf3 container running on that network
 podman run -d --name iperf3 --network test1 docker.io/networkstatic/iperf3 -s
 # A container running in the same network can reference another pod by its name
 podman run --rm -it --network test1 docker.io/networkstatic/iperf3 -c iperf3
 # A container outside the created network can't talk to other containers
 # This will fail
 podman run --rm -it docker.io/networkstatic/iperf3 -c iperf3
 ```
 ## Healthchecks
 TCP Healthchecks can be accomplished with netcat (netcat-openbsd)
 ```bash
 podman run -d --name iperf3 \
 --health-cmd='CMD-SHELL iperf3 -n 1 -c localhost' \
 --health-interval=10s --health-timeout=5s --health-retries=3 \
 docker.io/networkstatic/iperf3 iperf3 -s
 ```
 HTTP Healthchecks can be accomplished with curl
 ```bash
 podman run -d --name nginx \
 --health-cmd='CMD-SHELL curl --fail http://127.0.0.1:80 || exit 1' \
 --health-interval=10s --health-timeout=5s --health-retries=3 \
 docker.io/nginx
 ```
 Check healthchecks with
 ```bash
 podman --log-level debug healthcheck run <container>
 ```
 ## Volume Import and Export
 ```bash
 podman volume export myvol --output myvol.tar
 podman volume import myvol test.tar
 ```
@@ -30,11 +30,17 @@ sudo smbpasswd -a sambauser
 ## Create a SMB Share
 ```bash
 # Add the share user
 sudo useradd -r -M -s /usr/sbin/nologin sambauser
 sudo smbpasswd -a sambauser
 # Create share
-mkdir /srv/smb/sambauser
+sudo mkdir /srv/smb/sambauser
 sudo chown -R sambauser:sambauser /srv/smb/sambauser
 sudo chmod -R 2770 /srv/samba/sambauser
 # Set proper selinux labels for samba
-sudo semanage fcontext --add --type "samba_share_t" "/srv/smb(/.*)?"
+sudo semanage fcontext --add --type "samba_share_t" "/srv/smb/sambauser"
 # Run restorecon at the root of the btrfs subvolume
 sudo restorecon -FRv /srv
@@ -0,0 +1,44 @@
 FROM registry.fedoraproject.org/fedora-toolbox:44
 COPY vscodium.repo /etc/yum.repos.d/vscodium.repo
 RUN dnf install -y \
 vim git btop \
 virt-manager virt-install \
 podman podman-compose \
 wl-clipboard \
 jq yq rg \
 kubectl helm \
 pinentry \
 pandoc \
 iperf3 \
 nethogs \
 ansible \
 tmux \
 wine \
 unzip \
 ImageMagick \
 go \
 rust rustup \
 git git-lfs \
 net-tools \
 texlive-latex texlive-scheme-full \
 pwgen \
 reptyr \
 netcat \
 p7zip \
 make \
 gcc \
 codium \
 pass \
 stress-ng \
 fuse \
 bind-utils \
 nmap \
 poppler \
 nodejs24 \
 kitty \
 scrot \
 xorg-x11-server-Xvfb \
 ffmpeg \
 portaudio
@@ -0,0 +1,7 @@
 # Toolbox
 ## How to Use
 ```bash
 podman build -t localhost/fedora-toolbox:44 active/software_toolbx
 ```
@@ -0,0 +1,9 @@
 [gitlab.com_paulcarroty_vscodium_repo]
 name=gitlab.com_paulcarroty_vscodium_repo
 baseurl=https://paulcarroty.gitlab.io/vscodium-deb-rpm-repo/rpms/
 enabled=1
 gpgcheck=1
 repo_gpgcheck=1
 gpgkey=https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg
 metadata_expire=1h
@@ -40,7 +40,6 @@ Virtual Machine Management
 <https://libvirt.org/uri.html#ssh-transport>
 ### Configuring Aliases
 1. Edit `~/.config/libvirt/libvirt.conf`
 2. Add your aliases
@@ -0,0 +1,281 @@
 # vLLM Radiance on R9700
 - [vLLM Radiance on R9700](#vllm-radiance-on-r9700)
  - [Overview](#overview)
  - [Deployment Topology](#deployment-topology)
  - [Capacity Analysis (Single 32 GiB R9700)](#capacity-analysis-single-32-gib-r9700)
  - [FP8-KV Calibration: The Process](#fp8-kv-calibration-the-process)
    - [Why: The Uncalibrated Scale Problem](#why-the-uncalibrated-scale-problem)
    - [Step 1: Host Prep](#step-1-host-prep)
    - [Step 2: Stop the Service](#step-2-stop-the-service)
    - [Step 3: Run the Calibration](#step-3-run-the-calibration)
    - [Step 4: Verify the Artifact (CPU-only)](#step-4-verify-the-artifact-cpu-only)
    - [Step 5: Edit the Quadlet](#step-5-edit-the-quadlet)
    - [Step 6: Deploy via Ansible](#step-6-deploy-via-ansible)
    - [Step 7: Confirm at Runtime](#step-7-confirm-at-runtime)
  - [What Calibration Actually Does (Benefits)](#what-calibration-actually-does-benefits)
  - [Operational Notes](#operational-notes)
  - [Remaining Work: Matched-Control Quality Gate](#remaining-work-matched-control-quality-gate)
 ## Overview
 `vllm-radiance` is a qualified downstream vLLM distribution for the AMD Radeon AI
 PRO R9700 (gfx1201). The upstream-qualified environment is dual-GPU TP2; this
 deployment runs the single-GPU (TP1) configuration on `driveripper-ai` serving
 the AMD Quark MXFP4 27B checkpoint (W4A8: 4-bit weights, dynamic FP8
 activation quantization over RDNA4's native FP8 WMMA path).
 Served as `juggernaut` on the `ai-internal` network, port 8000.
 ## Deployment Topology
 | Thing              | Location                                                                                                                                                                                   |
 | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
 | Quadlet (source of |
 | truth)             | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/quadlets/vllm-radiance.container`                                                                                                    |
 | Ansible playbook   | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/playbook.yaml` (run from `~/Deployments`: `ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml`) |
 | Deployed quadlet   | `driveripper-ai:/home/ai/.config/containers/systemd/ai-vllm-radiance.container`                                                                                                            |
 | systemd unit       | **user** unit of `ai` (`systemctl --user status ai-vllm-radiance` — not in system scope)                                                                                                   |
 | Image              | `docker.io/magiccodingman/vllm-radiance:1.0.387`                                                                                                                                           |
 | Model              | `driveripper-ai:/home/ai/vllm-radiance-models/qwen3.8-27b-quark-awq-mxfp4` → `/models` (18.44 GiB on disk, 17.95 GiB in GPU)                                                               |
 | Compile caches     | `driveripper-ai:/home/ai/vllm-radiance-cache` → `/cache`                                                                                                                                   |
 | FP8-KV sidecar     | `driveripper-ai:/home/ai/vllm-radiance-calibration` → `/calibration` (ro)                                                                                                                  |
 The playbook copies the quadlet to `~/.config/containers/systemd/` and does
 `systemctl --user restart ai-vllm-radiance` with daemon-reload. **Any edit to
 the local quadlet lands on the next playbook run** — there is no separate
 "apply" step.
 Key quadlet settings (as of the calibration deploy, 2026-10-11):
 ```
 --tensor-parallel-size=1  --max-model-len=154000  --max-num-seqs=2
 --max-num-batched-tokens=4096  --kv-cache-dtype=fp8  --attention-backend=R4D
 --enable-prefix-caching  --mamba-cache-mode=align  --no-async-scheduling
 --gpu-memory-utilization=0.98
 Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
 Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
 HealthStartPeriod=600s  (health check kills on 3x10s failures after the start period)
 ```
 ## Capacity Analysis (Single 32 GiB R9700)
 Measured from the server startup logs (the authoritative numbers — do not
 re-derive from weight sizes alone; the GDN hybrid architecture makes
 per-token KV much cheaper than a plain transformer):
 ```
 Model loading took 17.95 GiB
 Actual usage: 19.38 GiB consumed (weights + non-torch)
             + 2.02 GiB peak activation
             + 0.41 GiB CUDA graphs
 Available KV cache memory: 9.82 GiB  (10.14 GiB at the 154K config)
 GPU KV cache size: 308,847 tokens   (309,466 at the 154K config)
 ```
 Budget math (KV budget ≈ 309K tokens):
 | Config                | KV needed | Fits?                                        |
 | --------------------- | --------- | -------------------------------------------- |
 | 150K × 2              | 300,000   | yes, ~3% headroom                            |
 | **154K × 2 (chosen)** | 308,000   | yes — `2.01x` per vLLM, ~1.5K token headroom |
 | 256K × 1              | 262,144   | yes                                          |
 | 256K × 2              | 524,288   | **no** — `1.18x` max concurrency             |
 The per-request C2 ceiling is `KV_budget / 2 ≈ 154,423` tokens; `--max-model-len=154000`
 sits exactly at it. A second full 256K request is only reachable via CPU KV
 offload (`RADIANCE_KV_OFFLOADING_SIZE`), which would need to roughly double the
 token budget. `MAX_NUM_SEQS` is an admission ceiling, not a promise that every
 admitted request can simultaneously reach `MAX_MODEL_LEN`.
 ## FP8-KV Calibration: The Process
 ### Why: The Uncalibrated Scale Problem
 The server runs `--kv-cache-dtype=fp8` with FP8 attention, but the checkpoint
 carries no q/k/v/prob scales. vLLM therefore used placeholder scale **1.0**,
 which the startup log flags:
 ```
 WARNING ... Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8
 attention. This may cause accuracy issues.
 ```
 Radiance ships `radiance_kv_calibration.py` (in the image, also in the
 `vllm-radiance` repo) plus the `RADIANCE_FP8_KV_SCALES` runtime loader to fix
 this with a small immutable sidecar. Full reference: `docs/FP8_KV_TUNABLEOP.md`
 in the vllm-radiance repo.
 ### Step 1: Host Prep
 The calibration dir did not exist and the vllm-radiance repo is not on
 `driveripper-ai`, so copy only the corpus:
 ```bash
 ssh driveripper-ai "mkdir -p /home/ai/vllm-radiance-calibration"
 scp ~/Projects/Experimental/vllm-radiance/benchmarks/fixtures/fp8-kv-calibration.jsonl \
    driveripper-ai:/home/ai/vllm-radiance-calibration/
 ```
 (Optionally extend the JSONL with representative traffic first — each line is
 `{"text": ...}` or `{"messages": [...], "tools": [...]}` plus optional `repeat`.)
 ### Step 2: Stop the Service
 Calibration needs the GPU released; the server holds ~97% of VRAM.
 ```bash
 ssh driveripper-ai "systemctl --user stop ai-vllm-radiance"
 ```
 ### Step 3: Run the Calibration
 An eager, BF16 full-precision-KV, `max_tokens=1` prefill sweep with PyTorch
 hooks on every full-attention layer (no source edits — public worker-extension
 API only). Budget ~10–30 min:
 ```bash
 ssh driveripper-ai "podman run --rm --ipc=host \
  --device=/dev/kfd --device=/dev/dri \
  --ulimit memlock=-1:-1 \
  -e HIP_VISIBLE_DEVICES=0 \
  -e RADIANCE_MXFP4=1 -e RADIANCE_MXFP4_W4A8=1 -e RADIANCE_MXFP4_W4A8_MIN_M=0 \
  -v /home/ai/vllm-radiance-models:/models:ro \
  -v /home/ai/vllm-radiance-calibration:/calibration:rw \
  -v /home/ai/vllm-radiance-calibration/fp8-kv-calibration.jsonl:/fixtures/fp8-kv-calibration.jsonl:ro \
  --entrypoint python \
  docker.io/magiccodingman/vllm-radiance:1.0.387 \
  -m radiance_kv_calibration calibrate \
  --model /models/qwen3.8-27b-quark-awq-mxfp4 \
  --quantization auto \
  --corpus /fixtures/fp8-kv-calibration.jsonl \
  --output /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
  --tensor-parallel-size 1 \
  --max-model-len 16384 \
  --batch-size 4 \
  --seed 17"
 ```
 Success line: `[radiance] wrote immutable FP8-KV sidecar: /calibration/...`.
 Outputs are **immutable** — a re-run needs a fresh filename. The sidecar and
 its sibling `.manifest.json` (sha256 + full source-checkpoint identity +
 per-layer observed amax + corpus/seed/software provenance) land in the host dir.
 ### Step 4: Verify the Artifact (CPU-only)
 No engine start — checks manifest schema, sidecar sha256, that every tensor is
 a positive finite scalar `*.q|k|v|prob_scale`, and (with `--model`) that it was
 calibrated against this exact checkpoint:
 ```bash
 ssh driveripper-ai "podman run --rm \
  -v /home/ai/vllm-radiance-models:/models:ro \
  -v /home/ai/vllm-radiance-calibration:/calibration:ro \
  --entrypoint python docker.io/magiccodingman/vllm-radiance:1.0.387 \
  -m radiance_kv_calibration verify \
  /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
  --model /models/qwen3.8-27b-quark-awq-mxfp4"
 ```
 ### Step 5: Edit the Quadlet
 Three lines in the local quadlet (source of truth), then nothing else:
 ```diff
 Volume=/home/ai/vllm-radiance-models:/models:ro,z
 Volume=/home/ai/vllm-radiance-cache:/cache:z
 +Volume=/home/ai/vllm-radiance-calibration:/calibration:ro,z
 ...
 Environment=WEIGHT_QUANTIZATION=auto
 +Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
 +Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
 ```
 ### Step 6: Deploy via Ansible
 ```bash
 cd ~/Deployments
 ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml
 ```
 ### Step 7: Confirm at Runtime
 All three must be present/absent:
 ```
 FP8-KV scales = /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors   (banner; was "checkpoint/default")
 Loading Radiance FP8-KV calibration sidecar ... (verify=True)          (checksum + identity passed at load)
 "uncalibrated" warning: gone
 /health: OK
 ```
 ## What Calibration Actually Does (Benefits)
 **The problem.** With placeholder scale 1.0, FP8 values are stored as
 `value / 1.0`. The calibration hooks measured the model's real pre-quantization
 activations: peak **23.0 (K, layer 31)** and **59.75 (V, layer 43)** across the
 16 full-attention layers. FP8 E4M3's max magnitude is 448 — so the raw values
 only used **5–13% of the representable range**. E4M3 has just 3 mantissa bits;
 when values cluster near the bottom of the range, the quantization steps
 between representable values are ~8–20× coarser than necessary. That is rounding
 noise injected into every attention score and KV lookup — worst for
 long-context retrieval, where KV detail is exactly what the task depends on.
 **The fix.** The calibration writes per-layer
 `scale = observed_amax × 1.05 / 448` (5% headroom for excursions the corpus
 didn't hit) and `prob_scale = 1.05 / 448` (softmax probabilities are bounded by
 1). At load time vLLM divides each tensor by that scale before FP8 conversion,
 so observed maxima land at ~95% of the FP8 range — the full 3 mantissa bits do
 useful work. Result: **~7.5× finer quantization steps on V, ~19.6× finer on K**
 (≈ +2.9 / +4.3 effective mantissa bits), and the "may cause accuracy issues"
 warning is resolved.
 **Provenance & safety.** The sidecar is immutable (sha256 in the manifest),
 bound to this exact checkpoint, and re-verified at every startup by
 `VERIFY=1` — a changed model makes the server **fail closed** instead of
 silently using wrong scales.
 **What it does NOT do.** No throughput change (expected neutral — a big TPS
 delta means something is wrong, not a win). No new kernels. No weight-path
 change. Purely a precision fix for the FP8 attention/KV arithmetic.
 ## Operational Notes
 - **Sidecar is per-checkpoint.** Any fine-tune or re-download changes
  activation ranges — re-run calibration (fresh filename) rather than reusing
  scales by architecture name, and never set `VERIFY=0` as a deployment
  setting (it exists only to diagnose a deliberate mismatch).
 - **Startup re-hash cost.** The model dir on `driveripper-ai` has no HF
  download metadata, so the manifest bound full weight sha256s — every startup
  re-hashes all 18.44 GiB of weights for identity verification. Adds a few
  minutes on cold restarts; worked fine in practice.
 - **Health check.** `HealthStartPeriod=600s` was chosen deliberately: the
  deployed (pre-calibration) quadlet had the health check disabled at 30s start
  period — a cold kernel-compile startup would fail 3×10s of health checks and
  `HealthOnFailure=kill` + `Restart=always` loops it.
 - **Capacity ceiling.** 154K×2 runs the KV tier to ~99.6% at full context by
  design. Normal mixed traffic stays well below. If you need two full 256K
  concurrent requests, that is a CPU-offload decision, not a flag tweak.
 - **Corpus.** The 2026-10-10 sidecar used the stock 8-prompt corpus (tool
  schema, code, 4.9K-token retrieval, reasoning, multilingual, math, structured
  data). If the real workload is dominated by content the stock corpus doesn't
  cover, re-calibrate with an extended corpus. Calibration was run at
  16K `max_model_len`; amax is largely context-independent, but if long-context
  numerics ever look off, that's the first thing to re-run with a longer
  `--max-model-len`.
 ## Remaining Work: Matched-Control Quality Gate
 Per `docs/FP8_KV_TUNABLEOP.md` and the project's no-claim-without-control
 policy: compare **calibrated vs default-scale** runs with identical image,
 target, prompts, seeds, sampling, graphs, and concurrency:
 - fixed-prompt greedy outputs (meaningful agreement)
 - sampled tool-schema checks
 - long-context retrieval
 - vision inputs if vision is deployed
 - BetterBench / sustained c1/c2/c4/c8 for the throughput-neutral claim
 Success criterion: **fidelity at-or-above the uncalibrated baseline with ~neutral
 throughput.** Only after that gate is this a qualified change, not just a
 plausible one.
@@ -1,24 +1,19 @@
 import json
 import os
 import smtplib
 import ssl
 import subprocess
 from email.message import EmailMessage
 from pathlib import Path
 from typing import Iterable, TypedDict, cast
 import requests
 from dotenv import dotenv_values, load_dotenv
 from openai import OpenAI
 from openai.types.chat import ChatCompletionMessageParam, ChatCompletionToolUnionParam
-class AWS_SES_DOTENV(TypedDict):
+class NTFY_DOTENV(TypedDict):
-    USER: str
+    TOKEN: str
    PASSWORD: str
    ENDPOINT: str
    TLS_PORT: str
    SENDER: str
    RECEIVER: str
 class ToolCallController:
@@ -209,39 +204,30 @@ def smartctl(device_path: str) -> str:
    return output
-def load_ses_creds() -> AWS_SES_DOTENV:
+def load_ntfy_creds() -> NTFY_DOTENV:
    ses_dotenv_location = Path(os.getenv("HOME", "/root"), ".env/aws_ses")
    print(f"Loading env from {ses_dotenv_location}")
    raw_values = dotenv_values(ses_dotenv_location)
    if raw_values:
-        aws_ses_config = cast(AWS_SES_DOTENV, raw_values)
+        ntfy_config = cast(NTFY_DOTENV, raw_values)
        # print(f"AWS SES Credentials loaded: {aws_ses_config}")
-        return aws_ses_config
+        return ntfy_config
-    print("No email credentials supplied. Exiting.")
+    print("No ntfy credentials supplied. Exiting.")
    exit(1)
 def alert_user(message: str) -> str:
-    ses_config = load_ses_creds()
+    ntfy_creds = load_ntfy_creds()
    port = int(ses_config["TLS_PORT"])
    user = ses_config["USER"]
    password = ses_config["PASSWORD"]
    sender = ses_config["SENDER"]
    receiver = ses_config["RECEIVER"]
-    # Create a secure SSL context
+    requests.post(
-    context = ssl.create_default_context()
+        "https://ntfy.reeseapps.com/servers",
-
+        data=message,
-    msg = EmailMessage()
+        headers={
-    msg["Subject"] = "Agent Disk Report"
+            "Authorization": f"Bearer {ntfy_creds['TOKEN']}",
-    msg["From"] = sender
+            "Title": "Agent Disk Report",
-    msg["To"] = receiver
+            "Priority": "default",
-    msg.set_content(message)
+        },
-
+    )
    with smtplib.SMTP_SSL(ses_config["ENDPOINT"], port, context=context) as server:
        server.login(user, password)
        result = server.send_message(msg)
        print(result)
    return "Sent."
@@ -322,13 +308,14 @@ def run_conversation(user_message: str, max_tool_calls=10):
 if __name__ == "__main__":
    load_dotenv()
    api_key = os.getenv("OPENAI_API_KEY", "")
-    client = OpenAI(base_url="https://llama-think.reeselink.com", api_key=api_key)
+    client = OpenAI(base_url="https://aipi.reeseapps.com", api_key=api_key)
    # Example usage
    print(
        run_conversation(
            "Check the btrfs pools on this system. Take the appropriate action if any pools aren't "
            "healthy. Don't run scrubs unless necessary. Also check the btrfs pool space and report "
            "if any are getting full. At the very end of performing your checks send a single, "
-            "concise message to the user explaining what you did and what concerns you might have."
+            "concise message to the user via the alert_user tool explaining what you did and "
            "what concerns you might have."
        )
    )
@@ -1,5 +0,0 @@
 # Proxy Automation
 1. Create or update the reeselink.com address in Unifi
 2. Create or update the entry in the ddns values
 3. Create or update the entry in the caddy values
@@ -1 +0,0 @@
 3.13
@@ -1,9 +0,0 @@
 # Update
 Updates every server in the fleet
 ## Usage
 ```bash
 uv run automations/updates/main.py
 ```
@@ -1,64 +0,0 @@
 import subprocess
 import time
 import yaml
 from tqdm import tqdm
 def main():
    print("Retrieving hosts")
    with open("ansible/inventory.yaml", "r") as f:
        all_hosts = yaml.load(f, yaml.SafeLoader)
    fedora_hosts = all_hosts["fedora"]["hosts"].keys()
    with open("update.log", "w") as log_file:
        for _, host in enumerate(tqdm(fedora_hosts, desc="Running system updates")):
            log_file.write(f"Updating {host}\n")
            log_file.flush()
            try:
                subprocess.run(
                    ["ssh", host, "dnf", "upgrade", "-y"],
                    stdout=log_file,
                    stderr=log_file,
                    check=True,
                )
            except Exception as e:
                log_file.write(f"Couldn't connect to {host}. Skipping...\n")
                continue
            log_file.flush()
            log_file.write(f"Rebooting {host}\n")
            log_file.flush()
            subprocess.run(
                ["ssh", host, "reboot"],
                stdout=log_file,
                stderr=log_file,
                check=True,
            )
            time.sleep(5)  # wait for reboot to take effect
            booted = False
            max_attempts = 10
            cur_attempts = 0
            while max_attempts > cur_attempts and not booted:
                try:
                    subprocess.run(
                        ["ssh", host, "echo"],
                        stdout=log_file,
                        stderr=log_file,
                        check=True,
                        timeout=2,
                    )
                    log_file.write(f"{host} booted!\n")
                    log_file.flush()
                    booted = True
                except Exception as e:
                    cur_attempts += 1
                    log_file.write(f"Waiting for {host} to reboot\n")
                    log_file.flush()
                    time.sleep(5)
            if cur_attempts >= max_attempts:
                exit(1)
 if __name__ == "__main__":
    main()
@@ -1,12 +0,0 @@
 [project]
 name = "updates"
 version = "0.1.0"
 description = "Update servers based on ansible inventory"
 readme = "README.md"
 requires-python = ">=3.13"
 dependencies = [
    "pyyaml>=6.0.3",
    "tqdm>=4.67.3",
    "types-pyyaml>=6.0.12.20250915",
    "types-tqdm>=4.67.3.20260205",
 ]
@@ -1,121 +0,0 @@
 version = 1
 revision = 3
 requires-python = ">=3.13"
 [[package]]
 name = "colorama"
 version = "0.4.6"
 source = { registry = "https://pypi.org/simple" }
 sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" }
 wheels = [
    { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
 ]
 [[package]]
 name = "pyyaml"
 version = "6.0.3"
 source = { registry = "https://pypi.org/simple" }
 sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" }
 wheels = [
    { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" },
    { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" },
    { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" },
    { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" },
    { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" },
    { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" },
    { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" },
    { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" },
    { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" },
    { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" },
    { url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" },
    { url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" },
    { url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" },
    { url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" },
    { url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" },
    { url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" },
    { url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" },
    { url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" },
    { url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" },
    { url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" },
    { url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" },
    { url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" },
    { url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" },
    { url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" },
    { url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" },
    { url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" },
    { url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" },
    { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" },
 ]
 [[package]]
 name = "tqdm"
 version = "4.67.3"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
    { name = "colorama", marker = "sys_platform == 'win32'" },
 ]
 sdist = { url = "https://files.pythonhosted.org/packages/09/a9/6ba95a270c6f1fbcd8dac228323f2777d886cb206987444e4bce66338dd4/tqdm-4.67.3.tar.gz", hash = "sha256:7d825f03f89244ef73f1d4ce193cb1774a8179fd96f31d7e1dcde62092b960bb", size = 169598, upload-time = "2026-02-03T17:35:53.048Z" }
 wheels = [
    { url = "https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf", size = 78374, upload-time = "2026-02-03T17:35:50.982Z" },
 ]
 [[package]]
 name = "types-pyyaml"
 version = "6.0.12.20250915"
 source = { registry = "https://pypi.org/simple" }
 sdist = { url = "https://files.pythonhosted.org/packages/7e/69/3c51b36d04da19b92f9e815be12753125bd8bc247ba0470a982e6979e71c/types_pyyaml-6.0.12.20250915.tar.gz", hash = "sha256:0f8b54a528c303f0e6f7165687dd33fafa81c807fcac23f632b63aa624ced1d3", size = 17522, upload-time = "2025-09-15T03:01:00.728Z" }
 wheels = [
    { url = "https://files.pythonhosted.org/packages/bd/e0/1eed384f02555dde685fff1a1ac805c1c7dcb6dd019c916fe659b1c1f9ec/types_pyyaml-6.0.12.20250915-py3-none-any.whl", hash = "sha256:e7d4d9e064e89a3b3cae120b4990cd370874d2bf12fa5f46c97018dd5d3c9ab6", size = 20338, upload-time = "2025-09-15T03:00:59.218Z" },
 ]
 [[package]]
 name = "types-requests"
 version = "2.32.4.20260107"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
    { name = "urllib3" },
 ]
 sdist = { url = "https://files.pythonhosted.org/packages/0f/f3/a0663907082280664d745929205a89d41dffb29e89a50f753af7d57d0a96/types_requests-2.32.4.20260107.tar.gz", hash = "sha256:018a11ac158f801bfa84857ddec1650750e393df8a004a8a9ae2a9bec6fcb24f", size = 23165, upload-time = "2026-01-07T03:20:54.091Z" }
 wheels = [
    { url = "https://files.pythonhosted.org/packages/1c/12/709ea261f2bf91ef0a26a9eed20f2623227a8ed85610c1e54c5805692ecb/types_requests-2.32.4.20260107-py3-none-any.whl", hash = "sha256:b703fe72f8ce5b31ef031264fe9395cac8f46a04661a79f7ed31a80fb308730d", size = 20676, upload-time = "2026-01-07T03:20:52.929Z" },
 ]
 [[package]]
 name = "types-tqdm"
 version = "4.67.3.20260205"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
    { name = "types-requests" },
 ]
 sdist = { url = "https://files.pythonhosted.org/packages/53/46/790b9872523a48163bdda87d47849b4466017640e5259d06eed539340afd/types_tqdm-4.67.3.20260205.tar.gz", hash = "sha256:f3023682d4aa3bbbf908c8c6bb35f35692d319460d9bbd3e646e8852f3dd9f85", size = 17597, upload-time = "2026-02-05T04:03:19.721Z" }
 wheels = [
    { url = "https://files.pythonhosted.org/packages/cc/da/7f761868dbaa328392356fab30c18ab90d14cce86b269e7e63328f29d4a3/types_tqdm-4.67.3.20260205-py3-none-any.whl", hash = "sha256:85c31731e81dc3c5cecc34c6c8b2e5166fafa722468f58840c2b5ac6a8c5c173", size = 23894, upload-time = "2026-02-05T04:03:18.48Z" },
 ]
 [[package]]
 name = "updates"
 version = "0.1.0"
 source = { virtual = "." }
 dependencies = [
    { name = "pyyaml" },
    { name = "tqdm" },
    { name = "types-pyyaml" },
    { name = "types-tqdm" },
 ]
 [package.metadata]
 requires-dist = [
    { name = "pyyaml", specifier = ">=6.0.3" },
    { name = "tqdm", specifier = ">=4.67.3" },
    { name = "types-pyyaml", specifier = ">=6.0.12.20250915" },
    { name = "types-tqdm", specifier = ">=4.67.3.20260205" },
 ]
 [[package]]
 name = "urllib3"
 version = "2.6.3"
 source = { registry = "https://pypi.org/simple" }
 sdist = { url = "https://files.pythonhosted.org/packages/c7/24/5f1b3bdffd70275f6661c76461e25f024d5a38a46f04aaca912426a2b1d3/urllib3-2.6.3.tar.gz", hash = "sha256:1b62b6884944a57dbe321509ab94fd4d3b307075e0c2eae991ac71ee15ad38ed", size = 435556, upload-time = "2026-01-07T16:24:43.925Z" }
 wheels = [
    { url = "https://files.pythonhosted.org/packages/39/08/aaaad47bc4e9dc8c725e68f9d04865dbcb2052843ff09c97b08904852d84/urllib3-2.6.3-py3-none-any.whl", hash = "sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4", size = 131584, upload-time = "2026-01-07T16:24:42.685Z" },
 ]
@@ -0,0 +1,174 @@
 # An Opinionated Guide to Home Labbing: from Hardware to Software
 ## Abstract
 Between increasingly required subscriptions, privacy terms changing overnight,
 and the difficulty of downloading your own data, it often feels like we rent our
 digital lives rather than own them. We'll walk you through hardware selection,
 the tools you'll need, the operating systems you'll install, and the software
 you might want to support for your homelab. We'll share our experience with our
 own variety of hardware, which ranges from tiny Raspberry Pis to mid-size laptop
 motherboards to overkill Threadripper servers. We'll talk about the operating
 systems we use and which ones have found the most stability and longevity in our
 deployments while comparing them to other homelab operating systems. We'll cover
 common homelab use-cases like media servers, Git servers, network storage, and
 local AI tooling. We'll also talk about safety and maintenance so you can weigh
 the choice of cutting the cloud cord properly. By the end of this talk you'll have a
 starting point for your own homelab journey, and maybe even a grocery list.
 Alex Oladele is a senior software engineer at Red Hat, working on Ansible Galaxy
 as part of the Ansible Automation Platform. He is a member of Black Python Devs
 and speaks at conferences about compassionate culture, Python, and practical
 engineering skills. He is passionate about mentoring and creating safe,
 inclusive spaces in tech. Outside of work, he builds silly side projects and
 runs a homelab he is very proud of
 Reese is DIY technology enthusiast with a passion for projects that make things
 easy. He's been working in development since 2017 with experience in risk,
 scripting automation, full stack web development, container infrastructure, amd
 homelab server hardware. He has a passion for mentoring, but even more of a
 passion for sharing the new tech he found last week with anyone who will listen.
 Currently, he leads the agentic vulnerability management team at Hungtington
 Bank.
 ## Structure
 "I'm lazy, I don't want my family to kill me, and it works"
 1. I want this in my house
 2. I want to connect outside my house
 3. I want my friends to connect
 4. I don't want this to go down
 5. I want to recover if there's an error
 6. My server died, what now?
 ## Thoughts
 Give 2 ideas per section. First for "I can't let this break my family will kill
 me". Second for "I have an understanding partner who is my cat and won't care."
 ## Story driven presentation
 I have decided to make a strong home server. Where do I even start?
 Overarching theme:
 - This is about privacy, control, and stability
 - This is not about saving money
 - This is not about keeping up with the bleeding edge of the industry
 - (Both of these things might happen though, and that's great when it does)
 1. Hardware: you find a box (old laptop, rpi) you're set.
   - Reese
     - Framework 13 motherboard for experiments, QA, Forgejo/Gitea runners, `experimental.reeseapps.com`
     - Raspberry pi for home assistant
     - Threadripper 3000 series with a bunch of disks for main apps
     - framework desktop for local AI
   - Alex
     - "Gaming PC Foundation": 5950x + ddr4 64GB ram (before memory shortage)
     - Framework motherboard
     - Raspberry pi cluster
 2. I'm a little familiar with linux but what do I install on that hardware?
   - Alex: Truenas, Raspberry Pi OS
   - Reese: Address the proxmox shaped elephant in the room, Fedora, Alma Linux, cloud init
 3. What does your homelab maintenance toolbox look like? What toolkit do you
   open to work on your homelab? What do you bring to the table to start
   tinkering?
   - Reese: btop, podman, uv, yarn, pi.dev, borg backup, distrobox (talk about misconceptions like bash history preservation)
     - AI in the homelab, safety, security, privacy
     - Discuss the progression: chat bots -> harnesses -> agentic integration
   - Alex: truenas catalog, ansible, opencode
     - Discuss security in your tooling like setting wait periods before yarn installing
     - mention uv acquisition
     - Building custom apps with AI
 4. So you want to launch some containers
   - We dislike docker compose
   - Truenas apps
   - quadlets + podman
   - kubernetes (k3s) and its downsides
 5. When to use VMs
   - An extra layer of isolation
   - Easier networking
   - Simpler to prevent resource exhaustion
     - Talk about "right-sizing": Every VM gets 2GB ram and 2cpu cores to start. On a 32GB system, assuming 4GB for the host, that creates 14 VM "slots".
     - Alma Linux 10 takes up about 450MB at boot with nothing installed
     - Double the memory when resource exhaustion happens
 6. I want to install a new app while I'm at friend's house
   - Truenas web portal (app page, both official and community) + tailscale
   - VPN and I need access to my computer
 7. I want to check my server status on my phone (updates, disks, memory pressure, error logs, services running)
   - Truenas web interface
   - Cockpit web interface
 8. I want to add more storage
   - Truenas ZFS storage pools
   - BTRFS pools
 9. I want to install a new alpha app without much support
   - Truenas custom docker compose images
   - Fedora clone and run (in a VM for style)
 10. I want to backup my photos
    - Google Photos: don't use git, images aren't meant for git
    - **Immich, with backups (tell stories about losing my image data)**
 11. I want a local copy of my code
    - Github
    - Gitea/Gitlab (talk about that transition)
    - Forgejo
 12. I want private document editing
    - Google drive, Obsidian (forces use of markdown as my standard)
    - VSCode + pandoc (commit markdown files as your documents)
    - Nextcloud (Collabora)
 13. I want a local, offline LLMs
    - llama.cpp is great, but has a very high learning curve
    - Ollama has pitfalls
      - You don't know which model you're downloading
      - Model inference is not optimized
 14. I want to watch media I own
    - Plex - I know that ruffles some jimmies. Give example: add letterbox support into Plex.
    - Jellyfin if you're cheap
 15. I want to know when something goes wrong
    - Uptime Kuma!
    - NTFY
    - N8N
    - Truenas sending emails if there's an error
    - Fedora requires a custom solution
 16. I want "reasonable availability"
    - Truenas hits 90%+ availability. Updates take it down for reboot (5-10
       minutes). Disk failure requires full shutdown, disk swap, and rebuild.
       This could be half a day.
    - Fedora hits 90%+ availability. Updates take it down for reboot (<1 min).
       Disk failures can be ignored by rebalancing. Disk failures still require
       full shutdown and resilver. This can take half a day.
 17. I want to automate my house
    - Home Assistant (raspberry pi or green)
 18. I want backups of all my data
    - No backups is an option
    - Local weekly backups to usb drives via Truenas data replication
    - Borg backup via CLI or Pika.
    - Full disk backups, app directory backups, hybrid model
    - Backblaze and S3 integration for Truenas
    - 3 copies of your data, 2 different media, 1 off site.
    1.  I want a private VPN
    - Tailscale, moved from wifiman, also moved from pivpn
    - unifi wireguard server, rawdog wireguard on a pi
@@ -16,6 +16,7 @@ dependencies = [
    "pytest>=9.0.2",
    "pyyaml>=6.0.3",
    "requests>=2.32.5",
    "rich>=15.0.0",
    "sse-starlette>=3.3.4",
    "tqdm>=4.67.3",
    "types-pyyaml>=6.0.12.20250915",
@@ -285,6 +285,7 @@ dependencies = [
    { name = "pytest" },
    { name = "pyyaml" },
    { name = "requests" },
    { name = "rich" },
    { name = "sse-starlette" },
    { name = "tqdm" },
    { name = "types-pyyaml" },
@@ -307,6 +308,7 @@ requires-dist = [
    { name = "pytest", specifier = ">=9.0.2" },
    { name = "pyyaml", specifier = ">=6.0.3" },
    { name = "requests", specifier = ">=2.32.5" },
    { name = "rich", specifier = ">=15.0.0" },
    { name = "sse-starlette", specifier = ">=3.3.4" },
    { name = "tqdm", specifier = ">=4.67.3" },
    { name = "types-pyyaml", specifier = ">=6.0.12.20250915" },
@@ -612,6 +614,18 @@ wheels = [
    { url = "https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl", hash = "sha256:e91464b71ae3ee7afd3017d9f358ef0baf158fd9a298db92f1d4761133824c36", size = 108180, upload-time = "2026-02-09T14:57:25.787Z" },
 ]
 [[package]]
 name = "markdown-it-py"
 version = "4.2.0"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
    { name = "mdurl" },
 ]
 sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" }
 wheels = [
    { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" },
 ]
 [[package]]
 name = "markupsafe"
 version = "3.0.3"
@@ -689,6 +703,15 @@ wheels = [
    { url = "https://files.pythonhosted.org/packages/9c/46/f6b4ad632c67ef35209a66127e4bddc95759649dd595f71f13fba11bdf9a/mcp-1.27.0-py3-none-any.whl", hash = "sha256:5ce1fa81614958e267b21fb2aa34e0aea8e2c6ede60d52aba45fd47246b4d741", size = 215967, upload-time = "2026-04-02T14:48:07.24Z" },
 ]
 [[package]]
 name = "mdurl"
 version = "0.1.2"
 source = { registry = "https://pypi.org/simple" }
 sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" }
 wheels = [
    { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
 ]
 [[package]]
 name = "mergedeep"
 version = "1.3.4"
@@ -1259,6 +1282,19 @@ wheels = [
    { url = "https://files.pythonhosted.org/packages/3f/51/d4db610ef29373b879047326cbf6fa98b6c1969d6f6dc423279de2b1be2c/requests_toolbelt-1.0.0-py2.py3-none-any.whl", hash = "sha256:cccfdd665f0a24fcf4726e690f65639d272bb0637b9b92dfd91a5568ccf6bd06", size = 54481, upload-time = "2023-05-01T04:11:28.427Z" },
 ]
 [[package]]
 name = "rich"
 version = "15.0.0"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
    { name = "markdown-it-py" },
    { name = "pygments" },
 ]
 sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" }
 wheels = [
    { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" },
 ]
 [[package]]
 name = "rpds-py"
 version = "0.30.0"
Author	SHA1	Message	Date
ducoterra	f9b5d293ba	update pi.md with real pi agent config Sync the documented models.json example with the actual ~/.pi/agent/models.json (maxTokens 32000 -> 32768) and add the full ~/.pi/agent/settings.json to the doc. LLM-Generated: true	2026-10-21 00:00:07 -04:00
ducoterra	8bcee5c17f	document ops-user pi install and sudoers escalation policy Adds the sysadmin setup recipe to the pi notes: run pi as a dedicated `ops` user with a tightly controlled sudoers allowlist instead of root, so an agent can do host work without unbounded privilege. Includes the AGENTS.md to drop at /home/ops/AGENTS.md. It records that the allowlist is discovered with `sudo -l` (hostnamectl, reboot, the dnf and systemctl subcommands, firewall-cmd, the ansible shim, libvirt/virsh and composer-cli, getfacl/setfacl, osbuild-auto.sh, btrfs scrub) and sets the hard rule for agents: when a command is not on that list, stop, do not attempt escalation workarounds, and ask the human to add the exact full-path sudoers line for review before proceeding. LLM-Generated: true	2026-10-17 12:26:40 -04:00
ducoterra	e94d4f718c	add README index for the Raspberry Pi systemd runbooks Ties the two runbooks together as one host-agnostic directory: which runbook solves which problem, a file -> target-path map for the units, and the shared deploy pattern (one tar \| ssh pipe into /root, then install with explicit modes rather than copying a whole tree into /etc). States the conventions both units follow - configs under /usr/local as the source of truth with thin systemd wrappers, idempotent installers that re-derive desired state and never delete what they did not create, and gap-limited self-healing so a broken dependency cannot make a unit flap. Also documents the redaction recipe for pasting a WireGuard config: match `PrivateKey[[:space:]]*=` because wg-quick strip emits spaces around the `=`, which a `PrivateKey=` regex misses. LLM-Generated: true	2026-10-16 09:08:16 -04:00
ducoterra	3cdbb405ab	add WireGuard wg0 watchdog units for always-on SSH When SSH to a Pi targets a WireGuard tunnel address, reachability depends on three things that each fail independently: the unit starting at boot, the NAT mapping staying open, and the peer hostname still resolving. Rebooting onto a network with no route to the endpoint previously meant no way back in. - wg-quick@wg0.service.d/10-boot-resilience.conf: retry a failed boot-time up forever (Restart=on-failure, RestartSec=15, StartLimitIntervalSec=0) and cap endpoint resolution (WG_ENDPOINT_RESOLUTION_RETRIES=60, TimeoutStartSec=120) instead of hanging on the stock infinity. - wg0-watchdog: POSIX sh healer, oldest symptom first - missing interface or peer restarts the unit, a missing PersistentKeepalive is set directly, a stale handshake re-points the peer at a freshly resolved endpoint, and a very stale one restarts. Restarts are gap-limited so a broken network cannot make the unit flap, endpoint resolution falls back to the kernel endpoint and then to a last-known-good cache (tunnel DNS may only be reachable through the tunnel), and nothing it did not create is ever removed. Supports --dry-run. - wg0-watchdog.service/.timer: oneshot on a 60s timer, first check 30s after boot, Persistent=true to catch up after downtime, TimeoutStartSec=120 and Nice=10 so a restart cannot wedge the watchdog. - wg0.conf.template: documents /etc/wireguard/wg0.conf with placeholders only. Deliberately omits the DNS key so name resolution does not depend on the tunnel (Debian wg-quick has no `DNS = none`), and sets PersistentKeepalive 25 to hold the NAT mapping for inbound SSH. - wireguard.md: runbook for the placeholder table, failure modes, deploy, and verification. The real wg0.conf and key material never enter the repo. LLM-Generated: true	2026-10-16 09:07:56 -04:00
ducoterra	bdee581480	add self-healing root SSH key systemd units Headless Pis lose /root/.ssh/authorized_keys to imaging, first-boot tooling, or config management, which locks out the only SSH account that can fix it. - ensure-root-ssh-key: idempotent POSIX sh installer. Appends managed public keys from /usr/local/share/root-ssh-keys/authorized_keys, never removes or rewrites foreign keys, skips lines whose first field is not a known key algorithm, and enforces 0700 / 0600 root:root on the key dir and file. - ensure-root-ssh-key.service: oneshot, ordered Before=ssh.service so the key lands before the daemon starts; ConditionPathExists skips it when the managed keys file is absent. - ensure-root-ssh-key.path: PathChanged= + PathModified= on authorized_keys so deletes, renames, and in-place edits are healed at runtime, not just at boot. Writes only when a key is missing, so the unit settles after one no-op run. - authorized_keys.example: template with commented placeholder keys only; the real file stays on the host. - root-ssh-key.md: runbook with deploy, verification, revocation steps, and the gotcha that systemd has no PathMissing= key. LLM-Generated: true	2026-10-16 09:05:54 -04:00
ducoterra	3f14b8ce14	add vllm-radiance R9700 deployment and fp8-kv calibration notes Documents the single-GPU (TP1) vllm-radiance deployment on driveripper-ai, measured KV capacity analysis for the 154K config, and the full FP8-KV calibration workflow (host prep through runtime verification) including quadlet changes, deployment steps, and the remaining matched-control quality gate. LLM-Generated: true	2026-10-12 10:31:43 -04:00
ducoterra	d9925b931d	add sudoers allowlist for ops user automation Passwordless sudo access scoped to specific commands (dnf, systemctl, firewall-cmd, ansible, qemu/virt, btrfs, acl) for ansible and maintenance automations on the fedora host. LLM-Generated: true	2026-10-12 10:31:36 -04:00
ducoterra	be7a4cf0ce	Add Raspberry Pi headless configuration notes Document bootfs headless setup: userconf.txt password hash, and discovery/login steps via nmap and ssh. LLM-Generated: true	2026-10-10 19:37:23 -04:00
ducoterra	54cd5096df	add GUI, screen capture, and audio packages to toolbx image kitty, scrot, and Xvfb enable headless browser/UI interaction; ffmpeg and portaudio cover screen recording and audio capture. LLM-Generated: true	2026-10-10 18:17:25 -04:00
ducoterra	5e8524761b	Add thinking level map to pi model definitions Documents the thinkingLevelMap for the juggernaut and turbo models in the ~/.pi/agent/models.json example, and lowers maxTokens from 128000 to 32000 to match the working configuration. LLM-Generated: true	2026-10-03 22:03:52 -04:00
ducoterra	1e45cace0b	Add Flatpak version installation reference Add documentation for installing a specific Flatpak commit version, including listing available commits and downgrading to a previous one. LLM-Generated: true	2026-09-21 21:11:09 -04:00
ducoterra	dc108a791e	Add devops days presentation, software_pi service, and update software_toolbx - Add presentations/devops_days_self_hosting_presentation.md - Add active/software_pi/ service directory - Update active/software_toolbx/Containerfile - Update retired/kubernetes_userspace submodule LLM-Generated: true	2026-09-19 12:22:09 -04:00
ducoterra	fc91437228	various updates	2026-09-07 11:37:56 -04:00