update pi.md with real pi agent config

Sync the documented models.json example with the actual ~/.pi/agent/models.json (maxTokens 32000 -> 32768) and add the full ~/.pi/agent/settings.json to the doc. LLM-Generated: true
document ops-user pi install and sudoers escalation policy
2026-10-21 00:00:07 -04:00 · 2026-10-17 12:26:40 -04:00 · 2026-10-16 09:08:16 -04:00 · 2026-10-16 09:07:56 -04:00 · 2026-10-16 09:05:54 -04:00 · 2026-10-12 10:31:43 -04:00
100 changed files with 3600 additions and 1422 deletions
@@ -42,6 +42,7 @@ or give me access to your servers.
    - [bash](#bash)
    - [Bulk File/Folder Renaming](#bulk-filefolder-renaming)
    - [Escaping a Stuck SSH Terminal](#escaping-a-stuck-ssh-terminal)
+    - [Curl output hidden by bash prompt](#curl-output-hidden-by-bash-prompt)
  - [SSH Setup](#ssh-setup)
  - [Git GPG Commit Signing](#git-gpg-commit-signing)
  - [Important Dates and Times](#important-dates-and-times)
@@ -133,6 +134,14 @@ done

 Press the following keys: enter + ~ + .

+### Curl output hidden by bash prompt
+
+Add this to your ~/.curlrc
+
+```conf
+-w "\n"
+```
+
 ## SSH Setup

 Generate a key (password protect it!)
@@ -212,15 +221,16 @@ signed you'll see an output. If not, nothing will show.

 ## Important Dates and Times

-| Time  | Day | Description                        |
-| ----- | --- | ---------------------------------- |
-| 00:00 | All | Automated builds                   |
-| 00:00 | All | NAS Snapshots                      |
-| 02:00 | All | Backups                            |
-| 04:00 | All | Bare Metal Server Security Updates |
-| 05:00 | All | VM Server Security Updates         |
-| 05:00 | All | Unifi Protect Firmware Updates     |
-| 06:00 | All | Unifi Network Firmware Updates     |
+| Time  | Day | Description                    |
+| ----- | --- | ------------------------------ |
+| 00:00 | All | Automated builds               |
+| 00:00 | All | NAS Snapshots                  |
+| 01:00 | All | Backups                        |
+| 03:00 | All | App Updates                    |
+| 04:00 | All | Bare Metal Server Updates      |
+| 05:00 | All | VM Server Updates              |
+| 05:00 | All | Unifi Protect Firmware Updates |
+| 06:00 | All | Unifi Network Firmware Updates |

 ## Project Lifecycle

@@ -0,0 +1,14 @@
+# AWS Bedrock
+
+## Endpoints
+
+Completions endpoint: <https://bedrock-mantle.us-east-1.api.aws/v1>
+
+## Model Analysis
+
+| Model ID                | Input Price (Mil) | Output Price (Mil) | Doodle Jump Notes                                                                                                                                   |
+| ----------------------- | ----------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- |
+| minimax.minimax-m2.5    | $ 0.30            | $ 1.20             | Way too fast and player moves too slow. Identified issue and fixed after second attempt                                                             |
+| zai.glm-5               | $ 1.00            | $ 3.20             | Perfect game with great graphics.                                                                                                                   |
+| mistral.devstral-2-123b | $ 0.40            | $ 2.00             | Broken game with very basic graphics. Second attempt broken as well.                                                                                |
+| deepseek.v3.2           | $1.35             | $5.40              | Way too fast, platform generation erratic, mobile controls were a nice touch. Fixed speed issue on second attempt but platform generation still bad |
@@ -1,48 +0,0 @@
-services:
-  gitea:
-    image: docker.gitea.com/gitea:1.25-rootless
-    container_name: gitea
-    environment:
-      - GITEA__database__DB_TYPE=postgres
-      - GITEA__database__HOST=postgres:5432
-      - GITEA__database__NAME=gitea
-      - GITEA__database__USER=gitea
-      - GITEA__database__PASSWD=gitea
-    security_opt:
-      - "label=disable"
-    restart: always
-    networks:
-      - gitea
-    volumes:
-      - /home/gitea/gitea_data:/data:Z
-      - /home/gitea/gitea_etc:/etc/gitea:Z
-      - /home/gitea/gitea_custom:/var/lib/gitea/custom:Z
-      - /etc/localtime:/etc/localtime:ro
-    ports:
-      - "3000:3000"
-      - "2222:2222"
-    depends_on:
-      - postgres
-    labels:
-      - "io.containers.autoupdate=registry"
-
-  postgres:
-    image: docker.io/library/postgres:15
-    container_name: postgres
-    security_opt:
-      - "label=disable"
-    restart: always
-    environment:
-      - POSTGRES_USER=gitea
-      - POSTGRES_PASSWORD=gitea
-      - POSTGRES_DB=gitea
-    networks:
-      - gitea
-    volumes:
-      - /home/gitea/gitea_postgres:/var/lib/postgresql/data:Z
-    labels:
-      - "io.containers.autoupdate=registry"
-
-networks:
-  gitea:
-    enable_ipv6: true
@@ -4,7 +4,7 @@ networks:

 services:
  server:
-    image: docker.gitea.com/gitea:1.25.3
+    image: docker.gitea.com/gitea:1.26.1
    container_name: gitea
    environment:
      - USER_UID=1001
@@ -1,22 +0,0 @@
-[Unit]
-Requires=postgres.service
-
-[Container]
-AutoUpdate=registry
-ContainerName=gitea
-Environment=GITEA__database__DB_TYPE=postgres GITEA__database__HOST=postgres:5432 GITEA__database__NAME=gitea GITEA__database__USER=gitea GITEA__database__PASSWD=gitea
-Image=docker.gitea.com/gitea:1.25-rootless
-Network=gitea.network
-PublishPort=3000:3000
-PublishPort=2222:2222
-SecurityLabelDisable=true
-Volume=/home/gitea/gitea_data:/data:Z
-Volume=/home/gitea/gitea_etc:/etc/gitea:Z
-Volume=/home/gitea/gitea_custom:/var/lib/gitea/custom:Z
-Volume=/etc/localtime:/etc/localtime:ro
-
-[Service]
-Restart=always
-
-[Install]
-WantedBy=default.target
@@ -1,14 +0,0 @@
-[Container]
-AutoUpdate=registry
-ContainerName=postgres
-Environment=POSTGRES_USER=gitea POSTGRES_PASSWORD=gitea POSTGRES_DB=gitea
-Image=docker.io/library/postgres:15
-Network=gitea.network
-SecurityLabelDisable=true
-Volume=/home/gitea/gitea_postgres:/var/lib/postgresql/data:Z
-
-[Service]
-Restart=always
-
-[Install]
-WantedBy=default.target
@@ -10,7 +10,7 @@ DB_DATA_LOCATION=/srv/immich-db/postgres
 # TZ=Etc/UTC

 # The Immich version to use. You can pin this to a specific version like "v2.1.0"
-IMMICH_VERSION=v2
+IMMICH_VERSION=v3

 # Connection secret for postgres. You should change it to a random password
 # Please use only the characters `A-Za-z0-9`, without special characters or spaces
@@ -1,19 +0,0 @@
-services:
-  keycloak:
-    container_name: keycloak
-    restart: always
-    image: quay.io/keycloak/keycloak:26.4.5
-    ports:
-      - "9443:443"
-      - "8443:8443"
-    volumes:
-      - /home/foobar/data:/var/app/data
-    security_opt:
-      - label=disable
-    userns_mode: keep-id
-    command: 
-    - "start"
-    - "--hostname"
-    - "https://keycloak.reeseapps.com"
-    - "--hostname-admin"
-    - "https://keycloak.reeselink.com:8443"
@@ -1,186 +1,10 @@
 # Podman keycloak

 - [Podman keycloak](#podman-keycloak)
-  - [Setup keycloak Project](#setup-keycloak-project)
-  - [Install Keycloak with Docker](#install-keycloak-with-docker)
-  - [Install Keycloak with Podman](#install-keycloak-with-podman)
-    - [Create the keycloak user](#create-the-keycloak-user)
-    - [Write the keycloak compose spec](#write-the-keycloak-compose-spec)
-      - [A Note on Volumes](#a-note-on-volumes)
-    - [Convert keycloak compose spec to quadlets](#convert-keycloak-compose-spec-to-quadlets)
-    - [Create any container-mounted directories](#create-any-container-mounted-directories)
-    - [Start and enable your systemd quadlet](#start-and-enable-your-systemd-quadlet)
-    - [Expose keycloak](#expose-keycloak)
-      - [firewalld](#firewalld)
-  - [Backup keycloak](#backup-keycloak)
-  - [Upgrade keycloak](#upgrade-keycloak)
-    - [Upgrade Quadlets](#upgrade-quadlets)
-  - [Uninstall](#uninstall)
-  - [Notes](#notes)
-    - [SELinux](#selinux)
+  - [Adding a Client](#adding-a-client)

-## Setup keycloak Project
+## Adding a Client

- [ ] Copy and rename this folder to active/container_keycloak
- [ ] Find and replace keycloak with the name of the service.
- [ ] Create the rootless user to run the podman containers
- [ ] Write the compose.yaml spec for your service
- [ ] Convert the compose.yaml spec to a quadlet
- [ ] Install the quadlet on the podman server
- [ ] Expose the quadlet service
- [ ] Install a backup service and timer
-
-## Install Keycloak with Docker
-
-<https://www.keycloak.org/getting-started/getting-started-docker>
-
-```bash
-# Test in dev mode
-docker run -p 8080:8080 -e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:26.4.7 start-dev
-```
-
-## Install Keycloak with Podman
-
-### Create the keycloak user
-
-```bash
-# SSH into your podman server as root
-useradd keycloak
-loginctl enable-linger $(id -u keycloak)
-systemctl --user --machine=keycloak@.host enable podman-restart
-systemctl --user --machine=keycloak@.host enable --now podman.socket
-mkdir -p /home/keycloak/.config/containers/systemd
-```
-
-### Write the keycloak compose spec
-
-<https://www.keycloak.org/getting-started/getting-started-podman>
-
-```bash
-# Based on the example
-podman run \
-p 127.0.0.1:8080:8080 \
-e KC_BOOTSTRAP_ADMIN_USERNAME=admin \
-e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
-quay.io/keycloak/keycloak:26.4.5 start-dev
-```
-
-#### A Note on Volumes
-
-Named volumes are stored at `/home/keycloak/.local/share/containers/storage/volumes/`.
-
-### Convert keycloak compose spec to quadlets
-
-Run the following to convert a compose.yaml into the various `.container` files for systemd:
-
-```bash
-# Generate the systemd service
-podman run \
--security-opt label=disable \
--rm \
-v $(pwd)/active/container_keycloak/:/compose \
-v $(pwd)/active/container_keycloak/quadlets:/quadlets \
-quay.io/k9withabone/podlet \
-f /quadlets \
-i \
--overwrite \
-compose /compose/compose.yaml
-
-# Copy the files to the server
-export PODMAN_SERVER=
-scp -r active/container_keycloak/quadlets/. $PODMAN_SERVER:/home/keycloak/.config/containers/systemd/
-ssh $PODMAN_SERVER chown -R keycloak:keycloak /home/keycloak/.config/containers/systemd/
-```
-
-### Create any container-mounted directories
-
-SSH into your podman server as root:
-
-```bash
-machinectl shell keycloak@
-podman unshare
-mkdir some_volume
-# Chown to the namespaced user with UID 1000
-# This will be some really obscure UID outside the namespace
-# This will also solve most permission denied errors
-chown -R 1000:1000 some_volume
-```
-
-### Start and enable your systemd quadlet
-
-SSH into your podman server as root:
-
-```bash
-machinectl shell keycloak@
-systemctl --user daemon-reload
-systemctl --user restart keycloak
-# Enable auto-update service which will pull new container images automatically every day
-systemctl --user enable --now podman-auto-update.timer
-```
-
-### Expose keycloak
-
-1. If you need a domain, follow the [DDNS instructions](/active/container_ddns/ddns.md#install-a-new-ddns-service)
-2. For a web service, follow the [Caddy instructions](/active/container_caddy/caddy.md#adding-a-new-caddy-record)
-3. Finally, follow your OS's guide for opening ports via its firewall service.
-
-#### firewalld
-
-```bash
-# command to get current active zone and default zone
-firewall-cmd --get-active-zones
-firewall-cmd --get-default-zone
-
-# command to open 443 on tcp
-firewall-cmd --permanent --zone=<zone> --add-port=443/tcp
-
-# command to open 80 and 443 on tcp and udp
-firewall-cmd --permanent --zone=<zone> --add-port={80,443}/{tcp,udp}
-
-# command to list available services and then open http and https
-firewall-cmd --get-services
-firewall-cmd --permanent --zone=<zone> --add-service={http,https}
-```
-
-## Backup keycloak
-
-Follow the [Borg Backup instructions](/active/systemd_borg/borg.md#set-up-a-client-for-backup)
-
-## Upgrade keycloak
-
-### Upgrade Quadlets
-
-Upgrades should be a repeat of [writing the compose spec](#convert-keycloak-compose-spec-to-quadlets) and [installing the quadlets](#start-and-enable-your-systemd-quadlet)
-
-```bash
-export PODMAN_SERVER=
-scp -r quadlets/. $PODMAN_SERVER$:/home/keycloak/.config/containers/systemd/
-ssh keycloak systemctl --user daemon-reload
-ssh keycloak systemctl --user restart keycloak
-```
-
-## Uninstall
-
-```bash
-# Stop the user's services
-systemctl --user disable podman-restart
-podman container stop --all
-systemctl --user disable --now podman.socket
-systemctl --user disable --now podman-auto-update.timer
-
-# Delete the user (this won't delete their home directory)
-# userdel might spit out an error like:
-# userdel: user keycloak is currently used by process 591255
-# kill those processes and try again
-userdel keycloak
-```
-
-## Notes
-
-### SELinux
-
-<https://blog.christophersmart.com/2021/01/31/podman-volumes-and-selinux/>
-
-:z allows a container to share a mounted volume with all other containers.
-
-:Z allows a container to reserve a mounted volume and prevents any other container from accessing.
+1. Create a new client
+2. Clients -> click on client -> Client scopes -> client-frontend-dedicated -> Scope -> turn off Full Scope Allowed
+3. 
@@ -1,3 +0,0 @@
-# Quadlets
-
-Put your quadlets here.
@@ -0,0 +1,168 @@
+# Make sure to update the credential placeholders with your own secrets.
+# We mark them with # CHANGEME in the file below.
+# In addition, we recommend to restrict inbound traffic on the host to langfuse-web (port 3000) and minio (port 9090) only.
+# All other components are bound to localhost (127.0.0.1) to only accept connections from the local machine.
+# External connections from other machines will not be able to reach these services directly.
+services:
+  langfuse-worker:
+    image: docker.io/langfuse/langfuse-worker:3
+    restart: always
+    ports:
+      - 127.0.0.1:3030:3030
+    environment: &langfuse-worker-env
+      NEXTAUTH_URL: ${NEXTAUTH_URL:-http://localhost:3000}
+      DATABASE_URL: ${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/postgres} # CHANGEME
+      SALT: ${SALT:-mysalt} # CHANGEME
+      ENCRYPTION_KEY: ${ENCRYPTION_KEY:-0000000000000000000000000000000000000000000000000000000000000000} # CHANGEME: generate via `openssl rand -hex 32`
+      TELEMETRY_ENABLED: ${TELEMETRY_ENABLED:-true}
+      LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES: ${LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES:-false}
+      CLICKHOUSE_MIGRATION_URL: ${CLICKHOUSE_MIGRATION_URL:-clickhouse://clickhouse:9000}
+      CLICKHOUSE_URL: ${CLICKHOUSE_URL:-http://clickhouse:8123}
+      CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}
+      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEME
+      CLICKHOUSE_CLUSTER_ENABLED: ${CLICKHOUSE_CLUSTER_ENABLED:-false}
+      LANGFUSE_USE_AZURE_BLOB: ${LANGFUSE_USE_AZURE_BLOB:-false}
+      LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE: ${LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE:-false}
+      LANGFUSE_OCI_AUTH_TYPE: ${LANGFUSE_OCI_AUTH_TYPE:-workload_identity}
+      LANGFUSE_S3_EVENT_UPLOAD_BUCKET: ${LANGFUSE_S3_EVENT_UPLOAD_BUCKET:-langfuse}
+      LANGFUSE_S3_EVENT_UPLOAD_REGION: ${LANGFUSE_S3_EVENT_UPLOAD_REGION:-auto}
+      LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID:-minio}
+      LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME
+      LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT: ${LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT:-http://minio:9000}
+      LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE:-true}
+      LANGFUSE_S3_EVENT_UPLOAD_PREFIX: ${LANGFUSE_S3_EVENT_UPLOAD_PREFIX:-events/}
+      LANGFUSE_S3_MEDIA_UPLOAD_BUCKET: ${LANGFUSE_S3_MEDIA_UPLOAD_BUCKET:-langfuse}
+      LANGFUSE_S3_MEDIA_UPLOAD_REGION: ${LANGFUSE_S3_MEDIA_UPLOAD_REGION:-auto}
+      LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID: ${LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID:-minio}
+      LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY: ${LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME
+      LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: ${LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT:-http://localhost:9090}
+      LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE:-true}
+      LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: ${LANGFUSE_S3_MEDIA_UPLOAD_PREFIX:-media/}
+      LANGFUSE_S3_BATCH_EXPORT_ENABLED: ${LANGFUSE_S3_BATCH_EXPORT_ENABLED:-false}
+      LANGFUSE_S3_BATCH_EXPORT_BUCKET: ${LANGFUSE_S3_BATCH_EXPORT_BUCKET:-langfuse}
+      LANGFUSE_S3_BATCH_EXPORT_PREFIX: ${LANGFUSE_S3_BATCH_EXPORT_PREFIX:-exports/}
+      LANGFUSE_S3_BATCH_EXPORT_REGION: ${LANGFUSE_S3_BATCH_EXPORT_REGION:-auto}
+      LANGFUSE_S3_BATCH_EXPORT_ENDPOINT: ${LANGFUSE_S3_BATCH_EXPORT_ENDPOINT:-http://minio:9000}
+      LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT: ${LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT:-http://localhost:9090}
+      LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID: ${LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID:-minio}
+      LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY: ${LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY:-miniosecret} # CHANGEME
+      LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE: ${LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE:-true}
+      LANGFUSE_INGESTION_QUEUE_DELAY_MS: ${LANGFUSE_INGESTION_QUEUE_DELAY_MS:-}
+      LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS: ${LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS:-}
+      REDIS_HOST: ${REDIS_HOST:-redis}
+      REDIS_PORT: ${REDIS_PORT:-6379}
+      REDIS_AUTH: ${REDIS_AUTH:-myredissecret} # CHANGEME
+      REDIS_TLS_ENABLED: ${REDIS_TLS_ENABLED:-false}
+      REDIS_TLS_CA: ${REDIS_TLS_CA:-/certs/ca.crt}
+      REDIS_TLS_CERT: ${REDIS_TLS_CERT:-/certs/redis.crt}
+      REDIS_TLS_KEY: ${REDIS_TLS_KEY:-/certs/redis.key}
+      EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS:-}
+      SMTP_CONNECTION_URL: ${SMTP_CONNECTION_URL:-}
+
+  langfuse-web:
+    image: docker.io/langfuse/langfuse:3
+    restart: always
+    ports:
+      - 3000:3000
+    environment:
+      <<: *langfuse-worker-env
+      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET:-mysecret} # CHANGEME
+      LANGFUSE_INIT_ORG_ID: ${LANGFUSE_INIT_ORG_ID:-}
+      LANGFUSE_INIT_ORG_NAME: ${LANGFUSE_INIT_ORG_NAME:-}
+      LANGFUSE_INIT_PROJECT_ID: ${LANGFUSE_INIT_PROJECT_ID:-}
+      LANGFUSE_INIT_PROJECT_NAME: ${LANGFUSE_INIT_PROJECT_NAME:-}
+      LANGFUSE_INIT_PROJECT_PUBLIC_KEY: ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY:-}
+      LANGFUSE_INIT_PROJECT_SECRET_KEY: ${LANGFUSE_INIT_PROJECT_SECRET_KEY:-}
+      LANGFUSE_INIT_USER_EMAIL: ${LANGFUSE_INIT_USER_EMAIL:-admin@ducoterra.net}
+      LANGFUSE_INIT_USER_NAME: ${LANGFUSE_INIT_USER_NAME:-admin}
+      LANGFUSE_INIT_USER_PASSWORD: ${LANGFUSE_INIT_USER_PASSWORD:-testpass}
+
+  clickhouse:
+    image: docker.io/clickhouse/clickhouse-server
+    restart: always
+    user: "101:101"
+    environment:
+      CLICKHOUSE_DB: default
+      CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}
+      CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEME
+    volumes:
+      - langfuse_clickhouse_data:/var/lib/clickhouse
+      - langfuse_clickhouse_logs:/var/log/clickhouse-server
+    ports:
+      - 127.0.0.1:8123:8123
+      - 127.0.0.1:9000:9000
+    healthcheck:
+      test: wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1
+      interval: 5s
+      timeout: 5s
+      retries: 10
+      start_period: 1s
+
+  minio:
+    image: cgr.dev/chainguard/minio
+    restart: always
+    entrypoint: sh
+    # create the 'langfuse' bucket before starting the service
+    command: -c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data'
+    environment:
+      MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minio}
+      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-miniosecret} # CHANGEME
+    ports:
+      - 9090:9000
+      - 127.0.0.1:9091:9001
+    volumes:
+      - langfuse_minio_data:/data
+    healthcheck:
+      test: ["CMD", "mc", "ready", "local"]
+      interval: 1s
+      timeout: 5s
+      retries: 5
+      start_period: 1s
+
+  redis:
+    image: docker.io/redis:7
+    restart: always
+    # CHANGEME: row below to secure redis password
+    command: >
+      --requirepass ${REDIS_AUTH:-myredissecret}
+      --maxmemory-policy noeviction
+    ports:
+      - 127.0.0.1:6379:6379
+    volumes:
+      - langfuse_redis_data:/data
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 3s
+      timeout: 10s
+      retries: 10
+
+  postgres:
+    image: docker.io/postgres:${POSTGRES_VERSION:-17}
+    restart: always
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      interval: 3s
+      timeout: 3s
+      retries: 10
+    environment:
+      POSTGRES_USER: ${POSTGRES_USER:-postgres}
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres} # CHANGEME
+      POSTGRES_DB: ${POSTGRES_DB:-postgres}
+      TZ: UTC
+      PGTZ: UTC
+    ports:
+      - 127.0.0.1:5432:5432
+    volumes:
+      - langfuse_postgres_data:/var/lib/postgresql/data
+
+volumes:
+  langfuse_postgres_data:
+    driver: local
+  langfuse_clickhouse_data:
+    driver: local
+  langfuse_clickhouse_logs:
+    driver: local
+  langfuse_minio_data:
+    driver: local
+  langfuse_redis_data:
+    driver: local
@@ -0,0 +1,167 @@
+# Podman langfuse
+
+- [Podman langfuse](#podman-langfuse)
+  - [Setup langfuse Project](#setup-langfuse-project)
+  - [Install langfuse](#install-langfuse)
+    - [Create the langfuse user](#create-the-langfuse-user)
+    - [Write the langfuse compose spec](#write-the-langfuse-compose-spec)
+      - [A Note on Volumes](#a-note-on-volumes)
+    - [Convert langfuse compose spec to quadlets](#convert-langfuse-compose-spec-to-quadlets)
+    - [Create any container-mounted directories](#create-any-container-mounted-directories)
+    - [Start and enable your systemd quadlet](#start-and-enable-your-systemd-quadlet)
+    - [Expose langfuse](#expose-langfuse)
+      - [firewalld](#firewalld)
+  - [Backup langfuse](#backup-langfuse)
+  - [Upgrade langfuse](#upgrade-langfuse)
+    - [Upgrade Quadlets](#upgrade-quadlets)
+  - [Uninstall](#uninstall)
+  - [Notes](#notes)
+    - [SELinux](#selinux)
+
+## Setup langfuse Project
+
+- [ ] Copy and rename this folder to active/container_langfuse
+- [ ] Find and replace langfuse with the name of the service.
+- [ ] Create the rootless user to run the podman containers
+- [ ] Write the compose.yaml spec for your service
+- [ ] Convert the compose.yaml spec to a quadlet
+- [ ] Install the quadlet on the podman server
+- [ ] Expose the quadlet service
+- [ ] Install a backup service and timer
+
+## Install langfuse
+
+### Create the langfuse user
+
+```bash
+# SSH into your podman server as root
+useradd langfuse
+loginctl enable-linger $(id -u langfuse)
+systemctl --user --machine=langfuse@.host enable podman-restart
+systemctl --user --machine=langfuse@.host enable --now podman.socket
+mkdir -p /home/langfuse/.config/containers/systemd
+```
+
+### Write the langfuse compose spec
+
+Edit the compose.yaml at active/container_langfuse/compose/compose.yaml
+
+#### A Note on Volumes
+
+Named volumes are stored at `/home/langfuse/.local/share/containers/storage/volumes/`.
+
+### Convert langfuse compose spec to quadlets
+
+Run the following to convert a compose.yaml into the various `.container` files for systemd:
+
+```bash
+# Generate the systemd service
+podman run \
+--security-opt label=disable \
+--rm \
+-v $(pwd)/active/container_langfuse/compose:/compose \
+-v $(pwd)/active/container_langfuse/quadlets:/quadlets \
+quay.io/k9withabone/podlet \
+-f /quadlets \
+-i \
+--overwrite \
+compose /compose/compose.yaml
+
+# Copy the files to the server
+export PODMAN_SERVER=
+scp -r active/container_langfuse/quadlets/. $PODMAN_SERVER:/home/langfuse/.config/containers/systemd/
+ssh $PODMAN_SERVER chown -R langfuse:langfuse /home/langfuse/.config/containers/systemd/
+```
+
+### Create any container-mounted directories
+
+SSH into your podman server as root:
+
+```bash
+machinectl shell langfuse@
+podman unshare
+mkdir some_volume
+# Chown to the namespaced user with UID 1000
+# This will be some really obscure UID outside the namespace
+# This will also solve most permission denied errors
+chown -R 1000:1000 some_volume
+```
+
+### Start and enable your systemd quadlet
+
+SSH into your podman server as root:
+
+```bash
+machinectl shell langfuse@
+systemctl --user daemon-reload
+systemctl --user restart langfuse
+# Enable auto-update service which will pull new container images automatically every day
+systemctl --user enable --now podman-auto-update.timer
+```
+
+### Expose langfuse
+
+1. If you need a domain, follow the [DDNS instructions](/active/container_ddns/ddns.md#install-a-new-ddns-service)
+2. For a web service, follow the [Caddy instructions](/active/container_caddy/caddy.md#adding-a-new-caddy-record)
+3. Finally, follow your OS's guide for opening ports via its firewall service.
+
+#### firewalld
+
+```bash
+# command to get current active zone and default zone
+firewall-cmd --get-active-zones
+firewall-cmd --get-default-zone
+
+# command to open 443 on tcp
+firewall-cmd --permanent --zone=<zone> --add-port=443/tcp
+
+# command to open 80 and 443 on tcp and udp
+firewall-cmd --permanent --zone=<zone> --add-port={80,443}/{tcp,udp}
+
+# command to list available services and then open http and https
+firewall-cmd --get-services
+firewall-cmd --permanent --zone=<zone> --add-service={http,https}
+```
+
+## Backup langfuse
+
+Follow the [Borg Backup instructions](/active/systemd_borg/borg.md#set-up-a-client-for-backup)
+
+## Upgrade langfuse
+
+### Upgrade Quadlets
+
+Upgrades should be a repeat of [writing the compose spec](#convert-langfuse-compose-spec-to-quadlets) and [installing the quadlets](#start-and-enable-your-systemd-quadlet)
+
+```bash
+export PODMAN_SERVER=
+scp -r quadlets/. $PODMAN_SERVER$:/home/langfuse/.config/containers/systemd/
+ssh langfuse systemctl --user daemon-reload
+ssh langfuse systemctl --user restart langfuse
+```
+
+## Uninstall
+
+```bash
+# Stop the user's services
+systemctl --user disable podman-restart
+podman container stop --all
+systemctl --user disable --now podman.socket
+systemctl --user disable --now podman-auto-update.timer
+
+# Delete the user (this won't delete their home directory)
+# userdel might spit out an error like:
+# userdel: user langfuse is currently used by process 591255
+# kill those processes and try again
+userdel langfuse
+```
+
+## Notes
+
+### SELinux
+
+<https://blog.christophersmart.com/2021/01/31/podman-volumes-and-selinux/>
+
+:z allows a container to share a mounted volume with all other containers.
+
+:Z allows a container to reserve a mounted volume and prevents any other container from accessing.
@@ -0,0 +1,21 @@
+[Container]
+Pod=langfuse.pod
+ContainerName=langfuse-clickhouse
+Environment=CLICKHOUSE_DB=default
+Environment=CLICKHOUSE_USER=clickhouse
+Environment=CLICKHOUSE_PASSWORD=clickhouse
+User=101:101
+Image=docker.io/clickhouse/clickhouse-server
+Volume=langfuse-clickhouse_data.volume:/var/lib/clickhouse
+Volume=langfuse-clickhouse_logs.volume:/var/log/clickhouse-server
+HealthCmd=wget --no-verbose --tries=1 --spider http://localhost:8123/ping
+HealthInterval=5s
+HealthTimeout=5s
+HealthRetries=10
+HealthStartPeriod=1s
+
+[Service]
+Restart=always
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,5 @@
+[Volume]
+VolumeName=langfuse_clickhouse_data
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,5 @@
+[Volume]
+VolumeName=langfuse_clickhouse_logs
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,20 @@
+[Container]
+Pod=langfuse.pod
+ContainerName=langfuse-minio
+Environment=MINIO_ROOT_USER=minio
+Environment=MINIO_ROOT_PASSWORD=miniosecret
+Image=cgr.dev/chainguard/minio
+Volume=langfuse-minio_data.volume:/data
+Entrypoint=sh
+Exec=-c 'mkdir -p /data/langfuse && minio server --address ":9000" --console-address ":9001" /data'
+HealthCmd=mc ready local
+HealthInterval=1s
+HealthTimeout=5s
+HealthRetries=5
+HealthStartPeriod=1s
+
+[Service]
+Restart=always
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,5 @@
+[Volume]
+VolumeName=langfuse_minio_data
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,20 @@
+[Container]
+Pod=langfuse.pod
+ContainerName=langfuse-postgres
+Environment=POSTGRES_USER=postgres
+Environment=POSTGRES_PASSWORD=postgres
+Environment=POSTGRES_DB=postgres
+Environment=TZ=UTC
+Environment=PGTZ=UTC
+Image=docker.io/postgres:17
+Volume=langfuse-postgres_data.volume:/var/lib/postgresql/data
+HealthCmd=pg_isready -U postgres
+HealthInterval=3s
+HealthTimeout=3s
+HealthRetries=10
+
+[Service]
+Restart=always
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,5 @@
+[Volume]
+VolumeName=langfuse_postgres_data
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,17 @@
+[Container]
+Pod=langfuse.pod
+ContainerName=langfuse-redis
+Environment=REDIS_AUTH=myredissecret
+Image=docker.io/redis:7
+Volume=langfuse-redis_data.volume:/data
+Exec=--requirepass myredissecret --maxmemory-policy noeviction
+HealthCmd=redis-cli ping
+HealthInterval=3s
+HealthTimeout=10s
+HealthRetries=10
+
+[Service]
+Restart=always
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,5 @@
+[Volume]
+VolumeName=langfuse_redis_data
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,71 @@
+[Unit]
+Requires=langfuse-postgres.service langfuse-clickhouse.service langfuse-redis.service langfuse-minio.service
+
+[Container]
+Pod=langfuse.pod
+ContainerName=langfuse-web
+Environment=NEXTAUTH_URL=http://localhost:3000
+Environment=DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
+Environment=SALT=mysalt
+Environment=ENCRYPTION_KEY=0000000000000000000000000000000000000000000000000000000000000000
+Environment=TELEMETRY_ENABLED=true
+Environment=LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES=false
+Environment=CLICKHOUSE_MIGRATION_URL=clickhouse://localhost:9000
+Environment=CLICKHOUSE_URL=http://localhost:8123
+Environment=CLICKHOUSE_USER=clickhouse
+Environment=CLICKHOUSE_PASSWORD=clickhouse
+Environment=CLICKHOUSE_CLUSTER_ENABLED=false
+Environment=LANGFUSE_USE_AZURE_BLOB=false
+Environment=LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=false
+Environment=LANGFUSE_OCI_AUTH_TYPE=workload_identity
+Environment=LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse
+Environment=LANGFUSE_S3_EVENT_UPLOAD_REGION=auto
+Environment=LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=minio
+Environment=LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=miniosecret
+Environment=LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=http://localhost:9000
+Environment=LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true
+Environment=LANGFUSE_S3_EVENT_UPLOAD_PREFIX=events/
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_BUCKET=langfuse
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_REGION=auto
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID=minio
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY=miniosecret
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT=http://localhost:9090
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE=true
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_PREFIX=media/
+Environment=LANGFUSE_S3_BATCH_EXPORT_ENABLED=false
+Environment=LANGFUSE_S3_BATCH_EXPORT_BUCKET=langfuse
+Environment=LANGFUSE_S3_BATCH_EXPORT_PREFIX=exports/
+Environment=LANGFUSE_S3_BATCH_EXPORT_REGION=auto
+Environment=LANGFUSE_S3_BATCH_EXPORT_ENDPOINT=http://localhost:9000
+Environment=LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT=http://localhost:9090
+Environment=LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID=minio
+Environment=LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY=miniosecret
+Environment=LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE=true
+Environment=LANGFUSE_INGESTION_QUEUE_DELAY_MS=
+Environment=LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=
+Environment=REDIS_HOST=localhost
+Environment=REDIS_PORT=6379
+Environment=REDIS_AUTH=myredissecret
+Environment=REDIS_TLS_ENABLED=false
+Environment=REDIS_TLS_CA=/certs/ca.crt
+Environment=REDIS_TLS_CERT=/certs/redis.crt
+Environment=REDIS_TLS_KEY=/certs/redis.key
+Environment=EMAIL_FROM_ADDRESS=
+Environment=SMTP_CONNECTION_URL=
+Environment=NEXTAUTH_SECRET=mysecret
+Environment=LANGFUSE_INIT_ORG_ID=
+Environment=LANGFUSE_INIT_ORG_NAME=
+Environment=LANGFUSE_INIT_PROJECT_ID=
+Environment=LANGFUSE_INIT_PROJECT_NAME=
+Environment=LANGFUSE_INIT_PROJECT_PUBLIC_KEY=
+Environment=LANGFUSE_INIT_PROJECT_SECRET_KEY=
+Environment=LANGFUSE_INIT_USER_EMAIL=admin@ducoterra.net
+Environment=LANGFUSE_INIT_USER_NAME=admin
+Environment=LANGFUSE_INIT_USER_PASSWORD=testpass
+Image=docker.io/langfuse/langfuse:3
+
+[Service]
+Restart=always
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,61 @@
+[Unit]
+Requires=langfuse-postgres.service langfuse-clickhouse.service langfuse-redis.service langfuse-minio.service
+
+[Container]
+Pod=langfuse.pod
+ContainerName=langfuse-worker
+Environment=NEXTAUTH_URL=http://localhost:3000
+Environment=DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
+Environment=SALT=mysalt
+Environment=ENCRYPTION_KEY=0000000000000000000000000000000000000000000000000000000000000000
+Environment=TELEMETRY_ENABLED=true
+Environment=LANGFUSE_ENABLE_EXPERIMENTAL_FEATURES=false
+Environment=CLICKHOUSE_MIGRATION_URL=clickhouse://localhost:9000
+Environment=CLICKHOUSE_URL=http://localhost:8123
+Environment=CLICKHOUSE_USER=clickhouse
+Environment=CLICKHOUSE_PASSWORD=clickhouse
+Environment=CLICKHOUSE_CLUSTER_ENABLED=false
+Environment=LANGFUSE_USE_AZURE_BLOB=false
+Environment=LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE=false
+Environment=LANGFUSE_OCI_AUTH_TYPE=workload_identity
+Environment=LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse
+Environment=LANGFUSE_S3_EVENT_UPLOAD_REGION=auto
+Environment=LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=minio
+Environment=LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=miniosecret
+Environment=LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=http://localhost:9000
+Environment=LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true
+Environment=LANGFUSE_S3_EVENT_UPLOAD_PREFIX=events/
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_BUCKET=langfuse
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_REGION=auto
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID=minio
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY=miniosecret
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT=http://localhost:9090
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE=true
+Environment=LANGFUSE_S3_MEDIA_UPLOAD_PREFIX=media/
+Environment=LANGFUSE_S3_BATCH_EXPORT_ENABLED=false
+Environment=LANGFUSE_S3_BATCH_EXPORT_BUCKET=langfuse
+Environment=LANGFUSE_S3_BATCH_EXPORT_PREFIX=exports/
+Environment=LANGFUSE_S3_BATCH_EXPORT_REGION=auto
+Environment=LANGFUSE_S3_BATCH_EXPORT_ENDPOINT=http://localhost:9000
+Environment=LANGFUSE_S3_BATCH_EXPORT_EXTERNAL_ENDPOINT=http://localhost:9090
+Environment=LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID=minio
+Environment=LANGFUSE_S3_BATCH_EXPORT_SECRET_ACCESS_KEY=miniosecret
+Environment=LANGFUSE_S3_BATCH_EXPORT_FORCE_PATH_STYLE=true
+Environment=LANGFUSE_INGESTION_QUEUE_DELAY_MS=
+Environment=LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=
+Environment=REDIS_HOST=localhost
+Environment=REDIS_PORT=6379
+Environment=REDIS_AUTH=myredissecret
+Environment=REDIS_TLS_ENABLED=false
+Environment=REDIS_TLS_CA=/certs/ca.crt
+Environment=REDIS_TLS_CERT=/certs/redis.crt
+Environment=REDIS_TLS_KEY=/certs/redis.key
+Environment=EMAIL_FROM_ADDRESS=
+Environment=SMTP_CONNECTION_URL=
+Image=docker.io/langfuse/langfuse-worker:3
+
+[Service]
+Restart=always
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,12 @@
+[Pod]
+PublishPort=3000:3000/tcp
+PublishPort=127.0.0.1:3030:3030/tcp
+PublishPort=127.0.0.1:8123:8123/tcp
+PublishPort=127.0.0.1:9000:9000/tcp
+PublishPort=9090:9000/tcp
+PublishPort=127.0.0.1:9091:9001/tcp
+PublishPort=127.0.0.1:6379:6379/tcp
+PublishPort=127.0.0.1:5432:5432/tcp
+
+[Install]
+WantedBy=default.target
@@ -11,7 +11,7 @@ HealthInterval=30s
 HealthRetries=3
 HealthStartPeriod=40s
 HealthTimeout=10s
-Image=ghcr.io/berriai/litellm-database:v1.83.14-stable.patch.3
+Image=ghcr.io/berriai/litellm-database:main-stable
 Volume=/home/ai/litellm-config.yaml:/app/config.yaml:z
 Exec=--config=/app/config.yaml

@@ -0,0 +1,24 @@
+# NTFY
+
+## Running ntfy
+
+```bash
+podman run \
+-v ntfy:/var/cache/ntfy \
+-v ./active/container_ntfy/server.yml:/etc/ntfy/server.yml:z \
+-p 8001:80 \
+-it \
+docker.io/binwiederhier/ntfy:latest \
+serve \
+--cache-file /var/cache/ntfy/cache.db
+```
+
+## Testing with Curl
+
+```bash
+# Publish a topic
+curl -d "Backup successful" localhost:8001/mytopic
+
+# Listen on a topic
+curl -s localhost:8001/mytopic
+```
@@ -0,0 +1 @@
+# base-url:
@@ -0,0 +1,16 @@
+{
+  "type": "micro",
+  "wake_word": "Alexa",
+  "author": "Kevin Ahrendt",
+  "website": "https://www.kevinahrendt.com/",
+  "model": "alexa.tflite",
+  "trained_languages": ["en"],
+  "version": 2,
+  "micro": {
+    "probability_cutoff": 0.9,
+    "sliding_window_size": 5,
+    "feature_step_size": 10,
+    "tensor_arena_size": 22348,
+    "minimum_esphome_version": "2024.7.0"
+  }
+}
@@ -1,9 +1,12 @@
 esphome:
-  name: tab2
-  friendly_name: M5Stack Tab5 2
+  name: dickhead
+  friendly_name: Dickhead
  on_boot:
-    # Set the charing icon to the correct state on boot
+    # Set the charging icon to the correct state on boot
    - then:
+        - logger.log: "Delaying backlight initialization"
+        - delay: 2s
+        - logger.log: "End delay"
        - if:
            condition:
              lambda: return id(charging).state;
@@ -17,6 +20,7 @@ esphome:
 esp32:
  board: esp32-p4-evboard
  flash_size: 16MB
+  cpu_frequency: 360MHz
  framework:
    type: esp-idf
    advanced:
@@ -36,6 +40,7 @@ esp32_hosted:

 logger:
  hardware_uart: USB_SERIAL_JTAG
+  level: DEBUG

 psram:
  mode: hex
@@ -53,8 +58,8 @@ ota:
  platform: esphome

 wifi:
-  ssid: !secret wifi_ssid
-  password: !secret wifi_password
+  ssid: !secret mobile_wifi_ssid
+  password: !secret mobile_wifi_pass
  fast_connect: true
  on_connect:
    - lvgl.label.update:
@@ -63,15 +68,50 @@ wifi:
    - select.set:
        id: dac_output
        option: "LINE1"
+    - lvgl.label.update:
+        id: lbl_ip
+        text: !lambda return id(ip_addr).state;
+    - lvgl.label.update:
+        id: lbl_ap
+        text: !lambda return id(ssid).state;
  on_disconnect:
    - lvgl.label.update:
        id: lbl_status
        text: "DISCONNECTED"
-#   ap:
-#     password: !secret hotspot_password
-#     ap_timeout: 90s
+  ap:
+    password: !secret hotspot_password
+    ap_timeout: 30s

-# captive_portal:
+captive_portal:
+
+text_sensor:
+  - platform: wifi_info
+    ip_address:
+      id: ip_addr
+      name: Device IP Address
+      address_0:
+        name: Device IP Address 0
+      address_1:
+        name: Device IP Address 1
+      address_2:
+        name: Device IP Address 2
+      address_3:
+        name: Device IP Address 3
+      address_4:
+        name: Device IP Address 4
+    ssid:
+      id: ssid
+      name: Device Connected SSID
+    bssid:
+      name: Device Connected BSSID
+    mac_address:
+      name: Device Mac Wifi Address
+    scan_results:
+      name: Device Latest Scan Results
+    dns_address:
+      name: Device DNS Address
+    power_save_mode:
+      name: Device Wifi Power Save Mode

 time:
  - platform: sntp
@@ -82,14 +122,13 @@ time:
      - 1.pool.ntp.org
      - 2.pool.ntp.org

-# wireguard:
-#   address: !secret tab1_wg_ip
-#   private_key: !secret tab1_wg_pk
-#   peer_endpoint: !secret wg_host
-#   peer_public_key: !secret wg_pubkey
-
-#   # Optional keepalive (disabled by default)
-#   peer_persistent_keepalive: 25s
+wireguard:
+  address: !secret tab1_wg_ip
+  private_key: !secret tab1_wg_pk
+  peer_endpoint: !secret wg_host
+  peer_public_key: !secret wg_pubkey
+  netmask: 0.0.0.0
+  peer_persistent_keepalive: 25s

 i2c:
  - id: bsp_bus
@@ -196,6 +235,39 @@ binary_sensor:
    pin:
      pi4ioe5v6408: pi4ioe1
      number: 7
+  - platform: lvgl
+    widget: volume_up_widget
+    name: Volume Up Button
+    on_press:
+      then:
+        - logger.log: "Button pressed"
+        - media_player.volume_up:
+            id: tab5_media_player
+        - delay: 100ms
+        - lvgl.label.update:
+            id: lbl_volume
+            text: !lambda return to_string(int(id(tab5_media_player).volume * 100));
+        - light.turn_on:
+            id: backlight
+            brightness: !lambda |-
+              float current_value = id(backlight).current_values.get_brightness();
+              return current_value < 0.6 ? 0.6 : current_value + 0.2;
+  - platform: lvgl
+    widget: volume_down_widget
+    name: Volume Down Button
+    on_press:
+      then:
+        - logger.log: "Button pressed"
+        - media_player.volume_down:
+            id: tab5_media_player
+        - lvgl.label.update:
+            id: lbl_volume
+            text: !lambda return to_string(int(id(tab5_media_player).volume * 100));
+        - light.turn_on:
+            id: backlight
+            brightness: !lambda |-
+              float current_value = id(backlight).current_values.get_brightness();
+              return current_value < 0.6 ? 0.6 : current_value - 0.2;

 sensor:
  - platform: ina226
@@ -255,8 +327,8 @@ touchscreen:
      - logger.log: "LVGL resuming"
      - lvgl.resume:
      - light.turn_on: backlight
-    on_release:
-      - media_player.stop:
+    # on_release:
+    # - media_player.stop:

 esp_ldo:
  - voltage: 2.5V
@@ -287,7 +359,7 @@ light:
    restore_mode: ALWAYS_ON
    default_transition_length: 250ms
    initial_state:
-      brightness: "100%"
+      brightness: "50%"

 image:
  defaults:
@@ -318,20 +390,41 @@ lvgl:
          transition_length: 15s
      - lvgl.pause:
  widgets:
+    - obj:
+        align: TOP_MID
+        width: 100%
+        height: 100%
+        layout:
+          type: flex
+          flex_flow: column
+          flex_align_main: START
+          flex_align_track: center
+          flex_align_cross: center
+        widgets:
+          - label:
+              align: TOP_MID
+              id: lbl_status
+              text_font: montserrat_48
+              text: "CONNECTING..."
+          - label:
+              align: TOP_MID
+              id: lbl_ap
+              text_font: montserrat_22
+              text: "CONNECTING..."
+          - label:
+              align: TOP_MID
+              id: lbl_ip
+              text_font: montserrat_22
+              text: "CONNECTING..."
    - image:
        id: listen_icon_widget
        src: va_idle
        align: CENTER
-    - label:
-        align: TOP_MID
-        id: lbl_status
-        text_font: montserrat_48
-        text: "CONNECTING..."
    - label:
        align: BOTTOM_LEFT
        id: lbl_version
        text_font: montserrat_12
-        text: "v0.5"
+        text: "v0.6"
    - label:
        align: BOTTOM_RIGHT
        id: lbl_battery
@@ -341,21 +434,38 @@ lvgl:
        id: charging_icon_widget
        src: charging_icon
        align: TOP_RIGHT
-    - slider:
-        id: backlight_slider
+    - button:
+        id: volume_up_widget
+        widgets:
+          - label:
+              text: "\uF028"
+              text_font: montserrat_48
+              text_align: CENTER
+              align: CENTER
        x: 20
-        y: 50
-        width: 30
-        height: 220
+        y: 20
+        width: 100
+        height: 100
        pad_all: 8
-        min_value: 0
-        max_value: 255
-        on_release:
-          - homeassistant.action:
-              action: light.turn_on
-              data:
-                entity_id: light.backlight
-                brightness: !lambda return int(x);
+    - button:
+        id: volume_down_widget
+        widgets:
+          - label:
+              text: "\uF027"
+              text_font: montserrat_48
+              text_align: CENTER
+              align: CENTER
+        x: 20
+        y: 140
+        width: 100
+        height: 100
+        pad_all: 8
+    - label:
+        x: 20
+        y: 260
+        id: lbl_volume
+        text_font: montserrat_28
+        text: !lambda return "%.1f",to_string(id(tab5_media_player).volume);

 # The DAC Output select needs to be manually (or with an automation) changed to `LINE1` for the onboard speaker
 select:
@@ -414,7 +524,7 @@ speaker:
    audio_dac: es8388_dac
    dac_type: external
    channel: mono
-    buffer_duration: 100ms
+    buffer_duration: 60000ms
    bits_per_sample: 16bit
    sample_rate: 48000

@@ -429,9 +539,12 @@ media_player:
 micro_wake_word:
  id: mww
  models:
-    - okay_nabu
-    - hey_mycroft
-    - hey_jarvis
+    - model:
+        type: local
+        path: alexa.json
+      id: alexa
+      probability_cutoff: 0.97
+      sliding_window_size: 5
  on_wake_word_detected:
    - voice_assistant.start:
        wake_word: !lambda return wake_word;
@@ -56,6 +56,13 @@ Make sure your permissions are set correctly
 sudo usermod -a -G dialout ducoterra
 ```

+or on Kinoite
+
+```bash
+grep -E '^dialout:' /usr/lib/group | sudo tee -a /etc/group
+sudo usermod -aG dialout $USER
+```
+
 Then "run" your config file

 ```bash
@@ -0,0 +1,16 @@
+{
+  "type": "micro",
+  "wake_word": "hey dick head",
+  "author": "peenfart",
+  "website": "https://www.kevinahrendt.com/",
+  "model": "./hey_dick_head.tflite",
+  "trained_languages": ["en"],
+  "version": 2,
+  "micro": {
+    "probability_cutoff": 0.97,
+    "sliding_window_size": 5,
+    "feature_step_size": 10,
+    "tensor_arena_size": 22860,
+    "minimum_esphome_version": "2024.7"
+  }
+}
@@ -0,0 +1,12 @@
+# Raspberry Pi
+
+## Headless Configuration
+
+1. Create a file called `ssh` in `bootfs`
+2. Create a file called `userconf.txt` in `bootfs`
+3. Run `echo 'YourPassword' | openssl passwd -6 -stdin` in your terminal to generate your encrypted password
+4. Edit `userconf.txt` and add a single line: `username:encrypted_password`
+5. Save `userconf.txt` and eject the SD card
+6. Plug in the pi
+7. Find the pi with `nmap -sn 10.0.0.0/24`
+8. Login with `ssh username@pi-ip-address`
@@ -0,0 +1,60 @@
+# Raspberry Pi — boot-time hardening units
+
+Two reusable runbooks for a headless Pi that must stay reachable over SSH.
+Deliberately host-agnostic: **no addresses, key material, or peer identities
+live in this repo** — those stay on each host. Set `HOST` and fill in the
+placeholders from the runbook before running anything.
+
+| Runbook | Solves |
+| --- | --- |
+| [`root-ssh-key.md`](root-ssh-key.md) | root `authorized_keys` gets wiped (imaging, config management) → key is re-installed at boot and re-applied on change |
+| [`wireguard.md`](wireguard.md) | SSH through a WireGuard tunnel address fails after reboot / NAT timeout / roaming to another network |
+
+Verified against Debian 13 (trixie) with systemd 257 and NetworkManager. The
+units are plain POSIX `sh` + systemd, so they carry to any other distribution
+with the same systemd version or newer.
+
+## Layout → target path on the host
+
+| File in this directory | Installs to |
+| --- | --- |
+| `ensure-root-ssh-key` | `/usr/local/sbin/ensure-root-ssh-key` (`0755`) |
+| `ensure-root-ssh-key.service` | `/etc/systemd/system/` |
+| `ensure-root-ssh-key.path` | `/etc/systemd/system/` |
+| `root-ssh-keys/authorized_keys.example` | `/usr/local/share/root-ssh-keys/authorized_keys` (`0644`) — copy, fill in your public key(s) |
+| `wg0-watchdog` | `/usr/local/sbin/wg0-watchdog` (`0755`) |
+| `wg0-watchdog.service` | `/etc/systemd/system/` |
+| `wg0-watchdog.timer` | `/etc/systemd/system/` |
+| `wg-quick@wg0.service.d-10-boot-resilience.conf` | `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` |
+| `wg0.conf.template` | documents `/etc/wireguard/wg0.conf` (`0600`) — the real file is never copied here |
+
+## Deploy pattern
+
+Every command in both runbooks follows the same shape, so a fresh Pi needs only
+`HOST` and the placeholders:
+
+```bash
+HOST=root@<pi-address>
+D=$PWD   # run the snippets from a checkout of this directory
+```
+
+Files travel over one `tar | ssh` pipe into `/root`, then `install` moves them
+into place with explicit modes (no `scp` of a whole tree into `/etc`, no
+ambiguous ownership). Reload and enable once at the end:
+
+```bash
+ssh "$HOST" 'systemctl daemon-reload && systemctl enable --now <units>'
+```
+
+## Conventions worth keeping on any host
+
+- Config files under `/usr/local/{sbin,share}` are the source of truth; systemd
+  units stay thin wrappers so they can be re-imaged independently.
+- Idempotent installers over clever state: every script re-derives the desired
+  state from its own config file and never deletes what it did not create.
+- Anything that must heal itself is gap-limited (a minimum interval between
+  restarts) so a broken dependency cannot make a unit flap.
+- Private keys and addresses are excluded from the repo; when dumping a
+  WireGuard config, redact with a regex that tolerates `PrivateKey = ` spacing
+  (`wg-quick strip` emits spaces, a `PrivateKey=` regex misses them):
+  `sed -E 's/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/'`
@@ -0,0 +1,61 @@
+#!/bin/sh
+# ensure-root-ssh-key - idempotently install managed SSH public keys into
+# /root/.ssh/authorized_keys. Runs at boot (ensure-root-ssh-key.service) and
+# can be run by hand at any time.
+#
+# Source of truth: /usr/local/share/root-ssh-keys/authorized_keys
+# One key per line, same format as an authorized_keys file. Blank lines and
+# lines starting with '#' are ignored. Keys already present are left alone;
+# foreign keys in authorized_keys are never removed.
+
+set -eu
+
+KEY_DIR="${KEY_DIR:-/root/.ssh}"
+AUTH_KEYS="$KEY_DIR/authorized_keys"
+MANAGED_KEYS="${MANAGED_KEYS:-/usr/local/share/root-ssh-keys/authorized_keys}"
+
+if [ ! -r "$MANAGED_KEYS" ]; then
+    echo "ensure-root-ssh-key: cannot read $MANAGED_KEYS" >&2
+    exit 1
+fi
+
+umask 077
+install -d -m 0700 "$KEY_DIR"
+[ -e "$AUTH_KEYS" ] || : > "$AUTH_KEYS"
+chmod 0600 "$AUTH_KEYS"
+chown root:root "$KEY_DIR" "$AUTH_KEYS"
+
+added=0
+checked=0
+while IFS= read -r line <&3; do
+    case "$line" in
+        '' | \#*) continue ;;
+    esac
+
+    # Basic shape check: <algo> <base64> [comment]
+    algo=${line%% *}
+    case "$algo" in
+        ssh-ed25519 | ssh-rsa | ssh-dss | ecdsa-sha2-* | sk-ecdsa-sha2-* | sk-ssh-ed25519-* | ssh-ed448) ;;
+        *)
+            echo "ensure-root-ssh-key: skipping unrecognised line: $line" >&2
+            continue
+            ;;
+    esac
+
+    checked=$((checked + 1))
+    if grep -qxF -- "$line" "$AUTH_KEYS" 2>/dev/null; then
+        continue
+    fi
+    printf '%s\n' "$line" >> "$AUTH_KEYS"
+    added=$((added + 1))
+    echo "ensure-root-ssh-key: installed key (${algo}, ${#line} chars)"
+done 3< "$MANAGED_KEYS"
+
+if [ "$added" -eq 0 ]; then
+    echo "ensure-root-ssh-key: $checked managed key(s) already present in $AUTH_KEYS"
+else
+    echo "ensure-root-ssh-key: added $added key(s), $checked managed key(s) total"
+fi
+
+chmod 0600 "$AUTH_KEYS"
+exit 0
@@ -0,0 +1,16 @@
+[Unit]
+Description=Watch /root/.ssh/authorized_keys and re-install the managed root SSH key
+Documentation=man:sshd(8)
+After=local-fs.target
+
+[Path]
+# PathChanged covers deletion / rename / creation of the file,
+# PathModified covers in-place edits (key stripped or file truncated).
+# ensure-root-ssh-key.service only writes when a managed key is missing, so
+# its own writes settle after one extra no-op run instead of looping.
+PathChanged=/root/.ssh/authorized_keys
+PathModified=/root/.ssh/authorized_keys
+Unit=ensure-root-ssh-key.service
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,15 @@
+[Unit]
+Description=Ensure managed SSH public keys are installed in /root/.ssh/authorized_keys
+Documentation=man:sshd(8)
+After=local-fs.target
+Before=ssh.service sshd.service
+ConditionPathExists=/usr/local/share/root-ssh-keys/authorized_keys
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/ensure-root-ssh-key
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,109 @@
+# Runbook: self-healing root SSH key
+
+`ensure-root-ssh-key.service` re-installs the managed SSH public key(s) into
+`/root/.ssh/authorized_keys` on every boot, and `ensure-root-ssh-key.path`
+re-applies it at runtime whenever that file is edited, deleted, or replaced.
+Useful when the key gets wiped by imaging, first-boot tooling, or config
+management — or when you need the key to come back after someone else edits the
+file.
+
+## Files on the host
+
+| Path | Purpose |
+| --- | --- |
+| `/etc/systemd/system/ensure-root-ssh-key.service` | `Type=oneshot`, `WantedBy=multi-user.target`, ordered `Before=ssh.service` so the key is in place before the daemon starts |
+| `/etc/systemd/system/ensure-root-ssh-key.path` | `PathChanged=` + `PathModified=` on `/root/.ssh/authorized_keys`, triggers the service |
+| `/usr/local/sbin/ensure-root-ssh-key` | Idempotent POSIX `sh` installer |
+| `/usr/local/share/root-ssh-keys/authorized_keys` | Source of truth — one public key per line, `#` comments allowed |
+
+## Behaviour
+
+- Appends any managed key that is missing; never removes or rewrites foreign keys.
+- Creates `/root/.ssh` (`0700`) and `authorized_keys` (`0600`, root:root) if absent.
+- Lines whose first field is not a known key algorithm are skipped with a warning.
+- Skipped, not failed, if the managed keys file is deleted.
+- Runtime self-healing: deleting, truncating, or renaming `authorized_keys` puts
+  the managed key(s) back within ~1s. That also means a managed key cannot be
+  revoked from the host alone — delete it from
+  `/usr/local/share/root-ssh-keys/authorized_keys` first.
+
+> Gotcha: systemd has no `PathMissing=` key — it is silently ignored with a
+> warning in `systemd-analyze verify`. `PathChanged=` is what fires on
+> delete/rename; `PathModified=` covers in-place appends and truncations. Set both.
+
+## Install
+
+```bash
+HOST=root@<pi-address>
+D=$PWD        # run the snippets from a checkout of this directory
+PUB=$HOME/.ssh/id_ed25519.pub          # the key that must always work
+
+ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /usr/local/share/root-ssh-keys'
+tar -C "$D" -cf - ensure-root-ssh-key ensure-root-ssh-key.service ensure-root-ssh-key.path | \
+ssh "$HOST" 'tar -xf - -C /root &&
+  install -m 0755 /root/ensure-root-ssh-key /usr/local/sbin/ &&
+  install -m 0644 /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path /etc/systemd/system/ &&
+  rm -f /root/ensure-root-ssh-key /root/ensure-root-ssh-key.service /root/ensure-root-ssh-key.path &&
+  systemctl daemon-reload'
+cat "$PUB" | ssh "$HOST" 'install -m 0644 /dev/stdin /usr/local/share/root-ssh-keys/authorized_keys'
+ssh "$HOST" 'systemctl enable --now ensure-root-ssh-key.service ensure-root-ssh-key.path'
+```
+
+Use `root-ssh-keys/authorized_keys.example` instead of `$PUB` when the host
+should carry more than one key.
+
+## Verify
+
+One-shot smoke test that proves both units without touching a live session
+(existing SSH sessions are unaffected; only new logins during the window would
+fail):
+
+```bash
+ssh "$HOST" 'cp -a /root/.ssh/authorized_keys /root/.ssh/authorized_keys.testbak
+  rm -f /root/.ssh/authorized_keys
+  systemctl start ensure-root-ssh-key.service && cat /root/.ssh/authorized_keys
+  rm -f /root/.ssh/authorized_keys.testbak'
+```
+
+Status and history:
+
+```bash
+ssh "$HOST" 'systemctl is-enabled ensure-root-ssh-key.service ensure-root-ssh-key.path
+  systemctl status ensure-root-ssh-key.service --no-pager | head -8
+  journalctl -u ensure-root-ssh-key.service -b --no-pager | tail'
+```
+
+The installer honours `KEY_DIR` and `MANAGED_KEYS`, so the create-if-missing and
+preserve-foreign-keys paths can be exercised in a scratch directory instead of
+`/root/.ssh`:
+
+```bash
+ssh "$HOST" 'install -d -m 0700 /tmp/k/src && cp /usr/local/share/root-ssh-keys/authorized_keys /tmp/k/src/
+  KEY_DIR=/tmp/k/root MANAGED_KEYS=/tmp/k/src/authorized_keys /usr/local/sbin/ensure-root-ssh-key
+  ls -ld /tmp/k/root /tmp/k/root/authorized_keys; rm -rf /tmp/k'
+```
+
+Checklist that has passed on a reference host:
+
+- `systemd-analyze verify` clean for both units, both `enabled`.
+- Repeated runs add no duplicates; a wiped file and a deleted file are both restored.
+- Missing `/root/.ssh` is created `0700`, file `0600`.
+- Foreign keys survive; no trigger loop once the path unit settles.
+- Boot ordering: `Before=ssh.service` is honoured inside the boot transaction;
+  prove it after a reboot with `systemd-analyze critical-chain ssh.service`.
+
+## Manage keys
+
+```bash
+ssh "$HOST" 'echo "<public key line>" >> /usr/local/share/root-ssh-keys/authorized_keys
+  systemctl restart ensure-root-ssh-key.service'
+```
+
+## Uninstall
+
+```bash
+ssh "$HOST" 'systemctl disable --now ensure-root-ssh-key.path ensure-root-ssh-key.service
+  rm -f /etc/systemd/system/ensure-root-ssh-key.{service,path} /usr/local/sbin/ensure-root-ssh-key
+  rm -rf /usr/local/share/root-ssh-keys
+  systemctl daemon-reload'
+```
@@ -0,0 +1,11 @@
+# Template for /usr/local/share/root-ssh-keys/authorized_keys on the host.
+# Keys listed here are (re)installed into /root/.ssh/authorized_keys at boot and
+# whenever that file changes. Copy to `authorized_keys`, add your public keys,
+# and keep the real file on the host - do not commit it.
+#
+# One key per line, authorized_keys format. Blank lines and lines starting with
+# '#' are ignored; a line whose first field is not a known key algorithm is
+# skipped with a warning.
+#
+# ssh-ed25519 AAAA<base64> comment@host
+# ssh-rsa     AAAA<base64> comment@host
@@ -0,0 +1,17 @@
+# /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf
+# The stock unit already has After=/Wants=network-online.target nss-lookup.target.
+# This makes a boot-time failure (DHCP/DNS not ready yet, endpoint host offline)
+# retry forever instead of leaving the tunnel down until someone logs in.
+[Unit]
+# Unlimited retries: do not let the start rate limiter give up on us.
+StartLimitIntervalSec=0
+
+[Service]
+# Fail fast instead of hanging forever on endpoint resolution (stock value is
+# infinity), then let Restart= try again. Same variable as the stock unit,
+# later assignment wins.
+Environment=WG_ENDPOINT_RESOLUTION_RETRIES=60
+TimeoutStartSec=120
+TimeoutStopSec=30
+Restart=on-failure
+RestartSec=15
@@ -0,0 +1,151 @@
+#!/bin/sh
+# wg0-watchdog - keep the WireGuard client tunnel up so SSH to the tunnel
+# address works after boot, after roaming to a new network, and after the NAT
+# mapping or endpoint IP goes stale.
+#
+# Decisions, oldest symptom first:
+#   interface or peer gone            -> restart wg-quick@wg0.service
+#   no PersistentKeepalive in kernel  -> set it (covers a config edited without reload)
+#   handshake older than STALE        -> re-point the peer at the endpoint
+#                                        (re-resolves the hostname, forces a handshake)
+#   handshake older than STALE_RESTART-> restart wg-quick@wg0.service
+#
+# Restarts are gap-limited so a broken network cannot make the unit flap.
+# Never removes peers, addresses, or routes it did not create.
+
+set -eu
+
+IFACE="${WG_WD_IFACE:-wg0}"
+UNIT="${WG_WD_UNIT:-wg-quick@wg0.service}"
+CONF="${WG_WD_CONF:-/etc/wireguard/$IFACE.conf}"
+KEEPALIVE="${WG_WD_KEEPALIVE:-25}"
+STALE="${WG_WD_STALE:-120}"              # seconds: poke the endpoint
+STALE_RESTART="${WG_WD_STALE_RESTART:-300}"   # seconds: restart the unit
+MIN_RESTART_GAP="${WG_WD_MIN_RESTART_GAP:-180}"  # seconds between restarts
+STATE_DIR="${WG_WD_STATE_DIR:-/var/lib/wg0-watchdog}"
+
+DRY=0
+[ "${1:-}" = "--dry-run" ] && DRY=1
+
+now=$(date +%s)
+log() { printf 'wg0-watchdog: %s\n' "$*"; }
+run() {
+    if [ "$DRY" -eq 1 ]; then
+        printf 'wg0-watchdog: DRY-RUN would run:'
+        printf ' %s' "$@"
+        printf '\n'
+    else
+        "$@"
+    fi
+}
+
+read_state() { cat "$1" 2>/dev/null || echo 0; }
+write_state() {
+    if [ "$DRY" -eq 1 ]; then
+        return 0
+    fi
+    mkdir -p "$STATE_DIR"
+    printf '%s\n' "$2" > "$1"
+}
+
+restart_unit() {
+    reason=$1
+    last=$(read_state "$STATE_DIR/last-restart")
+    case "$last" in ''|*[!0-9]*) last=0 ;; esac
+    since=$((now - last))
+    if [ "$since" -lt "$MIN_RESTART_GAP" ]; then
+        log "restart of $UNIT suppressed ($reason); last restart ${since}s ago"
+        return 0
+    fi
+    log "restarting $UNIT ($reason)"
+    write_state "$STATE_DIR/last-restart" "$now"
+    # --no-block: a wg-quick up stuck on endpoint resolution must not wedge the
+    # watchdog; the next run reports whether the restart worked.
+    run systemctl restart --no-block "$UNIT"
+}
+
+# --- 1. interface / peer present -------------------------------------------
+if ! ip link show "$IFACE" >/dev/null 2>&1; then
+    restart_unit "interface $IFACE is missing"
+    exit 0
+fi
+
+peer=$(wg show "$IFACE" peers 2>/dev/null | head -n 1)
+if [ -z "${peer:-}" ]; then
+    restart_unit "no peer configured on $IFACE"
+    exit 0
+fi
+
+# --- 2. persistent keepalive present ---------------------------------------
+live_ka=$(wg show "$IFACE" persistent-keepalive 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
+[ -n "${live_ka:-}" ] || live_ka=0
+if [ "$live_ka" != "$KEEPALIVE" ]; then
+    log "persistent-keepalive is $live_ka, setting $KEEPALIVE"
+    run wg set "$IFACE" peer "$peer" persistent-keepalive "$KEEPALIVE"
+fi
+
+# --- 3. handshake age -------------------------------------------------------
+hs=$(wg show "$IFACE" latest-handshakes 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
+case "${hs:-}" in ''|*[!0-9]*) hs=0 ;; esac
+if [ "$hs" -eq 0 ]; then
+    age=999999
+else
+    age=$((now - hs))
+fi
+
+# endpoint currently in use by the kernel, and the cached last-known-good one
+live_ep=$(wg show "$IFACE" endpoints 2>/dev/null | awk -v p="$peer" '$1 == p {print $2}')
+cache_file="$STATE_DIR/endpoint"
+cached_ep=""
+[ -f "$cache_file" ] && cached_ep=$(cat "$cache_file" 2>/dev/null || true)
+
+if [ "$age" -lt "$STALE" ]; then
+    if [ -n "$live_ep" ]; then
+        write_state "$cache_file" "$live_ep"
+    fi
+    log "healthy: handshake ${age}s ago, endpoint ${live_ep:-unknown}, keepalive ${KEEPALIVE}s"
+    exit 0
+fi
+
+log "stale: last handshake ${age}s ago"
+
+# Prefer the endpoint from the config file, freshly resolved. Fall back to the
+# kernel's current endpoint, then to the cache - important because a DNS server
+# from the tunnel config may be reachable only through this very tunnel.
+target_ep=""
+conf_ep=$(awk -F= 'tolower($1) ~ /^[[:space:]]*endpoint[[:space:]]*$/ { gsub(/[[:space:]]/, "", $2); print $2 }' "$CONF" 2>/dev/null | tail -n 1)
+if [ -n "${conf_ep:-}" ]; then
+    host=${conf_ep%:*}
+    port=${conf_ep##*:}
+    case "$host" in
+        *[!0-9.]*)
+            resolved=$(getent hosts "$host" 2>/dev/null | awk '{print $1; exit}' || true)
+            ;;
+        *)
+            resolved=$host
+            ;;
+    esac
+    if [ -n "${resolved:-}" ]; then
+        target_ep="$resolved:$port"
+    else
+        log "could not resolve $host (tunnel DNS?), falling back"
+    fi
+fi
+[ -n "$target_ep" ] || target_ep=${live_ep:-${cached_ep:-}}
+
+if [ "$age" -ge "$STALE_RESTART" ]; then
+    restart_unit "handshake ${age}s old"
+    exit 0
+fi
+
+if [ -z "$target_ep" ]; then
+    log "no endpoint to re-point at; will retry next run"
+    exit 0
+fi
+
+if [ "$target_ep" = "$live_ep" ]; then
+    log "endpoint already $target_ep; forcing a handshake"
+fi
+log "re-pointing peer at $target_ep"
+run wg set "$IFACE" peer "$peer" endpoint "$target_ep"
+exit 0
@@ -0,0 +1,12 @@
+[Unit]
+Description=Keep the WireGuard wg0 tunnel reachable for SSH
+Documentation=man:wg-quick(8)
+
+[Service]
+Type=oneshot
+# Never let a hung systemctl restart inside the script wedge the watchdog.
+TimeoutStartSec=120
+ExecStart=/usr/local/sbin/wg0-watchdog
+StandardOutput=journal
+StandardError=journal
+Nice=10
@@ -0,0 +1,14 @@
+[Unit]
+Description=Check the WireGuard wg0 tunnel every minute and heal it if needed
+
+[Timer]
+# First check shortly after boot, then every minute. Persistent=true catches up
+# if the machine was suspended/offline.
+OnBootSec=30s
+OnUnitActiveSec=60s
+AccuracySec=15s
+RandomizedDelaySec=5s
+Persistent=true
+
+[Install]
+WantedBy=timers.target
@@ -0,0 +1,20 @@
+# wg0.conf template — /etc/wireguard/wg0.conf on the host, mode 0600 root:root.
+# Never commit a real copy: this file holds the tunnel private key.
+# Generate the key with `umask 077; wg genkey | tee privatekey | wg pubkey`.
+
+[Interface]
+PrivateKey = <tunnel private key>
+Address = <TUNNEL_ADDR>
+# Deliberately no `DNS` key: any resolver listed here is pushed as a system
+# upstream while the tunnel is up, which makes all name resolution depend on the
+# tunnel. Omitting it keeps the LAN's DHCP resolvers. Debian's wg-quick has no
+# `DNS = none` - a non-IP value becomes a search domain.
+
+[Peer]
+PublicKey = <PEER_PUBKEY>
+AllowedIPs = <PEER_NETWORK>
+Endpoint = <WG_SERVER_HOST:PORT>
+# Required when the client is behind NAT and must be reachable inbound (SSH):
+# without it the mapping expires and the server cannot initiate. 25s survives
+# typical 30-60s NAT UDP timeouts.
+PersistentKeepalive = 25
@@ -0,0 +1,162 @@
+# Runbook: WireGuard client that survives reboot, NAT, and roaming
+
+For a Pi whose SSH address is a **WireGuard tunnel address** rather than a LAN
+address: the physical interface is ordinary DHCP and the Pi moves between
+networks, while the tunnel peer is a reachable server. Reachability then depends
+on three separate things, each of which fails independently — the unit starting
+at boot, the NAT mapping staying open, and the peer hostname still resolving.
+
+Placeholders used below:
+
+| Placeholder | Meaning |
+| --- | --- |
+| `$HOST` | `root@<address you can already reach>` |
+| `<TUNNEL_ADDR>` | this Pi's tunnel address, e.g. `<addr>/32` |
+| `<PEER_NETWORK>` | network routed through the tunnel, e.g. `<net>/<cidr>` |
+| `<WG_SERVER_HOST:PORT>` | tunnel server endpoint |
+| `<PEER_PUBKEY>` | the server's WireGuard public key |
+
+## Files on the host
+
+| Path | Purpose |
+| --- | --- |
+| `/etc/wireguard/wg0.conf` | tunnel config, mode `0600` — see `wg0.conf.template`, never stored in this repo |
+| `/etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf` | retry a failed boot-time `up`: `Restart=on-failure`, `RestartSec=15`, `StartLimitIntervalSec=0`, `WG_ENDPOINT_RESOLUTION_RETRIES=60`, `TimeoutStartSec=120` |
+| `/usr/local/sbin/wg0-watchdog` | heals the tunnel once a minute |
+| `/etc/systemd/system/wg0-watchdog.service` | oneshot that runs the script |
+| `/etc/systemd/system/wg0-watchdog.timer` | `OnBootSec=30s`, `OnUnitActiveSec=60s`, `Persistent=true` |
+| `/var/lib/wg0-watchdog/endpoint` | last-known-good endpoint IP, used when DNS is unavailable |
+
+## Failure modes this closes
+
+1. **No `PersistentKeepalive`.** Behind NAT the server can only send to the Pi
+   while the mapping lives, so once it expires the tunnel looks configured but
+   inbound SSH times out. `PersistentKeepalive = 25` punches UDP state often
+   enough for typical 30-60s NAT timeouts.
+2. **A boot-time failure is permanent.** `wg-quick up` fails outright if the
+   endpoint hostname does not resolve yet or the server is down, and the unit is
+   not retried — so the tunnel stays dead until someone logs in locally.
+   `Restart=on-failure` with `StartLimitIntervalSec=0` retries forever;
+   `WG_ENDPOINT_RESOLUTION_RETRIES` is lowered from the stock `infinity`
+   (which hangs the unit indefinitely) so a retry can actually happen.
+3. **Nothing heals a stale tunnel.** Roaming to a new network keeps the old
+   endpoint IP in the kernel, and a peer whose endpoint is unreachable simply
+   stops handshaking without the unit ever noticing, because the unit already
+   "succeeded". The watchdog acts on handshake age.
+
+## Install
+
+```bash
+HOST=root@<pi-address>
+D=$PWD        # run the snippets from a checkout of this directory
+
+ssh "$HOST" 'install -d -m 0755 /usr/local/sbin /etc/systemd/system/wg-quick@wg0.service.d'
+tar -C "$D" -cf - wg0-watchdog wg0-watchdog.service wg0-watchdog.timer \
+    'wg-quick@wg0.service.d-10-boot-resilience.conf' | \
+ssh "$HOST" 'tar -xf - -C /root &&
+  install -m 0755 /root/wg0-watchdog /usr/local/sbin/ &&
+  install -m 0644 /root/wg0-watchdog.service /root/wg0-watchdog.timer /etc/systemd/system/ &&
+  install -m 0644 "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" \
+    /etc/systemd/system/wg-quick@wg0.service.d/10-boot-resilience.conf &&
+  rm -f /root/wg0-watchdog /root/wg0-watchdog.service /root/wg0-watchdog.timer \
+    "/root/wg-quick@wg0.service.d-10-boot-resilience.conf" &&
+  systemctl daemon-reload && systemctl enable --now wg-quick@wg0.service wg0-watchdog.timer'
+```
+
+Keep `PersistentKeepalive = 25` in `[Peer]` (see `wg0.conf.template`) and check
+the config file is `0600` — a world-readable `wg0.conf` exposes the tunnel
+private key to every local account:
+
+```bash
+ssh "$HOST" 'ls -l /etc/wireguard/wg0.conf; wg show wg0 persistent-keepalive'
+```
+
+## Watchdog behaviour
+
+Oldest symptom first, one action per run:
+
+1. interface or peer gone → `systemctl restart --no-block wg-quick@wg0.service`
+2. no `PersistentKeepalive` in the kernel → `wg set ... persistent-keepalive <n>`
+   (covers a config edited without a reload)
+3. handshake older than `WG_WD_STALE` (default 120s) → re-point the peer at the
+   endpoint, preferring a fresh resolution of the hostname from the config file,
+   then the kernel's current endpoint, then the cached IP
+4. handshake older than `WG_WD_STALE_RESTART` (default 300s) → restart the unit
+
+Restarts are gap-limited to one per `WG_WD_MIN_RESTART_GAP` (default 180s) so a
+dead network cannot make the unit flap. `--no-block` keeps a hung `wg-quick up`
+from wedging the watchdog. Peers, addresses, and routes are only ever added.
+All thresholds and the interface/state paths are environment overrides, and
+`--dry-run` prints the decision without acting.
+
+The endpoint resolution order matters when the tunnel config also pushes a DNS
+server: that resolver is reachable only through the tunnel, so a dead tunnel can
+mean the endpoint hostname cannot be resolved exactly when it is needed — hence
+the cached-IP fallback.
+
+## DNS
+
+A `DNS = <resolver>` line in the interface section is pushed through
+`resolvconf`/`systemd-resolved` while the tunnel is up, which can silently make
+*all* name resolution on the host depend on the tunnel. If the Pi must resolve
+names on whatever network it lands on, omit the `DNS` key entirely and let the
+LAN's DHCP resolvers stand.
+
+Two gotchas:
+
+- Debian's `wg-quick` does **not** implement `DNS = none` — any non-IP value
+  becomes a search domain, so the line has to be absent, not set to `none`.
+- A record already pushed by a previous `wg-quick up` survives the config edit;
+  withdraw it (`resolvconf -d <record>`, name visible in `resolvconf --list`) and
+  confirm the local caching resolver's upstreams before believing it is gone.
+
+## Taking wg0 down on purpose
+
+Stop the watchdog first or the tunnel comes straight back:
+
+```bash
+ssh "$HOST" 'systemctl stop wg0-watchdog.timer wg-quick@wg0.service'
+ssh "$HOST" 'systemctl start wg0-watchdog.timer'      # resume
+```
+
+## Check on it
+
+```bash
+ssh "$HOST" 'wg show wg0
+  systemctl status wg-quick@wg0 --no-pager | head -6
+  systemctl list-timers "wg0-watchdog*" --no-pager | head -3
+  journalctl -u wg0-watchdog.service -b --no-pager | tail -15'
+```
+
+Dry-run the decision logic without touching the tunnel:
+
+```bash
+ssh "$HOST" 'WG_WD_STATE_DIR=/tmp/wd WG_WD_STALE=0 /usr/local/sbin/wg0-watchdog --dry-run; rm -rf /tmp/wd'
+```
+
+Validate an edited config without disturbing the running tunnel:
+
+```bash
+ssh "$HOST" 'wg-quick strip /etc/wireguard/wg0.conf | sed -E "s/^(PrivateKey[[:space:]]*=).*/\1 [REDACTED]/"'
+```
+
+## Diagnosis notes (generic, worth checking on any host)
+
+- `wg-quick@<iface>` can be `enabled` yet `inactive` while the interface exists
+  and passes traffic — it may have been created by hand or taken over by
+  NetworkManager as an assumed profile (`nmcli dev status` shows such a device
+  as `connected (externally)`). Nothing owns it: `systemctl reload` fails with
+  "unit not active", so apply changes with `wg set` or a full restart, and let
+  `systemctl restart` take ownership before trusting a reboot.
+- `Restart=on-failure` does work for `Type=oneshot` on systemd 257 — worth
+  proving on a new distribution with a throwaway unit under
+  `/run/systemd/system` before relying on it: `Type=oneshot` +
+  `ExecStart=/bin/false`, then check `systemctl show -p NRestarts`.
+- `systemd-analyze verify <unit>` catches mistyped unit keys, and it only warns —
+  read the output instead of trusting the exit status.
+- NetworkManager does not fight `wg-quick` as long as no keyfile exists for the
+  interface and its assumed profile keeps `autoconnect no`; check
+  `/etc/NetworkManager/system-connections/` before assuming that.
+- The real end-to-end test is a reboot: DHCP → endpoint resolution →
+  `wg-quick up` → inbound SSH through the tunnel. `systemd-analyze
+  critical-chain ssh.service` after boot shows whether the ordering held.
@@ -0,0 +1,48 @@
+# Reese's Workstation
+
+- [Reese's Workstation](#reeses-workstation)
+  - [Virtual Machines](#virtual-machines)
+
+## Virtual Machines
+
+1. Blueprints are stored in active/software_osbuild
+2. Builds are stored in ${HOME}/Images/builds
+
+```bash
+# These are saved in ~/.bashrc.d/osbuild.sh
+export OSBUILD_BLUEPRINT_DIR=active/software_osbuild
+export OSBUILD_CLOUD_DIR=${HOME}/Images/cloud
+export OSBUILD_BUILD_DIR=${HOME}/Images/builds
+export OSBUILD_IMAGE_DIR=${HOME}/Images/disks
+
+# Push a blueprint
+composer-cli blueprints push ${OSBUILD_BLUEPRINT_DIR}/fedora43-base.toml
+
+# Start a build
+composer-cli compose start fedora43-base qcow2
+
+# Watch the build
+watch composer-cli compose list running
+
+# Extract the build
+composer-cli compose image \
+--filename ${OSBUILD_BUILD_DIR}/fedora-43-base.qcow2 \
+92a2f1dd-9238-4d50-ab1d-cd7c8e355cac
+
+# Convert the build to raw
+qemu-img convert -f qcow2 -O raw \
+${OSBUILD_BUILD_DIR}/fedora-43-base.qcow2 \
+${OSBUILD_IMAGE_DIR}/fedora-boot.raw
+
+# Install (Change password for default user ducoterra!)
+virt-install \
+--boot uefi,firmware.feature0.name=secure-boot,firmware.feature0.enabled=no \
+--cpu host-passthrough --vcpus sockets=1,cores=8,threads=2 \
+--ram=8192 \
+--os-variant=fedora43 \
+--network default \
+--graphics none \
+--console pty,target.type=virtio \
+--name "fedora" \
+--import --disk "path=${OSBUILD_IMAGE_DIR}/fedora-boot.raw,bus=virtio"
+```
@@ -0,0 +1,89 @@
+# KDE Notes
+
+## Fixing alt tab or taskbar icons
+
+1. System Settings - Window Management - Window Rules - Add New...
+2. Bring the buggy app window side-by-side with system settings.
+3. Click "Detect Window Properties..." in the bottom.
+4. Move the crosshair to the buggy window and click.
+5. Click "Window class (application)".
+6. Click "Add Property..." in the bottm left.
+7. Click "Desktop file name".
+8. Enter the base name of the desktop file
+
+    For flatpak, check `/var/lib/flatpak/exports/share/applications/`
+
+    For system apps, check `/usr/share/applications/`
+
+    For user apps, check `~/.local/share/applications/`
+
+9. Click "Apply".
+10. The icon should be fixed immediately.
+
+## KDE Connect
+
+Allow KDE Connect via 1714-1764 tcp/udp
+
+```bash
+# Set source address to allow connections
+sudo firewall-cmd \
+--zone=drop \
+--permanent \
+--add-port=1714-1764/udp \
+--add-port=1714-1764/tcp
+
+sudo firewall-cmd --reload
+```
+
+## Automatic Display Switching
+
+```bash
+# List displays
+# Builtin: eDP-2
+# Roku: DP-13
+# Lenovo: DP-11
+kscreen-doctor -o
+
+# Put the builtin display back to normal
+kscreen-doctor \
+output.eDP-2.enable \
+output.eDP-2.position.0,0 \
+output.eDP-2.primary \
+output.eDP-2.mode.2560x1600@165 \
+output.eDP-2.scale.1.25
+
+# Show on Roku TV and Monitor
+kscreen-doctor \
+output.DP-11.enable \
+output.DP-11.position.0,0 \
+output.DP-11.mode.2560x1440@60 \
+output.DP-11.scale.1 \
+output.DP-13.enable \
+output.DP-13.position.0,0 \
+output.DP-13.mode.2560x1440@100 \
+output.DP-13.scale.1 \
+output.DP-13.primary \
+output.eDP-2.disable
+
+# Show only on the roku TV
+kscreen-doctor \
+output.DP-11.enable \
+output.DP-11.position.0,0 \
+output.DP-11.primary \
+output.DP-11.mode.3840x2160@60 \
+output.DP-11.scale.2 \
+output.DP-13.disable \
+output.eDP-2.disable
+
+# Mirror the builtin display to the roku tv
+kscreen-doctor \
+output.DP-11.enable \
+output.DP-11.position.0,0 \
+output.DP-11.primary \
+output.DP-11.mode.3840x2160@60 \
+output.DP-11.scale.2 \
+output.eDP-2.enable \
+output.eDP-2.mode.1920x1080@60 \
+output.eDP-2.scale.1 \
+output.eDP-2.position.0,0
+```
@@ -1,114 +1,204 @@
 # Fedora Kinoite

 - [Fedora Kinoite](#fedora-kinoite)
-  - [TPM2 Luks Decryption](#tpm2-luks-decryption)
-  - [Podman](#podman)
-    - [Docker Compose and Docker Buildkit with Rootless Podman](#docker-compose-and-docker-buildkit-with-rootless-podman)
-  - [rpm-ostree](#rpm-ostree)
-    - [Git, Vim, etc](#git-vim-etc)
-    - [Libvirt, Qemu, KVM](#libvirt-qemu-kvm)
-    - [ROCM](#rocm)
-    - [Ollama](#ollama)
-  - [Network](#network)
-    - [Hostname](#hostname)
-    - [VLAN Setup with nmcli](#vlan-setup-with-nmcli)
-  - [GPU Support in Distrobox](#gpu-support-in-distrobox)
-  - [Automatic Display Switching](#automatic-display-switching)
+  - [Adding users to groups (usermod replacement)](#adding-users-to-groups-usermod-replacement)
+  - [System Prereqs](#system-prereqs)
+  - [Toolbox](#toolbox)
+    - [Clipboard Sharing](#clipboard-sharing)
+    - [Running Host Commands](#running-host-commands)
+    - [Saving a Toolbox](#saving-a-toolbox)
+  - [Apps](#apps)
+    - [Gear Lever](#gear-lever)
+    - [VSCodium](#vscodium)
+    - [Ansible](#ansible)
+    - [OpenCode](#opencode)
+    - [Nextcloud](#nextcloud)
+    - [UV](#uv)
+    - [NVM](#nvm)
+  - [Set Hostname](#set-hostname)
+  - [rpm-ostree Notes](#rpm-ostree-notes)
+  - [Accent keys when holding a key](#accent-keys-when-holding-a-key)

-## TPM2 Luks Decryption
-
-Mostly taken from here:
-<https://gist.github.com/jdoss/777e8b52c8d88eb87467935769c98a95>
-
-PCR reference for `--tpm2-pcrs` args
-
-```text
-0: System firmware executable
-2: Kernel
-4: Bootloader
-7: Secure boot state
-8: Cmdline
-9: Initrd
-```
-
-Basic commands:
+## Adding users to groups (usermod replacement)

 ```bash
-# Show tpm2 devices
-systemd-cryptenroll --tpm2-device=list
-# Show crypto luks block devices
-blkid -t TYPE=crypto_LUKS
-
-# Enroll the tpm2 device with systemd-cryptenroll
-systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0,2,4,7,8,9 /dev/nvme0n1p3
-
-# Reenroll
-systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=0,2,4,7,8,9
-
-# Append to command line args
-rpm-ostree kargs --append=rd.luks.options=tpm2-device=auto
+grep -E '^weldr:' /usr/lib/group | sudo tee -a /etc/group
+sudo usermod -aG weldr $USER
 ```

-When you update you'll need to reenroll. Add this to your ~/.bashrc
+## System Prereqs

 ```bash
-# LUKS TPM2 commands
-alias tpm2-reenroll='sudo systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=0,2,4,7,8,9'
+# Appimage support
+rpm-ostree install fuse fuse-libs
+
+# Editor
+rpm-ostree install vim
+
+# Qemu/KVM
+rpm-ostree install virt-manager libvirt virt-install
+
+# OSBuild
+rpm-ostree install -y osbuild-composer composer-cli openscap-scanner scap-security-guide
+
+# Distrobox
+rpm-ostree install -y distrobox
+
+# BTRFS Snapshots
+rpm-ostree install -y snapper
+
+# Kitty Terminal
+# terminfo prevents the "WARNING: terminal is not fully functional" error
+rpm-ostree install -y kitty kitty-terminfo
 ```

-## Podman
+## Toolbox

-Since you'll be using podman for most container-based services, you'll want to set the
-the podman auth file to somewhere persistent, otherwise it'll get deleted every time you
-reboot.
+### Clipboard Sharing

-Add this to your `.bashrc`:
+Inside the toolbox run:

 ```bash
-# Podman auth file
-export REGISTRY_AUTH_FILE=$HOME/.podman-auth.json
+sudo dnf install -y wl-clipboard
 ```

-Source that and then run `podman login` to create the file.
+### Running Host Commands

-### Docker Compose and Docker Buildkit with Rootless Podman
+You'll want to use toolbox for development. Unfortunately, toolbox doesn't
+integrate well with host system operations.

-Allows you to use podman with full docker-compose compatibility.
-
-<https://emersion.fr/blog/2025/using-podman-compose-and-buildkit/>
+I create a `toolbox.sh` in `~/.bashrc.d` with functions that map to host
+commands like so:

 ```bash
-rpm-ostree install docker-compose docker-buildx
-reboot
-systemctl --user enable --now podman.socket
-docker context create podman --docker host=unix://$XDG_RUNTIME_DIR/podman/podman.sock
-docker context use podman
+# Use flatpak-spawn --host podman inside toolbox, regular podman outside
+podman() {
+    if [ -f /run/.containerenv ]; then
+        flatpak-spawn --host podman "$@"
+    else
+        command podman "$@"
+    fi
+}

-### Autostarting services with quadlets
-
-If you want to run something as your user at boot (like a systemd process, think ollama) you can
-create a user quadlets like so:
-
-```bash
-# Generate the .container file
-podman run --rm ghcr.io/containers/podlet --install --description "Local AI" \
-  podman run \
-  -d \
-  -v ollama:/root/.ollama \
-  -p 11434:11434 \
-  --name ollama \
-  --restart always \
-  docker.io/ollama/ollama > ~/.config/containers/systemd/ollama.container
-
-# Verify the service (Note the filename:service, this is required! You will get "Failed to prepare filename" without it)
-systemd-analyze verify ~/.config/containers/systemd/ollama.container:ollama.service
-
-# Start the service
-systemctl --user daemon-reload
-systemctl --user start ollama
+# Notify the user if they're trying to run a command that doesn't work in a toolbox
+rpm-ostree() {
+    if [ -f /run/.containerenv ]; then
+        echo "Command must be run outside toolbox."
+    else
+        command rpm-ostree "$@"
+    fi
+}
 ```

-## rpm-ostree
+I have the following functions mapped:
+
+1. podman
+2. rpm-ostree
+3. flatpak
+4. firewall-cmd
+
+### Saving a Toolbox
+
+After setting up a base toolbox you'll want to save it for future use. This is easy with podman commit.
+
+```bash
+# 1. Commit the running container to a new image
+podman commit fedora-toolbox-44 my-toolbox-snapshot
+
+# 2. Save the image to a tar file for later use
+podman save -o ~/my-toolbox-snapshot.tar my-toolbox-snapshot
+
+# Load the image
+podman load -i ~/my-toolbox-snapshot.tar
+
+# Create a new toolbox from it
+toolbox create --image my-toolbox-snapshot
+```
+
+## Apps
+
+Anything not mentioned here is installed via flatpak from flathub.
+
+### Gear Lever
+
+```bash
+flatpak install it.mijorus.gearlever
+```
+
+### VSCodium
+
+Install in toolbox.
+
+<https://vscodium.com/#install-on-fedora-rhel-centos-rockylinux-opensuse-rpm-package>
+
+```bash
+sudo tee -a /etc/yum.repos.d/vscodium.repo << 'EOF'
+[gitlab.com_paulcarroty_vscodium_repo]
+name=gitlab.com_paulcarroty_vscodium_repo
+baseurl=https://paulcarroty.gitlab.io/vscodium-deb-rpm-repo/rpms/
+enabled=1
+gpgcheck=1
+repo_gpgcheck=1
+gpgkey=https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg
+metadata_expire=1h
+EOF
+
+sudo dnf install codium
+```
+
+### Ansible
+
+```bash
+# Install pipx
+python3 -m ensurepip --user
+python3 -m pip install --user pipx
+pipx ensurepath
+
+# Install ansible
+pipx install --include-deps ansible
+```
+
+### OpenCode
+
+```bash
+curl -fsSL https://opencode.ai/install | bash
+
+# Install pyright LSP
+python -m pip install --upgrade pip
+python -m pip install --user pyright
+```
+
+### Nextcloud
+
+Download the AppImage. Install via Gear Lever.
+
+<https://github.com/nextcloud-releases/desktop/releases>
+
+| Source | Repo                       | Release file name |
+| ------ | -------------------------- | ----------------- |
+| Github | nextcloud-releases/desktop | *x86_64.AppImage  |
+
+### UV
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+```
+
+### NVM
+
+<https://www.nvmnode.com/guide/installation.html#nvm-install-for-linux-macos>
+
+```bash
+curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
+```
+
+## Set Hostname
+
+```bash
+sudo hostnamectl hostname reesework16
+```
+
+## rpm-ostree Notes

 Note: if you don't need them, remove extra repos

@@ -133,169 +223,6 @@ rpm-ostree apply-live
 rpm-ostree status
 ```

-### Git, Vim, etc
+## Accent keys when holding a key

-Some packages are nice to have at the system level.
-
-```bash
-rpm-ostree install git vim
-```
-
-### Libvirt, Qemu, KVM
-
-```bash
-rpm-ostree install virt-manager libvirt
-systemctl enable --now libvirtd
-```
-
-### ROCM
-
-Credit to:
-<https://medium.com/@seancheo/running-generative-ai-on-amd-in-fedora-40-28aa3bebb187>
-
-Here are most of the ROCM packages that you'll need.
-
-Fedora is currently working on a ROCM meta package here:
-<https://packages.fedoraproject.org/pkgs/rocm/rocm/>
-
-```bash
-rpm-ostree install \
-hipblas-devel \
-hipblaslt-devel \
-hipcc \
-hipcc-libomp-devel \
-hipcub-devel \
-hipfft-devel \
-hipfort-devel \
-hiprand-devel \
-hiprt-devel \
-hipsolver-devel \
-hipsparse-devel \
-rocalution-devel \
-rocblas-devel \
-rocfft-devel \
-rocm-clang-devel \
-rocm-clang-tools-extra-devel \
-rocm-cmake \
-rocm-comgr-devel \
-rocm-core-devel \
-rocm-hip-devel \
-rocm-libc++-devel \
-rocm-libc++-static \
-rocm-llvm-devel \
-rocm-omp-devel \
-rocm-runtime-devel \
-rocm-rpp-devel \
-rocm-smi-devel \
-rocminfo \
-rocdecode-devel \
-rocjpeg-devel \
-rocprim-devel \
-rocrand-devel \
-rocsolver-devel \
-rocsparse-devel \
-rocthrust-devel \
-roctracer-devel \
-miopen
-```
-
-### Ollama
-
-<https://ollama.com/download>
-
-Run the installation script as normal. Make sure you have the [ROCM](#rocm) drivers installed
-for GPU acceleration. The script *should* automatically pull the ROCM drivers after installing
-the base packages. If not, you should install them manually.
-
-For starting ollama as a service, follow the link below with one exception.
-
-<https://github.com/ollama/ollama/blob/main/docs/linux.md#adding-ollama-as-a-startup-service-recommended>
-
-```bash
-# Change the recommended home dir to /var/home/ollama
-sudo useradd -r -s /bin/false -U -m -d /var/home/ollama ollama
-```
-
-## Network
-
-### Hostname
-
-```bash
-sudo hostnamectl hostname reesework16
-```
-
-### VLAN Setup with nmcli
-
-```bash
-# VLAN 2
-nmcli conn
-export NMCLI_DEVICE=enp195s0f4u1u3
-nmcli connection add type VLAN con-name $NMCLI_DEVICE.2 dev $NMCLI_DEVICE id 2
-```
-
-## GPU Support in Distrobox
-
-Fix for `error="failed to check permission on /dev/kfd: open /dev/kfd: invalid argument"`
-
-```bash
-# You have to create the video and render group to /etc/group before you can use it
-sudo grep -E '^video:' /usr/lib/group | sudo tee -a /etc/group
-sudo grep -E '^render:' /usr/lib/group | sudo tee -a /etc/group
-sudo usermod -aG video $USER
-sudo usermod -aG render $USER
-```
-
-Logout and log back in to adopt new groups.
-
-## Automatic Display Switching
-
-```bash
-# List displays
-# Builtin: eDP-2
-# Roku: DP-13
-# Lenovo: DP-11
-kscreen-doctor -o
-
-# Put the builtin display back to normal
-kscreen-doctor \
-output.eDP-2.enable \
-output.eDP-2.position.0,0 \
-output.eDP-2.primary \
-output.eDP-2.mode.2560x1600@165 \
-output.eDP-2.scale.1.25
-
-# Show on Roku TV and Monitor
-kscreen-doctor \
-output.DP-11.enable \
-output.DP-11.position.0,0 \
-output.DP-11.mode.2560x1440@60 \
-output.DP-11.scale.1 \
-output.DP-13.enable \
-output.DP-13.position.0,0 \
-output.DP-13.mode.2560x1440@100 \
-output.DP-13.scale.1 \
-output.DP-13.primary \
-output.eDP-2.disable
-
-# Show only on the roku TV
-kscreen-doctor \
-output.DP-11.enable \
-output.DP-11.position.0,0 \
-output.DP-11.primary \
-output.DP-11.mode.3840x2160@60 \
-output.DP-11.scale.2 \
-output.DP-13.disable \
-output.eDP-2.disable
-
-# Mirror the builtin display to the roku tv
-kscreen-doctor \
-output.DP-11.enable \
-output.DP-11.position.0,0 \
-output.DP-11.primary \
-output.DP-11.mode.3840x2160@60 \
-output.DP-11.scale.2 \
-output.eDP-2.enable \
-output.eDP-2.mode.1920x1080@60 \
-output.eDP-2.scale.1 \
-output.eDP-2.position.0,0
-```
+Settings -> keyboard -> virtual keyboard -> disable virtual keyboard
@@ -4,6 +4,7 @@
  - [Installation](#installation)
  - [Setup SSH](#setup-ssh)
  - [DNF](#dnf)
+    - [Version lock](#version-lock)
  - [Power Profiles with Tuned](#power-profiles-with-tuned)
  - [Fail2Ban](#fail2ban)
  - [BTRFS Parent Volumes](#btrfs-parent-volumes)
@@ -84,6 +85,7 @@
  - [Sound Devices](#sound-devices)
  - [Install ffmpegthumbnailer, remove totem](#install-ffmpegthumbnailer-remove-totem)
  - [Add compatibility for HEIC to mogrify](#add-compatibility-for-heic-to-mogrify)
+  - [Redirecting ports for rootless services](#redirecting-ports-for-rootless-services)

 <https://docs.fedoraproject.org/en-US/fedora-server/installation/postinstallation-tasks/#_manage_system_updates>

@@ -127,6 +129,16 @@ dnf update --refresh -y
 dnf install -y git glances tmux vim python3-libdnf5 borgbackup tpm2-tools
 ```

+### Version lock
+
+```bash
+sudo dnf install dnf-plugin-versionlock
+sudo dnf list <package> --showduplicates
+sudo dnf versionlock add <package_name>-<version>.x86_64
+sudo dnf versionlock list
+sudo dnf versionlock delete <package_name>
+```
+
 ## Power Profiles with Tuned

 1. `dnf install tuned`
@@ -1486,3 +1498,29 @@ sudo dnf install ffmpegthumbnailer
 ```bash
 sudo dnf install libheic-freeworld
 ```
+
+## Redirecting ports for rootless services
+
+```bash
+# ipv4
+sudo firewall-cmd \
+--permanent \
+--add-forward-port=port=80:proto=tcp:toport=8080
+
+sudo firewall-cmd --add-rich-rule='forward-port port=80 protocol=tcp to-port=8080
+```
+
+Or in ansible
+
+```yaml
+- name: Forward ports
+    ansible.posix.firewalld:
+    port_forward:
+    - port: "{{ item.from }}"
+        proto: tcp
+        toport: "{{ item.to }}"
+    permanent: true
+    state: enabled
+    immediate: true
+    loop: "{{ forward_ports }}"
+```
@@ -0,0 +1,59 @@
+# This is the sudoers file for the ops user to give limited access to specific commands without a password
+# Used for automations like ansible and maintenance
+
+# Hostname
+ops ALL=(root) NOPASSWD: /usr/bin/hostnamectl hostname *
+
+# Reboot
+ops ALL=(root) NOPASSWD: /usr/bin/reboot
+
+# System updates
+ops ALL=(root) NOPASSWD: /usr/bin/dnf update*, /usr/bin/dnf upgrade*, /usr/bin/dnf install *, /usr/bin/dnf remove *
+
+# Package cache management
+ops ALL=(root) NOPASSWD: /usr/bin/dnf clean all, /usr/bin/dnf makecache, /usr/bin/dnf autoremove
+
+# Repos
+ops ALL=(root) NOPASSWD: /usr/bin/dnf config-manager addrepo *
+
+# Service management
+ops ALL=(root) NOPASSWD: /usr/bin/systemctl start *, /usr/bin/systemctl restart *, /usr/bin/systemctl reload *, /usr/bin/systemctl enable *
+
+# Firewall
+ops ALL=(root) NOPASSWD: /usr/bin/firewall-cmd *
+
+# Ansible
+ops ALL=(root) NOPASSWD: /bin/sh -c echo BECOME-SUCCESS-* ; /usr/bin/python3 /home/ops/.ansible*
+
+# qemu-img
+ops ALL=(root) NOPASSWD: /usr/bin/qemu-img *
+
+# virsh
+ops ALL=(root) NOPASSWD: /usr/bin/virsh *
+
+# virt-install
+ops ALL=(root) NOPASSWD: /usr/bin/virt-install *
+
+# composer-cli
+ops ALL=(root) NOPASSWD: /usr/bin/composer-cli *
+
+# osbuild
+ops ALL=(root) NOPASSWD: /usr/local/bin/osbuild-auto.sh
+
+# getfacl
+ops ALL=(root) NOPASSWD: /usr/sbin/getfacl*
+
+# set default rwx sbin
+ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -d -m u\:ops\:rwx /usr/local/sbin
+
+# set existing rwx sbin
+ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -m u\:ops\:rwx /usr/local/sbin
+
+# set default rwx /srv
+ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -d -m u\:ops\:rwx /srv/*
+
+# set existing rwx /srv
+ops ALL=(root) NOPASSWD: /usr/sbin/setfacl -R -m u\:ops\:rwx /srv/*
+
+# Start btrfs scrub
+ops ALL=(root) NOPASSWD: /usr/bin/btrfs scrub*
@@ -1,2 +0,0 @@
-[Network]
-Internal=true
@@ -1,14 +0,0 @@
-[Pod]
-Network=ai-internal.network
-# llama.cpp server
-PublishPort=8000:8000/tcp
-# llama.cpp embed
-PublishPort=8001:8001/tcp
-# llama.cpp instruct
-PublishPort=8002:8002/tcp
-# llama.cpp code
-PublishPort=8003:8003/tcp
-# stable-diffusion.cpp gen
-PublishPort=1234:1234/tcp
-# stable-diffusion.cpp edit
-PublishPort=1235:1235/tcp
@@ -0,0 +1,754 @@
+# Framework Desktop AI Deployment Guide
+
+Replicating my local AI inference stack: three [llama.cpp](https://github.com/ggml-org/llama.cpp) OpenAI-compatible model servers on a Framework Desktop (AMD Ryzen AI Max 300, Strix Halo iGPU).
+
+Key design decisions:
+
+- **Rootless Podman Quadlet** — no docker-compose, no system services. Everything runs under a dedicated `ai` user via systemd user units. `~/.config/containers/systemd/` is the single source of truth.
+- **Locally built image** — `llama.cpp` is cloned to the box and its official Vulkan Dockerfile is built with podman. The image entrypoint is `/app/llama-server`, so quadlet `Exec=` args are passed straight to the server.
+- **Air-gapped model network** — all model servers sit on an `Internal=true` podman network with no internet egress. Ports are published to the host for API access.
+- **Ansible-driven** — playbooks on the workstation copy the right quadlet into place over SSH and restart the user-scoped systemd service. Swapping models is a one-line playbook var change.
+
+```
+                      workstation (laptop)
+                      ansible-playbook over SSH as user `ai`
+                                     |
+                                     v
+-----------------------------------------------------------------------+
+|  Framework Desktop "deskwork"  (Fedora 43 Server Edition)            |
+|                                                                      |
+|  user `ai`  (loginctl linger on, rootless podman 5.x)                |
+|  /home/ai/.config/containers/systemd/                                |
+|  ├── ai-internal.network   -> network systemd-ai-internal            |
+|  │                               (Internal=true, no internet)        |
+|  ├── ai-embed.container    -> :8001  EmbeddingGemma-300M             |
+|  ├── ai-lite.container     -> :8002  gemma-4-E4B-it-qat (+MTP draft) |
+|  └── ai-turbo.container    -> :8003  Qwen3.6-35B-A3B (MTP)           |
+|                                                                      |
+|  image : localhost/llama-cpp-vulkan:latest   (built from git clone)  |
+|  models: /home/ai/models/{text,embedding}/...   (GGUF via hf CLI)    |
+|  GPU   : /dev/kfd + /dev/dri passed into every container (Vulkan)    |
+-----------------------------------------------------------------------+
+```
+
+| Port | Service  | Model                     | Source quadlet                                   |
+| ---- | -------- | ------------------------- | ------------------------------------------------ |
+| 8001 | ai-embed | EmbeddingGemma-300M       | `embed/quadlets/embeddinggemma-embed.container`  |
+| 8002 | ai-lite  | gemma-4-E4B-it-qat (+MTP) | `lite/quadlets/gemma4-e4b-qat-lite.container`    |
+| 8003 | ai-turbo | Qwen3.6-35B-A3B (MTP)     | `turbo/quadlets/qwen3.6-35b-a3b-turbo.container` |
+
+---
+
+## 1. Hardware
+
+Framework Desktop with an AMD Ryzen AI Max 300 (Strix Halo) — the iGPU is a Vulkan device with access to the unified memory. 128 GB unified memory recommended.
+
+### BIOS
+
+<https://knowledgebase.frame.work/en_us/changing-memory-allocation-amd-ryzen-ai-max-300-series-By1LG5Yrll>
+
+1. Set GPU memory to 512MB
+
+### Kernel args
+
+Edit `/etc/default/grub` and add the following to `GRUB_CMDLINE_LINUX`:
+
+```conf
+amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856
+```
+
+Then regenerate grub and reboot:
+
+```bash
+sudo grub2-mkconfig -o /boot/grub2/grub.cfg
+sudo reboot
+```
+
+After boot, verify the GPU is visible:
+
+```bash
+ls -l /dev/kfd /dev/dri/renderD128
+```
+
+Expected (note the world-writable modes — that's why the `ai` user needs no special groups):
+
+```text
+crw-rw-rw-. 1 root render 234, 0 ... /dev/kfd
+crw-rw-rw-. 1 root render 226, 128 ... /dev/dri/renderD128
+```
+
+## 2. Operating System
+
+Fedora 43 (Server Edition, minimal is fine). Requires cgroup v2 and a recent podman with Quadlet support (`podman-systemd` generator) — Fedora 40+ ships it.
+
+```bash
+sudo dnf install -y podman git
+podman --version        # 5.x
+podman info --format '{{.Host.CgroupsVersion}}'   # must print: v2
+```
+
+## 3. The `ai` user
+
+All containers run rootless as a dedicated `ai` user with linger enabled (so the user systemd manager + containers survive logout and start at boot).
+
+On the Framework Desktop:
+
+```bash
+sudo useradd -m ai
+sudo loginctl enable-linger ai
+
+# Give the ai user SSH access from your workstation
+sudo -u ai mkdir -p /home/ai/.ssh
+sudo -u ai tee /home/ai/.ssh/authorized_keys > /dev/null < ~/.ssh/id_ed25519.pub
+sudo -u ai chmod 700 /home/ai/.ssh
+sudo -u ai chmod 600 /home/ai/.ssh/authorized_keys
+```
+
+On your workstation's `~/.ssh/config`:
+
+```text
+Host deskwork-ai
+    HostName deskwork.reeselink.com
+    User ai
+```
+
+(Use whatever hostname/IP your Framework Desktop has. The inventory and playbooks reference the host as `deskwork-ai`.)
+
+## 4. Models
+
+As the `ai` user on the Framework Desktop.
+
+### Install the Hugging Face CLI
+
+<https://huggingface.co/docs/huggingface_hub/en/guides/cli#getting-started>
+
+```bash
+curl -LsSf https://hf.co/cli/install.sh | bash
+hf auth login
+```
+
+### Create the model dirs
+
+```bash
+mkdir -p /home/ai/models/{text,image,video,embedding,tts,stt}
+```
+
+### Download the three model sets
+
+```bash
+# --- embed (ai-embed, port 8001) ---
+mkdir -p /home/ai/models/embedding/emebeddinggemma-300m
+cd /home/ai/models/embedding/emebeddinggemma-300m
+# NOTE: the "emebeddinggemma" typo is intentional — it matches the quadlet below
+hf download --local-dir . ggml-org/EmbeddingGemma-300M-GGUF embeddinggemma-300M-BF16.gguf
+```
+
+```bash
+# --- lite (ai-lite, port 8002) ---
+mkdir -p /home/ai/models/text/gemma-4-e4b-it-qat
+cd /home/ai/models/text/gemma-4-e4b-it-qat
+# main model + MTP draft (speculative decoding) + vision projector
+hf download --local-dir . unsloth/gemma-4-E4B-it-qat-GGUF \
+  --include gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
+  --include mtp-gemma-4-E4B-it.gguf \
+  --include mmproj-BF16.gguf
+```
+
+```bash
+# --- turbo (ai-turbo, port 8003) ---
+mkdir -p /home/ai/models/text/qwen3.6-35b-a3b-mtp
+cd /home/ai/models/text/qwen3.6-35b-a3b-mtp
+hf download --local-dir . unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
+  --include Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
+  --include mmproj-F32.gguf
+```
+
+The turbo model is ~37 GB; make sure you have the space (`df -h /home`).
+
+## 5. llama.cpp + the Vulkan image
+
+As the `ai` user on the Framework Desktop:
+
+```bash
+cd /home/ai
+git clone https://github.com/ggml-org/llama.cpp.git
+cd llama.cpp
+podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
+```
+
+The multi-stage `.devops/vulkan.Dockerfile` builds the full toolset with the Vulkan backend; the final stage's entrypoint is `/app/llama-server`, so any args after the image name go straight to the server. Podman tags local builds under the `localhost/` namespace, which is why quadlets reference `localhost/llama-cpp-vulkan:latest`.
+
+Verify:
+
+```bash
+podman images | grep llama-cpp-vulkan
+```
+
+Optional extras (only if you use them):
+
+```bash
+# ROCm image (for the AMD dGPU boxes)
+podman build -t llama-cpp-rocm:latest -f .devops/rocm.Dockerfile .
+
+# Diffusion variant (diffusion-gemma turbo quadlet) — built from a fork that
+# has diffusion support wired in
+git clone https://github.com/danielhanchen/llama.cpp.git llama.cpp-diffusion
+cd llama.cpp-diffusion
+podman build -t llama-cpp-diffusion-vulkan:latest -f .devops/vulkan.Dockerfile .
+```
+
+## 6. Quadlet primer
+
+Podman Quadlet reads files from `~/.config/containers/systemd/` (rootless search path) and generates systemd user units via a systemd generator. No compose files, no hand-written units.
+
+Naming rules that matter here:
+
+| Quadlet file    | Generated systemd service | Podman resource                                             |
+| --------------- | ------------------------- | ----------------------------------------------------------- |
+| `foo.container` | `foo.service`             | container `systemd-foo` (unless `ContainerName=` overrides) |
+| `foo.network`   | `foo-network.service`     | network `systemd-foo`                                       |
+
+- `Network=ai-internal.network` in a container quadlet means: use the network from the `ai-internal.network` quadlet file **and** auto-add a dependency on `ai-internal-network.service`, so the network exists before the container starts.
+- Quadlet units are "transient" (generated), so `systemctl enable` doesn't apply — the generator applies the `[Install]` section itself. That's what `WantedBy=multi-user.target default.target` in every file below is for: start at boot.
+- Image pulls can exceed systemd's default 90s start timeout, hence `TimeoutStartSec=900`.
+- Debugging: `systemd-analyze --user verify <unit>.service` and `/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun`.
+
+Full quadlet reference: <https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html>
+
+## 7. The files
+
+Everything below lives in the `Deployments` repo on the workstation. Layout:
+
+```text
+Deployments/
+├── inventory.yaml
+└── reeseapps/ai/
+    ├── roles/containers-infra/
+    │   ├── tasks/main.yaml
+    │   └── templates/ai-internal.network
+    └── deployments/
+        ├── embed/
+        │   ├── playbook.yaml
+        │   └── quadlets/embeddinggemma-embed.container
+        ├── lite/
+        │   ├── playbook.yaml
+        │   └── quadlets/
+        │       ├── gemma4-e2b-qat-lite.container
+        │       ├── gemma4-e4b-qat-lite.container   <- active
+        │       └── lfm2.5-2.6b.container
+        └── turbo/
+            ├── playbook.yaml
+            └── quadlets/
+                ├── diffusion-gemma.container
+                ├── gemma-4-26b-a4b-turbo.container
+                ├── gemma-4-qat-turbo.container
+                ├── muse-glimmer-30b-juggernaut-turbo.container
+                └── qwen3.6-35b-a3b-turbo.container <- active
+```
+
+### 7.1 inventory.yaml
+
+Only the `ai` group is needed for this stack:
+
+```yaml
+ai:
+  hosts:
+    deskwork-ai:
+      ansible_python_interpreter: /usr/bin/python3
+```
+
+### 7.2 The `containers-infra` role
+
+Runs first in every playbook. Creates the quadlet dir and installs the air-gapped network.
+
+`reeseapps/ai/roles/containers-infra/tasks/main.yaml`:
+
+```yaml
+- name: Create /home/ai/.config/containers/systemd
+  ansible.builtin.file:
+    path: /home/ai/.config/containers/systemd
+    state: directory
+    mode: "0755"
+
+- name: Copy infra quadlets
+  template:
+    src: "{{ item }}"
+    dest: "/home/ai/.config/containers/systemd/{{ item }}"
+  loop:
+    - ai-internal.network
+```
+
+`reeseapps/ai/roles/containers-infra/templates/ai-internal.network`:
+
+```ini
+[Network]
+Internal=true
+```
+
+`Internal=true` gives the network NAT-style local connectivity but no internet egress — the model servers don't need it.
+
+### 7.3 The playbooks
+
+All three playbooks have the same shape: run the infra role, copy the selected quadlet (chosen by the `container_file` var) to a fixed destination name, then daemon-reload and restart the resulting user service.
+
+`reeseapps/ai/deployments/embed/playbook.yaml`:
+
+```yaml
+- name: Create Embedding AI Stack
+  hosts: deskwork-ai
+  vars:
+    container_file: quadlets/embeddinggemma-embed.container
+  roles:
+    - ../../roles/containers-infra
+  tasks:
+    - name: Copy ai-embed.container
+      copy:
+        src: "{{ container_file }}"
+        dest: /home/ai/.config/containers/systemd/ai-embed.container
+        owner: ai
+        group: ai
+        mode: "0644"
+    - name: Reload and start the ai-embed service
+      ansible.builtin.systemd_service:
+        state: restarted
+        name: ai-embed
+        daemon_reload: true
+        scope: user
+```
+
+`reeseapps/ai/deployments/lite/playbook.yaml`:
+
+```yaml
+- name: Create Lite Service
+  hosts: deskwork-ai
+  vars:
+    container_file: quadlets/gemma4-e4b-qat-lite.container
+  roles:
+    - ../../roles/containers-infra
+  tasks:
+    - name: Copy ai-lite.container
+      copy:
+        src: "{{ container_file }}"
+        dest: /home/ai/.config/containers/systemd/ai-lite.container
+        owner: ai
+        group: ai
+        mode: "0644"
+    - name: Reload and start the ai-lite service
+      ansible.builtin.systemd_service:
+        state: restarted
+        name: ai-lite
+        daemon_reload: true
+        scope: user
+```
+
+`reeseapps/ai/deployments/turbo/playbook.yaml`:
+
+```yaml
+- name: Create Turbo AI Stack
+  hosts: deskwork-ai
+  vars:
+    container_file: quadlets/qwen3.6-35b-a3b-turbo.container
+  roles:
+    - ../../roles/containers-infra
+  tasks:
+    - name: Copy ai-turbo.container
+      copy:
+        src: "{{ container_file }}"
+        dest: /home/ai/.config/containers/systemd/ai-turbo.container
+        owner: ai
+        group: ai
+        mode: "0644"
+    - name: Reload and start the ai-turbo service
+      ansible.builtin.systemd_service:
+        state: restarted
+        name: ai-turbo
+        daemon_reload: true
+        scope: user
+```
+
+To swap which model a service runs, change only the `container_file` var and re-run that playbook.
+
+### 7.4 The quadlets (currently active variants)
+
+`embed/quadlets/embeddinggemma-embed.container` → installed as `ai-embed.container`:
+
+```ini
+[Unit]
+Description=A Llama CPP Server For Embedding Models
+
+[Container]
+ContainerName=ai-embed
+
+# Internet-disconnected network
+Network=ai-internal.network
+
+# llama.cpp juggernaut
+PublishPort=8001:8001/tcp
+
+# Image is built locally via podman build
+Image=localhost/llama-cpp-vulkan:latest
+AutoUpdate=registry
+
+# Downloaded models volume
+Volume=/home/ai/models/embedding:/models:z
+
+# GPU Device
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+
+# Server command
+Exec=--port 8001 \
+    -c 0 \
+    -b 1024 \
+    -ub 1024 \
+    --perf \
+    --n-gpu-layers all \
+    --embedding \
+    -m /models/emebeddinggemma-300m/embeddinggemma-300M-BF16.gguf \
+    --alias embed
+
+# Health Check
+HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8001/health || exit 1
+HealthInterval=10s
+HealthRetries=3
+HealthStartPeriod=10s
+HealthTimeout=30s
+HealthOnFailure=kill
+
+# EnvironmentFile=/home/ai/.llama-api/keys.env
+
+[Service]
+Restart=always
+# Extend Timeout to allow time to pull the image
+TimeoutStartSec=900
+
+[Install]
+# Start by default on boot
+WantedBy=multi-user.target default.target
+```
+
+`lite/quadlets/gemma4-e4b-qat-lite.container` → installed as `ai-lite.container`:
+
+```ini
+[Unit]
+Description=A Llama CPP Server Running a Non-Reasoning Model
+
+[Container]
+ContainerName=ai-lite
+
+# Internet-disconnected network
+Network=ai-internal.network
+
+# llama.cpp juggernaut
+PublishPort=8002:8002/tcp
+# Image is built locally via podman build
+Image=localhost/llama-cpp-vulkan:latest
+AutoUpdate=registry
+
+# Downloaded models volume
+Volume=/home/ai/models/text:/models:z
+
+# GPU Device
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+
+# Server command
+Exec=--port 8002 \
+    --parallel 1 \
+    --temp 1.0 \
+    --top-p 0.95 \
+    --top-k 64 \
+    -fa on \
+    --no-mmap \
+    --kv-unified \
+    --perf \
+    --spec-type draft-mtp --spec-draft-n-max 3 \
+    --reasoning off \
+    --model /models/gemma-4-e4b-it-qat/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
+    --model-draft /models/gemma-4-e4b-it-qat/mtp-gemma-4-E4B-it.gguf \
+    --mmproj /models/gemma-4-e4b-it-qat/mmproj-BF16.gguf \
+    --alias lite
+
+# Health Check
+HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8002/health || exit 1
+HealthInterval=10s
+HealthRetries=3
+HealthStartPeriod=10s
+HealthTimeout=30s
+HealthOnFailure=kill
+
+# EnvironmentFile=/home/ai/.llama-api/keys.env
+
+[Service]
+Restart=always
+# Extend Timeout to allow time to pull the image
+TimeoutStartSec=900
+
+[Install]
+# Start by default on boot
+WantedBy=multi-user.target default.target
+```
+
+`turbo/quadlets/qwen3.6-35b-a3b-turbo.container` → installed as `ai-turbo.container`:
+
+```ini
+[Unit]
+Description=A Llama CPP Server Running a Reasoning Model
+
+[Container]
+ContainerName=ai-turbo
+
+# Internet-disconnected network
+Network=ai-internal.network
+
+# llama.cpp juggernaut
+PublishPort=8003:8003/tcp
+
+# Image is built locally via podman build
+# latest-mtp is for mtp testing
+# latest is main branch
+Image=localhost/llama-cpp-vulkan:latest
+AutoUpdate=registry
+
+# Downloaded models volume
+Volume=/home/ai/models/text:/models:z
+
+# GPU Device
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+
+# Server command
+Exec=--port 8003 \
+    --temp 0.7 \
+    --top-p 0.8 \
+    --top-k 20 \
+    --presence-penalty 1.5 \
+    --min-p 0.00 \
+    --chat-template-kwargs '{"enable_thinking":false}' \
+    -ctk q8_0 \
+    -ctv q8_0 \
+    --kv-unified \
+    --parallel 2 \
+    -fa on \
+    --load-mode none \
+    --image-min-tokens 4096 \
+    --image-max-tokens 4096 \
+    --n-gpu-layers all \
+    --perf \
+    --jinja \
+    --reasoning-preserve \
+    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
+    --mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \
+    --spec-type draft-mtp --spec-draft-n-max 3 \
+    --spec-draft-type-k q8_0 \
+    --spec-draft-type-v q8_0 \
+    --alias turbo
+
+# Health Check
+# CMD-SHELL is string form, CMD is array form
+HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8003/health || exit 1
+HealthInterval=10s
+HealthRetries=3
+HealthStartPeriod=30s
+HealthTimeout=30s
+HealthOnFailure=kill
+
+[Service]
+Restart=always
+# Extend Timeout to allow time to pull the image
+TimeoutStartSec=900
+
+[Install]
+# Start by default on boot
+WantedBy=multi-user.target default.target
+```
+
+Notes on the server flags:
+
+- `--alias` sets the model name served on `/v1/models`.
+- `--spec-type draft-mtp` + `--model-draft` (or the MTP head inside the main GGUF for Qwen) enables MTP speculative decoding for a big speedup.
+- `-fa on` = flash attention; `--n-gpu-layers all` = offload everything to the GPU; `--perf` prints timing stats to the log.
+- The healthcheck hits the server's `/health` endpoint; `HealthOnFailure=kill` + `Restart=always` means a wedged server is killed and restarted by systemd.
+- `:z` on the volume = SELinux shared-label for rootless containers.
+
+## 8. Ansible setup (workstation)
+
+From zero on the workstation:
+
+```bash
+# Install ansible (my setup uses pipx; uv tool install ansible works too)
+pipx install ansible
+ansible-playbook --version
+```
+
+No Ansible Galaxy collections are required for this stack — the playbooks only use `ansible.builtin` modules and the local `containers-infra` role (referenced by relative path `../../roles/containers-infra`, which is why the directory layout in §7.1 must be preserved).
+
+Put the files from §7 into a `Deployments/` directory on the workstation (or clone the full repo if you have access). Then, from the repo root:
+
+```bash
+cd Deployments
+
+# Order matters only for the first run: embed creates the shared network.
+ansible-playbook -i inventory.yaml reeseapps/ai/deployments/embed/playbook.yaml
+ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml
+ansible-playbook -i inventory.yaml reeseapps/ai/deployments/turbo/playbook.yaml
+```
+
+Each run is idempotent and safe to re-run. The first turbo start will take a while (37 GB model load) — that's what `TimeoutStartSec=900` is for.
+
+## 9. Verification
+
+On the Framework Desktop (as `ai`):
+
+```bash
+systemctl --user status ai-embed ai-lite ai-turbo
+podman ps
+podman network ls | grep systemd-ai-internal
+journalctl --user -fu ai-turbo   # watch startup, look for the --perf stats
+```
+
+Expected `podman ps`:
+
+```text
+ai-embed   localhost/llama-cpp-vulkan:latest   0.0.0.0:8001->8001/tcp
+ai-lite    localhost/llama-cpp-vulkan:latest   0.0.0.0:8002->8002/tcp
+ai-turbo   localhost/llama-cpp-vulkan:latest   0.0.0.0:8003->8003/tcp
+```
+
+From the host (OpenAI-compatible API):
+
+```bash
+# Health
+curl http://localhost:8001/health
+curl http://localhost:8002/health
+curl http://localhost:8003/health
+
+# List models
+curl -s http://localhost:8003/v1/models | jq
+
+# Chat completion (turbo)
+curl -s http://localhost:8003/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "turbo",
+    "messages": [
+      {"role": "user", "content": "Say hello in five words."}
+    ],
+    "max_tokens": 100
+  }' | jq
+
+# Chat completion (lite)
+curl -s http://localhost:8002/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "lite",
+    "messages": [
+      {"role": "user", "content": "Say hello in five words."}
+    ],
+    "max_tokens": 100
+  }' | jq
+
+# Embeddings (embed)
+curl -s http://localhost:8001/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "embed",
+    "input": "This is the reason you ended up here."
+  }' | jq '.data[0].embedding[0:5]'
+```
+
+## 10. Day-2 operations
+
+### Swap the model a service runs
+
+Change `container_file` in that service's playbook and re-run it:
+
+```bash
+# e.g. run the smaller 2B gemma on ai-lite instead
+$EDITOR reeseapps/ai/deployments/lite/playbook.yaml
+#    container_file: quadlets/gemma4-e2b-qat-lite.container
+ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml
+```
+
+Available swap-in quadlets:
+
+- lite: `gemma4-e2b-qat-lite.container`, `gemma4-e4b-qat-lite.container` (active), `lfm2.5-2.6b.container`
+- turbo: `qwen3.6-35b-a3b-turbo.container` (active), `gemma-4-26b-a4b-turbo.container`, `gemma-4-qat-turbo.container`, `muse-glimmer-30b-juggernaut-turbo.container`, `diffusion-gemma.container` (uses `localhost/llama-cpp-diffusion-vulkan:latest`)
+
+Make sure the model files the new quadlet references are already downloaded (see the `--model` / `--mmproj` / `--model-draft` paths inside it).
+
+### Update llama.cpp (image rebuild)
+
+On the Framework Desktop as `ai`:
+
+```bash
+cd /home/ai/llama.cpp
+git pull
+podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
+
+# Restart the servers so they pick up the new image
+systemctl --user restart ai-embed ai-lite ai-turbo
+```
+
+Tagged snapshots are a good habit:
+
+```bash
+export BUILD_TAG=$(date +"%Y-%m-%d-%H-%M-%S")
+podman build -t llama-cpp-vulkan:${BUILD_TAG} -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
+```
+
+Note: `AutoUpdate=registry` in the quadlets is effectively a no-op for `localhost/` images (podman auto-update needs a remote registry to compare against) — updates are manual, as above.
+
+### Logs and troubleshooting
+
+```bash
+journalctl --user -fu ai-turbo                  # follow logs
+systemctl --user status ai-embed ai-lite ai-turbo
+podman exec ai-turbo curl -s http://127.0.0.1:8003/health
+systemd-analyze --user verify ai-turbo.service  # validate generated unit
+/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun
+```
+
+Common failures:
+
+- `Unit ai-turbo.service not found` → the generator failed on a bad quadlet option (often a syntax error or an option your podman version doesn't know). Run the `podman-system-generator --dryrun` command above to see the error.
+- GPU errors at startup → confirm `/dev/kfd` and `/dev/dri/renderD128` exist and the user can read/write them; on a box where they're `0660`, add the user: `sudo usermod -aG render ai` then re-login.
+- Service times out on first start → normal while loading a big model; `TimeoutStartSec=900` covers ~15 min.
+- SELinux `avc` denials on model reads → make sure volume mounts use the `:z` suffix.
+
+### Manual escape hatch
+
+Skip systemd entirely and run a server by hand (useful for testing a new model):
+
+```bash
+podman run -it --rm \
+--device=/dev/kfd \
+--device=/dev/dri \
+-v /home/ai/models/text:/models:z \
+--entrypoint /bin/bash \
+localhost/llama-cpp-vulkan:latest
+
+./llama-server \
+-m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
+--mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \
+--port 8003 \
+-ctk q8_0 -ctv q8_0 \
+--kv-unified \
+--parallel 2 \
+-fa on \
+--load-mode none \
+--n-gpu-layers all \
+--perf \
+--jinja \
+--reasoning-preserve \
+--spec-type draft-mtp --spec-draft-n-max 3 \
+--spec-draft-type-k q8_0 \
+--spec-draft-type-v q8_0
+```
+
+## References
+
+- Podman Quadlet: <https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html>
+- llama.cpp server docs: <https://github.com/ggml-org/llama.cpp/tree/master/tools/server>
+- Framework Desktop hardware notes: `active/device_framework_desktop/framework_desktop.md`
+- Broader AI stack notes (model lists, benchmarks, vLLM, stable-diffusion.cpp): `active/software_ai_stack/ai_stack.md`
@@ -3,10 +3,6 @@
 - [Self Hosted AI Stack](#self-hosted-ai-stack)
  - [Notes](#notes)
    - [Podman Volume Locations](#podman-volume-locations)
-    - [List of Internal Links](#list-of-internal-links)
-  - [Quick Install](#quick-install)
-    - [Text Stack](#text-stack)
-    - [Image Stack](#image-stack)
  - [Setup](#setup)
    - [Create the AI user](#create-the-ai-user)
    - [Helper aliases](#helper-aliases)
@@ -31,18 +27,12 @@
        - [Nomic Embedding](#nomic-embedding)
  - [llama.cpp](#llamacpp)
  - [stable-diffusion.cpp](#stable-diffusioncpp)
+  - [STT whisper.cpp](#stt-whispercpp)
+  - [TTS coqui-tts](#tts-coqui-tts)
  - [open-webui](#open-webui)
+    - [Upgrade Open Webui](#upgrade-open-webui)
  - [lite-llm](#lite-llm)
-  - [Install Services with Quadlets](#install-services-with-quadlets)
-    - [API Keys](#api-keys)
-    - [Internal and External Pods](#internal-and-external-pods)
-    - [Llama CPP Server (Port 8000)](#llama-cpp-server-port-8000)
-    - [Llama CPP Embedding Server (Port 8001)](#llama-cpp-embedding-server-port-8001)
-    - [Llama CPP Instruct Server (Port 8002)](#llama-cpp-instruct-server-port-8002)
-    - [Stable Diffusion CPP (Port 1234 and 1235)](#stable-diffusion-cpp-port-1234-and-1235)
-    - [Open Webui (Port 8080)](#open-webui-port-8080)
-    - [Install the update script](#install-the-update-script)
-    - [Install Guest Open Webui with Start/Stop Services](#install-guest-open-webui-with-startstop-services)
+  - [Install the update script](#install-the-update-script)
  - [Benchmark Results](#benchmark-results)
  - [Testing with Curl](#testing-with-curl)
    - [OpenAI API](#openai-api)
@@ -74,33 +64,6 @@ hf-download ()

 `~/.local/share/containers/storage/volumes/`

-### List of Internal Links
-
- llama-cpp
- llama-embed
- llama-instruct
- image-gen
- image-edit
- openwebui
-
-## Quick Install
-
-### Text Stack
-
-```bash
-ansible-playbook \
-i ansible/inventory.yaml \
-active/software_ai_stack/install_ai_text_stack.yaml
-```
-
-### Image Stack
-
-```bash
-ansible-playbook \
-i ansible/inventory.yaml \
-active/software_ai_stack/install_ai_image_stack.yaml
-```
-
 ## Setup

 ### Create the AI user
@@ -183,8 +146,7 @@ rsync -av --progress /home/ai/models/ /srv/models/
 In my completely subjective opinion: 5 bit quant is usually the sweet spot for
 unsloth models. Q5_K_S is usually just fine.

-I usually download the F16 mmproj files. This is also completely subjective.
-BF16 is fine. F32 is overkill.
+I usually download the BF16 mmproj files. This is also completely subjective.

 #### Text models

@@ -388,20 +350,43 @@ podman run \
 --name llama-server-demo \
 --device=/dev/kfd \
 --device=/dev/dri \
-v /home/ai/models/text:/models:z \
+-v $(pwd):/models:z \
 -p 8010:8000 \
 --ipc host \
 localhost/llama-cpp-vulkan:latest \
 --host 0.0.0.0 \
 --port 8000 \
-c 128000 \
 --perf \
 --n-gpu-layers all \
 --jinja \
--models-max 1 \
--models-dir /models \
--chat-template-kwargs '{"enable_thinking": false}' \
-m /models/qwen3.5-35b-a3b
+--model /models/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
+--mmproj /models/mmproj-F16.gguf \
+--temp 1.0 \
+--top-p 0.95 \
+--top-k 64 \
+--chat-template-kwargs '{"enable_thinking":true}'
+
+podman run \
+--rm \
+--name llama-server-demo \
+--device=/dev/kfd \
+--device=/dev/dri \
+-v $(pwd):/models:z \
+-p 8010:8000 \
+--ipc host \
+localhost/llama-cpp-vulkan:latest \
+--host 0.0.0.0 \
+--port 8000 \
+--perf \
+--n-gpu-layers all \
+--jinja \
+--model /models/Qwen3.5-9B-UD-Q4_K_XL.gguf \
+--mmproj /models/mmproj-BF16.gguf \
+--temp 0.6 \
+--top-p 0.95 \
+--top-k 20 \
+--min-p 0.00 \
+--chat-template-kwargs '{"enable_thinking":true}'
 ```

 Embedding models
@@ -526,6 +511,28 @@ localhost/stable-diffusion-cpp:latest \
 -p "A normal business meeting. People discuss business for 2 seconds. Suddenly, a horde of furries carrying assault rifles bursts into the room and causes a panic. Hatsune Miku leads the charge screaming in rage."
 ```

+## STT whisper.cpp
+
+Whisper stt service.
+
+```bash
+podman run -it --rm -p "8081:8080" \
+-v /home/ai/models/stt:/models:z \
+--device /dev/kfd \
+--device /dev/dri \
+ghcr.io/ggml-org/whisper.cpp:main-vulkan \
+"whisper-server \
+--host 0.0.0.0 \
+-m /models/ggml-large-v3-turbo/ggml-large-v3-turbo-q5_0.bin \
+--convert"
+```
+
+## TTS coqui-tts
+
+```bash
+podman run --rm -it -p 5002:5002 --entrypoint /bin/bash ghcr.io/coqui-ai/tts-cpu
+```
+
 ## open-webui

 ```bash
@@ -552,6 +559,13 @@ Use the following connections:
 | stable-diffusion.cpp      | <http://host.containers.internal:1234/v1> |
 | stable-diffusion.cpp edit | <http://host.containers.internal:1235/v1> |

+### Upgrade Open Webui
+
+```bash
+podman image pull ghcr.io/open-webui/open-webui:main
+systemctl --user restart open-webui.service
+```
+
 ## lite-llm

 <https://docs.litellm.ai/docs/proxy/configs>
@@ -563,96 +577,7 @@ podman run \
 -p 4000:4000
 ```

-## Install Services with Quadlets
-
-### API Keys
-
-```bash
-mkdir -p /home/ai/.llama-api
-touch /home/ai/.llama-api/keys.env
-chmod 600 /home/ai/.llama-api/keys.env
-vim /home/ai/.llama-api/keys.env
-
-LLAMA_API_KEY=
-
-# Generate keys and append to file, then comma separate the keys
-openssl rand -base64 48 >> keys.env
-openssl rand -base64 48 >> keys.env
-openssl rand -base64 48 >> keys.env
-```
-
-### Internal and External Pods
-
-These will be used to restrict internet access to our llama.cpp and
-stable-diffusion.cpp services while allowing the frontend services to
-communicate with those containers.
-
-```bash
-scp -r active/software_ai_stack/ai-internal.* deskwork-ai:.config/containers/systemd/
-ssh deskwork-ai
-systemctl --user daemon-reload
-systemctl --user start ai-internal-pod.service
-```
-
-### Llama CPP Server (Port 8000)
-
-Installs the llama.cpp server to run our text models.
-
-```bash
-scp -r active/software_ai_stack/llama-think.container deskwork-ai:.config/containers/systemd/
-ssh deskwork-ai
-systemctl --user daemon-reload
-systemctl --user restart ai-internal-pod.service
-```
-
-### Llama CPP Embedding Server (Port 8001)
-
-Installs the llama.cpp server to run our embedding models
-
-```bash
-scp -r active/software_ai_stack/llama-embed.container deskwork-ai:.config/containers/systemd/
-ssh deskwork-ai
-systemctl --user daemon-reload
-systemctl --user restart ai-internal-pod.service
-```
-
-### Llama CPP Instruct Server (Port 8002)
-
-Installs the llama.cpp server to run a constant instruct (no thinking) model for quick replies
-
-```bash
-scp -r active/software_ai_stack/llama-instruct.container deskwork-ai:.config/containers/systemd/
-ssh deskwork-ai
-systemctl --user daemon-reload
-systemctl --user restart ai-internal-pod.service
-```
-
-### Stable Diffusion CPP (Port 1234 and 1235)
-
-Installs the stable-diffusion.cpp server to run our image models.
-
-```bash
-scp -r active/software_ai_stack/quadlets_stable_diffusion/* deskwork-ai:.config/containers/systemd/
-ssh deskwork-ai
-systemctl --user daemon-reload
-systemctl --user restart ai-internal-pod.service
-```
-
-### Open Webui (Port 8080)
-
-Installs the open webui frontend.
-
-```bash
-scp -r active/software_ai_stack/quadlets_openwebui/* deskwork-ai:.config/containers/systemd/
-ssh deskwork-ai
-systemctl --user daemon-reload
-systemctl --user restart ai-external-pod.service
-```
-
-Note, all services will be available at `host.containers.internal`. So llama.cpp
-will be up at `http://host.containers.internal:8000`.
-
-### Install the update script
+## Install the update script

 ```bash
 # 1. Builds the latest llama.cpp and stable-diffusion.cpp
@@ -664,18 +589,6 @@ chmod +x update-script.sh
 ./update-script.sh
 ```

-### Install Guest Open Webui with Start/Stop Services
-
-Optionally install a guest openwebui service.
-
-```bash
-scp -r active/software_ai_stack/systemd/. deskwork-ai:.config/systemd/user/
-ssh deskwork-ai
-systemctl --user daemon-reload
-systemctl --user enable open-webui-guest-start.timer
-systemctl --user enable open-webui-guest-stop.timer
-```
-
 ## Benchmark Results

 Benchmarks are run with [unsloth gpt-oss-20b Q8_0](https://huggingface.co/unsloth/gpt-oss-20b-GGUF/blob/main/gpt-oss-20b-Q8_0.gguf)
@@ -1,23 +0,0 @@
- name: Create Deskwork AI Stack
-  hosts: deskwork-ai
-  tasks:
-    - name: Create /home/ai/.config/containers/systemd
-      ansible.builtin.file:
-        path: /home/ai/.config/containers/systemd
-        state: directory
-        mode: "0755"
-    - name: Copy Quadlets
-      template:
-        src: "{{ item }}"
-        dest: "/home/ai/.config/containers/systemd/{{ item }}"
-      loop:
-        - ai-internal.network
-        - ai-internal.pod
-        - stable-diffusion-gen-server.container
-        - stable-diffusion-edit-server.container
-    - name: Reload and start the ai-internal-pod service
-      ansible.builtin.systemd_service:
-        state: restarted
-        name: ai-internal-pod.service
-        daemon_reload: true
-        scope: user
@@ -1,24 +0,0 @@
- name: Create Deskwork AI Stack
-  hosts: deskwork-ai
-  tasks:
-    - name: Create /home/ai/.config/containers/systemd
-      ansible.builtin.file:
-        path: /home/ai/.config/containers/systemd
-        state: directory
-        mode: "0755"
-    - name: Copy Quadlets
-      template:
-        src: "{{ item }}"
-        dest: "/home/ai/.config/containers/systemd/{{ item }}"
-      loop:
-        - ai-internal.network
-        - ai-internal.pod
-        - llama-embed.container
-        - llama-instruct.container
-        - llama-code.container
-    - name: Reload and start the ai-internal-pod service
-      ansible.builtin.systemd_service:
-        state: restarted
-        name: ai-internal-pod.service
-        daemon_reload: true
-        scope: user
@@ -1,22 +0,0 @@
- name: Create Deskwork AI Stack
-  hosts: driveripper-ai
-  tasks:
-    - name: Create /home/ai/.config/containers/systemd
-      ansible.builtin.file:
-        path: /home/ai/.config/containers/systemd
-        state: directory
-        mode: "0755"
-    - name: Copy Quadlets
-      template:
-        src: "{{ item }}"
-        dest: "/home/ai/.config/containers/systemd/{{ item }}"
-      loop:
-        - ai-internal.network
-        - ai-internal.pod
-        - llama-think.container
-    - name: Reload and start the ai-internal-pod service
-      ansible.builtin.systemd_service:
-        state: restarted
-        name: ai-internal-pod.service
-        daemon_reload: true
-        scope: user
@@ -1,49 +0,0 @@
-[Unit]
-Description=A Llama CPP Server Running a Coding Model
-
-[Container]
-# Shared AI internal pod without internet access
-Pod=ai-internal.pod
-
-# Image is built locally via podman build
-Image=localhost/llama-cpp-vulkan:latest
-
-# Downloaded models volume
-Volume=/home/ai/models/text:/models:z
-
-# GPU Device
-AddDevice=/dev/kfd
-AddDevice=/dev/dri
-
-# Server command
-Exec=--port 8003 \
-    -c 256000 \
-    -n 65536 \
-    --temp 0.7 \
-    --top-p 0.8 \
-    --top-k 20 \
-    --repeat-penalty 1.05 \
-    --perf \
-    --n-gpu-layers all \
-    --jinja \
-    -m /models/qwen3-coder-30b-a3b/Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf \
-    --alias code
-
-# Health Check
-HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8003/health || exit 1
-HealthInterval=10s
-HealthRetries=3
-HealthStartPeriod=10s
-HealthTimeout=30s
-HealthOnFailure=kill
-
-EnvironmentFile=/home/ai/.llama-api/keys.env
-
-[Service]
-Restart=always
-# Extend Timeout to allow time to pull the image
-TimeoutStartSec=900
-
-[Install]
-# Start by default on boot
-WantedBy=multi-user.target default.target
@@ -1,45 +0,0 @@
-[Unit]
-Description=A Llama CPP Server For Embedding Models
-
-[Container]
-# Shared AI internal pod without internet access
-Pod=ai-internal.pod
-
-# Image is built locally via podman build
-Image=localhost/llama-cpp-vulkan:latest
-
-# Downloaded models volume
-Volume=/home/ai/models/embedding:/models:z
-
-# GPU Device
-AddDevice=/dev/kfd
-AddDevice=/dev/dri
-
-# Server command
-Exec=--port 8001 \
-    -c 0 \
-    -b 1024 \
-    -ub 1024 \
-    --perf \
-    --n-gpu-layers all \
-    --embedding \
-    -m /models/emebeddinggemma-300m/embeddinggemma-300M-BF16.gguf \
-    --alias embed
-
-# Health Check
-HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8001/health || exit 1
-HealthInterval=10s
-HealthRetries=3
-HealthStartPeriod=10s
-HealthTimeout=30s
-HealthOnFailure=kill
-EnvironmentFile=/home/ai/.llama-api/keys.env
-
-[Service]
-Restart=always
-# Extend Timeout to allow time to pull the image
-TimeoutStartSec=900
-
-[Install]
-# Start by default on boot
-WantedBy=multi-user.target default.target
@@ -1,53 +0,0 @@
-[Unit]
-Description=A Llama CPP Server Running a Non-Reasoning Model
-
-[Container]
-# Shared AI internal pod without internet access
-Pod=ai-internal.pod
-
-# Image is built locally via podman build
-Image=localhost/llama-cpp-vulkan:latest
-
-# Downloaded models volume
-Volume=/home/ai/models/text:/models:z
-
-# GPU Device
-AddDevice=/dev/kfd
-AddDevice=/dev/dri
-
-# Server command
-Exec=--port 8002 \
-    -c 262144 \
-    -n 32768 \
-    --temp 0.7 \
-    --top-p 0.8 \
-    --min-p 0.0 \
-    --top-k 20 \
-    --repeat-penalty 1.0 \
-    --presence-penalty 1.5 \
-    --reasoning-budget 0 \
-    --perf \
-    --n-gpu-layers all \
-    --jinja \
-    -m /models/qwen3.6-35b-a3b/Qwen3.6-35B-A3B-UD-Q5_K_M.gguf \
-    --mmproj /models/qwen3.6-35b-a3b/mmproj-F16.gguf \
-    --chat-template-kwargs '{"enable_thinking": false}' \
-    --alias instruct
-
-# Health Check
-HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8002/health || exit 1
-HealthInterval=10s
-HealthRetries=3
-HealthStartPeriod=10s
-HealthTimeout=30s
-HealthOnFailure=kill
-EnvironmentFile=/home/ai/.llama-api/keys.env
-
-[Service]
-Restart=always
-# Extend Timeout to allow time to pull the image
-TimeoutStartSec=900
-
-[Install]
-# Start by default on boot
-WantedBy=multi-user.target default.target
@@ -1,52 +0,0 @@
-[Unit]
-Description=A Llama CPP Server Running a Reasoning Model
-
-[Container]
-# Shared AI internal pod without internet access
-Pod=ai-internal.pod
-
-# Image is built locally via podman build
-Image=localhost/llama-cpp-vulkan:latest
-
-# Downloaded models volume
-Volume=/home/ai/models/text:/models:z
-
-# GPU Device
-AddDevice=/dev/kfd
-AddDevice=/dev/dri
-
-# Server command
-Exec=--port 8000 \
-    -c 262144 \
-    -n 32768 \
-    --temp 0.7 \
-    --top-p 0.95 \
-    --top-k 20 \
-    --min-p 0.0 \
-    --presence-penalty 0.0 \
-    --repeat-penalty 1.0 \
-    --reasoning-budget 5000 \
-    -fa on \
-    --perf \
-    --n-gpu-layers all \
-    --jinja \
-    -m /models/qwen3.6-35b-a3b/Qwen3.6-35B-A3B-UD-Q5_K_M.gguf \
-    --mmproj /models/qwen3.6-35b-a3b/mmproj-F16.gguf \
-    --alias think
-
-# Health Check
-HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8000/health || exit 1
-HealthInterval=10s
-HealthRetries=3
-HealthStartPeriod=10s
-HealthTimeout=30s
-HealthOnFailure=kill
-
-[Service]
-Restart=always
-# Extend Timeout to allow time to pull the image
-TimeoutStartSec=900
-
-[Install]
-# Start by default on boot
-WantedBy=multi-user.target default.target
@@ -1,24 +0,0 @@
-[Unit]
-Description=An Open Webui Frontend for Local AI Services
-
-[Container]
-# Shared AI external pod
-PublishPort=8080:8080
-
-# Open Webui base image
-Image=ghcr.io/open-webui/open-webui:main
-
-# Nothing too complicated here. Open Webui will basically configure itself.
-Volume=open-webui-data:/app/backend/data
-
-# WEBUI_SECRET_KEY is required to prevent logout on Restart
-EnvironmentFile=/home/ai/.env/open-webui-env
-
-[Service]
-Restart=always
-# Extend Timeout to allow time to pull the image
-TimeoutStartSec=900
-
-[Install]
-# Start by default on boot
-WantedBy=multi-user.target default.target
@@ -1,44 +0,0 @@
-[Unit]
-Description=A Stable Diffusion CPP Server for Editing Images
-
-[Container]
-# Shared AI internal pod without internet access
-Pod=ai-internal.pod
-
-# Vulkan image for AMD GPU
-Image=localhost/stable-diffusion-cpp:latest
-
-# Shared models directory
-Volume=/home/ai/models:/models:z
-
-# GPU Device
-AddDevice=/dev/kfd
-AddDevice=/dev/dri
-
-# Override entrypoint to use server
-Entrypoint=/sd-server
-
-# Server args
-Exec=-l 0.0.0.0 \
-    --listen-port 1235 \
-    --diffusion-model /models/image/flux2-klein/flux-2-klein-9b-Q8_0.gguf \
-    --vae /models/image/flux2-klein/ae.safetensors \
-    --llm /models/image/flux2-klein/Qwen3-8B-Q4_K_M.gguf \
-    -v \
-    --sampling-method euler \
-    --cfg-scale 1.0 \
-    --vae-conv-direct \
-    --diffusion-conv-direct \
-    --fa \
-    --mmap \
-    --seed -1 \
-    --steps 5
-
-[Service]
-Restart=always
-# Extend Timeout to allow time to pull the image
-TimeoutStartSec=900
-
-[Install]
-# Start by default on boot
-WantedBy=multi-user.target default.target
@@ -1,43 +0,0 @@
-[Unit]
-Description=A Stable Diffusion CPP Server for Generating Images
-
-[Container]
-# Shared AI internal pod without internet access
-Pod=ai-internal.pod
-
-# Vulkan image for AMD GPU
-Image=localhost/stable-diffusion-cpp:latest
-
-# Shared models directory
-Volume=/home/ai/models:/models:z
-
-# GPU Device
-AddDevice=/dev/kfd
-AddDevice=/dev/dri
-
-# Override entrypoint to use server
-Entrypoint=/sd-server
-
-# Server args
-Exec=-l 0.0.0.0 \
-    --listen-port 1234 \
-    --diffusion-model /models/image/z-turbo/z_image_turbo-Q8_0.gguf \
-    --vae /models/image/z-turbo/ae.safetensors  \
-    --llm /models/image/z-turbo/Qwen3-4B-Instruct-2507-Q4_K_M.gguf \
-    -v \
-    --cfg-scale 1.0 \
-    --vae-conv-direct \
-    --diffusion-conv-direct \
-    --fa \
-    --mmap \
-    --seed -1 \
-    --steps 8
-
-[Service]
-Restart=always
-# Extend Timeout to allow time to pull the image
-TimeoutStartSec=900
-
-[Install]
-# Start by default on boot
-WantedBy=multi-user.target default.target
@@ -1,8 +0,0 @@
-[Unit]
-Description=Start open-webui-guest on schedule
-Wants=open-webui-guest.service
-After=open-webui-guest.service
-
-[Service]
-Type=oneshot
-ExecStart=/bin/systemctl --user start open-webui-guest.service
@@ -1,10 +0,0 @@
-[Unit]
-Description=Timer to start open-webui-guest at 08:00 Mon‑Fri
-
-[Timer]
-OnCalendar=Mon..Fri *-*-* 08:00:00
-Persistent=true
-Unit=open-webui-guest-start.service
-
-[Install]
-WantedBy=timers.target
@@ -1,7 +0,0 @@
-[Unit]
-Description=Stop open-webui-guest on schedule
-After=open-webui-guest.service
-
-[Service]
-Type=oneshot
-ExecStart=/bin/systemctl --user stop open-webui-guest.service
@@ -1,10 +0,0 @@
-[Unit]
-Description=Timer to stop open-webui-guest at 18:00 Mon‑Fri
-
-[Timer]
-OnCalendar=Mon..Fri *-*-* 18:00:00
-Persistent=true
-Unit=open-webui-guest-stop.service
-
-[Install]
-WantedBy=timers.target
@@ -0,0 +1,12 @@
+# Flatpak
+
+## Installing a specific version
+
+```bash
+# List the available commits
+flatpak remote-info --log flathub com.orcaslicer.OrcaSlicer
+
+# Install the commit version you want
+# Orca Slicer, for example
+sudo flatpak update --commit=acd5ab796f5640645938194da3a6822b3699dcf41c61059bfbd6accb798ebc7b com.orcaslicer.OrcaSlicer
+```
@@ -0,0 +1,5 @@
+# Konsole
+
+## Tips
+
+- To open new tabs with the same profile as the current tab, assign the "clone tab" keyboard shortcut to <ctrl>+<shift>+<T>.
@@ -0,0 +1,79 @@
+name = "fedora44-base"
+description = "Fedora Base Installation"
+version = "0.2.3"
+distro = "fedora-44"
+modules = []
+groups = []
+
+[customizations]
+hostname = "fedora44-base"
+
+[customizations.timezone]
+timezone = "America/New_York"
+
+[[customizations.disk.partitions]]
+type = "plain"
+label = "root"
+mountpoint = "/"
+fs_type = "ext4"
+minsize = "128 GiB"
+
+[customizations.firewall.services]
+enabled = ["ssh", "cockpit"]
+
+[customizations.services]
+enabled = ["sshd", "cockpit.socket"]
+
+[[packages]]
+name = "bash-completion"
+version = "*"
+
+[[packages]]
+name = "tmux"
+version = "*"
+
+[[packages]]
+name = "openssh-server"
+version = "*"
+
+[[packages]]
+name = "vim"
+version = "*"
+
+[[packages]]
+name = "git"
+version = "*"
+
+[[packages]]
+name = "cockpit"
+version = "*"
+
+[[packages]]
+name = "borgbackup"
+version = "*"
+
+[[packages]]
+name = "policycoreutils-python-utils"
+version = "*"
+
+[[packages]]
+name = "systemd-container"
+version = "*"
+
+[[customizations.files]]
+path = "/root/.inputrc"
+mode = "0644"
+user = "root"
+group = "root"
+data = """
+"\\C-h": backward-kill-word
+"""
+
+[[customizations.files]]
+path = "/root/.ssh/authorized_keys"
+mode = "0600"
+user = "root"
+group = "root"
+data = """
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGQa781Qj8mNlUdRquFFqg0O2ornG9SBHe705y4+1vPI ssh@ducoterra.net
+"""
@@ -29,6 +29,8 @@ dnf install -y osbuild-composer composer-cli
 systemctl enable --now osbuild-composer.socket

 # Optional: add your user to the weldr group
+# On fedora kinoite (atomic) you'll need to add the group manually for now
+grep -E '^weldr:' /usr/lib/group | sudo tee -a /etc/group
 sudo usermod -aG weldr $USER

 # Optional: cockpit dependency
@@ -0,0 +1,139 @@
+# Pi AI Harness
+
+## Prereqs
+
+I would recommend using `pass` to store your api key. This is installed with `dnf install pass` on Fedora systems.
+
+After installing pass, update your `~/.gnupg/gpg-agent.conf` to set the timeout to max so automated processes can use the values in your password store:
+
+```conf
+default-cache-ttl 2147483647
+max-cache-ttl 2147483647
+```
+
+Then add a `pass.sh` to you `~/.bashrc.d` that looks like
+
+```bash
+pass unlock
+```
+
+That way every time you log in your pass keyring is unlocked for your session so automations can use it.
+
+## Setup
+
+This is my ~/.pi/agent/models.json
+
+```json
+{
+  "providers": {
+    "reeseapps-aipi": {
+      "baseUrl": "https://aipi.reeseapps.com/v1",
+      "api": "openai-completions",
+      "apiKey": "!pass aipi/dev-token",
+      "models": [
+        {
+          "id": "juggernaut",
+          "input": ["text", "image"],
+          "reasoning": true,
+          "contextWindow": 262144,
+          "maxTokens": 32768,
+          "thinkingLevelMap": {
+            "off": "none",
+            "minimal": null,
+            "low": "low",
+            "medium": "medium",
+            "high": null,
+            "xhigh": "xhigh",
+            "max": null
+          }
+        },
+        {
+          "id": "turbo",
+          "input": ["text", "image"],
+          "reasoning": true,
+          "contextWindow": 262144,
+          "maxTokens": 32768,
+           "thinkingLevelMap": {
+            "off": "none",
+            "minimal": null,
+            "low": "low",
+            "medium": "medium",
+            "high": null,
+            "xhigh": "xhigh",
+            "max": null
+          }
+        }
+      ]
+    }
+  }
+}
+```
+
+This is my ~/.pi/agent/settings.json
+
+```json
+{
+  "lastChangelogVersion": "1.3.0",
+  "theme": "dark/dark",
+  "defaultProvider": "reeseapps-aipi",
+  "defaultModel": "turbo",
+  "httpIdleTimeoutMs": 0,
+  "enableInstallTelemetry": false,
+  "doubleEscapeAction": "tree",
+  "images": {
+    "autoResize": false
+  },
+  "editorPaddingX": 0,
+  "defaultThinkingLevel": "xhigh",
+  "hideThinkingBlock": false,
+  "thinkingBudgets": {
+    "minimal": 1024,
+    "low": 4096,
+    "medium": 10240,
+    "xhigh": 65536
+  },
+  "steeringMode": "all"
+}
+```
+
+## Installing for sysadmin work
+
+I would absolutely recommend installing pi under a special user (I call mine
+`ops`) with limited sudoers access that you tightly control. Pair that with the
+following AGENTS.md at `/home/ops/AGENTS.md`:
+
+```text
+# AGENTS.md
+
+## Sudo policy
+
+The `ops` user on this machine has **limited sudoers capabilities**. The exact
+allowlist is discoverable at any time with:
+
+sudo -l
+
+Allowed (NOPASSWD) commands currently include: `hostnamectl hostname`, `reboot`,
+`dnf update/upgrade/install/remove/clean/makecache/autoremove`,
+`dnf config-manager addrepo`, `systemctl start/restart/reload/enable`,
+`firewall-cmd`, the ansible shim, `qemu-img`, `virsh`, `virt-install`,
+`composer-cli`, `getfacl`, specific `setfacl` invocations,
+`/usr/local/sbin/osbuild-auto.sh`, and `btrfs scrub`.
+
+## Rule for agents
+
+If you (the agent) believe you need a permission that is **not** in the
+`sudo -l` allowlist — i.e., a command fails with a sudo permission error, or a
+task requires root access you do not have — you must:
+
+1. **Stop** and do not attempt workarounds that bypass sudo (no password
+   guessing, no escalating via other users, no trying to write to root-owned
+   paths without permission).
+2. **Ask the human** to update sudoers with the specific command(s) you need,
+   stating exactly which command(s) and arguments should be added so the user
+   can review and approve them. Requests must be correct and copy-pastable,
+   using full command paths in standard sudoers format, e.g.
+   `ops ALL=(root) NOPASSWD: /usr/sbin/btrfs scrub*`
+
+Do not proceed with root-only operations until the user has confirmed the
+sudoers entries have been added.
+```
@@ -0,0 +1,48 @@
+# Podman
+
+## Networks
+
+```bash
+# Create a podman network to group shared containers
+podman network create test1
+# Create an iperf3 container running on that network
+podman run -d --name iperf3 --network test1 docker.io/networkstatic/iperf3 -s
+# A container running in the same network can reference another pod by its name
+podman run --rm -it --network test1 docker.io/networkstatic/iperf3 -c iperf3
+# A container outside the created network can't talk to other containers
+# This will fail
+podman run --rm -it docker.io/networkstatic/iperf3 -c iperf3
+```
+
+## Healthchecks
+
+TCP Healthchecks can be accomplished with netcat (netcat-openbsd)
+
+```bash
+podman run -d --name iperf3 \
+--health-cmd='CMD-SHELL iperf3 -n 1 -c localhost' \
+--health-interval=10s --health-timeout=5s --health-retries=3 \
+docker.io/networkstatic/iperf3 iperf3 -s
+```
+
+HTTP Healthchecks can be accomplished with curl
+
+```bash
+podman run -d --name nginx \
+--health-cmd='CMD-SHELL curl --fail http://127.0.0.1:80 || exit 1' \
+--health-interval=10s --health-timeout=5s --health-retries=3 \
+docker.io/nginx
+```
+
+Check healthchecks with
+
+```bash
+podman --log-level debug healthcheck run <container>
+```
+
+## Volume Import and Export
+
+```bash
+podman volume export myvol --output myvol.tar
+podman volume import myvol test.tar
+```
@@ -30,11 +30,17 @@ sudo smbpasswd -a sambauser
 ## Create a SMB Share

 ```bash
+# Add the share user
+sudo useradd -r -M -s /usr/sbin/nologin sambauser
+sudo smbpasswd -a sambauser
+
 # Create share
-mkdir /srv/smb/sambauser
+sudo mkdir /srv/smb/sambauser
+sudo chown -R sambauser:sambauser /srv/smb/sambauser
+sudo chmod -R 2770 /srv/samba/sambauser

 # Set proper selinux labels for samba
-sudo semanage fcontext --add --type "samba_share_t" "/srv/smb(/.*)?"
+sudo semanage fcontext --add --type "samba_share_t" "/srv/smb/sambauser"

 # Run restorecon at the root of the btrfs subvolume
 sudo restorecon -FRv /srv
@@ -0,0 +1,44 @@
+FROM registry.fedoraproject.org/fedora-toolbox:44
+
+COPY vscodium.repo /etc/yum.repos.d/vscodium.repo
+
+RUN dnf install -y \
+vim git btop \
+virt-manager virt-install \
+podman podman-compose \
+wl-clipboard \
+jq yq rg \
+kubectl helm \
+pinentry \
+pandoc \
+iperf3 \
+nethogs \
+ansible \
+tmux \
+wine \
+unzip \
+ImageMagick \
+go \
+rust rustup \
+git git-lfs \
+net-tools \
+texlive-latex texlive-scheme-full \
+pwgen \
+reptyr \
+netcat \
+p7zip \
+make \
+gcc \
+codium \
+pass \
+stress-ng \
+fuse \
+bind-utils \
+nmap \
+poppler \
+nodejs24 \
+kitty \
+scrot \
+xorg-x11-server-Xvfb \
+ffmpeg \
+portaudio
@@ -0,0 +1,7 @@
+# Toolbox
+
+## How to Use
+
+```bash
+podman build -t localhost/fedora-toolbox:44 active/software_toolbx
+```
@@ -0,0 +1,9 @@
+[gitlab.com_paulcarroty_vscodium_repo]
+name=gitlab.com_paulcarroty_vscodium_repo
+baseurl=https://paulcarroty.gitlab.io/vscodium-deb-rpm-repo/rpms/
+enabled=1
+gpgcheck=1
+repo_gpgcheck=1
+gpgkey=https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/raw/master/pub.gpg
+metadata_expire=1h
+
@@ -40,7 +40,6 @@ Virtual Machine Management
 <https://libvirt.org/uri.html#ssh-transport>

 ### Configuring Aliases
-
 1. Edit `~/.config/libvirt/libvirt.conf`
 2. Add your aliases

@@ -0,0 +1,281 @@
+# vLLM Radiance on R9700
+
+- [vLLM Radiance on R9700](#vllm-radiance-on-r9700)
+  - [Overview](#overview)
+  - [Deployment Topology](#deployment-topology)
+  - [Capacity Analysis (Single 32 GiB R9700)](#capacity-analysis-single-32-gib-r9700)
+  - [FP8-KV Calibration: The Process](#fp8-kv-calibration-the-process)
+    - [Why: The Uncalibrated Scale Problem](#why-the-uncalibrated-scale-problem)
+    - [Step 1: Host Prep](#step-1-host-prep)
+    - [Step 2: Stop the Service](#step-2-stop-the-service)
+    - [Step 3: Run the Calibration](#step-3-run-the-calibration)
+    - [Step 4: Verify the Artifact (CPU-only)](#step-4-verify-the-artifact-cpu-only)
+    - [Step 5: Edit the Quadlet](#step-5-edit-the-quadlet)
+    - [Step 6: Deploy via Ansible](#step-6-deploy-via-ansible)
+    - [Step 7: Confirm at Runtime](#step-7-confirm-at-runtime)
+  - [What Calibration Actually Does (Benefits)](#what-calibration-actually-does-benefits)
+  - [Operational Notes](#operational-notes)
+  - [Remaining Work: Matched-Control Quality Gate](#remaining-work-matched-control-quality-gate)
+
+## Overview
+
+`vllm-radiance` is a qualified downstream vLLM distribution for the AMD Radeon AI
+PRO R9700 (gfx1201). The upstream-qualified environment is dual-GPU TP2; this
+deployment runs the single-GPU (TP1) configuration on `driveripper-ai` serving
+the AMD Quark MXFP4 27B checkpoint (W4A8: 4-bit weights, dynamic FP8
+activation quantization over RDNA4's native FP8 WMMA path).
+
+Served as `juggernaut` on the `ai-internal` network, port 8000.
+
+## Deployment Topology
+
+| Thing              | Location                                                                                                                                                                                   |
+| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Quadlet (source of |
+| truth)             | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/quadlets/vllm-radiance.container`                                                                                                    |
+| Ansible playbook   | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/playbook.yaml` (run from `~/Deployments`: `ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml`) |
+| Deployed quadlet   | `driveripper-ai:/home/ai/.config/containers/systemd/ai-vllm-radiance.container`                                                                                                            |
+| systemd unit       | **user** unit of `ai` (`systemctl --user status ai-vllm-radiance` — not in system scope)                                                                                                   |
+| Image              | `docker.io/magiccodingman/vllm-radiance:1.0.387`                                                                                                                                           |
+| Model              | `driveripper-ai:/home/ai/vllm-radiance-models/qwen3.8-27b-quark-awq-mxfp4` → `/models` (18.44 GiB on disk, 17.95 GiB in GPU)                                                               |
+| Compile caches     | `driveripper-ai:/home/ai/vllm-radiance-cache` → `/cache`                                                                                                                                   |
+| FP8-KV sidecar     | `driveripper-ai:/home/ai/vllm-radiance-calibration` → `/calibration` (ro)                                                                                                                  |
+
+The playbook copies the quadlet to `~/.config/containers/systemd/` and does
+`systemctl --user restart ai-vllm-radiance` with daemon-reload. **Any edit to
+the local quadlet lands on the next playbook run** — there is no separate
+"apply" step.
+
+Key quadlet settings (as of the calibration deploy, 2026-10-11):
+
+```
+--tensor-parallel-size=1  --max-model-len=154000  --max-num-seqs=2
+--max-num-batched-tokens=4096  --kv-cache-dtype=fp8  --attention-backend=R4D
+--enable-prefix-caching  --mamba-cache-mode=align  --no-async-scheduling
+--gpu-memory-utilization=0.98
+Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
+Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
+HealthStartPeriod=600s  (health check kills on 3x10s failures after the start period)
+```
+
+## Capacity Analysis (Single 32 GiB R9700)
+
+Measured from the server startup logs (the authoritative numbers — do not
+re-derive from weight sizes alone; the GDN hybrid architecture makes
+per-token KV much cheaper than a plain transformer):
+
+```
+Model loading took 17.95 GiB
+Actual usage: 19.38 GiB consumed (weights + non-torch)
+             + 2.02 GiB peak activation
+             + 0.41 GiB CUDA graphs
+Available KV cache memory: 9.82 GiB  (10.14 GiB at the 154K config)
+GPU KV cache size: 308,847 tokens   (309,466 at the 154K config)
+```
+
+Budget math (KV budget ≈ 309K tokens):
+
+| Config                | KV needed | Fits?                                        |
+| --------------------- | --------- | -------------------------------------------- |
+| 150K × 2              | 300,000   | yes, ~3% headroom                            |
+| **154K × 2 (chosen)** | 308,000   | yes — `2.01x` per vLLM, ~1.5K token headroom |
+| 256K × 1              | 262,144   | yes                                          |
+| 256K × 2              | 524,288   | **no** — `1.18x` max concurrency             |
+
+The per-request C2 ceiling is `KV_budget / 2 ≈ 154,423` tokens; `--max-model-len=154000`
+sits exactly at it. A second full 256K request is only reachable via CPU KV
+offload (`RADIANCE_KV_OFFLOADING_SIZE`), which would need to roughly double the
+token budget. `MAX_NUM_SEQS` is an admission ceiling, not a promise that every
+admitted request can simultaneously reach `MAX_MODEL_LEN`.
+
+## FP8-KV Calibration: The Process
+
+### Why: The Uncalibrated Scale Problem
+
+The server runs `--kv-cache-dtype=fp8` with FP8 attention, but the checkpoint
+carries no q/k/v/prob scales. vLLM therefore used placeholder scale **1.0**,
+which the startup log flags:
+
+```
+WARNING ... Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8
+attention. This may cause accuracy issues.
+```
+
+Radiance ships `radiance_kv_calibration.py` (in the image, also in the
+`vllm-radiance` repo) plus the `RADIANCE_FP8_KV_SCALES` runtime loader to fix
+this with a small immutable sidecar. Full reference: `docs/FP8_KV_TUNABLEOP.md`
+in the vllm-radiance repo.
+
+### Step 1: Host Prep
+
+The calibration dir did not exist and the vllm-radiance repo is not on
+`driveripper-ai`, so copy only the corpus:
+
+```bash
+ssh driveripper-ai "mkdir -p /home/ai/vllm-radiance-calibration"
+scp ~/Projects/Experimental/vllm-radiance/benchmarks/fixtures/fp8-kv-calibration.jsonl \
+    driveripper-ai:/home/ai/vllm-radiance-calibration/
+```
+
+(Optionally extend the JSONL with representative traffic first — each line is
+`{"text": ...}` or `{"messages": [...], "tools": [...]}` plus optional `repeat`.)
+
+### Step 2: Stop the Service
+
+Calibration needs the GPU released; the server holds ~97% of VRAM.
+
+```bash
+ssh driveripper-ai "systemctl --user stop ai-vllm-radiance"
+```
+
+### Step 3: Run the Calibration
+
+An eager, BF16 full-precision-KV, `max_tokens=1` prefill sweep with PyTorch
+hooks on every full-attention layer (no source edits — public worker-extension
+API only). Budget ~10–30 min:
+
+```bash
+ssh driveripper-ai "podman run --rm --ipc=host \
+  --device=/dev/kfd --device=/dev/dri \
+  --ulimit memlock=-1:-1 \
+  -e HIP_VISIBLE_DEVICES=0 \
+  -e RADIANCE_MXFP4=1 -e RADIANCE_MXFP4_W4A8=1 -e RADIANCE_MXFP4_W4A8_MIN_M=0 \
+  -v /home/ai/vllm-radiance-models:/models:ro \
+  -v /home/ai/vllm-radiance-calibration:/calibration:rw \
+  -v /home/ai/vllm-radiance-calibration/fp8-kv-calibration.jsonl:/fixtures/fp8-kv-calibration.jsonl:ro \
+  --entrypoint python \
+  docker.io/magiccodingman/vllm-radiance:1.0.387 \
+  -m radiance_kv_calibration calibrate \
+  --model /models/qwen3.8-27b-quark-awq-mxfp4 \
+  --quantization auto \
+  --corpus /fixtures/fp8-kv-calibration.jsonl \
+  --output /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
+  --tensor-parallel-size 1 \
+  --max-model-len 16384 \
+  --batch-size 4 \
+  --seed 17"
+```
+
+Success line: `[radiance] wrote immutable FP8-KV sidecar: /calibration/...`.
+Outputs are **immutable** — a re-run needs a fresh filename. The sidecar and
+its sibling `.manifest.json` (sha256 + full source-checkpoint identity +
+per-layer observed amax + corpus/seed/software provenance) land in the host dir.
+
+### Step 4: Verify the Artifact (CPU-only)
+
+No engine start — checks manifest schema, sidecar sha256, that every tensor is
+a positive finite scalar `*.q|k|v|prob_scale`, and (with `--model`) that it was
+calibrated against this exact checkpoint:
+
+```bash
+ssh driveripper-ai "podman run --rm \
+  -v /home/ai/vllm-radiance-models:/models:ro \
+  -v /home/ai/vllm-radiance-calibration:/calibration:ro \
+  --entrypoint python docker.io/magiccodingman/vllm-radiance:1.0.387 \
+  -m radiance_kv_calibration verify \
+  /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
+  --model /models/qwen3.8-27b-quark-awq-mxfp4"
+```
+
+### Step 5: Edit the Quadlet
+
+Three lines in the local quadlet (source of truth), then nothing else:
+
+```diff
+ Volume=/home/ai/vllm-radiance-models:/models:ro,z
+ Volume=/home/ai/vllm-radiance-cache:/cache:z
+Volume=/home/ai/vllm-radiance-calibration:/calibration:ro,z
+ ...
+ Environment=WEIGHT_QUANTIZATION=auto
+Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
+Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
+```
+
+### Step 6: Deploy via Ansible
+
+```bash
+cd ~/Deployments
+ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml
+```
+
+### Step 7: Confirm at Runtime
+
+All three must be present/absent:
+
+```
+FP8-KV scales = /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors   (banner; was "checkpoint/default")
+Loading Radiance FP8-KV calibration sidecar ... (verify=True)          (checksum + identity passed at load)
+"uncalibrated" warning: gone
+/health: OK
+```
+
+## What Calibration Actually Does (Benefits)
+
+**The problem.** With placeholder scale 1.0, FP8 values are stored as
+`value / 1.0`. The calibration hooks measured the model's real pre-quantization
+activations: peak **23.0 (K, layer 31)** and **59.75 (V, layer 43)** across the
+16 full-attention layers. FP8 E4M3's max magnitude is 448 — so the raw values
+only used **5–13% of the representable range**. E4M3 has just 3 mantissa bits;
+when values cluster near the bottom of the range, the quantization steps
+between representable values are ~8–20× coarser than necessary. That is rounding
+noise injected into every attention score and KV lookup — worst for
+long-context retrieval, where KV detail is exactly what the task depends on.
+
+**The fix.** The calibration writes per-layer
+`scale = observed_amax × 1.05 / 448` (5% headroom for excursions the corpus
+didn't hit) and `prob_scale = 1.05 / 448` (softmax probabilities are bounded by
+1). At load time vLLM divides each tensor by that scale before FP8 conversion,
+so observed maxima land at ~95% of the FP8 range — the full 3 mantissa bits do
+useful work. Result: **~7.5× finer quantization steps on V, ~19.6× finer on K**
+(≈ +2.9 / +4.3 effective mantissa bits), and the "may cause accuracy issues"
+warning is resolved.
+
+**Provenance & safety.** The sidecar is immutable (sha256 in the manifest),
+bound to this exact checkpoint, and re-verified at every startup by
+`VERIFY=1` — a changed model makes the server **fail closed** instead of
+silently using wrong scales.
+
+**What it does NOT do.** No throughput change (expected neutral — a big TPS
+delta means something is wrong, not a win). No new kernels. No weight-path
+change. Purely a precision fix for the FP8 attention/KV arithmetic.
+
+## Operational Notes
+
+- **Sidecar is per-checkpoint.** Any fine-tune or re-download changes
+  activation ranges — re-run calibration (fresh filename) rather than reusing
+  scales by architecture name, and never set `VERIFY=0` as a deployment
+  setting (it exists only to diagnose a deliberate mismatch).
+- **Startup re-hash cost.** The model dir on `driveripper-ai` has no HF
+  download metadata, so the manifest bound full weight sha256s — every startup
+  re-hashes all 18.44 GiB of weights for identity verification. Adds a few
+  minutes on cold restarts; worked fine in practice.
+- **Health check.** `HealthStartPeriod=600s` was chosen deliberately: the
+  deployed (pre-calibration) quadlet had the health check disabled at 30s start
+  period — a cold kernel-compile startup would fail 3×10s of health checks and
+  `HealthOnFailure=kill` + `Restart=always` loops it.
+- **Capacity ceiling.** 154K×2 runs the KV tier to ~99.6% at full context by
+  design. Normal mixed traffic stays well below. If you need two full 256K
+  concurrent requests, that is a CPU-offload decision, not a flag tweak.
+- **Corpus.** The 2026-10-10 sidecar used the stock 8-prompt corpus (tool
+  schema, code, 4.9K-token retrieval, reasoning, multilingual, math, structured
+  data). If the real workload is dominated by content the stock corpus doesn't
+  cover, re-calibrate with an extended corpus. Calibration was run at
+  16K `max_model_len`; amax is largely context-independent, but if long-context
+  numerics ever look off, that's the first thing to re-run with a longer
+  `--max-model-len`.
+
+## Remaining Work: Matched-Control Quality Gate
+
+Per `docs/FP8_KV_TUNABLEOP.md` and the project's no-claim-without-control
+policy: compare **calibrated vs default-scale** runs with identical image,
+target, prompts, seeds, sampling, graphs, and concurrency:
+
+- fixed-prompt greedy outputs (meaningful agreement)
+- sampled tool-schema checks
+- long-context retrieval
+- vision inputs if vision is deployed
+- BetterBench / sustained c1/c2/c4/c8 for the throughput-neutral claim
+
+Success criterion: **fidelity at-or-above the uncalibrated baseline with ~neutral
+throughput.** Only after that gate is this a qualified change, not just a
+plausible one.
@@ -1,24 +1,19 @@
 import json
 import os
 import smtplib
-import ssl
 import subprocess
 from email.message import EmailMessage
 from pathlib import Path
 from typing import Iterable, TypedDict, cast

+import requests
 from dotenv import dotenv_values, load_dotenv
 from openai import OpenAI
 from openai.types.chat import ChatCompletionMessageParam, ChatCompletionToolUnionParam


-class AWS_SES_DOTENV(TypedDict):
-    USER: str
-    PASSWORD: str
-    ENDPOINT: str
-    TLS_PORT: str
-    SENDER: str
-    RECEIVER: str
+class NTFY_DOTENV(TypedDict):
+    TOKEN: str


 class ToolCallController:
@@ -209,39 +204,30 @@ def smartctl(device_path: str) -> str:
    return output


-def load_ses_creds() -> AWS_SES_DOTENV:
+def load_ntfy_creds() -> NTFY_DOTENV:
    ses_dotenv_location = Path(os.getenv("HOME", "/root"), ".env/aws_ses")
    print(f"Loading env from {ses_dotenv_location}")
    raw_values = dotenv_values(ses_dotenv_location)
    if raw_values:
-        aws_ses_config = cast(AWS_SES_DOTENV, raw_values)
+        ntfy_config = cast(NTFY_DOTENV, raw_values)
        # print(f"AWS SES Credentials loaded: {aws_ses_config}")
-        return aws_ses_config
-    print("No email credentials supplied. Exiting.")
+        return ntfy_config
+    print("No ntfy credentials supplied. Exiting.")
    exit(1)


 def alert_user(message: str) -> str:
-    ses_config = load_ses_creds()
-    port = int(ses_config["TLS_PORT"])
-    user = ses_config["USER"]
-    password = ses_config["PASSWORD"]
-    sender = ses_config["SENDER"]
-    receiver = ses_config["RECEIVER"]
+    ntfy_creds = load_ntfy_creds()

-    # Create a secure SSL context
-    context = ssl.create_default_context()
-
-    msg = EmailMessage()
-    msg["Subject"] = "Agent Disk Report"
-    msg["From"] = sender
-    msg["To"] = receiver
-    msg.set_content(message)
-
-    with smtplib.SMTP_SSL(ses_config["ENDPOINT"], port, context=context) as server:
-        server.login(user, password)
-        result = server.send_message(msg)
-        print(result)
+    requests.post(
+        "https://ntfy.reeseapps.com/servers",
+        data=message,
+        headers={
+            "Authorization": f"Bearer {ntfy_creds['TOKEN']}",
+            "Title": "Agent Disk Report",
+            "Priority": "default",
+        },
+    )

    return "Sent."

@@ -322,13 +308,14 @@ def run_conversation(user_message: str, max_tool_calls=10):
 if __name__ == "__main__":
    load_dotenv()
    api_key = os.getenv("OPENAI_API_KEY", "")
-    client = OpenAI(base_url="https://llama-think.reeselink.com", api_key=api_key)
+    client = OpenAI(base_url="https://aipi.reeseapps.com", api_key=api_key)
    # Example usage
    print(
        run_conversation(
            "Check the btrfs pools on this system. Take the appropriate action if any pools aren't "
            "healthy. Don't run scrubs unless necessary. Also check the btrfs pool space and report "
            "if any are getting full. At the very end of performing your checks send a single, "
-            "concise message to the user explaining what you did and what concerns you might have."
+            "concise message to the user via the alert_user tool explaining what you did and "
+            "what concerns you might have."
        )
    )
@@ -1,5 +0,0 @@
-# Proxy Automation
-
-1. Create or update the reeselink.com address in Unifi
-2. Create or update the entry in the ddns values
-3. Create or update the entry in the caddy values
@@ -1 +0,0 @@
-3.13
@@ -1,9 +0,0 @@
-# Update
-
-Updates every server in the fleet
-
-## Usage
-
-```bash
-uv run automations/updates/main.py
-```
@@ -1,64 +0,0 @@
-import subprocess
-import time
-
-import yaml
-from tqdm import tqdm
-
-
-def main():
-    print("Retrieving hosts")
-    with open("ansible/inventory.yaml", "r") as f:
-        all_hosts = yaml.load(f, yaml.SafeLoader)
-    fedora_hosts = all_hosts["fedora"]["hosts"].keys()
-
-    with open("update.log", "w") as log_file:
-        for _, host in enumerate(tqdm(fedora_hosts, desc="Running system updates")):
-            log_file.write(f"Updating {host}\n")
-            log_file.flush()
-            try:
-                subprocess.run(
-                    ["ssh", host, "dnf", "upgrade", "-y"],
-                    stdout=log_file,
-                    stderr=log_file,
-                    check=True,
-                )
-            except Exception as e:
-                log_file.write(f"Couldn't connect to {host}. Skipping...\n")
-                continue
-            log_file.flush()
-            log_file.write(f"Rebooting {host}\n")
-            log_file.flush()
-            subprocess.run(
-                ["ssh", host, "reboot"],
-                stdout=log_file,
-                stderr=log_file,
-                check=True,
-            )
-            time.sleep(5)  # wait for reboot to take effect
-            booted = False
-            max_attempts = 10
-            cur_attempts = 0
-            while max_attempts > cur_attempts and not booted:
-                try:
-                    subprocess.run(
-                        ["ssh", host, "echo"],
-                        stdout=log_file,
-                        stderr=log_file,
-                        check=True,
-                        timeout=2,
-                    )
-                    log_file.write(f"{host} booted!\n")
-                    log_file.flush()
-                    booted = True
-                except Exception as e:
-                    cur_attempts += 1
-                    log_file.write(f"Waiting for {host} to reboot\n")
-                    log_file.flush()
-                    time.sleep(5)
-
-            if cur_attempts >= max_attempts:
-                exit(1)
-
-
-if __name__ == "__main__":
-    main()
@@ -1,12 +0,0 @@
-[project]
-name = "updates"
-version = "0.1.0"
-description = "Update servers based on ansible inventory"
-readme = "README.md"
-requires-python = ">=3.13"
-dependencies = [
-    "pyyaml>=6.0.3",
-    "tqdm>=4.67.3",
-    "types-pyyaml>=6.0.12.20250915",
-    "types-tqdm>=4.67.3.20260205",
-]
@@ -1,121 +0,0 @@
-version = 1
-revision = 3
-requires-python = ">=3.13"
-
-[[package]]
-name = "colorama"
-version = "0.4.6"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
-]
-
-[[package]]
-name = "pyyaml"
-version = "6.0.3"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" },
-    { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" },
-    { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" },
-    { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" },
-    { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" },
-    { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" },
-    { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" },
-    { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" },
-    { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" },
-    { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" },
-    { url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" },
-    { url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" },
-    { url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" },
-    { url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" },
-    { url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" },
-    { url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" },
-    { url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" },
-    { url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" },
-    { url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" },
-    { url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" },
-    { url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" },
-    { url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" },
-    { url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" },
-    { url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" },
-    { url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" },
-    { url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" },
-    { url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" },
-    { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" },
-]
-
-[[package]]
-name = "tqdm"
-version = "4.67.3"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "colorama", marker = "sys_platform == 'win32'" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/09/a9/6ba95a270c6f1fbcd8dac228323f2777d886cb206987444e4bce66338dd4/tqdm-4.67.3.tar.gz", hash = "sha256:7d825f03f89244ef73f1d4ce193cb1774a8179fd96f31d7e1dcde62092b960bb", size = 169598, upload-time = "2026-02-03T17:35:53.048Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf", size = 78374, upload-time = "2026-02-03T17:35:50.982Z" },
-]
-
-[[package]]
-name = "types-pyyaml"
-version = "6.0.12.20250915"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/7e/69/3c51b36d04da19b92f9e815be12753125bd8bc247ba0470a982e6979e71c/types_pyyaml-6.0.12.20250915.tar.gz", hash = "sha256:0f8b54a528c303f0e6f7165687dd33fafa81c807fcac23f632b63aa624ced1d3", size = 17522, upload-time = "2025-09-15T03:01:00.728Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/bd/e0/1eed384f02555dde685fff1a1ac805c1c7dcb6dd019c916fe659b1c1f9ec/types_pyyaml-6.0.12.20250915-py3-none-any.whl", hash = "sha256:e7d4d9e064e89a3b3cae120b4990cd370874d2bf12fa5f46c97018dd5d3c9ab6", size = 20338, upload-time = "2025-09-15T03:00:59.218Z" },
-]
-
-[[package]]
-name = "types-requests"
-version = "2.32.4.20260107"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "urllib3" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/0f/f3/a0663907082280664d745929205a89d41dffb29e89a50f753af7d57d0a96/types_requests-2.32.4.20260107.tar.gz", hash = "sha256:018a11ac158f801bfa84857ddec1650750e393df8a004a8a9ae2a9bec6fcb24f", size = 23165, upload-time = "2026-01-07T03:20:54.091Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/1c/12/709ea261f2bf91ef0a26a9eed20f2623227a8ed85610c1e54c5805692ecb/types_requests-2.32.4.20260107-py3-none-any.whl", hash = "sha256:b703fe72f8ce5b31ef031264fe9395cac8f46a04661a79f7ed31a80fb308730d", size = 20676, upload-time = "2026-01-07T03:20:52.929Z" },
-]
-
-[[package]]
-name = "types-tqdm"
-version = "4.67.3.20260205"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "types-requests" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/53/46/790b9872523a48163bdda87d47849b4466017640e5259d06eed539340afd/types_tqdm-4.67.3.20260205.tar.gz", hash = "sha256:f3023682d4aa3bbbf908c8c6bb35f35692d319460d9bbd3e646e8852f3dd9f85", size = 17597, upload-time = "2026-02-05T04:03:19.721Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/cc/da/7f761868dbaa328392356fab30c18ab90d14cce86b269e7e63328f29d4a3/types_tqdm-4.67.3.20260205-py3-none-any.whl", hash = "sha256:85c31731e81dc3c5cecc34c6c8b2e5166fafa722468f58840c2b5ac6a8c5c173", size = 23894, upload-time = "2026-02-05T04:03:18.48Z" },
-]
-
-[[package]]
-name = "updates"
-version = "0.1.0"
-source = { virtual = "." }
-dependencies = [
-    { name = "pyyaml" },
-    { name = "tqdm" },
-    { name = "types-pyyaml" },
-    { name = "types-tqdm" },
-]
-
-[package.metadata]
-requires-dist = [
-    { name = "pyyaml", specifier = ">=6.0.3" },
-    { name = "tqdm", specifier = ">=4.67.3" },
-    { name = "types-pyyaml", specifier = ">=6.0.12.20250915" },
-    { name = "types-tqdm", specifier = ">=4.67.3.20260205" },
-]
-
-[[package]]
-name = "urllib3"
-version = "2.6.3"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/c7/24/5f1b3bdffd70275f6661c76461e25f024d5a38a46f04aaca912426a2b1d3/urllib3-2.6.3.tar.gz", hash = "sha256:1b62b6884944a57dbe321509ab94fd4d3b307075e0c2eae991ac71ee15ad38ed", size = 435556, upload-time = "2026-01-07T16:24:43.925Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/39/08/aaaad47bc4e9dc8c725e68f9d04865dbcb2052843ff09c97b08904852d84/urllib3-2.6.3-py3-none-any.whl", hash = "sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4", size = 131584, upload-time = "2026-01-07T16:24:42.685Z" },
-]
@@ -0,0 +1,174 @@
+# An Opinionated Guide to Home Labbing: from Hardware to Software
+
+## Abstract
+
+Between increasingly required subscriptions, privacy terms changing overnight,
+and the difficulty of downloading your own data, it often feels like we rent our
+digital lives rather than own them. We'll walk you through hardware selection,
+the tools you'll need, the operating systems you'll install, and the software
+you might want to support for your homelab. We'll share our experience with our
+own variety of hardware, which ranges from tiny Raspberry Pis to mid-size laptop
+motherboards to overkill Threadripper servers. We'll talk about the operating
+systems we use and which ones have found the most stability and longevity in our
+deployments while comparing them to other homelab operating systems. We'll cover
+common homelab use-cases like media servers, Git servers, network storage, and
+local AI tooling. We'll also talk about safety and maintenance so you can weigh
+the choice of cutting the cloud cord properly. By the end of this talk you'll have a
+starting point for your own homelab journey, and maybe even a grocery list.
+
+Alex Oladele is a senior software engineer at Red Hat, working on Ansible Galaxy
+as part of the Ansible Automation Platform. He is a member of Black Python Devs
+and speaks at conferences about compassionate culture, Python, and practical
+engineering skills. He is passionate about mentoring and creating safe,
+inclusive spaces in tech. Outside of work, he builds silly side projects and
+runs a homelab he is very proud of
+
+Reese is DIY technology enthusiast with a passion for projects that make things
+easy. He's been working in development since 2017 with experience in risk,
+scripting automation, full stack web development, container infrastructure, amd
+homelab server hardware. He has a passion for mentoring, but even more of a
+passion for sharing the new tech he found last week with anyone who will listen.
+Currently, he leads the agentic vulnerability management team at Hungtington
+Bank.
+
+## Structure
+
+"I'm lazy, I don't want my family to kill me, and it works"
+
+1. I want this in my house
+2. I want to connect outside my house
+3. I want my friends to connect
+4. I don't want this to go down
+5. I want to recover if there's an error
+6. My server died, what now?
+
+## Thoughts
+
+Give 2 ideas per section. First for "I can't let this break my family will kill
+me". Second for "I have an understanding partner who is my cat and won't care."
+
+## Story driven presentation
+
+I have decided to make a strong home server. Where do I even start?
+
+Overarching theme:
+
+- This is about privacy, control, and stability
+- This is not about saving money
+- This is not about keeping up with the bleeding edge of the industry
+- (Both of these things might happen though, and that's great when it does)
+
+1. Hardware: you find a box (old laptop, rpi) you're set.
+
+   - Reese
+     - Framework 13 motherboard for experiments, QA, Forgejo/Gitea runners, `experimental.reeseapps.com`
+     - Raspberry pi for home assistant
+     - Threadripper 3000 series with a bunch of disks for main apps
+     - framework desktop for local AI
+   - Alex
+     - "Gaming PC Foundation": 5950x + ddr4 64GB ram (before memory shortage)
+     - Framework motherboard
+     - Raspberry pi cluster
+
+2. I'm a little familiar with linux but what do I install on that hardware?
+
+   - Alex: Truenas, Raspberry Pi OS
+   - Reese: Address the proxmox shaped elephant in the room, Fedora, Alma Linux, cloud init
+
+3. What does your homelab maintenance toolbox look like? What toolkit do you
+   open to work on your homelab? What do you bring to the table to start
+   tinkering?
+
+   - Reese: btop, podman, uv, yarn, pi.dev, borg backup, distrobox (talk about misconceptions like bash history preservation)
+     - AI in the homelab, safety, security, privacy
+     - Discuss the progression: chat bots -> harnesses -> agentic integration
+   - Alex: truenas catalog, ansible, opencode
+     - Discuss security in your tooling like setting wait periods before yarn installing
+     - mention uv acquisition
+     - Building custom apps with AI
+
+4. So you want to launch some containers
+
+   - We dislike docker compose
+   - Truenas apps
+   - quadlets + podman
+   - kubernetes (k3s) and its downsides
+
+5. When to use VMs
+
+   - An extra layer of isolation
+   - Easier networking
+   - Simpler to prevent resource exhaustion
+     - Talk about "right-sizing": Every VM gets 2GB ram and 2cpu cores to start. On a 32GB system, assuming 4GB for the host, that creates 14 VM "slots".
+     - Alma Linux 10 takes up about 450MB at boot with nothing installed
+     - Double the memory when resource exhaustion happens
+
+6. I want to install a new app while I'm at friend's house
+   - Truenas web portal (app page, both official and community) + tailscale
+   - VPN and I need access to my computer
+
+7. I want to check my server status on my phone (updates, disks, memory pressure, error logs, services running)
+   - Truenas web interface
+   - Cockpit web interface
+
+8. I want to add more storage
+   - Truenas ZFS storage pools
+   - BTRFS pools
+
+9. I want to install a new alpha app without much support
+   - Truenas custom docker compose images
+   - Fedora clone and run (in a VM for style)
+
+10. I want to backup my photos
+    - Google Photos: don't use git, images aren't meant for git
+    - **Immich, with backups (tell stories about losing my image data)**
+
+11. I want a local copy of my code
+    - Github
+    - Gitea/Gitlab (talk about that transition)
+    - Forgejo
+
+12. I want private document editing
+    - Google drive, Obsidian (forces use of markdown as my standard)
+    - VSCode + pandoc (commit markdown files as your documents)
+    - Nextcloud (Collabora)
+
+13. I want a local, offline LLMs
+    - llama.cpp is great, but has a very high learning curve
+    - Ollama has pitfalls
+      - You don't know which model you're downloading
+      - Model inference is not optimized
+
+14. I want to watch media I own
+    - Plex - I know that ruffles some jimmies. Give example: add letterbox support into Plex.
+    - Jellyfin if you're cheap
+
+15. I want to know when something goes wrong
+    - Uptime Kuma!
+    - NTFY
+    - N8N
+    - Truenas sending emails if there's an error
+    - Fedora requires a custom solution
+
+16. I want "reasonable availability"
+    - Truenas hits 90%+ availability. Updates take it down for reboot (5-10
+       minutes). Disk failure requires full shutdown, disk swap, and rebuild.
+       This could be half a day.
+    - Fedora hits 90%+ availability. Updates take it down for reboot (<1 min).
+       Disk failures can be ignored by rebalancing. Disk failures still require
+       full shutdown and resilver. This can take half a day.
+
+17. I want to automate my house
+    - Home Assistant (raspberry pi or green)
+
+18. I want backups of all my data
+    - No backups is an option
+    - Local weekly backups to usb drives via Truenas data replication
+    - Borg backup via CLI or Pika.
+    - Full disk backups, app directory backups, hybrid model
+    - Backblaze and S3 integration for Truenas
+    - 3 copies of your data, 2 different media, 1 off site.
+
+    1.  I want a private VPN
+    - Tailscale, moved from wifiman, also moved from pivpn
+    - unifi wireguard server, rawdog wireguard on a pi
@@ -16,6 +16,7 @@ dependencies = [
    "pytest>=9.0.2",
    "pyyaml>=6.0.3",
    "requests>=2.32.5",
+    "rich>=15.0.0",
    "sse-starlette>=3.3.4",
    "tqdm>=4.67.3",
    "types-pyyaml>=6.0.12.20250915",
@@ -285,6 +285,7 @@ dependencies = [
    { name = "pytest" },
    { name = "pyyaml" },
    { name = "requests" },
+    { name = "rich" },
    { name = "sse-starlette" },
    { name = "tqdm" },
    { name = "types-pyyaml" },
@@ -307,6 +308,7 @@ requires-dist = [
    { name = "pytest", specifier = ">=9.0.2" },
    { name = "pyyaml", specifier = ">=6.0.3" },
    { name = "requests", specifier = ">=2.32.5" },
+    { name = "rich", specifier = ">=15.0.0" },
    { name = "sse-starlette", specifier = ">=3.3.4" },
    { name = "tqdm", specifier = ">=4.67.3" },
    { name = "types-pyyaml", specifier = ">=6.0.12.20250915" },
@@ -612,6 +614,18 @@ wheels = [
    { url = "https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl", hash = "sha256:e91464b71ae3ee7afd3017d9f358ef0baf158fd9a298db92f1d4761133824c36", size = 108180, upload-time = "2026-02-09T14:57:25.787Z" },
 ]

+[[package]]
+name = "markdown-it-py"
+version = "4.2.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "mdurl" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" },
+]
+
 [[package]]
 name = "markupsafe"
 version = "3.0.3"
@@ -689,6 +703,15 @@ wheels = [
    { url = "https://files.pythonhosted.org/packages/9c/46/f6b4ad632c67ef35209a66127e4bddc95759649dd595f71f13fba11bdf9a/mcp-1.27.0-py3-none-any.whl", hash = "sha256:5ce1fa81614958e267b21fb2aa34e0aea8e2c6ede60d52aba45fd47246b4d741", size = 215967, upload-time = "2026-04-02T14:48:07.24Z" },
 ]

+[[package]]
+name = "mdurl"
+version = "0.1.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
+]
+
 [[package]]
 name = "mergedeep"
 version = "1.3.4"
@@ -1259,6 +1282,19 @@ wheels = [
    { url = "https://files.pythonhosted.org/packages/3f/51/d4db610ef29373b879047326cbf6fa98b6c1969d6f6dc423279de2b1be2c/requests_toolbelt-1.0.0-py2.py3-none-any.whl", hash = "sha256:cccfdd665f0a24fcf4726e690f65639d272bb0637b9b92dfd91a5568ccf6bd06", size = 54481, upload-time = "2023-05-01T04:11:28.427Z" },
 ]

+[[package]]
+name = "rich"
+version = "15.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "markdown-it-py" },
+    { name = "pygments" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" },
+]
+
 [[package]]
 name = "rpds-py"
 version = "0.30.0"
Author	SHA1	Message	Date
ducoterra	f9b5d293ba	update pi.md with real pi agent config Sync the documented models.json example with the actual ~/.pi/agent/models.json (maxTokens 32000 -> 32768) and add the full ~/.pi/agent/settings.json to the doc. LLM-Generated: true	2026-10-21 00:00:07 -04:00
ducoterra	8bcee5c17f	document ops-user pi install and sudoers escalation policy Adds the sysadmin setup recipe to the pi notes: run pi as a dedicated `ops` user with a tightly controlled sudoers allowlist instead of root, so an agent can do host work without unbounded privilege. Includes the AGENTS.md to drop at /home/ops/AGENTS.md. It records that the allowlist is discovered with `sudo -l` (hostnamectl, reboot, the dnf and systemctl subcommands, firewall-cmd, the ansible shim, libvirt/virsh and composer-cli, getfacl/setfacl, osbuild-auto.sh, btrfs scrub) and sets the hard rule for agents: when a command is not on that list, stop, do not attempt escalation workarounds, and ask the human to add the exact full-path sudoers line for review before proceeding. LLM-Generated: true	2026-10-17 12:26:40 -04:00
ducoterra	e94d4f718c	add README index for the Raspberry Pi systemd runbooks Ties the two runbooks together as one host-agnostic directory: which runbook solves which problem, a file -> target-path map for the units, and the shared deploy pattern (one tar \| ssh pipe into /root, then install with explicit modes rather than copying a whole tree into /etc). States the conventions both units follow - configs under /usr/local as the source of truth with thin systemd wrappers, idempotent installers that re-derive desired state and never delete what they did not create, and gap-limited self-healing so a broken dependency cannot make a unit flap. Also documents the redaction recipe for pasting a WireGuard config: match `PrivateKey[[:space:]]*=` because wg-quick strip emits spaces around the `=`, which a `PrivateKey=` regex misses. LLM-Generated: true	2026-10-16 09:08:16 -04:00
ducoterra	3cdbb405ab	add WireGuard wg0 watchdog units for always-on SSH When SSH to a Pi targets a WireGuard tunnel address, reachability depends on three things that each fail independently: the unit starting at boot, the NAT mapping staying open, and the peer hostname still resolving. Rebooting onto a network with no route to the endpoint previously meant no way back in. - wg-quick@wg0.service.d/10-boot-resilience.conf: retry a failed boot-time up forever (Restart=on-failure, RestartSec=15, StartLimitIntervalSec=0) and cap endpoint resolution (WG_ENDPOINT_RESOLUTION_RETRIES=60, TimeoutStartSec=120) instead of hanging on the stock infinity. - wg0-watchdog: POSIX sh healer, oldest symptom first - missing interface or peer restarts the unit, a missing PersistentKeepalive is set directly, a stale handshake re-points the peer at a freshly resolved endpoint, and a very stale one restarts. Restarts are gap-limited so a broken network cannot make the unit flap, endpoint resolution falls back to the kernel endpoint and then to a last-known-good cache (tunnel DNS may only be reachable through the tunnel), and nothing it did not create is ever removed. Supports --dry-run. - wg0-watchdog.service/.timer: oneshot on a 60s timer, first check 30s after boot, Persistent=true to catch up after downtime, TimeoutStartSec=120 and Nice=10 so a restart cannot wedge the watchdog. - wg0.conf.template: documents /etc/wireguard/wg0.conf with placeholders only. Deliberately omits the DNS key so name resolution does not depend on the tunnel (Debian wg-quick has no `DNS = none`), and sets PersistentKeepalive 25 to hold the NAT mapping for inbound SSH. - wireguard.md: runbook for the placeholder table, failure modes, deploy, and verification. The real wg0.conf and key material never enter the repo. LLM-Generated: true	2026-10-16 09:07:56 -04:00
ducoterra	bdee581480	add self-healing root SSH key systemd units Headless Pis lose /root/.ssh/authorized_keys to imaging, first-boot tooling, or config management, which locks out the only SSH account that can fix it. - ensure-root-ssh-key: idempotent POSIX sh installer. Appends managed public keys from /usr/local/share/root-ssh-keys/authorized_keys, never removes or rewrites foreign keys, skips lines whose first field is not a known key algorithm, and enforces 0700 / 0600 root:root on the key dir and file. - ensure-root-ssh-key.service: oneshot, ordered Before=ssh.service so the key lands before the daemon starts; ConditionPathExists skips it when the managed keys file is absent. - ensure-root-ssh-key.path: PathChanged= + PathModified= on authorized_keys so deletes, renames, and in-place edits are healed at runtime, not just at boot. Writes only when a key is missing, so the unit settles after one no-op run. - authorized_keys.example: template with commented placeholder keys only; the real file stays on the host. - root-ssh-key.md: runbook with deploy, verification, revocation steps, and the gotcha that systemd has no PathMissing= key. LLM-Generated: true	2026-10-16 09:05:54 -04:00
ducoterra	3f14b8ce14	add vllm-radiance R9700 deployment and fp8-kv calibration notes Documents the single-GPU (TP1) vllm-radiance deployment on driveripper-ai, measured KV capacity analysis for the 154K config, and the full FP8-KV calibration workflow (host prep through runtime verification) including quadlet changes, deployment steps, and the remaining matched-control quality gate. LLM-Generated: true	2026-10-12 10:31:43 -04:00
ducoterra	d9925b931d	add sudoers allowlist for ops user automation Passwordless sudo access scoped to specific commands (dnf, systemctl, firewall-cmd, ansible, qemu/virt, btrfs, acl) for ansible and maintenance automations on the fedora host. LLM-Generated: true	2026-10-12 10:31:36 -04:00
ducoterra	be7a4cf0ce	Add Raspberry Pi headless configuration notes Document bootfs headless setup: userconf.txt password hash, and discovery/login steps via nmap and ssh. LLM-Generated: true	2026-10-10 19:37:23 -04:00
ducoterra	54cd5096df	add GUI, screen capture, and audio packages to toolbx image kitty, scrot, and Xvfb enable headless browser/UI interaction; ffmpeg and portaudio cover screen recording and audio capture. LLM-Generated: true	2026-10-10 18:17:25 -04:00
ducoterra	5e8524761b	Add thinking level map to pi model definitions Documents the thinkingLevelMap for the juggernaut and turbo models in the ~/.pi/agent/models.json example, and lowers maxTokens from 128000 to 32000 to match the working configuration. LLM-Generated: true	2026-10-03 22:03:52 -04:00
ducoterra	1e45cace0b	Add Flatpak version installation reference Add documentation for installing a specific Flatpak commit version, including listing available commits and downgrading to a previous one. LLM-Generated: true	2026-09-21 21:11:09 -04:00
ducoterra	dc108a791e	Add devops days presentation, software_pi service, and update software_toolbx - Add presentations/devops_days_self_hosting_presentation.md - Add active/software_pi/ service directory - Update active/software_toolbx/Containerfile - Update retired/kubernetes_userspace submodule LLM-Generated: true	2026-09-19 12:22:09 -04:00
ducoterra	fc91437228	various updates	2026-09-07 11:37:56 -04:00
ducoterra	9995fb29ff	add framework desktop docs	2026-08-16 11:14:51 -04:00
ducoterra	b531e75413	update ai stack with turbo	2026-05-25 21:40:56 -04:00