add framework desktop docs

2026-08-16 11:14:51 -04:00
parent b531e75413
commit 9995fb29ff
1 changed files with 754 additions and 0 deletions
@@ -0,0 +1,754 @@
+# Framework Desktop AI Deployment Guide
+
+Replicating my local AI inference stack: three [llama.cpp](https://github.com/ggml-org/llama.cpp) OpenAI-compatible model servers on a Framework Desktop (AMD Ryzen AI Max 300, Strix Halo iGPU).
+
+Key design decisions:
+
+- **Rootless Podman Quadlet** — no docker-compose, no system services. Everything runs under a dedicated `ai` user via systemd user units. `~/.config/containers/systemd/` is the single source of truth.
+- **Locally built image** — `llama.cpp` is cloned to the box and its official Vulkan Dockerfile is built with podman. The image entrypoint is `/app/llama-server`, so quadlet `Exec=` args are passed straight to the server.
+- **Air-gapped model network** — all model servers sit on an `Internal=true` podman network with no internet egress. Ports are published to the host for API access.
+- **Ansible-driven** — playbooks on the workstation copy the right quadlet into place over SSH and restart the user-scoped systemd service. Swapping models is a one-line playbook var change.
+
+```
+                      workstation (laptop)
+                      ansible-playbook over SSH as user `ai`
+                                     |
+                                     v
+-----------------------------------------------------------------------+
+|  Framework Desktop "deskwork"  (Fedora 43 Server Edition)            |
+|                                                                      |
+|  user `ai`  (loginctl linger on, rootless podman 5.x)                |
+|  /home/ai/.config/containers/systemd/                                |
+|  ├── ai-internal.network   -> network systemd-ai-internal            |
+|  │                               (Internal=true, no internet)        |
+|  ├── ai-embed.container    -> :8001  EmbeddingGemma-300M             |
+|  ├── ai-lite.container     -> :8002  gemma-4-E4B-it-qat (+MTP draft) |
+|  └── ai-turbo.container    -> :8003  Qwen3.6-35B-A3B (MTP)           |
+|                                                                      |
+|  image : localhost/llama-cpp-vulkan:latest   (built from git clone)  |
+|  models: /home/ai/models/{text,embedding}/...   (GGUF via hf CLI)    |
+|  GPU   : /dev/kfd + /dev/dri passed into every container (Vulkan)    |
+-----------------------------------------------------------------------+
+```
+
+| Port | Service  | Model                     | Source quadlet                                   |
+| ---- | -------- | ------------------------- | ------------------------------------------------ |
+| 8001 | ai-embed | EmbeddingGemma-300M       | `embed/quadlets/embeddinggemma-embed.container`  |
+| 8002 | ai-lite  | gemma-4-E4B-it-qat (+MTP) | `lite/quadlets/gemma4-e4b-qat-lite.container`    |
+| 8003 | ai-turbo | Qwen3.6-35B-A3B (MTP)     | `turbo/quadlets/qwen3.6-35b-a3b-turbo.container` |
+
+---
+
+## 1. Hardware
+
+Framework Desktop with an AMD Ryzen AI Max 300 (Strix Halo) — the iGPU is a Vulkan device with access to the unified memory. 128 GB unified memory recommended.
+
+### BIOS
+
+<https://knowledgebase.frame.work/en_us/changing-memory-allocation-amd-ryzen-ai-max-300-series-By1LG5Yrll>
+
+1. Set GPU memory to 512MB
+
+### Kernel args
+
+Edit `/etc/default/grub` and add the following to `GRUB_CMDLINE_LINUX`:
+
+```conf
+amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856
+```
+
+Then regenerate grub and reboot:
+
+```bash
+sudo grub2-mkconfig -o /boot/grub2/grub.cfg
+sudo reboot
+```
+
+After boot, verify the GPU is visible:
+
+```bash
+ls -l /dev/kfd /dev/dri/renderD128
+```
+
+Expected (note the world-writable modes — that's why the `ai` user needs no special groups):
+
+```text
+crw-rw-rw-. 1 root render 234, 0 ... /dev/kfd
+crw-rw-rw-. 1 root render 226, 128 ... /dev/dri/renderD128
+```
+
+## 2. Operating System
+
+Fedora 43 (Server Edition, minimal is fine). Requires cgroup v2 and a recent podman with Quadlet support (`podman-systemd` generator) — Fedora 40+ ships it.
+
+```bash
+sudo dnf install -y podman git
+podman --version        # 5.x
+podman info --format '{{.Host.CgroupsVersion}}'   # must print: v2
+```
+
+## 3. The `ai` user
+
+All containers run rootless as a dedicated `ai` user with linger enabled (so the user systemd manager + containers survive logout and start at boot).
+
+On the Framework Desktop:
+
+```bash
+sudo useradd -m ai
+sudo loginctl enable-linger ai
+
+# Give the ai user SSH access from your workstation
+sudo -u ai mkdir -p /home/ai/.ssh
+sudo -u ai tee /home/ai/.ssh/authorized_keys > /dev/null < ~/.ssh/id_ed25519.pub
+sudo -u ai chmod 700 /home/ai/.ssh
+sudo -u ai chmod 600 /home/ai/.ssh/authorized_keys
+```
+
+On your workstation's `~/.ssh/config`:
+
+```text
+Host deskwork-ai
+    HostName deskwork.reeselink.com
+    User ai
+```
+
+(Use whatever hostname/IP your Framework Desktop has. The inventory and playbooks reference the host as `deskwork-ai`.)
+
+## 4. Models
+
+As the `ai` user on the Framework Desktop.
+
+### Install the Hugging Face CLI
+
+<https://huggingface.co/docs/huggingface_hub/en/guides/cli#getting-started>
+
+```bash
+curl -LsSf https://hf.co/cli/install.sh | bash
+hf auth login
+```
+
+### Create the model dirs
+
+```bash
+mkdir -p /home/ai/models/{text,image,video,embedding,tts,stt}
+```
+
+### Download the three model sets
+
+```bash
+# --- embed (ai-embed, port 8001) ---
+mkdir -p /home/ai/models/embedding/emebeddinggemma-300m
+cd /home/ai/models/embedding/emebeddinggemma-300m
+# NOTE: the "emebeddinggemma" typo is intentional — it matches the quadlet below
+hf download --local-dir . ggml-org/EmbeddingGemma-300M-GGUF embeddinggemma-300M-BF16.gguf
+```
+
+```bash
+# --- lite (ai-lite, port 8002) ---
+mkdir -p /home/ai/models/text/gemma-4-e4b-it-qat
+cd /home/ai/models/text/gemma-4-e4b-it-qat
+# main model + MTP draft (speculative decoding) + vision projector
+hf download --local-dir . unsloth/gemma-4-E4B-it-qat-GGUF \
+  --include gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
+  --include mtp-gemma-4-E4B-it.gguf \
+  --include mmproj-BF16.gguf
+```
+
+```bash
+# --- turbo (ai-turbo, port 8003) ---
+mkdir -p /home/ai/models/text/qwen3.6-35b-a3b-mtp
+cd /home/ai/models/text/qwen3.6-35b-a3b-mtp
+hf download --local-dir . unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
+  --include Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
+  --include mmproj-F32.gguf
+```
+
+The turbo model is ~37 GB; make sure you have the space (`df -h /home`).
+
+## 5. llama.cpp + the Vulkan image
+
+As the `ai` user on the Framework Desktop:
+
+```bash
+cd /home/ai
+git clone https://github.com/ggml-org/llama.cpp.git
+cd llama.cpp
+podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
+```
+
+The multi-stage `.devops/vulkan.Dockerfile` builds the full toolset with the Vulkan backend; the final stage's entrypoint is `/app/llama-server`, so any args after the image name go straight to the server. Podman tags local builds under the `localhost/` namespace, which is why quadlets reference `localhost/llama-cpp-vulkan:latest`.
+
+Verify:
+
+```bash
+podman images | grep llama-cpp-vulkan
+```
+
+Optional extras (only if you use them):
+
+```bash
+# ROCm image (for the AMD dGPU boxes)
+podman build -t llama-cpp-rocm:latest -f .devops/rocm.Dockerfile .
+
+# Diffusion variant (diffusion-gemma turbo quadlet) — built from a fork that
+# has diffusion support wired in
+git clone https://github.com/danielhanchen/llama.cpp.git llama.cpp-diffusion
+cd llama.cpp-diffusion
+podman build -t llama-cpp-diffusion-vulkan:latest -f .devops/vulkan.Dockerfile .
+```
+
+## 6. Quadlet primer
+
+Podman Quadlet reads files from `~/.config/containers/systemd/` (rootless search path) and generates systemd user units via a systemd generator. No compose files, no hand-written units.
+
+Naming rules that matter here:
+
+| Quadlet file    | Generated systemd service | Podman resource                                             |
+| --------------- | ------------------------- | ----------------------------------------------------------- |
+| `foo.container` | `foo.service`             | container `systemd-foo` (unless `ContainerName=` overrides) |
+| `foo.network`   | `foo-network.service`     | network `systemd-foo`                                       |
+
+- `Network=ai-internal.network` in a container quadlet means: use the network from the `ai-internal.network` quadlet file **and** auto-add a dependency on `ai-internal-network.service`, so the network exists before the container starts.
+- Quadlet units are "transient" (generated), so `systemctl enable` doesn't apply — the generator applies the `[Install]` section itself. That's what `WantedBy=multi-user.target default.target` in every file below is for: start at boot.
+- Image pulls can exceed systemd's default 90s start timeout, hence `TimeoutStartSec=900`.
+- Debugging: `systemd-analyze --user verify <unit>.service` and `/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun`.
+
+Full quadlet reference: <https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html>
+
+## 7. The files
+
+Everything below lives in the `Deployments` repo on the workstation. Layout:
+
+```text
+Deployments/
+├── inventory.yaml
+└── reeseapps/ai/
+    ├── roles/containers-infra/
+    │   ├── tasks/main.yaml
+    │   └── templates/ai-internal.network
+    └── deployments/
+        ├── embed/
+        │   ├── playbook.yaml
+        │   └── quadlets/embeddinggemma-embed.container
+        ├── lite/
+        │   ├── playbook.yaml
+        │   └── quadlets/
+        │       ├── gemma4-e2b-qat-lite.container
+        │       ├── gemma4-e4b-qat-lite.container   <- active
+        │       └── lfm2.5-2.6b.container
+        └── turbo/
+            ├── playbook.yaml
+            └── quadlets/
+                ├── diffusion-gemma.container
+                ├── gemma-4-26b-a4b-turbo.container
+                ├── gemma-4-qat-turbo.container
+                ├── muse-glimmer-30b-juggernaut-turbo.container
+                └── qwen3.6-35b-a3b-turbo.container <- active
+```
+
+### 7.1 inventory.yaml
+
+Only the `ai` group is needed for this stack:
+
+```yaml
+ai:
+  hosts:
+    deskwork-ai:
+      ansible_python_interpreter: /usr/bin/python3
+```
+
+### 7.2 The `containers-infra` role
+
+Runs first in every playbook. Creates the quadlet dir and installs the air-gapped network.
+
+`reeseapps/ai/roles/containers-infra/tasks/main.yaml`:
+
+```yaml
+- name: Create /home/ai/.config/containers/systemd
+  ansible.builtin.file:
+    path: /home/ai/.config/containers/systemd
+    state: directory
+    mode: "0755"
+
+- name: Copy infra quadlets
+  template:
+    src: "{{ item }}"
+    dest: "/home/ai/.config/containers/systemd/{{ item }}"
+  loop:
+    - ai-internal.network
+```
+
+`reeseapps/ai/roles/containers-infra/templates/ai-internal.network`:
+
+```ini
+[Network]
+Internal=true
+```
+
+`Internal=true` gives the network NAT-style local connectivity but no internet egress — the model servers don't need it.
+
+### 7.3 The playbooks
+
+All three playbooks have the same shape: run the infra role, copy the selected quadlet (chosen by the `container_file` var) to a fixed destination name, then daemon-reload and restart the resulting user service.
+
+`reeseapps/ai/deployments/embed/playbook.yaml`:
+
+```yaml
+- name: Create Embedding AI Stack
+  hosts: deskwork-ai
+  vars:
+    container_file: quadlets/embeddinggemma-embed.container
+  roles:
+    - ../../roles/containers-infra
+  tasks:
+    - name: Copy ai-embed.container
+      copy:
+        src: "{{ container_file }}"
+        dest: /home/ai/.config/containers/systemd/ai-embed.container
+        owner: ai
+        group: ai
+        mode: "0644"
+    - name: Reload and start the ai-embed service
+      ansible.builtin.systemd_service:
+        state: restarted
+        name: ai-embed
+        daemon_reload: true
+        scope: user
+```
+
+`reeseapps/ai/deployments/lite/playbook.yaml`:
+
+```yaml
+- name: Create Lite Service
+  hosts: deskwork-ai
+  vars:
+    container_file: quadlets/gemma4-e4b-qat-lite.container
+  roles:
+    - ../../roles/containers-infra
+  tasks:
+    - name: Copy ai-lite.container
+      copy:
+        src: "{{ container_file }}"
+        dest: /home/ai/.config/containers/systemd/ai-lite.container
+        owner: ai
+        group: ai
+        mode: "0644"
+    - name: Reload and start the ai-lite service
+      ansible.builtin.systemd_service:
+        state: restarted
+        name: ai-lite
+        daemon_reload: true
+        scope: user
+```
+
+`reeseapps/ai/deployments/turbo/playbook.yaml`:
+
+```yaml
+- name: Create Turbo AI Stack
+  hosts: deskwork-ai
+  vars:
+    container_file: quadlets/qwen3.6-35b-a3b-turbo.container
+  roles:
+    - ../../roles/containers-infra
+  tasks:
+    - name: Copy ai-turbo.container
+      copy:
+        src: "{{ container_file }}"
+        dest: /home/ai/.config/containers/systemd/ai-turbo.container
+        owner: ai
+        group: ai
+        mode: "0644"
+    - name: Reload and start the ai-turbo service
+      ansible.builtin.systemd_service:
+        state: restarted
+        name: ai-turbo
+        daemon_reload: true
+        scope: user
+```
+
+To swap which model a service runs, change only the `container_file` var and re-run that playbook.
+
+### 7.4 The quadlets (currently active variants)
+
+`embed/quadlets/embeddinggemma-embed.container` → installed as `ai-embed.container`:
+
+```ini
+[Unit]
+Description=A Llama CPP Server For Embedding Models
+
+[Container]
+ContainerName=ai-embed
+
+# Internet-disconnected network
+Network=ai-internal.network
+
+# llama.cpp juggernaut
+PublishPort=8001:8001/tcp
+
+# Image is built locally via podman build
+Image=localhost/llama-cpp-vulkan:latest
+AutoUpdate=registry
+
+# Downloaded models volume
+Volume=/home/ai/models/embedding:/models:z
+
+# GPU Device
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+
+# Server command
+Exec=--port 8001 \
+    -c 0 \
+    -b 1024 \
+    -ub 1024 \
+    --perf \
+    --n-gpu-layers all \
+    --embedding \
+    -m /models/emebeddinggemma-300m/embeddinggemma-300M-BF16.gguf \
+    --alias embed
+
+# Health Check
+HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8001/health || exit 1
+HealthInterval=10s
+HealthRetries=3
+HealthStartPeriod=10s
+HealthTimeout=30s
+HealthOnFailure=kill
+
+# EnvironmentFile=/home/ai/.llama-api/keys.env
+
+[Service]
+Restart=always
+# Extend Timeout to allow time to pull the image
+TimeoutStartSec=900
+
+[Install]
+# Start by default on boot
+WantedBy=multi-user.target default.target
+```
+
+`lite/quadlets/gemma4-e4b-qat-lite.container` → installed as `ai-lite.container`:
+
+```ini
+[Unit]
+Description=A Llama CPP Server Running a Non-Reasoning Model
+
+[Container]
+ContainerName=ai-lite
+
+# Internet-disconnected network
+Network=ai-internal.network
+
+# llama.cpp juggernaut
+PublishPort=8002:8002/tcp
+# Image is built locally via podman build
+Image=localhost/llama-cpp-vulkan:latest
+AutoUpdate=registry
+
+# Downloaded models volume
+Volume=/home/ai/models/text:/models:z
+
+# GPU Device
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+
+# Server command
+Exec=--port 8002 \
+    --parallel 1 \
+    --temp 1.0 \
+    --top-p 0.95 \
+    --top-k 64 \
+    -fa on \
+    --no-mmap \
+    --kv-unified \
+    --perf \
+    --spec-type draft-mtp --spec-draft-n-max 3 \
+    --reasoning off \
+    --model /models/gemma-4-e4b-it-qat/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
+    --model-draft /models/gemma-4-e4b-it-qat/mtp-gemma-4-E4B-it.gguf \
+    --mmproj /models/gemma-4-e4b-it-qat/mmproj-BF16.gguf \
+    --alias lite
+
+# Health Check
+HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8002/health || exit 1
+HealthInterval=10s
+HealthRetries=3
+HealthStartPeriod=10s
+HealthTimeout=30s
+HealthOnFailure=kill
+
+# EnvironmentFile=/home/ai/.llama-api/keys.env
+
+[Service]
+Restart=always
+# Extend Timeout to allow time to pull the image
+TimeoutStartSec=900
+
+[Install]
+# Start by default on boot
+WantedBy=multi-user.target default.target
+```
+
+`turbo/quadlets/qwen3.6-35b-a3b-turbo.container` → installed as `ai-turbo.container`:
+
+```ini
+[Unit]
+Description=A Llama CPP Server Running a Reasoning Model
+
+[Container]
+ContainerName=ai-turbo
+
+# Internet-disconnected network
+Network=ai-internal.network
+
+# llama.cpp juggernaut
+PublishPort=8003:8003/tcp
+
+# Image is built locally via podman build
+# latest-mtp is for mtp testing
+# latest is main branch
+Image=localhost/llama-cpp-vulkan:latest
+AutoUpdate=registry
+
+# Downloaded models volume
+Volume=/home/ai/models/text:/models:z
+
+# GPU Device
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+
+# Server command
+Exec=--port 8003 \
+    --temp 0.7 \
+    --top-p 0.8 \
+    --top-k 20 \
+    --presence-penalty 1.5 \
+    --min-p 0.00 \
+    --chat-template-kwargs '{"enable_thinking":false}' \
+    -ctk q8_0 \
+    -ctv q8_0 \
+    --kv-unified \
+    --parallel 2 \
+    -fa on \
+    --load-mode none \
+    --image-min-tokens 4096 \
+    --image-max-tokens 4096 \
+    --n-gpu-layers all \
+    --perf \
+    --jinja \
+    --reasoning-preserve \
+    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
+    --mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \
+    --spec-type draft-mtp --spec-draft-n-max 3 \
+    --spec-draft-type-k q8_0 \
+    --spec-draft-type-v q8_0 \
+    --alias turbo
+
+# Health Check
+# CMD-SHELL is string form, CMD is array form
+HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8003/health || exit 1
+HealthInterval=10s
+HealthRetries=3
+HealthStartPeriod=30s
+HealthTimeout=30s
+HealthOnFailure=kill
+
+[Service]
+Restart=always
+# Extend Timeout to allow time to pull the image
+TimeoutStartSec=900
+
+[Install]
+# Start by default on boot
+WantedBy=multi-user.target default.target
+```
+
+Notes on the server flags:
+
+- `--alias` sets the model name served on `/v1/models`.
+- `--spec-type draft-mtp` + `--model-draft` (or the MTP head inside the main GGUF for Qwen) enables MTP speculative decoding for a big speedup.
+- `-fa on` = flash attention; `--n-gpu-layers all` = offload everything to the GPU; `--perf` prints timing stats to the log.
+- The healthcheck hits the server's `/health` endpoint; `HealthOnFailure=kill` + `Restart=always` means a wedged server is killed and restarted by systemd.
+- `:z` on the volume = SELinux shared-label for rootless containers.
+
+## 8. Ansible setup (workstation)
+
+From zero on the workstation:
+
+```bash
+# Install ansible (my setup uses pipx; uv tool install ansible works too)
+pipx install ansible
+ansible-playbook --version
+```
+
+No Ansible Galaxy collections are required for this stack — the playbooks only use `ansible.builtin` modules and the local `containers-infra` role (referenced by relative path `../../roles/containers-infra`, which is why the directory layout in §7.1 must be preserved).
+
+Put the files from §7 into a `Deployments/` directory on the workstation (or clone the full repo if you have access). Then, from the repo root:
+
+```bash
+cd Deployments
+
+# Order matters only for the first run: embed creates the shared network.
+ansible-playbook -i inventory.yaml reeseapps/ai/deployments/embed/playbook.yaml
+ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml
+ansible-playbook -i inventory.yaml reeseapps/ai/deployments/turbo/playbook.yaml
+```
+
+Each run is idempotent and safe to re-run. The first turbo start will take a while (37 GB model load) — that's what `TimeoutStartSec=900` is for.
+
+## 9. Verification
+
+On the Framework Desktop (as `ai`):
+
+```bash
+systemctl --user status ai-embed ai-lite ai-turbo
+podman ps
+podman network ls | grep systemd-ai-internal
+journalctl --user -fu ai-turbo   # watch startup, look for the --perf stats
+```
+
+Expected `podman ps`:
+
+```text
+ai-embed   localhost/llama-cpp-vulkan:latest   0.0.0.0:8001->8001/tcp
+ai-lite    localhost/llama-cpp-vulkan:latest   0.0.0.0:8002->8002/tcp
+ai-turbo   localhost/llama-cpp-vulkan:latest   0.0.0.0:8003->8003/tcp
+```
+
+From the host (OpenAI-compatible API):
+
+```bash
+# Health
+curl http://localhost:8001/health
+curl http://localhost:8002/health
+curl http://localhost:8003/health
+
+# List models
+curl -s http://localhost:8003/v1/models | jq
+
+# Chat completion (turbo)
+curl -s http://localhost:8003/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "turbo",
+    "messages": [
+      {"role": "user", "content": "Say hello in five words."}
+    ],
+    "max_tokens": 100
+  }' | jq
+
+# Chat completion (lite)
+curl -s http://localhost:8002/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "lite",
+    "messages": [
+      {"role": "user", "content": "Say hello in five words."}
+    ],
+    "max_tokens": 100
+  }' | jq
+
+# Embeddings (embed)
+curl -s http://localhost:8001/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "embed",
+    "input": "This is the reason you ended up here."
+  }' | jq '.data[0].embedding[0:5]'
+```
+
+## 10. Day-2 operations
+
+### Swap the model a service runs
+
+Change `container_file` in that service's playbook and re-run it:
+
+```bash
+# e.g. run the smaller 2B gemma on ai-lite instead
+$EDITOR reeseapps/ai/deployments/lite/playbook.yaml
+#    container_file: quadlets/gemma4-e2b-qat-lite.container
+ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml
+```
+
+Available swap-in quadlets:
+
+- lite: `gemma4-e2b-qat-lite.container`, `gemma4-e4b-qat-lite.container` (active), `lfm2.5-2.6b.container`
+- turbo: `qwen3.6-35b-a3b-turbo.container` (active), `gemma-4-26b-a4b-turbo.container`, `gemma-4-qat-turbo.container`, `muse-glimmer-30b-juggernaut-turbo.container`, `diffusion-gemma.container` (uses `localhost/llama-cpp-diffusion-vulkan:latest`)
+
+Make sure the model files the new quadlet references are already downloaded (see the `--model` / `--mmproj` / `--model-draft` paths inside it).
+
+### Update llama.cpp (image rebuild)
+
+On the Framework Desktop as `ai`:
+
+```bash
+cd /home/ai/llama.cpp
+git pull
+podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
+
+# Restart the servers so they pick up the new image
+systemctl --user restart ai-embed ai-lite ai-turbo
+```
+
+Tagged snapshots are a good habit:
+
+```bash
+export BUILD_TAG=$(date +"%Y-%m-%d-%H-%M-%S")
+podman build -t llama-cpp-vulkan:${BUILD_TAG} -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
+```
+
+Note: `AutoUpdate=registry` in the quadlets is effectively a no-op for `localhost/` images (podman auto-update needs a remote registry to compare against) — updates are manual, as above.
+
+### Logs and troubleshooting
+
+```bash
+journalctl --user -fu ai-turbo                  # follow logs
+systemctl --user status ai-embed ai-lite ai-turbo
+podman exec ai-turbo curl -s http://127.0.0.1:8003/health
+systemd-analyze --user verify ai-turbo.service  # validate generated unit
+/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun
+```
+
+Common failures:
+
+- `Unit ai-turbo.service not found` → the generator failed on a bad quadlet option (often a syntax error or an option your podman version doesn't know). Run the `podman-system-generator --dryrun` command above to see the error.
+- GPU errors at startup → confirm `/dev/kfd` and `/dev/dri/renderD128` exist and the user can read/write them; on a box where they're `0660`, add the user: `sudo usermod -aG render ai` then re-login.
+- Service times out on first start → normal while loading a big model; `TimeoutStartSec=900` covers ~15 min.
+- SELinux `avc` denials on model reads → make sure volume mounts use the `:z` suffix.
+
+### Manual escape hatch
+
+Skip systemd entirely and run a server by hand (useful for testing a new model):
+
+```bash
+podman run -it --rm \
+--device=/dev/kfd \
+--device=/dev/dri \
+-v /home/ai/models/text:/models:z \
+--entrypoint /bin/bash \
+localhost/llama-cpp-vulkan:latest
+
+./llama-server \
+-m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
+--mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \
+--port 8003 \
+-ctk q8_0 -ctv q8_0 \
+--kv-unified \
+--parallel 2 \
+-fa on \
+--load-mode none \
+--n-gpu-layers all \
+--perf \
+--jinja \
+--reasoning-preserve \
+--spec-type draft-mtp --spec-draft-n-max 3 \
+--spec-draft-type-k q8_0 \
+--spec-draft-type-v q8_0
+```
+
+## References
+
+- Podman Quadlet: <https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html>
+- llama.cpp server docs: <https://github.com/ggml-org/llama.cpp/tree/master/tools/server>
+- Framework Desktop hardware notes: `active/device_framework_desktop/framework_desktop.md`
+- Broader AI stack notes (model lists, benchmarks, vLLM, stable-diffusion.cpp): `active/software_ai_stack/ai_stack.md`