add framework desktop docs

2026-08-16 11:14:51 -04:00
parent b531e75413
commit 9995fb29ff
1 changed files with 754 additions and 0 deletions
@@ -0,0 +1,754 @@
 # Framework Desktop AI Deployment Guide
 Replicating my local AI inference stack: three [llama.cpp](https://github.com/ggml-org/llama.cpp) OpenAI-compatible model servers on a Framework Desktop (AMD Ryzen AI Max 300, Strix Halo iGPU).
 Key design decisions:
 - **Rootless Podman Quadlet** — no docker-compose, no system services. Everything runs under a dedicated `ai` user via systemd user units. `~/.config/containers/systemd/` is the single source of truth.
 - **Locally built image** — `llama.cpp` is cloned to the box and its official Vulkan Dockerfile is built with podman. The image entrypoint is `/app/llama-server`, so quadlet `Exec=` args are passed straight to the server.
 - **Air-gapped model network** — all model servers sit on an `Internal=true` podman network with no internet egress. Ports are published to the host for API access.
 - **Ansible-driven** — playbooks on the workstation copy the right quadlet into place over SSH and restart the user-scoped systemd service. Swapping models is a one-line playbook var change.
 ```
                      workstation (laptop)
                      ansible-playbook over SSH as user `ai`
                                     |
                                     v
 +-----------------------------------------------------------------------+
 |  Framework Desktop "deskwork"  (Fedora 43 Server Edition)            |
 |                                                                      |
 |  user `ai`  (loginctl linger on, rootless podman 5.x)                |
 |  /home/ai/.config/containers/systemd/                                |
 |  ├── ai-internal.network   -> network systemd-ai-internal            |
 |  │                               (Internal=true, no internet)        |
 |  ├── ai-embed.container    -> :8001  EmbeddingGemma-300M             |
 |  ├── ai-lite.container     -> :8002  gemma-4-E4B-it-qat (+MTP draft) |
 |  └── ai-turbo.container    -> :8003  Qwen3.6-35B-A3B (MTP)           |
 |                                                                      |
 |  image : localhost/llama-cpp-vulkan:latest   (built from git clone)  |
 |  models: /home/ai/models/{text,embedding}/...   (GGUF via hf CLI)    |
 |  GPU   : /dev/kfd + /dev/dri passed into every container (Vulkan)    |
 +-----------------------------------------------------------------------+
 ```
 | Port | Service  | Model                     | Source quadlet                                   |
 | ---- | -------- | ------------------------- | ------------------------------------------------ |
 | 8001 | ai-embed | EmbeddingGemma-300M       | `embed/quadlets/embeddinggemma-embed.container`  |
 | 8002 | ai-lite  | gemma-4-E4B-it-qat (+MTP) | `lite/quadlets/gemma4-e4b-qat-lite.container`    |
 | 8003 | ai-turbo | Qwen3.6-35B-A3B (MTP)     | `turbo/quadlets/qwen3.6-35b-a3b-turbo.container` |
 ---
 ## 1. Hardware
 Framework Desktop with an AMD Ryzen AI Max 300 (Strix Halo) — the iGPU is a Vulkan device with access to the unified memory. 128 GB unified memory recommended.
 ### BIOS
 <https://knowledgebase.frame.work/en_us/changing-memory-allocation-amd-ryzen-ai-max-300-series-By1LG5Yrll>
 1. Set GPU memory to 512MB
 ### Kernel args
 Edit `/etc/default/grub` and add the following to `GRUB_CMDLINE_LINUX`:
 ```conf
 amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856
 ```
 Then regenerate grub and reboot:
 ```bash
 sudo grub2-mkconfig -o /boot/grub2/grub.cfg
 sudo reboot
 ```
 After boot, verify the GPU is visible:
 ```bash
 ls -l /dev/kfd /dev/dri/renderD128
 ```
 Expected (note the world-writable modes — that's why the `ai` user needs no special groups):
 ```text
 crw-rw-rw-. 1 root render 234, 0 ... /dev/kfd
 crw-rw-rw-. 1 root render 226, 128 ... /dev/dri/renderD128
 ```
 ## 2. Operating System
 Fedora 43 (Server Edition, minimal is fine). Requires cgroup v2 and a recent podman with Quadlet support (`podman-systemd` generator) — Fedora 40+ ships it.
 ```bash
 sudo dnf install -y podman git
 podman --version        # 5.x
 podman info --format '{{.Host.CgroupsVersion}}'   # must print: v2
 ```
 ## 3. The `ai` user
 All containers run rootless as a dedicated `ai` user with linger enabled (so the user systemd manager + containers survive logout and start at boot).
 On the Framework Desktop:
 ```bash
 sudo useradd -m ai
 sudo loginctl enable-linger ai
 # Give the ai user SSH access from your workstation
 sudo -u ai mkdir -p /home/ai/.ssh
 sudo -u ai tee /home/ai/.ssh/authorized_keys > /dev/null < ~/.ssh/id_ed25519.pub
 sudo -u ai chmod 700 /home/ai/.ssh
 sudo -u ai chmod 600 /home/ai/.ssh/authorized_keys
 ```
 On your workstation's `~/.ssh/config`:
 ```text
 Host deskwork-ai
    HostName deskwork.reeselink.com
    User ai
 ```
 (Use whatever hostname/IP your Framework Desktop has. The inventory and playbooks reference the host as `deskwork-ai`.)
 ## 4. Models
 As the `ai` user on the Framework Desktop.
 ### Install the Hugging Face CLI
 <https://huggingface.co/docs/huggingface_hub/en/guides/cli#getting-started>
 ```bash
 curl -LsSf https://hf.co/cli/install.sh | bash
 hf auth login
 ```
 ### Create the model dirs
 ```bash
 mkdir -p /home/ai/models/{text,image,video,embedding,tts,stt}
 ```
 ### Download the three model sets
 ```bash
 # --- embed (ai-embed, port 8001) ---
 mkdir -p /home/ai/models/embedding/emebeddinggemma-300m
 cd /home/ai/models/embedding/emebeddinggemma-300m
 # NOTE: the "emebeddinggemma" typo is intentional — it matches the quadlet below
 hf download --local-dir . ggml-org/EmbeddingGemma-300M-GGUF embeddinggemma-300M-BF16.gguf
 ```
 ```bash
 # --- lite (ai-lite, port 8002) ---
 mkdir -p /home/ai/models/text/gemma-4-e4b-it-qat
 cd /home/ai/models/text/gemma-4-e4b-it-qat
 # main model + MTP draft (speculative decoding) + vision projector
 hf download --local-dir . unsloth/gemma-4-E4B-it-qat-GGUF \
  --include gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
  --include mtp-gemma-4-E4B-it.gguf \
  --include mmproj-BF16.gguf
 ```
 ```bash
 # --- turbo (ai-turbo, port 8003) ---
 mkdir -p /home/ai/models/text/qwen3.6-35b-a3b-mtp
 cd /home/ai/models/text/qwen3.6-35b-a3b-mtp
 hf download --local-dir . unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
  --include Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
  --include mmproj-F32.gguf
 ```
 The turbo model is ~37 GB; make sure you have the space (`df -h /home`).
 ## 5. llama.cpp + the Vulkan image
 As the `ai` user on the Framework Desktop:
 ```bash
 cd /home/ai
 git clone https://github.com/ggml-org/llama.cpp.git
 cd llama.cpp
 podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
 ```
 The multi-stage `.devops/vulkan.Dockerfile` builds the full toolset with the Vulkan backend; the final stage's entrypoint is `/app/llama-server`, so any args after the image name go straight to the server. Podman tags local builds under the `localhost/` namespace, which is why quadlets reference `localhost/llama-cpp-vulkan:latest`.
 Verify:
 ```bash
 podman images | grep llama-cpp-vulkan
 ```
 Optional extras (only if you use them):
 ```bash
 # ROCm image (for the AMD dGPU boxes)
 podman build -t llama-cpp-rocm:latest -f .devops/rocm.Dockerfile .
 # Diffusion variant (diffusion-gemma turbo quadlet) — built from a fork that
 # has diffusion support wired in
 git clone https://github.com/danielhanchen/llama.cpp.git llama.cpp-diffusion
 cd llama.cpp-diffusion
 podman build -t llama-cpp-diffusion-vulkan:latest -f .devops/vulkan.Dockerfile .
 ```
 ## 6. Quadlet primer
 Podman Quadlet reads files from `~/.config/containers/systemd/` (rootless search path) and generates systemd user units via a systemd generator. No compose files, no hand-written units.
 Naming rules that matter here:
 | Quadlet file    | Generated systemd service | Podman resource                                             |
 | --------------- | ------------------------- | ----------------------------------------------------------- |
 | `foo.container` | `foo.service`             | container `systemd-foo` (unless `ContainerName=` overrides) |
 | `foo.network`   | `foo-network.service`     | network `systemd-foo`                                       |
 - `Network=ai-internal.network` in a container quadlet means: use the network from the `ai-internal.network` quadlet file **and** auto-add a dependency on `ai-internal-network.service`, so the network exists before the container starts.
 - Quadlet units are "transient" (generated), so `systemctl enable` doesn't apply — the generator applies the `[Install]` section itself. That's what `WantedBy=multi-user.target default.target` in every file below is for: start at boot.
 - Image pulls can exceed systemd's default 90s start timeout, hence `TimeoutStartSec=900`.
 - Debugging: `systemd-analyze --user verify <unit>.service` and `/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun`.
 Full quadlet reference: <https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html>
 ## 7. The files
 Everything below lives in the `Deployments` repo on the workstation. Layout:
 ```text
 Deployments/
 ├── inventory.yaml
 └── reeseapps/ai/
    ├── roles/containers-infra/
    │   ├── tasks/main.yaml
    │   └── templates/ai-internal.network
    └── deployments/
        ├── embed/
        │   ├── playbook.yaml
        │   └── quadlets/embeddinggemma-embed.container
        ├── lite/
        │   ├── playbook.yaml
        │   └── quadlets/
        │       ├── gemma4-e2b-qat-lite.container
        │       ├── gemma4-e4b-qat-lite.container   <- active
        │       └── lfm2.5-2.6b.container
        └── turbo/
            ├── playbook.yaml
            └── quadlets/
                ├── diffusion-gemma.container
                ├── gemma-4-26b-a4b-turbo.container
                ├── gemma-4-qat-turbo.container
                ├── muse-glimmer-30b-juggernaut-turbo.container
                └── qwen3.6-35b-a3b-turbo.container <- active
 ```
 ### 7.1 inventory.yaml
 Only the `ai` group is needed for this stack:
 ```yaml
 ai:
  hosts:
    deskwork-ai:
      ansible_python_interpreter: /usr/bin/python3
 ```
 ### 7.2 The `containers-infra` role
 Runs first in every playbook. Creates the quadlet dir and installs the air-gapped network.
 `reeseapps/ai/roles/containers-infra/tasks/main.yaml`:
 ```yaml
 - name: Create /home/ai/.config/containers/systemd
  ansible.builtin.file:
    path: /home/ai/.config/containers/systemd
    state: directory
    mode: "0755"
 - name: Copy infra quadlets
  template:
    src: "{{ item }}"
    dest: "/home/ai/.config/containers/systemd/{{ item }}"
  loop:
    - ai-internal.network
 ```
 `reeseapps/ai/roles/containers-infra/templates/ai-internal.network`:
 ```ini
 [Network]
 Internal=true
 ```
 `Internal=true` gives the network NAT-style local connectivity but no internet egress — the model servers don't need it.
 ### 7.3 The playbooks
 All three playbooks have the same shape: run the infra role, copy the selected quadlet (chosen by the `container_file` var) to a fixed destination name, then daemon-reload and restart the resulting user service.
 `reeseapps/ai/deployments/embed/playbook.yaml`:
 ```yaml
 - name: Create Embedding AI Stack
  hosts: deskwork-ai
  vars:
    container_file: quadlets/embeddinggemma-embed.container
  roles:
    - ../../roles/containers-infra
  tasks:
    - name: Copy ai-embed.container
      copy:
        src: "{{ container_file }}"
        dest: /home/ai/.config/containers/systemd/ai-embed.container
        owner: ai
        group: ai
        mode: "0644"
    - name: Reload and start the ai-embed service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-embed
        daemon_reload: true
        scope: user
 ```
 `reeseapps/ai/deployments/lite/playbook.yaml`:
 ```yaml
 - name: Create Lite Service
  hosts: deskwork-ai
  vars:
    container_file: quadlets/gemma4-e4b-qat-lite.container
  roles:
    - ../../roles/containers-infra
  tasks:
    - name: Copy ai-lite.container
      copy:
        src: "{{ container_file }}"
        dest: /home/ai/.config/containers/systemd/ai-lite.container
        owner: ai
        group: ai
        mode: "0644"
    - name: Reload and start the ai-lite service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-lite
        daemon_reload: true
        scope: user
 ```
 `reeseapps/ai/deployments/turbo/playbook.yaml`:
 ```yaml
 - name: Create Turbo AI Stack
  hosts: deskwork-ai
  vars:
    container_file: quadlets/qwen3.6-35b-a3b-turbo.container
  roles:
    - ../../roles/containers-infra
  tasks:
    - name: Copy ai-turbo.container
      copy:
        src: "{{ container_file }}"
        dest: /home/ai/.config/containers/systemd/ai-turbo.container
        owner: ai
        group: ai
        mode: "0644"
    - name: Reload and start the ai-turbo service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-turbo
        daemon_reload: true
        scope: user
 ```
 To swap which model a service runs, change only the `container_file` var and re-run that playbook.
 ### 7.4 The quadlets (currently active variants)
 `embed/quadlets/embeddinggemma-embed.container` → installed as `ai-embed.container`:
 ```ini
 [Unit]
 Description=A Llama CPP Server For Embedding Models
 [Container]
 ContainerName=ai-embed
 # Internet-disconnected network
 Network=ai-internal.network
 # llama.cpp juggernaut
 PublishPort=8001:8001/tcp
 # Image is built locally via podman build
 Image=localhost/llama-cpp-vulkan:latest
 AutoUpdate=registry
 # Downloaded models volume
 Volume=/home/ai/models/embedding:/models:z
 # GPU Device
 AddDevice=/dev/kfd
 AddDevice=/dev/dri
 # Server command
 Exec=--port 8001 \
    -c 0 \
    -b 1024 \
    -ub 1024 \
    --perf \
    --n-gpu-layers all \
    --embedding \
    -m /models/emebeddinggemma-300m/embeddinggemma-300M-BF16.gguf \
    --alias embed
 # Health Check
 HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8001/health || exit 1
 HealthInterval=10s
 HealthRetries=3
 HealthStartPeriod=10s
 HealthTimeout=30s
 HealthOnFailure=kill
 # EnvironmentFile=/home/ai/.llama-api/keys.env
 [Service]
 Restart=always
 # Extend Timeout to allow time to pull the image
 TimeoutStartSec=900
 [Install]
 # Start by default on boot
 WantedBy=multi-user.target default.target
 ```
 `lite/quadlets/gemma4-e4b-qat-lite.container` → installed as `ai-lite.container`:
 ```ini
 [Unit]
 Description=A Llama CPP Server Running a Non-Reasoning Model
 [Container]
 ContainerName=ai-lite
 # Internet-disconnected network
 Network=ai-internal.network
 # llama.cpp juggernaut
 PublishPort=8002:8002/tcp
 # Image is built locally via podman build
 Image=localhost/llama-cpp-vulkan:latest
 AutoUpdate=registry
 # Downloaded models volume
 Volume=/home/ai/models/text:/models:z
 # GPU Device
 AddDevice=/dev/kfd
 AddDevice=/dev/dri
 # Server command
 Exec=--port 8002 \
    --parallel 1 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64 \
    -fa on \
    --no-mmap \
    --kv-unified \
    --perf \
    --spec-type draft-mtp --spec-draft-n-max 3 \
    --reasoning off \
    --model /models/gemma-4-e4b-it-qat/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
    --model-draft /models/gemma-4-e4b-it-qat/mtp-gemma-4-E4B-it.gguf \
    --mmproj /models/gemma-4-e4b-it-qat/mmproj-BF16.gguf \
    --alias lite
 # Health Check
 HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8002/health || exit 1
 HealthInterval=10s
 HealthRetries=3
 HealthStartPeriod=10s
 HealthTimeout=30s
 HealthOnFailure=kill
 # EnvironmentFile=/home/ai/.llama-api/keys.env
 [Service]
 Restart=always
 # Extend Timeout to allow time to pull the image
 TimeoutStartSec=900
 [Install]
 # Start by default on boot
 WantedBy=multi-user.target default.target
 ```
 `turbo/quadlets/qwen3.6-35b-a3b-turbo.container` → installed as `ai-turbo.container`:
 ```ini
 [Unit]
 Description=A Llama CPP Server Running a Reasoning Model
 [Container]
 ContainerName=ai-turbo
 # Internet-disconnected network
 Network=ai-internal.network
 # llama.cpp juggernaut
 PublishPort=8003:8003/tcp
 # Image is built locally via podman build
 # latest-mtp is for mtp testing
 # latest is main branch
 Image=localhost/llama-cpp-vulkan:latest
 AutoUpdate=registry
 # Downloaded models volume
 Volume=/home/ai/models/text:/models:z
 # GPU Device
 AddDevice=/dev/kfd
 AddDevice=/dev/dri
 # Server command
 Exec=--port 8003 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --presence-penalty 1.5 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}' \
    -ctk q8_0 \
    -ctv q8_0 \
    --kv-unified \
    --parallel 2 \
    -fa on \
    --load-mode none \
    --image-min-tokens 4096 \
    --image-max-tokens 4096 \
    --n-gpu-layers all \
    --perf \
    --jinja \
    --reasoning-preserve \
    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
    --mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \
    --spec-type draft-mtp --spec-draft-n-max 3 \
    --spec-draft-type-k q8_0 \
    --spec-draft-type-v q8_0 \
    --alias turbo
 # Health Check
 # CMD-SHELL is string form, CMD is array form
 HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8003/health || exit 1
 HealthInterval=10s
 HealthRetries=3
 HealthStartPeriod=30s
 HealthTimeout=30s
 HealthOnFailure=kill
 [Service]
 Restart=always
 # Extend Timeout to allow time to pull the image
 TimeoutStartSec=900
 [Install]
 # Start by default on boot
 WantedBy=multi-user.target default.target
 ```
 Notes on the server flags:
 - `--alias` sets the model name served on `/v1/models`.
 - `--spec-type draft-mtp` + `--model-draft` (or the MTP head inside the main GGUF for Qwen) enables MTP speculative decoding for a big speedup.
 - `-fa on` = flash attention; `--n-gpu-layers all` = offload everything to the GPU; `--perf` prints timing stats to the log.
 - The healthcheck hits the server's `/health` endpoint; `HealthOnFailure=kill` + `Restart=always` means a wedged server is killed and restarted by systemd.
 - `:z` on the volume = SELinux shared-label for rootless containers.
 ## 8. Ansible setup (workstation)
 From zero on the workstation:
 ```bash
 # Install ansible (my setup uses pipx; uv tool install ansible works too)
 pipx install ansible
 ansible-playbook --version
 ```
 No Ansible Galaxy collections are required for this stack — the playbooks only use `ansible.builtin` modules and the local `containers-infra` role (referenced by relative path `../../roles/containers-infra`, which is why the directory layout in §7.1 must be preserved).
 Put the files from §7 into a `Deployments/` directory on the workstation (or clone the full repo if you have access). Then, from the repo root:
 ```bash
 cd Deployments
 # Order matters only for the first run: embed creates the shared network.
 ansible-playbook -i inventory.yaml reeseapps/ai/deployments/embed/playbook.yaml
 ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml
 ansible-playbook -i inventory.yaml reeseapps/ai/deployments/turbo/playbook.yaml
 ```
 Each run is idempotent and safe to re-run. The first turbo start will take a while (37 GB model load) — that's what `TimeoutStartSec=900` is for.
 ## 9. Verification
 On the Framework Desktop (as `ai`):
 ```bash
 systemctl --user status ai-embed ai-lite ai-turbo
 podman ps
 podman network ls | grep systemd-ai-internal
 journalctl --user -fu ai-turbo   # watch startup, look for the --perf stats
 ```
 Expected `podman ps`:
 ```text
 ai-embed   localhost/llama-cpp-vulkan:latest   0.0.0.0:8001->8001/tcp
 ai-lite    localhost/llama-cpp-vulkan:latest   0.0.0.0:8002->8002/tcp
 ai-turbo   localhost/llama-cpp-vulkan:latest   0.0.0.0:8003->8003/tcp
 ```
 From the host (OpenAI-compatible API):
 ```bash
 # Health
 curl http://localhost:8001/health
 curl http://localhost:8002/health
 curl http://localhost:8003/health
 # List models
 curl -s http://localhost:8003/v1/models | jq
 # Chat completion (turbo)
 curl -s http://localhost:8003/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "turbo",
    "messages": [
      {"role": "user", "content": "Say hello in five words."}
    ],
    "max_tokens": 100
  }' | jq
 # Chat completion (lite)
 curl -s http://localhost:8002/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lite",
    "messages": [
      {"role": "user", "content": "Say hello in five words."}
    ],
    "max_tokens": 100
  }' | jq
 # Embeddings (embed)
 curl -s http://localhost:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "embed",
    "input": "This is the reason you ended up here."
  }' | jq '.data[0].embedding[0:5]'
 ```
 ## 10. Day-2 operations
 ### Swap the model a service runs
 Change `container_file` in that service's playbook and re-run it:
 ```bash
 # e.g. run the smaller 2B gemma on ai-lite instead
 $EDITOR reeseapps/ai/deployments/lite/playbook.yaml
 #    container_file: quadlets/gemma4-e2b-qat-lite.container
 ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml
 ```
 Available swap-in quadlets:
 - lite: `gemma4-e2b-qat-lite.container`, `gemma4-e4b-qat-lite.container` (active), `lfm2.5-2.6b.container`
 - turbo: `qwen3.6-35b-a3b-turbo.container` (active), `gemma-4-26b-a4b-turbo.container`, `gemma-4-qat-turbo.container`, `muse-glimmer-30b-juggernaut-turbo.container`, `diffusion-gemma.container` (uses `localhost/llama-cpp-diffusion-vulkan:latest`)
 Make sure the model files the new quadlet references are already downloaded (see the `--model` / `--mmproj` / `--model-draft` paths inside it).
 ### Update llama.cpp (image rebuild)
 On the Framework Desktop as `ai`:
 ```bash
 cd /home/ai/llama.cpp
 git pull
 podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
 # Restart the servers so they pick up the new image
 systemctl --user restart ai-embed ai-lite ai-turbo
 ```
 Tagged snapshots are a good habit:
 ```bash
 export BUILD_TAG=$(date +"%Y-%m-%d-%H-%M-%S")
 podman build -t llama-cpp-vulkan:${BUILD_TAG} -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
 ```
 Note: `AutoUpdate=registry` in the quadlets is effectively a no-op for `localhost/` images (podman auto-update needs a remote registry to compare against) — updates are manual, as above.
 ### Logs and troubleshooting
 ```bash
 journalctl --user -fu ai-turbo                  # follow logs
 systemctl --user status ai-embed ai-lite ai-turbo
 podman exec ai-turbo curl -s http://127.0.0.1:8003/health
 systemd-analyze --user verify ai-turbo.service  # validate generated unit
 /usr/lib/systemd/system-generators/podman-system-generator --user --dryrun
 ```
 Common failures:
 - `Unit ai-turbo.service not found` → the generator failed on a bad quadlet option (often a syntax error or an option your podman version doesn't know). Run the `podman-system-generator --dryrun` command above to see the error.
 - GPU errors at startup → confirm `/dev/kfd` and `/dev/dri/renderD128` exist and the user can read/write them; on a box where they're `0660`, add the user: `sudo usermod -aG render ai` then re-login.
 - Service times out on first start → normal while loading a big model; `TimeoutStartSec=900` covers ~15 min.
 - SELinux `avc` denials on model reads → make sure volume mounts use the `:z` suffix.
 ### Manual escape hatch
 Skip systemd entirely and run a server by hand (useful for testing a new model):
 ```bash
 podman run -it --rm \
 --device=/dev/kfd \
 --device=/dev/dri \
 -v /home/ai/models/text:/models:z \
 --entrypoint /bin/bash \
 localhost/llama-cpp-vulkan:latest
 ./llama-server \
 -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
 --mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \
 --port 8003 \
 -ctk q8_0 -ctv q8_0 \
 --kv-unified \
 --parallel 2 \
 -fa on \
 --load-mode none \
 --n-gpu-layers all \
 --perf \
 --jinja \
 --reasoning-preserve \
 --spec-type draft-mtp --spec-draft-n-max 3 \
 --spec-draft-type-k q8_0 \
 --spec-draft-type-v q8_0
 ```
 ## References
 - Podman Quadlet: <https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html>
 - llama.cpp server docs: <https://github.com/ggml-org/llama.cpp/tree/master/tools/server>
 - Framework Desktop hardware notes: `active/device_framework_desktop/framework_desktop.md`
 - Broader AI stack notes (model lists, benchmarks, vLLM, stable-diffusion.cpp): `active/software_ai_stack/ai_stack.md`