homelab/active/software_ai_stack/ai_framework_desktop.md

Framework Desktop AI Deployment Guide

Replicating my local AI inference stack: three llama.cpp OpenAI-compatible model servers on a Framework Desktop (AMD Ryzen AI Max 300, Strix Halo iGPU).

Key design decisions:

• Rootless Podman Quadlet — no docker-compose, no system services. Everything runs under a dedicated ai user via systemd user units. ~/.config/containers/systemd/ is the single source of truth.
• Locally built image — llama.cpp is cloned to the box and its official Vulkan Dockerfile is built with podman. The image entrypoint is /app/llama-server, so quadlet Exec= args are passed straight to the server.
• Air-gapped model network — all model servers sit on an Internal=true podman network with no internet egress. Ports are published to the host for API access.
• Ansible-driven — playbooks on the workstation copy the right quadlet into place over SSH and restart the user-scoped systemd service. Swapping models is a one-line playbook var change.

                      workstation (laptop)
                      ansible-playbook over SSH as user `ai`
                                     |
                                     v
+-----------------------------------------------------------------------+
|  Framework Desktop "deskwork"  (Fedora 43 Server Edition)            |
|                                                                      |
|  user `ai`  (loginctl linger on, rootless podman 5.x)                |
|  /home/ai/.config/containers/systemd/                                |
|  ├── ai-internal.network   -> network systemd-ai-internal            |
|  │                               (Internal=true, no internet)        |
|  ├── ai-embed.container    -> :8001  EmbeddingGemma-300M             |
|  ├── ai-lite.container     -> :8002  gemma-4-E4B-it-qat (+MTP draft) |
|  └── ai-turbo.container    -> :8003  Qwen3.6-35B-A3B (MTP)           |
|                                                                      |
|  image : localhost/llama-cpp-vulkan:latest   (built from git clone)  |
|  models: /home/ai/models/{text,embedding}/...   (GGUF via hf CLI)    |
|  GPU   : /dev/kfd + /dev/dri passed into every container (Vulkan)    |
+-----------------------------------------------------------------------+

Port	Service	Model	Source quadlet
8001	ai-embed	EmbeddingGemma-300M	embed/quadlets/embeddinggemma-embed.container
8002	ai-lite	gemma-4-E4B-it-qat (+MTP)	lite/quadlets/gemma4-e4b-qat-lite.container
8003	ai-turbo	Qwen3.6-35B-A3B (MTP)	turbo/quadlets/qwen3.6-35b-a3b-turbo.container

---

1. Hardware

Framework Desktop with an AMD Ryzen AI Max 300 (Strix Halo) — the iGPU is a Vulkan device with access to the unified memory. 128 GB unified memory recommended.

BIOS

https://knowledgebase.frame.work/en_us/changing-memory-allocation-amd-ryzen-ai-max-300-series-By1LG5Yrll

1. Set GPU memory to 512MB

Kernel args

Edit /etc/default/grub and add the following to GRUB_CMDLINE_LINUX:

amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856

Then regenerate grub and reboot:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

After boot, verify the GPU is visible:

ls -l /dev/kfd /dev/dri/renderD128

Expected (note the world-writable modes — that's why the ai user needs no special groups):

crw-rw-rw-. 1 root render 234, 0 ... /dev/kfd
crw-rw-rw-. 1 root render 226, 128 ... /dev/dri/renderD128

2. Operating System

Fedora 43 (Server Edition, minimal is fine). Requires cgroup v2 and a recent podman with Quadlet support (podman-systemd generator) — Fedora 40+ ships it.

sudo dnf install -y podman git
podman --version        # 5.x
podman info --format '{{.Host.CgroupsVersion}}'   # must print: v2

3. The ai user

All containers run rootless as a dedicated ai user with linger enabled (so the user systemd manager + containers survive logout and start at boot).

On the Framework Desktop:

sudo useradd -m ai
sudo loginctl enable-linger ai

# Give the ai user SSH access from your workstation
sudo -u ai mkdir -p /home/ai/.ssh
sudo -u ai tee /home/ai/.ssh/authorized_keys > /dev/null < ~/.ssh/id_ed25519.pub
sudo -u ai chmod 700 /home/ai/.ssh
sudo -u ai chmod 600 /home/ai/.ssh/authorized_keys

On your workstation's ~/.ssh/config:

Host deskwork-ai
    HostName deskwork.reeselink.com
    User ai

(Use whatever hostname/IP your Framework Desktop has. The inventory and playbooks reference the host as deskwork-ai.)

4. Models

As the ai user on the Framework Desktop.

Install the Hugging Face CLI

https://huggingface.co/docs/huggingface_hub/en/guides/cli#getting-started

curl -LsSf https://hf.co/cli/install.sh | bash
hf auth login

Create the model dirs

mkdir -p /home/ai/models/{text,image,video,embedding,tts,stt}

Download the three model sets

# --- embed (ai-embed, port 8001) ---
mkdir -p /home/ai/models/embedding/emebeddinggemma-300m
cd /home/ai/models/embedding/emebeddinggemma-300m
# NOTE: the "emebeddinggemma" typo is intentional — it matches the quadlet below
hf download --local-dir . ggml-org/EmbeddingGemma-300M-GGUF embeddinggemma-300M-BF16.gguf

# --- lite (ai-lite, port 8002) ---
mkdir -p /home/ai/models/text/gemma-4-e4b-it-qat
cd /home/ai/models/text/gemma-4-e4b-it-qat
# main model + MTP draft (speculative decoding) + vision projector
hf download --local-dir . unsloth/gemma-4-E4B-it-qat-GGUF \
  --include gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
  --include mtp-gemma-4-E4B-it.gguf \
  --include mmproj-BF16.gguf

# --- turbo (ai-turbo, port 8003) ---
mkdir -p /home/ai/models/text/qwen3.6-35b-a3b-mtp
cd /home/ai/models/text/qwen3.6-35b-a3b-mtp
hf download --local-dir . unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
  --include Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
  --include mmproj-F32.gguf

The turbo model is ~37 GB; make sure you have the space (df -h /home).

5. llama.cpp + the Vulkan image

As the ai user on the Framework Desktop:

cd /home/ai
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .

The multi-stage .devops/vulkan.Dockerfile builds the full toolset with the Vulkan backend; the final stage's entrypoint is /app/llama-server, so any args after the image name go straight to the server. Podman tags local builds under the localhost/ namespace, which is why quadlets reference localhost/llama-cpp-vulkan:latest.

Verify:

podman images | grep llama-cpp-vulkan

Optional extras (only if you use them):

# ROCm image (for the AMD dGPU boxes)
podman build -t llama-cpp-rocm:latest -f .devops/rocm.Dockerfile .

# Diffusion variant (diffusion-gemma turbo quadlet) — built from a fork that
# has diffusion support wired in
git clone https://github.com/danielhanchen/llama.cpp.git llama.cpp-diffusion
cd llama.cpp-diffusion
podman build -t llama-cpp-diffusion-vulkan:latest -f .devops/vulkan.Dockerfile .

6. Quadlet primer

Podman Quadlet reads files from ~/.config/containers/systemd/ (rootless search path) and generates systemd user units via a systemd generator. No compose files, no hand-written units.

Naming rules that matter here:

Quadlet file	Generated systemd service	Podman resource
foo.container	foo.service	container systemd-foo (unless ContainerName= overrides)
foo.network	foo-network.service	network systemd-foo

• Network=ai-internal.network in a container quadlet means: use the network from the ai-internal.network quadlet file and auto-add a dependency on ai-internal-network.service, so the network exists before the container starts.
• Quadlet units are "transient" (generated), so systemctl enable doesn't apply — the generator applies the [Install] section itself. That's what WantedBy=multi-user.target default.target in every file below is for: start at boot.
• Image pulls can exceed systemd's default 90s start timeout, hence TimeoutStartSec=900.
• Debugging: systemd-analyze --user verify <unit>.service and /usr/lib/systemd/system-generators/podman-system-generator --user --dryrun.

Full quadlet reference: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html

7. The files

Everything below lives in the Deployments repo on the workstation. Layout:

Deployments/
├── inventory.yaml
└── reeseapps/ai/
    ├── roles/containers-infra/
    │   ├── tasks/main.yaml
    │   └── templates/ai-internal.network
    └── deployments/
        ├── embed/
        │   ├── playbook.yaml
        │   └── quadlets/embeddinggemma-embed.container
        ├── lite/
        │   ├── playbook.yaml
        │   └── quadlets/
        │       ├── gemma4-e2b-qat-lite.container
        │       ├── gemma4-e4b-qat-lite.container   <- active
        │       └── lfm2.5-2.6b.container
        └── turbo/
            ├── playbook.yaml
            └── quadlets/
                ├── diffusion-gemma.container
                ├── gemma-4-26b-a4b-turbo.container
                ├── gemma-4-qat-turbo.container
                ├── muse-glimmer-30b-juggernaut-turbo.container
                └── qwen3.6-35b-a3b-turbo.container <- active

7.1 inventory.yaml

Only the ai group is needed for this stack:

ai:
  hosts:
    deskwork-ai:
      ansible_python_interpreter: /usr/bin/python3

7.2 The containers-infra role

Runs first in every playbook. Creates the quadlet dir and installs the air-gapped network.

reeseapps/ai/roles/containers-infra/tasks/main.yaml:

- name: Create /home/ai/.config/containers/systemd
  ansible.builtin.file:
    path: /home/ai/.config/containers/systemd
    state: directory
    mode: "0755"

- name: Copy infra quadlets
  template:
    src: "{{ item }}"
    dest: "/home/ai/.config/containers/systemd/{{ item }}"
  loop:
    - ai-internal.network

reeseapps/ai/roles/containers-infra/templates/ai-internal.network:

[Network]
Internal=true

Internal=true gives the network NAT-style local connectivity but no internet egress — the model servers don't need it.

7.3 The playbooks

All three playbooks have the same shape: run the infra role, copy the selected quadlet (chosen by the container_file var) to a fixed destination name, then daemon-reload and restart the resulting user service.

reeseapps/ai/deployments/embed/playbook.yaml:

- name: Create Embedding AI Stack
  hosts: deskwork-ai
  vars:
    container_file: quadlets/embeddinggemma-embed.container
  roles:
    - ../../roles/containers-infra
  tasks:
    - name: Copy ai-embed.container
      copy:
        src: "{{ container_file }}"
        dest: /home/ai/.config/containers/systemd/ai-embed.container
        owner: ai
        group: ai
        mode: "0644"
    - name: Reload and start the ai-embed service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-embed
        daemon_reload: true
        scope: user

reeseapps/ai/deployments/lite/playbook.yaml:

- name: Create Lite Service
  hosts: deskwork-ai
  vars:
    container_file: quadlets/gemma4-e4b-qat-lite.container
  roles:
    - ../../roles/containers-infra
  tasks:
    - name: Copy ai-lite.container
      copy:
        src: "{{ container_file }}"
        dest: /home/ai/.config/containers/systemd/ai-lite.container
        owner: ai
        group: ai
        mode: "0644"
    - name: Reload and start the ai-lite service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-lite
        daemon_reload: true
        scope: user

reeseapps/ai/deployments/turbo/playbook.yaml:

- name: Create Turbo AI Stack
  hosts: deskwork-ai
  vars:
    container_file: quadlets/qwen3.6-35b-a3b-turbo.container
  roles:
    - ../../roles/containers-infra
  tasks:
    - name: Copy ai-turbo.container
      copy:
        src: "{{ container_file }}"
        dest: /home/ai/.config/containers/systemd/ai-turbo.container
        owner: ai
        group: ai
        mode: "0644"
    - name: Reload and start the ai-turbo service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-turbo
        daemon_reload: true
        scope: user

To swap which model a service runs, change only the container_file var and re-run that playbook.

7.4 The quadlets (currently active variants)

embed/quadlets/embeddinggemma-embed.container → installed as ai-embed.container:

[Unit]
Description=A Llama CPP Server For Embedding Models

[Container]
ContainerName=ai-embed

# Internet-disconnected network
Network=ai-internal.network

# llama.cpp juggernaut
PublishPort=8001:8001/tcp

# Image is built locally via podman build
Image=localhost/llama-cpp-vulkan:latest
AutoUpdate=registry

# Downloaded models volume
Volume=/home/ai/models/embedding:/models:z

# GPU Device
AddDevice=/dev/kfd
AddDevice=/dev/dri

# Server command
Exec=--port 8001 \
    -c 0 \
    -b 1024 \
    -ub 1024 \
    --perf \
    --n-gpu-layers all \
    --embedding \
    -m /models/emebeddinggemma-300m/embeddinggemma-300M-BF16.gguf \
    --alias embed

# Health Check
HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8001/health || exit 1
HealthInterval=10s
HealthRetries=3
HealthStartPeriod=10s
HealthTimeout=30s
HealthOnFailure=kill

# EnvironmentFile=/home/ai/.llama-api/keys.env

[Service]
Restart=always
# Extend Timeout to allow time to pull the image
TimeoutStartSec=900

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

lite/quadlets/gemma4-e4b-qat-lite.container → installed as ai-lite.container:

[Unit]
Description=A Llama CPP Server Running a Non-Reasoning Model

[Container]
ContainerName=ai-lite

# Internet-disconnected network
Network=ai-internal.network

# llama.cpp juggernaut
PublishPort=8002:8002/tcp
# Image is built locally via podman build
Image=localhost/llama-cpp-vulkan:latest
AutoUpdate=registry

# Downloaded models volume
Volume=/home/ai/models/text:/models:z

# GPU Device
AddDevice=/dev/kfd
AddDevice=/dev/dri

# Server command
Exec=--port 8002 \
    --parallel 1 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64 \
    -fa on \
    --no-mmap \
    --kv-unified \
    --perf \
    --spec-type draft-mtp --spec-draft-n-max 3 \
    --reasoning off \
    --model /models/gemma-4-e4b-it-qat/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
    --model-draft /models/gemma-4-e4b-it-qat/mtp-gemma-4-E4B-it.gguf \
    --mmproj /models/gemma-4-e4b-it-qat/mmproj-BF16.gguf \
    --alias lite

# Health Check
HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8002/health || exit 1
HealthInterval=10s
HealthRetries=3
HealthStartPeriod=10s
HealthTimeout=30s
HealthOnFailure=kill

# EnvironmentFile=/home/ai/.llama-api/keys.env

[Service]
Restart=always
# Extend Timeout to allow time to pull the image
TimeoutStartSec=900

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

turbo/quadlets/qwen3.6-35b-a3b-turbo.container → installed as ai-turbo.container:

[Unit]
Description=A Llama CPP Server Running a Reasoning Model

[Container]
ContainerName=ai-turbo

# Internet-disconnected network
Network=ai-internal.network

# llama.cpp juggernaut
PublishPort=8003:8003/tcp

# Image is built locally via podman build
# latest-mtp is for mtp testing
# latest is main branch
Image=localhost/llama-cpp-vulkan:latest
AutoUpdate=registry

# Downloaded models volume
Volume=/home/ai/models/text:/models:z

# GPU Device
AddDevice=/dev/kfd
AddDevice=/dev/dri

# Server command
Exec=--port 8003 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --presence-penalty 1.5 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}' \
    -ctk q8_0 \
    -ctv q8_0 \
    --kv-unified \
    --parallel 2 \
    -fa on \
    --load-mode none \
    --image-min-tokens 4096 \
    --image-max-tokens 4096 \
    --n-gpu-layers all \
    --perf \
    --jinja \
    --reasoning-preserve \
    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
    --mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \
    --spec-type draft-mtp --spec-draft-n-max 3 \
    --spec-draft-type-k q8_0 \
    --spec-draft-type-v q8_0 \
    --alias turbo

# Health Check
# CMD-SHELL is string form, CMD is array form
HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8003/health || exit 1
HealthInterval=10s
HealthRetries=3
HealthStartPeriod=30s
HealthTimeout=30s
HealthOnFailure=kill

[Service]
Restart=always
# Extend Timeout to allow time to pull the image
TimeoutStartSec=900

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

Notes on the server flags:

• --alias sets the model name served on /v1/models.
• --spec-type draft-mtp + --model-draft (or the MTP head inside the main GGUF for Qwen) enables MTP speculative decoding for a big speedup.
• -fa on = flash attention; --n-gpu-layers all = offload everything to the GPU; --perf prints timing stats to the log.
• The healthcheck hits the server's /health endpoint; HealthOnFailure=kill + Restart=always means a wedged server is killed and restarted by systemd.
• :z on the volume = SELinux shared-label for rootless containers.

8. Ansible setup (workstation)

From zero on the workstation:

# Install ansible (my setup uses pipx; uv tool install ansible works too)
pipx install ansible
ansible-playbook --version

No Ansible Galaxy collections are required for this stack — the playbooks only use ansible.builtin modules and the local containers-infra role (referenced by relative path ../../roles/containers-infra, which is why the directory layout in §7.1 must be preserved).

Put the files from §7 into a Deployments/ directory on the workstation (or clone the full repo if you have access). Then, from the repo root:

cd Deployments

# Order matters only for the first run: embed creates the shared network.
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/embed/playbook.yaml
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/turbo/playbook.yaml

Each run is idempotent and safe to re-run. The first turbo start will take a while (37 GB model load) — that's what TimeoutStartSec=900 is for.

9. Verification

On the Framework Desktop (as ai):

systemctl --user status ai-embed ai-lite ai-turbo
podman ps
podman network ls | grep systemd-ai-internal
journalctl --user -fu ai-turbo   # watch startup, look for the --perf stats

Expected podman ps:

ai-embed   localhost/llama-cpp-vulkan:latest   0.0.0.0:8001->8001/tcp
ai-lite    localhost/llama-cpp-vulkan:latest   0.0.0.0:8002->8002/tcp
ai-turbo   localhost/llama-cpp-vulkan:latest   0.0.0.0:8003->8003/tcp

From the host (OpenAI-compatible API):

# Health
curl http://localhost:8001/health
curl http://localhost:8002/health
curl http://localhost:8003/health

# List models
curl -s http://localhost:8003/v1/models | jq

# Chat completion (turbo)
curl -s http://localhost:8003/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "turbo",
    "messages": [
      {"role": "user", "content": "Say hello in five words."}
    ],
    "max_tokens": 100
  }' | jq

# Chat completion (lite)
curl -s http://localhost:8002/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lite",
    "messages": [
      {"role": "user", "content": "Say hello in five words."}
    ],
    "max_tokens": 100
  }' | jq

# Embeddings (embed)
curl -s http://localhost:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "embed",
    "input": "This is the reason you ended up here."
  }' | jq '.data[0].embedding[0:5]'

10. Day-2 operations

Swap the model a service runs

Change container_file in that service's playbook and re-run it:

# e.g. run the smaller 2B gemma on ai-lite instead
$EDITOR reeseapps/ai/deployments/lite/playbook.yaml
#    container_file: quadlets/gemma4-e2b-qat-lite.container
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml

Available swap-in quadlets:

• lite: gemma4-e2b-qat-lite.container, gemma4-e4b-qat-lite.container (active), lfm2.5-2.6b.container
• turbo: qwen3.6-35b-a3b-turbo.container (active), gemma-4-26b-a4b-turbo.container, gemma-4-qat-turbo.container, muse-glimmer-30b-juggernaut-turbo.container, diffusion-gemma.container (uses localhost/llama-cpp-diffusion-vulkan:latest)

Make sure the model files the new quadlet references are already downloaded (see the --model / --mmproj / --model-draft paths inside it).

Update llama.cpp (image rebuild)

On the Framework Desktop as ai:

cd /home/ai/llama.cpp
git pull
podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .

# Restart the servers so they pick up the new image
systemctl --user restart ai-embed ai-lite ai-turbo

Tagged snapshots are a good habit:

export BUILD_TAG=$(date +"%Y-%m-%d-%H-%M-%S")
podman build -t llama-cpp-vulkan:${BUILD_TAG} -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .

Note: AutoUpdate=registry in the quadlets is effectively a no-op for localhost/ images (podman auto-update needs a remote registry to compare against) — updates are manual, as above.

Logs and troubleshooting

journalctl --user -fu ai-turbo                  # follow logs
systemctl --user status ai-embed ai-lite ai-turbo
podman exec ai-turbo curl -s http://127.0.0.1:8003/health
systemd-analyze --user verify ai-turbo.service  # validate generated unit
/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun

Common failures:

• Unit ai-turbo.service not found → the generator failed on a bad quadlet option (often a syntax error or an option your podman version doesn't know). Run the podman-system-generator --dryrun command above to see the error.
• GPU errors at startup → confirm /dev/kfd and /dev/dri/renderD128 exist and the user can read/write them; on a box where they're 0660, add the user: sudo usermod -aG render ai then re-login.
• Service times out on first start → normal while loading a big model; TimeoutStartSec=900 covers ~15 min.
• SELinux avc denials on model reads → make sure volume mounts use the :z suffix.

Manual escape hatch

Skip systemd entirely and run a server by hand (useful for testing a new model):

podman run -it --rm \
--device=/dev/kfd \
--device=/dev/dri \
-v /home/ai/models/text:/models:z \
--entrypoint /bin/bash \
localhost/llama-cpp-vulkan:latest

./llama-server \
-m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
--mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \
--port 8003 \
-ctk q8_0 -ctv q8_0 \
--kv-unified \
--parallel 2 \
-fa on \
--load-mode none \
--n-gpu-layers all \
--perf \
--jinja \
--reasoning-preserve \
--spec-type draft-mtp --spec-draft-n-max 3 \
--spec-draft-type-k q8_0 \
--spec-draft-type-v q8_0

References

• Podman Quadlet: https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html
• llama.cpp server docs: https://github.com/ggml-org/llama.cpp/tree/master/tools/server
• Framework Desktop hardware notes: active/device_framework_desktop/framework_desktop.md
• Broader AI stack notes (model lists, benchmarks, vLLM, stable-diffusion.cpp): active/software_ai_stack/ai_stack.md