# Framework Desktop AI Deployment Guide

Replicating my local AI inference stack: three [llama.cpp](https://github.com/ggml-org/llama.cpp) OpenAI-compatible model servers on a Framework Desktop (AMD Ryzen AI Max 300, Strix Halo iGPU).

Key design decisions:

- **Rootless Podman Quadlet** — no docker-compose, no system services. Everything runs under a dedicated `ai` user via systemd user units. `~/.config/containers/systemd/` is the single source of truth.
- **Locally built image** — `llama.cpp` is cloned to the box and its official Vulkan Dockerfile is built with podman. The image entrypoint is `/app/llama-server`, so quadlet `Exec=` args are passed straight to the server.
- **Air-gapped model network** — all model servers sit on an `Internal=true` podman network with no internet egress. Ports are published to the host for API access.
- **Ansible-driven** — playbooks on the workstation copy the right quadlet into place over SSH and restart the user-scoped systemd service. Swapping models is a one-line playbook var change.

```
                      workstation (laptop)
                      ansible-playbook over SSH as user `ai`
                                     |
                                     v
+-----------------------------------------------------------------------+
|  Framework Desktop "deskwork"  (Fedora 43 Server Edition)            |
|                                                                      |
|  user `ai`  (loginctl linger on, rootless podman 5.x)                |
|  /home/ai/.config/containers/systemd/                                |
|  ├── ai-internal.network   -> network systemd-ai-internal            |
|  │                               (Internal=true, no internet)        |
|  ├── ai-embed.container    -> :8001  EmbeddingGemma-300M             |
|  ├── ai-lite.container     -> :8002  gemma-4-E4B-it-qat (+MTP draft) |
|  └── ai-turbo.container    -> :8003  Qwen3.6-35B-A3B (MTP)           |
|                                                                      |
|  image : localhost/llama-cpp-vulkan:latest   (built from git clone)  |
|  models: /home/ai/models/{text,embedding}/...   (GGUF via hf CLI)    |
|  GPU   : /dev/kfd + /dev/dri passed into every container (Vulkan)    |
+-----------------------------------------------------------------------+
```

| Port | Service  | Model                     | Source quadlet                                   |
| ---- | -------- | ------------------------- | ------------------------------------------------ |
| 8001 | ai-embed | EmbeddingGemma-300M       | `embed/quadlets/embeddinggemma-embed.container`  |
| 8002 | ai-lite  | gemma-4-E4B-it-qat (+MTP) | `lite/quadlets/gemma4-e4b-qat-lite.container`    |
| 8003 | ai-turbo | Qwen3.6-35B-A3B (MTP)     | `turbo/quadlets/qwen3.6-35b-a3b-turbo.container` |

---

## 1. Hardware

Framework Desktop with an AMD Ryzen AI Max 300 (Strix Halo) — the iGPU is a Vulkan device with access to the unified memory. 128 GB unified memory recommended.

### BIOS

<https://knowledgebase.frame.work/en_us/changing-memory-allocation-amd-ryzen-ai-max-300-series-By1LG5Yrll>

1. Set GPU memory to 512MB

### Kernel args

Edit `/etc/default/grub` and add the following to `GRUB_CMDLINE_LINUX`:

```conf
amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856
```

Then regenerate grub and reboot:

```bash
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot
```

After boot, verify the GPU is visible:

```bash
ls -l /dev/kfd /dev/dri/renderD128
```

Expected (note the world-writable modes — that's why the `ai` user needs no special groups):

```text
crw-rw-rw-. 1 root render 234, 0 ... /dev/kfd
crw-rw-rw-. 1 root render 226, 128 ... /dev/dri/renderD128
```

## 2. Operating System

Fedora 43 (Server Edition, minimal is fine). Requires cgroup v2 and a recent podman with Quadlet support (`podman-systemd` generator) — Fedora 40+ ships it.

```bash
sudo dnf install -y podman git
podman --version        # 5.x
podman info --format '{{.Host.CgroupsVersion}}'   # must print: v2
```

## 3. The `ai` user

All containers run rootless as a dedicated `ai` user with linger enabled (so the user systemd manager + containers survive logout and start at boot).

On the Framework Desktop:

```bash
sudo useradd -m ai
sudo loginctl enable-linger ai

# Give the ai user SSH access from your workstation
sudo -u ai mkdir -p /home/ai/.ssh
sudo -u ai tee /home/ai/.ssh/authorized_keys > /dev/null < ~/.ssh/id_ed25519.pub
sudo -u ai chmod 700 /home/ai/.ssh
sudo -u ai chmod 600 /home/ai/.ssh/authorized_keys
```

On your workstation's `~/.ssh/config`:

```text
Host deskwork-ai
    HostName deskwork.reeselink.com
    User ai
```

(Use whatever hostname/IP your Framework Desktop has. The inventory and playbooks reference the host as `deskwork-ai`.)

## 4. Models

As the `ai` user on the Framework Desktop.

### Install the Hugging Face CLI

<https://huggingface.co/docs/huggingface_hub/en/guides/cli#getting-started>

```bash
curl -LsSf https://hf.co/cli/install.sh | bash
hf auth login
```

### Create the model dirs

```bash
mkdir -p /home/ai/models/{text,image,video,embedding,tts,stt}
```

### Download the three model sets

```bash
# --- embed (ai-embed, port 8001) ---
mkdir -p /home/ai/models/embedding/emebeddinggemma-300m
cd /home/ai/models/embedding/emebeddinggemma-300m
# NOTE: the "emebeddinggemma" typo is intentional — it matches the quadlet below
hf download --local-dir . ggml-org/EmbeddingGemma-300M-GGUF embeddinggemma-300M-BF16.gguf
```

```bash
# --- lite (ai-lite, port 8002) ---
mkdir -p /home/ai/models/text/gemma-4-e4b-it-qat
cd /home/ai/models/text/gemma-4-e4b-it-qat
# main model + MTP draft (speculative decoding) + vision projector
hf download --local-dir . unsloth/gemma-4-E4B-it-qat-GGUF \
  --include gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
  --include mtp-gemma-4-E4B-it.gguf \
  --include mmproj-BF16.gguf
```

```bash
# --- turbo (ai-turbo, port 8003) ---
mkdir -p /home/ai/models/text/qwen3.6-35b-a3b-mtp
cd /home/ai/models/text/qwen3.6-35b-a3b-mtp
hf download --local-dir . unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
  --include Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
  --include mmproj-F32.gguf
```

The turbo model is ~37 GB; make sure you have the space (`df -h /home`).

## 5. llama.cpp + the Vulkan image

As the `ai` user on the Framework Desktop:

```bash
cd /home/ai
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
```

The multi-stage `.devops/vulkan.Dockerfile` builds the full toolset with the Vulkan backend; the final stage's entrypoint is `/app/llama-server`, so any args after the image name go straight to the server. Podman tags local builds under the `localhost/` namespace, which is why quadlets reference `localhost/llama-cpp-vulkan:latest`.

Verify:

```bash
podman images | grep llama-cpp-vulkan
```

Optional extras (only if you use them):

```bash
# ROCm image (for the AMD dGPU boxes)
podman build -t llama-cpp-rocm:latest -f .devops/rocm.Dockerfile .

# Diffusion variant (diffusion-gemma turbo quadlet) — built from a fork that
# has diffusion support wired in
git clone https://github.com/danielhanchen/llama.cpp.git llama.cpp-diffusion
cd llama.cpp-diffusion
podman build -t llama-cpp-diffusion-vulkan:latest -f .devops/vulkan.Dockerfile .
```

## 6. Quadlet primer

Podman Quadlet reads files from `~/.config/containers/systemd/` (rootless search path) and generates systemd user units via a systemd generator. No compose files, no hand-written units.

Naming rules that matter here:

| Quadlet file    | Generated systemd service | Podman resource                                             |
| --------------- | ------------------------- | ----------------------------------------------------------- |
| `foo.container` | `foo.service`             | container `systemd-foo` (unless `ContainerName=` overrides) |
| `foo.network`   | `foo-network.service`     | network `systemd-foo`                                       |

- `Network=ai-internal.network` in a container quadlet means: use the network from the `ai-internal.network` quadlet file **and** auto-add a dependency on `ai-internal-network.service`, so the network exists before the container starts.
- Quadlet units are "transient" (generated), so `systemctl enable` doesn't apply — the generator applies the `[Install]` section itself. That's what `WantedBy=multi-user.target default.target` in every file below is for: start at boot.
- Image pulls can exceed systemd's default 90s start timeout, hence `TimeoutStartSec=900`.
- Debugging: `systemd-analyze --user verify <unit>.service` and `/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun`.

Full quadlet reference: <https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html>

## 7. The files

Everything below lives in the `Deployments` repo on the workstation. Layout:

```text
Deployments/
├── inventory.yaml
└── reeseapps/ai/
    ├── roles/containers-infra/
    │   ├── tasks/main.yaml
    │   └── templates/ai-internal.network
    └── deployments/
        ├── embed/
        │   ├── playbook.yaml
        │   └── quadlets/embeddinggemma-embed.container
        ├── lite/
        │   ├── playbook.yaml
        │   └── quadlets/
        │       ├── gemma4-e2b-qat-lite.container
        │       ├── gemma4-e4b-qat-lite.container   <- active
        │       └── lfm2.5-2.6b.container
        └── turbo/
            ├── playbook.yaml
            └── quadlets/
                ├── diffusion-gemma.container
                ├── gemma-4-26b-a4b-turbo.container
                ├── gemma-4-qat-turbo.container
                ├── muse-glimmer-30b-juggernaut-turbo.container
                └── qwen3.6-35b-a3b-turbo.container <- active
```

### 7.1 inventory.yaml

Only the `ai` group is needed for this stack:

```yaml
ai:
  hosts:
    deskwork-ai:
      ansible_python_interpreter: /usr/bin/python3
```

### 7.2 The `containers-infra` role

Runs first in every playbook. Creates the quadlet dir and installs the air-gapped network.

`reeseapps/ai/roles/containers-infra/tasks/main.yaml`:

```yaml
- name: Create /home/ai/.config/containers/systemd
  ansible.builtin.file:
    path: /home/ai/.config/containers/systemd
    state: directory
    mode: "0755"

- name: Copy infra quadlets
  template:
    src: "{{ item }}"
    dest: "/home/ai/.config/containers/systemd/{{ item }}"
  loop:
    - ai-internal.network
```

`reeseapps/ai/roles/containers-infra/templates/ai-internal.network`:

```ini
[Network]
Internal=true
```

`Internal=true` gives the network NAT-style local connectivity but no internet egress — the model servers don't need it.

### 7.3 The playbooks

All three playbooks have the same shape: run the infra role, copy the selected quadlet (chosen by the `container_file` var) to a fixed destination name, then daemon-reload and restart the resulting user service.

`reeseapps/ai/deployments/embed/playbook.yaml`:

```yaml
- name: Create Embedding AI Stack
  hosts: deskwork-ai
  vars:
    container_file: quadlets/embeddinggemma-embed.container
  roles:
    - ../../roles/containers-infra
  tasks:
    - name: Copy ai-embed.container
      copy:
        src: "{{ container_file }}"
        dest: /home/ai/.config/containers/systemd/ai-embed.container
        owner: ai
        group: ai
        mode: "0644"
    - name: Reload and start the ai-embed service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-embed
        daemon_reload: true
        scope: user
```

`reeseapps/ai/deployments/lite/playbook.yaml`:

```yaml
- name: Create Lite Service
  hosts: deskwork-ai
  vars:
    container_file: quadlets/gemma4-e4b-qat-lite.container
  roles:
    - ../../roles/containers-infra
  tasks:
    - name: Copy ai-lite.container
      copy:
        src: "{{ container_file }}"
        dest: /home/ai/.config/containers/systemd/ai-lite.container
        owner: ai
        group: ai
        mode: "0644"
    - name: Reload and start the ai-lite service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-lite
        daemon_reload: true
        scope: user
```

`reeseapps/ai/deployments/turbo/playbook.yaml`:

```yaml
- name: Create Turbo AI Stack
  hosts: deskwork-ai
  vars:
    container_file: quadlets/qwen3.6-35b-a3b-turbo.container
  roles:
    - ../../roles/containers-infra
  tasks:
    - name: Copy ai-turbo.container
      copy:
        src: "{{ container_file }}"
        dest: /home/ai/.config/containers/systemd/ai-turbo.container
        owner: ai
        group: ai
        mode: "0644"
    - name: Reload and start the ai-turbo service
      ansible.builtin.systemd_service:
        state: restarted
        name: ai-turbo
        daemon_reload: true
        scope: user
```

To swap which model a service runs, change only the `container_file` var and re-run that playbook.

### 7.4 The quadlets (currently active variants)

`embed/quadlets/embeddinggemma-embed.container` → installed as `ai-embed.container`:

```ini
[Unit]
Description=A Llama CPP Server For Embedding Models

[Container]
ContainerName=ai-embed

# Internet-disconnected network
Network=ai-internal.network

# llama.cpp juggernaut
PublishPort=8001:8001/tcp

# Image is built locally via podman build
Image=localhost/llama-cpp-vulkan:latest
AutoUpdate=registry

# Downloaded models volume
Volume=/home/ai/models/embedding:/models:z

# GPU Device
AddDevice=/dev/kfd
AddDevice=/dev/dri

# Server command
Exec=--port 8001 \
    -c 0 \
    -b 1024 \
    -ub 1024 \
    --perf \
    --n-gpu-layers all \
    --embedding \
    -m /models/emebeddinggemma-300m/embeddinggemma-300M-BF16.gguf \
    --alias embed

# Health Check
HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8001/health || exit 1
HealthInterval=10s
HealthRetries=3
HealthStartPeriod=10s
HealthTimeout=30s
HealthOnFailure=kill

# EnvironmentFile=/home/ai/.llama-api/keys.env

[Service]
Restart=always
# Extend Timeout to allow time to pull the image
TimeoutStartSec=900

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target
```

`lite/quadlets/gemma4-e4b-qat-lite.container` → installed as `ai-lite.container`:

```ini
[Unit]
Description=A Llama CPP Server Running a Non-Reasoning Model

[Container]
ContainerName=ai-lite

# Internet-disconnected network
Network=ai-internal.network

# llama.cpp juggernaut
PublishPort=8002:8002/tcp
# Image is built locally via podman build
Image=localhost/llama-cpp-vulkan:latest
AutoUpdate=registry

# Downloaded models volume
Volume=/home/ai/models/text:/models:z

# GPU Device
AddDevice=/dev/kfd
AddDevice=/dev/dri

# Server command
Exec=--port 8002 \
    --parallel 1 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64 \
    -fa on \
    --no-mmap \
    --kv-unified \
    --perf \
    --spec-type draft-mtp --spec-draft-n-max 3 \
    --reasoning off \
    --model /models/gemma-4-e4b-it-qat/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
    --model-draft /models/gemma-4-e4b-it-qat/mtp-gemma-4-E4B-it.gguf \
    --mmproj /models/gemma-4-e4b-it-qat/mmproj-BF16.gguf \
    --alias lite

# Health Check
HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8002/health || exit 1
HealthInterval=10s
HealthRetries=3
HealthStartPeriod=10s
HealthTimeout=30s
HealthOnFailure=kill

# EnvironmentFile=/home/ai/.llama-api/keys.env

[Service]
Restart=always
# Extend Timeout to allow time to pull the image
TimeoutStartSec=900

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target
```

`turbo/quadlets/qwen3.6-35b-a3b-turbo.container` → installed as `ai-turbo.container`:

```ini
[Unit]
Description=A Llama CPP Server Running a Reasoning Model

[Container]
ContainerName=ai-turbo

# Internet-disconnected network
Network=ai-internal.network

# llama.cpp juggernaut
PublishPort=8003:8003/tcp

# Image is built locally via podman build
# latest-mtp is for mtp testing
# latest is main branch
Image=localhost/llama-cpp-vulkan:latest
AutoUpdate=registry

# Downloaded models volume
Volume=/home/ai/models/text:/models:z

# GPU Device
AddDevice=/dev/kfd
AddDevice=/dev/dri

# Server command
Exec=--port 8003 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --presence-penalty 1.5 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}' \
    -ctk q8_0 \
    -ctv q8_0 \
    --kv-unified \
    --parallel 2 \
    -fa on \
    --load-mode none \
    --image-min-tokens 4096 \
    --image-max-tokens 4096 \
    --n-gpu-layers all \
    --perf \
    --jinja \
    --reasoning-preserve \
    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
    --mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \
    --spec-type draft-mtp --spec-draft-n-max 3 \
    --spec-draft-type-k q8_0 \
    --spec-draft-type-v q8_0 \
    --alias turbo

# Health Check
# CMD-SHELL is string form, CMD is array form
HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8003/health || exit 1
HealthInterval=10s
HealthRetries=3
HealthStartPeriod=30s
HealthTimeout=30s
HealthOnFailure=kill

[Service]
Restart=always
# Extend Timeout to allow time to pull the image
TimeoutStartSec=900

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target
```

Notes on the server flags:

- `--alias` sets the model name served on `/v1/models`.
- `--spec-type draft-mtp` + `--model-draft` (or the MTP head inside the main GGUF for Qwen) enables MTP speculative decoding for a big speedup.
- `-fa on` = flash attention; `--n-gpu-layers all` = offload everything to the GPU; `--perf` prints timing stats to the log.
- The healthcheck hits the server's `/health` endpoint; `HealthOnFailure=kill` + `Restart=always` means a wedged server is killed and restarted by systemd.
- `:z` on the volume = SELinux shared-label for rootless containers.

## 8. Ansible setup (workstation)

From zero on the workstation:

```bash
# Install ansible (my setup uses pipx; uv tool install ansible works too)
pipx install ansible
ansible-playbook --version
```

No Ansible Galaxy collections are required for this stack — the playbooks only use `ansible.builtin` modules and the local `containers-infra` role (referenced by relative path `../../roles/containers-infra`, which is why the directory layout in §7.1 must be preserved).

Put the files from §7 into a `Deployments/` directory on the workstation (or clone the full repo if you have access). Then, from the repo root:

```bash
cd Deployments

# Order matters only for the first run: embed creates the shared network.
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/embed/playbook.yaml
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/turbo/playbook.yaml
```

Each run is idempotent and safe to re-run. The first turbo start will take a while (37 GB model load) — that's what `TimeoutStartSec=900` is for.

## 9. Verification

On the Framework Desktop (as `ai`):

```bash
systemctl --user status ai-embed ai-lite ai-turbo
podman ps
podman network ls | grep systemd-ai-internal
journalctl --user -fu ai-turbo   # watch startup, look for the --perf stats
```

Expected `podman ps`:

```text
ai-embed   localhost/llama-cpp-vulkan:latest   0.0.0.0:8001->8001/tcp
ai-lite    localhost/llama-cpp-vulkan:latest   0.0.0.0:8002->8002/tcp
ai-turbo   localhost/llama-cpp-vulkan:latest   0.0.0.0:8003->8003/tcp
```

From the host (OpenAI-compatible API):

```bash
# Health
curl http://localhost:8001/health
curl http://localhost:8002/health
curl http://localhost:8003/health

# List models
curl -s http://localhost:8003/v1/models | jq

# Chat completion (turbo)
curl -s http://localhost:8003/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "turbo",
    "messages": [
      {"role": "user", "content": "Say hello in five words."}
    ],
    "max_tokens": 100
  }' | jq

# Chat completion (lite)
curl -s http://localhost:8002/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lite",
    "messages": [
      {"role": "user", "content": "Say hello in five words."}
    ],
    "max_tokens": 100
  }' | jq

# Embeddings (embed)
curl -s http://localhost:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "embed",
    "input": "This is the reason you ended up here."
  }' | jq '.data[0].embedding[0:5]'
```

## 10. Day-2 operations

### Swap the model a service runs

Change `container_file` in that service's playbook and re-run it:

```bash
# e.g. run the smaller 2B gemma on ai-lite instead
$EDITOR reeseapps/ai/deployments/lite/playbook.yaml
#    container_file: quadlets/gemma4-e2b-qat-lite.container
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml
```

Available swap-in quadlets:

- lite: `gemma4-e2b-qat-lite.container`, `gemma4-e4b-qat-lite.container` (active), `lfm2.5-2.6b.container`
- turbo: `qwen3.6-35b-a3b-turbo.container` (active), `gemma-4-26b-a4b-turbo.container`, `gemma-4-qat-turbo.container`, `muse-glimmer-30b-juggernaut-turbo.container`, `diffusion-gemma.container` (uses `localhost/llama-cpp-diffusion-vulkan:latest`)

Make sure the model files the new quadlet references are already downloaded (see the `--model` / `--mmproj` / `--model-draft` paths inside it).

### Update llama.cpp (image rebuild)

On the Framework Desktop as `ai`:

```bash
cd /home/ai/llama.cpp
git pull
podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .

# Restart the servers so they pick up the new image
systemctl --user restart ai-embed ai-lite ai-turbo
```

Tagged snapshots are a good habit:

```bash
export BUILD_TAG=$(date +"%Y-%m-%d-%H-%M-%S")
podman build -t llama-cpp-vulkan:${BUILD_TAG} -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile .
```

Note: `AutoUpdate=registry` in the quadlets is effectively a no-op for `localhost/` images (podman auto-update needs a remote registry to compare against) — updates are manual, as above.

### Logs and troubleshooting

```bash
journalctl --user -fu ai-turbo                  # follow logs
systemctl --user status ai-embed ai-lite ai-turbo
podman exec ai-turbo curl -s http://127.0.0.1:8003/health
systemd-analyze --user verify ai-turbo.service  # validate generated unit
/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun
```

Common failures:

- `Unit ai-turbo.service not found` → the generator failed on a bad quadlet option (often a syntax error or an option your podman version doesn't know). Run the `podman-system-generator --dryrun` command above to see the error.
- GPU errors at startup → confirm `/dev/kfd` and `/dev/dri/renderD128` exist and the user can read/write them; on a box where they're `0660`, add the user: `sudo usermod -aG render ai` then re-login.
- Service times out on first start → normal while loading a big model; `TimeoutStartSec=900` covers ~15 min.
- SELinux `avc` denials on model reads → make sure volume mounts use the `:z` suffix.

### Manual escape hatch

Skip systemd entirely and run a server by hand (useful for testing a new model):

```bash
podman run -it --rm \
--device=/dev/kfd \
--device=/dev/dri \
-v /home/ai/models/text:/models:z \
--entrypoint /bin/bash \
localhost/llama-cpp-vulkan:latest

./llama-server \
-m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \
--mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \
--port 8003 \
-ctk q8_0 -ctv q8_0 \
--kv-unified \
--parallel 2 \
-fa on \
--load-mode none \
--n-gpu-layers all \
--perf \
--jinja \
--reasoning-preserve \
--spec-type draft-mtp --spec-draft-n-max 3 \
--spec-draft-type-k q8_0 \
--spec-draft-type-v q8_0
```

## References

- Podman Quadlet: <https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html>
- llama.cpp server docs: <https://github.com/ggml-org/llama.cpp/tree/master/tools/server>
- Framework Desktop hardware notes: `active/device_framework_desktop/framework_desktop.md`
- Broader AI stack notes (model lists, benchmarks, vLLM, stable-diffusion.cpp): `active/software_ai_stack/ai_stack.md`