diff --git a/active/software_ai_stack/ai_framework_desktop.md b/active/software_ai_stack/ai_framework_desktop.md new file mode 100644 index 0000000..0ddbbd5 --- /dev/null +++ b/active/software_ai_stack/ai_framework_desktop.md @@ -0,0 +1,754 @@ +# Framework Desktop AI Deployment Guide + +Replicating my local AI inference stack: three [llama.cpp](https://github.com/ggml-org/llama.cpp) OpenAI-compatible model servers on a Framework Desktop (AMD Ryzen AI Max 300, Strix Halo iGPU). + +Key design decisions: + +- **Rootless Podman Quadlet** — no docker-compose, no system services. Everything runs under a dedicated `ai` user via systemd user units. `~/.config/containers/systemd/` is the single source of truth. +- **Locally built image** — `llama.cpp` is cloned to the box and its official Vulkan Dockerfile is built with podman. The image entrypoint is `/app/llama-server`, so quadlet `Exec=` args are passed straight to the server. +- **Air-gapped model network** — all model servers sit on an `Internal=true` podman network with no internet egress. Ports are published to the host for API access. +- **Ansible-driven** — playbooks on the workstation copy the right quadlet into place over SSH and restart the user-scoped systemd service. Swapping models is a one-line playbook var change. + +``` + workstation (laptop) + ansible-playbook over SSH as user `ai` + | + v ++-----------------------------------------------------------------------+ +| Framework Desktop "deskwork" (Fedora 43 Server Edition) | +| | +| user `ai` (loginctl linger on, rootless podman 5.x) | +| /home/ai/.config/containers/systemd/ | +| ├── ai-internal.network -> network systemd-ai-internal | +| │ (Internal=true, no internet) | +| ├── ai-embed.container -> :8001 EmbeddingGemma-300M | +| ├── ai-lite.container -> :8002 gemma-4-E4B-it-qat (+MTP draft) | +| └── ai-turbo.container -> :8003 Qwen3.6-35B-A3B (MTP) | +| | +| image : localhost/llama-cpp-vulkan:latest (built from git clone) | +| models: /home/ai/models/{text,embedding}/... (GGUF via hf CLI) | +| GPU : /dev/kfd + /dev/dri passed into every container (Vulkan) | ++-----------------------------------------------------------------------+ +``` + +| Port | Service | Model | Source quadlet | +| ---- | -------- | ------------------------- | ------------------------------------------------ | +| 8001 | ai-embed | EmbeddingGemma-300M | `embed/quadlets/embeddinggemma-embed.container` | +| 8002 | ai-lite | gemma-4-E4B-it-qat (+MTP) | `lite/quadlets/gemma4-e4b-qat-lite.container` | +| 8003 | ai-turbo | Qwen3.6-35B-A3B (MTP) | `turbo/quadlets/qwen3.6-35b-a3b-turbo.container` | + +--- + +## 1. Hardware + +Framework Desktop with an AMD Ryzen AI Max 300 (Strix Halo) — the iGPU is a Vulkan device with access to the unified memory. 128 GB unified memory recommended. + +### BIOS + + + +1. Set GPU memory to 512MB + +### Kernel args + +Edit `/etc/default/grub` and add the following to `GRUB_CMDLINE_LINUX`: + +```conf +amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 +``` + +Then regenerate grub and reboot: + +```bash +sudo grub2-mkconfig -o /boot/grub2/grub.cfg +sudo reboot +``` + +After boot, verify the GPU is visible: + +```bash +ls -l /dev/kfd /dev/dri/renderD128 +``` + +Expected (note the world-writable modes — that's why the `ai` user needs no special groups): + +```text +crw-rw-rw-. 1 root render 234, 0 ... /dev/kfd +crw-rw-rw-. 1 root render 226, 128 ... /dev/dri/renderD128 +``` + +## 2. Operating System + +Fedora 43 (Server Edition, minimal is fine). Requires cgroup v2 and a recent podman with Quadlet support (`podman-systemd` generator) — Fedora 40+ ships it. + +```bash +sudo dnf install -y podman git +podman --version # 5.x +podman info --format '{{.Host.CgroupsVersion}}' # must print: v2 +``` + +## 3. The `ai` user + +All containers run rootless as a dedicated `ai` user with linger enabled (so the user systemd manager + containers survive logout and start at boot). + +On the Framework Desktop: + +```bash +sudo useradd -m ai +sudo loginctl enable-linger ai + +# Give the ai user SSH access from your workstation +sudo -u ai mkdir -p /home/ai/.ssh +sudo -u ai tee /home/ai/.ssh/authorized_keys > /dev/null < ~/.ssh/id_ed25519.pub +sudo -u ai chmod 700 /home/ai/.ssh +sudo -u ai chmod 600 /home/ai/.ssh/authorized_keys +``` + +On your workstation's `~/.ssh/config`: + +```text +Host deskwork-ai + HostName deskwork.reeselink.com + User ai +``` + +(Use whatever hostname/IP your Framework Desktop has. The inventory and playbooks reference the host as `deskwork-ai`.) + +## 4. Models + +As the `ai` user on the Framework Desktop. + +### Install the Hugging Face CLI + + + +```bash +curl -LsSf https://hf.co/cli/install.sh | bash +hf auth login +``` + +### Create the model dirs + +```bash +mkdir -p /home/ai/models/{text,image,video,embedding,tts,stt} +``` + +### Download the three model sets + +```bash +# --- embed (ai-embed, port 8001) --- +mkdir -p /home/ai/models/embedding/emebeddinggemma-300m +cd /home/ai/models/embedding/emebeddinggemma-300m +# NOTE: the "emebeddinggemma" typo is intentional — it matches the quadlet below +hf download --local-dir . ggml-org/EmbeddingGemma-300M-GGUF embeddinggemma-300M-BF16.gguf +``` + +```bash +# --- lite (ai-lite, port 8002) --- +mkdir -p /home/ai/models/text/gemma-4-e4b-it-qat +cd /home/ai/models/text/gemma-4-e4b-it-qat +# main model + MTP draft (speculative decoding) + vision projector +hf download --local-dir . unsloth/gemma-4-E4B-it-qat-GGUF \ + --include gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \ + --include mtp-gemma-4-E4B-it.gguf \ + --include mmproj-BF16.gguf +``` + +```bash +# --- turbo (ai-turbo, port 8003) --- +mkdir -p /home/ai/models/text/qwen3.6-35b-a3b-mtp +cd /home/ai/models/text/qwen3.6-35b-a3b-mtp +hf download --local-dir . unsloth/Qwen3.6-35B-A3B-MTP-GGUF \ + --include Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \ + --include mmproj-F32.gguf +``` + +The turbo model is ~37 GB; make sure you have the space (`df -h /home`). + +## 5. llama.cpp + the Vulkan image + +As the `ai` user on the Framework Desktop: + +```bash +cd /home/ai +git clone https://github.com/ggml-org/llama.cpp.git +cd llama.cpp +podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile . +``` + +The multi-stage `.devops/vulkan.Dockerfile` builds the full toolset with the Vulkan backend; the final stage's entrypoint is `/app/llama-server`, so any args after the image name go straight to the server. Podman tags local builds under the `localhost/` namespace, which is why quadlets reference `localhost/llama-cpp-vulkan:latest`. + +Verify: + +```bash +podman images | grep llama-cpp-vulkan +``` + +Optional extras (only if you use them): + +```bash +# ROCm image (for the AMD dGPU boxes) +podman build -t llama-cpp-rocm:latest -f .devops/rocm.Dockerfile . + +# Diffusion variant (diffusion-gemma turbo quadlet) — built from a fork that +# has diffusion support wired in +git clone https://github.com/danielhanchen/llama.cpp.git llama.cpp-diffusion +cd llama.cpp-diffusion +podman build -t llama-cpp-diffusion-vulkan:latest -f .devops/vulkan.Dockerfile . +``` + +## 6. Quadlet primer + +Podman Quadlet reads files from `~/.config/containers/systemd/` (rootless search path) and generates systemd user units via a systemd generator. No compose files, no hand-written units. + +Naming rules that matter here: + +| Quadlet file | Generated systemd service | Podman resource | +| --------------- | ------------------------- | ----------------------------------------------------------- | +| `foo.container` | `foo.service` | container `systemd-foo` (unless `ContainerName=` overrides) | +| `foo.network` | `foo-network.service` | network `systemd-foo` | + +- `Network=ai-internal.network` in a container quadlet means: use the network from the `ai-internal.network` quadlet file **and** auto-add a dependency on `ai-internal-network.service`, so the network exists before the container starts. +- Quadlet units are "transient" (generated), so `systemctl enable` doesn't apply — the generator applies the `[Install]` section itself. That's what `WantedBy=multi-user.target default.target` in every file below is for: start at boot. +- Image pulls can exceed systemd's default 90s start timeout, hence `TimeoutStartSec=900`. +- Debugging: `systemd-analyze --user verify .service` and `/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun`. + +Full quadlet reference: + +## 7. The files + +Everything below lives in the `Deployments` repo on the workstation. Layout: + +```text +Deployments/ +├── inventory.yaml +└── reeseapps/ai/ + ├── roles/containers-infra/ + │ ├── tasks/main.yaml + │ └── templates/ai-internal.network + └── deployments/ + ├── embed/ + │ ├── playbook.yaml + │ └── quadlets/embeddinggemma-embed.container + ├── lite/ + │ ├── playbook.yaml + │ └── quadlets/ + │ ├── gemma4-e2b-qat-lite.container + │ ├── gemma4-e4b-qat-lite.container <- active + │ └── lfm2.5-2.6b.container + └── turbo/ + ├── playbook.yaml + └── quadlets/ + ├── diffusion-gemma.container + ├── gemma-4-26b-a4b-turbo.container + ├── gemma-4-qat-turbo.container + ├── muse-glimmer-30b-juggernaut-turbo.container + └── qwen3.6-35b-a3b-turbo.container <- active +``` + +### 7.1 inventory.yaml + +Only the `ai` group is needed for this stack: + +```yaml +ai: + hosts: + deskwork-ai: + ansible_python_interpreter: /usr/bin/python3 +``` + +### 7.2 The `containers-infra` role + +Runs first in every playbook. Creates the quadlet dir and installs the air-gapped network. + +`reeseapps/ai/roles/containers-infra/tasks/main.yaml`: + +```yaml +- name: Create /home/ai/.config/containers/systemd + ansible.builtin.file: + path: /home/ai/.config/containers/systemd + state: directory + mode: "0755" + +- name: Copy infra quadlets + template: + src: "{{ item }}" + dest: "/home/ai/.config/containers/systemd/{{ item }}" + loop: + - ai-internal.network +``` + +`reeseapps/ai/roles/containers-infra/templates/ai-internal.network`: + +```ini +[Network] +Internal=true +``` + +`Internal=true` gives the network NAT-style local connectivity but no internet egress — the model servers don't need it. + +### 7.3 The playbooks + +All three playbooks have the same shape: run the infra role, copy the selected quadlet (chosen by the `container_file` var) to a fixed destination name, then daemon-reload and restart the resulting user service. + +`reeseapps/ai/deployments/embed/playbook.yaml`: + +```yaml +- name: Create Embedding AI Stack + hosts: deskwork-ai + vars: + container_file: quadlets/embeddinggemma-embed.container + roles: + - ../../roles/containers-infra + tasks: + - name: Copy ai-embed.container + copy: + src: "{{ container_file }}" + dest: /home/ai/.config/containers/systemd/ai-embed.container + owner: ai + group: ai + mode: "0644" + - name: Reload and start the ai-embed service + ansible.builtin.systemd_service: + state: restarted + name: ai-embed + daemon_reload: true + scope: user +``` + +`reeseapps/ai/deployments/lite/playbook.yaml`: + +```yaml +- name: Create Lite Service + hosts: deskwork-ai + vars: + container_file: quadlets/gemma4-e4b-qat-lite.container + roles: + - ../../roles/containers-infra + tasks: + - name: Copy ai-lite.container + copy: + src: "{{ container_file }}" + dest: /home/ai/.config/containers/systemd/ai-lite.container + owner: ai + group: ai + mode: "0644" + - name: Reload and start the ai-lite service + ansible.builtin.systemd_service: + state: restarted + name: ai-lite + daemon_reload: true + scope: user +``` + +`reeseapps/ai/deployments/turbo/playbook.yaml`: + +```yaml +- name: Create Turbo AI Stack + hosts: deskwork-ai + vars: + container_file: quadlets/qwen3.6-35b-a3b-turbo.container + roles: + - ../../roles/containers-infra + tasks: + - name: Copy ai-turbo.container + copy: + src: "{{ container_file }}" + dest: /home/ai/.config/containers/systemd/ai-turbo.container + owner: ai + group: ai + mode: "0644" + - name: Reload and start the ai-turbo service + ansible.builtin.systemd_service: + state: restarted + name: ai-turbo + daemon_reload: true + scope: user +``` + +To swap which model a service runs, change only the `container_file` var and re-run that playbook. + +### 7.4 The quadlets (currently active variants) + +`embed/quadlets/embeddinggemma-embed.container` → installed as `ai-embed.container`: + +```ini +[Unit] +Description=A Llama CPP Server For Embedding Models + +[Container] +ContainerName=ai-embed + +# Internet-disconnected network +Network=ai-internal.network + +# llama.cpp juggernaut +PublishPort=8001:8001/tcp + +# Image is built locally via podman build +Image=localhost/llama-cpp-vulkan:latest +AutoUpdate=registry + +# Downloaded models volume +Volume=/home/ai/models/embedding:/models:z + +# GPU Device +AddDevice=/dev/kfd +AddDevice=/dev/dri + +# Server command +Exec=--port 8001 \ + -c 0 \ + -b 1024 \ + -ub 1024 \ + --perf \ + --n-gpu-layers all \ + --embedding \ + -m /models/emebeddinggemma-300m/embeddinggemma-300M-BF16.gguf \ + --alias embed + +# Health Check +HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8001/health || exit 1 +HealthInterval=10s +HealthRetries=3 +HealthStartPeriod=10s +HealthTimeout=30s +HealthOnFailure=kill + +# EnvironmentFile=/home/ai/.llama-api/keys.env + +[Service] +Restart=always +# Extend Timeout to allow time to pull the image +TimeoutStartSec=900 + +[Install] +# Start by default on boot +WantedBy=multi-user.target default.target +``` + +`lite/quadlets/gemma4-e4b-qat-lite.container` → installed as `ai-lite.container`: + +```ini +[Unit] +Description=A Llama CPP Server Running a Non-Reasoning Model + +[Container] +ContainerName=ai-lite + +# Internet-disconnected network +Network=ai-internal.network + +# llama.cpp juggernaut +PublishPort=8002:8002/tcp +# Image is built locally via podman build +Image=localhost/llama-cpp-vulkan:latest +AutoUpdate=registry + +# Downloaded models volume +Volume=/home/ai/models/text:/models:z + +# GPU Device +AddDevice=/dev/kfd +AddDevice=/dev/dri + +# Server command +Exec=--port 8002 \ + --parallel 1 \ + --temp 1.0 \ + --top-p 0.95 \ + --top-k 64 \ + -fa on \ + --no-mmap \ + --kv-unified \ + --perf \ + --spec-type draft-mtp --spec-draft-n-max 3 \ + --reasoning off \ + --model /models/gemma-4-e4b-it-qat/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \ + --model-draft /models/gemma-4-e4b-it-qat/mtp-gemma-4-E4B-it.gguf \ + --mmproj /models/gemma-4-e4b-it-qat/mmproj-BF16.gguf \ + --alias lite + +# Health Check +HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8002/health || exit 1 +HealthInterval=10s +HealthRetries=3 +HealthStartPeriod=10s +HealthTimeout=30s +HealthOnFailure=kill + +# EnvironmentFile=/home/ai/.llama-api/keys.env + +[Service] +Restart=always +# Extend Timeout to allow time to pull the image +TimeoutStartSec=900 + +[Install] +# Start by default on boot +WantedBy=multi-user.target default.target +``` + +`turbo/quadlets/qwen3.6-35b-a3b-turbo.container` → installed as `ai-turbo.container`: + +```ini +[Unit] +Description=A Llama CPP Server Running a Reasoning Model + +[Container] +ContainerName=ai-turbo + +# Internet-disconnected network +Network=ai-internal.network + +# llama.cpp juggernaut +PublishPort=8003:8003/tcp + +# Image is built locally via podman build +# latest-mtp is for mtp testing +# latest is main branch +Image=localhost/llama-cpp-vulkan:latest +AutoUpdate=registry + +# Downloaded models volume +Volume=/home/ai/models/text:/models:z + +# GPU Device +AddDevice=/dev/kfd +AddDevice=/dev/dri + +# Server command +Exec=--port 8003 \ + --temp 0.7 \ + --top-p 0.8 \ + --top-k 20 \ + --presence-penalty 1.5 \ + --min-p 0.00 \ + --chat-template-kwargs '{"enable_thinking":false}' \ + -ctk q8_0 \ + -ctv q8_0 \ + --kv-unified \ + --parallel 2 \ + -fa on \ + --load-mode none \ + --image-min-tokens 4096 \ + --image-max-tokens 4096 \ + --n-gpu-layers all \ + --perf \ + --jinja \ + --reasoning-preserve \ + -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \ + --mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \ + --spec-type draft-mtp --spec-draft-n-max 3 \ + --spec-draft-type-k q8_0 \ + --spec-draft-type-v q8_0 \ + --alias turbo + +# Health Check +# CMD-SHELL is string form, CMD is array form +HealthCmd=CMD-SHELL curl --fail http://127.0.0.1:8003/health || exit 1 +HealthInterval=10s +HealthRetries=3 +HealthStartPeriod=30s +HealthTimeout=30s +HealthOnFailure=kill + +[Service] +Restart=always +# Extend Timeout to allow time to pull the image +TimeoutStartSec=900 + +[Install] +# Start by default on boot +WantedBy=multi-user.target default.target +``` + +Notes on the server flags: + +- `--alias` sets the model name served on `/v1/models`. +- `--spec-type draft-mtp` + `--model-draft` (or the MTP head inside the main GGUF for Qwen) enables MTP speculative decoding for a big speedup. +- `-fa on` = flash attention; `--n-gpu-layers all` = offload everything to the GPU; `--perf` prints timing stats to the log. +- The healthcheck hits the server's `/health` endpoint; `HealthOnFailure=kill` + `Restart=always` means a wedged server is killed and restarted by systemd. +- `:z` on the volume = SELinux shared-label for rootless containers. + +## 8. Ansible setup (workstation) + +From zero on the workstation: + +```bash +# Install ansible (my setup uses pipx; uv tool install ansible works too) +pipx install ansible +ansible-playbook --version +``` + +No Ansible Galaxy collections are required for this stack — the playbooks only use `ansible.builtin` modules and the local `containers-infra` role (referenced by relative path `../../roles/containers-infra`, which is why the directory layout in §7.1 must be preserved). + +Put the files from §7 into a `Deployments/` directory on the workstation (or clone the full repo if you have access). Then, from the repo root: + +```bash +cd Deployments + +# Order matters only for the first run: embed creates the shared network. +ansible-playbook -i inventory.yaml reeseapps/ai/deployments/embed/playbook.yaml +ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml +ansible-playbook -i inventory.yaml reeseapps/ai/deployments/turbo/playbook.yaml +``` + +Each run is idempotent and safe to re-run. The first turbo start will take a while (37 GB model load) — that's what `TimeoutStartSec=900` is for. + +## 9. Verification + +On the Framework Desktop (as `ai`): + +```bash +systemctl --user status ai-embed ai-lite ai-turbo +podman ps +podman network ls | grep systemd-ai-internal +journalctl --user -fu ai-turbo # watch startup, look for the --perf stats +``` + +Expected `podman ps`: + +```text +ai-embed localhost/llama-cpp-vulkan:latest 0.0.0.0:8001->8001/tcp +ai-lite localhost/llama-cpp-vulkan:latest 0.0.0.0:8002->8002/tcp +ai-turbo localhost/llama-cpp-vulkan:latest 0.0.0.0:8003->8003/tcp +``` + +From the host (OpenAI-compatible API): + +```bash +# Health +curl http://localhost:8001/health +curl http://localhost:8002/health +curl http://localhost:8003/health + +# List models +curl -s http://localhost:8003/v1/models | jq + +# Chat completion (turbo) +curl -s http://localhost:8003/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "turbo", + "messages": [ + {"role": "user", "content": "Say hello in five words."} + ], + "max_tokens": 100 + }' | jq + +# Chat completion (lite) +curl -s http://localhost:8002/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "lite", + "messages": [ + {"role": "user", "content": "Say hello in five words."} + ], + "max_tokens": 100 + }' | jq + +# Embeddings (embed) +curl -s http://localhost:8001/v1/embeddings \ + -H "Content-Type: application/json" \ + -d '{ + "model": "embed", + "input": "This is the reason you ended up here." + }' | jq '.data[0].embedding[0:5]' +``` + +## 10. Day-2 operations + +### Swap the model a service runs + +Change `container_file` in that service's playbook and re-run it: + +```bash +# e.g. run the smaller 2B gemma on ai-lite instead +$EDITOR reeseapps/ai/deployments/lite/playbook.yaml +# container_file: quadlets/gemma4-e2b-qat-lite.container +ansible-playbook -i inventory.yaml reeseapps/ai/deployments/lite/playbook.yaml +``` + +Available swap-in quadlets: + +- lite: `gemma4-e2b-qat-lite.container`, `gemma4-e4b-qat-lite.container` (active), `lfm2.5-2.6b.container` +- turbo: `qwen3.6-35b-a3b-turbo.container` (active), `gemma-4-26b-a4b-turbo.container`, `gemma-4-qat-turbo.container`, `muse-glimmer-30b-juggernaut-turbo.container`, `diffusion-gemma.container` (uses `localhost/llama-cpp-diffusion-vulkan:latest`) + +Make sure the model files the new quadlet references are already downloaded (see the `--model` / `--mmproj` / `--model-draft` paths inside it). + +### Update llama.cpp (image rebuild) + +On the Framework Desktop as `ai`: + +```bash +cd /home/ai/llama.cpp +git pull +podman build -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile . + +# Restart the servers so they pick up the new image +systemctl --user restart ai-embed ai-lite ai-turbo +``` + +Tagged snapshots are a good habit: + +```bash +export BUILD_TAG=$(date +"%Y-%m-%d-%H-%M-%S") +podman build -t llama-cpp-vulkan:${BUILD_TAG} -t llama-cpp-vulkan:latest -f .devops/vulkan.Dockerfile . +``` + +Note: `AutoUpdate=registry` in the quadlets is effectively a no-op for `localhost/` images (podman auto-update needs a remote registry to compare against) — updates are manual, as above. + +### Logs and troubleshooting + +```bash +journalctl --user -fu ai-turbo # follow logs +systemctl --user status ai-embed ai-lite ai-turbo +podman exec ai-turbo curl -s http://127.0.0.1:8003/health +systemd-analyze --user verify ai-turbo.service # validate generated unit +/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun +``` + +Common failures: + +- `Unit ai-turbo.service not found` → the generator failed on a bad quadlet option (often a syntax error or an option your podman version doesn't know). Run the `podman-system-generator --dryrun` command above to see the error. +- GPU errors at startup → confirm `/dev/kfd` and `/dev/dri/renderD128` exist and the user can read/write them; on a box where they're `0660`, add the user: `sudo usermod -aG render ai` then re-login. +- Service times out on first start → normal while loading a big model; `TimeoutStartSec=900` covers ~15 min. +- SELinux `avc` denials on model reads → make sure volume mounts use the `:z` suffix. + +### Manual escape hatch + +Skip systemd entirely and run a server by hand (useful for testing a new model): + +```bash +podman run -it --rm \ +--device=/dev/kfd \ +--device=/dev/dri \ +-v /home/ai/models/text:/models:z \ +--entrypoint /bin/bash \ +localhost/llama-cpp-vulkan:latest + +./llama-server \ +-m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf \ +--mmproj /models/qwen3.6-35b-a3b-mtp/mmproj-F32.gguf \ +--port 8003 \ +-ctk q8_0 -ctv q8_0 \ +--kv-unified \ +--parallel 2 \ +-fa on \ +--load-mode none \ +--n-gpu-layers all \ +--perf \ +--jinja \ +--reasoning-preserve \ +--spec-type draft-mtp --spec-draft-n-max 3 \ +--spec-draft-type-k q8_0 \ +--spec-draft-type-v q8_0 +``` + +## References + +- Podman Quadlet: +- llama.cpp server docs: +- Framework Desktop hardware notes: `active/device_framework_desktop/framework_desktop.md` +- Broader AI stack notes (model lists, benchmarks, vLLM, stable-diffusion.cpp): `active/software_ai_stack/ai_stack.md`