docs: What are the correct llama.cpp arguments for Qwen 3.8?

2026-09-01 14:29:56 +00:00
parent de68207f48
commit d15a3aea0b
1 changed files with 50 additions and 0 deletions
@@ -0,0 +1,50 @@
+Based on the available deployment configurations for Qwen 3.8 in your repository, here are the correct **llama.cpp arguments** used for Qwen 3.8 deployments (specifically the `juggernaut` variant):
+
+## Qwen 3.8 llama.cpp Arguments
+
+The key Qwen 3.8-specific arguments (consistent across both `qwen3.8-flash-next` and `qwen3.8-27b-epic-vulkan` deployments) are:
+
+| Argument | Value | Purpose |
+|----------|-------|---------|
+| `--ctk q8_0` | 8-bit quantization | Uses QLoRA-style 8-bit weight quantization for memory efficiency |
+| `--ctv q8_0` | Context tuning | Improves reasoning quality with context-aware tuning |
+| `--n-gpu-layers all` | All GPU layers | Utilizes every available GPU layer for inference (critical for performance) |
+
+### Full Example (from `Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-flash-next.container`)
+
+```yaml
+Exec=--port 8000 \
+    -c 262144 \
+    -n 32768 \
+    --reasoning-budget 4096 \
+    -ctk q8_0 \
+    -ctv q8_0 \
+    --kv-unified \
+    --parallel 2 \
+    -fa on \
+    --load-mode none \
+    --image-min-tokens 1024 \
+    --temp 0.6 \
+    --top-p 0.95 \
+    --top-k 20 \
+    --min-p 0.0 \
+    -m /models/qwen3.8-27b-a3b-mtp/Qwen3.8-35B-A3B-UD-Q6_K.gguf \
+    --mmproj /models/qwen3.8-35b-a3b-mtp/mmproj-F16.gguf \
+    --chat-template-file /models/qwen3.8-35b-a3b-mtp/chat_template.jinja \
+    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q6_K.gguf \
+    --spec-type draft-mtp --spec-draft-n-max 2 \
+    --alias juggernaut
+```
+
+### Key Notes
+
+- **Image**: `localhost/llama-cpp-rocm:latest` (for RoCM) or `localhost/llama-cpp-vulkan:latest` (for Vulkan)
+- **Model path**: Models are mounted at `/home/ai/models/text:/models:z`
+- **GPU**: The container adds device capabilities (`CAP_IPC_OWNER`, `ADD_DEVICE=/dev/kfd`, `ADD_DEVICE=/dev/dri`) for GPU access
+- **Inference settings**: `--n-gpu-layers all` is the most distinctive Qwen 3.8 flag — it ensures every layer on the GPU participates in inference, which is critical for this model size
+
+These arguments are drawn from the actual deployment configurations in:
+- `Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-flash-next.container`
+- `Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-epic-vulkan.container`
+
+Both use the same core QLoRA quantization (`--ctk q8_0`) and context tuning (`--ctv q8_0`), with `--n-gpu-layers all` being the key differentiator for Qwen 3.8's larger model sizes.