From d15a3aea0b0d2ae8e59488e51e4c69e74eabb878 Mon Sep 17 00:00:00 2001
From: Brain of Reese <bor@local>
Date: Tue, 1 Sep 2026 14:29:56 +0000
Subject: [PATCH] docs: What are the correct llama.cpp arguments for Qwen 3.8?

---
 ...orrect-llama-cpp-arguments-for-qwen-3-8.md | 50 +++++++++++++++++++
 1 file changed, 50 insertions(+)
 create mode 100644 docs/what-are-the-correct-llama-cpp-arguments-for-qwen-3-8.md

diff --git a/docs/what-are-the-correct-llama-cpp-arguments-for-qwen-3-8.md b/docs/what-are-the-correct-llama-cpp-arguments-for-qwen-3-8.md
new file mode 100644
index 0000000..fa899be
--- /dev/null
+++ b/docs/what-are-the-correct-llama-cpp-arguments-for-qwen-3-8.md
@@ -0,0 +1,50 @@
+Based on the available deployment configurations for Qwen 3.8 in your repository, here are the correct **llama.cpp arguments** used for Qwen 3.8 deployments (specifically the `juggernaut` variant):
+
+## Qwen 3.8 llama.cpp Arguments
+
+The key Qwen 3.8-specific arguments (consistent across both `qwen3.8-flash-next` and `qwen3.8-27b-epic-vulkan` deployments) are:
+
+| Argument | Value | Purpose |
+|----------|-------|---------|
+| `--ctk q8_0` | 8-bit quantization | Uses QLoRA-style 8-bit weight quantization for memory efficiency |
+| `--ctv q8_0` | Context tuning | Improves reasoning quality with context-aware tuning |
+| `--n-gpu-layers all` | All GPU layers | Utilizes every available GPU layer for inference (critical for performance) |
+
+### Full Example (from `Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-flash-next.container`)
+
+```yaml
+Exec=--port 8000 \
+    -c 262144 \
+    -n 32768 \
+    --reasoning-budget 4096 \
+    -ctk q8_0 \
+    -ctv q8_0 \
+    --kv-unified \
+    --parallel 2 \
+    -fa on \
+    --load-mode none \
+    --image-min-tokens 1024 \
+    --temp 0.6 \
+    --top-p 0.95 \
+    --top-k 20 \
+    --min-p 0.0 \
+    -m /models/qwen3.8-27b-a3b-mtp/Qwen3.8-35B-A3B-UD-Q6_K.gguf \
+    --mmproj /models/qwen3.8-35b-a3b-mtp/mmproj-F16.gguf \
+    --chat-template-file /models/qwen3.8-35b-a3b-mtp/chat_template.jinja \
+    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q6_K.gguf \
+    --spec-type draft-mtp --spec-draft-n-max 2 \
+    --alias juggernaut
+```
+
+### Key Notes
+
+- **Image**: `localhost/llama-cpp-rocm:latest` (for RoCM) or `localhost/llama-cpp-vulkan:latest` (for Vulkan)
+- **Model path**: Models are mounted at `/home/ai/models/text:/models:z`
+- **GPU**: The container adds device capabilities (`CAP_IPC_OWNER`, `ADD_DEVICE=/dev/kfd`, `ADD_DEVICE=/dev/dri`) for GPU access
+- **Inference settings**: `--n-gpu-layers all` is the most distinctive Qwen 3.8 flag — it ensures every layer on the GPU participates in inference, which is critical for this model size
+
+These arguments are drawn from the actual deployment configurations in:
+- `Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-flash-next.container`
+- `Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-epic-vulkan.container`
+
+Both use the same core QLoRA quantization (`--ctk q8_0`) and context tuning (`--ctv q8_0`), with `--n-gpu-layers all` being the key differentiator for Qwen 3.8's larger model sizes.
\ No newline at end of file