brain-of-reese-docs/docs/what-are-the-correct-llama-cpp-arguments-for-qwen-3-8.md

Based on the available deployment configurations for Qwen 3.8 in your repository, here are the correct llama.cpp arguments used for Qwen 3.8 deployments (specifically the juggernaut variant):

Qwen 3.8 llama.cpp Arguments

The key Qwen 3.8-specific arguments (consistent across both qwen3.8-flash-next and qwen3.8-27b-epic-vulkan deployments) are:

Argument	Value	Purpose
--ctk q8_0	8-bit quantization	Uses QLoRA-style 8-bit weight quantization for memory efficiency
--ctv q8_0	Context tuning	Improves reasoning quality with context-aware tuning
--n-gpu-layers all	All GPU layers	Utilizes every available GPU layer for inference (critical for performance)

Full Example (from Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-flash-next.container)

Exec=--port 8000 \
    -c 262144 \
    -n 32768 \
    --reasoning-budget 4096 \
    -ctk q8_0 \
    -ctv q8_0 \
    --kv-unified \
    --parallel 2 \
    -fa on \
    --load-mode none \
    --image-min-tokens 1024 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0 \
    -m /models/qwen3.8-27b-a3b-mtp/Qwen3.8-35B-A3B-UD-Q6_K.gguf \
    --mmproj /models/qwen3.8-35b-a3b-mtp/mmproj-F16.gguf \
    --chat-template-file /models/qwen3.8-35b-a3b-mtp/chat_template.jinja \
    -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q6_K.gguf \
    --spec-type draft-mtp --spec-draft-n-max 2 \
    --alias juggernaut

Key Notes

• Image: localhost/llama-cpp-rocm:latest (for RoCM) or localhost/llama-cpp-vulkan:latest (for Vulkan)
• Model path: Models are mounted at /home/ai/models/text:/models:z
• GPU: The container adds device capabilities (CAP_IPC_OWNER, ADD_DEVICE=/dev/kfd, ADD_DEVICE=/dev/dri) for GPU access
• Inference settings: --n-gpu-layers all is the most distinctive Qwen 3.8 flag — it ensures every layer on the GPU participates in inference, which is critical for this model size

These arguments are drawn from the actual deployment configurations in:

• Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-flash-next.container
• Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-epic-vulkan.container

Both use the same core QLoRA quantization (--ctk q8_0) and context tuning (--ctv q8_0), with --n-gpu-layers all being the key differentiator for Qwen 3.8's larger model sizes.