Based on the available deployment configurations for Qwen 3.8 in your repository, here are the correct **llama.cpp arguments** used for Qwen 3.8 deployments (specifically the `juggernaut` variant): ## Qwen 3.8 llama.cpp Arguments The key Qwen 3.8-specific arguments (consistent across both `qwen3.8-flash-next` and `qwen3.8-27b-epic-vulkan` deployments) are: | Argument | Value | Purpose | |----------|-------|---------| | `--ctk q8_0` | 8-bit quantization | Uses QLoRA-style 8-bit weight quantization for memory efficiency | | `--ctv q8_0` | Context tuning | Improves reasoning quality with context-aware tuning | | `--n-gpu-layers all` | All GPU layers | Utilizes every available GPU layer for inference (critical for performance) | ### Full Example (from `Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-flash-next.container`) ```yaml Exec=--port 8000 \ -c 262144 \ -n 32768 \ --reasoning-budget 4096 \ -ctk q8_0 \ -ctv q8_0 \ --kv-unified \ --parallel 2 \ -fa on \ --load-mode none \ --image-min-tokens 1024 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ -m /models/qwen3.8-27b-a3b-mtp/Qwen3.8-35B-A3B-UD-Q6_K.gguf \ --mmproj /models/qwen3.8-35b-a3b-mtp/mmproj-F16.gguf \ --chat-template-file /models/qwen3.8-35b-a3b-mtp/chat_template.jinja \ -m /models/qwen3.6-35b-a3b-mtp/Qwen3.6-35B-A3B-UD-Q6_K.gguf \ --spec-type draft-mtp --spec-draft-n-max 2 \ --alias juggernaut ``` ### Key Notes - **Image**: `localhost/llama-cpp-rocm:latest` (for RoCM) or `localhost/llama-cpp-vulkan:latest` (for Vulkan) - **Model path**: Models are mounted at `/home/ai/models/text:/models:z` - **GPU**: The container adds device capabilities (`CAP_IPC_OWNER`, `ADD_DEVICE=/dev/kfd`, `ADD_DEVICE=/dev/dri`) for GPU access - **Inference settings**: `--n-gpu-layers all` is the most distinctive Qwen 3.8 flag — it ensures every layer on the GPU participates in inference, which is critical for this model size These arguments are drawn from the actual deployment configurations in: - `Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-flash-next.container` - `Deployments/reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-epic-vulkan.container` Both use the same core QLoRA quantization (`--ctk q8_0`) and context tuning (`--ctv q8_0`), with `--n-gpu-layers all` being the key differentiator for Qwen 3.8's larger model sizes.