# vLLM Radiance on R9700 - [vLLM Radiance on R9700](#vllm-radiance-on-r9700) - [Overview](#overview) - [Deployment Topology](#deployment-topology) - [Capacity Analysis (Single 32 GiB R9700)](#capacity-analysis-single-32-gib-r9700) - [FP8-KV Calibration: The Process](#fp8-kv-calibration-the-process) - [Why: The Uncalibrated Scale Problem](#why-the-uncalibrated-scale-problem) - [Step 1: Host Prep](#step-1-host-prep) - [Step 2: Stop the Service](#step-2-stop-the-service) - [Step 3: Run the Calibration](#step-3-run-the-calibration) - [Step 4: Verify the Artifact (CPU-only)](#step-4-verify-the-artifact-cpu-only) - [Step 5: Edit the Quadlet](#step-5-edit-the-quadlet) - [Step 6: Deploy via Ansible](#step-6-deploy-via-ansible) - [Step 7: Confirm at Runtime](#step-7-confirm-at-runtime) - [What Calibration Actually Does (Benefits)](#what-calibration-actually-does-benefits) - [Operational Notes](#operational-notes) - [Remaining Work: Matched-Control Quality Gate](#remaining-work-matched-control-quality-gate) ## Overview `vllm-radiance` is a qualified downstream vLLM distribution for the AMD Radeon AI PRO R9700 (gfx1201). The upstream-qualified environment is dual-GPU TP2; this deployment runs the single-GPU (TP1) configuration on `driveripper-ai` serving the AMD Quark MXFP4 27B checkpoint (W4A8: 4-bit weights, dynamic FP8 activation quantization over RDNA4's native FP8 WMMA path). Served as `juggernaut` on the `ai-internal` network, port 8000. ## Deployment Topology | Thing | Location | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Quadlet (source of | | truth) | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/quadlets/vllm-radiance.container` | | Ansible playbook | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/playbook.yaml` (run from `~/Deployments`: `ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml`) | | Deployed quadlet | `driveripper-ai:/home/ai/.config/containers/systemd/ai-vllm-radiance.container` | | systemd unit | **user** unit of `ai` (`systemctl --user status ai-vllm-radiance` — not in system scope) | | Image | `docker.io/magiccodingman/vllm-radiance:1.0.387` | | Model | `driveripper-ai:/home/ai/vllm-radiance-models/qwen3.8-27b-quark-awq-mxfp4` → `/models` (18.44 GiB on disk, 17.95 GiB in GPU) | | Compile caches | `driveripper-ai:/home/ai/vllm-radiance-cache` → `/cache` | | FP8-KV sidecar | `driveripper-ai:/home/ai/vllm-radiance-calibration` → `/calibration` (ro) | The playbook copies the quadlet to `~/.config/containers/systemd/` and does `systemctl --user restart ai-vllm-radiance` with daemon-reload. **Any edit to the local quadlet lands on the next playbook run** — there is no separate "apply" step. Key quadlet settings (as of the calibration deploy, 2026-10-11): ``` --tensor-parallel-size=1 --max-model-len=154000 --max-num-seqs=2 --max-num-batched-tokens=4096 --kv-cache-dtype=fp8 --attention-backend=R4D --enable-prefix-caching --mamba-cache-mode=align --no-async-scheduling --gpu-memory-utilization=0.98 Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1 HealthStartPeriod=600s (health check kills on 3x10s failures after the start period) ``` ## Capacity Analysis (Single 32 GiB R9700) Measured from the server startup logs (the authoritative numbers — do not re-derive from weight sizes alone; the GDN hybrid architecture makes per-token KV much cheaper than a plain transformer): ``` Model loading took 17.95 GiB Actual usage: 19.38 GiB consumed (weights + non-torch) + 2.02 GiB peak activation + 0.41 GiB CUDA graphs Available KV cache memory: 9.82 GiB (10.14 GiB at the 154K config) GPU KV cache size: 308,847 tokens (309,466 at the 154K config) ``` Budget math (KV budget ≈ 309K tokens): | Config | KV needed | Fits? | | --------------------- | --------- | -------------------------------------------- | | 150K × 2 | 300,000 | yes, ~3% headroom | | **154K × 2 (chosen)** | 308,000 | yes — `2.01x` per vLLM, ~1.5K token headroom | | 256K × 1 | 262,144 | yes | | 256K × 2 | 524,288 | **no** — `1.18x` max concurrency | The per-request C2 ceiling is `KV_budget / 2 ≈ 154,423` tokens; `--max-model-len=154000` sits exactly at it. A second full 256K request is only reachable via CPU KV offload (`RADIANCE_KV_OFFLOADING_SIZE`), which would need to roughly double the token budget. `MAX_NUM_SEQS` is an admission ceiling, not a promise that every admitted request can simultaneously reach `MAX_MODEL_LEN`. ## FP8-KV Calibration: The Process ### Why: The Uncalibrated Scale Problem The server runs `--kv-cache-dtype=fp8` with FP8 attention, but the checkpoint carries no q/k/v/prob scales. vLLM therefore used placeholder scale **1.0**, which the startup log flags: ``` WARNING ... Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. ``` Radiance ships `radiance_kv_calibration.py` (in the image, also in the `vllm-radiance` repo) plus the `RADIANCE_FP8_KV_SCALES` runtime loader to fix this with a small immutable sidecar. Full reference: `docs/FP8_KV_TUNABLEOP.md` in the vllm-radiance repo. ### Step 1: Host Prep The calibration dir did not exist and the vllm-radiance repo is not on `driveripper-ai`, so copy only the corpus: ```bash ssh driveripper-ai "mkdir -p /home/ai/vllm-radiance-calibration" scp ~/Projects/Experimental/vllm-radiance/benchmarks/fixtures/fp8-kv-calibration.jsonl \ driveripper-ai:/home/ai/vllm-radiance-calibration/ ``` (Optionally extend the JSONL with representative traffic first — each line is `{"text": ...}` or `{"messages": [...], "tools": [...]}` plus optional `repeat`.) ### Step 2: Stop the Service Calibration needs the GPU released; the server holds ~97% of VRAM. ```bash ssh driveripper-ai "systemctl --user stop ai-vllm-radiance" ``` ### Step 3: Run the Calibration An eager, BF16 full-precision-KV, `max_tokens=1` prefill sweep with PyTorch hooks on every full-attention layer (no source edits — public worker-extension API only). Budget ~10–30 min: ```bash ssh driveripper-ai "podman run --rm --ipc=host \ --device=/dev/kfd --device=/dev/dri \ --ulimit memlock=-1:-1 \ -e HIP_VISIBLE_DEVICES=0 \ -e RADIANCE_MXFP4=1 -e RADIANCE_MXFP4_W4A8=1 -e RADIANCE_MXFP4_W4A8_MIN_M=0 \ -v /home/ai/vllm-radiance-models:/models:ro \ -v /home/ai/vllm-radiance-calibration:/calibration:rw \ -v /home/ai/vllm-radiance-calibration/fp8-kv-calibration.jsonl:/fixtures/fp8-kv-calibration.jsonl:ro \ --entrypoint python \ docker.io/magiccodingman/vllm-radiance:1.0.387 \ -m radiance_kv_calibration calibrate \ --model /models/qwen3.8-27b-quark-awq-mxfp4 \ --quantization auto \ --corpus /fixtures/fp8-kv-calibration.jsonl \ --output /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \ --tensor-parallel-size 1 \ --max-model-len 16384 \ --batch-size 4 \ --seed 17" ``` Success line: `[radiance] wrote immutable FP8-KV sidecar: /calibration/...`. Outputs are **immutable** — a re-run needs a fresh filename. The sidecar and its sibling `.manifest.json` (sha256 + full source-checkpoint identity + per-layer observed amax + corpus/seed/software provenance) land in the host dir. ### Step 4: Verify the Artifact (CPU-only) No engine start — checks manifest schema, sidecar sha256, that every tensor is a positive finite scalar `*.q|k|v|prob_scale`, and (with `--model`) that it was calibrated against this exact checkpoint: ```bash ssh driveripper-ai "podman run --rm \ -v /home/ai/vllm-radiance-models:/models:ro \ -v /home/ai/vllm-radiance-calibration:/calibration:ro \ --entrypoint python docker.io/magiccodingman/vllm-radiance:1.0.387 \ -m radiance_kv_calibration verify \ /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \ --model /models/qwen3.8-27b-quark-awq-mxfp4" ``` ### Step 5: Edit the Quadlet Three lines in the local quadlet (source of truth), then nothing else: ```diff Volume=/home/ai/vllm-radiance-models:/models:ro,z Volume=/home/ai/vllm-radiance-cache:/cache:z +Volume=/home/ai/vllm-radiance-calibration:/calibration:ro,z ... Environment=WEIGHT_QUANTIZATION=auto +Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors +Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1 ``` ### Step 6: Deploy via Ansible ```bash cd ~/Deployments ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml ``` ### Step 7: Confirm at Runtime All three must be present/absent: ``` FP8-KV scales = /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors (banner; was "checkpoint/default") Loading Radiance FP8-KV calibration sidecar ... (verify=True) (checksum + identity passed at load) "uncalibrated" warning: gone /health: OK ``` ## What Calibration Actually Does (Benefits) **The problem.** With placeholder scale 1.0, FP8 values are stored as `value / 1.0`. The calibration hooks measured the model's real pre-quantization activations: peak **23.0 (K, layer 31)** and **59.75 (V, layer 43)** across the 16 full-attention layers. FP8 E4M3's max magnitude is 448 — so the raw values only used **5–13% of the representable range**. E4M3 has just 3 mantissa bits; when values cluster near the bottom of the range, the quantization steps between representable values are ~8–20× coarser than necessary. That is rounding noise injected into every attention score and KV lookup — worst for long-context retrieval, where KV detail is exactly what the task depends on. **The fix.** The calibration writes per-layer `scale = observed_amax × 1.05 / 448` (5% headroom for excursions the corpus didn't hit) and `prob_scale = 1.05 / 448` (softmax probabilities are bounded by 1). At load time vLLM divides each tensor by that scale before FP8 conversion, so observed maxima land at ~95% of the FP8 range — the full 3 mantissa bits do useful work. Result: **~7.5× finer quantization steps on V, ~19.6× finer on K** (≈ +2.9 / +4.3 effective mantissa bits), and the "may cause accuracy issues" warning is resolved. **Provenance & safety.** The sidecar is immutable (sha256 in the manifest), bound to this exact checkpoint, and re-verified at every startup by `VERIFY=1` — a changed model makes the server **fail closed** instead of silently using wrong scales. **What it does NOT do.** No throughput change (expected neutral — a big TPS delta means something is wrong, not a win). No new kernels. No weight-path change. Purely a precision fix for the FP8 attention/KV arithmetic. ## Operational Notes - **Sidecar is per-checkpoint.** Any fine-tune or re-download changes activation ranges — re-run calibration (fresh filename) rather than reusing scales by architecture name, and never set `VERIFY=0` as a deployment setting (it exists only to diagnose a deliberate mismatch). - **Startup re-hash cost.** The model dir on `driveripper-ai` has no HF download metadata, so the manifest bound full weight sha256s — every startup re-hashes all 18.44 GiB of weights for identity verification. Adds a few minutes on cold restarts; worked fine in practice. - **Health check.** `HealthStartPeriod=600s` was chosen deliberately: the deployed (pre-calibration) quadlet had the health check disabled at 30s start period — a cold kernel-compile startup would fail 3×10s of health checks and `HealthOnFailure=kill` + `Restart=always` loops it. - **Capacity ceiling.** 154K×2 runs the KV tier to ~99.6% at full context by design. Normal mixed traffic stays well below. If you need two full 256K concurrent requests, that is a CPU-offload decision, not a flag tweak. - **Corpus.** The 2026-10-10 sidecar used the stock 8-prompt corpus (tool schema, code, 4.9K-token retrieval, reasoning, multilingual, math, structured data). If the real workload is dominated by content the stock corpus doesn't cover, re-calibrate with an extended corpus. Calibration was run at 16K `max_model_len`; amax is largely context-independent, but if long-context numerics ever look off, that's the first thing to re-run with a longer `--max-model-len`. ## Remaining Work: Matched-Control Quality Gate Per `docs/FP8_KV_TUNABLEOP.md` and the project's no-claim-without-control policy: compare **calibrated vs default-scale** runs with identical image, target, prompts, seeds, sampling, graphs, and concurrency: - fixed-prompt greedy outputs (meaningful agreement) - sampled tool-schema checks - long-context retrieval - vision inputs if vision is deployed - BetterBench / sustained c1/c2/c4/c8 for the throughput-neutral claim Success criterion: **fidelity at-or-above the uncalibrated baseline with ~neutral throughput.** Only after that gate is this a qualified change, not just a plausible one.