homelab/active/software_vllm_radiance/vllm_radiance.md

vLLM Radiance on R9700

• vLLM Radiance on R9700
  • Overview
  • Deployment Topology
  • Capacity Analysis (Single 32 GiB R9700)
  • FP8-KV Calibration: The Process
    • Why: The Uncalibrated Scale Problem
    • Step 1: Host Prep
    • Step 2: Stop the Service
    • Step 3: Run the Calibration
    • Step 4: Verify the Artifact (CPU-only)
    • Step 5: Edit the Quadlet
    • Step 6: Deploy via Ansible
    • Step 7: Confirm at Runtime
  • What Calibration Actually Does (Benefits)
  • Operational Notes
  • Remaining Work: Matched-Control Quality Gate

Overview

vllm-radiance is a qualified downstream vLLM distribution for the AMD Radeon AI PRO R9700 (gfx1201). The upstream-qualified environment is dual-GPU TP2; this deployment runs the single-GPU (TP1) configuration on driveripper-ai serving the AMD Quark MXFP4 27B checkpoint (W4A8: 4-bit weights, dynamic FP8 activation quantization over RDNA4's native FP8 WMMA path).

Served as juggernaut on the ai-internal network, port 8000.

Deployment Topology

Thing	Location
Quadlet (source of
truth)	~/Deployments/reeseapps/ai/deployments/vllm-radiance/quadlets/vllm-radiance.container
Ansible playbook	~/Deployments/reeseapps/ai/deployments/vllm-radiance/playbook.yaml (run from ~/Deployments: ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml)
Deployed quadlet	driveripper-ai:/home/ai/.config/containers/systemd/ai-vllm-radiance.container
systemd unit	user unit of ai (systemctl --user status ai-vllm-radiance — not in system scope)
Image	docker.io/magiccodingman/vllm-radiance:1.0.387
Model	driveripper-ai:/home/ai/vllm-radiance-models/qwen3.8-27b-quark-awq-mxfp4 → /models (18.44 GiB on disk, 17.95 GiB in GPU)
Compile caches	driveripper-ai:/home/ai/vllm-radiance-cache → /cache
FP8-KV sidecar	driveripper-ai:/home/ai/vllm-radiance-calibration → /calibration (ro)

The playbook copies the quadlet to ~/.config/containers/systemd/ and does systemctl --user restart ai-vllm-radiance with daemon-reload. Any edit to the local quadlet lands on the next playbook run — there is no separate "apply" step.

Key quadlet settings (as of the calibration deploy, 2026-10-11):

--tensor-parallel-size=1  --max-model-len=154000  --max-num-seqs=2
--max-num-batched-tokens=4096  --kv-cache-dtype=fp8  --attention-backend=R4D
--enable-prefix-caching  --mamba-cache-mode=align  --no-async-scheduling
--gpu-memory-utilization=0.98
Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
HealthStartPeriod=600s  (health check kills on 3x10s failures after the start period)

Capacity Analysis (Single 32 GiB R9700)

Measured from the server startup logs (the authoritative numbers — do not re-derive from weight sizes alone; the GDN hybrid architecture makes per-token KV much cheaper than a plain transformer):

Model loading took 17.95 GiB
Actual usage: 19.38 GiB consumed (weights + non-torch)
             + 2.02 GiB peak activation
             + 0.41 GiB CUDA graphs
Available KV cache memory: 9.82 GiB  (10.14 GiB at the 154K config)
GPU KV cache size: 308,847 tokens   (309,466 at the 154K config)

Budget math (KV budget ≈ 309K tokens):

Config	KV needed	Fits?
150K × 2	300,000	yes, ~3% headroom
154K × 2 (chosen)	308,000	yes — 2.01x per vLLM, ~1.5K token headroom
256K × 1	262,144	yes
256K × 2	524,288	no — 1.18x max concurrency

The per-request C2 ceiling is KV_budget / 2 ≈ 154,423 tokens; --max-model-len=154000 sits exactly at it. A second full 256K request is only reachable via CPU KV offload (RADIANCE_KV_OFFLOADING_SIZE), which would need to roughly double the token budget. MAX_NUM_SEQS is an admission ceiling, not a promise that every admitted request can simultaneously reach MAX_MODEL_LEN.

FP8-KV Calibration: The Process

Why: The Uncalibrated Scale Problem

The server runs --kv-cache-dtype=fp8 with FP8 attention, but the checkpoint carries no q/k/v/prob scales. vLLM therefore used placeholder scale 1.0, which the startup log flags:

WARNING ... Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8
attention. This may cause accuracy issues.

Radiance ships radiance_kv_calibration.py (in the image, also in the vllm-radiance repo) plus the RADIANCE_FP8_KV_SCALES runtime loader to fix this with a small immutable sidecar. Full reference: docs/FP8_KV_TUNABLEOP.md in the vllm-radiance repo.

Step 1: Host Prep

The calibration dir did not exist and the vllm-radiance repo is not on driveripper-ai, so copy only the corpus:

ssh driveripper-ai "mkdir -p /home/ai/vllm-radiance-calibration"
scp ~/Projects/Experimental/vllm-radiance/benchmarks/fixtures/fp8-kv-calibration.jsonl \
    driveripper-ai:/home/ai/vllm-radiance-calibration/

(Optionally extend the JSONL with representative traffic first — each line is {"text": ...} or {"messages": [...], "tools": [...]} plus optional repeat.)

Step 2: Stop the Service

Calibration needs the GPU released; the server holds ~97% of VRAM.

ssh driveripper-ai "systemctl --user stop ai-vllm-radiance"

Step 3: Run the Calibration

An eager, BF16 full-precision-KV, max_tokens=1 prefill sweep with PyTorch hooks on every full-attention layer (no source edits — public worker-extension API only). Budget ~10–30 min:

ssh driveripper-ai "podman run --rm --ipc=host \
  --device=/dev/kfd --device=/dev/dri \
  --ulimit memlock=-1:-1 \
  -e HIP_VISIBLE_DEVICES=0 \
  -e RADIANCE_MXFP4=1 -e RADIANCE_MXFP4_W4A8=1 -e RADIANCE_MXFP4_W4A8_MIN_M=0 \
  -v /home/ai/vllm-radiance-models:/models:ro \
  -v /home/ai/vllm-radiance-calibration:/calibration:rw \
  -v /home/ai/vllm-radiance-calibration/fp8-kv-calibration.jsonl:/fixtures/fp8-kv-calibration.jsonl:ro \
  --entrypoint python \
  docker.io/magiccodingman/vllm-radiance:1.0.387 \
  -m radiance_kv_calibration calibrate \
  --model /models/qwen3.8-27b-quark-awq-mxfp4 \
  --quantization auto \
  --corpus /fixtures/fp8-kv-calibration.jsonl \
  --output /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
  --tensor-parallel-size 1 \
  --max-model-len 16384 \
  --batch-size 4 \
  --seed 17"

Success line: [radiance] wrote immutable FP8-KV sidecar: /calibration/.... Outputs are immutable — a re-run needs a fresh filename. The sidecar and its sibling .manifest.json (sha256 + full source-checkpoint identity + per-layer observed amax + corpus/seed/software provenance) land in the host dir.

Step 4: Verify the Artifact (CPU-only)

No engine start — checks manifest schema, sidecar sha256, that every tensor is a positive finite scalar *.q|k|v|prob_scale, and (with --model) that it was calibrated against this exact checkpoint:

ssh driveripper-ai "podman run --rm \
  -v /home/ai/vllm-radiance-models:/models:ro \
  -v /home/ai/vllm-radiance-calibration:/calibration:ro \
  --entrypoint python docker.io/magiccodingman/vllm-radiance:1.0.387 \
  -m radiance_kv_calibration verify \
  /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
  --model /models/qwen3.8-27b-quark-awq-mxfp4"

Step 5: Edit the Quadlet

Three lines in the local quadlet (source of truth), then nothing else:

 Volume=/home/ai/vllm-radiance-models:/models:ro,z
 Volume=/home/ai/vllm-radiance-cache:/cache:z
+Volume=/home/ai/vllm-radiance-calibration:/calibration:ro,z
 ...
 Environment=WEIGHT_QUANTIZATION=auto
+Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
+Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1

Step 6: Deploy via Ansible

cd ~/Deployments
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml

Step 7: Confirm at Runtime

All three must be present/absent:

FP8-KV scales = /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors   (banner; was "checkpoint/default")
Loading Radiance FP8-KV calibration sidecar ... (verify=True)          (checksum + identity passed at load)
"uncalibrated" warning: gone
/health: OK

What Calibration Actually Does (Benefits)

The problem. With placeholder scale 1.0, FP8 values are stored as value / 1.0. The calibration hooks measured the model's real pre-quantization activations: peak 23.0 (K, layer 31) and 59.75 (V, layer 43) across the 16 full-attention layers. FP8 E4M3's max magnitude is 448 — so the raw values only used 5–13% of the representable range. E4M3 has just 3 mantissa bits; when values cluster near the bottom of the range, the quantization steps between representable values are ~8–20× coarser than necessary. That is rounding noise injected into every attention score and KV lookup — worst for long-context retrieval, where KV detail is exactly what the task depends on.

The fix. The calibration writes per-layer scale = observed_amax × 1.05 / 448 (5% headroom for excursions the corpus didn't hit) and prob_scale = 1.05 / 448 (softmax probabilities are bounded by 1). At load time vLLM divides each tensor by that scale before FP8 conversion, so observed maxima land at ~95% of the FP8 range — the full 3 mantissa bits do useful work. Result: ~7.5× finer quantization steps on V, ~19.6× finer on K (≈ +2.9 / +4.3 effective mantissa bits), and the "may cause accuracy issues" warning is resolved.

Provenance & safety. The sidecar is immutable (sha256 in the manifest), bound to this exact checkpoint, and re-verified at every startup by VERIFY=1 — a changed model makes the server fail closed instead of silently using wrong scales.

What it does NOT do. No throughput change (expected neutral — a big TPS delta means something is wrong, not a win). No new kernels. No weight-path change. Purely a precision fix for the FP8 attention/KV arithmetic.

Operational Notes

• Sidecar is per-checkpoint. Any fine-tune or re-download changes activation ranges — re-run calibration (fresh filename) rather than reusing scales by architecture name, and never set VERIFY=0 as a deployment setting (it exists only to diagnose a deliberate mismatch).
• Startup re-hash cost. The model dir on driveripper-ai has no HF download metadata, so the manifest bound full weight sha256s — every startup re-hashes all 18.44 GiB of weights for identity verification. Adds a few minutes on cold restarts; worked fine in practice.
• Health check. HealthStartPeriod=600s was chosen deliberately: the deployed (pre-calibration) quadlet had the health check disabled at 30s start period — a cold kernel-compile startup would fail 3×10s of health checks and HealthOnFailure=kill + Restart=always loops it.
• Capacity ceiling. 154K×2 runs the KV tier to ~99.6% at full context by design. Normal mixed traffic stays well below. If you need two full 256K concurrent requests, that is a CPU-offload decision, not a flag tweak.
• Corpus. The 2026-10-10 sidecar used the stock 8-prompt corpus (tool schema, code, 4.9K-token retrieval, reasoning, multilingual, math, structured data). If the real workload is dominated by content the stock corpus doesn't cover, re-calibrate with an extended corpus. Calibration was run at 16K max_model_len; amax is largely context-independent, but if long-context numerics ever look off, that's the first thing to re-run with a longer --max-model-len.

Remaining Work: Matched-Control Quality Gate

Per docs/FP8_KV_TUNABLEOP.md and the project's no-claim-without-control policy: compare calibrated vs default-scale runs with identical image, target, prompts, seeds, sampling, graphs, and concurrency:

• fixed-prompt greedy outputs (meaningful agreement)
• sampled tool-schema checks
• long-context retrieval
• vision inputs if vision is deployed
• BetterBench / sustained c1/c2/c4/c8 for the throughput-neutral claim

Success criterion: fidelity at-or-above the uncalibrated baseline with ~neutral throughput. Only after that gate is this a qualified change, not just a plausible one.