services/homelab
2
1
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

add vllm-radiance R9700 deployment and fp8-kv calibration notes

Browse Source 
Documents the single-GPU (TP1) vllm-radiance deployment on
driveripper-ai, measured KV capacity analysis for the 154K config, and
the full FP8-KV calibration workflow (host prep through runtime
verification) including quadlet changes, deployment steps, and the
remaining matched-control quality gate.

LLM-Generated: true
This commit is contained in:
ducoterra
2026-10-12 10:31:43 -04:00
parent d9925b931d
commit 3f14b8ce14
1 changed files with 281 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
active/software_vllm_radiance/vllm_radiance.md
+281
View File
@@ -0,0 +1,281 @@
# vLLM Radiance on R9700
- [vLLM Radiance on R9700](#vllm-radiance-on-r9700)
- [Overview](#overview)
- [Deployment Topology](#deployment-topology)
- [Capacity Analysis (Single 32 GiB R9700)](#capacity-analysis-single-32-gib-r9700)
- [FP8-KV Calibration: The Process](#fp8-kv-calibration-the-process)
- [Why: The Uncalibrated Scale Problem](#why-the-uncalibrated-scale-problem)
- [Step 1: Host Prep](#step-1-host-prep)
- [Step 2: Stop the Service](#step-2-stop-the-service)
- [Step 3: Run the Calibration](#step-3-run-the-calibration)
- [Step 4: Verify the Artifact (CPU-only)](#step-4-verify-the-artifact-cpu-only)
- [Step 5: Edit the Quadlet](#step-5-edit-the-quadlet)
- [Step 6: Deploy via Ansible](#step-6-deploy-via-ansible)
- [Step 7: Confirm at Runtime](#step-7-confirm-at-runtime)
- [What Calibration Actually Does (Benefits)](#what-calibration-actually-does-benefits)
- [Operational Notes](#operational-notes)
- [Remaining Work: Matched-Control Quality Gate](#remaining-work-matched-control-quality-gate)
## Overview
`vllm-radiance` is a qualified downstream vLLM distribution for the AMD Radeon AI
PRO R9700 (gfx1201). The upstream-qualified environment is dual-GPU TP2; this
deployment runs the single-GPU (TP1) configuration on `driveripper-ai` serving
the AMD Quark MXFP4 27B checkpoint (W4A8: 4-bit weights, dynamic FP8
activation quantization over RDNA4's native FP8 WMMA path).
Served as `juggernaut` on the `ai-internal` network, port 8000.
## Deployment Topology
| Thing | Location |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Quadlet (source of |
| truth) | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/quadlets/vllm-radiance.container` |
| Ansible playbook | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/playbook.yaml` (run from `~/Deployments`: `ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml`) |
| Deployed quadlet | `driveripper-ai:/home/ai/.config/containers/systemd/ai-vllm-radiance.container` |
| systemd unit | **user** unit of `ai` (`systemctl --user status ai-vllm-radiance` — not in system scope) |
| Image | `docker.io/magiccodingman/vllm-radiance:1.0.387` |
| Model | `driveripper-ai:/home/ai/vllm-radiance-models/qwen3.8-27b-quark-awq-mxfp4` → `/models` (18.44 GiB on disk, 17.95 GiB in GPU) |
| Compile caches | `driveripper-ai:/home/ai/vllm-radiance-cache` → `/cache` |
| FP8-KV sidecar | `driveripper-ai:/home/ai/vllm-radiance-calibration` → `/calibration` (ro) |
The playbook copies the quadlet to `~/.config/containers/systemd/` and does
`systemctl --user restart ai-vllm-radiance` with daemon-reload. **Any edit to
the local quadlet lands on the next playbook run** — there is no separate
"apply" step.
Key quadlet settings (as of the calibration deploy, 2026-10-11):
```
--tensor-parallel-size=1 --max-model-len=154000 --max-num-seqs=2
--max-num-batched-tokens=4096 --kv-cache-dtype=fp8 --attention-backend=R4D
--enable-prefix-caching --mamba-cache-mode=align --no-async-scheduling
--gpu-memory-utilization=0.98
Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
HealthStartPeriod=600s (health check kills on 3x10s failures after the start period)
```
## Capacity Analysis (Single 32 GiB R9700)
Measured from the server startup logs (the authoritative numbers — do not
re-derive from weight sizes alone; the GDN hybrid architecture makes
per-token KV much cheaper than a plain transformer):
```
Model loading took 17.95 GiB
Actual usage: 19.38 GiB consumed (weights + non-torch)
+ 2.02 GiB peak activation
+ 0.41 GiB CUDA graphs
Available KV cache memory: 9.82 GiB (10.14 GiB at the 154K config)
GPU KV cache size: 308,847 tokens (309,466 at the 154K config)
```
Budget math (KV budget ≈ 309K tokens):
| Config | KV needed | Fits? |
| --------------------- | --------- | -------------------------------------------- |
| 150K × 2 | 300,000 | yes, ~3% headroom |
| **154K × 2 (chosen)** | 308,000 | yes — `2.01x` per vLLM, ~1.5K token headroom |
| 256K × 1 | 262,144 | yes |
| 256K × 2 | 524,288 | **no** — `1.18x` max concurrency |
The per-request C2 ceiling is `KV_budget / 2 ≈ 154,423` tokens; `--max-model-len=154000`
sits exactly at it. A second full 256K request is only reachable via CPU KV
offload (`RADIANCE_KV_OFFLOADING_SIZE`), which would need to roughly double the
token budget. `MAX_NUM_SEQS` is an admission ceiling, not a promise that every
admitted request can simultaneously reach `MAX_MODEL_LEN`.
## FP8-KV Calibration: The Process
### Why: The Uncalibrated Scale Problem
The server runs `--kv-cache-dtype=fp8` with FP8 attention, but the checkpoint
carries no q/k/v/prob scales. vLLM therefore used placeholder scale **1.0**,
which the startup log flags:
```
WARNING ... Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8
attention. This may cause accuracy issues.
```
Radiance ships `radiance_kv_calibration.py` (in the image, also in the
`vllm-radiance` repo) plus the `RADIANCE_FP8_KV_SCALES` runtime loader to fix
this with a small immutable sidecar. Full reference: `docs/FP8_KV_TUNABLEOP.md`
in the vllm-radiance repo.
### Step 1: Host Prep
The calibration dir did not exist and the vllm-radiance repo is not on
`driveripper-ai`, so copy only the corpus:
```bash
ssh driveripper-ai "mkdir -p /home/ai/vllm-radiance-calibration"
scp ~/Projects/Experimental/vllm-radiance/benchmarks/fixtures/fp8-kv-calibration.jsonl \
driveripper-ai:/home/ai/vllm-radiance-calibration/
```
(Optionally extend the JSONL with representative traffic first — each line is
`{"text": ...}` or `{"messages": [...], "tools": [...]}` plus optional `repeat`.)
### Step 2: Stop the Service
Calibration needs the GPU released; the server holds ~97% of VRAM.
```bash
ssh driveripper-ai "systemctl --user stop ai-vllm-radiance"
```
### Step 3: Run the Calibration
An eager, BF16 full-precision-KV, `max_tokens=1` prefill sweep with PyTorch
hooks on every full-attention layer (no source edits — public worker-extension
API only). Budget ~10–30 min:
```bash
ssh driveripper-ai "podman run --rm --ipc=host \
--device=/dev/kfd --device=/dev/dri \
--ulimit memlock=-1:-1 \
-e HIP_VISIBLE_DEVICES=0 \
-e RADIANCE_MXFP4=1 -e RADIANCE_MXFP4_W4A8=1 -e RADIANCE_MXFP4_W4A8_MIN_M=0 \
-v /home/ai/vllm-radiance-models:/models:ro \
-v /home/ai/vllm-radiance-calibration:/calibration:rw \
-v /home/ai/vllm-radiance-calibration/fp8-kv-calibration.jsonl:/fixtures/fp8-kv-calibration.jsonl:ro \
--entrypoint python \
docker.io/magiccodingman/vllm-radiance:1.0.387 \
-m radiance_kv_calibration calibrate \
--model /models/qwen3.8-27b-quark-awq-mxfp4 \
--quantization auto \
--corpus /fixtures/fp8-kv-calibration.jsonl \
--output /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
--tensor-parallel-size 1 \
--max-model-len 16384 \
--batch-size 4 \
--seed 17"
```
Success line: `[radiance] wrote immutable FP8-KV sidecar: /calibration/...`.
Outputs are **immutable** — a re-run needs a fresh filename. The sidecar and
its sibling `.manifest.json` (sha256 + full source-checkpoint identity +
per-layer observed amax + corpus/seed/software provenance) land in the host dir.
### Step 4: Verify the Artifact (CPU-only)
No engine start — checks manifest schema, sidecar sha256, that every tensor is
a positive finite scalar `*.q|k|v|prob_scale`, and (with `--model`) that it was
calibrated against this exact checkpoint:
```bash
ssh driveripper-ai "podman run --rm \
-v /home/ai/vllm-radiance-models:/models:ro \
-v /home/ai/vllm-radiance-calibration:/calibration:ro \
--entrypoint python docker.io/magiccodingman/vllm-radiance:1.0.387 \
-m radiance_kv_calibration verify \
/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
--model /models/qwen3.8-27b-quark-awq-mxfp4"
```
### Step 5: Edit the Quadlet
Three lines in the local quadlet (source of truth), then nothing else:
```diff
Volume=/home/ai/vllm-radiance-models:/models:ro,z
Volume=/home/ai/vllm-radiance-cache:/cache:z
+Volume=/home/ai/vllm-radiance-calibration:/calibration:ro,z
...
Environment=WEIGHT_QUANTIZATION=auto
+Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
+Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
```
### Step 6: Deploy via Ansible
```bash
cd ~/Deployments
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml
```
### Step 7: Confirm at Runtime
All three must be present/absent:
```
FP8-KV scales = /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors (banner; was "checkpoint/default")
Loading Radiance FP8-KV calibration sidecar ... (verify=True) (checksum + identity passed at load)
"uncalibrated" warning: gone
/health: OK
```
## What Calibration Actually Does (Benefits)
**The problem.** With placeholder scale 1.0, FP8 values are stored as
`value / 1.0`. The calibration hooks measured the model's real pre-quantization
activations: peak **23.0 (K, layer 31)** and **59.75 (V, layer 43)** across the
16 full-attention layers. FP8 E4M3's max magnitude is 448 — so the raw values
only used **5–13% of the representable range**. E4M3 has just 3 mantissa bits;
when values cluster near the bottom of the range, the quantization steps
between representable values are ~8–20× coarser than necessary. That is rounding
noise injected into every attention score and KV lookup — worst for
long-context retrieval, where KV detail is exactly what the task depends on.
**The fix.** The calibration writes per-layer
`scale = observed_amax × 1.05 / 448` (5% headroom for excursions the corpus
didn't hit) and `prob_scale = 1.05 / 448` (softmax probabilities are bounded by
1). At load time vLLM divides each tensor by that scale before FP8 conversion,
so observed maxima land at ~95% of the FP8 range — the full 3 mantissa bits do
useful work. Result: **~7.5× finer quantization steps on V, ~19.6× finer on K**
(≈ +2.9 / +4.3 effective mantissa bits), and the "may cause accuracy issues"
warning is resolved.
**Provenance & safety.** The sidecar is immutable (sha256 in the manifest),
bound to this exact checkpoint, and re-verified at every startup by
`VERIFY=1` — a changed model makes the server **fail closed** instead of
silently using wrong scales.
**What it does NOT do.** No throughput change (expected neutral — a big TPS
delta means something is wrong, not a win). No new kernels. No weight-path
change. Purely a precision fix for the FP8 attention/KV arithmetic.
## Operational Notes
- **Sidecar is per-checkpoint.** Any fine-tune or re-download changes
activation ranges — re-run calibration (fresh filename) rather than reusing
scales by architecture name, and never set `VERIFY=0` as a deployment
setting (it exists only to diagnose a deliberate mismatch).
- **Startup re-hash cost.** The model dir on `driveripper-ai` has no HF
download metadata, so the manifest bound full weight sha256s — every startup
re-hashes all 18.44 GiB of weights for identity verification. Adds a few
minutes on cold restarts; worked fine in practice.
- **Health check.** `HealthStartPeriod=600s` was chosen deliberately: the
deployed (pre-calibration) quadlet had the health check disabled at 30s start
period — a cold kernel-compile startup would fail 3×10s of health checks and
`HealthOnFailure=kill` + `Restart=always` loops it.
- **Capacity ceiling.** 154K×2 runs the KV tier to ~99.6% at full context by
design. Normal mixed traffic stays well below. If you need two full 256K
concurrent requests, that is a CPU-offload decision, not a flag tweak.
- **Corpus.** The 2026-10-10 sidecar used the stock 8-prompt corpus (tool
schema, code, 4.9K-token retrieval, reasoning, multilingual, math, structured
data). If the real workload is dominated by content the stock corpus doesn't
cover, re-calibrate with an extended corpus. Calibration was run at
16K `max_model_len`; amax is largely context-independent, but if long-context
numerics ever look off, that's the first thing to re-run with a longer
`--max-model-len`.
## Remaining Work: Matched-Control Quality Gate
Per `docs/FP8_KV_TUNABLEOP.md` and the project's no-claim-without-control
policy: compare **calibrated vs default-scale** runs with identical image,
target, prompts, seeds, sampling, graphs, and concurrency:
- fixed-prompt greedy outputs (meaningful agreement)
- sampled tool-schema checks
- long-context retrieval
- vision inputs if vision is deployed
- BetterBench / sustained c1/c2/c4/c8 for the throughput-neutral claim
Success criterion: **fidelity at-or-above the uncalibrated baseline with ~neutral
throughput.** Only after that gate is this a qualified change, not just a
plausible one.