ducoterra
3f14b8ce14
Documents the single-GPU (TP1) vllm-radiance deployment on driveripper-ai, measured KV capacity analysis for the 154K config, and the full FP8-KV calibration workflow (host prep through runtime verification) including quadlet changes, deployment steps, and the remaining matched-control quality gate. LLM-Generated: true
282 lines
14 KiB
Markdown
282 lines
14 KiB
Markdown
# vLLM Radiance on R9700
|
||
|
||
- [vLLM Radiance on R9700](#vllm-radiance-on-r9700)
|
||
- [Overview](#overview)
|
||
- [Deployment Topology](#deployment-topology)
|
||
- [Capacity Analysis (Single 32 GiB R9700)](#capacity-analysis-single-32-gib-r9700)
|
||
- [FP8-KV Calibration: The Process](#fp8-kv-calibration-the-process)
|
||
- [Why: The Uncalibrated Scale Problem](#why-the-uncalibrated-scale-problem)
|
||
- [Step 1: Host Prep](#step-1-host-prep)
|
||
- [Step 2: Stop the Service](#step-2-stop-the-service)
|
||
- [Step 3: Run the Calibration](#step-3-run-the-calibration)
|
||
- [Step 4: Verify the Artifact (CPU-only)](#step-4-verify-the-artifact-cpu-only)
|
||
- [Step 5: Edit the Quadlet](#step-5-edit-the-quadlet)
|
||
- [Step 6: Deploy via Ansible](#step-6-deploy-via-ansible)
|
||
- [Step 7: Confirm at Runtime](#step-7-confirm-at-runtime)
|
||
- [What Calibration Actually Does (Benefits)](#what-calibration-actually-does-benefits)
|
||
- [Operational Notes](#operational-notes)
|
||
- [Remaining Work: Matched-Control Quality Gate](#remaining-work-matched-control-quality-gate)
|
||
|
||
## Overview
|
||
|
||
`vllm-radiance` is a qualified downstream vLLM distribution for the AMD Radeon AI
|
||
PRO R9700 (gfx1201). The upstream-qualified environment is dual-GPU TP2; this
|
||
deployment runs the single-GPU (TP1) configuration on `driveripper-ai` serving
|
||
the AMD Quark MXFP4 27B checkpoint (W4A8: 4-bit weights, dynamic FP8
|
||
activation quantization over RDNA4's native FP8 WMMA path).
|
||
|
||
Served as `juggernaut` on the `ai-internal` network, port 8000.
|
||
|
||
## Deployment Topology
|
||
|
||
| Thing | Location |
|
||
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||
| Quadlet (source of |
|
||
| truth) | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/quadlets/vllm-radiance.container` |
|
||
| Ansible playbook | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/playbook.yaml` (run from `~/Deployments`: `ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml`) |
|
||
| Deployed quadlet | `driveripper-ai:/home/ai/.config/containers/systemd/ai-vllm-radiance.container` |
|
||
| systemd unit | **user** unit of `ai` (`systemctl --user status ai-vllm-radiance` — not in system scope) |
|
||
| Image | `docker.io/magiccodingman/vllm-radiance:1.0.387` |
|
||
| Model | `driveripper-ai:/home/ai/vllm-radiance-models/qwen3.8-27b-quark-awq-mxfp4` → `/models` (18.44 GiB on disk, 17.95 GiB in GPU) |
|
||
| Compile caches | `driveripper-ai:/home/ai/vllm-radiance-cache` → `/cache` |
|
||
| FP8-KV sidecar | `driveripper-ai:/home/ai/vllm-radiance-calibration` → `/calibration` (ro) |
|
||
|
||
The playbook copies the quadlet to `~/.config/containers/systemd/` and does
|
||
`systemctl --user restart ai-vllm-radiance` with daemon-reload. **Any edit to
|
||
the local quadlet lands on the next playbook run** — there is no separate
|
||
"apply" step.
|
||
|
||
Key quadlet settings (as of the calibration deploy, 2026-10-11):
|
||
|
||
```
|
||
--tensor-parallel-size=1 --max-model-len=154000 --max-num-seqs=2
|
||
--max-num-batched-tokens=4096 --kv-cache-dtype=fp8 --attention-backend=R4D
|
||
--enable-prefix-caching --mamba-cache-mode=align --no-async-scheduling
|
||
--gpu-memory-utilization=0.98
|
||
Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
|
||
Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
|
||
HealthStartPeriod=600s (health check kills on 3x10s failures after the start period)
|
||
```
|
||
|
||
## Capacity Analysis (Single 32 GiB R9700)
|
||
|
||
Measured from the server startup logs (the authoritative numbers — do not
|
||
re-derive from weight sizes alone; the GDN hybrid architecture makes
|
||
per-token KV much cheaper than a plain transformer):
|
||
|
||
```
|
||
Model loading took 17.95 GiB
|
||
Actual usage: 19.38 GiB consumed (weights + non-torch)
|
||
+ 2.02 GiB peak activation
|
||
+ 0.41 GiB CUDA graphs
|
||
Available KV cache memory: 9.82 GiB (10.14 GiB at the 154K config)
|
||
GPU KV cache size: 308,847 tokens (309,466 at the 154K config)
|
||
```
|
||
|
||
Budget math (KV budget ≈ 309K tokens):
|
||
|
||
| Config | KV needed | Fits? |
|
||
| --------------------- | --------- | -------------------------------------------- |
|
||
| 150K × 2 | 300,000 | yes, ~3% headroom |
|
||
| **154K × 2 (chosen)** | 308,000 | yes — `2.01x` per vLLM, ~1.5K token headroom |
|
||
| 256K × 1 | 262,144 | yes |
|
||
| 256K × 2 | 524,288 | **no** — `1.18x` max concurrency |
|
||
|
||
The per-request C2 ceiling is `KV_budget / 2 ≈ 154,423` tokens; `--max-model-len=154000`
|
||
sits exactly at it. A second full 256K request is only reachable via CPU KV
|
||
offload (`RADIANCE_KV_OFFLOADING_SIZE`), which would need to roughly double the
|
||
token budget. `MAX_NUM_SEQS` is an admission ceiling, not a promise that every
|
||
admitted request can simultaneously reach `MAX_MODEL_LEN`.
|
||
|
||
## FP8-KV Calibration: The Process
|
||
|
||
### Why: The Uncalibrated Scale Problem
|
||
|
||
The server runs `--kv-cache-dtype=fp8` with FP8 attention, but the checkpoint
|
||
carries no q/k/v/prob scales. vLLM therefore used placeholder scale **1.0**,
|
||
which the startup log flags:
|
||
|
||
```
|
||
WARNING ... Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8
|
||
attention. This may cause accuracy issues.
|
||
```
|
||
|
||
Radiance ships `radiance_kv_calibration.py` (in the image, also in the
|
||
`vllm-radiance` repo) plus the `RADIANCE_FP8_KV_SCALES` runtime loader to fix
|
||
this with a small immutable sidecar. Full reference: `docs/FP8_KV_TUNABLEOP.md`
|
||
in the vllm-radiance repo.
|
||
|
||
### Step 1: Host Prep
|
||
|
||
The calibration dir did not exist and the vllm-radiance repo is not on
|
||
`driveripper-ai`, so copy only the corpus:
|
||
|
||
```bash
|
||
ssh driveripper-ai "mkdir -p /home/ai/vllm-radiance-calibration"
|
||
scp ~/Projects/Experimental/vllm-radiance/benchmarks/fixtures/fp8-kv-calibration.jsonl \
|
||
driveripper-ai:/home/ai/vllm-radiance-calibration/
|
||
```
|
||
|
||
(Optionally extend the JSONL with representative traffic first — each line is
|
||
`{"text": ...}` or `{"messages": [...], "tools": [...]}` plus optional `repeat`.)
|
||
|
||
### Step 2: Stop the Service
|
||
|
||
Calibration needs the GPU released; the server holds ~97% of VRAM.
|
||
|
||
```bash
|
||
ssh driveripper-ai "systemctl --user stop ai-vllm-radiance"
|
||
```
|
||
|
||
### Step 3: Run the Calibration
|
||
|
||
An eager, BF16 full-precision-KV, `max_tokens=1` prefill sweep with PyTorch
|
||
hooks on every full-attention layer (no source edits — public worker-extension
|
||
API only). Budget ~10–30 min:
|
||
|
||
```bash
|
||
ssh driveripper-ai "podman run --rm --ipc=host \
|
||
--device=/dev/kfd --device=/dev/dri \
|
||
--ulimit memlock=-1:-1 \
|
||
-e HIP_VISIBLE_DEVICES=0 \
|
||
-e RADIANCE_MXFP4=1 -e RADIANCE_MXFP4_W4A8=1 -e RADIANCE_MXFP4_W4A8_MIN_M=0 \
|
||
-v /home/ai/vllm-radiance-models:/models:ro \
|
||
-v /home/ai/vllm-radiance-calibration:/calibration:rw \
|
||
-v /home/ai/vllm-radiance-calibration/fp8-kv-calibration.jsonl:/fixtures/fp8-kv-calibration.jsonl:ro \
|
||
--entrypoint python \
|
||
docker.io/magiccodingman/vllm-radiance:1.0.387 \
|
||
-m radiance_kv_calibration calibrate \
|
||
--model /models/qwen3.8-27b-quark-awq-mxfp4 \
|
||
--quantization auto \
|
||
--corpus /fixtures/fp8-kv-calibration.jsonl \
|
||
--output /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
|
||
--tensor-parallel-size 1 \
|
||
--max-model-len 16384 \
|
||
--batch-size 4 \
|
||
--seed 17"
|
||
```
|
||
|
||
Success line: `[radiance] wrote immutable FP8-KV sidecar: /calibration/...`.
|
||
Outputs are **immutable** — a re-run needs a fresh filename. The sidecar and
|
||
its sibling `.manifest.json` (sha256 + full source-checkpoint identity +
|
||
per-layer observed amax + corpus/seed/software provenance) land in the host dir.
|
||
|
||
### Step 4: Verify the Artifact (CPU-only)
|
||
|
||
No engine start — checks manifest schema, sidecar sha256, that every tensor is
|
||
a positive finite scalar `*.q|k|v|prob_scale`, and (with `--model`) that it was
|
||
calibrated against this exact checkpoint:
|
||
|
||
```bash
|
||
ssh driveripper-ai "podman run --rm \
|
||
-v /home/ai/vllm-radiance-models:/models:ro \
|
||
-v /home/ai/vllm-radiance-calibration:/calibration:ro \
|
||
--entrypoint python docker.io/magiccodingman/vllm-radiance:1.0.387 \
|
||
-m radiance_kv_calibration verify \
|
||
/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \
|
||
--model /models/qwen3.8-27b-quark-awq-mxfp4"
|
||
```
|
||
|
||
### Step 5: Edit the Quadlet
|
||
|
||
Three lines in the local quadlet (source of truth), then nothing else:
|
||
|
||
```diff
|
||
Volume=/home/ai/vllm-radiance-models:/models:ro,z
|
||
Volume=/home/ai/vllm-radiance-cache:/cache:z
|
||
+Volume=/home/ai/vllm-radiance-calibration:/calibration:ro,z
|
||
...
|
||
Environment=WEIGHT_QUANTIZATION=auto
|
||
+Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors
|
||
+Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1
|
||
```
|
||
|
||
### Step 6: Deploy via Ansible
|
||
|
||
```bash
|
||
cd ~/Deployments
|
||
ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml
|
||
```
|
||
|
||
### Step 7: Confirm at Runtime
|
||
|
||
All three must be present/absent:
|
||
|
||
```
|
||
FP8-KV scales = /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors (banner; was "checkpoint/default")
|
||
Loading Radiance FP8-KV calibration sidecar ... (verify=True) (checksum + identity passed at load)
|
||
"uncalibrated" warning: gone
|
||
/health: OK
|
||
```
|
||
|
||
## What Calibration Actually Does (Benefits)
|
||
|
||
**The problem.** With placeholder scale 1.0, FP8 values are stored as
|
||
`value / 1.0`. The calibration hooks measured the model's real pre-quantization
|
||
activations: peak **23.0 (K, layer 31)** and **59.75 (V, layer 43)** across the
|
||
16 full-attention layers. FP8 E4M3's max magnitude is 448 — so the raw values
|
||
only used **5–13% of the representable range**. E4M3 has just 3 mantissa bits;
|
||
when values cluster near the bottom of the range, the quantization steps
|
||
between representable values are ~8–20× coarser than necessary. That is rounding
|
||
noise injected into every attention score and KV lookup — worst for
|
||
long-context retrieval, where KV detail is exactly what the task depends on.
|
||
|
||
**The fix.** The calibration writes per-layer
|
||
`scale = observed_amax × 1.05 / 448` (5% headroom for excursions the corpus
|
||
didn't hit) and `prob_scale = 1.05 / 448` (softmax probabilities are bounded by
|
||
1). At load time vLLM divides each tensor by that scale before FP8 conversion,
|
||
so observed maxima land at ~95% of the FP8 range — the full 3 mantissa bits do
|
||
useful work. Result: **~7.5× finer quantization steps on V, ~19.6× finer on K**
|
||
(≈ +2.9 / +4.3 effective mantissa bits), and the "may cause accuracy issues"
|
||
warning is resolved.
|
||
|
||
**Provenance & safety.** The sidecar is immutable (sha256 in the manifest),
|
||
bound to this exact checkpoint, and re-verified at every startup by
|
||
`VERIFY=1` — a changed model makes the server **fail closed** instead of
|
||
silently using wrong scales.
|
||
|
||
**What it does NOT do.** No throughput change (expected neutral — a big TPS
|
||
delta means something is wrong, not a win). No new kernels. No weight-path
|
||
change. Purely a precision fix for the FP8 attention/KV arithmetic.
|
||
|
||
## Operational Notes
|
||
|
||
- **Sidecar is per-checkpoint.** Any fine-tune or re-download changes
|
||
activation ranges — re-run calibration (fresh filename) rather than reusing
|
||
scales by architecture name, and never set `VERIFY=0` as a deployment
|
||
setting (it exists only to diagnose a deliberate mismatch).
|
||
- **Startup re-hash cost.** The model dir on `driveripper-ai` has no HF
|
||
download metadata, so the manifest bound full weight sha256s — every startup
|
||
re-hashes all 18.44 GiB of weights for identity verification. Adds a few
|
||
minutes on cold restarts; worked fine in practice.
|
||
- **Health check.** `HealthStartPeriod=600s` was chosen deliberately: the
|
||
deployed (pre-calibration) quadlet had the health check disabled at 30s start
|
||
period — a cold kernel-compile startup would fail 3×10s of health checks and
|
||
`HealthOnFailure=kill` + `Restart=always` loops it.
|
||
- **Capacity ceiling.** 154K×2 runs the KV tier to ~99.6% at full context by
|
||
design. Normal mixed traffic stays well below. If you need two full 256K
|
||
concurrent requests, that is a CPU-offload decision, not a flag tweak.
|
||
- **Corpus.** The 2026-10-10 sidecar used the stock 8-prompt corpus (tool
|
||
schema, code, 4.9K-token retrieval, reasoning, multilingual, math, structured
|
||
data). If the real workload is dominated by content the stock corpus doesn't
|
||
cover, re-calibrate with an extended corpus. Calibration was run at
|
||
16K `max_model_len`; amax is largely context-independent, but if long-context
|
||
numerics ever look off, that's the first thing to re-run with a longer
|
||
`--max-model-len`.
|
||
|
||
## Remaining Work: Matched-Control Quality Gate
|
||
|
||
Per `docs/FP8_KV_TUNABLEOP.md` and the project's no-claim-without-control
|
||
policy: compare **calibrated vs default-scale** runs with identical image,
|
||
target, prompts, seeds, sampling, graphs, and concurrency:
|
||
|
||
- fixed-prompt greedy outputs (meaningful agreement)
|
||
- sampled tool-schema checks
|
||
- long-context retrieval
|
||
- vision inputs if vision is deployed
|
||
- BetterBench / sustained c1/c2/c4/c8 for the throughput-neutral claim
|
||
|
||
Success criterion: **fidelity at-or-above the uncalibrated baseline with ~neutral
|
||
throughput.** Only after that gate is this a qualified change, not just a
|
||
plausible one.
|