diff --git a/active/software_vllm_radiance/vllm_radiance.md b/active/software_vllm_radiance/vllm_radiance.md new file mode 100644 index 0000000..a7e7958 --- /dev/null +++ b/active/software_vllm_radiance/vllm_radiance.md @@ -0,0 +1,281 @@ +# vLLM Radiance on R9700 + +- [vLLM Radiance on R9700](#vllm-radiance-on-r9700) + - [Overview](#overview) + - [Deployment Topology](#deployment-topology) + - [Capacity Analysis (Single 32 GiB R9700)](#capacity-analysis-single-32-gib-r9700) + - [FP8-KV Calibration: The Process](#fp8-kv-calibration-the-process) + - [Why: The Uncalibrated Scale Problem](#why-the-uncalibrated-scale-problem) + - [Step 1: Host Prep](#step-1-host-prep) + - [Step 2: Stop the Service](#step-2-stop-the-service) + - [Step 3: Run the Calibration](#step-3-run-the-calibration) + - [Step 4: Verify the Artifact (CPU-only)](#step-4-verify-the-artifact-cpu-only) + - [Step 5: Edit the Quadlet](#step-5-edit-the-quadlet) + - [Step 6: Deploy via Ansible](#step-6-deploy-via-ansible) + - [Step 7: Confirm at Runtime](#step-7-confirm-at-runtime) + - [What Calibration Actually Does (Benefits)](#what-calibration-actually-does-benefits) + - [Operational Notes](#operational-notes) + - [Remaining Work: Matched-Control Quality Gate](#remaining-work-matched-control-quality-gate) + +## Overview + +`vllm-radiance` is a qualified downstream vLLM distribution for the AMD Radeon AI +PRO R9700 (gfx1201). The upstream-qualified environment is dual-GPU TP2; this +deployment runs the single-GPU (TP1) configuration on `driveripper-ai` serving +the AMD Quark MXFP4 27B checkpoint (W4A8: 4-bit weights, dynamic FP8 +activation quantization over RDNA4's native FP8 WMMA path). + +Served as `juggernaut` on the `ai-internal` network, port 8000. + +## Deployment Topology + +| Thing | Location | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Quadlet (source of | +| truth) | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/quadlets/vllm-radiance.container` | +| Ansible playbook | `~/Deployments/reeseapps/ai/deployments/vllm-radiance/playbook.yaml` (run from `~/Deployments`: `ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml`) | +| Deployed quadlet | `driveripper-ai:/home/ai/.config/containers/systemd/ai-vllm-radiance.container` | +| systemd unit | **user** unit of `ai` (`systemctl --user status ai-vllm-radiance` — not in system scope) | +| Image | `docker.io/magiccodingman/vllm-radiance:1.0.387` | +| Model | `driveripper-ai:/home/ai/vllm-radiance-models/qwen3.8-27b-quark-awq-mxfp4` → `/models` (18.44 GiB on disk, 17.95 GiB in GPU) | +| Compile caches | `driveripper-ai:/home/ai/vllm-radiance-cache` → `/cache` | +| FP8-KV sidecar | `driveripper-ai:/home/ai/vllm-radiance-calibration` → `/calibration` (ro) | + +The playbook copies the quadlet to `~/.config/containers/systemd/` and does +`systemctl --user restart ai-vllm-radiance` with daemon-reload. **Any edit to +the local quadlet lands on the next playbook run** — there is no separate +"apply" step. + +Key quadlet settings (as of the calibration deploy, 2026-10-11): + +``` +--tensor-parallel-size=1 --max-model-len=154000 --max-num-seqs=2 +--max-num-batched-tokens=4096 --kv-cache-dtype=fp8 --attention-backend=R4D +--enable-prefix-caching --mamba-cache-mode=align --no-async-scheduling +--gpu-memory-utilization=0.98 +Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors +Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1 +HealthStartPeriod=600s (health check kills on 3x10s failures after the start period) +``` + +## Capacity Analysis (Single 32 GiB R9700) + +Measured from the server startup logs (the authoritative numbers — do not +re-derive from weight sizes alone; the GDN hybrid architecture makes +per-token KV much cheaper than a plain transformer): + +``` +Model loading took 17.95 GiB +Actual usage: 19.38 GiB consumed (weights + non-torch) + + 2.02 GiB peak activation + + 0.41 GiB CUDA graphs +Available KV cache memory: 9.82 GiB (10.14 GiB at the 154K config) +GPU KV cache size: 308,847 tokens (309,466 at the 154K config) +``` + +Budget math (KV budget ≈ 309K tokens): + +| Config | KV needed | Fits? | +| --------------------- | --------- | -------------------------------------------- | +| 150K × 2 | 300,000 | yes, ~3% headroom | +| **154K × 2 (chosen)** | 308,000 | yes — `2.01x` per vLLM, ~1.5K token headroom | +| 256K × 1 | 262,144 | yes | +| 256K × 2 | 524,288 | **no** — `1.18x` max concurrency | + +The per-request C2 ceiling is `KV_budget / 2 ≈ 154,423` tokens; `--max-model-len=154000` +sits exactly at it. A second full 256K request is only reachable via CPU KV +offload (`RADIANCE_KV_OFFLOADING_SIZE`), which would need to roughly double the +token budget. `MAX_NUM_SEQS` is an admission ceiling, not a promise that every +admitted request can simultaneously reach `MAX_MODEL_LEN`. + +## FP8-KV Calibration: The Process + +### Why: The Uncalibrated Scale Problem + +The server runs `--kv-cache-dtype=fp8` with FP8 attention, but the checkpoint +carries no q/k/v/prob scales. vLLM therefore used placeholder scale **1.0**, +which the startup log flags: + +``` +WARNING ... Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 +attention. This may cause accuracy issues. +``` + +Radiance ships `radiance_kv_calibration.py` (in the image, also in the +`vllm-radiance` repo) plus the `RADIANCE_FP8_KV_SCALES` runtime loader to fix +this with a small immutable sidecar. Full reference: `docs/FP8_KV_TUNABLEOP.md` +in the vllm-radiance repo. + +### Step 1: Host Prep + +The calibration dir did not exist and the vllm-radiance repo is not on +`driveripper-ai`, so copy only the corpus: + +```bash +ssh driveripper-ai "mkdir -p /home/ai/vllm-radiance-calibration" +scp ~/Projects/Experimental/vllm-radiance/benchmarks/fixtures/fp8-kv-calibration.jsonl \ + driveripper-ai:/home/ai/vllm-radiance-calibration/ +``` + +(Optionally extend the JSONL with representative traffic first — each line is +`{"text": ...}` or `{"messages": [...], "tools": [...]}` plus optional `repeat`.) + +### Step 2: Stop the Service + +Calibration needs the GPU released; the server holds ~97% of VRAM. + +```bash +ssh driveripper-ai "systemctl --user stop ai-vllm-radiance" +``` + +### Step 3: Run the Calibration + +An eager, BF16 full-precision-KV, `max_tokens=1` prefill sweep with PyTorch +hooks on every full-attention layer (no source edits — public worker-extension +API only). Budget ~10–30 min: + +```bash +ssh driveripper-ai "podman run --rm --ipc=host \ + --device=/dev/kfd --device=/dev/dri \ + --ulimit memlock=-1:-1 \ + -e HIP_VISIBLE_DEVICES=0 \ + -e RADIANCE_MXFP4=1 -e RADIANCE_MXFP4_W4A8=1 -e RADIANCE_MXFP4_W4A8_MIN_M=0 \ + -v /home/ai/vllm-radiance-models:/models:ro \ + -v /home/ai/vllm-radiance-calibration:/calibration:rw \ + -v /home/ai/vllm-radiance-calibration/fp8-kv-calibration.jsonl:/fixtures/fp8-kv-calibration.jsonl:ro \ + --entrypoint python \ + docker.io/magiccodingman/vllm-radiance:1.0.387 \ + -m radiance_kv_calibration calibrate \ + --model /models/qwen3.8-27b-quark-awq-mxfp4 \ + --quantization auto \ + --corpus /fixtures/fp8-kv-calibration.jsonl \ + --output /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \ + --tensor-parallel-size 1 \ + --max-model-len 16384 \ + --batch-size 4 \ + --seed 17" +``` + +Success line: `[radiance] wrote immutable FP8-KV sidecar: /calibration/...`. +Outputs are **immutable** — a re-run needs a fresh filename. The sidecar and +its sibling `.manifest.json` (sha256 + full source-checkpoint identity + +per-layer observed amax + corpus/seed/software provenance) land in the host dir. + +### Step 4: Verify the Artifact (CPU-only) + +No engine start — checks manifest schema, sidecar sha256, that every tensor is +a positive finite scalar `*.q|k|v|prob_scale`, and (with `--model`) that it was +calibrated against this exact checkpoint: + +```bash +ssh driveripper-ai "podman run --rm \ + -v /home/ai/vllm-radiance-models:/models:ro \ + -v /home/ai/vllm-radiance-calibration:/calibration:ro \ + --entrypoint python docker.io/magiccodingman/vllm-radiance:1.0.387 \ + -m radiance_kv_calibration verify \ + /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors \ + --model /models/qwen3.8-27b-quark-awq-mxfp4" +``` + +### Step 5: Edit the Quadlet + +Three lines in the local quadlet (source of truth), then nothing else: + +```diff + Volume=/home/ai/vllm-radiance-models:/models:ro,z + Volume=/home/ai/vllm-radiance-cache:/cache:z ++Volume=/home/ai/vllm-radiance-calibration:/calibration:ro,z + ... + Environment=WEIGHT_QUANTIZATION=auto ++Environment=RADIANCE_FP8_KV_SCALES=/calibration/qwen38-mxfp4-fp8kv-20261010.safetensors ++Environment=RADIANCE_FP8_KV_SCALES_VERIFY=1 +``` + +### Step 6: Deploy via Ansible + +```bash +cd ~/Deployments +ansible-playbook -i inventory.yaml reeseapps/ai/deployments/vllm-radiance/playbook.yaml +``` + +### Step 7: Confirm at Runtime + +All three must be present/absent: + +``` +FP8-KV scales = /calibration/qwen38-mxfp4-fp8kv-20261010.safetensors (banner; was "checkpoint/default") +Loading Radiance FP8-KV calibration sidecar ... (verify=True) (checksum + identity passed at load) +"uncalibrated" warning: gone +/health: OK +``` + +## What Calibration Actually Does (Benefits) + +**The problem.** With placeholder scale 1.0, FP8 values are stored as +`value / 1.0`. The calibration hooks measured the model's real pre-quantization +activations: peak **23.0 (K, layer 31)** and **59.75 (V, layer 43)** across the +16 full-attention layers. FP8 E4M3's max magnitude is 448 — so the raw values +only used **5–13% of the representable range**. E4M3 has just 3 mantissa bits; +when values cluster near the bottom of the range, the quantization steps +between representable values are ~8–20× coarser than necessary. That is rounding +noise injected into every attention score and KV lookup — worst for +long-context retrieval, where KV detail is exactly what the task depends on. + +**The fix.** The calibration writes per-layer +`scale = observed_amax × 1.05 / 448` (5% headroom for excursions the corpus +didn't hit) and `prob_scale = 1.05 / 448` (softmax probabilities are bounded by +1). At load time vLLM divides each tensor by that scale before FP8 conversion, +so observed maxima land at ~95% of the FP8 range — the full 3 mantissa bits do +useful work. Result: **~7.5× finer quantization steps on V, ~19.6× finer on K** +(≈ +2.9 / +4.3 effective mantissa bits), and the "may cause accuracy issues" +warning is resolved. + +**Provenance & safety.** The sidecar is immutable (sha256 in the manifest), +bound to this exact checkpoint, and re-verified at every startup by +`VERIFY=1` — a changed model makes the server **fail closed** instead of +silently using wrong scales. + +**What it does NOT do.** No throughput change (expected neutral — a big TPS +delta means something is wrong, not a win). No new kernels. No weight-path +change. Purely a precision fix for the FP8 attention/KV arithmetic. + +## Operational Notes + +- **Sidecar is per-checkpoint.** Any fine-tune or re-download changes + activation ranges — re-run calibration (fresh filename) rather than reusing + scales by architecture name, and never set `VERIFY=0` as a deployment + setting (it exists only to diagnose a deliberate mismatch). +- **Startup re-hash cost.** The model dir on `driveripper-ai` has no HF + download metadata, so the manifest bound full weight sha256s — every startup + re-hashes all 18.44 GiB of weights for identity verification. Adds a few + minutes on cold restarts; worked fine in practice. +- **Health check.** `HealthStartPeriod=600s` was chosen deliberately: the + deployed (pre-calibration) quadlet had the health check disabled at 30s start + period — a cold kernel-compile startup would fail 3×10s of health checks and + `HealthOnFailure=kill` + `Restart=always` loops it. +- **Capacity ceiling.** 154K×2 runs the KV tier to ~99.6% at full context by + design. Normal mixed traffic stays well below. If you need two full 256K + concurrent requests, that is a CPU-offload decision, not a flag tweak. +- **Corpus.** The 2026-10-10 sidecar used the stock 8-prompt corpus (tool + schema, code, 4.9K-token retrieval, reasoning, multilingual, math, structured + data). If the real workload is dominated by content the stock corpus doesn't + cover, re-calibrate with an extended corpus. Calibration was run at + 16K `max_model_len`; amax is largely context-independent, but if long-context + numerics ever look off, that's the first thing to re-run with a longer + `--max-model-len`. + +## Remaining Work: Matched-Control Quality Gate + +Per `docs/FP8_KV_TUNABLEOP.md` and the project's no-claim-without-control +policy: compare **calibrated vs default-scale** runs with identical image, +target, prompts, seeds, sampling, graphs, and concurrency: + +- fixed-prompt greedy outputs (meaningful agreement) +- sampled tool-schema checks +- long-context retrieval +- vision inputs if vision is deployed +- BetterBench / sustained c1/c2/c4/c8 for the throughput-neutral claim + +Success criterion: **fidelity at-or-above the uncalibrated baseline with ~neutral +throughput.** Only after that gate is this a qualified change, not just a +plausible one.