[mikasenghaas]

Summary

• abort stale vLLM requests at weight-update boundaries so cancelled evaluations cannot retain old-policy KV state or block broadcasts
• backport Garrett's focused BF16-trainer / FP8-inference path from feat(deepseek_v4): fp8 RL #3627:
  • power-of-two UE8M0 block scales
  • stochastic E4M3 weight rounding for unbiased sub-bin online updates
  • DeepSeek V4 SwiGLU clamp handling on the Triton FP8 fast path
• retain the existing DeepSeek V4 attention-sink loading fix from the base branch

Why stochastic rounding

Round-to-nearest can pin a weight to its initial E4M3 value until training moves it by half a quantization bin. Stochastic rounding instead selects the adjacent representable value according to the fractional distance, so the served value equals the BF16 trainer value in expectation. Values already on the FP8 grid remain exact.

Validation

• ruff passes
• full unit suite: 191 passed
• dedicated stochastic-rounding tests cover exact on-grid preservation and seeded midpoint unbiasedness
• 8 trainer + 4 inference node DSV4F smoke test completed all 5 steps on 96 B300 GPUs
• step-0 BF16-to-FP8 broadcast and reload succeeded; broadcasts/reloads continued through step 5
• step-0 and terminal online evals completed with 0% rollout errors and 0% truncation

Metric	Result
Slurm result	COMPLETED (exit 0)
End-to-end runtime	20m 41s
Post-compile throughput	39.9k-50.5k tok/s
Post-compile MFU	18.5%-23.4%
Peak trainer memory	153.3 / 267.7 GiB per B300 (57.3%)

On B300/SM103, DeepGEMM requires packed UE8M0 scales, so the validated config pairs fp8_ue8m0_weight_scales = true with VLLM_USE_DEEP_GEMM_E8M0=1. The older H200-style disabled setting fails during engine startup with Unsupported architecture or scaling factor types.

This PR is stacked on #3625 for the uneven-shard Muon and DSV4 attention-sink fixes. The FP8 commits preserve Garrett's authorship from #3627.

---

Note

Medium Risk
Changes weight-update pause semantics and online FP8 quantization paths that directly affect trainer–inference policy alignment; incorrect behavior would skew rollouts rather than typical UI-only risk.

Overview
Hardens online RL weight broadcast by pausing inference with mode=abort (orchestrator default) instead of draining in-flight work, and exposes mode / clear_cache on the vLLM /pause admin route so stale rollouts cannot block updates or resume under the old policy’s KV state.

Adds inference.fp8_ue8m0_weight_scales and inference.fp8_stochastic_weight_rounding, wired through worker env vars to vLLM patches that force power-of-two block scales and unbiased stochastic e4m3 rounding on online per_block_cast_to_fp8, so BF16 trainer weights round-trip dequantized checkpoints and track sub-bin updates in expectation. Trainer-side Triton FP8 casts are updated to be bit-identical to vLLM’s CUDA quantizer (RN divide, clamp, scale floor).

Additional vLLM backports: DeepSeek V4 request-level tools attach to the first existing system message (matches reference encoding), and Triton MoE SwiGLU clamp is honored on the fused FP8 fast path. Multi-node SLURM templates set -a when sourcing .env so vars propagate into srun child shells.

^{Reviewed by Cursor Bugbot for commit a3ffa1a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5748803. Configure here.}

[cursor]

Dynamo NCCL pause still uses keep

High Severity

The Dynamo NCCL weight-update path still pauses with mode=keep. Cancelled non-streaming requests can stay alive in vLLM and block that pause indefinitely, leaving KV from the previous policy across the update.

 

^{Reviewed by Cursor Bugbot for commit 5748803. Configure here.}

[mikasenghaas]

Superseded by the #3692 fork branch, which the DSV4F SFT run pins.