Repository navigation
feat(inference): opt-in online FP8 ue8m0 scales and stochastic weight rounding - #4041
Closed
garrett361 wants to merge 3 commits into
Closed
garrett361 wants to merge 3 commits into
garrett361 wants to merge 3 commits into
Conversation
garrett361
marked this pull request as ready for review
October 12, 2026 18:40
garrett361
force-pushed
the
feat/online-fp8-weight-rounding
branch
from
October 12, 2026 18:58
56236a6 to
5c2f018
Compare
…FP8 weight scales vLLM's online block-FP8 methods quantize weights with the scale amax / 448. Dequantized FP8 releases such as the DeepSeek V4 bf16 checkpoint already sit on the e4m3 grid under power-of-two block scales, so that scale choice injects the full e4m3 rounding error on weights that would otherwise round-trip exactly. The new opt-in field forces use_ue8m0=True in per_block_cast_to_fp8 through the PRIME_FP8_UE8M0_WEIGHT_SCALES env var. The option is independent of VLLM_USE_DEEP_GEMM_E8M0: vLLM's E8M0 requant finds power-of-two scales already in place and leaves the weights alone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ne FP8 weights Round-to-nearest pins a weight that starts on the e4m3 grid until the trainer has moved it half a bin, about 1000 steps at lr 1e-6, so the served policy stays at step 0 while the trainer drifts. Rounding each weight to its far neighbour with probability equal to the fractional distance makes the served weight unbiased and lets it track sub-bin updates in expectation. On-grid input rounds to nearest, so the checkpoint quantizes to identical bytes. The wrapper is registered after the UE8M0 one, so it composes with it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…fect fp8_ue8m0_weight_scales and fp8_stochastic_weight_rounding patch vLLM's online per-block FP8 caster, so they are silent no-ops unless vLLM quantizes linear or MoE weights online to per-block FP8. InferenceConfig now raises in that case, resolving the quantization shorthand and the quantization_config passthrough the way vLLM's resolve_quantization_config does. RLConfig also rejects fp8_stochastic_weight_rounding with an FP8 trainer (dense FP8 or DeepGEMM FP8 MoE compute). The trainer re-quantizes its weights with round-to-nearest every step, so stochastic rounding on the inference side only adds trainer/inference mismatch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
garrett361
force-pushed
the
feat/online-fp8-weight-rounding
branch
from
October 12, 2026 19:02
5c2f018 to
64c3a04
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Online FP8 serving rounds bf16 weights to nearest with
amax / 448block scales, which re-rounds dequantized FP8 releases and pins served weights under small RL updates. This adds two opt-ininferenceoptions, one for each problem.fp8_ue8m0_weight_scalesuses power-of-two block scales, so a dequantized FP8 checkpoint round-trips exactly.fp8_stochastic_weight_roundingrounds without bias, so served weights track sub-grid updates in expectation.🤖 Generated with Claude Code