Repository navigation
fix(serving): pin cuBLAS/cuDNN to deterministic modes for batch-shape invariance - #44094
Asthenia0412 wants to merge 3 commits into
Conversation
…nd for fp4 on SM100 (Blackwell)
FP4 Marlin is weight-only (W4A16) and does not support W4A4 GEMM.
FlashInfer's mm_fp4 rejects backend='marlin' on SM100+ with:
BackendSupportedError: mm_fp4 does not support backend 'marlin' with capability 100
When the user explicitly passes --fp4-gemm-backend marlin on Blackwell GPUs,
or auto-selection unexpectedly chooses marlin, fall back to flashinfer_cutedsl
(SM100) or flashinfer_cutlass (other SM12x) with a clear warning.
Fixes: #43988
Signed-off-by: yancy <yancy@users.noreply.github.com>
…eterministic-inference Issue #43956 reports that Qwen3.8-27B scores change with batch shape under BF16 and FP8 checkpoints, enough to flip borderline decisions. The root cause is cuBLAS picking a different GEMM reduction order for different M/N/K shapes, which crosses a BF16 rounding boundary on identical inputs. The fix does three things whenever --enable-deterministic-inference is set: 1. Sets CUBLAS_WORKSPACE_CONFIG=:4096:8 so cuBLAS resolves every GEMM call to its deterministic algorithm variant. This is the primary fix and must be applied before cuBLAS is lazily initialised, so the env var is set at the very top of the pipeline's deterministic hook (step 228 of 351). 2. Sets CUDNN_DETERMINISTIC=1 and CUDNN_BENCHMARK=0, plus the equivalent torch.backends.cudnn attributes, to keep cuDNN convolutions consistent across shapes. 3. Calls torch.use_deterministic_algorithms(True, warn_only=True) and pins flash- / mem-efficient-SDP and allow_bf16_reduced_precision_reduction so any remaining PyTorch ops that have shape-dependent behaviour are forced onto deterministic paths. All three env vars are also set in handle_environment_variables (step 318) as a belt-and-braces guard — setdefault means the earlier hook still wins when both run, but any code path that initialises cuBLAS between steps 228 and 318 is still covered. Signed-off-by: yancy <yancy@users.noreply.github.com>
…fp4_utils.py Signed-off-by: yancy <yancy@users.noreply.github.com>
|
🤖 CI gating notice: This PR's checks ( |
|
Hi! The CI currently shows skipped jobs because this repo requires the Could someone please add the @lm-sys @zhyncs @hnyls2002 @ispobock @BBuf @Fridge003 @kevin-mii |
|
Thanks for your interest in SGLang. This account opened 16 PRs in about eight hours, across unrelated subsystems (attention backends, speculative decoding, diffusion loading, scheduler IPC, KV-cache compression, docs), including two PRs for the same change (#44073 and #44117, both for #28808) and a fix for #43929 that is already on main. We can't review contributions at this volume, so we are closing the ones still open. If you want to pursue one, please pick a single change, verify it on a real run (a repro, a test, or numbers), and open a fresh PR. |
Summary
Fixes numerical drift across batch shapes reported in #43956: Qwen3.8-27B scores change enough to flip borderline decisions (FP8 / BF16) when the same token IDs are placed into different batch sizes.
Root cause
The reporter isolated the first projection drift to a pure-PyTorch
torch.nn.functional.linear(BF16 x BF16 → BF16) inmodel.language_model.layers.0.linear_attn.in_proj_ba. A BF16 intermediate dot product was near the rounding boundary and cuBLAS chose a different reduction order for different M/N/K shapes — this crossed the boundary and the difference propagated through the network. With FP8 checkpoints and the default BF16 LM head the drift reached 12.2 percentage points on a 0.5 P(positive) tie.torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction = Falsealone does not help because the problem is reduction order, not reduced-precision reduction.Fix
Wire cuBLAS and cuDNN determinism into the existing
--enable-deterministic-inferencecode path in the argument-resolution pipeline:CUBLAS_WORKSPACE_CONFIG=:4096:8— forces cuBLAS to pick the deterministic algorithm variant for every GEMM call, regardless of M/N/K shape. This is the primary fix and is set before any cuBLAS lazy initialisation.CUDNN_DETERMINISTIC=1/CUDNN_BENCHMARK=0plus the equivalenttorch.backends.cudnnattributes.torch.use_deterministic_algorithms(True, warn_only=True)with flash-/mem-efficient-SDP disabled andallow_bf16_reduced_precision_reduction=Falseso remaining PyTorch ops with shape-dependent behaviour are pinned to deterministic code paths.All three env vars are applied:
handle_deterministic_inference(pipeline step 228 of 351) — the earliest deterministic hook, well before model load or any GEMM.setdefaultinhandle_environment_variables(step 318) as a belt-and-braces guard for code paths that might initialise cuBLAS between the two steps.setdefaultpreserves the earlier hook’s values.Testing
I don’t have access to the reported H20 environment, so verification will need to come from the reporter’s existing reproduction script. The fix is additive to the existing
--enable-deterministic-inferenceflag and a no-op when that flag is not set.Related
Closes #43956.
CI States
Latest PR Test (Base): ❌ Run #38444677950
Latest PR Test (Extra): ❌ Run #38444677764
Latest PR Test (AMD ROCm 10): ❌ Run #38444677940