[Feature] NVFP4 Marlin fallback for non-Blackwell GPUs (SM75+)

[Godmook]

Motivation

Related Issue: #19491
NVFP4-quantized models (e.g., nvidia/Llama-3.1-8B-Instruct-NVFP4, nvidia/DeepSeek-V3-0324-FP4, mistralai/Minimax-M2.5-NVFP4) crash immediately on non-Blackwell GPUs because get_min_capability() returned 100.
This forces users on A100/A40/H100/RTX 3090 to fall back to less accurate quantization (AWQ/GPTQ) or switch to vLLM, which already supports this via Marlin fallback. This PR brings equivalent functionality to SGLang.

Key properties:

• Weights remain compressed in FP4 (2 values per byte) — no VRAM explosion unlike naive BF16 dequantization
• FP4 dequantization happens in-flight inside the Marlin kernel via fast bitwise operations during tensor core matmul
• Fully automatic — no user-side flags needed; fallback is triggered based on GPU capability detection
• All kernel changes are gated by w_type == kFE2M1f (FP4 only) — zero impact on existing INT4/INT8/FP8 quantization paths

---

Modifications

Summary

File	Type	Change
marlin_utils_fp4.py	New	Core NVFP4 Marlin utilities (prepare, apply, scale conversion)
test_nvfp4_marlin_fallback.py	New	6 unit tests covering full fallback stack
marlin/marlin_template.h (JIT)	Kernel fix	Fix FP4 scale stride/group mismatch — aligns with vLLM
marlin/marlin_template.h (sgl-kernel)	Kernel fix	Same fix as JIT — identical changes
marlin_moe/marlin_template.h	Kernel fix	Uncomment dequant_fp8_scales() for kFE2M1f; fix global_scale application
marlin_moe/moe_wna16_marlin.cuh	Modified	Guard FP4 kernel instantiation behind SGL_MOE_MARLIN_FP4 macro
moe_wna16_marlin.py	Modified	Add separate FP4 JIT module to avoid compile-time regression for non-FP4
fused_marlin_moe.py	Modified	Add w1/w2_global_scale params; detect FP4 Marlin mode
moe_runner/marlin.py	Modified	Add w13/w2_global_scale fields to MarlinMoeQuantInfo
compressed_tensors_w4a4_nvfp4.py	Modified	get_min_capability 100→75; Marlin fallback in linear path
compressed_tensors_w4a4_nvfp4_moe.py	Modified	get_min_capability 100→75; Marlin fallback in MoE path
modelopt_quant.py	Modified	get_min_capability 100→75; Marlin fallback in linear + MoE path
weight_utils.py	Modified	Explicit FP8/FP4 config routing for modelopt checkpoints
environ.py	Modified	Add SGLANG_FORCE_NVFP4_MARLIN env var
environment_variables.md	Modified	Document new env var

---

1. New: marlin_utils_fp4.py

Core utility module. All Marlin-specific preparation and inference logic is isolated here.

Function	Description
is_fp4_marlin_supported()	Returns True if SM ≥ 75
nvfp4_marlin_process_scales()	Converts per-group FP8-S1E4M3 scales to FP8-S0E5M3 format required by Marlin dequant kernel
nvfp4_marlin_process_global_scale()	Pre-adjusts FP4↔BF16 exponent bias for gptq_marlin_gemm
apply_fp4_marlin_linear()	Runs linear inference via gptq_marlin_gemm with float4_e2m1f scalar type
prepare_fp4_layer_for_marlin()	Repacks linear weights (gptq_marlin_repack), permutes + converts scales, creates workspace
prepare_moe_fp4_layer_for_marlin()	Same as above, applied per-expert for MoE layers

---

2. Kernel Bug Fixes

2a. Linear kernel: marlin/marlin_template.h (JIT + sgl-kernel)

Root cause: SGLang's Marlin kernel had incorrect FP4 scale stride calculations that diverged from vLLM's corrected implementation. The kernel assumed FP16-sized (2-byte) elements for scale strides, but NVFP4 uses FP8 scales (1 byte each). This caused the kernel to read every other scale row and misalign scale-to-weight-group mapping, producing garbage output on all non-Blackwell GPUs.

6 changes, all gated by w_type == kFE2M1f (zero impact on INT4/INT8/FP8 paths):

Variable	Before (FP4 path)	After (FP4 path)	Impact on non-FP4
s_gl_stride	prob_n / 8	prob_n / 16	None (ternary returns 8)
s_sh_stride	16 * thread_n_blocks / 8	16 * thread_n_blocks / 16	None (ternary returns 8)
s_tb_groups	thread_k_blocks / group_blocks / 2	thread_k_blocks / group_blocks	None (ternary returned /1)
s_gl_rd	... / 2 + ...	... + ...	None (ternary returned /1)
s_sh_rd	Special *2 + warp_row%2 block	Shared logic with other types	None (deleted block was FP4-only)
cur_group_id	k_blocks / (group_blocks * 2)	k_blocks / group_blocks	None (ternary returned *1)

These changes align SGLang's Marlin kernel with vLLM's corrected implementation.

2b. MoE kernel: marlin_moe/marlin_template.h

Same As Linear Kernel.

---

3. Fallback Integration — Both Quantization Formats

Both supported NVFP4 quantization formats now have full Marlin fallback:

Format	Files Changed	get_min_capability	Linear	MoE
compressed-tensors W4A4 NVFP4	compressed_tensors_w4a4_nvfp4.py, .._moe.py	100 → 75	✅	✅
ModelOpt NVFP4	modelopt_quant.py	100 → 75	✅	✅

Fallback trigger flow (automatic, no user action needed):

Model Load
└─ get_min_capability() = 75 → SM75+ GPU accepted
 
process_weights_after_loading()
└─ not is_blackwell_supported() AND is_fp4_marlin_supported()
   ├─ prepare_fp4_layer_for_marlin()      # linear layers
   ├─ prepare_moe_fp4_layer_for_marlin()  # MoE layers
   └─ layer.use_marlin_fallback = True
 
Inference (forward pass)
└─ if use_marlin_fallback:
   ├─ apply_fp4_marlin_linear()           # linear: gptq_marlin_gemm
   └─ runner.run(MarlinMoeQuantInfo)      # MoE: moe_wna16_marlin_gemm

Blackwell (SM100+) is completely unaffected — the fallback path is only taken when is_blackwell_supported() returns False. Native FP4 inference continues to use the existing Blackwell-optimized path. Users can force the Marlin fallback on Blackwell for testing via SGLANG_FORCE_NVFP4_MARLIN=1.

---

4. New: test/registered/quant/test_nvfp4_marlin_fallback.py

Test	What it validates
test_is_fp4_marlin_supported	SM capability detection (SM ≥ 75 → True)
test_min_capability_changed	ModelOptFp4Config.get_min_capability() == 75
test_nvfp4_marlin_process_scales	FP8 scale conversion produces no NaN
test_prepare_and_apply_fp4_marlin_linear	Linear layer prepare + inference end-to-end
test_fused_marlin_moe_fp4	MoE Marlin GEMM end-to-end (previously NaN, now fixed)
test_prepare_moe_fp4_layer_for_marlin	MoE weight repacking shape correctness

---

Accuracy Tests

Unit tests (A40, SM86)

All 6 unit tests pass:

$ python test/registered/quant/test_nvfp4_marlin_fallback.py
GPU: NVIDIA A40 (SM86)
SM86: Testing Marlin FP4 fallback (non-Blackwell GPU).
test_is_fp4_marlin_supported ... ok
test_min_capability_changed ... ok
test_nvfp4_marlin_process_scales ... ok
test_prepare_and_apply_fp4_marlin_linear ... ok
test_fused_marlin_moe_fp4 ... ok
test_prepare_moe_fp4_layer_for_marlin ... ok
 
Ran 6 tests in 8.191s
OK

---

Benchmarking and Profiling

This PR is a correctness fix that unblocks a previously unsupported code path. Comparison against native Blackwell FP4 is not applicable since the target hardware is different.

• Memory efficiency: weights remain in FP4 format (same VRAM as native FP4), unlike naive BF16 dequantization which would 4× the weight memory
• Throughput on non-Blackwell GPUs will be lower than native Blackwell FP4 — this is expected and noted in the startup warning. End-to-end throughput benchmarks against AWQ/GPTQ alternatives on A100 are left for follow-up

E2E benchmark (nvidia/Llama-3.1-8B-Instruct-NVFP4 on A100-40GB)

Benchmark	Metric	NVFP4 Marlin (this PR)	BF16 Baseline
GSM8K	Accuracy (CoT, 8-shot)	74.1%	84.5% ✅ Meta official, reproducible
ARC-Challenge	acc / acc_norm (25-shot)	56.3% / 60.8%	83.4% (0-shot) ❌ Unfair comparison — Meta internal 0-shot eval format; community lm-eval 25-shot results are typically ~60% (When I searched on Google, people usually said it is about 60%, but I'm not sure)
HellaSwag	acc / acc_norm (10-shot)	0.5849 / 0.7944	~0.60 / ~0.8777 ⚠️ Not in Meta official report; based on Open LLM Leaderboard community measurements

*BF16 baseline from Meta's published Llama 3.1 8B Instruct results. Note: FP4 quantization inherently trades accuracy for 4× weight compression. Different evaluation protocols (n-shot, prompt format) also affect absolute numbers. The key validation is that output is coherent and numerically reasonable — before this PR, the same model produced garbage text ("ίνα خوش DBHelper troubled WRITE...").

E2E benchmark (nvidia/Qwen3-30B-A3B-NVFP4 on A100-80GB)

Benchmark	Metric	NVFP4 Marlin (this PR)
GSM8K	Accuracy (CoT, 8-shot)	90.1%
GSM8K	Invalid	0.000
GSM8K	Latency	151.5 s
GSM8K	Output throughput	1130.4 token/s
ARC-Challenge	acc (25-shot)	0.6689 ± 0.0138
ARC-Challenge	acc_norm (25-shot)	0.6894 ± 0.0135

"text": "спіль688вати/autoload debaclerobat航空unctionistine(mappedatoi лиш/il Brofern.Delay"

After this PR (A100):

"text": "5. It is not real. To consider that 2 + 2 could"

---

Checklist

• Format your code according to the contribution guide
• Add unit tests — test/registered/quant/test_nvfp4_marlin_fallback.py (6 tests, all passing on A40/SM86)
• Update documentation according to the contribution guide
• Provide accuracy and speed benchmark results
• Follow the SGLang code style

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the compatibility of NVFP4-quantized models by introducing a Marlin kernel fallback for GPUs with compute capability SM75 and above, but below Blackwell (SM100). This change allows users on a wider range of NVIDIA hardware, including A100 and RTX 3090, to leverage the memory efficiency of FP4 quantization without encountering crashes. The implementation ensures that weights remain compressed, and the fallback is entirely automatic. A crucial bug causing NaN outputs in the Mixture-of-Experts (MoE) kernel for FP4 inference has also been addressed, improving the stability and correctness of the system.

Highlights

• NVFP4 Marlin Fallback for Non-Blackwell GPUs: Implemented a fallback mechanism allowing NVFP4-quantized models to run on non-Blackwell GPUs (SM75+), such as A100/A40/RTX 3090, preventing crashes and expanding hardware compatibility.
• Memory Efficiency Preserved: Ensured that weights remain compressed in FP4 format (2 values per byte) in VRAM, avoiding the memory explosion associated with naive BF16 dequantization.
• Automatic Fallback Detection: The system automatically detects GPU capability and triggers the Marlin fallback without requiring any user-side flags.
• MoE Kernel NaN Fix: Resolved a critical bug in the marlin_moe/marlin_template.h kernel where dequant_fp8_scales() was commented out for kFE2M1f, leading to NaN outputs for FP4 MoE inference.
• New Utility Module and Tests: Introduced marlin_utils_fp4.py for core NVFP4 Marlin utilities and test_nvfp4_marlin_fallback.py with 6 unit tests covering the full fallback stack.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• python/sglang/jit_kernel/csrc/gemm/marlin_moe/marlin_template.h
  • Uncommented and clarified the dequant_fp8_scales() function for kFE2M1f to correctly convert FP8 scales to BF16/FP16, resolving NaN output issues in MoE kernels.
• python/sglang/srt/layers/moe/fused_moe_triton/fused_marlin_moe.py
  • Added w1_global_scale and w2_global_scale parameters to fused_marlin_moe.
  • Implemented logic to detect FP4 Marlin mode based on the presence of global scales.
  • Adjusted assertions for scale dtypes to accommodate FP4 Marlin's special float8_e4m3fn format.
  • Modified scalar type assignment to use float4_e2m1f when in FP4 Marlin mode.
  • Passed w1_global_scale and w2_global_scale to the moe_wna16_marlin_gemm kernel.
• python/sglang/srt/layers/moe/moe_runner/marlin.py
  • Added w13_global_scale and w2_global_scale fields to MarlinMoeQuantInfo for FP4 Marlin-specific global scales.
  • Included global_num_experts in MarlinMoeQuantInfo.
  • Passed global_num_experts, w13_global_scale, and w2_global_scale to the fused_marlin_moe function.
• python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4.py
  • Updated get_min_capability() from 100 to 75 to enable Marlin FP4 fallback on older architectures.
  • Added params_dtype attribute to the layer.
  • Integrated Marlin FP4 fallback logic into process_weights_after_loading() for linear layers, including global scale consolidation and weight repacking.
  • Implemented apply_weights() to use apply_fp4_marlin_linear() when Marlin fallback is active.
• python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4_moe.py
  • Modified the constructor to check for is_fp4_marlin_supported() on non-Blackwell GPUs, raising an error if not supported and logging a warning if fallback is used.
  • Updated get_min_capability() from 100 to 75.
  • Added intermediate_size_per_partition to the layer.
  • Integrated Marlin FP4 fallback logic into process_weights_after_loading() for MoE layers, including creating w*_weight_scale_2 and calling prepare_moe_fp4_layer_for_marlin().
  • Modified create_moe_runner() to use MoeRunnerBackend.MARLIN when fallback is active.
  • Implemented apply_weights() to use MarlinMoeQuantInfo and runner.run() for Marlin fallback.
• python/sglang/srt/layers/quantization/marlin_utils_fp4.py
  • Added a new file containing utility functions for NVFP4 Marlin fallback.
  • Included is_fp4_marlin_supported() to check for SM75+ capability.
  • Defined nvfp4_marlin_process_scales() to convert FP8-S1E4M3 scales to FP8-S0E5M3.
  • Implemented nvfp4_marlin_process_global_scale() to pre-adjust global scales for exponent bias.
  • Provided apply_fp4_marlin_linear() for running FP4-quantized linear operations via the Marlin kernel.
  • Created prepare_fp4_layer_for_marlin() to repack linear layer weights and process scales/bias for Marlin.
  • Developed prepare_moe_fp4_layer_for_marlin() to repack MoE layer weights and process scales for Marlin.
• python/sglang/srt/layers/quantization/modelopt_quant.py
  • Updated get_min_capability() from 100 to 75.
  • Added params_dtype attribute to the layer.
  • Integrated Marlin FP4 fallback logic into process_weights_after_loading() for linear layers, including global scale consolidation and weight repacking.
  • Implemented apply() to use apply_fp4_marlin_linear() when Marlin fallback is active.
  • Modified the constructor of ModelOptNvFp4FusedMoEMethod to check for is_fp4_marlin_supported() on non-Blackwell GPUs, raising an error if not supported and logging a warning if fallback is used.
  • Integrated Marlin FP4 fallback logic into process_weights_after_loading() for MoE layers.
  • Modified create_moe_runner() to use MoeRunnerBackend.MARLIN when fallback is active.
  • Implemented apply() for MoE layers to use MarlinMoeQuantInfo and runner.run() for Marlin fallback.
• test/srt/test_nvfp4_marlin_fallback.py
  • Added a new test file for NVFP4 Marlin fallback functionality.
  • Included TestNvfp4MarlinLinear with tests for prepare_and_apply_fp4_marlin_linear and nvfp4_marlin_process_scales.
  • Added TestNvfp4MarlinMoe with tests for fused_marlin_moe_fp4 and prepare_moe_fp4_layer_for_marlin.
  • Included TestFp4MarlinSupport with tests for is_fp4_marlin_supported and get_min_capability_changed.

Activity

• All 6 unit tests were run and passed successfully on an NVIDIA A40 (SM86) GPU.
• Code formatted according to the contribution guide.
• Unit tests added and passed.
• Documentation updated.
• Accuracy and speed benchmark results provided (noting this is a correctness fix, not a performance comparison against native FP4).
• SGLang code style followed.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant feature: a Marlin fallback mechanism for NVFP4-quantized models, enabling them to run on non-Blackwell GPUs (SM75+). This is a great enhancement for broader hardware compatibility. The changes are well-structured, with the core logic encapsulated in the new marlin_utils_fp4.py module, and comprehensive unit tests are included. The PR also contains a critical bug fix for MoE layers that was causing NaN outputs. My review focuses on improving maintainability by refactoring duplicated code and enhancing clarity around some of the complex quantization logic.

[gemini-code-assist]

This block of code for the Marlin FP4 fallback path is identical to the one in python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a4_nvfp4_moe.py. To improve maintainability and reduce code duplication, I recommend refactoring this logic into a shared helper function.

For example, you could add a function like apply_fp4_marlin_moe to python/sglang/srt/layers/quantization/marlin_utils_fp4.py and call it from both places. This would centralize the logic for creating MarlinMoeQuantInfo and running the MoE layer.

Here's a sketch of what the helper could look like:

# In marlin_utils_fp4.py
def apply_fp4_marlin_moe(
    layer: torch.nn.Module,
    runner: "MoeRunner",
    moe_runner_config: "MoeRunnerConfig",
    dispatch_output: "StandardDispatchOutput",
) -> "CombineInput":
    from sglang.srt.layers.moe.moe_runner.marlin import MarlinMoeQuantInfo

    expert_map = None
    global_num_experts = -1
    if hasattr(layer, "dispatcher") and hasattr(
        layer.dispatcher, "local_expert_mapping"
    ):
        expert_map = layer.dispatcher.local_expert_mapping
        if expert_map is not None:
            global_num_experts = moe_runner_config.num_experts

    quant_info = MarlinMoeQuantInfo(
        w13_qweight=layer.w13_weight,
        w2_qweight=layer.w2_weight,
        w13_scales=layer.w13_weight_scale,
        w2_scales=layer.w2_weight_scale,
        w13_g_idx_sort_indices=None,
        w2_g_idx_sort_indices=None,
        weight_bits=4,
        w13_global_scale=layer.w13_weight_scale_2,
        w2_global_scale=layer.w2_weight_scale_2,
        expert_map=expert_map,
        global_num_experts=global_num_experts,
    )
    return runner.run(dispatch_output, quant_info)

[DarkSharpness]

cc @celve if you have any context

[ciprianveg]

cool. thank you for this.

[adhikjoshi]

This is actually great

[DarkSharpness]

/tag-and-rerun-ci

[BBuf]

I think the main concern on the kernel side is test coverage. The new tests mostly check shape/dtype and that the output is not NaN, but they do not compare the new FP4 Marlin path against a reference implementation numerically. For changes this deep in scale indexing/dequant logic, I would feel much better with a correctness test against a trusted baseline, not just a no-NaN check.

[Godmook]

/rerun-failed-ci

[Godmook]

/rerun-failed-ci

[Godmook]

/rerun-failed-ci

[Godmook]

python3 /actions-runner/_work/sglang/sglang/test/registered/quant/test_nvfp4_marlin_fallback.py
.
.

[CI Test Method] TestFp4MarlinSupport.test_is_fp4_marlin_supported
test_is_fp4_marlin_supported (__main__.TestFp4MarlinSupport) ... ok
[CI Test Method] TestFp4MarlinSupport.test_min_capability_changed
[CI Test Method] TestFp4MarlinSupport.test_should_use_fp4_marlin_fallback
test_min_capability_changed (__main__.TestFp4MarlinSupport)
get_min_capability() must return 75 (not 100). ... ok
test_should_use_fp4_marlin_fallback (__main__.TestFp4MarlinSupport)
should_use_fp4_marlin_fallback returns True on non-Blackwell SM>=75. ... ok
test_fake_apply_fp4_marlin_linear (__main__.TestNvfp4MarlinLinear)
Fake impl for PCG tracing must return the correct shape and dtype. ... ok
test_fp4_marlin_3d_input (__main__.TestNvfp4MarlinLinear)
Verify correct reshape for 3-D input (batch, seq_len, K). ... Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
ok
test_fp4_marlin_custom_op_registration (__main__.TestNvfp4MarlinLinear)
apply_fp4_marlin_linear must be registered as torch.ops.sglang for PCG. ... ok
test_fp4_marlin_linear_with_bias (__main__.TestNvfp4MarlinLinear)
Verify output_with_bias == output_no_bias + bias. ... ok
test_fp4_marlin_multiple_shapes (__main__.TestNvfp4MarlinLinear)
Numerical correctness across various (M, N, K) dimensions. ... ok
test_fp4_marlin_numerical_correctness (__main__.TestNvfp4MarlinLinear)
Kernel output vs BF16 dequant reference (cosine sim, MAE, assert_close). ... ok
test_fp4_marlin_registered_op_numerical (__main__.TestNvfp4MarlinLinear)
torch.ops.sglang.apply_fp4_marlin_linear matches the direct Python call. ... ok
test_nvfp4_marlin_scale_values_correctness (__main__.TestNvfp4MarlinLinear)
Verify scale conversion produces analytically correct values. ... ok
test_prepare_and_apply_fp4_marlin_linear (__main__.TestNvfp4MarlinLinear)
Smoke test: shape and dtype are correct after prepare + apply. ... ok
test_prepare_fp4_layer_permutes_bias (__main__.TestNvfp4MarlinLinear)
prepare_fp4_layer_for_marlin must permute layer.bias when present. ... ok
test_prepare_rejects_bad_weight_shape (__main__.TestNvfp4MarlinLinear)
prepare_fp4_layer_for_marlin must raise on mismatched weight shape. ... ok
test_fused_marlin_moe_fp4 (__main__.TestNvfp4MarlinMoe)
Smoke test: shape, dtype, no NaN for multi-expert MoE. ... <frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
ok
test_fused_marlin_moe_fp4_numerical (__main__.TestNvfp4MarlinMoe)
E=1, topk=1 MoE output vs dequant reference (SiLU-gated). ... ok
test_prepare_moe_fp4_layer_for_marlin (__main__.TestNvfp4MarlinMoe)
Weight repacking produces correct shapes for all expert tensors. ... Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel for MoE layers. This may degrade performance for compute-heavy workloads.
ok

----------------------------------------------------------------------
Ran 17 tests in 409.588s

@BBuf All nvpf4 Kernel tests are passed I think. I'll try to fix the other CI fails. Thanks for help!

[Godmook]

/rerun-failed-ci

[Godmook]

/rerun-failed-ci

[Godmook]

/rerun-failed-ci

[Godmook]

/rerun-failed-ci

[Godmook]

/rerun-failed-ci

[Godmook]

Thanks for help! @BBuf. It seems like I made things a bit hectic, so I’m sorry if I bothered you while you were busy. As a big fan of SGLang, I found it really fun! Thank you so much for your help, especially when you’re busy 😊

[Godmook]

@ciprianveg It is merged. You can use it! :)

[ciprianveg]

@ciprianveg It is merged. You can use it! :)

thank you very much for your relentless work here!

[b8zhong]

Merge conflict

[Godmook]

Hi. @b8zhong Thanks for the heads-up and sorry for this problems....
I traced all 16 files against current main and found 5 files with stale merge resolution from my earlier main merge. Here's the full list what I saw:

1. docs/references/environment_variables.md

SGLANG_FLASHINFER_FP4_GEMM_BACKEND line — removed by #21536, I accidentally kept it. I'll fix it.
2. python/sglang/srt/environ.py

SGLANG_HICACHE_MAX_PINNED_RATIO — removed by #21884
SGLANG_ENABLE_MM_SPLITTING — removed by #21899

3. python/sglang/srt/layers/moe/fused_moe_triton/fused_marlin_moe.py

silu_and_mul moved from sgl_kernel → sglang.jit_kernel.activation (#21766)
4. compressed_tensors_w4a4_nvfp4.py

Missing and not get_fp4_gemm_runner_backend().is_cutlass() guard on the flashinfer path
5. modelopt_quant.py

• cutlass_fp4_gemm import changed to top-level try/except
• New CUTLASS FP4 GEMM code path added
• Same .is_cutlass() guard missing

The remaining 11 files are identical to main. I'll rebase on latest main to resolve all of these cleanly If you approve my plan. Really sorry about that...

[b8zhong]

Merge conflict?

[Fridge003]

@Godmook This PR is just reverted due to the merge conflicts. Please rebase your branch and reland with a new PR, thanks!

[Godmook]

@Godmook This PR is just reverted due to the merge conflicts. Please rebase your branch and reland with a new PR, thanks!

Yap. I talked with @b8zhong and I'll firstly do more Benchmark tests and after saw that datas, I think we have to decide it is good to land or not. Thanks for help!