Add model gpt-oss

[hlu1]

Motivation

Authors @xutizhou, @liz-badada, @zhuofan1123, @linhu-nv from NVIDIA Solution Architect Team

Many thanks to @qsang-nv, @PerkzZheng, @yzh119 for helping integrate attention sink with FlashInfer attention kernels, @dongfengy for helping integrate the Triton MoE kernel.

SGLang now support GPT-OSS. Here is a simple guide provides step-by-step instructions for deploying GPT-OSS with SGLang or running benchmark tests. (Note: Current implementation focuses on functional completeness rather than performance optimization. Community contributions for performance improvements are welcome. Some workarounds exist in the current codebase and will be addressed in future updates.)

Main Modifications

• Add support to model GPT-OSS
• Model weight loading adaptation
• Integrated Triton MoE support (data types: especially MXFP4_W4A16, MXFP4_W4A8)
• Added multiple attention backends to support SWA with sinks (Flashinfer attention backend; Torch native attention backend)
• Added accuracy tests (GSM8K, MMLU, etc.) and reasoning mode as well as system_messages support.
• Args: --attention-backend [flashinfer, torch_native_sink], --enable-fp8-act for MoE MXFP4 W4A8
• Others

Prerequisites

• Hopper / Blackwell GPUs
• Docker on Hopper: lmsysorg/sglang:latest, on Blackwell: lmsysorg/sglang:v0.4.8-cu128-b200
• Transformers: newly released version
• Flashinfer: newly released version
• Triton 3.4.0.
• Sgl-kernel 0.3.1
• Torch 2.8

cd path-to/sglang
git submodule init && git submodule update

Model Weights

Get model weights from HuggingFace Model Card

• GPT-OSS-120B
• GPT-OSS-20B

Command Example

To serve GPT-OSS model, refer to the following command:

Launch Server

python3 -m sglang.launch_server \
    --model-path <path-to-gpt-oss> \
    --tp-size 4 \
    --enable-fp8-act \
    --cuda-graph-bs 256 \
    --max-running-requests 256 \
    --attention-backend flashinfer

Bench Serving

python3 -m sglang.bench_serving \
    --backend sglang \
    --dataset-name random \
    --num-prompts 6144 \
    --random-input-len 1000 \
    --random-output-len 2000 \
    --random-range-ratio 1

Note:

• To enable MXFP4 W4A8 triton MoE, try --enable-fp8-act.

Bench offline

To bench GPT-OSS model offline, refer to the following command:

python3 -m sglang.bench_offline_throughput \
        --model-path <path-to-gpt-oss>  \
        --tp-size 4 \
        --num-prompts 6144 \
        --enable-fp8-act \
        --cuda-graph-bs 256 \
        --dataset-name random \
        --random-input 1000 \
        --random-output 2000\
        --random-range-ratio 1 \
        --max-running-requests 256 \
        --attention-backend [flashinfer]

Simple Test

python3 test/srt/models/test_gpt_oss_models.py

Note

If meet with some issues on Blackwell, refer #7227

Modifications

Accuracy Test

Benchmark & Profiling

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[Edenzzzz]

Remove?

[Edenzzzz]

seems not needed, if not >= 0 flashinfer will not use sliding window

[merrymercy]

Summary:

• We will be based on Add initial support for gpt-oss #8824 going forward
• merge flashinfer_backend support to this PR Add initial support for gpt-oss #8824
• try vllm mxfp4 moe kernel instead of the one in this PR

[chriswritescode-dev]

So this will not work on Ada series GPUs?

[hlu1]

So this will not work on Ada series GPUs?

We never tested on Ada. Feel free to try it.

[chriswritescode-dev]

So this will not work on Ada series GPUs?

We never tested on Ada. Feel free to try it.
sglang/python/sglang/srt/model_loader/loader.py", line 140, in _get_quantization_config
raise ValueError(
ValueError: The quantization method mxfp4 is not supported for the current GPU. Minimum capability: 90. Current capability: 89.

Nope I guess its due to mxfp4. Any plans for this ?

[zhyncs]

@chriswritescode-dev please refer to #8833

[zhyncs]

Thank you for your contributions. Since we have already received support in pull requests #8824 and #8843, we will now close this PR.

[yiakwy-xpu-ml-framework-team]

hi @hlu1 great work!

May I know the reason to swizzle a tensor outside of main gpu kernel ? Swizzle is realted to shared memory layout, and it has nothing to do with its gpu main memory layout. I am really interested why this helps ?

[hlu1]

Swizzling is very important for the kernel performance for block scale formats on Hopper and Blackwell. It's required to get optimal global memory access.