Skip to content

WIP - SGL B200 FP8 8k1k #2866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cquil11 merged 7 commits into from
Sep 8, 2026
Merged

WIP - SGL B200 FP8 8k1k #2866

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ca1183f
perf(qwen3.5): refresh B200 FP8 SGLang tuning
kedarpotdar-nv Sep 7, 2026
19cd971
perf(qwen3.5): expand B200 FP8 search space
kedarpotdar-nv Sep 8, 2026
013eed1
perf(qwen3.5): sync B200 SGLang tuning args
kedarpotdar-nv Sep 8, 2026
b450926
perf: add Qwen3.5 B200 SGLang changelog
kedarpotdar-nv Sep 8, 2026
9318386
perf(qwen3.5): simplify B200 sweep ranges
kedarpotdar-nv Sep 8, 2026
e324b92
Merge branch 'main' into qwen-b200-sgl-fp8-v4
cquil11 Sep 8, 2026
2cb20fb
Update perf-changelog.yaml
cquil11 Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions benchmarks/single_node/fixed_seq_len/qwen3.5_fp8_b200.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,14 @@ PYTHONNOUSERSITE=1 python3 -m sglang.launch_server --model-path=$MODEL --host=0.
--quantization fp8 \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--mamba-full-memory-ratio 0.37 \
--linear-attn-prefill-backend flashinfer \
Comment on lines +43 to +44

@functionstackx functionstackx Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

per written InferenceX guidelines, can u add these flags to the sglang docs so that all users can see the recipes?

@kedarpotdar-nv kedarpotdar-nv Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

--attention-backend trtllm_mha \
--moe-runner-backend flashinfer_trtllm \
--cuda-graph-max-bs $CONC \
--max-prefill-tokens 16384 \
--chunked-prefill-size 16384 \
--mem-fraction-static 0.8 \
--max-prefill-tokens 32768 \
--chunked-prefill-size 32768 \
--mem-fraction-static 0.86 \
--stream-interval 50 \
--scheduler-recv-interval $( [[ $CONC -gt 4 ]] && echo 30 || echo 10 ) \
--tokenizer-worker-num 6 \
Expand Down
6 changes: 5 additions & 1 deletion configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1200,7 +1200,7 @@ dsv4-fp4-b300-sglang-mtp:
- { tp: 4, ep: 1, conc-start: 4, conc-end: 32, spec-decoding: mtp }

qwen3.5-fp8-b200-sglang:
image: lmsysorg/sglang:v0.5.14-cu130
image: lmsysorg/sglang:v0.5.19-cu130
model: Qwen/Qwen3.5-397B-A17B-FP8
model-prefix: qwen3.5
runner: cluster:b200-nscale
Expand All @@ -1212,8 +1212,12 @@ qwen3.5-fp8-b200-sglang:
- isl: 8192
osl: 1024
search-space:
- { tp: 8, conc-list: [1, 2] }
- { tp: 8, conc-start: 4, conc-end: 4 }
- { tp: 4, ep: 1, conc-start: 4, conc-end: 256 }
- { tp: 4, ep: 1, conc-list: [2] }
- { tp: 4, ep: 1, conc-start: 4, conc-end: 512 }
Comment thread
cursor[bot] marked this conversation as resolved.
Resolved
- { tp: 4, ep: 1, conc-list: [320, 384, 448, 640] }
Comment thread
cquil11 marked this conversation as resolved.
Resolved

qwen3.5-fp8-b200-sglang-agentic-mtp:
image: lmsysorg/sglang:v0.5.16-cu130
Expand Down