WIP - SGL B200 FP8 8k1k

[kedarpotdar-nv]

Note

Low Risk
Benchmark and config-only changes with no production serving or auth/data-path impact.

Overview

Retunes the Qwen3.5-397B-A17B FP8 on B200 fixed-seq-len (8k/1k) SGLang benchmark and bumps the image from lmsysorg/sglang:v0.5.14-cu130 to lmsysorg/sglang:v0.5.19-cu130.

Changes

benchmarks/single_node/fixed_seq_len/qwen3.5_fp8_b200.sh — server recipe tuning:

• Add --mamba-full-memory-ratio 0.37 and --linear-attn-prefill-backend flashinfer
• Raise --max-prefill-tokens and --chunked-prefill-size from 16384 → 32768
• Raise --mem-fraction-static from 0.8 → 0.86

configs/nvidia-master.yaml — expanded qwen3.5-fp8-b200-sglang 8k/1k search space:

• TP8: conc 1–4 (previously 4 only)
• TP4/EP1: conc 2–512 (previously 4–256), plus explicit points at 320, 384, 448, 640

perf-changelog.yaml — appended an entry recording the image bump, recipe flag changes, and sweep expansion.

All changes are 8k/1k-specific memory/scheduling tuning; model architecture, precision, and parallelism are unchanged.

Validation

• Full sweep + evals: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34175132645 (GSM8K em_strict 0.967 @ conc 320 / 0.971 @ conc 640, TP4 on b200-nscale)
• Merged upstream SGLang cookbook update covering the B200 FP8 FlashInfer linear-attention and MoE backends: docs(cookbook): Qwen3.5 FP8 on B200/B300 — trtllm-gen MoE + symm mem sgl-project/sglang#38374
• Published SGLang Qwen3.5 cookbook: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.5

^{Reviewed by Cursor Bugbot for commit 9318386. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 013eed1. Configure here.}

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34172920687
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=34172920687

[functionstackx]

per written InferenceX guidelines, can u add these flags to the sglang docs so that all users can see the recipes?

[kedarpotdar-nv]

done.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34175132645
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=34175132645

[kedarpotdar-nv]

/reuse-sweep-run 34175132645

[kedarpotdar-nv]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34175132645
• Verified that this PR passes evals. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34175132645
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

• Validation and eval run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34175132645
• Authorized reuse: /reuse-sweep-run 34175132645
• Merged upstream cookbook update covering the B200 FP8 FlashInfer linear-attention and MoE backends: docs(cookbook): Qwen3.5 FP8 on B200/B300 — trtllm-gen MoE + symm mem sgl-project/sglang#38374
• Published SGLang Qwen3.5 cookbook: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.5
• --mamba-full-memory-ratio 0.37, --mem-fraction-static 0.86, the prefill limits, and expanded concurrency range are 8k/1k-specific memory/scheduling tuning and do not alter model architecture, precision, parallelism, or kernel selection.
• Speculative-decoding and agentic-workload criteria are not applicable to this non-speculative fixed-sequence-length submission.
• This PR does not use append-only: true.

Signed: kedarpotdar-nv

[Klaud-Cold]

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — kedarpotdar-nv is a named owner of configs/nvidia-master.yaml; remaining changed paths fall under the catch-all.
✅ Check 1 (sweep on in-PR commit): PASS — head 9318386 has all single-node 8k1k / and both eval / check-runs success (executed, not skipped) in run 34175132645.
✅ Check 2 (evals pass): PASS — GSM8K em_strict 0.967 (conc 320) / 0.971 (conc 640) for qwen3.5 fp8 sglang TP4 on b200-nscale, run on the PR's exact image lmsysorg/sglang:v0.5.19-cu130.
✅ Check 3 (recipe merged & matches): PASS — sgl-project/sglang#38374 is MERGED (2026-09-08) and the published Qwen3.5 cookbook B200 FP8 397B recipe matches all major args: model, tp=4 (tp=8 latency), --enable-symm-mem, --attention-backend trtllm_mha, --moe-runner-backend flashinfer_trtllm, --linear-attn-prefill-backend flashinfer; --quantization fp8 is auto-detected from the FP8 checkpoint. Pre-existing --kv-cache-dtype fp8_e4m3 / --mamba-ssm-dtype bfloat16 are documented on the same cookbook page (FP8-KV-cache tip sanctions manual use; the H200 FP8 command uses the identical combo). --mamba-full-memory-ratio, prefill sizes, --mem-fraction-static 0.86, --disable-radix-cache, scheduler/stream/tokenizer knobs are InferenceX sweep/harness tuning (informational only).
✅ Check 4 (reuse command): PASS — /reuse-sweep-run 34175132645 posted by kedarpotdar-nv (COLLABORATOR).
✅ Check 5 (latest checklist): PASS — sign-off contains every current-template item, all checked.
✅ Check 6 (upstream image & ordering): PASS — lmsysorg/sglang:v0.5.19-cu130 is the upstream SGLang org on established B200; framework is SGLang itself, so ordering is moot.
✅ Check 7 (deprecated models): PASS — Qwen3.5 fp8 single-turn 8k1k is active per MODELS.md (only bf16 recipes and 1k1k/1k8k scenarios are retired).
✅ Check 8 (no architecture hacks): PASS — no --hf-overrides/config edits; changes are kernel-backend and memory/scheduling tuning only.
➖ Check 9 (spec-decode chat template): N/A — no speculative-decoding changes in this PR.
✅ Check 10 (no engine patches): PASS — no patch/heredoc/site-packages edits; only client-side pip install datasets pandas.
➖ Check 11 (agentic golden AL): N/A — no agentic spec-decode changes, and no simulated-acceptance knobs on this non-spec config.
➖ Check 12 (append-only): N/A — the new perf-changelog entry does not use append-only: true.

[kedarpotdar-nv]

@claude please update the PR description

[Klaud-Cold]

Claude finished @kedarpotdar-nv's task in 54s —— View job

---

PR description updated ✅

• Gather context (diff vs origin/main, review thread, validation run)
• Write updated description reflecting the final state of the PR
• Update the PR body

The description now reflects the final state of the branch at head 9318386:

• Image bump: lmsysorg/sglang:v0.5.14-cu130 → v0.5.19-cu130
• Recipe tuning in benchmarks/single_node/fixed_seq_len/qwen3.5_fp8_b200.sh: --mamba-full-memory-ratio 0.37, --linear-attn-prefill-backend flashinfer, prefill limits 16384 → 32768, mem-fraction-static 0.8 → 0.86
• Simplified sweep ranges as landed in the final commit (TP8 conc 1–4; TP4/EP1 conc 2–512 + explicit 320/384/448/640) — the earlier overlapping-range entries flagged by Bugbot are gone
• perf-changelog.yaml entry noted
• Validation section with the passing sweep/eval run (34175132645) and the merged upstream cookbook links from the sign-off

I kept the Bugbot footer line intact. The title still says "WIP" — I left it as-is since only the description was requested; let me know if you'd like that dropped too.

Added to recipe per comment