Skip to content

[NV] update qwen3.5 b200 configs #169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Fridge003 merged 4 commits into from
Feb 24, 2026
Merged

[NV] update qwen3.5 b200 configs #169

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4bbeb28
update qwen3.5 b200
kedarpotdar-nv Feb 19, 2026
1fa00ad
align argument format and remove placeholder variables
faradawn Feb 20, 2026
e3efbac
simplify code
faradawn Feb 20, 2026
05f8dea
add b200 config for mtp and non-mtp for qwen3.5
faradawn Feb 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions src/components/autoregressive/Qwen35ConfigGenerator/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,29 +50,28 @@ const Qwen35ConfigGenerator = () => {
{ id: 'disabled', label: 'Disabled', default: false },
{ id: 'enabled', label: 'Enabled', default: true }
],
commandRule: (value) => value === 'enabled' ? '--speculative-algo NEXTN \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4' : null
commandRule: (value) => value === 'enabled' ? '--speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4' : null
}
},

modelConfigs: {
h100: { bf16: { tp: 16, mem: 0.8 } },
h200: { bf16: { tp: 8, mem: 0.8 } },
b200: { bf16: { tp: 8, mem: 0.8 } }
b200: { bf16: { tp: 8, mem: 0.82 } }
},

generateCommand: function (values) {
const { hardware } = values;
const { hardware, speculative } = values;

const modelName = `${this.modelFamily}/Qwen3.5-397B-A17B`;

const hwConfig = this.modelConfigs[hardware].bf16;
const tpValue = hwConfig.tp;
const memFraction = hwConfig.mem;

// Initialize the base command
let cmd = 'python -m sglang.launch_server \\\n';
cmd += ` --model ${modelName}`;

// TP setting
cmd += ` \\\n --tp ${tpValue}`;

// Apply commandRule from all options
Expand All @@ -85,7 +84,18 @@ const Qwen35ConfigGenerator = () => {
}
});

// Memory fraction based on hardware
// Append B200-specific backend configurations
if (hardware === 'b200') {
cmd += ` \\\n --attention-backend trtllm_mha`;
cmd += ` \\\n --moe-runner-backend flashinfer_trtllm`;
cmd += ` \\\n --disable-radix-cache`;
cmd += ` \\\n --enable-flashinfer-allreduce-fusion`;
if (speculative === 'disabled') {
cmd += ` \\\n --tokenizer-worker-num 6`;
Copy link
Copy Markdown
Collaborator

@JustinTong0323 JustinTong0323 Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, how do you get the number of 6?

Copy link
Copy Markdown
Contributor

@faradawn faradawn Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reference: https://github.com/SemiAnalysisAI/InferenceX/pull/758/changes. Line 56 of benchmarks/single_node/qwen3.5_bf16_b200.sh. Let me know if there is any issue or suggestions!

}
}

// Add memory fraction
cmd += ` \\\n --mem-fraction-static ${memFraction}`;

return cmd;
Expand Down