Skip to content

fix(runners): scope b200-dgxc qwen3.5 fp4 model path to the V2 checkpoint / 修复:将 b200-dgxc 的 qwen3.5 fp4 模型路径限定到 V2 检查点 - #2414

Merged
Ankur-singh merged 1 commit into
qwen3.5-fp4-nvfp4-v2from
fix-2205-b200-qwen35-fp4-model-path
Jul 29, 2026
Merged

fix(runners): scope b200-dgxc qwen3.5 fp4 model path to the V2 checkpoint / 修复:将 b200-dgxc 的 qwen3.5 fp4 模型路径限定到 V2 检查点#2414
Ankur-singh merged 1 commit into
qwen3.5-fp4-nvfp4-v2from
fix-2205-b200-qwen35-fp4-model-path

Conversation

@Ankur-singh

Copy link
Copy Markdown
Collaborator

Targets qwen3.5-fp4-nvfp4-v2 (the branch behind #2205), not main.

Problem

In runners/launch_b200-dgxc.sh at #2205's head, the model-path branch keys only on MODEL_PREFIX + PRECISION:

elif [[ $MODEL_PREFIX == "qwen3.5" && $PRECISION == "fp4" ]]; then
    export MODEL_PATH="/scratch/fsw/models/Qwen3.5-397B-A17B-NVFP4-V2"

But two checkpoints now share that pair:

Config key model: at head
qwen3.5-fp4-b200-sglang, -sglang-mtp nvidia/Qwen3.5-397B-A17B-NVFP4-V2
qwen3.5-fp4-b200-trt, -trt-mtp nvidia/Qwen3.5-397B-A17B-NVFP4

Because the script later does export MODEL="$MODEL_PATH", the two TRT configs would serve V2 weights while publishing results under the old checkpoint name.

It is also runner-dependent: the b200 pool spans b200-cw_*, b200-nb_* and b200-dgxc_*, and only launch_b200-dgxc.sh pins a path. So the same config key served old weights on cw/nb and V2 on dgxc, depending on scheduling.

Origin: 25ef4a1 ("remove trt configs") reverted the TRT keys to NVFP4 but left the launcher on the V2 path — a partial revert.

Fix

Branch on the checkpoint so each key resolves to its own weights. The old-checkpoint arm restores the exact /lustre path used on main.

Verification

Extracted the real resolution chain from the file and ran the configs through it:

Config (model) main before (bug) after (fix)
…NVFP4-V2 (sglang) /lustre/…NVFP4 /scratch/…NVFP4-V2 /scratch/…NVFP4-V2
…NVFP4 (trt) /lustre/…NVFP4 /scratch/…NVFP4-V2 /lustre/…NVFP4

bash -n clean; fp8 and bf16 qwen3.5 arms unaffected.

Because the TRT arm now resolves identically to main, TRT behaviour is unchanged and no re-sweep is needed; #2205's existing sweep covers the sglang keys it actually changes.

Draft because #2205 itself still has open items (a CHANGES_REQUESTED review, and the SGLang cookbook still documenting the pre-V2 checkpoint). Merging this into that branch changes its head SHA, so its sweep evidence will need a /reuse-sweep-run authorization afterwards.

中文:launch_b200-dgxc.sh 的 qwen3.5+fp4 分支仅依据 model-prefix 与 precision 判断,而这两个字段现对应两个检查点,导致 TRT 配置在 b200-dgxc 运行器上加载 V2 权重却以旧检查点名称发布结果。改为按检查点分支;旧检查点恢复 main 上的 /lustre 路径,故 TRT 行为不变、无需重跑 sweep。

…oint / 修复:将 b200-dgxc 的 qwen3.5 fp4 模型路径限定到 V2 检查点

The qwen3.5 + fp4 MODEL_PATH branch in launch_b200-dgxc.sh keyed only on
MODEL_PREFIX and PRECISION, but two checkpoints now share that pair:

  qwen3.5-fp4-b200-sglang / -sglang-mtp  -> nvidia/Qwen3.5-397B-A17B-NVFP4-V2
  qwen3.5-fp4-b200-trt    / -trt-mtp     -> nvidia/Qwen3.5-397B-A17B-NVFP4

Since this script later does `export MODEL="$MODEL_PATH"`, the shared branch
made the two TRT configs serve V2 weights while still publishing results under
the old checkpoint name whenever they landed on a b200-dgxc_* runner
(launch_b200-cw.sh / launch_b200-nb.sh pin no path, so the same key served the
old weights there -- i.e. runner-dependent).

Introduced by 25ef4a1, which reverted the TRT keys to NVFP4 but left the
launcher pointing at the V2 path.

Branch on the checkpoint so each key resolves to its own weights. The
old-checkpoint arm restores the exact /lustre path used on main, so TRT
behaviour is unchanged and needs no re-sweep.

中文:launch_b200-dgxc.sh 中 qwen3.5 + fp4 的 MODEL_PATH 分支仅依据
MODEL_PREFIX 与 PRECISION 判断,而这两个字段现已对应两个不同的检查点,导致
TRT 配置在 b200-dgxc 运行器上会加载 V2 权重,却仍以旧检查点名称发布结果。
改为按检查点分支:V2 走 /scratch,旧检查点恢复为 main 上的 /lustre 路径,
因此 TRT 行为保持不变,无需重跑 sweep。
@Ankur-singh
Ankur-singh merged commit f85d3de into qwen3.5-fp4-nvfp4-v2 Jul 29, 2026
1 check passed
@Ankur-singh
Ankur-singh deleted the fix-2205-b200-qwen35-fp4-model-path branch July 29, 2026 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant