perf(recipe): add Qwen3.5 text H100 recipe

[yaoyu-33]

What does this PR do?

Adds the canonical text-only H100 performance recipe for
Qwen/Qwen3.5-35B-A3B, focused tests, a verified H100 performance item in the
model verification card, and reusable performance skill/campaign guidance.

This PR is stacked on the public runtime pin in #5449, which pins the current
Megatron-Core integration head published in NVIDIA/Megatron-LM#6370. The
model-card run used ancestor
606c01046bf411c54e730739844da9e613f84a76; later commits update tests and
repair only the inactive deterministic torch fallback's FLA 0.4.2 l2norm
call. The measured FlashQLA path is unchanged.

Frozen configuration

• Hardware/precision: 2 nodes, 16 H100s, BF16 compute
• Workload: sequence length 4096, MBS1, GBS1024, 50 optimizer steps
• Parallelism: TP1/PP1/CP1/EP8/ETP1
• Optimizer memory: precision-aware Adam, FP16 main parameters, BF16 gradients
  and moments, no parameter remainders
• GDN: FlashQLA 0.1.2, fused pre-GDR path, fused gated RMSNorm
• MoE: Hopper torch.grouped_mm, HybridEP, static capacity 1.05, 64-token
  chunks, 16 dispatcher SMs, 108 preprocessing SMs, and shared-expert overlap
• Execution: eager mode, no activation recompute, no DP/parameter-gather or
  combined EP communication overlap
• Runtime hygiene: rank-local TileLang, Triton, TorchInductor, and HybridEP JIT
  caches, plus a shared Torch extension build cache

The recipe retains the model-native local aux_loss coefficient and gradient
clipping. Forced expert balancing and mock data make this a performance
stability benchmark, not convergence evidence.

The TE cross-entropy path is intentional for this mock-data performance
recipe; it does not change the functional-training coverage, which remains
unverified. The card records the Hugging Face repository's top-level
Qwen3_5MoeForConditionalGeneration architecture, while the recipe selects
the supported text-only causal-LM provider.

Reproduce

git checkout fdc1ac339303330c1a73eaa3d1d9d4bab70c776b
git submodule update --init 3rdparty/Megatron-LM
./scripts/training/train.sh \
  --nodes 2 --gpus-per-node 8 \
  --recipe qwen35_text_35b_a3b_pretrain_16gpu_h100_bf16_config \
  --mode pretrain --max_steps 50 \
  logger.save_config_filepath=work/model-verification/qwen3.5-35b-a3b/h100-performance/resolved-config.yaml

The recipe-specific kernel overlay is required and hash-locked in
examples/model_verification_cards/qwen3.5-35b-a3b/runtime-requirements.txt.
Install it without replacing the base container's Torch stack before running
the card command:

uv pip install --system --no-deps --require-hashes \
  -r examples/model_verification_cards/qwen3.5-35b-a3b/runtime-requirements.txt

The caller supplies normal site-specific Slurm, container, credential, cache,
and storage wiring outside the card.

Verified result

A fresh public checkout of Bridge fdc1ac339303330c1a73eaa3d1d9d4bab70c776b
and MCore 606c01046bf411c54e730739844da9e613f84a76 completed the exact 50-step
card command. Steps 41-50 averaged 20,966.100 ms / 280.970
model TFLOP/s/GPU. LM loss was finite from 12.830670 to
0.008044560, skipped=0, NaN=0, the resolved post-setup configuration
was persisted, and every Slurm step exited 0:0.

The earlier frozen candidate reached 285.18 model TFLOP/s/GPU; the card uses a
portable 280 model TFLOP/s/GPU final-10 gate to tolerate ordinary node-to-node
variance.

The card remains pinned to fdc1ac339303330c1a73eaa3d1d9d4bab70c776b,
the exact public commit used by the accepted run. Later equal-input HybridEP
hardening and EOS-only launcher experiments were removed from the final
stack. The current recipe includes only a CI-required static declaration
rewrite for rank-local cache environment variables; its resolved runtime
environment and performance path are equivalent to the measured source, but
the source text is not byte-for-byte identical.

Compared with the previous public 24,561.870 ms / 239.820 TFLOP/s/GPU result,
the frozen candidate reduced step time by 15.90% and increased throughput by
18.91%.

Optimizations retained

• FlashQLA Hopper GDN backend plus streamed pre-GDR fusion
• fused GDN output gated RMSNorm (about +1.37% in the matched A/B)
• Hopper torch.grouped_mm experts with GPU-resident offsets
• HybridEP static capacity and tuned chunk/SM settings
• shared-expert overlap and local auxiliary balancing
• EP8 optimizer sharding plus FP16 precision-aware optimizer main parameters
• isolated compiler/JIT caches and a shared Torch extension build cache for
  deterministic multi-rank startup

Rejected experiments are intentionally not in the frozen recipe: FP8, MBS2,
GDN recompute, CPU offload, combined EP overlap, DP/parameter-gather overlap,
CUDA graphs, global fused residual RMSNorm, NCCL EP, and alternate chunk/SM
settings were slower, unstable, or exceeded memory.

Public PR stack

• refactor: extract and split common logic between GDN & GDN2 NVIDIA/Megatron-LM#5843: merged GDN/GDN2 refactor base
• Bump FLA to 0.5.1 and integrate GDN2 kernels NVIDIA/Megatron-LM#5765: GDN2/FLA integration ancestry
• [dev] moe(perf): Add FlashQLA backend for Hopper/SM90 gated delta net (GDN). NVIDIA/Megatron-LM#5564: source for the adapted FlashQLA Hopper backend
• [main] moe(perf): Pre-GDR kernel fusion NVIDIA/Megatron-LM#5532: source for the adapted streamed pre-GDR fusion
• perf(gdn): preserve Qwen3.5 H100 integration stack NVIDIA/Megatron-LM#6370: exact public integration used by this card
• build: pin Qwen3.5 H100 runtime stack #5449: Bridge MCore pin and hash-locked runtime overlay

The #6370 integration carries adaptations of NVIDIA/Megatron-LM#5982 and
NVIDIA/Megatron-LM#5396. The former is inactive because the accepted recipe
does not recompute; the latter's separate Q/K normalization path is bypassed
by pre-GDR fusion. Neither is credited for the result.

Validation

• exact 50-step 16xH100 card gate: 280.970 model TFLOP/s/GPU over
  steps 41-50, finite metrics, zero skipped/NaN iterations, exit 0:0
• focused Qwen3.5 recipe/runtime unit tests
• model-card schema and privacy validation
• Linux uv lock --check and all-extras dry-run
• uv run pre-commit run --all-files
• independent reproducibility, code, and PR-stack audits

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[svcnvidia-nemo-ci]

/nvskills-ci

[yaoyu-33]

/ok to test a767ee3

[claude]

Light review — Qwen3.5-35B-A3B H100 perf recipe + skill updates

Solid, well-documented addition. The recipe wiring is correct: qwen35_text_35b_a3b_pretrain_16gpu_h100_bf16_config calls the shared base qwen35_text_35b_a3b_pretrain_config (resolving to recipes/qwen/gb200/qwen35.py), overrides the H100 topology, and the unit test patches the correct module (recipes.qwen.gb200.qwen35) where AutoBridge/AutoConfig are looked up. set_cuda_graph_modules / cuda_graph_module_names are used correctly, and _benchmark_common sets train_iters=50 matching the test assertion.

A few small things to consider (none merge-blocking):

1. Perf gate reuses the Qwen3-30B number (Question). examples/model_verification_cards/qwen3.5-35b-a3b/card.yaml:215 sets the threshold to "at least 287.305 model TFLOP/s/GPU". That 287.305 value is the verified Qwen3-30B result (qwen3-30b-a3b/card.yaml:515 and the perf skills). The narrative correctly explains the best Qwen3.5 run reached only 225.8-225.9 (~21.4% below), so the item stays unverified — but pinning the gate to a different model number reads like a copy-paste. Confirm this is an intentional aspirational target, not a placeholder.

2. cross_entropy_fusion_impl silently flips to te (Low). The base recipe deliberately sets cross_entropy_fusion_impl = "native" (gb200/qwen35.py:66) with a comment that the TE path "currently warns about stability". The new perf recipe calls _benchmark_common(cfg) with the default cross_entropy_impl="te", reverting to the flagged path. This matches the qwen3_moe perf recipes convention, so likely intentional for throughput — but worth a one-line note, since the base recipe explicitly chose otherwise.

3. Card architecture field (Question). The card lists architecture: Qwen3_5MoeForConditionalGeneration while the summary and recipe are text-only (recipe overrides to Qwen3_5MoeForCausalLM). Consistent with how the gemma-4/minimax text-only cards document the HF repo top-level VLM architecture, so probably fine — just confirm intent.

The skill/card documentation updates are thorough, internally consistent, and cross-referenced. The moe_flex_dispatcher_num_sms=32 (recipe) vs moe_hybridep_num_sms=32 (_benchmark_common) fields are distinct and both handled by the migration guard in model_load_save.py.

Suggested test cases

• qwen35_text_35b_a3b_16gpu_h100_bf16_perf

Only a single BF16 / H100 / 16-GPU perf recipe is added; no fp8_cs, fp8_mx, or nvfp4 variants exist for this model, so no other perf cases are impacted. The added unit test test_qwen35_text_35b_a3b_h100_bf16_perf_recipe and the flat-recipe instantiation path in test_perf_config_integration.py cover the new recipe.

[svcnvidia-nemo-ci]

/nvskills-ci

[svcnvidia-nemo-ci]

/nvskills-ci

[svcnvidia-nemo-ci]

/nvskills-ci

[svcnvidia-nemo-ci]

/nvskills-ci

[svcnvidia-nemo-ci]

/nvskills-ci

[yaoyu-33]

/ok to test 14fe1a7

[yaoyu-33]

/ok to test 1f21e3f1cba9ef82db33bb411a8f179b19dad557

[copy-pr-bot]

/ok to test 1f21e3f1cba9ef82db33bb411a8f179b19dad557

@yaoyu-33, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

[yaoyu-33]

/ok to test 1f21e3f

[yaoyu-33]

/ok to test 07e7bca

[yaoyu-33]

/ok to test d414104

[yaoyu-33]

/ok to test 7e4a6d1

[svcnvidia-nemo-ci]

/nvskills-ci

[claude]

Light review — LGTM with minor notes

Solid, well-scoped PR: a fail-closed H100 perf recipe + model-specific runtime patch, extensively unit-tested. The runtime patch validates every assumption (SM90, 128-dim heads, BF16-only, SwiGLU, static rank capacity, GPU-resident counts) and raises rather than silently degrading, and the recipe only layers benchmark overrides onto the existing GB200 base config. Revision pins are consistently applied and asserted in tests.

Notes (non-blocking):

• Test gap — cache-env fallback branch. _qwen35_rank_local_kernel_cache_env (perf_recipes/qwen/h100/qwen35.py:35) has a fallback path (NEMO_HOME / XDG_CACHE_HOME / /tmp for the cache root, and RANK / default "0" for the rank) that no test exercises — test_qwen35_text_35b_a3b_h100_bf16_perf_recipe only sets MBRIDGE_KERNEL_CACHE_DIR and SLURM_PROCID. A quick monkeypatch.delenv(...) variant asserting the derived NEMO_HOME/megatron-bridge/qwen35-h100/rank-0/... paths would lock down the default-path logic that real launches without MBRIDGE_KERNEL_CACHE_DIR will hit.
• Unverifiable locally. The patch depends on MCore-internal hooks (GatedDeltaNet._apply_gated_norm, token_dispatcher.check_over_budget, _comm_manager.setup_metadata) that live in the pinned submodule (not checked out here). These are correctly the domain of the documented end-to-end run on the pinned MCore commit; flagging only so a reviewer with the submodule confirms those attribute/method names still match the pin.

Suggested test cases

• test_qwen35_text_35b_a3b_h100_bf16_perf_recipe (extend: assert fallback cache paths when MBRIDGE_KERNEL_CACHE_DIR and SLURM_PROCID/RANK are unset)
• test_qwen35_h100_perf_spec_replaces_grouped_expert_runtime
• test_perf_recipe_factory_builds_config[qwen35_text_35b_a3b_pretrain_16gpu_h100_bf16_config]
• test_all_perf_recipe_factories_are_exported

No functional/L-tier perf launcher targets a qwen35_*gpu_h100_bf16_perf case; coverage for this recipe is the unit tests above (the H100 perf run is validated via the model-verification card command, not a CI perf test).