perf(recipe): add Qwen3.5 text H100 recipe

docs/skills-index.md

@@ -58,6 +58,9 @@ skills/nemo-mbridge-perf-activation-recompute/SKILL
 ```{toctree}
 :hidden:
 
+skills/nemo-mbridge-perf-moe-optimization-workflow/references/qwen35-h100-campaign
+skills/nemo-mbridge-perf-moe-dispatcher-selection/references/qwen35-h100-hybridep
+skills/nemo-mbridge-perf-expert-parallel-overlap/references/qwen35-h100-overlap
 skills/nemo-mbridge-perf-nsys-analysis/references/pitfalls
 skills/nemo-mbridge-perf-nsys-analysis/references/sql-recipes
 ```

examples/model_verification_cards/qwen3.5-35b-a3b/card.yaml

@@ -0,0 +1,219 @@
+# Agent-readable model verification card.
+# status: unverified | verified | unsupported | not_applicable
+
+title: qwen35_text_35b_a3b
+summary: >
+  Performance scope: pretrain_performance.H100 uses the canonical text-only
+  Qwen3.5-35B-A3B performance recipe. Functional conversion, inference, and
+  training coverage remains unverified and is not implied by the performance
+  benchmark.
+verification_index:
+  model_level:
+    unverified:
+      - hf_to_megatron_cpu
+      - hf_to_megatron_gpu
+      - megatron_to_hf_cpu
+      - megatron_to_hf_gpu
+      - manual_forward_pass
+      - inference
+  training:
+    H100:
+      unverified: [pretrain, sft, sft_export_inference, sft_long_context, peft, checkpoint_resume]
+  performance:
+    H100: verified
+model:
+  hf_id: Qwen/Qwen3.5-35B-A3B
+  hf_revision: 59d61f3ce65a6d9863b86d2e96597125219dc754  # pragma: allowlist secret
+  architecture: Qwen3_5MoeForConditionalGeneration
+  min_transformers_version: "5.8.1"
+verification_environment:
+  base_container: nvcr.io/nvidia/nemo:26.06
+  bridge_commit: fdc1ac339303330c1a73eaa3d1d9d4bab70c776b  # pragma: allowlist secret
+
+items:
+  hf_to_megatron_cpu:
+    status: unverified
+    precision: bf16
+    command: null
+    last_verified: null
+    expected_result: >
+      A pinned CPU import must exit successfully, persist iter_0000000, and
+      reload the complete text-only tensor set without missing, unexpected,
+      shape-mismatched, dtype-mismatched, or value-mismatched tensors.
+
+  hf_to_megatron_gpu:
+    status: unverified
+    precision: bf16
+    command: null
+    last_verified: null
+    expected_result: >
+      A pinned distributed GPU import must exit successfully, persist
+      iter_0000000, and reload the complete text-only tensor set without
+      missing, unexpected, shape-mismatched, dtype-mismatched, or
+      value-mismatched tensors.
+
+  megatron_to_hf_cpu:
+    status: unverified
+    precision: bf16
+    command: null
+    last_verified: null
+    expected_result: >
+      A pinned CPU export must exit successfully and strictly reload as the
+      supported text-only Hugging Face model with the expected keys, shapes,
+      dtypes, and values.
+
+  megatron_to_hf_gpu:
+    status: unverified
+    precision: bf16
+    command: null
+    last_verified: null
+    expected_result: >
+      A pinned distributed GPU export must exit successfully and strictly
+      reload as the supported text-only Hugging Face model with the expected
+      keys, shapes, dtypes, and values.
+
+  manual_forward_pass:
+    status: unverified
+    precision: bf16
+    command: null
+    last_verified: null
+    expected_result: >
+      Verification requires a matching next-token prediction and cosine
+      similarity of at least 0.99 against the pinned Hugging Face revision,
+      with maximum and mean absolute logit differences recorded as diagnostics.
+
+  inference:
+    status: unverified
+    precision: bf16
+    command: null
+    last_verified: null
+    expected_result: >
+      A pinned Megatron checkpoint must complete deterministic greedy
+      generation for an exact token count and record the literal completion.
+
+  pretrain:
+    H100:
+      status: unverified
+      precision: bf16
+      enabled_features: {}
+      command: null
+      last_verified: null
+      metrics:
+        initial_loss: null
+        final_loss: null
+        last_10_steps_step_time_ms_avg: null
+        last_10_steps_model_tflops_per_gpu_avg: null
+      expected_result: >
+        A public H100 pretraining recipe must complete a bounded 100-step run
+        with finite loss, no skipped or NaN iterations, all four metrics, a
+        persisted post-setup configuration, and reloadable checkpoints.
+
+  sft:
+    H100:
+      status: unverified
+      precision: bf16
+      enabled_features: {}
+      command: null
+      last_verified: null
+      metrics:
+        initial_loss: null
+        final_loss: null
+        last_10_steps_step_time_ms_avg: null
+        last_10_steps_model_tflops_per_gpu_avg: null
+      expected_result: >
+        A public text-only H100 recipe must complete a pinned-data 100-step
+        full-SFT run with finite loss, no skipped or NaN iterations, all four
+        metrics, and a reloadable final checkpoint.
+
+  sft_export_inference:
+    H100:
+      status: unverified
+      precision: bf16
+      depends_on: sft
+      commands: null
+      last_verified: null
+      expected_result: >
+        The final full-SFT checkpoint must export to Hugging Face, reload
+        strictly, and complete deterministic greedy inference for an exact
+        token count.
+
+  sft_long_context:
+    H100:
+      status: unverified
+      precision: bf16
+      enabled_features: {}
+      command: null
+      last_verified: null
+      metrics:
+        initial_loss: null
+        final_loss: null
+        last_10_steps_step_time_ms_avg: null
+        last_10_steps_model_tflops_per_gpu_avg: null
+      expected_result: >
+        A public text-only H100 recipe must complete bounded long-context SFT
+        with finite loss, no skipped or NaN iterations, all four metrics, and
+        verified packing and context-parallel behavior.
+
+  peft:
+    H100:
+      status: unverified
+      precision: bf16
+      enabled_features: {}
+      command: null
+      last_verified: null
+      metrics:
+        initial_loss: null
+        final_loss: null
+        last_10_steps_step_time_ms_avg: null
+        last_10_steps_model_tflops_per_gpu_avg: null
+      expected_result: >
+        A public text-only H100 recipe must complete a pinned-data 100-step
+        PEFT run with finite loss, no skipped or NaN iterations, all four
+        metrics, and a reloadable final adapter checkpoint.
+
+  checkpoint_resume:
+    H100:
+      status: unverified
+      precision: bf16
+      depends_on: pretrain
+      command: null
+      last_verified: null
+      metrics:
+        initial_loss: null
+        final_loss: null
+        last_10_steps_step_time_ms_avg: null
+        last_10_steps_model_tflops_per_gpu_avg: null
+      resume_comparison:
+        reference_item: pretrain
+        sentinel_steps: [51, 100]
+        loss_relative_tolerance: 1.0e-2
+        loss_absolute_tolerance: 1.0e-6
+        sentinels_match: null
+      expected_result: >
+        A direct continuation must restore optimizer, scheduler, data-order,
+        and RNG state from the pretraining midpoint, complete the remaining
+        steps, and pass the declared loss-sentinel tolerance.
+
+  pretrain_performance:
+    H100:
+      status: verified
+      precision: bf16
+      command: >
+        ./scripts/training/train.sh
+        --nodes 2 --gpus-per-node 8
+        --recipe qwen35_text_35b_a3b_pretrain_16gpu_h100_bf16_config
+        --mode pretrain --max_steps 50
+        logger.save_config_filepath=work/model-verification/qwen3.5-35b-a3b/h100-performance/resolved-config.yaml
+      last_verified: 2026-08-08
+      metrics:
+        initial_loss: 12.830670
+        final_loss: 0.008044560
+        last_10_steps_step_time_ms_avg: 20966.100
+        last_10_steps_model_tflops_per_gpu_avg: 280.970
+      expected_result: >
+        The exact command must complete all 50 steps with finite loss,
+        skipped=0, NaN=0, exit 0:0, and a persisted post-setup configuration.
+        The verified run averaged 20966.100 ms and 280.970 model TFLOP/s/GPU
+        over steps 41-50, passing the 280.000 model TFLOP/s/GPU gate. Mock
+        data and forced expert balancing make this performance-stability
+        evidence, not convergence evidence.

skills/nemo-mbridge-multi-node-slurm/SKILL.md

@@ -198,6 +198,29 @@ uv run python -m torch.distributed.run --nproc_per_node=8 \
 | `No space left on device` during `uv` or `pip` | Container's `/root/.cache/` is full | Redirect: `export UV_CACHE_DIR=<SHARED_FS>/uv_cache` |
 | `ModuleNotFoundError: No module named 'megatron.core.activations'` | Container's pre-installed megatron-core conflicts with local `3rdparty/Megatron-LM` | Install local: `pip install -e 3rdparty/Megatron-LM --no-deps --no-build-isolation` |
 
+### QoS fragmentation workaround for diagnostic A/B runs
+
+If an exact 2-node interactive request remains pending under
+`QOSGrpNodeLimit` while lower-priority 1-node jobs repeatedly consume newly
+free nodes, a two-element Slurm array can remove that fragmentation for a
+diagnostic run:
+
+- request one exclusive node per array task with `#SBATCH --array=0-1`
+- let task 0 atomically publish its hostname to a unique file on shared storage
+- make both tasks wait for that file, then launch one
+  `torch.distributed.run` agent per node with `node_rank=$SLURM_ARRAY_TASK_ID`
+- give the first task a bounded rendezvous timeout long enough for the second
+  task to start
+- validate both array-task exit codes and both distributed-step exit codes
+
+This intentionally lets task 0 hold one node while task 1 waits for the next
+free node. Use it only for controlled diagnostics: partial array starts consume
+allocation time, a missing partner must fail closed, and the result does not
+replace a required final run in one atomic multi-node allocation. A matched
+Qwen3.5 16-H100 A/B on 2026-07-27 used this pattern after an atomic 2-node
+request was repeatedly fragmented; both array tasks and both distributed steps
+completed `0:0`.
+
 ---
 
 ## Debugging Multi-Node Failures

skills/nemo-mbridge-perf-activation-recompute/SKILL.md

@@ -1,17 +1,14 @@
 ---
 name: nemo-mbridge-perf-activation-recompute
-description: Validate and use selective and full activation recompute in Megatron Bridge to reduce GPU memory usage at the cost of extra compute.
+description: Validate and use selective and full activation recompute in Megatron Bridge to reduce GPU memory usage at the cost of extra compute. Use for GPU-memory reduction, recompute-related OOM or regression investigation, recompute_granularity, recompute_num_layers, recompute_modules, recompute_method, selective recompute, full recompute, or activation-memory OOM.
 license: Apache-2.0
-when_to_use: Reducing GPU memory via activation recompute, or investigating a commit that changed recompute settings and caused OOM or a regression; 'recompute_granularity', 'recompute_num_layers', 'recompute_modules', 'recompute_method', 'selective recompute', 'full recompute', 'activation memory OOM'.
 ---
 
 # Activation Recompute
 
 Stable docs: @docs/training/activation-recomputation.md
 Card: @skills/nemo-mbridge-perf-activation-recompute/card.yaml
 
-<!-- NVSkills CI refresh: 2026-06-15. No instruction changes. -->
-
 ## What It Is
 
 Activation recompute trades GPU compute for memory by discarding intermediate
@@ -34,6 +31,10 @@ how many layers via `recompute_num_layers`.
    @skills/nemo-mbridge-perf-memory-tuning/SKILL.md.
 2. For activation pressure, start with selective recompute:
    `recompute_granularity="selective"` and `recompute_modules=["core_attn"]`.
+   For a Gated Delta Net model, use `recompute_modules=["gdn"]` as a focused
+   alternative when GDN activations, rather than attention activations, own the
+   peak. Benchmark the exact model because this checkpoints the complete GDN
+   module, not only its recurrent kernel.
 3. Add modules by cost: `"layernorm"` is cheap but saves little, while `"mlp"`
    saves much more memory at a clear throughput cost.
 4. Use full-layer recompute only when selective recompute does not fit, and set
@@ -74,6 +75,7 @@ cfg.model.recompute_num_layers = 4
 | `moe_act` | MoE activation functions | low | small |
 | `shared_experts` | shared expert layers | moderate | moderate |
 | `mla_up_proj` | Multi-Latent Attention up projection | moderate | moderate |
+| `gdn` | complete GatedDeltaNet module: projections, convolution, recurrent rule, normalization, CP exchange, and output projection | architecture-dependent | potentially material on GDN-heavy hybrids |
 
 ### Performance harness CLI
 
@@ -109,6 +111,15 @@ uv run python scripts/performance/run_script.py \
 - `distribute_saved_activations=True` cannot be combined with `sequence_parallel=True`
 - Combining `mlp` + `core_attn` recompute is slightly worse than `mlp` alone
   due to double recompute overhead
+- `gdn` requires
+  `experimental_attention_variant="gated_delta_net"`. It is selective
+  submodule recompute and is distinct from full-layer recompute.
+- Fine-grained MoE expert-parallel overlap rejects full-layer recompute,
+  non-null `recompute_method`/`recompute_num_layers`, and
+  `recompute_modules` containing `moe`. On current MCore, selective non-MoE
+  modules such as `gdn` remain valid. This is a compatibility contract, not a
+  throughput guarantee; validate memory, step time, and numerical health on
+  the exact combined schedule.
 
 ## Measured Results
 
@@ -133,6 +144,34 @@ Key takeaways:
 - For this workload, the actual OOM fix was `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`
   (memory fragmentation, not capacity). See @skills/nemo-mbridge-perf-memory-tuning/SKILL.md.
 
+### Qwen3.5 GDN-heavy EP-overlap result
+
+On 2026-07-26, an exact 2-node/16-H100 Qwen3.5-35B-A3B run paired
+fine-grained EP overlap with `recompute_modules=["gdn"]`. The matched
+owner-stream-release/connections=1 overlap control averaged 44.4646 seconds
+over steps 2-3 and reached 72.028 GiB rank-0 peak allocated memory.
+
+Selective GDN recompute completed all three steps with finite loss and
+gradients and zero skipped/NaN iterations. Steps 2-3 were 25.5641 and 25.1589
+seconds, a 25.3615-second mean (about 232.27 model TFLOP/s/GPU), while
+iteration-2 rank-0 peak allocated/reserved memory fell to 59.516/65.123 GiB.
+This removed about 12.5 GiB of allocated peak and contracted the pathological
+overlap schedule by 42.96%.
+
+The result still ran 13.52% slower by step time than the accepted no-EP-overlap
+control at 22.340925 seconds, so it was rejected as a performance recipe.
+Treat architecture-specific selective recompute as a way to recover a
+memory-bound schedule, not as a guarantee that recompute plus overlap beats
+the simpler schedule.
+
+The recovered headroom can enable another memory feature without making the
+combination faster. Adding active MoE paged stash to the same selective-GDN
+run completed with finite numerics and held iteration-2 rank-0 peak allocation
+to 65.891 GiB, but steps 2-3 averaged 29.98755 seconds. That was 18.24% slower
+than GDN recompute alone because full-schedule stash pack/unpack work outweighed
+the allocator benefit. Validate stacked memory features end to end; capacity
+compatibility does not imply additive throughput benefit.
+
 ## Code Anchors
 
 ### Recompute modules enum and selective checkpoint logic
@@ -169,6 +208,19 @@ Key takeaways:
                     ], "full recompute is only supported with full iteration CUDA graph."
 ```
 
+### GDN selective recompute and EP-overlap validation (MCore)
+
+```text
+3rdparty/Megatron-LM/megatron/core/transformer/transformer_config.py
+3rdparty/Megatron-LM/megatron/core/ssm/gated_delta_net.py
+src/megatron/bridge/training/comm_overlap.py
+```
+
+`TransformerConfig` accepts `gdn` only for the gated-delta-net attention
+variant, and `GatedDeltaNet` uses it to checkpoint the whole module. Bridge and
+MCore EP-overlap validation forbid full/MoE recompute but do not forbid this
+selective GDN scope.
+
 ### CPU offloading PP incompatibility (MCore)
 
 ```1303:1306:3rdparty/Megatron-LM/megatron/core/transformer/transformer_config.py
@@ -187,6 +239,8 @@ Key takeaways:
 | `AssertionError: full recompute is only supported with full iteration CUDA graph` | layer-level recompute with TE-scoped graph capture | check `cuda_graph_impl` and `cuda_graph_scope` | use `selective`, set `cuda_graph_impl=none`, or use `local` + `full_iteration` |
 | ValueError: PP + CPU offloading | `cpu_offloading=True` with `pipeline_model_parallel_size > 1` | check PP config | disable CPU offloading or set PP=1 |
 | mlp+core_attn worse than mlp alone | double recompute overhead | compare Exp 1 vs Exp 2 | use mlp alone |
+| `gdn` rejected during config validation | the model is not using the gated-delta-net attention variant | inspect `experimental_attention_variant` | remove `gdn` or select a module implemented by that architecture |
+| EP-overlap recompute assertion | full-layer recompute, `recompute_method`/`recompute_num_layers`, or full `moe` recompute was enabled | inspect all four recompute fields after overrides | use selective non-MoE scope such as `gdn` and leave method/count null |
 
 ## Known Limitations