Skip to content

[skills] chore: clean up nemo mbridge skill references #4044

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cuichenx merged 1 commit into from
May 29, 2026
Merged

[skills] chore: clean up nemo mbridge skill references #4044

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion skills/nemo-mbridge-perf-cuda-graphs/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ def _delete_cuda_graphs(cuda_graph_helper):
recompute (`recompute_granularity="selective"` + `recompute_modules`),
disable CUDA graphs, or switch to `local` + `full_iteration`. Enforced
in MCore `transformer_config.py:2001-2005`. See also
@skills/perf-activation-recompute/SKILL.md.
@skills/nemo-mbridge-perf-activation-recompute/SKILL.md.

12. **Benchmark numbers are workload-specific**: graph wins are usually real
when host overhead is visible, but the exact gain depends on batch shape,
Expand Down
2 changes: 1 addition & 1 deletion skills/nemo-mbridge-perf-cuda-graphs/card.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ failure_modes:
- name: full_recompute_with_te_scoped
symptom: "AssertionError: full recompute is only supported with full iteration CUDA graph"
likely_cause: recompute_granularity=full with any TE-scoped graph (attn, mlp, moe_router, etc.). Common on FP8 CS configs that default to cuda_graph_impl=transformer_engine + scope=mlp.
fix: use recompute_granularity=selective with recompute_modules, or disable CUDA graphs (cuda_graph_impl=none), or switch to cuda_graph_impl=local + cuda_graph_scope=full_iteration. See skills/perf-activation-recompute/SKILL.md.
fix: use recompute_granularity=selective with recompute_modules, or disable CUDA graphs (cuda_graph_impl=none), or switch to cuda_graph_impl=local + cuda_graph_scope=full_iteration. See skills/nemo-mbridge-perf-activation-recompute/SKILL.md.
- name: packed_sequences_with_te_scoped
symptom: "AssertionError: CUDA graph accepts only Tensor inputs. packed_seq_params excluded"
likely_cause: packed_sequence=True passes a non-Tensor packed_seq_params input into TE-scoped capture
Expand Down
8 changes: 4 additions & 4 deletions skills/nemo-mbridge-perf-memory-tuning/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ When a training run OOMs or is close to the memory limit:
fragmentation-induced OOM with zero performance cost. Most Slurm launch
templates already include it.
2. **Add selective activation recompute** (`recompute_modules=[core_attn]`) if
not already enabled. See @skills/perf-activation-recompute/SKILL.md.
not already enabled. See @skills/nemo-mbridge-perf-activation-recompute/SKILL.md.
3. **Avoid increasing TP** as a memory fix — doubling TP dramatically increases
NVLink all-reduce volume and often kills throughput (-28% on Llama3 70B).
4. **Avoid increasing PP at the cost of DP** — halving DP doubles gradient
Expand Down Expand Up @@ -92,11 +92,11 @@ If the model genuinely does not fit (not fragmentation), adjust parallelism:
| Increase PP (keeping DP) | Fewer layers per stage | Moderate (~6% if DP halved) | Only if GPU count allows |
| Increase TP | Fewer params per GPU | Severe (-28% on 70B) | Last resort |
| Distributed optimizer | Shards optimizer state across DP ranks | ~1-2% | Recommended for large models |
| FSDP | Shards params + grads + optimizer | Varies | See @skills/perf-megatron-fsdp/SKILL.md |
| FSDP | Shards params + grads + optimizer | Varies | See @skills/nemo-mbridge-perf-megatron-fsdp/SKILL.md |

### Activation recompute

See @skills/perf-activation-recompute/SKILL.md for full details.
See @skills/nemo-mbridge-perf-activation-recompute/SKILL.md for full details.

### CPU offloading

Expand Down Expand Up @@ -175,7 +175,7 @@ offloading.` This approach is blocked for any model using PP > 1.

Selective activation recompute with `mlp` saved ~3 GB peak memory but cost
~16% GPU utilization on this workload. See
@skills/perf-activation-recompute/SKILL.md for full results.
@skills/nemo-mbridge-perf-activation-recompute/SKILL.md for full results.

## Code Anchors

Expand Down
2 changes: 1 addition & 1 deletion skills/nemo-mbridge-perf-moe-long-context/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ For long-context MoE training:
Useful references:

- @docs/training/activation-recomputation.md
- @skills/perf-cuda-graphs/SKILL.md
- @skills/nemo-mbridge-perf-cuda-graphs/SKILL.md

## Pitfalls

Expand Down
2 changes: 1 addition & 1 deletion skills/nemo-mbridge-perf-moe-optimization-workflow/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ or tightly controlled static-shape experiments.

Related references:

- @skills/perf-cuda-graphs/SKILL.md
- @skills/nemo-mbridge-perf-cuda-graphs/SKILL.md
- @docs/training/cuda-graphs.md
- @docs/training/activation-recomputation.md

Expand Down
2 changes: 1 addition & 1 deletion skills/nemo-rl-e2e-testing/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -465,7 +465,7 @@ If an attach helper enters a container that no longer sees the expected checkout

Attach helpers that use `--no-container-mount-home` can enter a minimal `/home/$USER` in follow-up steps even when the original run saw the real checkout. Keep metric dumping and assertions in the same container step as the run when possible. If a follow-up step must inspect compute-local artifacts, use paths under the node-local run directory and do not assume `$NEMO_RL_REPO` is visible.

For general Slurm debugging and multi-node patterns, read @skills/multi-node-slurm/SKILL.md.
For general Slurm debugging and multi-node patterns, read @skills/nemo-mbridge-multi-node-slurm/SKILL.md.

## Pass Criteria

Expand Down
4 changes: 2 additions & 2 deletions skills/verl-e2e-testing/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,7 @@ bash tests/special_e2e/run_ppo_trainer_megatron.sh \
++critic.megatron.override_transformer_config.gradient_accumulation_fusion=False
```

For Bridge-native FSDP behavior and constraints, also read @skills/perf-megatron-fsdp/SKILL.md.
For Bridge-native FSDP behavior and constraints, also read @skills/nemo-mbridge-perf-megatron-fsdp/SKILL.md.

## Convergence / Learning Signal

Expand Down Expand Up @@ -431,7 +431,7 @@ If an attach helper enters a container that no longer sees the expected checkout

On CUDA/H100 clusters, some launchers set both `CUDA_VISIBLE_DEVICES` and ROCm variables such as `ROCR_VISIBLE_DEVICES`. If verl workers fail before model construction with `Please don't set ROCR_VISIBLE_DEVICES when HIP/CUDA_VISIBLE_DEVICES is set`, fix the launcher/container environment or apply a temporary local verl workaround that drops `ROCR_VISIBLE_DEVICES` when CUDA is already set. Do not report this as a Bridge provider failure.

For general Slurm debugging and multi-node patterns, read @skills/multi-node-slurm/SKILL.md.
For general Slurm debugging and multi-node patterns, read @skills/nemo-mbridge-multi-node-slurm/SKILL.md.

## Pass Criteria

Expand Down
Loading