From 7bc07eb6d28f0052c349581c12dcf97e7014c185 Mon Sep 17 00:00:00 2001 From: Chen Cui Date: Thu, 28 May 2026 17:09:34 -0700 Subject: [PATCH] [skills] clean up nemo mbridge cross references Signed-off-by: Chen Cui --- skills/nemo-mbridge-perf-cuda-graphs/SKILL.md | 2 +- skills/nemo-mbridge-perf-cuda-graphs/card.yaml | 2 +- skills/nemo-mbridge-perf-memory-tuning/SKILL.md | 8 ++++---- skills/nemo-mbridge-perf-moe-long-context/SKILL.md | 2 +- .../nemo-mbridge-perf-moe-optimization-workflow/SKILL.md | 2 +- skills/nemo-rl-e2e-testing/SKILL.md | 2 +- skills/verl-e2e-testing/SKILL.md | 4 ++-- 7 files changed, 11 insertions(+), 11 deletions(-) diff --git a/skills/nemo-mbridge-perf-cuda-graphs/SKILL.md b/skills/nemo-mbridge-perf-cuda-graphs/SKILL.md index c2ea168594..4daa38eae2 100644 --- a/skills/nemo-mbridge-perf-cuda-graphs/SKILL.md +++ b/skills/nemo-mbridge-perf-cuda-graphs/SKILL.md @@ -303,7 +303,7 @@ def _delete_cuda_graphs(cuda_graph_helper): recompute (`recompute_granularity="selective"` + `recompute_modules`), disable CUDA graphs, or switch to `local` + `full_iteration`. Enforced in MCore `transformer_config.py:2001-2005`. See also - @skills/perf-activation-recompute/SKILL.md. + @skills/nemo-mbridge-perf-activation-recompute/SKILL.md. 12. **Benchmark numbers are workload-specific**: graph wins are usually real when host overhead is visible, but the exact gain depends on batch shape, diff --git a/skills/nemo-mbridge-perf-cuda-graphs/card.yaml b/skills/nemo-mbridge-perf-cuda-graphs/card.yaml index 0f9cfa5f6d..18e687a6e3 100644 --- a/skills/nemo-mbridge-perf-cuda-graphs/card.yaml +++ b/skills/nemo-mbridge-perf-cuda-graphs/card.yaml @@ -241,7 +241,7 @@ failure_modes: - name: full_recompute_with_te_scoped symptom: "AssertionError: full recompute is only supported with full iteration CUDA graph" likely_cause: recompute_granularity=full with any TE-scoped graph (attn, mlp, moe_router, etc.). Common on FP8 CS configs that default to cuda_graph_impl=transformer_engine + scope=mlp. - fix: use recompute_granularity=selective with recompute_modules, or disable CUDA graphs (cuda_graph_impl=none), or switch to cuda_graph_impl=local + cuda_graph_scope=full_iteration. See skills/perf-activation-recompute/SKILL.md. + fix: use recompute_granularity=selective with recompute_modules, or disable CUDA graphs (cuda_graph_impl=none), or switch to cuda_graph_impl=local + cuda_graph_scope=full_iteration. See skills/nemo-mbridge-perf-activation-recompute/SKILL.md. - name: packed_sequences_with_te_scoped symptom: "AssertionError: CUDA graph accepts only Tensor inputs. packed_seq_params excluded" likely_cause: packed_sequence=True passes a non-Tensor packed_seq_params input into TE-scoped capture diff --git a/skills/nemo-mbridge-perf-memory-tuning/SKILL.md b/skills/nemo-mbridge-perf-memory-tuning/SKILL.md index 8e3b33104f..bac0728cb0 100644 --- a/skills/nemo-mbridge-perf-memory-tuning/SKILL.md +++ b/skills/nemo-mbridge-perf-memory-tuning/SKILL.md @@ -54,7 +54,7 @@ When a training run OOMs or is close to the memory limit: fragmentation-induced OOM with zero performance cost. Most Slurm launch templates already include it. 2. **Add selective activation recompute** (`recompute_modules=[core_attn]`) if - not already enabled. See @skills/perf-activation-recompute/SKILL.md. + not already enabled. See @skills/nemo-mbridge-perf-activation-recompute/SKILL.md. 3. **Avoid increasing TP** as a memory fix — doubling TP dramatically increases NVLink all-reduce volume and often kills throughput (-28% on Llama3 70B). 4. **Avoid increasing PP at the cost of DP** — halving DP doubles gradient @@ -92,11 +92,11 @@ If the model genuinely does not fit (not fragmentation), adjust parallelism: | Increase PP (keeping DP) | Fewer layers per stage | Moderate (~6% if DP halved) | Only if GPU count allows | | Increase TP | Fewer params per GPU | Severe (-28% on 70B) | Last resort | | Distributed optimizer | Shards optimizer state across DP ranks | ~1-2% | Recommended for large models | -| FSDP | Shards params + grads + optimizer | Varies | See @skills/perf-megatron-fsdp/SKILL.md | +| FSDP | Shards params + grads + optimizer | Varies | See @skills/nemo-mbridge-perf-megatron-fsdp/SKILL.md | ### Activation recompute -See @skills/perf-activation-recompute/SKILL.md for full details. +See @skills/nemo-mbridge-perf-activation-recompute/SKILL.md for full details. ### CPU offloading @@ -175,7 +175,7 @@ offloading.` This approach is blocked for any model using PP > 1. Selective activation recompute with `mlp` saved ~3 GB peak memory but cost ~16% GPU utilization on this workload. See -@skills/perf-activation-recompute/SKILL.md for full results. +@skills/nemo-mbridge-perf-activation-recompute/SKILL.md for full results. ## Code Anchors diff --git a/skills/nemo-mbridge-perf-moe-long-context/SKILL.md b/skills/nemo-mbridge-perf-moe-long-context/SKILL.md index 487ef30833..abfe8ca952 100644 --- a/skills/nemo-mbridge-perf-moe-long-context/SKILL.md +++ b/skills/nemo-mbridge-perf-moe-long-context/SKILL.md @@ -118,7 +118,7 @@ For long-context MoE training: Useful references: - @docs/training/activation-recomputation.md -- @skills/perf-cuda-graphs/SKILL.md +- @skills/nemo-mbridge-perf-cuda-graphs/SKILL.md ## Pitfalls diff --git a/skills/nemo-mbridge-perf-moe-optimization-workflow/SKILL.md b/skills/nemo-mbridge-perf-moe-optimization-workflow/SKILL.md index ff31f7b8ba..73dded436f 100644 --- a/skills/nemo-mbridge-perf-moe-optimization-workflow/SKILL.md +++ b/skills/nemo-mbridge-perf-moe-optimization-workflow/SKILL.md @@ -131,7 +131,7 @@ or tightly controlled static-shape experiments. Related references: -- @skills/perf-cuda-graphs/SKILL.md +- @skills/nemo-mbridge-perf-cuda-graphs/SKILL.md - @docs/training/cuda-graphs.md - @docs/training/activation-recomputation.md diff --git a/skills/nemo-rl-e2e-testing/SKILL.md b/skills/nemo-rl-e2e-testing/SKILL.md index bdb8a81d62..968ca3657b 100644 --- a/skills/nemo-rl-e2e-testing/SKILL.md +++ b/skills/nemo-rl-e2e-testing/SKILL.md @@ -465,7 +465,7 @@ If an attach helper enters a container that no longer sees the expected checkout Attach helpers that use `--no-container-mount-home` can enter a minimal `/home/$USER` in follow-up steps even when the original run saw the real checkout. Keep metric dumping and assertions in the same container step as the run when possible. If a follow-up step must inspect compute-local artifacts, use paths under the node-local run directory and do not assume `$NEMO_RL_REPO` is visible. -For general Slurm debugging and multi-node patterns, read @skills/multi-node-slurm/SKILL.md. +For general Slurm debugging and multi-node patterns, read @skills/nemo-mbridge-multi-node-slurm/SKILL.md. ## Pass Criteria diff --git a/skills/verl-e2e-testing/SKILL.md b/skills/verl-e2e-testing/SKILL.md index 7e25ca27ab..d89e3d7bff 100644 --- a/skills/verl-e2e-testing/SKILL.md +++ b/skills/verl-e2e-testing/SKILL.md @@ -336,7 +336,7 @@ bash tests/special_e2e/run_ppo_trainer_megatron.sh \ ++critic.megatron.override_transformer_config.gradient_accumulation_fusion=False ``` -For Bridge-native FSDP behavior and constraints, also read @skills/perf-megatron-fsdp/SKILL.md. +For Bridge-native FSDP behavior and constraints, also read @skills/nemo-mbridge-perf-megatron-fsdp/SKILL.md. ## Convergence / Learning Signal @@ -431,7 +431,7 @@ If an attach helper enters a container that no longer sees the expected checkout On CUDA/H100 clusters, some launchers set both `CUDA_VISIBLE_DEVICES` and ROCm variables such as `ROCR_VISIBLE_DEVICES`. If verl workers fail before model construction with `Please don't set ROCR_VISIBLE_DEVICES when HIP/CUDA_VISIBLE_DEVICES is set`, fix the launcher/container environment or apply a temporary local verl workaround that drops `ROCR_VISIBLE_DEVICES` when CUDA is already set. Do not report this as a Bridge provider failure. -For general Slurm debugging and multi-node patterns, read @skills/multi-node-slurm/SKILL.md. +For general Slurm debugging and multi-node patterns, read @skills/nemo-mbridge-multi-node-slurm/SKILL.md. ## Pass Criteria