Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
20776f2
Update Megatron Inference
tdene Apr 27, 2026
adcd0e9
Fix issues from the rebase
tdene May 15, 2026
835cb43
Address reviewer comments
tdene May 18, 2026
229cecd
Address reviewer comments
tdene May 18, 2026
3bb2a52
Address reviewer comments
tdene May 18, 2026
74b2a64
Address reviewer comments
tdene May 21, 2026
e54f42e
Address reviewer comments
tdene May 21, 2026
f2971c2
Address reviewer comments
tdene May 21, 2026
64664bc
Address reviewer comments
tdene May 21, 2026
841e593
Self-review
tdene May 26, 2026
6e2e528
Update for latest Megatron-LM
tdene May 26, 2026
fd2d580
lint
tdene May 26, 2026
3c48917
Apply suggestions from code review
tdene May 27, 2026
77e7a65
Address reviewer comments
tdene May 27, 2026
d27c1bd
Unfactor MegatronRefitController
tdene May 27, 2026
d64a81f
Address reviewer comments
tdene May 28, 2026
cc5f5e4
Address reviewer comments
tdene May 28, 2026
0867638
Fix NVLink behavior
tdene May 29, 2026
20acfea
lint
tdene May 29, 2026
278f8c8
lint
tdene May 29, 2026
52c7397
Fix non-Gym and colocated paths
tdene Jun 1, 2026
f7658b9
Fix stop words and sampling params
tdene Jun 1, 2026
f5e0497
Add more functional test coverage
tdene Jun 1, 2026
89208b5
Add MInf unit tests
tdene Jun 2, 2026
4a9a43b
lint
tdene Jun 2, 2026
892dd87
Merge remote-tracking branch 'origin/main' into tde/megatron_inference
tdene Jun 2, 2026
a116ccc
lint
tdene Jun 2, 2026
c1e8086
lint
tdene Jun 2, 2026
c7301d6
lint
tdene Jun 3, 2026
7337fbb
lint
tdene Jun 3, 2026
359eced
Merge remote-tracking branch 'origin/main' into tde/megatron_inference
tdene Jun 4, 2026
42d8c85
Fix merge issues
tdene Jun 5, 2026
0fbe5c9
Fix old unit tests
tdene Jun 5, 2026
41b317b
Merge remote-tracking branch 'origin/main' into tde/megatron_inference
tdene Jun 9, 2026
4d504a0
Fix tests
tdene Jun 10, 2026
7d1e2ca
Fix unit tests
tdene Jun 11, 2026
c71e3b5
Fix merge
tdene Jun 11, 2026
6d3233a
Merge remote-tracking branch 'origin/main' into tde/megatron_inference
tdene Jun 11, 2026
1caf9b8
Fix NeMo Gym merge
tdene Jun 11, 2026
e7949e2
Fix CI
tdene Jun 11, 2026
a5057f8
Fix CI
tdene Jun 12, 2026
4c717df
Merge remote-tracking branch 'origin/main' into tde/megatron_inference
tdene Jun 12, 2026
a44233b
Fix fp32 test issue
tdene Jun 12, 2026
27d320f
Fix CI
tdene Jun 13, 2026
155f7e8
Fix CI
tdene Jun 13, 2026
df4f663
Merge remote-tracking branch 'origin/main' into tde/megatron_inference
tdene Jun 14, 2026
e1b5f85
Update value worker
tdene Jun 14, 2026
2ba8f0b
Add an early preview of #2612 to fix CI
tdene Jun 15, 2026
0984e27
Fix functional test configs
tdene Jun 15, 2026
f14d5cb
Skip MInf functional tests on Blackwell
tdene Jun 15, 2026
dce23c4
Merge remote-tracking branch 'origin/main' into tde/megatron_inference
tdene Jun 15, 2026
ae64407
Hotfix; see NVIDIA-NeMo/Megatron-Bridge#4375
tdene Jun 15, 2026
a84cfba
Address reviewer comments
tdene Jun 16, 2026
2162b37
Merge remote-tracking branch 'origin/main' into tde/megatron_inference
tdene Jun 16, 2026
71da32a
Disable top_p / top_k test for Megatron Inference
tdene Jun 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/cicd-main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -697,6 +697,8 @@ jobs:
runner: ${{ needs.org-member-pre-flight.outputs.runner_prefix }}
- script: L1_Functional_Tests_Megatron_3
runner: ${{ needs.org-member-pre-flight.outputs.runner_prefix }}
- script: L1_Functional_Tests_Megatron_4
runner: ${{ needs.org-member-pre-flight.outputs.runner_prefix }}
- script: L1_Functional_Tests_AutoModel
runner: ${{ needs.org-member-pre-flight.outputs.runner_prefix }}
- script: L1_Functional_Tests_SGLang
Expand Down Expand Up @@ -764,6 +766,8 @@ jobs:
runner: ${{ vars.GB200_RUNNER }}
- script: L1_Functional_Tests_Megatron_3
runner: ${{ vars.GB200_RUNNER }}
- script: L1_Functional_Tests_Megatron_4
runner: ${{ vars.GB200_RUNNER }}
- script: L1_Functional_Tests_AutoModel
runner: ${{ vars.GB200_RUNNER }}
- script: L1_Functional_Tests_SGLang
Expand Down Expand Up @@ -832,6 +836,8 @@ jobs:
runner: ${{ needs.org-member-pre-flight.outputs.runner_prefix }}
- script: L1_Functional_Tests_Megatron_3
runner: ${{ needs.org-member-pre-flight.outputs.runner_prefix }}
- script: L1_Functional_Tests_Megatron_4
runner: ${{ needs.org-member-pre-flight.outputs.runner_prefix }}
- script: L1_Functional_Tests_AutoModel
runner: ${{ needs.org-member-pre-flight.outputs.runner_prefix }}
- script: L1_Functional_Tests_Gym
Expand Down
14 changes: 12 additions & 2 deletions examples/configs/grpo_math_1B.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -307,13 +307,23 @@ policy:
stop_token_ids: null
stop_strings: null
mcore_generation_config:
buffer_size_gb: 10 # Total buffer size is 2x this value (active requests + paused requests)
async_engine: false
max_model_len: ${policy.max_total_sequence_length} # Engine-side max sequence length.
cuda_graph_impl: "local" # CUDA-graph implementation. Options: "none", "local", "transformer_engine", "full_iteration".
inference_cuda_graph_scope: "block" # Inference CUDA-graph scope. Options: "none", "layer", "block".
buffer_size_gb: 10
num_cuda_graphs: 4 # Number of CUDA graphs to pre-compile for different batch sizes
block_size_tokens: 256 # Size of each KV cache block in tokens (affects memory granularity)
use_cuda_graphs_for_non_decode_steps: true # Enable CUDA graphs for prefill/context processing
enable_chunked_prefill: true # Split long prefills into chunks for better memory management
unified_memory_level: 0 # Unified memory usage level (0=disabled, higher values enable more aggressive paging)
enable_prefix_caching: false # Reuse KV blocks across requests sharing a prompt prefix.
max_tokens: 16384 # Maximum number of tokens to use in a single step. Analogous to vllm's max_num_batched_tokens
kv_cache_management_mode: "persist" # KV cache lifecycle across suspend/resume. Options: "persist", "offload". To select "recompute", set grpo.async_grpo.recompute_kv_cache_after_weight_updates=true.
materialize_only_last_token_logits: true
num_speculative_tokens: 0
refit_backend: "nvshmem" # Copy-service backend for non-colocated megatron weight refit. Options: "gloo" or "nvshmem".
parsers: [] # OpenAI tool-call / response parser names exposed by the HTTP server (only used if expose_http_server=true).
expose_http_server: false # Start an OpenAI-compatible HTTP server alongside the persistent engine. Required for NeMo-Gym.
vllm_cfg:
async_engine: false
precision: ${policy.precision}
Expand Down
11 changes: 1 addition & 10 deletions examples/configs/grpo_math_1B_megatron.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ policy:
moe_enable_deepep: false
moe_token_dispatcher_type: "alltoall"
moe_shared_expert_overlap: false
#gives ~20% training perf speedup with sequence packing
#gives ~20% training perf speedup with sequence packing
apply_rope_fusion: True

peft:
Expand Down Expand Up @@ -175,15 +175,6 @@ policy:
temperature: 1.0
top_p: 1.0
top_k: null
mcore_generation_config:
buffer_size_gb: 10 # Total buffer size is 2x this value (active requests + paused requests)
num_cuda_graphs: 4 # Number of CUDA graphs to pre-compile for different batch sizes
block_size_tokens: 256 # Size of each KV cache block in tokens (affects memory granularity)
use_cuda_graphs_for_non_decode_steps: true # Enable CUDA graphs for prefill/context processing
enable_chunked_prefill: false # Split long prefills into chunks for better memory management
unified_memory_level: 0 # Unified memory usage level (0=disabled, higher values enable more aggressive paging)
max_tokens: 16384 # Maximum number of tokens to use in a single step. Analogous to vllm's max_num_batched_tokens

vllm_cfg:
tensor_parallel_size: 1
gpu_memory_utilization: 0.6
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
defaults: ../../grpo_math_1B.yaml
grpo:
max_num_steps: 30
checkpointing:
enabled: false
checkpoint_dir: results/grpo-llama3.2-1b-instruct-2n8g-megatron_generation-noncolocated
save_period: 100
policy:
model_name: meta-llama/Llama-3.2-1B-Instruct
tokenizer:
name: meta-llama/Llama-3.2-1B-Instruct
optimizer: null
megatron_cfg:
enabled: true
scheduler:
lr_warmup_iters: 50
dtensor_cfg:
enabled: false
make_sequence_length_divisible_by: 1
generation:
backend: megatron
max_new_tokens: 512
vllm_cfg:
max_model_len: 512
colocated:
enabled: false
resources:
gpus_per_node: 8
num_nodes: 1
data:
max_input_seq_length: 512
logger:
log_dir: logs/grpo-llama3.2-1b-instruct-2n8g-megatron_generation-noncolocated
wandb_enabled: true
tensorboard_enabled: true
wandb:
project: nemo-rl
name: grpo-llama3.2-1b-instruct-2n8g-megatron_generation-noncolocated
cluster:
gpus_per_node: 8
num_nodes: 2
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
defaults: ../../../nemo_gym/grpo_nanov3.yaml
grpo:
async_grpo:
enabled: true
in_flight_weight_updates: true
checkpointing:
checkpoint_dir: results/grpo-nanov3-30BA3B-2n8g-megatron_generation-async-gym
policy:
model_name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16
tokenizer:
name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
train_global_batch_size: 16
max_total_sequence_length: 2048
sequence_packing:
enabled: false
generation:
backend: megatron
colocated:
enabled: false
resources:
gpus_per_node: 8
num_nodes: 1
logger:
wandb_enabled: true
tensorboard_enabled: true
wandb:
project: nemo-rl
name: grpo-nanov3-30BA3B-2n8g-megatron_generation-async-gym
cluster:
num_nodes: 2
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
defaults: ../../grpo_math_1B.yaml
grpo:
num_prompts_per_step: 2
num_generations_per_prompt: 8
checkpointing:
checkpoint_dir: results/grpo-nanov3-30BA3B-2n8g-megatron_generation
policy:
model_name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16
tokenizer:
name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
train_global_batch_size: 16
train_micro_batch_size: 1
logprob_batch_size: 1
max_total_sequence_length: 2048
dtensor_cfg:
enabled: false
megatron_cfg:
enabled: true
bias_activation_fusion: false
tensor_model_parallel_size: 2
expert_model_parallel_size: 8
sequence_parallel: true
sequence_packing:
enabled: false
generation:
backend: megatron
mcore_generation_config:
transformer_impl: inference_optimized
activation_checkpointing: false
tensor_model_parallel_size: 2
expert_model_parallel_size: 8
sequence_parallel: true
async_engine: true
colocated:
enabled: false
resources:
gpus_per_node: 8
num_nodes: 1
logger:
wandb_enabled: true
tensorboard_enabled: true
wandb:
project: nemo-rl
name: grpo-nanov3-30BA3B-2n8g-megatron_generation
cluster:
gpus_per_node: 8
num_nodes: 2
36 changes: 36 additions & 0 deletions examples/nemo_gym/grpo_nanov3.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,42 @@ policy:
top_k: null
stop_token_ids: null
stop_strings: null
mcore_generation_config:
Comment thread
yuki-97 marked this conversation as resolved.
Resolved
# Scoped to generation workers only; training keeps its own defaults
# via the base yaml. The mcore_generation_config block is merged into
# megatron_cfg for inference workers before model setup runs.
max_model_len: ${policy.max_total_sequence_length}
transformer_impl: "inference_optimized"
cuda_graph_impl: "local"
inference_cuda_graph_scope: "block"
activation_checkpointing: false
mamba_inference_ssm_states_dtype: "float32"
inference_moe_token_dispatcher_type: "nccl" # Fall back to NCCL for now
inference_grouped_gemm_backend: "vllm"
moe_router_num_groups: null # InferenceTopKRouter requires num_groups=None
moe_router_group_topk: null # paired with moe_router_num_groups=null
pipeline_model_parallel_size: 1
expert_tensor_parallel_size: 1
expert_model_parallel_size: 8
sequence_parallel: True
context_parallel_size: 1
tensor_model_parallel_size: 2
buffer_size_gb: 20 # Total GPU memory (in GB) allocated for KV cache buffers
num_cuda_graphs: -1 # Number of CUDA graphs to pre-compile for different batch sizes
block_size_tokens: 256 # Size of each KV cache block in tokens (affects memory granularity)
use_cuda_graphs_for_non_decode_steps: true # Enable CUDA graphs for prefill/context processing
enable_chunked_prefill: true
max_tokens: 16384 # Maximum number of tokens to use in a single step. Analogous to vllm's max_num_batched_tokens
kv_cache_management_mode: "persist" # KV cache lifecycle across suspend/resume. Options: "persist", "offload". To select "recompute", set grpo.async_grpo.recompute_kv_cache_after_weight_updates=true.
materialize_only_last_token_logits: true
num_speculative_tokens: 0
refit_backend: "nvshmem" # Copy-service backend for non-colocated megatron weight refit. Options: "gloo" or "nvshmem".
async_engine: true
expose_http_server: true
enable_prefix_caching: true
parsers:
- deepseek-r1-reasoning
- qwen3-coder-tool
vllm_cfg:
# NB: can re-enable prefix cache on vllm >= 0.11.2.
# enable_prefix_caching: false
Expand Down
18 changes: 18 additions & 0 deletions examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,24 @@ policy:
tool_parser: hermes
# Enable the appropriate reasoning parser here. Since this model is an instruct model, we comment it out.
# reasoning_parser: deepseek_r1
mcore_generation_config:
async_engine: false
max_model_len: ${policy.max_total_sequence_length}
cuda_graph_impl: "local"
inference_cuda_graph_scope: "block"
buffer_size_gb: 10
num_cuda_graphs: 4
block_size_tokens: 256
use_cuda_graphs_for_non_decode_steps: true
enable_chunked_prefill: true
enable_prefix_caching: false
max_tokens: 16384
kv_cache_management_mode: "persist"
materialize_only_last_token_logits: true
num_speculative_tokens: 0
refit_backend: "nvshmem"
parsers: []
expose_http_server: false

# Needs to be set to whatever backend TP size.
make_sequence_length_divisible_by: ${policy.megatron_cfg.tensor_model_parallel_size}
Expand Down
26 changes: 18 additions & 8 deletions nemo_rl/algorithms/async_utils/trajectory_collector.py
Original file line number Diff line number Diff line change
Expand Up @@ -437,9 +437,9 @@ def resume(self) -> None:
def prepare_for_refit(self) -> None:
"""Pause new generation starts and optionally wait for pending generations.

For vLLM V1 async engine, leverages in-flight weight updates via collective_rpc,
allowing ongoing generations to continue with their current KV caches while
weights are updated. This significantly improves async performance.
For backends with an async engine in-flight weight updates allows ongoing generations
to continue with their current KV caches while weights are updated.
This significantly improves async performance.

For non-async engines, waits for all pending generations to complete before refit.
"""
Expand All @@ -450,19 +450,29 @@ def prepare_for_refit(self) -> None:
self._refit_pause_cleared.clear()
print("⏸️ New generation starts paused")

# Check if we're using vLLM async engine
vllm_cfg = self.master_config.policy.get("generation", {}).get("vllm_cfg", {})
is_async_engine = vllm_cfg.get("async_engine", False)
# Check if we're using async engine
Comment thread
yuki-97 marked this conversation as resolved.
Resolved
generation_cfg = self.master_config.policy.get("generation", {})
backend = generation_cfg.get("backend", "")
if backend == "vllm":
is_async_engine = generation_cfg.get("vllm_cfg", {}).get(
"async_engine", False
)
elif backend == "megatron":
is_async_engine = generation_cfg.get("mcore_generation_config", {}).get(
"async_engine", False
)
else:
is_async_engine = False
in_flight_weight_updates = self.master_config.grpo.get("async_grpo", {}).get(
"in_flight_weight_updates", False
)

if is_async_engine and in_flight_weight_updates:
# vLLM V1 async engine supports in-flight weight updates
# async engines support in-flight weight updates
# Ongoing generations will continue with their current KV caches
# New generations (after weight update) will use the updated weights
print(
"🚀 Using vLLM V1 in-flight weight update - skipping wait for pending generations"
f"🚀 Using {backend} in-flight weight update - skipping wait for pending generations"
)
print(
f" {len(self._inflight_threads)} ongoing generations will complete with current weights"
Expand Down
Loading
Loading