cp: Merge main to super-v3.5-posttraining - 2026-08-10

docs/guides/nemotron-3-ultra.md

@@ -203,13 +203,16 @@ Set the following before each `bash examples/nemo_gym/nemotron-3-ultra/ultra_lau
 | `NL2BASH_JUDGE_MODEL` | NL2Bash / general-purpose judge: HF repo id or local path. Default judge is `Qwen/Qwen3-235B-A22B-Instruct-2507-FP8`. |
 | `SAFETY_JUDGE_MODEL` | Content-safety judge: HF repo id or local path. Default is [`nvidia/Nemotron-Content-Safety-Reasoning-4B`](https://huggingface.co/nvidia/Nemotron-Content-Safety-Reasoning-4B). |
 
-> **Serving GenRM as a standalone service.** The GenRM judge does not have to run
-> inside the training job. You can bring it up separately — any OpenAI-compatible
-> vLLM endpoint, or the external-GenRM service launcher under `tools/external_genrm/`
-> (runs the judge fleet in a second Slurm hetgroup behind a load balancer) — and
-> point the run at it with `GENRM_BASE_URL=http://<host>:<port>/v1`. Judging is then
-> routed to that endpoint instead of being served from the gym pool, which frees
-> those GPUs and lets one GenRM deployment back many training runs.
+> **Serving GenRM outside Gym.** For a separately deployed OpenAI-compatible
+> endpoint, set `GENRM_BASE_URL=http://<host>:<port>/v1`. Judging is then routed
+> to that endpoint instead of being served from the Gym pool, allowing one
+> deployment to back multiple training runs.
+>
+> To co-schedule dedicated model servers with training in one Slurm
+> heterogeneous allocation, use the
+> [external Gym vLLM pool helpers](https://github.com/NVIDIA-NeMo/RL/blob/main/tools/external_gym_vllm/README.md).
+> They place the server fleet in a second hetgroup, start load balancers on the
+> training component, and inject the resolved endpoints into the driver command.
 
 Optional knobs:
 

docs/guides/ppo.md

@@ -38,9 +38,24 @@ We define a [ValueInterface](../../nemo_rl/models/value/interfaces.py) that cont
 
 The value model supports the **Megatron-Core backend** (`value.megatron_cfg.enabled: true`) and the **DTensor backend** (`value.dtensor_cfg.enabled: true`). It uses the same architecture and tokenizer as the policy (configured via `value.model_name`), but is trained with a separate MSE loss on GAE returns.
 
-### Colocated Architecture
+### Deployment Architectures
 
-PPO uses a colocated architecture where the **policy**, **value model**, and **vLLM generation engine** share the same set of GPUs. GPU memory is managed by offloading models to CPU between stages: the value model is loaded to GPU only during its inference and training phases, then offloaded to make room for other components.
+By default, PPO uses a colocated architecture where the **policy**, **value model**, and **generation engine** share one `RayVirtualCluster`. GPU memory is managed by offloading models to CPU between stages: the value model is loaded to GPU only during its inference and training phases, then offloaded to make room for the other components.
+
+PPO also supports non-colocated vLLM generation. In this mode, the policy and value model continue to time-share one training `RayVirtualCluster`, while vLLM runs on a separate inference `RayVirtualCluster` in the same Ray cluster. Updated policy weights are transferred to vLLM through the cross-cluster collective refit path.
+
+```yaml
+policy:
+  generation:
+    backend: vllm
+    colocated:
+      enabled: false
+      resources:
+        gpus_per_node: 2
+        num_nodes: null
+```
+
+When only one node remains for policy and generation after other resources are reserved, `gpus_per_node` reserves that many GPUs for generation and `num_nodes` must be `null` or `1`. When more than one node remains for training and generation, generation uses complete nodes: set `num_nodes` to the number of inference nodes and `gpus_per_node` equal to `cluster.gpus_per_node`. Non-colocated SGLang generation is not currently supported by PPO.
 
 ### Value Model Configuration
 
@@ -217,6 +232,8 @@ ppo:
   seed: 42
   use_dynamic_sampling: false
   overlong_filtering: false
+  # null logs mismatch metrics without masking; set a threshold to mask sequences.
+  seq_logprob_error_threshold: null
 
   adv_estimator:
     name: "gae"
@@ -257,6 +274,7 @@ value_loss_fn:
 **PPO-specific parameters:**
 - **`ppo.ppo_epochs`**: Number of training updates per rollout batch
 - **`ppo.policy_training_start_step`**: Number of critic-only warmup steps before policy training begins
+- **`ppo.seq_logprob_error_threshold`**: Nullable sequence-level multiplicative probability-error threshold. PPO always logs sequence-level train/generation mismatch metrics; when this is set, sequences above the threshold are excluded from advantage and loss computation.
 - **`ppo.adv_estimator.name`**: Set to `"gae"` for GAE advantage estimation (PPO default)
 - **`ppo.adv_estimator.gae_lambda`**: GAE $\lambda$ parameter (bias-variance tradeoff, typically 0.95)
 - **`ppo.adv_estimator.gae_gamma`**: Discount factor $\gamma$ (typically 1.0 for outcome-supervised tasks)
@@ -268,7 +286,7 @@ All other parameters (clipping, KL, importance sampling, dynamic sampling, rewar
 
 ## Metrics
 
-PPO logs all the same metrics as GRPO (see [GRPO Metrics](grpo.md#metrics)). In addition, the following critic-specific metrics are logged:
+PPO logs all the same metrics as GRPO (see [GRPO Metrics](grpo.md#metrics)). It also logs the following PPO-specific metrics:
 
 | Metric | Description |
 |--------|-------------|
@@ -279,6 +297,14 @@ PPO logs all the same metrics as GRPO (see [GRPO Metrics](grpo.md#metrics)). In
 | `critic/values_max` | Maximum predicted value |
 | `critic/returns_mean` | Mean of GAE returns |
 | `critic/explained_var` | Explained variance: $1 - \text{Var}(R - V) / \text{Var}(R)$. Higher is better; values near 1.0 indicate the critic accurately predicts returns. |
+| `max_seq_mult_prob_error` | Maximum sequence-level multiplicative probability error between generation and training logprobs before optional masking. |
+| `mean_seq_mult_prob_error` | Mean sequence-level multiplicative probability error before optional masking. |
+| `min_seq_mult_prob_error` | Minimum sequence-level multiplicative probability error before optional masking. |
+| `max_seq_mult_prob_error_after_mask` | Maximum sequence-level multiplicative probability error among sequences retained after optional masking. |
+| `mean_seq_mult_prob_error_after_mask` | Mean sequence-level multiplicative probability error among sequences retained after optional masking. |
+| `min_seq_mult_prob_error_after_mask` | Minimum sequence-level multiplicative probability error among sequences retained after optional masking. |
+| `num_masked_seqs_by_logprob_error` | Number of sequences excluded by `ppo.seq_logprob_error_threshold`. |
+| `masked_correct_pct` | Fraction of sequences excluded by `ppo.seq_logprob_error_threshold` that received a reward of 1. |
 
 ## Evaluate the Trained Model
 

examples/configs/distillation_math.yaml

@@ -201,6 +201,9 @@ policy: &POLICY_BASE
         temperature: 1.0
         top_p: 1.0
         top_k: null
+        val_temperature: ${.temperature}
+        val_top_p: ${.top_p}
+        val_top_k: ${.top_k}
         stop_token_ids: null
         stop_strings: null
         vllm_cfg:

examples/configs/evals/eval.yaml

@@ -12,6 +12,9 @@ generation:
   temperature: 0.0
   top_p: 1.0
   top_k: -1 # -1 means disable
+  val_temperature: ${.temperature}
+  val_top_p: ${.top_p}
+  val_top_k: ${.top_k}
   num_prompts_per_step: -1 # -1 means pass all prompts at once
   model_name: "Qwen/Qwen2.5-Math-1.5B-Instruct"
   stop_token_ids: null

examples/configs/evals/mmau.yaml

@@ -11,6 +11,9 @@ generation:
   temperature: 0.0
   top_p: 1.0
   top_k: -1
+  val_temperature: ${.temperature}
+  val_top_p: ${.top_p}
+  val_top_k: ${.top_k}
   num_prompts_per_step: -1
   model_name: "Qwen/Qwen2.5-Omni-3B"
   stop_token_ids: null

examples/configs/grpo_math_1B.yaml

@@ -15,6 +15,9 @@ grpo:
   advantage_clip_low: null
   advantage_clip_high: null
   max_val_samples: 256
+  # Validation rollouts per prompt; k > 1 also reports the pass_k metric.
+  # max_val_samples counts PROMPTS: total validation rollouts = max_val_samples * k.
+  val_num_generations_per_prompt: 1
   # Early stop once this metric (e.g. accuracy or pass_k) reaches the threshold; null disables.
   stop_at_validation_metric: null
   # Required when stop_at_validation_metric is set.
@@ -344,6 +347,10 @@ policy:
     temperature: 1.0
     top_p: 1.0
     top_k: null
+    # Validation-only sampling; defaults follow the train values above.
+    val_temperature: ${.temperature}
+    val_top_p: ${.top_p}
+    val_top_k: ${.top_k}
     stop_token_ids: null
     stop_strings: null
     # null = topology default (IPC colocated, NCCL non-colocated).

examples/configs/ppo_math_1B.yaml

@@ -47,6 +47,9 @@ ppo:
     source_max: 1.0
     target_min: -1.0  # DAPO: scale rewards to [-1, 1]
     target_max: 1.0
+  # Log train/generation probability mismatch. Set a threshold to mask
+  # high-error sequences; null keeps metrics-only behavior.
+  seq_logprob_error_threshold: null
 
 loss_fn:
   disable_ppo_ratio: false
@@ -234,6 +237,9 @@ policy:
     temperature: 1.0
     top_p: 1.0
     top_k: null
+    val_temperature: ${.temperature}
+    val_top_p: ${.top_p}
+    val_top_k: ${.top_k}
     stop_token_ids: null
     stop_strings: null
     mcore_generation_config:
@@ -459,3 +465,9 @@ logger:
 cluster:
   gpus_per_node: 1
   num_nodes: 1
+  # Port range for the distributed master address (TCPStore / NCCL rendezvous)
+  # and per-worker available ports.  Kept below the OS ephemeral range
+  # (32768-60999 on stock Linux).  See ray.sub for the full port layout.
+  master_port_range_low: 1400
+  master_port_range_high: 1999
+  segment_size: null  # Nodes per NVLink domain segment for topology-aware alignment; null to disable

examples/configs/recipes/llm/grpo-nanov3-30BA3B-2n8g-megatron_generation-async-gym.yaml

@@ -15,8 +15,6 @@ policy:
   megatron_cfg:
     pipeline_model_parallel_size: 1
     context_parallel_size: 2
-  sequence_packing:
-    enabled: false
   generation:
     backend: megatron
     colocated:

examples/configs/recipes/llm/mopd-qwen3-1.7b-3n8g-megatron-pack.yaml

@@ -2,7 +2,6 @@ defaults: ../../grpo_math_1B.yaml
 grpo:
   num_prompts_per_step: 8
   num_generations_per_prompt: 4
-  num_val_generations_per_prompt: 1
   max_num_steps: 5
   val_period: 1000
   overlong_filtering: true

examples/configs/recipes/llm/ppo-qwen2.5-1.5b-gsm8k-1n8g-automodel-noncolocated.yaml

@@ -0,0 +1,59 @@
+defaults: ../../ppo_math_1B.yaml
+ppo:
+  num_prompts_per_step: 1024
+  num_generations_per_prompt: 1
+  max_num_epochs: 15
+  ppo_epochs: 1
+  val_period: 1
+  overlong_filtering: true
+  reward_shaping:
+    enabled: false
+  adv_estimator:
+    gae_lambda_value: 1.0
+    gae_lambda_policy: 1
+  reward_scaling:
+    enabled: false
+loss_fn:
+  ratio_clip_max: 0.2
+  ratio_clip_c: 3
+value_loss_fn:
+  scale: 1.0
+  cliprange: 0.5
+checkpointing:
+  checkpoint_dir: results/ppo-qwen2.5-1.5b-gsm8k-1n8g-automodel-noncolocated
+policy:
+  model_name: Qwen/Qwen2.5-1.5B-Instruct
+  train_global_batch_size: 256
+  max_total_sequence_length: 1024
+  generation:
+    max_new_tokens: 512
+    vllm_cfg:
+      gpu_memory_utilization: 0.4
+      max_model_len: 1024
+    colocated:
+      enabled: false
+      resources:
+        gpus_per_node: 4
+value:
+  model_name: Qwen/Qwen2.5-1.5B-Instruct
+  train_micro_batch_size: 4
+data:
+  max_input_seq_length: 512
+  train:
+    dataset_name: gsm8k
+    split: train
+  validation:
+    dataset_name: gsm8k
+    split: test
+  default:
+    system_prompt_file: examples/prompts/gsm8k.txt
+env:
+  math:
+    math_verify_impl: hf_math_verify
+logger:
+  log_dir: logs/ppo-qwen2.5-1.5b-gsm8k-1n8g-automodel-noncolocated
+  wandb:
+    project: nemo-rl
+    name: ppo-qwen2.5-1.5b-gsm8k-1n8g-automodel-noncolocated
+cluster:
+  gpus_per_node: 8

examples/configs/recipes/llm/ppo-qwen2.5-1.5b-gsm8k-2n8g-megatron-valuetp2sp-dynbatch-noncolocated.yaml

@@ -0,0 +1,83 @@
+defaults: ../../ppo_math_1B_megatron.yaml
+ppo:
+  num_prompts_per_step: 1024
+  num_generations_per_prompt: 1
+  max_num_epochs: 15
+  ppo_epochs: 1
+  val_period: 1
+  overlong_filtering: true
+  reward_shaping:
+    enabled: false
+  adv_estimator:
+    gae_lambda_value: 1.0
+    gae_lambda_policy: 1
+  reward_scaling:
+    enabled: false
+loss_fn:
+  ratio_clip_max: 0.2
+  ratio_clip_c: 3
+value_loss_fn:
+  scale: 1.0
+  cliprange: 0.5
+checkpointing:
+  checkpoint_dir: results/ppo-qwen2.5-1.5b-gsm8k-2n8g-megatron-valuetp2sp-dynbatch-noncolocated
+policy:
+  model_name: Qwen/Qwen2.5-1.5B-Instruct
+  train_global_batch_size: 256
+  max_total_sequence_length: 1024
+  megatron_cfg:
+    tensor_model_parallel_size: 2
+    context_parallel_size: 2
+    sequence_parallel: true
+    optimizer:
+      weight_decay: 0.01
+    scheduler:
+      start_weight_decay: 0.01
+      end_weight_decay: 0.01
+      lr_warmup_iters: 0
+      lr_warmup_init: 0
+  make_sequence_length_divisible_by: 8
+  generation:
+    max_new_tokens: 512
+    colocated:
+      enabled: false
+      resources:
+        gpus_per_node: 8
+        num_nodes: 1
+    vllm_cfg:
+      gpu_memory_utilization: 0.4
+      max_model_len: 1024
+value:
+  model_name: Qwen/Qwen2.5-1.5B-Instruct
+  train_micro_batch_size: 4
+  megatron_cfg:
+    tensor_model_parallel_size: 2
+    sequence_parallel: true
+    optimizer:
+      lr: 1.0e-05
+      weight_decay: 0.01
+    scheduler:
+      lr_warmup_iters: 0
+  dynamic_batching:
+    enabled: true
+data:
+  max_input_seq_length: 512
+  train:
+    dataset_name: gsm8k
+    split: train
+  validation:
+    dataset_name: gsm8k
+    split: test
+  default:
+    system_prompt_file: examples/prompts/gsm8k.txt
+env:
+  math:
+    math_verify_impl: hf_math_verify
+logger:
+  log_dir: logs/ppo-qwen2.5-1.5b-gsm8k-2n8g-megatron-valuetp2sp-dynbatch-noncolocated
+  wandb:
+    project: nemo-rl
+    name: ppo-qwen2.5-1.5b-gsm8k-2n8g-megatron-valuetp2sp-dynbatch-noncolocated
+cluster:
+  num_nodes: 2
+  gpus_per_node: 8