[BREAKING][algo] feat: Rollout Correction for General Off-Policy Problems

[szrlee]

Summary

This PR introduces a comprehensive overhaul of the rollout correction system with typed configuration, mathematical documentation, and performance optimizations.

If you find the PR useful, please consider citing:

@online{liu-li-2025-rl-collapse,
  title = {When Speed Kills Stability: Demystifying {RL} Collapse from the Training-Inference Mismatch},
  author = {Liu, Jiacai and Li, Yingru and Fu, Yuqian and Wang, Jiawei and Liu, Qian and Shen, Yu},
  year = {2025},
  month = sep,
  url = {https://richardli.xyz/rl-collapse}
}

⚠️ BREAKING CHANGE: Removes backward compatibility. Users must migrate to typed config.

---

What's New

1. Typed Configuration with Presets

Before (deprecated):

algorithm:
  rollout_is: true
  rollout_is_threshold: 2.0
  rollout_is_level: token

After (Python - Recommended):

from verl.trainer.config.algorithm import RolloutCorrectionConfig

# Use validated presets
config.algorithm.rollout_correction = RolloutCorrectionConfig.token_is()
config.algorithm.rollout_correction = RolloutCorrectionConfig.seq_is_rs()

After (YAML):

algorithm:
  rollout_correction:
    rollout_is: token
    rollout_is_threshold: 2.0

10 validated presets:

• token_is() / token_tis() - Per-token IS
• seq_is() - Sequence-level IS
• seq_is_rs() / seq_mis() - Sequence IS + rejection sampling
• geo_rs() / geo_mis() - Geometric RS + veto
• ppo_is_bypass() - Bypass mode (performance)
• pure_is() - Pure policy gradient (no PPO clipping)
• disabled() - Metrics only

Complete document can be find on https://verl.readthedocs.io/en/latest/advance/rollout_corr.html.

YAML Configuration (Advanced)

For advanced customization or YAML-based configs:

algorithm:
  rollout_correction:
    rollout_is: token                      # IS weights: "token", "sequence", or null
    rollout_is_threshold: 2.0              # Upper threshold for IS weights
    rollout_rs: null                       # Rejection sampling: "token", "sequence", "geometric", or null
    rollout_rs_threshold: null             # RS upper threshold (required if rollout_rs is enabled)
    rollout_rs_threshold_lower: null       # RS lower threshold (auto-reciprocal if null)
    rollout_token_veto_threshold: null     # Per-token veto threshold (null = disabled)
    bypass_old_logprob_for_rollout: false  # Skip old_log_prob computation
    use_pure_rollout_correction: false     # Pure policy gradient with IS

# REQUIRED: Enable log prob calculation
actor_rollout_ref:
  rollout:
    calculate_log_probs: true

2. Mathematical Documentation

New comprehensive document: docs/advance/rollout_corr_math.md (585 lines)

Theoretical foundation:

• REINFORCE → PPO → Decoupled PPO progression
• Batch size invariance: Decoupling proximal policy from behavior policy
• Three-policy framework: π_rollout, π_old, π_θ

Complete formulations for:

• Off-policy REINFORCE (pure_is)
• Standard PPO and bypass mode
• Decoupled PPO (token_is, seq_is, seq_is_rs)
• Rejection sampling (geo_rs)

Diagnostic metrics:

• KL divergence (direct and K3 estimators)
• Perplexity and perplexity ratio
• χ² divergence (token and sequence level)

Quality:

• Objective technical descriptions
• All formulas mathematically verified
• Cross-document consistency validated

3. Training Modes

Mode	Config	Policies	Speed	Description
Standard	bypass=false, pure=false	3	Standard	Full decoupled PPO with batch size invariance
Bypass	bypass=true, pure=false	2	Fast	PPO clips against rollout (faster)
Pure IS	bypass=true, pure=true	2	Fast	Off-policy REINFORCE without clipping

Example:

# Bypass mode for performance
config = RolloutCorrectionConfig.ppo_is_bypass(threshold=2.0)

# Pure IS for research
config = RolloutCorrectionConfig.pure_is(threshold=2.0)

4. Chi-Squared Divergence Metrics

Quantify off-policy severity:

rollout_corr/chi2_token    # E[ρ²] - 1
rollout_corr/chi2_seq      # E[(∏ρ)²] - 1

Interpretation:

• χ² = 0: Perfect on-policy
• χ² < 1: Low off-policiness, stable
• χ² ≥ 10: High off-policiness, need correction

Cleanup:

• Removed mismatch_ prefix
• All metrics under rollout_corr/ namespace

5. Bug Fix

Critical fix:

• rollout_rs="token" with rollout_rs_threshold=None silently failed
• Now raises ValueError with clear error message

---

Migration Guide

Example 1: Basic Token-level IS

# Old (no longer works)
config.algorithm.rollout_is = True
config.algorithm.rollout_is_threshold = 2.0
config.algorithm.rollout_is_level = "token"

# New
config.algorithm.rollout_correction = RolloutCorrectionConfig.token_is(threshold=2.0)

Example 2: Sequence IS + Rejection Sampling

# Old (no longer works)
config.algorithm.rollout_is_level = "sequence"
config.algorithm.rollout_is_mode = "mask"

# New
config.algorithm.rollout_correction = RolloutCorrectionConfig.seq_is_rs(
    is_threshold=2.0,
    rs_threshold=2.0
)

Example 3: Disable

# Old
rollout_is: false

# New
rollout_correction: null

---

References

Liu, Li, Fu, Wang, Liu, Shen (2025). When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch. Blog

[gemini-code-assist]

Code Review

This is a comprehensive refactoring that modernizes the configuration and terminology for rollout correction, moving from a flat structure to a more organized nested configuration under algorithm.rollout_correction. The breaking change is well-documented with a clear migration guide, and the separation of Importance Sampling (IS) and Rejection Sampling (RS) into distinct parameters (rollout_is and rollout_rs) greatly improves clarity and control. The code is cleaner, and the documentation updates across all 23 files are thorough. I found one critical issue where the implementation for the rollout_rs_threshold fallback is missing, which contradicts the documentation and will cause a runtime error for users relying on this feature. Once this is addressed, this will be an excellent pull request.

[szrlee]

Regarding the critical issue you identified: we've addressed it, but intentionally took a different approach than the suggested fallback. Here's our reasoning:

Our Solution: Explicit Configuration (No Fallback)

We removed the fallback behavior entirely and updated the documentation to match. This follows Python's "explicit is better than implicit" principle.

Why no fallback to rollout_is_threshold?

1. Prevents silent failures: The original bug silently skipped rejection sampling when rollout_rs_threshold=None. The fallback suggestion would still allow implicit behavior that could hide misconfiguration.

2. Clearer user intent: IS and RS serve different purposes (variance reduction vs. outlier rejection). Forcing users to explicitly set thresholds makes their intent clear and prevents accidentally using the wrong threshold.

3. Smart default exists: The function signature has rollout_rs_threshold: Optional[float] = 2.0, which provides a sensible default when the parameter is omitted.

Current Behavior:

# ✅ Works - uses default 2.0
compute_rollout_correction_and_rejection_mask(
    rollout_rs="token"  # rollout_rs_threshold defaults to 2.0
)

# ✅ Works - uses explicit value
compute_rollout_correction_and_rejection_mask(
    rollout_rs="token",
    rollout_rs_threshold=5.0
)

# ❌ Raises clear error - catches misconfiguration
compute_rollout_correction_and_rejection_mask(
    rollout_rs="token",
    rollout_rs_threshold=None  # ValueError with clear message
)

What Changed (commit ef7f8f57):

• Raise ValueError when rollout_rs is enabled but rollout_rs_threshold is explicitly None
• Updated docs: (uses rollout_is_threshold if null) → (required if rollout_rs is enabled)
• All 11 tests passing with this explicit approach

The key distinction: we only raise an error when someone explicitly passes None, not when they omit the parameter (which uses the default). This catches configuration errors while maintaining ease of use.

Does this approach work for your use case, or do you see a scenario where the fallback would be necessary?

[ISEEKYAN]

@szrlee The new APIs are very powerful and general, but I would say it a little complicated for a new user to get started. We cannot expect our users use the APIs after reading and understand all the related papers, so it will be very crucial to provide more verified combinations of the parameters and their alias, such as TIS, MIS. If I were a user with no background, I would search the key work MIS in the codebase, and I would be very confused with no direct code and docs regarding to it.

So I think it will be very helpful for users to add docs about:

1. combinations of TIS and MIS, and describe why their parameters are like that.
2. Add migrated combination of verified examples, such as megatron training crash when long term DAPO with qwen3-14b/qwen3-30b-a3b #3597 (comment). We have costed at lease several tens of thousand GPU hours to verify these workable combination.

Thanks again for the incredible contributions!

[wuxibin89]

@szrlee The new APIs are very powerful and general, but I would say it a little complicated for a new user to get started. We cannot expect our users use the APIs after reading and understand all the related papers, so it will be very crucial to provide more verified combinations of the parameters and their alias, such as TIS, MIS. If I were a user with no background, I would search the key work MIS in the codebase, and I would be very confused with no direct code and docs regarding to it.

So I think it will be very helpful for users to add docs about:

1. combinations of TIS and MIS, and describe why their parameters are like that.
2. Add migrated combination of verified examples, such as megatron training crash when long term DAPO with qwen3-14b/qwen3-30b-a3b #3597 (comment). We have costed at lease several tens of thousand GPU hours to verify these workable combination.

Thanks again for the incredible contributions!

Yes, we should add a typed config class, e.g RolloutCorrectionConfig, and set default value to best practice.

[szrlee]

@szrlee The new APIs are very powerful and general, but I would say it a little complicated for a new user to get started. We cannot expect our users use the APIs after reading and understand all the related papers, so it will be very crucial to provide more verified combinations of the parameters and their alias, such as TIS, MIS. If I were a user with no background, I would search the key work MIS in the codebase, and I would be very confused with no direct code and docs regarding to it.
So I think it will be very helpful for users to add docs about:

1. combinations of TIS and MIS, and describe why their parameters are like that.
2. Add migrated combination of verified examples, such as megatron training crash when long term DAPO with qwen3-14b/qwen3-30b-a3b #3597 (comment). We have costed at lease several tens of thousand GPU hours to verify these workable combination.

Thanks again for the incredible contributions!

Yes, we should add a typed config class, e.g RolloutCorrectionConfig, and set default value to best practice.

@ISEEKYAN @wuxibin89 this refactor (commit f8bd558) replaces dict-based config
with typed RolloutCorrectionConfig class and factory methods.

[tongyx361]

There are some unexpected memory / dependency issues (e.g., FSDP, vLLM) in the CI, which should be unrelated to this PR. We plan to retry the CI first, while investigating the root causes.

[ISEEKYAN]

leave some questions

[ISEEKYAN]

rollout_rs == "token", so it should be rollout_rs_token_masked_fraction instread of rollout_rs_seq_masked_fraction? maybe we can use the same metric name rollout_rs_masked_fraction for both modes?

[ISEEKYAN]

why do geo and seq modes have difference methods of calculating rollout_rs_ratio_fraction? I thought they are all sequence level, with only difference on how to calculate mismatch(mean vs sum on logp).

[lizipao]

Hello, I want to know why I get an error when using the following settings:

rollout_is=null
rollout_is_threshold=1.001
rollout_rs=geometric
rollout_rs_threshold=1.001
rollout_rs_threshold_lower=0.99
rollout_token_veto_threshold=1e-4

The error message is:

File "verl/verl/trainer/ppo/rollout_corr_helper.py", line 754, in compute_offpolicy_metrics
    assert response_mask.any(), "Expected at least one valid token in response_mask"
AssertionError: Expected at least one valid token in response_mask

After testing, I found that the all-zero mask is the modified_response_mask after compute_rollout_rejection_mask. Is this reasonable?

[wanghongqu]

File "verl/verl/trainer/ppo/rollout_corr_helper.py", line 754, in compute_offpolicy_metrics
assert response_mask.any(), "Expected at least one valid token in response_mask"
AssertionError: Expected at least one valid token in response_mask

the same error，any one can help me ?????

[ISEEKYAN]

File "verl/verl/trainer/ppo/rollout_corr_helper.py", line 754, in compute_offpolicy_metrics assert response_mask.any(), "Expected at least one valid token in response_mask" AssertionError: Expected at least one valid token in response_mask

the same error，any one can help me ?????

I think this situation is rare and data related, maybe you can increase the batch size to lower the probability of all invalid. Or you should add some function to skip this training step and redo the rollout until you have at least one valid data for training.