[misc] fix: deprecate rollout.mode config option

[yurekami]

Summary

Addresses #4604 - Deprecates the actor_rollout_ref.rollout.mode config option since only async mode is now supported.

Changes:

1. RolloutConfig validation (verl/workers/config/rollout.py):

   • Added validation in __post_init__ to raise ValueError for "sync" mode
   • Added DeprecationWarning for unknown modes

2. Simplified main_ppo.py:

   • Removed sync mode check (moved to config validation)
   • Always uses AsyncActorRolloutRefWorker instead of conditional selection

3. Removed dead code paths:

   • fsdp_workers.py: Removed sync mode switch-to-trainer-mode code
   • megatron_workers.py: Same cleanup
   • ray_trainer.py: Simplified async mode checks since sync is no longer valid

Backward Compatibility:

The mode field is kept in the config for backward compatibility with existing shell scripts that set actor_rollout_ref.rollout.mode=async. These scripts will continue to work.

Follow-up Work:

Shell scripts and documentation that explicitly set mode=async can be updated in separate PRs to remove the now-redundant setting.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[gemini-code-assist]

Code Review

This pull request effectively deprecates the rollout.mode configuration option, correctly removing the 'sync' mode and simplifying the codebase to only support 'async' mode. The changes are well-contained and the validation logic has been appropriately moved to the configuration class. I have one suggestion to make the validation stricter to avoid silent behavior changes for users with misconfigured modes, which could be easily missed.

[gemini-code-assist]

Using warnings.warn for unknown rollout.mode values can lead to silent behavior changes that might be missed by users, as DeprecationWarning is often suppressed by default. Since only 'async' mode is supported now, any other value should be treated as an error to prevent unexpected behavior. For example, a typo like mode=asyn would have previously fallen back to sync-like behavior, but with this change, it would be treated as 'async' with just a potentially hidden warning. Raising a ValueError for all non-'async' modes is safer and more explicit about the supported configuration.

        if self.mode != "async":
            if self.mode == "sync":
                raise ValueError(
                    "Rollout mode 'sync' has been removed. Please set "
                    "`actor_rollout_ref.rollout.mode=async` or remove the mode setting entirely."
                )
            raise ValueError(
                f"Unknown rollout mode '{self.mode}'. Only 'async' mode is supported. "
                "Please set `actor_rollout_ref.rollout.mode=async` or remove the mode setting entirely."
            )