[rollout] fix: Add "non_block" argument compatibility to collective_rpc()

[kevssim]

What does this PR do?

This PR fixes a TypeError that occurs when newer versions of vLLM (v0.11+) attempt to call ExternalZeroMQDistributedExecutor.collective_rpc.

The issue stems from a recent vLLM update (vllm-project/vllm#24219) that added the keyword argument non_block to the Executor.collective_rpc interface. Since the verl implementation of collective_rpc did not define this parameter, calling it with non_block=True resulted in the error: TypeError: ExternalZeroMQDistributedExecutor.collective_rpc() got an unexpected keyword argument 'non_block'.

By using **extra_kwargs in the function signature, we ensure compatibility with both legacy and modern vLLM interfaces without affecting the existing ZeroMQ non-blocking logic.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[gemini-code-assist]

Code Review

This pull request aims to fix a TypeError in collective_rpc when used with newer versions of vLLM by adding **extra_kwargs to the method signature. While this correctly resolves the signature mismatch, the implementation has a critical flaw: the arguments captured by **extra_kwargs are not passed on to the remote worker. This will cause any new keyword arguments to be silently ignored, potentially leading to incorrect behavior. I've left a critical comment with a suggested fix to properly merge these arguments before they are sent.

[gemini-code-assist]

While adding **extra_kwargs fixes the TypeError, the collected arguments are then discarded, as they are not used in the function body. This means any new keyword arguments from vllm (like non_block, but also any others) will be silently ignored instead of being passed to the remote worker via the pickled message. This is a critical issue as it can lead to unexpected behavior if the remote method relies on these arguments.

The pull request description mentions not wanting to affect the existing non-blocking logic. You can achieve this while still correctly handling other arguments by merging extra_kwargs and then explicitly handling or removing non_block.

I recommend updating the function body to merge the arguments:

    all_kwargs = kwargs or {}
    all_kwargs.update(extra_kwargs)
    
    # Now `all_kwargs` contains all keyword arguments.
    # You can handle `non_block` if needed, for example, by ignoring it:
    all_kwargs.pop("non_block", None)

    # ...
    
    message = pickle.dumps((sent_method, args, all_kwargs))
    # ...

This ensures forward compatibility with any other keyword arguments that might be added in the future, while still addressing the immediate TypeError. Since the function body is not in this diff, I cannot provide a direct code suggestion.

[huaiyizhao]

Thanks, it works!

[huaiyizhao]

verl is TOO DEEPLY coupled with inference and training backend using its hybrid engine. I think we need a better tradeoff between compatibility and efficiency.