[Bugfix] Fix message queue initialization order for cross-node DP

[jianzs]

Summary

Fixes a bug where _init_message_queues() was called before worker.init_device(), causing failures when nnodes_within_dp > 1.

Problem

When nnodes_within_dp > 1 (cross-node DP group), the message queue initialization requires get_inner_dp_world_group() which is only available after parallel groups are initialized during init_device(). The incorrect order caused:

AssertionError: inner dp world group is not initialized

Root Cause

Dependency chain:

_init_message_queues() (Branch B)
    └── requires get_inner_dp_world_group()
            └── requires _INNER_DP_WORLD variable
                    └── initialized in initialize_model_parallel()
                            └── called in worker.init_device()

Fix

Move _init_message_queues() call to after worker.init_device() and worker.load_model().

Affected Configurations

This bug affects ANY configuration where nnodes_within_dp > 1, including:

• Uniform cross-node DP (e.g., PP=4, TP=2, DP=1 across 2 nodes)
• Non-uniform topology scenarios

Test Plan

• E2E-20: Uniform cross-node PP=4, TP=2, DP=1 (nnodes_within_dp=2) - PASSES with fix
• Verified same test FAILS on main branch commit a60985b07

[gemini-code-assist]

Code Review

This pull request addresses an initialization order bug for cross-node data parallelism by moving the _init_message_queues() call, ensuring that required distributed process groups are initialized before use and resolving the AssertionError. While the fix itself is logically sound, enabling this functionality exposes a critical security vulnerability in the distributed communication layer. The use of unauthenticated ZMQ sockets combined with the pickle serialization format allows for potential data leakage and Remote Code Execution (RCE) across the network, especially in multi-node deployments. The underlying architecture requires urgent security hardening.

[gemini-code-assist]

The initialization of message queues via _init_message_queues (now correctly ordered for cross-node configurations) introduces a significant security risk when used in multi-node environments. The underlying MessageQueue implementation (in vllm/distributed/device_communicators/shm_broadcast.py) uses unauthenticated ZMQ sockets and the pickle serialization format.

In cross-node setups (where nnodes_within_dp > 1), the driver binds to a network-accessible IP (obtained via get_ip()), allowing any attacker on the network to eavesdrop on sensitive data (such as prompts and generated tokens) or achieve Remote Code Execution (RCE) on worker nodes by sending malicious pickled objects to the ZMQ ports.

While vLLM defines a VLLM_ALLOW_INSECURE_SERIALIZATION environment variable, it is not currently enforced in the MessageQueue component, which uses pickle unconditionally. It is highly recommended to implement ZMQ authentication (e.g., using the CURVE security mechanism) and transition to a safer serialization format (e.g., msgpack or protobuf) for distributed communication.

[njhill]

Thanks @jianzs, I think this is a duplicate of #35892.

[jianzs]

Thanks @jianzs, I think this is a duplicate of #35892.

Got it.