fix: Port contention issues between Ray/vLLM/Gym and sandbox workers by saumishr · Pull Request #3103 · NVIDIA-NeMo/RL

saumishr · 2026-07-07T20:16:10Z

What does this PR do ?

Eliminates port contention between all NeMo RL services (Ray, vLLM, SGLang, Gym, Sandbox) and the kernel's ephemeral port range by pinning every service port below the observed ephemeral floor.

Fix

Pin all service ports below 9000. Final layout (documented in ray.sub and nemo_rl/distributed/virtual_cluster.py):

Ray worker gRPC moved to 2000-2999 (MIN_WORKER_PORT/MAX_WORKER_PORT in ray.sub), and --min-worker-port/--max-worker-port added to ray start --head so the head node is constrained to the same range as workers.
Ray GCS + management ports moved below the floor: GCS/client 1200-1201, node/object/runtime-env/dashboard-agent/metrics ports to 1301-1312, dashboard-agent-listen to 1311.
Master address / TCPStore moved to 1400-1999 (DEFAULT_MASTER_PORT_RANGE_* in virtual_cluster.py).
NeMo RL generation HTTP moved to 3000-4999 (port_range_low/high in the example configs and DEFAULT_GENERATION_PORT_RANGE_* in virtual_cluster.py).
NeMo Gym HTTP moved to 5000-5999 (example configs and DEFAULT_GYM_PORT_RANGE_*, referenced from nemo_gym.py).
Sandbox Nginx stays at 6000; uWSGI workers moved to 6001+ by exporting SANDBOX_WORKER_BASE_PORT=6001 and NGINX_PORT=6000 into the sandbox srun (the container's start-with-nginx.sh already supports these env vars — no image change needed).
vLLM engine TP/DP rendezvous moved from 20001 to 7000 (DEFAULT_VLLM_PORT_RANGE_LOW in virtual_cluster.py, consumed by vllm_worker.py).
SGLang engine ports (server / nccl / dist_init) moved from 15000 to the same 7000-8999 band (vLLM and SGLang are mutually exclusive backends); free-port helper defaults lowered from 10000. The SGLang router HTTP port already allocated from 3000-4000 and is unchanged.
Propagated the new bands to all example configs that hardcoded the old ranges, kept the frozen tests/unit/reference_configs/* copies in sync, and refreshed stale port-range comments.

Issues

None

Usage

None

copy-pr-bot · 2026-07-07T20:16:14Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

saumishr · 2026-07-08T01:49:11Z

/ok to test a42fc85

saumishr · 2026-07-08T19:58:21Z

/ok to test dc8705e

ananthsub · 2026-07-09T17:33:42Z

/ok to test 7a4ac77

terrykong

Reviewed with a coordinated agent team (RL expert, test agent, bug finder, devil's advocate). Nice, thorough fix — the new layout is internally consistent across ray.sub, virtual_cluster.py, all example configs, and the frozen reference configs, and we verified the key premises (sandbox NGINX_PORT/SANDBOX_WORKER_BASE_PORT are consumed by NeMo-Skills' start-with-nginx.sh; the 1301-1312 odd/even head-offset math checks out; the shrunken master/generation/gym ranges can't realistically be exhausted; head+worker[0] sharing 2000-2999 is safe since worker ports are bind-probed).

Local verification: tests/unit/distributed/test_virtual_cluster.py passes (31 passed) including the updated vLLM port parametrizations, test_config_v2.py::test_reference_configs_up_to_date passes (7 passed), and all pre-commit hooks pass.

Findings are inline (2 doc syncs, 1 naming convention, plus optional nits). One soft ask: it would be helpful to add a line to the PR description noting what job validated the fix end-to-end (e.g. N-node run on the affected GB200 cluster with no EADDRINUSE) so future readers know the layout was battle-tested.

Generated by Claude Code

saumishr · 2026-07-09T23:20:13Z

/ok to test 1d66c6a

…workers (cherry picked from commit 5816e9131753720b50c3892b5cf4f74c5c74c5c2) Signed-off-by: Saurabh Mishra <sauramishra@nvidia.com> [NemoRL] Move Ray management ports as well below the ephemeral port range (cherry picked from commit d993e7b5be8a6d196ce6d57b118ebcdd30cef613) Signed-off-by: Saurabh Mishra <sauramishra@nvidia.com> chore: propagate below-ephemeral port ranges to remaining configs and tests The port-contention fix moved the default port bands below the 9000 ephemeral floor (generation 3000-4999, gym 5000-5999, vLLM 7000+, master 1400-1999). Several example configs still hardcoded the old ranges, which would override the new defaults with colliding values. This propagates the new layout to those configs, mirrors it into the frozen reference configs (tests/unit/reference_configs), refreshes stale port-range comments, and updates the vLLM port-assignment test to the new DEFAULT_VLLM_PORT_RANGE_LOW. Signed-off-by: Saurabh Mishra <sauramishra@nvidia.com> fix: move SGLang engine ports below the ephemeral floor SGLang allocated its engine ports (server / nccl / dist_init) starting at 15000, above the 9000 ephemeral floor, and the free-port helpers defaulted to 10000 — both prone to the same TOCTOU collisions the port layout is meant to avoid. SGLang and vLLM are mutually exclusive backends, so SGLang now shares the below-floor engine rendezvous band (7000-8999). Refreshes the stale port-range comment and notes SGLang in the central port-layout maps (ray.sub, virtual_cluster.py). The router HTTP port already allocated from 3000-4000 (generation band) and is unchanged. Signed-off-by: Saurabh Mishra <sauramishra@nvidia.com>

ananthsub · 2026-07-10T17:35:45Z

/ok to test f673c74

vLLM 0.25's RayExecutorV2 picks the torch.distributed TCPStore port with a bind-probe (Step 3) but only binds it much later, in the rank-0 worker's init_process_group. In between, Step 4 builds the broadcast MessageQueue; when the engine spans nodes that queue needs a real TCP socket, so it calls get_open_port() and binds and holds the result (shm_broadcast.py: remote_subscribe_port = get_open_port(), then remote_socket.bind(...)). Both searches start at VLLM_PORT, so the queue takes the very port the probe just released and startup dies with EADDRINUSE (DeepSeek-V3 generation TP=32, observed on port 7000). Engines that fit on one node bind an ipc:// socket instead and never allocate a TCP port here, which is why only node-spanning engines are affected. Fix: patch _select_tcpstore_port to start its search at VLLM_PORT + 32, past the queue's scan range. Both ports stay inside the engine's 100-port window and therefore below the OS ephemeral floor (as low as 9000 on some nodes). vLLM applies the same disjoint-window idea to co-located DP engines a few lines below, seeding them from master_port + 100 + rank * 32. Deliberately NOT fixed by leaving VLLM_PORT unset for these engines: that sends vLLM to _get_open_port()'s s.bind(("", 0)) fallback, i.e. kernel-assigned ephemeral ports, which is exactly the TOCTOU contention the reserved port layout exists to prevent (#2380, #3103). The port assignment in configure_worker is therefore unchanged from main. The patch verifies its own result by reading the file back, so a patch that silently fails to land is visible in worker logs instead of degrading to the collision at runtime. Signed-off-by: Terry Kong <terryk@nvidia.com>

vLLM 0.25's RayExecutorV2 picks the torch.distributed TCPStore port with a bind-probe (Step 3) but only binds it much later, in the rank-0 worker's init_process_group. In between, Step 4 builds the broadcast MessageQueue; when the engine spans nodes that queue needs a real TCP socket, so it calls get_open_port() and binds and holds the result (shm_broadcast.py: remote_subscribe_port = get_open_port(), then remote_socket.bind(...)). Both searches start at VLLM_PORT, so the queue takes the very port the probe just released and startup dies with EADDRINUSE (DeepSeek-V3 generation TP=32, observed on port 7000). Engines that fit on one node bind an ipc:// socket instead and never allocate a TCP port here, which is why only node-spanning engines are affected. Fix: patch _select_tcpstore_port to search from VLLM_PORT + 32, past the queue's scan range. Both ports stay inside the engine's 100-port window and therefore below the OS ephemeral floor (as low as 9000 on some nodes). The offset must run *before* the local_dp_rank test, not inside it. _select_tcpstore_port's disjoint-window branch reads as though it only serves co-located DP engines, but ParallelConfig.__post_init__ takes its offline-SPMD path for every engine NeMo-RL builds and assigns data_parallel_rank_local = VLLM_DP_RANK_LOCAL (0 by default) and data_parallel_master_port = VLLM_DP_MASTER_PORT (0 by default). A plain non-DP engine therefore arrives with local_dp_rank=0, not None: the None branch is dead code, and the DP branch searches from 0 + 100 + 0*32 = 100, fails all 32 attempts on the privileged range, and falls through to get_open_port() -- straight back to VLLM_PORT, the exact port the MessageQueue holds. An earlier revision of this commit put the offset inside the None branch and was silently inert on hardware. See RL-1104. Deliberately NOT fixed by leaving VLLM_PORT unset: that sends vLLM to _get_open_port()'s s.bind(("", 0)) fallback, i.e. kernel-assigned ephemeral ports, which is exactly the TOCTOU contention the reserved port layout exists to prevent (#2380, #3103). The port assignment in configure_worker is therefore unchanged from main. tests/unit/models/generation/test_vllm_tcpstore_port.py pins the arithmetic using the values vLLM actually passes (local_dp_rank=0, master_port=0) and the real patch applied to a copy of the installed executor, so a fix on a dead branch fails the suite instead of passing review. It also pins the anchor snippet so an upstream rename cannot silently turn the patch into a no-op. The patch additionally re-reads the file and warns if the marker is absent. Signed-off-by: Terry Kong <terryk@nvidia.com>

vLLM 0.25's MessageQueue picks the port for its remote (TCP) socket with get_open_port(), which binds a probe socket, releases it, and returns the number; ZMQ binds it for real several statements later. That window is a TOCTOU race, and on a non-driver node it is lost reliably rather than occasionally. Every RayWorkerProc on a non-driver node takes n_local_reader=0 (ray_executor_v2.py::_init_message_queues), so every one of them needs a real TCP port and they all scan from the same VLLM_PORT -- 7000 for a node-spanning engine. _init_message_queues runs immediately after init_device(), whose process-group setup is a collective barrier, so all workers on the node reach the probe within microseconds of each other, all see the same port free, and all but one die with: zmq.error.ZMQError: Address already in use (addr='tcp://10.65.1.9:7000') Workers on the driver node take n_local_reader=1 and use an ipc:// socket, which is why only engines spanning >= 2 nodes are affected -- and why no nightly test catches it, since none runs an engine whose tensor_parallel_size * pipeline_parallel_size exceeds cluster.gpus_per_node. Fix the race at the bind rather than the probe: retry, advancing past the port that was lost. This terminates because a port a peer already holds with ZMQ is visible to the next probe. Ports stay anchored at VLLM_PORT rather than falling back to kernel-ephemeral ones, which is the contention the reserved sub-ephemeral band exists to prevent (#2380, #3103). Patching the bind rather than handing each worker a private start port also covers every other MessageQueue with a remote reader, including the executor's own rpc_broadcast_mq. The code is byte-identical on vLLM main; this is an upstream bug, so the same change is worth proposing there. Verified against the real vLLM 0.25.1 MessageQueue with concurrent processes: unpatched, 6/8 and 30/32 workers die (3/3 runs each); patched, 8/8 and 32/32 bind distinct ports inside the reserved band. Signed-off-by: Terry Kong <terryk@nvidia.com>

vLLM 0.25's RayExecutorV2 picks the torch.distributed TCPStore port with a bind-probe (Step 3) but only binds it much later, in the rank-0 worker's init_process_group. In between, Step 4 builds the broadcast MessageQueue; when the engine spans nodes that queue needs a real TCP socket, so it calls get_open_port() and binds and holds the result (shm_broadcast.py: remote_subscribe_port = get_open_port(), then remote_socket.bind(...)). Both searches start at VLLM_PORT, so the queue takes the very port the probe just released and startup dies with EADDRINUSE (DeepSeek-V3 generation TP=32, observed on port 7000). Engines that fit on one node bind an ipc:// socket instead and never allocate a TCP port here, which is why only node-spanning engines are affected. Fix: patch _select_tcpstore_port to search from VLLM_PORT + 32, past the queue's scan range. Both ports stay inside the engine's 100-port window and therefore below the OS ephemeral floor (as low as 9000 on some nodes). The offset must run *before* the local_dp_rank test, not inside it. _select_tcpstore_port's disjoint-window branch reads as though it only serves co-located DP engines, but ParallelConfig.__post_init__ takes its offline-SPMD path for every engine NeMo-RL builds and assigns data_parallel_rank_local = VLLM_DP_RANK_LOCAL (0 by default) and data_parallel_master_port = VLLM_DP_MASTER_PORT (0 by default). A plain non-DP engine therefore arrives with local_dp_rank=0, not None: the None branch is dead code, and the DP branch searches from 0 + 100 + 0*32 = 100, fails all 32 attempts on the privileged range, and falls through to get_open_port() -- straight back to VLLM_PORT, the exact port the MessageQueue holds. An earlier revision of this commit put the offset inside the None branch and was silently inert on hardware. See RL-1104. Deliberately NOT fixed by leaving VLLM_PORT unset: that sends vLLM to _get_open_port()'s s.bind(("", 0)) fallback, i.e. kernel-assigned ephemeral ports, which is exactly the TOCTOU contention the reserved port layout exists to prevent (#2380, #3103). The port assignment in configure_worker is therefore unchanged from main. tests/unit/models/generation/test_vllm_tcpstore_port.py pins the arithmetic using the values vLLM actually passes (local_dp_rank=0, master_port=0) and the real patch applied to a copy of the installed executor, so a fix on a dead branch fails the suite instead of passing review. It also pins the anchor snippet so an upstream rename cannot silently turn the patch into a no-op. The patch additionally re-reads the file and warns if the marker is absent. Signed-off-by: Terry Kong <terryk@nvidia.com>

vLLM 0.25's MessageQueue picks the port for its remote (TCP) socket with get_open_port(), which binds a probe socket, releases it, and returns the number; ZMQ binds it for real several statements later. That window is a TOCTOU race, and on a non-driver node it is lost reliably rather than occasionally. Every RayWorkerProc on a non-driver node takes n_local_reader=0 (ray_executor_v2.py::_init_message_queues), so every one of them needs a real TCP port and they all scan from the same VLLM_PORT -- 7000 for a node-spanning engine. _init_message_queues runs immediately after init_device(), whose process-group setup is a collective barrier, so all workers on the node reach the probe within microseconds of each other, all see the same port free, and all but one die with: zmq.error.ZMQError: Address already in use (addr='tcp://10.65.1.9:7000') Workers on the driver node take n_local_reader=1 and use an ipc:// socket, which is why only engines spanning >= 2 nodes are affected -- and why no nightly test catches it, since none runs an engine whose tensor_parallel_size * pipeline_parallel_size exceeds cluster.gpus_per_node. Fix the race at the bind rather than the probe: retry, advancing past the port that was lost. This terminates because a port a peer already holds with ZMQ is visible to the next probe. Ports stay anchored at VLLM_PORT rather than falling back to kernel-ephemeral ones, which is the contention the reserved sub-ephemeral band exists to prevent (#2380, #3103). Patching the bind rather than handing each worker a private start port also covers every other MessageQueue with a remote reader, including the executor's own rpc_broadcast_mq. The code is byte-identical on vLLM main; this is an upstream bug, so the same change is worth proposing there. Verified against the real vLLM 0.25.1 MessageQueue with concurrent processes: unpatched, 6/8 and 30/32 workers die (3/3 runs each); patched, 8/8 and 32/32 bind distinct ports inside the reserved band. Signed-off-by: Terry Kong <terryk@nvidia.com>

saumishr changed the title ~~[NemoRL] Fix port contention issues between Ray/vLLM/Gym and sandbox workers~~ fix: Port contention issues between Ray/vLLM/Gym and sandbox workers Jul 7, 2026

saumishr marked this pull request as ready for review July 7, 2026 20:20

saumishr requested review from a team as code owners July 7, 2026 20:20

saumishr force-pushed the fix/ray-port-contention branch 2 times, most recently from 6aa6039 to a42fc85 Compare July 8, 2026 01:35

saumishr added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Jul 8, 2026

saumishr force-pushed the fix/ray-port-contention branch from a42fc85 to dc8705e Compare July 8, 2026 19:55

saumishr requested review from ananthsub and terrykong July 8, 2026 22:20

ananthsub previously approved these changes Jul 8, 2026

View reviewed changes

saumishr force-pushed the fix/ray-port-contention branch from dc8705e to 7a4ac77 Compare July 9, 2026 16:50

terrykong reviewed Jul 9, 2026

View reviewed changes

Comment thread ray.sub Outdated

Comment thread ray.sub

Comment thread nemo_rl/distributed/virtual_cluster.py

Comment thread tests/unit/distributed/test_virtual_cluster.py

Comment thread ray.sub Outdated

saumishr dismissed ananthsub’s stale review via 7801856 July 9, 2026 21:01

saumishr force-pushed the fix/ray-port-contention branch from 7a4ac77 to 7801856 Compare July 9, 2026 21:01

saumishr requested a review from a team as a code owner July 9, 2026 21:01

github-actions Bot added Documentation Improvements or additions to documentation labels Jul 9, 2026

saumishr force-pushed the fix/ray-port-contention branch from 7801856 to 1d66c6a Compare July 9, 2026 23:03

saumishr force-pushed the fix/ray-port-contention branch from 1d66c6a to f673c74 Compare July 10, 2026 04:54

This was referenced Jul 10, 2026

refactor(sglang): allocate ports from reserved ranges #2997

Merged

perf(ray.sub): tune GCS for large-scale multi-node init bursts #3158

Merged

terrykong enabled auto-merge (squash) July 11, 2026 01:04

terrykong approved these changes Jul 11, 2026

View reviewed changes

terrykong merged commit c1868b1 into NVIDIA-NeMo:main Jul 11, 2026
136 of 139 checks passed

This was referenced Jul 25, 2026

build: bump vllm 0.20.0 → 0.25.1 #3280

Merged

fix(vllm): apply the TCPStore port offset on the branch vLLM actually takes #3350

Closed

terrykong mentioned this pull request Jul 26, 2026

fix(vllm): retry MessageQueue remote bind so node-spanning engines start #3355

Merged

terrykong mentioned this pull request Aug 2, 2026

[Dynamo] managed generation backend cleanup tracking issue #3459

Open

21 tasks

terrykong mentioned this pull request Aug 23, 2026

mcore inference HTTP server port pick is TOCTOU-prone; mirror vLLM's held-socket reservation #3775

Open

yfw mentioned this pull request Aug 27, 2026

fix: isolate SingleController service port ranges #3884

Merged

Navigation Menu

Uh oh!

fix: Port contention issues between Ray/vLLM/Gym and sandbox workers - #3103

fix: Port contention issues between Ray/vLLM/Gym and sandbox workers#3103
terrykong merged 1 commit into
NVIDIA-NeMo:mainfrom
saumishr:fix/ray-port-contention

saumishr commented Jul 7, 2026

Uh oh!

copy-pr-bot Bot commented Jul 7, 2026

Uh oh!

saumishr commented Jul 8, 2026

Uh oh!

saumishr commented Jul 8, 2026

Uh oh!

ananthsub commented Jul 9, 2026

Uh oh!

terrykong left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saumishr commented Jul 9, 2026

Uh oh!

ananthsub commented Jul 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

saumishr commented Jul 7, 2026

What does this PR do ?

Fix

Issues

Usage

Uh oh!

copy-pr-bot Bot commented Jul 7, 2026

Uh oh!

saumishr commented Jul 8, 2026

Uh oh!

saumishr commented Jul 8, 2026

Uh oh!

ananthsub commented Jul 9, 2026

Uh oh!

terrykong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saumishr commented Jul 9, 2026

Uh oh!

ananthsub commented Jul 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants