Skip to content

shm_broadcast: bind XPUB port atomically to avoid pinned-port race - #1

Closed
charlotte12l wants to merge 1 commit into
mainfrom
xingyuliu/fix-shm-broadcast-port-race
Closed

charlotte12l wants to merge 1 commit into
mainfrom
xingyuliu/fix-shm-broadcast-port-race

Conversation

@charlotte12l

@charlotte12l charlotte12l commented Jun 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

When SGLANG_PORT is pinned and multiple MessageQueue writers start
concurrently on the same node, the XPUB local socket setup uses
get_open_port() followed by a later bind() — a non-atomic
check-then-bind. Two writers can be handed the same port and then collide
on bind() with EADDRINUSE.

One scheduler subprocess dies at init:

zmq.error.ZMQError: Address already in use (addr='tcp://127.0.0.1:50000')
  File ".../sglang/srt/distributed/parallel_state.py", line 464, in __init__
    self.mq_broadcaster = MessageQueue.create_from_process_group(self.cpu_group, 1 << 22, 6)
  File ".../sglang/srt/distributed/device_communicators/shm_broadcast.py", line 214, in __init__
    self.local_socket.bind(socket_addr)

which then cascades into a gloo failure on every rank:

RuntimeError: [gloo/transport/tcp/pair.cc] Connection reset by peer
  File ".../sglang/srt/model_executor/model_runner.py", line 1189, in get_available_gpu_memory
    torch.distributed.all_reduce(tensor, op=ReduceOp.MIN, group=group)

This replaces get_open_port() + bind() with zmq's bind_to_random_port,
which picks and binds a free port atomically. When SGLANG_PORT is set the
search starts at that base (min=base, max=base + 8) to preserve the
existing port range; otherwise it falls back to the default ephemeral range.

A similar atomic fix is proposed upstream in sgl-project#15885 (PR sgl-project#15903).

Test

Concurrent MessageQueue writers under a pinned SGLANG_PORT no longer
hit the EADDRINUSE error; each binds a distinct port.


CI States

Latest PR Test (Base): ❌ Run #28311918658
Latest PR Test (Extra): ❌ Run #28311918565

When SGLANG_PORT is pinned and multiple MessageQueue writers start
concurrently on the same node, get_open_port() followed by a later bind()
is a non-atomic check-then-bind: two writers can be handed the same port
and then collide on bind() with EADDRINUSE. The loser dies, drops its
gloo peer, and cascades into "connection reset by peer" on the next
collective.

Replace get_open_port()+bind() with zmq's bind_to_random_port, which
picks and binds a free port atomically. When SGLANG_PORT is set the
search starts at that base (min=base, max=base+8) to preserve the
existing port range; otherwise it falls back to the default ephemeral
range.

A similar atomic fix is proposed upstream in sgl-project#15885
(PR sgl-project#15903).

Signed-off-by: xingyuliu <xingyuliu@fb.com>
@charlotte12l

Copy link
Copy Markdown
Owner Author

Retargeting to upstream sgl-project/sglang.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant