Skip to content

fix(sglang): Port collision when colocating 2 workers on same node with DP - #584

Merged
weireweire merged 1 commit into
NVIDIA:mainfrom
trevor-m:colocate-dp-ports
Oct 16, 2026
Merged

weireweire merged 1 commit into
NVIDIA:mainfrom
trevor-m:colocate-dp-ports

Conversation

@trevor-m

@trevor-m trevor-m commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

When colocating multiple workers on a single node, srt-slurm assigns each worker's kv event publisher a port sequentially:

Worker 0 gets port 5200

^[[2m2026-10-06T19:35:29.513690Z^[[0m ^[[32m INFO^[[0m ^[[2margs.parse_args^[[0m^[[2m:^[[0m Derived use_kv_events=True from kv_events_config={"publisher": "zmq", "topic": "kv-events", "endpoint": "tcp://*:5200"}

Worker 1 gets port 5201

^[[2m2026-10-06T19:35:28.575815Z^[[0m ^[[32m INFO^[[0m ^[[2margs.parse_args^[[0m^[[2m:^[[0m Derived use_kv_events=True from kv_events_config={"publisher": "zmq", "topic": "kv-events", "endpoint": "tcp://*:5201"}

However, if we are using DP then sglang will apply dp worker rank as an offset to the base kv event publisher port to determine that DP worker's port. This will lead to a collision since worker 0 rank 1 will get port 5200+1 which collides with worker 1 rank 0 port.

[2026-10-06 12:38:42 DP0 TP0 EP0] Starting ZMQ publisher thread
[2026-10-06 12:38:42 DP1 TP1 EP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 6082, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 705, in __init__
    self.init_kv_events_publisher()
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2393, in init_kv_events_publisher
    self.kv_events_publisher = SchedulerKvEventsPublisher( 
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 11, in __init__
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler_components/kv_events_publisher.py", line 57, in __post_init__
    self.init_kv_events(self.kv_events_config)
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler_components/kv_events_publisher.py", line 64, in init_kv_events
    self.kv_event_publisher = EventPublisherFactory.create(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/kv_events.py", line 646, in create
    return constructor(attn_dp_rank=attn_dp_rank, **config_dict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/kv_events.py", line 394, in __init__
    self._socket_setup()
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/kv_events.py", line 472, in _socket_setup
    self._pub.bind(self._endpoint)
  File "/opt/sglang/lib/python3.12/site-packages/zmq/sugar/socket.py", line 322, in bind
    super().bind(addr)
  File "zmq/backend/cython/_zmq.py", line 1009, in zmq.backend.cython._zmq.Socket.bind
    _check_rc(rc)
    ^^^^^^^^^^^
  File "zmq/backend/cython/_zmq.py", line 190, in zmq.backend.cython._zmq._check_rc
    raise ZMQError(errno)
    ^^^^^^^^^^^
zmq.error.ZMQError: Address already in use (addr='tcp://*:5201')

To fix this, we spread out the base kv event publisher port for each worker based on the dp size, so worker 0 gets 5200 and worker 1 gets 5202 for example.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

src/srtctl diff: +50 -28

SGLang binds each KV-event publisher at the configured port plus its DP
rank (the attention-DP rank, or the replica rank without attention DP).
Two DP workers on one node got consecutive base ports, so worker 0's rank 1
bound worker 1's base and failed with "Address already in use".

NVIDIA#473 reserved a dp-size block, but only in Dynamo sidecar mode with KV
events, and it read only dp-size. Reserve the block for every SGLang
process, as the vLLM backend does for its local DP ranks, and size it with
SGLangBackend.num_dp_ranks (dp-size x attn-dp-size). sgl-project/sglang#41818
added --attn-dp-size and left dp-size counting replicas; the deprecated
dp-size + enable-dp-attention spelling resolves to the same layout. The
sidecar follower-publisher check, the replica NCCL-port rule and the AIPerf
follower scrape now read the same resolver, so --attn-dp-size works there
too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.73684% with 1 line in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@4406bb7). Learn more about missing BASE report.

Files with missing lines Patch % Lines
src/srtctl/backends/sglang.py 94.44% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #584   +/-   ##
=======================================
  Coverage        ?   85.38%           
=======================================
  Files           ?      162           
  Lines           ?    23470           
  Branches        ?        0           
=======================================
  Hits            ?    20041           
  Misses          ?     3429           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@weireweire
weireweire merged commit ae3d687 into NVIDIA:main Oct 16, 2026
12 checks passed
nvpohanh added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 16, 2026
* chore: bump srt-slurm to v2.59.1

Move the srt-slurm submodule from v2.51.1 (545233a0) to v2.59.1 (ae3d6877).
v2.59.1 adds NVIDIA/srt-slurm#584, which reserves one KV-event port per DP
rank for SGLang workers, so two attention-DP workers on one node no longer
collide on base port + DP rank.

The pin now includes NVIDIA/srt-slurm#573 and #602, so drop their backport
patches. The #602 patch no longer applies to v2.59.1 and would fail every
srt-slurm job at the launcher's git apply. Upstream #573 merged a different
design (idle GPUs on worker nodes are excused only after a worker GPU UUID
probe) than the backported filter.

Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep the participating-GPU filter on srt-slurm v2.59.1

v2.59.1 includes NVIDIA/srt-slurm#573, which keeps every exporter row and
excuses idle worker-node GPUs only after a DCGM worker GPU UUID probe. The
InferenceX power validator (infx/results/power/multinode.py) recomputes
device validity without that excusal. An NVIDIA point that uses part of an
exclusive node would store publication_valid=true, recompute
unexpected_device, and lose its power metrics.

Keep 573-participating-gpus.patch, which still applies to v2.59.1, until
the validator follows the upstream rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants