[ROCm] Enable DeepEP ROCm as all2allbackend for AMD GPUs. by lcskrishna · Pull Request #34692 · vllm-project/vllm

lcskrishna · 2026-02-17T11:53:31Z

This PR is to integrate changes required to run DeepEP as an all2allbackend on AMD GPUs.

co-authored by : @itej89

The following changes are performed:

The current codebase is modified to run DeepEP backend OOB on AMD GPUs without any issues.
FusedBatchedMoE kernels can run on AMD GPUs with float8_e4m3fnuz format added.

Related steps:

Docker update with ROCm/DeepEP installation steps. check [AMD][Build] Add DeepEP to ROCm Dockerfile #36086
Unit tests to add.

cc: @kenroche

Purpose

Test Plan

Accuracy Test

MI300 + CX7

Serve on 1N DeepSeek-V3 with DeepEP - Low Latency | High Throughput

ROCSHMEM_MAX_NUM_CONTEXTS=256  \ # Required for LL
HSA_NO_SCRATCH_RECLAIM=1 \
ROCSHMEM_HEAP_SIZE=7524589824 \
VLLM_ENGINE_READY_TIMEOUT_S=7800 \
vllm serve /models/DeepSeek-V3 \
    --tensor-parallel-size 1 \
    --data-parallel-size 8 \
    --enforce-eager \
    --enable-expert-parallel \
    --host 127.0.0.1 \
    --port 2000 \
    --block-size 1 \
    --all2all-backend deepep_low_latency | deepep_high_throughput

Serve on 2N DeepSeek-V3 with DeepEP - Low Latency

# NODE 1 
ROCSHMEM_MAX_NUM_CONTEXTS=256  \ # Required for LL
HSA_NO_SCRATCH_RECLAIM=1 \
ROCSHMEM_HEAP_SIZE=7524589824 \
VLLM_ENGINE_READY_TIMEOUT_S=7800 \
vllm serve /models/DeepSeek-V3 \   
    --tensor-parallel-size 1 \
    --data-parallel-size 16 \
    --data-parallel-size-local 8 \
    --enforce-eager \
    --enable-expert-parallel \
    --host 127.0.0.1 \
    --port 2000 \
    --block-size 1 \
    --data-parallel-address 10.158.214.101 \
    --data-parallel-rpc-port 13345 \
    --api-server-count=8 \
    --all2all-backend deepep_low_latency | deepep_high_throughput

# NODE 2 
HSA_NO_SCRATCH_RECLAIM=1 \
ROCSHMEM_MAX_NUM_CONTEXTS=256  \ # Required for LL
ROCSHMEM_HEAP_SIZE=7524589824 \
VLLM_ENGINE_READY_TIMEOUT_S=7800 \
vllm serve /models/DeepSeek-V3 \   
    --tensor-parallel-size 1 \
    --data-parallel-size 16 \
    --data-parallel-size-local 8 \
    --enforce-eager \
    --enable-expert-parallel \
    --block-size 1 \
    --data-parallel-address 10.158.214.101 \
    --data-parallel-rpc-port 13345 \
    --data-parallel-start-rank 8 \
    --headless \
    --all2all-backend deepep_low_latency | deepep_high_throughput

Serve 1P1D through RIXL

# PREFILL
 VLLM_NIXL_SIDE_CHANNEL_HOST=\${host_ip} \
    VLLM_NIXL_SIDE_CHANNEL_PORT=5557 \
    UCX_TLS=rc,sm,self,rocm_copy,rocm_ipc,tcp \
    UCX_NET_DEVICES=mlx5_0:1 \
    UCX_SOCKADDR_TLS_PRIORITY=rdmacm,tcp \
    UCX_SOCKADDR_CM_ENABLE=y \
    UCX_RDMA_CM_ENABLED=y \
    UCX_MEMTYPE_CACHE=y \
    UCX_RNDV_SCHEME=get_zcopy \
    UCX_RNDV_THRESH=4k \
    UCX_ROCM_IPC_MIN_ZCOPY=0 \
    HSA_ENABLE_SDMA=1 \
    UCX_LOG_LEVEL=info \
    NIXL_LOG_LEVEL=DEBUG \
    HSA_ENABLE_SDMA=1 \
    ROCSHMEM_HEAP_SIZE=15049179648 \
    vllm serve \${MODEL_PATH} \
    --port 30000 \
	--tensor-parallel-size 8 \
 	--quantization="fp8 \
 	--enforce-eager \
 	--no-enable-prefix-caching \
 	--block-size 1 \
        --trust-remote-code \
        --disable-log-requests \
        --kv-transfer-config '{\"kv_connector\": \"NixlConnector\", \"engine_id\": \"pd-run\", \"kv_role\": \"kv_producer\", \"kv_parallel_size\": 8, \"kv_rank\": 0, \"kv_buffer_size\": 5000000000, \"kv_buffer_device\": \"cuda\", \"kv_ip\": \"'\"\${host_ip}\"'\", \"kv_port\": 14600}'"

# DECODE
VLLM_NIXL_SIDE_CHANNEL_HOST=\${host_ip} \
    VLLM_NIXL_SIDE_CHANNEL_PORT=5557 \
    UCX_TLS=rc,sm,self,rocm_copy,rocm_ipc,tcp \
    UCX_NET_DEVICES=mlx5_0:1 \
    UCX_SOCKADDR_TLS_PRIORITY=rdmacm,tcp \
    UCX_SOCKADDR_CM_ENABLE=y \
    UCX_RDMA_CM_ENABLED=y \
    UCX_MEMTYPE_CACHE=y \
    UCX_RNDV_SCHEME=get_zcopy \
    UCX_RNDV_THRESH=4k \
    UCX_ROCM_IPC_MIN_ZCOPY=0 \
    HSA_ENABLE_SDMA=1 \
    UCX_LOG_LEVEL=info \
    NIXL_LOG_LEVEL=DEBUG \
    HSA_ENABLE_SDMA=1 \
    ROCSHMEM_HEAP_SIZE=15049179648 \
    vllm serve \${MODEL_PATH} \
    --port 30000 \
	--tensor-parallel-size 8 \
 	--quantization="fp8 \
 	--enforce-eager \
 	--no-enable-prefix-caching \
 	--block-size 1 \
        --trust-remote-code \
        --disable-log-requests \
        --kv-transfer-config '{\"kv_connector\": \"NixlConnector\", \"engine_id\": \"llama8b-run\", \"kv_role\": \"kv_consumer\", \"kv_parallel_size\": 8, \"kv_rank\": 0, \"kv_buffer_size\": 5000000000, \"kv_buffer_device\": \"cuda\", \"kv_ip\": \"'\"\${host_ip}\"'\", \"kv_port\": 14600}'"

GSM8K Evaluation

lm_eval --model local-completions \
    --tasks gsm8k \
    --model_args model=/models/DeepSeek-V3,base_url=http://localhost:2000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False

Test Result

Accuracy Results

GSM8K Evaluation Results
Serve on 1N DeepSeek-V3 with DeepEP - Low Latency

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9484|±  |0.0061|
|     |       |strict-match    |     5|exact_match|↑  |0.9462|±  |0.0062|

Serve on 1N DeepSeek-V3 with DeepEP - High Throughput

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9484|±  |0.0061|
|     |       |strict-match    |     5|exact_match|↑  |0.9469|±  |0.0062|

Serve on 2N DeepSeek-V3 with DeepEP - Low Latency

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9492|±  | 0.006|
|     |       |strict-match    |     5|exact_match|↑  |0.9492|±  | 0.006|

Serve on 2N DeepSeek-V3 with DeepEP - High Throughput

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9492|±  |0.0060|
|     |       |strict-match    |     5|exact_match|↑  |0.9454|±  |0.0063|

Serve 1P1D on RIXL

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9439|±  |0.0063|
|     |       |strict-match    |     5|exact_match|↑  |0.9447|±  |0.0063|

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

github-actions · 2026-02-17T11:53:40Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

gemini-code-assist

Code Review

This pull request enables DeepEP on ROCm for AMD GPUs by adding platform-specific logic, particularly for handling the float8_e4m3fnuz data type and adjusting API arguments. The changes are consistent across the modified files. My main feedback is to refactor a piece of duplicated code in all2all.py to improve maintainability.

lcskrishna · 2026-02-17T12:02:10Z

cc: @itej89 @amathews-amd

robertgshaw2-redhat · 2026-02-17T17:12:39Z

cc @tlrmchlsmth @varun-sundar-rabindranath

lcskrishna · 2026-02-19T12:12:46Z

Ran the following tests locally and all succeeded.

# pytest -v tests/kernels/moe/test_deepep_moe.py |& tee test_log.txt
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /app/vllm
configfile: pyproject.toml
plugins: anyio-4.12.1, asyncio-1.3.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collecting ... collected 56 items

tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-1-128-128-dtype0] PASSED [  1%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-1-128-128-dtype1] PASSED [  3%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-2-128-512-dtype0] PASSED [  5%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-2-128-512-dtype1] PASSED [  7%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-3-1024-2048-dtype0] PASSED [  8%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-3-1024-2048-dtype1] PASSED [ 10%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-32-128-1024-dtype0] PASSED [ 12%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-32-128-1024-dtype1] PASSED [ 14%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-45-512-2048-dtype0] PASSED [ 16%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-45-512-2048-dtype1] PASSED [ 17%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-64-1024-1024-dtype0] PASSED [ 19%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-64-1024-1024-dtype1] PASSED [ 21%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-222-1024-2048-dtype0] PASSED [ 23%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[False-world_dp_size0-6-32-222-1024-2048-dtype1] PASSED [ 25%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-1-128-128-dtype0] PASSED [ 26%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-1-128-128-dtype1] PASSED [ 28%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-2-128-512-dtype0] PASSED [ 30%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-2-128-512-dtype1] PASSED [ 32%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-3-1024-2048-dtype0] PASSED [ 33%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-3-1024-2048-dtype1] PASSED [ 35%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-32-128-1024-dtype0] PASSED [ 37%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-32-128-1024-dtype1] PASSED [ 39%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-45-512-2048-dtype0] PASSED [ 41%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-45-512-2048-dtype1] PASSED [ 42%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-64-1024-1024-dtype0] PASSED [ 44%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-64-1024-1024-dtype1] PASSED [ 46%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-222-1024-2048-dtype0] PASSED [ 48%]
tests/kernels/moe/test_deepep_moe.py::test_deep_ep_moe[True-world_dp_size0-6-32-222-1024-2048-dtype1] PASSED [ 50%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-1-128-2560-dtype0] PASSED [ 51%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-1-128-2560-dtype1] PASSED [ 53%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-2-128-2560-dtype0] PASSED [ 55%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-2-128-2560-dtype1] PASSED [ 57%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-3-1024-2560-dtype0] PASSED [ 58%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-3-1024-2560-dtype1] PASSED [ 60%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-32-128-2560-dtype0] PASSED [ 62%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-32-128-2560-dtype1] PASSED [ 64%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-45-512-2560-dtype0] PASSED [ 66%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-45-512-2560-dtype1] PASSED [ 67%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-64-1024-2560-dtype0] PASSED [ 69%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-64-1024-2560-dtype1] PASSED [ 71%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-222-1024-2560-dtype0] PASSED [ 73%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[True-world_dp_size0-6-32-222-1024-2560-dtype1] PASSED [ 75%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-1-128-2560-dtype0] PASSED [ 76%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-1-128-2560-dtype1] PASSED [ 78%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-2-128-2560-dtype0] PASSED [ 80%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-2-128-2560-dtype1] PASSED [ 82%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-3-1024-2560-dtype0] PASSED [ 83%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-3-1024-2560-dtype1] PASSED [ 85%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-32-128-2560-dtype0] PASSED [ 87%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-32-128-2560-dtype1] PASSED [ 89%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-45-512-2560-dtype0] PASSED [ 91%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-45-512-2560-dtype1] PASSED [ 92%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-64-1024-2560-dtype0] PASSED [ 94%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-64-1024-2560-dtype1] PASSED [ 96%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-222-1024-2560-dtype0] PASSED [ 98%]
tests/kernels/moe/test_deepep_moe.py::test_low_latency_deep_ep_moe[False-world_dp_size0-6-32-222-1024-2560-dtype1] PASSED [100%]

=============================== warnings summary ===============================
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

tests/kernels/moe/test_deepep_moe.py:84
  /app/vllm/tests/kernels/moe/test_deepep_moe.py:84: PytestCollectionWarning: cannot collect test class 'TestConfig' because it has a __init__ constructor (from: tests/kernels/moe/test_deepep_moe.py)
    @dataclasses.dataclass

tests/kernels/moe/test_deepep_moe.py:94
  /app/vllm/tests/kernels/moe/test_deepep_moe.py:94: PytestCollectionWarning: cannot collect test class 'TestTensors' because it has a __init__ constructor (from: tests/kernels/moe/test_deepep_moe.py)
    @dataclasses.dataclass

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================== 56 passed, 4 warnings in 464.09s (0:07:44) ==================
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

lcskrishna · 2026-02-26T16:18:28Z

cc: @HAIAI

mergify · 2026-02-28T00:20:35Z

Documentation preview: https://vllm--34692.org.readthedocs.build/en/34692/

mergify · 2026-02-28T00:20:37Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lcskrishna.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Tej Kiran <vpolamre@amd.com>

itej89 · 2026-02-28T23:16:52Z

Please find the DeepEP Unit Test Pass Results on the image built using the newly added docker file "docker/Dockerfile.rocm_deepep" below

rocm-deepep-unit-test.log

HAIAI · 2026-03-18T09:28:04Z

@HAIAI , I've made the requested changes. Also, for the latest pull from main, we might need changes in the run command w.r.t to --block-size parameter, Setting "--block-size 1" is giving issues. I am testing it now.

(Worker_DP1_EP1 pid=5348) ERROR 03-17 16:26:59 [multiproc_executor.py:932] raise ValueError(f"No common block size for {kv_manager_block_size}. ") (Worker_DP1_EP1 pid=5348) ERROR 03-17 16:26:59 [multiproc_executor.py:932] ValueError: No common block size for 1.

Above can be ignored - since:
#35752 had done some refactor on the block size argument on various backends.
https://github.com/vllm-project/vllm/pull/36185/changes#diff-2a5bf4cfb22305cacd4db8f0bf2c155afbbd2d45180970f0a4cb6de7eff7bd07
updated with the removal of blocksize 1 and moved to the common select blocksize from PR#35752 above from triton mla.

itej89 · 2026-03-18T18:54:43Z

Please find the updated test results in alignment with the latest updates below

Test Plan

Accuracy

MI300 + CX7

Serve on 1N DeepSeek-V3 with DeepEP - Low Latency | High Throughput

ROCSHMEM_MAX_NUM_CONTEXTS=256  \ # Required for LL
HSA_NO_SCRATCH_RECLAIM=1 \
ROCSHMEM_HEAP_SIZE=7524589824 \
VLLM_ENGINE_READY_TIMEOUT_S=7800 \
vllm serve /models/DeepSeek-V3 \
    --tensor-parallel-size 1 \
    --data-parallel-size 8 \
    --enforce-eager \
    --enable-expert-parallel \
    --gpu-memory-utilization 0.8 \
    --host 127.0.0.1 \
    --port 2000 \
    --all2all-backend   deepep_low_latency | deepep_high_throughput

Serve on 2N DeepSeek-V3 with DeepEP - Low Latency

# NODE 1 
ROCSHMEM_MAX_NUM_CONTEXTS=256  \ # Required for LL
HSA_NO_SCRATCH_RECLAIM=1 \
ROCSHMEM_HEAP_SIZE=7524589824 \
VLLM_ENGINE_READY_TIMEOUT_S=7800 \
vllm serve /models/DeepSeek-V3 \
    --tensor-parallel-size 1 \
    --data-parallel-size 16 \
    --data-parallel-size-local 8 \
    --enforce-eager \
    --enable-expert-parallel \
    --host  10.158.215.236 \
    --port 2000 \
    --data-parallel-address  10.158.215.236 \
    --data-parallel-rpc-port 13345 \
    --api-server-count=8 \
    --gpu-memory-utilization 0.8 \
    --all2all-backend   deepep_low_latency | deepep_high_throughput

# NODE 2 
HSA_NO_SCRATCH_RECLAIM=1 \
ROCSHMEM_MAX_NUM_CONTEXTS=256  \ # Required for LL
ROCSHMEM_HEAP_SIZE=7524589824 \
VLLM_ENGINE_READY_TIMEOUT_S=7800 \
vllm serve /models/DeepSeek-V3 \
    --tensor-parallel-size 1 \
    --data-parallel-size 16 \
    --data-parallel-size-local 8 \
    --enforce-eager \
    --enable-expert-parallel \
    --data-parallel-address  10.158.215.236 \
    --data-parallel-rpc-port 13345 \
    --data-parallel-start-rank 8 \
    --headless \
    --gpu-memory-utilization 0.8 \
    --all2all-backend   deepep_low_latency | deepep_high_throughput

GSM8K Evaluation

lm_eval --model local-completions \
    --tasks gsm8k \
    --model_args model=/models/DeepSeek-V3,base_url=http://localhost:2000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False

Test Results

Accuracy

GSM8K Evaluation Results
Serve on 1N DeepSeek-V3 with DeepEP - Low Latency

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9507|±  |0.0060|
|     |       |strict-match    |     5|exact_match|↑  |0.9484|±  |0.0061|

Serve on 1N DeepSeek-V3 with DeepEP - High Throughput

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9522|±  |0.0059|
|     |       |strict-match    |     5|exact_match|↑  |0.9500|±  |0.0060|

Serve on 2N DeepSeek-V3 with DeepEP - Low Latency

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9477|±  |0.0061|
|     |       |strict-match    |     5|exact_match|↑  |0.9469|±  |0.0062|

Serve on 2N DeepSeek-V3 with DeepEP - High Throughput

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9469|±  |0.0062|
|     |       |strict-match    |     5|exact_match|↑  |0.9439|±  |0.0063|

HAIAI · 2026-03-20T04:27:20Z

@houseroad @njhill This PR is ready to merge, would you mind to have a look?

lcskrishna · 2026-03-20T04:28:33Z

The CI failures seem unrelated to the PRs and the AMD-CI is successful. This PR is ready for merge with whoever has merge access.

njhill · 2026-03-20T22:39:46Z

Thanks @lcskrishna @HAIAI! Could you merge in main one more time?

itej89 · 2026-03-20T22:54:15Z

Thanks @lcskrishna @HAIAI! Could you merge in main one more time?

@njhill As requested, I've merged the main. Thanks!!

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com>

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com> Signed-off-by: Nithin Chalapathi <nithin.ch10@gmail.com>

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com>

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com> Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com>

mergify Bot added the rocm Related to AMD ROCm label Feb 17, 2026

github-project-automation Bot added this to AMD Feb 17, 2026

github-project-automation Bot moved this to Todo in AMD Feb 17, 2026

gemini-code-assist Bot reviewed Feb 17, 2026

View reviewed changes

Comment thread vllm/distributed/device_communicators/all2all.py Outdated

raviguptaamd mentioned this pull request Feb 17, 2026

[ROCm] Enable DBO (Dynamic Batch Optimization) on ROCm #34726

Merged

mergify Bot added documentation Improvements or additions to documentation ci/build labels Feb 28, 2026

mergify Bot added the needs-rebase label Feb 28, 2026

lcskrishna and others added 6 commits February 28, 2026 21:58

ROCm DeepEP integration changes

811860e

Signed-off-by: Tej Kiran <vpolamre@amd.com>

update condition logic in more cleaner way

79b4bbd

Signed-off-by: Tej Kiran <vpolamre@amd.com>

add low latency changes

e929e24

Signed-off-by: Tej Kiran <vpolamre@amd.com>

Included verified Dockerfile for DeepEP and rixl

b1009bc

Signed-off-by: Tej Kiran <vpolamre@amd.com>

Updated the instructions for gpu-arch and nic selection

459aa26

Signed-off-by: Tej Kiran <vpolamre@amd.com>

Updated the link path in nixl connector usage document

7b4f617

Signed-off-by: Tej Kiran <vpolamre@amd.com>

itej89 force-pushed the rocm-deepep branch from 22a11f4 to 7b4f617 Compare February 28, 2026 22:18

lcskrishna marked this pull request as ready for review March 4, 2026 17:00

lcskrishna requested review from gshtras and tjtanaa as code owners March 4, 2026 17:00

HAIAI approved these changes Mar 18, 2026

View reviewed changes

itej89 and others added 2 commits March 18, 2026 20:08

Merge branch 'main' into rocm-deepep

9fa1fdb

Merge branch 'vllm-project:main' into rocm-deepep

d548099

WoosukKwon added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 20, 2026

Merge branch 'main' into rocm-deepep

21d03fc

HAIAI merged commit 3982bc2 into vllm-project:main Mar 21, 2026
134 checks passed

github-project-automation Bot moved this from Todo to Done in AMD Mar 21, 2026

SouthWest7 pushed a commit to SouthWest7/vllm that referenced this pull request Mar 27, 2026

[ROCm] Enable DeepEP ROCm as all2allbackend for AMD GPUs. (vllm-proje…

61ff0bf

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com>

khairulkabir1661 pushed a commit to khairulkabir1661/vllm that referenced this pull request Mar 27, 2026

[ROCm] Enable DeepEP ROCm as all2allbackend for AMD GPUs. (vllm-proje…

88fb4be

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com>

JiantaoXu pushed a commit to JiantaoXu/vllm that referenced this pull request Mar 28, 2026

[ROCm] Enable DeepEP ROCm as all2allbackend for AMD GPUs. (vllm-proje…

19e2649

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com>

mtparet pushed a commit to blackfuel-ai/vllm that referenced this pull request Apr 9, 2026

[ROCm] Enable DeepEP ROCm as all2allbackend for AMD GPUs. (vllm-proje…

5968019

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com>

mystous pushed a commit to mystous/vllm_hybrid that referenced this pull request May 10, 2026

[ROCm] Enable DeepEP ROCm as all2allbackend for AMD GPUs. (vllm-proje…

cda22fe

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com>

plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026

[ROCm] Enable DeepEP ROCm as all2allbackend for AMD GPUs. (vllm-proje…

fdb518b

…ct#34692) Signed-off-by: Tej Kiran <vpolamre@amd.com> Co-authored-by: Tej Kiran <vpolamre@amd.com>

Rohan138 mentioned this pull request Aug 5, 2026

[ROCm] Work around DeepEP teardown SIGSEGV in MoE test harness #51174

Merged

Uh oh!

Uh oh!

Conversation

lcskrishna commented Feb 17, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Accuracy Test

MI300 + CX7

Test Result

Accuracy Results

Uh oh!

github-actions Bot commented Feb 17, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

lcskrishna commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertgshaw2-redhat commented Feb 17, 2026

Uh oh!

lcskrishna commented Feb 19, 2026

Uh oh!

lcskrishna commented Feb 26, 2026

Uh oh!

mergify Bot commented Feb 28, 2026

Uh oh!

mergify Bot commented Feb 28, 2026

Uh oh!

itej89 commented Feb 28, 2026

Uh oh!

HAIAI commented Mar 18, 2026

Uh oh!

itej89 commented Mar 18, 2026

Test Plan

Accuracy

MI300 + CX7

Test Results

Accuracy

Uh oh!

HAIAI commented Mar 20, 2026

Uh oh!

lcskrishna commented Mar 20, 2026

Uh oh!

njhill commented Mar 20, 2026

Uh oh!

itej89 commented Mar 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

lcskrishna commented Feb 17, 2026 •

edited by github-actions Bot

Loading

lcskrishna commented Feb 17, 2026 •

edited

Loading