[serve][llm] Data Parallel Attention: Public API and Documentation #58301
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
kouroshHakha
changed the title
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
ray-gardener
bot
added
serve
richardliaw
approved these changes
Nov 8, 2025
eicherseiji
reviewed
Nov 8, 2025
| - gpu | ||
| instance_type: gpu-large | ||
| commands: | ||
| - RAYCI_DISABLE_TEST_DB=1 bazel run //ci/ray_ci:test_in_docker -- //doc/... llm |
Contributor
There was a problem hiding this comment.
Should we add
//python/ray/llm/...
so potential multi_gpu_4 targets added there will be picked up too?
| Deploy with: | ||
| ```bash | ||
| serve deploy dp_config.yaml |
Contributor
There was a problem hiding this comment.
Suggested change
| serve deploy dp_config.yaml | |
| serve run dp_config.yaml |
landscapepainter
pushed a commit
to landscapepainter/ray
that referenced
this pull request
Nov 17, 2025
Aydin-ab
pushed a commit
to Aydin-ab/ray-aydin
that referenced
this pull request
Nov 19, 2025
…ay-project#58301) Signed-off-by: Aydin Abiar <[email protected]>
ykdojo
pushed a commit
to ykdojo/ray
that referenced
this pull request
Nov 27, 2025
SheldonTsen
pushed a commit
to SheldonTsen/ray
that referenced
this pull request
Dec 1, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds data parallel attention support to the public API, creates comprehensive documentation, and sets up CI testing infrastructure for multi-GPU examples.
Changes
Public API
DPRankAssignerprivate (_DPRankAssigner)build_dp_deployment()andbuild_dp_openai_app()toray.serve.llmpublic APIDPServeras public API inray.serve.llm.deploymentDocumentation
data-parallel-attention.mdcovering DP patterns, configuration, and usagedp_basic_example.py- Basic DP deployment with OpenAI ingressdp_pd_example.py- DP + Prefill-decode disaggregationCI Infrastructure
multi_gpu/folder for examples requiring 4+ GPUsg6-large, multi-GPU tests ongpu-large(4 GPUs)multi_gpu_4tagTesting
literalincludedirectives