Skip to content

[Doc] Switch K8S examples to default MP mode - #39400

Merged
hmellor merged 17 commits into
vllm-project:mainfrom
panpan0000:ray2mp
Jun 10, 2026
Merged

[Doc] Switch K8S examples to default MP mode#39400
hmellor merged 17 commits into
vllm-project:mainfrom
panpan0000:ray2mp

Conversation

@panpan0000

@panpan0000 panpan0000 commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Fix #38113

per @ed-pai, previous k8s yaml won't work since default docker images has removed ray.
so switch the k8s yaml example to default mp(multiprocessing) as distributed backend.

Background:

This PR preserves the old ray code there as comments, in case some people still need it for reference or pip install ray by themselves.

Also add explicit --distributed-executor-backend ray for examples/online_serving/multi-node-serving.sh

Test Plan

apply the example LWS.md

Test Result

the actual vllm command as below , which aligns https://docs.vllm.ai/en/latest/serving/parallelism_scaling/#running-vllm-with-multiprocessing

cloudshell-worker-npvr4qhxq5:~# kubectl exec -it vllm-0 -- ps -lefww
....  /usr/bin/python3 /usr/local/bin/vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size 2 --nnodes 2 --node-rank 0 --master-addr vllm-0.vllm.default --port 8080

cloudshell-worker-npvr4qhxq5:~# kubectl exec -it vllm-0-1 -- ps -lefww
....  /usr/bin/python3 /usr/local/bin/vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size 2 --nnodes 2 --node-rank 1 --master-addr vllm-0.vllm.default --headless


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

@mergify

mergify Bot commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--39400.org.readthedocs.build/en/39400/

@mergify mergify Bot added the documentation Improvements or additions to documentation label Apr 9, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates documentation and example configurations for distributed vLLM serving. It transitions from using a Ray-based multi-node serving script to direct vllm serve commands, explicitly configuring nodes, ranks, and master addresses, often leveraging the mp distributed executor backend. The changes include updating lws.md and kthena.md deployment examples, commenting out old Ray commands, and a minor typo fix. The review comments suggest improving consistency in argument formatting within the kthena.md documentation by using spaces instead of equals signs for parameters like --nnodes and --node-rank to enhance readability and reduce confusion.

Comment thread docs/deployment/integrations/kthena.md Outdated
Comment thread docs/deployment/integrations/kthena.md Outdated
@chaunceyjiang chaunceyjiang added ready ONLY add when PR is ready to merge/full CI is needed labels Apr 9, 2026
@panpan0000

Copy link
Copy Markdown
Contributor Author

@hmellor @yankay can you please take a look ?

Comment thread docs/deployment/frameworks/lws.md
Comment thread docs/deployment/frameworks/lws.md Outdated
@panpan0000

Copy link
Copy Markdown
Contributor Author

comments addressed , thanks @hmellor

Comment thread docs/deployment/integrations/kthena.md Outdated
Comment thread docs/deployment/integrations/kthena.md Outdated
Comment thread docs/deployment/integrations/kthena.md Outdated
Comment thread docs/deployment/integrations/kthena.md Outdated
Comment thread docs/deployment/integrations/kthena.md

@hmellor hmellor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks this is looking better. The terminology used is inconsistent though. Please name the tabs Multiprocessing (default), Ray

@panpan0000

Copy link
Copy Markdown
Contributor Author

Thanks this is looking better. The terminology used is inconsistent though. Please name the tabs Multiprocessing (default), Ray

all set . thanks @hmellor

@mergify

mergify Bot commented May 23, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @panpan0000.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label May 23, 2026
panpan0000 and others added 7 commits May 24, 2026 14:31
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
panpan0000 and others added 4 commits May 24, 2026 14:31
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
@panpan0000

Copy link
Copy Markdown
Contributor Author

This pull request has merge conflicts that must be resolved before it can be merged. Please rebase the PR, @panpan0000.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

already addressed @mergify

@panpan0000

Copy link
Copy Markdown
Contributor Author

all PR are rusted... Lol

@panpan0000

Copy link
Copy Markdown
Contributor Author

Hi, @hmellor , PTAL again, thanks

@hmellor hmellor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a few nits for consistency

Comment thread docs/deployment/frameworks/lws.md Outdated
Comment thread docs/deployment/frameworks/lws.md Outdated
Comment thread docs/deployment/frameworks/lws.md Outdated
Comment thread docs/deployment/frameworks/lws.md Outdated
Comment thread docs/deployment/frameworks/lws.md Outdated
Comment thread docs/deployment/integrations/kthena.md Outdated
hmellor added 2 commits June 10, 2026 17:58
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
hmellor enabled auto-merge (squash) June 10, 2026 16:59
Comment thread docs/deployment/integrations/kthena.md Outdated
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Comment thread docs/deployment/integrations/kthena.md Outdated
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
hmellor merged commit 3d300ae into vllm-project:main Jun 10, 2026
15 checks passed
@panpan0000

Copy link
Copy Markdown
Contributor Author

THank you @hmellor

wcynb1023 pushed a commit to wcynb1023/vllm that referenced this pull request Jun 11, 2026
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
Saddss pushed a commit to Saddss/vllm that referenced this pull request Jun 14, 2026
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
vivek8123 pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Jun 18, 2026
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
divineearthly pushed a commit to divineearthly/vllm that referenced this pull request Jun 19, 2026
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
Signed-off-by: divineearthly <divineearthly@gmail.com>
nkzhenhua pushed a commit to nkzhenhua/vllm that referenced this pull request Jun 24, 2026
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
Dao007forever pushed a commit to Dao007forever/vllm that referenced this pull request Jul 18, 2026
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
philippesic pushed a commit to philippesic/vllm-semantic-cache that referenced this pull request Jul 19, 2026
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
plasticchris pushed a commit to plasticchris/vllm that referenced this pull request Jul 20, 2026
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Signed-off-by: Peter Pan <peter.pan@daocloud.io>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Installation]: Ray not present in Container Image

4 participants