Describe the feature
Please add first-class support for multi-node multiprocessing deployment in production-stack.
Right now, the distributed Kubernetes path is centered around the RayCluster template:
https://github.com/vllm-project/production-stack/blob/main/helm/templates/ray-cluster.yaml
That template currently hardcodes Ray-based execution, while recent vLLM changes indicate that multi-node deployment should also support the multiprocessing path:
vllm-project/vllm#39400
It would be useful if production-stack exposed a supported way to configure multi-node MP deployment parameters such as nnodes, node-rank, master-addr, master-port, and headless.
Why do you need this feature?
I am trying to deploy a single distributed model across 4 Kubernetes nodes with production-stack.
At the moment, the chart supports the Ray-based path, but does not provide a first-class way to deploy multi-node vLLM with multiprocessing. That makes it difficult to follow the newer vLLM deployment direction without maintaining custom manifests outside of production-stack.
Additional context
Concrete improvements that would help:
- avoid hardcoding Ray-only execution in the current distributed template
- expose multi-node multiprocessing settings through chart values
- provide an official Kubernetes example for multi-node MP deployment
Describe the feature
Please add first-class support for multi-node
multiprocessingdeployment inproduction-stack.Right now, the distributed Kubernetes path is centered around the RayCluster template:
https://github.com/vllm-project/production-stack/blob/main/helm/templates/ray-cluster.yaml
That template currently hardcodes Ray-based execution, while recent vLLM changes indicate that multi-node deployment should also support the multiprocessing path:
vllm-project/vllm#39400
It would be useful if
production-stackexposed a supported way to configure multi-node MP deployment parameters such asnnodes,node-rank,master-addr,master-port, andheadless.Why do you need this feature?
I am trying to deploy a single distributed model across 4 Kubernetes nodes with
production-stack.At the moment, the chart supports the Ray-based path, but does not provide a first-class way to deploy multi-node vLLM with multiprocessing. That makes it difficult to follow the newer vLLM deployment direction without maintaining custom manifests outside of
production-stack.Additional context
Concrete improvements that would help: