Skip to content

feature: add first-class multi-node multiprocessing deployment support #930

Description

@observerw

Describe the feature

Please add first-class support for multi-node multiprocessing deployment in production-stack.

Right now, the distributed Kubernetes path is centered around the RayCluster template:
https://github.com/vllm-project/production-stack/blob/main/helm/templates/ray-cluster.yaml

That template currently hardcodes Ray-based execution, while recent vLLM changes indicate that multi-node deployment should also support the multiprocessing path:
vllm-project/vllm#39400

It would be useful if production-stack exposed a supported way to configure multi-node MP deployment parameters such as nnodes, node-rank, master-addr, master-port, and headless.

Why do you need this feature?

I am trying to deploy a single distributed model across 4 Kubernetes nodes with production-stack.

At the moment, the chart supports the Ray-based path, but does not provide a first-class way to deploy multi-node vLLM with multiprocessing. That makes it difficult to follow the newer vLLM deployment direction without maintaining custom manifests outside of production-stack.

Additional context

Concrete improvements that would help:

  • avoid hardcoding Ray-only execution in the current distributed template
  • expose multi-node multiprocessing settings through chart values
  • provide an official Kubernetes example for multi-node MP deployment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions