Multi-node multi-instance vLLM via gym eval submit

[oyilmaz-nvidia]

Summary

Adds multi-node vLLM service support to the Slurm orchestration layer (gym eval submit), building on top of the existing single-node multi-instance support (separate PR: onur/multi-instance-vllm-service). A vllm service can now span multiple physical Slurm nodes, either for a single replica's tensor/pipeline-parallel footprint, or for multi-node data-parallel replicas.

What's NOT included in this PR is that multi-node TP with multiple instance which is a rare case. So, in order to use the features in this PR, the model has to fit into a single node.

What's new

• ray distributed backend — a new distributed_backend option (alongside the existing single-node mp backend) using vLLM's Ray core executor (--distributed-executor-backend ray, not the ray.serve library) to span a service across nodes.
• Automatic backend selection by node count — if compute.node_pools total more than one node, distributed_backend is automatically forced to ray, overriding anything set/defaulted at the service level. No need to write distributed_backend: {type: ray} yourself.
• Multi-node data-parallel validation — number_of_instances must divide evenly across the node count (each node hosts an equal share of replicas).
• Sbatch script generation (slurm_script.py):
  • injects a Ray head-node-IP prelude (scontrol show hostnames, HEAD_NODE_IP/RAY_HEAD_NODE_IP) when any service uses the ray backend,
  • single-instance multi-node: wraps the vLLM command in ray symmetric-run (falling back to manual ray start --head/--block for older Ray), spanning TP/PP across nodes,
  • multi-instance multi-node: uses vLLM's native multi-node data-parallel pattern — head node serves the OpenAI API, worker nodes run --headless with a --data-parallel-start-rank offset — no Ray involved for that path,
  • adds --nodes=/--ntasks= to each service's srun step on multi-node compute (--nodes=1 --ntasks=1 for the driver).

New example config

• examples/slurm_vllm_ray_multi_node.yaml — 2-node compute, 8 replicas (TP2) spanning both nodes, backend auto-selected as ray from node count alone.

[prokotg]

Thank you for the PR! I left comments in placees where I think some changes are critical. Let me know if I can provide more feedback

[prokotg]

this function can be removed

[prokotg]

Might be tackled in another PR but we need to check how multi-rank logs are written (we should be able to differentiate logs from instances)

[prokotg]

Approved, thank you! Agreed upon items to remember for the upcoming PRs:

• Multi-node multi-instnace model handling
• Partial allocation for service and collision avoidance (multiple services should have separate GPUs)
• Logging of multi-rank services should be handled

[oyilmaz-nvidia]

/ok to test 968b57c

[oyilmaz-nvidia]

/ok to test 3b5da65

[oyilmaz-nvidia]

@prokotg To have a record here, so the following configs are supported for multi node. Assuming each node has 8 GPUs.

1. nodes: 2, gpus_per_node: 8 and tensor_parallel_size: 8, pipeline_parallel_size: 2, number_of_instances: 1
2. nodes: 2, gpus_per_node: 8 and tensor_parallel_size: 2, pipeline_parallel_size: 1, number_of_instances: 8

but the following is not supported.

1. nodes: 4, gpus_per_node: 8 and tensor_parallel_size: 8, pipeline_parallel_size: 2, number_of_instances: 2

And this is due to a bug on the vLLM side.

[prokotg]

Thanks @oyilmaz-nvidia could you share more details on the vllm bug?