Skip to content

feat(vllm): support configurable direct serve binary - #409

Merged
ishandhanani merged 1 commit into
mainfrom
feat/direct-vllm-rs-serve
Sep 9, 2026
Merged

ishandhanani merged 1 commit into
mainfrom
feat/direct-vllm-rs-serve

Conversation

@jiahanc

@jiahanc jiahanc commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add backend.vllm_serve_binary for direct aggregate frontend.type: vllm jobs.
  • Keep vllm as the default, preserving all existing recipes.
  • Allow vllm-rs or an absolute executable path when the Rust binary is not on PATH.
  • Leave Dynamo, sidecar, and vllm-router launch paths unchanged.
  • Document a managed-engine Rust frontend recipe and cover command construction plus schema round-tripping.

Motivation

The direct vLLM backend currently hardcodes vllm serve. Some vLLM images ship the managed-engine Rust OpenAI frontend as vllm-rs, occasionally only at a package-local absolute path. Recipes need a first-class way to select that executable while retaining srtctl's existing model, topology, port, device, profiling, and vLLM CLI argument construction.

Duplicate-work check

No open PR matched searches for vllm-rs, Rust frontend vllm, or configurable serve binary in NVIDIA/srt-slurm.

Tests

  • .venv/bin/python -m pytest tests/ -v (1753 passed, 2 skipped, 6 deselected)
  • .venv/bin/ruff check src/srtctl/
  • .venv/bin/ruff format --check src/srtctl/ tests/test_configs.py
  • .venv/bin/ty check src/srtctl/backends/vllm.py
  • git diff --check

AI assistance was used to prepare this change. The submitter is responsible for reviewing and understanding every changed line before merge.

Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants