[WIP][Feature] Optional vLLM router for internal rollout gateway

[CalvinXKY]

Purpose

Add support for spawning vllm-router as the HTTP rollout gateway in addition to the existing sglang-router (sgl-model-gateway) path.

• New flag --router-impl {sglang,vllm} (args.router_impl): choose which gateway binary Slime starts when no external router is configured (--sglang-router-ip unset).
• New module slime.utils.router: register shared --router-* / --worker-urls CLI, build RouterArgs, assert the selected package is installed, map worker control-plane to the modern /workers API when router_impl=vllm, and sanitize RouterArgs for vLLM when both wheels are installed (e.g. sglang’s max_concurrent_requests=-1 is invalid for vLLM’s unsigned Rust fields).
• slime/utils/http_utils.py: run_router dispatches to vllm_router.launch_router or sglang_router.launch_router based on impl.
• requirements.txt: add vllm-router>=0.1.14 alongside sglang-router.
  Default behavior remains sglang; external routers are unchanged.

Test Plan

• Install deps: pip install -r requirements.txt
• Train / rollout with --router-impl vllm (internal router) and confirm log line: Launch HTTP router (impl=vllm) ...
• Smoke with --router-impl sglang (or omit flag) to confirm no regression

running:

export PYTHONPATH=/root/Megatron-LM
SCRIPT_DIR="/data/nfs/kaiyuan/RL/vime/scripts"
source "${SCRIPT_DIR}/models/qwen3-0.6B.sh"

python train.py \
  ${MODEL_ARGS[@]} \
  \
  --hf-checkpoint /data/nfs/kaiyuan/models/Qwen3-0.6B \
  --ref-load /data/nfs/kaiyuan/models/Qwen3-0.6B_torch_dist \
  --load /work/Qwen3-0.6B_slime_ckpt \
  --save /work/Qwen3-0.6B_slime_ckpt \
  --save-interval 10 \
  --train-memory-margin-bytes 2147483648 \
  \
  --prompt-data /data/nfs/kaiyuan/datasets/dapo-math-17k/dapo-math-17k.jsonl \
  --input-key prompt \
  --label-key label \
  --apply-chat-template \
  --rollout-shuffle \
  \
  --rm-type deepscaler \
  --num-rollout 100 \
  --rollout-batch-size 4 \
  --n-samples-per-prompt 2 \
  --num-steps-per-rollout 1 \
  --global-batch-size 8 \
  \
  --actor-num-nodes 1 \
  --actor-num-gpus-per-node 8 \
  --rollout-num-gpus 8 \
  --rollout-num-gpus-per-engine 2 \
  --rollout-max-response-len 1024 \
  --sglang-mem-fraction-static 0.8 \
  \
  --attention-dropout 0.0 \
  --hidden-dropout 0.0 \
  --accumulate-allreduce-grads-in-fp32 \
  --attention-softmax-in-fp32 \
  --attention-backend flash \
  --optimizer adam \
  --lr 1e-6 \
  --lr-decay-style constant \
  --weight-decay 0.1 \
  --adam-beta1 0.9 \
  --adam-beta2 0.98 \
  --router-impl vllm \
  \
  --colocate

Test Result