diff --git a/README.md b/README.md index 1b6ee2665..153cea52d 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ We also provide examples for some use cases not covered in the quick start guide Arguments in Vime are divided into three categories: 1. **Megatron arguments**: Vime reads all arguments in Megatron. You can configure Megatron by passing arguments like `--tensor-model-parallel-size 2`. -2. **vLLM arguments**: vLLM server and engine options are exposed with a `--vllm-` prefix (for example, `--vllm-gpu-memory-utilization`). Router-related options use the `--router-` prefix. See [slime/backends/vllm_utils/arguments.py](slime/backends/vllm_utils/arguments.py) for the full surface. +2. **vLLM arguments**: vLLM server and engine options are exposed with a `--vllm-` prefix (for example, `--vllm-gpu-memory-utilization`). Router options live under two prefixes: vllm-router's native options are passed with `--router-` (for example, `--router-policy round_robin`), while Vime-side orchestration knobs that tell Vime *where* the router lives use `--vllm-router-` (`--vllm-router-ip`, `--vllm-router-port`, `--vllm-router-request-timeout-secs`). See [slime/backends/vllm_utils/arguments.py](slime/backends/vllm_utils/arguments.py) for the full surface. 3. **Framework-specific arguments**: Shared slime/Vime orchestration flags (rollout GPUs, data paths, RL algorithms, etc.). Please refer to [slime/utils/arguments.py](slime/utils/arguments.py). `--rollout-num-gpus-per-engine` sets the tensor parallel size of each vLLM engine. The default rollout entry is `slime.rollout.vllm_rollout.generate_rollout`. @@ -78,7 +78,6 @@ Vime is derived from slime. The following upstream resources and in-repo guides - Upstream repository: [THUDM/slime](https://github.com/THUDM/slime) - English docs in this repo: [docs/en/](docs/en/) - Chinese docs in this repo: [docs/zh/](docs/zh/) -- vLLM backend design notes (Chinese): [docs/zh/advanced/slime_vllm_backend_design_v1.md](docs/zh/advanced/slime_vllm_backend_design_v1.md) ## FAQ & Acknowledgements diff --git a/README_zh.md b/README_zh.md index cad5dbb50..54b121da9 100644 --- a/README_zh.md +++ b/README_zh.md @@ -45,7 +45,7 @@ Vime 继承 slime 的广泛模型支持，包括： Vime 的参数分为三类： 1. **Megatron 参数**：Vime 会读取 Megatron 中的全部参数，可通过传入如 `--tensor-model-parallel-size 2` 的方式配置 Megatron； -2. **vLLM 参数**：vLLM server 与 engine 相关选项以 `--vllm-` 为前缀（例如 `--vllm-gpu-memory-utilization`）；路由相关选项使用 `--router-` 前缀。完整参数见 [slime/backends/vllm_utils/arguments.py](slime/backends/vllm_utils/arguments.py)。 +2. **vLLM 参数**：vLLM server 与 engine 相关选项以 `--vllm-` 为前缀（例如 `--vllm-gpu-memory-utilization`）。路由相关选项分两类前缀：vllm-router 自身的选项以 `--router-` 传入（例如 `--router-policy round_robin`），Vime 侧用于告诉 Vime *router 在哪里* 的编排参数则以 `--vllm-router-` 为前缀（`--vllm-router-ip`、`--vllm-router-port`、`--vllm-router-request-timeout-secs`）。完整参数见 [slime/backends/vllm_utils/arguments.py](slime/backends/vllm_utils/arguments.py)。 3. **框架参数**：与 slime/Vime 编排相关的开关（rollout GPU、数据路径、RL 算法等），见 [slime/utils/arguments.py](slime/utils/arguments.py)。 `--rollout-num-gpus-per-engine` 对应每个 vLLM engine 的 tensor parallel size。默认 rollout 入口为 `slime.rollout.vllm_rollout.generate_rollout`。 @@ -78,7 +78,6 @@ Vime 由 slime 衍生而来。以下上游资源与本仓库文档仍沿用 slim - 上游仓库：[THUDM/slime](https://github.com/THUDM/slime) - 本仓库英文文档：[docs/en/](docs/en/) - 本仓库中文文档：[docs/zh/](docs/zh/) -- vLLM 后端设计说明：[docs/zh/advanced/slime_vllm_backend_design_v1.md](docs/zh/advanced/slime_vllm_backend_design_v1.md) ## 常见 Q&A 与致谢 diff --git a/docs/_static/image/arch.png b/docs/_static/image/arch.png deleted file mode 100644 index b809bd3ff..000000000 Binary files a/docs/_static/image/arch.png and /dev/null differ diff --git a/docs/_static/image/blogs/release_v0.1.0/cuda_vmm.png b/docs/_static/image/blogs/release_v0.1.0/cuda_vmm.png deleted file mode 100644 index 5bc852b9e..000000000 Binary files a/docs/_static/image/blogs/release_v0.1.0/cuda_vmm.png and /dev/null differ diff --git a/docs/_static/image/blogs/release_v0.1.0/overrall.png b/docs/_static/image/blogs/release_v0.1.0/overrall.png deleted file mode 100644 index ff0293ceb..000000000 Binary files a/docs/_static/image/blogs/release_v0.1.0/overrall.png and /dev/null differ diff --git a/docs/_static/image/sglang_config.png b/docs/_static/image/sglang_config.png deleted file mode 100644 index c4cde43ab..000000000 Binary files a/docs/_static/image/sglang_config.png and /dev/null differ diff --git a/docs/_static/image/vllm_config.png b/docs/_static/image/vllm_config.png new file mode 100644 index 000000000..9c48219b9 Binary files /dev/null and b/docs/_static/image/vllm_config.png differ diff --git a/docs/en/advanced/fault-tolerance.md b/docs/en/advanced/fault-tolerance.md index 5fe09d78a..da10c372b 100644 --- a/docs/en/advanced/fault-tolerance.md +++ b/docs/en/advanced/fault-tolerance.md @@ -6,7 +6,7 @@ To enable the fault tolerance function in slime, please set `--use-fault-toleran ## Rollout Fault Tolerance -During the rollout process, slime periodically sends heartbeat requests (`/health_generate`) to all SGLang servers. If a heartbeat times out, that SGLang server will be stopped. After the current rollout round is complete, the server will be restarted and its parameters will be correctly updated. +During the rollout process, slime periodically sends heartbeat requests (`/health`) to all vLLM engines. If a heartbeat times out, that vLLM engine will be stopped. After the current rollout round is complete, the engine will be restarted and its parameters will be correctly updated. - `--rollout-health-check-first-wait`: Since some large MoE models require compilation on their first run, slime will wait for `rollout_health_check_first_wait` seconds before the first rollout to start sending heartbeats. Defaults to 300s. - `--rollout-health-check-interval`: The interval between heartbeat checks. Defaults to 10s. diff --git a/docs/en/advanced/low-precision.md b/docs/en/advanced/low-precision.md index b5415390b..951d5c163 100644 --- a/docs/en/advanced/low-precision.md +++ b/docs/en/advanced/low-precision.md @@ -63,7 +63,7 @@ Here's a quick explanation of how FP8 training is currently implemented in slime 2. Training: During training, weights and activations are quantized online to nvfp8 format, and cuBLAS FP8 GEMM is called for various GEMM computations in forward and backward passes. -3. Weight updates: During RL weight updates, Megatron first dequantizes FP8 weights to bf16 format, then slime quantizes these bf16 weights to fp8 format and sends them to sglang. (This additional dequantization and quantization is not elegant, but we haven't modified the interface yet for framework compatibility.) +3. Weight updates: During RL weight updates, Megatron first dequantizes FP8 weights to bf16 format, then slime quantizes these bf16 weights to fp8 format and sends them to vLLM. (This additional dequantization and quantization is not elegant, but we haven't modified the interface yet for framework compatibility.) 4. Save checkpoint: Similar to weight updates, if checkpoints need to be saved from the training engine, they will also be dequantized back to bf16 and saved to `torch_dist` format checkpoints. diff --git a/docs/en/advanced/megatron-config.md b/docs/en/advanced/megatron-config.md index 973363ca8..fb9f1b68d 100644 --- a/docs/en/advanced/megatron-config.md +++ b/docs/en/advanced/megatron-config.md @@ -2,7 +2,7 @@ `--megatron-config-path` is a YAML-based configuration system for applying role-specific overrides on top of the shared Megatron CLI arguments. Today it is mainly intended for PPO actor / critic configuration. -Unlike `--sglang-config`, `--megatron-config-path` does not manage deployment, routing, or GPU orchestration. Its only job is to decide which training arguments each role should finally use. +Unlike `--vllm-config`, `--megatron-config-path` does not manage deployment, routing, or GPU orchestration. Its only job is to decide which training arguments each role should finally use. --- diff --git a/docs/en/advanced/on-policy-distillation.md b/docs/en/advanced/on-policy-distillation.md index 1930ff8e4..b4efae4c0 100644 --- a/docs/en/advanced/on-policy-distillation.md +++ b/docs/en/advanced/on-policy-distillation.md @@ -7,9 +7,9 @@ On-policy distillation (OPD) enables a student model to learn from a larger teac | Argument | Description | |----------|-------------| | `--use-opd` | Enable on-policy distillation. Required flag to use OPD. | -| `--opd-type` | Type of OPD: `sglang` or `megatron`. Required when `--use-opd` is set. | +| `--opd-type` | Type of OPD: `vllm` or `megatron`. Required when `--use-opd` is set. | | `--opd-kl-coef` | OPD KL penalty coefficient (default: 1.0). Controls the weight of the distillation signal relative to the RL advantage. | -| `--opd-teacher-load` | Path to teacher Megatron checkpoint. **Required** when `--opd-type=megatron`, **must not be set** when `--opd-type=sglang`. | +| `--opd-teacher-load` | Path to teacher Megatron checkpoint. **Required** when `--opd-type=megatron`, **must not be set** when `--opd-type=vllm`. | | `--opd-teacher-ckpt-step` | Optional checkpoint step for teacher model. | ## How It Works @@ -26,14 +26,14 @@ This means OPD can be combined with any advantage estimator, including GRPO, PPO ## Two Teacher Modes -### SGLang Mode (`--opd-type sglang`) +### vLLM Mode (`--opd-type vllm`) -The teacher runs on an external SGLang server. Teacher log-probs are obtained during the rollout phase. +The teacher runs on an external vLLM server. Teacher log-probs are obtained during the rollout phase. **When to use**: The teacher has a different architecture from the student, or the teacher is too large to load alongside the training model. **How it works**: -1. An external SGLang server runs the teacher model. +1. An external vLLM server runs the teacher model. 2. During rollout, the custom reward function (`slime.rollout.on_policy_distillation.reward_func`) sends each sample to the teacher server to obtain token-level log-probs. 3. The custom post-processing function (`slime.rollout.on_policy_distillation.post_process_rewards`) trims the teacher log-probs to the response span and stores them in `sample.teacher_log_probs`. 4. During training, the KL penalty is computed from the stored teacher log-probs and applied to advantages. @@ -41,11 +41,11 @@ The teacher runs on an external SGLang server. Teacher log-probs are obtained du **Configuration**: ```bash --use-opd ---opd-type sglang +--opd-type vllm --opd-kl-coef 1.0 --custom-rm-path slime.rollout.on_policy_distillation.reward_func --custom-reward-post-process-path slime.rollout.on_policy_distillation.post_process_rewards ---rm-url http://:/generate +--rm-url http://:/inference/v1/generate ``` ### Megatron Mode (`--opd-type megatron`) @@ -73,7 +73,7 @@ The teacher model is loaded directly into Megatron via `--opd-teacher-load`. Tea Complete example scripts are provided in `examples/on_policy_distillation/`: -### SGLang Teacher +### vLLM Teacher ```bash # 1. Download models and data diff --git a/docs/en/advanced/reproducibility.md b/docs/en/advanced/reproducibility.md index 79eb7b77e..da87cd7c5 100644 --- a/docs/en/advanced/reproducibility.md +++ b/docs/en/advanced/reproducibility.md @@ -1,19 +1,18 @@ # Reproducibility -Reproducibility is a bedrock of scientific progress. 通过结合 SGLang 提供的 [确定性推理](https://lmsys.org/blog/2025-09-22-sglang-deterministic/) 和 Megatron-LM 的确定性模式，slime 可以提供完全确定性（bitwise）的实验复现能力。 - -为了开启确定性训练，你需要通过 `pip uninstall flash_attn_3 -y` 卸载 flash attention 3，并设置： +Reproducibility is a bedrock of scientific progress. By combining vLLM's [batch-invariant deterministic inference](https://vllm.ai/blog/2025-11-10-bitwise-consistent-train-inference) with Megatron-LM's deterministic mode, slime supports bitwise experiment reproduction. +To enable deterministic training, you need to first uninstall the flash attention 3 in the docker with `pip uninstall flash_attn_3 -y` and set: ```bash - # sglang config - --sglang-enable-deterministic-inference - --sglang-attention-backend flashinfer + # vLLM config + --vllm-enable-deterministic-inference + --vllm-attention-backend flashinfer # megatron config --deterministic-mode ``` -以及设置如下环境变量： +And set the following environment variables: ```bash "env_vars": { @@ -24,9 +23,9 @@ Reproducibility is a bedrock of scientific progress. 通过结合 SGLang 提供 } ``` -我们提供了一个完全确定性的，用 Qwen2.5 0.5B 训练 GSM8K 的脚本。 +Here we provide the script to do RL training on Qwen2.5 0.5B model and GSM8K dataset with full deterministic. -可以用如下脚本初始化训练数据和 ckpt： +For data and checkpoint preparation, please run: ```bash # download @@ -43,10 +42,10 @@ PYTHONPATH=/root/Megatron-LM/ python \ --save /root/Qwen2.5-0.5B-Instruct_torch_dist/ ``` -可以使用如下脚本进行训练： +And to run training, ```bash bash script/run-qwen2.5-0.5B-reproducibility.sh ``` -这个 PR 中记录了 wandb 的截图 [pull#370](https://github.com/THUDM/slime/pull/370). +For screen shots of the wandb, please refer to [pull#370](https://github.com/THUDM/slime/pull/370). diff --git a/docs/en/advanced/speculative-decoding.md b/docs/en/advanced/speculative-decoding.md index bfca86936..9029edcc4 100644 --- a/docs/en/advanced/speculative-decoding.md +++ b/docs/en/advanced/speculative-decoding.md @@ -4,22 +4,31 @@ Speculative decoding is a key optimization for speeding up rollouts. Instead of ## Accelerating Inference with Speculative Decoding -For models with MTP layers (e.g., GLM-4.7, DeepSeek-V3/R1), simply add: +vLLM exposes speculative decoding as a single JSON config (`SpeculativeConfig`), +which slime forwards via `--vllm-speculative-config`. For models with MTP layers +(e.g., GLM-4.7, DeepSeek-V3/R1), pass: ```bash ---sglang-speculative-algorithm EAGLE ---sglang-speculative-num-steps 3 ---sglang-speculative-eagle-topk 1 ---sglang-speculative-num-draft-tokens 4 +--vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3}' ``` -If you want to use a separately trained draft model (e.g., one trained with [SpecForge](https://docs.sglang.ai/SpecForge/)), also set: +To use a separately trained draft model, set `model` (and optionally `draft_tensor_parallel_size`) +in the same JSON: ```bash ---sglang-speculative-draft-model-path /your/draft/model/path +--vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3,"model":"/your/draft/model/path"}' ``` -For detailed parameter meanings and configuration, see SGLang’s speculative decoding [documentation](https://docs.sglang.ai/advanced_features/speculative_decoding.html). +To train a draft model from scratch, see [TorchSpec](https://github.com/lightseekorg/TorchSpec) +and [vllm-project/speculators](https://github.com/vllm-project/speculators). +TorchSpec provides torch-native, disaggregated draft training. +Speculators supports EAGLE-3, DFlash, and MTP-style drafts, ships pre-trained +checkpoints on Hugging Face (see the `RedHatAI/*-speculator.*` collection), and +saves drafts in a format that `vllm serve ` can deploy directly. + +For the full list of `SpeculativeConfig` fields (including `disable_by_batch_size`, +`acceptance_method`, draft TP, etc.), see vLLM's speculative-decoding +[documentation](https://docs.vllm.ai/en/latest/features/speculative_decoding/). ## Online SFT for the Draft Model diff --git a/docs/en/advanced/sglang-config.md b/docs/en/advanced/vllm-config.md similarity index 74% rename from docs/en/advanced/sglang-config.md rename to docs/en/advanced/vllm-config.md index 70af72d2e..bd6949f08 100644 --- a/docs/en/advanced/sglang-config.md +++ b/docs/en/advanced/vllm-config.md @@ -1,33 +1,33 @@ -# SGLang Config: Advanced Engine Deployment +# vLLM Config: Advanced Engine Deployment -`--sglang-config` is a YAML-based configuration system for fine-grained control over SGLang engine deployment in slime. It enables **multi-model serving**, **Prefill-Decode (PD) disaggregation**, **heterogeneous server groups**, and can even serve as a **standalone SGLang launcher** for complex inference topologies. +`--vllm-config` is a YAML-based configuration system for fine-grained control over vLLM engine deployment in slime. It enables **multi-model serving**, **Prefill-Decode (PD) disaggregation**, **heterogeneous server groups**, and can even serve as a **standalone vLLM launcher** for complex inference topologies. --- ## Architecture Overview -In the default setup (without `--sglang-config`), slime deploys a single model behind a single router with uniform server groups: +In the default setup (without `--vllm-config`), slime deploys a single model behind a single router with uniform server groups: -![architecture overview](../../_static/image/arch.png) +![architecture overview](../../../imgs/arch.png) -With `--sglang-config`, the SGLang deployment expands into a multi-model, multi-router topology: +With `--vllm-config`, the vLLM deployment expands into a multi-model, multi-router topology: -![sglang-config architecture](../../_static/image/sglang_config.png) +![vllm-config architecture](../../_static/image/vllm_config.png) **Key design principles:** - **Each model gets its own router.** Models are isolated at the routing layer, allowing independent load balancing and fault tolerance. -- **Server groups within a model can be heterogeneous.** Different groups can have different TP sizes, worker types (prefill/decode/regular), and SGLang server argument overrides. +- **Server groups within a model can be heterogeneous.** Different groups can have different TP sizes, worker types (prefill/decode/regular), and vLLM engine argument overrides. - **Weight sync is per-model.** Only models with `update_weights: true` receive weight updates from training. Frozen models (reference, reward, etc.) are served as-is. --- ## Config Format -The config file is a YAML document with a top-level `sglang` key containing a list of model definitions: +The config file is a YAML document with a top-level `vllm` key containing a list of model definitions: ```yaml -sglang: +vllm: - name: # Required. Unique identifier for this model. model_path: # Optional. HF checkpoint path. Defaults to --hf-checkpoint. update_weights: # Optional. Whether to sync weights from training. Auto-inferred. @@ -36,7 +36,7 @@ sglang: - worker_type: # Required. One of: regular, prefill, decode, placeholder. num_gpus: # Required. Total GPUs allocated to this group. num_gpus_per_engine: # Optional. TP size override for this group. - overrides: # Optional. SGLang ServerArgs field overrides. + overrides: # Optional. vLLM EngineArgs field overrides. ``` ### Field Reference @@ -45,7 +45,7 @@ sglang: | Field | Type | Default | Description | |-------|------|---------|-------------| -| `name` | `str` | **Required** | Unique name for this model (e.g., `"actor"`, `"ref"`, `"reward"`). Used as the key in `args.sglang_model_routers`. | +| `name` | `str` | **Required** | Unique name for this model (e.g., `"actor"`, `"ref"`, `"reward"`). Used as the key in `args.vllm_model_routers`. | | `model_path` | `str` | `args.hf_checkpoint` | HuggingFace checkpoint path. All server groups within a model must use the same model path. | | `update_weights` | `bool` | Auto | Whether this model receives weight updates from training. When not set, automatically inferred: `true` if `model_path` matches `--hf-checkpoint`, `false` otherwise. | | `num_gpus_per_engine` | `int` | `args.rollout_num_gpus_per_engine` | Default TP size for server groups in this model. Individual groups can override. | @@ -58,13 +58,13 @@ sglang: | `worker_type` | `str` | **Required** | Engine type: `regular` (standard), `prefill` (PD prefill worker), `decode` (PD decode worker), or `placeholder` (reserve GPU slots without launching engines). | | `num_gpus` | `int` | **Required** | Total number of GPUs for this group. Must be > 0. | | `num_gpus_per_engine` | `int` | Model's `num_gpus_per_engine` | TP size override. Number of GPUs per engine instance. | -| `overrides` | `dict` | `{}` | SGLang `ServerArgs` field overrides. Applied on top of `--sglang-*` CLI args with highest priority. | +| `overrides` | `dict` | `{}` | vLLM `EngineArgs` field overrides. Applied on top of `--vllm-*` CLI args with highest priority. | ### Worker Types | Type | Description | Use Case | |------|-------------|----------| -| `regular` | Standard SGLang engine | Default mode, handles both prefill and decode | +| `regular` | Standard vLLM engine | Default mode, handles both prefill and decode | | `prefill` | PD disaggregation prefill worker | Dedicated to prompt processing; paired with `decode` workers | | `decode` | PD disaggregation decode worker | Dedicated to token generation; paired with `prefill` workers | | `placeholder` | Reserves GPU slots, no engine created | Reserve GPUs for training co-location or future use | @@ -78,8 +78,8 @@ sglang: The simplest config replicates the default behavior: ```yaml -# sglang_basic.yaml -sglang: +# vllm_basic.yaml +vllm: - name: default server_groups: - worker_type: regular @@ -88,7 +88,7 @@ sglang: ```bash python train.py \ - --sglang-config sglang_basic.yaml \ + --vllm-config vllm_basic.yaml \ --rollout-num-gpus 8 \ --rollout-num-gpus-per-engine 2 \ ... @@ -101,8 +101,8 @@ This creates 4 engines (8 GPUs ÷ 2 GPUs/engine) behind a single router. Separate prefill and decode phases onto dedicated server groups for better throughput in multi-turn and agentic workloads: ```yaml -# sglang_pd.yaml -sglang: +# vllm_pd.yaml +vllm: - name: actor server_groups: - worker_type: prefill @@ -115,7 +115,7 @@ sglang: ```bash python train.py \ - --sglang-config sglang_pd.yaml \ + --vllm-config vllm_pd.yaml \ --rollout-num-gpus 16 \ ... ``` @@ -125,15 +125,15 @@ python train.py \ - Using larger TP for decode (lower latency) - Independent scaling of prefill vs. decode capacity -> **Note:** PD disaggregation uses SGLang Model Gateway (sgl-router) with `pd_disaggregation=True`. +> **Note:** PD disaggregation uses vllm-router with `pd_disaggregation=True`. ### 3. Multi-Model Serving Deploy multiple models simultaneously, each behind its own router: ```yaml -# sglang_multi_model.yaml -sglang: +# vllm_multi_model.yaml +vllm: - name: actor update_weights: true # receives weight updates from training server_groups: @@ -160,7 +160,7 @@ sglang: ```bash python train.py \ - --sglang-config sglang_multi_model.yaml \ + --vllm-config vllm_multi_model.yaml \ --rollout-num-gpus 16 \ --hf-checkpoint /path/to/actor_model \ --rollout-function-path my_rollout.generate_rollout \ @@ -170,7 +170,7 @@ python train.py \ **Accessing models in custom rollout functions:** ```python -from slime.rollout.sglang_rollout import get_model_url +from slime.rollout.vllm_rollout import get_model_url from slime.utils.http_utils import post async def my_generate(args, sample, sampling_params): @@ -189,15 +189,15 @@ async def my_generate(args, sample, sampling_params): ... ``` -The `get_model_url()` helper reads from `args.sglang_model_routers`, a dict mapping model names to `(ip, port)` tuples that is automatically populated after engine startup. +The `get_model_url()` helper reads from `args.vllm_model_routers`, a dict mapping model names to `(ip, port)` tuples that is automatically populated after engine startup. ### 4. Multi-Model with PD Disaggregation Combine multi-model and PD disaggregation for maximum flexibility: ```yaml -# sglang_full.yaml -sglang: +# vllm_full.yaml +vllm: - name: actor update_weights: true server_groups: @@ -222,7 +222,7 @@ sglang: Use `placeholder` groups to reserve GPU slots without creating engines. This is useful for co-located training where some GPUs need to be reserved for training: ```yaml -sglang: +vllm: - name: actor server_groups: - worker_type: regular @@ -234,10 +234,10 @@ sglang: ### 6. Per-Group ServerArgs Overrides -Use `overrides` to apply SGLang `ServerArgs` fields to specific server groups without affecting others: +Use `overrides` to apply vLLM `ServerArgs` fields to specific server groups without affecting others: ```yaml -sglang: +vllm: - name: actor server_groups: - worker_type: regular @@ -250,23 +250,23 @@ sglang: enable_torch_compile: true ``` -Overrides take **highest priority**, overriding both the base `--sglang-*` CLI args and model-level defaults. This is especially useful for: +Overrides take **highest priority**, overriding both the base `--vllm-*` CLI args and model-level defaults. This is especially useful for: - Different memory configurations per group - Different context lengths for prefill vs. decode - Enabling experimental features on specific groups -### 7. Standalone SGLang Launcher +### 7. Standalone vLLM Launcher -While `--sglang-config` is designed for slime's training pipeline, it also works as a powerful launcher for pure inference scenarios using the `--rollout-external` pattern or by configuring slime to focus solely on serving. +While `--vllm-config` is designed for slime's training pipeline, it also works as a powerful launcher for pure inference scenarios using the `--rollout-external` pattern or by configuring slime to focus solely on serving. **Using external engines with a pre-launched topology:** -For complex production deployments, you may want to pre-launch SGLang engines independently and connect them to slime: +For complex production deployments, you may want to pre-launch vLLM engines independently and connect them to slime: ```bash -# Step 1: Launch SGLang engines externally -python -m sglang.launch_server --model-path /path/to/model --port 10090 ... -python -m sglang.launch_server --model-path /path/to/model --port 10091 ... +# Step 1: Launch vLLM engines externally +vllm serve /path/to/model --port 10090 ... +vllm serve /path/to/model --port 10091 ... # Step 2: Connect slime to external engines python train.py \ @@ -275,39 +275,39 @@ python train.py \ ... ``` -> **Note:** `--sglang-config` and `--rollout-external` are mutually exclusive. Use `--sglang-config` when you want slime to manage the full engine lifecycle; use `--rollout-external` when engines are pre-deployed. +> **Note:** `--vllm-config` and `--rollout-external` are mutually exclusive. Use `--vllm-config` when you want slime to manage the full engine lifecycle; use `--rollout-external` when engines are pre-deployed. --- ## Router Configuration -Each model in the config gets its own independent router (SGLang Model Gateway by default). +Each model in the config gets its own independent router (vllm-router by default). ### Router Policies You can configure the routing policy: ```bash ---router-policy round_robin # Simple round-robin ---router-policy consistent_hashing # Session affinity for multi-turn ---router-policy cache_aware # Cache-aware routing (default) +--router-policy round_robin # Simple round-robin +--router-policy consistent_hash # Session affinity for multi-turn +--router-policy cache_aware # Cache-aware routing (default) ``` ### Session-Affinity Routing for Multi-Turn Agents For multi-turn dialogues and agentic workloads, session affinity ensures that all requests belonging to the same conversation are routed to the same backend worker. This significantly improves prefix cache hit rates because the worker already has the conversation history cached. -slime automatically assigns each sample a unique `session_id` (stored in `sample.session_id`). When the router policy is `consistent_hashing`, this ID is passed as the `X-SMG-Routing-Key` header, and SGLang Model Gateway uses it to deterministically route all turns of the same session to the same worker. +slime automatically assigns each sample a unique `session_id` (stored in `sample.session_id`). When the router policy is `consistent_hash`, this ID is passed as the `x-session-id` header, and vllm-router uses it to deterministically route all turns of the same session to the same worker. ```bash ---router-policy consistent_hashing +--router-policy consistent_hash ``` **How it works:** 1. Each sample is assigned a unique `session_id` via UUID -2. On each request, slime passes `X-SMG-Routing-Key: ` in the HTTP header -3. SGLang Model Gateway's consistent hashing policy maps this key to a specific worker +2. On each request, slime passes `x-session-id: ` in the HTTP header +3. vllm-router's consistent-hash policy maps this key to a specific worker 4. Subsequent turns reuse the same `session_id`, ensuring they hit the same worker --- @@ -327,7 +327,7 @@ When the config is loaded, slime applies the following resolution cascade: ## Mutual Exclusion -`--sglang-config` is mutually exclusive with: +`--vllm-config` is mutually exclusive with: | Flag | Conflict Reason | |------|----------------| @@ -340,10 +340,10 @@ When the config is loaded, slime applies the following resolution cascade: Below is a complete example showing a multi-model setup for agentic RL training with PD disaggregation on 32 GPUs: -**Config file (`sglang_agent.yaml`):** +**Config file (`vllm_agent.yaml`):** ```yaml -sglang: +vllm: - name: actor update_weights: true server_groups: @@ -379,7 +379,7 @@ sglang: ```bash python train.py \ - --sglang-config sglang_agent.yaml \ + --vllm-config vllm_agent.yaml \ --hf-checkpoint /data/models/Qwen3-8B \ --rollout-num-gpus 32 \ --rollout-function-path my_agent.rollout.generate_rollout \ @@ -392,7 +392,7 @@ python train.py \ **Custom rollout function (`my_agent/rollout.py`):** ```python -from slime.rollout.sglang_rollout import get_model_url +from slime.rollout.vllm_rollout import get_model_url from slime.utils.http_utils import post async def generate_with_models(args, sample, sampling_params): @@ -442,12 +442,12 @@ No. All server groups within a model must share the same `model_path`. This is v ### Q: How do I access the router address for a specific model at runtime? -Use `get_model_url(args, "model_name", "/endpoint")` from `slime.rollout.sglang_rollout`. It reads from `args.sglang_model_routers`, which is a dict `{ model_name: (ip, port) }` populated automatically. +Use `get_model_url(args, "model_name", "/endpoint")` from `slime.rollout.vllm_rollout`. It reads from `args.vllm_model_routers`, which is a dict `{ model_name: (ip, port) }` populated automatically. -### Q: Can I use `--sglang-config` without training (inference only)? +### Q: Can I use `--vllm-config` without training (inference only)? -While `--sglang-config` is designed for slime's training loop, you can effectively use it for inference-only scenarios by configuring a rollout-only run. For fully standalone SGLang serving, consider using SGLang's native `launch_server` directly or the `--rollout-external` mode for connecting to pre-deployed engines. +While `--vllm-config` is designed for slime's training loop, you can effectively use it for inference-only scenarios by configuring a rollout-only run. For fully standalone vLLM serving, consider using vLLM's native `vllm serve` directly or the `--rollout-external` mode for connecting to pre-deployed engines. -### Q: What is the relationship between `--sglang-config` and `--prefill-num-servers`? +### Q: What is the relationship between `--vllm-config` and `--prefill-num-servers`? -`--prefill-num-servers` is the legacy way to enable PD disaggregation (it creates a single model with prefill + decode groups). `--sglang-config` is the newer, more flexible approach. They are mutually exclusive. We recommend migrating to `--sglang-config` for all new deployments. +`--prefill-num-servers` is the legacy way to enable PD disaggregation (it creates a single model with prefill + decode groups). `--vllm-config` is the newer, more flexible approach. They are mutually exclusive. We recommend migrating to `--vllm-config` for all new deployments. diff --git a/docs/en/blogs/introducing_slime.md b/docs/en/blogs/introducing_slime.md deleted file mode 100644 index 06eb60f20..000000000 --- a/docs/en/blogs/introducing_slime.md +++ /dev/null @@ -1,124 +0,0 @@ -# slime: An SGLang-Native Post-Training Framework for RL Scaling - -> This article was first released in [lmsys.org](https://lmsys.org/blog/2025-07-09-slime/). - -## Vision That Drives slime - -We believe in RL. We believe RL is the final piece toward AGI. - -If you feel the same way, you'll share our vision: - -- Every field should be end-to-end RLed and every task should become an agent environment. -- Every RL run should last longer, and every model should scale larger. -- RL systems should integrate seamlessly with existing infrastructure, letting us focus on new ideas instead of boilerplate engineering. - -That's why we present [slime](https://github.com/THUDM/slime), a post-training framework designed to be: - -- **Versatile** – with a fully customizable rollout interface and flexible training setups (colocated or decoupled, synchronous or asynchronous, RL or SFT cold start). -- **Performant** - integrating SGLang for inference and Megatron-LM for training, natively. -- **Maintainable** - with a lightweight codebase and smooth transition from Megatron pretraining to SGLang deployment. - -In short, a post-training framework for RL scaling. - -Here’s how we made it happen. - -## Customizability Brings Freedom - -> We should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries. -> -> -> — *The Bitter Lesson* -> - -A prevailing misconception within the RL community is the need for separate frameworks for different tasks: one for plain math, one for multi-turn tool calling, one for asynchronous training, one for agentic tasks, and so on. Forking and maintaining multiple frameworks is dreadful, leading to time-wasting bugfix cherry-picking, or worse, training crashes by missing patches. - -It wasn’t always like this: no one forks PyTorch just for a new dataloader. We believe the current chaos stems from the trap of dictating how people should build their applications. If we insist on defining a universal template for every rollout scenario, we’ll inevitably create an RL framework that meets only a fraction of real-world needs. - -slime views the data sampling in RL differently. We manage all SGLang servers within slime with [sgl-router](https://github.com/sgl-project/sglang/tree/main/sgl-router) and provide an interface for the data generation component, **allowing users to inject custom logic and freely interact with SGLang servers**. Unleash their creativity. - -![slime architecture](../../_static/image/arch.png) - -With the sgl-router, users only need to send HTTP requests to a single endpoint. By exposing this endpoint, complex agent environments can directly interact with slime through an OpenAI-compatible API — no need to modify the environment, and training-deployment consistency is preserved. - -Regarding training schemes, slime uses Ray for resource management, enabling **colocated** (same GPUs) or **decoupled** (separate GPUs) setups with a single flag (`--colocate`). - -And with Ray's asynchronous execution via `.remote()`, slime naturally supports asynchronous training. Changing synchronization behavior is as simple as moving the `ray.get` operation. And to make experimenting with different strategies easy, we didn't wrap the code with trainer classes, but simply exposed the training loop in entrypoint `train.py`. - -## Built for Performance - -**A decent RL framework must be fast and consistently fast.** - -**Fast** means leveraging the fastest inference and training frameworks. - -Unlike pre-training, RL workloads involve tons of online sampling during training, which makes the inference performance crucial. Therefore, slime exclusively integrates SGLang, and deliberately delivers an SGLang-native experience. - -So what does ‘SGLang-native’ mean? It means you can take full advantage of all SGLang optimizations — using SGLang inside slime feels just like using it standalone. To make that possible: - -- slime internally launches SGLang servers in a **server-based mode**. -- slime implements **seamless pass-through** for all SGLang parameters (with a `--sglang` prefix), ensuring that all optimization options can be enabled. For instance, you can pass `--sglang-enable-ep-moe`, `--sglang-enable-dp-attention` and `--sglang-enable-deepep-moe` for the powerful multi-node MoE inference capabilities. -- slime provides an **SGLang-only debug mode** (`--debug-rollout-only`) for easy performance tuning. - -Together, we can reproduce the standalone performance of SGLang within slime. Even the base image of slime is built on `lmsysorg/sglang:dev`. - -For training, slime integrates the battle-tested Megatron-LM, aiming for a similarly native pre-training experience: - -- slime also implements **seamless pass-through** for all Megatron parameters. -- slime supports **all Megatron parallelisms** (TP, PP, EP, CP) and monitors training MFU. -- slime offers a **Megatron-only debug mode** (`--debug-train-only`) and supports storing sampling data for reproducibility. - -Megatron can be notoriously complex, so we also provide checkpoint conversion tools to simplify its use. - -**Consistently fast** means keeping pace with the evolving inference and training frameworks. - -If you ever followed the [SGLang PR list](https://github.com/sgl-project/sglang/pulls), you will be astonished by its rapid evolution. Megatron, on the other hand, is often heavily customized, with every organization maintaining its own fork. slime is designed to keep pace with upstream changes in SGLang and adapt to optimizations in in-house Megatron variants. This is another reason why we pursue native support for SGLang and Megatron. The parameter pass-through makes upgrading effortless. - -Beyond optimizing inference and training frameworks, we also tackled RL-specific workloads. When SGLang needs changes to support these workflows, we work closely with the SGLang team to upstream patches—so slime can stay native, even as RL logic evolves. Examples include: - -**Optimizing weight updates**: Unlike inference tasks, RL training involves frequent updates to model weights. To address this, we’ve introduced several optimizations in SGLang: - - - Parameter updates for MoE models under various parallelism strategies ([#6265](https://github.com/sgl-project/sglang/pull/6265), [#6308](https://github.com/sgl-project/sglang/pull/6308), [#6311](https://github.com/sgl-project/sglang/pull/6311)). - - Bucketed parameter update support to reduce overhead ([#7292](https://github.com/sgl-project/sglang/pull/7292)). - -**`/abort_request` for dynamic sampling**: In RL algorithms that require oversampling, such as [DAPO](https://arxiv.org/abs/2503.14476), some requests may continue running even after sufficient data has been collected. In collaboration with the [AReal](https://github.com/inclusionAI/AReaL) team, we designed an new endpoint: `/abort_request`. This endpoint enables: - - - Immediate termination of on-going requests. - - Reclaiming partially generated content, which enables partial rollouts. - -Implemented in [#6698](https://github.com/sgl-project/sglang/pull/6698), [#6855](https://github.com/sgl-project/sglang/pull/6855), [#6184](https://github.com/sgl-project/sglang/pull/6184), [#5966](https://github.com/sgl-project/sglang/pull/5966). - - -## Lightweight and Extensible - -Focusing on customization and performance, slime: - -1. Provides a customizable rollout interface. -2. Uses Ray for GPU management and asynchronous execution. -3. Integrates SGLang for inference and Megatron for training. -4. Provides weight updates between training and inference. - -Pretty straightforward, right? slime transfers complexity from the framework to user-defined pipelines and core libraries (SGLang and Megatron), resulting in a lightweight, easily maintainable codebase. - -But it doesn’t stop at RL. - -Thanks to its modular design and powerful backends, slime can naturally extend to other post-training workflows with minimal extra code: - -- **SFT**: Load Megatron and use token prediction loss. -- **Rejection Sampling**: Use SGLang for filter, followed by Megatron SFT. - -*(Note that SFT feature is now in experimental state.)* - -Beyond that, slime's native integration **seamlessly bridges pre-training to online services**. We can use Megatron for pre-training, switch to slime (which integrates both Megatron and SGLang) for post-training, and finally use SGLang directly for evaluation and deployment. This eliminates the cumbersome and error-prone steps of converting checkpoint formats and aligning precision between frameworks. - -The unified pipeline saves us from tedious glue code, freeing us to focus on what really matters: better RL. Hurray! - -## Roadmap - -The journey of RL scaling has just begun, and slime is continuously evolving. In the next phase, we will focus on: - -1. Collaborating with the SGLang team to explore optimal RL training strategies for large-scale MoE models. -2. Supporting broader post-training workflows, strengthening the pre-training-to-production bridge. -3. Adding native PyTorch training backend support to lower the entry barrier. - -We hope slime accelerates your RL scaling journey and turns your innovative ideas into reality. Contributions and conversations are always welcome! - -Special thanks to the AMD GenAI - Foundation Model Team for Day-1 AMD hardware support. \ No newline at end of file diff --git a/docs/en/blogs/release_v0.1.0.md b/docs/en/blogs/release_v0.1.0.md deleted file mode 100644 index 2a1b93e6e..000000000 --- a/docs/en/blogs/release_v0.1.0.md +++ /dev/null @@ -1,158 +0,0 @@ -# v0.1.0: Redefining High-Performance RL Training Frameworks - -> The origin version of this article is in Chinese and was first released in [zhihu](https://zhuanlan.zhihu.com/p/1945237948166547268). - -With the help of the community, we've finally released the first version of **slime**, **v0.1.0**, just two months after it was open-sourced. - -In a nutshell, this version can be summarized as follows: - -> **slime v0.1.0 provides all the essential performance optimizations needed for large-scale MoE RL training.** - -Specifically, this version brings the following improvements: - - - **Performance**: - - Provides **efficient inference for MoE models**, especially with **fp8 rollout + deepep + mtp**. - - Designed a generic **training framework memory offload solution** to save more KV Cache space, thus increasing inference concurrency. - - **Faster parameter updates**. - - Achieves more training with fewer GPUs through **CPU Adam**. - - Supports **all of Megatron's parallel strategies** as well as **deepep**. - - **Features**: - - Added support for **GSPO** for MoE model training. - - Added support for **TIS** for fp8 rollout. - - **Correctness**: - - Implemented **Dense and MoE model CI** to strictly check metrics like kl. - -We hope to use slime v0.1.0 to demonstrate our understanding of high-performance RL training frameworks and have it become a baseline for future performance comparisons. - -Next, I'll elaborate on the design philosophy behind these features. - ------ - -## Performance Optimization: Pushing the Limits of RL Training Speed - -In traditional deep learning training, there's a universal solution for speedup: **add more GPUs**. By reducing the amount of data processed per GPU, you can significantly lower end-to-end training latency. - -However, this method doesn't work for RL training because **inference latency cannot be reduced by adding more GPUs**. Even with more GPUs, we still have to wait for the longest sample to finish decoding. While increasing throughput can improve the amount of training data per rollout, the off-policy issues caused by an excessively large inference batch size still have some limitations. - -I believe this is the biggest challenge for infrastructure under the current RL paradigm, which is: - -> **We want to scale inference compute, but we cannot scale inference latency.** - -The decoding speed of a single data point determines the upper limit of RL training speed. For larger MoE models, there are currently three common optimization methods to push this limit, and we've tried all of them: - -1. **Reduce memory access through quantization**: Considering that long calibration is not feasible in RL training, slime opts for fp8 quantization. -2. **Use deepep low-latency mode to reduce all2all latency across machines**: To work with deepep, slime recommends using blockwise quantization with fp8 to enable related SGLang configurations. -3. **Enable Speculative Sampling**: slime allows loading any draft model for the inference part (currently, it doesn't support updating the draft model during training). - -![](../../_static/image/blogs/release_v0.1.0/overrall.png) - -By using the three optimizations mentioned above, we can increase a model like **GLM4.5 355B-A32B** from less than 10 tokens/s for a single data point to **60-70 tokens/s**, which significantly raises the upper limit of RL training speed. - -In addition to monitoring inference throughput, slime also monitors `perf/longest_sample_tokens_per_sec` to better understand the potential for performance optimization in the inference part. - ------ - -## Doing More Experiments with Fewer GPUs: Fully Offloading Megatron - -After optimizing the upper limit, we noticed another characteristic of RL training: as long as the **KV Cache doesn't overflow**, increasing the inference batch size doesn't significantly affect training latency. - -**KV Cache overflow** occurs during inference when the response lengths of the data are all very long, leading to insufficient KV Cache space. This requires kicking out some half-generated data from the queue and then re-running prefill and subsequent inference steps after other data has been processed and freed up space. If a data point with a response length of 64k has to wait for 32k tokens to be decoded by other data during its inference, its total time is equivalent to decoding 96k tokens. This greatly impacts the RL training speed. - -Therefore, a more suitable training configuration is to calculate the minimum number of GPUs needed to prevent KV Cache overflow based on the inference batch size, the average response length, and the available KV Cache space on a single server. A group of these GPUs is then used for training. For example, if we have 512 GPUs and the calculation shows that 256 GPUs provide enough KV Cache, we should run two experiments in parallel instead of launching one experiment with all 512 GPUs. - -Based on this consideration, we noticed two points for optimization: - -1. **The optimal number of GPUs may not be sufficient to load the training part**. Inference only needs to load the fp8 parameters, while training generally requires more than 18 times the parameter size of GPU memory (bf16 param, fp32 grad, fp32 master param, fp32 m and v). To solve this, slime uses **Megatron's built-in CPU Adam** to save GPU memory for the training part. This strategy allowed us to provide solutions for training GLM 4.5 355B-A32B with 8 nodes and DeepSeek R1 with 16 nodes. -2. **Increase the KV Cache space available per SGLang Server**, which means increasing `mem_fraction`. For the more common integrated training and inference tasks, the main limitation for a larger `mem_fraction` is the residual GPU memory after offloading the training part to the CPU. Therefore, we need to find a generic way to offload the GPU memory used by the Megatron part. - -### How to Offload GPU Tensors Generically - -One crude approach is to find all the GPU Tensors allocated by Megatron and call `.to("cpu")` on all of them. This method has three difficulties: - - - It's hard to capture all GPU Tensors allocated by Megatron. - - Because Megatron's distributed optimizer reorganizes all parameters into some contiguous GPU buffers and then divides them with various slices, it's difficult to properly handle all references to correctly free the GPU Tensors. - - It requires checking the source code again with every new Megatron version, which is hard to maintain. - -Is there a more generic solution? - -We noticed that SGLang's `torch_memory_saver` and VLLM's `cumem_allocator` provide a more general offload solution. Their principle is that CUDA 10.2 provides a **series of Virtual Memory Management APIs**, similar to an operating system's virtual and physical addresses (VA and PA). When allocating GPU memory, they return a handle to a memory mapping instead of the actual physical address. Therefore, when offloading, we only need to "secretly" release the memory corresponding to this mapping and reallocate it when this memory is needed. The upper-level application doesn't need to be aware of this. - -![](../../_static/image/blogs/release_v0.1.0/cuda_vmm.png) - -A natural idea is to use this method to take over the entire training process in RL. However, this prevents the reuse of PyTorch's `CUDACachingAllocator`, and without the cache, memory fragmentation becomes more pronounced, easily leading to **OOM** during training. - -To continue reusing the native, cached allocator, we cannot use `CUDAPluggableAllocator`. Noticing again that slime's architecture has training and inference in different processes, we only need to **directly replace `cudaMalloc` and `cudaFree` used by `CUDACachingAllocator` in the training process with VMM APIs via `LD_PRELOAD`**. This allows us to completely and generically offload all GPU Tensors allocated by PyTorch. - -At the same time, we must also note one detail: VMM APIs and cudaIPC APIs (such as `cudaIpcGetMemHandle`) are incompatible. Therefore, for integrated training and inference tasks and DeepEP, we need to disable the `LD_PRELOAD` replacement and switch back to `cudaMalloc`. - -With help from the SGLang community, we updated `torch_memory_saver` for slime's needs, implementing this offload solution. - -### How to Offload NCCL - -After thoroughly offloading the GPU Tensors in Megatron, we found that a large amount of GPU memory still remained, which was caused by **NCCL**. In PyTorch, each NCCL group involved in communication allocates a substantial buffer. This issue is particularly noticeable for larger MoE models due to the various parallel strategies, potentially taking up **more than 10GB**. - -The `LD_PRELOAD` solution mentioned above doesn't handle the NCCL issue well, and we don't want to modify the NCCL source code to avoid having to maintain a separate NCCL fork in addition to slime. So, slime's approach is to use `destroy_process_group` to destroy the NCCL group when offloading Megatron and then recreate it before loading Megatron. To do this, we mimicked the VMM API and monkey patched `dist.new_group` to add a layer of `ReloadableProcessGroup`. - -In this way, we achieved a generic **NCCL offload**. However, because we need to rebuild the NCCL group, this operation has a slight impact on the speed of the first communication in each training iteration. But we believe this approach offers a significant advantage in terms of maintainability and the GPU memory it saves. - -Combining these two optimizations, we reduced Megatron's residual GPU memory from around **15-18GB** to **3-5GB**, which allows us to increase the `mem_fraction` for MoE models to **0.7-0.8**. This significantly boosts the available KV Cache, increases the concurrency each server can support, and allows us to launch more training tasks with fewer GPUs. - ------ - -## Parameter Update Optimization - -Parameter update is another special step in RL training. For this, slime v0.1.0 provides the best optimization solution for scenarios where training and inference are in different processes. This work was heavily optimized by Biao He. I recommend reading his blog post: - - - [Efficient Reinforcement Learning Training - Optimizing Weight Synchronization in slime](https://hebiao064.github.io/rl-weight-sync) - -Currently, slime can complete weight synchronization for a GLM4.5 355B-A32B model with bf16 weights in **48s** and complete fp8 blockwise quantization + parameter update in **100s** (the fp8 branch is still being optimized). - ------ - -## Training Optimization - -For the pure training part of slime, we believe Megatron already provides ample optimizations, so our main focus was to **ensure compatibility with all of Megatron's parallel strategies**. - -During this adaptation, we found an interesting bugfix: we discovered that when SGLang enabled mtp, the Megatron part couldn't start DeepEP. It turned out that when mtp is enabled, SGLang disables the overlap schedule, which causes a certain metadata communication to use nccl instead of gloo after being offloaded to the CPU, and this conflicts with DeepEP. - ------ - -## Performance Optimization Check List - -Since the release of slime, I've often been asked about its performance comparison with other frameworks. - -My understanding of benchmarks is that they should not be used as a weapon for frameworks to attack each other, but rather as a **tool for identifying gaps**. To that end, we will gradually release performance benchmarks that slime focuses on to improve ourselves. - -I also believe that before running benchmarks, you can analyze a framework's focus on performance from a qualitative perspective. Here's a basic feature check list for optimizations: - - - Does it support MoE training? (Currently, large-scale experiments are focused on MoE) - - Can the internal sglang `mem_fraction` or vllm `gpu_utilization` be adjusted to over 0.7? (Ensures KV Cache space) - - Does it support fp8 or lower precision inference? (Reduces inference memory access, boosts speed) - - Does it support enabling deepep for both training and inference? (Optimizes MoE all2all communication) - - Does it support speculative sampling? (Improves inference latency and throughput) - - Does it have an efficient training backend, such as Megatron or torchtitan, and support all necessary parallelization strategies? (Reuses mature training optimizations) - -slime v0.1.0 has made preliminary attempts at all the above optimizations, and there's still a lot of room for improvement. We hope this version can serve as a baseline for future slime versions or for performance comparisons between different frameworks. We also welcome all friends who share our pursuit of performance to try out slime and join the slime community\! - ------ - -## New Algorithm Support - -To better train MoE models and perform fp8 rollouts, we implemented **GSPO** and **TIS**. Additionally, community experts have helped implement algorithms like reinforce++ and reinforce++ baseline. - ------ - -## Correctness Verification - -slime v0.1.0 adds **end-to-end CI**: we run single-machine GLM4 9B and Qwen3 30B-A3B training for each PR, ensuring correctness through strict checks. For example, we explicitly require: - - - The recomputed log prob of the first rollout must be exactly equal to the log prob of the reference model. - - The ppo_kl of the first training step within each rollout must be exactly 0. - -Such precise verification is rarely achieved in training frameworks, and it's something we are very proud of. - ------ - -This is a brief introduction to slime v0.1.0. I hope it sparks your curiosity about slime and that it can be helpful in your work. - -Everyone is welcome to join the slime community. Let's work together to build an open RL Infra and contribute to RL scaling\! \ No newline at end of file diff --git a/docs/en/developer_guide/ci.md b/docs/en/developer_guide/ci.md index f84f46879..82ee9cbcc 100644 --- a/docs/en/developer_guide/ci.md +++ b/docs/en/developer_guide/ci.md @@ -6,7 +6,7 @@ slime uses GitHub Actions for CI. Tests are triggered by **PR labels** — addin The workflow is defined in `.github/workflows/pr-test.yml` (auto-generated from `pr-test.yml.j2`). Each CI job: -1. Runs on a self-hosted GPU runner via `docker run`; most tests use `slimerl/slime:latest`, while image validation uses `slimerl/slime-test:latest`. +1. Runs on a self-hosted GPU runner via `docker run`; most tests use `inferactinc/public:vime-vllm-cu129-latest`, while image validation uses `inferactinc/public:vime-vllm-cu129-latest`. 2. Installs slime with `pip install -e . --no-deps`. 3. Acquires the required GPUs via `tests/ci/gpu_lock_exec.py --count `. 4. Executes the test file: `python .py` or `python tests/.py`, depending on whether the test lives under `tests/` or a subdirectory such as `tests/plugin_contracts/`. @@ -23,7 +23,7 @@ Add a label to your PR to trigger the corresponding test suite: | `run-ci-megatron` | `e2e-test-megatron` | Core Megatron training tests covering dense, MoE, PPO, MTP, OPD, etc. | | `run-ci-precision` | `e2e-test-precision` | Numerical precision validation (parallel check). | | `run-ci-ckpt` | `e2e-test-ckpt` | Checkpoint save/load correctness (sync and async-save). | -| `run-ci-image` | `e2e-test-image` | Full test suite run on `slimerl/slime-test:latest` image (for image validation). | +| `run-ci-image` | `e2e-test-image` | Full test suite run on `inferactinc/public:vime-vllm-cu129-latest` image (for image validation). | | `run-ci-changed` | `e2e-test-changed` | **Dynamically** detects new/modified test files in the PR and runs only those. | All labels also run when triggered via `workflow_dispatch` (manual run from the Actions tab). @@ -40,11 +40,11 @@ This is the most useful label for development. When you add a new test file or m This means you don't need to manually register your new test in the workflow — just make sure your test file has a top-level `NUM_GPUS = ` constant and `run-ci-changed` will pick it up. -**Example**: If your PR adds `tests/test_qwen3_8B_opd_sglang.py` with `NUM_GPUS = 8`, adding the `run-ci-changed` label will automatically run that test on 8 GPUs. +**Example**: If your PR adds `tests/test_qwen3_8B_opd_vllm.py` with `NUM_GPUS = 8`, adding the `run-ci-changed` label will automatically run that test on 8 GPUs. ### `run-ci-image` — Full Suite on Test Image -This runs **all** registered tests on the `slimerl/slime-test:latest` Docker image. Use this label to: +This runs **all** registered tests on the `inferactinc/public:vime-vllm-cu129-latest` Docker image. Use this label to: - Validate a newly built Docker image before release. - Run the entire test suite for a comprehensive pre-merge check. @@ -57,7 +57,7 @@ This is the primary label for validating Megatron-backend changes. It covers: - Dense models: GLM4-9B, Qwen3-4B (PPO) - MoE models: Qwen3-30B-A3B (with DeepEP + FP8), Qwen3.6-35B-A3B PD + Mooncake, Moonlight-16B-A3B -- Specialized: MiMo-7B MTP, Qwen2.5-0.5B debug rollout-then-train, OPD with sglang teacher +- Specialized: MiMo-7B MTP, Qwen2.5-0.5B debug rollout-then-train, OPD with vLLM teacher All tests use 8 GPUs. If you are modifying Megatron training logic, loss computation, or checkpoint conversion, this is the label to use. diff --git a/docs/en/developer_guide/debug.md b/docs/en/developer_guide/debug.md index 4f8218a17..ec8f33478 100644 --- a/docs/en/developer_guide/debug.md +++ b/docs/en/developer_guide/debug.md @@ -7,8 +7,8 @@ During the development of slime, it is often necessary to check if the model's p 1. **First Training Step** 1. Check if the generated `rollout` is coherent. If not, there are two possible reasons: * Parameters were not loaded correctly. You need to check the logs for a confirmation that Megatron successfully loaded the checkpoint (ckpt). - * There was an error in updating the parameters. You can check if all parameters were converted and mapped correctly, or if the parameter names were converted according to the parallelization strategy (e.g., when `pp_size > 1`, check if the layer IDs for the parameters provided by the second stage are correct). A thorough method is to save all parameters in the `load_weights` implementation of the corresponding model in SGLang and verify that they are consistent with the loaded checkpoint. - * If all parameters are updated correctly and the problem persists, it's possible that some special buffers in SGLang were released during the release process. + * There was an error in updating the parameters. You can check if all parameters were converted and mapped correctly, or if the parameter names were converted according to the parallelization strategy (e.g., when `pp_size > 1`, check if the layer IDs for the parameters provided by the second stage are correct). A thorough method is to save all parameters in the `load_weights` implementation of the corresponding model in vLLM and verify that they are consistent with the loaded checkpoint. + * If all parameters are updated correctly and the problem persists, it's possible that some special buffers in vLLM were released during the release process. * If you are testing with a pretrained model, you can switch to an instruct version of a model with the same architecture to see if this garbled output is specific to the pretrained model. 2. Check the printed rollout stats to see if `log_probs` and `ref_log_probs` are exactly equal (meaning KL divergence is 0 in the first step) and their values are small. @@ -36,11 +36,11 @@ Specifically, slime currently provides the following parameters for separate deb 1. `--debug-rollout-only` - When enabled, slime will not load Megatron and will only initialize SGLang. You can use this method to debug the inference part. + When enabled, slime will not load Megatron and will only initialize vLLM. You can use this method to debug the inference part. 2. `--debug-train-only` - When enabled, slime will not load SGLang and will only initialize Megatron. You can use this method to debug the training part. + When enabled, slime will not load vLLM and will only initialize Megatron. You can use this method to debug the training part. 3. `--save-debug-rollout-data /your/saved/debug/data_{rollout_id}.pt` @@ -48,15 +48,15 @@ Specifically, slime currently provides the following parameters for separate deb 4. `--load-debug-rollout-data /your/saved/debug/data_{rollout_id}.pt` - When enabled, data will be loaded from `args.load_debug_rollout_data.format(rollout_id=rollout_id)`, and SGLang will not be initialized (automatically setting `debug_train_only=True`). This method allows you to fix the input for the training part to tune it, for example, by switching between different parallelization strategies. + When enabled, data will be loaded from `args.load_debug_rollout_data.format(rollout_id=rollout_id)`, and vLLM will not be initialized (automatically setting `debug_train_only=True`). This method allows you to fix the input for the training part to tune it, for example, by switching between different parallelization strategies. ## INT4 / Compressed-Tensors Quantization Checkpoint Issues -When using INT4-quantized models (e.g., `compressed-tensors` with `W4A16`), the checkpoint's `config.json` contains a `quantization_config.ignore` list that specifies which parameters should **not** be quantized. During online weight updates (Megatron → SGLang), slime also reads this ignore list to decide which parameters to INT4-quantize. An incorrect ignore list can cause silent errors: +When using INT4-quantized models (e.g., `compressed-tensors` with `W4A16`), the checkpoint's `config.json` contains a `quantization_config.ignore` list that specifies which parameters should **not** be quantized. During online weight updates (Megatron → vLLM), slime also reads this ignore list to decide which parameters to INT4-quantize. An incorrect ignore list can cause silent errors: 1. **MoE router weights (`mlp.gate.weight`) become all zeros** - The MoE router weight (`mlp.gate.weight`, shape `[num_experts, hidden_size]`) is a plain 2D weight tensor, but it is **not** a Linear layer weight. If it is not in the ignore list, the online quantizer will INT4-quantize it into `weight_packed`, `weight_scale`, `weight_zero_point`, etc. However, SGLang does not expect quantized names for the router, so these parameters are silently skipped during `load_weights`, resulting in all-zero gate weights. + The MoE router weight (`mlp.gate.weight`, shape `[num_experts, hidden_size]`) is a plain 2D weight tensor, but it is **not** a Linear layer weight. If it is not in the ignore list, the online quantizer will INT4-quantize it into `weight_packed`, `weight_scale`, `weight_zero_point`, etc. However, vLLM does not expect quantized names for the router, so these parameters are silently skipped during `load_weights`, resulting in all-zero gate weights. **Fix**: Ensure `config.json` contains `"re:.*mlp\\.gate\\..*"` in the ignore list. @@ -89,12 +89,12 @@ When using INT4-quantized models (e.g., `compressed-tensors` with `W4A16`), the 4. **How to diagnose** - - Use `--check-weight-update-equal` to verify that weights after a Megatron → SGLang sync match the expected values. If a parameter shows all zeros on the SGLang side, it was likely incorrectly quantized or missing from the checkpoint. - - Use `--debug-rollout-only` with a small number of GPUs to quickly test whether SGLang can generate coherent text from the quantized checkpoint alone. + - Use `--check-weight-update-equal` to verify that weights after a Megatron → vLLM sync match the expected values. If a parameter shows all zeros on the vLLM side, it was likely incorrectly quantized or missing from the checkpoint. + - Use `--debug-rollout-only` with a small number of GPUs to quickly test whether vLLM can generate coherent text from the quantized checkpoint alone. -## Debug sglang illegal memory access (IMA) +## Debug vllm illegal memory access (IMA) -When running large scale RL, we will occationally meet the IMA in SGLang, there are some debug suggestions based on our experience: +When running large scale RL, we will occationally meet the IMA in vLLM, there are some debug suggestions based on our experience: 1. Enable `CUDA_LAUNCH_BLOCKING=1` diff --git a/docs/en/developer_guide/profiling.md b/docs/en/developer_guide/profiling.md index 93b8ea5b3..a91eae493 100644 --- a/docs/en/developer_guide/profiling.md +++ b/docs/en/developer_guide/profiling.md @@ -1,6 +1,6 @@ # Profiling -In slime, we can perform detailed performance analysis of the rollout process using the profiling interface provided by SGLang. +In slime, we can perform detailed performance analysis of the rollout process using the profiling interface provided by vLLM. ## 1. Sleeping the Rollout Process @@ -16,9 +16,9 @@ python train.py \ This function will make the rollout process enter an infinite wait loop, allowing you to manually send requests or run stress testing tools. -## 2. Obtaining SGLang Engine List +## 2. Obtaining vLLM Engine List -SGLang engines (workers) are registered with the router. You can retrieve the list of all active engines by accessing the `/workers` endpoint of the router. +vLLM engines (workers) are registered with the router. You can retrieve the list of all active engines by accessing the `/workers` endpoint of the router. The router address is typically printed in the startup logs: ``` @@ -61,6 +61,6 @@ python tools/profile_rollout.py --router-url http://127.0.0.1:3000 --action stop While the Rollout process is in a waiting state via `sleep_rollout`, you can: 1. Start profiling using `tools/profile_rollout.py`. -2. Use stress testing tools (such as SGLang's built-in benchmark tools) to send requests to the router or directly to the engines. +2. Use stress testing tools (such as vLLM's built-in benchmark tools) to send requests to the router or directly to the engines. 3. Wait for profiling to complete (if `num_steps` was set) or stop it manually. 4. Collect the `.json` trace files from the `output_dir` and view them using `chrome://tracing` in Chrome or [Perfetto](https://ui.perfetto.dev/). diff --git a/docs/en/developer_guide/trace.md b/docs/en/developer_guide/trace.md index 6fb0c6967..5d24a6c4b 100644 --- a/docs/en/developer_guide/trace.md +++ b/docs/en/developer_guide/trace.md @@ -36,7 +36,7 @@ By default it also starts a local static server so you can open the generated HT - Each row corresponds to one sample. - Bars represent spans, while point markers represent instant events. - Span attributes recorded at the start or end of `trace_span(...)` are shown in the details panel. -- When SGLang returns PD disaggregation timings, the viewer adds synthetic `[P]` and `[D]` lanes to break out prefill/decode work. +- When vLLM returns PD disaggregation timings, the viewer adds synthetic `[P]` and `[D]` lanes to break out prefill/decode work. - When PD is not enabled, those virtual lanes are omitted automatically and the base trace still renders normally. ## Instrument custom code @@ -101,19 +101,18 @@ If you need to add attrs after part of the function has executed, use an inner ` - `trace_function(...)` for the outer function-level lifecycle span - nested `trace_span(...)` for important sub-steps such as generation, RM, filtering, or post-processing -If you want to record SGLang generation metadata in a consistent way, reuse `build_sglang_meta_trace_attrs`: +For per-turn attrs around an HTTP call, wrap it in `trace_span` directly: ```python -from slime.utils.trace_utils import build_sglang_meta_trace_attrs, trace_span +from slime.utils.trace_utils import trace_span -with trace_span(sample, "sglang_generate") as span: +with trace_span(sample, "vllm_generate", attrs={"max_tokens": params["max_new_tokens"]}): output = await post(url, payload) - span.update(build_sglang_meta_trace_attrs(output["meta_info"])) ``` ## Tips - Save a small number of rollouts first; the viewer is easiest to read when each dump contains a manageable number of samples. - The viewer is built from the saved `.pt` dump, so traces can be inspected offline on another machine. -- For GPU/kernel-level SGLang profiling traces, see [Profiling](./profiling.md). +- For GPU/kernel-level vLLM profiling traces, see [Profiling](./profiling.md). diff --git a/docs/en/examples/deepseek-r1.md b/docs/en/examples/deepseek-r1.md index ab10502ee..acbbf1135 100644 --- a/docs/en/examples/deepseek-r1.md +++ b/docs/en/examples/deepseek-r1.md @@ -5,7 +5,7 @@ This is an example of doing DeepSeek R1 RL training using 128xH100 GPUs. We will use bf16 for training, and an fp8 format with 128x128 blockwise quantization for inference. The maximum response length is 32k, and dynamic sampling will be used to filter data during training. -Regarding parallelism, for sglang we will enable EP64, activate dp attention, and deepep. For the Megatron part, we will use TP8, PP4, EP32, and CP4. +Regarding parallelism, for vLLM we will enable expert parallelism with DP. For the Megatron part, we will use TP8, PP4, EP32, and CP4. ⚠️ To save GPU memory, we will use CPU Adam. Each node (8xH100) will occupy 1.4\~1.5TB of host memory. If a single machine's host memory is insufficient, this can be resolved by adding more GPUs to expand the parallelism. @@ -73,7 +73,7 @@ for WORKER_IP in $(awk '{print $1}' $BASE_DIR/mpi_hostfile); do fi echo "Starting Ray worker on ${WORKER_IP}" ssh root@"${WORKER_IP}" \ - "pkill -9 sglang ; ray stop --force ; pkill -9 python ; ray start --address=${MASTER_ADDR}:6379 --num-gpus 8 --node-ip-address ${WORKER_IP} --disable-usage-stats" & + "pkill -9 vllm ; ray stop --force ; pkill -9 python ; ray start --address=${MASTER_ADDR}:6379 --num-gpus 8 --node-ip-address ${WORKER_IP} --disable-usage-stats" & done wait ``` @@ -91,7 +91,7 @@ This reads the model's config from [scripts/models/deepseek-v3.sh](https://githu ```bash CKPT_ARGS=( - # HF ckpt required by sglang, we also read the tokenizer from here + # HF ckpt required by vLLM, we also read the tokenizer from here --hf-checkpoint $BASE_DIR/DeepSeek-R1/ #--hf-checkpoint $BASE_DIR/DeepSeek-R1-bf16/ --ref-load $BASE_DIR/DeepSeek-R1_torch_dist/ @@ -102,7 +102,7 @@ CKPT_ARGS=( ) ``` -slime will perform online quantization during training based on the quantization configuration in `hf_checkpoint`. For instance, in the current example, we are using the fp8 checkpoint of DeepSeek R1. This means that when updating parameters, we will first perform blockwise quantization on the parameters before passing them to sglang. +slime will perform online quantization during training based on the quantization configuration in `hf_checkpoint`. For instance, in the current example, we are using the fp8 checkpoint of DeepSeek R1. This means that when updating parameters, we will first perform blockwise quantization on the parameters before passing them to vLLM. #### PERF\_ARGS @@ -167,30 +167,23 @@ OPTIMIZER_ARGS=( ) ``` -#### SGLANG\_ARGS +#### VLLM\_ARGS -These are the parameters required by sglang. Here, `--rollout-num-gpus-per-engine` basically corresponds to sglang's `tp_size`. Other sglang parameters are passed to slime by adding a `--sglang-` prefix. To fully leverage sglang's large EP inference capabilities, we have added configurations like ep64, dp\_attention dp8, and deepep mode auto. +These are the parameters required by vLLM. Here, `--rollout-num-gpus-per-engine` corresponds to vLLM's `tp_size`. Other vLLM parameters are passed to slime by adding a `--vllm-` prefix. -The final `--sglang-server-concurrency` is a parameter specific to slime. It is used to prevent the sglang server's concurrent requests from becoming too large and crashing the HTTP server. The default is 512. However, since we now have one server for 8 nodes, we have adjusted it to 1024 to ensure that each dp rank can have a concurrency of 128. +`--vllm-server-concurrency` is a parameter specific to slime. It is used to prevent the vLLM engine's concurrent requests from becoming too large and crashing the HTTP server. The default is 512. However, since we now have one server for 8 nodes, we have adjusted it to 1024 to ensure that each dp rank can have a concurrency of 128. ```bash -SGLANG_ARGS=( +VLLM_ARGS=( --rollout-num-gpus-per-engine 64 - --sglang-mem-fraction-static 0.7 - ----sglang-ep-size 64 + --vllm-gpu-memory-utilization 0.7 + --vllm-enable-expert-parallel - # dp attention - --sglang-enable-dp-attention - --sglang-dp-size 8 - --sglang-moe-dense-tp-size 1 - --sglang-enable-dp-lm-head - - # enable deepep for sglang - --sglang-moe-a2a-backend deepep - --sglang-deepep-mode auto + # data parallelism for the attention block + --vllm-data-parallel-size 8 # make every dp rank have 128 concurrency - --sglang-server-concurrency 1024 + --vllm-server-concurrency 1024 ) ``` diff --git a/docs/en/examples/glm4-9B.md b/docs/en/examples/glm4-9B.md index 4a934ee14..9ea7d3883 100644 --- a/docs/en/examples/glm4-9B.md +++ b/docs/en/examples/glm4-9B.md @@ -2,7 +2,7 @@ ## Environment Setup -After pulling the `slimerl/slime:latest` image, initialize the image environment as follows: +After pulling the `inferactinc/public:vime-vllm-cu129-latest` image, initialize the image environment as follows: ```bash cd /root/ @@ -72,7 +72,7 @@ MODEL_ARGS += ( --rotary-base 10000 ) ```bash CKPT_ARGS=( - # HF checkpoint required by sglang; we also read the tokenizer from here + # HF checkpoint required by vLLM; we also read the tokenizer from here --hf-checkpoint /root/GLM-Z1-9B-0414 # Checkpoint for the reference model --ref-load /root/GLM-Z1-9B-0414_torch_dist @@ -191,17 +191,17 @@ OPTIMIZER_ARGS=( ) ``` -#### SGLANG\_ARGS +#### VLLM\_ARGS -Parameters required by sglang. Here, `--rollout-num-gpus-per-engine` basically corresponds to sglang's `tp_size`. Other sglang parameters are passed to slime by adding the `--sglang-` prefix. +Parameters required by vLLM. Here, `--rollout-num-gpus-per-engine` corresponds to vLLM's `tp_size`. Other vLLM parameters are passed to slime by adding the `--vllm-` prefix. ```bash -SGLANG_ARGS=( +VLLM_ARGS=( --rollout-num-gpus-per-engine 2 ) ``` -⚠️ slime uses `sgl-router` to schedule multiple sglang servers. `dp_size` is not supported when DP attention is disabled. +⚠️ slime uses `vllm-router` to schedule multiple vLLM engines. ### Co-located Training and Inference @@ -231,7 +231,7 @@ ray job submit ... \ In this case, both training and inference will share these 8 GPUs. -⚠️ When using co-located training and inference, Megatron will always occupy some GPU memory. Therefore, you need to adjust `--sglang-mem-fraction-static` to reduce the proportion of memory occupied by sglang. +⚠️ When using co-located training and inference, Megatron will always occupy some GPU memory. Therefore, you need to adjust `--vllm-gpu-memory-utilization` to reduce the proportion of memory occupied by vLLM. ### Dynamic Sampling diff --git a/docs/en/examples/glm4.7-30B-A3B.md b/docs/en/examples/glm4.7-30B-A3B.md index e677561aa..ff8a549b6 100644 --- a/docs/en/examples/glm4.7-30B-A3B.md +++ b/docs/en/examples/glm4.7-30B-A3B.md @@ -67,38 +67,35 @@ GLM-4.7-Flash is a Mixture-of-Experts (MoE) model with 64 routed experts (top-4 ) ``` -3. Enable MoE optimization in SGLang with DP attention: +3. Enable MoE expert parallelism in vLLM. GLM-4.7-Flash is a non-MLA model, + so we use 8-way data parallelism on the attention block with expert + parallelism on the experts: ```bash - SGLANG_ARGS=( + VLLM_ARGS=( --rollout-num-gpus-per-engine 8 - --sglang-mem-fraction-static 0.7 - --sglang-enable-dp-attention - --sglang-dp-size 8 - --sglang-enable-dp-lm-head - --sglang-moe-dense-tp-size 1 + --vllm-gpu-memory-utilization 0.7 + --vllm-data-parallel-size 8 + --vllm-enable-expert-parallel ... ) ``` #### MTP Speculative Decoding (Inference Acceleration) -GLM-4.7-Flash includes 1 MTP (Multi-Token Prediction) layer, which can be used for speculative decoding during inference to speed up rollout generation. To enable this, add the following to `SGLANG_ARGS`: +GLM-4.7-Flash includes 1 MTP (Multi-Token Prediction) layer, which can be used for speculative decoding during inference to speed up rollout generation. To enable this, add the following to `VLLM_ARGS`: ```bash -SGLANG_ARGS=( +VLLM_ARGS=( ... # MTP speculative decoding (EAGLE) - --sglang-speculative-algorithm EAGLE - --sglang-speculative-num-steps 3 - --sglang-speculative-eagle-topk 1 - --sglang-speculative-num-draft-tokens 4 + --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3}' ) ``` -This enables SGLang to use the model's MTP layer as a draft model for EAGLE-style speculative decoding. The MTP layer predicts multiple future tokens, and SGLang verifies them in parallel, leading to faster generation. +This enables vLLM to use the model's MTP layer as a draft model for EAGLE-style speculative decoding. The MTP layer predicts multiple future tokens, and vLLM verifies them in parallel, leading to faster generation. -> ⚠️ **Note**: Speculative decoding requires additional GPU memory. If you encounter OOM issues, try reducing `--sglang-mem-fraction-static` or disabling speculative decoding. +> ⚠️ **Note**: Speculative decoding requires additional GPU memory. If you encounter OOM issues, try reducing `--vllm-gpu-memory-utilization` or disabling speculative decoding. #### MTP Training @@ -140,17 +137,15 @@ Key modifications for multi-node: - Remove CPU Adam configurations (distributed optimizer reduces per-GPU memory usage). - Adjust parallelism: e.g., TP=4, PP=2, EP=8, CP=2. -When the total number of GPUs is not a multiple or divisor of the total number of experts (64), you can use `--sglang-ep-num-redundant-experts` to add redundant experts. For example, in a 24-GPU scenario: +When the total number of GPUs is not a multiple or divisor of the total number of experts (64), enable vLLM's EPLB (Expert Parallelism Load Balancer) and configure redundant experts via `--vllm-eplb-config`. For example, in a 24-GPU scenario: ```bash -SGLANG_ARGS=( +VLLM_ARGS=( --rollout-num-gpus-per-engine 24 - --sglang-mem-fraction-static 0.7 - --sglang-ep-size 24 - --sglang-enable-dp-attention - --sglang-dp-size 3 - --sglang-moe-dense-tp-size 1 - --sglang-enable-dp-lm-head - --sglang-ep-num-redundant-experts 16 + --vllm-gpu-memory-utilization 0.7 + --vllm-data-parallel-size 3 + --vllm-enable-expert-parallel + --vllm-enable-eplb + --vllm-eplb-config '{"num_redundant_experts": 16}' ) ``` diff --git a/docs/en/examples/glm4.7-355B-A32B.md b/docs/en/examples/glm4.7-355B-A32B.md index 8035f6782..18d74b691 100644 --- a/docs/en/examples/glm4.7-355B-A32B.md +++ b/docs/en/examples/glm4.7-355B-A32B.md @@ -83,39 +83,35 @@ GLM-4.7 is a Mixture-of-Experts (MoE) model with 160 routed experts (top-8 activ ) ``` -3. Enable MoE optimization in SGLang with DP attention: +3. Enable MoE expert parallelism in vLLM. GLM-4.7 is non-MLA, so we use 4-way + data parallelism on the attention block with expert parallelism on the + experts (EP size is auto-derived as `tensor_parallel_size × data_parallel_size`): ```bash - SGLANG_ARGS=( + VLLM_ARGS=( --rollout-num-gpus-per-engine 32 - --sglang-mem-fraction-static 0.7 - --sglang-enable-dp-attention - --sglang-dp-size 4 - --sglang-ep-size 32 - --sglang-enable-dp-lm-head - --sglang-moe-dense-tp-size 1 + --vllm-gpu-memory-utilization 0.7 + --vllm-data-parallel-size 4 + --vllm-enable-expert-parallel ... ) ``` #### MTP Speculative Decoding (Inference Acceleration) -GLM-4.7 includes MTP (Multi-Token Prediction) layers that can be used for speculative decoding during inference to speed up rollout generation. To enable this, add the following to `SGLANG_ARGS`: +GLM-4.7 includes MTP (Multi-Token Prediction) layers that can be used for speculative decoding during inference to speed up rollout generation. To enable this, add the following to `VLLM_ARGS`: ```bash -SGLANG_ARGS=( +VLLM_ARGS=( ... # MTP speculative decoding (EAGLE) - --sglang-speculative-algorithm EAGLE - --sglang-speculative-num-steps 3 - --sglang-speculative-eagle-topk 1 - --sglang-speculative-num-draft-tokens 4 + --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3}' ) ``` -This lets SGLang use the model's MTP layer as the draft model for EAGLE-style speculative decoding. +This lets vLLM use the model's MTP layer as the draft model for EAGLE-style speculative decoding. -> ⚠️ **Note**: Speculative decoding requires additional GPU memory. If you encounter OOM issues, try reducing `--sglang-mem-fraction-static` or disabling speculative decoding. +> ⚠️ **Note**: Speculative decoding requires additional GPU memory. If you encounter OOM issues, try reducing `--vllm-gpu-memory-utilization` or disabling speculative decoding. #### MTP Training @@ -146,13 +142,18 @@ This example already targets multi-node training. Before launching: - Set `MASTER_ADDR` to an address reachable by all nodes. - Unset proxy variables before starting Ray workers. - Provide a `HOSTFILE` listing worker IPs (one per line) and export `HOSTFILE=/path/to/hostfile` before launching. -- Adjust parallelism coherently. The default example uses TP=8, PP=4, EP=16, CP=2, while rollout uses 32 GPUs per engine with SGLang DP attention. +- Adjust parallelism coherently. The default example uses TP=8, PP=4, EP=16, CP=2, while rollout uses 32 GPUs per engine with DP=4 + expert parallel. -If your rollout GPU count does not divide the expert count cleanly, you can use `--sglang-ep-num-redundant-experts` to add redundant experts. +If your rollout GPU count does not divide the expert count cleanly, enable vLLM's EPLB (Expert Parallelism Load Balancer) and configure redundant experts via `--vllm-eplb-config`, e.g.: + +```bash +--vllm-enable-eplb +--vllm-eplb-config '{"num_redundant_experts": 16}' +``` ## FP8 Rollout -The open-source FP8 checkpoint of GLM-4.7 uses per-channel quantization, which cannot currently enable DeepEP in SGLang. You can convert it to a 128x128 per-block FP8 checkpoint with the tool provided in slime: +The open-source FP8 checkpoint of GLM-4.7 uses per-channel quantization, which cannot currently enable DeepEP in vLLM. You can convert it to a 128x128 per-block FP8 checkpoint with the tool provided in slime: ```bash cd /root/slime @@ -165,25 +166,18 @@ python tools/convert_hf_to_fp8.py \ Then switch `--hf-checkpoint` to `$BASE_DIR/GLM-4.7-355B-A32B-FP8/` to enable FP8 rollout. -An example FP8 `SGLANG_ARGS` setup is: +An example FP8 `VLLM_ARGS` setup is: ```bash -SGLANG_ARGS=( +VLLM_ARGS=( --rollout-num-gpus-per-engine 32 - --sglang-mem-fraction-static 0.7 - --sglang-enable-dp-attention - --sglang-dp-size 32 - --sglang-ep-size 32 - --sglang-moe-dense-tp-size 1 - --sglang-enable-dp-lm-head - --sglang-cuda-graph-bs 1 2 4 8 $(seq 16 8 128) - - --sglang-speculative-algorithm EAGLE - --sglang-speculative-num-steps 3 - --sglang-speculative-eagle-topk 1 - --sglang-speculative-num-draft-tokens 4 - - --sglang-moe-a2a-backend deepep - --sglang-deepep-mode auto + --vllm-gpu-memory-utilization 0.7 + --vllm-data-parallel-size 32 + --vllm-enable-expert-parallel + --vllm-cudagraph-capture-sizes 1 2 4 8 $(seq 16 8 128) + + --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3}' + + --vllm-all2all-backend deepep_high_throughput ) ``` diff --git a/docs/en/examples/qwen3-30B-A3B.md b/docs/en/examples/qwen3-30B-A3B.md index 04d973adf..d413fa829 100644 --- a/docs/en/examples/qwen3-30B-A3B.md +++ b/docs/en/examples/qwen3-30B-A3B.md @@ -56,23 +56,21 @@ Here, we will briefly introduce the MoE-related parts in the [run-qwen3-30B-A3B. ) ``` -3. Enable MoE optimization supported by SGLang. The current configuration is ep8: +3. Enable MoE expert parallelism in vLLM. EP size is auto-derived as + `tensor_parallel_size × data_parallel_size`, so for an 8-GPU engine + `--vllm-enable-expert-parallel` alone gives you EP=8: ```bash - SGLANG_ARGS=( + VLLM_ARGS=( --rollout-num-gpus-per-engine 8 - --sglang-mem-fraction-static 0.7 - --sglang-ep-size 8 - --sglang-cuda-graph-bs 1 2 4 8 $(seq 16 8 256) + --vllm-gpu-memory-utilization 0.7 + --vllm-enable-expert-parallel + --vllm-cudagraph-capture-sizes 1 2 4 8 $(seq 16 8 256) ) ``` - Similarly, you can also add DP attention, for example, by configuring: - - ```bash - --sglang-enable-dp-attention - --sglang-dp-size 8 - ``` + For DP on the attention block plus EP on the experts, combine + `--vllm-data-parallel-size N` with `--vllm-enable-expert-parallel`. ### BF16 Training with FP8 Inference @@ -103,18 +101,15 @@ For a multi-node environment, the following modifications are necessary: In addition, you can make the following changes: - - When the total number of GPUs is not a multiple or divisor of the total number of experts, you can use `--sglang-ep-num-redundant-experts` to add redundant experts. For example, in a 24-GPU scenario, you can configure it as follows: + - When the total number of GPUs is not a multiple or divisor of the total number of experts, enable vLLM's EPLB (Expert Parallelism Load Balancer) and configure redundant experts via `--vllm-eplb-config`. For example, in a 24-GPU scenario: ```bash - SGLANG_ARGS=( + VLLM_ARGS=( --rollout-num-gpus-per-engine 24 - --sglang-mem-fraction-static 0.7 - --sglang-ep-size 24 - --sglang-enable-dp-attention - --sglang-dp-size 3 - - --sglang-moe-dense-tp-size 1 - --sglang-enable-dp-lm-head - --sglang-ep-num-redundant-experts 16 + --vllm-gpu-memory-utilization 0.7 + --vllm-data-parallel-size 3 + --vllm-enable-expert-parallel + --vllm-enable-eplb + --vllm-eplb-config '{"num_redundant_experts": 16}' ) ``` diff --git a/docs/en/examples/qwen3-4B.md b/docs/en/examples/qwen3-4B.md index 035066e12..11ec73957 100644 --- a/docs/en/examples/qwen3-4B.md +++ b/docs/en/examples/qwen3-4B.md @@ -2,7 +2,7 @@ ## Environment Setup -After pulling the `slimerl/slime:latest` image, initialize the image environment as follows: +After pulling the `inferactinc/public:vime-vllm-cu129-latest` image, initialize the image environment as follows: ```bash cd /root/ diff --git a/docs/en/examples/qwen3-4b-base-openhermes.md b/docs/en/examples/qwen3-4b-base-openhermes.md index 1ae784902..55ea305f8 100644 --- a/docs/en/examples/qwen3-4b-base-openhermes.md +++ b/docs/en/examples/qwen3-4b-base-openhermes.md @@ -52,7 +52,7 @@ bash script/run-qwen3-4B-base-sft.sh You can compare [run-qwen3-4B-base-sft.sh](https://github.com/THUDM/slime/blob/main/scripts/run-qwen3-4B-base-sft.sh) with [run-qwen3-4B.sh](https://github.com/THUDM/slime/blob/main/scripts/run-qwen3-4B.sh). You will find that besides changing the model from the instruct version to the base model, the main adjustments are as follows: -1. Removed `SGLANG_ARGS` and `GRPO_ARGS`. This is because it is not necessary to start SGLang or configure GRPO-related settings during the SFT process. +1. Removed `VLLM_ARGS` and `GRPO_ARGS`. This is because it is not necessary to start vLLM or configure GRPO-related settings during the SFT process. 2. Renamed `ROLLOUT_ARGS` to `SFT_ARGS` and configured it as follows: @@ -73,7 +73,7 @@ You can compare [run-qwen3-4B-base-sft.sh](https://github.com/THUDM/slime/blob/m ) ``` - SFT actually reuses the custom rollout functionality of slime. By using `--rollout-function-path`, the data generation part is switched from the RL rollout that uses `sglang` to the SFT version that reads data from a file, which is `slime.rollout.sft_rollout.generate_rollout`. + SFT actually reuses the custom rollout functionality of slime. By using `--rollout-function-path`, the data generation part is switched from the RL rollout that uses `vLLM` to the SFT version that reads data from a file, which is `slime.rollout.sft_rollout.generate_rollout`. For SFT, it is recommended to set `rollout_batch_size` and `global_batch_size` to the same value and not to configure `n_samples_per_prompt`. This is equivalent to training one batch right after reading one batch. @@ -81,6 +81,6 @@ You can compare [run-qwen3-4B-base-sft.sh](https://github.com/THUDM/slime/blob/m As for `--calculate-per-token-loss`, this is because `slime` defaults to calculating the per-sample mean for GRPO. In general SFT training, the average is taken over all unmasked tokens in a batch, so it is recommended to configure this. - Finally, `--disable-compute-advantages-and-returns` indicates that there is no need to pre-calculate log probabilities during the SFT process, and `--debug-train-only` means that `sglang` does not need to be initialized. + Finally, `--disable-compute-advantages-and-returns` indicates that there is no need to pre-calculate log probabilities during the SFT process, and `--debug-train-only` means that `vLLM` does not need to be initialized. 3. Used `train_async.py` instead of `train.py`. This is to leverage the asynchronous training process to implement data prefetching. diff --git a/docs/en/get_started/customization.md b/docs/en/get_started/customization.md index 79d2e8264..c913958f8 100644 --- a/docs/en/get_started/customization.md +++ b/docs/en/get_started/customization.md @@ -33,7 +33,7 @@ Below is a summary of all available customization interfaces and their purposes. ### 1. Rollout Function (`--rollout-function-path`) -**Default**: `slime.rollout.sglang_rollout.generate_rollout` +**Default**: `slime.rollout.vllm_rollout.generate_rollout` **Purpose**: Override the entire rollout generation logic. @@ -406,7 +406,7 @@ Stabilize MoE RL training by recording and replaying expert routing decisions to | Argument | Description | | --- | --- | | `--use-routing-replay` | Forward-backward routing consistency in training. ([arXiv:2507.18071](https://arxiv.org/abs/2507.18071)) | -| `--use-rollout-routing-replay` | R3: Replay routing from rollout during training. Supported by slime's default `sglang_router` path. ([arXiv:2510.11370](https://arxiv.org/abs/2510.11370)) | +| `--use-rollout-routing-replay` | R3: Replay routing from rollout during training. Supported by slime's default `vllm_rollout` path. ([arXiv:2510.11370](https://arxiv.org/abs/2510.11370)) | ## Testing Custom Function Paths diff --git a/docs/en/get_started/qa.md b/docs/en/get_started/qa.md index 2344e4b1c..78611282e 100644 --- a/docs/en/get_started/qa.md +++ b/docs/en/get_started/qa.md @@ -43,21 +43,21 @@ Yes. Data packing refers to the process of concatenating samples of varying lengths during training to improve GPU utilization. slime performs this operation by default. -8. **What should I do if the sglang component shows a `Max retries exceeded with url: /get_model_info (Caused by NewConnectionError)` error?** +8. **What should I do if the vLLM component shows a `Max retries exceeded with url: /health (Caused by NewConnectionError)` error?** - This issue primarily stems from port conflicts caused by multiple sglang servers running on a single machine. We are currently working with the sglang team to resolve this. A temporary workaround is to minimize the number of sglang servers on a single machine, for example, by setting `tp=8`. + slime probes `GET /health` to wait for each vLLM engine to come up; this error means the readiness probe never connected. The most common cause is port conflicts when multiple vLLM engines share a single machine. A temporary workaround is to minimize the number of vLLM engines per machine, for example, by setting `tp=8`. 9. **My gradient norm is very high and the training crashes. What should I do?** First, ensure that your data and model are compatible. For example, if your data already uses a chat template, check if this template matches the one used by the original model. If the data is correct, please refer to our [Debug Guide](../developer_guide/debug.md) for a more in-depth analysis. -10. **My sglang generation takes an extremely long time, GPU power is maxed out, and there's no output for a long while. Why?** +10. **My vLLM generation takes an extremely long time, GPU power is maxed out, and there's no output for a long while. Why?** Please verify that the model corresponding to `--hf-checkpoint` has its stop tokens configured correctly. If not, you can set them using the `--rollout-stop` or `--rollout-stop-token-ids` arguments. -11. **Sglang shows an `an illegal memory access was encountered` error.** +11. **vLLM shows an `an illegal memory access was encountered` error.** - According to [SGLang documentation](https://docs.sglang.io/references/faq.html), this could be an OOM error. Consider reducing the value of `--sglang-mem-fraction-static`. + This is often an OOM. Consider reducing `--vllm-gpu-memory-utilization`. 12. **A `JSONDecodeError` occurs related to torch compile/inductor.** diff --git a/docs/en/get_started/quick_start.md b/docs/en/get_started/quick_start.md index ea792928e..d993eaa69 100644 --- a/docs/en/get_started/quick_start.md +++ b/docs/en/get_started/quick_start.md @@ -5,7 +5,7 @@ This document will guide you through setting up the environment and getting star ## Basic Environment Setup -Since slime may contain temporary patches for sglang/megatron, to avoid potential environment configuration issues, we strongly recommend **users to use our latest Docker image**, which comes pre-configured with all dependencies. +Since slime may contain temporary patches for vllm/megatron, to avoid potential environment configuration issues, we strongly recommend **users to use our latest Docker image**, which comes pre-configured with all dependencies. ### Hardware Support @@ -28,12 +28,12 @@ Please execute the following commands to pull the latest image and start an inte ```shell # Pull the latest image -docker pull slimerl/slime:latest +docker pull inferactinc/public:vime-vllm-cu129-latest # Start the container docker run --rm --gpus all --ipc=host --shm-size=16g \ --ulimit memlock=-1 --ulimit stack=67108864 \ - -it slimerl/slime:latest /bin/bash + -it inferactinc/public:vime-vllm-cu129-latest /bin/bash ``` ### Install slime @@ -286,17 +286,17 @@ OPTIMIZER_ARGS=( ) ``` -### SGLANG_ARGS: SGLang Service Parameters +### VLLM_ARGS: vLLM Service Parameters -This part of parameters is used to configure SGLang inference service. -- `--rollout-num-gpus-per-engine`: Basically equivalent to SGLang's `tp_size`. -- Other SGLang parameters can be passed to slime by adding the `--sglang-` prefix, and slime will automatically forward them to SGLang. For example, to set SGLang's `--log-level INFO` parameter, just use `--sglang-log-level INFO`. +This part of parameters is used to configure the vLLM inference service. +- `--rollout-num-gpus-per-engine`: Equivalent to vLLM's `tp_size`. +- Other vLLM parameters can be passed to slime by adding the `--vllm-` prefix, and slime will automatically forward them to vLLM. For example, to set vLLM's `--log-level INFO` parameter, just use `--vllm-log-level INFO`. > ⚠️ **Note**: -> slime uses `sgl-router` to schedule multiple SGLang Servers. Without enabling DP Attention, `dp_size` will be calculated through `rollout-num-gpus/rollout-num-gpus-per-engine`. +> slime uses `vllm-router` to schedule multiple vLLM engines. `dp_size` is calculated through `rollout-num-gpus / rollout-num-gpus-per-engine`. ```bash -SGLANG_ARGS=( +VLLM_ARGS=( --rollout-num-gpus-per-engine 2 ) ``` @@ -332,7 +332,7 @@ ray job submit ... \ At this time, training and inference will share all 8 GPUs. > ⚠️ **Note**: -> In training-inference integration mode, Megatron will occupy a certain amount of GPU memory before it can be offloaded after initialization. You need to adjust the `--sglang-mem-fraction-static` parameter to reduce SGLang's GPU memory usage ratio to avoid insufficient GPU memory. We usually recommend 0.8. +> In training-inference integration mode, Megatron will occupy a certain amount of GPU memory before it can be offloaded after initialization. You need to adjust the `--vllm-gpu-memory-utilization` parameter to reduce vLLM's GPU memory usage ratio to avoid insufficient GPU memory. We usually recommend 0.8. ### Dynamic Sampling @@ -477,7 +477,7 @@ First, specify a custom asynchronous Python function through the `--custom-gener **Core Implementation Points**: 1. **Build Interaction Loop**: Create a loop to control maximum interaction rounds (such as `for _ in range(max_turns):`). -2. **Call Model to Generate Action**: In each round of the loop, call SGLang service to let the model generate the next action (such as `query`) based on the current conversation history. +2. **Call Model to Generate Action**: In each round of the loop, call the vLLM service to let the model generate the next action (such as `query`) based on the current conversation history. 3. **Parse and Execute Action**: Parse model output, identify actions and parameters, and call external tools or APIs (such as Google search). 4. **Build Observation Results**: Format the results returned by tools and append them to the conversation history as input for the next round. 5. **Handle Loss Masking**: This is the key to Agent training. @@ -495,7 +495,7 @@ async def generate(args, sample: Sample, sampling_params) -> Sample: for _ in range(max_turns): # 1. Model generates action - model_output = await call_sglang(prompt + full_response, ...) + model_output = await call_vllm(prompt + full_response, ...) # ... tokenization and appending ... loss_masks += [1] * len(model_tokens) # loss_mask = 1 full_response += model_output @@ -565,7 +565,7 @@ ray job submit --address="http://127.0.0.1:8265" \ } }' \ -- python3 train.py \ - --... # Other Megatron/SGLang/slime arguments + --... # Other Megatron/vLLM/slime arguments ``` Optionally, the following environment variables may be needed based on your environment. For example, when there are multiple IPs and the wrong one is chosen in a Docker or SLURM envionment. We provide an example used in a SLURM + enroot multi-node system as follows: diff --git a/docs/en/get_started/usage.md b/docs/en/get_started/usage.md index cbd012220..4afa17588 100644 --- a/docs/en/get_started/usage.md +++ b/docs/en/get_started/usage.md @@ -7,7 +7,7 @@ When using slime, parameters are primarily passed for the following purposes: 1. To allocate a portion of the GPUs in the cluster for training and another portion for inference. 2. To load Megatron for the training portion. -3. To load SGLang for the inference portion. +3. To load vLLM for the inference portion. 4. To configure the hyperparameters required for RL training. Following this order, we need to configure these parameters: @@ -19,8 +19,7 @@ There are four main parameters for cluster resource allocation: - `--actor-num-nodes`: The number of nodes required for RL actor training. - `--actor-num-gpus-per-node`: The number of GPUs per node for RL actor training. - `--rollout-num-gpus`: The total number of GPUs required for rollout (inference). - - `--rollout-num-gpus-per-engine`: The number of GPUs per inference engine. This parameter is similar to SGLang's `tp_size`. When performing multi-node serving, this value should be the total number of GPUs. For example, if serving one model with 2 nodes and 16 GPUs, this value should be 16. - The reason for not using a parameter like `--sglang-tp-size` is that we might consider supporting SGLang's `dp_size` parameter in the future, which means an engine could contain multiple SGLang servers (currently, only `--sglang-dp-size` under the `--sglang-enable-dp-attention` condition is supported). + - `--rollout-num-gpus-per-engine`: The number of GPUs per inference engine. This parameter is similar to vLLM's `tp_size`. When performing multi-node serving, this value should be the total number of GPUs. For example, if serving one model with 2 nodes and 16 GPUs, this value should be 16. With the default configuration, we use these parameters to allocate `actor_num_nodes * actor_num_gpus_per_node` GPUs for training and `rollout_num_gpus` GPUs for inference via Ray, thus achieving a separation of training and inference resources. @@ -38,7 +37,7 @@ slime supports multiple training backends, which can be selected via the `--trai ### Loading Megatron -Unlike tools such as SGLang, vLLM, or Hugging Face Trainer, Megatron cannot directly read Hugging Face checkpoints. Instead, the user must configure the parameters for the model to be trained and load Megatron's own checkpoint format. +Unlike tools such as vLLM or Hugging Face Trainer, Megatron cannot directly read Hugging Face checkpoints. Instead, the user must configure the parameters for the model to be trained and load Megatron's own checkpoint format. Generally, we need to perform three preparatory steps: @@ -135,20 +134,20 @@ Note: - Regardless of the checkpoint storage method (i.e., however `--ckpt-format` is set), Megatron can load both `torch` and `torch_dist` formats. -### Loading SGLang +### Loading vLLM -Loading SGLang is very simple. You only need: +Loading vLLM is very simple. You only need: - - `--hf-checkpoint`: The Hugging Face checkpoint used to initialize SGLang. + - `--hf-checkpoint`: The Hugging Face checkpoint used to initialize vLLM. Note: - - Before the first training step, slime will synchronize the parameters from Megatron to SGLang. Therefore, the `--hf-checkpoint` does not need to contain the latest training parameters, and you do not need to change the HF checkpoint when resuming training. - - By default, SGLang reads the maximum context length from the `config.json` in the Hugging Face checkpoint. You can use the `--sglang-context-length` parameter to override this value to support longer inference. - - During co-located training and inference, although Megatron and SGLang will offload sequentially, they still need to leave some memory for each other. You need to adjust SGLang's total VRAM usage by reducing `--sglang-mem-fraction-static`. - - slime supports passing through sgl-router parameters by adding a `router` prefix to the original parameter name. For example, sgl-router's `--balance-abs-threshold` parameter should be set as `--router-balance-abs-threshold`. Since sgl-router uses cache-aware routing by default, it may cause uneven request distribution. You can set `--router-balance-abs-threshold 0` to force balanced distribution, but this may affect prefix cache hit rate in multi-turn conversation scenarios. + - Before the first training step, slime will synchronize the parameters from Megatron to vLLM. Therefore, the `--hf-checkpoint` does not need to contain the latest training parameters, and you do not need to change the HF checkpoint when resuming training. + - By default, vLLM reads the maximum context length from the `config.json` in the Hugging Face checkpoint. You can use the `--vllm-max-model-len` parameter to override this value to support longer inference. + - During co-located training and inference, although Megatron and vLLM will offload sequentially, they still need to leave some memory for each other. You need to adjust vLLM's total VRAM usage by reducing `--vllm-gpu-memory-utilization`. + - slime supports passing through vllm-router parameters by adding a `router` prefix to the original parameter name. For example, vllm-router's `--balance-abs-threshold` parameter should be set as `--router-balance-abs-threshold`. Since vllm-router uses cache-aware routing by default, it may cause uneven request distribution. You can set `--router-balance-abs-threshold 0` to force balanced distribution, but this may affect prefix cache hit rate in multi-turn conversation scenarios. -For details on some of SGLang's customizations and the principles behind how slime incorporates SGLang, please see the "How to Use SGLang" section. +For details on some of vLLM's customizations and the principles behind how slime incorporates vLLM, please see the "How to Use vLLM" section. ### Data Format @@ -278,9 +277,9 @@ megatron: slime supports customizing data generation (rollout) to various degrees. - - By default, it uses the `generate_rollout` function from [slime/rollout/sglang_rollout.py](https://github.com/THUDM/slime/blob/main/slime/rollout/sglang_rollout.py) for data generation. This file implements an asynchronous (asyncio) data generation flow based on SGLang and supports features like dynamic sampling and partial rollout. + - By default, it uses the `generate_rollout` function from [slime/rollout/vllm_rollout.py](https://github.com/vllm-project/vime/blob/main/slime/rollout/vllm_rollout.py) for data generation. This file implements an asynchronous (asyncio) data generation flow based on vLLM and supports features like dynamic sampling and partial rollout. - - You can completely replace the `generate_rollout` in sglang\_example.py by using the `--rollout-function-path` parameter. You just need to ensure that the function signature passed via `--rollout-function-path` is as follows: + - You can completely replace the default `generate_rollout` by using the `--rollout-function-path` parameter. You just need to ensure that the function signature passed via `--rollout-function-path` is as follows: ```python def generate_rollout(args, rollout_id, data_source, evaluation=False) -> RolloutFnTrainOutput | RolloutFnEvalOutput: @@ -325,71 +324,73 @@ slime supports customizing data generation (rollout) to various degrees. TOKENIZER = AutoTokenizer.from_pretrained(args.hf_checkpoint, trust_remote_code=True) # send request to router + prompt_token_ids = TOKENIZER(sample.prompt, add_special_tokens=False)["input_ids"] output = await post( - f"http://{args.sglang_router_ip}:{args.sglang_router_port}/generate", + f"http://{args.vllm_router_ip}:{args.vllm_router_port}/inference/v1/generate", { - "text": sample.prompt, - "sampling_params": sampling_params, + "model": args.hf_checkpoint, + "token_ids": prompt_token_ids, + "sampling_params": {"max_tokens": sampling_params["max_new_tokens"]}, } ) - prompt_tokens_ids = TOKENIZER(sample.prompt, add_special_tokens=False)["input_ids"] - response_token_ids = TOKENIZER(output["text"], add_special_tokens=False)["input_ids"] + choice = output["choices"][0] + response_token_ids = list(choice.get("token_ids") or []) # set sample - sample.tokens = prompt_tokens_ids + response_token_ids + sample.tokens = prompt_token_ids + response_token_ids sample.response_length = len(response_token_ids) - finish_reason = output["meta_info"]["finish_reason"]["type"] + finish_reason = choice.get("finish_reason") or "stop" if finish_reason == "length": sample.status = Sample.Status.TRUNCATED - elif finish_reason == "abort": + elif finish_reason in ("abort", "cancelled"): sample.status = Sample.Status.ABORTED else: sample.status = Sample.Status.COMPLETED - sample.response = output["text"] + sample.response = TOKENIZER.decode(response_token_ids) if response_token_ids else "" return sample ``` - For a more complete version, please refer to [slime/rollout/sglang_rollout.py](https://github.com/THUDM/slime/blob/main/slime/rollout/sglang_rollout.py). + For a more complete version, please refer to [slime/rollout/vllm_rollout.py](https://github.com/vllm-project/vime/blob/main/slime/rollout/vllm_rollout.py). - Sometimes, you may also need to support a custom reward model. This can be configured by setting `--custom-rm-path`. -## How to Use SGLang +## How to Use vLLM -slime implements a server-based engine using SGLang via the `HttpServerEngineAdapter` as an intermediary. +slime runs vLLM in server mode and talks to it over HTTP. ### Parameter Configuration -slime incorporates almost all SGLang parameters by using SGLang's `ServerArgs.add_cli_args`. When setting an SGLang parameter, you need to add the `--sglang-` prefix. For example: +slime incorporates almost all vLLM parameters by forwarding vLLM's `EngineArgs` CLI flags. When setting a vLLM parameter, you need to add the `--vllm-` prefix. For example: - - In co-located training and inference, you often need to limit `--mem-fraction-static`. This parameter should be changed to `--sglang-mem-fraction-static`. - - During training, if you want SGLang to infer beyond the maximum context length specified in the Hugging Face checkpoint's `config.json`, you need to use `--context-length`, which becomes `--sglang-context-length` in slime. - - For multi-node large EP inference, you might need `--ep-size`, `--enable-dp-attention`, `--dp-size`, `--moe-a2a-backend deepep`, etc. These can be passed as `--sglang-ep-size`, `--sglang-enable-dp-attention`, `--sglang-dp-size`, and `--sglang-moe-a2a-backend deepep` respectively. + - In co-located training and inference, you often need to limit GPU memory utilization. Pass it as `--vllm-gpu-memory-utilization`. + - During training, if you want vLLM to infer beyond the maximum context length specified in the Hugging Face checkpoint's `config.json`, you need to use `--max-model-len`, which becomes `--vllm-max-model-len` in slime. + - For multi-node large EP inference, you might need `--enable-expert-parallel`, `--data-parallel-size`, etc. These can be passed as `--vllm-enable-expert-parallel` and `--vllm-data-parallel-size` respectively. Some parameters related to slime's resource scheduling are configured by slime itself, for example: - - `--tp-size` in slime is set using `--rollout-num-gpus-per-engine`. - - `--model-path` in slime is set using `--hf-checkpoint`. + - `--tensor-parallel-size` in slime is set using `--rollout-num-gpus-per-engine`. + - `--model` in slime is set using `--hf-checkpoint`. -The way SGLang parameters are integrated into slime can be found in [slime/backends/sglang_utils/arguments.py](https://github.com/THUDM/slime/blob/main/slime/backends/sglang_utils/arguments.py). +The way vLLM parameters are integrated into slime can be found in [slime/backends/vllm_utils/arguments.py](https://github.com/vllm-project/vime/blob/main/slime/backends/vllm_utils/arguments.py). ### How to Use the Router -slime uses [sglang-router](https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway) to manage the SGLang servers during the training process. You can configure the address of the [sglang-router](https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway) using `--sglang-router-ip` and `--sglang-router-port`. If not configured, a router will be started by default within the cluster. +slime uses [vllm-router](https://github.com/vllm-project/router) to manage the vLLM engines during the training process. You can configure the address of the router using `--vllm-router-ip` and `--vllm-router-port`. If not configured, a router will be started by default within the cluster. -After starting, all SGLang servers will register with the router via the `/add_worker` endpoint. When actually generating data, you only need to send HTTP requests to the router, which will perform load balancing and forward the requests to the servers. +After starting, all vLLM engines will register with the router. When actually generating data, you only need to send HTTP requests to the router, which will perform load balancing and forward the requests to the engines. -When you configure an external router using `--sglang-router-ip` and `--sglang-router-port`, slime will not start an internal router. Instead, it will register all its servers with this external router. You can then use this external router's address to implement more complex data generation workflows. Note that the router supports OpenAI-compatible APIs. +When you configure an external router using `--vllm-router-ip` and `--vllm-router-port`, slime will not start an internal router. Instead, it will register all its engines with this external router. You can then use this external router's address to implement more complex data generation workflows. Note that the router supports OpenAI-compatible APIs. -### Advanced Engine Configuration (--sglang-config) +### Advanced Engine Configuration (--vllm-config) -For advanced deployments, you can use `--sglang-config` with a YAML file to configure server groups, multi-model serving, and selective weight updates. +For advanced deployments, you can use `--vllm-config` with a YAML file to configure server groups, multi-model serving, and selective weight updates. **Multi-model deployment** allows serving multiple models simultaneously (e.g., an actor model that receives weight updates and a frozen reference/reward model): ```yaml -sglang: +vllm: - name: actor update_weights: true # receives weight updates from training (default) server_groups: @@ -405,11 +406,11 @@ sglang: num_gpus_per_engine: 2 ``` -Each model gets its own router. The per-model router info is accessible via `args.sglang_model_routers` (a dict mapping model name to `(ip, port)` tuples). Custom rollout functions can use `get_model_url(args, "ref")` from `slime.rollout.sglang_rollout` to route requests to a specific model. +Each model gets its own router. The per-model router info is accessible via `args.vllm_model_routers` (a dict mapping model name to `(ip, port)` tuples). Custom rollout functions can use `get_model_url(args, "ref")` from `slime.rollout.vllm_rollout` to route requests to a specific model. **Server group features:** - `worker_type`: `regular`, `prefill`, `decode`, or `placeholder` (reserves GPU slots without creating engines) -- `overrides`: Dict of SGLang `ServerArgs` field overrides applied on top of `--sglang-*` CLI args +- `overrides`: Dict of vLLM `EngineArgs` field overrides applied on top of `--vllm-*` CLI args - `num_gpus_per_engine`: Per-group TP size override ## How to Use Megatron diff --git a/docs/en/index.rst b/docs/en/index.rst index d5ca098e0..c74814e41 100644 --- a/docs/en/index.rst +++ b/docs/en/index.rst @@ -3,7 +3,7 @@ slime Documentation slime is an LLM post-training framework for RL scaling, providing two core capabilities: -- High-Performance Training: Supports efficient training in various modes by connecting Megatron with SGLang; +- High-Performance Training: Supports efficient training in various modes by connecting Megatron with vLLM; - Flexible Data Generation: Enables arbitrary training data generation workflows through custom data generation interfaces and server-based engines. slime is the RL-framework behind GLM-4.7, GLM-4.6 and GLM-4.5. Apart from models from Z.ai, we also supports the following models: @@ -47,7 +47,7 @@ slime is the RL-framework behind GLM-4.7, GLM-4.6 and GLM-4.5. Apart from models advanced/reproducibility.md advanced/fault-tolerance.md advanced/pd-disaggregation.md - advanced/sglang-config.md + advanced/vllm-config.md advanced/megatron-config.md advanced/arch-support-beyond-megatron.md @@ -70,15 +70,3 @@ slime is the RL-framework behind GLM-4.7, GLM-4.6 and GLM-4.5. Apart from models developer_guide/trace.md developer_guide/profiling.md -.. toctree:: - :maxdepth: 1 - :caption: Hardware Platforms - - platform_support/amd_tutorial.md - -.. toctree:: - :maxdepth: 1 - :caption: Blogs - - blogs/release_v0.1.0.md - blogs/introducing_slime.md diff --git a/docs/zh/advanced/fault-tolerance.md b/docs/zh/advanced/fault-tolerance.md index b3664f8c8..0dd79efd1 100644 --- a/docs/zh/advanced/fault-tolerance.md +++ b/docs/zh/advanced/fault-tolerance.md @@ -6,7 +6,7 @@ ## rollout 容灾 -slime 会在 rollout 过程中，定期向所有 SGLang server 发送心跳请求（`/health_generate`），如果心跳超时，则会停止这个 SGLang server。并在这轮 rollout 完成之后进行重启和正确的参数更新。 +slime 会在 rollout 过程中，定期向所有 vLLM 引擎发送心跳请求（`/health`），如果心跳超时，则会停止这个 vLLM 引擎。并在这轮 rollout 完成之后进行重启和正确的参数更新。 - `--rollout-health-check-first-wait`：由于一些大的 MoE 模型在第一次运行时需要处理一些编译，我们会在第一次 rollout 前等待 `rollout_health_check_first_wait` 秒再开始发送心跳，默认为 300s； - `--rollout-health-check-interval`：心跳检查间隔，默认为 10s； diff --git a/docs/zh/advanced/low-precision.md b/docs/zh/advanced/low-precision.md index e076661a1..27affe47f 100644 --- a/docs/zh/advanced/low-precision.md +++ b/docs/zh/advanced/low-precision.md @@ -55,7 +55,7 @@ bash scripts/low_precision/run-qwen3-30b-a3b-fp8.sh 1. **初始化**：如果启用了 FP8 方案，相关层将在 FP8 上下文中构建。 2. **训练过程**：在训练期间，权重和激活值会在线量化为 `nvfp8` 格式，并在前向和反向传播中调用 `cuBLAS FP8 GEMM` 进行计算。 -3. **权重更新**：在强化学习（RL）权重更新期间，Megatron 首先将 FP8 权重反量化为 BF16 格式，然后 slime 再将这些 BF16 权重重新量化为 FP8 并发送给 sglang。（这种“反量化+再量化”的操作虽然不够优雅，但为了框架兼容性，目前尚未修改接口。） +3. **权重更新**：在强化学习（RL）权重更新期间，Megatron 首先将 FP8 权重反量化为 BF16 格式，然后 slime 再将这些 BF16 权重重新量化为 FP8 并发送给 vLLM。（这种"反量化+再量化"的操作虽然不够优雅，但为了框架兼容性，目前尚未修改接口。） 4. **保存 ckpt**：与权重更新类似，从训练引擎保存 ckpt 时，也会反量化回 BF16 并以 `torch_dist` 格式保存。 ### 待办事项 (TODO) diff --git a/docs/zh/advanced/megatron-config.md b/docs/zh/advanced/megatron-config.md index 1616c9f83..02acaadab 100644 --- a/docs/zh/advanced/megatron-config.md +++ b/docs/zh/advanced/megatron-config.md @@ -2,7 +2,7 @@ `--megatron-config-path` 是一个基于 YAML 的配置系统，用于在公共 Megatron CLI 参数之上，为不同训练角色追加覆盖。目前它主要用于 PPO 场景中的 actor / critic 配置。 -与 `--sglang-config` 不同，`--megatron-config-path` 不负责部署、路由或 GPU 资源编排；它只负责决定“这个角色最终使用哪些训练参数”。 +与 `--vllm-config` 不同，`--megatron-config-path` 不负责部署、路由或 GPU 资源编排；它只负责决定“这个角色最终使用哪些训练参数”。 --- diff --git a/docs/zh/advanced/on-policy-distillation.md b/docs/zh/advanced/on-policy-distillation.md index 3f7dc124b..883601cc3 100644 --- a/docs/zh/advanced/on-policy-distillation.md +++ b/docs/zh/advanced/on-policy-distillation.md @@ -7,9 +7,9 @@ | 参数 | 说明 | |------|------| | `--use-opd` | 启用在策略蒸馏。使用 OPD 的必需标志。 | -| `--opd-type` | OPD 类型：`sglang` 或 `megatron`。启用 `--use-opd` 时必须设置。 | +| `--opd-type` | OPD 类型：`vllm` 或 `megatron`。启用 `--use-opd` 时必须设置。 | | `--opd-kl-coef` | OPD KL 惩罚系数（默认值：1.0）。控制蒸馏信号相对于 RL advantage 的权重。 | -| `--opd-teacher-load` | 教师模型的 Megatron checkpoint 路径。`--opd-type=megatron` 时**必须**设置，`--opd-type=sglang` 时**不可**设置。 | +| `--opd-teacher-load` | 教师模型的 Megatron checkpoint 路径。`--opd-type=megatron` 时**必须**设置，`--opd-type=vllm` 时**不可**设置。 | | `--opd-teacher-ckpt-step` | 可选的教师模型 checkpoint 步数。 | ## 原理 @@ -26,14 +26,14 @@ $$ ## 两种教师模式 -### SGLang 模式 (`--opd-type sglang`) +### vLLM 模式 (`--opd-type vllm`) -教师模型运行在外部 SGLang 服务器上，教师的 log-probs 在 rollout 阶段获取。 +教师模型运行在外部 vLLM 服务器上，教师的 log-probs 在 rollout 阶段获取。 **适用场景**：教师与学生架构不同，或教师模型太大无法与训练模型同时加载。 **工作流程**： -1. 外部 SGLang 服务器运行教师模型。 +1. 外部 vLLM 服务器运行教师模型。 2. 在 rollout 阶段，自定义 reward 函数（`slime.rollout.on_policy_distillation.reward_func`）将每个样本发送到教师服务器以获取 token 级 log-probs。 3. 自定义后处理函数（`slime.rollout.on_policy_distillation.post_process_rewards`）将教师 log-probs 裁剪到 response 范围并存储到 `sample.teacher_log_probs` 中。 4. 在训练阶段，从存储的教师 log-probs 计算 KL 惩罚并应用到 advantages 上。 @@ -41,11 +41,11 @@ $$ **配置**： ```bash --use-opd ---opd-type sglang +--opd-type vllm --opd-kl-coef 1.0 --custom-rm-path slime.rollout.on_policy_distillation.reward_func --custom-reward-post-process-path slime.rollout.on_policy_distillation.post_process_rewards ---rm-url http://:/generate +--rm-url http://:/inference/v1/generate ``` ### Megatron 模式 (`--opd-type megatron`) @@ -73,7 +73,7 @@ $$ 完整的示例脚本在 `examples/on_policy_distillation/` 中： -### SGLang 教师 +### vLLM 教师 ```bash # 1. 下载模型和数据 diff --git a/docs/zh/advanced/reproducibility.md b/docs/zh/advanced/reproducibility.md index 409d1294c..3502034f6 100644 --- a/docs/zh/advanced/reproducibility.md +++ b/docs/zh/advanced/reproducibility.md @@ -1,18 +1,19 @@ -# Reproducibility +# 可复现性 -Reproducibility is a bedrock of scientific progress. By combining the [deterministic inference](https://lmsys.org/blog/2025-09-22-sglang-deterministic/) of SGLang and the deterministic mode of Megatron-LM, slime supports bitwise experiment reproduction. +可复现性是科研进展的基础。slime 通过结合 vLLM 的 [batch-invariant 确定性推理](https://vllm.ai/blog/2025-11-10-bitwise-consistent-train-inference) 与 Megatron-LM 的 deterministic 模式，支持 bitwise 级的实验复现。 + +为了开启确定性训练，你需要通过 `pip uninstall flash_attn_3 -y` 卸载 flash attention 3，并设置： -To enable deterministic training, you need to first uninstall the flash attention 3 in the docker with `pip uninstall flash_attn_3 -y` and set: ```bash - # sglang config - --sglang-enable-deterministic-inference - --sglang-attention-backend flashinfer + # vLLM config + --vllm-enable-deterministic-inference + --vllm-attention-backend flashinfer # megatron config --deterministic-mode ``` -And set the following environment variables: +以及设置如下环境变量： ```bash "env_vars": { @@ -23,9 +24,9 @@ And set the following environment variables: } ``` -Here we provide the script to do RL training on Qwen2.5 0.5B model and GSM8K dataset with full deterministic. +我们提供了一个完全确定性的，用 Qwen2.5 0.5B 训练 GSM8K 的脚本。 -For data and checkpoint preparation, please run: +可以用如下脚本初始化训练数据和 ckpt： ```bash # download @@ -42,10 +43,10 @@ PYTHONPATH=/root/Megatron-LM/ python \ --save /root/Qwen2.5-0.5B-Instruct_torch_dist/ ``` -And to run training, +可以使用如下脚本进行训练： ```bash bash script/run-qwen2.5-0.5B-reproducibility.sh ``` -For screen shots of the wandb, please refer to [pull#370](https://github.com/THUDM/slime/pull/370). +这个 PR 中记录了 wandb 的截图 [pull#370](https://github.com/THUDM/slime/pull/370)。 diff --git a/docs/zh/advanced/speculative-decoding.md b/docs/zh/advanced/speculative-decoding.md index c2508be22..ad5723040 100644 --- a/docs/zh/advanced/speculative-decoding.md +++ b/docs/zh/advanced/speculative-decoding.md @@ -4,22 +4,28 @@ ## 使用投机采样加速推理 -对于有 MTP 层的模型（例如 GLM-4.6、Deepseek-V3/R1），只需要添加： +vLLM 把投机采样的所有配置收敛到一个 JSON（`SpeculativeConfig`），slime 通过 +`--vllm-speculative-config` 透传。对于有 MTP 层的模型（例如 GLM-4.7、DeepSeek-V3/R1），传入： ```bash ---sglang-speculative-algorithm EAGLE ---sglang-speculative-num-steps 3 ---sglang-speculative-eagle-topk 1 ---sglang-speculative-num-draft-tokens 4 +--vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3}' ``` -如果要使用单独训练的 draft model（例如 [SpecForge](https://docs.sglang.ai/SpecForge/) 训练的），还需要额外设置： +如果要使用单独训练的 draft model，在同一个 JSON 里加上 `model`（可选还可加 +`draft_tensor_parallel_size` 等）： ```bash ---sglang-speculative-draft-model-path /your/draft/model/path +--vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3,"model":"/your/draft/model/path"}' ``` -详细参数含义及配置方法，请参考 SGLang 的 speculative decoding [文档](https://docs.sglang.ai/advanced_features/speculative_decoding.html) +要从头训练一个 draft model，可以参考 [TorchSpec](https://github.com/lightseekorg/TorchSpec) +和 [vllm-project/speculators](https://github.com/vllm-project/speculators)。 +TorchSpec 提供 torch-native 的 disaggregated draft training。 +Speculators 支持 EAGLE-3、DFlash 以及 MTP 风格的 draft，HuggingFace 上已有预训练 ckpt +（参见 `RedHatAI/*-speculator.*` 集合），产物可被 `vllm serve ` 直接部署。 + +`SpeculativeConfig` 的完整字段（`disable_by_batch_size`、`acceptance_method`、 +draft TP 等）请参考 vLLM 的 speculative decoding [文档](https://docs.vllm.ai/en/latest/features/speculative_decoding/)。 ## 在线 SFT draft model diff --git a/docs/zh/advanced/sglang-config.md b/docs/zh/advanced/vllm-config.md similarity index 73% rename from docs/zh/advanced/sglang-config.md rename to docs/zh/advanced/vllm-config.md index e68c52c84..10bfb3fd5 100644 --- a/docs/zh/advanced/sglang-config.md +++ b/docs/zh/advanced/vllm-config.md @@ -1,33 +1,33 @@ -# SGLang Config：高级引擎部署 +# vLLM Config：高级引擎部署 -`--sglang-config` 是一个基于 YAML 的配置系统，用于在 slime 中精细控制 SGLang 引擎的部署。它支持**多模型服务**、**Prefill-Decode (PD) 分离**、**异构服务器组**，甚至可以作为复杂推理拓扑的**独立 SGLang 启动器**。 +`--vllm-config` 是一个基于 YAML 的配置系统，用于在 slime 中精细控制 vLLM 引擎的部署。它支持**多模型服务**、**Prefill-Decode (PD) 分离**、**异构服务器组**，甚至可以作为复杂推理拓扑的**独立 vLLM 启动器**。 --- ## 架构概览 -在默认配置（不使用 `--sglang-config`）下，slime 部署单个模型，放在单个 router 后面，使用统一的服务器组： +在默认配置（不使用 `--vllm-config`）下，slime 部署单个模型，放在单个 router 后面，使用统一的服务器组： -![架构概览](../../_static/image/arch.png) +![架构概览](../../../imgs/arch.png) -使用 `--sglang-config` 后，SGLang 部署扩展为多模型、多 router 拓扑： +使用 `--vllm-config` 后，vLLM 部署扩展为多模型、多 router 拓扑： -![sglang-config 架构](../../_static/image/sglang_config.png) +![vllm-config 架构](../../_static/image/vllm_config.png) **核心设计原则：** - **每个模型拥有独立的 router。** 模型在路由层隔离，支持独立的负载均衡和容错。 -- **同一模型内的服务器组可以异构。** 不同组可以有不同的 TP 大小、worker 类型（prefill/decode/regular）和 SGLang server 参数覆盖。 +- **同一模型内的服务器组可以异构。** 不同组可以有不同的 TP 大小、worker 类型（prefill/decode/regular）和 vLLM server 参数覆盖。 - **权重同步按模型维度。** 只有 `update_weights: true` 的模型会接收来自训练的权重更新。冻结的模型（reference、reward 等）保持原样。 --- ## 配置格式 -配置文件是一个 YAML 文档，顶层 `sglang` 键包含一个模型定义列表： +配置文件是一个 YAML 文档，顶层 `vllm` 键包含一个模型定义列表： ```yaml -sglang: +vllm: - name: # 必填。模型的唯一标识符。 model_path: # 可选。HF checkpoint 路径。默认使用 --hf-checkpoint。 update_weights: # 可选。是否从训练同步权重。自动推断。 @@ -36,7 +36,7 @@ sglang: - worker_type: # 必填。可选：regular、prefill、decode、placeholder。 num_gpus: # 必填。分配给该组的 GPU 总数。 num_gpus_per_engine: # 可选。该组的 TP 大小覆盖。 - overrides: # 可选。SGLang ServerArgs 字段覆盖。 + overrides: # 可选。vLLM EngineArgs 字段覆盖。 ``` ### 字段参考 @@ -45,7 +45,7 @@ sglang: | 字段 | 类型 | 默认值 | 说明 | |------|------|--------|------| -| `name` | `str` | **必填** | 模型唯一名称（如 `"actor"`、`"ref"`、`"reward"`）。用作 `args.sglang_model_routers` 的 key。 | +| `name` | `str` | **必填** | 模型唯一名称（如 `"actor"`、`"ref"`、`"reward"`）。用作 `args.vllm_model_routers` 的 key。 | | `model_path` | `str` | `args.hf_checkpoint` | HuggingFace checkpoint 路径。同一模型内的所有服务器组必须使用相同的 model path。 | | `update_weights` | `bool` | 自动推断 | 该模型是否接收训练权重更新。未设置时自动推断：如果 `model_path` 与 `--hf-checkpoint` 匹配则为 `true`，否则为 `false`。 | | `num_gpus_per_engine` | `int` | `args.rollout_num_gpus_per_engine` | 该模型服务器组的默认 TP 大小。各组可单独覆盖。 | @@ -58,13 +58,13 @@ sglang: | `worker_type` | `str` | **必填** | 引擎类型：`regular`（标准）、`prefill`（PD prefill worker）、`decode`（PD decode worker）或 `placeholder`（占位，不启动引擎）。 | | `num_gpus` | `int` | **必填** | 该组的 GPU 总数。必须 > 0。 | | `num_gpus_per_engine` | `int` | 模型的 `num_gpus_per_engine` | TP 大小覆盖。每个引擎实例的 GPU 数量。 | -| `overrides` | `dict` | `{}` | SGLang `ServerArgs` 字段覆盖。优先级最高，覆盖 `--sglang-*` CLI 参数和模型级默认值。 | +| `overrides` | `dict` | `{}` | vLLM `ServerArgs` 字段覆盖。优先级最高，覆盖 `--vllm-*` CLI 参数和模型级默认值。 | ### Worker 类型 | 类型 | 说明 | 使用场景 | |------|------|----------| -| `regular` | 标准 SGLang 引擎 | 默认模式，同时处理 prefill 和 decode | +| `regular` | 标准 vLLM 引擎 | 默认模式，同时处理 prefill 和 decode | | `prefill` | PD 分离的 prefill worker | 专门处理 prompt；与 `decode` worker 配对 | | `decode` | PD 分离的 decode worker | 专门生成 token；与 `prefill` worker 配对 | | `placeholder` | 占位，不创建引擎 | 为训练共置预留 GPU 或留作未来使用 | @@ -78,8 +78,8 @@ sglang: 最简单的配置，复现默认行为： ```yaml -# sglang_basic.yaml -sglang: +# vllm_basic.yaml +vllm: - name: default server_groups: - worker_type: regular @@ -88,7 +88,7 @@ sglang: ```bash python train.py \ - --sglang-config sglang_basic.yaml \ + --vllm-config vllm_basic.yaml \ --rollout-num-gpus 8 \ --rollout-num-gpus-per-engine 2 \ ... @@ -101,8 +101,8 @@ python train.py \ 将 prefill 和 decode 阶段分离到专用服务器组，以提升多轮和 agentic 场景的吞吐量： ```yaml -# sglang_pd.yaml -sglang: +# vllm_pd.yaml +vllm: - name: actor server_groups: - worker_type: prefill @@ -115,7 +115,7 @@ sglang: ```bash python train.py \ - --sglang-config sglang_pd.yaml \ + --vllm-config vllm_pd.yaml \ --rollout-num-gpus 16 \ ... ``` @@ -125,15 +125,15 @@ python train.py \ - 为 decode 使用更大的 TP（降低延迟） - 独立扩展 prefill 和 decode 的容量 -> **注意：** PD 分离使用 SGLang Model Gateway (sgl-router)，并设置 `pd_disaggregation=True`。 +> **注意：** PD 分离使用 vllm-router (vllm-router)，并设置 `pd_disaggregation=True`。 ### 3. 多模型服务 同时部署多个模型，每个模型拥有独立的 router： ```yaml -# sglang_multi_model.yaml -sglang: +# vllm_multi_model.yaml +vllm: - name: actor update_weights: true # 接收训练权重更新 server_groups: @@ -160,7 +160,7 @@ sglang: ```bash python train.py \ - --sglang-config sglang_multi_model.yaml \ + --vllm-config vllm_multi_model.yaml \ --rollout-num-gpus 16 \ --hf-checkpoint /path/to/actor_model \ --rollout-function-path my_rollout.generate_rollout \ @@ -170,7 +170,7 @@ python train.py \ **在自定义 rollout 函数中访问模型：** ```python -from slime.rollout.sglang_rollout import get_model_url +from slime.rollout.vllm_rollout import get_model_url from slime.utils.http_utils import post async def my_generate(args, sample, sampling_params): @@ -189,15 +189,15 @@ async def my_generate(args, sample, sampling_params): ... ``` -`get_model_url()` 从 `args.sglang_model_routers`（一个将模型名称映射到 `(ip, port)` 元组的字典）中读取，该字典在引擎启动后自动填充。 +`get_model_url()` 从 `args.vllm_model_routers`（一个将模型名称映射到 `(ip, port)` 元组的字典）中读取，该字典在引擎启动后自动填充。 ### 4. 多模型 + PD 分离 将多模型与 PD 分离结合，实现最大灵活性： ```yaml -# sglang_full.yaml -sglang: +# vllm_full.yaml +vllm: - name: actor update_weights: true server_groups: @@ -222,7 +222,7 @@ sglang: 使用 `placeholder` 组来预留 GPU 而不创建引擎。这在共置训练场景中很有用，部分 GPU 需要为训练预留： ```yaml -sglang: +vllm: - name: actor server_groups: - worker_type: regular @@ -234,10 +234,10 @@ sglang: ### 6. 按组覆盖 ServerArgs -使用 `overrides` 将 SGLang `ServerArgs` 字段应用到特定服务器组，而不影响其他组： +使用 `overrides` 将 vLLM `ServerArgs` 字段应用到特定服务器组，而不影响其他组： ```yaml -sglang: +vllm: - name: actor server_groups: - worker_type: regular @@ -250,23 +250,23 @@ sglang: enable_torch_compile: true ``` -覆盖具有**最高优先级**，会覆盖基础的 `--sglang-*` CLI 参数和模型级默认值。这对以下场景特别有用： +覆盖具有**最高优先级**，会覆盖基础的 `--vllm-*` CLI 参数和模型级默认值。这对以下场景特别有用： - 不同组使用不同的内存配置 - prefill 和 decode 使用不同的 context length - 在特定组上启用实验性功能 -### 7. 独立 SGLang 启动器 +### 7. 独立 vLLM 启动器 -虽然 `--sglang-config` 是为 slime 的训练流水线设计的，但它也可以作为纯推理场景的强大启动器，通过 `--rollout-external` 模式或配置 slime 仅关注推理服务。 +虽然 `--vllm-config` 是为 slime 的训练流水线设计的，但它也可以作为纯推理场景的强大启动器，通过 `--rollout-external` 模式或配置 slime 仅关注推理服务。 **使用预启动的外部引擎：** -对于复杂的生产部署，你可能希望独立预启动 SGLang 引擎，然后将其连接到 slime： +对于复杂的生产部署，你可能希望独立预启动 vLLM 引擎，然后将其连接到 slime： ```bash -# 步骤 1：外部启动 SGLang 引擎 -python -m sglang.launch_server --model-path /path/to/model --port 10090 ... -python -m sglang.launch_server --model-path /path/to/model --port 10091 ... +# 步骤 1：外部启动 vLLM 引擎 +vllm serve /path/to/model --port 10090 ... +vllm serve /path/to/model --port 10091 ... # 步骤 2：将 slime 连接到外部引擎 python train.py \ @@ -275,39 +275,39 @@ python train.py \ ... ``` -> **注意：** `--sglang-config` 和 `--rollout-external` 互斥。当你希望 slime 管理完整的引擎生命周期时，使用 `--sglang-config`；当引擎已预部署时，使用 `--rollout-external`。 +> **注意：** `--vllm-config` 和 `--rollout-external` 互斥。当你希望 slime 管理完整的引擎生命周期时，使用 `--vllm-config`；当引擎已预部署时，使用 `--rollout-external`。 --- ## Router 配置 -每个模型都有独立的 router（默认使用 SGLang Model Gateway）。 +每个模型都有独立的 router（默认使用 vllm-router）。 ### Router 策略 你可以配置路由策略： ```bash ---router-policy round_robin # 简单轮询 ---router-policy consistent_hashing # 多轮会话亲和 ---router-policy cache_aware # 缓存感知路由（默认） +--router-policy round_robin # 简单轮询 +--router-policy consistent_hash # 多轮会话亲和 +--router-policy cache_aware # 缓存感知路由（默认） ``` ### 多轮 Agent 的会话亲和路由 对于多轮对话和 agentic 场景，会话亲和确保同一对话的所有请求路由到同一个 backend worker。这可以显著提升 prefix cache 命中率，因为 worker 已经缓存了对话历史。 -slime 自动为每个 sample 分配一个唯一的 `session_id`（存储在 `sample.session_id` 中）。当 router 策略为 `consistent_hashing` 时，该 ID 通过 `X-SMG-Routing-Key` header 传递，SGLang Model Gateway 使用它将同一会话的所有轮次确定性地路由到同一个 worker。 +slime 自动为每个 sample 分配一个唯一的 `session_id`（存储在 `sample.session_id` 中）。当 router 策略为 `consistent_hash` 时，该 ID 通过 `x-session-id` header 传递，vllm-router 使用它将同一会话的所有轮次确定性地路由到同一个 worker。 ```bash ---router-policy consistent_hashing +--router-policy consistent_hash ``` **工作原理：** 1. 每个 sample 通过 UUID 分配唯一的 `session_id` -2. 每次请求时，slime 在 HTTP header 中传递 `X-SMG-Routing-Key: ` -3. SGLang Model Gateway 的 consistent hashing 策略将该 key 映射到特定的 worker +2. 每次请求时，slime 在 HTTP header 中传递 `x-session-id: ` +3. vllm-router 的 consistent-hash 策略将该 key 映射到特定的 worker 4. 后续轮次复用相同的 `session_id`，确保命中同一个 worker --- @@ -327,7 +327,7 @@ slime 自动为每个 sample 分配一个唯一的 `session_id`（存储在 `sam ## 互斥 -`--sglang-config` 与以下选项互斥： +`--vllm-config` 与以下选项互斥： | 选项 | 冲突原因 | |------|----------| @@ -340,10 +340,10 @@ slime 自动为每个 sample 分配一个唯一的 `session_id`（存储在 `sam 下面是一个完整的示例，展示在 32 个 GPU 上使用 PD 分离进行多模型 agentic RL 训练： -**配置文件 (`sglang_agent.yaml`)：** +**配置文件 (`vllm_agent.yaml`)：** ```yaml -sglang: +vllm: - name: actor update_weights: true server_groups: @@ -379,7 +379,7 @@ sglang: ```bash python train.py \ - --sglang-config sglang_agent.yaml \ + --vllm-config vllm_agent.yaml \ --hf-checkpoint /data/models/Qwen3-8B \ --rollout-num-gpus 32 \ --rollout-function-path my_agent.rollout.generate_rollout \ @@ -392,7 +392,7 @@ python train.py \ **自定义 rollout 函数 (`my_agent/rollout.py`)：** ```python -from slime.rollout.sglang_rollout import get_model_url +from slime.rollout.vllm_rollout import get_model_url from slime.utils.http_utils import post async def generate_with_models(args, sample, sampling_params): @@ -442,12 +442,12 @@ async def generate_with_models(args, sample, sampling_params): ### Q: 运行时如何获取特定模型的 router 地址？ -使用 `slime.rollout.sglang_rollout` 中的 `get_model_url(args, "model_name", "/endpoint")`。它从 `args.sglang_model_routers`（一个 `{ model_name: (ip, port) }` 字典）中读取，该字典在引擎启动后自动填充。 +使用 `slime.rollout.vllm_rollout` 中的 `get_model_url(args, "model_name", "/endpoint")`。它从 `args.vllm_model_routers`（一个 `{ model_name: (ip, port) }` 字典）中读取，该字典在引擎启动后自动填充。 -### Q: 可以不训练，只用 `--sglang-config` 做推理吗？ +### Q: 可以不训练，只用 `--vllm-config` 做推理吗？ -虽然 `--sglang-config` 是为 slime 的训练循环设计的，但你可以通过配置仅 rollout 的运行来实现纯推理场景。对于完全独立的 SGLang 推理服务，建议直接使用 SGLang 原生的 `launch_server`，或使用 `--rollout-external` 模式连接预部署的引擎。 +虽然 `--vllm-config` 是为 slime 的训练循环设计的，但你可以通过配置仅 rollout 的运行来实现纯推理场景。对于完全独立的 vLLM 推理服务，建议直接使用 vLLM 原生的 `launch_server`，或使用 `--rollout-external` 模式连接预部署的引擎。 -### Q: `--sglang-config` 和 `--prefill-num-servers` 是什么关系？ +### Q: `--vllm-config` 和 `--prefill-num-servers` 是什么关系？ -`--prefill-num-servers` 是启用 PD 分离的旧方式（它创建一个带有 prefill + decode 组的单模型）。`--sglang-config` 是更新、更灵活的方式。两者互斥。我们推荐所有新部署迁移到 `--sglang-config`。 +`--prefill-num-servers` 是启用 PD 分离的旧方式（它创建一个带有 prefill + decode 组的单模型）。`--vllm-config` 是更新、更灵活的方式。两者互斥。我们推荐所有新部署迁移到 `--vllm-config`。 diff --git a/docs/zh/blogs/introducing_slime.md b/docs/zh/blogs/introducing_slime.md deleted file mode 100644 index 0e54a97be..000000000 --- a/docs/zh/blogs/introducing_slime.md +++ /dev/null @@ -1,119 +0,0 @@ -# slime：为 RL Scaling 设计的 SGLang-Native 后训练框架 - -> 本文由英文版翻译而来，首发于 [lmsys.org](https://lmsys.org/blog/2025-07-09-slime/)。 - -## 愿景 - -我们相信强化学习。我们相信强化学习是通往 AGI 的最后一块拼图。 - -所以我们相信： - - - 应该在每个领域上都尝试端到端强化学习，把所有任务都转变为 agent 环境。 - - 强化学习训练运行都应持续更久，每个模型都应扩展得更大。 - - 强化学习系统应与现有基础设施无缝集成，让我们能够专注于新的想法，而不是琐碎的杂活。 - -所以我们实现了 [slime](https://github.com/THUDM/slime)，它是一个专为后训练设计的框架，旨在实现以下特性： - - - **多功能性**——具有完全可定制的推理接口和灵活的训练设置（同地或解耦，同步或异步，强化学习或 SFT 冷启动）。 - - **高性能**——原生集成了 SGLang 进行推理，以及 Megatron-LM 进行训练。 - - **可维护性**——代码库轻量，并能从 Megatron 预训练平滑过渡到 SGLang 部署。 - -简而言之，它是一个为强化学习扩展而生的后训练框架。 - -以下是我们如何实现这一切的。 - -## 自定义才能自由 - -> 我们应该停止尝试用简单的方式来思考心智的内容，例如简单地思考空间、物体、多智能体或对称性。 -> -> — *The Bitter Lesson* - -强化学习社区中一个普遍的误解是，不同任务需要不同的框架：一个用于纯粹的数学，一个用于多轮工具调用，一个用于异步训练，一个用于智能体任务，等等。维护和分叉多个框架令人沮丧，这导致了浪费时间的 bug 修复挑选，甚至更糟的是，因为遗漏补丁而导致的训练崩溃。 - -事情并非总是如此：没有人会为了一个新的数据加载器而分叉 PyTorch。我们认为目前的混乱源于一种陷阱，即规定人们应该如何构建他们的应用。如果我们坚持为每种推理场景定义一个通用模板，我们最终只会创建一个只满足一小部分实际需求的强化学习框架。 - -slime 以不同的方式看待强化学习中的数据采样。我们在 slime 内部通过 [sgl-router](https://github.com/sgl-project/sglang/tree/main/sgl-router) 管理所有 SGLang 服务器，并为数据生成组件提供一个接口，**允许用户注入自定义逻辑并自由地与 SGLang 服务器交互**。这能释放他们的创造力。 - -使用 sgl-router，用户只需向一个单一端点发送 HTTP 请求。通过暴露这个端点，复杂的智能体环境可以直接通过一个与 OpenAI 兼容的 API 与 slime 交互——无需修改环境，并且训练与部署的一致性也得到了保留。 - -在训练方案方面，slime 使用 Ray 进行资源管理，通过一个简单的标志 (`--colocate`)，即可启用**同地**（相同 GPU）或**解耦**（不同 GPU）的设置。 - -凭借 Ray 通过 `.remote()` 实现的异步执行，slime 自然支持异步训练。改变同步行为就像移动 `ray.get` 操作一样简单。为了便于尝试不同的策略，我们没有将代码封装在训练器类中，而是简单地将训练循环暴露在入口文件 `train.py` 中。 - -## 为性能而生 - -**一个合格的强化学习框架必须既快，又持续地快。** - -**快**意味着要利用最快的推理和训练框架。 - -与预训练不同，强化学习工作负载在训练过程中涉及大量的在线采样，这使得推理性能至关重要。因此，slime 专门集成了 SGLang，并刻意提供了 SGLang 原生体验。 - -那么，“SGLang 原生”意味着什么？这意味着你可以充分利用所有 SGLang 的优化——在 slime 内部使用 SGLang 就像单独使用它一样。为了实现这一点： - - - slime 在内部以**服务器模式**启动 SGLang 服务器。 - - slime 对所有 SGLang 参数实现了**无缝传递**（带有 `--sglang` 前缀），确保所有优化选项都可以启用。例如，你可以传递 `--sglang-enable-ep-moe`、`--sglang-enable-dp-attention` 和 `--sglang-enable-deepep-moe`，以实现强大的多节点 MoE 推理功能。 - - slime 提供了一个**仅限 SGLang 的调试模式** (`--debug-rollout-only`)，以便轻松进行性能调优。 - -通过这些，我们可以在 slime 内部重现 SGLang 的独立性能。甚至 slime 的基础镜像也是基于 `lmsysorg/sglang:dev` 构建的。 - -对于训练，slime 集成了久经考验的 Megatron-LM，旨在提供同样原生的预训练体验： - - - slime 也对所有 Megatron 参数实现了**无缝传递**。 - - slime 支持**所有 Megatron 并行策略**（TP, PP, EP, CP），并监控训练 MFU。 - - slime 提供了**仅限 Megatron 的调试模式** (`--debug-train-only`)，并支持存储采样数据以供重现。 - -Megatron 可能非常复杂，因此我们还提供了 ckpt 转换工具来简化其使用。 - -**持续地快**意味着要跟上不断发展的推理和训练框架。 - -如果你曾关注 [SGLang 的 PR 列表](https://github.com/sgl-project/sglang/pulls)，你会被其快速的演进所震惊。另一方面，Megatron 通常被深度定制，每个组织都维护着自己的分叉。slime 旨在跟上游 SGLang 的变化，并适应内部 Megatron 变体中的优化。这也是我们追求对 SGLang 和 Megatron 原生支持的另一个原因。参数传递使得升级毫不费力。 - -除了优化推理和训练框架，我们还处理了强化学习特有的工作负载。当 SGLang 需要修改以支持这些工作流时，我们与 SGLang 团队紧密合作，将补丁合并到上游——这样即使强化学习逻辑演变，slime 也能保持原生。例如： - -**优化权重更新**：与推理任务不同，强化学习训练涉及频繁的模型权重更新。为了解决这个问题，我们在 SGLang 中引入了几项优化： - - - 在各种并行策略下对 MoE 模型进行参数更新（[\#6265](https://github.com/sgl-project/sglang/pull/6265)、[\#6308](https://github.com/sgl-project/sglang/pull/6308)、[\#6311](https://github.com/sgl-project/sglang/pull/6311)）。 - - 支持桶式参数更新以减少开销（[\#7292](https://github.com/sgl-project/sglang/pull/7292)）。 - -**用于动态采样的 `/abort_request`**：在需要过采样的强化学习算法中，例如 [DAPO](https://arxiv.org/abs/2503.14476)，即使已收集到足够的数据，某些请求可能仍会继续运行。我们与 [AReal](https://github.com/inclusionAI/AReaL) 团队合作，设计了一个新的端点：`/abort_request`。这个端点能够： - - - 立即终止正在进行的请求。 - - 重新获取部分生成的内容，从而实现部分推理。 - -这些功能在 [\#6698](https://github.com/sgl-project/sglang/pull/6698)、[\#6855](https://github.com/sgl-project/sglang/pull/6855)、[\#6184](https://github.com/sgl-project/sglang/pull/6184)、[\#5966](https://github.com/sgl-project/sglang/pull/5966) 中实现。 - -## 轻量且可扩展 - -slime 专注于可定制性和性能： - -1. 提供了一个可定制的推理接口。 -2. 使用 Ray 进行 GPU 管理和异步执行。 -3. 集成 SGLang 用于推理，Megatron 用于训练。 -4. 提供训练和推理之间的权重更新。 - -很简单，对吧？slime 将复杂性从框架转移到用户定义的管道和核心库（SGLang 和 Megatron），从而形成一个轻量、易于维护的代码库。 - -但它并不仅限于强化学习。 - -由于其模块化设计和强大的后端，slime 可以通过最少的额外代码自然地扩展到其他后训练工作流： - - - **SFT**：加载 Megatron 并使用 token 预测损失。 - - **Rejection Sampling**：使用 SGLang 进行过滤，然后使用 Megatron SFT。 - -*（请注意，SFT 功能目前处于实验阶段。）* - -除此之外，slime 的原生集成**无缝连接了预训练到在线服务**。我们可以使用 Megatron 进行预训练，切换到 slime（它集成了 Megatron 和 SGLang）进行后训练，最后直接使用 SGLang 进行评估和部署。这消除了转换检查点格式和对齐框架之间精度的繁琐且易出错的步骤。 - -统一的管道将我们从繁琐的“胶水代码”中解放出来，让我们能够专注于真正重要的事情：更好的强化学习。太棒了！ - -## 发展蓝图 - -强化学习扩展的旅程才刚刚开始，slime 也在不断演进。在下一阶段，我们将专注于： - -1. 与 SGLang 团队合作，探索大规模 MoE 模型的最佳强化学习训练策略。 -2. 支持更广泛的后训练工作流，加强从预训练到生产的桥梁。 -3. 添加对原生 PyTorch 训练后端的支持，以降低入门门槛。 - -我们希望 slime 能加速你的强化学习扩展之旅，并将你的创新想法变为现实。欢迎随时提出贡献和进行交流！ - -特别感谢 AMD GenAI - Foundation Model Team 在第一天就提供了 AMD 硬件支持。 diff --git a/docs/zh/blogs/release_v0.1.0.md b/docs/zh/blogs/release_v0.1.0.md deleted file mode 100644 index ff95a7e3c..000000000 --- a/docs/zh/blogs/release_v0.1.0.md +++ /dev/null @@ -1,146 +0,0 @@ -# v0.1.0: 重新定义高性能 RL 训练框架 - -> 本文首发于[知乎](https://zhuanlan.zhihu.com/p/1945237948166547268). - -在社区的帮助下，我们在开源 2 个月后终于为 **slime** 打上了第一个版本号：**v0.1.0**。 - -用一句话来概括这个版本 - -> **slime v0.1.0 提供了大规模 MoE RL 训练所需的所有基本性能优化。** - -具体来说，这个版本带来了以下改进： - -- **性能**： - - 提供了 **MoE 模型的高效推理**，特别是 fp8 rollout + deepep + mtp。 - - 设计了通用的**训练框架显存 offload 方案**，节省出更多 KV Cache 空间，从而提升推理并发度。 - - **更快的参数更新**。 - - 通过 **CPU Adam** 实现用更少的 GPU 也能进行更多训练。 - - 支持了 **Megatron 的全部并行策略**以及 deepep。 -- **功能**： - - 针对 MoE 模型训练支持了 **GSPO**。 - - 针对 fp8 rollout 支持了 **TIS**。 -- **正确性**： - - 加入了 **Dense 与 MoE 模型 CI**，严格检查 kl 等指标。 - -我们希望通过 slime v0.1.0 展示我们对高性能 RL 训练框架的理解，并有机会成为未来性能对比的基准（baseline）。 - -接下来，我将展开介绍上述功能背后的设计思路。 - ------- - -## 性能优化：提升 RL 训练速度上限 - -在传统的深度学习训练中，有一个万能的提速方案：**加卡**。通过降低每张卡上的计算数据，可以显著降低端到端的训练延迟。 - -然而，这个方法在 RL 训练中却行不通，因为**推理的延迟无法通过增加 GPU 来降低**。即使我们有再多的 GPU，也只能等待最长的样本解码（decode）结束。虽然可以通过增加吞吐量来提升每个 rollout 的训练数据量，但过大的推理 batch size 带来的 off-policy 问题目前来看仍有一些局限性。 - -我认为这是当前 RL 范式下对基础设施（Infra）的最大挑战，即： - -> **我们希望 scale inference compute，但是我们无法 scale inference latency。** - -单条数据的解码速度决定了 RL 训练速度的上限。对于较大的 MoE 模型，目前主要有以下 3 种常规优化方案来提升这一上限，我们也在每个方向上都做了尝试： - -1. **通过量化来降低访存**：考虑到 RL 训练中不能进行长时间的 calibration，slime 选择进行 fp8 量化。 -2. **使用 deepep low latency 模式来降低跨机 all2all 的时延**：为了配合 deepep，slime 推荐使用 fp8 的 blockwise 量化来开启 SGLang 的相关配置。 -3. **开启投机采样（Speculative Sampling）**：slime 允许推理部分加载任意的 draft model（目前还不支持训练中更新 draft model）。 - -![](../../_static/image/blogs/release_v0.1.0/overrall.png) - -使用上述 3 种优化，我们可以将 **GLM4.5 355B-A32B** 这样的模型，从单条数据小于 10 token/s，提升至 **60~70 token/s**，从而极大提升 RL 训练速度的上限。 - -slime 也会在监控推理的吞吐之外，监控 `perf/longest_sample_tokens_per_sec`，从而更好地掌握推理部分的性能优化空间。 - -## 用更少的卡做更多的实验：充分 Offload Megatron - -在对上限进行优化后，我们注意到 RL 训练的另外一个特性：只要 **KV Cache 不溢出**，推理 batch size 的提升并不会明显影响训练的延迟。 - -**KV Cache 溢出**指的是推理过程中，当数据的回复长度都很长时，KV Cache 空间不足，需要将某些生成到一半的数据先踢出队列，等其他数据推理完并腾出空间后，再重新进行 prefill 和后续的推理。如果一条回复长度为 64k 的数据在推理过程中等待了其他数据解码 32k token，相当于它的总时长对应了解码 96k token，这对 RL 训练速度影响很大。 - -因此，一个比较合适的训练配置是根据推理部分的 batch size、数据的平均回复长度和单个 server 能预留的 KV Cache 空间，计算一个在 KV Cache 不溢出情况下的最少 GPU 数量，并以这些 GPU 为一组进行训练。例如，我们有 512 张卡，计算出来 256 卡的 KV Cache 就已经充足，那么就应该并行运行 2 个实验，而不是用 512 卡一起启动实验。 - -基于这样的考量，我们注意到了 2 个优化点： - -1. **最佳卡数可能不足以支持加载训练部分**。推理只需要加载 fp8 参数就够了，而训练一般需要 18 倍参数量以上的显存（bf16 param、fp32 grad、fp32 master param、fp32 m 和 v）。为了解决这个问题，slime 选择开启 **Megatron 自带的 CPU Adam** 来节省训练部分的显存。我们也是基于这样的策略提供了 8 节点训练 GLM 4.5 355B-A32B 以及 16 节点训练 DeepSeek R1 的方案。 -2. **提升每个 SGLang Server 能预留的 KV Cache**，也就是开大 `--mem-fraction`。对于现在更为常见的训推一体的训练任务，限制 `mem_fraction` 的主要是将训练部分 offload 至 CPU 后的残留显存。因此，我们需要找到一个通用的方法，将 Megatron 部分占用的显存 offload 到 GPU。 - -### 如何通用地 Offload GPU Tensor - -一个比较粗暴的方式是找到 Megatron 分配的所有 GPU Tensor，然后把它们全部 `.to("cpu")`。这种做法有 3 个难点： - -- 很难捕获 Megatron 分配的全部 GPU Tensor。 -- 由于 Megatron 的 distributed optimizer 会把所有参数重新整理到一些连续的 GPU Buffer 里，然后再通过各种 slice 划分出去，很难处理好所有的引用从而正确释放 GPU Tensor。 -- Megatron 每次版本更新都要重新查一遍，不太好维护。 - -有没有一个更通用的方案呢？ - -我们注意到 SGLang 中的 `torch_memory_saver` 和 VLLM 中的 `cumem_allocator` 提供了一种较为通用的 offload 方案。它们的原理大致是，CUDA 10.2 提供了**一系列 Virtual Memory Management API**，类似于操作系统的虚拟地址与物理地址（VA 和 PA）。在分配显存时会返回一个显存映射的句柄（handle），而不是实际的物理地址。因此，我们在 offload 时只需要偷偷释放这个映射对应的显存，然后在需要这段显存时重新分配就可以了，上层的应用无需感知。 - -![](../../_static/image/blogs/release_v0.1.0/cuda_vmm.png) - -一个自然的想法就是用这个方式接管 RL 中训练部分的全流程。但是，这会导致无法复用 PyTorch 的 `CUDACachingAllocator`，没有缓存会导致显存碎片更明显，训练过程很容易 **OOM**。 - -为了能继续复用原生的带缓存的 allocator，我们不能使用 `CUDAPluggableAllocator` 了。再次注意到 slime 的架构中训练和推理是在不同进程的，所以我们只需要通过 `LD_PRELOAD` **直接替换训练进程中 `CUDACachingAllocator` 使用的 `cudaMalloc` 和 `cudaFree` 为 VMM API**。这样我们就可以完整且通用地 offload PyTorch 分配的所有 GPU Tensor。 - -同时，我们还要注意一个细节，那就是 VMM API 与 `cudaIPC API`（例如 `cudaIpcGetMemHandle`）是不匹配的。因此，对于训推一体的参数更新，或是 DeepEP，我们需要关闭 `LD_PRELOAD` 的替换，用回 `cudaMalloc`。 - -在 SGLang 社区的帮助下，我们针对 slime 的需求更新了 `torch_memory_saver`，实现了这个 offload 方案。 - -### 如何 Offload NCCL - -在彻底 offload 了 Megatron 中的 GPU Tensor 后，我们发现还会有大量的显存残留，这是 **NCCL** 导致的。在 PyTorch 中，每个参与通信的 NCCL group 都会分配一份不小的 buffer。对于较大的 MoE 模型，由于引入了各种并行策略，这个问题尤为明显，可能会占到 **10GB 以上**。 - -上述 `LD_PRELOAD` 的方案不太好处理 NCCL 的问题，我们也不想去修改 NCCL 源码，以免在维护 slime 之余还要维护一个 NCCL fork。所以 slime 采用的方案是在 offload Megatron 的时候利用 `destroy_process_group` 来销毁 NCCL group，然后在 load Megatron 之前重建。为此，我们模仿 VMM API，对 `dist.new_group` 进行了 `monkey patch`，增加了一层 `ReloadableProcessGroup`。 - -这样，我们就实现了通用的 **NCCL offload**。不过，因为我们需要重建 NCCL group，这样的操作会对每轮训练的第一次通信速度有一定影响。但我们认为从可维护性以及节省的显存上，这个方案有很大的优势。 - -结合上述两项优化，我们将 Megatron 的残留显存从约 **15~18GB** 降低到了 **3~5GB**，从而将 MoE 模型的 `mem_fraction` 提升至 **0.7~0.8**，明显提升了预留的 KV Cache，提升了每个 server 能支持的并发度，实现了用更少的 GPU 启动更多的训练任务。 - -## 参数更新优化 - -参数更新是另外一个 RL 训练中特殊的环节。在这方面，slime v0.1.0 提供了训练推理在不同进程条件下的最佳优化方案。这部分工作由 Biao He 老师进行了大量优化，推荐阅读他的这篇博文： - -- [高效强化学习训练 - 优化 slime 中的权重同步](https://hebiao064.github.io/rl-weight-sync) - -目前 slime 可以做到 **48s** 完成训推一体下 GLM4.5 355B-A32B 模型 bf16 权重的参数同步，以及 **100s** 完成 fp8 blockwise 量化 + 参数更新（fp8 分支还在优化中）。 - -## 训练优化 - -对于 slime 的纯训练部分，我们认为 Megatron 已经提供了充足的优化，所以我们主要是**保证了对 Megatron 全部并行策略的适配**。 - -适配过程中有一个有趣的 bugfix：我们发现在 SGLang 开启 mtp 后，Megatron 部分无法启动 DeepEP。后来发现是因为在开启 mtp 时，SGLang 会 disable overlap schedule，导致在被 offload 到 CPU 后仍用 nccl 而非 gloo 进行某个 metadata 的通信，并与 DeepEP 产生了冲突。 - -## 性能优化 Check List - -slime 发布以来，我经常被问到它与其他框架的性能对比。 - -我对基准测试（benchmark）的理解是，benchmark 不应该成为框架之间相互攻击的武器，而应该成为**查缺补漏的工具**。为此，我们会逐渐推出 slime 关注的性能 benchmark，用于自我提升。 - -同时，我也认为在跑分之前，可以从很多定性的角度来分析一个框架对性能的重视程度。这里提供一个基础优化的 feature check list： - -- 是否支持 MoE 的训练？（目前大规模实验集中于 MoE） -- 是否支持内部的 sglang `mem_fraction` 或 vllm `gpu_utilization` 能调至 0.7 以上？（保证 KV Cache 空间） -- 是否支持 fp8 或更低精度量化的推理？（降低推理访存，提升速度） -- 是否支持训练和推理均开启 deepep？（优化 MoE all2all 通信） -- 是否支持投机采样？（提升推理时延与吞吐） -- 是否有高效的训练 backend，如 Megatron、torchtitan，并支持各项必要的并行？（复用成熟的训练优化） - -slime v0.1.0 对上述的所有优化都进行了初步的尝试，当然提升空间依然很大。我们也希望这个版本能成为未来 slime 版本或者不同框架之间性能对比的基准。我们更欢迎所有在性能上和我们有相同追求的朋友来试用 slime，参与 slime 社区！ - -## 新算法支持 - -为了更好地训练 MoE 模型以及进行 fp8 rollout，我们实现了 **GSPO** 与 **TIS**。同时，社区的大佬也帮忙实现了例如 reinforce++、reinforce++ baseline 这样的算法。 - -## 正确性验证 - -slime v0.1.0 增加了**端到端 CI**：我们会对每个 PR 运行单机的 GLM4 9B 和 Qwen3 30B-A3B 训练，通过一些严格的检查来保证更新的正确性。例如，我们会明确要求： - -- 第一个 rollout 的重算 log prob 和 reference model 的 log prob 完全相等。 -- 每个 rollout 内的第一个训练步的 ppo_kl 严格为 0。 - -这样的精确验证是训练框架中很少能做到的，也是我们非常自豪的一点。 - ------- - -以上大致是 slime v0.1.0 的一个简单介绍，希望能让大家对 slime 产生一丝好奇，也希望 slime 能对大家的工作有所帮助。 - -欢迎大家都来参与 slime 社区，让我们共同建设开放的 RL Infra，一起为 RL scaling 添砖加瓦！ \ No newline at end of file diff --git a/docs/zh/developer_guide/ci.md b/docs/zh/developer_guide/ci.md index b8830ecc7..f6378e01a 100644 --- a/docs/zh/developer_guide/ci.md +++ b/docs/zh/developer_guide/ci.md @@ -6,7 +6,7 @@ slime 使用 GitHub Actions 进行 CI。测试通过 **PR label** 触发—— 工作流定义在 `.github/workflows/pr-test.yml`（由 `pr-test.yml.j2` 自动生成）。每个 CI 任务会： -1. 在自托管 GPU runner 上通过 `docker run` 运行；大多数测试使用 `slimerl/slime:latest`，镜像验证使用 `slimerl/slime-test:latest`。 +1. 在自托管 GPU runner 上通过 `docker run` 运行；大多数测试使用 `inferactinc/public:vime-vllm-cu129-latest`，镜像验证使用 `inferactinc/public:vime-vllm-cu129-latest`。 2. 通过 `pip install -e . --no-deps` 安装 slime。 3. 通过 `tests/ci/gpu_lock_exec.py --count ` 获取所需数量的 GPU。 4. 执行测试文件：`python .py` 或 `python tests/.py`。如果测试位于 `tests/plugin_contracts/` 这样的子目录，CI 也会自动处理。 @@ -23,7 +23,7 @@ slime 使用 GitHub Actions 进行 CI。测试通过 **PR label** 触发—— | `run-ci-megatron` | `e2e-test-megatron` | 核心 Megatron 训练测试，覆盖 Dense、MoE、PPO、MTP、OPD 等。 | | `run-ci-precision` | `e2e-test-precision` | 数值精度校验（并行一致性检查）。 | | `run-ci-ckpt` | `e2e-test-ckpt` | Checkpoint 保存/加载正确性（同步和异步保存）。 | -| `run-ci-image` | `e2e-test-image` | 在 `slimerl/slime-test:latest` 镜像上运行**全部**测试（用于镜像验证）。 | +| `run-ci-image` | `e2e-test-image` | 在 `inferactinc/public:vime-vllm-cu129-latest` 镜像上运行**全部**测试（用于镜像验证）。 | | `run-ci-changed` | `e2e-test-changed` | **动态**检测 PR 中新增或修改的测试文件，仅运行这些测试。 | 所有 label 也可通过 `workflow_dispatch`（在 Actions 页面手动触发）来运行。 @@ -40,11 +40,11 @@ slime 使用 GitHub Actions 进行 CI。测试通过 **PR label** 触发—— 这意味着你不需要手动在 workflow 中注册新测试——只需确保测试文件顶部有 `NUM_GPUS = ` 常量，`run-ci-changed` 就会自动识别并运行。 -**示例**：如果你的 PR 新增了 `tests/test_qwen3_8B_opd_sglang.py`（其中 `NUM_GPUS = 8`），添加 `run-ci-changed` label 后会自动在 8 张 GPU 上运行该测试。 +**示例**：如果你的 PR 新增了 `tests/test_qwen3_8B_opd_vllm.py`（其中 `NUM_GPUS = 8`），添加 `run-ci-changed` label 后会自动在 8 张 GPU 上运行该测试。 ### `run-ci-image` — 在测试镜像上运行全部测试 -这会在 `slimerl/slime-test:latest` Docker 镜像上运行**所有**已注册的测试。适用于： +这会在 `inferactinc/public:vime-vllm-cu129-latest` Docker 镜像上运行**所有**已注册的测试。适用于： - 验证新构建的 Docker 镜像是否可用。 - 在合并前做全面的测试检查。 @@ -57,7 +57,7 @@ slime 使用 GitHub Actions 进行 CI。测试通过 **PR label** 触发—— - Dense 模型：GLM4-9B、Qwen3-4B（PPO） - MoE 模型：Qwen3-30B-A3B（DeepEP + FP8）、Qwen3.6-35B-A3B PD + Mooncake、Moonlight-16B-A3B -- 特殊场景：MiMo-7B MTP、Qwen2.5-0.5B debug rollout-then-train、OPD（sglang teacher 模式） +- 特殊场景：MiMo-7B MTP、Qwen2.5-0.5B debug rollout-then-train、OPD（vLLM teacher 模式） 所有测试使用 8 张 GPU。如果你正在修改 Megatron 训练逻辑、loss 计算或 checkpoint 转换，应该使用这个 label。 diff --git a/docs/zh/developer_guide/debug.md b/docs/zh/developer_guide/debug.md index d964a4e82..d21e95087 100644 --- a/docs/zh/developer_guide/debug.md +++ b/docs/zh/developer_guide/debug.md @@ -7,8 +7,8 @@ 1. 训练第一步 1. rollout 的生成是否是人话，如果不是，有以下 2 种可能： - 参数没有正常加载。需要查看是否有 megatron 成功加载 ckpt 的日志； - - 更新参数有误。可以查看是不是所有的参数都做了转换和参数对应，或者参数名是不是根据并行做了转换（例如 pp_size > 1 时，第二个 stage 提供的参数的 layer id 是不是正确的）。一个比较彻底的方法是在对应模型的 sglang 实现的 `load_weights` 中保存所有的参数，查看和加载的 ckpt 中是否一致； - - 如果所有参数更新都正确，还出现问题，有可能是 sglang 里有一些特殊的 buffer 在 release 的时候被释放了； + - 更新参数有误。可以查看是不是所有的参数都做了转换和参数对应，或者参数名是不是根据并行做了转换（例如 pp_size > 1 时，第二个 stage 提供的参数的 layer id 是不是正确的）。一个比较彻底的方法是在对应模型的 vllm 实现的 `load_weights` 中保存所有的参数，查看和加载的 ckpt 中是否一致； + - 如果所有参数更新都正确，还出现问题，有可能是 vllm 里有一些特殊的 buffer 在 release 的时候被释放了； - 如果是用 pretrain 模型进行的测试，可以换成同结构模型的 instruct 版本，查看这种乱码是不是 pretrain 模型特有的。 2. 查看打印的 rollout stats 的 `log_probs` 和 `ref_log_probs` 是否完全相等（即第一步 kl=0），且值较小 - 如果不是完全相等的，一般是 transformer engine 中的某些 non-deterministic kernel 导致的，例如： @@ -34,11 +34,11 @@ slime 支持将训练部分和推理部分分开进行调试，从而实现： 1. `--debug-rollout-only` - 开启后，slime 将不会加载 megatron，只初始化 sglang ，可以用这个方法来进行推理部分的调试。 + 开启后，slime 将不会加载 megatron，只初始化 vllm ，可以用这个方法来进行推理部分的调试。 1. `--debug-train-only` - 开启后，slime 将不会加载 sglang，只初始化 megatron ，可以用这个方法来进行训练部分的调试。 + 开启后，slime 将不会加载 vllm，只初始化 megatron ，可以用这个方法来进行训练部分的调试。 2. `--save-debug-rollout-data /your/saved/debug/data_{rollout_id}.pt` @@ -46,15 +46,15 @@ slime 支持将训练部分和推理部分分开进行调试，从而实现： 3. `--load-debug-rollout-data /your/saved/debug/data_{rollout_id}.pt` - 开启后，会从 `args.load_debug_rollout_data.format(rollout_id=rollout_id)` 来加载数据，并且不会初始化 sglang（自动设置 `debug_train_only=True`）。可以以这种方式来固定训练部分的输入，对训练部分进行调优，例如切换各种并行。 + 开启后，会从 `args.load_debug_rollout_data.format(rollout_id=rollout_id)` 来加载数据，并且不会初始化 vllm（自动设置 `debug_train_only=True`）。可以以这种方式来固定训练部分的输入，对训练部分进行调优，例如切换各种并行。 ## INT4 / Compressed-Tensors 量化 Checkpoint 问题 -使用 INT4 量化模型（如 `compressed-tensors` 的 `W4A16`）时，checkpoint 的 `config.json` 中有一个 `quantization_config.ignore` 列表，指定哪些参数**不**做量化。在线权重更新（Megatron → SGLang）时，slime 也会读取这个 ignore list 来决定哪些参数需要 INT4 量化。ignore list 不正确会导致静默错误： +使用 INT4 量化模型（如 `compressed-tensors` 的 `W4A16`）时，checkpoint 的 `config.json` 中有一个 `quantization_config.ignore` 列表，指定哪些参数**不**做量化。在线权重更新（Megatron → vLLM）时，slime 也会读取这个 ignore list 来决定哪些参数需要 INT4 量化。ignore list 不正确会导致静默错误： 1. **MoE 路由权重（`mlp.gate.weight`）变成全零** - MoE 的路由权重（`mlp.gate.weight`，shape `[num_experts, hidden_size]`）是一个普通的 2D weight tensor，但它**不是** Linear 层的权重。如果它不在 ignore list 中，在线量化器会把它 INT4 量化为 `weight_packed`、`weight_scale`、`weight_zero_point` 等。然而 SGLang 不会以量化名称来加载路由权重，因此这些参数在 `load_weights` 时被静默跳过，导致 gate 权重全零。 + MoE 的路由权重（`mlp.gate.weight`，shape `[num_experts, hidden_size]`）是一个普通的 2D weight tensor，但它**不是** Linear 层的权重。如果它不在 ignore list 中，在线量化器会把它 INT4 量化为 `weight_packed`、`weight_scale`、`weight_zero_point` 等。然而 vLLM 不会以量化名称来加载路由权重，因此这些参数在 `load_weights` 时被静默跳过，导致 gate 权重全零。 **修复方法**：确保 `config.json` 的 ignore list 中包含 `"re:.*mlp\\.gate\\..*"`。 @@ -87,12 +87,12 @@ slime 支持将训练部分和推理部分分开进行调试，从而实现： 4. **如何排查** - - 使用 `--check-weight-update-equal` 验证 Megatron → SGLang 权重同步后的值是否正确。如果某个参数在 SGLang 侧全为零，说明它可能被错误量化或在 checkpoint 中缺失。 - - 使用 `--debug-rollout-only` 配合少量 GPU，快速测试 SGLang 能否从量化 checkpoint 正常生成文本。 + - 使用 `--check-weight-update-equal` 验证 Megatron → vLLM 权重同步后的值是否正确。如果某个参数在 vLLM 侧全为零，说明它可能被错误量化或在 checkpoint 中缺失。 + - 使用 `--debug-rollout-only` 配合少量 GPU，快速测试 vLLM 能否从量化 checkpoint 正常生成文本。 -## Debug sglang illegal memory access (IMA) +## Debug vllm illegal memory access (IMA) -在进行大规模 RL 时，不时会遇到 SGLang IMA 的问题，以下是我们的一些 debug 建议： +在进行大规模 RL 时，不时会遇到 vLLM IMA 的问题，以下是我们的一些 debug 建议： 1. 开启 `CUDA_LAUNCH_BLOCKING=1` diff --git a/docs/zh/developer_guide/profiling.md b/docs/zh/developer_guide/profiling.md index 11609ec55..cade3546a 100644 --- a/docs/zh/developer_guide/profiling.md +++ b/docs/zh/developer_guide/profiling.md @@ -1,6 +1,6 @@ # 性能分析 (Profiling) -在 slime 中，我们可以通过 SGLang 提供的 profiling 接口对 rollout 过程进行详细的性能分析。 +在 slime 中，我们可以通过 vLLM 提供的 profiling 接口对 rollout 过程进行详细的性能分析。 ## 1. 使 Rollout 进程进入等待状态 (Sleep Rollout) @@ -16,9 +16,9 @@ python train.py \ 该函数会让 rollout 进程进入无限循环等待状态，方便你手动发送请求或运行压测工具。 -## 2. 获取 SGLang 引擎列表 +## 2. 获取 vLLM 引擎列表 -SGLang 引擎（workers）注册在 router 上。你可以通过访问 router 的 `/workers` 接口来获取所有活跃引擎的列表。 +vLLM 引擎（workers）注册在 router 上。你可以通过访问 router 的 `/workers` 接口来获取所有活跃引擎的列表。 通常 router 地址会在启动日志中打印： ``` @@ -61,6 +61,6 @@ python tools/profile_rollout.py --router-url http://127.0.0.1:3000 --action stop 在 Rollout 进程通过 `sleep_rollout` 处于等待状态时，你可以： 1. 使用 `tools/profile_rollout.py` 启动 profiling。 -2. 使用压测工具（如 `sglang` 自带的 benchmark 工具）向 router 或直接向引擎发送请求。 +2. 使用压测工具（如 `vllm` 自带的 benchmark 工具）向 router 或直接向引擎发送请求。 3. 等待 profiling 完成（如果设置了 `num_steps`）或手动停止。 4. 在 `output_dir` 中获取 `.json` trace 文件，并使用 `chrome://tracing` 或 [Perfetto](https://ui.perfetto.dev/) 查看。 diff --git a/docs/zh/developer_guide/trace.md b/docs/zh/developer_guide/trace.md index 09588bd32..a5e24f93c 100644 --- a/docs/zh/developer_guide/trace.md +++ b/docs/zh/developer_guide/trace.md @@ -36,7 +36,7 @@ python tools/trace_timeline_viewer.py /path/to/debug/rollout_0.pt - 每一行对应一条 sample。 - 条形块表示 span，点表示瞬时事件。 - `trace_span(...)` 在开始和结束时记录的属性，都会显示在详情面板里。 -- 当 SGLang 返回 PD 分离相关时延时，viewer 会自动补出 `[P]` 和 `[D]` 两条虚拟 lane，用来拆开展示 prefill/decode。 +- 当 vLLM 返回 PD 分离相关时延时，viewer 会自动补出 `[P]` 和 `[D]` 两条虚拟 lane，用来拆开展示 prefill/decode。 - 如果没有开启 PD，这两条虚拟 lane 不会出现，基础 trace 也仍然可以正常渲染。 ## 给自定义代码打点 @@ -101,19 +101,18 @@ async def custom_rollout_batch(samples, **kwargs): - 外层用 `trace_function(...)` 表示整个函数生命周期 - 内层用 `trace_span(...)` 标记 generation、RM、filter、post-process 等关键子步骤 -如果想统一记录 SGLang 返回的 generation 元信息，可以复用 `build_sglang_meta_trace_attrs`： +如果想在某个 HTTP 调用周围记录每轮的 attrs，可以直接套一层 `trace_span`： ```python -from slime.utils.trace_utils import build_sglang_meta_trace_attrs, trace_span +from slime.utils.trace_utils import trace_span -with trace_span(sample, "sglang_generate") as span: +with trace_span(sample, "vllm_generate", attrs={"max_tokens": params["max_new_tokens"]}): output = await post(url, payload) - span.update(build_sglang_meta_trace_attrs(output["meta_info"])) ``` ## 使用建议 - 先保存少量 rollout；单个 dump 的 sample 数量适中时，viewer 会更容易阅读。 - viewer 直接基于保存下来的 `.pt` dump 工作，因此可以把文件拷到别的机器离线分析。 -- 如果你想看的是 SGLang 自身的 GPU / kernel 级 profiling trace，请参考 [性能分析](./profiling.md)。 +- 如果你想看的是 vLLM 自身的 GPU / kernel 级 profiling trace，请参考 [性能分析](./profiling.md)。 diff --git a/docs/zh/examples/deepseek-r1.md b/docs/zh/examples/deepseek-r1.md index 368653844..237540040 100644 --- a/docs/zh/examples/deepseek-r1.md +++ b/docs/zh/examples/deepseek-r1.md @@ -4,7 +4,7 @@ 我们会使用 bf16 进行训练，128x128 blockwise quant 的 fp8 格式进行推理，模型最大回复长度为 32k，并训练中会使用 dynamic sampling 对数据进行筛选。 -在并行上，sglang 方面我们会启用 ep64，开启 dp attention 与 deepep；megatron 部分我们采用 tp8、pp4、ep32、cp4。 +在并行上，vLLM 方面我们会启用 expert parallelism 与 data parallelism；megatron 部分我们采用 tp8、pp4、ep32、cp4。 ⚠️ 为了节省 GPU 显存，我们会使用 CPU Adam，每个 node（8xH100）会占用 1.4~1.5B 内存。如果单机的内存不够，可以通过增加 GPU，扩大并行的方式解决。 @@ -72,7 +72,7 @@ for WORKER_IP in $(awk '{print $1}' $BASE_DIR/mpi_hostfile); do fi echo "Starting Ray worker on ${WORKER_IP}" ssh root@"${WORKER_IP}" \ - "pkill -9 sglang ; ray stop --force ; pkill -9 python ; ray start --address=${MASTER_ADDR}:6379 --num-gpus 8 --node-ip-address ${WORKER_IP} --disable-usage-stats" & + "pkill -9 vllm ; ray stop --force ; pkill -9 python ; ray start --address=${MASTER_ADDR}:6379 --num-gpus 8 --node-ip-address ${WORKER_IP} --disable-usage-stats" & done wait ``` @@ -91,7 +91,7 @@ source "${SCRIPT_DIR}/models/deepseek-v3.sh" ```bash CKPT_ARGS=( - # sglang 需要的 hf ckpt，我们也会从这里读 tokenizer + # vLLM 需要的 hf ckpt，我们也会从这里读 tokenizer --hf-checkpoint $BASE_DIR/DeepSeek-R1/ #--hf-checkpoint $BASE_DIR/DeepSeek-R1-bf16/ --ref-load $BASE_DIR/DeepSeek-R1_torch_dist/ @@ -102,7 +102,7 @@ CKPT_ARGS=( ) ``` -slime 会根据 `hf_checkpoint` 中的量化配置从而在训练中进行在线量化。例如当前的例子中，我们使用的是 DeepSeek R1 的 fp8 ckpt，那么在进行参数更新的时候，我们会首先将参数进行 blockwise quant，再传至 sglang。 +slime 会根据 `hf_checkpoint` 中的量化配置从而在训练中进行在线量化。例如当前的例子中，我们使用的是 DeepSeek R1 的 fp8 ckpt，那么在进行参数更新的时候，我们会首先将参数进行 blockwise quant，再传至 vLLM。 #### PERF_ARGS @@ -167,30 +167,23 @@ OPTIMIZER_ARGS=( ) ``` -#### SGLANG_ARGS +#### VLLM_ARGS -sglang 所需的参数，这里 `--rollout-num-gpus-per-engine` 基本对应 sglang 的 `tp_size`，除此之外的 sglang 参数均通过添加 `--sglang-` 的前缀来传给 slime。为了充分利用 sglang 的大 EP 推理能力，我们加上了 ep64、dp_attention dp8、deepep mode auto 等配置。 +vLLM 所需的参数，这里 `--rollout-num-gpus-per-engine` 对应 vLLM 的 `tp_size`，除此之外的 vLLM 参数均通过添加 `--vllm-` 的前缀来传给 slime。 -最后的 `--sglang-server-concurrency` 是 slime 的特有参数，是为了方式同时发给 sglang server 的并发太大打爆 http server，默认为 512。但是我们现在是 8 机一个 server，为了保证每个 dp rank 能有 128 的并发，我们调整为 1024。 +`--vllm-server-concurrency` 是 slime 的特有参数，用于防止同时发给 vLLM 引擎的并发太大打爆 HTTP server，默认为 512。但是我们现在是 8 机一个 server，为了保证每个 dp rank 能有 128 的并发，我们调整为 1024。 ```bash -SGLANG_ARGS=( +VLLM_ARGS=( --rollout-num-gpus-per-engine 64 - --sglang-mem-fraction-static 0.7 - --sglang-ep-size 64 + --vllm-gpu-memory-utilization 0.7 + --vllm-enable-expert-parallel - # dp attention - --sglang-enable-dp-attention - --sglang-dp-size 8 - --sglang-moe-dense-tp-size 1 - --sglang-enable-dp-lm-head - - # enable deepep for sglang - --sglang-moe-a2a-backend deepep - --sglang-deepep-mode auto + # data parallelism for the attention block + --vllm-data-parallel-size 8 # make every dp rank has 128 concurrency - --sglang-server-concurrency 1024 + --vllm-server-concurrency 1024 ) ``` diff --git a/docs/zh/examples/glm4-9B.md b/docs/zh/examples/glm4-9B.md index 8cb562c1f..fb53c0715 100644 --- a/docs/zh/examples/glm4-9B.md +++ b/docs/zh/examples/glm4-9B.md @@ -2,7 +2,7 @@ ## 环境准备 -拉取 `slimerl/slime:latest` 镜像后，用如下方式初始化镜像环境： +拉取 `inferactinc/public:vime-vllm-cu129-latest` 镜像后，用如下方式初始化镜像环境： ```bash cd /root/ @@ -72,7 +72,7 @@ MODEL_ARGS += ( --rotary-base 10000 ) ```bash CKPT_ARGS=( - # sglang 需要的 hf ckpt，我们也会从这里读 tokenizer + # vLLM 需要的 hf ckpt，我们也会从这里读 tokenizer --hf-checkpoint /root/GLM-Z1-9B-0414 # reference model 的 ckp --ref-load /root/GLM-Z1-9B-0414_torch_dist @@ -191,17 +191,17 @@ OPTIMIZER_ARGS=( ) ``` -#### SGLANG_ARGS +#### VLLM_ARGS -sglang 所需的参数，这里 `--rollout-num-gpus-per-engine` 基本对应 sglang 的 `tp_size`，除此之外的 sglang 参数均通过添加 `--sglang-` 的前缀来传给 slime。 +vLLM 所需的参数，这里 `--rollout-num-gpus-per-engine` 对应 vLLM 的 `tp_size`，除此之外的 vLLM 参数均通过添加 `--vllm-` 的前缀来传给 slime。 ```bash -SGLANG_ARGS=( +VLLM_ARGS=( --rollout-num-gpus-per-engine 2 ) ``` -⚠️ slime 会用 sgl-router 来调度多个 sglang server，在不开启 dp attention 的情况下不支持 `dp_size`。 +⚠️ slime 会用 vllm-router 来调度多个 vLLM 引擎。 ### 训推一体 @@ -231,7 +231,7 @@ ray job submit ... \ 此时，训练和推理就会共用这 8 张卡了。 -⚠️ 在训推一体的训练时，megatron 始终会占据一些显存，所以需要通过调整 `--sglang-mem-fraction-static` 来降低 sglang 占据的显存比例。 +⚠️ 在训推一体的训练时，megatron 始终会占据一些显存，所以需要通过调整 `--vllm-gpu-memory-utilization` 来降低 vLLM 占据的显存比例。 ### dynamic sampling diff --git a/docs/zh/examples/glm4.7-30B-A3B.md b/docs/zh/examples/glm4.7-30B-A3B.md index cdb2b4562..290880910 100644 --- a/docs/zh/examples/glm4.7-30B-A3B.md +++ b/docs/zh/examples/glm4.7-30B-A3B.md @@ -66,38 +66,34 @@ GLM-4.7-Flash 是一个 MoE（混合专家）模型，包含 64 个路由专家 ) ``` -3. 开启 SGLang 支持的 MoE 优化，使用 DP attention： +3. 在 vLLM 侧开启 MoE expert parallelism。GLM-4.7-Flash 是非 MLA 模型， + 这里用 attention 上 8 路 data parallel + expert 上 expert parallel： ```bash - SGLANG_ARGS=( + VLLM_ARGS=( --rollout-num-gpus-per-engine 8 - --sglang-mem-fraction-static 0.7 - --sglang-enable-dp-attention - --sglang-dp-size 8 - --sglang-enable-dp-lm-head - --sglang-moe-dense-tp-size 1 + --vllm-gpu-memory-utilization 0.7 + --vllm-data-parallel-size 8 + --vllm-enable-expert-parallel ... ) ``` #### MTP 投机解码（推理加速） -GLM-4.7-Flash 包含 1 层 MTP（Multi-Token Prediction）层，可用于推理时的投机解码来加速 rollout 生成。要启用此功能，在 `SGLANG_ARGS` 中添加以下配置： +GLM-4.7-Flash 包含 1 层 MTP（Multi-Token Prediction）层，可用于推理时的投机解码来加速 rollout 生成。要启用此功能，在 `VLLM_ARGS` 中添加以下配置： ```bash -SGLANG_ARGS=( +VLLM_ARGS=( ... # MTP 投机解码 (EAGLE) - --sglang-speculative-algorithm EAGLE - --sglang-speculative-num-steps 3 - --sglang-speculative-eagle-topk 1 - --sglang-speculative-num-draft-tokens 4 + --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3}' ) ``` -这会让 SGLang 使用模型的 MTP 层作为 EAGLE 风格投机解码的 draft 模型。MTP 层预测多个未来 token，SGLang 并行验证它们，从而加速生成。 +这会让 vLLM 使用模型的 MTP 层作为 EAGLE 风格投机解码的 draft 模型。MTP 层预测多个未来 token，vLLM 并行验证它们，从而加速生成。 -> ⚠️ **注意**：投机解码会占用额外的 GPU 显存。如果遇到 OOM 问题，可以尝试降低 `--sglang-mem-fraction-static` 或关闭投机解码。 +> ⚠️ **注意**：投机解码会占用额外的 GPU 显存。如果遇到 OOM 问题，可以尝试降低 `--vllm-gpu-memory-utilization` 或关闭投机解码。 #### MTP 训练 @@ -139,17 +135,15 @@ bash scripts/run-glm4.7-30B-A3B.sh - 去掉 CPU Adam 相关的配置，因为使用了 distributed optimizer，多机环境下 optimizer 的显存占比会明显下降。 - 调整并行度：例如 TP=4, PP=2, EP=8, CP=2。 -当总卡数并不能被 expert 总数（64）乘除时，可以使用 `--sglang-ep-num-redundant-experts` 来增加冗余的 expert。例如对于 24 卡的场景： +当总卡数并不能被 expert 总数（64）整除时，可以开启 vLLM 的 EPLB（Expert Parallelism Load Balancer），通过 `--vllm-eplb-config` 配置冗余 expert 数量。例如对于 24 卡的场景： ```bash -SGLANG_ARGS=( +VLLM_ARGS=( --rollout-num-gpus-per-engine 24 - --sglang-mem-fraction-static 0.7 - --sglang-ep-size 24 - --sglang-enable-dp-attention - --sglang-dp-size 3 - --sglang-moe-dense-tp-size 1 - --sglang-enable-dp-lm-head - --sglang-ep-num-redundant-experts 16 + --vllm-gpu-memory-utilization 0.7 + --vllm-data-parallel-size 3 + --vllm-enable-expert-parallel + --vllm-enable-eplb + --vllm-eplb-config '{"num_redundant_experts": 16}' ) ``` diff --git a/docs/zh/examples/glm4.7-355B-A32B.md b/docs/zh/examples/glm4.7-355B-A32B.md index 0954721da..798b3c60a 100644 --- a/docs/zh/examples/glm4.7-355B-A32B.md +++ b/docs/zh/examples/glm4.7-355B-A32B.md @@ -83,39 +83,35 @@ GLM-4.7 是一个 MoE（混合专家）模型，包含 160 个路由专家（top ) ``` -3. 在 SGLang 中开启带 DP attention 的 MoE 优化： +3. 在 vLLM 侧开启 MoE expert parallelism。GLM-4.7 是非 MLA 模型， + 这里用 attention 上 4 路 data parallel + expert 上 expert parallel + （EP size 由 `tensor_parallel_size × data_parallel_size` 自动推导）： ```bash - SGLANG_ARGS=( + VLLM_ARGS=( --rollout-num-gpus-per-engine 32 - --sglang-mem-fraction-static 0.7 - --sglang-enable-dp-attention - --sglang-dp-size 4 - --sglang-ep-size 32 - --sglang-enable-dp-lm-head - --sglang-moe-dense-tp-size 1 + --vllm-gpu-memory-utilization 0.7 + --vllm-data-parallel-size 4 + --vllm-enable-expert-parallel ... ) ``` #### MTP 投机解码（推理加速） -GLM-4.7 包含 MTP（Multi-Token Prediction）层，可以在推理阶段用于投机解码，加速 rollout 生成。启用方法是在 `SGLANG_ARGS` 中加入： +GLM-4.7 包含 MTP（Multi-Token Prediction）层，可以在推理阶段用于投机解码，加速 rollout 生成。启用方法是在 `VLLM_ARGS` 中加入： ```bash -SGLANG_ARGS=( +VLLM_ARGS=( ... # MTP 投机解码 (EAGLE) - --sglang-speculative-algorithm EAGLE - --sglang-speculative-num-steps 3 - --sglang-speculative-eagle-topk 1 - --sglang-speculative-num-draft-tokens 4 + --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3}' ) ``` -这样 SGLang 就会使用模型自带的 MTP 层作为 EAGLE 风格投机解码的 draft model。 +这样 vLLM 就会使用模型自带的 MTP 层作为 EAGLE 风格投机解码的 draft model。 -> ⚠️ **注意**：投机解码会额外占用 GPU 显存。如果遇到 OOM，可以尝试降低 `--sglang-mem-fraction-static` 或暂时关闭投机解码。 +> ⚠️ **注意**：投机解码会额外占用 GPU 显存。如果遇到 OOM，可以尝试降低 `--vllm-gpu-memory-utilization` 或暂时关闭投机解码。 #### MTP 训练 @@ -146,13 +142,18 @@ MTP_ARGS=( - `MASTER_ADDR` 设置为所有节点都能访问到的地址； - 在启动 Ray worker 前先取消代理； - 提供一个 `HOSTFILE` 列出 worker IP（每行一个），并在启动前 `export HOSTFILE=/path/to/hostfile`； -- 并行度需要成套调整。默认示例使用 TP=8、PP=4、EP=16、CP=2，rollout 侧则使用 32 张卡 / engine + SGLang DP attention。 +- 并行度需要成套调整。默认示例使用 TP=8、PP=4、EP=16、CP=2，rollout 侧则使用 32 张卡 / engine + DP=4 + expert parallel。 -如果 rollout GPU 数与 expert 数（160）之间不能整除，可以通过 `--sglang-ep-num-redundant-experts` 增加冗余 expert。 +如果 rollout GPU 数与 expert 数（160）之间不能整除，可以开启 vLLM 的 EPLB（Expert Parallelism Load Balancer），通过 `--vllm-eplb-config` 配置冗余 expert，例如： + +```bash +--vllm-enable-eplb +--vllm-eplb-config '{"num_redundant_experts": 16}' +``` ## FP8 Rollout -开源版 GLM-4.7 的 FP8 checkpoint 使用的是 per-channel 量化，目前无法在 SGLang 中直接启用 DeepEP。可以利用 slime 自带工具将其转换为 128x128 的 per-block FP8 checkpoint： +开源版 GLM-4.7 的 FP8 checkpoint 使用的是 per-channel 量化，目前无法在 vLLM 中直接启用 DeepEP。可以利用 slime 自带工具将其转换为 128x128 的 per-block FP8 checkpoint： ```bash cd /root/slime @@ -165,25 +166,18 @@ python tools/convert_hf_to_fp8.py \ 随后把 `--hf-checkpoint` 改成 `$BASE_DIR/GLM-4.7-355B-A32B-FP8/` 即可开启 FP8 rollout。 -一个可参考的 FP8 `SGLANG_ARGS` 配置如下： +一个可参考的 FP8 `VLLM_ARGS` 配置如下： ```bash -SGLANG_ARGS=( +VLLM_ARGS=( --rollout-num-gpus-per-engine 32 - --sglang-mem-fraction-static 0.7 - --sglang-enable-dp-attention - --sglang-dp-size 32 - --sglang-ep-size 32 - --sglang-moe-dense-tp-size 1 - --sglang-enable-dp-lm-head - --sglang-cuda-graph-bs 1 2 4 8 $(seq 16 8 128) - - --sglang-speculative-algorithm EAGLE - --sglang-speculative-num-steps 3 - --sglang-speculative-eagle-topk 1 - --sglang-speculative-num-draft-tokens 4 - - --sglang-moe-a2a-backend deepep - --sglang-deepep-mode auto + --vllm-gpu-memory-utilization 0.7 + --vllm-data-parallel-size 32 + --vllm-enable-expert-parallel + --vllm-cudagraph-capture-sizes 1 2 4 8 $(seq 16 8 128) + + --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3}' + + --vllm-all2all-backend deepep_high_throughput ) ``` diff --git a/docs/zh/examples/qwen3-30B-A3B.md b/docs/zh/examples/qwen3-30B-A3B.md index 8413d1d1a..a1bc8296d 100644 --- a/docs/zh/examples/qwen3-30B-A3B.md +++ b/docs/zh/examples/qwen3-30B-A3B.md @@ -55,23 +55,21 @@ bash scripts/run-qwen3-30B-A3B.sh ) ``` -3. 开启 sglang 支持的 moe 优化，当前配置为 ep8： +3. 在 vLLM 侧开启 MoE expert parallelism。vLLM 中 EP size 由 + `tensor_parallel_size × data_parallel_size` 自动推导，所以 8 卡 engine 只需 + `--vllm-enable-expert-parallel` 就是 EP=8： ```bash - SGLANG_ARGS=( + VLLM_ARGS=( --rollout-num-gpus-per-engine 8 - --sglang-mem-fraction-static 0.7 - --sglang-ep-size 8 - --sglang-cuda-graph-bs 1 2 4 8 $(seq 16 8 256) + --vllm-gpu-memory-utilization 0.7 + --vllm-enable-expert-parallel + --vllm-cudagraph-capture-sizes 1 2 4 8 $(seq 16 8 256) ) ``` - 类似地，也可以加入 dp attention，例如配置上： - - ```bash - --sglang-enable-dp-attention - --sglang-dp-size 8 - ``` + 如果要在 attention 上做 DP 同时在 expert 上做 EP，可以加 `--vllm-data-parallel-size N` + 配合 `--vllm-enable-expert-parallel`。 ### bf16 训练 fp8 推理 @@ -101,18 +99,15 @@ hf download Qwen/Qwen3-30B-A3B-FP8 --local-dir /root/Qwen3-30B-A3B-FP8 除此之外，还可以进行如下的修改： -- 当总卡数并不能被 expert 总数乘除时，可以使用 `--sglang-ep-num-redundant-experts` 来增加冗余的 expert，例如对于 24 卡的场景，可以配置： +- 当总卡数并不能被 expert 总数整除时，可以开启 vLLM 的 EPLB（Expert Parallelism Load Balancer），通过 `--vllm-eplb-config` 配置冗余 expert。例如对于 24 卡的场景： ```bash - SGLANG_ARGS=( + VLLM_ARGS=( --rollout-num-gpus-per-engine 24 - --sglang-mem-fraction-static 0.7 - --sglang-ep-size 24 - --sglang-enable-dp-attention - --sglang-dp-size 3 - - --sglang-moe-dense-tp-size 1 - --sglang-enable-dp-lm-head - --sglang-ep-num-redundant-experts 16 + --vllm-gpu-memory-utilization 0.7 + --vllm-data-parallel-size 3 + --vllm-enable-expert-parallel + --vllm-enable-eplb + --vllm-eplb-config '{"num_redundant_experts": 16}' ) ``` diff --git a/docs/zh/examples/qwen3-4B.md b/docs/zh/examples/qwen3-4B.md index a2dc8beef..c02301cb5 100644 --- a/docs/zh/examples/qwen3-4B.md +++ b/docs/zh/examples/qwen3-4B.md @@ -2,7 +2,7 @@ ## 环境准备 -拉取 `slimerl/slime:latest` 镜像后，用如下方式初始化镜像环境： +拉取 `inferactinc/public:vime-vllm-cu129-latest` 镜像后，用如下方式初始化镜像环境： ```bash cd /root/ diff --git a/docs/zh/examples/qwen3-4b-base-openhermes.md b/docs/zh/examples/qwen3-4b-base-openhermes.md index 6abb6ac64..3de99a22b 100644 --- a/docs/zh/examples/qwen3-4b-base-openhermes.md +++ b/docs/zh/examples/qwen3-4b-base-openhermes.md @@ -51,7 +51,7 @@ bash script/run-qwen3-4B-base-sft.sh 可以将 [run-qwen3-4B-base-sft.sh](https://github.com/THUDM/slime/blob/main/scripts/run-qwen3-4B-base-sft.sh) 与 [run-qwen3-4B.sh](https://github.com/THUDM/slime/blob/main/scripts/run-qwen3-4B.sh) 进行对比。会发现除了我们将模型由 instruct 模型换为了 base 模型之外，主要进行了如下的几个调整： -1. 移除了 `SGLANG_ARGS` 和 `GRPO_ARGS`。这是因为 sft 的过程中不需要启动 sglang 或者做 grpo 相关的配置； +1. 移除了 `VLLM_ARGS` 和 `GRPO_ARGS`。这是因为 sft 的过程中不需要启动 vLLM 或者做 grpo 相关的配置； 2. 将 `ROLLOUT_ARGS` 改名为了 `SFT_ARGS`，并配置为： @@ -72,7 +72,7 @@ bash script/run-qwen3-4B-base-sft.sh ) ``` - slime 中的 sft 实际上是复用了 slime 的 custom rollout 功能，通过 `--rollout-function-path` 将数据生成部分从使用 sglang 的 RL rollout，切换成了从文件中读取数据的 sft 版本，即 `slime.rollout.sft_rollout.generate_rollout`。 + slime 中的 sft 实际上是复用了 slime 的 custom rollout 功能，通过 `--rollout-function-path` 将数据生成部分从使用 vLLM 的 RL rollout，切换成了从文件中读取数据的 sft 版本，即 `slime.rollout.sft_rollout.generate_rollout`。 对于 sft 来说，建议将 `rollout_batch_size` 与 `global_batch_size` 设置成相同的，并不要配置 `n_samples_per_prompt`，这样相当于是读一个 batch 就训一个 batch。 @@ -80,6 +80,6 @@ bash script/run-qwen3-4B-base-sft.sh 至于 `--calculate-per-token-loss`，这是因为 slime 默认是以 GRPO 的 per sample mean 进行计算的，而一般 sft 训练都是按一个 batch 的所有不被 mask 的 token 取平均，所以建议配置上。 - 最后 `--disable-compute-advantages-and-returns` 表示 sft 的过程中不需要预先计算 log prob，`--debug-train-only` 表示不需要初始化 sglang。 + 最后 `--disable-compute-advantages-and-returns` 表示 sft 的过程中不需要预先计算 log prob，`--debug-train-only` 表示不需要初始化 vLLM。 3. 使用了 `train_async.py` 而不是 `train.py`。这是为了利用异步训练的流程，来实现数据 prefetch。 diff --git a/docs/zh/get_started/customization.md b/docs/zh/get_started/customization.md index 509badf3f..b885ca58b 100644 --- a/docs/zh/get_started/customization.md +++ b/docs/zh/get_started/customization.md @@ -33,7 +33,7 @@ slime 通过函数路径参数提供了广泛的自定义能力。这些参数 ### 1. Rollout 函数 (`--rollout-function-path`) -**默认值**: `slime.rollout.sglang_rollout.generate_rollout` +**默认值**: `slime.rollout.vllm_rollout.generate_rollout` **用途**: 覆盖整个 rollout 生成逻辑。 @@ -408,7 +408,7 @@ def custom_hook(args, rollout_id, step_id, model, optimizer, opt_param_scheduler | 参数 | 说明 | | --- | --- | | `--use-routing-replay` | 训练中前向-反向路由一致性。([arXiv:2507.18071](https://arxiv.org/abs/2507.18071)) | -| `--use-rollout-routing-replay` | R3：在训练时重放 rollout 阶段的路由。slime 默认的 `sglang_router` 路径支持该功能。([arXiv:2510.11370](https://arxiv.org/abs/2510.11370)) | +| `--use-rollout-routing-replay` | R3：在训练时重放 rollout 阶段的路由。slime 默认的 `vllm_rollout` 路径支持该功能。([arXiv:2510.11370](https://arxiv.org/abs/2510.11370)) | ## 自定义函数路径的测试 diff --git a/docs/zh/get_started/qa.md b/docs/zh/get_started/qa.md index 05c73890b..ae722c0fe 100644 --- a/docs/zh/get_started/qa.md +++ b/docs/zh/get_started/qa.md @@ -43,21 +43,21 @@ data packing 是指在训练过程中，将长短不一的 sample 拼接到一起，从而提升训练的利用率。slime 默认会进行这样的操作。 -1. **sglang 部分出现 `Max retries exceeded with url: /get_model_info (Caused by NewConnectionError` 的问题怎么办？** +1. **vLLM 部分出现 `Max retries exceeded with url: /health (Caused by NewConnectionError)` 的问题怎么办？** - 这个问题主要来源于单机内多个 sglang server 导致的端口冲突，目前我们仍在和 sglang 团队一起解决这个问题。一个临时的缓解方案是尽可能减少单机内的 sglang server 数量，例如设置 tp=8。 + slime 通过 `GET /health` 探测 vLLM engine 是否就绪;这个错误意味着 readiness 探测从未连通。最常见的原因是单机内多个 vLLM 引擎导致的端口冲突。一个临时的缓解方案是尽可能减少单机内的 vLLM 引擎数量，例如设置 tp=8。 1. **grad norm 好高，训练训崩了怎么办？** 首先请确保数据和模型是匹配的，例如说，如果数据是实现已经做好 chat template 的了，这个 chat template 是否和原模型一致。如果数据正确的话，可以参考 [debug 指南](../developer_guide/debug.md) 进行更深入的分析。 -1. **我的 sglang 生成时间特别特别久，gpu 功率都打满了，跑了好久好没有输出是为什么？** +1. **我的 vLLM 生成时间特别特别久，gpu 功率都打满了，跑了好久好没有输出是为什么？** 请确认一下 `--hf-checkpoint` 对应的模型是否正确设置了 stop token，如果没有，可以通过 `--rollout-stop` 或者 `--rollout-stop-token-ids` 来进行设置。 -1. **sglang 出现 an illegal memory access was encountered** +1. **vLLM 出现 an illegal memory access was encountered** - 根据 sglang 的文档（https://docs.sglang.ai/references/troubleshooting.html），有可能是 OOM 了，可以考虑缩小 `--sglang-mem-fraction-static`。 + 常见原因是 OOM，可以考虑缩小 `--vllm-gpu-memory-utilization`。 1. **出现 torch compile/inducer 的 `JSONDecodeError`** diff --git a/docs/zh/get_started/quick_start.md b/docs/zh/get_started/quick_start.md index a57d93452..0dff33a4c 100644 --- a/docs/zh/get_started/quick_start.md +++ b/docs/zh/get_started/quick_start.md @@ -4,7 +4,7 @@ ## 基础环境搭建 -由于 slime 可能会包含针对 sglang/megatron 的临时补丁（patch）。为避免潜在的环境配置问题，强烈建议**用户使用我们提供的最新 Docker 镜像**，它已预置好所有依赖。 +由于 slime 可能会包含针对 vllm/megatron 的临时补丁（patch）。为避免潜在的环境配置问题，强烈建议**用户使用我们提供的最新 Docker 镜像**，它已预置好所有依赖。 ### 硬件支持说明 @@ -27,12 +27,12 @@ ```shell # 拉取最新镜像 -docker pull slimerl/slime:latest +docker pull inferactinc/public:vime-vllm-cu129-latest # 启动容器 docker run --rm --gpus all --ipc=host --shm-size=16g \ --ulimit memlock=-1 --ulimit stack=67108864 \ - -it slimerl/slime:latest /bin/bash + -it inferactinc/public:vime-vllm-cu129-latest /bin/bash ``` ### 安装 slime @@ -285,17 +285,17 @@ OPTIMIZER_ARGS=( ) ``` -### SGLANG_ARGS: SGLang 服务参数 +### VLLM_ARGS: vLLM 服务参数 -这部分参数用于配置 SGLang 推理服务。 -- `--rollout-num-gpus-per-engine`: 基本等同于 SGLang 的 `tp_size`。 -- 其他 SGLang 参数可以通过添加 `--sglang-` 前缀传递给 slime, slime 会自动透传给 SGLang。例如，要设置 SGLang 的 `--log-level INFO` 参数，只需使用 `--sglang-log-level INFO` 即可。 +这部分参数用于配置 vLLM 推理服务。 +- `--rollout-num-gpus-per-engine`: 等同于 vLLM 的 `tp_size`。 +- 其他 vLLM 参数可以通过添加 `--vllm-` 前缀传递给 slime，slime 会自动透传给 vLLM。例如，要设置 vLLM 的 `--log-level INFO` 参数，只需使用 `--vllm-log-level INFO` 即可。 > ⚠️ **注意**： -> slime 使用 `sgl-router` 调度多个 SGLang Server。在不开启 DP Attention 的情况下， `dp_size` 会通过 `rollout-num-gpus / rollout-num-gpus-per-engine` 计算得到。 +> slime 使用 `vllm-router` 调度多个 vLLM 引擎。`dp_size` 会通过 `rollout-num-gpus / rollout-num-gpus-per-engine` 计算得到。 ```bash -SGLANG_ARGS=( +VLLM_ARGS=( --rollout-num-gpus-per-engine 2 ) ``` @@ -332,7 +332,7 @@ ray job submit ... \ 此时，训练和推理将共享全部 8 张 GPU。 > ⚠️ **注意**： -> 在训推一体化模式下，Megatron 初始化后才能被 offload 掉，会占据一定量的显存。您需要通过调整 `--sglang-mem-fraction-static` 参数来降低 SGLang 的显存占用比例，以避免显存不足。通常我们建议为 0.8。 +> 在训推一体化模式下，Megatron 初始化后才能被 offload 掉，会占据一定量的显存。您需要通过调整 `--vllm-gpu-memory-utilization` 参数来降低 vLLM 的显存占用比例，以避免显存不足。通常我们建议为 0.8。 > 此外，[torch_memory_saver](https://github.com/fzyzcjy/torch_memory_saver) 里面的一些优化只能在训推一体模式中使用，因为需要释放 GPU 显存。训推分离模式暂不支持。 @@ -482,7 +482,7 @@ ROLLOUT_ARGS=( **核心实现要点**: 1. **构建交互循环**: 创建循环以控制最大交互轮次（如 `for _ in range(max_turns):`）。 -2. **调用模型生成动作**: 每轮循环中，调用 SGLang 服务，让模型根据当前对话历史生成下一步动作（如 `query`）。 +2. **调用模型生成动作**: 每轮循环中，调用 vLLM 服务，让模型根据当前对话历史生成下一步动作（如 `query`）。 3. **解析并执行动作**: 解析模型输出，识别动作与参数，并调用外部工具或 API（如 Google 搜索）。 4. **构建观察结果**: 将工具返回的结果格式化后，追加到对话历史中，作为下一轮的输入。 5. **处理 Loss Masking**: 这是 Agent 训练的关键。 @@ -501,7 +501,7 @@ async def generate(args, sample: Sample, sampling_params) -> Sample: for _ in range(max_turns): # 1. 模型生成动作 - model_output = await call_sglang(prompt + full_response, ...) + model_output = await call_vllm(prompt + full_response, ...) # ... tokenization and appending ... loss_masks += [1] * len(model_tokens) # loss_mask = 1 full_response += model_output @@ -571,7 +571,7 @@ ray job submit --address="http://127.0.0.1:8265" \ } }' \ -- python3 train.py \ - --...（其他 Megatron/SGLang/slime 参数） + --...（其他 Megatron/vLLM/slime 参数） ``` slime 针对大规模混合专家（MoE）模型的分布式训练进行了深度优化。我们提供了一些端到端的训练案例以供参考： diff --git a/docs/zh/get_started/usage.md b/docs/zh/get_started/usage.md index dede29482..16b277480 100644 --- a/docs/zh/get_started/usage.md +++ b/docs/zh/get_started/usage.md @@ -6,7 +6,7 @@ 1. 把集群中一部分 GPU 分配做训练，一部分分配做推理； 2. 训练的部分加载 megatron； -3. 推理部分加载 sglang； +3. 推理部分加载 vLLM； 4. 配置 RL 训练需要的超参。 按照这个顺序，我们需要配置这些参数： @@ -21,9 +21,7 @@ - `--rollout-num-gpus`：rollout （inference）一共需要多少卡； -- `--rollout-num-gpus-per-engine`：每个 inference engine 有多少卡，这个参数会比较像 sglang 的 `tp_size`，也就是在进行多机 serving 的时候，这个数值应该是总卡数，例如 2 机 16 卡 serving 一个模型，这里的值应该是 16。 - - 这里不像其他的 sglang 参数那样引入 `--sglang-tp-size` 是因为未来也许会考虑支持 sglang 的 dp_size 参数，也就是一个 engine 里面其实是有多个 sglang server 的（目前只支持 `--sglang-enable-dp-attention` 情况下的 `--sglang-dp-size`）。 +- `--rollout-num-gpus-per-engine`：每个 inference engine 有多少卡，这个参数会比较像 vLLM 的 `tp_size`，也就是在进行多机 serving 的时候，这个数值应该是总卡数，例如 2 机 16 卡 serving 一个模型，这里的值应该是 16。 在默认的配置下，我们会根据这些参数，通过 ray 给训练部分分配 `actor_num_nodes * actor_num_gpus_per_node` 张 GPU，给推理分配 `rollout_num_gpus` 张 GPU，也就是实现了训推分离。 @@ -41,7 +39,7 @@ slime 支持多种训练后端，可以通过 `--train-backend` 参数进行选 ### 加载 megatron -megatron 与 sglang, vllm 或者 huggingface trainer 之类的工具不同，他不能直接读取 huggingface ckpt，而是需要用户配置好要训练的模型的参数，并且加载 megatron 自己的 ckpt。 +megatron 与 vLLM 或者 huggingface trainer 之类的工具不同，它不能直接读取 huggingface ckpt，而是需要用户配置好要训练的模型的参数，并且加载 megatron 自己的 ckpt。 一般来说，我们需要做 3 点准备： @@ -139,20 +137,20 @@ torch 格式是 megatron 的老存储格式，里面的结构大约是一些 `mp - 不管进行何种方式存储 ckpt，即无论如何设置 `--ckpt-format`，megatron 都可以加载 torch 或 torch_dist 格式 -### 加载 sglang +### 加载 vLLM -sglang 的加载非常简单，只需要： +vLLM 的加载非常简单，只需要： -- `--hf-checkpoint`：初始化 sglang 用的 huggingface ckpt； +- `--hf-checkpoint`：初始化 vLLM 用的 huggingface ckpt； 注意： -- 在第一个训练步之前，slime 会把 megatron 里的参数同步给 sglang，所以 `--hf-checkpoint` 中不需要有最新的训练参数，在续训得时候也不需要更换 hf ckpt； -- sglang 默认会从 huggingface ckpt 中 `config.json` 读取模型的最大 context length，可以使用 `--sglang-context-length` 参数来对这个值进行覆盖，从而支持进行更长的推理； -- 在训推一体的训练过程中，虽然 megatron 和 sglang 会先后 offload，但是还是需要为对方留有一些空间，需要通过减小 `--sglang-mem-fraction-static` 来调整 sglang 的显存占用总量。 -- slime 支持透传 sgl-router 的参数，方式是在原参数名前加上 `router` 前缀。例如，sgl-router 的 `--balance-abs-threshold` 参数需要设置为 `--router-balance-abs-threshold`。由于 sgl-router 默认使用 cache-aware routing，可能会导致请求分配不均衡的问题。可以通过设置 `--router-balance-abs-threshold 0` 来强制均衡分配，但这可能会影响多轮对话场景下 prefix cache 的命中率。 +- 在第一个训练步之前，slime 会把 megatron 里的参数同步给 vLLM，所以 `--hf-checkpoint` 中不需要有最新的训练参数，在续训的时候也不需要更换 hf ckpt； +- vLLM 默认会从 huggingface ckpt 中 `config.json` 读取模型的最大 context length，可以使用 `--vllm-max-model-len` 参数来对这个值进行覆盖，从而支持进行更长的推理； +- 在训推一体的训练过程中，虽然 megatron 和 vLLM 会先后 offload，但是还是需要为对方留有一些空间，需要通过减小 `--vllm-gpu-memory-utilization` 来调整 vLLM 的显存占用总量。 +- slime 支持透传 vllm-router 的参数，方式是在原参数名前加上 `router` 前缀。例如，vllm-router 的 `--balance-abs-threshold` 参数需要设置为 `--router-balance-abs-threshold`。由于 vllm-router 默认使用 cache-aware routing，可能会导致请求分配不均衡的问题。可以通过设置 `--router-balance-abs-threshold 0` 来强制均衡分配，但这可能会影响多轮对话场景下 prefix cache 的命中率。 -对于一些 sglang 的自定义以及 slime 引入 sglang 的原理，请见 sglang 使用方法一节。 +对于一些 vLLM 的自定义以及 slime 引入 vLLM 的原理，请见 vLLM 使用方法一节。 ### 数据格式 @@ -282,9 +280,9 @@ megatron: slime 支持不同程度的自定义数据生成（rollout）。 -- 默认会使用 [slime/rollout/sglang_rollout.py](https://github.com/THUDM/slime/blob/main/slime/rollout/sglang_rollout.py) 中的 `generate_rollout` 函数进行数据生成。这个文件中实现了基于 sglang 的异步（asyncio）数据生成流程，并支持了例如 dynamic sampling，partial rollout 等功能； +- 默认会使用 [slime/rollout/vllm_rollout.py](https://github.com/vllm-project/vime/blob/main/slime/rollout/vllm_rollout.py) 中的 `generate_rollout` 函数进行数据生成。这个文件中实现了基于 vLLM 的异步（asyncio）数据生成流程，并支持了例如 dynamic sampling，partial rollout 等功能； -- 可以通过 `--rollout-function-path` 参数，完全替换 sglang_rollout.py 中的 `generate_rollout`，只需要保证 `--rollout-function-path` 传入的函数签名满足： +- 可以通过 `--rollout-function-path` 参数，完全替换默认的 `generate_rollout`，只需要保证 `--rollout-function-path` 传入的函数签名满足： ```python def generate_rollout(args, rollout_id, data_source, evaluation=False) -> RolloutFnTrainOutput | RolloutFnEvalOutput: @@ -327,71 +325,73 @@ slime 支持不同程度的自定义数据生成（rollout）。 TOKENIZER = AutoTokenizer.from_pretrained(args.hf_checkpoint, trust_remote_code=True) # send request to router + prompt_token_ids = TOKENIZER(sample.prompt, add_special_tokens=False)["input_ids"] output = await post( - f"http://{args.sglang_router_ip}:{args.sglang_router_port}/generate", + f"http://{args.vllm_router_ip}:{args.vllm_router_port}/inference/v1/generate", { - "text": sample.prompt, - "sampling_params": sampling_params, + "model": args.hf_checkpoint, + "token_ids": prompt_token_ids, + "sampling_params": {"max_tokens": sampling_params["max_new_tokens"]}, } ) - prompt_tokens_ids = TOKENIZER(sample.prompt, add_special_tokens=False)["input_ids"] - response_token_ids = TOKENIZER(output["text"], add_special_tokens=False)["input_ids"] + choice = output["choices"][0] + response_token_ids = list(choice.get("token_ids") or []) # set sample - sample.tokens = prompt_tokens_ids + response_token_ids + sample.tokens = prompt_token_ids + response_token_ids sample.response_length = len(response_token_ids) - finish_reason = output["meta_info"]["finish_reason"]["type"] + finish_reason = choice.get("finish_reason") or "stop" if finish_reason == "length": sample.status = Sample.Status.TRUNCATED - elif finish_reason == "abort": + elif finish_reason in ("abort", "cancelled"): sample.status = Sample.Status.ABORTED else: sample.status = Sample.Status.COMPLETED - sample.response = output["text"] + sample.response = TOKENIZER.decode(response_token_ids) if response_token_ids else "" return sample ``` - 更完备的版本请查看 [slime/rollout/sglang_rollout.py](https://github.com/THUDM/slime/blob/main/slime/rollout/sglang_rollout.py)。 + 更完备的版本请查看 [slime/rollout/vllm_rollout.py](https://github.com/vllm-project/vime/blob/main/slime/rollout/vllm_rollout.py)。 - 有的时候，我们还需要支持自定义的 reward model，可以通过配置 `--custom-rm-path` 来进行配置。 -## sglang 使用方法 +## vLLM 使用方法 -slime 通过 `HttpServerEngineAdapter` 作为中介，实现了基于 sglang 的 server based engine。 +slime 以 server 模式运行 vLLM，通过 HTTP 与之通信。 ### 参数配置 -slime 通过引入 sglang 的 `ServerArgs.add_cli_args`，从而引入了几乎所有的 sglang 参数，在设置一个 sglang 参数的时候，需要在参数前加上 `--sglang` 的前缀，例如： +slime 通过转发 vLLM 的 `EngineArgs` CLI 参数，引入了几乎所有的 vLLM 参数。在设置一个 vLLM 参数的时候，需要在参数前加上 `--vllm-` 的前缀，例如： -- 在训推一体的训练时，往往需要限制 `--mem-fraction-static`，这个参数需要转变为 `--sglang-mem-fraction-static`； -- 在训练中，希望 sglang 能推理超过 huggingface checkpoint 的 `config.json` 中标识的最长 context length，需要使用 `--context-length`，那么在 slime 中需要使用 `--sglang-context-length`； -- 在进行多机大 ep 推理的时候，需要 `--ep-size`、`--enable-dp-attention`、`--dp-size`、`--moe-a2a-backend deepep` 等，则可以对应地传入 `--sglang-ep-size`、`--sglang-enable-dp-attention`、`--sglang-dp-size`、`--sglang-moe-a2a-backend deepep` 。 +- 在训推一体的训练时，往往需要限制 GPU 显存占用，传入 `--vllm-gpu-memory-utilization`； +- 在训练中，希望 vLLM 能推理超过 huggingface checkpoint 的 `config.json` 中标识的最长 context length，需要使用 `--max-model-len`，那么在 slime 中需要使用 `--vllm-max-model-len`； +- 在进行多机大 ep 推理的时候，需要 `--enable-expert-parallel`、`--data-parallel-size` 等，则可以对应地传入 `--vllm-enable-expert-parallel`、`--vllm-data-parallel-size`。 -有部分参与和 slime 的资源调度相关，会由 slime 自行配置，例如： +有部分参数和 slime 的资源调度相关，会由 slime 自行配置，例如： -- `--tp-size` 在 slime 中会使用 `--rollout-num-gpus-per-engine` -- `--model-path` 在 slime 中会使用 `--hf-checkpoint` +- `--tensor-parallel-size` 在 slime 中会使用 `--rollout-num-gpus-per-engine` +- `--model` 在 slime 中会使用 `--hf-checkpoint` -sglang 参数引入 slime 的方式可以参考 [slime/backends/sglang_utils/arguments.py](https://github.com/THUDM/slime/blob/main/slime/backends/sglang_utils/arguments.py)。 +vLLM 参数引入 slime 的方式可以参考 [slime/backends/vllm_utils/arguments.py](https://github.com/vllm-project/vime/blob/main/slime/backends/vllm_utils/arguments.py)。 ### router 使用方法 -slime 会用 [sglang-router](https://github.com/sgl-project/sglang/tree/main/sgl-router) 来管理训练过程中的 sglang server。可以通过 `--sglang-router-ip` 与 `--sglang-router-port` 来配置 [sglang-router](https://github.com/sgl-project/sglang/tree/main/sgl-router) 的地址。如果不进行配置，则会在集群中默认启动一个 router。 +slime 会用 [vllm-router](https://github.com/vllm-project/router) 来管理训练过程中的 vLLM 引擎。可以通过 `--vllm-router-ip` 与 `--vllm-router-port` 来配置 router 的地址。如果不进行配置，则会在集群中默认启动一个 router。 -所有的 sglang server 在启动后，会通过 `/add_worker` 申请加入 router。在实际进行数据生成的时候，只需要向 router 发送 http 请求，router 会进行 load balancing 操作，将请求转发给 server 们。 +所有的 vLLM 引擎在启动后会注册到 router。在实际进行数据生成的时候，只需要向 router 发送 http 请求，router 会进行 load balancing 操作，将请求转发给引擎。 -当通过 `--sglang-router-ip` 与 `--sglang-router-port` 来配置传入一个外部的 router，此时 slime 不再会在内部启动一个 router，而是会把所有的 server 都注册在这个外部 router 上。这时可以利用这个外部的 router 地址来实现更复杂的数据生成流程。注意 router 是支持 openai compatible api 的。 +当通过 `--vllm-router-ip` 与 `--vllm-router-port` 来配置传入一个外部的 router，此时 slime 不再会在内部启动一个 router，而是会把所有的引擎都注册在这个外部 router 上。这时可以利用这个外部的 router 地址来实现更复杂的数据生成流程。注意 router 是支持 openai compatible api 的。 -### 高级引擎配置（--sglang-config） +### 高级引擎配置（--vllm-config） -对于高级部署场景，可以使用 `--sglang-config` 指定一个 YAML 文件，来配置服务器组、多模型部署以及选择性权重更新。 +对于高级部署场景，可以使用 `--vllm-config` 指定一个 YAML 文件，来配置服务器组、多模型部署以及选择性权重更新。 **多模型部署**允许同时服务多个模型（例如一个接收权重更新的 actor 模型和一个冻结的 reference/reward 模型）： ```yaml -sglang: +vllm: - name: actor update_weights: true # 接收训练的权重更新（默认） server_groups: @@ -407,11 +407,11 @@ sglang: num_gpus_per_engine: 2 ``` -每个模型都有自己独立的 router。每个模型的 router 信息可通过 `args.sglang_model_routers`（一个将模型名映射到 `(ip, port)` 元组的字典）访问。自定义 rollout 函数可以使用 `slime.rollout.sglang_rollout` 中的 `get_model_url(args, "ref")` 来将请求路由到指定模型。 +每个模型都有自己独立的 router。每个模型的 router 信息可通过 `args.vllm_model_routers`（一个将模型名映射到 `(ip, port)` 元组的字典）访问。自定义 rollout 函数可以使用 `slime.rollout.vllm_rollout` 中的 `get_model_url(args, "ref")` 来将请求路由到指定模型。 **服务器组功能：** - `worker_type`：`regular`、`prefill`、`decode` 或 `placeholder`（预留 GPU 位置但不创建引擎） -- `overrides`：SGLang `ServerArgs` 字段覆盖字典，会叠加在 `--sglang-*` CLI 参数之上 +- `overrides`：vLLM `EngineArgs` 字段覆盖字典，会叠加在 `--vllm-*` CLI 参数之上 - `num_gpus_per_engine`：每组的 TP 大小覆盖 ## megatron 使用方法 diff --git a/docs/zh/index.rst b/docs/zh/index.rst index 36d3d79eb..98b960608 100644 --- a/docs/zh/index.rst +++ b/docs/zh/index.rst @@ -3,7 +3,7 @@ slime 文档 slime 是一个面向 RL Scaling 的 LLM 后训练框架，提供两大核心能力： -- 高性能训练：通过连接 Megatron 与 SGLang，支持多种模式下的高效训练； +- 高性能训练：通过连接 Megatron 与 vLLM，支持多种模式下的高效训练； - 灵活的数据生成：通过自定义数据生成接口与基于服务器的引擎，实现任意训练数据生成流程。 slime 是 GLM-4.7、GLM-4.6、GLM-4.5 背后的 RL 训练框架。除此之外，slime 还支持: @@ -47,7 +47,7 @@ slime 是 GLM-4.7、GLM-4.6、GLM-4.5 背后的 RL 训练框架。除此之外 advanced/reproducibility.md advanced/fault-tolerance.md advanced/pd-disaggregation.md - advanced/sglang-config.md + advanced/vllm-config.md advanced/megatron-config.md advanced/arch-support-beyond-megatron.md @@ -70,9 +70,3 @@ slime 是 GLM-4.7、GLM-4.6、GLM-4.5 背后的 RL 训练框架。除此之外 developer_guide/trace.md developer_guide/profiling.md -.. toctree:: - :maxdepth: 1 - :caption: 博客 - - blogs/release_v0.1.0.md - blogs/introducing_slime.md