diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7b8b7ba51b..232a9c5c27 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,7 +6,7 @@ Thank you for your interest in contributing to slime! We deeply appreciate every ## Collaboration Scope -slime is the RL training infrastructure behind [GLM-4.5 through GLM-5.1](https://z.ai) and a large number of internal experiments at Z.ai. We open-sourced slime because we believe the training scenarios used internally cover the majority of cutting-edge RL algorithm requirements, and we hope to provide the community with a correct and efficient large-scale RL training infrastructure. +slime is the RL training infrastructure behind [GLM-4.5 through GLM-5.2](https://z.ai) and a large number of internal experiments at Z.ai. We open-sourced slime because we believe the training scenarios used internally cover the majority of cutting-edge RL algorithm requirements, and we hope to provide the community with a correct and efficient large-scale RL training infrastructure. Our goal for open-source collaboration is focused on **bug fixes** and **general-purpose large-scale RL optimizations**. We have had several successful collaborations with the community in this area, including: diff --git a/README.md b/README.md index 9337b0065f..5d49a92049 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ This makes slime one of the most battle-tested open RL post-training frameworks: ## Why This Design Matters -- **Battle-tested by frontier model training**: slime is the RL framework behind [GLM-5.1](https://z.ai/blog/glm-5.1), [GLM-5](https://z.ai/blog/glm-5), [GLM-4.7](https://z.ai/blog/glm-4.7), [GLM-4.6](https://z.ai/blog/glm-4.6), and [GLM-4.5](https://z.ai/blog/glm-4.5). This validates the full post-training loop, not only isolated examples. +- **Battle-tested by frontier model training**: slime is the RL framework behind [GLM-5.2](https://z.ai/blog/glm-5.2), [GLM-5.1](https://z.ai/blog/glm-5.1), [GLM-5](https://z.ai/blog/glm-5), [GLM-4.7](https://z.ai/blog/glm-4.7), [GLM-4.6](https://z.ai/blog/glm-4.6), and [GLM-4.5](https://z.ai/blog/glm-4.5). This validates the full post-training loop, not only isolated examples. - **Correctness-first infrastructure**: RL bugs are often silent. slime keeps the dataflow explicit, supports separate rollout-only and train-only debugging paths, and documents reproducibility, fault tolerance, tracing, profiling, and CI as first-class engineering concerns. - **Native by design**: slime passes Megatron arguments through directly and exposes installed SGLang arguments with a `--sglang-` prefix. New upstream training and serving optimizations can be used without adding another abstraction layer inside slime. - **Maximum data-generation freedom**: math, code, search, tools, sandboxes, verifiers, environments, multi-agent systems, and long-horizon agentic workflows plug in as data generation or reward workflows. They do not fork the training kernel. diff --git a/README_zh.md b/README_zh.md index 0f62b1b8a6..5240fa22f5 100644 --- a/README_zh.md +++ b/README_zh.md @@ -17,7 +17,7 @@ slime 的设计目标，是让这两大能力彼此强化，同时避免把系 ## 为什么这个设计重要 -- **经过 frontier model 训练验证**：slime 是 [GLM-5.1](https://z.ai/blog/glm-5.1)、[GLM-5](https://z.ai/blog/glm-5)、[GLM-4.7](https://z.ai/blog/glm-4.7)、[GLM-4.6](https://z.ai/blog/glm-4.6)、[GLM-4.5](https://z.ai/blog/glm-4.5) 背后的 RL 训练框架。这验证的是完整 post-training loop，而不是孤立 example。 +- **经过 frontier model 训练验证**：slime 是 [GLM-5.2](https://z.ai/blog/glm-5.2)、[GLM-5.1](https://z.ai/blog/glm-5.1)、[GLM-5](https://z.ai/blog/glm-5)、[GLM-4.7](https://z.ai/blog/glm-4.7)、[GLM-4.6](https://z.ai/blog/glm-4.6)、[GLM-4.5](https://z.ai/blog/glm-4.5) 背后的 RL 训练框架。这验证的是完整 post-training loop，而不是孤立 example。 - **以正确性为先的基础设施**：RL bug 往往不会立刻报错。slime 保持显式的数据流，支持 rollout-only 和 train-only 分离调试，并把可复现性、容错、trace、profiling 和 CI 作为一等工程问题来维护。 - **从设计开始就是 native**：slime 直接透传 Megatron 参数，并通过 `--sglang-` 前缀暴露当前安装版本 SGLang 支持的参数。新的上游训练和 serving 优化可以直接使用，不需要在 slime 里再加一层抽象。 - **最大化的数据生成自由度**：math、code、search、tool、sandbox、verifier、environment、multi-agent system 以及 long-horizon agentic workflow 都可以作为 data generation 或 reward workflow 接入，而不需要 fork training kernel。 diff --git a/docs/en/examples/glm5.2-744B-A40B.md b/docs/en/examples/glm5.2-744B-A40B.md new file mode 100644 index 0000000000..4057926cea --- /dev/null +++ b/docs/en/examples/glm5.2-744B-A40B.md @@ -0,0 +1,177 @@ +# GLM-5.2 744B-A40B with 256xH100 + +This is the recommended 32-node, 256-H100 training example for [GLM-5.2](https://z.ai/blog/glm-5.2). + +The recipe uses the GLM-5.2 BF16 checkpoint for Megatron training and the FP8 checkpoint for SGLang rollout. It assumes two Hugging Face repositories will be available: + +- BF16: `zai-org/GLM-5.2` +- FP8: `zai-org/GLM-5.2-FP8` + +## Environment Setup + +For environment setup and dataset download, see [Example: Qwen3-4B](qwen3-4B.md). For multi-node training, make sure every node can access the same `$BASE_DIR` path. + +### Download Model + +```bash +hf download zai-org/GLM-5.2 --local-dir $BASE_DIR/GLM-5.2 +hf download zai-org/GLM-5.2-FP8 --local-dir $BASE_DIR/GLM-5.2-FP8 +``` + +The open-source GLM-5.2 config uses `model_type: glm_moe_dsa`, which slime maps onto +the DeepSeek-V3.2 bridge (`slime_plugins.mbridge.deepseek_v32`) since the two share the +same DSA weight layout. + +### Convert Checkpoint + +The training side needs the BF16 Hugging Face checkpoint converted to the Megatron torch_dist format. The torch_dist format is reshardable, so the conversion parallel layout does **not** need to match training; we use a layout that satisfies Megatron's expert-group constraint on the conversion node count. + +Run the following on 4 nodes / 32 GPUs: + +```bash +cd /root/slime +pip install -e . --no-deps +source scripts/models/glm5.2-744B-A40B.sh +PYTHONPATH=/root/Megatron-LM/ torchrun \ + --nproc-per-node 8 \ + --master-addr ${MASTER_ADDR} --master-port 12345 \ + --nnodes=4 --node-rank ${NODE_RANK} \ + tools/convert_hf_to_torch_dist.py \ + ${MODEL_ARGS[@]} \ + --tensor-model-parallel-size 8 \ + --pipeline-model-parallel-size 2 \ + --decoder-last-pipeline-num-layers 40 \ + --expert-model-parallel-size 16 \ + --expert-tensor-parallel-size 1 \ + --hf-checkpoint $BASE_DIR/GLM-5.2/ \ + --save $BASE_DIR/GLM-5.2_torch_dist/ +``` + +Here, `MASTER_ADDR` is the IP of node0, and `NODE_RANK` is the current node index. + +`MODEL_ARGS` includes `--allgather-cp`, a slime-only flag, so `tools/convert_hf_to_torch_dist.py` registers it too (it is a no-op for conversion). On 32 GPUs, Megatron requires `expert_tp(1) * expert_model_parallel * pp` to divide the world size, so we convert with `EP=16` (`1*16*2=32`). The resulting checkpoint still loads at training-time `EP=32` because torch_dist is reshardable. + +## Run Training + +From node0: + +```bash +cd /root/slime +export BASE_DIR=/shared/path +export MASTER_ADDR= +export HOSTFILE=$BASE_DIR/hostfile # one worker IP per line, all 32 nodes +bash scripts/run-glm5.2-744B-A40B.sh +``` + +If `HOSTFILE` is not set, join the other nodes to the Ray cluster manually. + +### Parameter Introduction + +#### Model Configuration + +`scripts/models/glm5.2-744B-A40B.sh` contains the GLM-5.2 DSA + cross-layer index sharing configuration: 256 routed experts, top-8 activation, 1 shared expert, and 78 layers total (3 dense + 75 MoE). + +The DSA index sharing schedule, such as `index_topk_freq=4` and `index_skip_topk_offset=3`, is read from the Hugging Face config. The Megatron side uses the shared `slime_plugins.models.glm5.glm5:get_glm5_spec` provider and enables: + +```bash +--allgather-cp +``` + +This makes DSA + context parallel use the allgather-CP layout, and the index-share provider gathers index K/V across the CP group. + +#### Training Parallelism + +The default script targets 32 nodes and 256 GPUs: + +```bash +PERF_ARGS=( + --tensor-model-parallel-size 4 + --pipeline-model-parallel-size 8 + --decoder-first-pipeline-num-layers 14 + --decoder-last-pipeline-num-layers 16 + --context-parallel-size 8 + --expert-model-parallel-size 32 + --expert-tensor-parallel-size 1 + ... +) +``` + +`TP=4 * PP=8 * CP=8 = 256` GPUs form one training group (`DP=1`). The expert group constraint `expert_tp(1) * EP(32) * PP(8) = 256` divides the world size exactly (`expert_dp=1`). + +DSA cross-layer index sharing requires every pipeline stage to **start** on a "computing" layer. With `index_topk_freq=4` / `index_skip_topk_offset=3`, the computing layers are 1, 2, 3, 7, 11, ..., 75. A uniform `78/8` split would start stages on skip layers and fail the index-share assertion in `get_glm5_spec`. We therefore use `--decoder-first-pipeline-num-layers 14` and `--decoder-last-pipeline-num-layers 16`, leaving 6 middle stages of `(78-14-16)/6 = 8` layers each. The stage starts land on global layers 1, 15, 23, 31, 39, 47, 55, 63 — all computing layers. + +#### BF16 Training + FP8 Rollout + +The launcher writes the default paths directly in `CKPT_ARGS` and `ROLLOUT_ARGS`, matching the style of the other example scripts: + +```bash +CKPT_ARGS=( + --hf-checkpoint $BASE_DIR/GLM-5.2-FP8 + --ref-load $BASE_DIR/GLM-5.2_torch_dist + --load $BASE_DIR/GLM-5.2_slime + --save $BASE_DIR/GLM-5.2_slime + --save-interval 20 +) + +ROLLOUT_ARGS=( + --prompt-data $BASE_DIR/dapo-math-17k/dapo-math-17k.jsonl + ... +) +``` + +`--hf-checkpoint` provides FP8 weights and the tokenizer for SGLang rollout; `--ref-load` is the Megatron torch_dist checkpoint converted from BF16. To debug BF16 rollout, change `--hf-checkpoint` in the script to `$BASE_DIR/GLM-5.2`. + +#### SGLang Configuration + +The rollout side runs with **prefill/decode (PD) disaggregation**: 1 prefill engine (64 GPU) + 3 decode engines (192 GPU) = 256 GPUs total (which must equal the colocated `rollout_num_gpus`). Each engine spans 64 GPUs with DP attention and `EP=64` (DeepEP's dispatch config map supports up to 160 EP ranks, so a single 256-GPU engine would be invalid). Prefill uses the `auto` DeepEP path; decode uses `low_latency` + `deep_gemm`. The split is configured via the `--sglang-config` YAML: + +```yaml +sglang: + - name: default + server_groups: + - worker_type: prefill + num_gpus: 64 + num_gpus_per_engine: 64 + overrides: { deepep_mode: auto, ... } + - worker_type: decode + num_gpus: 192 + num_gpus_per_engine: 64 + overrides: { deepep_mode: low_latency, moe_runner_backend: deep_gemm, ... } +``` + +PD transfer runs over RDMA/IB with the mooncake backend: + +```bash +--sglang-disaggregation-transfer-backend mooncake +--sglang-disaggregation-ib-device mlx5_100,...,mlx5_107 +``` + +The rest of the rollout uses FP8 KV cache and the NSA + DeepEP backends: + +```bash +SGLANG_ARGS=( + --sglang-enable-dp-attention + --sglang-ep-size 64 + --sglang-dp-size 64 + --sglang-kv-cache-dtype fp8_e4m3 + --sglang-nsa-decode-backend flashmla_kv + --sglang-nsa-prefill-backend flashmla_sparse + --sglang-attention-backend nsa + ... +) +``` + +MTP / EAGLE speculative decoding is enabled using the model's own next-token-prediction layer (the GLM-5.2 checkpoint ships an MTP layer), so no separate draft model is needed: + +```bash +--sglang-speculative-algorithm EAGLE +--sglang-speculative-num-steps 4 +--sglang-speculative-eagle-topk 1 +--sglang-speculative-num-draft-tokens 5 +``` + +`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` must cover the largest decode batch: `max cuda_graph_max_bs (decode group = 12) * speculative_num_draft_tokens (5) = 60`, rounded up to `64`. A value below this trips the DeepEP low-latency dispatch buffer assertion during the decode group's CUDA-graph capture. + +#### Networking + +DeepEP/NVSHMEM communication across nodes needs the IB-aware NCCL settings in the Ray runtime env (`NCCL_SOCKET_IFNAME`, `NCCL_IB_*`, `NCCL_NET_GDR_LEVEL`, `NCCL_P2P_LEVEL=NVL`, `NCCL_NVLS_ENABLE=0`, `MC_IB_PCI_RELAXED_ORDERING`, ...). The script defaults to `SOCKET_IFNAME=eth0`; set `SOCKET_IFNAME` before launch if your environment differs, and it will be written to `GLOO_SOCKET_IFNAME`, `TP_SOCKET_IFNAME`, and `NCCL_SOCKET_IFNAME`. DeepEP also requires `NVSHMEM_DISABLE_NCCL=1`. diff --git a/docs/en/get_started/quick_start.md b/docs/en/get_started/quick_start.md index 70132bbdb1..d0bd3fb780 100644 --- a/docs/en/get_started/quick_start.md +++ b/docs/en/get_started/quick_start.md @@ -587,6 +587,7 @@ export NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME=$(ip -o -4 addr show | awk '$4 ~ /^10\. slime has been deeply optimized for distributed training of large-scale Mixture of Experts (MoE) models. We provide some end-to-end training cases for reference: - [Example: 8xH100 Training GLM-4.7-Flash](../examples/glm4.7-30B-A3B.md) +- [Example: 32xH100 Training GLM-5.2](../examples/glm5.2-744B-A40B.md) - [Example: 64xH100 Training GLM-4.7](../examples/glm4.7-355B-A32B.md) - [Example: 128xH100 Training DeepSeek-R1](../examples/deepseek-r1.md) - Scripts such as `scripts/run_qwen3_30b_a3b.py` and `scripts/run_glm45_355b_a32b.py` also support multi-node training. Their documentation is still being expanded. diff --git a/docs/en/index.rst b/docs/en/index.rst index 4617d5beaf..f3401ce3a7 100644 --- a/docs/en/index.rst +++ b/docs/en/index.rst @@ -13,7 +13,7 @@ This makes slime one of the most battle-tested open RL post-training frameworks: Why This Design Matters ----------------------- -- **Battle-tested by frontier model training**: slime is the RL framework behind `GLM-5.1 `_, `GLM-5 `_, `GLM-4.7 `_, `GLM-4.6 `_, and `GLM-4.5 `_. +- **Battle-tested by frontier model training**: slime is the RL framework behind `GLM-5.2 `_, `GLM-5.1 `_, `GLM-5 `_, `GLM-4.7 `_, `GLM-4.6 `_, and `GLM-4.5 `_. - **Native by design**: slime passes Megatron arguments through directly and exposes installed SGLang arguments with a ``--sglang-`` prefix, so upstream training and serving optimizations remain available without adding another wrapper layer. - **SGLang-focused rollout**: slime chooses one rollout backend intentionally. This avoids flattening multiple inference engines into a lowest-common-denominator abstraction and lets RL workloads use SGLang-specific serving, routing, caching, disaggregation, and weight-sync behavior directly. - **Agentic workflows as data generation**: tool use, sandbox interaction, verifier rewards, environment feedback, multi-agent loops, and long-horizon agentic workflows plug into the same training / rollout / Data Buffer path instead of forking the training kernel. @@ -67,6 +67,7 @@ Start by Use Case examples/glm4.7-30B-A3B.md examples/qwen3-30B-A3B.md + examples/glm5.2-744B-A40B.md examples/glm4.7-355B-A32B.md examples/deepseek-r1.md diff --git a/docs/zh/examples/glm5.2-744B-A40B.md b/docs/zh/examples/glm5.2-744B-A40B.md new file mode 100644 index 0000000000..8f5bc9e0ea --- /dev/null +++ b/docs/zh/examples/glm5.2-744B-A40B.md @@ -0,0 +1,175 @@ +# 256xH100 训练 GLM-5.2 744B-A40B + +这里是使用 32 节点、256 张 H100 训练 [GLM-5.2](https://z.ai/blog/glm-5.2) 的推荐配置示例。 + +这个配置使用 GLM-5.2 的 BF16 checkpoint 做 Megatron 训练，使用 FP8 checkpoint 做 SGLang rollout。下面假设 Hugging Face 上会提供两个地址： + +- BF16: `zai-org/GLM-5.2` +- FP8: `zai-org/GLM-5.2-FP8` + +## 环境准备 + +搭建环境与下载数据的方法可以参考 [示例：Qwen3-4B](qwen3-4B.md)。多机启动前，请确保所有节点都能访问同一个 `$BASE_DIR` 路径。 + +### 下载模型 + +```bash +hf download zai-org/GLM-5.2 --local-dir $BASE_DIR/GLM-5.2 +hf download zai-org/GLM-5.2-FP8 --local-dir $BASE_DIR/GLM-5.2-FP8 +``` + +开源 GLM-5.2 的 config 使用 `model_type: glm_moe_dsa`，slime 将其映射到 DeepSeek-V3.2 的 bridge（`slime_plugins.mbridge.deepseek_v32`），因为两者共享相同的 DSA 权重布局。 + +### 转换 Checkpoint + +训练侧需要把 BF16 Hugging Face checkpoint 转换为 Megatron 可加载的 torch_dist 格式。torch_dist 格式支持重新切分，所以转换时的并行布局**不需要**与训练一致；我们使用一个能满足 Megatron expert group 约束（在转换的节点数下成立）的布局即可。 + +可以在 4 台机器 / 32 卡上分别执行： + +```bash +cd /root/slime +pip install -e . --no-deps +source scripts/models/glm5.2-744B-A40B.sh +PYTHONPATH=/root/Megatron-LM/ torchrun \ + --nproc-per-node 8 \ + --master-addr ${MASTER_ADDR} --master-port 12345 \ + --nnodes=4 --node-rank ${NODE_RANK} \ + tools/convert_hf_to_torch_dist.py \ + ${MODEL_ARGS[@]} \ + --tensor-model-parallel-size 8 \ + --pipeline-model-parallel-size 2 \ + --decoder-last-pipeline-num-layers 40 \ + --expert-model-parallel-size 16 \ + --expert-tensor-parallel-size 1 \ + --hf-checkpoint $BASE_DIR/GLM-5.2/ \ + --save $BASE_DIR/GLM-5.2_torch_dist/ +``` + +其中 `MASTER_ADDR` 是 node0 的 IP，`NODE_RANK` 表示当前机器编号。 + +`MODEL_ARGS` 里包含 `--allgather-cp` 这个 slime 自定义参数，所以 `tools/convert_hf_to_torch_dist.py` 也注册了它（转换时是 no-op）。在 32 卡上，Megatron 要求 `expert_tp(1) * expert_model_parallel * pp` 能整除 world size，因此转换时使用 `EP=16`（`1*16*2=32`）。由于 torch_dist 支持重新切分，转换出的 checkpoint 仍然可以在训练时以 `EP=32` 加载。 + +## 执行训练 + +从 node0 执行： + +```bash +cd /root/slime +export BASE_DIR=/shared/path +export MASTER_ADDR= +export HOSTFILE=$BASE_DIR/hostfile # 每行一个 worker IP，共 32 个节点 +bash scripts/run-glm5.2-744B-A40B.sh +``` + +如果不设置 `HOSTFILE`，需要手动在其他节点加入 Ray 集群。 + +### 参数简介 + +#### 模型配置 + +`scripts/models/glm5.2-744B-A40B.sh` 使用 GLM-5.2 的 DSA + cross-layer index sharing 配置：256 个 routed experts、top-8 激活、1 个 shared expert，模型共 78 层（3 层 dense + 75 层 MoE）。 + +DSA index sharing 的 schedule（例如 `index_topk_freq=4`、`index_skip_topk_offset=3`）从 Hugging Face config 中读取。Megatron 侧使用共享的 `slime_plugins.models.glm5.glm5:get_glm5_spec` provider，并开启： + +```bash +--allgather-cp +``` + +这会让 DSA + context parallel 使用 allgather-CP layout，并在 index-share provider 中对 index K/V 做 CP group gather。 + +#### 训练并行 + +默认脚本按 32 节点 256 卡配置： + +```bash +PERF_ARGS=( + --tensor-model-parallel-size 4 + --pipeline-model-parallel-size 8 + --decoder-first-pipeline-num-layers 14 + --decoder-last-pipeline-num-layers 16 + --context-parallel-size 8 + --expert-model-parallel-size 32 + --expert-tensor-parallel-size 1 + ... +) +``` + +`TP=4 * PP=8 * CP=8 = 256` 卡构成一个训练组（`DP=1`）。expert group 约束 `expert_tp(1) * EP(32) * PP(8) = 256` 正好整除 world size（`expert_dp=1`）。 + +DSA cross-layer index sharing 要求每个 pipeline stage 都必须**从 computing layer 开始**。在 `index_topk_freq=4` / `index_skip_topk_offset=3` 下，computing layer 是第 1、2、3、7、11、...、75 层。如果直接 `78/8` 均分，stage 会从 skip layer 开始，触发 `get_glm5_spec` 里的 index-share 断言。因此我们使用 `--decoder-first-pipeline-num-layers 14` 和 `--decoder-last-pipeline-num-layers 16`，中间 6 个 stage 各 `(78-14-16)/6 = 8` 层。各 stage 的起始全局层为 1、15、23、31、39、47、55、63，全部是 computing layer。 + +#### BF16 训练 + FP8 Rollout + +训练脚本直接在 `CKPT_ARGS` 和 `ROLLOUT_ARGS` 中写入默认路径，风格与其他示例脚本保持一致： + +```bash +CKPT_ARGS=( + --hf-checkpoint $BASE_DIR/GLM-5.2-FP8 + --ref-load $BASE_DIR/GLM-5.2_torch_dist + --load $BASE_DIR/GLM-5.2_slime + --save $BASE_DIR/GLM-5.2_slime + --save-interval 20 +) + +ROLLOUT_ARGS=( + --prompt-data $BASE_DIR/dapo-math-17k/dapo-math-17k.jsonl + ... +) +``` + +`--hf-checkpoint` 提供 SGLang rollout 所需的 FP8 权重和 tokenizer；`--ref-load` 是从 BF16 checkpoint 转换出的 Megatron torch_dist 权重。需要调试 BF16 rollout 时，可以直接把脚本里的 `--hf-checkpoint` 改成 `$BASE_DIR/GLM-5.2`。 + +#### SGLang 配置 + +rollout 侧采用 **prefill/decode (PD) 分离**:1 个 prefill engine(64 卡)+ 3 个 decode engine(192 卡)= 256 卡(必须等于 colocate 的 `rollout_num_gpus`)。每个 engine 64 卡,开 DP attention、`EP=64`(DeepEP 的 dispatch config map 只支持到 160 个 EP rank,所以单个 256 卡 engine 非法)。prefill 用 `auto` DeepEP 路径,decode 用 `low_latency` + `deep_gemm`。切分通过 `--sglang-config` YAML 配置: + +```yaml +sglang: + - name: default + server_groups: + - worker_type: prefill + num_gpus: 64 + num_gpus_per_engine: 64 + overrides: { deepep_mode: auto, ... } + - worker_type: decode + num_gpus: 192 + num_gpus_per_engine: 64 + overrides: { deepep_mode: low_latency, moe_runner_backend: deep_gemm, ... } +``` + +PD 传输走 RDMA/IB,使用 mooncake backend: + +```bash +--sglang-disaggregation-transfer-backend mooncake +--sglang-disaggregation-ib-device mlx5_100,...,mlx5_107 +``` + +其余 rollout 配置使用 FP8 KV cache 和 NSA + DeepEP backend: + +```bash +SGLANG_ARGS=( + --sglang-enable-dp-attention + --sglang-ep-size 64 + --sglang-dp-size 64 + --sglang-kv-cache-dtype fp8_e4m3 + --sglang-nsa-decode-backend flashmla_kv + --sglang-nsa-prefill-backend flashmla_sparse + --sglang-attention-backend nsa + ... +) +``` + +MTP / EAGLE speculative decoding 直接使用模型自带的 next-token-prediction 层（GLM-5.2 checkpoint 自带 MTP 层），因此不需要单独的 draft model： + +```bash +--sglang-speculative-algorithm EAGLE +--sglang-speculative-num-steps 4 +--sglang-speculative-eagle-topk 1 +--sglang-speculative-num-draft-tokens 5 +``` + +`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` 需要覆盖最大的 decode batch:`max cuda_graph_max_bs (decode 组 = 12) * speculative_num_draft_tokens (5) = 60`,向上取整到 `64`。低于该值会在 decode 组 CUDA graph capture 时触发 DeepEP low-latency dispatch buffer 的断言。 + +#### 网络 + +DeepEP/NVSHMEM 的跨节点通信需要在 Ray runtime env 中配置 IB 相关的 NCCL 参数（`NCCL_SOCKET_IFNAME`、`NCCL_IB_*`、`NCCL_NET_GDR_LEVEL`、`NCCL_P2P_LEVEL=NVL`、`NCCL_NVLS_ENABLE=0`、`MC_IB_PCI_RELAXED_ORDERING` 等）。脚本默认使用 `SOCKET_IFNAME=eth0`，如环境不同可在启动前设置 `SOCKET_IFNAME`，它会同时写入 `GLOO_SOCKET_IFNAME`、`TP_SOCKET_IFNAME` 和 `NCCL_SOCKET_IFNAME`。DeepEP 还要求设置 `NVSHMEM_DISABLE_NCCL=1`。 diff --git a/docs/zh/get_started/quick_start.md b/docs/zh/get_started/quick_start.md index 1daa0a502a..345a5d10be 100644 --- a/docs/zh/get_started/quick_start.md +++ b/docs/zh/get_started/quick_start.md @@ -584,5 +584,6 @@ ray job submit --address="http://127.0.0.1:8265" \ slime 针对大规模混合专家（MoE）模型的分布式训练进行了深度优化。我们提供了一些端到端的训练案例以供参考： - [示例：8xH100 训练 GLM-4.7-Flash](../examples/glm4.7-30B-A3B.md) +- [示例：32xH100 训练 GLM-5.2](../examples/glm5.2-744B-A40B.md) - [示例：64xH100 训练 GLM-4.7](../examples/glm4.7-355B-A32B.md) - [示例：128xH100 训练 DeepSeek-R1](../examples/deepseek-r1.md) diff --git a/docs/zh/index.rst b/docs/zh/index.rst index edcd8be376..7075a28b84 100644 --- a/docs/zh/index.rst +++ b/docs/zh/index.rst @@ -13,7 +13,7 @@ slime 的设计目标，是让这两大能力彼此强化，同时避免把系 为什么这个设计重要 ------------------ -- **经过 frontier model 训练验证**：slime 是 `GLM-5.1 `_、`GLM-5 `_、`GLM-4.7 `_、`GLM-4.6 `_、`GLM-4.5 `_ 背后的 RL 训练框架。 +- **经过 frontier model 训练验证**：slime 是 `GLM-5.2 `_、`GLM-5.1 `_、`GLM-5 `_、`GLM-4.7 `_、`GLM-4.6 `_、`GLM-4.5 `_ 背后的 RL 训练框架。 - **从设计开始就是 native**：slime 直接透传 Megatron 参数，并通过 ``--sglang-`` 前缀暴露当前安装版本 SGLang 支持的参数。新的上游训练和 serving 优化可以直接使用，不需要在 slime 里再加一层 wrapper。 - **专注 SGLang rollout**：slime 有意选择单一 rollout backend，避免为了同时兼容多个 inference engine 而被迫抽象成 lowest-common-denominator 的公共能力子集，从而可以直接发挥 SGLang-specific 的 serving、routing、caching、disaggregation 和 weight-sync 能力。 - **Agentic workflow 就是数据生成**：tool use、sandbox interaction、verifier reward、environment feedback、multi-agent loop 和 long-horizon agentic workflow 都接入同一条 training / rollout / Data Buffer 路径，而不是 fork training kernel。 @@ -67,6 +67,7 @@ slime 的设计目标，是让这两大能力彼此强化，同时避免把系 examples/glm4.7-30B-A3B.md examples/qwen3-30B-A3B.md + examples/glm5.2-744B-A40B.md examples/glm4.7-355B-A32B.md examples/deepseek-r1.md diff --git a/scripts/low_precision/run-kimi-k2-Thinking-int4.sh b/scripts/low_precision/run-kimi-k2-Thinking-int4.sh index 7a89241a47..c11779e037 100644 --- a/scripts/low_precision/run-kimi-k2-Thinking-int4.sh +++ b/scripts/low_precision/run-kimi-k2-Thinking-int4.sh @@ -159,6 +159,7 @@ RUNTIME_ENV_JSON="{ \"env_vars\": { \"PYTHONPATH\": \"/root/Megatron-LM/\", \"CUDA_DEVICE_MAX_CONNECTIONS\": \"1\", + \"NVSHMEM_DISABLE_NCCL\": \"1\", \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\", \"NCCL_TIMEOUT_MS\":\"360000000\", \"no_proxy\": \"${no_proxy}\", diff --git a/scripts/low_precision/run-moonlight-16B-A3B-int4.sh b/scripts/low_precision/run-moonlight-16B-A3B-int4.sh index 55026c9999..2218df3cbe 100644 --- a/scripts/low_precision/run-moonlight-16B-A3B-int4.sh +++ b/scripts/low_precision/run-moonlight-16B-A3B-int4.sh @@ -141,6 +141,7 @@ RUNTIME_ENV_JSON="{ \"env_vars\": { \"PYTHONPATH\": \"/root/Megatron-LM/\", \"CUDA_DEVICE_MAX_CONNECTIONS\": \"1\", + \"NVSHMEM_DISABLE_NCCL\": \"1\", \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\", \"OPEN_TRAINING_INT4_FAKE_QAT_FLAG\": \"1\", \"OPEN_TRAINING_INT4_GROUP_SIZE\": \"128\" diff --git a/scripts/low_precision/run-qwen3-235B-A22B-int4.sh b/scripts/low_precision/run-qwen3-235B-A22B-int4.sh index abcb2a97d7..6d41eabd85 100644 --- a/scripts/low_precision/run-qwen3-235B-A22B-int4.sh +++ b/scripts/low_precision/run-qwen3-235B-A22B-int4.sh @@ -143,6 +143,7 @@ RUNTIME_ENV_JSON="{ \"env_vars\": { \"PYTHONPATH\": \"/root/Megatron-LM/\", \"CUDA_DEVICE_MAX_CONNECTIONS\": \"1\", + \"NVSHMEM_DISABLE_NCCL\": \"1\", \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\", \"NCCL_TIMEOUT_MS\":\"360000000\", \"no_proxy\": \"${no_proxy}\", diff --git a/scripts/low_precision/run-qwen3-30B-A3B-int4.sh b/scripts/low_precision/run-qwen3-30B-A3B-int4.sh index 1770eef9f9..30cbcb5b84 100644 --- a/scripts/low_precision/run-qwen3-30B-A3B-int4.sh +++ b/scripts/low_precision/run-qwen3-30B-A3B-int4.sh @@ -139,6 +139,7 @@ RUNTIME_ENV_JSON="{ \"env_vars\": { \"PYTHONPATH\": \"/root/Megatron-LM/\", \"CUDA_DEVICE_MAX_CONNECTIONS\": \"1\", + \"NVSHMEM_DISABLE_NCCL\": \"1\", \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\", \"OPEN_TRAINING_INT4_FAKE_QAT_FLAG\": \"1\", \"OPEN_TRAINING_INT4_GROUP_SIZE\": \"128\" diff --git a/scripts/low_precision/run-qwen3-30b-a3b-fp8.sh b/scripts/low_precision/run-qwen3-30b-a3b-fp8.sh index cdce4a3b1b..402076e64b 100644 --- a/scripts/low_precision/run-qwen3-30b-a3b-fp8.sh +++ b/scripts/low_precision/run-qwen3-30b-a3b-fp8.sh @@ -155,6 +155,7 @@ RUNTIME_ENV_JSON="{ \"env_vars\": { \"PYTHONPATH\": \"/root/Megatron-LM/\", \"CUDA_DEVICE_MAX_CONNECTIONS\": \"1\", + \"NVSHMEM_DISABLE_NCCL\": \"1\", \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\", \"NVTE_FP8_BLOCK_SCALING_FP32_SCALES\": \"1\", \"NCCL_TIMEOUT_MS\":\"36000000\" diff --git a/scripts/models/glm5.2-744B-A40B.sh b/scripts/models/glm5.2-744B-A40B.sh new file mode 100644 index 0000000000..bd39c93594 --- /dev/null +++ b/scripts/models/glm5.2-744B-A40B.sh @@ -0,0 +1,62 @@ +MOE_ROUTED_EXPERTS=256 +MOE_ACTIVE_ROUTED_EXPERTS=8 +MOE_SHARED_EXPERTS=1 + +NHIDDEN=6144 +MOE_FFN_HIDDEN=2048 +MOE_SHARED_EXPERT_INTERMEDIATE_SIZE=$(($MOE_FFN_HIDDEN * $MOE_SHARED_EXPERTS)) +FFN_HIDDEN=12288 +N_DENSE_LAYERS=3 +N_MOE_LAYERS=75 +NHEADS=64 + +# GLM-5.2 744B-A40B with DSA *cross-layer index sharing*. +# Only the computing layers (layer 1,2,3,7,11,...,75 in Megatron 1-indexing) +# carry indexer weights and compute the sparse top-k; the remaining layers reuse +# the most recent computing layer's indices. The schedule (index_topk_freq=4, +# index_skip_topk_offset=3) is read from the HF config by the shared glm5 provider +# (cross-layer sharing activates automatically when index_topk_freq > 1). +MODEL_ARGS=( + --spec "slime_plugins.models.glm5.glm5" "get_glm5_spec" + --moe-layer-freq [0]*$N_DENSE_LAYERS+[1]*$N_MOE_LAYERS + --num-experts $MOE_ROUTED_EXPERTS + --moe-shared-expert-intermediate-size $MOE_SHARED_EXPERT_INTERMEDIATE_SIZE + --moe-router-topk $MOE_ACTIVE_ROUTED_EXPERTS + --moe-grouped-gemm + --moe-permute-fusion + --moe-ffn-hidden-size $MOE_FFN_HIDDEN + --moe-router-score-function sigmoid + --moe-router-pre-softmax + --moe-router-enable-expert-bias + --moe-router-bias-update-rate 0 + --moe-router-load-balancing-type seq_aux_loss + --moe-router-topk-scaling-factor 2.5 + --moe-aux-loss-coeff 0 + --moe-router-dtype fp32 + --make-vocab-size-divisible-by 16 + --num-layers $((N_DENSE_LAYERS + N_MOE_LAYERS)) + --hidden-size $NHIDDEN + --ffn-hidden-size $FFN_HIDDEN + --num-attention-heads $NHEADS + --disable-bias-linear + --swiglu + --untie-embeddings-and-output-weights + --position-embedding-type rope + --no-position-embedding + --normalization RMSNorm + --qk-layernorm + --multi-latent-attention + --q-lora-rank 2048 + --kv-lora-rank 512 + --qk-head-dim 192 + --v-head-dim 256 + --kv-channels 192 + --qk-pos-emb-head-dim 64 + --vocab-size 154880 + --rotary-base 8000000 + --enable-experimental + + # DSA + context parallel uses the sequential allgather-CP layout (not zigzag); + # the index-share provider gathers index_k/kv across the CP group to match. + --allgather-cp +) diff --git a/scripts/run-deepseek-r1.sh b/scripts/run-deepseek-r1.sh index b2a4d824e9..8c4c7de0ea 100644 --- a/scripts/run-deepseek-r1.sh +++ b/scripts/run-deepseek-r1.sh @@ -161,6 +161,7 @@ ray job submit --address="http://127.0.0.1:8265" \ "MASTER_ADDR": "${MASTER_ADDR}", "PYTHONPATH": "/root/Megatron-LM/", "CUDA_DEVICE_MAX_CONNECTIONS": "1", + "NVSHMEM_DISABLE_NCCL": "1", "LD_LIBRARY_PATH": "/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/sgl-workspace/nvshmem/install/lib/" } }' \ diff --git a/scripts/run-glm4.7-30B-A3B.sh b/scripts/run-glm4.7-30B-A3B.sh index bec6bfc492..62c093c6e0 100644 --- a/scripts/run-glm4.7-30B-A3B.sh +++ b/scripts/run-glm4.7-30B-A3B.sh @@ -154,6 +154,7 @@ RUNTIME_ENV_JSON="{ \"env_vars\": { \"PYTHONPATH\": \"/root/Megatron-LM/\", \"CUDA_DEVICE_MAX_CONNECTIONS\": \"1\", + \"NVSHMEM_DISABLE_NCCL\": \"1\", \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\" } }" diff --git a/scripts/run-glm4.7-355B-A32B.sh b/scripts/run-glm4.7-355B-A32B.sh index f9e98d7fb9..5c8a289298 100644 --- a/scripts/run-glm4.7-355B-A32B.sh +++ b/scripts/run-glm4.7-355B-A32B.sh @@ -177,6 +177,7 @@ RUNTIME_ENV_JSON=$(cat </dev/null | grep -o 'NV[0-9][0-9]*' | wc -l) +if [ "$NVLINK_COUNT" -gt 0 ]; then + HAS_NVLINK=1 +else + HAS_NVLINK=0 +fi +echo "HAS_NVLINK: $HAS_NVLINK (detected $NVLINK_COUNT NVLink references)" + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)" +source "${SCRIPT_DIR}/models/glm5.2-744B-A40B.sh" + +if [ -z "${BASE_DIR:-}" ]; then + echo "BASE_DIR is not set. Please set it to a shared path visible from every node." + exit 1 +fi + +SOCKET_IFNAME=${SOCKET_IFNAME:-eth0} + +CKPT_ARGS=( + --hf-checkpoint $BASE_DIR/GLM-5.2-FP8 + --ref-load $BASE_DIR/GLM-5.2_torch_dist + --load $BASE_DIR/GLM-5.2_slime + --save $BASE_DIR/GLM-5.2_slime + --save-interval 20 +) + +ROLLOUT_ARGS=( + --prompt-data $BASE_DIR/dapo-math-17k/dapo-math-17k.jsonl + --input-key prompt + --label-key label + --apply-chat-template + --rollout-shuffle + + --rm-type deepscaler + + --num-rollout 3000 + --rollout-batch-size 8 + --n-samples-per-prompt 8 + --rollout-max-response-len 65536 + --rollout-temperature 1.0 + + --global-batch-size 64 +) + +# TP=4, PP=8, CP=8 consumes all 256 GPUs (32 nodes) for one training group; DP=1. +# Experts use EP=32: expert_tp(1) * ep(32) * pp(8) = 256 = world_size (expert_dp=1). +# +# DSA cross-layer index sharing requires every pipeline stage to START on a +# "computing" layer (index_topk_freq=4, index_skip_topk_offset=3 -> computing +# layers are 1,2,3,7,11,...,75). A uniform 78/8 split would start stages on skip +# layers and fail. We instead use first=14, last=16, leaving 6 middle stages of +# (78-14-16)/6 = 8 layers each. Stage starts land on global layers +# 1,15,23,31,39,47,55,63 -- all computing layers. +PERF_ARGS=( + --tensor-model-parallel-size 4 + --sequence-parallel + --pipeline-model-parallel-size 8 + --decoder-first-pipeline-num-layers 14 + --decoder-last-pipeline-num-layers 16 + --context-parallel-size 8 + --expert-model-parallel-size 32 + --expert-tensor-parallel-size 1 + + --recompute-granularity full + --recompute-method uniform + --recompute-num-layers 1 + + --use-dynamic-batch-size + --max-tokens-per-gpu 8192 + --data-pad-size-multiplier 1024 + --log-probs-chunk-size 16384 +) + +GRPO_ARGS=( + --advantage-estimator grpo + --kl-loss-coef 0.00 + --kl-loss-type low_var_kl + --kl-coef 0.00 + --entropy-coef 0.00 + --eps-clip 0.2 + --eps-clip-high 0.28 + + --use-tis + --tis-clip-low 0.5 + --tis-clip 2.0 +) + +OPTIMIZER_ARGS=( + --optimizer adam + --lr 1e-6 + --lr-decay-style constant + --weight-decay 0.1 + --adam-beta1 0.9 + --adam-beta2 0.98 + --optimizer-cpu-offload + --overlap-cpu-optimizer-d2h-h2d + --use-precision-aware-optimizer +) + +WANDB_ARGS=( + # --use-wandb + # --wandb-project slime-dev + # --wandb-group glm5.2-744B-A40B +) + +SGLANG_CONFIG_FILE=$(mktemp /tmp/sglang_glm52_744B_A40B_XXXXXX.yaml) +# PD disaggregation: 1 prefill engine (64 GPU) + 3 decode engines (192 GPU) = 256. +# Each engine spans 64 GPUs (EP=64, within DeepEP's supported rank set). Prefill +# uses the auto DeepEP path; decode uses low_latency + deep_gemm for throughput. +cat > "${SGLANG_CONFIG_FILE}" <