docs: align vLLM docs with current defaults - #240
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the documentation to reflect changes in supported GPU platforms (adding GB200/GB300 and A100), simplifies the training backend documentation to focus on Megatron-LM, and updates the default routing policy to consistent_hash. It also removes the EPLB configuration section and updates multi-node training instructions. Feedback highlights that hardcoding the English language in the Read the Docs configuration prevents automatic translation builds (such as Chinese), and suggests explicitly marking consistent_hash as the default policy in the advanced configuration guides for consistency.
| - cd docs | ||
| - mkdir -p "$READTHEDOCS_OUTPUT/html" | ||
| - VIME_DOC_LANG=en sphinx-build -b html -D language=en --conf-dir ./ ./en "$READTHEDOCS_OUTPUT/html" | ||
| - VIME_DOC_LANG=en sphinx-build -b html -D language=en -c docs docs/en "$READTHEDOCS_OUTPUT/html" |
There was a problem hiding this comment.
Hardcoding en and docs/en in the build command prevents Read the Docs from building translations (such as the Chinese version in docs/zh) automatically. Using the standard READTHEDOCS_LANGUAGE environment variable allows Read the Docs to build the correct language version dynamically based on the project configuration.
- VIME_DOC_LANG=${READTHEDOCS_LANGUAGE:-en} sphinx-build -b html -D language=${READTHEDOCS_LANGUAGE:-en} -c docs docs/${READTHEDOCS_LANGUAGE:-en} "$READTHEDOCS_OUTPUT/html"| --router-policy round_robin # 简单轮询 | ||
| --router-policy consistent_hash # 多轮会话亲和 | ||
| --router-policy cache_aware # 缓存感知路由（默认） | ||
| --router-policy cache_aware # 缓存感知路由 |
There was a problem hiding this comment.
Since consistent_hash is now the default router policy, it would be helpful to also update line 292 to explicitly mark it as （默认） for clarity and consistency with the updated Chinese usage guide.
There was a problem hiding this comment.
I think it's actually the same:
--router-policy consistent_hash # Session affinity for multi-turn (default)
--router-policy cache_aware # Cache-aware routing
d45a3d3 to
c329322
Compare
|
Says docs-only, but tools/profile_rollout.py changed. Can we split or update the PR description? |
Squash to 2–3 commits: (1) doc content, (2) assets/badges, (3) tools. |
| print(f"Starting profile on {worker_url} for {args.num_steps} steps...") | ||
| response = requests.post(f"{worker_url}/start_profile", json=payload) | ||
| print(f"Starting profile on {worker_url}...") | ||
| response = requests.post(f"{worker_url}/start_profile", json={}) |
There was a problem hiding this comment.
Does vLLM actually ignore the old payload?
| parser.add_argument("--router-url", type=str, required=True, help="Router URL (e.g., http://127.0.0.1:3000)") | ||
| parser.add_argument("--action", type=str, choices=["start", "stop"], default="start", help="Action to perform") | ||
| parser.add_argument("--output-dir", type=str, default="/tmp/vllm_profile", help="Output directory for traces") | ||
| parser.add_argument("--num-steps", type=int, default=3, help="Number of steps to profile (default: 3)") |
There was a problem hiding this comment.
Anyone still passing those flags will break silently — did we check?
There was a problem hiding this comment.
No, it will not breaks but just silently do the start profile.
| - vLLM 默认会从 huggingface ckpt 中 `config.json` 读取模型的最大 context length，可以使用 `--vllm-max-model-len` 参数来对这个值进行覆盖，从而支持进行更长的推理； | ||
| - 在训推一体的训练过程中，虽然 megatron 和 vLLM 会先后 offload，但是还是需要为对方留有一些空间，需要通过减小 `--vllm-gpu-memory-utilization` 来调整 vLLM 的显存占用总量。 | ||
| - vime 支持透传 vllm-router 的参数，方式是在原参数名前加上 `router` 前缀。例如，vllm-router 的 `--balance-abs-threshold` 参数需要设置为 `--router-balance-abs-threshold`。由于 vllm-router 默认使用 cache-aware routing，可能会导致请求分配不均衡的问题。可以通过设置 `--router-balance-abs-threshold 0` 来强制均衡分配，但这可能会影响多轮对话场景下 prefix cache 的命中率。 | ||
| - vime 支持透传 vllm-router 的参数，方式是在原参数名前加上 `router` 前缀。例如，vllm-router 的 `--balance-abs-threshold` 参数需要设置为 `--router-balance-abs-threshold`。vime 默认使用 `consistent_hash` 路由策略。暂时不支持 cache-aware routing。可以通过设置 `--router-balance-abs-threshold 0` 来强制均衡分配，但这可能会影响多轮对话场景下 prefix cache 的命中率。 |
There was a problem hiding this comment.
cache-aware routing is not supported for now — but --router-policy cache_aware is still in argparse choices and vllm-config examples.
| --router-policy round_robin # 简单轮询 | ||
| --router-policy consistent_hash # 多轮会话亲和 | ||
| --router-policy cache_aware # 缓存感知路由（默认） | ||
| --router-policy cache_aware # 缓存感知路由 |
|
|
||
| #### EPLB | ||
|
|
||
| 当总卡数并不能被 expert 总数整除时，可以开启 vLLM 的 EPLB（Expert Parallelism Load Balancer），通过 `--vllm-eplb-config` 配置冗余 expert。例如对于 24 卡的场景： |
There was a problem hiding this comment.
silent delete
"EPLB still works via --vllm-enable-eplb" or link to vLLM docs; or explain why example is outdated.
| **vime** 支持多种 NVIDIA GPU 硬件平台： | ||
|
|
||
| - **B200 系列**：完全支持，运行步骤与 H 系列完全相同 | ||
| - **GB200 / GB300 / B200 / 300 系列**：完全支持，运行步骤与 H 系列完全相同 |
There was a problem hiding this comment.
what is 300 Series? B300?
| - B 卡基本功能稳定，可作为开发和测试参考，但暂无 CI 保护 | ||
| - 两种硬件平台使用完全相同的安装和启动流程 | ||
|
|
||
| - 对于不方便使用 docker 的场景，请参考 [build_conda.sh](https://github.com/vllm-project/vime/blob/main/build_conda.sh)。 |
There was a problem hiding this comment.
Why remove the non-Docker path? GB guard doesn't replace conda install.
There was a problem hiding this comment.
We don't have conda for now, also I don't think vllm has a conda release.
|
|
||
| - 对于不方便使用 docker 的场景，请参考 [build_conda.sh](https://github.com/vllm-project/vime/blob/main/build_conda.sh)。 | ||
| **GB Support Guard**： | ||
| - 对于 GB 系列训练，请将 `--actor-num-gpus-per-node` 设为 `4`。 |
There was a problem hiding this comment.
Do we need GB-specific notes in quick start, or in a hardware/advanced doc?
There was a problem hiding this comment.
I would just remove it and then add it to a gb200 glm3.5 106B example.
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
355e043 to
6ef4146
Compare
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
CI override（maintainer approved）Decision: Merge with failing
|
2 participants
Update docs for current vLLM defaults and example guidance:\n\n- remove outdated EPLB section from Qwen3-30B-A3B example docs\n- update supported GPU wording in quick start\n- clarify multi-node Ray/Kubernetes wording\n- update training backend wording\n- note consistent_hash as the default router policy in usage docs and remove the old default note from vllm-config docs\n\nThis PR is docs-only.