Repository navigation
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
This pull request has merge conflicts that must be resolved before it can be |
8e08fb9 to
4a102e4
Compare
|
@liulanze Thanks for the reviews so far. I've rebased the code. Is there more you'd like to be addressed? |
|
This pull request has merge conflicts that must be resolved before it can be |
moe_sort_routing only ran the invalid-id remap (ids outside [0, n_expert) -> n_expert) when an expert_map or build_inverse was given. moe_permute and moe_permute_with_scratch, used by CutlassExpertsFp8 on a single EP rank, pass neither, so -1 routes for CUDA-graph padding tokens reached the sort raw. The radix sort places them after all experts while the offset scan assumes signed ascending keys, so expert_first_token_offset came out wrong and valid tokens ran through the wrong experts. Always run the remap, return early from preprocessTopkIdLauncher for an empty batch, and add a regression test for -1 routes without expert_map. Fixes vllm-project#59686 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jerry Quinn <jlquinn@us.ibm.com>
…th invalid routes With invalid routes (e.g. -1 for padding tokens) sorted past the valid rows, the grouped GEMM never writes those rows of mm1_out, but the dynamic per-tensor scale of act_out reduces over every row, so stale workspace data skewed the scale for valid tokens. run_cutlass_moe_fp8 only zeroed the workspace with an expert_map or the batched format; zero it whenever the activation scale is per-tensor. Apply the same guard to run_cutlass_moe_w4a8_fp8, and always bound the Humming scatter path's input quantization to the valid routes. Add a regression test running per-tensor CUTLASS fp8 MoE with -1 padding routes and a garbage-filled workspace against an unpadded reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jerry Quinn <jlquinn@us.ibm.com>
4a102e4 to
b5b1095
Compare
… MoE path run_cutlass_moe_w4a8_fp8 asserts per_act_token, so the `if not per_act_token` fill added there could never run. With per-token activation scales, rows the grouped GEMM leaves unwritten can't affect the valid rows' scales anyway. The fill that matters stays in run_cutlass_moe_fp8, which allows per-tensor scales. Signed-off-by: Jerry Quinn <jlquinn@us.ibm.com>
|
@bnellnm Thanks. Can you add the |
This branch has not been deployed
3 participants
Overview
Fixes #59686.
moe_permutereturns wrongexpert_first_token_offsetwhentopk_idscontains-1(the id the router writes for CUDA-graph padding tokens withVLLM_MOE_SKIP_PADDING=1, the default) and noexpert_mapis given. The CUTLASS fp8 MoE path then runs some valid tokens through the wrong expert. This always applies the existing invalid-id remap inmoe_sort_routing.Claims
CutlassExpertsFp8(and the othermoe_permutecallers:cutlass_moe.pyW4A8 path, Humming non-scatter path) run without anexpert_map.VLLM_CUTLASSbackend (first-token collapse to'\n', off-topic sampled output).expert_mapis given (the remap already ran there).Validation
H100 (SM 9.0), vLLM
main47de9d4d04,_moe_C_stable_libtorchrebuilt for sm_90 with CUDA 13.1; unpatched = the precompiled nightly extension for the same source.CutlassExpertsFp8(E=128, top-4, K=3072, N=1536, per-token act / per-channel weight fp8): sizes where valid rows differ from an unpadded calltest_moe_permute_invalid_expert_ids_without_expert_maptests/kernels/moe/test_moe_permute_unpermute.pyModel evaluation, 128-expert top-4 fp8 MoE checkpoint (compressed-tensors
FP8_DYNAMIC), TP=2, default backends (VLLM_CUTLASSMoE, CUDA graphs on). Next-token distributions vs. the bf16 checkpoint on 90 short chat prompts x thinking on/off; "foreign mass" = probability the fp8 model puts on its top-10 tokens that are outside the bf16 top-100 (0 for bf16 vs. itself).Commands:
Details
Root cause:
CubKeyValueSorterradix-sorts only the lowfloor(log2(2E-1)) + 1bits, so a raw-1(all ones) sorts after every real expert, butfindTotalEltsLessThanTargetbinary-searches the keys as signed ascending ints. Probes landing in the trailing-1s push offsets too high, so an expert's grouped-GEMM segment can cover the next expert's rows; when padding outnumbers valid tokens every offset equals the total route count and the valid MoE output is zero.preprocessTopkIdKernelalready maps ids outside[0, n_expert)to then_expertsentinel the sort expects, but it only ran with anexpert_maporbuild_inverse;moe_permute/moe_permute_with_scratchpass neither on a single EP rank.Cost: one
n_token * topkint32 copy and one elementwise kernel permoe_permutecall, no host sync, CUDA-graph safe.preprocessTopkIdLaunchernow returns early for an empty batch, since the unconditional call would otherwise computeblock = 0. This overlaps #58790 (empty routing inmoe_permute, open); whichever lands second can drop its guard.Not duplicate: open PRs on the same
-1padding sentinel target other backends (AITER #57036, FlashInfer b12x #57946, XPU #55231) or empty routing (#58790); none changes themoe_sort_routingremap gate. Searchedmoe_permute,expert_first_token_offset,topk_ids -1 padding,VLLM_MOE_SKIP_PADDING,preprocessTopkId.Limitations: validated on H100 only; the sm_90 kernel build used CUDA 13.1.
AI assistance: Claude Code was used to localize the bug, write the repros, the fix and the test, and run the validation. The human submitter reviewed every changed line and the results.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
🤖 Generated with Claude Code