Repository navigation
Conversation
VLLM_MOE_SKIP_PADDING (enabled by default) lets the topk kernels mark cudagraph padding rows by writing -1 into topk_ids, so downstream MoE kernels can skip work for them. The AITER fused-expert path never handled that sentinel and passed topk_ids straight into aiter.fused_moe, where moe_sorting indexes the expert weights with the raw id. A value of -1 reads outside the weight buffer and trips a GPU memory access fault: Memory access fault by GPU node-9 on address 0x7f255b5b2000. Reason: Unknown. This reproduces on DeepSeek-V4.1-Flash (MXFP4 experts, TP=4, MI355X): the profiling run builds a fully synthetic batch whose rows are all marked as padding, so every routing index is -1 and all ranks fault before the server can start. It is not limited to startup -- trailing cudagraph padding rows are marked the same way on real batches. Neutralize the sentinel where the other AITER dtype fixups happen: route padding rows to expert 0 with zero weight, contributing nothing to the output. The rewrite is branchless so it remains cudagraph-capturable. Signed-off-by: Xiao Yu <xiao.yu.dc@outlook.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Hi @xiao-llm thanks for your fix. Have we measured the perf-wise impact? Those extra elementwise operations can be expensive cc @shen-shanshan |
|
@xiao-llm please also take a look at this ROCm/aiter#5295 , do you think this fix can address your problem here? |
| # nothing to the output. Done branchlessly to stay cudagraph-capturable. | ||
| is_padding = topk_ids < 0 | ||
| topk_ids = torch.where(is_padding, torch.zeros_like(topk_ids), topk_ids) | ||
| topk_weights = torch.where(is_padding, torch.zeros_like(topk_weights), topk_weights) |
There was a problem hiding this comment.
These ops can be quite expensive, can you test the kernel level and e2e cost first?
There was a problem hiding this comment.
sure, will verify it
It looks like solve the same issue, will verify if it works for DeepSeek-V4.1-Flash mxfp4. |
|
@micah-wil is bumping the aiter version and it should help us to solve the problem |
|
@xiao-llm is this PR still needed? |
This branch has not been deployed
4 participants
VLLM_MOE_SKIP_PADDING (enabled by default) lets the topk kernels mark cudagraph padding rows by writing -1 into topk_ids, so downstream MoE kernels can skip work for them. The AITER fused-expert path never handled that sentinel and passed topk_ids straight into aiter.fused_moe, where moe_sorting indexes the expert weights with the raw id. A value of -1 reads outside the weight buffer and trips a GPU memory access fault:
Memory access fault by GPU node-9 on address 0x7f255b5b2000.
Reason: Unknown.
This reproduces on DeepSeek-V4.1-Flash (MXFP4 experts, TP=4, MI355X): the profiling run builds a fully synthetic batch whose rows are all marked as padding, so every routing index is -1 and all ranks fault before the server can start. It is not limited to startup -- trailing cudagraph padding rows are marked the same way on real batches.
Neutralize the sentinel where the other AITER dtype fixups happen: route padding rows to expert 0 with zero weight, contributing nothing to the output. The rewrite is branchless so it remains cudagraph-capturable.
Purpose
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.