Repository navigation
Conversation
#9443 wraps MoE runners eagerly and publishes a no-adapter (slot-0) mapping before the first forward so LoRA-typed MoE kernels have a context on HunyuanImage-3's baseline generate() path (#9232). On GPU that is free: punica sets no_lora_flag for slot 0 and the TRTLLM / Triton LoRA experts return early. Ascend's unquantized MoE path instead branches on whether a context is published at all, not on whether any token selects an adapter: vllm_ascend/ops/fused_moe/routed_experts.py runs the argsort-based routing recovery plus moe_lora_apply_w13/w2 under `if lora_context is not None`, token_dispatcher.py adds an all_to_all of the LoRA indices per layer, and vllm_ascend/lora/punica_npu.py:add_lora_fused_moe has no no_lora short-circuit, so bgmv shrink/expand still launch over every row for a zero delta. Measured on A3 (910) HunyuanImage-3 nightly perf, same nodes and day: with the idle context published, e2e is 7.90 / 8.25 / 8.44 s and HunyuanImage3Pipeline.model.forward 6.31 / 6.53 / 6.67 ms, against a pre-#9443 baseline of 5.43 / 5.70 / 5.81 s and 3.79 / 3.99 / 4.13 ms (tp2_cfgp2 / tp2_sp2 / tp4). H200 is unaffected. Keep the eager wrapping and the GPU publish, and detach the context while no adapter is active on runners that expose set_lora_context(). The hook is attached by capability, so GPU wrappers, which have no such entry point and whose experts assert the context exists, are unchanged. Activation, re-arm and rollback all republish through _publish_moe_lora_context, so an active adapter still gets a real context; deactivation now also releases the cost instead of keeping it published. Signed-off-by: tzhouam <tzhouam@connect.ust.hk> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes the HunyuanImage-3 NPU performance regression introduced by #9443 (reached
release/v0.32.0through batch-7 backport #9470) without reverting it, so #9232 stays fixed.#9443 wraps MoE runners eagerly and publishes a no-adapter (slot-0) mapping before the first forward, so LoRA-typed MoE kernels have a context on HunyuanImage-3's baseline
generate(). On GPU this is free: punica setsno_lora_flagfor slot 0 and the TRTLLM / Triton LoRA experts return early.Ascend's unquantized MoE path instead branches on whether a context is published at all, not on whether any token selects an adapter:
vllm_ascend/ops/fused_moe/routed_experts.pyruns theargsort-based routing recovery plusmoe_lora_apply_w13/moe_lora_apply_w2underif lora_context is not None:vllm_ascend/ops/fused_moe/token_dispatcher.pyadds anall_to_allof the LoRA indices per layer, plus permute/unpermutevllm_ascend/lora/punica_npu.py: add_lora_fused_moehas nono_lorashort-circuit, sobgmv_shrink/bgmv_expand_slicestill launch over every row for a zero delta(The Ascend quantized path already gates on
has_lora(lora_context)inops/fused_moe/moe_mlp.py; only the unquantized path keys on presence.)Change
Keep the eager wrapping and the GPU publish; detach the context while no adapter is active on runners that expose
set_lora_context(). The hook is attached by capability, so GPU wrappers — which have no such entry point and whose experts assert the context exists — are unchanged. Activation, re-arm and rollback all republish through_publish_moe_lora_context(), so an active adapter still gets a real context; deactivation now also releases the cost instead of leaving it published..buildkite/common/ci_source_file_dependencies.yml: addvllm_omni/diffusion/lora/to thehunyuan_image3anchor. HunyuanImage-3 is the MoE + PEFT model whose baseline path runs this code, and without it the HunyuanImage-3 perf/accuracy steps are filtered out of a LoRA-only diff — which is how this regression reached the release branch unnoticed.Test Plan
vLLM version: 0.32.0 Base:
release/v0.32.0@2dbc92bc0dpytest -q tests/diffusion/lora/test_lora_manager.py(two new cases: idle detaches and publishes nothing; active still publishes the slot-1 mapping)test_hunyuan_image_tp2_cfgp2,test_hunyuan_image_tp2_sp2,test_hunyuan_image_tp4Test Result
stage_0_gen(s)model.forward(ms)H200 is unaffected either way: the 10-28
mainnightly (18 commits ahead of #9443) is flat at 2189 / 2381 / 2265 ms.Follow-up filed upstream: vllm-ascend should gate its unquantized MoE path on
has_lora(lora_context)the way the quantized one already does, and/or short-circuitadd_lora_fused_moeonpunica_wrapper.no_lora. That is the general fix and also helps LLM serving batches that contain no LoRA tokens, but it needs a vllm-ascend release plus an image bump, so it cannot land for v0.32.0. The two changes compose.