Repository navigation
Merge upstream v0.32.0 into release/0.32 - #57
Merged
Merged
Conversation
#9443 publishes a no-adapter (slot-0) MoE LoRA mapping before the first forward so LoRA-typed MoE kernels have a context on HunyuanImage-3's baseline generate() (fixes #9232). Only TrtLlmBf16LoRAExperts needs that while idle: apply() asserts the context exists and then returns through _batch_has_no_lora(). Every other LoRA-capable MoE kernel treats context presence as the LoRA switch and keeps working for a zero delta — Triton disables skip_invalid and still runs apply_w13_lora/apply_w2_lora ahead of the no_lora_flag return in add_lora_fused_moe (~6% of HunyuanImage-3 DiT on H100), and the Ascend unquantized path adds routing recovery, an all_to_all of the LoRA indices per layer and bgmv shrink/expand launches (40-50% e2e on A3). Keep the eager wrapping, and decide per wrapper while no adapter is active: wrappers whose experts expose _batch_has_no_lora keep set_mapping(slot 0); the rest are detached through the Ascend runner idle hook or fused_experts/prepare_finalize set_lora_context(None); anything that cannot be detached keeps the mapping rather than being left without one. Activation is unchanged, so every wrapper still gets the slot-1 mapping. NPU A3 nightly perf, build #10126 vs the pre-regression distribution of 13 nightlies (tp2_cfgp2 / tp2_sp2 / tp4): e2e 5.95 / 5.79 / 5.77 s against medians 5.43 / 5.70 / 5.81 s, model.forward 3.79 / 3.99 / 4.11 ms against 3.79 / 3.99 / 4.13 ms, all inside range; with the idle context published those were 7.90 / 8.25 / 8.44 s and 6.31 / 6.53 / 6.67 ms. Upstream follow-up: vllm-project/vllm-ascend#20078 asks for the unquantized Ascend path to gate on has_lora() as its quantized path already does. Fixes #9514 Signed-off-by: tzhouam <tzhouam@connect.ust.hk> Co-authored-by: zzh <67720593+BLANKETusers@users.noreply.github.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
release/0.32 was cut one commit before the upstream v0.32.0 tag. This merges the tag, which brings in the one missing commit:
It touches diffusion LoRA only. After the merge, v0.32.0 is an ancestor of release/0.32.
Tests:
tests/diffusion/loraon H100, 105 passed.