Fixes from PR#1635 applied to v1.22.0_next branch #1647
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MohitIntel
requested review from
PatrykWo,
afierka-intel,
jikunshang,
kzawora-intel,
madamczyk-intel,
mgawarkiewicz-intel,
michalkuligowski,
mswiniarsk,
vivekgoe and
xuechendi
as code owners
MohitIntel
force-pushed
the
mdeopujari/new_rebase_PR1635
branch
from
542611e to
8e07f7f
Compare
michalkuligowski
previously requested changes
Jul 24, 2025
FusedSDPA kernel with window_size+causal only works when seq_len is multiple of SLICE_SIZE. If not, fallback to the original implementation which creates attention_mask with window_size
-Move the seq_len check for use_sdpa_window to attn_metadata -Automatically set all environment variable
Additional changes from PR1597
remove print statement
MohitIntel
force-pushed
the
mdeopujari/new_rebase_PR1635
branch
from
e01eae7 to
518dab2
Compare
Author
It has now been rebased on vllm-fork v1.22.0_next after yesterday's merge of PR1616. |
MohitIntel
dismissed
michalkuligowski’s stale review
This branch has now been rebased on latest vllm-fork v1.22.0_next after yesterdays' PR1616 got merged.
Author
|
@michalkuligowski , This is ready to merge to v1.22.0_next. |
|
/run-gaudi-tests |
michalkuligowski
requested changes
Jul 28, 2025
| setuptools>=77.0.3 | ||
| setuptools-scm>=8 | ||
| vllm-hpu-extension @ git+https://github.com/HabanaAI/vllm-hpu-extension.git@5135570 | ||
| vllm-hpu-extension @ git+https://github.com/HabanaAI/vllm-hpu-extension.git@009adb2 |
There was a problem hiding this comment.
This PR is almost identical to #1660. Please apply comments from there
|
closing this is done in #1660 |
wpyszka
pushed a commit
that referenced
this pull request
Aug 1, 2025
This PR contains following changes 1. Port Gemma3 SLIDING_WINDOW FusedSDPA feature from habana_main + Add a few extra fixes including.. - Sliding FusedSDPA kernel, we are adding threshold variable to enable or disable to use optimized kernel. This kernel will be performance/memory benefit for longer sequence. We are providing environment variable to control per customer request. - Based on the threshold, choose different prompt bucket, if it's smaller than the threshold, use PROMPT_BUCKET_STEP, otherwise use SLICE_SIZE. - Added mark_step before SLIDING FusedSDPA is run. - Misc fixes for bucket related issue. 2. upstream fixes vllm-project#18732 vllm-project#21479 vllm-project#19788 3. optimized Gemma3RMSNorm with FusedRMSNorm Dependent on #1647 Run command with. VLLM_FUSEDSDPA_SLIDE_THLD=2048 VLLM_EXPONENTIAL_BUCKETING=false VLLM_PROMPT_BS_BUCKET_MAX=64 VLLM_PROMPT_SEQ_BUCKET_STEP=1024 VLLM_PROMPT_SEQ_BUCKET_MAX=20480 PT_HPU_SDPA_QKV_SLICE_MODE_FWD=1 --------- Signed-off-by: Lukas Geiger <[email protected]> Signed-off-by: Hongmin Fan <[email protected]> Co-authored-by: Henry Tang <[email protected]> Co-authored-by: Mohit Deopujari <[email protected]> Co-authored-by: Shiv Kaul <[email protected]> Co-authored-by: Shiv Kaul <[email protected]> Co-authored-by: Libin Tang <[email protected]> Co-authored-by: Lukas Geiger <[email protected]> Co-authored-by: Hongmin Fan <[email protected]> Co-authored-by: Harish Subramony <[email protected]> Co-authored-by: Jianhong-Zhang <[email protected]> Co-authored-by: Libin Tang <[email protected]> Co-authored-by: Michał Kuligowski <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.