Repository navigation
[Misc]feat: adapt to vLLM main (54503ece) - #12372
vllm-ascend-ci wants to merge 1 commit into
Conversation
Signed-off-by: main2main-bot <main2main-bot@users.noreply.github.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request upgrades the project's vLLM dependency to a newer commit and introduces necessary adaptations to support the newly added LongCat-Flash-Lite model variant. Additionally, it includes a stability fix for Hunyuan VL model processing to ensure robust input position handling. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
Suggested PR Title:
[Attention][BugFix] Add longcat_flash_ngram support and patch Hunyuan VL xdrope input positionsSuggested PR Summary:
### What this PR does / why we need it?
This PR adds support for the `longcat_flash_ngram` model type in KV transfer, speculative config, quantization, and model runner components. It also patches `get_xdrope_input_positions` for Hunyuan VL to guard against `IndexError` when `image_start_indices` exceeds `image_grid_thw` entries.
Feedback:
In `vllm_ascend/patch/hunyuan_vl_processor_compat.py`, a potential `RuntimeError` due to shape mismatch can occur if `pos + token_num` exceeds the length of `w_index`. A boundary check should be added to ensure the copy slice does not exceed the tensor bounds.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No test details provided.| token_num = (llm_grid_w + 1) * llm_grid_h | ||
| w_index[pos : pos + token_num].copy_( | ||
| torch.arange(0, llm_grid_w + 1).reshape(1, -1).expand(llm_grid_h, -1).reshape(-1) | ||
| ) | ||
| h_index[pos : pos + token_num].copy_( | ||
| torch.arange(0, llm_grid_h).reshape(-1, 1).expand(-1, llm_grid_w + 1).reshape(-1) | ||
| ) | ||
| t_index[pos : pos + token_num] = image_index |
There was a problem hiding this comment.
If pos + token_num exceeds the length of w_index (for example, if the input tokens are truncated or the image start token is near the end of the input), slicing w_index[pos : pos + token_num] will return a tensor shorter than token_num. Attempting to .copy_() a tensor of size token_num into a smaller slice will raise a RuntimeError due to shape mismatch.
To prevent this, we should calculate the actual number of elements that can be copied (copy_num) and slice the source tensors accordingly.
token_num = (llm_grid_w + 1) * llm_grid_h
copy_num = min(token_num, len(w_index) - pos)
if copy_num > 0:
w_index[pos : pos + copy_num].copy_(
torch.arange(0, llm_grid_w + 1).reshape(1, -1).expand(llm_grid_h, -1).reshape(-1)[:copy_num]
)
h_index[pos : pos + copy_num].copy_(
torch.arange(0, llm_grid_h).reshape(-1, 1).expand(-1, llm_grid_w + 1).reshape(-1)[:copy_num]
)
t_index[pos : pos + copy_num] = image_index
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
|
Superseded by #12420. |
1 participant
What this PR does / why we need it?
Upgrade vLLM commit to
54503ecevllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_connector.py,vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_layerwise_connector.py,vllm_ascend/patch/platform/patch_speculative_config.py,vllm_ascend/worker/model_runner_v1.pydue to 08dfd686num_attn_module=2like longcat_flashvllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_connector.py,vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_layerwise_connector.py,vllm_ascend/patch/hunyuan_vl_processor_compat.py,vllm_ascend/patch/platform/patch_speculative_config.py,vllm_ascend/quantization/modelslim_config.py,vllm_ascend/worker/model_runner_v1.pydue to 0b6636cbnum_attn_module=2like longcat_flashvllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_connector.py,vllm_ascend/distributed/kv_transfer/kv_p2p/mooncake_layerwise_connector.py,vllm_ascend/patch/hunyuan_vl_processor_compat.py,vllm_ascend/patch/platform/patch_speculative_config.py,vllm_ascend/quantization/modelslim_config.py,vllm_ascend/worker/model_runner_v1.pydue to 54503ecenum_attn_module=2like longcat_flash