[BugFix][310P] Fall back for unsupported penalty kernels

[UniversePeak]

What this PR does / why we need it?

Fixes #15161. Triton-Ascend penalty kernels are not supported on Ascend 310P, but vllm-ascend v0.23.0 selected them whenever Triton was available. This causes requests using presence_penalty/frequency_penalty/repetition_penalty to fail during sampling.

The regular and speculative samplers now use the upstream vLLM penalty implementation on 310P while retaining the Triton path on other devices.

Does this PR introduce any user-facing change?

Yes. Presence, frequency, and repetition penalties no longer route to unsupported Triton kernels on 310P.

How was this patch tested?

• Added tests/ut/_310p/sample/test_penalties_310.py covering regular and speculative sampler routing.
• ruff check --fix vllm_ascend/sample/sampler.py vllm_ascend/sample/rejection_sampler.py tests/ut/_310p/sample/test_penalties_310.py
• ruff format --check vllm_ascend/sample/sampler.py vllm_ascend/sample/rejection_sampler.py tests/ut/_310p/sample/test_penalties_310.py
• python -m py_compile vllm_ascend/sample/sampler.py vllm_ascend/sample/rejection_sampler.py tests/ut/_310p/sample/test_penalties_310.py
• git diff --check

The full pytest environment could not load because the available host has vllm 0.26.0 without the repository-required vllm._C, uvloop, and torch_npu dependencies; 310P hardware validation is left to CI/maintainer hardware.

---

Signed-off-by: UniversePeak UniversePeak@users.noreply.github.com

• vLLM main: vllm-project/vllm@84030bb

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a compatibility issue where Triton-Ascend penalty kernels were incorrectly invoked on Ascend 310P hardware, leading to sampling failures. By updating the sampler and rejection sampler logic to detect 310P devices, the system now correctly falls back to the standard vLLM penalty implementation, ensuring stability for users of these devices while preserving optimized performance elsewhere.

Highlights

• Penalty Kernel Compatibility: Disabled Triton-Ascend penalty kernels on Ascend 310P devices to prevent runtime failures during sampling.
• Fallback Mechanism: Implemented a fallback to the upstream vLLM penalty implementation for 310P hardware while maintaining existing Triton paths for other devices.
• Testing: Added comprehensive unit tests in tests/ut/test_penalties_310p.py to verify correct sampler routing on 310P.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

---

Tip

💡 Consider Linking a Related Issue or RFC

Your PR title contains the [BugFix] tag, indicating a bug fix or new feature.

Linking a related issue or RFC in the PR description is strongly encouraged — it gives reviewers helpful context and speeds up the review. You can use any of these keywords:

• Fixes #<issue_number>
• Closes #<issue_number>
• Resolves #<issue_number>
• Refs #<rfc_or_issue_number> (for RFCs)

🙏 Thanks for helping us keep the project well-organized!

[gemini-code-assist]

Code Review

Suggested PR Title:

[Ops][BugFix] Fallback to vLLM penalties on 310P and fix rejection sampler shape mismatch

Suggested PR Summary:

### What this PR does / why we need it?
This pull request introduces a fallback to the default vLLM penalty implementation on Ascend 310P devices because Triton-Ascend penalty kernels are not supported on this hardware. This fallback is implemented in both the standard sampler and the rejection sampler.

However, a critical issue was identified in the rejection sampler fallback: calling `Sampler.apply_penalties` directly with the original `sampling_metadata` will cause a `RuntimeError` due to shape mismatch. In speculative decoding, the `logits` tensor is expanded, but the penalty tensors are not. To resolve this, the penalty tensors in `sampling_metadata` must be repeated using `repeat_indices` before invoking the fallback.

### Does this PR introduce _any_ user-facing change?
No, this is an internal fallback mechanism for Ascend 310P devices to ensure compatibility and prevent execution failures.

### How was this patch tested?
The changes are covered by new unit tests in `tests/ut/test_penalties_310p.py` which mock the device type and Triton availability to verify that the fallback is correctly triggered.

[gemini-code-assist]

In speculative decoding (rejection sampler), the logits tensor has an expanded shape of [num_tokens + batch_size, vocab_size] to account for all draft and bonus tokens, whereas the penalty tensors in sampling_metadata (such as presence_penalties, frequency_penalties, etc.) have a shape of [batch_size].\n\nDirectly calling Sampler.apply_penalties(logits, sampling_metadata, output_token_ids) without repeating/expanding the penalty tensors using repeat_indices will result in a guaranteed runtime RuntimeError due to shape mismatch on 310P devices.\n\nTo fix this, we should create a copy of sampling_metadata with the penalty tensors repeated by repeat_indices before passing it to Sampler.apply_penalties. This also allows us to update the warning message to accurately reflect that Triton is unsupported on 310P (matching the updated warning in sampler.py).\n\nHere is the suggested fix for this block:\n\npython\n if not HAS_TRITON or is_310p():\n logger.warning_once(\n "[sample/rejection_sampler] Triton unavailable or unsupported on 310P, falling back to vLLM default "\n "penalty implementation in rejection sampler. Rejection sampling performance "\n "may be degraded on NPU. "\n )\n repeated_sampling_metadata = replace(\n sampling_metadata,\n prompt_token_ids=sampling_metadata.prompt_token_ids[repeat_indices] if sampling_metadata.prompt_token_ids is not None else None,\n presence_penalties=sampling_metadata.presence_penalties[repeat_indices],\n frequency_penalties=sampling_metadata.frequency_penalties[repeat_indices],\n repetition_penalties=sampling_metadata.repetition_penalties[repeat_indices],\n )\n return Sampler.apply_penalties(logits, repeated_sampling_metadata, output_token_ids)\n

[UniversePeak]

The code and pre-commit checks are passing, but the current ci-gate failure is a repository workflow gate: selected tests are present and the job requires ready-precise or ready-all before it will run them. Could a maintainer add the appropriate ready label and rerun the gate? The exact failed job log is https://github.com/vllm-project/vllm-ascend/actions/runs/33196293083/job/98936133887

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[UniversePeak]

Resolved the merge conflict by rebasing the branch onto the latest vllm-ascend/main and pushed the updated branch. The 310P penalty fallback and speculative-decoding regression coverage are unchanged; git diff --check, py_compile, and focused Ruff checks pass locally. The repository checks are now running again, with the remaining ready-precise/ready-all CI gate still requiring maintainer labeling.

[UniversePeak]

Correction to my previous update: the branch was updated by merging the latest vllm-ascend/main, not by rebasing. The resulting PR head is 752e272a; the merge conflict is resolved and the repository checks have been restarted. The local focused checks remain green, while the ready-precise/ready-all gate still requires maintainer labeling.

[UniversePeak]

The latest workflow run for the current head 67a95fa still stops at ci-gate before the selected test matrix starts. The gate reports that this source change needs a maintainer label: ready-precise (recommended) or ready-all; the pre-commit, test recommendation, and test selection jobs passed, while the test jobs remain pending. Could a maintainer add ready-precise so the selected tests can run? https://github.com/vllm-project/vllm-ascend/actions/runs/33719490939

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[UniversePeak]

Rebased onto the latest main (cbc9d3b) via a fresh merge (e7a542da) after the base moved past our previous anchor; the only conflict was the HardwareCapability enum in vllm_ascend/device/hardware_profile.py (kept both the new upstream MLAPO comment and our TRITON_PENALTIES entry), and the local pre-commit hooks (ruff, codespell, typos, meta-kernel shape checks) pass on the merged file.

The PR is mergeable again. Since ci-gate still requires the ready-precise label to run the test matrix, could a maintainer please add it when convenient? Happy to address any review feedback in the meantime.

[UniversePeak]

The E2E run exposed a real bug in my fallback: Sampler.apply_penalties is also called in _rejection_sample (speculative decoding path), where sampling_metadata is a SimpleNamespace-style container rather than a dataclass, so the previous dataclasses.replace-based expansion raised TypeError: replace() should be called on dataclass instances.

Fixed in 68a96e3: the metadata is now cloned by re-instantiating the container with expanded penalty tensors (falling back to the original metadata when no expansion is needed), so the same fix now covers both the regular sampling path and the rejection sampler path. Focused checks pass locally; the PR branch and fork copy are in sync.

[UniversePeak]

The CPU unit test failure is fixed by 68a96e3: on the latest E2E run, pre-commit and cpu-ut are both green (https://github.com/vllm-project/vllm-ascend/actions/runs/35507518895). ci-gate still fails only because the branch needs a maintainer label — could someone please add ready-precise so the selected tests can run? Happy to address any review feedback meanwhile.