Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ascend] add ascend graph mode #2647

Merged
merged 7 commits into from
Oct 25, 2024

Conversation

CyCle1024
Copy link
Collaborator

Motivation

Add ascend's graph mode for following models when tp=1:
Llama2, InternLM2, Qwen2

Other models aren't verified on ascend's graph mode, may or may not be worked, we show warning when using ascend's
graph mode on other models.

Modification

Add AscendGraphRunner, silu_and_mul ops and rotary_embedding ops on ascend.

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
  3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

@lvhan028 lvhan028 added the enhancement New feature or request label Oct 25, 2024
@lvhan028 lvhan028 requested a review from grimoire October 25, 2024 06:18
@lvhan028 lvhan028 mentioned this pull request Oct 25, 2024
@lvhan028 lvhan028 merged commit 44a0cd3 into InternLM:main Oct 25, 2024
5 checks passed
AllentDan pushed a commit to AllentDan/lmdeploy that referenced this pull request Nov 13, 2024
* [pytorch] ascend enable atbgraph

* add paged prefill attention

* refine ascend-update-step-ctx (InternLM#26)

refine ascend-update-step-ctx

---------

Co-authored-by: CyCle1024 <[email protected]>

* fix: rewrite enable graph for ascend

* fix backend error due to folder refactor

* remove unnecessary comment

* fix rotary_embedding (InternLM#27)

---------

Co-authored-by: jinminxi104 <[email protected]>
Co-authored-by: tangzhiyi11 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants