Add Eagle and Eagle3 support to Transformers modeling backend#30340
Add Eagle and Eagle3 support to Transformers modeling backend#30340hmellor merged 10 commits intovllm-project:mainfrom
Conversation
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for Eagle and Eagle3 speculative decoding to the Transformers modeling backend. The changes include implementing the SupportsEagle and SupportsEagle3 interfaces in the Base model class. The forward method is updated to optionally return auxiliary hidden states required by Eagle3. I've found a critical issue where an attribute is accessed without being initialized, which could lead to a crash. My review includes a suggestion to fix this.
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
github-actions
bot
added
the
ready
…roject#30340) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Ubuntu <mjtaheri68@gmail.com>
…roject#30340) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: dsuhinin <suhinin.dmitriy@gmail.com>
3 participants
Adds Eagle and Eagle3 support to all Transformers modeling backend classes by adding suport to
Base.Depends on:
torch.autocastif it will have an effect huggingface/transformers#42747