Conversation
|
@qubvel @Rocketknight1 Could you help review this PR ? Let me know if you have any suggestions. Thank you in advance for your time and assistance. |
qubvel
left a comment
qubvel
left a comment
There was a problem hiding this comment.
Hi @AlanPonnachan! Thanks for working on the model 🤗
Please see other model implementations in the repo to follow code style and patterns (e.g. how the attention module should be implemented).
It's better to reuse existing blocks rather than defining new ones. You can also utilize modular converter for inheritance.
Please see similar PRs:
Thanks!
| @@ -0,0 +1,225 @@ | |||
| # coding=utf-8 | |||
There was a problem hiding this comment.
Please refactor it to follow mllama model conversion format. It should be a KEY_MAPPING dict instead of create_rename_keys
| return image | ||
|
|
||
|
|
||
| @torch.no_grad() |
There was a problem hiding this comment.
Do we need no_grad here? The one is used in the code below
| ) | ||
| query, key, value = qkv.unbind(0) | ||
|
|
||
| context_layer = F.scaled_dot_product_attention(query, key, value, attn_mask=mask) |
There was a problem hiding this comment.
Please see other models on how the attention module should be structured.
|
@AlanPonnachan Are you still working on this and if you don't have the bandwidth then I can take it from here and refine the code. |
|
@yaswanth19 you can try to do modular converter part. I am facing some issues on that part. Further I can help you |
3 participants
What does this PR do?
Fixes #35351
This PR adds AIMv2 support in Transformers. AIMv2 showed better performance than SigLIP.
TODO
Who can review?
@qubvel
@Rocketknight1