Upgrade to Transformers 4.34#475
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
| """ | ||
| cos, sin = self.cos_sin(seq_len, past_key_values_length, position_ids, query.device, query.dtype) | ||
|
|
||
| # Query and key's shapes are [bs * num_heads, seq_len, dim], might need manual expansion. Ifs and elses used to |
There was a problem hiding this comment.
@regisss I think we should discard this change. It leads to a significant performance drop. Also, it can handle cases where the expansion_factor > 1 even without this change
There was a problem hiding this comment.
@regisss From internal tests it turned out this is needed for some configurations. Without this they got graph compilation error. Please bring this change back.
There was a problem hiding this comment.
Done.
What about the performance drop? Was it when doing text generation or during training?
There was a problem hiding this comment.
@schoi-habana We have to remove it because the performance drop is quite important for both training and inference. I don't seen any compilation error with SynapseAI 1.12.0.
@libinta Maybe something to monitor to make sure everything works as expected for the release of 1.13?
regisss
requested review from
bhargaveede,
libinta,
mandy-li,
ssarkar2 and
vivekgoe
as code owners
4 participants
What does this PR do?
As per title.
Before submitting