Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix the incorrect permutation of gguf #31788

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Fix the incorrect permutation of gguf #31788

wants to merge 6 commits into from

Conversation

PenutChen
Copy link

What does this PR do?

Fixes #31766. The permutation of q_proj and k_proj needs to consider both num_attention_heads and num_key_value_heads.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc

@LysandreJik
Copy link
Member

Thanks for your PR @PenutChen, we really appreciate it!

@SunMarc it'd be awesome if you can review this when you have a minute.

SunMarc
SunMarc reviewed Jul 4, 2024
View reviewed changes
Copy link
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the deep dive @PenutChen Really good job ! Can you add a few tests since it was not working with a bigger llama ! If possible, we can test this on a q4 model to not take too much memory !

src/transformers/modeling_gguf_pytorch_utils.py Outdated Show resolved Hide resolved
src/transformers/modeling_gguf_pytorch_utils.py Outdated Show resolved Hide resolved
src/transformers/modeling_gguf_pytorch_utils.py Outdated Show resolved Hide resolved
@PenutChen
Copy link
Author

@SunMarc Thanks for the advice! I noticed that there is already a Q4 Llama3 test. This update will change the expected text. Should I just update the expected text?

SunMarc
SunMarc approved these changes Jul 5, 2024
View reviewed changes
Copy link
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for iterating ! Could you are the perplexity of the model before and after for llama 3 ? If the llama 3 perpelxity was so high, it's a bit strange that it was able to pass the test / generate coherent text

@PenutChen
Copy link
Author

The perplexity of Llama3 Q4 is 1713.8865 before the fix and 6.4626 after the fix. The model comes from NousResearch/Meta-Llama-3-8B-GGUF.

They still produce reasonable output if the sequence length is short. This might be because only the q_proj and k_proj weights are abnormal, and their weights are relatively low. Other weights are robust to a certain extent, which could be the cause. This is just my guess.

@SunMarc
Copy link
Member

SunMarc commented Jul 5, 2024

Thanks for the confirmation !

@SunMarc SunMarc requested a review from LysandreJik July 5, 2024 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SunMarc SunMarc SunMarc approved these changes

@LysandreJik LysandreJik Awaiting requested review from LysandreJik

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

The conversion of the llama3 model back from gguf seems weird.
3 participants
@PenutChen @LysandreJik @SunMarc