Skip to content

edit mixtral quantization config file (#114)#1739

Merged
regisss merged 1 commit into
huggingface:mainfrom
HabanaAI:auto-pr-0efa7fe
Feb 5, 2025
Merged

edit mixtral quantization config file (#114)#1739
regisss merged 1 commit into
huggingface:mainfrom
HabanaAI:auto-pr-0efa7fe

Conversation

@dudilester
Copy link
Copy Markdown
Contributor

No description provided.

@dudilester dudilester requested a review from regisss as a code owner February 3, 2025 14:51
@libinta libinta changed the title [SW-215893] edit mixtral quantization config file (#114) edit mixtral quantization config file (#114) Feb 3, 2025
@libinta
Copy link
Copy Markdown
Collaborator

libinta commented Feb 3, 2025

@regisss let's hold this until I get accuracy from team

@dudilester
Copy link
Copy Markdown
Contributor Author

@libinta , @regisss this PR corrects the quant_config context, as the old version has invalid regular-expression strings that are not supported, as well the use of both allow-list and block-list which is invalid. the new version quantize the exact same layers as the old file did, just using correct syntax and logic, so practically these 2 files should result with the same model quantization from user perspective. Regarding the QA testing of the new config file, it was done with the new fp8 dynamic moe code which is not yet upstreamed but should be in this release (1.20).
However even if the new fp8 dynamic moe changes are not upstreamed and we remain with old static moe code this config file should be upstreamed as the old is incorrect.

@libinta libinta added the run-test Run CI for PRs from external contributors label Feb 4, 2025
@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Copy Markdown
Collaborator

@regisss regisss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@regisss regisss merged commit df43900 into huggingface:main Feb 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-test Run CI for PRs from external contributors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants