Skip to content

Update Mixtral-8x7B fp8 hqt example#756

Merged
regisss merged 1 commit into
huggingface:mainfrom
jychen21:update-readme-fp8-mixtral-8x7b
Mar 5, 2024
Merged

Update Mixtral-8x7B fp8 hqt example#756
regisss merged 1 commit into
huggingface:mainfrom
jychen21:update-readme-fp8-mixtral-8x7b

Conversation

@jychen21
Copy link
Copy Markdown

@jychen21 jychen21 commented Mar 4, 2024

What does this PR do?

Update fp8 hqt example of mixtral-8x7b (1x) to README

Test with bs16_output2048 on 1 card:
Input/outputs:
input 1: ('DeepSpeed is a machine learning framework',)
output 1: ('DeepSpeed is a machine learning framework that enables training of large models on a single machine with a single GPU. It is designed to be easy to use and efficient, and it can be used to train models on a variety of tasks.\n\n## ...

input 2: ('He is working on',)
output 1: ("He is working on a new album, which is expected to be released in 2019.\n\n## ...

input 3: ('He has a',)
output 1: ('He has a new book out, and he’s on a book tour.\n\n ...

input 4: ('He got all',)
output 1: ('He got all the way to the top of the mountain, but he didn’t know what to do when he got there.\n\n ...

input 5: ('Everyone is happy and I can',)
output 1: ('Everyone is happy and I can’t stop smiling.\n\n ...

input 6: ('The new movie that got Oscar this year',)
output 1: ('The new movie that got Oscar this year, “The Shape of Water” is a fantasy drama film directed by Guillermo del Toro and written by del Toro and Vanessa Taylor. It stars Sally Hawkins, Michael Shannon, Richard Jenkins, Doug Jones, Michael Stuhlbarg, and Octavia Spencer. Set in Baltimore, Maryland, in 1962, the story follows a mute custodian at a high-security government laboratory who befriends a captured humanoid amphibian creature.\n\n ...

...

input 15: ('In the far far distance from our galaxy,',)
output 1: ('In the far far distance from our galaxy, there is a planet called “Earth”. It is a planet that is full of life and is the home of many different species. One of these species is called “Humans”. Humans are a very intelligent species and are the most advanced species on the planet. They have created many different technologies that have helped them to survive and thrive on the planet.\n\n ...

input 16: ('Peace is the only way',)
output 1: ('Peace is the only way to end the war in Syria.\n\n ...

Throughput (including tokenization) = 645.77 tokens/second

@jychen21 jychen21 requested a review from regisss as a code owner March 4, 2024 05:29
@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@regisss regisss added the run-test Run CI for PRs from external contributors label Mar 5, 2024
Copy link
Copy Markdown
Collaborator

@regisss regisss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@regisss regisss merged commit 7df8006 into huggingface:main Mar 5, 2024
puneeshkhanna pushed a commit to puneeshkhanna/optimum-habana-fork that referenced this pull request Mar 11, 2024
HolyFalafel pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Mar 11, 2024
gplutop7 pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Oct 15, 2025
Co-authored-by: Iman Gohari <s.m.iman.gohari@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-test Run CI for PRs from external contributors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants