Enable Llama2 70B to run with hqt on single card (#50) by HolyFalafel · Pull Request #762 · huggingface/optimum-habana

HolyFalafel · 2024-03-05T11:24:15Z

Add disk_offload flag that controls device_map=auto. Setting this flag enbales weights offload to disk when cpu memory runs OOM.
Add const serialization path flag that gets a path for where to serialize const sections, so if there is no space on device to save all const sections they will be offloaded to disk.

What does this PR do?

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

Add disk_offload flag that controls device_map=auto. Setting this flag enbales weights offload to disk when cpu memory runs OOM. Add const serialization path flag that gets a path for where to serialize const sections, so if there is no space on device to save all const sections they will be offloaded to disk.

HuggingFaceDocBuilderDev · 2024-03-05T11:57:22Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

libinta

@HolyFalafel can you provide a cmd in readme for run with hqt on single card?

libinta

Please add how to run hqt on single card in README

regisss

Agree with Libin, please provide an example.

Can you also rebase your branch on main and run the following please?

pip install -U ruff
make style

…ce#679)

Co-authored-by: regisss <15324346+regisss@users.noreply.github.com>

…ace#675)

….0 (huggingface#699)

Co-authored-by: Sayantan Sarkar <sasarkar@habana.ai> Co-authored-by: Libin Tang <litang@habana.ai> Co-authored-by: Jimin Ha <jha@habana.ai> Co-authored-by: Yeonsil Yoon <yyoon@habana.ai> Co-authored-by: Sayantan Sarkar <supersarkar@gmail.com>

…ace#705)

This reverts commit ddb4c02.

…uggingface#736)

Co-authored-by: Akihiro Takahashi <akihiro.takahashi@intel.com> Co-authored-by: Iman Gohari <s.m.iman.gohari@intel.com> Co-authored-by: Libin Tang <litang@habana.ai>

Co-authored-by: ZhaiFeiyue <80079571+ZhaiFeiyue@users.noreply.github.com> Co-authored-by: Yaser Afshar <ya.afshar@gmail.com> Co-authored-by: regisss <15324346+regisss@users.noreply.github.com>

…ace#704) Co-authored-by: regisss <15324346+regisss@users.noreply.github.com> Co-authored-by: edward.mascarenhas <emascare@gaudi-user-hf-1.amr.corp.intel.com>

…e#755) Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

Co-authored-by: regisss <15324346+regisss@users.noreply.github.com>

Fix Llama-70B-FSDP model loading issue (#63)

Expose Llama Fused OPs control from run_lora_clm.py (#23) * Expose Llama Fused OPs control from run_lora_clm.py * Update as per review comments Co-authored-by: Vivek Goel <vgoel@habana.ai>

) Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

HolyFalafel requested a review from regisss as a code owner March 5, 2024 11:24

libinta reviewed Mar 6, 2024

View reviewed changes

Comment thread examples/text-generation/utils.py

libinta added the synapse 1.15 label Mar 6, 2024

libinta reviewed Mar 7, 2024

View reviewed changes

regisss reviewed Mar 11, 2024

View reviewed changes

nngokhale and others added 23 commits March 11, 2024 11:19

Add ControlNet Pipeline (huggingface#585)

23da729

Fix tests (huggingface#669)

6680add

Add Flan T5 to model table (huggingface#677)

dbee23b

Update dev version (huggingface#678)

0014f9d

Change capture logic for HPU graphs in Diffusers pipelines (huggingfa…

dd22d63

…ce#679)

Update llama-7b command to include eval (huggingface#665)

48840fb

Co-authored-by: regisss <15324346+regisss@users.noreply.github.com>

Added check if reuse_cache supported by selected model_type (huggingf…

4f8b2d2

…ace#675)

Update examples (huggingface#682)

707803f

Add FSDP test to Gaudi2 CI (huggingface#683)

274d459

Bring back workaround for Falcon with SynapseAI 1.13 (huggingface#685)

25de760

Update examples (huggingface#686)

8850c37

Fix version check when console output is disabled (huggingface#688)

7ae066e

Patch for Gaudi Text-Generation Pipeline (huggingface#690)

2332b87

pt22 roberta fix (huggingface#701)

ade04fc

Add PPO and stack_llama support (huggingface#615)

0743a19

Updated requirements for image-classification samples: datasets>=2.14…

e67d875

….0 (huggingface#699)

Update example diff file for image classification (huggingface#703)

1b1d162

Add instruction in README to checkout latest stable release (huggingf…

1ccbc4b

…ace#705)

Update diff file (huggingface#706)

cc76cc7

To fix LLAMA-V2-70B-FT-HF (8x) for eager mode (huggingface#709)

f28bd06

Fix Llama initialization

91156c5

Revert "Fix Llama initialization"

65f2d0b

This reverts commit ddb4c02.

yeonsily and others added 21 commits March 11, 2024 11:19

Adding a flag whether to save checkpoint or not in run_lora_clm.py (h…

d4ac144

…uggingface#736)

Refactor and update CI for encoder-decoders (huggingface#742)

29a3687

SDXL fine tuning (huggingface#667)

1facf71

Co-authored-by: Akihiro Takahashi <akihiro.takahashi@intel.com> Co-authored-by: Iman Gohari <s.m.iman.gohari@intel.com> Co-authored-by: Libin Tang <litang@habana.ai>

Make style for ruff v0.3

9df7a87

Enable Mixtral-8x7B (huggingface#739)

17d5c71

Co-authored-by: ZhaiFeiyue <80079571+ZhaiFeiyue@users.noreply.github.com> Co-authored-by: Yaser Afshar <ya.afshar@gmail.com> Co-authored-by: regisss <15324346+regisss@users.noreply.github.com>

Support speech recognition with whisper models and seq2seq. (huggingf…

5699b75

…ace#704) Co-authored-by: regisss <15324346+regisss@users.noreply.github.com> Co-authored-by: edward.mascarenhas <emascare@gaudi-user-hf-1.amr.corp.intel.com>

Update example diff files

0fdbd76

Fix throughput calculation for diffusion models (huggingface#715)

44d5572

Fix command in speech recognition example README

5748277

Update CI baselines (huggingface#749)

40e3262

Fix crash in llama mode in llava image-to-text generation (huggingfac…

ea71b50

…e#755) Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

Update comment added in huggingface#507

d2b8cd0

Update Mixtral-8x7B fp8 hqt example (huggingface#756)

98cb8e5

Controlnet training (huggingface#650)

2e0e3ce

Co-authored-by: regisss <15324346+regisss@users.noreply.github.com>

Fix get_dtype and convert_into_dtypes (huggingface#769)

3cfd71f

Override sdpa option in Gaudi (huggingface#771)

2f18086

Fix Llama-70B-FSDP model loading issue (huggingface#752)

e6ccd26

Fix Llama-70B-FSDP model loading issue (#63)

Expose Llama Fused OPs control from run_lora_clm.py (huggingface#751)

08512d0

Expose Llama Fused OPs control from run_lora_clm.py (#23) * Expose Llama Fused OPs control from run_lora_clm.py * Update as per review comments Co-authored-by: Vivek Goel <vgoel@habana.ai>

Add image-to-text and visual question answering example (huggingface#738

125a6b5

) Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

Fixing tests by making static_shapes False (huggingface#778)

569b4d5

stylize code

2fdc24e

HolyFalafel requested review from ZhaiFeiyue, bhargaveede, hlahkar, mandy-li, ssarkar2 and vivekgoe as code owners March 11, 2024 09:21

HolyFalafel requested a review from a user March 11, 2024 09:21

HolyFalafel mentioned this pull request Mar 11, 2024

Enable Llama2 70B to run with hqt on single card (#50) #780

Merged

HolyFalafel closed this Mar 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Llama2 70B to run with hqt on single card (#50)#762

Enable Llama2 70B to run with hqt on single card (#50)#762
HolyFalafel wants to merge 63 commits into
huggingface:mainfrom
HabanaAI:dev/dsemiat/llama_single_card_pre_1.10.0

HolyFalafel commented Mar 5, 2024

Uh oh!

HuggingFaceDocBuilderDev commented Mar 5, 2024

Uh oh!

libinta left a comment

Uh oh!

Uh oh!

libinta left a comment

Uh oh!

regisss left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

HolyFalafel commented Mar 5, 2024

What does this PR do?

Before submitting

Uh oh!

HuggingFaceDocBuilderDev commented Mar 5, 2024

Uh oh!

libinta left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

libinta left a comment

Choose a reason for hiding this comment

Uh oh!

regisss left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants