From cea9694f6a51914c2394038bc0c5fec2cdd77cc8 Mon Sep 17 00:00:00 2001 From: mandy-li Date: Tue, 16 Jan 2024 10:40:27 -0800 Subject: [PATCH] Add use_flash_attention to Llama2-70B finetuning command in README --- examples/language-modeling/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/examples/language-modeling/README.md b/examples/language-modeling/README.md index ca947f30ef..909593427d 100644 --- a/examples/language-modeling/README.md +++ b/examples/language-modeling/README.md @@ -549,7 +549,8 @@ python3 ../gaudi_spawn.py --use_deepspeed --world_size 8 run_lora_clm.py \ --throughput_warmup_steps 3 \ --lora_rank 4 \ --lora_target_modules "q_proj" "v_proj" "k_proj" "o_proj" \ - --validation_split_percentage 4 + --validation_split_percentage 4 \ + --use_flash_attention True ``` - Multi-card finetuning of Falcon-180B: