Support for fine-tuning Llama 3.1 Omni.

[Jintao-Huang]

model: https://modelscope.cn/models/ICTNLP/Llama-3.1-8B-Omni

github: https://github.com/ictnlp/LLaMA-Omni

[Jintao-Huang]

ictnlp/LLaMA-Omni#1

[Jintao-Huang]

inference

CUDA_VISIBLE_DEVICES=0 swift infer --model_type llama3_1-8b-omni

sft

NPROC_PER_NODE=4 CUDA_VISIBLE_DEVICES=0,1,2,3 swift sft --model_type llama3_1-8b-omni --dataset aishell1-zh-mini#20000 --deepspeed default-zero2 --sft_type lora --output_dir output

[satheeshkola-532]

inference

CUDA_VISIBLE_DEVICES=0 swift infer --model_type llama3_1-8b-omni

sft

NPROC_PER_NODE=4 CUDA_VISIBLE_DEVICES=0,1,2,3 swift sft --model_type llama3_1-8b-omni --dataset aishell1-zh-mini#20000 --deepspeed default-zero2 --sft_type lora --output_dir output

hey @Jintao-Huang , i have fine tuned the llama omni by using the custom instruction dataset on another language using the above given cli command, it is generating the promising text responses in the desired language, but the audio generated is not supporting the desired lanugage(means vocoder isn't generating audio in fine tuned lanugae), so could you please check this issue once to completely fine tune the entire pipeline?

[fromlimbo]

@Jintao-Huang 你好，我在A800上面CUDA_VISIBLE_DEVICES=0是可以正常sft的，改成CUDA_VISIBLE_DEVICES=0,1多卡就会报错NotImplementedError: Cannot copy out of meta tensor; no data!
，请问默认模式下是不支持多卡吗？
CUDA_VISIBLE_DEVICES=0 swift sft \ --model_type llama3_1-8b-omni \ --model_id_or_path ICTNLP/Llama-3.1-8B-Omni \ --sft_type lora \ --dataset aishell1-zh-mini#5000

[SteinsHead]

inference

CUDA_VISIBLE_DEVICES=0 swift infer --model_type llama3_1-8b-omni

sft

NPROC_PER_NODE=4 CUDA_VISIBLE_DEVICES=0,1,2,3 swift sft --model_type llama3_1-8b-omni --dataset aishell1-zh-mini#20000 --deepspeed default-zero2 --sft_type lora --output_dir output

hey @Jintao-Huang , i have fine tuned the llama omni by using the custom instruction dataset on another language using the above given cli command, it is generating the promising text responses in the desired language, but the audio generated is not supporting the desired lanugage(means vocoder isn't generating audio in fine tuned lanugae), so could you please check this issue once to completely fine tune the entire pipeline?

Hello, have you solved the problem yet? We are also trying other languages and are facing the same issue. @satheeshkola-532 @Jintao-Huang

[ParikshitGehlaut]

Does anyone facing issue with hugging face accelerate version ? latest version is 1.0.1, But I getting error that data_seed requires Accelerate version accelerate >= 1.1.0.

[kadirnar]

@Jintao-Huang ,

I have never used the Swift library. I will try it for the first time. I want to train the Llama-Omni model from scratch. I want to use Llama3.3 70B and whisper large v3 model. Is this possible?