Qwen2.5 does not support Qlora?

[mathCrazyy]

I can't use qlora_qwen2_5_3b , and didn't find the qlora in lora_qwen2_5_0_5b, lora_qwen2_5_14b_base, lora_qwen2_5_14b_instruct, lora_qwen2_5_1_5b_base, lora_qwen2_5_1_5b_instruct, lora_qwen2_5_32b_base, lora_qwen2_5_32b_instruct, lora_qwen2_5_3b, lora_qwen2_5_72b_base, lora_qwen2_5_72b_instruct, lora_qwen2_5_7b_base, lora_qwen2_5_7b_instruct, qwen2_5_0_5b, qwen2_5_14b_base, qwen2_5_14b_instruct, qwen2_5_1_5b_base, qwen2_5_1_5b_instruct, qwen2_5_32b_base, qwen2_5_32b_instruct, qwen2_5_3b, qwen2_5_72b_base, qwen2_5_72b_instruct, qwen2_5_7b_base, qwen2_5_7b_instruct, qwen2_5_tokenizer,

[joecummings]

It definitely does support QLoRA, we just didn't add it b/c we didn't see an immediate use-case for quantizing models that were already so small. However, enabling it is as simple as the following code snippet:

# This will tell the builder to quantize the base model into NF4Tensors
qlora_qwen2_5_3b = partial(lora_qwen2_5_3b, quantize_base=True) 

And if you want to contribute this to the library, I'd be happy to look over a PR.

[felipemello1]

I think that in the config you can do

tune run <recipe> --config <my_config> model.quantize_base=True

I didn't test it, but i dont see why it wouldnt work

[ebsmothers]

As @felipemello1 mentioned, this is supported directly from the CLI by setting model.quantize_base=True when running any of the Qwen 2.5 recipes. So I am gonna mark this as resolved. @mathCrazyy please feel free to reopen if you encounter any issues