support for quantization and qlora models?

[thistleknot]

would love to use this with quantization and adapters

[Miraclemarvel55]

yes quite need. that is amazing

[Gregory-Ledray]

I am not a maintainer.

This is a duplicate of #744

[LopezGG]

I am using vllm with qlora. I merge the adapters with

model_for_merge = AutoModelForCausalLM.from_pretrained(
        base_model_path_or_name,
        torch_dtype=torch.float16,
    )
    full_model = PeftModel.from_pretrained(model_for_merge,
                                          model_id=script_args.adapter_dir,
                                        )
    full_model = full_model.base_model.merge_and_unload()  
    full_model.save_pretrained(script_args.full_model_dir)
    tokenizer.save_pretrained(script_args.full_model_dir)

and point vllm to this dir. Given that the A & B matrices are added to original weight matrices, there is no change in dimension or architecture. So, vllm can consume it. I would like a way to pass the model directly after load without saving it though

[hmellor]

Closing in favour of #3225 because quantization and LoRA are both supported. We just need QLoRA support.