Bayesian PEFT

This repo contains the code for our NeurIPS 2024 paper:
BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models
Yibin Wang*, Haizhou Shi*, Ligong Han, Dimitris Metaxas, Hao Wang
Thirty-eighth Conference on Neural Information Processing Systems, 2024
[Paper] [OpenReview] [Talk] [Slides] [Poster]

Important Note:

This repository is currently in its initial development stage. It contains the foundational code required to replicate the in-distribution experiments (Table 1 in the paper).

The complete version of this repository, including the implementation of LAP baseline and additional extensions, is scheduled to be uploaded by November 30, 2024.

To Run the Code

To install the required conda environment, run:

conda create --name <env>
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
pip install transformers datasets evaluate accelerate bitsandbytes jaxtyping torchmetrics setproctitle ipdb peft wandb nltk scikit-learn

Before you run the code, there are a couple of settings you might want to modify:

• wandb_entity: at utils/args.py line 139, change to your own wandb account;

The following command in the terminal could reproduce the basic results of in-distribution experiment:

bash scripts/blob/blob-llama-all.sh
bash scripts/blob/blob-roberta-all.sh

Note: The number of GPUs used for parallel training, the type of GPUs, and the model quantization settings can result in slight differences in the final performance.

References

BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models

@inproceedings{BLoB,
  title={BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models},
  author={Wang, Yibin and Shi, Haizhou and Han, Ligong and Metaxas, Dimitris and Wang, Hao},
  booktitle={Advances in Neural Information Processing Systems},
  year={2024}
}