Adding additional tokens to vocabulary

[adibMosharrof]

I am using models like EleutherAI/gpt-j-6B and llama-7b-hf for text generation.
I have added special tokens to the vocabulary as I want a structured output.

Prompt
"<|begincontext|>I want to make a restaurant reservation for 2 people at half past 11 in the morning.<|endcontext|>",
Target
"<|begintarget|><|begindsts|><|begindst|><|beginintent|>FindRestaurants<|endintent|><|beginbelief|><|endbelief|><|enddst|><|enddsts|><|beginuseraction|>INFORM_INTENT->Restaurants^intent~FindRestaurants<|enduseraction|><|beginaction|>REQUEST->Restaurants^city~<|endaction|><|beginresponse|>Do you have a specific which you want the eating place to be located at?<|endresponse|><|endtarget|>"

I have an example Colab Notebook
https://colab.research.google.com/drive/16qKy92cGoNPWrlQ4zlvntVGeSgjrknVF?usp=sharing

I am able to train the model without any errors.
However, when I perform inference, it does not produce any structured output, it just produces some random generation.

Here is a sample generation
<|endintent|> I\'ll make the reservation for 6 o"clock in the evening, for two people. I\'ll make the reservation for 6 o"clock in the evening, for two people. I\'ll make the reservation for 6 o"clock in the evening, for two people.

In my original code, when I train on a lot of data and plot the train/eval loss I can see that the train/eval loss decreases to values, train_loss=0.2163, eval_loss = 0.2416. With such low loss values, I am surprised why the generation has absolutely no structure. With a GPT-2 model, training for a few steps with a small amount of data produces a structured output.

This issue #326 talks about additional tokens in the vocabulary, which is similar to what I want to do.

Can you please give me some pointers on where I am going wrong.

[pacman100]

Hello, during full finetuning, the embedding layer with additional tokens is also trained which is not the case when using PEFT LoRA as per the code you shared. I think this might work if you also train only the embedding layers along with LoRa layers. To do that, specify modules_to_save in LoraConfig like below for gpt-j:

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=target_modules,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    base_model_name_or_path=model_name,
   modules_to_save=["wte"]
)

Also, use the main branch of PEFT

[adibMosharrof]

thank you very much for your response. I will make the changes you suggested and get back to you.

[SOCSChamp]

Hello, during full finetuning, the embedding layer with additional tokens is also trained which is not the case when using PEFT LoRA as per the code you shared. I think this might work if you also train only the embedding layers along with LoRa layers. To do that, specify modules_to_save in LoraConfig like below for gpt-j:

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=target_modules,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    base_model_name_or_path=model_name,
   modules_to_save=["wte"]
)

Also, use the main branch of PEFT

As I mention in this issue:

##349 (comment)

Adding the embedding layer on any of the transformer models I've tried gives me a float error. Adding it to the LoRA target argument gives me an attribute error for bias. Has anyone actually tested using this out of the box, or am I missing something?

[iMountTai]

I have a same error.

[pacman100]

#349 (comment)

[pacman100]

Hello @adibMosharrof, see this comment please: #337 (comment)

[adibMosharrof]

@pacman100 I had tried using what u suggested before but that did not seem to work.
I will try this new suggestion.
P.s. I have been traveling, so I am sorry for not being responsive.

[adibMosharrof]

@pacman100 Thank you very much for looking into this issue.

In your example notebook, I can see that you no longer use
model = prepare_model_for_int8_training(model)

Initially I had that in my code, and I was having the same issues as before, but once I removed that I was getting better generation after training. Could you please explain why this is happening.

In my actual training script, I have a validation set which is similar to the csv file I shared.
I train for more epochs with a bigger batch size by using grad accumulation.
However, I see that the eval loss becomes nan.

All these concepts are quite new to me, so I really dont understand how to solve this.

[adibMosharrof]

@pacman100
I ran the code from the notebook you shared in #337.

The only changes I made was to load the model in 8bit

`model_name = "EleutherAI/gpt-j-6B"

model = AutoModelForCausalLM.from_pretrained(
model_name,
load_in_8bit=True,
device_map="auto"
)
`
However, in the 2nd logging step, I see that my loss becomes 0.0.

`
{'loss': 4.8516, 'learning_rate': 2.5e-05, 'epoch': 0.04}

{'loss': 0.0, 'learning_rate': 5e-05, 'epoch': 0.08}

{'loss': 0.0, 'learning_rate': 7.5e-05, 'epoch': 0.12}

{'loss': 0.0, 'learning_rate': 0.0001, 'epoch': 0.16}

16%|█████████████████████████████████▊
| 20/123 [00:58<04:34, 2.66s/it]
`

Below is the output of generation

<|begincontext|><|user|>I am feeling hungry so I would like to find a place to eat.<|system|>Do you have a specific which you want the eating place to be located at?<|user|>I would like for it to be in San Jose.<|system|>Is there a specific cuisine type you enjoy, such as Mexican, Italian or something else?<|beginlastuserutterance|>I usually like eating the American type of food.<|endlastuserutterance|><|endcontext|>!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

`peft: 0.4.0.dev0 4fd374e

bitsandbytes = 0.38.1
`

I am confused why I am getting different results. It would be great if you could shed some light on this.

[adibMosharrof]

I was able to make things work and all my requirements have been fulfilled.
@pacman100 Thank you very much for supporting me with this issue. I am really grateful for the effort you put in.

A running code example can be found in the notebook I had shared initially

https://colab.research.google.com/drive/16qKy92cGoNPWrlQ4zlvntVGeSgjrknVF?usp=sharing#scrollTo=tpfeUu0NKQRs

I would like to share a few things I stumbled upon so that others dont face the same issue.

8-bit Training

If you want to train in 8 bit, you actually need the line below
model = prepare_model_for_int8_training(model)

{'eval_loss': 11.521041870117188, 'eval_runtime': 300.9554, 'eval_samples_per_second': 0.984, 'eval_steps_per_second': 0.083, 'epoch': 1.71}
{'eval_loss': 10.17847728729248, 'eval_runtime': 300.8959, 'eval_samples_per_second': 0.984, 'eval_steps_per_second': 0.083, 'epoch': 3.42}
{'eval_loss': 7.587922096252441, 'eval_runtime': 301.4311, 'eval_samples_per_second': 0.982, 'eval_steps_per_second': 0.083, 'epoch': 5.13}
{'eval_loss': 2.641119956970215, 'eval_runtime': 302.047, 'eval_samples_per_second': 0.98, 'eval_steps_per_second': 0.083, 'epoch': 6.84}

Bits And Bytes Version

I had to revert to bitsandbytes=0.37.2 as well.

In version 0.38.1, I would get out of memory exception when I called
model.save_pretrained(training_args.output_dir)

Additional tokens

Since I increased my vocabulary size by adding additional tokens, I had to add to the modules_to_save option in LoraConfig.

For Gpt-J model, I had to use modules_to_save = ["lm_head", "wte"] and for Llama, facebook opt I had to use modules_to_save = ["lm_head", "embed_tokens"]

Cuda Call errors

Another caveat I found, is that you cannot call model.cuda() as it messes things up internally somehow.
I load the model in 8 bit and it is loaded in the gpu.