Packing in SFT

[wdykas]

I understand how packing is allowed in pretraining but I was looking for some clarification on how we are allowed to pack samples for SFT with ConstantLengthDataset. I see that an EOS token is put between samples https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L567 but how are attention masks handled to make sure that the samples don't attend to each others context or is just putting EOS enough signal?

Could you also point me to the code for how multiple answers are handled in generation for loss calculations?

[younesbelkada]

Hi @wdykas !

Packing is a common practice and a trick to enable pre-training / fine-tuning on more sequences. I think that adding the EOS token is an enough signal for the model.

The trick seems to has been introduced in the T5 paper: https://arxiv.org/pdf/1910.10683.pdf

Could you also point me to the code for how multiple answers are handled in generation for loss calculations?

Can you elaborate more on this question? for loss calculation we do not perform generation we only compute the next token once.

[wdykas]

Sorry I meant more along the lines of how can calculate loss on the tokens without cross contamination in attention between sequences? Is there code where you create special attention masks? This seemed to be an issue opened here: huggingface/transformers#25452

[lvwerra]

Since the sequences are generally not correlated there is no issue in cross contamination. Otherwise you would need to pass a dedicated attention mask but this is not currently done.

[wdykas]

thanks for the answer! One last QQ, a problem that I see in this implementation is that we create a long continuous buffer of text separated by [EOS]. Then it seems we create examples of seq_len by naively looping over https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L579C35-L579C48 without considering EOS.
Doesn't this mean that some examples, if not many, will start with sequences that are cutoff and are in the middle?

[lvwerra]

Yes, that's the idea of packing. If you want to avoid that you can disable packing and each sequence will start at the beginning and filled with padding tokens to the max sequence length in the batch.

[yuanenming]

I think it is better to add a feature to make the sequence packing without any kinds of cross contamination like the figure above. Specifically in the supervised finetuning stage. Because in the SFT stage, We usually have short conversations (e.g. << 4096), which may cause some "training-inference gap" if we don't add the attention mask.

Notice that there is an implicit affect of the very long sequence attention (if we do not add the attention mask) -- the entropy [1].

[1] https://kexue.fm/archives/8823

[minhbtrc]

I think it is better to add a feature to make the sequence packing without any kinds of cross contamination like the figure above. Specifically in the supervised finetuning stage. Because in the SFT stage, We usually have short conversations (e.g. << 4096), which may cause some "training-inference gap" if we don't add the attention mask.

Notice that there is an implicit affect of the very long sequence attention (if we do not add the attention mask) -- the entropy [1].

[1] https://kexue.fm/archives/8823

Thank you for the answer!

[Qubitium]

Since the sequences are generally not correlated there is no issue in cross contamination. Otherwise you would need to pass a dedicated attention mask but this is not currently done.

I am are very confused on the current sft packing implementation.

My dataset is small, less than 1k rows, and very correlated.

As such, based on what I have gathered in this issue, the current packing code causes my full finetune training result to go off-the-rails in terms of quality vs packing false.

So my question is:

1. How do I manually tweak the attention mask as to fix the strong correlation issue between our sequences within a packed final output row? Is there an example code I can reference to?

2. How can the training quality survive cut-offs where some pre-packed sequences are split into end and beg of output example rows? Since the current code naively split into fixed sized final output.

Thanks.

[lvwerra]

Packing is mainly a throughput/efficiency technique at scale. For 1000 samples, why don't you just turn packing off?

[Qubitium]

Packing is mainly a throughput/efficiency technique at scale. For 1000 samples, why don't you just turn packing off?

Because gpu waste is waste at any scale. The idea of packing allows us to reduce small/micro training time to be completed in 1/2 of the time vs packing off. Imagine 1 minute training time (packing=on) vs 2 minutes (packing=off) but repeat that N times over the course of the day for a developer that is trying to maximize work/time efficiency. We have verified the training time diff for packing=on but result quality is an issue for us.

I don't think packing should be reserved for large data. For many cases, 1000 dataset for finetuning is actually overkill. Quality of data trumps dataset row count. So 1k may be tiny, but actually for a special laser focused training, it's quite large.

Just that the current implementations is not optimized for correlated/small datasets, and/or that we are not using it properly, thus my questions on the two points that is causing confusion/problem for us.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[hanyin88]

Thanks for sharing this very important topic.

Another likely very relevant consideration is that, in Huggingface's BetterTransformer (Fast Kernel in LLaMA-recipes) does not support attention_mask:

The PyTorch-native scaled_dot_product_attention operator can only dispatch to Flash Attention if no attention_mask is provided. Thus, by default in training mode, the BetterTransformer integration drops the mask support and can only be used for training that do not require a padding mask for batched training. BetterTransformer is not suited for the fine-tuning of models on tasks that requires a padding mask.