Cuda out of memory

[DmitriiSavin]

I'm encountering a strange error:
Fine-tuning on a large dataset occurs without problems and very good performance is achieved on val and test

But when fine-tuning on a smaller dataset on the 21st epoch, I get the following error when calculating the forest, namely on this line of code
https://github.com/Peterande/D-FINE/blob/master/src/misc/box_ops.py#L54

This error occurs
torch.OutOfMemoryError: CUDA out of memory

I have restarted training several times and the same error occurs every time at the same epoch on this dataset. There is no such problem with another dataset even using the same training parameters (learning rate, batch size, augmentations and so on)

Has anyone seen this problem before?