Failure in training HMR on Agora dataset

[dldaisy]

hello,
I preprocess agora dataset(3840*2160 / smpl image using agora data converter, and use it to train HMR model. For the adversarial training part, I first replace the cmu_mosh with pose/shape parameters extracted from the preprocessed agora npz file, but whether I use adversarial training or not, the loss collapse problem exists: I find the loss would suddenly become large(from ~250 - ~72000), as shown below in the screenshot.

Do you have any idea about why this happens? Thanks a lot.

[dldaisy]

The bug seems solved. Will update after training finished.