Exploding gradient problem

[9-Time]

I have tried training on open image dataset but i kept getting exploding gradient with the bounding box after 1 epoch

Any ideas on how to fix them?

[qfgaohao]

@scarmaten have you tried smaller learning rates?

[9-Time]

yup i have tried it with learning rate all the way till 1e-20. still no luck.
The problem was partially solved when I removed Expand(mean), RandomSampleCrop(), RandomMirror() under the trainAugmentation in transformation.py but that would mean that the model's accuracy is much worse

[qfgaohao]

Hi @scarmaten , one possible way to debug is use a very small, verified dataset as both train and val data to train overfitted models. If you can get a overfitted model with 100% (or almost) accuracy, that means the architecture and system is fine, and the problem might be the data. Otherwise, there is something wrong in the system design.

[tamyiuchau]

I have also come across exploding gradient problem when trying to use mobilenet v2 ssd on Widerface. I am able to bisect the problem to the smooth L1 loss at

pytorch-ssd/vision/nn/multibox_loss.py

Line 45 in 1e86eb3

smooth_l1_loss = F.smooth_l1_loss(predicted_locations, gt_locations, size_average=False)

.
I suspect it is caused by no default boxes matched. After changing torch.nn.functional.smooth_l1_loss( reduction='sum'), problem is solved.

[tamyiuchau]

The question then arise, should I adjust the prior boxes for my specific purposes, or just let it go?(As it is kind of working now)